Implementation Strategy Fidelity and Organizational Outcomes in a Randomized Trial: Implications for Re-Entry from Corrections to Community Treatment

Abstract

Purpose.

This study examined fidelity of implementation strategies used in an organizational process improvement intervention (OPII) designed to improve evidence-based practices related to assessments for drug-involved clients exiting incarceration. Leadership was studied as a moderating factor between fidelity and outcomes.

Methods.

A mixed-method cluster randomized design was used to randomize 21 sites to early- or delayed-start within 9 research centers. Parent study protocol was reviewed to develop fidelity constructs (i.e., responsiveness, dose, quality, adherence). Outcomes were site-level success in achieving goals and objectives completed during the OPII (e.g., percent goals achieved). Correlations, analyses of covariance, regressions and moderation analyses were performed. Qualitative interviews assessed facilitators/barriers to implementation.

Results.

Fidelity constructs related to outcomes. No differences were found in fidelity by early or delayed condition. At low levels of leadership, high staff responsiveness (i.e., engagement in the OPII) related to poorer outcome.

Conclusions.

It is important to consider contextual factors (e.g., leadership) that may influence implementation strategy fidelity when deploying evidence-based practices. Findings are relevant to researchers, clinicians, administrators and policy makers, and suggest that goal completion during implementation of evidence-based practices requires monitoring of leadership competence, fidelity to implementation strategies (i.e., staff responsiveness to strategies) and attendance to goal importance.

Introduction

Implementation Science (IS) facilitating evidence-based practice (EBP): Justice settings

Substance use is a substantial problem during transitioning from incarceration to community (Carson & Sabol, 2012; Glaze & Bonczar, 2011; SAMHSA, 2012). Assessment and services during transition are critical to improve outcomes. Unfortunately, use of EBPs for assessment, case-planning and service delivery during transition is not widespread (Henderson et al., 2008, 2009). Successful outcomes may be enhanced by evidence-based assessment processes matching needs with services. IS involves understanding factors related to adoption, implementation and sustainment of EBPs (McGovern et al., 2013).

Many implementation theories exist promoting effective implementation, but there is much overlap among them, and variations in what each covers (see Damschroder et al., 2009). Consolidated Framework for Implementation Research (CFIR) offers an overarching typology to promote theory development, and verification regarding what works, and how, across multiple contexts (Damschroder et al., 2009). It has five domains: Intervention characteristics (e.g., complexity), outer/inner settings (e.g., economics/leadership), characteristics of individuals involved (e.g., years of experience), and implementation process. Implementation process has steps (i.e., strategies) to achieve EBP use, and central to this is carrying out implementation according to plan (i.e., fidelity). Domains interact in complex ways influencing EBP implementation, although little work has been done to understand this dynamic (Damschroder et al., 2009).

Fidelity

Fidelity of implementation can mean the degree to which program providers (i.e., teachers, therapists) implement EBPs as intended by program developers (Dusenbury et al., 2003). However, fidelity has had many definitions and uses, making it difficult to understand how fidelity is conceptualized and defined (Breitenstein et al., 2010; Toomey et al., 2020). Beyond fidelity of intervention as delivered by service providers, fidelity may be broadly conceptualized as the degree to which each stage of a protocol is deployed as intended, or where an assumption can be made about what actually happened during deployment (Prowse & Nagle, 2015). Such assumptions may include client understanding and use of information conveyed during provider use of an EBP (Toomey et al., 2020). In this sense, fidelity is a chain (Toomey et al., 2020), spanning provider implementation, client understanding/use, and systemic strategies supporting proper EBP use (e.g., training, supervision, policy, management support, etc). Because conceptualization of fidelity can be confusing, it is important to be pragmatic (Harvey et al., 2015), and define fidelity carefully for an intended purpose (Toomey et al., 2020).

One review (Slaughter et al., 2015) examined fidelity to implementation strategies based on Dane and Schneider’s (1998) seminal work identifying five elements important to fidelity: Adherence (strategies delivered as intended), dose (strategy exposure in terms of number, length, frequency), delivery quality (aspects not necessarily related to strategy content, such as attitudes of persons receiving strategies), responsiveness (recipient enthusiasm/engagement) and differentiation (a safeguard against receiving unspecified or contraindicated strategies); although differentiation is often ignored in the literature (Dusenbury et al., 2003; Elliott & Mihalic, 2004). Most research assessing fidelity focuses on fidelity of an EBP and not on fidelity of implementation strategies employed to encourage use of the EBP (Slaughter et al., 2015). EBP fidelity focuses on quality of clinical practice to influence client outcome, whereas fidelity to implementation strategies focuses on caliber of strategies to influence service provider behavior (Slaughter et al., 2015). The focus of the current study is on fidelity to implementation strategies, henceforth referred to as implementation fidelity. Reporting fidelity to implementation strategies allows assessment of the extent to which implementation success is influenced by strategies deployed (Slaughter et al., 2015). It facilitates selection of optimal strategies, better replication and therefore more effective transfer of EBPs into practice (Slaughter et al., 2015).

Organizational process improvement intervention (OPII)

The OPII protocol was designed to improve evidence-based practices in assessment: Instrumentation, case plan development, transfer of case plans to community treatment partners and use of case plans for drug-involved persons during community re-entry from corrections. OPII was part of the Criminal Justice Drug Abuse Treatment Studies-2 (CJDATS-2), a national collaborative, funded by the National Institute on Drug Abuse (Shafer et al., 2014). OPII may be considered a bundle, consisting of several implementation strategies (Powell et al., 2015), and central to these is facilitation of a Local Change Team (LCT), which includes a Champion.

Main outcomes indicated OPII resulted in improvements to the assessment process (use of valid instruments, individualized case plans, and service delivery) that were sustained over time (Welsh et al., 2015). Follow-up analyses found CFIR’s inner setting and staff characteristics (e.g., staffing levels, rehabilitation attitude) impacted staff perceptions of accomplishing goals and objectives set during OPII (Prendergast et al., 2017). However, fidelity to implementation strategies was not examined, nor was the interaction between this fidelity and other CFIR domains (e.g., staff characteristics, inner setting).

Purpose

This study investigated whether fidelity to OPII implementation strategies related to site-level success in achieving goals/objectives set by the LCT. Fidelity to implementation strategies was conceptualized in terms of adherence, dose, quality and responsiveness (see above). Contextual factors (e.g., leadership) were examined to determine whether these moderated the relationship between implementation strategy fidelity and organizational outcomes (i.e., meeting goals/objectives). Success in achieving goals/objectives is an important implementation outcome as it serves as a precondition for attaining changes in service and client outcomes (Proctor et al., 2011). Success should be based on stakeholder experience of the change (i.e., improved assessment process) to be implemented (Proctor et al., 2011). Qualitative data were used to explain quantitative results (Palinkas et al., 2011).

This study is important because: 1) Research detailing inter-professional efforts between systems to make change is rare, yet for substance-involved persons leaving corrections, such collaboration is crucial. 2) The study focuses on factors impacting use of evidence-based assessment in correctional settings; and these are settings in great need of EBPs (Henderson et al, 2008; 2009). 3) Studies on fidelity to implementation strategies (let alone contextual factors impacting fidelity) are novel, yet such studies are needed to better transfer evidence to practice. 4) This study provides a framework in which to place fidelity to implementation strategies using CFIR and provides a definition of implementation strategy fidelity (i.e., adherence, dose, etc.); this is relatively uncommon in the literature (see Slaughter et al., 2015). Employing such a framework stands to assist theory development and promote better understanding of systems change (Damschroder et al., 2009).

Methods

OPII was developed as part of CJDATS-2, a multi-site study involving nine research centers. Relevant OPII aspects are described below. Complete descriptions of OPII and CJDATS-2 are available elsewhere (Ducharme et al., 2013; Shafer et al., 2014).

OPII

The OPII bundle consisted of several implementation strategies (Powell et al., 2015) including formal commitments from corrections/community partners; academic partnerships; Local Change Team (LCT) comprised of corrections and community staff; ongoing expert coaching provided by an external Facilitator; a Champion (corrections LCT member dedicated to supporting the initiative); LCT meetings; needs assessment, conducted by the LCT, identifying barriers/facilitators and blue-print for change; and the LCT monitoring change efforts for effectiveness. Under Facilitator guidance, LCT and Champion were charged with implementing the EBP and were therefore central to implementation. Their activities were embedded in several phases with recommended duration as described below.

Pre-phase (1-2 months) - “Kick off” meeting was held (Facilitator/research center described the protocol to stakeholders and LCT formed). Needs Assessment (3-4 months) - Facilitator worked with each LCT to perform an organizational assessment determining quality of existing processes in four areas: Measurement (valid instruments), case plans (tailored to individual needs), conveyance and utility (data sharing between corrections/community settings), and service activation (treatment based on valid assessment). Process Improvement Plan (3-4 months) - LCT worked to identify and achieve specific goals with objectives in one or more of the four areas. Goals/objectives were specific to problems identified by each LCT and reflected priorities and available resources of local agencies. Therefore, there was considerable variation in goals/objectives across sites. Implementation (6 months) - Following plan approval by correctional agency executives, LCT took steps to accomplish goals/objectives. Follow-up/Sustainability (3 months) - Included an assessment regarding sustainability of both achieved goals/objectives, and any ongoing efforts to improve assessment practice.

Each LCT consisted of 6-12 individuals from the participating correctional agency and at least one community-based substance abuse treatment agency receiving referrals from the correctional agency. Staff members, including supervisors, were eligible to be on the LCT if they were responsible for assessment, case planning, referrals, and/or substance abuse treatment planning. Each LCT had a Champion. Persons were eligible to be a Champion if they had a direct line of communication to the chief executive officer of the correctional agency. The Champion served as the communication and decision-making pipeline with the corrections agency executive, and facilitated operational change processes identified by the LCT. Finally, each LCT was guided through OPII by a Facilitator, who served as a neutral mediator within the LCT. The Facilitator maintained communication among stakeholders, and acted as a liaison between LCT and research center. Facilitators were determined by research centers in consultation with correctional agencies. To encourage standardized procedures across sites, Facilitators attended a 1.5 day training before beginning OPII, and booster training (1.5 days) a year later. Each of nine research centers had at least two implementation sites; one had an additional delayed site (see Study Design), and one had two pairs of sites. Therefore, in total there were 21 LCTs.

Design

Parent study design details are published elsewhere (Prendergast et al., 2017; Shafer et al., 2014). In short, OPII was evaluated using a multi-site cluster randomized design. Cluster randomization involves randomization of groups (clusters) rather than individuals. Each of the nine research centers identified two or more participating sites. Sites consisted of a criminal justice agency and one or more associated community treatment providers. Within each research center, sites were randomly assigned (RAND function, Microsoft Excel) to early- (10 sites) or delayed-start condition (11 sites) by the CJDATS-2 Steering Committee Chair (for more detail, see Shafer et al., 2014). No sites were lost or excluded. When the early-start condition was at the end of Implementation phase, implementation of OPII was begun for the delayed-start condition. Time between starting early and delayed sites, on average, was 14 months. LCT’s were blind to condition. Data were collected 2010 – 2013.

Quality Control

Quality of OPII implementation strategies was monitored and maintained through mechanisms set forth by the OPII protocol including a) Facilitator experience; b) OPII manual (Shafer & Hiller, 2010) to guide Facilitator work with LCTs; c) routine Facilitator calls to review progress, address barriers, provide ongoing training; d) “end-of-phase” reports developed by LCTs and delivered to agency executives; e) tracking LCT meeting attendance; f) notes taken during LCT meetings; g) Facilitator “logs” describing the LCT’s next steps and milestones accomplished; h) monthly “implementation checklist” of specific activities to be conducted during each of the five phases; i) monthly LCT membership tracking including turnover; and j) monthly reports of time spent within each phase for each site. Researchers from across nine centers monitored implementation throughout the protocol (e.g., time to complete phase), and engaged in problem-solving with Facilitators as needed.

Measures

Implementation Strategy Fidelity

To assess adherence, dose, quality and responsiveness, data were reviewed for candidate indicators including LCT surveys, Facilitator attendance to coaching calls, percent phase reports completed and other activities. As an example, Working Alliance Inventory (WAI; Neale & Rosenheck, 1995), Goal Commitment (Klein et al., 2001), completion of Facilitator logs and number of LCT meetings were candidates to develop measures of quality, responsiveness, adherence and dose, respectively. A total of 24 candidate indicators were reviewed. Two senior authors independently categorized each candidate into adherence, dose, quality and responsiveness, and agreed on 70%. Disagreements were resolved through consensus with all authors. CJDATS2 assessed major outcomes at the end of Implementation phase; therefore survey data collected at or near this phase were utilized. See Table 1 for timing of measures.

Table 1.

Design and phased timing for sources of measures.

Early Start	Delayed Start
Phase	Phase
-Measurement Source	-Measurement Source
Kickoff Phase (1-2 mo)
-Leadership assessed
-Fidelity Checklist
Needs Assessment Phase (3-4 mo)
Planning Phase (3-4 mo)
-LCT Meetings
-Fidelity Checklist
-Goals (feasibility/ dedication)
Implementation Phase (6 mo)
-LCT Meetings
-Phase Report
-Executive Meeting
-Fidelity Checklist
-WAI
-Staff Satisfaction (burden/cohesion)
-Management Support
	Kickoff Phase (1-2 mo)
	-Leadership assessed
	-Fidelity Checklist
	Needs Assessment Phase (3-4 mo)
	Planning Phase (3-4 mo)
	-LCT Meetings
	-Fidelity Checklist
	-Goals (feasibility/ dedication)
	Implementation Phase (6 mo)
	-LCT Meetings
	-Phase Report
	-Executive Meeting
	-Fidelity Checklist
	-WAI
	-Staff Satisfaction (burden/ cohesion)
	-Management Support

Notes. WAI = Working Alliance Inventory; LCT = Local Change Team, mo = months. See text for detail on how each measures source was used.

Scores across indicators were transformed to z-scores. To determine which indicators to retain/discard, internal reliability analyses were examined within each fidelity construct. For each site, a total score was obtained for each fidelity construct by summing z-score values of respective fidelity indicators.

Responsiveness, was comprised of five indicators including (1) Goal Dedication and (2) Goal Feasibility (Klein et al., 2001); (3) Protocol Burden and (4) Protocol Cohesion (Knudsen et al, 2007); and (5) Perceived Management Support for protocol activities, which was created for the study. For survey measures see National Addiction & HIV Data Archive Program: www.icpsr.umich.edu/icpsrweb/NAHDAP/studies/35082. Again, each of the five indicators was transformed to a z-score, summed and internal reliability was obtained for the final scale (α = 0.68).

Adherence, included fidelity checklists, which were to be submitted at least monthly during Implementation phase. To the extent sites submitted forms more frequently than monthly, total number of forms exceeded number expected. Therefore, percent of checklists was allowed to exceed 100% to reflect more frequent use of the form. The resulting fidelity measure was comprised of six indicators: (1) percent fidelity checklists completed (M = 103.55, SD = 19.17), (2) number of weeks to complete Implementation phase (M = 25.29, SD = 6.36), (3) weeks to complete Implementation phase report (M = 25.33, SD = 16.19), (4) weeks to conduct meeting with correctional agency executives during Implementation phase (M = 21.24, SD = 16.63), (5) percent Facilitator Logs completed for LCT meetings (M = 75.40, SD = 35.48), and (6) percent of notes uploaded after LCT meetings (M = 59.41, SD = 41.64). Internal reliability for the final scale was α = 0.76.

Quality, included four indicators: (1) Bond with Facilitator and (2) Agreement with Facilitator on Tasks from WAI (Neale & Rosenheck, 1995); and (3) Champion stability (M = 0.90, SD = 1.14) and (4) LCT member stability (M = 4.71, SD = 4.06). Stability was coded as number added + number left. Internal reliability for the scale was α = 0.69. Dose consisted of a single indicator, (1) number of LCT meetings held.¹

Goals/Objectives

Each LCT produced an End of Implementation Report using standardized format with sections on process improvement goals, rationale for goals, extent to which goals were achieved, and barriers to implementation. For each report, 2-3 trained researchers (master- or doctoral-level) independently rated degree to which an LCT accomplished each goal and objective using a Likert scale anchored at 0 (not begun) and 6 (completed). A total of 69 goals and 169 objectives were rated across 21 LCTs. Raters, blind to condition, then participated in a consensus call to review ratings, discuss discordant ratings, and assign a consensus score for any discordant items. For reliability, Krippendorff’s Alpha (2013; α = 0.77) was computed on ratings conducted prior to consensus. Final ratings indicated the degree to which an LCT was successful in achieving its goals and objectives. Average number of goals met, average number of objectives met, percent goals met, and percent objectives met per site were utilized in analyses. A goal/objective was defined as being met if it was rated 4 (nearly completed) through 6 (completed); it was unmet if rated as 0 (not started) through 3 (moderately completed). Across sites, average number of goals met was 4.10 (SD = 1.46), average number of objectives met was 4.21 (SD = 1.50), average percent goals met was 59.33 (SD = 39.06), and average percent objectives met was 66.91 (SD = 29.76).

Covariates/Moderator

Covariates considered for analyses, measured at baseline, included: 1) Staff experiences of burnout and efficacy; and 2) stressful agency climate, and agency leadership (CFIR’s recipient characteristics and inner setting domains, respectively). Prior research indicates importance of these constructs in IS, and measures to assess them (see Broome et al., 2009; Lehman et al., 2012). Each of these constructs was based on reports from correctional and treatment agency staff involved in assessment processes. A single covariate was chosen given relatively high correlations with dependent variables and low correlations with independent variables across 21 sites. Leadership emerged as the best covariate across proposed analyses. To conserve space, detailed description and psychometrics are not provided on other potential covariates, but can be provided upon request. Given its relation to outcomes, leadership was also utilized as a moderator in outcomes analyses.

Leadership was assessed using 9-items (Broome et al., 2009) with ratings on a Likert scale from 1 (disagree strongly) to 5 (agree strongly). It is a global assessment of transformational (“Supervisor leads by example”) and transactional behaviors (“Supervisor gives special recognition when work is very good”), with lower and higher scores reflecting more poorly and more highly rated leadership, respectively. Staff (N = 1052) completed items in reference to their supervisor; and supervisors completed ratings on themselves. Staff-types completing the measure were 6.1% corrections directors, 7.6% treatment directors, 40.7% corrections officer and 45.6% treatment staff. Across sites, M = 35.46, SD = 3.20 and α = 0.94.

Analyses

Fidelity validity/stability.

Validity was examined by calculating correlations between fidelity constructs and goals and objectives, and correlations among fidelity constructs. To determine whether fidelity was relatively stable during implementation of OPII across study conditions (early vs. delayed), analyses of covariance (ANCOVA) was conducted with study condition as the independent variable, each fidelity construct as the dependent variable, and leadership as a covariate (again, site was unit of analyses).

Leadership moderation

Given the relationship between leadership and goals/objectives, moderator analyses were conducted to determine whether leadership moderated effects of fidelity on attainment of goals/objectives. Hayes’ (2018) SPSS PROCESS macro was used to implement path analysis. Each fidelity construct (responsiveness, adherence, quality, dose) was run in separate analyses for each dependent variable (M number of goals, M number of objectives, % goals, % objectives met). Fidelity construct and leadership were centered. The model included fidelity construct, leadership, and fidelity construct by leadership interaction. Following a significant fidelity construct by leadership interaction, conditional effects of fidelity on the dependent variable were examined at low (−1SD from the M), average (moderator M) and high (+1SD from the M) leadership levels.

Qualitative interviews

Structured qualitative interviews were conducted immediately following the Implementation phase. LCT members were individually asked, “What factors were helpful during implementation?” and “What factors were barriers?” Interviews were 15-25 minutes, audio-recorded, transcribed and coded for thematic content using Atlas.ti qualitative software. A researcher with over a decade of experience in collecting/analyzing qualitative data over-saw qualitative data collection/analyses. An open coding process was used, characterizing conceptual content. Emergent themes were regularly discussed with the CJDATS2 qualitative research core to confirm consistency. Following this initial round of coding, axial coding allowed for identification of relations among initial codes to develop and then code for emergent themes related to LCT experiences of implementation facilitators/barriers.

Multiple hypothesis tests

Again, analyses were conducted at the site level, given the thrust of the study (i.e., site-level fidelity). In addition, strict alpha adjustment was not employed due to the exploratory nature of the research, which deals with topics not yet studied at length and assists in developing theory (see Gaus et al., 2015; Kimmelman et al., 2014). Effect sizes and p-levels are provided to assist with interpretation of results (Hojat & Xu, 2004).

Results

As the current study does not present results of the randomized clinical trial (RCT), a CONSORT diagram is not presented. CONSORT diagram for and results of the RCT are presented elsewhere (Welsh et al., 2015). Descriptives for LCT respondents, Facilitators and staff asked to report on leadership are provided in Table 2. Facilitators had about 13 years of experience, LCT members had about 11 years of experience, and staff were in their current position for about 4 years.

Table 2.

Description of respondents.

	Local Change Team		Facilitators		Staff Returning Leader Evaluations
Characteristic	M, SD	%, N=251	M, SD	%, N=9	M, SD	%, N=1052
Gender
-Man	-	37.8	-	66.7	-	38.8
-Woman	-	61.0	-	33.3	-	55.4
-Mis/ Ref/ Oth	-	1.2		0.0	-	5.8
Age (years)	46.20, 10.42	-	45.33, 10.07	-	43.62, 11.01	-
Race
-White	-	73.3	-	88.9	-	63.1
-African Am	-	17.5	-	11.1	-	19.8
-Oth	-	5.2	-	0.0	-	7.4
-Mis/ Ref	-	4.0		0.0	-	9.7
Ethnicity
-Hispanic	-	6.0	-	11.1	-	7.7
-Non-Hispan	-	85.3	-	88.9	-	79.4
-Mis/ Ref/ Oth	-	8.7		0.0	-	12.9
Yrs Exper	10.62, 10.40	-	12.97, 8.93	-	-	-
Mo in Positn	-	-	-	-	50.71, 52.94	-
Degree
-MA+	-	33.1	-	66.6	-	32.9
-BA/ BS	-	39.0	-	0.0	-	45.0
-< BA/ BS	-	13.1	-	11.1	-	20.3
-Mis/ Ref/ Oth	-	14.8		22.3b	-	1.8
Discipline
-Addic Couns	-	34.7	-	11.1	-	37.8
-Psychol	-	3.2	-	11.1	-	3.8
-Admin	-	6.0	-	11.1	-	2.2
-CJ/ Law Enf	-	25.5	-	11.1	-	27.9
-SW/ Hum Sv	-	10.8	-	0.0	-	5.9
-Oth	-	9.2a	-	33.3c	-	18.5d
-Mis/ Ref	-	10.6		22.3b	-	3.9b

Notes. Mis/ Ref/ Oth = Missing/ Refused/ Other, Hispan = Hispanic, Yrs Exper = Years of Experience, Mo in Positn = Months in Current Position, MA+ = More than Masters, BA/ BS = Bachelor of Arts/ Bachelor of Science, < = Less than, Addic Couns = Addictions Counseling, Psychol = Psychology, Admin = Administration, CJ/ Law Enf = Criminal Justice/ Law Enforcement, SW/ Hum Sv = Social Work/ Human Services.

^aCase manager, director, re-entry counselor, other counselor, medicine, education.

^bContains only missing data.

^cCase manager, director, re-entry counselor.

^dCounseling, education, medicine, military, vocational rehabilitation.

The median correlation among implementation fidelity constructs was 0.19, and the median correlation was 0.23 between implementation fidelity constructs and measures of goals and objectives. Due to small sample and effect size, p-values were generally non-significant across correlations. ANCOVA models (Table 3) showed that implementation strategy fidelity did not vary as a function of randomization to study condition (early vs. delayed condition). Leadership was not found to moderate relationships between dose, quality, or adherence and LCT attainment of goals/objectives. Moderator analyses showed significant effects for the responsiveness by leadership interaction for both percent goals and percent objectives achieved (Table 4). At lower levels of leadership, higher responsiveness was significantly related to poorer outcomes in comparison to lower responsiveness, which related to better outcomes (Table 5 and Figure 1). It may be that relatively poor leadership in combination with a responsive LCT created (or was symptomatic of) conflict, which led to relatively fewer goals and objectives being achieved. Alternatively, it may be that under poorer leadership, easier goals and objectives were wisely chosen, requiring less responsiveness to accomplish. Qualitative data were used to explore these possibilities.

Table 3.

Analyses of covariance for study condition (early vs. delayed) predicting implementation strategy fidelity constructs while controlling for leadership.

DV	ME (SD)	MD (SD)	F(1,18)	p	d	95% CI of d
Responsiveness	.55 (3.15)	−.50 (3.70)	.026	.873	.32a	−.54; 1.19
Dose	.27 (1.21)	−.24 (.74)	.82	.378	.47a	−.40; 1.34
Quality	1.29 (2.58)	−1.18 (2.74)	3.73	.069	.85b	−.04; 1.75
Adherence	1.91 (3.75)	−1.76 (3.89)	3.08	.098	.93b	−.02; 1.88

Notes. DV = Dependent variable; E = Early condition; D = Delayed condition; M = Mean; SD = Standard deviation; p = p-value; d = Cohen’s d; CI = Confidence interval

^a= small effect size

^b= large effect size (Cohen, 1988).

Table 4.

Regression results examining leadership as a moderator of the relationship between implementation fidelity constructs and attainment of LCT goals and objectives.

DV = % Goals Achieved
IV	R 2	F[3,17](p)	ΔR2 (p)	IV b(p)	Leadership b (p)	IV*Leadership b(p)
Responsiveness	.44	4.48(.02)	.15(.045)a	−.04(.56)	.29(.27)	.15(.045)*
Dose	.24	1.77(.19)	.00(.91)d	.13(.56)	.45(.07)	−.02(.91)
Quality	.26	1.96(.16)	.03(.40)d	.04(.60)	.47(.04)	.09(.40)
Adherence**	.28	1.93(.17)	.02(.57)d	.04(.54)	.58(.13)	−.50(.57)
DV = Mean Goal Rating
IV	R 2	F[3,17](p)	ΔR2(p)	IV b(p)	Leadership b (p)	IV*Leadership b(p)
Responsiveness	.46	4.76(.01)	.10(.09)c	−.06(.39)	.42(.12)	.12(.09)
Dose	.35	2.99(.06)	.02(.44)d	.26(.23)	.51(.03)	−.15(.44)
Quality	.33	2.75(.08)	.02(.53)d	.09(.28)	.49(.03)	.06(.53)
Adherence**	.45	4.09(.03)	.10(.11)c	.07(.23)	.77(.03)	−.14(.11)
DV = % Objectives Achieved
IV	R 2	F[3,17](p)	ΔR2(p)	IV b(p)	Leadership b (p)	IV*Leadership b(p)
Responsiveness	.50	5.66(.01)	.16(.03)a	−.05(.42)	.34(.18)	.15(.03)*
Dose	.26	1.99(.15)	.00(.98)d	.10(.65)	.48(.05)	−.01(.98)
Quality	.29	2.29(.12)	.02(.46)d	.06(.43)	.48(.03)	.08(.46)
Adherence**	.32	2.40(.11)	.02(.52)d	.06(.33)	.57(.12)	−.06(.52)
DV = Mean Objective Rating
IV	R 2	F[3,17](p)	ΔR2(p)	IV b(p)	Leadership b(p)	IV*Leadership b(p)
Responsiveness	.46	4.85(.01)	.12(.06)b	−.07(.29)	.37(.16)	.13(.06)
Dose	.27	2.09(.14)	.02(.54)d	.20(.38)	.47(.05)	−.13(.54)
Quality	.26	1.94(.16)	.01(.57)d	.06(.46)	.46(.05)	.06(.57)
Adherence**	.34	2.56(.09)	.05(.32)c	.07(.27)	.61(.10)	−.09(.32)

Notes. LCT = local change team; IV = Independent variable; DV = dependent variable; ΔR² = change in R² due to interaction; All n = 21, except for Adherence due to missing data

^*See Table 6 and Figure 1 for significant interactions

^**for Adherence, degrees of freedom for overall F are (3,15) with n = 19.

Effect sizes are a = large, b = medium-large, c = medium, d = small (Nandy, 2012).

Table 5.

Conditional effects for responsiveness.

	DV = % Goals Achieved		DV = % Objectives Achieved
Leadership	b	p	b	p
Low	−.19	.03*	−.20	.01*
Moderate	−.04	.56	−.05	.42
High	.11	.33	.01	.35

Notes. DV = Dependent variable

^*p < .05.

Figure 1a.

Influence of responsiveness on % goals achieved at different levels of leadership.

To better understand the moderating effects of leadership, qualitative data were reviewed for factors facilitating and impeding implementation. In terms of facilitating factors, relationships emerged as a key theme. The following illustrates the importance of the leadership-LCT relationship:

“…the fact that the [CORRECTIONS AGENCY LEADER] was really engaged…and…interested in the outcome gave [THE LCT] the chutzpah whereby they could make these groups talk to them and work…through the…objective. In terms of other facilitative factors…[THE LCT LEADER]…was really a good choice…just the fact that [THE LCT] would get all this extra work…speaks a lot about their commitment…that level of commitment and professionalism was definitely important.”

In terms of barriers, a key theme was conflict. The second excerpt shows difficulties stemming from leadership and LCT interactions. The third illustrates how one LCT compensated in the context of leadership that did not support change:

“…just getting [CORRECTIONS AGENCY LEADER] to approve the Process Improvement Plan was excruciating…We initially presented it to him. He gave us some caveats that we had to address. We got back to him really quickly and then waited …and…prodded him as much as we could…so [THE LCT] did what they could officially to try to get the process going. But…we finally got a word from the [ADMINISTRATOR]. It didn’t come from the [CORRECTIONS AGENCY LEADER]…it just kind of like dropped from heaven one day…that was a huge delay.”

“…the changes that [THE LCT] made were very small…They didn't have dramatic impact…the changes that the team most wanted to make were impossible…because they were …mandated by…department rule or regulation. If the team had actually put in…goals that addressed some of those system-wide things, …they [WOULD HAVE BEEN] vetoed by the [CORRECTIONS AGENCY LEADERSHIP]…”

The first quote illustrates influence of engaged leadership on enthusiastic response of the LCT and importance of the Champion in accomplishing objectives. The second quote illustrates influence of disengaged leadership on a responsive LCT, the down-stream communication process, and adverse impact on goals. Finally, quote three articulates the tactic used by some LCTs to choose lesser goals when leadership prohibits (or cannot support) needed change within a hierarchical and bureaucratic system. This is consistent with empirical findings.

Discussion

Descriptive statistics indicated fidelity to implementation strategies was generally acceptable or better during the study. For example, percents of phase reports and executive meetings held were so high as to have no variation and so were removed from analyses (see footnote 1). In addition, there was relative stability in Facilitators and agencies. Also the percent of Facilitator calls on which the research center was represented, and that the Facilitator attended, was generally high. Separate indicators of fidelity to implementation strategies are important; however, combining these indicators into broad meaningful constructs (i.e., dose, adherence, etc.) facilitates further analyses of implementation strategy fidelity and contextual factors that can impact fidelity of implementation strategies.

Although these data suggested fidelity to implementation strategies, it is important that there be no variation by study condition. ANCOVA results indicated implementation strategy fidelity did not vary as a function of randomization to early or delayed site. This bolsters confidence in outcomes analyses presented elsewhere (Prendergast et al., 2017; Welsh et al., 2015). However, it is important to note effect sizes for quality and adherence were in the large range (Cohen, 1988), suggesting further investigation with a larger number of sites is appropriate. Correlational analyses provided some evidence for the validity of implementation fidelity constructs, given relatively small-medium effect sizes among them and somewhat larger effect sizes between implementation fidelity constructs and goals and objectives.

Leadership at multiple levels, including systems, organizations and work-groups, is key to creating a climate conducive to EBP implementation (Aarons et al., 2014). Therefore, it may not be surprising that under poor leadership (i.e., uninspiring leadership; few meaningful transactions with leadership), even when implementation strategies were carried out with fidelity (i.e., LCT responsiveness), little progress was made towards rolling out an EBP to improve assessment practices for persons released to the community.

In the context of poor leadership, it also may not be surprising that some LCTs, charged with making improvements, were rather unresponsive to OPII. What is more striking is that strides were made towards rolling out EBPs in this case. This may be explained by the hierarchical nature of criminal justice settings, which are often characterized by mechanistic operations with clear division of labor creating a narrow range of duties (Dias & Vaughn, 2006). In such circumstances, LCTs may rely on a narrower range of duties afforded to probation officers in order to fulfill even small goals so as to conform to executive directives to participate effectively in improving assessment practices. Perhaps with little inspiration to choose loftier goals, and without the means to accomplish them, less engaged LCTs wisely chose more manageable tasks.

Methods to track fidelity to implementation strategies are relatively rare, including contextual factors impacting fidelity, yet such studies are needed to better transfer evidence to practice (Slaughter et al., 2015). In addition, CFIR’s five domains (EBP characteristics, outer/inner settings, characteristics of persons involved in implementation, implementation strategies) dynamically interact in complex ways, though little work has been done to elucidate this dynamic (Damschroder et al., 2009). The current study illustrated how implementation strategies can be conceptualized and measured in terms of fidelity constructs (responsiveness, dose, adherence and quality). Compared to other fidelity constructs, responsiveness in relation to leadership was associated to outcome. The impact of leadership interacted in both expected (i.e., high responsiveness in the context of poor leadership related to poor goal accomplishment) and unexpected ways, (i.e., poor leadership and low responsiveness related to goal accomplishment). In this study, change agent engagement and receptivity to tasks (i.e., responsiveness) was relatively more important than other fidelity constructs including adherence (delivery as intended), dose (exposure in terms of number of meetings), and quality (aspects not necessarily related to strategy content, such team stability).

In the context of poor leadership, it is difficult to argue that change agents (i.e., LCTs) should be less responsive. Although replication is needed, policy implications for EBP implementation would be to prepare and maximize leadership (Aarons et al., 2015), and attend to change agent engagement (i.e., responsiveness) and their choice of goals. Given the current trial, it may be optimal to track responsiveness of change agents, leadership competence, and goal completion and importance throughout implementation, and intervene where needed (e.g., on leadership, and via Facilitator’s work with LCT and Champion).

Operationalizing and measuring responsiveness is challenging in development of effective protocols (Kutash et al., 2012). This study was able to operationalize and measure responsiveness, and link this construct to contextual factors (e.g., leadership) during implementation. Harvey and colleagues (2018) found Facilitator recruitment/selection/training, ability to apply knowledge/skills, mentorship from experienced Facilitators, ability to work collaboratively, and managerial support are critical to implementation outcome. These findings are consistent with the current study in that OPII attended expressly to Facilitator recruitment/selection/training, and regular mentorship. Managerial support for Facilitators was not specifically tracked; however, perceived managerial support of LCT activities with Facilitator was evaluated and integrated into fidelity measurement (i.e., responsiveness). In terms of working collaboratively, LCTs rated working alliance with Facilitator as part of fidelity (i.e., quality). Although somewhat different approaches were used to study fidelity to an implementation strategy using facilitation, results have similarities.

Limitations/Recommendations

LCT self-reports were included in creation of fidelity constructs, and as such, may be biased. However, inclusion of more objective indicators (e.g., number of LCT meetings held), somewhat mitigates this concern. Although internal consistencies were in the acceptable range (George & Mallory, 2003), alphas for fidelity constructs could be improved in the future, perhaps by having access to more candidate indicators. In addition, the measure of leadership included reports from several types of staff, minimizing over-reliance on LCT reports alone. Because N = 21 sites, analyses were limited. For example, nesting of staff within LCT, and LCT within research center could not be conducted, nor could factor analyses be conducted to develop fidelity constructs. Future studies may seek to utilize designs that allow for nesting and include more sites for increased power to detect statistically significant results with strict alpha adjustment and to conduct factor analyses. Due to small sample size, alpha was not corrected, but indicators of effect size are provided in tables to assist with interpretation of results (Hojat, & Xu, 2004).

Important organization-level outcomes were measured (meeting goals/objectives); however, client-level (i.e., drug use) outcomes were not measured. Future studies may seek to include client-level outcomes in addition to important organizational outcomes. Although organizational climate variables (i.e., leadership) are typically represented by aggregate staff ratings (Welsh et al., 2016), degree to which raters (correctional vs treatment staff) at a site were in agreement was not examined. Due to small sample size, the simplest model was used in analyses (a single leadership moderator).

Of note, organizations that participated were not randomly chosen. Future studies may wish to match sites on organizational characteristics such as size, resources, etc. Finally, as with all studies, replication is needed to determine if findings are robust, and larger sample size would improve significance testing in a fully powered trial. Future studies may also wish to use qualitative data to better understand unique aspects of inter-professional collaboration (during LCT meetings) that assisted or deterred outcomes.

Conclusions

It is possible to measure and achieve fidelity of implementation strategies intended to improve evidence-based assessment practice for substance-involved persons during community re-entry. Further, impact of leadership on organizational functioning is potentially very important (Aarons et al., 2016; Proctor, 2007). Even when staff members are more responsive in attempting to accomplish organizational goals, it appears that this does not compensate for impact of relatively weak leadership on actually realizing those goals.

Prior to engaging in organizational change, it behooves agencies to review leadership quality to determine how to optimize accomplishing goals and objectives. Leadership can act as a leverage point (Proctor, 2004). Preparing leaders for change may be a viable strategy to support organizations implementing EBPs (Aarons et al., 2015). Few, if any, studies directly examine the role of implementation strategy fidelity in the context of agency factors, such as leadership (CFIR’s inner context domain). Assessing fidelity of implementation strategies and contextual factors affecting this fidelity are important to settings installing evidence-based practices. This includes correctional treatment settings that wish to implement organizational processes for improving assessment and case planning for substance-involved clients as they transition to communities. In such scenarios, it behooves agencies to track responsiveness of change agents, leadership competence, and goal completion and goal importance throughout implementation, and intervene as needed.

Exploratory research deals with problems that have not yet been studied at length (Schwab & Held, 2020), which reflects the current study. Exploratory analysis identifies novel information, may include multiple comparisons, points out new research directions to prepare for more fully powered studies (Gaus et al., 2015), and assists in developing theory (Kimmelman et al., 2014). This study produced important and novel evidence that supports a fully powered study examining impact of implementation strategy fidelity in relation to contextual factors (see Kimmelman et al., 2014).

Figure 1b.

Influence of responsiveness on % objectives achieved at different levels of leadership.

Abbreviations in the text

ANCOVA

Analysis of Covariance

PROCESS

Conditional process macro

CFIR

Consolidated Framework for Implementation Research

CONSORT

Consolidated Standards of Reporting Trials

CJDATS-2

Criminal Justice-Drug Abuse Treatment Studies-2

EBP

Evidence-Based Practice

HIV

Human Immunodeficiency virus

IS

Implementation Science

LCT

Local Change Team

M

Mean

NIDA

National Institute on Drug Abuse

OPII

Organizational Process Improvement Intervention

RAND

Random function, Microsoft Excel

SPSS

Statistical Package for the Social Sciences

SD

Standard Deviation

WAI

Working Alliance Inventory