Validation of the Implementation Leadership Scale (ILS) in Substance Use Disorder Treatment Organizations

Abstract

There have been recent calls for pragmatic measures to assess factors that influence evidence-based practice (EBP) implementation processes and outcomes. The Implementation Leadership Scale (ILS) is a brief and efficient measure that can be used for research or organizational development purposes to assess leader behaviors and actions that actively support effective EBP implementation. The ILS was developed and validated in mental health settings. This study validates the ILS factor structure with providers in alcohol and other drug (AOD) use treatment agencies. Participants were 323 service providers working in 72 workgroups from three AOD use treatment agencies. Confirmatory factor analyses and reliability analyses were conducted to examine the psychometric properties of the ILS. Convergent and discriminant validity were also assessed. Confirmatory factor analyses demonstrated good fit to the hypothesized first and second order factor structure. Internal consistency reliability was excellent. Convergent and discriminant validity was supported. The ILS psychometric characteristics, reliability, and validity were supported in AOD use treatment agencies. The ILS is a brief and pragmatic measure that can be used for research and practice to assess leadership for EBP implementation in AOD use treatment agencies.

Although there are a variety of evidence-based practices (EBPs) available to treat substance use disorders (e.g., motivational interviewing, contingency management), the effective uptake and fidelity of these practices in routine care continues to be an issue of concern (Hall, Staiger, Simpson, Best, & Lubman, In press). To address these concerns, researchers have increasingly been focusing on understanding factors that impact successful uptake of EBPs (e.g., Aarons, Hurlburt, & Horwitz, 2011; Garner, 2009; Raghavan, Inoue, Ettner, & Hamilton, 2010). Some of the barriers identified relate to the complexity of implementing EBPs in community-based organizations, including both outer (i.e., system) and inner (i.e., organizational) context factors that impact the success of the implementation process (Aarons, Hurlburt, et al., 2011). For example, research has shown that individual-level factors such as clinician attitudes toward EBPs can affect whether clinicians are willing to adopt a practice into routine care with their clients (Henggeler et al., 2008; Smith & Manfredo, 2011). Additionally, broader organizational-level factors such as organizational climate for implementation interact with individual factors and play a role in whether a new practice is effectively implemented (Gotham, 2004; Jacobs et al., 2015; Simpson, 2002).

One major theme in research on organizational factors that play a role in successful implementation is leadership. Whether it be upper-level leaders setting the strategy for the organization and making decisions about funding the implementation of EBPs, or work group supervisors managing the day-to-day implementation issues and addressing their staff's problems and concerns (Aarons, Ehrhart, Farahnak, & Sklar, 2014), leaders across levels are important agents in effectively guiding the organization through the implementation process. Empirical research on the role of leaders in implementation supports this assertion, and a number of leadership behaviors have been associated with implementation-related outcomes (e.g., Aarons, 2006; Aarons, Sommerfeld, & Willging, 2011; Michaelis, Stegmaier, & Sonntag, 2009, 2010; Roman & Johnson, 2002; Sloan & Gruman, 1988).

Although prior research addresses general leadership behaviors, it does not address the behaviors leaders can perform to specifically support implementation efforts. For example, leaders can take a proactive, problem solving approach and engage and work with their supervisees to address implementation challenges. Other literatures provide some guidance on this issue; for instance, in the literature on services management and safety, researchers have begun to investigate the leadership behaviors associated with the achievement of specific strategic outcomes in those areas, studying such concepts as service leadership (e.g., Schneider, Ehrhart, Mayer, Saltz, & Niles-Jolly, 2005) and safety leadership (e.g., Barling, Loughlin, & Kelloway, 2002; Zohar, 2002), respectively. In the services literature in particular, recent meta-analytic evidence has shown that service leadership is more strongly related to service climate than general leadership (Hong, Liao, Hu, & Jiang, 2013). In the same way, focused or strategic leadership behaviors for supporting EBP implementation may also facilitate effective implementation (Aarons, Ehrhart, Farahnak, & Hurlburt, 2015).

In line with calls for efficient and pragmatic (i.e., brief, reliable, valid) measures which help to capture implementation constructs (Aarons, Hurlburt, et al., 2011; Damschroder & Hagedorn, 2011; Lewis et al., 2015), Aarons and colleagues (2014) developed the Implementation Leadership Scale (ILS) to deepen our understanding of the specific leadership behaviors that are critical for effective implementation. The ILS has four dimensions: 1) proactive leadership, 2) knowledgeable leadership, 3) supportive leadership, and 4) perseverant leadership. Proactive leadership addresses the degree to which the leader establishes clear goals, plans, and removes obstacles that may hinder EBP implementation. Knowledgeable leadership is the degree to which the leader is knowledgeable about the EBP being implemented and is able to successfully address staff questions. Supportive leadership is the degree to which a leader is supportive of staff efforts to use and learn about EBPs, and recognizes and acknowledges their efforts. Lastly, perseverant leadership is the degree to which the leader moves forward persevering through the ups and downs of the implementation process. The importance of leadership during the EBP implementation process is gaining more recognition in the substance abuse treatment literature (e.g., Hunter, Schwartz, & Friedmann, 2016); however, the ILS has not been validated in AOD treatment agencies. Although there are some similarities and differences in different service sectors and settings (Aarons, Hurlburt, et al., 2011), there is a need to examine implementation constructs across sectors to determine their psychometrics, performance, and generalizability.

The purpose of this study is to examine the factor structure, reliability, and validity of the ILS in AOD treatment agencies with a sample of alcohol and other drugs use treatment service providers. We hypothesized that the ILS would demonstrate a strong factor structure, high reliability, and show support for convergent and discriminant validity, supporting its use in AOD treatment settings. Specifically, we predicted that the ILS would have high correlations with another measure of leadership because dimensions of leadership across measures are likely to tap into the overall construct of leadership. We also predicted that the ILS would have low to moderate correlations with measures of organizational climate because, while leadership has been shown to be associated with organizational climate, climate is a distinct construct in its own right.

Method

Participants

Participants were 323 service providers employed in three AOD treatment agencies in California and New York. Of 363 eligible providers, 327 (90.1%) participated in the survey. Data from four participants were excluded due to missing data, resulting in a final analytic sample of 323 providers. Providers were organized into 65 workgroups, with an average workgroup size of 4.49 (SD=3.15; range=1–13). We defined `workgroup' as all providers who report to the same supervisor. As shown in Table 1, the sample was 62.9% female and participants had an average age of 46.49 years (SD=11.61, range 21–71). The racial distribution of the sample was 59.7% Caucasian, 18.9% African-American, 1.3% Native-American, 2.8% Asian-American or Pacific Islander, and 17.3% other. In addition, 28.5% of participants identified as Hispanic/Latino. Participant experience in substance abuse treatment averaged 7 years and job tenure averaged 3.7 years. The majority of participants had at least some college education (90.5%).

Table 1.

Participant Demographics

Characteristics	Values
Race
Caucasian	59.7%
African-American	18.9%
Asian-American or Pacific Islander	2.8%
Native American	1.3%
“Other”	17.3%
Ethnicity
Hispanic	28.5%
Non-Hispanic	71.5%
Education
No college	9.5%
Some college	32%
College degree	23.4%
Some graduate work	5.7%
Master's degree	27.5%
Ph.D. or M.D.	1.9%
Gender
Female	62.9%
Male	37.1%
Primary discipline
Drug/Alcohol Counseling	69.7%
Marriage & family therapy	11.1%
Psychology	8%
Social Work	8%
Other	3.2%
Age
Mean (SD)	46.5(11.6)
Tenure with agency (years)
Mean (SD)	3.7 (3.7)
Tenure in substance abuse treatment (years)
Mean (SD)	7.0 (6.1)

Note: Due to missing data, values may not add up to total N of 323.

Procedure

The research team made initial contact with agency executives to describe the study and recruit supervisors and providers for participation. Upon approval, providers were then contacted via email and phone. Eligibility in the study required that participants had prior experience with one or more EPBs and had been working with their identified supervisor for 3 months or longer. A telephone orientation meeting was conducted with each recruited team where a project overview was provided and questions were answered regarding including those pertaining to definitions of EBP.

This study was approved by the Institutional Review Board of San Diego State University. Data were collected via online (n=220) and in-person (n=103) surveys, and the survey took approximately 20–30 minutes to complete. The method of data collection (online vs. in-person) was determined by agencies' preferences and distance from the research team. There were no significant differences in any of the measures as a function of the method of survey administration. Participants provided informed consent and received a $15 gift certificate for their participation. For online surveys each participant was emailed a unique password and username, in addition to the link to the survey. For in-person data collection research staff administered the survey during a regularly occurring team meeting. If participants were not able to complete the survey in-person and collecting data online was not practical, surveys were mailed or left at the agency. Providers reported about their primary supervisor's implementation leadership and transformational and transaction leadership behaviors, as well as organizational climate for their unit.

Measures

Implementation leadership scale (ILS)

The ILS is comprised of 12 items. Subscales include Proactive Leadership (α =.94, 4 items), Knowledgeable Leadership (α =.97, 4 items), Supportive Leadership (α =.93, 4 items), and Perseverant Leadership (α =.94, 4 items). The mean of the subscales is computed to create the ILS total mean score (α =.97). The ILS is scored on a 5-point scale indicating the degree to which the leader performs specific behavior consisting from 0 (`not at all') to 4 (`to a very great extent'). The complete ILS measure including scoring instructions can be found at no cost in the “additional files” link accompanying the original measure development study (Aarons, Ehrhart, & Farahnak, 2014).

Multifactor leadership questionnaire (MLQ)

The MLQ (Bass & Avolio, 1995) is a well-known measure used to assess leadership in organizations. We selected the MLQ because it is the most widely researched measure of leadership. The MLQ assesses transformational leadership with four dimensions: individualized consideration (α =.92), idealized influence (α =.91, 8 items), inspirational motivation (α =.91, 4 items), and intellectual stimulation (α =.89, 4 items). Transactional leadership is also assessed, and is measured with the contingent reward (α =.84, 4 items) dimension. MLQ items were scored indicating how frequently the leader performed specific behaviors from 0 (`not at all') to 4 (`frequently, if not always') scale.

Organizational climate measure (OCM)

The OCM (Patterson et al., 2005) consists of a total of 17 dimensions capturing the domains of the competing values framework (Quinn & Rohrbaugh, 1983). The OCM was selected because it is a well-supported measure that has multiple dimensions likely to be related to EBP implementation. To assess discriminant validity, we utilized the following four dimensions: autonomy (α =.65, 5 items), formalization (α =.67, 5 items), efficiency (α =.89, 4 items), and performance feedback (α =.89, 5 items). OCM items were scored indicating the degree to which specific aspects of climate were absent or present in the workgroup 0 (`definitely false') to 3 (`definitely true') scale.

Statistical Analyses

Confirmatory factor analyses (CFA) were conducted using Mplus statistical software (Muthén & Muthén, 1998–2016) accounting for the nested data structure and using maximum likelihood estimation with robust standard errors to adjust the standard error and chi-square values. Although minimal, missing data were imputed using full information maximum likelihood (FIML) estimation. In order to assess model fit, several descriptive fit indexes and recommended cutoffs were utilized: comparative fit index (CFI) greater than .95, the root mean square error of approximation (RMSEA) less than .06, and the standardized root mean square residual (SRMR) less than .08 indicating good model fit (Hu & Bentler, 1999). Cronbach's alpha was also assessed for each of the subscales and the overall ILS. Intraclass correlations (ICC[1]s) and the average correlation within group (a_wg(j)) for each subscale were calculated to evaluate whether aggregation of the individual-level (i.e., provider) responses to the unit (i.e., workgroup level was warranted. ICC(1) represents the proportion of variance that is between units as opposed to within units. The a_wg(1) is calculated as one minus the quotient of two times the observed variance divided by the maximum possible variance, and a_wg(j) is the sum of a_wg(1) values for items divided by the number of items for a scale. Values of a_wg(j) range from −1.00 to 1.00, with values greater than 0.60 representing acceptable agreement (Brown & Hauenstein, 2005). Convergent and discriminant validity were also assessed based on the correlations between the ILS and the MLQ and OCM.

Results

Table 2 shows the ILS item and means, SDs, reliabilities, and aggregation statistics. Cronbach's alphas for the subscales and ILS total score ranged from .93–.97, demonstrating excellent internal consistency reliability. The ICC(1) for the overall ILS scale was .24 and the ICC(1) for the subscales ranged from .19 to .24. The a_wg(j) values for the total ILS scale and all but one of the four ILS dimensions were strong, ranging from .74 to .76. The pattern of all the aggregation statistics support the ILS subscales and total scale as unit-level constructs in AOD use treatment agencies.

Table 2.

Summary Statistics for the ILS Subscales and Total Scale

	Mean	SD	α	ICC(1)	awg	Factor loadings
Proactive leadership	2.51	1.13	.94	.19	.76	.92
Has developed a plan to facilitate implementation of EBP	2.49	1.23			.75	.91
Removed obstacles to implementation of EBP	2.40	1.23			.76	.93
Has established clear department standards for the implementation of EBP	2.63	1.16			.76	.89
Knowledgeable leadership	2.95	1.00	.97	.24	.76	.87
Is knowledgeable about EBP	3.01	1.00			.77	.94
Is able to answer staffs questions about EBP	2.90	1.07			.76	.96
Knows what he or she is talking about when it comes to EBP	2.93	1.03			.77	.97
Supportive leadership	2.90	1.02	.93	.20	.74	.90
Recognizes and appreciates employee efforts toward successful implementation of EBP	2.79	1.13			.74	.85
Supports employee efforts to learn more about EBP	2.95	1.08			.73	.93
Supports employee efforts to use EBP	2.96	1.04			.74	.94
Perseverant leadership	2.72	1.06	.94	.24	.76	.90
Perseveres through the ups and downs of implementing EBP	2.72	1.06			.78	.94
Carries on through the challenges of implementing EBP	2.75	1.09			.76	.97
Reacts to critical issues regarding the implementation of EBP by effectively addressing the problems	2.66	1.21			.75	.85
Implementation Leadership Total (12 items)	2.77	.97	.97	.24	.76

Note: Response anchors ranged from 0 = not at all, 4 = to a very great extent; SD = Standard deviation; α = Cronbach's alpha; ICC = Intraclass correlation; a_wg = average within group correlation; Factor loadings are all standardized and statistically significant at p<.001.

CFA results provided strong support for the four-factor implementation leadership model with a second-order overall factor indicating a higher order latent construct of implementation leadership. The model fit for the higher-order factor model was good (χ²(50)=158.18, p<0.001; CFI=0.966, RMSEA=0.082, 90% C.I. [.068, .096], probability RMSEA ≤ .05 = .000; SRMR=0.042). As shown in Table 2, the first-order standardized factor loadings ranged from .85 to .97 and the second-order standardized factor loadings ranged from .87 to .92. All factor loading were statistically significant (p's < 0.001).

Table 3 presents the convergent and discriminant validity results. The ILS dimensions and total scale were moderately to strongly correlated with both transformational and transactional leadership with correlations ranging from .57 to .77. These moderate to high correlations with the MLQ provides support for convergent validity. For discriminant validity, the ILS total scale and dimensions were weakly to moderately correlated with the OCM autonomy (range .19 to .27), formalization (range .21 to .28), efficiency (range .29 to .41), and feedback (range .47 to .57) dimensions. Although overall correlations were found to be somewhat higher than the original measurement development paper in a mental health sample (Aarons, Ehrhart, & Farahnak, 2014), correlations were still weak enough to provide support for discriminant validity of the ILS compared to the OCM.

Table 3.

Correlations of the Implementation Leadership Scale Scores with Multifactor Leadership Questionnaire [convergent validity] and Organizational Climate Measure [divergent validity] Scores

		Implementation Leadership Subscales				ILS Total
		Proactive	Knowledgeable	Support	Perseverant
MLQ	Transformational Leadership
	Intellectual Stimulation	.68**	.65**	.74**	.73**	.77**
	Inspirational Motivation	.66**	.64**	.72**	.72**	.74**
	Individual Consideration	.70**	.64**	.76**	.71**	.77**
	Idealized Influence	.69**	.64**	.75**	.74**	.77**
	Transactional Leadership
	Contingent Reward	.67**	.57**	.71**	.65**	.71**
Climate	Autonomy	.24**	.19**	.27**	.27**	.26**
	Formalization	.25**	.28**	.24**	.21**	.28**
	Efficiency	.29**	.36**	.38**	.34**	.41**
	Feedback	.52**	.47**	.57**	.50**	.57**

Note: Sample size of correlations varies from 316 to 323; ILS = Implementation Leadership Scale; MLQ = Multifactor Leadership Questionnaire; Climate = General Organizational Climate Measure;

^*p < .05,

^**p < .01,

^***p < .001.

Discussion

This study supports the first and second order factor structure, reliability and validitity of the ILS in a sample of AOD use treatment providers. Similar to previous ILS validation studies in other health sectors (Aarons, Ehrhart, & Farahnak, 2014; Finn, Torres, Ehrhart, Roesch, & Aarons, In press), both the first and second order factor structure of the ILS demonstrated strong model fit. The ILS demonstrated strong internal consistency rebliability, as well as convergent and discriminant validity providing further support for its use in AOD use treament agencies. Interestingly, the OCM “Feedback” scale demonstrated moderate correlations with the ILS, stronger correlations than for the other OCM subscales. This is likely because feedback is a construct that is more tied to interactions with a supervisor or designated leader. For example, feedback is provided during annual performance reviews and for guidance on performance of assigned tasks and duties. This begs the question of how leader self-perceptions and provider perceptions of the leader may impact organizational culture and climate (Aarons, Ehrhart, Farahnak, Sklar, & Horowitz, In press; Ehrhart, Schneider, & Macey, 2014). For example, discrepancies in ILS ratings in mental health are associated with feedback climate (Aarons, Ehrhart, Torres, Finn, & Beidas, In review).

The ILS is brief, efficient, and pragmatic measure that can be administered quickly, typically taking no more than a few minutes to complete. This supports the pragmatic nature of the measure as it reduces the amount of time taken away from job duties for those completing the measure. The ILS maps onto the leadership construct in the Exploration, Prepration, Implementation, Sustainment (EPIS) implementation framework that is increasingly be utilized in implementation studies in addictions (Aarons, Hurlburt, et al., 2011; Knight et al., 2015). Additionally, the ILS could be used in leadership training programs to help leaders identify and develop specific leader behaviors that help to facilitate a strong implementation climate. One such leadership training program is the Leadership and Organizational Change for Implementation (LOCI) strategy that involves improving general leadership and implementation leadership, while also enhancing organizational supports for EBP implementation (Aarons et al., 2015) that is currently being tested in AOD treament agencies. In this study both general leadership (i.e., transormational/transactional) and implementation leadership development are supported through assessment of current leadership, tailored leadership development plans, and brief coaching to support leader behavior change.

Some limitations of this study should be noted. First, this study focused on model fit, reliability, aggregation statistics, and convergent and discriminant validity, Future studies should examine the predictive validity of the measure for implementation-related outcomes including proximal outcomes such as change in leadership behaviors. Second, more complex models examining the predictive power of implementation leadership are needed, including subsequent affects on implementation climate (Ehrhart, Aarons, & Farahnak, 2014), implementation citizenship behavior (Ehrhart, Aarons, & Farahnak, 2015), and implementation outcomes (Proctor et al., 2011). Third, leadership ratings in this study were only from the persepectives of staff about their immediate supervisor. There may be discrepancies between leader and follower ratings that can affect organizational culture or climate (Aarons et al., In press). Finally, as the organizational structure of substance use treatment programs are typically multilevel, future research should examine supervior and staff perceptions of the implementation leadership for middle and upper-level management.

Conclusion

This study provides support for the generalizability of the ILS in AOD treament agencies. Past research has identified the critical role leaders can play in the EBP implementation process. This study demonstrated that ILS dimensions of being knowledgeable, supportive, proactive, and perseverant during the implementation process are relevant for AOD treament agencies. Further research should further examine how leadership impacts the efficiency and the effectiveness of EBP implementation, with the aim of improving services to improve the quality and outcomes of substance abuse treatment services.

Highlights.

• This study validates the Implementation Leadership Scale (ILS) for use in substance use disorder treatment organizations.

• The ILS assesses specific leader behaviors that support the implementation of EBPs.

• The ILS is a very brief and pragmatic measure that can be used for research, organizational development, and leadership development.