The Roles of System and Organizational Leadership in System-Wide Evidence-Based Intervention Sustainment: A Mixed-Method Study

Abstract

If evidence-based interventions (EBIs) are not sustained, investments are wasted and public health impact is limited. Leadership has been suggested as a key determinant of implementation and sustainment; however, little empirical work has examined this factor. This mixed-methods study framed using the Exploration, Preparation, Implementation, Sustainment (EPIS) conceptual framework examines leadership in both the outer service system context and inner organizational context in eleven system-wide implementations of the same EBI across two U.S. states and 87 counties. Quantitative data at the outer context (i.e., system) and inner context (i.e., team) levels demonstrated that leadership predicted future sustainment and differentiated between sites with full, partial, or no sustainment. In the outer context positive sustainment leadership was characterized as establishing a project’s mission and vision, early and continued planning for sustainment, realistic project plans, and having alternative strategies for project survival. Inner context frontline transformational leadership predicted sustainment while passive-avoidant leadership predicted non-sustainment. Qualitative results found that sustainment was associated with outer context leadership characterized by engagement in ongoing supportive EBI championing, marketing to stakeholders; persevering in these activities; taking action to institutionalize the EBI with funding, contracting, and system improvement plans; and fostering ongoing collaboration between stakeholders at state and county, and community stakeholder levels. For frontline leadership the most important activities included championing the EBI and providing practical support for service providers. There was both convergence and expansion that identified unique contributions of the quantitative and qualitative methods. Greater attention to leadership in both the outer system and inner organizational contexts is warranted to enhance EBI implementation and sustainment.

Evidence-based interventions (EBIs) are treatments, interventions, or practices with outcomes supported by rigorous scientific evidence (Chambless & Ollendick, 2001). While a number of similar terms have been used (e.g., evidence-based treatment, evidence-based practice, empirically supported treatment), the common thread among these terms is reliance on research evidence as a key feature (APA Presidential Task Force on Evidence-Based Practice, 2006; Institute of Medicine [IOM], 2001). Despite increasing demand for EBIs in public-sector service systems (Wike et al., 2014), there is a need for greater understanding of factors that facilitate their implementation and sustainment. If EBIs are not sustained, investments in their development and implementation are wasted and public health impact will be limited. Most pertinent to the study for sustainment is the finding of a 55% failure rate for implemented home-based treatments (Wright, Catty, Watt, & Burns, 2004). For programs still “identifiable” after implementation, many key elements were no longer practiced. A systematic review of dissemination and implementation research in children’s mental health research also determined that only 10% (n=8) of 80 the reviewed articles focused on sustainment (Novins, Green, Legha, & Aarons, 2013). Key sustainment factors elucidated in these few studies include the importance of ongoing supervision/support, the fit of EBI with organization, staff, and clients, and a supportive organizational culture. However, there has been little empirical work centered on the role of leadership in sustainment across both the outer service system and inner organizational contexts (Aarons, Hurlburt, & Horwitz, 2011).

In the present study we use mixed-methods to examine the role of system level and organizational level (i.e., front-line leadership) in sustainment of a home-based EBI across eleven unique service systems in two states involving 87 counties. This study adds to the implementation science literature in three distinct ways. First, while there are numerous studies of implementation, there are few studies of sustainment. This study examines large-scale EBI sustainment in public sector systems for vulnerable populations. Second, leadership is a component or construct in a number of implementation frameworks, but there is little empirical research that elucidates the types of leadership or outer and inner context impacts on implementation or sustainment. This study examines leadership from multiple perspectives consistent with one of the five most highly cited implementation frameworks currently being used – the Exploration, Preparation, Implementation, Sustainment (EPIS) framework (Aarons, Hurlburt, et al., 2011; Skolarus & Sales, 2014). Third, there have been calls for use of mixed-methods in implementation science. The study avoids the pitfall of lack of mixed-method integration by addressing leadership for sustainment through complementary methodological approaches that integrates mixed-methods to quantitatively answer the questions of whether leadership at the system and organizational levels impact sustainment, and uses functions of convergence and expansion with quantitative findings to further elucidate the nature of leadership that supports sustainment across outer context and inner context.

This study focuses on leadership because leaders can play a unique role in positively or negatively impacting the capacity to foster change and innovation (Damanpour & Schneider, 2006; Gumusluoglu & Ilsev, 2009; Jung, Chow, & Wu, 2003; Scott & Bruce, 1994). Leaders and their behaviors are instrumental in facilitating a positive climate for innovation and positive attitudes toward EBI during implementation (Aarons, 2006; Aarons & Sommerfeld, 2012). While there is a vast literature on leadership, the role of leadership in EBI implementation is often discussed, and almost universally acknowledged as critical, but is rarely empirically analyzed. The limited empirical research in this area supports the presence of a relationship between general leadership ability and implementation of innovative practices (Michaelis, Stegmaier, & Sonntag, 2010), but focuses less on identifying characteristics of leadership across levels. While studies vary in setting (e.g., human services, business etc.), sample size, number of organizational units, and rigor, there is consensus that leadership is important in implementation and change.

Outer and Inner Context of Implementation and Sustainment

Implementation conceptual frameworks illustrate the complexities of EBI implementation and sustainment, with many approaching implementation as a process involving stakeholders operating in complex systems at multiple levels (Aarons, Hurlburt, et al., 2011; Damschroder et al., 2009; Meyers, Durlak, & Wandersman, 2012). Developed with a focus on public-sector service settings, the four-phase Exploration, Preparation, Implementation, and Sustainment (EPIS) model explicitly addresses both context and process. The EPIS framework emphasizes the role of outer (i.e., system) and inner (i.e., organization) context factors (Aarons, Hurlburt, et al., 2011) and their interplay through each implementation phase. Outer context factors shape the broader environment that affects operations in a service system and encompasses policies, funding, contracting and relationships with provider organizations, system-level leadership, inter-organizational networks, and academic-community partnerships. Inner context factors are those specific to the organizations and groups tasked with delivering EBIs. They include leadership at the organization and team/workgroup level, organizational culture and climate, and characteristics of service providers, such as work attitudes, adaptability, experience, and training.

Both implementation theories and leadership theories emphasize leadership in supporting innovative practices such as EBIs. Implementation scholars assert the importance of leadership in terms of obtaining funding, dispersing resources, and enforcing policies in support of implementation (Aarons, Horowitz, Dlugosz, & Ehrhart, 2012). For example, research from the United Kingdom’s Collaboration for Leadership in Applied Health Research and Care addresses the importance of leaders managing implementation projects, obtaining senior management support, serving as clinical opinion leaders, and fostering organizational learning climates (Harvey et al., 2011). Other research suggests that leader roles can include interpreting research evidence, applying it to organizational contexts, and making research-informed implementation decisions (Kyratsis, Ahmad, & Holmes, 2012). Weiner’s organizational theory of innovation implementation suggests that leaders are critical in creating readiness for change, ensuring innovation-values fit, and developing plans, practices, structures, and strategies to support implementation (Weiner, 2009). Recent work on assessing implementation leadership has identified specific leader characteristics in regard to EBI implementation and sustainment including being knowledgeable about EBIs, supportive of staff in the implementation process, proactive in problem solving implementation issues, and perseverant through the ups-and-downs of implementation (Aarons, Ehrhart, & Farahnak, 2014).

There is also growing empirical evidence for the importance of leadership in predicting the success of implementation efforts. For example, transformational leadership (i.e., the degree to which a leader can inspire and motivate others; (Bass & Avolio, 1995) may predict employees’ reported use of an innovative practice being implemented in their organization (Michaelis, Stegmaier, & Sonntag, 2009; Michaelis et al., 2010). Consistent with transactional leadership (e.g., providing reinforcement or rewards for desired behaviors), perceived support from one’s supervisor is also associated with employees’ participation in implementation (Sloan & Gruman, 1988). Empirical research on leadership and implementation identifies mechanisms through which leaders affect implementation including facilitating positive organizational climate (Aarons, Sommerfeld, & Willging, 2011), supportive team climate (Bain, Mann, & Pirola-Merlo, 2001), and positive work attitudes (Kinjerski & Skrypnek, 2008). Research has also shed light on the role of leaders in influencing employee attitudes toward EBI (Aarons, 2006), commitment to organizational change (Hill, Seo, Kang, & Taylor, 2012), and improving leader EBI support behaviors (Aarons, Ehrhart, Farahnak, & Hurlburt, 2015).

A number of implementation factors span outer and inner contexts. These include engaging strong leadership across system and organizational levels, use of specific management strategies, attending to both organizational and individual factors, and anchoring new programs across system levels (Larsen & Samdal, 2007). A recent review highlights how leadership at the outer and inner contexts may be critical for effective implementation and sustainment (Aarons, Ehrhart, Farahnak, & Sklar, 2014). There is a need to identify unique sustainment factors that can lead to improvements in processes and efficiencies not evident during initial implementation (Grimes, Kurns, & Tilly, 2006). Thus, additional empirical attention should be given to the roles of both outer and inner context leadership in EBI sustainment.

Outer context leadership can be considered in a number of ways; however, the notion of leadership competence is one that may be key to program sustainability and is assessed in the Program Sustainability Index (PSI; Mancini & Marek, 2004). The PSI Leadership Competence (PSI-LC) scale assesses specific leader actions, such as establishing mission and vision, early planning for sustainment, continued planning for sustainment, developing and following a realistic project plan, and using multiple strategies for project survival. These factors may be critical in supporting not only implementation, but also long-term sustainment.

Regarding inner context leadership, the Full-Range Leadership (FRL) model includes empirically supported dimensions of transformational leadership, transactional leadership, and passive-avoidant leadership (Avolio, Bass, & Jung, 1999). Transformational leadership is motivational, individually considerate, intellectually stimulating, and can engage staff in supporting the mission and vision of a leader and promote a climate for innovation and change. Transactional leadership involves provision of reinforcements for positive behaviors and monitoring of quality standards. Passive-Avoidant leadership, also called “non-leadership,” describes an absent leader who is not actively engaged. These aspects of leadership are well-researched and are associated with organizational functioning and attitudes toward adopting EBI (Aarons, 2006). Much of the literature on leadership investigates work groups and organizational levels. Yet, organizations can be strongly influenced by the decisions and policies made or instantiated by leaders at the system level that concern funding, disbursement of resources, and policy making to support EBI implementation (Stamatakis, Vinson, & Kerner, 2012). Different leadership approaches may be more effective depending upon whether an organization is in a stable-operating state or undergoing change (Eggleston & Bhagat, 1993). We extend this line of reasoning and suggest that leadership may be important at both the outer and inner context levels across the EPIS phases and that effects of leadership and ongoing leadership will be evident in the sustainment phase.

The Present Study

Our goal for the present study was to examine how outer and inner context leadership was related to system-wide EBI sustainment. Qualitative and quantitative data were drawn from a larger mixed-method investigation of EBI sustainment (Aarons, Green, et al., 2014). Previous work from this project has examined the role of collaboration in implementing and sustaining EBIs (Green et al., 2016), policymaker’s perspectives on EBI sustainment (Willging, Green, Gunderson, Chaffin, & Aarons, 2015), and the role of performance-based contracting in EBI sustainment (Willging et al., In press). The current study examines the roles of leadership in the outer system and inner organizational context (i.e., first-level leadership) in sustaining EBI service delivery. We predicted that service systems demonstrating more positive leadership at the system level and more positive and less negative leadership at the organizational level would be more likely to demonstrate EBI sustainment.

Methods

Study Context

This study utilizes quantitative and qualitative methods and data from 11 separate service systems that all implemented SafeCare® (SC), an EBI to reduce child maltreatment. Included are one statewide child-welfare service system and 10 county-wide systems, representing public health (n=1), child welfare (n=8), and mental health (n=1). SafeCare implementation began between 2 and 10 years prior to study participation. Systems had differing paths but all navigated through EPIS exploration, preparation, and implementation phases prior to this examination of sustainment. Service systems span two U.S. states, with one state utilizing a state-wide system, and the other state being county-based in which each county independently operates its own child welfare, public health, or mental health systems. California county systems are similar to the statewide system in a number of ways. California county systems operate highly autonomously from state oversight, similar to the statewide system in the current study. In addition, the statewide system is very similar in size to one of the county systems, and is similar in approach to services (i.e., contracting with CBOs for service delivery), CBO coverage, service model, provider characteristics, and the client population The statewide service system has a population of approximately 3.7 million residents, almost 42% of whom live in rural areas (U.S. Census Bureau, 2010). In this state, SC was implemented through a state-operated child welfare system with all services guided, contracted, and funded by the state government. Local community-based organizations (CBOs) contracted to deliver SC were mostly private non-profit service organizations that provided various services including mental health, child-welfare, substance abuse, and other services. These CBOs bid for contracts from the state agency to provide SC as part of an existing home-based service delivery infrastructure. Although the CBOs competed for contracts, at times they also collaborated or partnered with one another to bid for contracts [identified as collaboration, competition, or “co-opetition” (Bunger et al., 2014)], in order to cover more service areas, share resources, and/or strategically offer services. A local academic institution that had a long standing relationship with the state health and human services agency was influential in the initial selection of SC, training, ongoing fidelity monitoring, and coaching. The institution also collaborated with government stakeholders and CBOs as part of a large federally-funded experimental effectiveness trial of SC (Chaffin, Hecht, Bard, Silovsky, & Beasley, 2012) and maintains its ongoing collaborations with these partners on research and evaluation projects.

In the second state, service systems in each county are largely independent with some accountability to the state and federal initiatives. The ten service systems implementing SC in this state include six primarily urban and four primarily rural counties ranging in population from approximately 150,000 to 3.2 million residents. Counties in this state implemented SC through different partnerships and shifting funding arrangements (e.g., monies from county sources for service provision, foundation funding for scale-up). Federal dollars from the Centers for Disease Control, the Administration for Children Youth and Families, and the National Institutes of Health supported some training and research activities. Each scale-up project had multiple stakeholders involved. For example, the initial decision to implement SC in one county arose from meetings involving key personnel from county child welfare, nonprofit CBOs contracted to deliver child welfare services, a private foundation interested in initially funding an EBI, academic researchers, and EBI developers. Researchers at an academic institution also partnered with stakeholders within the 10 counties as part of federally-funded research studies that examined cascading diffusion models for EBI implementation, adoption, and adaptation in implementation (Aarons, Green, et al., 2012). In other counties implementation was facilitated through federal funds garnered by a prominent organization focused on addressing child maltreatment.

The Implemented EBI SafeCare®

SafeCare is a manualized curriculum-based EBI to reduce child maltreatment through home-based skills training and education for caregivers of children, ages zero to five, who are at-risk for, or reported for, child neglect (Chaffin et al., 2012; Lutzker & Edwards, 2009). SafeCare enhances problem solving and communications to improve parent or caregiver skills and behaviors to address home safety, child health, and parent-child or parent-infant interactions. SafeCare requires three primary professional roles to implement the program with fidelity: 1) home visitors who deliver the EBI to caregivers; 2) coaches who provide assistance to and conduct monthly monitoring of home visitors to ensure high levels of fidelity to the EBI; and 3) trainers who are certified to train and coach new home visitors. In the present study, all SC sites took advantage of this structure to facilitate self-sustainment and create resilience to workforce turnover by localizing training and quality control within the service system.

Participants

Table 1 shows demographics for all participants, including those who participated in the quantitative and/or qualitative components of the study. Consistent with recommendations for sampling in mixed-methods, a purposive sampling approach was used to identify and recruit the most relevant individuals with knowledge of the EBI implementation and sustainment in each service system (Aarons, Fettes, Sommerfeld, & Palinkas, 2012; Collins, Onwuegbuzie, & Jiao, 2007; Teddlie & Yu, 2007). In particular, system and organization administrator level respondents must have had involvement or knowledge of the implementation of SafeCare in their service system. Participants associated with the outer context were state, county, and agency administrators (n=44) completed a web-based survey that measured system-level leadership for EBI sustainment. For the 11 service systems the average number of participants in each system was 4 (SD = 3.07; range = 1 to 11). Participants for the inner context were front-line service providers (home visitors, n=162) employed by the CBOs providing SafeCare who completed a web-based survey assessing the leadership of their immediate. There were 39 teams with an average of 5.25 providers per team (SD = 3.73; range = 1 to 25).

Table 1.

Response Rate, Sample, and Participant Demographics

	Administrators	Service Providers
Response Rate	93%	94%
Sample Size	n=45	n=212
State/County	n=27	–
CBO	n=18	–
Gender
Female	87%	90%
Male	13%	10%
Education
High school	0%	1%
Some College	9%	22%
College Graduate	15%	55%
Master’s Degree	69%	22%
PhD	7%	0%
Ethnicity	22% Hispanic	43% Hispanic
Race
Caucasian	73%	43.7%
Black	2%	8.7%
American Indian	5%	15%
Asian	9%	3.1%
Multiple race	–	2%
Other race	11%	18%

Focus groups were conducted in-person with a subset, representing each service system, all agency administrators (n=44) completed the web-based survey prior to their interview. Of home visitors who completed the web-survey, individual interviews were conducted with each state, county (n=96). Participants were offered a small gift card as an incentive for participation in quantitative and qualitative components of the study. This study was approved by the [name withheld for blinding] Institutional Review board and informed consent was obtained from all participants prior to data collection.

Measures

Outer Context Leadership

Outer Context Leadership was measured with the Leadership Competence Scale of the Program Sustainability Index (PSI; Mancini & Marek, 2004). This measure was first used in mixed-method studies of community-based program personnel (Mancini & Marek, 1998). It comprises 5 subscales that demonstrate good psychometric properties with Cronbach’s alphas ranging from. 76–.88. We used the Leadership Competence (LC) subscale that consists of 5 items that assess the degree to which leaders: 1) establish a project’s mission and vision; 2) engage in early planning for sustainment; 3) continue planning for sustainment; 4) develop and follow a realistic project plan; and 5) identify alternative strategies for project survival. In regard to validity, the PSI is associated with ongoing planning process, confidence in project long-term survival, and meeting the needs of clients (Mancini & Marek, 2004). More importantly, the present study has the potential to add to the literature and provides support for the predictive validity of the PSI in predicting sustainment. Items were rated on a 5 point Likert scale from 0, “Not at all” to 4, “To a very great extent.” The PSI-LC subscale has excellent internal consistency reliability in this study (α=0.84).

Inner Context Leadership

Inner Context Leadership was assessed with the Multifactor Leadership Questionnaire (MLQ) 45-item Form 5X (Bass & Avolio, 1995). Service providers rated their immediate supervisor’s leadership behaviors. We examined transformational leadership, transactional leadership, and passive-avoidant leadership. The Cronbach’s alphas from the present study indicated excellent reliability for transformational leadership (α=0.96) and passive-avoidant leadership (α=0.86), and good reliability for transactional leadership (α=0.76). The validity of the MLQ is supported by studies demonstrating its associations and prediction of important organizational outcomes including change in practice (Leithwood & Jantzi, 2006), attitudes toward EBIs and capacity to train providers and implement EBIs (Aarons, 2006; Bonham, Sommerfeld, Willging, & Aarons, 2014), organizational culture (Aarons, Ehrhart, Farahnak, Sklar, & Horowitz, In press), knowledge sharing (Chen & Barnes, 2006), employee creativity (Dhar, 2015; Jyoti & Dev, 2015), innovation performance (Saad & Mazzarol, 2014), enhanced EBI receptivity, ongoing use, and ability to implement and sustain EBIs (Stetler, Ritchie, Rycroft-Malone, Schultz, & Charns, 2009), organizational climate expectations (von Thiele Schwarz, Hasson, & Tafvelin, 2016), and patient and consumer outcomes (Corrigan, Lickey, Campion, & Rashid, 2000; Wong & Giallonardo, 2015). Home visitors indicated the extent to which their supervisor exhibited specific behaviors on a 5 point Likert-type scale from “Not at all” to “To a very great extent.” Individual leadership scores were computed as item averages.

Sustainment

Sustainment was defined consistent with Stirman and colleagues’ (2012) systematic review, that recommends classifying an EBI as “fully” sustained if core elements are maintained or delivered at a sufficient level of fidelity after initial implementation support has been withdrawn, and adequate capacity exists to continue maintaining these core elements. For SC, core elements of fidelity include both “structural fidelity” (e.g. appropriate caseload sizes, monthly coaching visits, regularly scheduled team meetings) and “content fidelity” (e.g. fidelity to the EBI content for modules and sessions within modules). “Partial sustainment” sites were those where only some core elements continued after the withdrawal of initial implementation support. “Non-sustainment” sites were those in which certified home visitors were not implementing SC or its core elements with structural or content fidelity. Sites were categorized according to level of sustainment at the time of data collection as “full sustainment” (sites meeting key fidelity requirements; n=7), “partial sustainment” (sites meeting some of the model fidelity requirements; n=1), and “no sustainment” (sites no longer providing the EBI; n=3). Classification of sites into each category was based on independent review and then consensus of the first and second author, supported by consultation from the EBI developers. Those sites with “no sustainment” no longer provided SC services. The site classified as “partial sustainment” had providers trained and certified in the EBI who were actively providing services, but no longer provided recommended ongoing coaching and fidelity monitoring. The “full sustainment” sites have certified SC providers who conduct SC sessions, there is ongoing coaching and fidelity monitoring, and SC team meetings in accordance with model developer standards.

Sequencing of Data Collection

Data are from a prospective study of sustainment. For our leadership predictor variables we used web-based surveys and data from the last available wave of data for each site. Thus, time of data collection varied from 2010 to 2013 depending on whether a site was still active. Thus, the leadership ratings were based on current leadership during the EPIS active implementation phase (Aarons, Hurlburt, et al., 2011) as a predictor of future sustainment. Sustainment status was assessed in 2015. The qualitative interviewers were conducted by ethnographers that were not local academic collaborators or intervention developers. Although they were aware of which sites had no providers with whom to conduct focus groups, they were informed of the sustainment status of the sites for which there were current providers.

Qualitative Interview and Focus Groups

Outer Context

To assess outer context leadership, state/county and CBO administrators were asked about leadership during semi-structured interviews. Example questions and probes included: 1) “How are leaders within your state/county/agency influencing the ongoing use of SafeCare?” 2) “Who are these leaders? What did they do? Why?” 3) “Who are the most important decision-makers, or stakeholders, to influence whether SafeCare continues to be implemented? Why are they the important stakeholders?”

Inner Context

Consistent with our inner context quantitative measurement, the following questions and probes were posed to front-line service providers: 1) “How have leaders within your team supported use of SafeCare? What have they done to potentially undermine the use of SafeCare?” 2) “Who are the most important decision-makers, or stakeholders, to influence whether SafeCare continues to be implemented? Why are they the important stakeholders?”

Analyses

Quantitative Analyses

Ordinal regressions were conducted to examine the role of leadership during implementation in predicting future sustainment level (non, partial, full) at the system level. All analyses accounted for the nested data structure for system and organizational (i.e., team) levels, respectively. Because there were different respondents for outer and inner context analyses, respondents, and leadership variables at the system and team levels, separate outer context and inner context ordinal regressions adjusting standard errors for the relevant nesting unit of analysis were utilized. This approach adjusts for dependencies within each context. Thus, outer context analyses were nested by system (i.e., state or county) while inner context analyses were nested at the team level. Mplus version 7.3 (Muthén & Muthén, 1998–2016) was utilized specifying the model as “complex” which utilizes maximum likelihood estimation with robust standard errors to adjust for the nested data structure. Sustainment level was coded into three categories: 0=non-sustainment, 1=partial sustainment, and 2=full sustainment. Thus we examined 1) outer context leadership (i.e., PSI-LC), and 2) inner context leadership (i.e., transformational, transactional, passive-avoidant) in predicting sustainment.

Provider age, job tenure, and sex were controlled for due to their potential influence on both leadership perceptions and SafeCare sustainment. For example, older participants are more likely to have greater work experience, and therefore have more exposure to different leaders and their behaviors. Researchers have also demonstrated differences in perceptions of leadership depending on whether a respondent is male or female (e.g., Boatwright & Forrest, 2000). Hence, participant age and job tenure may impact attitudes and/or uptake of evidence-based practices (Aarons, 2006; Aarons & Sawitzky, 2006; Gray, Elhai, & Schmidt, 2007; Henggeler et al., 2008).

Qualitative Analyses

Between 2012 and 2014, two anthropologists collected qualitative data regarding system-level (outer context) and frontline (inner context) leadership via individual semi-structured interviews with state, county, and CBO administrators and focus groups with providers. As described previously, questions centered on identifying and describing the role of leadership in both the implementation and sustainment of SC, forms of leadership support, interactions with other SC stakeholders, and remaining needs for SC provision. Interviews and focus groups were digitally-recorded, professionally transcribed, and reviewed for accuracy by at least one author. Two members of the research team used an iterative process to review the transcripts using NVivo 10 qualitative data analysis software (QSR International, 2012). Segments of text ranging from a phrase to several paragraphs were assigned codes based a priori on the topic areas and questions that made up the interview guides (Patton, 2002, 2015). These codes centered on key sensitizing concepts from the implementation literature (e.g., implementation, sustainment, leadership support, and stakeholder interaction). These concepts provided “a general sense of reference” for our analysis and allowed us to analyze their salience and meaning for stakeholders through participants’ reflections on their own perceptions and experiences (Patton 2015, p. 545). Focused coding was then used to determine which of these concepts or themes emerged frequently and which represented unusual or particular concerns to the research participants. Each team member independently coded sets of transcripts, created detailed memos that both described and linked codes to each theme and shared their work with one another for review. Through the process of comparing and contrasting codes with one another (Corbin & Strauss, 2008; Glaser & Strauss, 1967), codes with similar content or meaning were grouped together into broad themes linked to segments of text. Qualitative results were classified into themes related to system-level leadership (i.e. state, county, and CBO administrators) and those related to frontline leadership (i.e., supervisors).

Mixed-Methods Integration

We followed recommendations for mixed-methods research designed to integrate qualitative and quantitative method philosophies, designs, strategies, analytic approaches, and interpretations (Aarons, Fettes, et al., 2012; Greene, 2006; Johnson, Onwuegbuzie, & Turner, 2007; Tashakkori & Teddlie, 2003). Mixed-methods research is increasingly being recognized as critical for studies of innovation implementation in health and human service settings (Demakis, McQueen, Kizer, & Feussner, 2000; Greenhalgh et al., 2010; Palinkas et al., 2011; Soh et al., 2011; Stetler et al., 2006). As such, we utilize two mixed-methods functions of convergence (i.e., determine whether the two methods support or provide corroboration across methods) and expansion (i.e., the degree to which one method provides new or additional insights into a given phenomenon or concern).

Results

Quantitative Results

Tables 2 and 3 show means and standard deviations for leadership measures for outer context and inner context participants, respectively.

Table 2.

Observed Means and Standard Deviations for Leadership Competence by Type of Sustainment

	n	Leadership Competence
Full Sustainment	36	3.15 (.48)
Partial Sustainment	2	3.20 (.28)
Non-Sustainment	6	1.37 (.51)

Note: This table represents outer context leadership

Table 3.

Observed Means and Standard Deviations for Transformational Leadership, Transactional Leadership, and Passive-Avoidant Leadership by Type of Sustainment

	Type of Leadership
	n	Transformational	Transactional	Passive-Avoidant
	Full Sustainment	196	2.73 (1.26)	2.00 (.99)	0.57 (.71)
Partial Sustainment	5	2.70 (.27)	2.28 (.60)	0.58 (.89)
Non-Sustainment	9	2.24 (.37)	2.11 (.38)	2.18 (.29)

Note: This table represents inner context frontline/team leadership

Outer Context: PSI Leadership Competence

We found that leadership competence scores predicted sustainment while controlling for time since sustainment (months) and system population size (β=.697, p <.001). For every one unit increase in leadership competence, there was a.697 increase in the log odds of attaining a given level of sustainment (non to partial, partial to full). The log ratio for leadership competence indicates that for a one unit increase in leadership competence, the odds of the non-sustainment and partial sustainment versus the full sustainment are 17.167 times greater. The same increase, 17.167 times, is found between non-sustainment and the combined categories of partial and full sustainment. Leadership competence, time from implementation, and system size variables accounted for approximately 67% (R²=.667) of the variance in sustainment in this model.

Inner context: MLQ Transformational, transactional, and passive-avoidant leadership

Ordinal regression analyses indicated that transformational, transactional, and passive-avoidant leadership – controlling for time from implementation, system size, provider age, sex, job tenure – predicted future EBI sustainment (see Table 5). The ordinal regression analyses showed that transformational leadership (β=.165, p <.05) significantly predicted sustainment, such that greater transformational leadership predicted sustainment. Thus, for every one unit increase in transformational leadership, there was a .165 increase in the log odds of moving from a given level of sustainment to the next level. The log ratio for transformational leadership indicates that for a one unit increase in leadership competence, the odds of the non-sustainment and partial sustainment versus the full sustainment are 1.563 times greater. The same increase, 1.563 times, is found between non-sustainment and the combined categories of partial and full sustainment. In the transformational leadership analysis – the combined predictor and covariates accounted for approximately 46% of the variance in sustainment (R²=.461).

Table 5.

Ordinal Logistic Regression Analyses of Effects of Transformational, Transactional, and Passive-Avoidant Leadership on Sustainment Controlling for Time from Implementation and Respondent Age, Sex, and Job Tenure

Independent variable	B	SE	z ratio	p-value	Odds Ratio
Transformational Leadership	.165	.081	2.038	.042	1.563
Age	−.052	.128	−.407	.684	.989
Sex	.070	.088	.796	.426	–
Job Tenure	.300	.175	1.739	.082	1.158
Time (months)	.590	.108	5.440	.000	1.032
Transactional Leadership	−.143	.074	−1.930	.054	.609
Age	−.048	.143	−.336	.737	.990
Sex	.078	.093	.840	.401	–
Job Tenure	.370	.193	1.915	.055	1.195
Time (months)	.555	.106	5.235	.000	1.030
Passive-Avoidant Leadership	−.395	.105	−3.762	.000	.229
Age	−.121	.129	−.943	.346	.969
Sex	.026	.069	.378	.705	–
Job Tenure	.189	.124	1.522	.128	1.114
Time (months)	.655	.119	5.499	.000	1.042

Note: N=211; Sustainment coded as 0=non-sustainment, 1=partial sustainment, 2=full sustainment; Time (months) indicates time in months from SafeCare® implementation for each service system.

Passive-avoidant leadership also significantly predicted sustainment (β=−.395, p <.001), however in the opposite direction than transformational leadership. Greater passive-avoidant leadership predicted failure to sustain. For every one unit increase in passive-avoidant leadership we expect a .395 decrease in the log odds of moving from a given level of sustainment. The log ratio for passive-avoidant leadership indicates that for a one unit increase in leadership competence, the odds of the non-sustainment and partial sustainment versus the full sustainment are .609 times smaller. The same decrease, .609 times, is found between non-sustainment and the combined categories of partial and full sustainment. Passive-avoidant leadership and covariates accounted for approximately 61% of the variance in sustainment (R²=.608).

The relationship between transactional leadership and sustainment was marginally supported (β=−.143, p =.05). Therefore, it is expected that for every one unit increase in transactional leadership there is a .143 decrease in the log odds of moving from a given level of sustainment. The log ratio for transactional leadership indicates that for a one unit increase in leadership competence, the odds of the non-sustainment and partial sustainment versus the full sustainment are .229 times smaller. The same decrease, .229 times, is found between non-sustainment and the combined categories of partial and full sustainment. Transactional leadership and covariates in the model accounted for approximately 45% of the variance in sustainment (R²=.447).

Qualitative Results

Outer Context System-level Leadership

Three primary themes were identified in the system-level/outer context qualitative data: ongoing championing of EBIs and SafeCare, institutionalizing SafeCare in the service system, and collaboration at the system level.

Ongoing championing of EBIs and SafeCare

In sustaining sites, participants perceived their state and county administrators as “supportive,” “believing in,” and “in favor” of EBIs and SC. Similarly, administrators in these sites were also aware of their responsibility to serve as “champions” for the intervention. For example, when asked about his/her leadership role, one county administrator answered, “We still need to maintain a steadfast valuing of evidence based practices and SafeCare in particular.” During the sustainment phase, championing SC in these sites took two important forms. First, system-level leaders reportedly worked to continually sell the EBI to decision-makers. When asked about key leaders locally, one CBO director pointed to two county administrators who “have access to those decision makers even outside of themselves that we don’t have access to.” This director described the importance of these administrators’ work: “You’ve really got to have champions that can help communicate that [SC] is worth the investment, it’s worth the time, [and] it’s worth having to wrangle all these different partners. They’re the face of that message.” Secondly, system-level leaders continued to champion SC to providers. One state administrator emphasized, “We just make it very clear to staff that this [SC] is what you have to do. It’s our evidence-based practice, and we stand by it.”

The ongoing championing of SC at the system level was especially significant in contexts of turnover and system change. In one sustaining site, CBO directors reflected positively on the fact that during “multiple shifts in administration,” there was “definitely a shared vision” maintained in the service system. In contrast, participants in other sites expressed apprehension about the continuity of high-level commitment to SC in their systems. In a site that was having trouble sustaining SC, a retired county administrator reflected, “I don’t know whether or not [my successor] has the same passion in SafeCare that I had. It was new enough when I left that it really would have taken somebody who continued to push to make sure the providers were doing that evidence-based work and not going back to what we had been doing before.”

Institutionalizing SafeCare in the service system

In addition to functioning as SC champions into the sustainment phase, system-level leaders in the sustaining sites worked to strategically and proactively institutionalize the intervention locally. This typically took the form of making certain that adequate funding was available for SC. In nearly all of the study sites, participants viewed allocation of funding as a crucial part of system-level leadership. As explained by a CBO director from a sustaining site: “The moment [county officials] decide that they want to use that funding for something else then we have 17 [home visitors], 2 managers, and 3 office assistants who would not be funded.” In some sites, system-level leaders navigated disruptions in funding streams by allocating money from other flexible sources, thus ensuring continuity of service provision. This work involved thinking ahead and being proactive. One county administrator in a sustaining site described his/her leadership in this vein: “I am extraordinarily involved in the next five year planning cycle. I am ensuring our voice is being heard, advocating for continued services, providing data to show that it’s [SC’s] working.” Echoing her/his counterparts in other sustaining sites, this administrator added, “My ongoing role is to ensure that we have adequate funding [and] that our contractors [CBOs] are getting what they need.”

System-level leaders also worked to institutionalize SC by embedding the intervention within contracts and official plans. For example, in one sustaining site, county administrators wrote SC into a five-year system improvement plan. Here, an administrator clarified that this particular move protected the EBI from system changes within the foreseeable future: “It’s been institutionalized in that way and it’s been highlighted as one of the ways we’re going to improve child welfare outcomes. It doesn’t matter who sits in my position.” Similarly, administrators in another sustaining site wrote SC into contracts with CBOs.

Collaboration at the system level

Another key aspect of system-level leadership in the sustainment of SC was “collaboration” between county and state administrators and the CBOs tasked with delivering the EBI. One state administrator in a sustaining site attributed the success of SC in part to ongoing contractual relationships with local CBOs. Similarly, a county administrator in another sustaining site indicated that productive working relationships with such CBOs was a key to the sustainment of SC, in contrast to other sites where county-provider relationships were less strong. Despite the top-down nature of contracts between state/county governments and CBOs, trust, openness, respect, and the ability to agree among system-level stakeholders contributed to enduring relationships often characterized as collaborative. These relationships reportedly engendered continuing commitment to SC and a spirit of shared accountability, ensuring that neither state/county administrators nor CBOs put a premature end to SC. In this sustaining site, CBO directors described a sense of greater stability resulting from this collective responsibility, with one explaining that sustainment of SC was “not all incumbent on one [government] agency and we don’t just look to the county for everything.”

Inner Context: Frontline Leadership

Two primary themes were identified in the frontline-level/inner context qualitative data: ongoing championing of EBIs and SC and practical support for home visitors.

Ongoing championing of EBIs and SafeCare

Like system-level leaders, home visitors acknowledged that frontline supervisors also worked to champion SC in their individual service systems. In sustaining sites, home visitors described their supervisors as “gung ho,” and “wonderful cheerleaders. I mean they’re so into it [SC].” One home visitor said, “They make you want to come to SafeCare. They make you want to come to work.”

Supervisors also functioned as champions of SC in interactions with other child welfare stakeholders, especially government-employed social workers who interacted with many of the same families and agencies as the home visitors. In many of the focus groups, negotiating the referral process with social workers was portrayed as a problem for home visitors. Many home visitors felt that social workers and other government staff simply lacked knowledge of SC. However, the supervisors sought to address this issue through their efforts to advocate for SC. For example, one group of home visitors described their supervisor: “She’ll remind the social workers that it would be a great idea for SafeCare. Have you looked at SafeCare? Why isn’t she doing SafeCare? So just throwing the word out.” Supervisors were characterized as “running interference” or building relationships with government entities whose work sometimes conflicted with SC provision. Supervisors also reportedly acted as intermediaries for home visitors in dealing with state or county administrators. One participant commented, “If we have a problem with [the state agency], they’ll [the supervisor] say, ‘What caused the problem?’ They’re behind us every step. Or in front of us, whatever. They’re our blockade.” Supervisors thus championed SC and the work of SC providers at multiple levels of the service system.

Practical support for home visitors

The most common form of supervisor leadership identified by home visitors was the provision of practical support, including answering questions, giving advice, and finding resources. Home visitors indicated that their supervisors provided “fresh ideas” or “another set of eyes” on a client’s situation. They appreciated having them “just around the corner” in their office or available by “call, text, or email” to help them access resources, supplies, and information. One home visitor commented, “[The supervisors] are really good at finding resources. Anything that comes up, they’ll send it to you, they’ll look for it.” Supervisors’ ability to answer questions and meet needs was related to their knowledge of SC and of the realities of providing home visitation services. In particular, home visitors appreciated when their supervisors made the effort to be trained in SC or to learn about the EBI from them. Home visitors also indicated the importance of their supervisors’ awareness of what it was like to deliver SC and negotiate their different service systems on a daily basis. For example, in one focus group in a sustaining site, home visitors shared their conviction that, “I couldn’t get [guidance] from the higher ups. I think they were clueless. They sit at a desk all day and they maybe haven’t been in the field in 15, 20 years. Don’t talk to me if you haven’t been in there in five years.” Supervisors’ leadership thus most commonly took the form of practical guidance and support rooted in their own knowledge and experience.

Mixed-Methods Integration

We examined both convergence and expansion of the quantitative and qualitative data. Table 6 illustrates convergence and shows that both sets of results converged regarding conclusions drawn from the two methods. Both sets supported the importance of leadership in sustainment. Specifically, quantitative and qualitative results both affirmed the importance of outer context leadership in system level sustainment. Qualitative analyses identified leadership that is supportive, perseverant in the implementation process and that demonstrates that EBIs and SC are important in service delivery were associated with sustainment. Inner context leadership was also associated with sustainment. While transformational leadership was associated with sustainment, passive-avoidant leadership was associated with non-sustainment. Quantitative and qualitative data converged in that inner context transformational leadership supporting sustainment was evident in qualitative data where frontline supervisors were seen as champions, role modeling enthusiasm, engagement, and commitment to the EBI. While passive-avoidant leadership was associated with non-sustainment, qualitative data illustrated that leaders who were not passive and who were proactive, involved, and intervening as needed were identified as important in EBI sustainment.

Table 6.

Mixed method Results Demonstrating Convergence of Findings

Method	Quantitative	Qualitative
Question	Is Outer Context Leadership related to Sustainment?	Is Outer Context Leadership related to Sustainment?
Answer	Yes: the PSI-LC scores were associated with type of sustainment. Yes: higher PSI-LC scores were associated with greater sustainment	Yes: System and agency leadership was identified as important for SC sustainment.
Question	Is inner context leadership related to sustainment?	Is inner context leadership related to sustainment?
Answer	Yes: Transformational leadership was associated with greater sustainment Yes: Passive-avoidant leadership was associated with non-sustainment	Yes: Sustainment was associated with leaders who created a vision for the EBI and engaged their staff and created enthusiasm for the EBI Yes: Sustainment was associated with leaders who responded and fixed problems during implementation
Question	What types of outer context leadership are associated with sustainment?	What types of outer context leadership are associated with sustainment?
Answer	Leadership Competence included a number of actions including: establish mission and vision, early planning for sustainment, continued planning for sustainment, developing and followed a realistic project plan, and using multiple strategies for project survival.	Outer context leadership for sustainment was characterized as supportive, perseverant, and valuing EBIs and SC. Outer context leadership took steps to institutionalize SC in the system through funding, system improvement plans, and proactive planning.
Question	What types of inner context leadership are associated with sustainment	What types of inner context leadership are associated with sustainment?
Answer	Transformational leadership (creating vision, engaging staff, being a role model) was associated with greater likelihood of sustainment Passive-avoidant leadership was associated with non-sustainment. Transactional leadership was not associated with sustainment	Sustainment was associated with frontline supervisors being champions (creating vision), being enthusiastic and engaging staff, and role modeling commitment to the EBI. Proactive and involved leadership was associated with sustainment

Table 7 illustrates expansion of quantitative and qualitative methods where integrating those methods can provide additional depth of understanding. In regard to leadership findings across methods, quantitative data suggested that decision makers could provide funding, policies, and support for EBP, qualitative results expanded on this and described how leaders can not only create policies that provide funding, but also establish collaborations that support sustainment. Inner context role modeling, vision, engagement, and problem solving were important in quantitative analyses. Qualitative analyses expanded identifying the importance of leaders working on day-to-day issues that arose, but also being engaged in supporting providers in their EBI use. In regard to outer context leadership, constructs in the leadership measure addressed establishing mission and vision, planning for sustainment, realistic planning, and using multiple strategies for sustainment. Qualitative data identified the importance of valuing EBIs, providing supportive and perseverant leadership, while also institutionalizing the EBI in the system in multiple ways. Such strategies included establishing formal funding, system improvement plans, and proactive planning. For the inner context, high transformational leadership was associated with sustainment and passive-avoidant leadership was associated with non-sustainment. Qualitative data expanded on these constructs finding that being knowledgeable about the EBI, proactive in problem solving, and preserving through the ups-and-downs of sustainment was important. However, in contrast to the quantitative finding that transactional leadership was not associated with sustainment, qualitative data identified that it was important for frontline leaders to attend to how the EBI was being used, and intervene as needed when quality standards weren’t being met.

Table 7.

Mixed method Results Demonstrating Expansion of Findings

Method	Quantitative	Qualitative
Question	Is leadership similar across levels?	How does leadership differ across levels?
Answer	In the outer context, decision makers could provide funding, policies, and support. In the inner context, leaders role modeled, provided vision for the EBI, engaged staff in the EBI, and problem-solved.	In the outer context, leaders set the stage by creating policies and supporting funding, and creating the necessary collaborations for sustainment. In the inner context, leaders worked with day-to-day exigencies of engaging and supporting providers in delivering the EBI.
Question	What aspects of outer context leadership are related to sustainment?	What additional aspects of outer context leadership are evident in sustainment?
Answer	Leadership Competence included a number of actions including: establish mission and vision, early planning for sustainment, continued planning for sustainment, developing and followed a realistic project plan, and using multiple strategies for project survival.	Outer context leadership for sustainment was characterized as supportive, perseverant, and valuing EBIs and SC. Outer context leadership took steps to institutionalize SC in the system through funding, system improvement plans, and proactive planning.
Question	What aspects of inner context leadership are related to sustainment?	What additional aspects of inner context leadership are evident in sustainment?
Answer	Transformational leadership (creating vision, engaging staff, being a role model) was associated with greater likelihood of sustainment Passive-avoidant leadership was associated with non-sustainment. Transactional leadership was not associated with sustainment	Additional leadership characteristics were also congruent with recent conceptualizations of implementation leadership as being knowledgeable about EBIs, being proactive in problem solving, and persevering through the ups-and-downs of implementation and sustainment (Aarons, Ehrhart, & Farahnak, 2014). Sustainment was associated with frontline leaders attending to how the EBI was being used and intervened as needed.

Discussion

Both qualitative and quantitative data and analyses supported the role and importance of leadership for EBI sustainment. Quantitative analyses implicated the role of both system and frontline leadership in differentiating between sites that had not sustained, and those that had sustained. At the system level, leadership competence assessed in the EPIS implementation phase predicted differences between sustainment and non-sustainment in the EPIS sustainment phase. The impact of leadership on sustainment was striking in that there was a seventeen times increase of likelihood of sustainment for stronger levels of leadership. Leadership competence as measured by the PSI, has multiple aspects consistent with established leadership dimensions from the FRL model and from the Implementation Leadership Scale (ILS: Aarons, Ehrhart, & Farahnak, 2014). These include the degree to which leaders establish a project’s mission and vision (e.g., FRL transformational leadership dimension subscale inspirational motivation), engaged in early planning for sustainment (e.g., ILS proactive leadership from the ILS), continued planning for sustainment (e.g., ILS perseverant leadership), developed and followed a realistic project plan (e.g., ILS proactive leadership and knowledgeable leadership), and identified alternative strategies for project survival (e.g., ILS proactive leadership). The present study also adds to the literature on sustainment leadership and provides support for the predictive validity of the PSI in predicting sustainment.

At the inner context level both transformational leadership and passive-avoidant leadership were associated with sustainment or non-sustainment, respectively. Transactional leadership was at the near significant level in predicting sustainment. However this is consistent with other work in different settings that find stronger effects for transformational leadership (Spinelli, 2006). While we expected that transformational leadership would be associated with sustainment, it is notable that passive-avoidant leadership was strongly associated with non-sustainment. Our findings highlight how the failure of leaders to be engaged and active in EBI implementation and sustainment may lead to failures to sustain SC with fidelity. Consistent with emerging work regarding implementation leadership, leaders should be knowledgeable about the EBI or EBIs being implemented, proactive and perseverant in the sustainment process, and should be supportive of their staff members’ efforts to use EBIs.

The dimensions of the FRL are sensitive to context (Antonakis, Avolio, & Sivasubramaniam, 2003) so it is not surprising that the PSI-LC has some conceptual similarities to aspects of the FRL model. Indeed the FRL model has been shown to be applicable in large healthcare settings (Spinelli, 2006). However, the present study is unique in highlighting multilevel leadership in relation to sustainment reported for both the outer system context and inner context. It may be useful in future studies to examine settings where there are differing perspectives regarding leadership across levels. However, our results are consistent with conceptual frameworks and approaches that call for congruence or alignment of leadership activities across levels in order to create system and organizational context conducive to effective EBI implementation and sustainment (Aarons, Ehrhart, Farahnak, et al., 2014). Future research should delve more deeply into mechanisms by which system level leadership influences organizational and team level leadership as well as the potential for reciprocal influences within and across levels.

Qualitative Findings

Qualitative data and analyses also supported the importance of leadership in sustainment. While qualitative questions were general in nature, themes and dimensions of leadership that they helped identify were consistent with the FRL and implementation leadership research. In particular, participants identified “champions” of EBIs in general and SC in particular as vital to sustainment of SC at the system level. Some scholars suggest that context may trump leadership in understanding variability in leadership ratings and correlations with other measures (Hetland & Sandal, 2003). However, our findings support the conclusion that leadership is indeed important in EBI sustainment. Sustainment is a much more objective outcome relative to studies that rely on respondent ratings on multiple measures where common method variance can impact associations and conclusions (Podsakoff, MacKenzie, & Podsakoff, 2003). Consistent with implementation leadership theory, qualitative data also highlighted specific characteristics of leaders at the system and frontline levels, including being proactive, perseverant, knowledgeable, and supportive during implementation, while remaining mindful of funding considerations that could impact provision of the EBI. Greater attention to leadership and evidence-based leadership training is warranted in large system-wide implementation efforts to best realize the benefits for EBIs for children and families.

Mixed-Methods Integration

Both quantitative and qualitative methods contributed to improving our understanding of how leaders relates to sustainment. However, the qualitative methods not only corroborated quantitative findings, but added a more nuanced and expanded perspective on the types of leadership in the outer and inner context, that can support sustained use of EBIs.

For convergence we found that quantitative and qualitative results did evidence a number of consistencies regarding the importance of leadership in sustainment. This was apparent in that leadership was important in both the outer system context and the inner team/organizational context. In addition the types of leadership important for sustainment had to do not only with creating a vision and mission at system and team levels, but also backing this up with the appropriate structures and processes including early planning, realistic planning, and using multiple strategies for project survival.

In regard to expansion, there were some additional insights garnered from the mixed-method approach. For example, there were very real differences in the scope and types of influence across levels. As noted above, decision makers could provide funding, policies, and support for EBP, and qualitative results expanded describing how leaders can not only create policies that provide funding, but also establish collaborations that support sustainment. Qualitative analyses enhanced a more nuanced understanding of the importance of leaders working on day-to-day issues that arose, but also being consistently available and engaged in supporting providers in their EBI use. Thus, there are some commonalities in the types of leadership across outer and inner context, but these may manifest differently in keeping with roles and responsibilities at different levels. Consistent with emerging work on implementation leadership, qualitative data identified the importance of leaders being knowledgeable about the EBI, proactive in problem solving, and preserving through the ups-and-downs of sustainment (Aarons, Ehrhart, & Farahnak, 2014). One key difference in quantitative and qualitative results was that quantitative results found that transactional leadership was not associated with sustainment but qualitative data identified that it was important for frontline leaders to attend to how the EBI was being used, and intervene as needed when quality standards weren’t being met. This qualitative finding is very consistent with the transactional leadership dimension of “active management by exception” where leaders attend to performance standards and provide corrective guidance. However, too much, or an overly harsh approach to management by exception can result in employee dissatisfaction and negative responses to new initiatives such as EBI implementation. For the provider workforce, the implementation of an appropriate EBI along with supportive fidelity coaching can lead to positive outcomes including lower emotional exhaustion and higher staff retention (Aarons, Fettes, Flores, & Sommerfeld, 2009; Aarons, Sommerfeld, Hecht, Silovsky, & Chaffin, 2009).

Implications

Higher-level leaders in systems and organizations should attend to how leadership is being utilized. For example, research suggests that lower- and middle-level leaders who do not support a change initiated by their superiors may use their leadership skills to impede the implementation process (Conger & Kanungo, 1988; Guth & Macmillan, 1986; Rogers & Farson, 1955). Thus, it is important to consider strategies to support the development of effective leaders and congruence of leadership and communications across levels so that work group leaders can provide optimal support to their employees in implementing and using EBI. Although most leadership research has focused on individual leaders, studies have demonstrated the importance of alignment across multiple levels of leadership (Hunt, 1991; O’Reilly, Caldwell, Chatman, Lapiz, & Self, 2010; Wooldridge & Floyd, 1990). At the system level, Chreim and colleagues (2012) examined the factors that influenced implementation processes during the transformation of health care service delivery to a new model within one Canadian province. They found that implementation was propelled by fostering agreement, active participation, commitment, and congruence of support at all levels of leadership. At the work group level, the degree to which providers agree about the strategy or change being implemented predicts implementation success (Stagner, 1969). Similarly, the aggregate of multiple levels of leadership predicts organizational outcomes as a function of strategic implementation efforts (O’Reilly et al., 2010). We propose that such leadership congruence is effective because it sends a clear message about the importance of EBI and facilitates a positive implementation climate among stakeholders. This should be examined where similar measures can be collected across outer context and inner context settings.

Finally, leaders at system, organization, and team levels should consider the use of “climate embedding mechanisms”, or actions that leaders can take that signal their support and the importance for a strategic initiative (Aarons, Ehrhart, Farahnak, et al., 2014; Schein, 2010). It generally requires a number of coordinated and concerted approaches to support the effective implementation and sustainment of new initiatives. The results presented here demonstrate that it is not only leadership style or leadership behavior, but how leaders at different levels coordinate, collaborate, and lead their systems or teams to actively support EBI implementation and sustainment. Such multi-level influence can lead to positive system and team outcomes that results in benefits to the patients and clients who most need effective and evidence-based health and allied health services.

Limitations

Some limitations of the present study should be noted. First, our measures of leadership were different for outer and inner contexts. However, this allowed us to assess leadership most pertinent to those levels. Second, sample size for quantitative analyses was small for some groups and this likely limited finding statistically significant results. However, sample size and number of organizational units (e.g., service teams) are common challenges in studies of leadership, particularly as higher organizational or system levels are studied (Klein, Dansereau, & Hall, 1994; Klein & Kozlowski, 2000). Mixed-methods (quantitative-qualitative) can help to mitigate these concerns when triangulation can be used to corroborate findings and quantitative and qualitative data are integrated to answer key implementation questions (Aarons, Fettes, et al., 2012). Third, interviews and focus groups covered a broad range of issues related to factors impacting EBI sustainment. More targeted assessment tied to quantitative measures may have provided greater convergence of findings.

Conclusions

Consistent with the EPIS implementation framework, this study demonstrates that leadership at both the outer system and inner organizational contexts are important in EBI sustainment. Future research should focus on identifying ways to improve leadership during system and organizational change. For example, system leadership and managers could receive training and coaching in order to improve critical leadership knowledge and skills. In addition, leadership training could include an emphasis on creating a positive climate for implementation (Ehrhart, Aarons, & Farahnak, 2014; Jacobs, Weiner, & Bunger, 2014; Klein & Sorra, 1996) and alignment across outer and inner context settings (Aarons, Ehrhart, Farahnak, et al., 2014). Such approaches hold promise to improve EBI implementation and sustainment and quality and outcomes of care.

Table 4.

Ordinal Logistic Regression Analysis of Effects of Leadership Competence on Sustainment Controlling for Time from Implementation and Service System Size

Independent variable	B	SE	z ratio	p-value	Odds Ratio
Leadership Competence	.697	.142	4.911	.000	17.167
Time (months)	.338	.214	1.577	.115	1.025
Population	.112	.200	0.560	.575	1.011

Note: N=44; Sustainment coded as 0=non-sustainment, 1=partial sustainment, 2=full sustainment; Time (months) indicates time in months since SafeCare® implementation for each service system; Population indicates population size for service system state or county per US Census data.