The good coach: implementation and sustainment factors that affect coaching as evidence-based intervention fidelity support

Abstract

Purpose:

Evidence-based interventions (EBIs) for human services unfold within complicated social and organizational circumstances and are influenced by the attitudes and behaviors of diverse stakeholders situated within these environments. Coaching is commonly regarded as an effective strategy to support service providers in delivering EBIs and attaining high levels of fidelity over time. The purpose of this paper is to address a lacuna in research examining the factors influencing coaching, an important EBI support component.

Methodology:

The authors use the Exploration, Preparation, Implementation, and Sustainment framework to consider inner- and outer-context factors that affect coaching over time. This case study of coaching draws from a larger qualitative data set from three iterative investigations of implementation and sustainment of a home visitation program, SafeCare®. SafeCare is an EBI designed to reduce child neglect.

Findings:

The authors elaborate on six major categories of findings derived from an iterative data coding and analysis process: perceptions of “good” and “bad” coaches by system sustainment status; coach as peer; in-house coaching capacity; intervention developer requirements vs other outer-context needs; outer- context support; and inner-context support.

Practical implications:

Coaching is considered a key component for effective implementation of EBIs in public-sector systems, yet is under-studied. Understanding inner- and outer-context factors illuminates the ways they affect the capacity of coaches to support service delivery.

Originality:

This paper demonstrates that coaching can accomplish more than provision of EBI fidelity support. Stakeholders characterized coaches as operating as boundary spanners who link inner and outer contexts to enable EBI implementation and sustainment.

1. Introduction

Implementation of evidence-based interventions (EBIs)—health or human service interventions supported by rigorous scientific research—represents an increasingly accepted strategy for improving the wellbeing of families at risk for child maltreatment (Novins et al., 2013). However, it has long been known that the implementation of EBIs within day-to-day service delivery milieus is often inconsistent and ineffective, underscoring the continued challenges of transferring science to service (Horwitz et al., 2010, Novins et al., 2013). Researchers consider the implementation of innovative human service technologies, such as EBIs, to be more complicated than other types of technology, because they are delivered by individuals and organizations within complex, multi-layered social contexts (Aarons et al., 2011, Fixsen et al., 2009). In this paper, we consider how systematic “consultation,” or “coaching,” can assist providers and organizations in implementing and sustaining EBIs.

Our case study of coaching, defined here as ongoing support, or technical assistance, from a specialist to improve EBI implementation after training (Beidas et al., 2013, Duda and Barrett, 2013, Kucharczyk et al., 2012), draws from a larger qualitative dataset documenting implementation and sustainment of a home visitation program, SafeCare^®, an EBI to reduce child neglect (Chaffin et al., 2012b, Gershater-Molko et al., 2003, Whitaker et al., 2012). Interviews and focus groups conducted with government administrators and the staff of community-based organizations (CBOs) contracted to deliver SafeCare^® in 11 United States human service (e.g., child welfare) systems. We examine how stakeholder perceptions and experiences of coaching changed during different phases of implementation, describe the significance of these changes, and discuss factors influencing coaching support in these systems.

Studies show that while training alone (i.e. “one-shot workshops”) is unlikely to result in practice change, let alone fidelity to an EBI (Odom, 2009), there are strategies to improve adherence to EBIs that range from didactic methods (Perepletchikova et al., 2007) to systematic review of care such as audit and feedback (Foy et al., 2005, Ivers et al., 2012), to observations performed in vivo or through recordings or one-way mirrors (Forgatch et al., 2005, Sheidow et al., 2008). Such strategies vary in effectiveness (Forgatch et al., 2005, Ivers et al., 2012). Studies of audit and feedback suggest limited evidence on reliability and how to best employ this approach, although intensive feedback appears to enhance effectiveness (Jamtvedt et al., 2006, Foy et al., 2005). Observation can better assess fidelity than self-report; continuous supervision, role plays, and periodic booster trainings may also reduce deviation from intervention protocols (Perepletchikova et al., 2007). Although potentially more costly, labor-intensive, and intrusive to practitioner and client than other strategies, observation as part of fidelity monitoring can contribute to greater adherence to and positive outcomes for home visitation interventions (Forgatch et al., 2005, Weiss et al., 2006).

Coaching is an implementation strategy that may involve in vivo observation, intensive feedback, and other methods described above to support use of and fidelity to EBIs, including SafeCare (Chaffin et al., 2012a). Researchers point to the critical role that coaching can play in implementing and sustaining EBIs with fidelity by ensuring ongoing support after initial training (Duda and Barrett, 2013, Edmunds et al., 2013, Fixsen et al., 2009). Coaches supply implementation support, for example, with therapists delivering cognitive-behavioral therapy to youth (Beidas et al., 2013), by the National Professional Development Center on Autism Spectrum Disorders for improved EBI provision to students with autism (Kucharczyk et al., 2012), and for preschool teachers to promote school readiness for at risk students with the Children’s School Success Project (Odom, 2009, Odom et al., 2010). Recognized characteristics of effective coaches include abilities both to negotiate diverse stakeholder needs in hierarchically-structured organizations and service systems (Duda and Barrett, 2013), and to cultivate the attitudes and behaviors among frontline workers that are most likely to result in successful implementation (Kucharczyk et al., 2012). Adequately trained and supported coaches can build trust, develop productive relationships, and create accountability for EBI implementation outcomes (Duda and Barrett, 2013).

Although researchers suggest that coaching improves fidelity to EBIs, the specific strategies and interactions by which coaching results in such desired outcomes is unclear (Beidas et al., 2013). EBIs are only as good as the support systems in place to ensure their integration into practice (Odom, 2009), and more research is needed to shed light on the nuances of implementing these systems (Fixsen et al., 2009, Fixsen et al., 2005). Our study offers a unique opportunity to examine coaching as a support system by exploring the situated perspectives of multiple stakeholders implementing the same EBI within varied service systems.

1.1. Conceptual Framework

Several frameworks define implementation as a complicated process involving a series of stages and factors at multiple levels, i.e., system, organization, provider, and client (Aarons et al., 2011, Damschroder et al., 2009, Meyers et al., 2012). The Exploration, Preparation, Implementation, and Sustainment (EPIS) model (Figure 1), developed for public service settings (e.g. child welfare, mental health care, substance abuse treatment), segments processes of instantiating an intervention into four phases: Exploration (consideration of new innovations, evidence, and “fit” within the service system), Preparation (planning for innovation implementation), Implementation (training in and provision), and Sustainment (maintaining with fidelity).

Figure 1: EPIS implementation framework.

Note: This figure depicts some examples of factors in the outer and inner contexts to be considered in each phase of the EPIS framework

Source: Adapted from Aarons et al. (2011)

The EPIS model applies the analytic categories of “inner context” and “outer context” to illuminate the factors that can impact implementation and effectiveness of coaching across phases. Inner-context factors related to coaching include organizational-level variables, such as CBO leadership, organizational culture and climate, job demands, and workforce retention, as well as provider-level factors, such as disposition toward EBIs, work stress, and job autonomy (Novins et al., 2013). Outer-context factors are pertinent to the system level of the broader environment in which organizations operate (Aarons et al., 2011, Damschroder et al., 2009). A primary outer-context variable that may shape coaching is leadership in government-administered human service systems, which may determine the nature of contracts, allocation of training, coaching, or other support systems, and interorganizational relationships involving the sharing of resources (e.g., training and coaching) among CBOs.

Through this case study on the instantiation of a single EBI, we found that coaches play an important role in implementation because their activities span inner and outer contexts, influencing how system-level ideas and processes concerning EBI fidelity are translated and operationalized locally. Coaches are answerable to both frontline workers and CBO management in the inner context, and government administrators in the outer context. The present study elaborates the range of inner- and outer-context factors that can impinge on coaching and its effectiveness over time, adding to our understanding of how coaching can support the implementation and sustainment of EBIs more broadly within human service systems.

2. Methodology

2.1. Study context

This examination of coaching processes utilizes secondary analysis of qualitative data collected during three iterative mixed-method studies of SafeCare implementation and sustainment in one state-operated and ten county-operated human service systems two states (Aarons et al., 2014c, Aarons et al., 2012, Aarons et al., 2009, Green et al., 2016, Willging et al., 2015a). In most systems, state or county government agencies contracted with CBOs to deliver SafeCare, specifying both staffing and reporting requirements. SafeCare addresses vulnerabilities in families at risk or reported for child maltreatment by enhancing the parenting skills of primary caregivers. The contracts required CBOs to deliver three SafeCare modules on child health, home safety, and parent-child or parent-infant interaction. Staff called “home visitors,” typically case managers responsible for working directly with families, took part in a one-week training to administer the modules to caregivers through explanation, assigned homework, role-play, and hands-on demonstrations.

Coaching is a key feature of SafeCare and other EBIs to ensure the intervention is delivered in the way research has found to be effective (McHugh and Barlow, 2010). Coaching for SafeCare is modeled on tenets of effective supervision and a developmental consultation approach (Stoltenberg, 2005, Stoltenberg and Delworth, 1987), and is required by the National SafeCare Training and Research Center (NSTRC), the entity that developed SafeCare. Coaches are the “support and fidelity monitoring system” for home visitors (Aarons et al., 2009). Per the NSTRC’s coaching manual, coaching is a “collaborative process” through which coaches help home visitors refine their skills as they learn how to provide parenting feedback to families. Previous research on SafeCare found that in vivo coaching for fidelity monitoring can function as quality control and enhance the working alliances between home visitors and client (Chaffin et al., 2012b).

In all study systems, a subset of home visitors was trained to become coaches and tasked with accompanying other home visitors to see clients monthly, using a checklist to document adherence to the core components of each module. Thus, coaches were responsible for assessing fidelity and offering technical support. Each system designated a “lead coach” who communicated with the NSTRC, taught and certified coaches and home visitors in SafeCare, and provided implementation oversight. Coaches carried caseloads as home visitors and therefore also received coaching. In several service systems, coaches convened as a group and with a lead coach for continued support. Some CBOs relied on personnel from other CBOs to coach home visitors. During the Preparation and Implementation phases, the NSTRC also made their trainers and coaches available to support home visitors locally, and helped CBO administrators select trainees to become internal coaches and trainers. Table 1 describes these roles.

Table 1:

Job Roles

	Personnel	Function
CBO administrators	Executive directors, area directors, program managers	Oversee the general operations of CBOs, but are not directly involved in the day-to-day implementation of the EBI; provide administrative support to the agency and the EBI
CBO supervisors	Team leader, frontline supervisor, clinical supervisor	Responsible for general clinical and workplace management of home visitor teams. Some supervisors additionally serve multiple EBI-related roles as trainers and coaches
SafeCare trainers or lead coaches	Persons trained and certified to train others in delivering the EBI. In the preparation phase, trainers were employed by the intervention developers rather than within the service systems	Train home visitors in EBI practices during a 5-day workshop. Trainers or local lead coaches also coach home visitors after initial training. A minority of coaches are certified as trainers
SafeCare coaches	Individuals trained and certified to coach others in delivering the EBI. Coaches are frontline supervisors or other direct service staff. In the preparation phase, intervention developer coaches were employed to start and scale-up the EBI until local coaches became certified	Conduct monthly shadowing visits of each home visitor to monitor model fidelity and provide supportive coaching. The coaches serve as expert guides for home visitors for asking questions related to the EBI provision to clients
SafeCare home visitors	Direct in-home service staff trained in the EBI	Home visitors provide in-home EBI services to families

2.2. Sample

For more than ten years, we conducted individual semi-structured interviews (n=166), small group interviews with an average of three participants (n=13), and focus groups with an average of six participants (n=80) with a wide range of stakeholders in all 11sevice systems. System-level stakeholders involved in planning and managing services included government administrators (e.g., directors of child welfare agencies), CBO administrators (e.g., executive directors and program managers), and funders (e.g., directors of funding agencies). Stakeholders at the frontline were clinical supervisors of home visitation staff, SafeCare coaches, and home visitors. As shown in Table 1, coaches were distinct from clinical supervisors, who oversaw the cases and overall wellbeing of families, but were usually not SafeCare trained and certified. However, four of the systems had individuals who performed the roles of both supervisor and coach. Most participants were women, particularly among the frontline workers, as is common in helping professions (Willging et al., 2017). Participants generally reflected the cultural demographics of workers and the service populations in their respective systems. Ages ranged widely, and most held at least a bachelor’s degree, with master’s degrees more common among supervisors and higher-level administrators.

For this analysis, we classified each system by sustainment status as recommended by Stirman and colleagues (2012): “full” (n=7); “partial” (n=1); and “non-” (n=3). In full-sustainment systems, core implementation elements of SafeCare, i.e. ongoing delivery of modules, monthly coaching, and weekly team meetings, were maintained after initial implementation support had been withdrawn, and adequate capacity existed to maintain these elements. Partial sustainment described systems meeting only some elements after withdrawal of initial implementation guidance (e.g., service delivery continued but the system did not conduct model-required coaching sessions). In non-sustaining systems, the EBI was no longer being provided by the system.

2.3. Data Collection and Analysis

Data were collected at three time points: Time 1 (T1; the Preparation/Implementation phase; 2006–2008), Time 2 (T2; the Implementation phase; 2009–2011), and Time 3 (T3; the Sustainment phase; 2012–2014). Data were collected in at least one system each year across all three time periods. Individual systems began implementation at different times or had divergent trajectories of implementation; the bulk of data collection occurred in T3 when the service systems had all been implementing the EBI for a minimum of two years. During T3, all coaching needs were being addressed by local stakeholders. Several stakeholders with long tenures in their positions were interviewed multiple times across time periods.

Interview and focus group guides in T1, T2, and T3 examined inner- and outer-context factors related to the Preparation, Implementation, and Sustainment phases of SafeCare in each service system, including the role of coaches and the influence of coaching on implementation and sustainment. Separate guides were developed for system and organizational stakeholders. Questions in T1 guides related to decision-making to implement the EBI and first impressions of the EBI; T2 questions concerned impressions of training, coaching, and impacts of EBI implementation on systems, CBOs, and staff; and T3 questions centered on the role of leadership and collaboration in EBI implementation and sustainment, and experiences delivering and coaching the EBI. The University of California, San Diego Institutional Review Board approved the research design, sampling method, and consent procedures.

All interviews and focus groups were digitally recorded, professionally transcribed, and checked for accuracy by at least one author. We used NVivo 10 qualitative data analysis software (QSR International, 2012) for data management and to support iterative review and analysis. First, transcripts for each study in our database were coded by research assistants to condense the data into analyzable units. Codes were assigned segments of text ranging from a phrase to several paragraphs based a priori on the topic areas and questions making up the interview guides (Patton, 2015). For the secondary analysis, we re-reviewed and recoded the transcripts for content specific to coaches and coaching. As part of this secondary analysis, we created additional codes based on key sensitizing concepts from the implementation literature, including coaching or consultation, (clinical) supervision, and support. These concepts provided “a general sense of reference” for the secondary analysis and supplied descriptive data based in the actual words of research participants, which enabled us to examine their salience and meaning for different types of stakeholders over time (Patton, 2015 p. 545). We then used open and focused coding to locate new issues related to coaching in the transcripts (e.g., trainers, implementation support, scheduling coach visits) and to determine which issues emerged frequently or represented particular concerns of the stakeholders (Corbin and Strauss, 2008). By comparing and contrasting codes in both the primary and secondary analyses of the data, we grouped codes with similar content into broad categories of findings linked to segments of text in our database (Corbin and Strauss, 2008, Glaser and Strauss, 1967). Within this database, we also incorporated detailed memos describing and linking codes to the larger categories which, in turn, illuminated the role of coaches in implementing and sustaining the EBI and how different types of stakeholders perceived and experienced coaching. Finally, accuracy checks of the findings and assistance with interpreting data were provided by persons involved in organizing and implementing the coach role in study service systems (Patton, 2015).

3. Results

The six major categories of findings are described below in a format that tells the story of SafeCare implementation across important EPIS phases, and in a way that underscores their connectedness: perceptions of “good” and “bad” coaches by system sustainment status; coach as peer; in-house coaching capacity; intervention developer requirements versus other outer-context needs; outer-context support; and inner-context support. Table 2 organizes results by sustainment status and EPIS phase to highlight how inner- and outer-context factors influenced coaching over time.

Table 2:

Coaching summary by sustainment status and EPIS phase

System	Preparation (data from T1)	Implementation (data from T2 and T3)	Sustainment (data from T2 and T3)
Fully sustaining (n=7)	Inner: CBO administrators, home visitors, and some supervisors learn about implementation and the coach role. CBOs experience turnover of veteran case workers in the transition to the EBI. Some home visitors express confusion about the coach’s advisory (vs supervisory) role. Some CBO staff are unhappy with not having an in-house coach Outer: systems initiate EBI with external coaching and transition to their local team post-certification. Some external trainers are characterized by CBO staff as difficult, to the point of threatening EBI continuation	Inner: CBO administrators make changes to better support coaches (e.g. mandatory consultation) and increase staff buy-in. Staff opinions of coaches improve but remain mixed. Scheduling challenges around coaching visits abound. Tensions exist between coaches and clinical supervisors regarding the coach role. Coaches depend on a lead coach for support Outer: communication issues with coaches and about coaching begin to resolve. Challenges of integrating coaching and the EBI into systems remain. Some systems expand coaching capacity as program grows	Inner: coaches are generally integrated and valued throughout systems as part of quality assurance. Coaches maintain that it is awkward to coach peers and some supervisors remain confused about coaching role. Coaches continue to rely on lead coach Outer: government administrators appreciate coaches as providing built-in accountability, allowing them to focus on other work. Funding for coaching and ongoing trainings is a universal concern. Turnover in leadership requires some coaches to act as “boundary spanners”
Partial sustaining (n=1)	Inner: CBO administrators, home visitors, and some supervisors learn about implementation and the coach role. There is discomfort at all levels with the idea of having a peer “coach” the other providers Outer: external trainers are not well received	Inner: as with sustaining and non-sustaining systems, coaches claim discomfort coaching peers Outer: the system loses one coach, leaving only the primary lead coach who refuses to travel to coach home visitors who are not nearby, leaving these more remote CBO staff without consistent coaching	Inner: due to the lack of consistent coaching, home visitors are not implementing the EBI with full fidelity Outer: little support from government administrators in sharing resources after the loss of one coach
Non-sustaining (n=3)	Inner: stakeholders learn about the implementation and the coach role. Staff express some resistance to the EBI Outer: some systems do not have local coaches and rely on external coaches. Multiple turnovers of lead coach position result in sporadic coaching in two systems	Inner: coaching is inconsistent due to turnover and perception among home visitors that external coaching is unhelpful Outer: systems insufficiently or inappropriately support coach or EBI due to financial and/or organizational issues. EBI implementation is negatively impacted by insufficient access to coaches	Inner: the EBI is no longer being implemented. CBO administrators note that adding the EBI to staff’s existing responsibilities posed challenges Outer: government administrators state they should have written the contract with specific EBI needs and language in mind

3.1. Perceptions of “good” and “bad” coaches by system sustainment status

Several characteristics of a “good” coach were identified during Implementation by stakeholders in sustaining and partial-sustaining service systems. Inner-context stakeholders deemed coaches effective when they validated and encouraged the work of home visitors and helped them advance their skills, often by modeling correct techniques: “They’re able to guide the staff in a way that supervisors aren’t,” stated one CBO administrator. Home visitors added that feedback was detailed and framed in a positive light. Stakeholders concurred that good coaches were “available,” “accessible,” and skilled at building “rapport” with clients and home visitors, paralleling techniques taught to home visitors in training, and as designed by the intervention developers. Describing her/his coach, a home visitor said, “[S/he] builds on [your] strengths so much and gets a twist on things you still have to work on.” Home visitors and supervisors, especially at the onset of implementation, also valued coaches for offering useful suggestions not covered within the EBI materials, such as information about autism and domestic violence resources. Supervisors in sustaining systems clarified that good coaches possessed excellent organizational skills and could keep track of complex schedules and great quantities of paperwork.

Coaches in sustaining and partial-sustaining systems referred to themselves as “problem-solvers,” “mentors,” and “helpers.” In contrast, government and CBO administrators commonly characterized the coaches as the “experts” charged with learning and teaching the EBI. This difference in understanding of the coaches’ role between inner- and outer-context stakeholders reflected the range of roles and responsibilities of participants. Home visitors and supervisors in the inner context depended on the coaches’ more quotidian technical support, while government administrators in the outer context emphasized the broader structure of implementation accountability that they perceived coaches as providing.

Stakeholders across service systems shared similar understandings of characteristics associated with a “bad” coach. While home visitors were trained to think of coaches as peers, a bad coach was perceived as overly directive and a face of authority. They described bad coaches as making critical remarks, or correcting them in the presence of clients. One home visitor in a sustaining system shared, “I was taught that [coaches] are supposed to help us and to teach us and to support us, but I get criticized… And in front of the clients, that’s even worse.” Another added that the occurrence of such scenarios causes clients to think, “This case manager doesn’t know how.” On the other hand, two home visitors agreed that while their coach was “energetic” about SafeCare, “[S/he] just displayed some really bad boundaries with my client.” A bad coach was also described as not available for or late to home visits, or as taking calls on their cell phone in the presence of clients. Bad coaches reportedly did not provide much if any feedback, leaving home visitors insecure about their fidelity ratings and their implementation ability and competence. A supervisor observed that when the coaches were “absent,” or not reachable on the phone or in the office, the home visitors were unhappy with them.

Home visitors in non-sustaining systems experienced inconsistent and infrequent coaching, often due to turnover, a lack of in-house coaches to provide in-person support (see 3.3 below), and/or poor placement of individuals in coach and lead coach positions. The CBO administrators in one non-sustaining system recalled the home visitors feeling as though an external trainer/coach had demeaned them in a way that impacted implementation negatively. Sustaining and partial-sustaining systems reported similar descriptions but overcame this conflict through assiduous effort to complete the EBI training and with the help of the other external trainers provided by the intervention developers. As shown in Table 2, reports of bad coaching diminished over time as home visitors, coaches, and supervisors grew more comfortable with the EBI and as administrators learned what qualities make for a good coach.

3.2. Coach as peer

The EBI’s coaching manual defined coaching as the “support and monitoring system” for home visitors delivering SafeCare, but initial trainers during the Preparation phase in the 11 systems emphasized coaches as complementary “peers,” engendering some confusion and discomfort among both coaches and home visitors. While coaches were trained to provide encouraging feedback, they were also charged with reviewing and addressing implementation fidelity, which heightened anxieties among home visitors that the coach would judge their work performance negatively. One coach in a partial-sustaining system explained:

We started [implementing the EBI] at the same time. I don’t know more than they do. They don’t know more than I do. I don’t feel like that’s a good or fair position to put somebody in…. We’re like equal coworkers so to be critiqued by your coworker is not a natural or comfortable situation.

Stakeholders in sustaining systems expressed similar perspectives. For example, one home visitor stated during Implementation, “I’m sure a part of it is evaluating,” In a second system, two supervisors confessed to calling coaches “monitors” among themselves and said they only use the term “coach” when in conversation with home visitors. A CBO administrator described coaching during Implementation as “a role that’s a little bit confusing because the person is seen monitoring the work of their peers and yet they’re not in a supervisory position.” Stakeholders in non-sustaining systems did not reach an EPIS phase to comment extensively on coaches as peers.

This lack of clarity and consistency regarding the coaches’ supportive vs. supervisory role was also commonly expressed by outer-context stakeholders. Some government administrators who lacked familiarity with the EBI treated, used, or talked about coaches as monitors. Government administrators who were not part of the Exploration and Preparation phases even sought to leverage the coach role to monitor the CBO home visitors and their billable hours. The coaches’ supervisors in one service system reportedly called a meeting and explained to the government administrators that coaches were not “supervisors or watchdogs.” One government administrator even wanted to become a coach specifically to monitor staff fidelity to the EBI. In these cases, outer-context stakeholders misunderstood or sought to recast the coaches’ supportive role in the inner context.

For their part, coaches were generally concerned with not looking like the “bad guy” and consciously endeavored to frame their comments to home visitors positively; those who had to coach supervisors especially expressed discomfort. Many felt more comfortable coaching home visitors from outside of their training cohort. Confusion about the coach role was most ubiquitous during the Preparation and Implementation phases and more confined to new hires in the Sustainment phase as stakeholders grew accustomed to coaching as part of service delivery and quality assurance.

3.3. In-house coaching capacity

Over the EPIS phases, it became clear in sustaining and partial-sustaining systems that home visitors benefitted from having their coaches housed within their own workplaces, rather than in another, possibly geographically distant, CBO. One home visitors remarked, “I do go to my coach when [s/he’s] there, but do I call to ask [her/him] something? No, I don’t. I call [the lead coach] or when I see the other coaches in the office, I ask them.” A lead coach in one system noted that, early in Implementation, “turf wars” for referrals surfaced because coaches were only based in a single CBO, leaving some home visitors to believe that those with in-house coaches were favored within the system. According to this individual, tensions were reduced among CBO staff once coaches were finally housed in multiple CBOs.

Similarly, a government administrator in another system with geographically dispersed coaches wondered if the system had enough coaches while entering the Sustainment phase and planned to ask the intervention developers for help. The participants in a focus group of coaches in this system agreed that it was harder to coach geographically dispersed home visitors, “We don’t really have that closeness with their management over there. We don’t have that much communication.” A government administrator in another sustaining system claimed that system stakeholders would not have agreed to do SafeCare without a local lead coach.

A minority of staff appreciated the opportunity to interact with coaches outside their CBOs, or to travel to new localities to perform coaching. For example, one home visitor said s/he liked having a coach from another CBO because the coach provided her/him with fresh ideas.

3.4. Intervention developer requirements versus other outer-context needs

Although government and CBO administrators were generally supportive of coaching, challenges in maintaining coaching capacity arose over time in many service systems. In these cases, coaches were affected by the unintended consequences of policies, procedures, and funding arrangements originating in the outer context. For example, one CBO administrator instituted mandatory consultation times as some home visitors were not “eager” to schedule them. Government administrators in another system established a rule that home visitors must arrange coaching visits by the 15^th of each month, to ameliorate rampant scheduling challenges. Coaches in this system said that because they had to report those home visitors who did not schedule on time, resentment ensued among those who missed deadlines, undermining what was meant to be a supportive relationship. Home visitors also felt unfairly blamed for scheduling issues reportedly resulting from a reduction in coaching hours, which they believed were decreased by government administrators for financial reasons. Coaching became less consistent because of decreased hours, and coach participation in trainings for new home visitors was also limited to save money, both of which fed into contradictory messages about the value accorded to coaching. The coaches believed that they no longer bonded as before with home visitors, nor were they privy to how the training information was presented, hindering their ability to help home visitors.

In one system, the inability to maintain coaching in compliance with EBI requirements contributed to SafeCare’s failure to progress beyond Implementation. Here, CBO administrators had trouble reconciling and integrating outer context reporting requirements prescribed by the funder, local government, and intervention developer. The inability to efficiently report and bill for coaching reportedly led to fewer clients, and therefore lower levels of—and often unproductive—coaching. The failure to resolve the compliance issues around coaching contributed to CBO administrators deciding not to sustain the program.

3.5. Outer-context support

Administrators in the outer context affected the level of support for coaches in critical ways. For example, coaches wanted administrators to watch out for the danger of allocating large caseloads to home visitors that threatened fidelity. A coach during the Sustainment phase asserted that, “If [home visitors] have any more than eight to ten cases I don’t feel that SafeCare is implemented in the way that it could be.” When asked about what was needed to sustain, one home visitor responded that they were dependent on “consultant and supervisor availability.” Finally, one lead coach asked to have her/his job requirements clarified and written down after experiencing greater and expanded responsibilities beyond what s/he perceived was appropriate.

Gaps in leadership in some systems caused by outer-context turnover during the Sustainment phase necessitated that coaches act as intermediaries between outer and inner contexts. Coaches said they had to communicate between the CBO home visitors and the government administrators to facilitate continued sustainment. Home visitors and other stakeholders in some systems considered their coaches to be liaisons to administrators, often underscoring the pressures placed on them to span inner and outer contexts. One coach explained,

Sometimes I feel like we’re that middleman where we don’t have enough say sometimes to do certain things, and then sometimes they ask us for too much to do. So it’s like we’re in that middle situation where we’re not their managers or their bosses but then sometimes they ask us questions that are for them.

Coaches benefited from having access to a coordinator dedicated to the SafeCare program, who simultaneously functioned as their lead coach; such individuals usually came from a supervisory administrative level. Coaches throughout Implementation and Sustainment phases characterized lead coaches as “key” to their ability to do their jobs. Government and CBO administrators also emphasized the importance of the lead coach during both phases, and some talked of having to replace an individual who was poorly-performing in this position with a more “appropriate” person for effective implementation to progress. Lead coaches with multiple responsibilities (e.g., supervision, home visiting) commented in the Sustainment phase that ideally they would be exclusively coordinating the program in their vicinity. After multiple turnovers in the lead coach position, a CBO administrator in what became a non-sustaining system suggested that the organization should have specifically recruited for, rather than assigned existing staff to, the lead coach role.

3.6. Inner-context support

In addition to outer-context support, coaches were affected by the level of support they received within the inner context. A CBO administrator described having to communicate sternly to staff during Implementation that their CBO was committed to EBIs,

[We had] a direct conversation, ‘We’ve gotta move forward.’ You know, ‘We’ve gotta get on board. What do we need to do?’ And so we changed how we were meeting with the consultant [coach]. We invited him to meetings. We changed our entire interactions…

The CBO administration concentrated on changes that would explicitly support the coach and thus, the implementation of the EBI.

Supervisors in the inner-context were particularly positioned to help or hinder coaches, according to lead coaches, coaches, and home visitors. Generally, supervisors were not trained in the EBI because administrators in both inner and outer contexts reasoned that supervisors could do their full-time administrative jobs without needing costly EBI-specific training. While some supervisors were trained with the intention of becoming coaches or attended portions of trainings when there was time or money for them to do so, most supervisors claimed that they did not have time to learn the model and that was a reason they valued the coaches. However, a lead coach said that it was the leadership of supervisors in the inner context that reinforced the coach role by helping home visitors understand and buy into the implementation:

It makes a difference when you have a supervisor who understands, and who lets you know that, “This is what you have to do.” So you have that support from the supervisor versus someone who just really doesn’t care. Then guess what? Your staff are not going to care either and they’re not going to do it and then that’s going to be my struggle.

Supervisors with a firm understanding of implementation processes greatly valued their coaches and how much the coaches helped them do their jobs better. Illustrating this point, one supervisor said that her/his peers were better able to oversee administrative procedures and could trust the coach to handle the EBI end of things.

However, supervisors also observed that coaches sometimes became tangled within a web of varied CBO policies and procedures when helping home visitors overcome implementation challenges. For example, there were instances when a coach suggested a solution that clashed with a CBO’s existing policies (e.g., no home visitations with families after dark). Supervisors in the Sustainment phase said they were especially careful to keep issues of SafeCare implementation separate from their other work with home visitors. One observed, “Mixed messages can happen, that’s why I never speak about SafeCare. I immediately refer them back to their coach.” Coaches in a sustaining system elsewhere also recognized this overlay, with one explaining, “It’s because [our CBO has its] own set of rules. The other visitors may not follow the same guidelines so sometimes we get caught up in their management.” In these cases, coaches, particularly those not working in-house, struggled to adhere to outer context requirements while managing inner context realities. Over time, in sustaining systems, some home visitors became supervisors drawing from their previous role, knowledge, and expertise to support the EBI. Lack of buy-in among some supervisors reduced support for coaches in the inner context of systems unable to sustain..

4. Discussion

This qualitative research benefited from a series of interrelated studies that made it possible to longitudinally investigate a multitude of factors affecting coaching, a core support component of SafeCare and other EBIs (McHugh and Barlow, 2010). Our findings clarify factors helping and hindering coaching support for EBI implementation and sustainment in complex human service systems. The findings suggest that coaches perceived as successful were those who effectively navigated and negotiated shifting inner and outer contexts throughout the Implementation phase and into the Sustainment phase.

In sustaining systems, coaches enabled both implementation and fidelity, despite changing procedures, paperwork, and personnel. Coaches helped home visitors incorporate these changes and providing a csource of constructive feedback and solutions to everyday challenges. However, our study also indicates that coaching can affect much more than intervention fidelity, as the efforts of coaches had implications beyond their defined role. Throughout the Implementation and Sustainment phases, for example, coaches were vital in communicating the intervention’s importance to home visitors, a notable aspect of organizational climate for EBI implementation (Aarons et al., 2014b). In prior research, coaching also contributed to staff support for SafeCare and reduced staff turnover in child welfare systems (Aarons et al., 2012).

Our findings suggest a strong connection between home visitors’ positive evaluations of their coaches and the ability of these coaches to communicate constructively with them. In fact, the coaches perceived as “good” were practicing the principles of the EBI itself, such as inviting the home visitor to instruct the coach on her/his preferences (e.g., if s/he preferred the coach to model EBI techniques, participate, or simply observe), using specific versus general praise, and giving rationales for criticism. By operationalizing the same techniques that home visitors were to use with clients, coaches engaged in favorable consultation practices, a finding resonating with other implementation research (Beidas et al., 2013, Stoltenberg, 2005).

However, coaches in our study felt constrained in cultivating communicative and trusting relationships with home visitors by outer-context factors, such as when funding for their work hours was cut and they were unable to participate with home visitors in trainings. Home visitors also reported that they were less able to form trusting and supportive relationships with coaches who were not based within their daily workspace. Coaches, in turn, perceived that home visitors who received coaching remotely or did not have coaches nearby did not receive adequate assistance and demonstrated lower levels of fidelity to the intervention. This inner-context situation was an unintentional result of outer-context decisions about how to apportion limited amounts of funding, as well as perhaps the desire to control training and coaching integrity by locating coaches in one or a few organizations.

For their part, coaches depended on lead coaches and supervisors for support. Supervisors who understood the intervention and the intended role of coaches greatly influenced home visitor buy-in. In contrast, supervisors not on board with, or knowledgeable about, the EBI were not considered useful to home visitors, and in the worst cases, negatively influenced home visitor relationships with coaches and the EBI itself. This was problematic when home visitors received inconsistent messages about whether coaches were considered peers or monitors. In several systems, supervisors and government administrators, especially those unfamiliar with the EBI, sometimes treated, used, or talked about coaches as having only a fidelity monitoring function in a way that reinforced frontline anxieties about coaching. The negative effects of such misperceptions are consistent with scholarly understandings of the dimensions of effective implementation leadership (Aarons et al., 2015).

We recommend that the insecurities and initial turnover that affected home visitors can be diminished by government and CBO administrators proactively clarifying the coach role with the initial trainers during the Preparation phase, and priming trainees to understand and embrace the support offered by coaches. Furthermore, rather than rely on trial and error to hire coaches, greater knowledge of the role of coaching in the EBI during the Preparation phase, and the characteristics of a good coach, will enable system and CBO administrators to recruit and select individuals for critical implementation positions who will support the EBI throughout the Sustainment phase (Ehrhart et al., 2014, Schein, 2010) and increase fidelity. Additionally, recruitment, selection, and hiring of individuals with desirable characteristics to support EBI implementation signify to employees the importance of the EBI (Ehrhart et al., 2014). This is in keeping with an implementation climate that communicates to employees what is expected, supported, and rewarded in organizations (Jacobs et al., 2014). Knowledgeable leaders can tailor the coach position to their individual implementation environments and balance it with other job assignments (e.g., supervisory functions) to prevent added responsibilities from presenting scheduling challenges like those that negatively affected the coaches in our study.

Our findings suggest that even when leaders lacked knowledge of the appropriate role for coaches and expected them to act as monitors, coaches and their supervisors could resist such outer-context pressures by explaining to administrators that coaches were meant to serve in a supportive capacity. In situations like these, coaches and supervisors intersect with stakeholders differently positioned within child welfare system hierarchies (Duda and Barrett, 2013), thereby spanning inner and outer contexts. Coaches were often conceptualized by other stakeholders in this spanning capacity as communication liaisons. While having a liaison between inner and outer contexts was valuable for other system stakeholders, our study results suggest that the boundary-spanning position puts coaches at risk of being asked to fulfill duties above and beyond their intended role, such as reporting beyond fidelity assurance for the EBI. We suggest that turnover in leadership positions, funding issues, and changing relationships between CBOs and government contracting entities—common characteristics of human service systems—risk exacerbating this issue, and may compromise the implementation support a coach can provide for frontline workers. Consequently, we recommend that CBO managers be prepared to support and protect coaches as built-in quality assurance for EBI delivery by buffering them against additional demands. Furthermore, government administrators must ensure that contracted CBOs and coaches have the tools they need to best assess and augment coaching in their systems. This includes anticipating and proactively addressing funding changes, strategically applying resources for ongoing implementation, and directing sufficient provisions for enough coaches throughout the service system (Aarons et al., 2014a, Willging et al., 2017).

This case study further demonstrates the importance of understanding the interaction between inner- and outer-context factors by illuminating the ways that they affect the capacity of coaches to support EBI implementation and sustainment. This builds on previous research that found evidence of outer-context decisions impacting inner-context processes (Willging et al., 2015b, Willging et al., 2015a). Our findings indicate that the effect of coaching on implementation depends largely on a coach’s ability to navigate, and help others navigate, dynamic service delivery environments; indeed, context may be considered the primary mediator to consider in order for an EBI to progress to the Sustainment phase (Dopson et al., 2008). In these environments, support components for EBIs, including coaching, work together and are limited in their ability to function independently to influence implementation (Fixsen et al., 2009, Wandersman et al., 2012). Leaders of CBOs and government agencies must be willing to collaborate to make organizational changes that strengthen the integration of coaching processes throughout EPIS phases (Aarons et al., 2011, Green et al., 2016). Ultimately, our study indicates that concepts of “good” and “bad” coaches relate to how well the coach role is integrated within CBOs, among home visitors, and within service systems as a whole.

The confluence of multiple “embedding mechanisms” may create a strategic climate that promotes EBI use (Schein, 2010). Primary embedding mechanisms are strategies that leaders at multiple levels (e.g., system, CBO, and team or workgroup) can use to develop such climates. Examples include what leaders focus on regularly, how leaders react to crises, how they allocate resources and rewards, how they teach or role model, and how they recruit staff (Aarons et al., 2014b). We contend that climates in which leaders at the system and organizational levels recognize the importance of coaching are likely to be conducive to EBI provision. Leaders demonstrate coach support by conveying this importance to others, problem-solving when problems with coaching arise, capitalizing coaching appropriately, teaching others about, acknowledging, and rewarding coaches for their services, and are purposeful when selecting coaches.

For home visitors, stories about important events and people associated the emergence and maintenance of the coach role may also inspire stakeholders to take coaching seriously, and perhaps to become coaches themselves. Stories, which we were privy to throughout this investigation, are themselves secondary embedding mechanisms that can bolster support among government and CBO administrators for coaching as a crucial implementation quality control measure.

Other secondary embedding mechanisms concern how leaders design and structure systems, organizations, and teams to support EBI, including development of reimbursement structures to compensate for coaching services or the sharing of coaching resources across CBOs, or how coaching-home visitation teams may be configured for fidelity monitoring and quality improvement purposes. This type of design can include formal statements highlighting the value of both the EBI and its support systems, i.e., coaching. Here, we recommend that government and CBO administrators clearly document descriptions of the coach position in service contracts and agency policies and procedures to maintain institutional memory that upholds the integrity of the coach role. This recommendation is consistent with implementation approaches that address the need for job redesign as part of comprehensive implementation strategies (Glisson and Schoenwald, 2005). Moreover, such actions will help mitigate the risk of coaches being blamed as “bad,” when in fact a contextual factor may be culpable. Blaming the coach damages the support mechanism in place for the implementation. Documenting coach roles early may ameliorate challenges related to turnover in government positions that led to new hires attempting to co-opt coaches as monitors. In our case study, communicating role expectations to all stakeholders may have helped avoid resentment among home visitors toward the coaches when financial decisions led to issues with scheduling and inconsistent guidance.

Second, detailed agreements to fund ongoing training and coaching or consultation will help maintain the quality of coaching processes. Coaches who, for financial reasons, were reduced in both training and billable coaching hours expressed concerns about having a more distant relationship with home visitors and a limited ability to support them in implementing. A CBO in a nonsustaining system that was challenged to efficiently bill and report according to funder and government requirements may have been able to sustain the implementation had they streamlined agreements first between the three entities.

Third, measuring and documenting outcomes will help guide improvements for the EBI, including for coaches (Aarons et al., 2011). Delineating reporting processes and consistent language agreed upon by system stakeholders ease communication challenges for coaches working across CBO agencies with different policies (Duda and Barrett, 2013), while simply describing coaches as peer support during training may generate confusion when coaches must rate the integrity of the implementation and provide guidance to home visitors. Adopting clearly articulated processes would also support the coaches who became entangled in the different CBO policies and procedures when trying to solve home visitors’ implementation challenges.

5. Limitations

Although the service systems where this research occurred were varied, the study took place in only two US states, thus limiting generalizability. Additionally, the study included participants from each of the non-sustaining systems, but not all stakeholder groups were represented in later phases, as we could only access government and/or CBO administrators; coaches and home visitors no longer occupied the same positions related to the EBI. We also recognize that while our categorization of systems by sustainment status may allow us to suggest characteristics common to successful and less successful implementation environments, they may not fully capture the nuanced differences in implementation across service systems. We are also limited in our capacity to compare across the categorizations, as the inclusion of only one partial-sustaining system limits understanding of the degree to which elements of SafeCare were in play.

Whereas this case study emphasized perceptions of coaching among different stakeholders, and we examined the SafeCare coach training manual, future research might include the clients who received the intervention. Moreover, we contend that favorable components, characteristics, and procedures are dependent on local contextual factors. Nevertheless, this study benefits from 11 natural settings that enable us to examine in-depth the circumstances in which coaching occurs over time.

6. Conclusion

This paper elaborated outer- and inner-context factors affecting coaching in EBI implementation and sustainment. Our findings indicate that successful coaching involves trusting relationships and open communication, not only between coaches and home visitors, but also between coaches, outer-context stakeholders, and CBO administrators. Inner-context factors that facilitated sustainment of the EBI included CBO administrators enacting changes to better support coaches, use of in-house coaches, and buy-in among supervisors and home visitors of the EBI. Outer-context factors facilitating sustainment included continued funding from government administrators, an understanding of the scope of work for coaches, and collaborating with CBO administrators to incorporate coaching and trainings into contracts.

Leaders in both the outer and inner context can use embedding mechanisms to support coaching. In the present study, coaching can be considered an embedding mechanism signaling the importance of maintaining fidelity support as a critical part of EBI implementation. Where sites attained full sustainment, coaching and fidelity support were judged by system and CBO leaders as critical to high quality service delivery. Coaching, then, might be a valuable addition to other parenting interventions (e.g., Positive Parenting Program or Triple P; Seng et al., 2006) that may not emphasize this type of ongoing fidelity support at the level required by SafeCare (Chaffin et al., 2012b, Funderburk et al., 2015). We recommend further institutionalizing the coach role through job redesign, clarification of coach requirements in job descriptions and performance evaluations, and recruitment for favorable coaching characteristics. Colocation of coaches in the offices of home visitors might also prove advantageous. Ciruclating stories about the positive role of coaching and coaches will likely reinforce coaching as a critical support system among stakeholders at multiple levels of a service system.

A clear contribution of this study is the identification of a common understanding of coaches as boundary spanners linking inner and outer contexts and facilitating communication to support implementation. However, while it may be valuable for coaches to span inner and outer contexts, doing so may also compromise the integrity of their specific role within EBI implementation and sustainment. Consequently, inner- and outer-context stakeholders must devote further attention to support coaching to improve fidelity and client outcomes in public sector service systems.