The Role of Collaborations in Sustaining an Evidence-Based Intervention to Reduce Child Neglect

Abstract

Child neglect is the most prevalent form of child maltreatment and represents 79.5% of open child-welfare cases. A recent study found the evidence-based intervention (EBI) SafeCare^® (SC) to significantly reduce child neglect recidivism rates. To fully capitalize on the effectiveness of such EBIs, service systems must engage in successful implementation and sustainment; however, little is known regarding what factors influence EBI sustainment. Collaborations among stakeholders are suggested as a means for facilitating EBI implementation and sustainment. This study combines descriptive quantitative survey data with qualitative interview and focus group findings to examine the role of collaboration within the context of public-private partnerships in 11 child welfare systems implementing SC. Participants included administrators of government child welfare systems and community-based organizations, as well as supervisors, coaches, and home visitors of the SC program. Sites were classified as fully-, partially-, and non-sustaining based on implementation fidelity. One-way analysis of variance was used to examine differences in stakeholder reported Effective Collaboration scores across fully-sustaining, partially-sustaining, and non-sustaining sites. Qualitative transcripts were analyzed via open and focused coding to identify the commonality, diversity, and complexity of collaborations involved in implementing and sustaining SC. Fully-sustaining sites reported significantly greater levels of effective collaboration than non-sustaining sites. Key themes described by SC stakeholders included shared vision, building on existing relationships, academic support, problem solving and resource sharing, and maintaining collaborations over time. Both quantitative and qualitative results converge in highlighting the importance of effective collaboration in EBI sustainment in child welfare service systems.

Approximately 3.9 million incidents of suspected child maltreatment were reported to state child protective service systems in the United States (U.S.) in 2013 (U.S. Department of Health and Human Services, 2015). Child neglect is the most prevalent form of child maltreatment, representing 79.5% of child victims and contributing to 80% of U.S. child welfare fatalities (U.S. Department of Health and Human Services, 2015). Neglected children have difficulties with social, emotional, and language development (Tyler, Allison, & Winsler, 2006), are at risk for cognitive deficiencies (Twardosz & Lutzker, 2010), and present the least positive and the most negative affect of all maltreated children (Hildyard & Wolfe, 2002). Given these high prevalence rates and negative outcomes, there is a pressing need to successfully implement and sustain early intervention and prevention efforts.

Historically, there has been a dearth of evidence-based interventions (EBIs) designed to specifically address and prevent child neglect (Chaffin, 2006). However, a recent statewide effectiveness study found the home visitation EBI SafeCare^® (SC) to significantly reduce child neglect recidivism rates compared to services as usual (Chaffin, Hecht, Bard, Silovsky, & Beasley, 2012). Despite increasing demand for EBIs in child welfare systems (Wike et al., 2014), less systematic understanding exists regarding factors that facilitate their implementation and sustainment, such as the role of collaboration among stakeholders within service systems. Without effective implementation and sustainment, initial investments in EBIs are wasted, the subsequent impact of beneficial interventions is limited, and outcomes for children and youth are compromised.

Researchers have suggested that collaborations between government administrators, CBO administrators, academic researchers, intervention purveyors, and service providers can facilitate overall implementation by bridging the research and practice gap through the pursuit of shared goals and visions (Aarons, Hurlburt, & Horwitz, 2011; Jones & Wells, 2007; Mendel, Meredith, Schoenbaum, Sherbourne, & Wells, 2008; Proctor et al., 2009). However, there is a lack of empirical research specifically examining the role of collaboration in the implementation and sustainment of EBIs within child welfare service systems (Proctor et al., 2009). Although collaboration can refer to a variety of relationships, it is also defined here as the process through which stakeholders who see different aspects of a problem work together towards a common goal in order to bring about change beyond what they are able to accomplish as individual entities (Bunger et al., 2014a; Gray, 1989; Lasker, Weiss, & Miller, 2001). Collaboration may be preplanned, highly structured, and formalized (Butterfoss, 2007) or may develop more organically based on the needs and characteristics of the services system. The current study examines: 1) how collaborations operate in the context of public-private partnerships; 2) the role of external partners (e.g., academic researchers, intervention purveyors, outside funders) in supporting EBI; and 3) how less formally developed collaborations of multiple stakeholders can affect implementation and sustainment of an EBI within child welfare service systems.

Collaboration in Child Welfare

Public child welfare services are increasingly provided by private community-based organizations (CBOs), leading to greater dependence on public-private collaborations (Collins-Camargo, McBeath, & Ensign, 2011; Johnston & Romzek, 2008). In privatized child welfare programs, relationships between public and private agencies require ongoing collaboration and information exchange between government agencies and CBOs (Collins-Camargo et al., 2011) as well as among CBOs who may need to coordinate service delivery to effectively meet the needs of clients (Johnston & Romzek, 2008). Contracting also leads to competition among CBOs, and in some cases, competition can increase the overall benefits of contracting. In most child welfare services systems, arrangements characterized by both collaboration and competition, referred to as co-opetition, are the norm (Bunger et al., 2014a). These relationships, such as sub-contracting between two CBOs for service delivery, can have mutual benefits while bridging gaps in services.

Balancing the priorities of many stakeholders can enhance shared decision-making, but can also introduce tension and influence power dynamics within a system (Horwath & Morrison, 2007). For example, contracting out child welfare services to private CBOs increasingly necessitates the redistribution of decision-making power related to implementation activities from public agencies to private contractors (Collins-Camargo et al., 2011). This can prove challenging for government administrators who may have traditionally exerted overall authority for implementation (Willging et al., 2015). Conversely, some stakeholders may feel coerced into a collaboration by those perceived as possessing greater power (Kothari & Cooke, 2001).

Complexities in the relationships among academic-community collaborations are also important to consider when implementing and sustaining EBIs in social service settings (Palinkas & Soydan, 2012; Proctor, 2003). Within the field of social work, Proctor (2003) calls for cultivation of academic-practice collaborations to carry out “trench to bench, and back to trench” research that aids in uptake of EBIs. Proctor emphasizes stakeholder engagement for incremental progression of knowledge to determine acceptability and applicability of new interventions. Consistent with this stance, approaches have been developed that formalize collaboration between community practitioners and researchers, such as community-based participatory research (CBPR) or community-partnered participatory research (CPPR) (Jones & Wells, 2007; Wallerstein & Duran, 2010). Prevailing CBPR and CPPR approaches focus on power-sharing in decision-making and joint academic-community ownership of research initiatives. Shared control over fiscal resources and expenditures is also an important feature. In contrast to these planned approaches, many implementation collaborations are less prescribed and formalized, yet attempt to align the interests of multiple stakeholders including government administrators, CBO administrators, direct service providers, academic researchers, and intervention purveyors, to undertake activities that promote implementation and sustainment of an EBI in the face of competing interests and priorities. Greater understanding of less formalized collaborations, developed organically to support EBI implementation in child welfare service systems, may elucidate factors that support or inhibit EBI sustainment.

There is a tendency to underestimate the complexity involved in collaboration and to minimize the stakeholder perceptions, priorities, agendas, and idiosyncrasies that may influence the change processes that occur with EBI implementation in public sector services (Green & Aarons, 2011). Stakeholders often need to operate within structural arrangements that may differ from those with which they are accustomed, which requires clear communication, comprehensive planning, and an understanding that the continuum of power and responsibilities may shift depending on the stage of implementation and collaboration (Butterfoss, 2007). Key elements of successful collaborations include empowering community stakeholders, shared focus, similar attributes and interests, cumulative advantages for members, frank discussion of concerns of all stakeholders, building trust, and valuing the needs of all stakeholders (Dahlander & McFarland, 2013; Maurana & Goldenberg, 1996). However, diverse stakeholders are often influenced by different organizational cultures and values, and challenges to collaborations can occur when stakeholders do not share the same understandings and expectations, or are guided by competing organizational and individual goals and agendas. Potential barriers to successful collaboration include lack of trust and respect, inequitable distribution of power, conflicting priorities and belief, conflicts over funding, and time consumption (Israel, Schulz, Parker, & Becker, 1998). When multiple stakeholders with differing views are at the table, negotiations that take problem-solving, pro-social, and less egoistic orientations may facilitate collaboration and achievement of joint outcomes (Campbell & Mark, 2006; De Dreu, Weingart, & Kwon, 2000). Reliance on collaborations for EBI implementation and sustainment is not risk free and has the potential to impede or undermine implementation as well as potentially support it (Bunger et al., 2014b; Kano, Willging, & Rylko-Bauer, 2009). Although there has been much research conducted on formalized collaborative processes in community initiatives (Israel et al., 1998; Wallerstein & Duran, 2010), few studies focus on the role of collaborations borne of necessity in EBI implementation and sustainment in public-sector service settings such as child welfare.

Current Study

As part of a larger mixed-method investigation of SC sustainment (Aarons et al., 2014c), the current study examines the role of collaborations in sustaining SC service delivery across 11 state/county-level implementation efforts. This manualized curriculum-based EBI aims to reduce child neglect through home-based skills training and education for caregivers of children ages zero to five who are at-risk, or have been reported for, child neglect (Chaffin et al., 2012; Lutzker & Edwards, 2009). The intervention enhances problem solving and communications to improve parent or caregiver skills and behaviors to address home safety, child health, and parent-child or parent-infant interactions. Implementation of the model requires three types of professionals: 1) home visitors who deliver the EBI to caregivers; 2) coaches who provide assistance to, and conduct monthly monitoring of, home visitors to ensure high levels of fidelity to the EBI; and 3) trainers who are certified to train and coach new home visitors. In the present study, all participating service systems took advantage of this structure to facilitate self-sustainment by localizing training and quality control (i.e., fidelity oversight) to create resilience to workforce turnover at a relatively modest cost. This study combines qualitative interviews and focus group data from government and CBO administrators, SC supervisors, SC coaches, and SC home visitors to examine how collaborations that were not part of a formalized approach operate in the implementation and sustainment of an EBI in child welfare service systems. Additionally, quantitative system-level data provided by government and CBO administrators are used to examine the hypothesis that service systems with greater levels of sustainment would report increased levels of effective collaboration.

Methods

Study Context

This study includes 11 separate child welfare systems. Included are one statewide system referred to here as “State A” and 10 county-wide systems based in “State B.” Training in SC began between 2 and 10 years prior to study participation. According to the 2010 U.S. Census Bureau, State A has a population of approximately 3.7 million residents, almost 42% of whom live in rural areas. In State A, SC was implemented through a state-operated child welfare system with all services guided, contracted, and funded by the state government. Local CBOs bid for contracts from the state agency to provide SC as part of home-based services. In some cases, CBOs also partner with one another to bid for contracts, to cover more service areas, and to share resources. Academic researchers were involved in the initial selection of, and training in, SC and its ongoing implementation and sustainment. The academic researchers collaborated with government administrators and CBOs as part of a large federally-funded effectiveness trial of SC and continue to partner with them on research and evaluation projects. State A has also contracted with the academic researchers to oversee training and coaching of home visitors.

State B includes six primarily urban and four primarily rural counties involved in implementing SC, ranging in population from approximately 3.2 million to just over 150,000 residents. Counties in State B implemented SC through different collaborations and shifting funding arrangements. For example, the initial decision to implement SC in one county arose from meetings involving key personnel from the local government, nonprofit CBOs contracted to deliver child welfare services, a private foundation interested in initially funding an EBI, academic researchers, and intervention developers. Researchers at an academic institution in State B also partnered with stakeholders within the 10 counties as part of federally-funded research studies that examined cascading diffusion models for EBI implementation and dynamic adaptation processes. The academic researchers in State B no longer have funding to provide coaching or training for the counties. Formal research partnerships pertaining to SC delivery ended in all but one county in 2012 and that county currently collaborates with the academic researchers on multiple federal research grants.

Defining Sustainment

Sustainment is operationalized in the current study based on Stirman and colleagues’ (2012) systematic review, which states that an EBI can be classified as sustained if core elements are maintained or delivered at a sufficient level of fidelity after initial implementation support has been withdrawn, and adequate capacity exists to continue maintaining these core elements. In the case of SC, initial implementation support consisted of training, coaching, and monitoring by the intervention developers and trainers. For SC, core elements of fidelity include not only “content fidelity” (i.e. fidelity to the EBI content during sessions with families) but also “structural fidelity” (e.g. appropriate caseload sizes, monthly coaching/fidelity monitoring visits, regularly scheduled team meetings). Stirman et al. (2012) use the terms “full” and “partial” to describe sites which meet all versus only some of the core elements after the withdrawal of initial implementation support. These terms are retained in this study. We use the term “non-sustainment” for sites that no longer have certified home visitors implementing SC. The first and senior/last author of the manuscript, who have been involved with each of the 11 implementations, classified programs into the three groups based on level of implementation fidelity (derived from the standards set by the National SafeCare Training and Research Center: NSTRC). Fully sustaining sites had trained coaches and home visitors who provided SC to clients on a weekly basis and maintained fidelity to the SC manual. Additionally, these sites had certified coaches who conducted fidelity monitoring visits and held team meetings according to NSTRC standards (n=7; total time implementing ranged from 2 to 10 years). Non-sustaining sites were those that no longer provided SC services (n=3; total time implementing ranged from 1.5 to 2.3 years). There was 1 partially sustaining site (total time implementing was 4 years). This site had trained home visitors and utilized the SC manuals but did not use SC coaches to conduct fidelity monitoring and provide support after the initial implementation year.

Quantitative Data Collection

Participants

Across the 11 SC sites, 53 of 60 (88.3%) invited upper-level stakeholders, including state and county child welfare administrators (“government administrators”); CBO directors, deputy directors, program managers (“CBO administrators”), and SC supervisors, completed the Program Sustainability Index (PSI). These individuals also took part in our individual and small group interviews as described below; however, seven of our interviewees did not complete this measure. Prior to completing the PSI, participants responded to brief demographic questions. Participants were 87.5% female and indicated their highest education as Master’s Degree (62.5%), Bachelor’s Degree (21.4%), Doctorate (7.1%), or some college (8.9%). The sample’s race was Caucasian (73.2%), African American (7.1%), Asian or Pacific Islander (7.1%), Native American (3.6%), or “other” (8.9%). Twenty-three percent indicated Hispanic ethnicity. The average tenure at participants’ current agency was 15 years (SD=9.89).

Measure

The Program Sustainability Index (PSI) was developed to assess five domains of sustainability in community programs on a 5-point scale from 0-not at all, to 4-to a very great extent, and demonstrates good psychometric properties (Mancini & Marek, 2004). The current study utilizes the Effective Collaboration subscale (10 items, current sample alpha=.93), measuring the extent to which stakeholders are perceived to support program goals and have a shared vision to sustain the effort. Example items include: “There is a shared vision among collaborators,” “Turf issues are resolved,” “Collaborators have clearly defined roles and responsibilities,” “Collaborators share credit for SafeCare success,” and “Local decision-makers are SafeCare collaborators.”

Data Analysis

Effective Collaboration scores were examined across level of sustainment: fully sustaining, partially-sustaining, and non-sustaining. A One-way Analysis of Variance (ANOVA) with Tukey’s Post Hoc test (based on the total of two possible comparisons) was conducted to determine whether there were significant differences in level of effective collaboration among the three sustainment levels. Effect sizes are reported using Cohen’s d (Cohen, 1988).

Qualitative Data Collection

Participants

Government administrators, CBO Administrators, and SC supervisors (all but seven of whom who also completed the PSI and demographic questions), participated in individual (n=52), or small group interviews (n=8) with an average of three participants. The SC coaches and home visitors (n=122) participated in one of 21 focus groups, with an average of six participants per group. Focus groups were specific to each agency and conducted separately with coaches and home visitors. Participants were primarily female (88%). Highest reported education was Bachelor’s Degree (59.3%), Masters’ Degree (20.3%), some graduate work (11.0%), some college (7.6%), and high school (1.7%). The sample was Caucasian (62.2%), African American (9.0%), Native American (7.2%), Asian or Pacific Islander (4.5%), and “other” (17.1%). Hispanic ethnicity was reported by 39% of the participants. Participants had been with their agencies for an average of 4.11 years (SD=4.67). See Table 1 for agency tenure, gender, and Hispanic ethnicity by stakeholder type.

Table 1.

Demographic characteristics of study participants by stakeholder type

	Female %	Hispanic %	Yrs. at Agency Mean (SD)
Government Administrators (n=18)	94.4%	11.1%	18.50 (9.89)
CBO Administrators (n=27)	81.5%	30.8%	14.73 (8.13)
SC Supervisors (n=8)	100%	25.0%	6.78 (1.72)
SC Coaches (n=28)	92.0%	44.0%	6.53 (6.62)
SC Home Visitors (n=94)	87.6%	36.8%	3.47 (3.33)

Procedure

Interview and focus group guides were developed by the authors of the current manuscript to examine factors related to implementation and sustainment of EBIs in child welfare service settings. Questions addressed positive and negative influences on SC implementation and sustainment, including but not limited to issues of leadership, decision- and policy-making, contracting processes, and stakeholders roles. Interview and focus group questions included several probes to facilitate a deeper examination of experiences related to possible collaborations: “Who are the most important decision-makers, or stakeholders, to influence whether SC continues in [name of site]? Why are they the important stakeholders? What role do they play?” “What types of interaction have you had with SC stakeholders outside your [team or agency] who play a role in making sure SC gets delivered at [name of site]. How did these interactions shape your own ideas, opinions, and attitudes about SafeCare? How did these interactions influence how you practice SafeCare?” Two anthropologists with expertise in qualitative methodology conducted the semi-structured interviews and focus groups with stakeholders from each of the 11 SC sites. Qualitative data collection occurred between 2012 and 2014; and two to ten years post initial SC training and implementation. Interviews were typically one hour in length and focus groups lasted about 90 minutes.

Data Analysis

All interviews and focus groups were digitally recorded, professionally transcribed, and checked for accuracy by at least one author. An iterative process was used to review the textual data and NVivo 10 qualitative data analysis software facilitated this work (QSR International, 2012). The transcripts were coded to condense the data into analyzable units. Segments of text ranging from a phrase to several paragraphs were assigned codes based a priori on the particular topic areas and questions that made up the interview guides (Patton, 2015). These codes thus centered on key sensitizing concepts, such as leadership, stakeholder interaction, and collaboration. The use of such concepts from the implementation literature provided “a general sense of reference” for our analysis and enabled us to examine both their salience and meaning for stakeholders through the provision of descriptive data based in the actual words of participants and directly reflecting their own perceptions and experiences (Patton 2015, p. 545). During our review of the transcripts, we used open coding to locate new themes and issues that emerged from the transcripts (e.g., “perceived role of academic partner,” “stakeholder ‘buy in’ and support,” and “factors related to partnership maintenance”) (Corbin & Strauss, 2008). Focused coding was then used to determine which themes emerged frequently and which represented unusual or particular concerns to the research participants. In this staged approach to analysis, three authors independently coded sets of transcripts, created detailed memos that both described and linked codes to each theme and issue, and shared their work with one another for review. Through the process of constantly comparing and contrasting codes with one another (Corbin & Strauss, 2008; Glaser & Strauss, 1967) via ongoing discussion, codes with similar content or meaning were grouped together into broad themes linked to segments of text. The final list of codes, constructed through a consensus of the authors, consisted of a list of themes that explicate how collaborations operate in the context of public-private partnerships, the role of external partners (e.g., academic researchers, intervention purveyors, outside funders) in supporting implementation and sustainment, and how less formally developed collaborations across multiple stakeholders play out in the implementation and sustainment of EBIs in child welfare service systems.

Results

Collaboration by Level of SC Sustainment

There was an average of five (SD=3.87) stakeholder reporters for each implementation site. Results of the One-way ANOVA indicated a significant difference among the three levels of sustainment, f(2)= 14.46, p<.01. Post hoc analysis indicated significantly higher effective collaboration scores for fully-sustaining sites (M=3.22, SD=.51) compared to non-sustaining sites (M=1.77, SD=1.27), Cohen’s d=1.48. No significant differences were found between fully-sustaining and partially-sustaining (M=2.65, SD=.49) sites, however scores were higher for fully-sustaining sites (Cohen’s d=.77). The effect size for the non-significant difference between partially-sustaining and non-sustaining effective collaboration scores was large (Cohen’s d=.91).

Qualitative Findings

Five major but interrelated themes were derived from the qualitative data analysis process: shared vision; building on existing relationships; academic support; problem solving and resource sharing; and maintaining collaborations over time. Themes are described in an order that moves broadly from the general characteristics of successful collaboration (i.e., shared vision; building on existing relationships) to discussion of particular types of collaboration (i.e., academic support; problem solving and resource sharing), with the last theme (i.e., maintaining collaboration over time) pointing toward future needs and goals. Quotations exemplifying the views and experiences of stakeholders are provided to illuminate each theme, and indicate differences by level of sustainment where applicable. Expressions such as “um” and “you know” and redundant wording were eliminated to enhance readability.

Shared Vision

Several stakeholders commented on sharing commitment, values, and goals across levels. Recalling initial meetings about the implementation of SC, government administrators emphasized “the importance of coming together.” One government administrator remembered seeing stakeholders from multiple counties and different system levels at the same meeting and thinking, “Okay we’re serious here.” Another government administrator described the shared commitment of CBOs and county administrators from multiple departments in the child welfare system, including social services and mental health, as a crucial foundation to their work, “We have made commitments to each other that we are going to support implementation and expansion of evidence-based practices serving children and families of the child welfare system. It’s just there. It’s one of the cornerstones of what we’re working on.”

Shared vision and commitment had several positive effects on the implementation and sustainment of SC. Several stakeholders underscored the importance of “buy-in” from stakeholders at all levels. Government administrators appreciated feedback from CBO administrators and providers that indicated buy-in, while SC providers and supervisors were encouraged by the positive attitude of government administrators. In one site, a focus group of CBO administrators agreed that it was “the partnership, the buy in, the ownership of SafeCare” in their county that made implementation successful. One SC supervisor also explained that shared goals made the difference between successful and unsuccessful collaboration, even when different parties disagree:

I can’t say that [the interactions with other stakeholders] are always 100% positive on the highest level. There might be some other stuff that happens in the conversation, but everyone’s intention I believe is for the good of the program. That impacts my attitude in a real good way. Whenever I can just really be conscious of the fact that everyone wants what’s best for the program I’m good.

Another CBO administrator agreed that “you can have excellent training and excellent support and crash, still, as an organization if you don’t have a similar buy-in.”

Stakeholders also explained that shared vision, values, and goals created accountability. For some, this accountability meant following through on their commitment to implement SC with fidelity. A SC supervisor from a fully-sustaining site recalled a conference call with stakeholders from another site where “it didn’t feel like they were all coming together on it, like we did here.” Instead, turnover in coaching and leadership left no one to ensure that fidelity was maintained until “they weren’t really doing SafeCare.” For that reason, s/he said, “I think that here we all agree, ‘Okay we’re all going to move forward with this, and we’re going to do it and then there’s some accountability because we all believe that this is a good thing for families.’” While these SC providers experienced an informal sense of accountability to one another from their shared commitment, more formal types of accountability, such as reporting requirements, were distinctly top-down in focus. One CBO administrator described “an ongoing stream of accountability” between levels in the form of data that had to be reported to the funder each month. This accountability ensured that CBOs met their goals, simply because “we want to keep the services going.” While this administrator appreciated the requirement to be accountable to the funder, it is important to note that in this case, collaboration operated in one direction (i.e., from CBOs to the funder) and was directive in nature.

In contrast, sites that had difficulty sustaining SC often reported a lack of shared vision across stakeholder groups, particularly as time went on. For example, a government administrator in a non-sustaining site described that in “a very large department that has lots of different priorities in play, and we had several evidence-based practices or worked with other foundations,” a disproportionate burden to maintain SC fell on one CBO and connections between stakeholders disappeared. In one fully-sustaining site, a CBO administrator worried about the departure of a key SC visionary at the government level, commenting that, “I’m not sure we have a champion within the Department of Human Services right now.” These examples also indicate that while buy-in from CBOs and home visitors was important for SC sustainment, it was in fact impossible without the commitment of higher-level government stakeholders. When asked about the most important decision-makers in the sustainment of SC, one CBO administrator pointed to “the higher-ups” at the government level and explained simply, “They’re the ultimate decision-makers.”

Building on Existing Relationships

In several of the fully-sustaining sites, collaboration on SC implementation was strengthened by preexisting relationships, particularly between counties, academic researchers, and CBOs, as well as among CBOs. The CBO administrators and government administrators commonly reported that they intentionally developed SC relationships with those whom they had partnered in the past because they knew they could work together. In one site, a collaboration central to SC implementation between child welfare and mental health departments was part of a larger network of “stakeholders who work on children’s issues and meet monthly.” Preexisting relationships allowed different stakeholders to operate from a place of trust and mutual respect. One government administrator stated, “We have really good [CBOs] so I trust them. I have a long working history with them; it’s a really good partnership. It’s definitely a good fit and so when they raise concerns, they’re legitimate concerns. It’s about how to problem-solve and not about people taking a stance.” Trust between the county and the CBOs based on existing relationships helped to minimize the possibility of political conflicts and helped contribute to the ongoing success of the SC program.

Preexisting relationships between counties and the academic researchers were also viewed as key to successful collaboration related to SC. One government administrator preferred working with the academic researchers over the intervention developers based on the ongoing relationship and their knowledge of local context:

It’s just because of the relationship that I have built with them and I trust their work. They kind of understand unique characteristics of [our county] because they’ve worked with our clientele, so I thought that questions and answers I may get from them are more specific to [our county] than national center people who might be able to give me more of a general answer.

As this government administrator articulated, the relationship with the academic researchers gave him/her confidence that they knew the county and its unique characteristics factoring into SC delivery. This county then worked with the academic collaborators to develop ongoing guidelines and evaluation activities to promote the sustainment of SafeCare.

Relationships between CBOs were particularly highlighted in fully-sustaining sites, where they had histories of collaborating on grant applications, sharing contracts and resources, and communicating openly about problem-solving and work strategies. In fact, in one site, the CBOs’ natural inclination to collaborate with one another coexisted uneasily with the government agency’s efforts to establish a competitive bidding process for child welfare contracts. A staff member from the agency recalled, “They didn’t want to compete against each other whereas at [the funding agency], we wanted the competition because our CEO kept saying we want the best agency so we wanted to see them submit different applications. And they wanted to partner with each other so it was interesting.” At the funding level some worried that there was “collusion” among the CBOs, or even “a conspiracy to shut out potential competitors.” However, as one CBO administrator explained, “We’re all friends…. We don’t like it when we have to go head to head like that. It happens. Its business, but, we prefer to work otherwise.” Other CBO administrators in this sustaining site described an ethic of collaboration that predated SC. This ethic was recognized by government administrators, one of whom observed, “[The team approach is] the spirit [the CBOs have] had forever and now it is part of the contract. But I think it’s more than being contractual, it’s just how the nonprofit community in [this county] works.” Another government administrator stated, “We’re a lot of good dirt, you know, it’s like good dirt, plant the seeds, because we are so collaborative by nature.”

Academic Support

Collaborations with academic researchers reportedly played a particularly prominent role in fully-sustaining sites throughout the implementation and sustainment of SC. Although government and CBO administrators in non-sustaining sites had little to say about the academic collaborators, government administrators from fully sustaining sites acknowledged their role in championing SC and considered them partners in helping to bring the EBI into the service system. Government administrators particularly noted the academic researchers’ contribution to overall understanding and acceptance of the SC model. A government administrator explained that the initial introduction to SC was better received by the CBO administrators and service providers from the academic researchers than it would have been from a funding agency or a county department. S/he explained: “I think when they get it [introduction to SC] from the funding agency, such as our department, it seems more as a compliance, contractual kind of demand versus a quality assurance type of conversation.” As this comment indicates, academic collaborators were able to play a crucial role in coordinating stakeholders to implement SC as a benefit to the service system in a manner that was perceived as collaborative, rather than top-down. In addition to providing financial support through grant-writing, government administrators also described the academic collaborators as the central providers of support through the implementation and stressed their openness and availability to answering questions and giving advice. In fact, a government administrator stated that the academic researchers were “the only ones I know” who provided any support.

Although government administrators in fully-sustaining sites repeatedly acknowledged the prominent leadership role that the academic researchers performed, CBO staff—including administrators, supervisors, coaches, and home visitors—also regarded them as leaders and visionaries during the implementation of SC. One executive director referred to the academic researchers as “the icons of the implementation” who “have been there all along.” For coaches and home visitors in these sites, the academic researchers represented reliable sources of support; they were often the ones who had trained them in SC and who continued to be embraced as authorities for addressing questions and concerns regarding both implementation and outcomes.

Stakeholders at all levels also pointed to the academic researchers’ role in “bridging the gap” between stakeholders. For example, a government administrator recounted how the academic researchers helped to translate information between the CBOs and the SC developers. This individual opined that other leadership “would’ve driven this [SC implementation] a whole different direction without them [academic researchers] kind of pulling it together and helping everyone see that what the providers brought to the table was really important, as well as the research piece.” Similarly, a CBO administrator described how the academic researchers helped him/her overcome his/her initial resistance to SC by proactively responding to the implementation needs and concerns identified by the home visitors. Acknowledging that s/he was “one of the resisters at the beginning,” this CBO administrator appreciated changes made specifically to administrative procedures viewed as burdensome by the home visitors. S/he explained, “We were meeting so often with [an academic researcher] and the [required] paperwork was changed to be friendlier. I think that’s when [I thought] ‘Oh, they are listening, they really want this to work.’”

In contrast, although input from the academic researchers helped ameliorate early concerns about SC in sustaining sites, home visitors in non-sustaining sites retained their initially negative impressions of the intervention’s perceived rigidity and rigorous fidelity requirements. In these sites, stakeholders reported receiving little support in translating SC to local contexts. In one non-sustaining site, CBO administrators and home visitors also felt that the assessments required by both the model and the research were too much of a burden on their already over-full schedules, and also expressed concerns about the applicability of the EBI itself to their service region.

Finally, the academic researchers were characterized as key to the sustainment of SC in sustaining service systems. In addition to providing ongoing support answering questions about the model, addressing concerns, and ensuring fidelity among home visitors, both government-and CBO-level administrators in sustaining sites indicated that the academic researchers offered a source of a reliable and ongoing assurance of their service systems’ continued commitment to SC. One government administrator explained, “We’ve really relied on that partnering with the university, with being able to really collect and have the data that supports those services.” A CBO administrator agreed that “I feel supported and kind of protected in a way by having that partnership with [the university].” This CBO administrator also reported that the academic collaboration ensured SC’s relevance to the service system, recounting a conversation with an academic researcher: “I said ‘When are we going to get that [new SC] model about how to have communication within relationships?’ [S/he] was saying, “It’s a year out and I’m like, ‘No, we need it now.’ It was just so exciting to have that partnership.” The CBO administrators in sustaining sites also understood that the research on SC outcomes would be influential in determining whether the program would continue to be funded by the local government and thus implemented.

Problem-Solving and Resource-Sharing

Collaborations contributed to joint problem-solving and resource-sharing, especially among CBO administrators and SC supervisors and home visitors, which helped to support implementation success. In one fully-sustaining site, SC home visitors from different CBOs were trained and supervised together in interagency collaborative teams, which provided a novel opportunity CBO administrators, supervisors, and home visitors from different CBOs to support and learn from one another. In this case, the CBO administrators felt that their collaboration allowed for clients to move “seamlessly” between regions without interrupting their SC services. The home visitors also valued the opportunity to learn with and from their counterparts in other agencies. One home visitor explained,

It’s been very valuable that it was an [opportunity] for us to learn from each other, talk about our experiences, our fears, what we don’t understand, sharing resources, I found that very, very helpful because everybody has different strengths and it really, really helped because whenever somebody would talk about something everybody had input to help that person.

Home visitors in interagency collaborative teams also reported sharing ideas and strategies for addressing client needs with their counterparts from other agencies. For example, one home visitor was celebrated by his peers for developing and then sharing an innovative and low-cost strategy to help homeless clients benefit from SC, designed for in-home delivery, using printed pictures. Overall, the interagency collaborative teams afforded this fully-sustaining site a valuable form of collaboration in service provision.

In some cases, SC home visitors also formed collaborative problem-solving and resource-sharing relationships with social workers who refer SC cases. Productive relationships with social workers involved open communication about the needs of clients and their suitability for SC. In the best cases, social workers were educated about SC and were able to make careful and appropriate referrals, often with input from home visitors. For example, in one fully-sustaining site, a CBO administrator described a relationship with county social workers that had evolved from “competitive” to “collegial” based in part on an improved understanding of appropriate services. In contrast, in sites where social workers were unaware of SC, home visitors reported conflicts and inappropriate referrals, which resulted in considerable frustration and wasted time.

Resource-sharing was also key in several sites in relation to funding of SC training and implementation. Several implementations involved unique configurations of funding sources. For example, in one site, a charitable organization interested in addressing child neglect partnered with the county child welfare system to fund the initial SC training. In another county, federal funds were leveraged so that public health nurses could be trained to provide SC. A contract with the local mental health department provided SC funding in a third county.

Interestingly, although these forms of resource-sharing allowed for SC implementation in places where funding was scarce and encouraged shared commitment and accountability between stakeholders, they also had the potential to present obstacles to the sustainment of SC, even in fully-sustaining sites. In some sites, the transition to new funding sources once initial financing for implementation had ended was a source of concern for government and CBO administrators. For example, in one site where the child welfare department collaborated with the mental health department for funding, government administrators could foresee a change in direction from the mental health department that might put an end to their funding of SC. A second site, which hired volunteers from an international aid organization to undertake the initial implementation of SC, faced the considerable expense of hiring and training new staff as the volunteers’ service terms ended. These complications sometimes resulted in a scarcity of resources for home visitation staff. In yet another site where SC expenses were being subsidized with money from other programs, a SC home visitor lamented,

[The child welfare agency] came to us and said, ‘We’d like you guys to do this [SC], but we can’t fund it’ – like literally they can’t even buy us binders, can’t buy us health supplies, so we’ve had to come up with that money on our own. To implement a program where you have no money is really difficult. We’re doing it, but we are missing some things.

In these places, resource-sharing constituted a stop-gap measure where support dedicated specifically for SC was unavailable, placing SC in a precarious position once the initial implementation was complete. Creative collaborations with and between diverse funding sources were thus vital for SC implementation; however, sustainment of SC was threatened when these collaborations could not be maintained or replaced over time.

Maintaining Collaboration Over Time

Maintaining collaborative relationships was a challenge for all SC sites that necessitated commitment from the government administrators and relationships between CBOs and CBO staff. At the most basic level, participants in the sustaining sites pointed to regular meetings as a key to ongoing collaboration and sustainment. A SC coach explained,

One of the things that we do, that we stayed very, very consistent from the beginning is our Thursday meetings. We’ve never really deviated off of our Thursday meetings. We might have changed the formats of the meetings but it’s always been our leader’s idea that we need to come together as a team. As much as we’re all in different agencies, we need to come together as a team.

Similarly, one CBO administrator opined that “what makes it [SC] feel really sustainable for me” is that “there’s all this support and continuity. There’s no lapse in when they’re going to meet. There’s no ifs ands or buts. They meet and it’s regular.”

In contrast, participants in other sites reported difficulties in maintaining regular meetings. Even in the fully-sustaining site where home visitors were trained in interagency teams, groups of home visitors fell out of touch with their counterparts in other agencies. Many felt that the travel required to meet was too much trouble. In the partial and non-sustaining sites, breakdowns in communication also led to a disintegration of collaborative relationships. One CBO administrator described struggles to maintain collaborative relationships with government administrators as a result of irregular communication: “Because of the lack of opportunity to meet on a regular basis and to communicate effectively, we never know who we’re supposed to communicate with. Is it this person? Is it that person now? Then we keep getting ‘No, that’s not a question I can answer.’” This CBO administrator described a combination of communication challenges and turnover of key leadership that created a “perfect storm” of challenges to the sustainment of SC. Similar concerns were reported by stakeholders in other partial and nonsustaining sites, where turnover among leaders who had initially championed SC resulted in the weakening of ongoing support for SC.

The ability to maintain key collaborative relationships for coaching and training home visitors was also highlighted as supporting sustainment. In the site with interagency home visitor teams, home visitors went on to train and coach new teams. In the same site, home visitors and coaches repeatedly indicated that collaborations between two lead CBOs had allowed a great deal of in-house expertise on SC to develop. However, even in the fully-sustaining sites, maintaining relationships to support the necessary infrastructure for SC over time was a challenge. In the same site where in-house expertise to support and train home visitors was extremely strong among the first CBOs to implement SC, other CBOs worried that this centralization of expertise constituted an unfair advantage and compromised the ability of other CBOs to develop skills and support their own home visitors. One home visitor complained that “because [the SC trainer] is in [another CBO], a lot of times it feels like the focus is what’s happening at [that CBO] versus what’s happening here.” In this case, although the initial CBO collaboration remained strong, administrators and staff from other CBOs suggested that they felt marginalized over time.

Collaborating on training and coaching home visitors also proved difficult to maintain in the partial and non-sustaining sites, often because infrastructure for SC sustainment was lacking. For example, SC was abandoned in one site as a result of confusion over billing procedures that could not accommodate SC, combined with a lack of in-house coaching staff for home visitors. The CBO leadership complained that “the cart was placed in front of the horse” when home visitors were trained in SC without an existing infrastructure of funding and supportive staff. Additionally, in the partially sustaining site, a lack of a strong collaborative relationship among the CBOs resulted in a failure to provide coaching and fidelity monitoring for much of the county outside of the region where a trained coach was located.

Discussion

Both quantitative and qualitative results support the role of collaboration in EBI sustainment. Quantitative effective collaboration scores were significantly higher in fully sustaining systems compared to non-sustaining systems, and effect sizes were large. Although non-significant, the direction of effects for other comparisons was consistent with expectations and effect sizes were large. The non-significant results are possibly a function of the small sample sizes, illustrating one of the inherent analytic issues in implementation research where limited units of analysis (e.g., service systems) and key informants may impact quantitative analyses and results.

Qualitative results showed that in fully-sustaining sites, stakeholders often emphasized the importance of relationships across multiple levels in ensuring both the existence of infrastructure and the practice of SC within service systems. Collaboration in these sites was also marked by the longevity of relationships, many of which predated the introduction of SC, and prioritizing interactions within and across stakeholder groups. A feature of these collaborations included the involvement of and increased connections with academic researchers, who provided guidance related to EBI implementation and evaluation services and access to financial resources to support SC locally via their grant writing activities. Academic researchers were thus able to function as de facto coordinators of the different aspects of SC implementation without being negatively perceived as imposing the intervention on providers in the same way that entities inside the service system (e.g., government administrators) might have been. This involvement was both encouraged and recognized as a key to success by government administrators and formalized in contacts between service systems and CBOs that included requirements to work with the academic researchers. Notably, stakeholders in sustaining sites continued to turn to the academic researchers for support related to SC after these requirements or grant funding had ended and continued to collaborate with them on projects that did not center specifically on SC.

In keeping with the broader collaboration literature (Bunger et al., 2014a; Gray, 1989), stakeholders in fully-sustaining sites highlighted the role of shared vision and commitment to SC among stakeholders. This vision provided motivation to support and sustain SC and was made manifest through continued interactions between and within stakeholder groups. Home visitors, for example, were further motivated and nurtured in their work when they had opportunities to meet and learn from other home visitors. This nurturance was also fueled by academic collaborators who bridged gaps between stakeholders, as well as by government administrators who demonstrated to home visitors and the CBOs that they were steadfast advocates of SC and valued their efforts to deliver the EBI to families in need. Shared vision appeared to give way to collaborative problem solving and resource sharing, which ensured that the necessary training, coaching, and overall support infrastructure were in place to support long-term SC delivery. However, it is important to note that while stakeholders in the fully sustaining sites experienced a sense of shared vision, this collaborative spirit did not replace the hierarchical relationships that structured the service systems. The CBOs and home visitors were still required to meet system standards, while government administrators were compelled to report data to funders. Shared vision created a sense of collaboration that helped to mitigate experiences of these requirements as directive; nonetheless, the essentially directive nature of the implementation remained intact. In order to foster a shared vision and commitment, implementation initiatives should allow input from all stakeholders at initial planning meetings, verbalize and document stakeholders’ commitment to supporting EBI implementation and sustainment, and create a clear plan that outlines each entity’s responsibilities and mechanisms for ensuring accountability.

This study also provided a unique opportunity to clarify the dynamics of collaboration in partially- and non-sustaining sites. Both quantitative and qualitative results suggest that collaboration was not strong in these sites. For example, meetings between stakeholders were less frequent, and higher-level government administrators were less invested in embedding SC within service systems. This lack of “buy in,” in turn, may have dissuaded CBOs in investing their own human capital into SC. Whether or not collaboration emerged as a positive force for EBI sustainment ultimately appeared dependent on government administrators who occupied positions of authority at the highest system levels. Without local champions behind SC, it is clear that the establishment of training, coaching, and other infrastructural supports, particularly those that could be shared across CBOs and home visitors, was not prioritized at these sites. Hence, it is important to identify champions for EBI implementation, provide them with support and recognitions, and develop secondary champions to buffer the effects of potential turnover.

Stakeholders in the partially- and non-sustaining sites also spoke less enthusiastically about the benefits of collaborating with the academic researchers and did not pursue ongoing relationships after the initial implementation period. In contrast to stakeholders in the sustaining sites, who viewed the collaboration as positive for the CBOs and home visitors when there was mutual feedback (e.g. the CBO administrator who was happy that the academic researchers responded to concerns about the paperwork), stakeholders in the non-sustaining sites viewed the research requirements of the academic researchers as a burden to implementation that did not provide tangible benefits to CBOs and home visitors. Moreover, these stakeholders were less likely to turn to the academic researchers for guidance and more likely to express the belief the SC was too rigid in its structure and inappropriate for their service milieu.

Importantly, the types of collaborations that characterized these service systems were not as methodically cultivated as is usually the case in successful planned CBPR and CPPR initiatives.(Israel, Eng, Schulz, & Parker, 2005; Jones & Wells, 2007). Instead, these collaborations typically arose out of necessity, i.e., to share training and coaching expertise across multiple CBOs in order to ensure adequate funding for day-to-day service provision. It is possible that the lack of a formalized and planned collaborative approach may have contributed to communication breakdowns and struggles to maintain relationships among stakeholders that characterized non- and partially-sustaining sites. Additionally, enhanced collaboration with academic researchers and intervention purveyors during the implementation phase may have led to increased ability to adapt the EBI to local contexts, hence enhancing its applicability and decreasing concerns regarding the rigidity of the model.

In fully sustaining sites, collaboration to support EBI was also most successful when resource sharing was built into infrastructure and institutionalized. For example, the interagency home visitor teams were able to maximize resources and expertise across their service system. In contrast, resource-sharing that functioned as a stop-gap measure had the potential to increase the vulnerability of SC, as in counties where temporary and ad hoc funding arrangements threatened SC sustainment. Collaboration also appeared to be a prominent factor in SC sustainment in systems where lead administrators of CBOs were less likely to consider themselves competitors of other CBOs and worked together in ways that strengthen the overall service delivery system rather than their own specific organizations. It remains to be seen whether funding agency or government efforts to increase competition among CBOs in service systems may threaten existing collaborations over time. For example, all CBOs in the current study were located within the service region they covered; however, attempts to expand contracting process to out-of-state/county service providers could potentially destabilize existing collaborations (Willging et al., 2015).

Increased formalization of collaborative processes is not without its own drawbacks if power imbalances across stakeholder groups are neglected. This research suggests that in the fully-sustaining sites, collaboration is not simply about imposing requirements on specific stakeholder groups to partner with other stakeholder groups, but is about providing space for ongoing interaction, co-learning, and troubleshooting to occur. Having this space enables stakeholders across levels to collectively reflect on and celebrate the outcomes of SC, which, in turn enhances their investment in the program, and may even result in higher ratings of worker satisfaction and job commitment (Chang, Ma, Chiu, Lin, & Lee, 2009). However, in contrast to CBPR where the goal is to overturn power imbalances across stakeholder groups, it is not so easy to equalize the playing field when it comes to EBI implementation in public service systems. Yet, what appears to make collaboration so central to EBI sustainment in this study is when meetings are not simply imposed by stakeholders at the higher levels but when this critical space for reflection is nurtured and all participants find that their perspectives have a bearing on EBI implementation. If collaboration is mandated but not supported for all participants, then some stakeholders may dismiss it as a bureaucratic requirement and may be less likely to fully engage in collaborative relationships that can support EBI practice. This study suggests that the participation of coordinators, such as the academic researchers, can facilitate the collaboration necessary to implement and sustain an EBI without being perceived as overly directive by system stakeholders.

Limitations

This work occurred in two U.S. states and examines efforts to sustain one specific EBI, which constrains generalizability. Additionally, as the study was naturalistic and levels of collaboration were not systematically varied, our results and conclusions are limited to observations of differences among existing service systems. However, our honed focus on 11 separate services systems allowed us to conduct rich in-depth interviews and focus groups with a diversity of stakeholders playing key roles in state- and county-run child welfare systems. Additionally, the focus on one EBI adds to our ability to document similarities and differences in the role of collaborations within similar implementation parameters. The sample size and the fact that not all stakeholders completed the quantitative measure limits our quantitative analysis. Despite the small sample, a diversity of stakeholders in systems with long-term sustainment, partial sustainment, and failed sustainment were interviewed and relatively high levels of saturation, or consistency of data, were found. Due to our focus on the sustainment phase, power struggles that affected the early stages of collaboration around SC within these sites were not examined but are described elsewhere (Aarons et al., 2014c; Hurlburt et al., 2014). Similarly, having an in-depth focus on collaboration limited our ability to describe other relevant factors that likely impact sustainment such as leadership, client populations, and funding/policy initiatives. This study is further limited by the lack of input from child welfare service recipients. Although the inclusion of such voices is part of the foundation of more formalized collaborative initiatives such as CPPR and CBPR (Jones & Wells, 2007; Wallerstein & Duran, 2010), these service recipients did not play a direct collaborative role in the more naturalistic implementation efforts described here. Yet, these service recipients may exert an indirect role in the sustainment of SC, as evaluation data such as recidivism rates, service retention, and satisfaction scores may influence a site’s decision to continue, adapt, or terminate SC services (Aarons et al., 2012).

Conclusion

The present study provides evidence of the supporting role of effective collaborations in EBI sustainment across 11 different service systems. Attention to collaboration is increasingly essential in public child welfare service systems, where diffusion of functions and responsibilities across multiple private agencies is the norm. Without attention to effective collaboration, the involvement of multiple stakeholders may lead to chaotic operations, reduced accountability, and lack of cohesion. Future research should examine the degree to which inter-organizational networks can enhance collaborations and the degree to which key formal and informal leaders can facilitate or inhibit the development and/or maintenance of collaborations critical for the effective implementation and sustainment of EBIs to improve outcomes for service systems and the clients they serve. Although there is a fast growing literature on how to capacitate or empower organizations for effective EBI uptake and provision (Aarons, Ehrhart, Farahnak, & Sklar, 2014a; Damschroder et al., 2009), further research must attend to strategies for nurturing collaborations in multi-level service systems to facilitate not only EBI delivery, but sustainment as well (Aarons et al., 2014b). Research on such strategies must not ignore the reality of power imbalances that can affect their implementation and must also acknowledge service users as stakeholders who may have important contributions to share. Attention to such issues is crucial for participatory approaches to public health intervention and are deserving of greater regard in relation to EBI sustainment.