Towards creating synergy among policy, procedures, and implementation of evidence-based models in child welfare systems: Two case examples

Abstract

Over the past four to five decades, multiple randomized controlled trials have verified that preventive interventions targeting key parenting skills can have far reaching effects on improving a diverse array of child outcomes. Further, these studies have shown that parenting skills can be taught; and they are malleable. Given these advances, prevention scientists are in a position to make solid empirically-based recommendations to public child service systems on using parent-mediated interventions to optimize positive outcomes for the children and families that they serve. Child welfare systems serve some of this country’s most vulnerable children and families, yet they have been slow (compared to juvenile justice and mental health systems) to adopt empirically-based interventions. This paper describes two child welfare initiated, policy-based case studies that have sought to scale-up research-based parenting skills into the routine services that caseworkers deliver to the families that they serve. In both case studies, the child welfare system leaders worked with evaluators and model developers to tailor policy, administrative, and fiscal system practices to institutionalize and sustain evidence-based practices into usual foster care services. Descriptions of the implementations, intervention models, and preliminary results are described.

Children and adolescents engaged in the child welfare system (CWS) are among the most vulnerable in our society. They have typically experienced trauma, are poor, and have been exposed to multiple other early adverse experiences. As noted by Biglan (2016), research in prevention science has shown that it is possible to reduce the psychological and biological impacts of such toxic stress, to repair children’s confidence by focusing on their individual strengths, and to protect children from opportunities to participate in risky behavior. Half of boys and girls in foster care experience mental health problems at a clinically significant level, making them challenging to parent (NSCAW Research Group, 2002). To properly care for and nurture these youngsters, we need to prepare and support foster and relative/kinship parents to use parenting skills that have been proven to be effective. With few exceptions such as the examples discussed here, our U.S. child welfare systems have been slow to implement evidence-based parenting interventions even though these models have been well established in numerous research studies as being effective for over a decade. These evidence-based parenting strategies, although straightforward, are often the hardest to implement in situations where children need them most. Foster care is certainly one of those situations.

KEEP (Keeping foster and kin parents supported and trained) and Parent Management Training Oregon (PMTO) are well-researched parenting models developed at the Oregon Social Learning Center as part of a suite of programs (Dishion, Forgatch, Chamberlain & Pelham, 2016) designed to strengthen parent skills and supports for nurturing children and adolescents towards optimal development. The goal of these models is to help parents (foster, relative, biological) learn and practice daily strategies that optimize the child’s prosocial development and increase the probability that the home setting is experienced by the child as a nurturing place rather than a traumatic one. For example, previous research shows that providing CWS foster parents with training and support yields positive results for children and adolescents and for the CWS (Price et al., 2008). Positive youth outcomes include lower rates of high-risk behaviors such as substance use, delinquency, and teenage pregnancy, and positive outcomes for the CWS include shorter lengths of stay in foster care and fewer placement disruptions (Horwitz, Chamberlain, Landsverk, & Mullican, 2010). In addition, key mechanisms that drive outcomes have been identified in several previous studies (Chamberlain et al., 2008; Price et al., 2008). For example, high levels of child/adolescent externalizing problems predict placement disruptions from foster care. Disruptions from foster home placements and multiple placements further drive children’s experiences of trauma and increase mental health problems that are costly to the individual, the CWS, and society. On the other hand, parental reinforcement, close supervision, the use of non-harsh consistent limit setting, and school involvement are well-documented protective factors (Leve et al., 2012).

In this paper we describe two examples where child welfare system leaders initiated policy-driven reforms in their foster care services by scaling up two of the above referenced evidence-based parenting interventions: KEEP for foster/relative parents and Parenting Through Change for Reunification (PTC-R) for biological parents. (PTC-R was adapted from the PMTO model for biological parents with children in foster care; Forgatch & Patterson, 2010.) In addition, we developed and implemented a new casework practice model, R³, designed to integrate the KEEP and PTC-R principles into the daily interactions that casework supervisors have with caseworkers. The implementation of these models into the existing CWS cultures involved a phased multi-level effort informed by the EPIS conceptual framework (Exploration, Preparation, Implementation, and Sustainability; Aarons, Hurlburt, & Horwitz, 2011), which identifies global factors that influence outer and inner contextual variables affecting implementation of child and family interventions in child welfare and other public service sectors. Examples of outer context factors are the overall service environment, political climate, and funding; inner context includes factors such as leadership, agency characteristics, and worker attitudes. As described below, our experiences in the implementation in both case examples affirmed that different contextual variables played critical roles at different phases of the implementation process, as is posited by the EPIS model.

The implementation of KEEP, PTC-R, and R³ into existing program policies and administrative and fiscal procedures involved the coordination of multiple levels of interrelated initiatives designed to work synergistically with the interventions to optimize their effects. The first example took place in a large urban child welfare system that contracts out all foster care casework management to private agencies. The second is a state-run system with centralized management where the scale up began in four rural regions and is expected to expand statewide. In both examples the scale ups were initiated by policy directives from the CWS agency leadership aimed at improving outcomes for children and families in foster care. Prior to the decision to implement the reforms described below, the CWS leadership teams in both locations had engaged with Fred Wulczyn and colleagues at the Center for State Child Welfare Data at Chapin Hall, University of Chicago to understand and identify their system needs and to explore strategic plans to improve their “business as usual” practices to improve specific outcomes such as decreasing length of stay in care and rates of placement disruption. This entailed identifying where, within each system, what reforms would be launched and how the selected interventions designed to accomplish the reforms could be supported by modifying existing program policies, administrative rules, and fiscal procedures. The integration of interventions into the CWS at multiple levels (i.e., policy, administrative, and fiscal) has tremendous advantages over implementing interventions as solitary, niche models in terms of institutionalization of new practices and the potential for long-term sustainability. First, we provide a brief overview of the two intervention models that were selected to accomplish the reforms and the practice model that was newly developed to strengthen staff implementation of the principles of intervention models. Then, we provide two case examples aimed at illustrating the ways in which systems with dissimilar cultures and geographies implemented policy-driven reforms and adapted administrative procedures and fiscal incentives to work in unison with key features of the intervention models.

Description of the Interventions

KEEP

KEEP is a training and support intervention developed with direct input from foster and relative parents that targets the following outcomes: (a) increasing parenting skills and confidence, (b) decreasing the number of foster care placement disruptions (lateral moves and step-ups to group care placements), (c) improving child behavioral and emotional problems, and (d) increasing the number of positive placement changes (e.g., reunification, adoption). KEEP has been found to be effective at achieving these outcomes in randomized controlled trials (Chamberlain et al., 2008; Price, Chamberlain, Landsverk, & Reid, 2009), and quasi-experimental designs (Greeno et al., 2015). More specifically, KEEP has been shown to improve child problem behavior which in turn mediates placement stability outcomes for children in foster and relative care (Chamberlain et al., 2008; Price et al., 2008).

A major principle of KEEP is that foster and relative parents can serve as key agents of change for children. This is accomplished by strengthening caregivers’ confidence and skills so they can change their child’s behaviors, teaching effective parent management strategies, and providing the caregivers with support. The intervention is delivered in the context of a parent group, where foster parents interact with one another guided by two group leaders. Parents are encouraged to complete a home practice each week related to the session content. Each session begins by debriefing the home practice with parents and tailoring the KEEP strategies to the situation in their home with their child(ren).

To learn the model, group leaders participate in a 5-day experiential training that includes information about the program’s theory and practice in the delivery of group sessions. During training, each trainee role plays facilitating several key sessions while other trainees act as foster/relative parents. KEEP is delivered in 16 weekly group meetings (90 minutes each), and includes detailed manuals for group leaders and for foster/relative parents. Group leaders tailor the session content based on issues and ideas raised by the group participants.

Fidelity to the KEEP curriculum is monitored closely and is measured across three dimensions including: (1) content, (2) process, and (3) structure, using a standardized rating protocol (Facilitator Adherence Rating; FAR; Buchanan, Chamberlain, Price, & Sprengelmeyer, 2013). During groups, the facilitators record each session using a laptop with software that enables the recording to be uploaded to a HIPAA-compliant, secure website designed to allow for direct observation of group sessions (Fidelity Observation System; FIDO). KEEP expert consultants view the recordings, rate them using the FAR, and identify areas for reinforcement and feedback. The recordings then are used in weekly consultation meetings (1.5 hours each). Prior to the consultation, group leaders complete a session review form with questions about what went well and challenges experienced. They also complete weekly forms on parent attendance and engagement ratings. Each of these measures informs the consultation process.

Parenting Through Change for Reunification (PTC-R)

PTC-R is a parent skill building and support model training program for biological parents whose children are living in foster care. PTC-R was adapted from the Parent Management Training Oregon model (PMTO; Forgatch & Patterson, 2010). PTC-R addresses key clinical issues in a parent group format. It is designed to increase effective parenting practices to decrease child behavioral and emotional problems and increase parent and child prosocial behavior. Parents learn to provide a nurturing, consistent, and safe family environment with an emphasis on strategies that deal with a wide variety of contextual adversities and stressors. Parents are engaged in exercises that teach core intervention components to decrease coercive and inconsistent parenting and increase effective parenting (skill encouragement, limit setting, monitoring, positive involvement, emotional regulation, and problem solving). Numerous studies have reported improvements in these parenting practices, which in turn produce positive outcomes for children, including reduction in behavior problems, police arrests, rates of out-of-home placement, delinquency, deviant peer association, and depression and improved academic performance and social skills (Forgatch, Patterson, DeGarmo, & Beldavs, 2009; Patterson, Forgatch, & DeGarmo, 2010; Forgatch et al., 2009; Forgatch & DeGarmo, 1999). The program has been replicated in numerous randomized controlled trials and quasi-experimental designs by independent research teams nationally and internationally (e.g., Akin et al., 2014; Kjøbli, Hukkelberg, & Ogden, 2013; Ogden & Hagen, 2008). PTC-R is delivered in 10 group sessions followed by an additional 6 sessions near the time of reunification.

Similar to KEEP, PTC-R trainees participate in a 5-day experiential pre-service training. Once trained, PTC-R interventionists deliver the program to parent groups in 90-minute sessions. Group leaders recorded each group session using a laptop with software that enables the recording to be uploaded to FIDO. The recordings are then viewed by the developers, coded for fidelity, and used in weekly consultation meetings (1.5 h each). Prior to the consultation, facilitators complete a session review form with questions about what went well and challenges experienced, and complete weekly forms on parent attendance and engagement ratings.

Because KEEP and PTC-R were shaped from the same behavioral and social learning frameworks often utilized in interventions developed at OSLC, the integration of the two in the project proved to be a natural fit. In the first case example, CWS leadership determined that all participating caseworkers and supervisors would be trained to provide both KEEP (for foster/relative parents) and PTC-R (for biological/adoptive parents). The reform was designed so that caseworkers would implement these two interventions as part of their usual casework routines, rather than the previous strategy of referring parents out for support and parenting services.

R³

To strengthen the reach and impact of KEEP and PTC-R principles, a casework practice model was developed by OSLC researchers (Saldana, Chamberlain, & Chapman, 2016) at the request of CWS leadership. R³ stems from the same social learning principles of reinforcement used in KEEP and PTC-R. The idea of R³ was to embed reinforcement strategies into all interactions that frontline staff have with children and families in the foster care system To broadly implement R³ we elected to train casework supervisors to use the R³ reinforcement principles with their caseworkers in routine group supervision meetings and, in turn, supervisors trained the caseworkers to reinforce the efforts, strengths, and small steps towards achievement of case goals of the parents on their caseloads (biological, foster, relative) in the context of their daily interactions.

Case Example 1: New York City

Prior to the foster care reform, under the Bloomberg administration, the New York City Administration for Children and Families (ACS) had implemented numerous evidence-based preventive interventions models in juvenile justice and in the CWS, including working with the Oregon Social Learning Center intervention development team. Capitalizing on this experience, ACS elected to initiate a foster care system reform in 2012 to achieve the following four goals:

1. Decrease the rate of foster care placement disruptions by increasing the supports for and skills training of foster and relative parents.

2. Decrease the number of children placed per home to a maximum of 3 (unless children were related; sibling groups).

3. Decrease the average length of stay in foster care by developing a parent/caregiver skill building and engagement model.

4. Decrease re-entry rates into foster care for families who had been reunified.

To accomplish these goals, ACS elected to focus on strengthening caseworkers’ skills and supports for interacting effectively with foster/relative parents using KEEP and with biological parents using PTC-R. Additionally, ACS requested that a supervisor practice model be developed to extend the reach of KEEP and PTC-R principles to interactions between supervisors and caseworkers. KEEP and PTC-R share roots in social learning theory, which posits that people modify their behavior by observing reinforcement received by others, and that all individuals exist within and respond to their environments in an adaptive way (behaviors that they are being reinforced for increase). The R³ model capitalizes on the idea that the social context within the system involves relationships between multiple agents (e.g., leadership, supervisors, and caseworkers) that ultimately affect outcomes for families. Supervisors were selected as the target for the R³ model because they are positioned central to the organizational leadership and caseworkers, the on-the-ground workforce who make daily decisions that affect the outcomes of the families being served.

Context and Preparation

ACS contracts with over 35 private agencies to do all case planning and case management services for families involved in the New York City foster care system. ACS wanted to select five “pilot” agencies to implement the reform that varied in size, geography, culture, and level of past performance on key outcomes. ACS leaders consulted with Fred Wulczyn and colleagues at Chapin Hall to select five contracted agencies serving a total of over 2,000 children per year (ages 2–18 years; approximately 20% of the foster care population at the time). Chapin Hall researchers utilized data from their Center for State Child Welfare Data archives to assist ACS in selecting representative agencies.

Program Policy

A major policy change enacted by ACS under this reform was to have caseworkers implement the KEEP and PTC-R interventions themselves rather than to refer parents to outside sources for these services as had been done previously. This new ACS policy was motivated by the desire to provide consistent, high quality, evidence-based parenting interventions to all caretakers of foster children (foster, biological, relative, adoptive). Having caseworkers directly provide these services had several potential advantages for families including the possibility of enhanced family engagement with the CWS, greater coherence in their individual case plan, and the opportunity to experience a more supportive relationship with caseworkers. From a system improvement perspective, caseworkers had first-hand knowledge of the parent’s level of skill, confidence, and commitment, and caseworkers potentially benefited by improving their own skills after being trained in well-researched intervention that is shown to improve parenting skills and decrease child problem behaviors. We hypothesize (although we have yet to test) that the R³ practice model has the potential to decrease caseworker stress.

Administrative

As described more fully by Wulczyn and Feldman (in press), the following administrative changes were introduced. Caseworker caseloads were reduced to no more than 12 cases to allow time for the caseworkers to be trained and to implement the evidence-based models. All agency caseworkers and supervisors participated in 10 days of in-person training over approximately 3 months (5 days for KEEP; 5 days for PTC-R). Staff in each agency was divided into two cohorts so that while one half of the caseworkers and supervisors participated in trainings, the other half could continue to provide all usual services during the training days. During the experiential training, each caseworker participated in role plays by facilitating several key sessions while other caseworkers acted as participating parents (foster/relative in KEEP; biological in PTC-R). After caseworkers completed the training, they conducted 1–3 “mock” groups to practice key content elements and gain experience with the fidelity monitoring video uploading system. This system was used to monitor the delivery of the interventions, to track participation rates, parent engagement levels, child behavior problems, parent stress levels, and other activities related to the implementation. This information was used for assessing progress and barriers, and for providing feedback and consultation to group leaders on a weekly basis during telephone conferences. Once KEEP and PTC-R groups were up and running, caseworkers and supervisors were trained in R³, a model that aims to influence the “how” (not the “what”) of casework practice. In the R³ model, supervisors are provided with training and weekly consultation. The supervisory role is critical to organizational climate because they are centrally positioned in the system between caseworkers and leadership. The three Rs include reinforcement of: (1) parental efforts, (2) parental relationships and roles, and (3) parents’ small steps toward goal achievement. These three reinforcement targets are encouraged in interactions between supervisors and their supervisee-caseworkers and between caseworkers and the families they serve.

Fiscal

As detailed in Wulczyn and Feldman (in press), the reform was designed so that the potential cost savings generated by the outcomes of KEEP and PTC-R would offset the up-front investments made to facilitate the administrative changes noted above plus the costs of implementing the two EBPs and R³. The lowered caseloads and implementation and training costs were considered necessary core investments in the project that were expected to be offset by the savings generated from the EBPs and the other innovations. Therefore, in order to achieve savings sufficient to neutralize costs of the reform, ACS calculated that a 17% reduction in each of three key project goals (placement disruptions, length of stay, and re-entry rates) would need to be attained.

Agencies were reimbursed by ACS for their participation in the trainings and for providing financial incentives to parents (foster and biological) for attending weekly group sessions. The incentives were intended to cover costs such as transportation and to provide parents with a modest stipend to acknowledge their commitment to participate; parents received $25 per session for attendance and if they attended 80% of sessions, they received a $100 bonus.

Evaluation

The evaluation was conducted by Chapin Hall researchers under a separate contract. First, it aimed to examine whether the KEEP and PTC-R models were implemented as designed and whether the reforms had the intended impact. ACS asked Chapin Hall to address the following research questions in line with the goals of the reform:

• What was the impact on children’s lengths of stay in foster care, both for children who entered placement after implementation as well as children in-care when the models were implemented?

• What was the impact on the stability of children’s placements?

• What was the likelihood of re-entry into care once permanence was achieved?

Four sources of data are used to evaluate project results. First, Chapin Hall research staff conducted one-on-one interviews with system stakeholders over the course of the first year, including provider agency staff, public agency staff, and the developers of the clinical models. The interviews were designed to get an in-depth description of the implementation experience from various perspectives, particularly as it related to changes in the process of care, the quality of care, and the extent to which the necessary system (administrative) capacity adjustments were made to support implementation efforts.

Second, an online survey was administered to caseworkers and supervisors from the five agencies. The survey was completed at the middle and end of the first year of implementation and included questions about employee job satisfaction and the extent to which job satisfaction had changed.

Third, data was used from the KEEP/PTC-R web-based fidelity management system that included video uploads of all KEEP and PTC-R sessions, fidelity ratings, attendance, parent engagement ratings, and other associated data. These data were used in conjunction with data from administrative records, the fourth source of data. Administrative data included information related to children’s placements in out-of-home care such of dates of entry and exit, type of placement, and placement moves. Together, these data were used to create an analytic data file that allowed for the measurement of potential project-specific effects.

Results on length of stay and placement stability outcomes are reported in detail by Chamberlain and colleagues (2016) and by Wulczyn and Feldman (in press). Analyses on reentry rates have not yet been conducted. As this was not a randomized controlled trial, alternative available comparison groups were used that included: (1) outcomes for children who had been served by the 5 agencies prior to the reform, (2) outcomes for children who were in-care at non-project agencies prior to the time the project was operational, and (3) children who were admitted to non-project agencies after the date the project became operational. To assess the rate of permanency, controlling for agency and period effects, children exposed to KEEP, PTC-R, and R³ were compared with children served by non-project agencies together with children served by the agencies outside of the project period. The rate of permanency (days to reunification) for children in the five intervention agencies was greater than the rate for all other children in the comparison group. This difference was statistically significant. Placement stability was assessed controlling for agency and period effects. An 18% slower rate of placement movements was observed in the KEEP-exposed homes; this effect did not reach statistical significance although the effect showed a trend towards increased placement stability in intervention agencies.

Sustainment

We implemented a full transfer model of sustainment as conceptualized by Forgatch and colleagues (2016). In the full transfer model, KEEP and PTC-R group leaders who have met performance and work load criteria are eligible to be trained as local trainers and consultants. Once trained, they provide the ongoing services for their agency staff that model developer team provided during the implementation phase of the project. These services include viewing video uploads of group sessions, fidelity ratings, weekly coaching for group leaders, and training new staff members. The model developer teams continue to conduct periodic fidelity assessments and provide additional training/consultation as needed. All five agencies in NYC have achieved full transfer in both KEEP and PTC-R.

Case Example 2: Tennessee

The State of Tennessee was selected by the U.S. Department of Health and Human Services as one of eight Title IV-E Waiver states in 2013. Awarded states are allowed greater flexibility in how they utilize their federal funds to address their most challenging child welfare problems. Waivers do not provide new money to the state, but rather allow them to utilize portions of their dollars in ways not traditionally covered by federal dollars, such as adopting new practice strategies in their communities. The Tennessee leadership elected to implement KEEP and R³ as part of the In Home Tennessee model to address their waiver goals which are:

1. To increase permanency for youth,

2. To reduce time in foster care, and

3. To promote positive outcomes for children and families in their homes.

Context and Preparation

Tennessee decided to first focus their efforts for KEEP and R³ in four regions (out of 12 total) in East Tennessee that showed relatively higher placement disruption rates and children remaining in care for periods above the state average; KEEP was first implemented in 10 counties in the 4 regions and then R³. A second cohort of an additional 14 counties in those same 4 regions was then brought on board with KEEP. The aim is to eventually roll out these models in all 95 Tennessee counties in the additional 8 regions over the next three years. The State IV-E waiver is time limited so it is imperative to Tennessee’s CWS leadership that there is a clear path and strategy to maintain the investments in workforce training that they are making under their waiver plan. We collaborated with Tennessee CWS leaders to structure a full transfer plan whereby a solid internal infrastructure for training and maintaining KEEP intervention strategies was devised. In February 2015, we began planning with leaders at Tennessee’s Department of Children’s Services (DCS) to implement KEEP with children placed in relative and foster care homes and to implement R³ with all supervisors of custodial (foster care) and non-custodial casework staff. We estimate that once rolled out statewide, 175 regional administrators, 300 supervisors, and 1,200 caseworkers and over 8,000 children will be exposed to KEEP and R³.

Program Policy

As in NYC, the Tennessee CWS leadership changed their service delivery policy such that frontline staff directly delivered the KEEP intervention to foster parents with the aim of providing them with consistent evidence-based parenting skills and to give caseworker and other frontline staff training and support for enacting those skills. Additionally, to attempt to extend the reach of KEEP, supervisors are being trained in R³. Tennessee leadership elected to contract with private agencies to implement services to biological parents; plans are underway at this time to implement the Nurturing Parent model to support biological parents (www.nurturingparenting.com).

KEEP is being implemented with foster and relative parents of 4–12 year olds, and R³ is being implemented with all supervisors of custodial (foster care) and non-custodial staff. Fidelity monitoring occurs for both models using a web-based fidelity monitoring system which has been continually upgraded to include the ability to generate reports relevant to CWS leadership, local offices, and regional administrators. In addition to tracking attendance, completion rates, session fidelity scores and parent engagement, we now also track staff participation in consultation and training. These data will be used by the Chapin Hall research team to evaluate the effects of dosage on child and system outcomes. As with KEEP, feedback is provided to supervisors quickly to maximize the potential that families receive consistent exposure to the model’s core principles. Additionally, as with KEEP sessions, video recordings of R³ supervision sessions are sent to the model developer for fidelity coding and feedback. This level of observation-based, rapid feedback, allows for more immediate reinforcement and modeling of the KEEP and R³ strategies during weekly consultation between Oregon and Tennessee.

Administrative

In Tennessee the state DCS provides all case management services for foster homes through their regional and local offices rather than contracting case management services out to private agencies as was done in New York City (Case Example 1).

Using lessons learned from the NYC scale up of KEEP, we modified the rollout strategy for Tennessee. In NYC, all agency caseworkers were trained to deliver the interventions whereas in Tennessee only a small group of caseworkers were selected to be KEEP group leaders in each of the four regions. The advantage of training all workers as we did in NYC is that all became familiar with the intervention goals and theory and therefore were in a good position to support foster and relative parents to enact the KEEP parenting skills. Three disadvantages of intensively training the total caseworker workforce were: 1) the cost of training large groups of caseworkers, 2) the time required to be off-line to receive the training, and 3) the fact that once their initial cadre of foster/relative parents completed KEEP, numbers of new parents requiring the training in subsequent years were smaller thereby needing the services of fewer KEEP group leaders that we had trained.

In Tennessee, in order to maintain a culture of caseworker support for and knowledge of KEEP and to conduct a more cost-effective rollout, in partnership with DCS leaders we decided to expose all caseworkers and supervisors to a two-day Foundational Training focused on the theory behind KEEP and an overview of the parenting skills being taught and reinforced. Like the intensive five-day training, this was interactive and included role plays, exercises, and games. After the Foundational Training, we collaborated with DCS leaders to select 4–10 staff from each region to receive the five-day intensive KEEP training.

Fiscal

As mentioned above, waiver funding allows for the redirection of funding to implement practices to improve child and family outcomes. Further, the Tennessee DCS fiscal team elected to reimburse parents for completing KEEP by increasing their daily foster care reimbursement rate. This is a creative use of fiscal incentives that appears to be meaningful to foster parents in that it has facilitated the successful recruitment of parents to participate in KEEP groups.

Evaluation

As part of the federal waiver requirements, states are required to include an evaluation plan to determine if their goals are impacted. Like in NYC, Tennessee contracted with Chapin Hall to conduct this independent evaluation. Administrative data records will be used to determine length of stay and placement disruption outcomes. Qualitative interviews with frontline staff and leadership will also be conducted. The implementation of KEEP and R³ are underway. Therefore, the evaluation and results of the waiver goals are not yet available for this project.

Summary and Conclusions

During the past decade, the reasons for lack of use of evidence-based interventions in public service systems are becoming better understood (Saldana, Chamberlain, Wang, & Brown, 2012) although there is still much to be learned about how researchers and policy decision makers can partner to plan for, select, accomplish, and sustain implementations that result in better outcomes for children, youth, and families. The EPIS model provides a conceptual framework that underscores common factors, barriers, and facilitators that relate to implementing and sustaining interventions in public service systems such as the ones described here. As described in EPIS, the outer contexts of both of the described system reforms were influenced by and an array of political, social, and structural factors. For example, each of these CWS systems was involved in obtaining or implementing system-wide Title IV-E waivers. In each case the interventions described here represent only part of those larger waiver efforts; other waiver activities are simultaneously taking place (such as enhanced assessments, influences on case load size, and other selected services). In both examples, the intention is to accomplish the implementation of the interventions during the waiver period and to then continue to sustain the interventions long-term after the waiver period ends.

In other ways the outer contexts were quite different for the two case examples. Case Example 1 takes place in a city-run system in a large urban area where all case management services are contracted out to private agencies, and group leaders and supervisors who implement the interventions are employees of those private agencies. In Case Example 2 the interventions are implemented by state-employed caseworkers and supervisors working in a centralized state-run system that includes both rural and urban areas. In Case Example 1 the private agencies belong to a network of private providers which gives them a voice to negotiate with the funders (i.e., the Administration for Children’s Services) around specific aspects of the implementation such as caseload size reductions; a situation not present in Case Example 2. In terms of the inner contexts, in Case Example 1, the five participating agencies vary on organizational characteristics as might be expected. Examples of some of those variances include: some had experience implementing EBPs, others did not; some agencies specialize in serving specific populations (e.g., Hispanic families), others are generalists; agency size, the number of layers of management staff, and where within those layers the KEEP and PTC-R program champions are situated vary as well. While there is some organizational variance in the different DCS regions in Case Example 2, there is also the desire by this state-wide centralized system to achieve consistency in administrative and fiscal policies and procedures. How and whether these differences relate to implementation success and sustainment will be explored in future work.

Although this work represents a potential contribution to the literature in that it describes details of how multiple theory-driven interventions are being implemented in public child welfare systems, there are also limitations. Some of these relate to the fact that the interventions are not implemented in the context of well-controlled research trials but rather are being conducted in real-world CWSs within the context of larger waiver-driven reform projects. Waivers provide the motivation and opportunity for systems to make innovative reforms but they are also time-limited. The work described here focuses on implementations enacted during waiver periods. Although the intention is to change casework and supervisor practices in ways consistent with the goals of the interventions, it is unknown at this time if interventions will be sustained post-waiver when outer contexts are likely to go through considerable changes. Second, in each case example there are additional waiver activities beyond the interventions described here that are being conducted in each system. It is likely to be challenging to determine whether it was the interventions per se or the interventions plus the additional waiver activities that drive any positive outcomes that are observed. Similarly, it is unclear how the individual interventions contribute to the overall outcomes relative to each other within this multi-intervention context. Additionally, it will be difficult to quantify how apparently key factors such as changes in leadership at the middle and upper management levels and changes at the frontline caseworker and supervisor levels will affect implementation processes and sustainment outcomes. Partnering with qualitative researchers may improve our ability to understand the impacts of these inevitable changes.

The two case examples describe work that was initiated child welfare system leaders to address the real-world demands that they and the children and families they serve face on a daily basis. In these two case examples we have had the opportunity to help create systems and strategies for caseworkers and supervisors to provide evidence-based services directly to families and to integrate those services into the daily work of their complex systems. This rewarding work has included high levels of compromise, coordination, detailed planning, and above all close communication and partnerships among CWS system leaders, program evaluators, and model developers.

Finally, as noted in Wulczyn and Feldman (in press), in the field of health care, interventions that are synergistic with policy, administrative, and fiscal procedures are becoming more commonplace; (Huang, Drewnosksi, Kumanyika, & Glass, 2009; Trickett & Beehler, 2013) but there remain few examples of such multilevel interventions in the child welfare literature. Moreover, child welfare systems are administratively and financially interdependent. Single-prong interventions that address only one level of the service system while ignoring the interdependencies may ignore the potential for fully integrating and sustaining evidence-based interventions in child welfare and other complex social service systems. The promise of such partnerships is that the results for children and families will be optimized through the work of frontline child welfare caseworkers and supervisors who are supported to deliver research-based services in synergy with their system’s internal policies and procedures.