Summary of Comparison between FFT-CW® and Usual Care Sample from Administration for Children’s Services

Abstract

This evaluation compared the efficiency and effectiveness of Functional Family Therapy-Child Welfare (FFT-CW^®, n=1625) to Usual Care (UC: n=2250) in reducing child maltreatment. FFT-CW^® is a continuum of care model based on the family’s risk status. In a child welfare setting, families received either UC or FFT-CW^® in a quasi-experimental, stepped wedge design across all five boroughs of New York City. The families were matched using stratified propensity scoring on their pre-service risk status and followed for 16 months. The ethnically diverse sample included African American (36%), Asian (4%); Hispanic (49%), and Non-Hispanic White (6%) or Other (6%) participants. Referral reasons included abuse or neglect (57.4%), child service needs (56.9%) or child health and safety concerns (42.8%). Clinical process variables included staff fidelity, service duration, and number of contacts. Positive outcomes included whether all clinical goals were met and negative outcomes included transfers, outplacement, recurring allegations and service participation within 16 months of the case open date. Families receiving FFT-CW^® completed treatment more quickly than UC and they were significantly more likely to meet all of the planned service goals. Higher treatment fidelity was associated with more favorable outcomes. Fewer FFT-CW^® families were transferred to another program at closing, and they had fewer recurring allegations. FFT-CW^® had fewer out-of-home placements in families with higher levels of risk factors. The FFT-CW^® program was more efficient in completing service, and more effective than UC in meeting treatment goals while also avoiding adverse outcomes.

Introduction

The main objective of this secondary data analysis project was to evaluate the impact of an adaptation of Functional Family Therapy (Alexander, Waldron, Robbins, & Neeb, 2013; Robbins, Alexander, Turner, & Hollimon, 2016) on the quality of treatment and prevention services for child maltreatment and allied problem behaviors in a child welfare setting. This adaptation called Functional Family Therapy-Child Welfare^® (FFT-CW^®) was described in a detailed clinical manual (Alexander et al., 2011) and was put into practice with families (n=1625) in all five boroughs of New York City following a detailed implementation protocol (Rowlands & Davidson, 2011). The program was a collaborative effort of the New York Foundling (NYF) organization which provides child welfare services throughout New York City and FFT LLC which provides training and supervision in FFT nationally and internationally.

Consequences of Child Maltreatment

Abuse and neglect pose serious threats to children in the United States with more than 1 million new cases reported each year, and more than 3 million receiving child protective services (US Department of Health and Human Services, 2016). Hussey, Chang, and Kotch (2006) found that childhood maltreatment including supervision neglect, physical neglect, physical assault or sexual abuse were all associated with increased risk (odds ratio) of adolescent substance abuse. Maltreatment contributes to child mortality and morbidity as well as problems such as depression, violence, delinquency and sexual promiscuity (Gilbert et al., 2009; Lucenko, Sharkova, Huber, Jemelka, & Mancuso, 2015).

The impact of child maltreatment is profound and enormous, with these effects reverberating through individuals, families, and institutions including medical and mental health resources, law enforcement, judicial systems, public social services, and nonprofit helping agencies (Gilbert et al., 2009). In 2012, the financial costs of child abuse and neglect were estimated at $220 million each day, or approximately $80 billion each year (Gelles & Perlman, 2012). Fang, Brown, Florence, and Mercy (2012) estimated the lifetime economic burden in the United States from new child maltreatment cases in 2008 to range between $124 to $585 billion.

As maltreated youth enter adulthood, they are more likely to engage in maltreatment of their own children (Jonson-Reid, Kohl, & Drake, 2012; Perepletchikova & Kaufman, 2010) representing a cross generational transmission of maltreatment, substance abuse, delinquency, and risky sexual behavior. These negative health consequences extend throughout adult life (Lucenko et al., 2015; Miller, Chen & Parker, 2011). These adverse influence processes may be particularly acute for families in poor neighborhoods that do not provide informal social control to reduce abuse and neglect (Emery, Trung, & Wu, 2015). Recurring maltreatment may reflect both characteristics of families and their social environments. Even when formal support services exist, families may need assistance in accessing these resources. Community agencies can facilitate access to these resources and family therapy may enable members to benefit from these services.

Adapting FFT to a child welfare organization: Functional Family Therapy-Child Welfare (FFT-CW^®)

This paper examines one approach to address some of these issues – an adaptation of the Functional Family Therapy (FFT) model (Alexander et al., 2013), which has been widely disseminated with over 400,000 families receiving services in 14 countries (Robbins, et al., 2016). The efficacy and effectiveness of FFT for adolescents with behavior problems is well established (Robbins et al., 2016). More model research is available at the FFT website (http://fftllc.com). FFT is an integrated model that combines systems and cognitive-behavioral theories to address a full range of adolescent behavior problems. FFT provides a coherent theory for understanding family relationship patterns and identifying the relational “payoff” or “function” of behaviors within the family. Interventions are organized into distinct phases of treatment (see description of FFT-CW^®-High Risk below) and are matched to relational functions of the family to increase adaptive behaviors and to decrease or eliminate maladaptive behaviors. This focus permits therapists to tailor interventions that are appropriate to the unique characteristics (e.g. strengths, culture, resources) of each family member.

The FFT model is adapted for Child Welfare clients by incorporating a developmental focus to meet the needs of youths across the entire age range (0–18). FFT-CW^® is a relational approach that matches interventions to the relational configurations of families. With delinquent or substance abusing adolescents, this approach often involves accommodating to families in which the youth’s problem behaviors have considerable power to engage and motivate family members into treatment. However, with younger children in FFT-CW^®, it is necessary to implement more “parent-driven” intervention strategies to build skills and create a family context in which youth can flourish. Another adaptation involves expanding the primary treatment focus from a target youth (e.g., delinquent adolescent) to multiple family members. The specific services received through the FFT-CW^® program address mental health, substance abuse, domestic violence, and other needs of family members and tailors treatment to their risks.

The current investigation

The aims of this study were: (1) To evaluate the effectiveness of FFT-CW^® compared to UC services in reducing the rates of recurring child abuse and neglect as well as duration and rates of out of home placement; (2) To evaluate the comparative efficiency of FFT-CW^® on duration of child welfare service while avoiding adverse outcomes, and (3) To evaluate hypothesized moderating and mediating effects of therapists/interventionists (e.g., fidelity), and family (e.g., co-occurring risk factors) on problem behaviors.

Prior investigations (e.g., Chaffin, Hecht, Bard, Silovsky, & Beasley, 2012; Chamberlain, Feldman, Wulczyn, Saldana, & Forgatch, 2016; Green et al. 2015) have identified a number of pitfalls concerning the use of child welfare administrative sets. We have attempted to address these concerns in this report. For example, this study capitalizes upon two features to help mitigate possible selection bias as an alternative explanation for the findings. The first factor was the presence of a staggered implementation of FFT-CW^® across four regions in New York City which helped to control for possible historical confounds. Second was the selection of a matched comparison sample from the more than 15,000 families each year in New York who are referred from ACS for preventive services often after an allegation of child maltreatment.

This research applied procedures to strictly protect client confidentiality since the administrative datasets in these analyses contained sensitive information about clients. The first stage of the project involved electronic datasets accessed through the New York’s Administration for Children’s Services (ACS) which provided extensive information about families before, during, and after the evaluated service episode. The pre and post-service episode data included records of all “allegation investigations” and all foster care placement(s) start and end dates for each family member. These data permitted a comparison between clients in the FFT-CW^® program and clients receiving usual care for at least a 16-month period following the case open date. Authorized ACS staff extracted client records, de-identified them, and created pseudo-IDs which permitted an integration of ACS family records from six different sources. The data from ACS had risk indicators for the families so that each could be stratified using comparable proxy variables to identify the type of interventions which the FFT-CW^® or UC clients needed. The research procedures were approved by (XX) Institutional Review Boards.

1 Method

1.1 Research Design

This secondary analysis project capitalizes, in part, upon the features of a stepped wedge design (Brown & Lilford, 2006) to evaluate system changes over time and a sample matching approach based upon stratified propensity scores to assess selection bias. The study can be represented as a 2 (Usual Care, FFT-CW^®) × 36 (Monthly Cohorts) × 5 (New York City Boroughs: Brooklyn, Bronx, Manhattan, Queens, and Staten-Island) between participant factorial design. The 36 levels of the Time independent variable are determined by the family’s case open date between June 1, 2011 and May31, 2014. Families were followed for 16 months after the case open date. The stepped wedge aspect of the study was based upon the training and implementation schedule of FFT-CW^® clinical teams which occurred in Manhattan during September, 2009, in the Bronx during October, 2011; in Queens during March, 2012, and in Brooklyn and Staten Island which were launched together in July, 2012. We classified families using seven risk dimensions: substance abuse, domestic violence, and mental health safety concerns as well as the family’s history of allegation, their out of home placement, and whether their transfer into the program was from foster care or another child welfare program The FFT-CW^® program was designed to tailor treatments to preservice risk levels using indices comparable to those available in this evaluation.

1.2 Participants

Approximately 60 families per month were enrolled in FFT-CW (n = 1625) at one of the NYF service sites throughout the five boroughs of NYC, with less in the first year due to the staggered implementation. The usual care sample (n = 2250) included approximately 60 families per month who received services at one of 58 agencies that provide child welfare preventive services to 80,000 families each year. ACS randomly sampled families (n ~ 12) each of 36 successive months from each of the five boroughs. These monthly cohorts of families were linked to FFT-CW^® families enrolled in the same month and the same borough. We were able to capitalize upon the logic of stepped wedge designs in which the onset of an innovation is replicated multiple times. We could control for changes over the calendar years (i.e., pre intervention to post intervention) within each of these replications. With this approach, we tried to control for transient historical events (e.g., economic recession, seasonal effects, hurricanes, or political elections) that could influence results. Table 1 summarizes demographic characteristics of clients in the two conditions. The proportion of female children in UC (49%) and FFT-CW^® (48%) was comparable in the two study conditions. Most of the primary caregivers were females (UC: 92%; FFT-CW^®: 93%) who had a spouse or other parent figure (UC: 52%; FFT-CW^®: 49%) in the home while 21% of families had another adult household member.

Table 1.

Demographic Characteristics of Study Samples.

Sample Characteristics	Usual Care M (SD)	FFT-CW M (SD)
Child Age (Years)	8.75 (5.23)	9.22 (5.26)
Primary Caregiver Age	36.75 (10.28)	37.63 (10.97)
Hispanic Family Size	4.33 (1.72)	4.39 (1.76)
African American Family Size	4.36 (1.83)	4.35 (1.93)
Other Family Size	4.11 (1.61)	4.34 (1.82)
Ethnicity/Race/Gender (%)
African American	36.8%	33.4%
Asian	5.7%	1.0%
Hispanic	44.2%	54.3%
Non-Hispanic White	5.0%	5.2%
Other/Unknown	8.3%	6.0%

Note: Cell entries are the means (M) and standard deviations (SD) or the percentage of the sample within the FFT-CW or the Usual Care programs that possesses the indicated attribute.

The caregivers ranged in age from 15 to 79 (M = 39, SD = 11.93). Most of the caregivers were a biological parent (92%), while others were grandparents (3%) or another extended family member. The families had an average of 2.28 (SD = 1.3) children with 33% having a single child. The children’s ages (M = 9.7, SD = 5.1) range from 0 – 19 years, and 60% of families had at least one child older than 10 years. Entry to the FFT-CW^® was unrestricted with regard to gender and race/ethnicity. The ethnic/racial diversity of FFT-CW^® clients reflected the profile of referrals to the NYF Preventive Services, and the distribution of UC sample was comparable. NYF received approximately 60 referrals from ACS and 2 self-referrals per month.

Inclusion/Exclusion criteria

Inclusion into FFT-CW^® required that families present an indication of child maltreatment or other evidence of a concern for a child’s safety or health. Youth/families met criteria for preventive outpatient or intensive outpatient services. Families with a member(s) that were exhibiting severe mental or behavioral problems requiring more intensive services were referred for crisis stabilization or hospitalization. Families often included multiple youth that were referred for services. Referral reasons in the current sample included abuse or neglect (57.4%), child service needs (56.9%) or child health and safety concerns (42.8%). Some families (6.3%) entered treatment voluntarily (walk-ins) because of concerns for a child’s well-being. Referral source details are available in Supplement S.1.

FFT-CW^® Interventionists/Therapists

Interventionists (n = 62) had at least a BA and prior experience in child welfare and nearly all (98%) of the therapists (n = 55) had an MA or MSW. The staff was recruited from their communities so they represent the ethnic/cultural background of residents. Interventionists and therapists received eight days (four 2-day trainings) of didactic, on-site workshops and one-hour weekly group consultation calls led by expert FFT-LLC consultants over one year. Supplement S.2 presents more detailed staff characteristics.

1.3 Measures

Family Assessment and Service Plan (FASP) and Clinician Rating Form

The FASP was completed by a case worker during the intake process. It is used to characterize family strengths, needs, and risks. Each child is rated on 7 dimensions: physical health, mental health, child development, cognitive skills, child behavior problems, alcohol/drug use, and child-family relationships. The primary caregiver is rated on their (1) relationship with other caregivers; (2) ability to cope with stress; (3) motivation/readiness to change; (4) expectations of children (e.g., developmentally appropriate?); (5) acceptance of children; (6) discipline of children (e.g., age appropriateness, any physical discipline?); and (7) problem solving skills. An additional risk assessment is made of the family in terms of (1) housing; (2) financial resources; (3) available social support; (4) domestic violence; (5) alcohol abuse; (6) drug; (7) serious mental health; (8) cognitive skills; (9) debilitating physical condition; (10) realistic expectations of children; and (11) recognition or attention to needs of children. Using the information from these assessments, the intake workers determine if a family is eligible for child welfare services. In the FFT-CW condition, a specified set of decision rules found in the Clinician Rating Form (CRF) are used to determine the family’s risk level by which they are assigned to either the lower or higher intensity service track (Rowland & Davidson, 2011).

SCR-State Central Registry is New York State’s computerized system where reports of abuse and neglect are recorded. This registry conforms to national standards for reporting abuse and neglect. The staff at New York’s ACS extracted initial and recurring reports of abuse and neglect from this data base for clients in the UC condition and those families who received FFT-CW^® at NYF. These allegations and investigations were time tagged so that it was possible to establish an index of the lag time from an initial report to any recurring reports and when they occurred in relation to the evaluated service episode. The family members in this study had 69,470 investigations and 60% of those were determined to be indicated (i.e., supported or substantiated by the investigation). Only instances of indicated abuse or neglect were included in this evaluation. Each investigation also determined whether evidence existed of a substance abuse, domestic violence, or mental health safety concern. The ACS data bases also contained records of all out of home placements for each family member including the beginning and end date of placement (or spell) as well as the type of placement (e.g., relative, foster family, group setting). These files included any foster care spells before, during, and after the current episode.

1.4 Clinical Interventions

FFT-CW^® Low Risk

The FFT-CW^® Low Risk (FFT-LR) intervention is a manualized case management approach for prevention workers charged with providing services to families in the community (Alexander et al., 2011). The FFT-LR program is implemented in three distinct phases: Engagement/Motivation, Support/Monitor, and Generalization. During the first phase, the primary focus is on engaging and motivating youth and families to be a part of a change process by decreasing family conflict and increasing their hope about the possibility for change. Examples of intervention strategies include reframing, creating a strength-based relational focus, and interrupting within family conflict. During the Support/Monitor Phase, the focus is to identify resources and interventions best suited to the family, and to support linkages for those change programs. Interventionists are expected to utilize their case management skills to maintain and enhance the impact of evidence-based interventions on family members. In the final phase, the focus is on helping youth/families to generalize change into other systems and to anticipate and plan for potential barriers or future challenges that youth and families may face.

FFT-CW^®-High Risk

The FFT-CW^® High Risk (FFT-HR) Intervention is also a manualized treatment that is based on traditional FFT. This approach is an evidence based, widely disseminated, cost effective family intervention for at-risk and juvenile justice involved families (Robbins et al., 2016). The FFT-HR model includes five phases: (1) Engagement, (2) Motivation; (3) Relational Assessment (4) Behavior Change; and (5) Generalization (see Alexander et al., 2013). Each phase includes specific techniques of intervention involving a strong cognitive component which is integrated into systemic training in effective family communication, parenting skills, conflict management, and numerous other coping strategies linked to a variety of syndromes and referral problems. FFT focuses on in-session therapist techniques and family interaction processes, which are predictive of positive change. One notable process change appears in family communications, especially negative/blaming patterns.

UC services

Limited information was available about the staff members providing services in the UC condition. Clients and staff were randomly sampled from 58 organizations that provide services to children and their families referred to ACS in New York City (see website). The typical caseload was approximately 12 families. No information was available concerning the training, supervision or fidelity monitoring of the UC programs. All programs were funded through New York’s ACS.

FFT-CW^® training and supervision

A critical collaborative task between the community agency and the program developer was training and supervision. Model adherence and fidelity are important determinants of successful interventions in community settings (Aarons et al., 2012; Chamberlain et al., 2016). Each staff member initially began providing services to four families, while receiving weekly supervision from a trained expert in FFT-CW^®. The clinical staff were expected to present one of their cases during the group supervision sessions. They were then rated on their fidelity to the model as well as their adherence to their administrative responsibilities (Supplement 3 has more details concerning the evaluation of model fidelity).

1.5 Dependent Variables of the Evaluated Service Episode

The critical unit of analysis was an evaluated service episode for each family. For the FFT-CW^® clients, their episode was defined by the opening and closing dates of the family’s participation in the FFT-CW^® program at NYF. Approximately 60 UC families who had initiated service were selected each month without regard for any prior service episodes. The open date for this new episode established the beginning of their evaluated service episode. Families in both conditions had a range of measures that were distributed to three time periods: 1) the pre evaluated service episode period, before the open date, 2) the evaluated service episode, the time between case open and closing dates, and 3) the post evaluated service episode period, after the closing date. The measures extracted from the various administrative data sets had a date field attached so that we could assign events into one of these three time periods. Some measures were linked to individual family members (e.g., indicated investigations or allegations, out of home placements). Other measures were associated with the family or case including referral source, service open and closing dates, closing status, and referral reason(s).

Many of the families had multiple service episodes. For each month from June 1st, 2011 through May 31, 2014, ACS randomly sampled cases which had a service episode opened in that designated month. These cases were paired with the FFT-CW^® cases enrolled during the same month. The open date for their service episode was contingent upon the month in which they were sampled regardless of any prior or subsequent service episode. This approach created 36 cohorts of families. We realized that a variety of social, economic, and environmental changes might occur which could influence administrative decisions about the provision of Child Welfare services. We adopted this cohort sampling strategy so that we could assess changes in the service patterns across the same time interval as the FFT-CW^® program was implemented.

1.6 Analysis Issues

The FFT-CW^® and the UC samples were selected to be approximately comparable in the two conditions. However, neither sample could be considered a random sample of those receiving services. Furthermore, differential selection bias could influence which families entered into the study samples. We recognize that selection biases could create important pretreatment differences factors that could influence the outcomes of treatment. The first phase of the analysis compared UC and FFT-CW^® clients on the measures obtained from ACS.

Propensity analysis

To address the potential of selection biases, we adopted the logic of propensity matching procedures to control for pretreatment differences between the UC and FFT-CW^® samples (Guo & Fraser, 2010). In this approach, we first identified differences in risk factors between the two conditions. Although the UC and FFT-CW^® families were originally sampled to be similar, we found a number of differences. For example, the samples differed in terms of referral source, referral reasons, pre-service indicated investigations (allegations), and pre-service out of home spells for the primary caregiver and other family members (see Table 2).

Table 2.

Current Referral Reasons (*) or Prior Risk Factors (**) by Treatment Conditions.

Current Referral Reason* or Prior Risk Factor**	Usual Care (n = 2250)	FFT-CW (n = 1625)	χ2	P	dϕ
Abuse or Neglect*	61.00%	52.40%	28.01	0.000	0.17
Child Health/Safety needs*	43.20%	42.20%	0.40	0.530	0.02
Truancy misses classes*	10.40%	8.40%	4.48	0.034	0.04
Family court contact*	5.20%	3.50%	6.22	0.013	0.03
Beyond parents control*	7.60%	12.00%	20.33	0.000	−0.09
Risk of replacement*	2.40%	0.50%	20.63	0.000	0.04
Substance Abuse**	11.40%	16.80%	22.61	0.000	−0.15
Domestic Violence**	9.70%	11.10%	1.98	0.159	−0.05
Mental Health**	8.00%	12.40%	20.77	0.000	−0.15
Allegation History**	41.60%	53.30%	51.03	0.000	−0.23
Transfer from PPRS**	4.10%	11.80%	80.52	0.000	−0.29
Caregiver Foster Care**	8.60%	13.00%	18.64	0.000	−0.14
Other Foster Care**	2.00%	4.40%	18.22	0.000	−0.14

Note: Cell entries are the percent of the samples in the two conditions that experienced a current referral reason or a prior risk factor. Effect sizes (d_ϕ) represent differences in proportions.

Propensity matching is a statistical procedure to adjust for differences between study samples when random assignment is not possible (Chaffin et al., 2012; Guo & Fraser, 2010; Rosenbaum & Rubin, 1983). One propensity matching approach is to create a one-to-one matching of individuals in the treatment samples with a comparable person in the comparison sample. We adopted a different approach because we wanted to include all UC cases, and we wanted to examine the interactive effects of various risk factors with treatment conditions. In this approach, we stratified the two samples on the multiple indicators that differentiate the two study samples. In the simplest example, the families might be stratified into two groups based upon the presence or absence of one specific risk factor such as safety concerns about domestic violence or substance abuse. In a more complex approach, families may be simultaneously stratified on a number of risk factors such as allegation history, substance abuse, and mental health safety concerns. This approach permitted an assessment of the interaction among these risk factors.

Effect size calculations

Due to the large sample sizes in this report, the criteria for statistical hypothesis testing may be misleading. Some effects may be significant by conventional statistical criteria but may not be clinically meaningful. To facilitate the evaluation of findings, we have reported three different effect size indices based upon Cohen’s criteria. The first effect size is Cohen’s d = (M₁–M₂)/SD_pooled which is useful in comparing two groups on a continuous dependent variable (Cohen, 1988; Lipsey, 1990). A second index is η² (eta squared), and it is useful for describing the differences among three or more groups on a continuous measure. The third effect size d_ϕ is useful for comparing proportions in two groups. The statistic d_ϕ = (2*(ϕ₁–ϕ₂)). The index ϕ = [arcsin(p ^1/2)] is computed for each group with a dichotomous dependent variable (Cohen, 1988; Lipsey, 1990).

2 Results

Referral Reasons and Risk Factors by Program Type

Clients in the two conditions were sampled using a strategy to create comparable groups but some differences could occur and we conducted analyses to identify potential differences. We compared the current referral reasons (marked with *) and preservice risk factors (marked with **) between the FFT-CW^® and the Usual Care conditions (see Table 2). While the rates for many of these indices were statistically significant, the effect sizes (d_ϕ) for most of the comparisons were quite small.

Preliminary analysis of risk factors in UC

We conducted a preliminary analysis within the UC sample to determine whether the various risk factors used to assign level of service for the FFT-CW^® program were associated with negative outcomes at the case closing for the usual care sample. The dichotomous negative outcome dependent variable was defined as the presence of any of the following circumstances: (1) out of home placement: (2) transfer to another service program; and (3) recommendation for a higher level of service. The analysis was a binomial regression model with substance abuse, mental health, prior allegation history, and transfer from a previous preventive service provider (PPRS) to the current provider as predictors of negative outcomes in the UC sample. The results (see Table 3) indicated that all four risk factors were statistically significantly associated with an increased likelihood of an adverse outcome.

Table 3.

Summary of Binomial Regression for Families in the Usual Care using Pre-service Risk Factors to Predict Negative Outcomes at Closing.

Covariate	B	SE	Wald	df	P	Exp(B)
Substance Abuse	0.61	0.18	12.10	1	.001	1.84
Mental Health	0.46	0.20	5.15	1	.023	1.59
Prior Allegation History	0.37	0.06	38.53	1	.000	1.45
Transfer from PPRS	0.82	0.26	10.26	1	.001	2.28
Constant	−2.52	0.12	443.00	1	.000	0.08

Note: The analysis was based upon a forward entry of covariates. The dependent variable was scored as 1.0 for the presence of any of three negative outcomes and 0.0 other wise.

2.1 Program Efficiency

Engagement rate as a clinical process measure

A specific objective of the first phase of implementing the FFT-CW^® model is to focus upon the engagement of families in therapy. These engagement techniques were used both in the LR and HR tracks. We conducted a series of analysis to assess whether the engagement and retention rates differed for FFT-CW^® and UC. We examined rates of non-engagement and found that the UC condition (16.1%) had a higher rate of non-engagement than FFT-CW^® (11.3%) [χ²(1) = 17.25, p < .001].

Length of stay (duration) in service episode

To examine duration of service, we calculated the number of days between the case open and case closing dates. We also assessed whether family ethnic or racial backgrounds was associated with program efficiency. We conducted a 2 (Service Program: FFT-CW^®, UC) × 5 (Ethnicity: African American, Asian, Hispanic, White, Other/Unknown) factorial analysis of variance with the duration of service as the dependent variable. We found that FFT-CW^® (M = 222.4, SD = 116.5) had a shorter length of stay than the UC (M = 316.0, SD = 193.8), [F(1,3766) = 91.81, p < .0001, d = .31]. The results for the Ethnic group factor are presented in Supplement S.4.

2.2 Program Effectiveness

All goals met

At the time of the proposed case closing, the clinical staff provided documentation to ACS concerning the status of the specific problems that were the basis of the initial referral. ACS staff then certified the case closure. The primary outcome variable for this section was based upon the certified closing status indicating whether all of the treatment goals had (scored as 1) or had not (scored as 0) been achieved. An important indicator of a successful treatment is that the service episode was closed after the family had met all of the goals established by ACS at the time of referral. This measure does not indicate that all family problems were resolved but that the issues leading to the current referral were successfully addressed. An evaluation of the closing status of clients indicated that the FFT-CW^® (55%) condition had a statistically significantly [χ²(1) = 30.97, p < .0001, d_ϕ = .43] higher rate of meeting all goals than the UC (35%) programs.

All Goals met by family race/ethnicity

The condition differences occurred in families with Hispanic [FFT-CW^® = 59%; UC = 35%; χ²(1) = 102.54, p < .0001; d_ϕ = 0.48] or African American primary caregivers [FFT-CW = 52%; UC = 29%; χ²(1) = 74.68, p < .0001, d_ϕ = ϕ_FFT−ϕ_UC = 0.48]. The families with other caregivers (Not Hispanic, Not African American) also were also more likely to meet all goals [χ²(1) = 4.18, p < .04, d_ϕ = 0.19] in FFT-CW^® (56.1%) than UC (46.6%) but the effect sizes were smaller (0.19 versus 0.48). The Other category included 4.0% Asian, 6.0% Non-Hispanic White, and 5.8% Other/unknown.

All goals met by referral reason

We also assessed whether the likelihood of all goals being met occurred for six of the most frequently occurring referral reasons leading to the initiation of services (see Table 4). The FFT-CW^® Treatment Condition had a higher rate of meeting all goals when referral was for abuse or neglect d_ϕ = 0.43), health or safety concerns for child (d_ϕ = 0.42), child habitually late for classes (d_ϕ = 0.68) truancy or missing classes (d_ϕ = 0.59), parent substance abuse (d_ϕ = 0.35), child beyond parental control (d_ϕ = 0.48).

Table 4.

Percent of Cases Meeting All Goals in Usual Care and FFT-CW by Referral Reason.

Referral Reason	Treatment Condition		χ2	p	dϕ
	Usual Care	FFT-CW
	Abuse neglect	35.1%				56.4%	93.25	.000	−0.43
Health and safety of child	35.4%	56.0%	66.93	.000	−0.42
Habitually late for classes	19.8%	51.8%	18.18	.000	−0.68
Truancy misses classes	21.0%	48.5%	28.92	.000	−0.59
Substance abuse	31.7%	48.7%	7.48	.006	−0.35
Beyond parents control	29.0%	52.3%	19.32	.000	−0.48

Note: Cell entries are the proportion of cases in Usual Care and FFT-CW that closed with all goals being met. The χ² was a continuity corrected estimate. The effect size estimate was d_ϕ.

All goals met by safety concerns

We examined the potential interacting effects of mental health, domestic violence, and substance abuse safety concerns on the all goals met outcome measure. The safety concerns were raised during the most recent allegation investigation prior to the service episode. We selected cases with a mental health safety concern and then identified families with or without a domestic violence or substance abuse concern within the two service conditions. Table 5 reports the percent of families classified as having all goals met by the combination of these risk factors by service condition as well as the effect size for the difference between the conditions. The results indicated that FFT-CW^® was associated with a large effect size improvement over usual care when the domestic violence concerns existed in combination with mental health concerns (d_ϕ = 0.91, 0.98). A moderate effect size was present when only the mental health concern existed (d_ϕ. = 0.39). An unexpected finding was that the substance abuse safety concern did not appear to influence the outcomes (d_ϕ =.02)

Table 5.

All goals met (%) at closing in UC or FFT-CW^® among families with a mental health and either domestic violence or substance abuse safety concerns.

Service Condition	Neither	Domestic Violence (DV)	Substance Abuse (SA)	DV + SA
UC	30% (70)	21% (24)	44% (55)	18% (22)
FFT-CW®	49% (87)	64% (25)	45% (56)	65% (31)
Effect Size (dϕ)	0.39	0.91	0.02	0.98

Note: All families have a mental health safety concern. The cell entries are the percent and sample size (n) of families with either a substance abuse or domestic violence safety concern. The effect size statistic reflects the difference in proportions between the two service conditions.

2.3 Avoiding Adverse Outcomes

The FFT-CW^® program was designed to be more efficient and to achieve goals more quickly than UC, and the findings are consistent with this premise. Moreover, the findings provide evidence that the FFT-CW program was more effective than the UC program in achieving all of the goals established for the service. However, we recognized that unintended consequences might result from a program that was designed to be more efficient. Not all clients achieved positive outcomes either in FFT-CW^® or Usual Care, and the accelerated program might have more adverse consequences including more transfers to other programs, out of home placements, more frequent return to service after the case closed, and more recurring allegations of abuse or neglect. We examined the frequency of these adverse outcomes.

Transfers to other programs at case closing

One indication of a successful service episode is that the family achieved enough progress that they did not need to be referred to another program at the closing date. We compared the percent of transfers in the FFT-CW^® and the UC cases at closing. The results indicated that FFT-CW^® (4.9%) had a significantly [χ²(1) = 30.97, p < .001, d_ϕ = 0.19] lower percentage of families transferred into another ACS sponsored Preventive Services than UC (9.8%) although the effect size was modest.

Out of home placement

One important outcome variable is the percent of families that had an out of home placement as a closing reason. We first evaluated the unadjusted prevalence of placement in the FFT-CW^® and the UC programs. The results indicated that the FFT-CW^® program (3.6%) had a statistically significantly [χ²(1) = 7.27, p < .007, d_ϕ = 0.09] higher rate of placement than the UC program (2.1%) although the effect size was quite small. One challenge in interpreting these findings was that the FFT-CW^® families had significantly higher pre-service levels of risk than the UC families on dimensions that were related to risk of out of home placement. These risk factors were initial transfer from a prior ACS sponsored Preventive Service, substance abuse and mental health safety concerns, caregiver foster care history and other family member foster care history. After statistically controlling for these factors in a binary logistic regression, the difference in placement rates between FFT-CW^® (3.2%) and UC (2.4%) was not statistically significant [B = 0.31, SE = 0.21, Wald (1) = 2.15, p < .14, Exp(B) = 1.36], and the small effect size (d_ϕ = 0.04) may not be clinically meaningful.

Return to service

More FFT-CW^® (17.3%) than UC (13.6%) families had an additional service episode following their evaluated episode, but the effect size for the difference (d_ϕ = 0.10) was small. Some of the cases were closed and then reopened while other cases were transferred directly without the case being formally close. A higher percentage of families in FFT-CW (9.3%) than UC (4.2%) were closed and then reopened for a Child Protective Service issue (d_ϕ = 0.20). A Thus, the FFT-CW^® program was more likely to close and then reopen cases while the UC program was more likely to transfer them.

Monthly service participation rates after case open date

One consequence of the shorter length of stay for the FFT-CW^® clients was that a higher percentage of them were available to return to service as compared to the UC families who remained in service. In other words, the UC families were more likely to remain in service and not be available to re-enter service. We developed an analysis strategy to assess whether a higher likelihood of a second service episode leads to a higher overall frequency of service. Using ACS records, we created an index for each family indicating whether they were in service for each of the 16 months following their initial opening date for the evaluated service episode. A comparison of the likelihood of being in service for FFT-CW^® and UC clients determined that the two conditions did not differ by more than 15% for the first 6 months of service.

However, the FFT-CW^® program had a lower likelihood of being in service at the 8^th (UC = 77%; FFT-CW^® = 55%; d_ϕ = 0.45), the 12^th (UC = 57%; FFT-CW^® = 26%; d_ϕ = 0.62) and the 16^th month (UC = 57%; FFT-CW^® = 26%; d_ϕ = 0.39) after case opening. In order to examine the effects of prior allegation history on these participation rates, families were classified as having Low (< 2), Moderate (2 through 7) or High (>7) rates of indicated allegations prior to the case However, the differences among the High history group was much smaller at all three time points (8^th month d_ϕ = 0.36; 12^th month d_ϕ = 0.22; 16-month d_ϕ = 0.03) partly because those families remained in service in both conditions (See Figure 1).

Figure 1. Family Monthly Service Participation Rates After Case Opening.

Note: Each data point represents the percent of the sample that had at least one service participation day during each month following the open date for evaluated service episode. These rates include any return to service following the initial evaluated service episode.

Recurring allegations

Another important variable potentially reflecting adverse outcomes was the recurrence of allegations after the case closing date. These events may reflect insufficient service to meet the problems faced by the family or they may also reflect new risk factors for the family. We examined effects of preservice risk factors and Service Condition (FFT-CW^®, Usual Care) on these outcomes. The ACS database provided records of each investigation start date, determination and close date for the 16,832 individuals included in the study. These records contained 26,180 occurrences before the service episode began and 6,839 occurrences after it began. Most (71.1%) of the individuals had at least one indicated allegation prior to the episode while 27.7% had an allegation after the service episode started.

The conditions differed in the percent of recurring allegations (2±) prior to the service episode (UC = 32%; FFT-CW^® 44%, d_ϕ = 0.25), and subsequent allegation analyses controlled for these different rates of recurring allegations. The first analysis selected individuals (n = 6249) who had recurring (i.e., ≥ 1) preservice allegations. To evaluate new allegations, we identified those indicated investigations that occurred in the 18 to 24-month period when nearly all families had ended service. We conducted a binomial regression with the Treatment Condition (UC = 0; FFT-CW^® =1) as the independent variable and the presence (1) or absence (0) of an allegation in the time period as the dependent variable. The results indicated that individuals in the FFT-CW^® (8%) condition had a statistically significant [B = −0.24, S.E. = 0.091, Wald(1) = 6.78, p ≤ .009, Exp(B) = 0.788] lower rate of recurrent allegations than the UC condition (10%).

Next, we examined the effects of treatments on the family level of recurrence. We created dichotomous indices for the presence or absence of caregiver allegation history, domestic violence, mental health, and substance abuse safety concerns. We selected cases with at least one prior indicated allegation and examined the association of mental health and domestic violence safety concerns to any new (i.e., recurring) allegations in the family for the period from 18–24 months (see Table 6). The biggest difference between FFT-CW^® and UC occurred in higher risk families who had both a mental health and a domestic violence concern (UC= 26%; FFT-CW^® = 13%; d_ϕ = 0.35). The difference was statistically significant [B = −1.03, S.E. = 0.53, Wald (1) = 3.75, p ≤ .05, Exp(B) = 0.36]. No significant condition differences occurred in lower risk families, without domestic violence safety concerns.

Table 6.

Recurring Allegations (%) in Families with Preservice Allegation Histories Associated with Mental Health or Domestic violence but no Substance Abuse.

Service Condition	No Domestic Violence		Domestic Violence		Combined
	No Mental Health	Mental Health	No Mental Health	Mental Health
ACS-UC	15%	10%	10%	26%	15%
FFT-CW	14%	10%	4%	13%	10%
dϕ	0.02	0.001	0.23	0.35	0.15

Note: Cell entries are the percent of families with new allegations during the period from 18–24 months following the case open date. The d_ϕ is the effect size comparing Service Conditions.

3 Discussion

The main objective of this secondary data analysis project was to evaluate the impact of an innovative approach, FFT-CW^®, on treatment outcomes for youth and families in a child welfare setting (Alexander et al., 2011; Robbins et al., 2016; Rowland & Davidson, 2011). This investigation was the first large scale, systematic study of the FFT-CW^® program. Results showed that FFT-CW^®, as implemented by NYF across five boroughs of NYC, was more efficient than UC in engaging youth and families into treatment, completing treatment more quickly and requiring fewer contacts with youth and families (Supplement S.5). This efficiency of service allowed more families in need to access crucial services.

The families presented with a complex array of referral problems, and the high acceptance rate (e.g., 89% engagement) provides support for the feasibility of providing FFT-CW^® for a diverse child welfare population. The speed with which FFT-CW^® was delivered to youth and families has important cost implications; moreover, it demonstrates the effectiveness of FFT-CW^® in addressing a range of problems that often keep families in child welfare systems indefinitely. This evaluation also showed that it was possible to triage families into appropriate levels of care, which may be a critical factor in improving both the efficiency and effectiveness of FFT-CW^®. Approximately 10% of the families were switched to a different level of service after the first few sessions based upon changes in the family’s risk status. By having a common initial intervention in the early sessions, the therapists and families could change the intensity of service with minimal disruption.

With respect to outcomes, FFT-CW^® was more effective than UC in achieving all planned treatment goals (as defined by ACS), especially when the family had multiple complex risk patterns. Furthermore, FFT-CW^® was more successful than UC with both Hispanic and African American families. However, the FFT-CW^® was not consistently more effective than UC with an “Other/Unknown” ethnic group, which was composed of families with Asian and Non-Hispanic White or Unknown primary caregivers. In addition, the substance abuse safety concern did not appear to influence the outcomes. One explanation for this finding is that many of the substance abusing clients in UC and FFT-CW^® received services for this problem in external programs so that the two conditions did not differ in the treatment they received for this risk factor.

The present findings also demonstrate the feasibility of training and supervision for child welfare staff to provide FFT-CW^®. The staff was able to demonstrate competence in all aspects of implementation, including using the Clinician Rating Form to identify family-level risk and triage families into appropriate levels of care as well as provide both the LR and HR tracks of FFT-CW^® with fidelity to the model. FFT-CW^® was also more effective than UC in avoiding adverse outcomes. The outcomes to be avoided were transfers to other programs at the end of the service episode, frequent participation in service after the case was initially closed, recurring investigations for allegations of abuse and neglect, and out of home placements to residential or foster care settings.

ACS has defined a target of 7 months for the duration of services for youth and families in NYC. When we examined the likelihood that families remained in service or returned to service after the target date, we found that FFT-CW^® families were less likely to be in service after the target closing date. When we controlled for pre-existing risk factors, the FFT-CW^® families had a lower likelihood of recurring allegations (Indicated Investigations) after the end of service episodes while maintaining a comparable out of home placements rate.

Limitations

The risk of recurring child maltreatment can be understood as the interaction between neighborhood and family characteristics (Emery et al., 2015) but the present study did not have a direct measure of neighborhood factors that might impact recurring allegations. Numerous problems exist in using child welfare administrative data sets to address research questions about the effectiveness of intervention programs (Chaffin et al., 2012; Chamberlain et al., 2016; Green et al., 2015, Lucenko et al, 2015). For example, official records may misrepresent the actual incidence of abuse and neglect. Hospital records of abuse may primarily reflect instances of serious physical harm. These records often focus primarily upon indicators that have important administrative cost implications such as length of stay. In spite of these limitations, administrative data sets have other advantages such as providing a longitudinal record of recurring events that are difficult to obtain through survey or self-report methodology (Green et al., 2015). Since the present study was a quasi-experimental rather than a randomized clinical trial. the results may not be equivalent to findings in an RCT which can provide more effective control over selection bias and historical factors. We adapted the logic of stepped wedge designs and propensity modeling to minimize the impact of selection bias and other historical events.

3.1 Summary and Conclusions

This study was the first large scale evaluation of the implementation of FFT-CW^® program in a community based child welfare setting. The findings indicated that FFT-CW^® was more efficient in completing service, and more effective than UC in meeting treatment goals while also avoiding adverse outcomes in a child welfare setting. Further research is needed to evaluate longer term results, to assess the sustainability of the treatment gains, and to compare the outcomes for other evidence based models and in other communities.