Juvenile Justice Risk Factors and Functional Family Therapy Fidelity on Felony Recidivism

Abstract

Families (n = 5,884) received Functional Family Therapy (FFT) provided as part of court-ordered probation services by 11 community sites throughout Florida. Sites provided home-based FFT to families with male (72%) or female (28%) delinquent youth. Juvenile justice courts referred clients to these services in an effort to redirect them away from incarceration. Clients were Hispanic (18%), Black (41%), and White Non-Hispanic (36%), while therapists (female, 79%) were of Hispanic (28%), Black (20%), and White Non-Hispanic (50%) ethnic/racial origins. Analyses of clients’ pretreatment recidivism risk and therapist’s caseload of risky clients demonstrated that both individual and treatment site case-mix of client criminal risk levels were associated with higher adjudicated felony recidivism. Furthermore, clinical process indicators suggest that therapists with larger rather than smaller caseloads of high-risk clients provided treatment with greater fidelity. Results suggest that experience in working with challenging clients is critical for achieving fidelity with these cases.

Adolescent delinquency is a significant public health issue with broad implications for individuals, families and communities (Alexander, Waldron, Robbins, & Neeb, 2013; Drake, Aos, & Miller, 2009; Lipsey, 2009). The costs for detaining youth are staggering. Nationwide estimates for 2005 indicate that the daily detention census in the United States was 93,000 youths, and the daily costs per offender was approximately $241 (Justice Policy Institute, 2009). The annual costs for detention exceeded $8.19 billion with $5.7 billion due to residential placement. The trends for youth involvement in the juvenile justice system, however, appear to be moving in a positive direction. For example, the National Juvenile Court Data Archive indicates that the number of referred youth has declined from 1.8 million to 1.1 million from 1997 to 2013 (Hockenberry & Puzzanchera, 2015). The Office of Juvenile Justice and Delinquency Prevention (2016) statistics indicate that the number of juveniles in residential settings has declined from a high of 105,000 in 1997, to approximately 54,000 in 2013 (Hockenberry, 2016).

Numerous factors influence treatment with youth in the juvenile justice system, including the expanded use of empirically established, community-based interventions. Functional Family Therapy (FFT: Alexander et al., 2013; Robbins, Alexander, Turner, & Holliman, 2016) is one evidence-based treatment (EBT) that has been widely disseminated into “real world” community settings in the past two decades to address adolescent delinquent behavior. Grounded in systemic and cognitive-behavioral theories, the clinical model identifies and addresses dysfunctional family relationship patterns, as well as individual and community risk and protective factors. Treatment aims to increase adaptive behaviors and decrease maladaptive behaviors while accommodating the family’s relational functions. This goal is achieved through intervention techniques with defined therapist goals and skills, using a strong cognitive component that includes systemic skill training in family communication, parenting skills, conflict management, and numerous other skills linked to a variety of syndromes and referral problems. Weekly supervision sessions assist therapists in adapting techniques to the unique problems encountered in each family (Alexander et al., 2013).

Approximately 50,000 families receive home-based FFT services each year (total ~ 500,000) from 1,600 therapists associated with 350 agencies in 11 countries (Robbins et al., 2016). Evaluations comparing FFT to usual probation services provides an estimated 8.94 benefit-to-cost ratio from the reduction of criminal recidivism (Lee, Aos, & Pennucci, 2015; Taxy, Liberman, Roman, & Downey, 2012). Despite the positive outcomes noted above, not all families benefit from FFT. The present investigation sought to examine this further by testing hypothesized interactions of client risk, site risk, fidelity, and clinical process variables on delinquency outcomes. Because FFT is effective with a very large, highly diverse set of clients and settings, it provides a useful platform to evaluate the potential interactive effects of client risk, therapist fidelity, and case experience on recidivism.

Challenges in Disseminating Evidence-Based Therapies

Numerous barriers exist when transporting EBTs, such as FFT, into community settings (Fixsen, Naoom, Blasé, Friedman, & Wallace, 2005). The process of transporting EBTs is complex and multifaceted, involving the interaction of client and treatment setting characteristics (Henggeler & Schaeffer, 2016; Kazdin, 2008; Schoenwald, Sheidow, & Chapman, 2009). These factors include the type of intervention models (e.g., individual, family, or group), the mix of clients served (e.g., variable client risk and caseload risk associated with reoffending), therapist training and experience, type of community agencies, and the service delivery setting (residential, office-based, or home-based). The results in real world implementation of EBTs are less powerful than the results observed in controlled evaluations (Schoenwald et al., 2009; Wilson & Hoge, 2013).

Implementation fidelity and treatment outcomes.

A fundamental assumption in implementation and dissemination research is that the integrity or fidelity of the intervention is a major determinant of successfully transporting efficacious treatments to community practice settings (Fixsen et al., 2005; Simpson & Flynn, 2007). Fidelity of an intervention reflects both adherence to change principles in the treatment model, as well as the competence of service delivery. Researchers have operationalized competence to encompass therapist skill and experience, appropriateness and timing of service delivery, and responsiveness to client behaviors (Hogue et al., 2008). From this perspective, higher model fidelity and therapist competence should produce better outcomes.

Prior research examining family therapy has demonstrated that better outcomes occurred with higher fidelity of service delivery (Robbins et al., 2011). Yet, some family therapy investigations found that fidelity did not directly predict outcomes, but instead the relationship appears to be curvilinear (Hogue et al., 2008) with better outcomes for intermediate fidelity. These investigators suggest that being too lax might result in weak and ineffective interventions, while rigidity (i.e., strict fidelity to the model) may lead to negative outcomes and diminished therapeutic alliance if model guidelines restrict the therapist’s flexibility to address complex problems for some clients. This rigid adherence to the model may be more problematic in community settings when therapists apply EBT guidelines with clients and in settings that differ from those used to develop the model. Regardless of the explanation, these findings indicate that fidelity to outcomes may not be a purely linear one.

Client risk and therapist fidelity.

Extensive research with adolescent and adult offenders has demonstrated the importance of client pretreatment risk as a predictor of justice system outcomes (Morgan, Kroner, Mills, Serna, & McDonald, 2013). This research indicates that it is important to match the type and intensity of treatment to the client’s criminogenic risks and needs (Vincent, Guy & Grisso, 2012). FFT research is consistent with this premise, and it indicates that clinical outcomes depend upon the interactive effects of client risk and therapist integrity in applying the intervention (Sexton & Turner, 2010). Higher fidelity in delivering FFT has a greater positive effect on outcomes for higher as compared to lower recidivism risk clients (Turner, Robbins, Rowlands, & Weaver, 2017).

Client risk, caseload risk, therapist competence, and model adherence.

Important factors to consider are the interaction between client risk, caseload risk, and therapist competence, when examining model adherence and outcomes in a community-based implementation. The caseload risk, or extent to which a therapist’s cases are comprised of riskier clients, may impact clinical adherence and the efficacy of the therapeutic alliance. One perspective is that a high caseload of difficult clients may lead to pessimism and discouragement resulting in ineffective treatment.

Another more positive view emphasizes the skill development associated with treating challenging cases. Therapists and supervisors with caseloads consisting of a larger mix of risky clients (high-risk caseload) gain more experience in treating these cases. They are more likely to develop skills in working with these families (Sexton & Turner, 2010; Taxy et al., 2012). With experience, therapists become more adept at discriminating among high-risk cases to identify those youth and families who are most at risk for failure. These cases can become a primary focus of case planning and staffing. Increased experience with difficult cases and more supervision may in turn lead to higher rates of fidelity or adherence to the model. Elevated recidivism risk creates increased opportunities for developing skills with the riskiest clients, which in turn may lead therapists to practice with greater fidelity. Hence, higher fidelity may result from rather than cause recidivism.

In contrast, therapists who work with a smaller case mix of risky clients (low-risk caseload) may be relatively successful with most of their low-risk clients, even at low to moderate levels of fidelity, but may have serious difficulties in working with their infrequent high-risk clients. These therapists may fail to recognize the complex risk factors of their clients. They might have less experience or supervisor support in planning interventions to address the needs of their clients. High-risk cases may pull therapists with less experience with these cases “off model” (Sexton & Turner, 2010), which in turn may lead to worse outcomes. Thus, although the overall performance with low-risk caseloads may be good since most clients are at low risk for recidivism, high-risk clients may be at greater risk for failure. An important implication of this reasoning, if substantiated, is that supervisors may need to pay careful attention to the treatment of high-risk clients by therapists with predominately low-risk caseloads. Since each felony can cost social institutions $50,000 with long-term adverse outcomes for victims and perpetrators, enhanced treatment of these juveniles is a major priority (Lee et al., 2015). If the current reasoning is accurate, then high-risk caseloads may be associated with greater therapist competence and adherence, but may also have high recidivism. Low-risk caseloads on the other hand may have lower fidelity and lower recidivism. If we ignore client and caseload risk variables, we might inappropriately conclude that intermediate, rather than high therapist adherence, leads to lower recidivism.

Current Study

This secondary data analysis examined hypotheses that client risk and caseload risk interact to influence clinical process indicators (e.g., therapist fidelity, length of service) and felony recidivism rates in a statewide implementation of FFT. The Florida Legislature and Department of Juvenile Justice (FDJJ) spearheaded the statewide Redirection Project to divert youth from incarceration to evidence-based, family-focused community interventions (Baglivio, Jackowski, Greenwald, & Wolff, 2014; Winokur, Hand, & Blankenship, 2012).

The research commenced with preliminary analysis designed to test underlying assumptions regarding the association between client risk, as well as caseload risk, and felony recidivism outcomes. Specifically, we sought to confirm prior research (Morgan et al., 2013) documenting the association between client risk and outcomes. We also examined the underlying assumption that the risk and needs assessment instrument used to evaluate clients’ risk to reoffend in Florida, the Positive Achievement Change Tool (PACT), significantly and positively predicts recidivism. The PACT was developed by FDJJ and was largely based upon the Washington State Juvenile Court Assessment, Back On Track! (Barnoski, 1997; Barnoski, 2004). Prior research has documented the predictive validity of the PACT not only with the overall population of youth referred to FDJJ (Baglivio, 2009; Winokur, Hand, & Blankenship, 2012), but also across subsamples based on gender, race, and ethnicity (Baglivio & Jackowski, 2011). We sought to confirm these results in the current sample and performed two preliminary analysis to assess whether: a) High-risk clients have higher felony recidivism than low-risk clients, and b) High-risk caseloads have higher felony recidivism than low-risk caseloads.

Primary Hypotheses

The primary focus of the study was to evaluate the effectiveness of FFT in reducing felony recidivism and the efficiency of case progression, while controlling for differences in client risk, caseload risk, therapist experience, and model adherence. We explored the impact of working with a high-risk caseload on fidelity and outcomes. Prior research suggests that the relationship between model adherence and outcomes may be curvilinear, whereby “intermediate adherence, representing a balance between protocol integrity and clinically flexible deviation” results in better outcomes than “high (rigid) adherence or low (lax) adherence” (Hogue et al., 2008, p.545).

We also explored whether a therapist’s competence in working with risky clients is related to case progression (completing a full treatment dose), and if so, whether this relationship is moderated by a caseload mix of mostly high-risk clients (high-risk caseload). There are conflicting findings in the literature regarding therapist competence. We sought to explore whether the efficacy of FFT in achieving successful outcomes is related to therapist fidelity and caseload mix, and whether fidelity is associated with higher rates of treatment completion. Since clients were nested within therapists, treatment settings, judicial circuit courts, and geographical regions, we also examined criminal context variables (county incarceration and violent crime rates) to assess their impact on the relationships between client risk, caseload risk, and fidelity.

Within this framework, we tested the following research hypotheses:

Hypothesis 1: Higher client risk and caseload risk are associated (1a) with lower fidelity and (1b) poorer case progress.

Hypothesis 2: (a) When therapists acquire greater experience with high-risk clients, they are more likely to complete a full dose of treatment in high-risk caseload sites, but they are less likely in low-risk caseload sites. (b) Therapists with higher risk caseloads improve their fidelity ratings more rapidly than therapist with lower risk caseloads.

Hypothesis 3: Higher levels of community crime (client risk, caseload risk, county violent crime rates, and county incarceration rates) are positively associated with therapist model adherence (3a) and recidivism rates (3b).

Method

Participants

Participants were families (n = 5,884) who were mandated for treatment by the Florida juvenile courts to FFT treatment to address an offense committed by an adolescent family member. Nearly all of the clients received home-based services, and all were treated by a trained and certified FFT therapist (n = 169) who was associated with one of 11 different community service providers across Florida. The FFT LLC organization provided the training and supervision of therapists in all of these settings.

Gender and minority inclusion.

Adolescents (male = 72%; female = 28%) and their families participated in this research. They ranged in age from 10.5 to 19.2 years (M = 16.18, SD = 1.38). The diverse sample included Non-Hispanic African American or Black (41.1%), Non-Hispanic Whites (35.5%), Hispanic Whites (17.8%), or Other racial/ethnic backgrounds (5.8%).

Therapists.

Most of the male (23%) and female (77%) therapists were master’s level social workers, mental health counselors, or marriage and family therapists. They were affiliated with one of eleven community service providers that managed referrals, therapist training and supervision, and service delivery. Therapists’ race/ethnicity included African American or Black (20%), Haitian (2%), Hispanic (28%), Other/Unknown (4%), and Non-Hispanic White (50%). The therapists acquired extensive case experience with FFT (M = 47, SD = 42, range 3 – 249) during the study period. Follow-up outcome data were available for clients of 135 therapists.

Design

This secondary data analysis can be viewed as a 2 (Caseload Risk: Low or High) x 4 (Client Risk: Low, Moderate, Moderate-High, and High) factorial design with client gender as a covariate. Clients were referred to one of the 11 community service providers between 2004 and 2013. The implementation at the various sites was staggered over time, permitting an assessment of the effects of case experience that was not fully confounded with calendar time. We categorized clients based on their assessed criminogenic risks and needs, as determined using the standardized and validated FDJJ PACT assessment, which classifies youth into four levels of pretreatment recidivism risk: low, moderate, moderate-high, and high (Winokur Early et al., 2012). We created a dichotomous crime hazard classification (low/high-risk) for each of the therapist’s caseloads in the 11 service sites based on the mean scores of all clients on the criminal history subscale of the PACT, adapting the logic of hierarchical linear modeling to address effects of nesting of clients and therapists within settings. Prior studies indicated that male clients had higher recidivism rates than female clients (Baglivio et al., 2014) and we incorporated gender as a covariate.

Data Sources

Data for the current study were made available through the collaboration/cooperation of the FDJJ, the Justice Research Center (JRC), and FFT LLC. Official delinquency indicators were obtained from the FDJJ Juvenile Justice Information System (JJIS), the computerized administrative system that contains records of all juvenile referrals, charges, dates of alleged infractions, detentions, adjudications, dispositions, placement and service history, and risk assessments data. This system conforms to national standards for reporting juvenile justice referrals and dispositions. The JRC used their authorized access to this system to extract all juvenile referral and placement records from both before and after a client’s participation in the program. Unique identifying information was removed from the record and replaced by a coded index so that the JRC could track and link client records over time. The JRC matched JJIS records to official juvenile adult criminal arrest and placement data maintained by the Florida Department of Law Enforcement and the Florida Department of Corrections. These time tagged data enabled us to establish the sequential order of clinical and juvenile justice events, as well as permitted us to evaluate subsequent juvenile and adult recidivism for all study youth.

The PACT risk assessment data for study youth was extracted from the JJIS information system. The PACT is administered to all juveniles referred to FDJJ for a criminal offense or violation of probation. The PACT produces a Criminal History Score and Social History Score upon which risk level classifications are then calculated (low, moderate, moderate-high, and high), and prior research demonstrates that these categories predict the likelihood of juvenile offender recidivism (Baglivio & Jackowski, 2013; Winokur Early et al, 2012). As elaborated upon in more detail below, the PACT was used to operationalize client risk and caseload risk.

The FFT LLC provided access to indicators of FFT clinical processes and therapist fidelity. These measures reflected clinical progress in treatment, as well as therapist adherence to the FFT model and therapist competence. Primary outcomes included measures of felony recidivism involving all youth from the case open date to one year after their release from FFT services. Operational definitions for all study measures are outlined below. The findings from the study have the potential to improve the tailoring of interventions to juvenile offenders and to provide better criteria for tracking treatment fidelity.

Study Variables and Procedures

We extracted variables from multiple administrative data sets accessed through the FDJJ or FFT LLC. The FDJJ authorized the JRC staff to see client names and criminal records and to assign a randomly generated, case identifier to each data set to link clients across data sets. The researchers received only de-identified data with a random case identifier. The National Institute on Drug Abuse (NIDA) project officer, the FDJJ, and IRBs approved study procedures.

Client risk.

All youth referred to FDJJ for a delinquency petition receive a PACT prescreen risk assessment. Probation officers use the PACT in a structured interview with the youths and their family to assess for multiple risk and protective factors. The PACT has extensive psychometric information that it is both a reliable and valid tool to predict the likelihood of further involvement in the juvenile justice system (Baglivio & Jackowski, 2013; Winokur, Hand, & Blankenship, 2012). A panel of nationally recognized criminal justice experts developed the original version of the assessment tool (Barnoski, 2004).

The FDJJ has used the PACT to assess recidivism risk for more than 100,000 adolescents in the state of Florida. The Criminal History scale includes 10-items that assess pretreatment criminal behavior. The total score on this scale primarily reflects individual differences in the types of prior offenses (e.g., felony, misdemeanor, weapons, crimes against person), and judicial actions such as confinement orders, or arrest warrants for failure to appear (alpha = .71). The Social History scale assesses a range of social problems including school behaviors, use of free time, peer relationships, gang involvement, family, mental health problems, prior neglect or abuse, running away from home, and deviant and prosocial attitudes and behaviors (alpha = .57).

Site caseload risk.

We used the PACT assessment to classify pretreatment risk factors that may be associated with clinical processes and outcomes. We created an index of the site caseload mix of risky clients using the mean Criminal History Score on the PACT for all clients on the caseload. We dichotomized caseload risk into higher and lower case mixes of risk depending upon whether they were above or below the mean on this index.

Community risk.

We examined the influence of community-level criminal justice risk using annual Index Crime and incarceration rates for each county in Florida and we assigned the appropriate regional score to each family. The Florida Department of Law Enforcement (FDLE) website provided annual Index Crime rates/100,000 residents for each county in Florida. The Florida Department of Corrections website provided estimates of county incarceration rates. We assigned crime and incarceration rates to each client based upon their county of residence. Information concerning these measures are available in Table P1 in the Online Supplement.

Therapist supervision procedures and fidelity assessment.

FFT therapists received standard FFT training and supervision procedures (Alexander et al., 2013). The training included ten days (five, 2-day trainings) of didactic, on-site workshops. Additional training and supervision occurred in one-hour weekly group consultation calls led by expert FFT-LLC consultants over 1 year, and ongoing support to an on-site clinical supervisor via telephone calls after the first phase of training. In addition to providing feedback on cases, the consultants (initial phase) and supervisors (after initial phase) rated the therapist’s compliance with administrative procedures (e.g., completing case notes), and the therapist’s fidelity to FFT principles. The supervisor rating scale ranged from 0 to 6, and we dichotomized these ratings into scores above or below the minimally accepted threshold (> 3.0).

The supervisor provided these weekly ratings as additional feedback to the therapists, and they recorded these values in the electronic record for each supervision session. We created a sequential order for these weekly ratings starting with the date of the therapist’s first supervision. Using these scores, we calculated a mean fidelity rating for each month that the therapist received supervision. We assigned the mean score to each client who enrolled during the month of the rating. This procedure permitted us to explore the nesting of therapist’s clients. We estimated the internal consistency of supervisor’s ratings (ICC = .84, α = .94) for each therapist during months 1–3 for each client to reflect the temporal stability of the therapist’s ratings (Turner et al., 2017). Any fidelity ratings analyses only used data after 2008 when the current procedure was adopted.

Dependent Variables

We identified adjudicated recidivism as occurring either during the service episode or during the 1 year period after the service period ended. Recidivism was defined as a subsequent adjudication or adjudication withheld for a law violation, and we classified the offense type as either a felony or any other offense type (misdemeanor, violation of probation due to a new law violation), exclusive of technical violations of probation. The JRC has conducted previous studies (Winokur, Blankenship, & Hand, 2009; Winokur et al., 2012) demonstrating that they can link juvenile records in the FDJJ JJIS system to adult arrest and corrections data maintained by the Florida Department of Law Enforcement and Florida Department of Corrections information with 98% matching accuracy. These systems provided a report of date and type of subsequent adjudications, convictions and any participation in mandated treatment or confinement. The FDJJ Common Definitions Report (2011) classifies offenses in terms of severity.

Statistical Considerations

Exploring nesting effects.

We recognized the possibility of data dependency resulting from the nesting of families within therapists who were nested within sites or community service providers (see Spybrook et al., 2011). Therapists participated in only one of these organizations. We examined these nesting effects and the results are presented in the Online Supplement.

Analysis approach.

We examined three types of measures: felony recidivism outcomes, clinical indicators reflecting progress in therapy, and therapist fidelity. Due to the large sample sizes, very small effect sizes could be statistically significant without being clinically meaningful. Thus, we primarily focused upon effect size estimates as a basis for making inferences about the effects of therapist level and client level risk factors. We used binary logistic regression procedures for dichotomous dependent variables and factorial analysis of variance for continuous variables.

Stratified matching procedure.

We adapted the logic of stratified propensity score matching (Guo & Fraser, 2010; Rosenbaum & Rubin, 1983; Turner et al., 2017) to control for the effects of client risk and caseload risk. One matching approach uses propensity scores to create one-to-one matches between cases in two different study samples (e.g., Baglivio et al., 2014). Our matching procedure equates subsets of cases rather than individuals on key variables. We matched based on client risk, caseload risk, therapist case experience, and client gender.

Effect size calculations.

To facilitate the evaluation of findings, we have reported three different effect size indices based upon Cohen’s (1988) criteria. The first effect size is Cohen’s d = (M₁ –M ₂)/SD_pooled which is useful in comparing two groups on a continuous dependent variable (Cohen, 1988; Lipsey, 1990). A second index is η² (eta squared), and it is useful for describing the differences among three or more groups on a continuous measure. The third effect size d_ϕ is useful for comparing proportions in two groups. The statistic d_ϕ = (2*(ϕ₁– ϕ₂)). The index ϕ = [arcsin(p ^1/2)] is computed for dichotomous dependent variables (Lipsey, 1990).

Results

Preliminary Analyses

The preliminary analysis assessed the effects of individual client risk and site caseload mix of risky clients on adjudicated felony recidivism occurring between the case open date and the 1 year period following the release from services (median time = 15 months). We used a 2 (caseload risk) x 4 (client risk) factorial design with client gender as a covariate; the observed recidivism rates for these combinations are presented in Figure 1. Numerical values and gender details of the analysis are presented in Table S1 of the online supplement. The results indicated that both client risk and caseload risk were associated with higher recidivism. As expected, the findings are consistent with prior research linking risk factors to recidivism.

Figure 1.

Percent of youths with an adjudicated felony recidivism charge between case open date and one year following the end of probation supervision (total time ~ 16 months) by client pretreatment risk category on the PACT and site caseload risk. Cell entries are the percent of juveniles within the high and low-risk caseload sites by the four-category PACT risk and the site caseload risk combinations. Numeric values are presented in the web based online supplement Table S1.

Primary Hypotheses

Effects of client risk and caseload risk on clinical process measures.

We reasoned that a therapist’s fidelity and case progress with high-risk clients might depend upon the site’s caseload mix. We recognized that case experience with risky clients could lead to either deteriorating or to enhanced clinical performance and our analyses evaluated these two possibilities. Therapists with a small (low-risk) caseload of very risky clients have little experience in treating these clients and few opportunities for supervision, potentially resulting in low confidence and discouragement. An alternative outcome may occur for supervised therapists with a larger (high-risk) caseload of risky clients. These therapists have more opportunities to gain expertise from treating these clients especially when receiving group supervision and sharing experiences with other therapists.

We examined the hypothesis about therapists either with low or with high-risk caseloads who are treating clients with higher recidivism risk. We assessed whether increased client risk was associated with lower therapist fidelity to the FFT model and poorer progress in treatment, resulting in the restriction of the treatment dose for clients. To address potential nesting effects, we included the therapist factor as a random effect and site caseload risk and client risk variables as fixed effect factors in a mixed effects ANOVA with the therapist fidelity score as the dependent variable. The results revealed significant main effects for caseload risk [F(1, 4157) = 9.54, p < .002], client risk [F(3, 4157) = 6.19, p < .001] and the caseload risk by client risk [F(3, 4157) = 3.04, p < .028] interaction.

We conducted pairwise comparisons among the main effect means using a Bonferroni adjusted confidence level [α/6; p < (.05/6) = .008]. The results indicated that the therapists treating the lowest risk clients received significantly higher fidelity scores (M = 3.64 S.E. = 0.04) than the other three client risk groups (moderate, moderate-high, and high). The comparison of main effect means for the caseload risk variable indicated that therapists working at sites with a high-risk caseload had higher fidelity scores (M = 3.56, S.E. = 0.03) than those working at sites with a low-risk caseload (M = 3.43, S.E. = 0.04). See Figure 2 (or Table S2 in the supplement).

Figure 2.

The effects of client pretreatment risk and site caseload risk on the mean monthly supervisor fidelity ratings of therapists treating clients in each category. Weekly fidelity ratings were pooled across four-week periods to produce more stable estimates. The monthly score was assigned to all clients entering treatment during the month of the ratings. The mean rating reflects all clients and range from 0–6 with a 3.0 reflecting minimally acceptable performance. Numeric values are presented in the web based online supplement Table S2.

To evaluate the interaction term, we conducted simple main effect comparisons within each of the caseload risk conditions (low and high). First, we compared the four client risk groups within the high-risk caseload sites and the results indicated that the four client risk groups were not significantly different from each other. Second, we compared the four client risk groups among the low-risk caseload sites. See Figure 2 (or Supplement Table S2). These comparisons indicated that the therapists received significantly higher fidelity scores when they were working with their least risky clients than when they worked with their higher risk clients (d_ϕ, = 0.51). The effect size (d_ϕ, = 0.35) among the high-risk caseload sites was of a medium magnitude.

Effects of risk factors on progress during treatment.

The first hypothesis also concerned the therapist progress in achieving interim treatment goals. To assess this element of the hypothesis, we examined the likelihood that clients achieved at least eight sessions, which corresponds to the end of the behavior change phase of FFT, an important interim treatment goal (Alexander et al., 2014). This goal reflects whether therapists are able to engage and retain clients sufficiently to achieve a minimal dose. Hypothesis 1b states that increased client risk and caseload risk are associated with a reduced likelihood of achieving a minimal treatment dose.

We conducted a binary logistic regression analysis using caseload risk, client risk, and gender as independent variables and the dichotomous minimal dose measure as the dependent variable.¹ Within the low-risk caseload sites, the differences in achieving a minimum dose between their lowest risk (85%) and their highest risk (63%) clients resulted in a relative large effect size (d_ϕ = 0.51; see Figure 3). Within high-risk caseload sites, the difference in the rate of achieving a minimal dose for the lowest individual risk clients (86%) versus the highest individual risk clients (72%) reflected a moderate effect size (d_ϕ = 0.35, See Figure 3). We also computed an effect size (d_ϕ = 0.19), which was in the small range, for the difference in rates among the highest risk clients between the low and high-risk caseload sites. In summary, these results indicated that therapists in low-risk versus the high-risk caseload sites had more difficulty with clinical processes including achieving a minimal dose treatment with their high-risk clients.

Figure 3.

The effects of client pretreatment risk and site caseload risk on the percent of clients receiving a minimal dose of therapy (8 sessions). Cell entries are the percent of clients who achieved at least 8 sessions. This dose reflects a quality assurance goal representing successful engagement, retention, and progress through the behavior change interventions. This dose usually does not include generalization training, which clinicians schedule to occur during sessions 9–12. The web based online supplement presents numerical values in Table S3.

Effects of risk factors and therapist experience on case completion.

The second hypothesis stated that therapists with high risk caseloads and more case experience working with risky clients are more likely to complete a full dose of treatment (12 or more sessions) than are therapists with a low-risk caseload and less experience with high risk clients.⁴ For this analysis, we determined how many cases each therapist had treated prior to the current case. We categorized the prior case experience into three groups (0–20, 21–40, and 41–60). We also pooled the low and moderate risk groups together (LR Client) and the moderate-high and high-risk youth (HR Client) together to create larger samples sizes. The therapists required approximately 6, 12, and 18 months to acquire these sets of cases.

We conducted a binary logistic regression analysis using caseload risk, the three-group case experience measure, client binary risk (low/mod, mod-high/high), and gender as independent variables. The dichotomous dependent variable indicated whether the client received a full dose (0 = No, 1 = Yes). The results indicated that the Caseload Risk x (p < .007, Exp(B) = 1.40), Client Risk (p < .001, Exp(B) = 0.76), and Therapist Case Experience (p < .001; Exp(B) = 1.15) were statistically significant (see Figure 4 and Supplemental Table 4). The gender effect was not significant. To test hypothesis 2, we compared the full dose index within the high-risk caseload sites, and the results indicated that the early cases (0–20: 47%) did not differ from the middle cases (21–40: 47%, d_ϕ = −0.058), but did differ from the late cases (41–60: 65%, d_ϕ = 0.38). To test the second hypothesis further, we compared the full dose index within the low-risk caseload sites, and the results indicated that the early cases (0–20: 49%) did not differ from middle cases (21–40: 46%, d_ϕ = −0.07), or from late cases (41–60: 52%, d_ϕ = 0.06).

Figure 4.

Percent of clients who received a full scheduled dose of treatment (12 sessions) by therapist prior case experience and by sites with either a lower risk caseload [LR Site (gray, above)] or a higher risk caseload [HR Site (black, above)] and the client’s pretreatment risk. Clients in the low and moderate risk groups were pooled [LR Client (dashed lines, above)] as were the clients in the moderate high and high-risk groups [HR Client (solid lines, above)] to increase the sample size for each observation point. The effect size for the comparison of the LR Sites (33%) and HR Sites (55%) among the higher risk clients at the 41–60 month assessment was d_ϕ = −0.44. Tabled values are presented in the web based online supplement Table S4.

We also examined Caseload Risk x Binary Client Risk interaction among late cases (41–60). The results (see Figure 4) indicated that the Low Risk Caseload-High Risk Client group (35%) had significantly lower completion rates than the Low-risk Caseload-Low Client Risk group (50%, d_ϕ = 0.31), or the High Risk Caseload-Low-risk Client group (60%, d_ϕ = 0.50), or the High Risk Caseload-High Risk Client group (55%, d_ϕ = 0.41). In summary, the findings are consistent with hypothesis 2a but not 2b. Thus, the findings for High-Risk Client indicated that case experience in High Risk Caseloads are linked to better clinical outcomes but worse outcomes in Low Risk Caseload sites.

Exploratory analysis of the relationship of case experience and fidelity.

We examined whether therapists would improve in model fidelity with experience, especially when accompanied by expert supervision, if their case mix included a higher proportion of challenging cases. We identified therapists’ fidelity ratings and their case experience within sites that had a larger or smaller caseload of clients with risk related criminal histories (recall Figure 1). We assessed whether early case experience was associated with changes in their supervisor’s fidelity ratings of the therapist. We combined pairs of cases (e.g., 1–2, 3–4) to produce more stable estimates. All clients began treatment after the FFT LLC adopted the current supervision and ratings procedures in 2008 (Robbins et al., 2016).

We determined the percent of clients whose therapists had fidelity ratings above the clinical threshold during the second month of their treatment. This time usually corresponds to the behavior change stage of treatment. We defined acceptable fidelity ratings as a supervisor score of 3 or higher. To make these comparisons, we computed an effect size (d_ϕ) based upon Cohen’s (1988) procedure for comparing proportions. The analyses for early cases indicated that therapists with larger caseloads of challenging cases had fewer acceptable ratings (medium-large effect sizes) than therapists with less challenging cases (1–2: d_ϕ = 0.23; 3–4: d_ϕ = 0.52; 5–6: d_ϕ = 0.27). However, an examination of later cases (i.e., 9–14) produced the opposite pattern. Therapists with a larger caseload mix of risky clients improved more than therapists with a smaller risky case mix or low-risk caseload so that, over time, they had more ratings (medium-large effect sizes) above the clinical threshold (9–10: d_ϕ = −0.33; 11–12: d_ϕ = −0.52; d_ϕ = −0.37).

After the 15^th case, therapists working with low-risk caseloads improved so that a small effect size difference occurred between the case mix sites, being negligible by the 24^th case. We extended these comparisons through the first 80 cases of each therapist, which required approximately 2 years. We pooled the scores into groups of 10 clients (e.g., 1–10). The proportion above acceptable fidelity thresholds improved with case experience (1–10 = 24%; 11–20 = 33%; 21–30 = 42%; 31–40 = 53%; 41–50 = 60%; 51–60 = 67%; 61–70 = 82%; 71–80 = 82%). These findings suggest that most therapists improve over a 2 year period.

We conducted an exploratory analyses to address a possible confound concerning the effects of therapist case experience. We recognized the possibility that the changes within a site caseload risk level might result from changes in the way that clients were assigned to therapists as they gained more experience. Specifically, we anticipated that the sites might withhold their more challenging cases from their newest therapists and assign these cases to their more experienced therapists. Thus, experienced therapists might be assigned to more difficult cases.

To examine this possibility, we conducted an exploratory analysis to compare the case mix of lower risk (low, moderate) and higher risk (moderate-high, high) clients by the therapist’s case experience in blocks of 10 clients. We summarized results in Figure 6 (and Supplement Table S6), and the findings indicated that the high-risk caseload sites have a greater percentage of the high risk clients through each block of cases for their first 50 cases. The effect sizes for the difference in caseload risk ranged from 0.31 to 0.50. Thus, case mix difference between low-risk and high-risk caseloads persists throughout the therapist’s case experience in their first year.

Figure 6.

Percent of clients with pretreatment moderate-high or high PACT risk categories by site caseload (low-risk or high-risk) and the sequential order of the therapist’s cases. We pooled the therapist’s cases into groups of 10 to produce more stable estimates over time. We also pooled the moderate-high and high PACT risk groups to stabilize estimates. The cell entries reflect the percentage of higher risk clients (Moderate High + High) during each 10 case grouping. Numeric values are presented in the web based online supplement Table S6.

In summary, sites with a low-risk caseload had lower fidelity scores, especially when treating their highest risk clients. High-risk caseload sites had higher fidelity scores and these ratings were not significantly different by client risk level. These findings are consistent with the premise that sites containing more of the challenging high-risk cases were associated with greater model compliance as reflected in supervisor ratings. Any differences due to client risk and therapist experience were not due to shifting assignments of risky clients to therapists.

Criminal history context and therapist fidelity.

We examined the hypothesis that therapists working in higher crime environments would be more faithful to FFT, which would be reflected in higher supervisor fidelity ratings. We selected four measures to reflect the criminal context for clients. The first was the client’s own Criminal History Score from the PACT, the second was the therapist’s case mix of risky clients, the third was a measure of the violent crime rate for the county in which the clients received services, and the fourth was the incarceration rate per capita in the client’s county (see Supplement Table P1). We extracted these last two variables from official Florida criminal records and imputed the county scores to each client in that county. We obtained the county specific rates for 2010 (the midpoint of the study period).

We regressed these four criminal context scores on fidelity scores after 2008–09 fiscal year, when a new ratings approach was adopted. The overall model was statistically significant [adjusted R² = .11, F(4, 2943) = 23.02, p < .001]. Caseload Risk variable [β = 0.14, t(2943) = 7.01, p <.001], County Incarceration Rate [β = 0.14, t(2943) = 7.01, p < .001], and County Violent Crime Rate [β = 0.14, t(2943) = 7.01, p <.001] were associated with higher fidelity scores. The individual Client Risk score based upon the PACT Criminal History score was not significantly associated with therapist fidelity [β = −0.02, t(2943) = 1.05, p < .294]. In summary, these findings are consistent with the premise that a higher criminal context is associated with greater adherence to the clinical intervention.

Effects of risk factors and clinical process measures on felony recidivism.

The last hypotheses extend the previous ones to examine the joint predictive effects of four criminal context measures, client gender, and two clinical process measures on the dichotomous Total Felony Recidivism Index. This recidivism measure indicated whether any adjudicated felony offenses that occurred between the case open date until the end of court supervision (0 = None; 1 = Any). We estimated separate binary logistic regression models within either the low or high-risk caseload sites. Three other criminal context variables included the Client Risk (PACT) categories, the client’s county incarceration rate, and the county violent crime rate. The clinical process measures were therapists’ case experience, and supervisor’s fidelity rating.

The results indicated that recidivism was elevated for clients with higher PACT Criminal History scores both in the Low (p < .001), and High-Risk Caseload Sites (p < .005). Males had higher recidivism rates for both the Low (p < .001) and the High Risk-Caseload (p < .001) sites. Higher therapist fidelity ratings (p < .028) and more case experience [p < .030] were associated with lower recidivism in the Low-Risk Caseload Sites.² These variables did not predict recidivism for the High-Risk Caseload Sites (p’s > .55). Neither the county violent crime rate nor the incarceration rate was statistically significant for Site Caseload Risk (p’s > .10).

A summary of findings for low-risk caseload sites indicates that higher client risk and lower fidelity were associated with worse recidivism. Low fidelity was also associated with higher client risk and poorer progress in treatment. The fidelity gap between lower and higher risk clients widens as therapists gain case experience. High-risk clients have less chance for favorable clinical and recidivism outcomes as clinicians gain experience in low caseload sites.

Discussion

This secondary data analysis study examined hypotheses concerning the effects of individual client and community risk as well as therapist’s caseload risk and experience on clinical processes, therapist competence, and treatment outcomes of a large scale, statewide implementation of FFT. The present findings demonstrate the importance of having a complex, multi-faceted framework for understanding how quality of treatment interacts with therapist and youth/family factors to facilitate successful outcomes. For example, the results of the preliminary analyses indicated that both therapist caseload mix of risky clients and individual client risk levels were associated with higher recidivism (see Figure 1). These effects were also associated with differences in therapist model fidelity (Hypotheses 1a). Therapists with a smaller case mix of risky clients (low-risk caseload) had lower fidelity ratings than those working with a larger number of risky clients on their caseload (Figure 2). Therapists with a low-risk caseload had worse fidelity ratings for their highest versus their lowest risk clients. Fidelity was not differentially associated with client risk when therapists had a larger caseload mix of risky clients. A key pair of findings was that therapists treating higher risk clients had both better fidelity and more recidivism. These findings are inconsistent with the premise that higher fidelity necessarily leads to better outcomes (Fixsen et al., 2005; Simpson & Flynn, 2007).

The present findings address two alternative hypotheses concerning the relationship of fidelity and outcomes. A first explanation is that a curvilinear relationship exists between fidelity and outcomes. That is, very high or low fidelity are associated with worse outcomes. A possible explanation for curvilinear effects is that poor fidelity leads to ineffective interventions while very high fidelity may lead to overly rigid implementation of models. Both processes may contribute to higher recidivism (Hogue et al., 2008). This explanation posits that intermediate fidelity may be optimal because therapists are flexible in applying their model.

A second explanation is that high fidelity is the result and not the cause of high recidivism risk. The findings suggest that therapist’s caseload mix, case experience, and youth/family risk may interact with treatment quality to impact recidivism. For example, there were significant differences in patterns of progress in treatment by caseload mix. When therapists had a smaller case mix of high-risk clients, they were less likely to successfully engage and retain clients (especially their highest risk clients). They did not make sufficient progress to achieve a minimal dose (Figure 3) or a full dose of treatment with their higher risk clients.

In contrast, therapists with a larger caseload mix of risky clients had better fidelity and treatment progress with their highest risk clients than therapists with fewer risky clients (Figure 2 and 3). Moreover, as shown in Figure 4, therapists with a high-risk caseload also got better at completing treatment with their high-risk clients as they gained more experience, as evidenced by the percentage of families who received 12 or more sessions. Furthermore, supervisors provided more favorable fidelity ratings to therapists with a larger caseload mix of risky clients (Figure 5). We recognized that therapists case experience and caseload mix might be confounded so that therapists with greater experience may receive cases that are more challenging. As presented in Figure 6 therapist caseload mix did not change with experience.

Figure 5.

Percent of therapists’ cases that were above acceptable fidelity threshold by case experience at the sites with a lower risk caseload or higher risk caseload. Cell entries are the percent of cases above the minimally acceptable threshold (3 or greater) for successive pairs of clients in the therapist’s case experience from 1 to 24. Therapists require approximately six months to enroll these clients. The effect sizes (d_ϕ) for the comparisons of acceptable threshold values in lower risk and higher risk caseload sites at sequence 3–4 = −0.52 but at sequence 9–10 = 0.34, at sequence 11–12 = 0.53; and at sequence 13–14 = 0.37. Numeric values are presented in the web based online supplement Table S5.

Does higher risk promote higher fidelity?

The present findings provide evidence that in certain contexts higher fidelity is associated with worse juvenile justice outcomes. We hypothesized that high-risk families lead therapists to adhere more closely to model guidelines in an attempt to mitigate the risk factors associated with their clients. From this perspective, high fidelity is a result rather than a cause of high recidivism. Therapist’s experience with a larger mix of risky clients and neighborhoods may heighten therapists’ awareness to issues across all levels of risk, including being able to identify those cases that are at highest risk of having a negative outcome. This sensitivity may be particularly likely for home-based treatments where therapists see first-hand the family and neighborhood risks that are often associated with negative client outcomes. Experience and ongoing assessment in these settings may increase the therapists, supervisors, and team’s ability to identify the highest risk cases and to devote time and resources to plan and deliver interventions that match the level of risk. Thus, the high-perceived risk for recidivism may increase model fidelity to mitigate the negative effect of these risk factors. High fidelity may result from high risk of recidivism rather than the reverse.

As noted, the current findings provide support for the perspective that higher community and client risk levels are associated with higher therapist fidelity of implementation. Regression analyses addressing Hypothesis 3 indicated that higher community risk of incarceration, county violent crime rates, and caseload risk were associated with higher fidelity. These results are consistent with extensive research (Risk, Need, Reactivity or R-N-R) demonstrating the importance of client risk as a predictor of criminal justice outcomes (Morgan et al., 2013). This research also indicates that it is important to match the type or intensity of treatment to the client’s risk of recidivism as well as their potential responsivity to treatment.

Clinical process indices for the sites with a larger case mix of risky clients (high-risk caseload) indicated that they were more likely to retain clients in treatment and to provide a full dose of treatment for their highest risk clients. These clinical process indicators were consistent with supervisor ratings of the therapist’s fidelity. The enhanced model fidelity did not completely offset the negative influence emanating from community and client risk factors

Limitations of the Present Study

The present analysis was restricted to a single clinical model in the evaluation of the interactive effects of client recidivism risk, therapist caseload risk, and case experience on felony recidivism. Within the FFT model, higher risk clients had better fidelity and worse criminal outcomes. This association may suggest that high fidelity contributes to poor outcomes. However, previous research concerning FFT indicates that the largest effect size differences between FFT and usual care occurred with high risk clients who received high fidelity treatment (Sexton & Turner, 2011; Turner et al., 2017). In other words, the negative effects of risk and fidelity was stronger in usual care models as compared to FFT. Without random assignment, it is not possible to rule out the possibility that some unmeasured confounding variable was responsible for the outcomes assessed. This analysis relied heavily on administrative data sets. While we used a variety of quality control procedures, we could not match approximately 15% of the clinical process measures to the juvenile justice records.

Clinical Implications

The present findings have important implications for clinical supervision. Therapists with smaller caseloads of risky clients may need additional training and supervision to work effectively with higher risk clients. Although they have fewer of these clients in their practice, they appear to have greater difficulty in engaging them in a full course of treatment. Therapists with a larger caseload of risky clients have more urgency to consult their supervisor and greater opportunities to implement lessons learned with these clients.

These findings are immediately relevant to family-based approaches that are being widely implemented in community settings. The reality of real world practice is that therapists have limited time to devote to supervision and planning. The current results provide guidance to supervisors about how to be strategic in selecting which cases to discuss in supervision. For FFT LLC, an organization that oversees the training and implementation of FFT to more than 1,600 therapists in 350 settings annually (Robbins et al., 2016), the current results suggest how outcomes can be improved by using client and community risk to drive consultation. While the present findings reflect relatively small effect sizes, the costs associated with each felony recidivism failure are substantial ($40–50K, Taxy et al., 2012; Winokur et al., 2012). The cumulative savings associated with each avoided offense provides a potential pool of resources to improve clinical services. It is possible that the consultation process can be fine-tuned to include ongoing feedback about client risk and caseload risk to accelerate therapist development and improve fidelity for the highest risk cases.

Authors’ Bio Sketch

Charles W. Turner, Ph.D., received his doctorate in Social Psychology from the University of Wisconsin, Madison, he was a faculty member at the University of Utah Psychology Department for 35 years, and he is now a Senior Scientist at the Oregon Research Institute. He has participated in 20 federally funded projects as a lead methodologist, and his research has focused primarily upon the evaluation of prevention and treatment outcomes for children and adolescents with behavior problems. He has been involved for 40 years in process and outcome research on Functional Family Therapy.

Michael S. Robbins, Ph.D., received his degree in clinical psychology from the University of Utah and he is now a Senior Scientist at the Oregon Research Institute as well as being the Clinical and Research Director for FFT LLC. Dr. Robbins has extensive experience conducting clinical research on family therapy with drug using, delinquent adolescents and their families, and he has directly overseen the training of hundreds of family therapists, both nationally and internationally. Dr. Robbins is recognized as a leader in the areas of process and outcome research in adolescent drug abuse treatment, and he has led numerous federally-funded research studies that involve the examination of in-session processes as well as clinical outcomes.

Kristin Winokur Early, Ph.D., received her doctorate from the College of Criminology and Criminal Justice at Florida State University, and she is now a professor of criminology in the graduate program at Purdue University Global and the research director for the Justice Research Center in Tallahassee, Florida. Her research focuses on evaluating the effectiveness of juvenile justice interventions on youth and family outcomes. She was appointed by Governor Rick Scott to serve on Florida’s Juvenile Justice and Delinquency Prevention (JJDP) State Advisory Group (SAG), serving on the Executive Committee and as the Chair of the Grants and Contracts Committee.

Julia L Blankenship, MSW, received her master in social work from Florida State University and she is now the President of the Justice Research Center in Tallahassee, Florida. She is an expert in the analyses of large administrative data sets and her current research focuses on assessing outcomes of at-risk youth and youth involved with the juvenile justice system. She has been the Principal Investigator and co-investigator on numerous studies, resulting in reports and articles on program, process, and outcome research.

Lisa R. Weaver, BA, received her Bachelor’s degree from Rutgers University in Political Science and she is currently a data analyst at the Oregon Research Institute (ORI). She has participated as a research analyst and data manager on large, complex data sets associated with several NIH grants at ORI that examine treatments for families and youth with serious problem behaviors. She also has extensive experience in working with children and youth with special needs.