Enhancing “usual practice” Treatment Foster Care: Findings from a randomized trial on improving youth outcomes

Abstract

Objectives

This article reports the initial findings from a randomized trial to enhance Treatment Foster Care (TFC) in “usual care” agencies. The intervention, Together Facing the Challenge, was built upon a combination of practice-based elements from a prior descriptive study of TFC and selected elements from Chamberlain’s evidence-based model (MTFC) to fill conspicuous gaps in usual practice. The study was designed to examine whether additional training and consultation to staff and treatment parents improved outcomes for youth.

Methods

The study was conducted with 247 youth in TFC and their treatment parents from 14 TFC agencies in a southeastern state. Half of the agencies were randomized to the intervention condition and received study-provided training and consultation. Control agencies continued to provide training and treatment as usual. Data for the current analyses come from interviews with treatment parents at baseline, 6, and 12 months.

Results

Youth in the intervention group showed significant improvement (compared to the youth in the control group) on the three focal domains – symptoms, behaviors, and strengths. Effects were larger for behaviors and symptoms than for strengths.

Conclusions

This study employs a “hybrid” model to improve practice. It builds upon current practices in existing agencies and infuses additional training and consultation to overcome observed deficits. Such an approach has tremendous potential for moving beyond a singular focus on disseminating evidence-based interventions to a broader view of improving practice in a wide range of agencies.

For the past 30 years, the field of children’s mental health has undergone a number of paradigmatic changes. Beginning in the late 1970s, the focus was on addressing the conspicuous gaps and problems in the overall systems that served children with mental health problems ^1–3 Enthusiasm about developing and expanding “systems of care” was challenged with a series of findings in the final years of the 20^th century that while it was possible to improve systems, it was not clear that such improvements led to increased benefit for children and families.^4,5 This contentious period led to a new focus in which systems came be viewed as potentially necessary but not sufficient to bring about desired individual-level outcomes. This led, in turn, to a heightened interest in understanding the “black box” of treatment. Currently, near the end of the first decade in the 21^st century, this emphasis on “evidence-based” practice has become the new rallying cry. Within the children’s mental health literature, there has been a growing recognition that these paradigms and trends should be brought together: for the most difficult-to-treat youth, significant improvements in outcomes are likely to require practice-level changes that involve provision of effective treatments and supports within the framework created by systems of care.^6,7 Much of the current effort (within systems of care and beyond) focuses on how to develop, evaluate, and disseminate evidence-based treatment. While this emphasis on treatments’ ability to create desired change is critical to the field, it also has the potential to focus on a small number of treatments in relatively few settings and to thereby limit our thinking and recognition about the range of services, treatments, and systems that serve the vast majority of children with serious emotional disturbance and their families in the country.

Treatment Foster Care, one of the few community-based treatment options for youth for which there is a substantial evidence base,^8–10 has thrived in this shifting environment. It is appealing from a “system of care” perspective because it provides least-restrictive residential treatment, allows for family-based living and community-based opportunities for learning and development, and is individualized to address a child’s strengths and needs. Chamberlain and colleagues have done a series of studies to develop the evidence base and have carefully described the model in ways that make it replicable by others.^{8, 10–13} The evidence-based model, Multidimensional Treatment Foster Care (MTFC), was developed in Oregon and is currently being disseminated in over 50 sites in the U.S. and internationally. While this is tremendously promising, there are at least 1,600 programs that provide Treatment Foster Care across the U.S. So, as with most treatment approaches, the vast majority of agencies providing TFC are not providing the evidence-based version.¹⁴

Interviews with directors of TFC agencies in a state-wide sample suggest that there are both practical and philosophical reasons that agencies do not see the current evidence-based model as viable for them (e.g., lack of resources to hire required staff, opposition to use of points charts in homes, use of contract (rather than “in house”) clinicians), but most directors express a desire to improve quality and outcomes in their TFC programs. Quantitative data from these same surveys show that there is certainly a need to improve practice in usual care TFC agencies.¹⁵

The current paper reports on a randomized trial that attempted to enhance TFC in these usual care settings by (a) drawing from evidence-based practice and well-established treatment approaches, and (b) building from “evidence” based on research about “what works” in existing usual care TFC.^16–18 The study was designed to evaluate the impact of increased training and consultation for TFC supervisors and treatment parents in an effort to change practice and improve youth-level outcomes.

Methods

Overall Design and Sample

The study was a randomized trial of TFC agencies in a southeastern state, (conducted in 2003–2008). Randomizing was done at the agency level: 7 agencies were randomized to the intervention group; 7 agencies to the control group. Agencies randomized to the intervention arm received study-provided training and consultation. Agencies in the control arm continued to provide training and services as usual. All youth served by these agencies during the 18-month recruitment period were eligible for inclusion in the study. Overall, 247 youth and their treatment foster parents. Data were collected at baseline, 6, and 12 months from treatment parents and youth. Results presented here reflect an “intent to treat” approach, with all participating youth included in analyses.

Sample Characteristics

A prior state-wide study of TFC provided a list of agencies, from which a subset was selected for the RCT.^{15, 19} The resulting sample included programs distributed throughout the state, two of which were operated by public mental health authorities (one randomly assigned to each study arm) while the others were privately managed. Overall, programs had been operating for 2–15 years and had 13–50 licensed homes at the time of study baseline. The sample included youth who lived in TFC homes in participating agencies at the time the study started as well as all youth who entered the agencies during the following 18 months.

Youth in the intervention and control conditions were comparable on a range of domains. They had an average age of approximately 13 years (range=2–21 at baseline), approximately half were female, and two-thirds were from minority racial/ethnic groups. At study baseline, youth had been living in their current TFC home for an average of 20 months (with a range of less than 1 month to over 12 years).

Treatment parents were also similar between conditions. The majority were female and from minority racial/ethnic groups (mostly African American). Approximately 60% were married. Households included an average of 4 people, and in nearly one-third of homes, there was more than one TFC youth placed (at the time of the baseline interview).

The current analyses use an intent-to-treat approach, so that all available data are used to assess outcomes, regardless of “dose” of treatment. In viewing the results, however, it should be noted that a substantial portion of youth moved between TFC homes or were discharged during the 12-month follow up. By 6 months, 74% remained in TFC (66% in the same TFC home where they were recruited); by 12 months, 61% remained in TFC (47% in the same home). Rates of movement or discharge were not significantly different in the two study arms.

Intervention

The intervention was developed by bringing together data from our previous state-wide descriptive study of TFC with elements of Chamberlain’s evidence-based MTFC. Many of the components that are considered to be critical to TFC were already evident in usual care practice. These included care coordination/case management, a view of treatment parents as key change agents, a team approach to treatment, respite, and work with youths’ families. However, compared to the evidence-based version of TFC, usual care TFC was conspicuously lacking in two areas: intensity of supervision/support of treatment parents by TFC supervisory staff, and proactive teaching-oriented approaches to problem behaviors. Therefore, the study provided training on these two potentially critical areas.

Training with TFC supervisors and treatment parents followed a study-developed protocol titled “Together Facing the Challenge.” ^20,21 This “train the trainer” model included two full days of training with TFC supervisors prior to training with treatment parents. The two-day training with supervisors provided (a) an overview of the upcoming training to be done with treatment parents, (b) discussion about their current practices/interactions with treatment parents, and (c) opportunities to practice skills and training elements so that they could serve as co-facilitators in the treatment parent training. Follow-up consultation visits were held monthly for one year after this initial training. These group-format consultation sessions focused on a combination of preplanned topics as well as discussion/problem-solving on emergent and salient issues from the supervisors.

Training with treatment parents was conducted over a six week period, with 2.5 hour sessions once a week (sessions were held in the evening and a meal and child care were provided). All training sessions were led by the study’s Intervention Director, with assistance from agency TFC supervisors. Topics for the six weeks included: (1) building relationships and teaching cooperation; (2) setting expectations; (3) use of effective parenting tools to enhance cooperation; (4) Implementing effective consequences; (5) Preparing youth for the future; and (6) Taking care of self. All sessions included didactic instruction, role plays/exercises, and homework assignments for the treatment parent to do during the week.²¹

Much of the training built from established parent-training approaches found in MTFC. In addition, two additional elements emerged from our previous study of usual care TFC and were included in the intervention. In contrast to the relatively short-term focus of MTFC, nearly half of the youth in our previous study were in TFC for over two years.^15,19 With these extended stays, two issues emerged that were not formally addressed in MTFC nor in existing treatment as usual TFC: preparation for adulthood and previous trauma. Therefore, focus on transition-related issues was included in the training and consulting work with supervisors. Previous trauma was addressed via training/consultation with local clinicians who worked with youth from the participating TFC programs in Trauma-Focused Cognitive Behavioral Therapy (TF-CBT).²² The study provided a cadre of trained clinicians in each participating community. Whether a specific youth received such treatment was decided by agencies and clinicians on an individual basis.

Data

Data come from in-person interviews with treatment parents at baseline, 6 and 12 months. Primary variables for the current analyses focus on youth-level outcomes. These include the Strengths and Difficulties Questionnaire (SDQ) ²³; Parent Daily Report (PDR) ²⁴; and the Behavioral and Emotional Rating Scale (BERS).²⁵

The SDQ provides an indication of clinical severity of the youth’s problems. This 25-item measure includes five subscales (emotional symptoms, conduct problems, inattention-hyperactivity, peer problems, prosocial behavior) as well as a “total difficulties” score (composed of the first four subscales).²⁶ Current analyses use the total difficulties score.

The PDR obtains information about the number of types of problematic behavior the youth displayed in the past 24 hours. It has been used in previous studies of youth in TFC and foster care to examine patterns of problem behavior over time.²⁸ In MTFC (Chamberlain’s evidence-based model), the PDR is also used for daily data collection and feedback on behaviors to inform treatment.⁹ The PDR was not used in this latter way in the current study; it served only as a data-collection instrument by the research staff at each follow-up interval.

The BERS provides an indication of a youth’s strengths.²⁵ This 52-item instrument includes 5 subscales (interpersonal strength, family involvement, intrapersonal strength, school functioning, and affective strength) and an overall strength quotient, which was used in the current analyses.

All three of these measures have been used previously in research on youth with emotional and behavioral problems and have acceptable psychometric properties. For the SDQ and PDR, a higher score indicates more problems. For the BERS, a higher score indicates more strengths.

Written informed consent was obtained from each youth’s parent/legal guardian for the youth’s inclusion in the study. Written consent was obtained from each participating treatment parent prior to interviews.

Data Analysis

The study included 247 youth, in agencies randomly assigned to one of two conditions: enhanced TFC (n=136) or treatment as usual TFC (n=111). To establish the validity of the randomization, baseline characteristics of participants in the two study arms were compared on a variety of dimensions. As noted in Table 1, there were no significant differences between conditions on the range of included factors. There were also no significant relationships between randomization or baseline characteristics and attrition. Analyses were based on the intent-to-treat sample using full likelihood methods. Such methods provide valid estimates and inferences with incomplete data when assumptions of ignorability are satisfied.

Table 1.

Baseline characteristics of sample¹

	Full Sample n=247	Intervention Sites n=137	Control Sites n=110
Variable
Child age	12.9 ±3.8	12.7 ±3.8	13.2 ±3.8
Child race
White	33%	34%	33%
Black	57%	56%	58%
Other	10%	11%	9%
Child sex (female)	45%	40%	51%
Length of Stay (months)	20.5 ±25.1	20.3 ±26.8	20.7 ±22.9
TP age	48.5 ±10.0	49.0 ±9.1	47.8 ±10.9
TP race
White	22%	25%	18%
Black	74%	71%	78%
Other	4%	4%	4%
TP sex (female)	90%	89%	92%
TP married	59%	61%	56%
Number of homes with multiple TFC youth	31%	28%	35%
SDQ2	16.3 ±6.9	17.4 ±6.8	14.6 ±6.8
BERS3	86.8 ±15.9	86.4 ±15.8	87.3 ±16.2
PDR4	5.8 ±4.8	5.9 ±4.8	5.6 ±4.9

¹No significant differences between groups on any examined variables.

²Possible scores range from 0–40, with higher scores indicating more symptoms

³Scores are normally distributed with a mean of 100 ± 15 for the general population, with higher scores indicating more strengths

⁴Possible scores range from 0–34, with higher scores indicating more behavior problems

Each focal outcome (PDR, BERS, SDQ) was mean-centered and standardized to facilitate comparisons between measures.²⁸ Z-scores were differenced from the baseline to form the primary outcome measure. Therefore, results represent mean-centered changes in standard deviation (i.e., z-score) units.

Using repeated measures linear regression procedures (SAS: Proc Mixed), each outcome was regressed on a model that included wave (as a class variable), a dichotomous proxy variable denoting randomization status, and an interaction term crossing these two. In addition, the baseline z-score for the focal outcome was included to control for differences among subjects in baseline levels of the relevant outcome measure. Covariance structure was specified as first-order autoregressive based on inspection and a comparison of −2 log likelihood values for competing covariance structures. Post hoc contrasts between the differenced z-scores across the two study arms were estimated at 6 and 12 months after adjusting to control for multiple comparisons.²⁹

Because wave and group were entered categorically, results are based on Type 3 tests. Therefore, we present Type 3 components for each level of wave and group as derived from the L matrix of contrasts as well as an F-test evaluating the overall significance of each effect. Type 3 tests are generally considered superior to Type 1 and Type 2 tests and are particularly useful in models with categorical interaction terms where effects are distributed over multiple measures.³⁰ The interaction term, representing the differential rate of change in the mean-centered standardized outcome measure(s) from baseline to 12 months among youth randomized to the enhanced TFC condition, provides the primary test of the study hypotheses.

Results

Tests of the differences on outcomes are presented in Table 2 and Table 3. For all three examined outcomes, there was a significant difference in the rate of change between the two conditions. In each instance, youth in the intervention group showed improvements across time. In contrast, youth in the control group remained relatively stable (or showed minor worsening) across time. Significant wave-by-group interaction terms for the PDR and SDQ measures show that rates of change were significantly greater among youth in the intervention group than the control group. The wave-by-group interaction for the BERS was marginally significant (p<.08).

Table 2.

Outcome models (z-scores)

PDR
	Type 3 L Components for Fixed Effects:					Type 3 Tests of Fixed Effects:
Effect	Coeff Est	Std Error	DF	t Value	Prob > |t|	F Value	Prob > F
Baseline PDR	−.211	.032	1, 200	−6.62	<.001	43.87	<.001
E-TFC	−.233	.073	1, 200	−3.20	<.01	10.25	<.01
Wave						3.37	<.05
Baseline	0
Month 6	.101	.064	2, 198	1.59	.114
Month 12	−.107	.082	2, 198	−1.31	.192
Interaction						3.74	<.05
BL × E-TFC	0
Month 6 × E-TFC	−.289	.127	2, 198	−2.27	<.05
Month 12 × E-TFC	−.370	.163	2, 198	−2.27	<.05
BERS
	Type 3 L Components for Fixed Effects:					Type 3 Tests of Fixed Effects:
Effect	Coeff Est	Std Error	DF	t Value	Prob > |t|	F Value	Prob > F
Baseline PDR	−.275	.033	1, 229	−8.27	<.001	68.45	<.001
E-TFC	.068	.068	1, 229	−.99	.322	.98	.322
Wave						.64	.528
Baseline	0
Month 6	−.009	.065	2, 317	−.14	.888
Month 12	−.079	.074	2, 317	−1.07	.287
Interaction						2.57	<.100
BL × E-TFC	0
Month 6 × E-TFC	.264	.130	2, 317	2.03	<.05
Month 12 × E-TFC	.001	.148	2, 317	0	.997
SDQ
	Type 3 L Components for Fixed Effects:					Type 3 Tests of Fixed Effects:
Effect	Coeff Est	Std Error	DF	t Value	Prob > |t|	F Value	Prob > F
Baseline PDR	−.264	.034	1, 184	−7.69	<.001	59.20	<.001
E-TFC	−.176	.076	1, 184	−2.30	<.05	5.28	<.05
Wave						2.48	<.10
Baseline	0
Month 6	−.130	.073	2, 206	−1.77	<.10
Month 12	−.163	.085	2, 206	−1.93	.055
Interaction						4.84	<.01
BL × E-TFC	0
Month 6 × E-TFC	−.457	.147	2, 206	−3.11	<.01
Month 12 × E-TFC	−.207	.169	2, 206	−1.23	.222

Table 3.

Outcome contrasts (z-scores) at 6 and 12 months

PDR
	Estimate	Std Error	DF	t Value	Prob > |t|
Contrast:
Month 6	0.302	0.109	198	2.78	<0.01
Month 12	0.383	0.140	198	2.74	<0.01
BERS
	Estimate	Std Error	DF	t Value	Prob > |t|
Contrast:
Month 6	−0.243	0.107	317	−2.26	<0.05
Month 12	0.020	0.118	317	0.17	0.866
SDQ
	Estimate	Std Error	DF	t Value	Prob > |t|
Contrast:
Month 6	0.411	0.124	206	3.31	<0.001
Month 12	0.161	0.141	206	1.14	0.2542

Figure 2 depicts these changes over time. The effects were strongest and most sustained for the PDR. In the enhanced TFC arm, rates of problem behaviors decreased across time, and the difference between the two conditions was significant at both 6 and 12 months. Youth in the comparison condition showed slight increases in problem behaviors by 6 months and subsequently remained constant. The SDQ presented a similar pattern across time for the comparison group. Youth in the enhanced TFC condition were markedly improved by 6 months and remained substantially below the comparison condition at 12 months. However, by 12 months, the difference was not statistically significant. The BERS showed the smallest difference between groups. Youth in the Enhanced condition showed small improvements in strengths during the first 6 months (p<.05), but by 12 months, the two groups were no longer significantly different.

Figure 2.

Outcomes across waves (baseline, 6, and 12 months)

Units in above analyses are expressed as z-scores. Converting these back to the units of the focal variables provides a more intuitive and concrete view of the effects. For the PDR, for example, these results suggest that a youth in the intervention group who had a mean level of problems at baseline (5.5) would have, on average, a PDR score of 4.2 at 6 months and 3.8 at 12 months. Similarly, movement on the SDQ would go from 17.5 at baseline, to 14.3 at 6 months, and 16.0 at 12 months. As indicated above, amount of change was smallest for the BERS: from 86.3 at baseline, to 91.0 at 6 months, back to 86.4 at 12 months.

These findings suggest that the observed changes were not strictly linear, suggesting a quadratic model might be more appropriate. To test this, all models were re-estimated after adding a quadratic term for time, including its interaction with group. In no instance was model fit significantly improved by the addition of the quadratic term (results not shown), suggesting that a linear fit, although not perfect, was adequate.

Discussion

This paper reports the initial youth-level outcomes of a randomized trial of enhanced Treatment Foster Care. This study is the first of its kind to examine the effects of increased training and consultation for supervisors and treatment parents in usual care TFC agencies. On the three focal outcomes – symptoms, problem behaviors, and strengths -- the enhanced model showed significant improvements over usual care TFC, particularly at 6 months, when most youth were still in TFC.

The reported study used a somewhat different approach to addressing change than has been done in much of the dissemination research in children’s mental health. Together Facing the Challenge built upon two sources of input – elements from evidence-based practice and data from “what was working” in existing TFC agencies. This hybrid model was designed to infuse existing agencies with improved practices that had been shown in randomized trials and observational study to be associated with better outcomes for youth. It attempted to recognize the resource limitations and philosophical orientations of existing agencies, while providing increased training and support in critical areas. The study focused on programs that were representative of existing statewide agencies, rather than on agencies that had resources, leadership, or incentives to initiate the change process. Therefore, it moves beyond the early adopters³¹ to examine potential for change across the range of existing programs. The current approach and findings apply only to TFC. However, it may be useful to assess whether such a hybrid approach may have implications for improving usual care treatment in other interventions.

While these results are promising, it should be recognized that the level of change seen in any individual outcome is in the small to moderate range. The preponderance of evidence across multiple outcomes – all in favor of the intervention group – bolsters support for the intervention. The decrease in effects by 12 months for two of the outcomes needs additional work to fully understand. At this point, it is not clear whether this reflects a true diminishing of the effect post-intervention, a lack of uptake to sustain the intervention beyond the study-driven training phase, or if the intent to treat design is diluting effects for specific subgroups (e.g., youth who remained in TFC throughout the period, youth successfully discharged, youth who received different sets of other services). Additional analyses are needed to examine processes and key factors that underlie these intent to treat findings (e.g., quality of implementation, receipt of additional services beyond TFC).

Conclusions

These results suggest the positive potential of this hybrid approach to improving outcomes in a wide range of TFC agencies. At this point, much more analysis is necessary to understand the processes by which these changes were created and to examine whether there are subgroups (of agencies, families, or youth) for whom the intervention was particularly successful. Such analyses will help identify potential causal mechanisms, the importance of the various intervention components, and, hopefully, targeted subgroups for future work. Such work should lead to improved knowledge of “what works” across a broad range of agencies and programs to elevate the overall practice and outcomes for youth in community-based residential placements.

Figure 1.

Recruitment and Enrollment