Revolutionizing Child Welfare with Outcomes Management

Abstract

Outcomes management technology holds great promise for improving the quality of services provided to youth in the child welfare system. Advantages include better detection of behavioral health and trauma-related issues, early indicators of case progress or risk of failure and program- and system-level learning. Yet organizational barriers to implementation persist. Attention is spent in this paper on addressing these barriers so the use of outcomes management technology becomes a common practice. A model for predicting resiliency is presented, along with case examples demonstrating its potential use for treatment planning and monitoring progress.

Introduction

In the last 10 years, outcomes management technology has advanced to the point where it is affordable; easy to use and implement; and capable of collecting, analyzing, and reporting data that can transform the delivery of care for youth who have been abused and neglected. The project discussed in this paper is a glimpse into the possibilities that exists. And the lessons imparted here show an understanding that it will take courage to move the child welfare field forward—but they also suggest that realizing the full potential of outcomes management technology is not so far out of reach.

The promise of outcomes management systems and the role they can play in improving clinical care and service delivery was first discussed more than a decade ago.¹ Outcomes management was distinguished from research then as a methodology of collecting and using clinical data to alter treatment results—a direct means to translate research into practice. The following key components were identified as necessary to a successful outcomes management system: collecting data from the client at periodic intervals; a focus on strengths and skills; and real-time actionable feedback to inform and guide treatment course decisions.²

Research quickly accumulated on the importance of using clinical feedback to improve case-level outcomes, ³ and youth-serving organizations began examining whether or not to integrate outcomes assessment into their practices. Early systems relied on archaic strategies for data collection, such as manual data entry, which did not provide immediate feedback to clinicians. As technology evolved, real-time reporting and adaptive response became possible; clinicians now had up-to-date information on a youth’s progress that they could use to facilitate discussions with the family regarding treatment planning and progress.

The advent of real-time feedback, in particular, allowed clinicians to see much earlier in the treatment process which youth in their case loads were improving, stagnating or declining. Over time, data reporting became more sophisticated, and began to include benchmarking of outcomes in relationship to similarly placed youth.

Despite these advances and the millions of dollars spent to launch new service demonstration projects, outcomes at the aggregate and system level often showed little improvement.^{4, 5} Many of these projects provided states with in-kind match funds to build systems of care for youth, but failed because they were ultimately designed based on what services were fundable—rather than on what services were the most effective or appropriate. These demonstration projects often did not use assessment, reporting, and high-level analytics to determine optimal service mixes for specific clinical subgroups, such as abused youth. Without the macro-level view of what works best for whom, these projects suffered from a significant handicap from the start—one that persists today. This is an opportunity lost.

With tight resources and bloated health care costs, the pressure to produce better outcomes for less money—especially for high-cost groups such as youth who have been abused or neglected —has never been greater for the child welfare system. The consequences of not using outcomes management routinely are already evident. Among youth in child welfare ages 6 to 17, 40% meet criteria for one or more DSM diagnoses; those in foster care use mental health services up to 15 times more often than other children.⁶ Exposure to abuse in childhood frequently precedes adult psychiatric disorders, including borderline personality, dissociative disorder/identity, suicidality, self-harm behaviors (e.g., cutting, substance abuse, sociopathy), and generational violence.^{7, 8} Serious long-term consequences include negative/confused self-concept,^9,10 difficulty setting boundaries, and dealing with conflict.¹¹ All of these are associated with a pattern of future victimization and abuse. ¹⁰

Predictive analytics and outcomes management systems have demonstrated their ability to improve adult behavioral health; ¹² these technological advances may similarly help child welfare agencies achieve better case-level results, improve overall system performance, and document the value of their services. Unfortunately, establishing a supportive culture for implementing outcomes management systems has not come as far as the technology in the last decade. For example, the perception by staff that data will be used punitively, rather than to “learn,” continues to hinder the widespread application of these valuable systems. To the extent that workers feel data will be used to reinforce positive service models and support them in their work, experience has shown that the implementation of outcomes management systems tends to go well. On the other hand, when data are used to discipline staff or when the focus of the implementation is solely at the executive level to control utilization of services, outcomes management is often seen as something negative, and its full potential is far from being realized. Outcomes management must be framed as an organizational learning opportunity to discover “what works for whom.”

Overcoming Barriers to Widespread Adoption of Outcomes Management

Establishing outcomes management requires multiple levels of organizational planning and coordination. At first glance, the start-up costs and the learning curve may seem like too steep a price to pay.

Step one, as described above, is to change the organizational culture to accept measurement as a valuable tool to better match individuals to the care they need. ^{1, 13} Unfortunately, providers are often unwilling to alter already established clinical practices.¹⁴ Step two is accepting that implementation may take longer than an organization would like. The following is a summary of current literature on the lessons learned from the implementation of outcomes management systems, which reveal the significant, though not insurmountable, obstacles to revolutionizing the child welfare system:

Get buy in from top management early

The upper echelon must see that the benefits of collecting outcomes data outweigh the initial costs before implementation begins. Once they recognize the value across the case, program, and system levels, they will be more effective in setting up operational supports, such as providing staff for training, encouraging regular case review of data, and assisting staff in understanding how to use data to improve services. Perhaps most importantly, they will be more likely to use data positively to improve quality.

Focus on training

In the public sector, the high turnover rate of clinicians and case managers continues to be a significant challenge. Successful large-scale implementations require multiple, redundant plans for the departure and arrival of staff. Newer technologies allow for Web-based, paced training and testing to ensure new staff are quickly introduced to outcomes measurement procedures, and current staff can take refresher courses without overwhelming limited resources.

Focus on long-term use of data

Many organizations initially implement outcomes management technology to comply with accreditation or other regulatory requirements, but this is generally not sufficient to keep an organization engaged for the long-term. In these scenarios, outcomes management programs are seen as little more than a burdensome obligation. But outcomes management systems can help improve clinical capacity, develop new programs and identify cost-effective treatments, among other benefits. With a culture of measurement established, outcomes management and prediction become the norm for organizational learning.

Strength-based assessment –vital to a successful outcomes management program

Simply implementing an outcomes management system that contains symptomology and/or needs is not enough. Families do not want to be seen as simply a constellation of symptoms. Nor should they be. Numerous international and U.S. child development studies show that acknowledging and using youth strengths for planning behavioral health treatment and monitoring progress is a vital factor to realizing better outcomes.^15–18 Many youth who are entering services today contain a combination of psychological and trauma-related symptomology, and are often also economically and socially disadvantaged.¹⁹ The literature suggests that despite the complexity of these cases, these youth frequently possess modifiable protective and vulnerability factors that may be addressed in programs such as mentoring and social skills training.^20–22

But attempts to incorporate the strength-based perspective in case management continue to be hindered by a preference for a definition of recovery that is based on the absence of symptoms. The alternative is a more comprehensive view that also includes the presence of skills and strengths that will give the youth a better chance at becoming fully functioning students, employees and members of society.²³ Progress in this area has largely centered on resiliency.^{16, 17} Increasingly, the definition of resilience has expanded to include the construct that true resilience is demonstrated in the face of adversity.²⁴ Research shows that intentional self-regulation and development of cognitive and behavioral skills related to resilience may triumph over the influence of negative environments.²⁵ The challenge of today’s social worker: Understanding how youth learn from adversity and applying the lessons of resilience research to help improve outcomes for youth. It is here where outcomes management can come into play.

An advantage of outcomes management is its ability to continuously and methodically provide feedback to caseworkers and families. For example, an outcomes management system that uses strength-based measurement can measure and monitor building blocks of resilience, such as learning to come up with multiple ways to view problems, seek support from others, and manage anger and frustration. This data can then be used to influence the course of care, and to motivate children and their families. Research has shown that feedback is associated with improvements in specialist medical care; how this feedback loop will work in the behavioral health field or with youth in child welfare who are constantly challenged is currently being investigated.²⁵ Early indications suggest that the continuous feedback when combined with strength-based assessment may improve learning and thus, outcomes.²⁶

The behavioral health outcomes management system used for the following case study was designed specifically for youth in the child welfare system who have been abused and/or neglected. The National Institute of Child Health and Human Development funded its development (Grant 5 R44 HD049955-03). The computer-based system is designed to be supportive of mental health and recovery from abuse, and measures a number of possible long-term outcomes, including improvement in resilience, symptomology, and permanency of the youth in out-of-home placement. For all of the outcomes domains, research has been focused on creating content that provides preliminary clinical data to support improved treatment planning and then the prediction of varying outcomes.

In this paper, the prediction of resilience is used to show the reader just one example of how content from an outcomes management system can be used to illuminate different case patterns of progress and/or decline. Additional steps beyond this point include determining how model variables can be practically translated into treatment plans and how data can be used to monitor whether interventions provided to the youth are effective. The discussion below focuses specifically on how certain features of outcomes management systems may be designed to support provider systems wishing to focus on building strengths, including: ^{27, 28}

Focus on the positives

Effective report design allows consumers to quickly understand their progress and where they need to focus their efforts. The reports, produced in real-time, highlight visual diagrams that display youth strength scale scores first (e.g., assessment of resilience), and then summarize client symptomology through descriptive scales that are scientifically- grounded and normed. This format allows for a straightforward review of reports with the client, which helps to improve communication between the youth, family, and caseworker and supports better engagement throughout the course of treatment. ²⁹

Increase the chances that other informants will participate and reinforce client gains

Participation in treatment planning can empower youth and families, ³⁰ but often staff struggle with including families in the process for a variety of reasons, including the added time it can take. Automating a portion of the process eliminates at least one hurdle, which makes family participation more feasible. Youth and parent self-assessment data gives everyone a clear “voice” in treatment planning.

Provide opportunity for disagreement or to reveal uncomfortable information

Two important components of nearly all therapies are skills training and cognitive modification of false belief systems. Client-friendly reports provide a platform for discussion with clients to explore contradictory information and resolve issues in the therapeutic relationship, including negative feelings about the provider. ³ Using outcomes technology may offer another advantage: Youth may reveal negative or shame-based behaviors (e.g., self-harm, cutting) through computer-based self-assessment more often than he/she ordinarily would share with the therapist. ^3,31

The example below shows how the selected strength-based content from the larger child welfare outcomes management system could be used to provide clinically meaningful information that can be turned into treatment goals and then monitored via the outcomes management system.

Method

Data Source

Data were collected from 2007 to 2009 using the computer-based behavioral health outcomes management system described above. Interagency treatment team members (counselors, case managers, and therapists), youth, and their families (birth, foster, or other caregivers) completed online assessments to provide information useful in monitoring mental health status and progress.

Although performance sites suggested in discussions that strength development in youth and families was important, few sites had specific programming. The most common outcomes that the child welfare programs were concerned with centered on permanency of youth in out-of-home placement.

Agencies participating in the study were located in Pennsylvania, Iowa, Missouri, and North Carolina. All performance site protocols, as well as the parent protocol for the project, were approved through either the site’s Institutional Review Board (IRB) and/or through an external IRB. In many of the sites, in addition to project approval from the National Institutes of Health (NIH) and an external IRB, a secondary approval was required from the county and/or affiliated university. A Certificate of Confidentiality covering the project data was obtained, as well for data collection at all of the performance sites. All relevant staff at the sites were provided with training on how to use the assessment software and interpret the self- and clinician-rated reports generated by the system. Although outside the scope of this paper, the investigators of this grant were directed to build additional training technology to support the often-rapid staff changes that occurred throughout the duration of the project.

Participants and Recruitment

The primary participants were youth, ages 10 to 19, who were receiving services through child welfare systems due to familial abuse and/or neglect. Youth were only enrolled in one child welfare agency or county at the time of this study and their data were only counted once in the results. The sample consisted of youth placed out of home who completed outcomes assessments using the computer-based system. Inclusion criteria included youth who were (1) victims of confirmed abuse; (2) in treatment programs providing abuse recovery services; and (3) not living in the environment where the abuse occurred. Youth and parents were told that the purpose of the assessment was to assist case managers in planning effective treatment that would help them overcome the effects of abuse. The sample was constructed to support modeling of permanency as well as resilience.

The project used a consecutive sampling strategy for recruitment of youth and parent informants. Youth aged 10 and older (and their biological or foster parents) were asked to complete assessments along with the primary case manager/clinician for the youth. The youth and/or guardians were approached by case managers and caseworkers who explained the benefits and risks of enrolling in the project. In all cases, written assent from youth and at least one parent was obtained before any assessments were completed. Youth and parents completed the assessment at intake and again at 90 and 180 days (or upon adoption/reunification, if prior to 180 days) during the study.

To increase participation, the project reimbursed youth and participating parents $33 per completed follow-up assessment for the youth and/or parent; the money came from the National Institutes of Health (NIH) grant (5 R44 HD049955-03) that funded the project. The county agencies that participated were not reimbursed because their mandates did not allow it.

Although intake data were available for 476 youth, follow-up data were often more difficult to collect despite the monetary incentive to families. Of youth enrolled in the project 30.6% of the original sample met the criteria (n=146) of having at least one intake and one follow-up available for modeling. Of the 146 youth used in the analyses, youth averaged 15.43 years of age at intake (s.d.=1.95), ranging from 9.96 to 18.99 years. Nearly half (48.6%) of the youth were white, 29.5% were African-American, 2.7% were Native American Indian and 19.2% were other races. About twenty-three percent (22.6%) were Hispanic/Latino (all races).

The analysis sample was similar to patients without follow-up data regarding age, gender, Hispanic ethnicity, the Resiliency Scale score, the Abuse-Victimization Adjustment Scale score (AVA), and several measures of symptomology, including Conduct Disorder and PTSD at intake. The analysis sample, however, had a higher percentage of whites (48.6% versus 33.1%), a lower percentage of African Americans (29.5% versus 52.6%), and a similar percentage of other races compared to the non-follow-up group, χ2(2)=21.85, p < .001. The analysis sample also appeared more severe on the measures of Depression (F(1,474)=10.10, p < .002) and ADHD (F(1,474)=10.79, p < .001).

During the data collection period, caseworkers were encouraged to share the reports with youth and families, in line with the values of the Child and Adolescent Service System Program Model (CASSP). Sites indicated that they shared the reports at least half of the time, and that doing so increased a family’s willingness to complete future assessments. Reasons for limited sharing included a lack of time or a perception that the parents and/or youth were “not ready” to see the data. This is an example of the “push and pull” nature of the process of change; upon further questioning, some clinicians admitted they were uncomfortable not being in charge and the consultative role with the family was unfamiliar. These revelations point to the ongoing need for training to help staff accept the altered roles they will play as outcomes management technology becomes more prominent.

Instruments and Measures

In general, during the project there were three software components that could be completed independently—a youth self-report, a parent/caregiver report of their own strengths and problems, and a clinician assessment that includes evaluations of both the youth and the caregiver.

The youth component of the assessment is appropriate for ages 10 to 18. As part of the original scientific validation of the tool, a series of items were constructed to measure whether or not a youth could complete the assessment reliably. More than 99% of the youth in the intake sample (n=476) reported that they either answered all or most of the questions as well as they could; only 5% of the youth reported they did not understand some or most of the items in the assessment.

The youth assessment that was completed by the study sample contained the following symptom and strength scales:

Youth Abuse Victimization-Adjustment (AVA)

The AVA scale identifies factors associated with recovery from abuse, and its impact on self and relationships. It also measures changes in schema related to abuse. Its content draws heavily upon research that shows that abuse and the related high levels of shame are linked to (1) poor adjustment;³² (2) relational problems such as projective fantasy and approach-avoidance conflicts;³³ and (3) the use of self-silencing techniques such as conflict avoidance, suppressing anger, and putting the needs of others ahead of oneself.³⁴ The AVA scale consists of 11 attitudinal items (e.g., it doesn’t matter if I am hurt because I am bad; I blame myself for bad experiences) that ask the youth to discuss the effects of abuse, as opposed to defining and discussing actual abuse events. All AVA scale items are on a six-point scale ranging from Strongly Agree to Strongly Disagree. Internal consistency in the NIH validation sample was .93.

Youth PTSD

A Post Traumatic Stress Disorder (PTSD) scale was designed by recasting selected symptoms for self-report from the Diagnostic and Statistical Manual, Fourth Edition to address a youth’s intrusive, hyperarousal, and avoidance PTSD symptomology. Additional items were added after reviewing relevant literature, especially as it pertains to dissociation from severe abuse, and by consultation with child welfare experts. The PTSD scale contains 10 items that ask the youth to indicate, based on a “bad experience” (e.g., hitting, “bad” touching, verbal abuse), how often a specific PTSD-related symptom has occurred in the past month. For example, a response option is “can’t stop thinking about the bad experience.” Internal consistency is .77.

Youth Assessment of Symptomology

Youth symptom scales for Depression/Anxiety (alpha of .86), ADHD (alpha .81), Conduct Disorder/Oppositional Defiance (alpha .85), screens for chemical dependency, and other mental health problems (e.g., Psychosis, Autism, OCD) were included in the assessment. These scales were taken from a behavioral health outcomes management system designed for 10- to 18-year-olds who were enrolled in community mental health programs.³⁴ Youth respond to questions asking whether, and how often, they have experienced specific symptoms in the past month (every day, most days, many days, some days, few days, no days).

Youth Strength Scales

Three developmental strength scales were included that have been determined to have predictive validity for treatment outcomes with youth with severe behavioral health problems.³⁵ Based on principal component analysis: (1) Resilience (alpha .89), a youth’s perception of their feelings and behaviors in response to adversity; (2) Life Satisfaction (alpha .93), a youth’s perception of the extent to which they are able to internalize positive relationships and experiences such as enjoying daily life, having hobbies, participating in community or school activities, or feeling hopeful about the future; and (3) Youth/Parent Relationship (alpha .91).

Data Analysis

The purpose of the analysis was to develop a predictive model of resilience that could be translated into effective case management reports. Resiliency Scale scores from youth self-report assessments were obtained at intake and at the most recent follow-up assessment available. Data were from the first follow-up for 63.0% of the cases, from the second follow-up for 21.2%, and from the third to sixth follow-up for 15.8%. The youth averaged 188.97 days between the intake and follow-up assessment (s.d.=133.11), ranging from 7.03 to 731.91 days.

Change in the youth Resiliency Scale score served as the dependent variable, with positive change scores signifying improvement in resiliency. Twenty-eight percent (28.1%) of youth had reliable improvement (i.e., improvement beyond 1.96 * standard error of the difference between two scores or SE_diff ³⁶), 55.5% did not change, and 16.4% had reliably lower scores at follow-up compared to intake.

Power analysis, using GPower 3.1.2, indicated that a maximum of 17 predictors could be used in a sample of 146 patients, assuming alpha = .05, power = .80 and a moderate effect size (f² = 0.15), to test a one-group fixed model for R2 deviation > 0. Two predictors—days since intake and the initial Resiliency Scale score—were chosen a priori to control for treatment “dose” and initial level of the outcome. The remaining item pool included 243 self-report items and eight summary scales. The items covered the following domains: demographics, youth strengths, family and interpersonal relationships, past/current stressors, internalization of blame and abuse-victimization adjustment, substance use, and motivation to change. In addition, mental health items assessed PTSD, ADHD, Depression/Anxiety, and Oppositional Defiance/Conduct Disorder behaviors, and screened for symptoms of Panic Disorder, OCD, Bipolar Disorder, and other indicators of potentially severe psychiatric disturbance (e.g., hallucinations). The scale scores were Parent-Child Relationships, Mastery, Life Satisfaction, Abuse-Victimization Adjustment, PTSD, ADHD, Depression/Anxiety, and Conduct Disorder.

A primary analytic task was to narrow the pool of additional potential predictors to 17 or fewer, including the two identified above. Because the goal was not explanatory, decisions regarding which predictor variables to include were chiefly empirically driven. Items with large amounts of missing data (> 10% missing) or severely restricted response ranges were excluded. Next, univariate analyses and scatterplots explored the relationships between each of the items and change in resiliency. Items with statistically significant relationships (p < .05) were considered first, with other variables added in later stages of the analysis. Inter-item correlations were examined to identify variables that were essentially co-linear. In such cases, items having the strongest correlations with the dependent variable were retained for further analysis.

Through this process the pool of potential predictors was reduced to less than 100 items. A series of linear multiple regression analyses was used to find the best set of predictors. In each analysis, days since intake and the initial Resiliency Scale score were entered in a single block. A second block, including up to 12 additional variables from the pool, were evaluated using stepwise regression. Variables from the second block that entered the equation were retained for further testing. This process continued in an iterative manner until the final set of 11 predictors was identified. These were:

• Intake Resiliency Scale

• Days since intake

• Trying to change life (AVA)

• Does some things well (Resiliency)

• Others concerned about drug/alcohol use (Substance Screen)

• Concerned about own drug/alcohol use (Substance Screen)

• Uses drugs/alcohol to change mood (Risk Factor)

• Feels has caused trouble to parents (Parent-Child Relationship)

• Feels wouldn’t be liked if known (AVA)

• Feels on edge, and (PTSD)

• Feels cranky or grumpy (PTSD)

After the item pool was reduced, a standard multiple regression analysis was performed between change in resiliency as the dependent variable and all eleven predictors. The sample was too small to allow for validation with a hold-out sample. However, the coefficient of cross-validation was estimated using the Wherry and Browne formulas. ³⁷

Results

Parameter estimates are displayed in Table 1. Together these items accounted for 55.4% of the variance in resiliency change scores (Adjusted R-Square = 51.7%). R was significantly different from 0, F(11, 134) =15.12, p < .001. Using the Wherry/Browne formula, the coefficient of cross validation was estimated at 0.48, suggesting modest shrinkage across samples.³⁷

Table 1.

Standard Multiple Regression Analysis of Change in Resilience for 146 Youth in Child Welfare Programs Using Youth Self-Report Data.

Predictor	B	SE B	t	p
Constant	3.57	0.41	8.67	.000
Resilience at Intake	−0.62	0.08	−8.18	.000
Days since intake	0.00	0.00	−0.26	.794
Trying to change life	0.35	0.13	2.72	.007
Does some things well	−0.16	0.07	−2.19	.030
Others concerned about drug/alcohol use	−1.08	0.27	−3.92	.000
Concerned about drug/alcohol use	1.04	0.30	3.42	.001
Uses drugs/alcohol to change mood	−0.68	0.20	−3.29	.001
Feels they have caused trouble to parents	−0.26	0.08	−3.18	.002
Feels that they wouldn’t be liked if known	0.14	0.04	3.40	.001
Feels on edge	−0.33	0.13	−2.64	.009
Feels cranky/grumpy	0.35	0.14	2.40	.018

Overall Model R-Square=0.55; Adjusted R-Square=0.52, F(11,134)=15.12, p<.000.

Wherry/Browne Coefficient of Cross-Validation: 0.48

^*Note: PASW Statistics 18.0.3

Prediction of Resiliency as an Outcomes Management Tool

The following graphs display case studies of change in resiliency for three different youth taken from the project’s databases. Little is known about the interventions they received, but it is possible to evaluate their progress in these graphs. Each graph includes the actual Resiliency Scale score, the predicted Resiliency score and error boundaries based on the standard error of the estimate (+/− 1SYX). Although each figure contains data for a full year of treatment, in practice the graph would be sent to the clinician in charge of the youth’s case immediately upon completion of each assessment. At intake, the graph would display only the prediction lines and the first “actual” score. The “actual” score allows the clinician to quickly identify the youth’s current level of resiliency skills. The prediction line shows the expected change in resiliency given the current “mix” of services offered at that site. This information can be used to identify treatment goals and to target appropriate additional services as necessary. Subsequent versions of the graph, updated when each new assessment is completed, allows the clinician to monitor progress, to again make appropriate changes to services, and to monitor the impact of such changes on an on-going basis.

Figure 1 shows data for a youth with moderate resiliency skills at intake (4.5 on the 6-point scale). As can be seen in the figure, this youth progressed as predicted (+/− 1 SYX), maintaining a moderate level of skills by the end of the treatment year.

Figure 1.

Youth A: Actual and Predicted Resiliency Scores by Time in Treatment Demonstrating Progress as Expected.

Figure 2 shows progress for another youth. Like Youth A, this one was expected to maintain intake-level resiliency scores during treatment. Youth B, however, improved beyond what was expected, reporting the highest resiliency score by the ninth month in the program.

Figure 2.

Youth B: Actual and Predicted Resiliency Scores by Time in Treatment Demonstrating More Progress than Expected.

The third case (Figure 3) shows a youth with moderate resiliency skills at intake who is predicted to make substantial improvements during treatment. At the first post-intake assessment, Youth C’s skills were essentially unchanged since intake, still far below the predicted level. At this point, the clinician might consider a change in services. Resiliency scores remained below the predicted level for the remainder of the year.

Figure 3.

Youth C: Actual and Predicted Resiliency Scores by Time in Treatment Demonstrating Less Progress than Expected.

Discussion

Advances in information technology have made it possible to collect, store, and analyze large volumes of clinical data at a cost that is feasible for child welfare agencies. The growing database of information from many episodes of youth care would allow for the derivation of patterns of factors associated with success or failure in recovery from abuse. Consequently, youth would have more focused care from the outset, resulting in fewer instances of youth being under-treated or receiving the wrong type of services. This is the promise of outcomes management: a system that supports case- and system-level learning to ultimately improve the delivery of care and outcomes.

In this paper, factors related to resilience were used as an example of one way an outcomes management system can contribute to organizational learning. With the youth samples discussed above in mind, parents, youth, and caseworkers could use the scores for resiliency to initially determine the need for possible interventions that seek to improve the strength of family relationships and coping skills, and address behavioral health, and then determine the effectiveness of these interventions with subsequent scores. This information can then influence organizational programming so that youth in child welfare receive services based on need, rather than just what is available. And the data only become more meaningful over time.

The experiences of this project showed that many of the hurdles to effective use of outcomes management were related to organizational environment. It is hard for organizations to convince workers and families that using this type of technology will not be punitive. As positive reinforcement of the power of outcomes management, in this project agencies received case reports and quarterly aggregate reports, which allowed them to see for the first time the percent of youth who improved, stayed the same, or got worse. In some cases, there were surprises, such as youth whose strengths improved while their problem areas worsened. The combination of data on strength and symptomology from multi-informants (youth, parent, clinicians) made it clear that all are important to determining what works best for whom.

Limitations

The predictive model used for this project was developed to illustrate the basic method of how such data from an outcomes management system could be used in clinical practice. For this project, the model was designed using a fairly small sample size with no validation study, which limits its direct generalizability to other service settings. This is especially important given the data mining nature of the analyses and the inevitable capitalization on chance relationships. The model cannot be applied without attention to site-specific calibration, a process similar to that commonly used in medical settings when validating a new diagnostic technique. This testing period can be time-consuming and expensive, with little immediate clinical benefit until enough data are collected to establish norms and to validate trends. But the initial costs are worth the long-term payout. Once established, the on-going collection of outcomes data provides a foundation for a learning system, including data-driven feedback and decision supports that impact outcomes for single patients and the system as a whole.

Because the purpose of the analysis was prediction and not explanation, the importance of variables that made the final model should not be overstated. Other items might have served equally well, but were eliminated due to the manner in which the item pool was reduced. As more data becomes available, the model can be further developed and refined, with a focus on identifying clinically actionable factors that can be used in tailored treatment planning.

In addition, the period between assessments in the data was large, averaging almost 4.5 months from intake to the first follow-up (mean=17.65, s.d.=9.35), preventing us from modeling early response to treatment. Given a large enough sample with multiple measures soon after intake, hierarchical linear modeling could replicate individual growth trajectories over time, predicting not only the final status, but the expected rate of change as well. Potential “treatment failures” could be identified earlier in the process and clinical intervention could change the course of outcomes.

Implications for Behavioral Health

Despite the lack of ideal settings, and budgets during the duration of the project, this paper provides a strategy for systematic data collection that could improve a variety of treatment outcomes, especially when programming is in place that considers the “whole” child. The technology platform for the outcomes management system used in this project allows for training, assessment, and reporting – all features critical to successful implementation.

This paper also looks at the less tangible components “not in the box” that are very important to creating an organizational culture where data can be used effectively, including acceptance of technology and the idea that “prediction” is possible or even, desirable. Predictive capabilities, touched on in the resiliency model, become refined the more an outcomes management system is used. From these “smart” systems, optimal service models can be developed by matching service and cost data with clinical data. Predictive models can also establish a “failure boundary” at the case level, which shows that a given child is in danger of failing to achieve an outcome in adequate time to alter the treatment strategy.

Despite the complex behavioral health, systematic, and environmental hurdles child welfare youth face, this project supports the belief that with outcomes management technology the true potential of building strengths in families can be realized. Continuous feedback combined with assessment content can impact treatment, placement decisions, and outcomes.

Large health care systems have bridged the gap between the ideals of outcomes management and successful implementation; the public sector can, too. The choice that needs to be made is whether to focus on short-term goals and putting out the fires of today versus investing in outcomes and other learning processes whose benefits will mean fewer fires tomorrow.