Change What? Identifying Quality Improvement Targets by Investigating Usual Mental Health Care

Abstract

Efforts to improve community-based children’s mental health care should be based on valid information about effective practices and current routine practices. Emerging research on routine care practices and outcomes has identified discrepancies between evidence-based practices and “usual care.” These discrepancies highlight potentially potent quality improvement interventions. This article reviews existing research on routine or “usual care” practice, identifies strengths and weaknesses in routine psychotherapeutic care, as well as gaps in knowledge, and proposes quality improvement recommendations based on existing data to improve the effectiveness of children’s mental health care. The two broad recommendations for bridging the research-practice gap are to implement valid, feasible measurement feedback systems and clinician training in common elements of evidence-based practice.

Introduction

“It is difficult and perhaps foolhardy to try to improve what you do not understand” (Hoagwood and Kolko 2009; p. 35). If we want to improve children’s mental health care, we need to understand its current functioning, identify what is working well and what is broken, and tailor quality improvement efforts for maximal potential impact. Effective, sustainable prescriptions for change should be based on accurate knowledge about the current system, including organizational structures and culture, clinical practices, values, and the identified strengths and weaknesses of all these. Unfortunately, our knowledge about usual care (UC) mental health care for children and families is surprisingly limited. Several researchers have noted the striking dearth of research on UC practice (Bickman 2000; Burns et al. 1999; Hoagwood and Kolko 2009; Weisz et al. 2006); and UC psychotherapy practice, in particular, remains a largely unexamined “black box” (Bickman 2000). Our lack of valid, reliable information about UC is shocking given that more money is spent in the United States on care for children’s mental health problems than for any other childhood health issue (Soni 2009). In our efforts to bridge the research—practice gap, we need to conduct more research on typical or usual practice to identify quality improvement targets.

In this first of a series of commissioned articles for this special issue, we will underscore the importance of understanding the current status quo in children’s mental health care as a necessary foundation for identifying and implementing needed changes and innovations. We will highlight recent research on treatment processes and outcomes in UC psychotherapy practice and propose quality improvement initiatives that address identified limitations in current UC and thus have great potential to enhance the effectiveness of community-based mental health care. We focus on psychosocial care delivered once children and families enter treatment, and thus do not address important public health challenges associated with significant unmet need for services and/or disparities in access to care (which are addressed in other articles in this special issue). In addition, given that non-residential care is the most common type of service received by children and families (Burns et al. 1999), we focus primarily on psychosocial interventions in this context, while acknowledging the importance of research on other types of services such as residential care (e.g., Curry 2004) and medication treatment (e.g., Olfson et al. 2002). Given the current reality that an estimated 15 million children receive mental health care (U.S. Department of Health and Human Services 1999) at a cost over $9.6 billion in 2006 (Soni 2009), what can we do to maximize the potential benefit, utilizing the significant fiscal, workforce, and family resources currently devoted to this care more efficiently?

Why Bother Examining “Usual Care” (UC) for Children?

One cannot expect to close the science-practice gap without knowing the nature and scope of UC. A clear understanding of the science-practice gap is informed both by (1) the articulation of empirically-supported practices, and (2) articulation of the range of current usual practices in community-based care outside of research contexts. The former task has been pursued vigorously for over 15 years, and has fueled dynamic and productive scholarly debate. Understanding usual care practice has received comparatively far less effort or interest, and thus, the true magnitude of the disconnect problem remains largely unknown. This is not a problem that will be solved with more randomized clinical trials, or with increasingly well-tuned definitions of evidence based practices. We need data addressing the other half of the problem namely, what type of treatments are being delivered in UC.

Without reliable data on UC practice, we can only speculate about what treatment approaches are being delivered, the extent to which treatment resembles empirically supported treatment, what factors are related to effective treatment, and how care varies by location, provider, or patient characteristics. Rigorous research on UC practice could provide a “road-map” of care, identifying specific discrepancies between evidence-based (EB) treatment approaches and UC practice that may be the most potent targets for quality improvement efforts, and identifying provider and patient characteristics associated with different types of care. Quality improvement programs have been shown to have lasting impact on psychosocial care (e.g., Wells et al. 2007). UC data would also provide baseline benchmarks on which to assess the impact of quality improvement interventions. In addition, research identifying links between specific UC practices and child/family outcomes could potentially identify treatment approaches that are particularly promising in the UC context, thus complementing our efforts to identify evidence-based practices with practice-based evidence. Although “what works” is usually interpreted as a question of efficacy, in this context we would add what works can also mean what is practical, feasible and affordable, and therefore what is effective. Although the findings on the effectiveness of UC are far from encouraging, millions of families are engaging in treatment and understanding more about successful delivery models in current care may provide a link to more effective implementation of quality improvement strategies.

Rich data on UC contexts (e.g., organizational structure, financing, culture and climate, providers’ attitudes and skills, clients’ needs and preferences) is essential to inform quality improvement efforts. Such data can forecast potential challenges for efforts to integrate EB practices and can be used to tailor intervention efforts to best fit the context, thus optimizing the likelihood of sustainability. Innovation diffusion theories suggest that innovation adoption is facilitated by adaptations to specific contexts (Rogers 2003). More specifically, research in mental health has demonstrated that training efforts to change provider behavior are more effective when they are tailored to the current care context (Casper 2007).

Why Don’t We Know More About Usual Care Psychotherapeutic Practices?

The lack of research on UC is likely due to several interacting factors, including professional politics and priorities, as well as methodological challenges. It is difficult and potentially “dangerous” to examine usual care practices (Bickman 2000). Critical examination of existing practice may challenge providers’ (agencies’ and individuals’) livelihoods, core beliefs, and “raison d’etre”. From a research funding perspective, studies that “simply” characterize existing practice may not be perceived as innovative or exciting compared to studies that test new innovations. In fact, the proportion of funding devoted to practice-based research (research based in existing service systems) is minimal compared to other areas of health care research (Westfall et al. 2007). However, even with adequate funding, rigorous investigation of UC psychotherapeutic practice and outcomes is methodologically complex (Garland et al. 2009). While there is a history of epidemiological and health services research describing service use patterns broadly in terms of number and type of services used or delivered, there is far less research on the psychotherapeutic intervention processes delivered, i.e., the interactions between providers and clients within “sessions” or visits.

There are few established methods or measures of psychotherapeutic processes and no consensus on the level of analysis on which to assess such processes (e.g., Chorpita and Daleiden 2009). Given that traditional definitions of practice in the efficacy literature have used manuals to define treatments (Chambless and Hollon 1998), there is little guidance for determining appropriate metrics for characterizing UC psychotherapeutic practices that are likely not driven by manuals. Thus, it is difficult to contrast treatment techniques or content in research-based manualized psychosocial interventions to those currently delivered in UC interventions. Development of a taxonomy to describe a comprehensive array of psychotherapeutic practice elements would be particularly useful to this field. Chorpita et al. (2005) define practice elements as discrete clinical techniques or strategies used as part of a larger intervention plan, such as relaxation training for children or training parents in developing a time-out program, etc.

Methods for outcome measurement have improved but implementation of standardized outcome assessment in UC contexts is challenging (Bickman 2008a, b). Accountability is difficult in all areas of health care, but is perhaps more challenging in mental health due to the arguably more subjective definitions of diagnoses, treatment practices and outcomes (compared with other specialties). However, the complexity in assessing practices and outcomes in mental health care is not an acceptable excuse for not conducting such assessments. Given the inherent complexities in assessing treatment practice and outcome, there is significant risk of over-simplification in accountability efforts; for example, relying exclusively on measures of consumer satisfaction may be important in its own right but should not necessarily be interpreted as assessment of clinical effectiveness since satisfaction has not been strongly linked to clinical improvement (Lambert et al. 1998; Garland et al. 2003b). Regulatory agencies have identified quality indicators for children’s mental health care, such as the Healthcare Effectiveness and Data Information Set (HEDIS) measures used by the National Committee for Quality Assurance (NCQA). These types of indicators, such as follow up after hospitalization for mental illness or medication management, are not designed to assess quality of psychotherapeutic processes or content in that they usually rely on administrative data sources. Thus these types of quality control mechanisms do not address quality of psychotherapeutic interventions. A recent review by Bickman and colleagues (Bickman et al. in press) found no research evidence that ties these NCQA measures to clinical outcomes and furthermore there appeared to be no evidence that states were measuring quality of UC in a systematic and scientifically valid fashion as indicated by their web sites.

The recent support for research in UC practice settings (such as NIDA and NIMH support for Practice based networks) and research to assess meaningful indicators of the quality and impact of UC treatment (summarized in the next section) are steps in the right direction. We call for more efforts to expand our knowledge of UC treatment processes and outcomes, as well as tests of the generalizability of emerging findings.

What Do We Know Already About Usual Care?

In this section we briefly highlight some of the available information on UC psychotherapeutic interventions for children and families, including research examining (a) outcomes; (b) common treatment practices; (c) linkages between outcomes and practices; (d) characteristics of the patient population; (e) characteristics of providers, including training and supervision; and (f) provider attitudes, values, and perceptions.

Research on UC Outcomes

Although most clients report benefiting from UC treatment (Garland et al. 2003a; Lambert et al. 1998), data on average improvements in clinical outcomes in UC psychotherapy are quite discouraging (e.g., Bickman et al. 2000a, b; Weiss et al. 1999, 2006). For example, in a meta-analysis of the effectiveness of UC treatment for children with various diagnoses, the average effect size for UC treatment, compared to a variety of control groups, was close to zero, reflecting minimal impact on children’s symptoms and functioning (Weisz et al. 2006).

However, not all findings have been negative. In a system-wide review of outcomes achieved in a statewide system of care, Daleiden et al. (2006) found that outcomes were modest but favorable prior to a planned system improvement effort, with mean rates of improvement on the Child and Adolescent Functional Assessment Scale (CAFAS; Hodges and Wong 1996) of around 1.1 units of improvement per month. Over a 4-year period of quality improvement efforts, the rates of improvement more than doubled. Although without a comparison group, it is difficult to pinpoint what caused these changes, the findings do raise the critical question of when UC might no longer be “usual.” More will be said about that point below, but clearly the quality of outcomes in usual care can differ across time and location, and it is likely that some of these differences will depend upon one’s definitions and metrics for what constitutes UC practice.

Research on UC Treatment Practices

Epidemiological and health services research has documented patterns of mental health service utilization for children, reporting on amount and broad types of care and factors associated with service utilization (e.g., Farmer et al. 2003; Garland et al. 2005; Hazen et al. 2004; Jones et al. 2009), and psychotherapy research has investigated psychotherapy processes in depth for research-based samples, but there has been limited research at the intersection of these fields, examining psychotherapeutic processes utilized in community-based UC practice contexts. The few existing studies UC treatment practices utilize different methods including case record review (Zima et al. 2005), therapist self-report (e.g., Baumann et al. 2006; Bearsley-Smith et al. 2008; Weersing et al. 2002), and observation (Garland et al. in press), and thus yield different perspectives on practice patterns. In their case record review of care for 813 children in publicly funded out-patient care in California, Zima et al. (2005) found wide variation in adherence to pre-determined indicators of quality care for ADHD, depression, and/or conduct disorder conditions (indicators represented both EB practice and expert-endorsed best practice guidelines for assessment, psychosocial and psychopharmacological intervention and monitoring). On average, patients’ charts passed 46% of the basic treatment quality indicators. Adherence to clinical assessment indicators (e.g., suicide risk assessment), was relatively high, but adherence to medication-specific monitoring indicators (e.g., height, weight, blood pressure monitoring) was relatively low (<25%). Contrary to expectations, the study reported that adherence to quality indicators did not differ significantly by child demographics or clinic factors. This study identifies several quality improvement targets for UC services and indicates that such improvement efforts are needed in all types of clinic settings, serving all types of patients.

Studies assessing therapists’ self-reports on psychotherapeutic practice similarly report great variation in practices. In general, results reflect eclectic treatment patterns with psychotherapists often explicitly identifying themselves as “eclectic” and also endorsing several treatment approaches derived from different theoretical orientations (Baumann et al. 2006; Kazdin et al. 1990; Koocher and Pedula 1977; Weersing et al. 2002). Interestingly, in one study, UC psychotherapists strongly endorsed the value of practice elements common in EB treatments, but observational data of sessions from these same clinics indicated that several of these same EB practice elements were delivered rarely (Brookman-Frazee et al. 2008). The extent to which attitudes about practice predict practice behavior is not well established. Related research indicates that the concordance between different data sources (e.g., therapist self-reports vs. observer ratings) on psychotherapeutic practice elements is not strong (Carroll and Rounsaville 2007; Hurlburt et al. 2009).

In an observational study of UC psychotherapy practice, Garland et al. (in press) followed 218 children ages 4–13 with disruptive behavior problems served in six publicly-funded clinics for 16 months. There was great variability in amount and type of care received; children attended many sessions (mean = 22; range = 0–63), and caregivers attended at least part of most sessions. Most children received additional services including psychoactive medications (62%), and school-based services (88%). Videotapes of therapy sessions were collected (n = 3241) and a random sample of 1215 was coded for an array of 27 practice elements. Results indicate that therapists delivered many different elements within and across treatment sessions targeting children and their caregivers, but average intensity of these observed elements was quite low (2.3 on a 1–6 scale). Thus, the treatment could be characterized as reflecting great breadth of practice elements, but not depth for any specific intervention strategy.

The investigators examined the extent to which UC resembled practice elements common in EB treatments for this patient population. Some common elements of EB practice were observed frequently (e.g., positive reinforcement for children, psychoeducation for caregivers), but they were not observed at the intensity specified in an EB protocol. Other common EB elements (including directive, skill-building techniques such as role-playing/rehearsal, modeling, and assigning/reviewing homework) were observed rarely (i.e., <25% of sessions) at any intensity.

These data offer a rare glimpse inside the “black box” of publicly-funded UC psychotherapy. Generalizability to other service contexts is yet to be determined, but the initial findings support some of the presumed strengths and weaknesses of UC psychotherapy. On average, children received a lot of services, but the treatment lacked depth and specificity. Therapists were observed to be very supportive and empathic and clients’ ratings of the quality of the therapeutic alliance were quite positive. However, therapists were not often observed delivering the types of active, directive, intensive intervention strategies which are common in EB treatment for this population. The positive perceptions of therapeutic alliance are consistent with other studies (Tryon et al. 2007). Clients (youths and caregivers) generally rated the alliance more positively than did the therapist, which is consistent with Bickman’s current research suggesting that counselors may not be accurate in their judgment of their clients’ perception of the therapeutic alliance.

Several technical reports from the Hawaii system give a similar portrait of the characteristics of UC, using an entirely different methodology. As part of electronic billing practices, the Hawaii Child and Adolescent Mental Health Division now requires providers to report practice elements used in the past month for all youth registered in the system (CAMHD 2008), using a checklist of practices that has been used to code the treatment outcome literature as well (Chorpita and Daleiden 2009). Since establishing this methodology, the Hawaii CAMHD system has annually published reports of the common practices used in its system across a variety of child problem areas (e.g., Daleiden et al. 2004). Daleiden and colleagues drew four basic conclusions, consistent with those outlined by other researchers above: (1) most UC involves a moderate usage of strategies that are also found in EB practices; (2) UC practice also included a moderate number of practices that were not ever used in EB practices for a given problem type; (3) when strategies from EB practices were used, there tended to be greater variety than found with the average structured treatment protocol, and (4) these findings held regardless of whether youth had “pure” (single disorder) or comborbid diagnostic profiles. These results support those obtained by observational analysis, showing that empirically indicated techniques are used in UC, but with less depth than in structured treatment protocols supported in the clinical outcome literature.

Research on Linkages Between Practices and Outcomes

Emerging work in the Hawaii system has begun to examine whether these patterns of clinician-reported practices are associated with improved outcomes to the extent that they include higher doses of procedures that are common among EB practices (Mueller et al. 2009). Preliminary results demonstrated that across all youth in the state system with ADHD, the average rate of improvement was higher among those youth whose services included the use of practices common among EB psychosocial treatments for ADHD (e.g., problem-solving, time out) relative to those whose services did not include such practices.

Research on Characteristics of the Patient Population

One of the oft-cited challenges of implementing evidence-based treatments in routine care settings are the reports that the characteristics of the patients differ from research contexts to UC contexts (e.g., Southam-Gerow et al. 2008; Baker-Ericzen et al. 2009). Early findings suggested that patients presenting to UC settings tend to be more ethnically diverse, and to exhibit more comorbidity (Southam-Gerow et al. 2003), and more recent work has shown that even in the context of similar symptom severity, families reporting to UC settings tend to be significantly lower in income and significantly higher in psychosocial stressors (Southam-Gerow et al. 2008). Recent findings by Baker-Ericzen et al. (2009) similarly showed that families served in publicly-funded UC present with more life stressors including lower socio-economic status and higher rates of parental depression compared with most research trial samples. Although there are mixed reports regarding significant discrepancies between samples included in research trials and patients served in usual care, this issue is of concern to practitioners and managers (Essock et al. 2003). The extent to which such differences may moderate the effectiveness of treatment is not known.

Research on Characteristics of Providers, Including Training and Supervision in UC

There is no national database providing descriptive data on mental health care providers. However, a fairly recent national study provides the most comprehensive data to date regarding children’s mental health care. The Clinic Systems Project (CSP), within the MacArthur Foundation funded Research Network on Children’s Mental Health, collected data from 1200 individual clinical providers from 100 clinics nationwide (representing 26 states). Details of the study are provided in Schoenwald et al. 2008, and Glisson et al. 2008. In sum, they report much variability in the organizational characteristics of agencies (funding, management structure, public/private partnerships, etc.). In terms of individual providers, it is noteworthy that fewer than 10% were doctoral-level providers; the majority held master’s degrees (67%). In addition, the majority of clinicians were female (76%) and Caucasian (71%), with a mean age of 38 (SD = 11, range 21–74). Schoenwald et al. (2008) also report that the vast majority of clinics provide clinical supervision as an ongoing activity. More research is needed to understand more about how clinical supervision is delivered in the UC context.

Research on Providers’ Attitudes/Values/Perceptions

Over the past decade, there have been several surveys and/or qualitative studies examining mental health care providers’ attitudes about evidence-based, and/or manualized treatment, to elucidate potential barriers to implementation and to inform training efforts (e.g., Aarons 2004; Addis and Krasnow 2000; Baumann et al. 2006; Borntrager et al. 2009; Brookman-Frazee et al. 2008; Essock et al. 2003; Rubin and Parrish 2007; Sheehan et al. 2007). Some common themes emerge across diverse studies. In general, there is significant variability in definitions and interpretation of evidence-based treatment and/or manualized interventions. Individual providers often express concern about how implementation of prescribed evidence-based practices may limit their autonomy and flexible use of clinical judgment. Concern about the lack of time and resources for training and consultation are also common among individual providers and program managers.

There has been less attention to evaluating providers’ attitudes about the effectiveness of the current service system, yet this is critical for assessing “readiness to change” (Prochaska and DiClemente 1982). Limited research suggests that most UC providers have confidence in the effectiveness of the care they are providing (Garland et al. 2003b). It may be particularly challenging to motivate anyone to change their behavior when they do not perceive that changes are necessary. This is not to say that UC providers are uninterested in expanding their clinical skills; in fact, we have found that providers are motivated to learn about new clinical approaches, but this doesn’t necessarily translate to changes in actual practices. Bickman and colleagues have developed a theoretical approach to change based on several social psychological approaches including goal setting, cognitive dissonance and attribution theories that posit that motivation for change will best occur when the clinician has the goal for providing effective treatment, has experienced dissonance concerning this goal and attributes this shortfall to something she can do something about (Riemer et al. 2005; Sapyta et al. 2005). If clinicians are confident in the effectiveness of current care, then this may prove to be a barrier to implementing and sustaining quality improvement efforts that change practice behaviors.

Our anecdotal experience reinforces perceptions of prevailing values among many UC psychotherapists, which places paramount importance on building and maintaining a supportive therapeutic relationship with clients. Data and common sense support the significance of the therapeutic alliance in psychotherapy, but prioritizing the perceived quality of the relationship should not interfere with delivery of effective therapeutic interventions, especially as noted earlier that the alliance is usually rated very high in UC. Therapists often express concern that some more directive, skill-building therapeutic approaches may interfere with relationship building. Many of the therapists with whom we’ve worked express ambivalence about interventions perceived as “directive.” Some express reluctance to implement a therapy model that requires agenda-setting, arguing that therapists should not impose their own agenda, but rather always maintain flexibility to “meet the client where they are.” Dissemination of research that indicates that therapeutic alliances are not weakened and in fact may be strengthened in the context of active, directive, skill-building interventions may ultimately help to shift provider attitudes in significant ways to inspire readiness to change.

More research is needed on workforce development in mental health (see the paper on workforce development in this special issue). We need to learn more about the range of attitudes, training and practice behaviors in the diverse UC mental health workforce, including attention to training programs across disciplines, supervision practices, continuing education, and the extent to which these experiences are associated with differential effectiveness of treatment, which is a hotly debated topic (e.g., Krause and Lutz 2009). In addition, we need more information about clinical decision-making and providers’ use of clinical data, including preferences in content and format of clinical data. This information could be used to advance the development and implementation of measurement feedback systems (as described in the recommendations section later). For example, research suggests that providers prefer feedback on specific items as opposed to global scores (Bickman et al. 2000a, b). Decades of research have noted the limitations of sole reliance upon clinical judgment for treatment planning and evaluation of effectiveness (e.g. Garb 1989, 1998, or Dawes 1996). We need to conduct more collaborative research with providers to design effective mechanisms to improve reliability and validity of problem identification, diagnostic assessment, treatment planning, and evaluation of treatment processes and outcomes. Challenges in all these areas have been documented in UC (Bickman 2008b; Garland et al. 2003a, b; Haine et al. 2007; Hawley and Weisz 2003; Lewczyk et al. 2003; Jensen and Weisz 2003).

What More Do We Need to Learn About Current Mental Health Care for Children?

Given the limited research on UC psychotherapeutic practice, we have limited information on potentially informative variations in care by geographic area, service/provider types, and/or patient characteristics. We don’t know, for example, how publicly-funded care may differ from privately-funded care in terms of practices or outcomes. In this broad overview we have not addressed important questions regarding diagnostic specificity in treatment approaches, or even the major challenges in achieving valid, reliable diagnostic assessment. There is likely great variability in the extent to which UC practice relies upon detailed assessment to drive treatment delivery, or whether treatment approaches are delivered more generically. The psychotherapy literature has for decades reflected the tension between emphasis on specific treatment techniques that are usually diagnostically specific and broad “common factors” that may be the active ingredients in effective psychotherapy. Recent calls for dissemination of transdiagnostic psychotherapeutic interventions suggest that these approaches may be a better ecological fit for community-based UC settings (McHugh et al. 2009). Achieving an optimal balance of fidelity to empirically supported interventions with documented potential for clinical benefit and feasible adaptation to local UC realities (organizational, cultural, clinical realities) is the goal of implementation efforts. Methods for achieving an optimal balance of fidelity and flexibility are currently being developed and tested (Chorpita and Regan 2009; McHugh et al. 2009).

Implementation science in mental health is relatively new and thus we are just learning about the most effective methods for training mental health providers and/or achieving sustainable change in organizations and provider behavior, although recent adaptation of behavior change theories are very promising (e.g., Perkins et al. 2007). We need to incorporate theory and methods on behavior change from associated fields and gain a greater understanding of motivation and incentives for changing practice (at the organizational, individual provider, and patient levels). Emerging research on clinical decision-making may also advance our knowledge of essential meta-cognitive skills for effective practice. Research on UC patients’ preferences, treatment selection, and treatment engagement is also critical for informing quality improvement. At present, potential patients usually have no information about potential effectiveness of various types of treatment or data on specific providers upon which to make an informed choice. At best, they may have “hearsay” information about a provider’s reputation. Ironically, consumers have easy access to extensive data to inform selection of most types of goods and services, but are left virtually “in the dark” regarding selection of mental health care which has greater significance for their well-being.

Finally, we need to understand what may actually be working in UC practice. Millions of families continue to engage in treatment, despite the fact that data suggests there is minimal average change on measurable outcomes. There is something rewarding and reinforcing about participating in UC (for both clients and providers), or else it would not have been sustained as long as it has. The majority of participants perceive significant benefit and express satisfaction in UC treatment; our outcome measures likely do not capture many of the benefits. However, there is great variability in service engagement and there are high dropout rates in UC, so perceived benefit is not universal. McKay et al. (2004) have developed interventions to address practical and attitudinal barriers to attendance in community-based mental health care for children and have effectively improved engagement rates. The extent to which improved engagement, as defined by attendance at more sessions, is associated with greater benefit is not well established.

How to Enhance the Effectiveness of Usual Care: Quality Improvement Efforts

Critical examination of potential costs and benefits of quality reform efforts requires knowledge about the status quo. What governs how services are delivered? What role does effectiveness play in determining who will be served, who will provide the services and what services will be provided. As noted earlier it is estimated that $9.6 billion dollars was spent on mental health care for children and adolescents in 2006. We cannot find data on how much is being spent on administration and enforcement of all the rules and regulations. But in general health this number is estimated to cost the practices for interactions with health care plans alone at least $23 billion to $31 billion each year (Sakowski et al. 2009). Total administrative cost in the United States is estimated to be $320.1 billion. The reforms we are presenting here would cost a very small fraction of what we are paying to maintain the status quo. We need to start managing our services with data, capitalizing on the data we already have and not relying on complex rules and regulations that absorb huge amounts of resources and in the process squeeze out resources needed for real reform.

Given the litany of insufficiencies and challenges we have highlighted in current children’s mental health care, it is tempting to suggest creating an entirely different model of mental health services. Some of the articles in this special issue advocate for new paradigms for children’s mental health services, whereas others advocate for incremental steps with the potential to significantly improve the effectiveness of services within current care structures. Regardless of the strategy chosen, we know it is extremely challenging to change service delivery systems and routine practices. Although it is tempting to point fingers, the truth is we have not figured out how to change services in an efficient and acceptable manner. This problem is not unique to child and adolescent mental health services but is a part of all health care as witnessed by the controversies on health care reform and by the significant federal investment in translational research (e.g., see: http://nihroadmap.nih.gov/clinicalresearch/overview-translational.asp).

In addition to exploring entirely new and different mental health care delivery models and paradigms, we believe there are essential quality improvement efforts that must be implemented now. These complementary quality improvement initiatives are informed by our review of the strengths and weaknesses of the “status quo” and are thus tailored to fill identified gaps in care and to fit within the context of current care systems; these complementary initiatives have potential to significantly improve the effectiveness of care. The first is the development of measurement feedback systems to improve accountability and build ongoing evaluation of impact into treatment (Bickman 2008a, b; Chorpita et al. 2008). The second is integration of common elements of evidence-based practice, including attention to provider training and supervision (e.g., Daleiden and Chorpita 2005; Garland et al. 2008).

Development of Measurement Feedback Systems (MFS)

It should no longer be acceptable for funders or accreditation agencies to require the collection of data that are too often left to gather dust in some office or be hidden in a mass of computer files. The data collection efforts must be designed to be useful to clinicians and administrators in order to provide effective treatment. To be clinically useful the information must be fed back to clinicians in a measurement feedback system (MFS). A MFS is a battery of comprehensive measures that are administered frequently concurrent with treatment, and provides timely feedback to clinicians and supervisors (Bickman 2008a). The MFS must not only be able to measure client progress but also report on key clinical processes and when using an EB treatment to measure adherence to EB protocols. A good MFS should have measures that are short, psychometrically sound, and are useful in everyday practice by clinicians. Further, MFSs should be comprehensive by assessing several domains by multiple reporters that include treatment progress (e.g. youth and family outcomes) and treatment processes (e.g. therapeutic alliance and treatment activities). Such a MFS provides systematic feedback that can be used to enhance clinical decision-making, improve accountability, drive program planning, and inform treatment effectiveness (Chorpita et al. 2008; Kelley and Bickman 2009).

Several studies have already demonstrated the ability of feedback to affect outcomes with adults (Slade et al. 2008; Davies et al. 2008). Bickman’s center has recently completed a large scale study of the implementation of a MFS they call Contextualized Feedback Intervention and Training or CFIT. Preliminary analysis shows that weekly feedback can affect clinician practice behavior and improve outcomes for youth. However, implementation was difficult with over half the feedback reports not even accessed. What they have learned from this 6 year study has resulted in revised training and software systems (CFIT 2.0™) that will be subject to further testing and evaluation but that has already been adopted by some service providers.

The introduction of MFSs into our current mental health services can provide a window on UC. The large data sets produced by MFSs can be studied to provide imperfect answers to the perennial question of what works for whom and under what conditions. These questions cannot be answered by the limited number of randomized clinical trials that we can afford. MFSs have the potential to reinforce accountability and attention to clinical improvement that is not often a focus of current system-level evaluation. MFSs enhance management’s ability to identify and reward the most effective institutions, programs, providers and practices. MFSs may alter practice not only by supporting clinical decision-making but also by helping assure that treatments labeled as EB treatments, when brought into community settings, actually are effective. The introduction of a MFS in mental health fits with the current emphasis on electronic record keeping, although the development of the required software programs is complex and costly, but technological advances are constantly improving potential efficiencies and feasibility. MFSs potentially have great benefits but these benefits cannot be realized if they are not used properly in practice. We know very little about how best to implement and sustain such interventions. The execution and success of MFSs hinge on a complex mixture of organizational and clinician level variables. Much change is needed at the clinician, provider and policy level to help ensure the success of MFSs.

Integration of Common Elements of Evidence-Based Practice

In addition to implementation of MFS’s, continued efforts are needed to shift care toward more effective delivery of EB practices. To date, efforts to implement EB practice have relied primarily on training providers in individual treatment models that are often diagnostic specific. Innovative complementary approaches to achieving sustainable change in provider behavior should also be explored, including training on practice elements common across individual treatment models (Chorpita et al. 2005; Chorpita and Regan 2009; Daleiden and Chorpita 2005; Garland et al. 2008; McHugh et al. 2009). There are several arguments for identifying and selecting elements of empirically supported treatments that are common across multiple individual treatment protocols. For clinicians from diverse educational backgrounds, training in common elements of EB practice can provide a foundation for improved practice, emphasizing the ongoing development of critical treatment skills that are likely to apply to a variety of clients. Rather than learning multiple individual treatment protocols, it may be more practical for individual clinicians to learn and master several core elements for different types of presenting problems. These core common elements could be delivered flexibly for the clients who present with multiple different types of problems. Implementation models perceived as more flexible may inspire stronger intentions to change practice, thereby increasing the likelihood that EB treatment strategies are actually adopted and sustained over time (Casper 2008; Perkins et al. 2007). A focus on flexibly applying and individually tailoring common EB treatment elements to the client may fit better with the diagnostic complexity and comorbidity so common in community-based patient populations (Chorpita et al. 2005). Thus, a common elements approach to implementation of EB practice may better meet the diverse clinical needs of children and families served in community settings, and may more readily fit within the existing service context (e.g., organizational structures, clinicians’ preferences, patient characteristics).

Chorpita and colleagues’ have developed a “Distillation and Matching Model” for deriving profiles of common elements of EB practice. They have demonstrated that it is possible to “distill” out specific practice elements that are uniquely associated with EB treatment protocols as applied to particular youth populations (e.g., depressed adolescents). With this approach, one can derive EB “practice element profiles,” representing elements of EB protocols unique to different clinical conditions and patient characteristics. These profiles can, in turn, inform the development of treatment plans that include procedures common to EB practices that have been applied successfully to youth with similar characteristics. As noted earlier, preliminary reports from the State of Hawaii suggest that this effort of “building treatments from elements” was part of a sustained and significant system-wide improvement in measured rates of functional impairment for children receiving services.

However, although promising, this approach has not been tested in a controlled trial, and a common EB practice elements approach to improving care remains speculative. Research is needed to determine whether this way of conceptualizing EB practice can be operationalized effectively and translated into improvements in the effectiveness of care. Simply because a treatment element is common to multiple efficacious individual treatment models does not necessarily imply that it is a potent mechanism of therapeutic change. However, in the absence of definitive research on the most potent mechanisms of therapeutic change, it seems reasonable to speculate that such common elements of EB treatment element may be worth investigating. Given these points, we feel that the approach of designing treatment plans using prescribed elements, as opposed to using formal EB packages, should only be performed in conjunction with an existing MFS, so that the immediate effects of chosen elements can be documented rather than assumed.

Conclusions

The limited available data on UC practices and outcomes indicate that significant improvements in the mental health care system are needed. We have briefly summarized existing data on usual care practices and outcomes and highlighted potential incremental quality improvement efforts that address identified problems in current services. These modest complementary service improvement changes call for (a) systematic, ongoing, and clinically useful measurement of treatment processes and outcomes, and (b) integration of common elements of evidence-based practices in routine care. These suggested reforms address identified gaps in knowledge and service impact. These reforms also reflect important themes implicit across the papers in this special issue, namely, the need to (a) improve accountability and empirically informed decision-making at the individual provider and service system levels, and (b) improve the translation of research to practice by capitalizing on decades of clinical research that has identified practice elements most likely to have beneficial impact for children and families.