A Measurement Feedback System (MFS) Is Necessary to Improve Mental Health Outcomes

This essay takes the position that mental health (MH) services for youth are unlikely to improve without a system of measurement that is administered frequently, is concurrent with treatment and provides feedback. The system, which I characterize as a measurement feedback system (MFS), should include clinical processes (mediators), contexts (moderators), outcomes, and feedback to clinicians and supervisors. In spite of the routine call to collect and use outcome data in real world treatment, progress has been painstakingly slow.^1–3 For example, Garland and colleagues found that even when outcome assessments were required, over 90% of the clinicians surveyed used their own judgment and paid little heed to the data.⁴ A more recent national survey of MH service organizations serving children and families indicated that almost 75% reported collecting some standardized outcome data.⁵ However, just collecting data on an annual basis will not result in improvement.

Measurement is not enough

Feedback from clients and families naturally occurs in treatment but it is highly filtered, biased, and subject to distortions caused by the use of cognitive heuristics and schemas.⁶ This informal and flawed feedback needs to be supplemented by a measurement feedback system (MFS) that uses valid, reliable and standardized measures. This system is central to quality improvement, professional development, as well as enhancing accountability.

Feedback has been successfully applied outside of MH for several decades.^7–8 However the application of a fully implemented MFS is in its infancy in mental health. A MFS has been shown to improve outcomes in adult MH, especially for those clients who were either not improving or deteriorating while in therapy.⁹ It has rarely been applied in children’s mental health. Yet researchers have demonstrated that the benefits of feedback can be substantial and replicable.¹⁰ The idea of using systematic data in treatment is not new ¹¹ but computer technology and advances in psychometrics makes a MFS more feasible. Despite this, use of a MFS has not been widely accepted and is often subtly rejected with arguments about scarcity of resources rather than patent opposition. Learning to what extent MH services are a good investment and implementing quality improvement efforts to make them better should be one of the highest priorities. However, there are many barriers to implementing a MFS --and few incentives.

What are the barriers to the adoption of a MFS?

Practitioners and managers report several reasons for not using a MFS, including: amount of paperwork, the large time burden, insufficient resources, low clinical usefulness, confidentiality, potential misuse, low scientific merit, and value differences.^2,12–13 There are five not so discernible barriers to the adoption of a MFS that are particularly important.

1. Improving MH outcomes has no obvious financial value

Payment for services is typically based on just the number of hours or days and the location of the services (e.g., hospital, outpatient clinic). Occasionally more experienced or educated providers will be paid more. In general, however, one unit of MH service, be it a visit or a day, is equivalent to another, thus making it a commodity. Because there is no widespread use of measures of effectiveness or quality of services, the commoditization process results in competition being based primarily on price. This is an advantage to funders since it should result in lower prices. However, it also may result in less effective services and a disincentive for improving services. Although pay for performance (P4P) schemes are growing in popularity in the general health sector, they are rare in MH.¹⁴ However, P4P should not be undertaken unless a mature measurement system with high integrity and security is in place. It is likely that MFSs will not succumb to a similar commoditization process because of the existence of indicators of measurement quality such as validity and reliability.

There is good evidence that decreases in the price of MH services have occurred, especially when compared to other health services. It has been reported that the value of behavioral health benefits decreased 54 percent from 1988 to 1998.¹⁵ From 1980 to 1997, the share of total claims accounted for by MH and substance abuse declined from 7.8% to 1.9%.¹⁶ The number of specialty MH providers grew at half the average annual rate of non-specialty providers from 1993 to 2003.¹⁷ Overall, annual costs per youth decreased $157 (14.4%) between 1997 and 2000. The decrease was driven by a combination of fewer outpatient visits (−1.3%) and a decline in payments per outpatient visit (−6.1%).¹⁸ The point here is not that we should pay more for MH services of unknown effectiveness but that in order to reverse the downward spiral on the price, services must be able to be differentiated on meaningful indicators of effectiveness.

2. Organizational and psychological factors

Currently everyone except the client appears to benefit from not having a MFS. Since effectiveness data are not available, states (typically the funder) can claim they are meeting the needs of their citizens without dealing with ineffective and problematic (in contrast to scandalous) services. Service providers can maintain that their use of public funds is justified because they use evidence-based treatments (EBTs), which may or may not be effective or implemented properly. Supervisors can continue to do their supervision, paradoxically based primarily on what their supervisees tell them. Clinicians can avoid disconfirmation of their effectiveness and the potential negative psychological effects by not considering alternative sources of information other than their own observations and intuition. All participants in the system can avoid the political, financial, and organizational problems of implementing a MFS and the data that it produces. These barriers to the implementation of a MFS have an immediate impact but hopefully the long term benefits of MFSs will prevail over the barriers. The benefits of a MFS are mostly in the long term and depend upon clinically effective treatments as well as solving many issues in developing, implementing and sustaining the system. From this perspective it is very understandable why MFSs are slow to be adopted.

3. Evidence-based treatments (EBT) and practice guidelines

Strong confidence in the putative effectiveness of EBTs, both pharmacological and psychosocial, may be misplaced. For example, Weisz and his colleagues found that although EBTs for youth were modestly superior to usual care, that advantage only occurred when the EBT was evaluated by the developer of that EBT.¹⁹ The factors determining the conditions under which EBTs are effective still remain to be clearly identified. Even if the effectiveness of EBTs are taken at face value, there are additional problems associated with depending on them to improve outcomes in the real world.

In an extensive review of the literature on continuing medical education, researchers found that the quality of the evidence for its effects on knowledge, attitudes, skills, practice, and outcomes to be low or very low.²⁰ In mental health, Ganju concluded, “a key finding is that training alone, even when it is fairly intensive, appears to increase knowledge but has a limited impact on practice.”²¹(^p4) It would be imprudent to depend only on the training of clinicians in EBTs to attain high fidelity implementation and the maintenance of fidelity. However, a national survey found little evidence of the monitoring of the ongoing effectiveness of EBTs.²²

Without a MFS it is unlikely that whatever was learned in a workshop will continue to be used reliably. In addition, even if a treatment model is strictly followed, there is little assurance that the treatment will be equally effective in different contexts. Ongoing measurement is necessary to maintain fidelity and to understand how treatments may be successfully adapted in different environments. MFSs do not compete with EBTs, but help to sustain and improve them. The concept of assessing the results of any treatment is a key step in the classic evidence-based medicine approach to practice.

Without monitoring an EBT it is illusory to claim that the EBT is effective. However, of greater concern is that EBTs will be used as a substitute for a MFS. That is, training in EBTs becomes the goal and displaces the original purpose of providing effective treatment. It is much easier to claim success because some number of clinicians have been trained rather than demonstrate it improved client outcomes. There is no longer any excuse for misconstruing inputs with outcomes. EBTs are not structured in a way that they can be mechanically implemented without variations introduced by the clinician and the service organization. MH services will not be successful in removing the influence of the clinician or “clinician proofing” treatments anymore than the field of education has been successful in “teacher proofing” the curriculum.²³

4. Accreditation and licensing

There is no substantial empirical data that licensing and accreditation affect the outcome of services. For example, a national survey of state licensing, regulating, and monitoring of residential facilities for children did not list evidence of effectiveness or measurement as one of the procedures required to obtain licensure or certification.²⁴ The National Committee for Quality Assurance (NCQA) has only two measures as part of its HEDIS system that specifically target youth. One is follow-up after hospitalization for mental illness and the second is follow-up for children prescribed ADHD medication. In 1998 the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) expanded its ORYX™ performance measurement initiative to behavioral health.²⁵ There are no specific performance measures for child and adolescent behavioral health and no specific criteria for measures. The primary purpose of these mechanisms appears to be to ensure a minimal standard of quality and safety. However, the Commission on Accreditation of Rehabilitation Facilities (CARF) has comprehensive standards for meeting its performance improvement criterion. These standards include measurement at intake, discharge, post discharge and other intervals of reliable and valid clinical data that are used at least annually in a performance analysis.²⁶ Whether these more rigorous standards are followed and affect outcomes is yet to be determined, but CARF is moving in the right direction.

5. Clinical experience and judgment

Clinicians, as all other professionals, see themselves as competent. Why stay in a profession if you think you are ineffective? One implication of this sense of efficacy is to lower motivation to adopt anything new. Anecdotal feedback from some clinicians in our current MFS study (described below) is that a MFS is probably good for new clinicians, but not needed for experienced ones.

There is little research support for the belief that more experienced clinicians produce better outcomes. It has been found that the number of years of experience, amount of supervision, and accreditation were not related to judges’ ratings of competence.²⁷ A review of the literature did not find a consistent relationship between “clinical competence” and outcomes.²⁸ Michael and his colleagues conducted a meta-analysis of child and adolescent treatment studies for depression and found that professionals and graduate students produced equivalent outcomes.²⁹ Because we have yet to find substantial evidence that general training or experience affects outcomes does not imply that clinicians do not affect these outcomes. A multilevel real world data-analysis of 1998 psychotherapy outpatients and 60 clinicians found that 17% of the variance in rates of improvement was explained by therapist differences.³⁰ This was twice as high than that found in similar studies^31–33, which suggests that naturalistic samples may include a wider range of therapist skill than do controlled clinical trials since the best managed trials appear to show the smallest therapist effects.³⁴

The barriers for adoption of MFSs are not solely due to service providers. Researchers have been slow to develop valid and feasible measures, feedback systems, or methods to integrate feedback into clinical practice. It is likely that the low demand for MFSs has contributed to the prolonged pace of development. Consumers seem more focused on access to services rather than the effectiveness of services. However, this may be changing as consumers see information on effectiveness of their treatment as their right, not a management option. Most importantly, payers have not made the implementation of a MFS a serious funding priority.

Moving in the right direction

The practice-based evidence approach as advocated in this paper promotes the systematic and frequent measurement of treatment progress and process in a continuous quality improvement framework.^35–36 This orientation is usually aligned with common factors, strengths-based and client–centered values.^37–38 In addition, there recently has been an increased emphasis on good measurement in what has been called evidence-based assessment. This approach to assessment not only requires good psychometric quality but emphasizes clinical usefulness.^39–40

The Substance Abuse and Mental Health Services Administration (SAMHSA) has had a major role in funding meetings, roundtables, and pilot research. With SAMHSA’s support, several states are implementing some form of a measurement system. For example, Ohio has spent more than a decade developing and implementing a measurement system that is now required state-wide. Massachusetts has also instituted measurement systems.¹⁴ Lambert and Burlingame, pioneers in the field of real-time outcome measurement, have collaborated with the state of Utah to implement a very brief concurrent measurement system in which data are entered on PDAs and are immediately available to the clinician.⁴¹ Since 2003 the MacArthur Foundation has also invested in research to develop a MFS with the cooperation of the state of Hawaii.⁴² MH services in the U.S. are not alone in promoting measurement. Great Britain has been in the forefront of routine measurement of adult and child and adolescent MH services, yet their child services is struggling to implement a voluntary measurement system.⁴³ Australia has been one of the first countries to support routine mental health outcome measurement, however implementation has been problematic.^44–45

CFIT, an example of a MFS

The Center for Evaluation and Program Improvement at Peabody College has developed an evidence-based, outcome-driven continuous quality improvement system called Contextualized Feedback Intervention and Training (CFIT). It can be used for continuing professional development and quality improvement. CFIT enables provider organizations to make data-based decisions and transform themselves into learning organizations.⁴⁶ CFIT is based on a theory of change, and grounded in psychological and organizational research.^47–48

CFIT has four major components: organizational assessment, treatment progress measurement, feedback, and training. CFIT is designed to affect and be affected by the culture of the organization. Each application of CFIT begins with an assessment of the organization's needs and readiness for change. This information can be used to tailor the implementation process of the system to the specific organizational context.⁴⁹ Although research on CFIT may lead to a better understanding of the contextual influences on implementation, the field has not yet developed scientifically based interventions designed to improve implementation and to sustain it in the long run. However, this is a great improvement over simply ignoring the important role of organizational context and personnel.⁵⁰

In order to make measurement concurrent with treatment feasible brief instruments were developed to collect weekly data on multiple domains from youth, caregiver, and clinician. The instrument battery assesses both processes (therapeutic alliance, treatment motivation, and session impact) and clinical outcomes (life satisfaction, hope, symptoms and functioning). It is available for free at http://peabody.vanderbilt.edu/ptpb.

The feedback can be used at all levels of the organization—by the clinician, supervisor, and administrator. The goal is to revolutionize the way these groups operate by providing them with shared information. The feedback is provided online in a user-friendly format. Individual scores can be compared to organization-specific norms. The data can also be aggregated to compare clinicians, clinics, provider organizations, and types of treatment. Such information can transform MH services from being viewed as a commodity to a service that is selected for its effectiveness.

CFIT’s clinical training is based on a common factors approach—those factors that are common across almost all therapies—rather than a specific therapeutic school.⁵¹ CFIT uses a combination of online modules, written manuals, teletraining, and on-site training sessions. The main objective is to provide clinicians and supervisors with information on how to optimize the therapeutic processes. Clinicians and supervisors can evaluate the success of their interventions, as well as refine interventions, through a continuing review of the feedback reports. With the support of a NIMH grant the effectiveness of CFIT to influence outcomes is currently being evaluated in a large scale field experiment.

Conclusions

Developing, implementing and sustaining a MFS is neither simple or a total solution for obtaining client improvement. Yes, it can be used with a broad range of clients, clinicians, treatments and contexts. Yes, it measures implementation and outcomes, and therefore problems are less likely to be hidden. However, all of the barriers noted earlier will affect its implementation and sustainability. Much more needs to be learned about how to successfully implement a MFS. While the importance of the organizational and cultural context of a MFS is recognized, it is not known under which settings a MFS will be feasible and effective. Measurement of client improvement is both conceptually and logistically complex. There is concern about how to measure change, what should be measured and the lack of agreement among respondents. It is unlikely that the level of commitment and funding will be ideal. It is likely that computer programs, which to the inexperienced seem simple, will remain immensely complicated and expensive to develop and test. All of these problems are likely to occur and there is not sufficient knowledge and technology to deal with them.²¹ However, the only way to learn about how to solve implementation problems is though continued implementation of MFSs.

Dalton Conley, a sociologist and winner of a $500,000 National Science Foundation award, is quoted as saying, “I would like to argue that sociology is among the hardest sciences of all — harder than the proverbial rocket science”.⁵² Establishing and sustaining a MFS is even more difficult. Real change in the real world is really hard.