A Framework for Measurement Feedback to Improve Decision-Making in Mental Health

Abstract

The authors present a multi-level framework for conceptualizing and designing measurement systems to improve decision-making in the treatment and prevention of child and adolescent mental health problems as well as the promotion of well-being. Also included is a description of the recommended drivers of the development and refinement of these measurement systems and the importance of the architecture upon which these measurement systems are built. The authors conclude with a set of recommendations for the next steps for the field.

It has been documented across a wide array of domains that knowledge of results is a critical ingredient in facilitating change (e.g., Kluger and Denisi 1996). However, for knowledge of results to be fed back to change agents (i.e., practitioners and policymakers), a rigorous, reliable, and valid measurement system must be in place and routinely utilized.

Recently, Bickman (2008) made a compelling argument for a measurement feedback system (MFS) for individual child and adolescent mental health practitioners and underscored the barriers to large-scale adoption. In this paper, we build on Bickman’s work by describing a framework for the different levels for which measurement systems are needed, the features or characteristics that should drive the development or refinement of such systems, and the importance of the architecture upon which these measurement systems are built. Throughout, we make recommendations for the next steps that will be needed before we are able to make significant progress in the accomplishment of these goals.

A Framework for Measurement Feedback Systems

For this framework, we are not simply interested in a feedback system for the treatment of psychopathology, but instead with a more comprehensive framework that is concerned with promotion of well-being and prevention of pathology, as well as treatment. This broader perspective is supported by recent advances in prevention and the recognition of the importance of promotion of competencies (e.g., National Research Council Institute of Medicine 2009).

While the client is the fundamental unit for whom we hope to move toward increased well-being and competency development (and away from pathology), these clients are, by definition, nested within larger service units. These service units, which include provider-client micro-settings and service agency/organizations, are also embedded within larger systems of state policies and regulations. A comprehensive health promotion, illness prevention, and treatment system needs to develop reliable and valid measurement instruments that are cost-effective and that can be implemented within and across each of these levels. Moreover, the assessment of treatment outcome alone is not sufficient. To maximize the utility of feedback to improve outcomes at multiple levels of the mental health system, sensitive, reliable, and valid measures of mediating processes and practices are also necessary. This framework is illustrated in the following table.

 

Target of measurement	Nature of metric
	Outcomes	Mediating processes and practices
Client-provider	(1)	(2)
Agency/organization	(3)	(4)
Policy-level	(5)	(6)

Decision-Making Needs at the Client-Provider Level

There are many potential outcomes to assess at the client-provider level, including symptom severity/diagnoses, functioning, consumer perspectives (e.g., satisfaction, quality of life), environments (e.g., social supports, family, neighborhood stability) and systems (e.g., service use, costs, etc.) (Hoagwood et al. 1996). To date, there has been extensive development of clinical assessment instruments to measure symptom severity, and to a lesser extent functional outcomes of clients (cell #1), with less attention to other outcome domains (Jensen et al. 1996; Kazdin 2000). Specifically, there are several instruments that measure child and adolescent psychopathology, many of which focus on making categorical decisions, such as general adaptive functioning, or specific diagnostic categories. There are also continuous measures of symptoms, though they are less often utilized in practice settings. Both types of instruments vary in length, ease, and cost of administration, as well as in the evidence of their reliability and validity, and cross-cultural sensitivity. However, even the best of these instruments rarely report evidence on short-term change. This evidence is critical if the instruments are to generate useful feedback to practitioners, thereby facilitating changes in service delivery. The mental health field lacks a comprehensive review of outcome measures, especially as evaluated with regard to the proceeding criteria. Only with this information can these measures be utilized and implemented as part of a comprehensive measurement feedback system. As a result, our first recommendation is a call for a status report on the quality and utility of available child, adolescent and family clinical outcome measures for large-scale implementation as part of a measurement feedback system for mental health and child welfare services.

There has been far less development of reliable, valid, user-friendly, and cost-effective measures of child and adolescent competencies for use as part of a feedback system. Reductions in psychopathology alone do not represent well-being and health. There is a need for further development and implementation of measures of competencies. Thus, our second recommendation is a call for further development and validation of child and adolescent competency instruments for large-scale implementation as part of a measurement feedback system for mental health and promotion services.

Feedback of client change data to the service provider is more useful in facilitating further change when it is linked with data on change in the mediating processes and practices (see cell #2). There has been minimal research addressing valid and feasible measurement of treatment processes in mental health care. Recently however, Chorpita and Dalieden (2009) have creatively employed analytic procedures to articulate specific treatment procedures associated with positive outcomes in the context of randomized clinical trials. The development of measures for the provision of specific treatment practices and other potential mediating processes is essential to a complete model of feedback for clinicians. With the development of such measures, outcome-based feedback has the potential to become instrumental in effecting real-time improvements in treatment (not just informative about clinical conditions). For example, when given feedback about negative outcomes, the clinician also needs additional information about discrepancies between the current and the expected course of action in order to make decisions about constructive change in treatment strategies (i.e., receiving feedback about poor outcomes versus receiving feedback about why treatment is not progressing well). Thus, when feedback on outcomes is paired with feedback on practices, particularly when those observed practices can be compared with practices expected based on the clinical treatment outcome literature, there is a much greater chance of improving the clinical decision-making process. The third recommendation is to further develop, improve, and validate the measures of practices, processes, and events that are a part of both routine care and evidence-based practices.

Decision-Making Needs at the Agency/Organization Level

When we begin to think about selective or universal approaches to intervention, as opposed to treatment (National Research Council Institute of Medicine 2009), we often employ universal and selective interventions at higher and larger levels of social organization, such as mental health, child welfare agencies, and school settings. Here, the focus of intervention is often on the practices and mediating processes at the organizational level (see cell #4) that are linked to aggregate-level individual outcomes (cell #3).

In 1976, the Mental Health Statistical Improvement Program (MHSIP) was the first initiative to suggest reporting standards based on the organization as the principle reporting unit (Leginski et al. 1989). In 1989, a MHSIP report was issued that recommended a decision support model, including standard data elements and minimum data sets, for data related to clients, clinical events, human resources, and financial and organizational performance indicators. By these standards, the clinical event was viewed as the basic unit of analysis (NIMH 1983). However, neither of these initiatives addressed broader organizational performance indicators, which are important. Borrowing from the organizational behavior literature, Glisson (2002) has demonstrated the salience of organizational norms such as culture and climate in mental health and child welfare agencies. He reported impressive relationships of organizational climate and culture to service providers’ job satisfaction and attrition, as well as to their clients’ mental health outcomes. This area requires increased theoretical attention and measurement development if child-serving mental health, welfare, and educational organizations are to use feedback to improve their decision-making and become more effective at the promotion of mental health and the treatment and prevention of mental disorders. The fourth recommendation is for further conceptualization and development of the measurement of organizational-level indicators of mediating processes that are related to aggregate-level individual outcomes. And in a related vein, there is a need to conduct a status report on the conceptualization and development of the measurement of organizational-level mediating processes.

Decision-Making Needs at the Policy-Level

Decision-making at the local, state, or federal levels about mental health policy often build on the measures developed and employed at the two lower levels, though in aggregate form (cell #5). For policymakers to improve their decision-making, they need to know how policies (cell #6) are associated with aggregate-level agency outcomes (cell #5). For example, data should be available to help policymakers establish incentives for better agency-level treatment outcomes or decide whether to invest in treatment rather than prevention services. This is an area that has received very little attention, either in the conceptual or measurement arenas. Thus, our fifth recommendation is a call for increased resources and attention devoted to conceptualizing and measuring the effectiveness of policy variations as these variations relate to changes in aggregate-level mental health outcomes.

Information Technology Architecture Versus Measurement Feedback Systems

At each of these three levels (client-provider, organization, system-wide policy), it is conceivable that a variety of measurement instruments could be employed, assuming that each instrument used has demonstrated reliability, validity, ease of usage, cost-effectiveness, and sensitivity to change. However, we emphasize that consideration about the information technology architecture of a feedback system, which includes data capture, storage, reporting, security, and privileging considerations, is an independent issue from instrumentation. It is likely that the particular instruments chosen for a measurement and feedback system will continue to evolve over time as psychometric and basic research continue to produce increasingly sophisticated measurement tools. The design of data architecture to support a measurement system must address such issues as how data can be aggregated and configured for feedback reports or displays, and how those reports are built to inform key decisions in common clinical and administrative functions, so that the feedback can be efficient and timely (Chorpita et al. 2008). Data system design should emphasize flexible organization of reporting information (e.g., including multiple forms of continuous data and discrete events in a single time series, aggregating and disaggregating reports across organizational units or staff roles), role-based privileges (e.g., giving supervisors, therapists, and clerical staff different views of the data), and an open platform for evolving metrics (e.g., allowing measures to be replaced or edited over time). This need is timely given current national attention to universal implementation of electronic health records. The sixth recommendation is to call for increased consideration and development of design principles relevant to the data architecture of measurement feedback systems that offer the greatest reporting flexibility and ease of use.

Summary

We have offered a multi-level framework and a set of recommendations on how to conceptualize, develop, and improve a feedback system that provides relevant and timely information about practices, processes, and events, as well as with the associated outcomes. When deployed for clinicians, organizations, and system-wide policymakers, this multi-level feedback system can be instrumental in improving clinical and organizational decision-making. The recommendations are:

1. Conduct a status report on the quality and utility of available child and adolescent psychopathology categorical and continuous instruments for large-scale implementation as part of a measurement feedback system of mental health and child welfare services.

2. Further develop and validate child and adolescent competency instruments for large-scale implementation as part of a measurement feedback system of mental health and promotion services.

3. Further develop, improve, and validate the measurement of practices, events, and mediating processes at the service deliverer-client level.

4. Further conceptualize and develop the measurement of organizational-level practices and processes, and conduct a status assessment on the conceptualization and development of indicators for organizational-level practices and processes.

5. Devote increased attention and resources to conceptualizing and measuring the effectiveness of policy variations and changes on mental health outcomes in the aggregate.

6. Develop a data architecture model for these different levels of measurement feedback systems that offers the greatest flexibility in feedback reporting and ease of use.