New trends in assessing the outcomes of mental health interventions

Abstract

Assessing the outcomes of interventions in mental health care is both important and challenging. The aim of this paper is to advance the field of outcomes research by proposing a taxonomy of the decisions that clinicians and researchers need to consider when evaluating outcomes. Our taxonomy has eight components, framed as decisions: Whose outcome will be considered? Which scientific stage is being investigated? What outcome domain(s) matter? What level of assessment will be used? Will clinical and/or recovery outcomes be assessed? Whose perspective will be considered? Will deficits and/or strengths be the focus? Will invariant or individualized measures be preferred? We propose a future focus on understanding what matters most to people using mental health services, and on the use of measures rated by service users as the primary approach to evaluating outcome.

Assessing the outcomes of interventions in mental health care is both important and challenging. It is important because producing significant outcomes, i.e., health gains attributable to an intervention 1, is the main goal of mental health services. Other important attributes of an intervention, such as accessibility, acceptability, efficiency and cost-effectiveness, need only to be considered where the intervention produces significant outcomes. Assessing outcomes is also challenging, because choosing methods, outcome domains and outcome measures all involve the balancing of conceptual, ethical and clinical considerations 2,3.

The aim of this paper is to propose a taxonomy of the decisions that clinicians and researchers need to consider when evaluating outcomes. Our taxonomy has eight components, each of which involves making explicit underpinning assumptions. We therefore frame these components as decisions.

DECISION 1: WHOSE OUTCOME?

It might be thought that the outcome for the patient is of primary importance, but the needs of at least three other stakeholder groups also need to be considered.

First, the patient's informal carers – their friends and family – often have substantially more contact with the patient than mental health staff, which may have powerful consequences. A UK study estimated that 4.8% of carers had terminated employment and 15.5% took a mean of 12.5 days off work per year as a result of their carer role 4. Carers also provide emotional and practical support that otherwise would be required from mental health services – estimated for people with schizophrenia being looked after by family as involving 5.6 hours per day 5. Informal carers will have their own perspectives on valued outcomes for the patient and for themselves 6. Evaluating the impact of their caring role on their mental and physical health may be a cost-effective element of an evaluation strategy, and carer-focussed measures are available 7–9.

Second, the wellbeing of staff may be considered as an outcome, for two specific reasons. The clinical rationale is that there is now robust evidence 10 that “parallel processes” exist in mental health services – the experience of staff within the system influences how they work with people using services. If services are for example to promote hope and empowerment, then staff need to experience hope and empowerment in their work role. The economic rationale is that providing mental health services is expensive, and the primary cost driver is human resources. A workforce with low morale, high sickness rates and poor performance is an inefficient investment 11. For both these reasons, monitoring outcomes such as staff wellbeing and morale might be justified.

Finally, members of the public largely fund mental health systems in many countries, whether directly through health insurance or indirectly through taxation. Therefore the public have a legitimate interest in the return on their investment 12. Outcomes of interest to the public might include reductions in antisocial behaviour such as aggressive begging, or of “odd” behaviour such as shouting at voices.

For the remainder of this paper, we focus on outcomes for patients.

DECISION 2: WHICH SCIENTIFIC STAGE?

Developed as an analogue of the phases of pharmacological product development, recent formulations have considered phases of complex psychosocial interventions 13, or more generally the phases of the translational medicine continuum 14, shown in Figure 1.

Figure 1.

Phases of the translational medicine continuum

In this conceptualization, outcomes may vary according to the scientific stage of enquiry. At Phase 0 (scientific discovery) the key outcome may be the formulation of a new aetiological pathway or candidate risk factor for a disorder. At Phase 1 (early human trials) the key outcomes of interest are likely to be tolerability of the new intervention and dose-response. In Phase 2 (early clinical trials) the critical outcome issue is estimating the effect size of the intervention – in other words, is there an early indication that the intervention is effective, and how effective? At Phase 3 (late clinical trials) the key outcome is more specific – exactly how effective is the intervention among routine populations with the condition of interest? Finally, in Phase 4 (implementation) the outcome focus becomes more pragmatic, namely how far can Phase 2 and 3 benefits be replicated in routine clinical practice?

DECISION 3: WHAT OUTCOME DOMAIN?

An outcome domain is a conceptually distinct component of outcome. A systematic review identified seven categories of outcome domains: wellbeing, cognition/emotion, behaviour, physical health, interpersonal, societal and services 15. Choosing the outcome domain or domains to evaluate should be a separate and prior decision to choosing the outcome measure 16. In our experience, this distinction is often not maintained, with the more common starting point being identification of measures. Conflating the choice of outcome domain with choosing the outcome measure leads to three problems: inconsistency, unimportance and unfairness.

When evaluating outcome for a specific intervention, it remains common to under-specify the intended mechanisms of action and the causal pathway from intervention to outcome. This is despite the scientific consensus that evaluation should involve identification of the theory base for an intervention 13. The absence of a testable model means that the rationale for the choice of outcome domain cannot be stated, and so the choice is likely to reflect current clinical assumptions about “what matters”. There has been a general movement from using service data (e.g., readmission rates) towards clinical outcomes (e.g., symptomatology) and more recently towards health-related quality of life as clinical endpoints in outcomes research. However, the result is that the evidence base remains poor for interventions targeting some important outcome domains, such as hope and empowerment 17. Explicitly identifying, with a rationale, the choice of outcome domain will increase theoretical coherence between intervention and outcome.

When evaluating whole-system interventions such as service models or when introducing outcome assessment into routine clinical settings, consideration of outcome domains is also an important first step. Mental health systems need to meet many goals, including patient benefit, harm minimization, public protection, and value-for-money. The choice of outcome domains sends a clear message about the relative balance of these goals, and hence is an influence on organizational culture. It is one means by which an organization communicates what is important, in other words its “core business”.

Finally, explicitly identifying the outcome domain reduces the extent to which success is unfairly judged in relation to aspects of a patient's life which are outside the control of the mental health service. Social determinants of mental ill-health such as poverty and social inequality are well-established 18, and as services in general cannot influence wider social determinants, measures of health-related quality of life may be insufficiently sensitive. An alternative approach is to identify more proximal outcome domains, such as symptomatology or recovery support.

DECISION 4: WHAT LEVEL OF ASSESSMENT?

It is important to have clarity about the level of assessment, from the individual intra-psychic level (e.g., symptoms) through the inter-personal and immediate social environment (e.g., carers, social networks) to the broader environmental level (e.g., stigma). For example, in relation to interventions related to stigma and discrimination associated with mental illness, one can assess the outcomes of a national programme, such as the Time to Change campaign in England 19, using whole population surveys 20, or in terms of sub-populations such as journalists 21, or in terms of the outcomes rated by individual mental health services users 22, all of which can be seen as valid and indeed complementary outcome measures.

DECISION 5: CLINICAL OR RECOVERY OUTCOMES?

Outcome assessment internationally remains primarily focussed on traditional clinical outcomes such as symptomatology, social disability and service use (e.g., admission rates). The four most commonly used measures assess social disability (Health of the Nation Outcome Scale, HONOS 23), symptoms (Clinical Outcomes in Routine Evaluation – Outcome Measure, CORE-OM 24; Outcome Questionnaire-45, OQ-45 25), and needs (Camberwell Assessment of Need, CAN 26). These are mandated for national or large regional use in Australia 27, Canada 28, England 29, Netherlands 30 and New Zealand 1. These measures have in common that they assess clinical outcomes.

Internationally there is an emerging consensus that services should be recovery-oriented 31. Recovery has been defined as “a deeply personal, unique process of changing one's attitudes, values, feelings, goals, skills, and/or roles” and “a way of living a satisfying, hopeful, and contributing life even within the limitations caused by illness” 32. International best practice is emerging 33, and it is becoming clear that organizational transformation is needed to develop a recovery orientation 34. Some dimensions of transformation include a greater emphasis on the biomedical ethical imperative of promoting autonomy 35, a changed workforce 36, a greater emphasis on patient choice, and, most relevantly, different goals of mental health care. The challenge is summarized by Repper and Perkins 37: “Traditional yardsticks of success – the alleviation of symptoms and discharge from services – are replaced by questions about whether people are able to do the things that give their lives meaning and purpose, irrespective of whether their problems continue and whether or not they continue to need help and support”. The challenge is to measure recovery as an outcome in a way which is both aggregatable and meaningful.

How might this be done? A systematic review of recovery frameworks identified five key recovery processes: connectedness (social inclusion, community integration), hope and optimism, development of a positive identity, meaningfulness in life, and empowerment – the CHIME Framework 38. If the goal of a mental health system is to promote recovery, then these recovery outcomes are the appropriate domains to target. New measures are becoming available 39.

One proposal is that outcome assessment should measure valued social roles which reinforce social identity, and individual goals which contribute to personal identity 40.

Valued social roles include employee, partner, family member, friend, citizen, free (i.e. non-detained) person, etc. Their value is relatively invariant – most (but of course not all) people want a job, a relationship, contact with their family, some close friends, the ability to exercise citizenship rights such as voting, not to be held in hospital or prison, etc. Assessment tends to be quantitative and dichotomous (or at least on an ordinal scale, such as unemployed – voluntary work – part-time work – full-time work), and hence easy to aggregate with little loss of meaning. The primary advantage of these outcome measures is that they are based on normal social values, and so avoid illness-related lowering of expectations either by staff, in an effort to be realistic, or by patients with internalized stigmatizing beliefs about what they can expect in life 41. Since most valued social roles occur outside the mental health system, they orientate the actions of the service towards increasing integration and participation by the person into his/her social environment, rather than encouraging a decontextualized and service-focussed view of the person. Their primary disadvantage is their invariance – some people get along very well in life without friends, or a partner, or a job.

Individual goals differ from person to person. No standardized measure will have items such as “swim with dolphins” or any of the other idiosyncratic goals individuals set and attain on their recovery journey. Any attempt to squeeze personal identity into predefined boxes can be justifiably criticized for its loss of meaning. This does not, of course, mean that personal goals should not be included in outcome evaluation – they remain central, despite the difficulties in assessing individual goal attainment. Rather, as McNamara 42 put it, “the challenge is to make the important measurable, not the measurable important”. So, an overall outcome evaluation strategy might measure two things. First, objective quality of life indicators, such as adequacy of housing, friendship, safety, employment and close relationships. Second, progress towards personal goals.

DECISION 6: WHOSE PERSPECTIVE?

Assuming that the outcome for the patient is the main focus, the question remains of whose perspective is used. Two perspectives have primarily been used to evaluate outcome.

First, and in our view most central, is the patient perspective. An emerging distinction in relation to patient-rated measures is between assessment oriented towards the experience of using mental health services and systems – patient rated experience measures (PREMs) – and assessment capturing direct health gain – patient rated outcome measures (PROMs), especially using patient-generated PROMs (PG-PROMs) 43. A range of PROMs exist, spanning both clinical and recovery outcomes 44–47. The development of PREMs is earlier stage, and has primarily focussed on satisfaction and experience of care. The main limitation of PREMs is that they may reduce the focus on a “life beyond illness”. People who use mental health services long-term can live in a “virtual institution”, in which key aspects of identity (social network, sense of self, housing, etc.) are all indexed on the mental illness 48. PREMs such as satisfaction are a normative judgment influenced by the person's reference group, so in people using mental health services positive ratings may be obtained because of an atypical reference group. This vulnerability of PREMs to being rated positively because of lowered expectations means that mental health systems should as far as possible evaluate success using outcome rather than experience measures.

Second, and perhaps the traditional focus in mental health systems, is the perspective of the clinician 49–51. Staff-rated measures exist for most outcome domains. This perspective has been called the “objective” assessment and the patient rating called the “subjective” assessment, but in fact staff assessments are themselves prone to bias due for example to professional training 52, and some studies have found patient rather than staff assessments to be more reliable 53. The reality is that both staff and patient perspectives are influenced by a range of factors, and both provide useful and complementary information on outcome. The relative balance given to the two perspectives should be based on scientific, ethical, professional and pragmatic considerations.

We now consider outcome assessment from the patient's perspective.

DECISION 7: DEFICITS OR STRENGTHS?

The World Health Organization (WHO) declares that health is “a state of complete physical, mental and social well-being and not merely the absence of disease or infirmity” 54. However, creating health-oriented rather than illness-oriented services has proved rather more difficult than the clarity of this declaration would suggest. In relation to outcome, the substantial majority of measures used in research and practice are focussed on mental illness – they assess amelioration of undesirable experiences such as symptoms or cognitive problems, reduction in risk factors such as stress, or attainment of an adequate level of functioning. Very few assess mental health, such as the use of strengths 55, the development of protective factors such as resilience, or the attainment of positive wellbeing 56.

Some argue that mental health is a distinct construct. The Complete State Model of Mental Health posits that mental health and mental illness lie on orthogonal spectrums 57. People with mental illness range from those who are “floundering” (when mental health is absent), through those experiencing moderate mental health, to those who are “struggling” (when mental health is present) as they work towards “flourishing” (high mental health, low mental illness). Epidemiological studies of adults (n=3,032) 57 and adolescents (n=1,234) 58 confirm that mental health and mental illness according to these definitions co-exist in the general population.

An alternative view is that mental health is better understood as lying on a single spectrum with positive mental health at one end and negative mental health at the other. Measures based on this approach have been developed, such as the Subjective Happiness Scale 59, which includes items such as “Compared to most of my peers, I consider myself…”, with Likert ratings from 1 (less happy) to 7 (more happy). Some wellbeing measures include only positively worded items, which are compatible with both understandings of mental health. Examples include the WHO-5 Well-Being Index (e.g., “I have felt cheerful and in good spirits”) 60 and the Warwick-Edinburgh Mental Well-Being Scale (WEMWBS) (e.g., “I've been feeling useful”) 61.

More generally, Vaillant identifies six models of mental health 62. The first model, being “above normal”, relates to superior functioning in a wide range of activities, such that life's problems never get out of hand. The goal of the second model, positive psychology, is intervention to maximize positive qualities, such as self-efficacy. A recent systematic review has identified indicators of wellbeing in psychosis 63, which are now being used to inform a new intervention based on positive psychology principles 64. The third model involves maturity, shown by attainment of developmental tasks such as identity, intimacy, generativity and integrity 65. The fourth model is emotional or social intelligence, i.e., the ability to read other people's emotions. Subjective wellbeing, i.e., the experience of positive mental health, is the fifth model, and the last model is resilience, which is linked to the adaptive value of coping mechanisms.

DECISION 8: INVARIANT VERSUS INDIVIDUALIZED?

Normal practice in outcome evaluation is to use standardized measures, for which key psychometric criteria have been established as adequate. More recent attention has enlarged the focus from the usual reliability and validity concerns to also consider feasibility and clinical relevance 66. However, standardized measures have the feature of invariance – the same outcome domain is assessed for each patient. The advantage of this approach is that it allows statements about the impact of an intervention or service on a specific outcome domain, such as symptomatology. The emerging important disadvantage, however, is that the outcome domain may or may not be important to the patient.

We learn from the reports of people who use services that recovery is very individual, varying greatly from person to person 67. As well as symptomatic or functional improvement, the tipping point towards starting to develop an identity as a person in recovery can be developing a supportive relationship with a mental health worker who treated them as a person not a patient 68, or non-clinical changes such as spiritual growth 69. This variation highlights the need for caution about viewing improvement in any single domain as universally important, and the outcome evaluation challenge of capturing individual importance using standardized assessments.

One technology that can be used to personalize evaluation is goal attainment scaling (GAS) 70. This approach involves patients prospectively identifying a personally-important goal and associated progress indicators on a typically five-point scale, using these indicators to assess progress at outcome evaluation, and then standardizing the results to allow aggregation. GAS has been used to identify and then evaluate a valued outcome in randomized controlled trials, primarily in rehabilitation medicine 71 and with older adults 72. Two systematic reviews have investigated this use of GAS. In relation to pharmacy practice, the conclusion was that GAS demonstrated high reliability, variable validity, excellent responsiveness, and was a useful methodology for evaluating effectiveness 73. In relation to physical rehabilitation, GAS was described as a sound measure, with reliability and sensitivity needing further investigation 74. Concern has been raised about sensitivity to subtle changes, responsiveness, inter-rater reliability, validity (content and construct), scaling non-linearity and lack of uni-dimensionality 75. For example, agreement on progress between a patient's therapist and an independent assessor is low 76. To these concerns, we would add that administration burden can be high, and that the GAS score 77 is not intuitive to interpret.

A new approach to address some of these issues is called the Personal Primary Outcome (PPO) list. Designed for use in randomized controlled trials and other evaluations, the PPO list comprises several outcome domains, each of which is (invisibly) linked to a relevant standardized outcome measure. At baseline, the patient chooses the outcome domain that is most closely linked to his/her goal in using mental health services, and then he/she completes the associated measure. The measure is re-administered at follow-up. The PPO list approach is currently being evaluated as a methodology for trials 78.

A second approach is to develop a standardized measure where items are selected according to patient preference. An example is the INSPIRE measure (downloadable at http://www.researchintorecovery.com/inspire) of recovery support, where for each item about support from a mental health worker, respondents are first asked if the item matters to them, and only if it does are they asked to rate support from the worker 47. The INSPIRE score therefore reflects the respondent's preferences, yet produces a quantitative score which can be used for monitoring change over time or can be aggregated with the scores of others.

CONCLUSION: WHAT OUTCOMES REALLY MATTER FOR SERVICE USERS?

Perhaps the most important insight developed in the last decade is that it is the point of view of the patient or service user that is the most important in deciding which outcomes to assess, and in making the actual outcome ratings. We know, for example, that quality of life is not closely related to users' needs as rated by staff of mental health services, but is closely associated with unmet needs as rated by service users 79,80. It follows that the emerging literature reporting service user views on measures 44,81 and developing new measures 82 is of paramount importance. New measures, such as the Recovery Star 83, can be independently evaluated 84 and incorporated into clinical practice 85. An additional advantage of making service user rated outcomes a principal focus is that it side-steps the issue that has bedevilled services in recent years, namely how to incentivize staff to make frequent, complete and valid outcome ratings on a long-term sustainable basis.

If we were unwisely to try to predict the central issues in mental health outcome measurement over the next decade, then we propose a relentless attention to the detail of what matters most to service users, as rated by service users.