Outcome measures and needs assessment tools for schizophrenia and related disorders

Abstract

Background

There has been a recent trend to encourage routine outcome measurement and needs assessment as an aid to decision making in clinical practice and patient care. Standardised instruments have been developed which measure clinical symptoms of disorders such as schizophrenia, wider health related quality of life and patients' needs. Such measures might usefully be applied to aid the recognition of psychosocial problems and to monitor the course of patients' progress over time in terms of disease severity and associated deficits in health related quality of life. They might also be used to help clinicians to make decisions about treatment and to assess subsequent therapeutic impact. Such an approach is not, however, without cost and the actual benefit of the adoption of routine outcome and needs assessment in the day‐to‐day care of those with schizophrenia remains unclear.

Objectives

To establish the value of the routine administration of outcome measures and needs assessment tools and the feedback they provide in improving the management and outcome of patients with schizophrenia and related disorders.

Search methods

The reviewers undertook electronic searches of the British Nursing Index (1994 to Sept 1999), the Cochrane Library (Issue 2, 2002), the Cochrane Schizophrenia Group Trials Register (2002), EMBASE (1980‐2002), MEDLINE (1966‐2002), and PsycLIT (1887‐2002), together with hand searches of key journals. References of all identified studies were searched for further trials, and the reviewers contacted authors of trials.

Selection criteria

Randomised controlled trials comparing the feedback of routine standardised outcome measurement and needs assessment, to routine care for those with schizophrenia.

Data collection and analysis

Reviewers evaluated data independently. Studies which randomised clinicians or clinical teams (rather than individual patients) were considered to be the most robust. However only those which took account of potential clustering effects were considered further. Where possible and appropriate, risk ratios (RR) and their 95% confidence intervals (CI) were calculated. For continuous data Weighted Mean Differences (WMD) were calculated. Data were inspected for heterogeneity.

Main results

No randomised data were found which addressed the specified objectives. One unpublished and one ongoing trial was identified.

Authors' conclusions

The routine use of outcomes measures and needs assessment tools is, as yet, unsupported by high quality evidence of clinical and cost effectiveness. Clinicians, patients and policy makers alike may wish to see randomised evidence before this strategy is routinely adopted.

Plain language summary

Outcome measures and needs assessment tools for schizophrenia and related disorders

The routine use of outcome measures such as scales assessing the mental state is thought to improve decision making and patient care in the treatment of people with schizophrenia. This review, however, did not find a single randomised controlled trial which examined this strategy.

Background

1. Routine outcome measurement and needs assessment 
 The measurement of outcomes has risen in prominence over the past 30 years (Donabedian 1966, Lohr 1988, Ellwood 1988). Standardised instruments have been developed which measure clinical aspects of illness, and more recently 'patient based measures' have been developed. In schizophrenia, standardised instruments traditionally define disease severity and change in clinical status by counting the number and severity of symptoms and signs ‐ such as delusions and hallucinations ('symptom based measures' ‐ e.g. Overall 1962). Patient based measures, however, assess the impact of illness on the individual (Jenkinson 1994). These are often referred to as health status, health related quality of life (HRQoL) or functional status measures (Bowling 1997). They measure more than just clinical symptoms, since they incorporate some combination of the following domains; physical health, mental health, social functioning, role functioning, general perceptions of health and well‐being, cognitive capacity and patient satisfaction.

Outcomes measures have come to be used for a number of purposes, which include: 1. the evaluation of the clinical and cost‐effectiveness of interventions in experimental situations such as trials, 2. the monitoring of population health, 3. clinical audit and 4. facilitating clinical decision making in routine practice and patient care (Faden 1992, Fitzpatrick 1992, Fitzpatrick 1994, Ware 1995). It is the last of these uses in schizophrenia and related disorders that will be considered in this review.

A related development has been the introduction of formal 'needs assessment' tools in the care of those with severe and enduring mental illnesses, such as schizophrenia. Such needs assessment tools are intended to define health and social needs at both a population level and, ideally, at an individual level (Thornicroft 1992), so that healthcare provision might be more rational, responsive and 'appropriate' (Stevens 1998, Wright 1998). Examples of individual patient needs assessment tools for use in severe mental illness include the Camberwell Assessment of Need (CAN) (Phelan 1995) and the MRC Needs for Care Assessment (Brewin 1993).

Both the role of outcomes measurement instruments and needs assessment tools in the day‐ to‐day care of those with schizophrenia are considered in this review. A separate review is being undertaken to assess the value of routine outcomes measurement in those with disorders such as depression and anxiety (Gilbody 2000a). This review is expected to be published in December 2002.

2. The potential benefits of routine outcome measurement 
 When used as aids to decision making in routine care, outcome measures and needs assessment tools are thought to be useful in improving patient care in a number of ways. Firstly, they may identify problems which might not otherwise be recognised by clinicians or those responsible for care. For example, clinicians are often unaware of a substantial proportion of a patient's social and psychological problems (Sprangers 1992), and the identification of these problems might trigger an appropriate response and improve the overall quality of patient care. Secondly, they function as mechanisms for monitoring the course of patients' progress over time enabling informed decisions about treatment and assessments of subsequent therapeutic impact to be made. Thirdly, surveys have suggested that clinicians find these data useful in formulating a more comprehensive assessment of the patient (Young 1987, Kazis 1990). Finally, patients often welcome the opportunity to provide clinicians with information regarding their health status, particularly when they perceive that this information is not otherwise comprehensively assessed, thus aiding effective patient‐doctor communication (Nelson 1990).

Routine outcome measurement has been advocated as an adjunct to patient care within psychiatric services (Marks 1998), where measures of psychiatric symptomatology might be applied in order to measure therapeutic response and to inform management decisions. In addition, broader measures of health related quality of life might also be usefully adopted. In the case of schizophrenia, impairments in quality of life and health status are often unrelated to the number or severity of symptoms (such as delusions and hallucinatory experiences) (Anthony 1995, Becker 1993). This is especially important, since it is the level of symptomatology which forms the major focus of clinical consultations and practice, and is the major criterion by which the success or failure of treatment is judged in both practice and research (Revicki 1994). Consequently, clinicians' perceptions of these wider problems are often poor and it has been empirically demonstrated that clinicians underestimate the health status or health related quality of life of patients when patient and clinician ratings are compared (Sainfort 1996, Becker 1993, Lehman 1983a, Lehman 1983b). The use of more comprehensive outcome measures, which capture both symptoms and wider health‐related quality of life, might therefore be useful in identifying needs, monitoring clinical response and making clinical decisions in those with severe mental illness.

The adoption of routine outcome measurement has also become central to government policy formulations. For example, in the UK, there have been a number of initiatives in recent years aimed at the introduction of outcomes measurement tools into routine mental health practice, as part of a government health strategy to 'improve significantly the health and social function of mentally ill people' (DoH 1991). In response, the Health of the Nation Outcome Scale (HoNOS) has been developed with a number of uses in mind, including the assessment of local service requirements and psychiatric morbidity at a population level (Wing 1994). However, a key aim of the developers of the HoNOS is that it should be useful to clinicians in actual individual care planning, since without this feature it would not be widely used and the data which would ultimately inform decisions made at a population level would not be collected (Stein 1999).

3. Possible disadvantages of routine outcome measurement 
 The routine measurement of outcome has not been without its critics (e.g. Crombie 1997), and concerns have been raised that outcomes measures are un‐interpretable, unwieldy and a bureaucratic hindrance to successful patient care.

One way in which the success or usefulness of outcome measures in everyday routine care might be judged is by evaluating of the degree to which their adoption improves the outcome and quality of care. The results of research in other specialities has generally not been positive in this respect (Kazis 1990), nor has the use of these measures been shown to improve the management of common psychiatric disorders in non‐psychiatric settings (Gilbody 2001).

The measurement of outcome in the context of individual patient care is not without cost. Instruments must be developed, administered, coded, stored and retrieved ‐ all of which have resource implication in terms of time, direct cost and opportunity cost. There is also a danger that outcome measurement triggers resource intensive interventions which are of no proven benefit to patients, and which might actually harm them. Perhaps, more subtly, there is also a danger that the uptake of outcome measurement in this context represents a marketing ploy, in which measurement is used to demonstrate an institution's 'customer orientation', but which does not inform the provision of care (Fitzpatrick 1994).

Because the case for the benefit of routine outcome measurement is not clear‐cut, we decided to systematically review the best available evidence on the value of routine outcome and needs assessment in the day‐to‐day care of those with schizophrenia and related disorders.

Objectives

To review the effects of feedback of the results of outcome measures and needs assessment tools to clinicians/clinical teams on the management and outcome of patients with schizophrenia and related disorders.

Methods

Criteria for considering studies for this review

Types of studies

Randomised controlled trials.

Types of participants

People with schizophrenia or related disorders, however defined. Studies relating primarily to children or adolescents, people whose primary problem was one of substance abuse or who were managed in specialist substance abuse services, and those with learning disabilities or dementia were excluded.

Types of interventions

1. Outcome or needs assessment 
 These interventions involved information from routine outcome/needs assessment being fed back to the clinician/clinical team or incorporated into routine care procedures and rituals (such as outpatient assessment, hospital admission or routine discharge planning). Outcome or need could be assessed in both intervention and control conditions, but the information had to be fed back to the clinician or clinical team.

Potentially relevant assessment instruments that can be actively incorporated into routine care include: 
 1.1 Standardised measures of psychiatric symptomatology (e.g. measures such as the Brief Psychiatric Rating Scale (Overall 1962).

1.2 Standardised measures of health‐related quality of life (HRQoL) or Health Status such as the generic Short Form 36 (Ware 1992), or those especially designed to be applied in schizophrenia and related disorders (e.g. Lehman's Quality of Life Scale, Lehman 1983b). The latter generally include some combination of physical functioning, social functioning, role functioning, mental well‐being, cognitive capacity, general health perceptions and subjective well‐being (Ware 1987, Ware 1995).

1.3 Standardised assessments of patient need (e.g. instruments such as the Camberwell Assessment of Need ‐ Phelan 1995). Some measures combine various components of the above (e.g. the Health of the Nation Outcome Scale ‐ Wing 1994) and were included if they were explicitly identified as measures of 'need' or 'outcome'.

2. Routine care 
 This was defined as routine care without the feedback of outcome and needs assessment tools to clinicians/clinical teams.

Types of outcome measures

Trial endpoint was studied as defined/measured by the authors of the study, with particular attention to the impact of routinely administered outcome/needs assessment tools on the following:

All outcomes were reported for the short term (up to 6 weeks), medium term (7‐26 weeks) and long term (over 26 weeks).

Additionally, self‐reports of the use of outcome information in changing patient management were sought.

It should be noted that several of these trial endpoints may have included the measurement instrument which was the focus of the evaluation i.e. that which was actively incorporated into routine care by being fed back to the clinician.

Primary outcomes

1. Clinical symptoms (improved/not improved, average score, change)

Secondary outcomes

1. Health related quality of life/health status/patient need/global functioning (improved/not improved, average score, change)

2. Satisfaction 
 2.1 Patient with care 
 2.2 Patient perceptions of the usefulness or acceptability of measurement instruments 
 2.3 Clinician perceptions of the usefulness or acceptability of measurement instruments

3. Starting or changing drug treatment

4. Psychosocial intervention (e.g. family intervention)

5. Service use (both psychiatric and non psychiatric, referral to outside agencies such as housing or social worker)

6. Hospital status, (discharge, readmission or length of stay)

7. Cost (direct and indirect).

Search methods for identification of studies

Electronic searches

1. We used the Cochrane Schizophrenia Group's Collaborative Review Group optimal RCT search strategy (see Group Module) on the following databases:

1.1 British Nursing Index/RCN (1994 to Sept 1999, searched Sept 1999 SilverPlatter) with the following strategy: 
 #1 health status 
 #2 status indicator* 
 #3 (outcome* or process*) near3 assessment* 
 #4 health outcome* 
 #5 quality of life 
 #6 outcome* measure* 
 #7 assess* 
 #8 score* or scoring 
 #9 index 
 #10 indices 
 #11 scale* 
 #12 monitor* 
 #13 #7 or #8 or #9 or #10 or #11 or #12 
 #14 outcome* 
 #15 #14 near3 #13 
 #16 #1 or #2 or #3 or #4 or #5 or #6 
 #17 #16 or #15

1.2 Cochrane Controlled Trials Register CD‐ROM (2002, issue 2) with the following strategy: 
 #1 HEALTH‐STATUS‐INDICATORS:ME 
 #2 OUTCOME‐AND‐PROCESS‐ASSESSMENT‐HEALTH‐CARE:ME 
 #3 OUTCOME‐ASSESSMENT‐HEALTH‐CARE:ME 
 #4 QUALITY‐OF‐LIFE:ME 
 #5 OUTCOME:TI AND MEASURE*:TI 
 #6 OUTCOME:AB AND MEASURE*:AB 
 #7 HEALTH:TI AND OUTCOME*:TI 
 #8 HEALTH:AB AND OUTCOME*:AB 
 #9 QUALITY:TI NEAR LIFE:TI 
 #10 QUALITY:AB NEAR LIFE:AB 
 #11 MEASURE:TI OR MEASURE:AB 
 #12 ASSESS*:TI OR ASSESS*:AB 
 #13 SCORE*:TI OR SCORING:TI OR SCORE*:AB OR SCORING:AB 
 #14 INDEX:TI OR INDEX:AB 
 #15 INDICES:TI OR INDICES:AB 
 #16 SCALE*:TI OR SCALE*AB 
 #17 MONITOR*:TI OR MONITOR*:AB 
 #18 #11 OR #13 OR #14 OR #15 OR #16 OR #17 
 #19 OUTCOME*:TI OR OUTCOME*:AB 
 #20 #19 AND #18 
 #21 #1 OR #2 OR #3 OR #4 OR #5 OR #6 OR #7 OR #8 OR #9 OR #10 
 #22 #21 OR #20

1.3 The Cochrane Schizophrenia Group Trials Register (Jan 2002) with the following strategy: 
 ["health status" or "health outcome*" or "status indicator*" or "quality of life" or "outcome measure*" or "outcomes measure*" or "outcome assessment*" or "outcomes assessment*" or "outcome score*" or "outcomes score*" or "outcome scoring" or "outcomes scoring" or "outcome index" or "outcomes index" or "outcome indices" or "outcomes indices" or "outcome scale*" or "outcomes scale*" or "outcome monitor*" or "outcomes monitor*" or "treatment outcome*" or treatment‐outcomes or outcome‐and‐process‐assessment or outcome‐assessment or quality‐of‐life or health‐survey or outcomes‐research or health‐status‐indicators or "psychological‐assessment" or "needs assessment*"]

1.4 CINAHL (1982 to Feb 2002, searched May 2002, SilverPlatter) with the following strategy: 
 #1 explode "Health‐Status"/ all topical subheadings / all age subheadings 
 #2 explode "Health‐Status‐Indicators"/ all topical subheadings / all age subheadings 
 #3 explode "Outcome‐Assessment"/ all topical subheadings / all age subheadings 
 #4 "Outcomes‐(Health‐Care)"/ all topical subheadings / all age subheadings 
 #5 explode "Quality‐of‐Life"/ all topical subheadings / all age subheadings 
 #6 health outcome* in ti,ab 
 #7 quality of life in ti,ab 
 #8 outcome measure* in ti,ab 
 #9 measure* in ti,ab 
 #10 assess* in ti,ab 
 #11 (score* or scoring) in ti,ab 
 #12 index in ti,ab 
 #13 indices in ti,ab 
 #14 scale* in ti,ab 
 #15 monitor* in ti,ab 
 #16 #9 or #10 or #11 or #12 or #13 or #14 or #15 
 #17 outcome* in ti,ab 
 #18. #17 near3 #16 
 #19. #1 or #2 or #3 or #4 or #5 or #6 or #7 or #8 
 #20. #18 or #19

1.5 EMBASE (1980 to Feb 2002, searched May 2002, SilverPlatter) with the following strategy: 
 #1 "health‐survey"/ all subheadings 
 #2 explode "quality‐of‐life"/ all subheadings 
 #3 "outcomes‐research"/ all subheadings 
 #4 health outcome* in ti,ab 
 #5 quality of life in ti,ab 
 #6 outcome measure* in ti,ab 
 #7 measure* in ti,ab 
 #8 (score* or scoring) in ti,ab 
 #9 index in ti,ab 
 #10 indices in ti,ab 
 #11 scale* in ti,ab 
 #12 monitor* in ti,ab 
 #13 assess* in ti,ab 
 #14 #7 or #8 or #9 or #10 or #11 or #12 or #13 
 #15 outcome* in ti,ab 
 #16 #15 near3 #14 
 #17 #1 or #2 or #3 or #4 or #5 or #6 
 #18 #16 or #17

1.6 MEDLINE (1966 to Feb 2002, searched May 2002, SilverPlatter) with the following strategy: 
 #1 "Health‐Status‐Indicators" 
 #2 "Outcome‐and‐Process‐Assessment‐(Health‐Care)"/ all subheadings 
 #3 "Outcome‐Assessment‐(Health‐Care)"/ all subheadings 
 #4 "Quality‐of‐Life"/ all subheadings 
 #5 (outcome measure*) in ti,ab 
 #6 (health outcome*) in ti,ab 
 #7 (quality of life) in ti,ab 
 #8 measure* in ti,ab 
 #9 assess* in ti,ab 
 #10 (score* or scoring) in ti,ab 
 #11 index in ti,ab 
 #12 indices in ti,ab 
 #13 scale* in ti,ab 
 #14 monitor* in ti,ab 
 #15 #8 or #9 or #10 or #12 or #11 or #13 or #14 
 #16 outcome* in ti,ab 
 #17 #16 near3 #15 
 #18 #1 or #2 or #3 or #4 or #5 or #6 or #7 
 #19 #17 or #18

1.7 PsycLIT (1887 to Feb 2002, searched May 2002, SilverPlatter) with the following strategy: 
 #1 explode "Treatment‐Outcomes" 
 #2 explode "Psychological‐Assessment" 
 #3 explode "Quality‐of‐Life" 
 #4 (outcome* or process*) near3 assessment* 
 #5 health status indicator* 
 #6 health status 
 #7 health outcome* in ti,ab 
 #8 quality of life in ti,ab 
 #9 outcome measure* in ti,ab 
 #10 measure* in ti,ab 
 #11 assess* in ti,ab 
 #12 (score* or scoring) in ti,ab 
 #13 index in ti,ab 
 #14 indices in ti,ab 
 #15 scale* in ti,ab 
 #16 monitor* in ti,ab 
 #17 #10 or #11 or #12 or #13 or #14 or #15 or #16 
 #18 outcome* in ti,ab 
 #19 #18 near3 #17 
 #20 #1 or #2 or #3 or #4 or #5 or #6 or #7 or #8 or #9 
 #21 #19 or #20

Searching other resources

1. Citation searches on located trials 
 We hand searched the citation lists of all potentially relevant studies (included and excluded).

2. Contacts 
 We contacted first authors of all potentially relevant studies (included and excluded).

3. Hand search 
 We hand searched the contents pages of the following journals: 
 Medical Care (Jan 1980 to Jan 2002) and Quality of Life Research (Jan 1996‐Jan 2002).

Data collection and analysis

1. Selection of trials 
 All potential trials were independently examined by two reviewers. Each determined if the trial fulfilled inclusion criteria. Disagreements were resolved by discussion, and with reference to a third reviewer.

2. Assessment of quality 
 Randomised trials of behavioural interventions, designed to influence clinical practice have to cope with the problem of 'cross contamination' between patients (Gail 1996). For example, a key aim of research summarised in this review is to encourage clinicians to become more responsive to psychosocial problems through the active feedback of standardised questionnaires. However, if clinicians see some patients with and some patients without questionnaire scores, there is a danger that clinical management will change for both intervention and control groups. In other words, receiving feedback for some patients may sensitise the clinician to psychosocial problems amongst all their patients. The consequence of this 'cross contamination' is that any real advantage of the intervention may be diluted, and no difference will be found between control and intervention arms. One solution to this problem is to use individual clinicians as the unit of randomisation, so that clinicians are randomised to receive feedback for all their patients if the clinician is allocated to the intervention arm, and to not receive feedback for any of their patients if the clinician is allocated to a control/usual care arm (Elbourne 1997, Bland 1997, Ukoumunne 1999). Hence, those studies which used 'cluster randomisation', where clinicians or clinical teams are the unit of randomisation were judged to be more robust than those which randomised individual patients.

We also assessed methodological quality in accordance with the Cochrane Handbook (Clarke 2002), and with the validated scale of Jadad (Jadad 1996) which considers method of randomisation, allocation concealment and intention to treat. All studies which were described as 'randomised' were included. Those with adequate randomisation and concealment of allocation (as defined by Jadad 1996) were compared with those where this was incompletely described in a sensitivity analysis.

3. Data collection 
 Data were extracted by one reviewer (SG) and independently checked by a second reviewer. Disagreements were resolved by discussion, and with reference to a third reviewer.

4. Data synthesis 
 Given the potential heterogeneity of studies, in terms of type of intervention, patient population, setting, outcome etc, data were handled in two ways:

4.1 We presented a detailed description of each study, its design and results in the 'other tables' section, together with a narrative overview of the main findings and important differences between individual studies.

4.2 For those studies which were sufficiently similar, in terms of the above variables, we undertook a quantitative data synthesis.

4.2.1 Intention to treat analysis 
 The analysis maintained the study groups according to the original randomisation procedure. For those lost to follow up, we assumed there had been no improvement from baseline.

4.2.2 Binary outcomes, 
 For outcomes such as readmission or intervention for a mental disorder, we calculated risk ratios.

4.2.3 Continuous outcomes. 
 Such as patient scores on the Brief Psychiatric Rating Scale, we calculated weighted mean differences (WMD) where there was a common metric between studies. Where different scales were used, which appeared to measure the same thing, we inspected the general direction of effect.

4.2.4 Cluster trials 
 For studies which employed a 'cluster randomisation' (such as randomisation by clinician or practice), we sought evidence that clustering had been accounted for by the authors of studies in their analysis. 
 However, the analysis and pooling of clustered data poses several problems: Firstly, authors often fail to account for intra class correlation in clustered studies, leading to a 'unit of analysis' error (Divine 1992) ‐ whereby p values are spuriously low, confidence intervals unduly narrow and statistical significance overestimated ‐ causing type I errors (Bland 1997, Gulliford 1999). Secondly, Revman does not currently support the meta‐analytic pooling of clustered dichotomous data, even when these are correctly analysed by the authors of primary studies, since the 'design effect' (a statistical correction for clustering) cannot be incorporated.

Where clustering was not accounted for in primary studies, we presented the data in a table, with an (*) symbol ‐ to indicate the presence of a unit of analysis error. In subsequent versions of this review we will attempt to contact first authors of studies to seek intra‐class correlation co‐efficients of their clustered data and to adjust for this using accepted methods (Gulliford 1999). If clustering had been incorporated into the analysis of primary studies, then we presented these data in a table. However, no further secondary analysis (including meta‐analytic pooling) will be attempted until there is consensus on the best methods of doing so, and until Revman software allows this. A Cochrane Statistical Methods Workgroup is currently addressing this issue. In the interim, individual studies will be very crudely classified as positive or negative, according to whether a statistically significant result (p<0.05) was obtained for the outcome in question, using an analytic method which allowed for clustering.

5. Scale‐derived data 
 A wide range of rating scales are available to measure outcomes in mental health trials. These scales vary in quality and many are not validated, or even ad hoc. It is generally accepted that measuring instruments should have the properties of reliability (the extent to which a test effectively measures anything at all) and validity (the extent to which a test measures that which it is supposed to measure). Before publication of an instrument, most scientific journals insist that reliability and validity be demonstrated to the satisfaction of referees. It was therefore decided, as a minimum standard, not to include any data from a rating scale in this review unless its properties had been published in a peer‐reviewed journal. In addition, the following minimum standards for rating scales were set: the rating scale should either be 1. a self‐report, or 2. completed by an independent rater or relative. More stringent standards for instruments may be set in future editions of this review.

Whenever possible we took the opportunity to make direct comparisons between trials that used the same measurement instrument to quantify specific outcomes. Where continuous data was presented from different scales rating the same effect, both sets of data were presented and the general direction of effect inspected.

6. Economic data 
 There are major difficulties involved in producing a systematic overview of cost effectiveness data (Gilbody 1999). Firstly, studies often adopt different perspectives, account for different types of cost data, use different methods of discounting future healthcare costs and are conducted at different points in time. Secondly, they are conducted in different countries and healthcare settings with different funding and reimbursement systems, making international comparisons difficult. Thirdly, economic evaluations often make fundamental errors in the analysis of data, by for example, applying parametric analyses to highly skewed cost data. Where cost data are presented and a formal cost effectiveness analysis is undertaken, their methods and results were simply described. No formal statistical pooling was attempted for cost or cost effectiveness data.

7. Heterogeneity 
 As well as inspecting the graphical presentations, the reviewers checked whether the differences among the results of trials were greater than would be expected by chance alone using tests of heterogeneity. A significance level less than 0.10 was interpreted as evidence of heterogeneity. Heterogenous data were not pooled, and sources of heterogeneity were sought.

8. Addressing publication bias 
 Psychiatric research is especially prone to publication bias (Gilbody 2000b), and we sought evidence of any inherent publication bias in our results by plotting 'funnel plots' where this was feasible (Egger 1997). Skewed funnel plots have a number of causes, including true heterogeneity, multiple publication of smaller studies, scientific fraud and publication bias. We sought to exclude heterogeneity and multiple publication bias before assuming publication bias or fraud to be the most likely cause.

Results

Description of studies

We inspected 3960 abstracts, and obtained 57 reports for further consideration ‐ including trials of the effect of routine outcomes assessment on depression, in order to check whether they included people with schizophrenia (Gilbody 2001).

1. Included studies 
 No studies could be included.

2. Excluded studies 
 We found seventeen studies that investigated the effect of routine outcomes assessment on the management of depression, which were excluded from the present review, but will be included in the companion review to this which is expected to be published in December 2002 (Gilbody 2000a).

3. Studies awaiting assessment 
 We found the methodological details of one study, which has now been completed (Marshall 2001). This study, which was conducted in Manchester, UK, will eventually report the impact of routine needs assessment on the process and outcome of care of those with severe mental illness who have taken part in the Care Programme Approach (CPA). The details of this study will be included in subsequent versions of this review, when study details and results are available in the public domain.

4. Ongoing studies 
 We found one ongoing study (Slade 2002), which is investigating the effect of routine outcome assessment on the management of people with severe mental illness. This study will establish the impact of the routine feedback of a battery of outcome measures, including the Camberwell Assessment of Need. Investigators plan to recruit 200 participants. The study is being conducted at the Institute of Psychiatry in London, UK.

Risk of bias in included studies

There were no studies that fulfilled the criteria for inclusion.

Effects of interventions

See description of studies.

Discussion

This is, as far as we are aware, the first attempt which has been made to systematically search for and appraise evidence to support or refute the value of routine outcome measurement and needs assessment for those with schizophrenia and related disorders. We were surprised that such a widely advocated strategy should be unsupported by robust evidence of clinical and cost effectiveness. Research to investigate the effect of routine outcome assessment on depression has been conducted, and the absence of a similar body of research amongst those with severe mental illness represents a gap in our understanding of how best to deliver effective care.

Evidence suggests that clinicians do not like collecting outcome measures in the context of the routine care of patients with serious mental illnesses, such as schizophrenia (Slade 1999, Gilbody 2002). This could be for many reasons. Clinicians may intuitively recognise that standardised measures add little to the overall psychosocial assessment of the well being of people with schizophrenia. The clinicians may feel that the measures themselves are of questionable psychometric value in the context of individual care. Of course, they could genuinely be uninterested in the use of standardised questionnaires, even if they are potentially of value, or even the psychosocial outcomes and patients' needs. The use of standardised measures is outside of most clinical 'cultures' and the implication of scrutiny though numerical monitoring may put clinicians off. If, however, the use of outcomes measures in the context of the routine care of those with schizophrenia and related disorders is genuinely of value, then the failure to use them is an omission. Conversely, if they are of little direct benefit to either people with schizophrenia or clinicians, then their collection is a bureaucratic exercise. This review is unable to give guidance in this respect.

Indirect evidence from other specialities suggests that the incorporation of routine outcomes measures into routine care does little to affect the quality of care that is offered to patients, nor has it been shown to be effective in actually improving outcome ‐ the major criterion of success of such a strategy (Kazis 1990). More direct evidence from the management of those with less severe mental disorders, such as depression and anxiety, suggests that the routine administration and feedback of standardised measures is largely ineffective in improving the quality and outcome of care (Gilbody 2001). There could be a number of reasons for this absence of effect. These include the fact that the provision of the results of routine outcomes measures is just another piece of information which clinicians feel unable to either interpret or act on (Nelson 1990). Similarly, the psychometric properties of many measures are such that they are un‐interpretable at the individual patient level because they are subject to wide variation when administered on successive occasions to the same patient (Dunn 1996).

The uncertainty of the true value of this strategy is likely to continue to frustrate those who wish to see the wider use of these measures, but who lack evidence to support this implementation. Equally, those with a genuine scepticism about the value of the approach are likely to be frustrated by continued requests that they undertake this activity in the absence of supportive evidence. We await with anticipation the results and outcomes of the ongoing and unpublished research identified in this review.

Authors' conclusions

Implications for practice.

The use of outcomes measurement in the context of routine care of those with schizophrenia is unsupported by robust clinical evidence of the clinical and cost effectiveness of this strategy.

1. For clinicians 
 Clinicians should judge for themselves whether the measurement of outcome and need is a reasonable use of their finite time and resources when the true value of this approach has yet to be demonstrated.

2. For people with schizophrenia 
 Invitations and edicts to complete and collect complex outcomes measures, questionnaires and needs assessment tools have not yet been shown to improve the quality of the care that those with schizophrenia and related disorders will receive.

3. For policy makers 
 Attempts to use the results of outcomes measures in planning the care of populations with severe mental illnesses are likely to be frustrated or of limited value, when so few clinicians collect and use these data in the care of their individual patients. Policy edicts to collect and use these data may not represent the best use of clinicians' and patients' time and finite healthcare resources.

Implications for research.

The true value of the use of outcomes measurement and needs assessment in improving the quality and outcome of the care of those with schizophrenia and related disorders needs to be established as a matter of some priority. The optimum design of such an evaluation would be a cluster‐randomised trial, framed within routine care settings. We await with interest the results of as yet unpublished and ongoing studies.

What's new

Date	Event	Description
31 October 2008	Amended	Converted to new review format.

Characteristics of studies

Characteristics of excluded studies [ordered by study ID]

Study	Reason for exclusion
Calkins 1994	Allocation: randomised. Participants: people with depression, no one with schizophrenia.
Callahan 1994	Allocation: randomised. Participants: people with depression, no one with schizophrenia.
Dorwick 1995	Allocation: randomised. Participants: people with depression, no one with schizophrenia.
German 1987	Allocation: randomised. Participants: people with depression, no one with schizophrenia.
Gold 1989	Allocation: randomised. Participants: people with depression, no one with schizophrenia.
Hoeper 1984	Allocation: randomised. Participants: people with depression, no one with schizophrenia.
Johnstone 1976	Allocation: randomised. Participants: people with depression, no one with schizophrenia.
Lewis 1996	Allocation: randomised. Participants: people with depression, no one with schizophrenia.
Linn 1980	Allocation: randomised. Participants: people with depression, no one with schizophrenia.
Magruder 1990	Allocation: randomised. Participants: people with depression, no one with schizophrenia.
Mathias 1994	Allocation: randomised. Participants: people with depression, no one with schizophrenia.
Moore 1997	Allocation: randomised. Participants: people with depression, no one with schizophrenia.
Reilfer 1996	Allocation: randomised. Participants: people with depression, no one with schizophrenia.
Weatherall 2000	Allocation: randomised. Participants: people with depression, no one with schizophrenia.
Whooley 2000	Allocation: randomised. Participants: people with depression, no one with schizophrenia.
Williams 1999	Allocation: randomised. Participants: people with depression, no one with schizophrenia.
Zung 1983	Allocation: randomised. Participants: people with depression, no one with schizophrenia.

Characteristics of ongoing studies [ordered by study ID]

Slade 2002.

Trial name or title	FOCUS ‐ Feedback on Outcome to Staff
Methods
Participants	People with severe mental illness. N=180.
Interventions	1. Administration and feedback of Camberwell Assessment of Need. 2. No feedback.
Outcomes	Service use. Mental state: BPRS. Needs: CAN.
Starting date	2001
Contact information	Dr Mike Slade Instiute of Psychiatry King's College London De Crespigny Park London UK SE5 8AF e‐mail: m.slade@iop.kcl.ac.uk
Notes

BPRS ‐ Brief Psychiatric Rating Scale 
 CAN ‐ Camberwell Assessment of Need

Contributions of authors

Simon Gilbody ‐ formulated and wrote protocol, initiated literature searches, extracted data and wrote the review.

Allan House ‐ commented on protocol, extracted data and commented on drafts of review.

Trevor Sheldon ‐ commented on protocol, extracted data and commented on drafts of review.

Sources of support

Internal sources

• University of York, UK.

• University of Leeds, UK.

External sources

• UK Medical Research Fellowship Programme, UK.

Declarations of interest

The authors began this review with a genuine uncertainty regarding the true value of routine outcomes measurement in improving the quality and outcome of care for those with schizophrenia. In the absence of such evidence, they are keen to seek funds to examine this question by conducting primary research.

Edited (no change to conclusions)