Involving users in the delivery and evaluation of mental health services: systematic review

Abstract

Objectives

To identify evidence from comparative studies on the effects of involving users in the delivery and evaluation of mental health services.

Data sources

English language articles published between January 1966 and October 2001 found by searching electronic databases.

Study selection

Systematic review of randomised controlled trials and other comparative studies of involving users in the delivery or evaluation of mental health services.

Data extraction

Patterns of delivery of services by employees who use or who used to use the service and professional employees and the effects on trainees, research, or clients of mental health services.

Results

Five randomised controlled trials and seven other comparative studies were identified. Half of the studies considered involving users in managing cases. Involving users as employees of mental health services led to clients having greater satisfaction with personal circumstances and less hospitalisation. Providers of services who had been trained by users had more positive attitudes toward users. Clients reported being less satisfied with services when interviewed by users.

Conclusions

Users can be involved as employees, trainers, or researchers without detrimental effect. Involving users with severe mental disorders in the delivery and evaluation of services is feasible.

What is already known on this topic

Involving health service users in the NHS is recommended in UK government policy

Involving users in mental health services is generally seen as worthwhile, but the effects of involving users have not been thoroughly evaluated, and few attempts to draw evaluations together have been made

What this study adds

The few comparative studies of users' involvement that have been published indicate that involving users as employees, trainers, or researchers has no negative effect on services and may be of benefit

Introduction

The Department of Health in the United Kingdom is committed to involving patients in the NHS; it is establishing the Commission for Patient and Public Involvement in Health. Users and carers have been involved in delivering and evaluating mental health services, but the effects of this involvement have not been rigorously assessed.^1–3

We found randomised controlled trials and other comparative studies containing evidence about positive or negative effects of involving users in the delivery or evaluation of mental health services.⁴ We sought evidence on involving users and the outcomes of involvement on clients (those receiving services). Initially the search encompassed users who were involved in planning services, but we found no comparative studies. We also investigated carers' involvement but found too few studies; only one involved carers as well as users,⁵ and one other explicitly mentioned a carer's relative with psychiatric history.⁶

Methods

We searched Medline, Embase, CINAHL, PsycINFO, HealthSTAR, Cochrane Controlled Trials Register, Web of Science, HMIC, and BIDS for references in English between January 1966 and October 2001 for the terms given in box B1. Searches equivalent to the Medline search were used for other databases.

Box 1.

Terms used in Medline search

MeSH terms

Consumer participation/

Consumer advocacy/

Patient advocacy/

Consumer organizations/

Consumer satisfaction/

Caregivers/

Family relations/

Mental disorders/

Mental health/

Mental health services/

Community mental health centers/

Psychiatry/

Key and text words ($ is a wildcard)

user$

consumer$

client$

carer$

caregiver$

involv$

participat$

We wrote to experts and organisations who had an interest in involving healthcare users. We searched the references in all papers for additional studies, whether we included them or not. We searched collections by hand in the Health Sciences Library of the University of Leeds.

Inclusion and exclusion criteria

We included evaluations of the impact of research on services if users had an active role in the design or in collecting data. We also included studies about users who delivered services by training mental health professionals.

We included studies about delivery involving users in partnership with others if services were integrated by health professionals and users working together in a team; cross-consultation; or recruitment, training, supervision, or payment of users by healthcare providers. We excluded studies which dealt only with the criteria in box B2. Box B3 gives the type of data we extracted.

Box 2.

Exclusion criteria

We excluded studies if they dealt with only

• Learning disabilities
• Involvement in decisions about a user's own treatment
• Providing information to users
• User satisfaction surveys that were researched by the provider (which do not require users' partnership)
• General health services not specifically aimed at mentally ill people
• Forensic services
• Services for mentally ill people which are not health related, such as housing or vocational rehabilitation
• Services with no contact with professionals or which could not be run by professionals which operate outside the mental health system—for example, self help groups

Box 3.

Type of data extracted from databases

• Mechanism of involving users, including support available
• Numbers of users involved and diagnoses
• Service or setting of involvement
• Study design, including numbers in comparison groups
• All measures of the process of involving users
• All measures of outcomes for employees who were or who had been users and their clients

To assess the quality of the data, we sought the method of randomisation, evidence of blinding during data collection, and an intention to treat analysis.⁴ We checked papers for inclusion and exclusion criteria and extracted data onto a standardised form independently by both authors. Meta-analysis was unacceptable because of heterogeneity in the study design and outcome measures so we summarised these qualitatively.⁴

Results

We identified five randomised controlled trials and seven other comparative studies.^5–16 Comparisons were mostly of services involving users compared with services with non-users in similar roles. One study compared involvement of more severely disordered users with those less severely disordered¹⁴; one study compared lots of contact with involved users with less contact.¹⁵

The nature of users' involvement

Eight studies focused on involving users as service providers, mainly working as case managers in services for clients with severe mental illness (table 1). Case managers need to engage clients, coordinate agencies, and help maintain effective delivery; the necessary skills are organisational and interpersonal rather than therapeutic. Two studies looked at the effects of involving users as trainers (table 2), and two studies considered involving users as interviewers (table 3).

Table 1.

Involving current or former users of mental health services as providers in mental health services

Study	Involvement of	No of users involved and inclusion criteria	Study design (n=No of clients)	Measures of client* outcomes or service delivery patterns	Differences between groups
Solomon and Draine, 1994-6, USA†7 17 19	Case managers in community mental health service	4 in team (population changed over time); recent use of psychiatric services	Randomised controlled trial; 2 case management team conditions: employing users (n=48) and employing non-users (n=48)	Delivery: dates, locations, and manner of contact with clients	User employees: more face to face, fewer telephone or office based contacts
				Outcomes: income, level of functioning, quality of life, attitude to drugs compliance, social contacts, symptoms, inpatient days, treatment satisfaction	1 year: clients of user employees less satisfied with treatment, less family contact; 2 years: none
Paulson et al, 1997-2000, USA†8 18	Case managers in assertive community treatment programme	5 in team (population changed over time)	Randomised controlled trial; 3 conditions: assertive community treatment employing users (n=58), employing non-users (n=59), and usual care (n=61)	Delivery: time spent on categories of case manager activities	User employees (compared with non-user ACT employees): longer in supervision, more flexible scheduling
				Outcomes: time until first hospitalisation, arrest, emergency hospital care, or homelessness	Clients of user employees: longer before hospital admission, fewer hospitalised, or had emergency care
O'Donnell et al, 1998-9, Australia5 23	Client advocates attached to case management service	Number not stated	Randomised controlled trial; 3 case management conditions: clients focused with advocacy (n=45), clients focused (n=39), and standard care (n=35)	Outcomes: satisfaction with service, quality of life, functioning, family burden, inpatient days, use of crisis services	Family burden lower for client focused (2 groups combined) than for standard case management
Klein et al, 1998, USA10	Peer counsellors alongside case management service	Number not stated; recovering from addiction	Comparative study; 2 case management conditions: with peer support (n=10) and standard (n=51)	Outcomes: hospital admissions, crisis events, social support, functioning, quality of life, drug use, satisfaction with service	Clients of peer support: fewer inpatient days, better social functioning, some quality of life improvements
Felton et al, 1995, USA11	Peer specialists on case management teams	3	Comparative study; 3 case management conditions: additional employees who were users (n=125), additional non-user employees (n=118), and no additional employees (n=68)	Outcomes: self esteem, engagement in programme, attitude to recovery, social support, quality of life, inpatient days, life problems, symptoms	Clients of user employees (compared with other 2 groups combined): more satisfied with living situations and finances, fewer reported life problems, less decline in contact with case managers
Chinman et al, 2000, USA12	Case managers in outreach service	Number not stated; prior psychiatric treatment	Descriptive study; case management service sites separated into 2 conditions: sites with ⩾10 clients of user employees (n=113) and sites with all or most services from non-user employees (n=630)	Outcomes: symptoms, quality of life, days of homelessness, social support, employment, relationship between client and case manager	None
Chinman et al, 2001, USA13	Service providers in community outreach service	3 in team (population changed over time)	Comparative study; 2 conditions: programme with user employees (n=92) and matched sample of clients receiving usual care (n=79)	Outcomes: number of readmissions to hospital, inpatient days	None
Lyons et al, 1996, USA14	Users as service providers in mobile crisis assessment service	8; prior psychiatric hospitalisation and medication or prior outpatient treatment	Descriptive study; compared working pairs in which: 1 or both of the pair had history of hospitalisation and neither user employee had a history of hospitalisation	Delivery: time spent on categories of duties, pattern of hospitalising clients	Working pairs in which at least 1 user employee had previous hospitalisation: more mobile outreach, fewer emergency responses, more hospitalising of clients involuntarily during routine dispatch

^*Clients are recipients of services in which users are employed. 

^†These studies are also described in other publications cited elsewhere.²⁰ 

Table 2.

 Involving current or former users of mental health services as trainers of mental health service providers

Study	Users	Users involved	Study design	Outcome measures	Differences between groups
Cook et al, 1995, USA6	Training mental health professionals	One person with bipolar disorder	Randomised controlled trial of 57 trainees trained by the user trainer or a non-user trainer	Trainee attitudes toward user employees; stigmatising factors of mental illness; likelihood of recovery	Trainees in the user trainer group had significantly more positive attitudes toward user employees and stigmatising factors of mental illness
Wood and Wilson-Barnet, 1999, UK15	Student nurse classroom education	Not stated	Comparative study of 2 groups of students (n=15; n=14) differing in exposure to involving users in training	Student approach to mental health assessment; qualitative themes; empathy; individualised approach	Students with more and earlier exposure to user involvement, less jargon, more empathy, more individualised approach

Table 3.

 Involving current or former users of a mental health service as interviewers of recipients of the service (clients) in evaluating mental health services

Study	Users involved	Design of study	Measurement of clients' views of service	Significant differences between groups
Clark, 1999, Canada9	Four with severe mental disorder and prior psychiatric hospitalisation	Randomised controlled trial of user interviewers (n=60) and staff interviewers (n=60)	Extremely positive and negative responses and general satisfaction	Clients interviewed by user interviewers gave more extremely negative responses about services
Polowczykl, 1993, USA16	People with schizophrenia or affective disorder in remission	Comparative study of user interviewers (n=225) and staff interviewer (n=305)	Satisfaction score	Clients interviewed by user interviewers gave lower service satisfaction scores

The users who were involved were current or former users of mental health services who had had serious psychiatric illness—most commonly schizophrenia or bipolar disorder; many had been hospitalised. Employees who were or who had been users of mental health care services and interviewers had similar disorders to their clients.

Interviewers and employees who were or who had been users all received training. Where applicable, this training was similar to that received by employees who had not been users of mental health services. Payment was mentioned in most studies, and support workers were available to nearly all of the employees were or who had been users of services.

Effects of users' involvement

The process of service delivery of employees who were or who had been users of mental health services differed from that of employees who had not. Users spent longer in supervision,⁸ in face-to-face contact with clients,¹⁷ or doing outreach work,¹⁴ and they spent less time on telephone or office work.¹⁷ Employees who were or who had been users had a higher turnover rate and had less distinct professional boundaries.⁸

Employing users in, or alongside, case management services did not have any detrimental effect on clients in terms of symptoms,⁷,¹² functioning,⁵,⁷,¹⁰,¹² or quality of life.⁵,⁷,¹² Clients of these services had some improved quality of life¹⁰,¹¹; they had fewer reported life problems and improved social functioning.¹¹,¹⁰ Some clients were less of a burden to their families.⁵,⁷,¹² In some studies, clients of employees who were or who had been users went for longer until hospital admission and fewer clients needed to be admitted to hospital,¹⁰,¹¹,¹⁸ or stay in hospital was shorter,¹⁰ although time in hospital was not significantly different in all studies.⁵,⁷,¹¹,¹³ Services employing people who were or who had been users did not have lower client satisfaction.⁵,⁷,¹⁰,¹² In one study, clients of employees who were or who had been users were less satisfied with treatment at follow up after one year,¹⁹ but they were not after two years.⁷

Involving users in training gave trainees a more positive attitude toward employees who had been mentally ill and mental illness in general,⁶ or they looked at users as individuals.¹⁵ Clients reported being less satisfied with services when interviewed by other users of the service in evaluation research.⁹,¹⁶

Design of study and interpretation

Our review of 298 papers about involving users in delivery of mental health services²⁰ included only 12 comparative studies. We found five randomised trials, only one of which indicated the randomisation method used (alternate allocation according to an alphabetically ordered list of surnames).⁶ Researchers collecting data were not blinded to treatment group in any of the studies. Four of the trials used intention to treat analysis.⁶,⁷,⁹,¹⁸ Of the other seven studies, researchers were blinded to treatment group in one study.¹¹ No intention to treat analysis was done in these studies.

Some studies were not set up to investigate users' involvement and the results were from a later analysis of routinely collected data.¹¹ Some studies had more than two study groups and did not directly compare involving users with involving those who had not been users.¹¹

Few standardised outcome measures were used unmodified. Measures included adapted versions or selected subscales of existing scales.⁵,⁷,^10–12,¹⁶ Some outcome measures were constructed for the particular study.⁶,¹¹,¹⁵ Users were involved in the design of a questionnaire developed for one study.⁹ The use of modified rating scales could have led to bias, as has been shown for unpublished scales.²¹

Only small numbers of users were involved, with numbers ranging from one user to eight users in a team, making it difficult to apply findings to involving users in general.⁶,¹⁴ More users were involved in some studies because some users dropped out, generally for unstated reasons, and were replaced.⁸,¹³,¹⁷

Sample sizes of studies were small, so estimates of effect were of low power. Clients were not always willing to see staff whom the clients knew had had mental illness.¹⁰

Authors interpreted their findings, saying, for example, that when users were less likely to hospitalise clients, it might be because of their own previous bad experiences or because they had more tolerance for behaviour arising from symptoms, used previous experience to help clients stay out of hospital, or more readily engaged with clients needing hospitalisation.¹⁴,¹⁸ That interviewers who had been users obtained a higher proportion of negative satisfaction scores might be due to clients feeling more able to be honest with users, thus increasing validity, or it might be that they perceive dissatisfaction as the socially desirable response.⁹,¹⁶ These possibilities were not explored.

Discussion

The studies that we identified suggest that users of mental health services can be involved as employees of such services, trainers, or researchers without damaging them. In some studies, benefit was indicated for clients of employees who were or who had been users of services, and, although this was not present across all studies, there were no serious disadvantages. The influence of trainers who had been users on the attitudes of trainees was positive; interviewers who had been users may have brought out negative opinions of services that would not otherwise have been obtained.

Studies suggest that users with a history of severe disorders can be involved in services. This may depend on adequate support, as all of the studies we found included details of the support provided to involved users. This included training and payment for involvement. Service providers have given practical and personal support to users—for example, discussing issues of confidentiality or advising on work matters.⁶,¹⁷ This support is clearly distinguished from treatment. Our review of non-comparative research supports these findings.²⁰

We found no comparative studies of users' involvement in planning mental health services, but other evaluations of users' involvement in planning in health services—including mental health services—have recently been reviewed.²²

Most of the studies we identified involved few users and have substantial methodological weaknesses. Studies of users as service providers mostly originated in the United States and were confined to a case management model. Government policy in the United Kingdom strongly supports the development of involving users in the delivery and evaluation of mental health services. Little evidence exists on the effectiveness of such programmes, and more formal evaluations are needed.