Cluster randomised controlled trial of the effectiveness of primary care mental health workers

Abstract

Background

Mental health issues are a core part of the work of primary care and are the second most common reason for consultations. There is some evidence that the quality of primary care mental health provision is variable.

Aim

To evaluate the effectiveness of primary care mental health workers with regards to satisfaction with care, mental health symptoms, use of the voluntary sector, and cost effectiveness of care.

Design of study

Cluster randomised controlled trial.

Setting

Practices in the Heart of Birmingham Primary Care Trust, Birmingham, England.

Method

Nineteen practices and 368 patients (18 to 65 years of age) with a diagnosis of a new or ongoing common mental health problem were recruited. Sixteen practices and 284 patients completed the trial.

Results

Patients in intervention practices had a higher mean level of general satisfaction than those in control practices (difference between group scores of 8.3, 95% confidence interval = 1.3 to 15.3, P = 0.023). The two groups did not differ in mental health symptom scores or use of the voluntary sector.

Conclusion

For patients with common mental health problems, primary care mental health workers may be effective at increasing satisfaction with an episode of care.

INTRODUCTION

Mental health issues are a core part of the work of primary care and the second most common reason for consultations.¹ However, there is some evidence that the quality of primary care mental health provision is variable, with inappropriate treatment and under diagnosis of common mental health problems.^2,3 Improving mental health services is a key strategic policy priority.⁴ Within this clinical and policy context, The NHS plan called for a new workforce of primary care mental health workers able to support primary care in delivering good quality primary care mental health.⁵ This role has since been developed through policy documents⁶ and role specific guidance from the Department of Health.^7,8 In spite of policy imperatives, the evidence to support the introduction and guide the implementation of the role is limited.^9–11

There has been no trial of the effectiveness of mental health workers in achieving the stated policy aims that underpinned their implementation strategy. This study aimed to explore the effectiveness of primary care mental health workers with regards to key issues of patient satisfaction with care, changes in mental health symptoms, use of the voluntary sector, and cost-effectiveness.

METHOD

Participants

The study took place in the Heart of Birmingham Teaching Primary Care Trust (PCT) in Birmingham, England. The Trust has a population of 310 000 and more than 150 GPs. Seventy per cent of practices are single-handed and 70% of the population are from black and ethnic minority communities.

Randomisation

A cluster randomised controlled trial design was used, clustered at general practice level, stratified by practice size. Practices were allocated to either the intervention (access to a mental health worker) or control group (no access to a worker) using computer generated random numbers.

Intervention

Workers had three broadly defined areas of responsibility, all of which were in accordance with national guidance on the role of mental health workers.⁷ Their patient work included brief evidence-based interventions, such as anxiety management for people with common mental health problems, information, assessment, screening if required (for example, using easily administered mental health rating scales), onward referral to the voluntary sector and support for self-help, and mental health promotion. Practice teamwork included providing support for audit, development of a mental illness register, in-house training, and initiatives to increase involvement of patients and carers at practice level. They also compiled a practice folder of local voluntary and community sector services, visiting each one in the practice locality to gain more in depth information about waiting times, referral mechanisms, treatment packages, and costs to patients.

Work in the wider community included liaising with primary care team members, statutory and non-statutory sector services, and specialised services for patients who are managed in primary care. These roles were defined in accordance with the national guidance to help ensure that findings were relevant beyond a single PCT.

In January 2002, the Trust employed five psychology graduates to train and then work as primary care mental health workers. All underwent a bespoke 12-week training course described elsewhere,¹² followed by a 3-week practice-based induction and ongoing training on a day release basis. Although shorter than subsequent centrally-funded training courses, its intensive nature covered all the knowledge, skills, and attitudes required in their job description.

Each worker was allocated to one or two practices, depending on the practice list size, so that each was responsible for a population of approximately 7000 patients. This number was based on historical referral rates to community mental health teams in the Trust and therefore potential workload. Workers were based in primary care, employed by the Trust, and received 1 hour of individual clinical supervision each week from a psychologist in the local mental health trust.

How this fits in

Mental health issues are a core part of the work of primary care, but there is evidence that the quality of primary care mental health provision is variable. For patients with common mental health problems, primary care mental health workers may be effective at increasing satisfaction with their episode of care. However, workers do not appear to have a statistically significant effect on patients' mental health symptoms or use of the voluntary sector.

Study outcomes

The primary outcome was patient satisfaction with the episode of care, as measured by a modified version of the self-report 18-item Consultation Satisfaction Questionnaire (CSQ) at 3 months.¹³ Patient satisfaction was chosen as the primary outcome because previous and contemporaneous work in the area of primary care mental health suggests that issues such as the depth of the relationship with the health professional, duration of the consultation, provision of information, and general satisfaction with care are critical issues in a primary care consultation.^14,15 Satisfaction with care is also related to other important issues including adherence to treatment.¹⁶ Previous work suggests that patients with mental health problems are often less satisfied with their primary care than the general population.^17,18

The CSQ was modified to include four additional questions, relating to provision of information, that were not previously addressed by this instrument (desire to be told more about services; information about services from practice staff; knowledge of who to contact if experiencing problems, and feeling that staff listened to needs). After discussion with the author of the scale, questions were further modified to relate to the ‘episode of care’ rather than a single consultation with a GP (R Baker, personal communication, 2002). Because the CSQ was modified for this study, the primary analysis was defined a priori to be the three items within the general satisfaction element of the scale so that comparison with other studies using this measure could be made.¹⁹ In addition, the overall score and each component score were also estimated and constituted secondary outcome measures. Another secondary outcome was change in mental health symptoms measured by the Clinical Outcomes in Routine Evaluation–Outcome Measure (CORE–OM).²⁰ CORE–OM is a 34-item self-report questionnaire that measures patient distress, including subjective wellbeing, commonly experienced problems or symptoms and life, and/or social functioning.

Participants completed the CORE–OM while in the surgery (with the help of an interpreter trained in the administration of this instrument if necessary). They were sent a further CORE–OM and the CSQ 3 months after recruitment, by post. If an interpreter was needed at the time of recruitment, then arrangements were made for the interpreter to be available to help complete the follow-up questionnaires. Patients who did not respond at 3 months received at least two reminders and were followed up by telephone or home visit where necessary.

Use of any voluntary sector in the time between recruitment and the 3-month follow up was also recorded by patient self-report on the returned questionnaires. To assess resource use and the cost-effectiveness of the role, the frequency of primary care consultations, referrals to secondary mental health care and non-mental healthcare services, mental health inpatient stays and mental health drug costs during the 12 months after recruitment were recorded from patient primary care records,

Patients

Patients identified by their GP as having an ongoing or newly diagnosed common mental health problem during the course of a normal consultation on study data-collection days, were invited to provide data for the study. The study was described as an opportunity to gain patients' views on primary care and mental health rather than as a direct evaluation of workers, as many patients in the trial either had no access to, or were not referred to, the workers. Eligible patients were referred to a researcher based at the practice, given the study information leaflet, and if agreeable, consented to participate. Study days were randomly selected and were initially once or twice a month per practice. The frequency of study days was later varied as a response to patient recruitment rates in each practice. Patients were excluded if they were under 18 years or over 65 years of age, unable or unwilling to give informed consent, or had organic brain disease.

Blinding

It was not possible to blind the research team to the allocation of practice status because of the presence of mental health workers in the intervention practices. However, the CSQ and CORE–OM are patient self-report measures, protecting against attribution bias.

Statistical methods and analysis

The trial aimed to identify the effect of the presence of a primary care mental health worker on the primary and secondary outcome measures. The posited treatment effects were thus most appropriately conceived at a practice level, although they were experienced by, and must be estimated from, the outcomes of patients who received care from each practice. Thus, practice-level treatment effects were estimated from a series of measures from patients which cannot be considered independent.

Analysis for the primary outcome and other continuous outcomes used generalised mixed models with the practice defined as a random variable and, thus, accounted for clustering at the practice level.²¹ Where baseline data were available (for example, the CORE–OM questionnaire) these were used as patient-level covariates in the analysis. Analysis of the effects of the mental health workers on the use of voluntary sector services by patients was undertaken similarly, using a non-linear mixed model. As the variable ‘mental health workers’ is largely confounded with practice, the specific effects of different workers could not be investigated, although the overall results were conditioned for variability on this strata using the strategy described above. The denominator degrees of freedom for the treatment effects of interest were derived from the practice strata. All analyses were conducted according to the intention-to-treat principal.

Analysis was also carried out for costs of resource use for 12 months after recruitment. Costs for resources were obtained from Curtis and Netten²² and the NHS resource costs.²³ Total costs for resource use over the 12-month period were calculated for each patient and mean costs were calculated by treatment group. A bootstrap procedure with 100 000 replications was used to calculate 95% confidence intervals (CI) for the difference in costs between the two groups. The effect of clustering at the practice level on the cost estimate was investigated using a mixed model based on the log-transformed cost per patient, with practices as random effects.

Sample size calculation

In randomised trials where patients are randomised by practice, allowance must be made in the planning and analysis stages for the likely clustering at the level of the practice (that patients within a practice may be more alike than those in the population as a whole). Taking the given number of practices that expressed an interest in participating (19 initially, then 17 of 77 in the PCT) at the start of the study (nine intervention and eight control practices), presuming the availability of 25 patients per practice, a standard deviation (SD) for the CSQ of 12.6, and imputing a large between-practice component of variance (SD = 5.7; equivalent to an intra-practice coefficient of 0.20), the study had 90% power as planned to find a 10.4% difference in the mean change in CSQ scores as statistically significant at the conventional α value (two sided) of 0.05.

RESULTS

Practice recruitment

All 77 practices in the PCT in 2002 were invited to participate in the study by letter from the PCT mental health lead and the research team. The role of the workers, practice responsibilities, and implication of involvement in the study were explained to interested GPs and practice managers at a series of meetings in Spring 2002. Nineteen practices volunteered to participate. Ten practices were randomised to intervention status and nine to control status. Two intervention practices and one control practice withdrew from the study almost immediately between randomisation and the start of data collection (Figure 1).

Figure 1.

Practice and patient flow through the trial [Consort Diagram].

Patient recruitment

Patient recruitment began on 1 February 2003 and continued until the 30 November 2004. Four hundred and fifty-eight patients were approached and 368 patients were recruited. Fifty-seven patients declined to participate in the control practices and 33 in the intervention practices. Patient baseline characteristics are shown in Table 1.

Table 1.

Baseline characteristics of patients and practices included in the trial.

	Intervention group (n = 180)	Control group (n =188)
Demographics
Mean age years (SD)	38.1 (10.4)	38.5 (11.3)
Females (%)	124 (68.86)	121 (64.4)
Full-time employment (%)	38 (21.1)	61 (32.4)
Part-time employment (%)	18 (10.0)	14 (7.4)
Unemployed (%)	29 (16.1)	38 (20.2)
On sickness benefit (%)	34 (18.9)	28 (14.9)
House person (%)	37 (20.6)	24 (12.8)
Other (%)	22 (12.2)	42 (22.3)
Ethnicity
Asian (%)	97 (53.9)	50 (26.6)
White (%)	34 (18.9)	100 (53.2)
Black (%)	25 (13.9)	20 (10.6)
Other (%)	22 (12.2)	17 (9.0)
Required an interpreter (%)	12 (6.6)	46 (24.5)
CORE, mean (SD)	7.2 (2.6)	7.3 (2.6)
Practices Variable, mean (SD) Townsend Score	13.2 (2.9)	12.8 (4.2)
Whole Time Equivalent GPs, mean (SD)	1.9 (0.9)	2.1 (0.8)

SD = standard deviation. CORE = clinical outcomes in routine evaluation.

Patient response at 3 months

In the intervention group, 142 out of 180 patients (78.9%) provided outcome data at 3 months, compared with 140 out of 188 patients in the control practices (74.5%). The recruitment of practices and flow of patients through the trial is described in Figure 1.

Primary outcome

In the general satisfaction component of the CSQ, the intervention practices had a higher mean level of satisfaction compared with the control practices which was modestly statistically significant (difference in CSQ general satisfaction of 8.3 [95% CI = 1.3 to 15.3]).

Secondary outcome measures

A modest benefit was also seen in the overall total CSQ score (difference between groups 30.3, 95% CI 0.6 to 60.0), and the duration of consultation and provision of information components of the score. No effect was observed on the professionalism component or the depth of relationship components (Table 2).

Table 2.

Results of the primary and secondary outcomes at 3 months.

Outcome	Control Group	Intervention Group	Estimate of difference between groups (95% CI)	P-value
CSQ General satisfaction score	50.23	58.15	8.31 (1.33 to 15.30)	0.023
CSQ_ Total score	224.76	254.07	30.31 (0.66 to 59.96)	0.046
CSQ_ Professional aspects of the consultations	31.10	32.56	1.54 (−1.28 to 4.36)	0.26
CSQ_ Depth of the relationship with the health professional	52.23	54.32	2.49 (−5.21 to 10.19)	0.50
CSQ_ Perceived duration of the consultation	40.08	50.18	9.65 (0.46 to 18.84)	0.04
CSQ_ Provision of information	51.09	58.32	7.35 (0.03 to 14.66)	0.049
Total CORE	6.06	6.60	-0.35 (−1.22 to 0.52)	0.41
Use of the voluntary sector (odds and odds ratio)	0.27	0.15	1.79 (0.91 to 3.54)	0.09

Difference in the reported mean values or odds for each group do not equal the difference reported in the table because the latter accounts for clustering at the practice level, and also accounts for the patient baseline value where available. CSQ = consultation satisfaction questionnaire.

The CORE–OM symptom questionnaire provided no evidence that the presence of a mental health worker improved patient symptoms (difference in means −0.35, 95% CI = 0.52 to −1.22) where a negative score indicates improved symptoms in the intervention group. The confidence intervals suggest that the study was sufficiently powered to explore mental health symptoms and satisfaction with care.

Resource use and cost-effectiveness

Forty-one participants were lost to follow up in terms of the availability of their primary care notes. As a result, data on resource use was available for only 327 participants (165 from the control group). Of the 162 participants in the intervention group for whom GP note-based follow-up data were available, 122 participants did not see a mental health worker, 38 participants saw a mental health worker at least once and no data were available for two participants. For the 38 participants who saw a mental health worker, the number of visits ranged from one to four, with a median of one visit. Average resource utilisation for the intervention group and control group was examined 12 months after recruitment to the study (Table 3). Resource use was similar across both groups.

Table 3.

Mean for resource use 12-months after randomisation by treatment group.

Resource	Control group (n = 165)	Intervention group (n = 162)
GP consultations	7.69 (6.41)a	7.88 (6.46)
Practice nurse consultations	1.69 (2.66)	1.70 (2.54)
PCMHW consultations	–	0.36 (0.76)
Counsellor consultations	0.05 (0.25)	0.06 (0.23)
Secondary care: non-mental health consultations	1.16 (1.99)	0.94 (1.35)
Psychiatrists consultations	0.11 (0.37)	0.19 (0.58)
Community psychiatric nurse consultations	0.05 (0.23)	0.07 (0.26)
Community mental health team consultations	0.09 (0.45)	0.06 (0.27)
Days as a mental health inpatient	0.31 (1.63)	0.73 (4.08)

PCMHW = primary care mental health worker.

^aStandard deviation.

Table 4 shows mean costs for resource utilisation and the difference in mean costs with the bootstrapped 95% CI. Resources have been grouped by setting (primary care and secondary care). For secondary care consultations, costs are presented separately for mental health consultations, non-mental health consultations, and mental health inpatient stays. The cost for drugs includes those prescribed before randomisation although, only costs incurred from the date of recruitment to the study were considered.

Table 4.

Mean resource costs (£) and difference in means costs (95% CI) by treatment group.

Resource	Control group	Intervention group	Difference in means (95% Cl)
Primary care consultations	232.21	261.79	29.58 (–49.07 to 110.51)
Secondary care: non-mental health consultations	147.2	118.69	–28.51 (–111.30 to 41.97)
Secondary care: mental health consultations	24.20	34.36	10.16 (–5.61 to 29.52)
Drugs costs	40.43	25.12	–15.31 (–48.40 to 13.48)
Total: drug costs and outpatient consultations (primary care and secondary care)	444.05	439.97	–4.08 (–121.19 to 109.17)
Inpatient stays	58.73	138.40	79.67 (–59.23 to 396.53)
Total costs (including inpatient stays)	502.77	578.36	75.59 (–132.65 to 392.60)

To account for the clustering at practice level, the effect of treatment group on costs was modelled using practice as a random variable. A model using log-transformed costs showed no significant effect of treatment. Standard errors on the estimate of difference in log cost were 58% wider, although accounting for practice in this way had no appreciable effect on the size of the estimate of effect when compared with a model that did not include the effect of practice. Thus, to account for clustering at practice level, the width of the bootstrapped CIs were inflated by about 58%. Similarly the standard errors on the estimate of difference in log cost were 54% wider for the combined total of outpatient visits and drugs prescribed, 95% wider for primary care consultations, 63% wider for non-mental health referrals to secondary care, and 73% wider for inpatient stays. Accounting for practice did not lead to a change in the standard error of estimate of difference in log costs for mental health referrals to secondary care. As with total costs, the bootstrapped CIs were inflated by the appropriate factor.

On average, the control group incurred a total annual cost of £502.77, while the intervention group had a total cost of £578.36. The difference in mean costs was £75.59 (95% CI = 132.65 to 392.60). Although the intervention group appeared to have higher costs, the 95% CI indicates a large range of plausible values for differences in cost, and shows that the difference in scores is not significant. Analyses of sub-costs indicated no significant difference between the two treatment groups for costs of primary care consultations, non-mental health consultations in secondary care, mental health consultations in secondary care, drug costs, and inpatient stays. Differences in costs between the two groups were relatively small for primary care visits, non-mental health referrals, mental health referrals, and drug costs.

The main source of variation appears to be costs for inpatient episodes, where CIs were much wider. An analysis of the total costs for each group not including inpatient stays showed slightly lower costs for the intervention group (£439.97) when compared with the control group (£444.05). However, this difference was not significant (difference in means = −4.08, 95% CI −132.65 to 392.60). It is important to note that the study was designed mainly to test for effectiveness of mental health workers with patient satisfaction as the primary outcome measure. Cost-effectiveness was a secondary outcome and as such the study lacks adequate power to detect differences in costs. Costs of activities that workers were involved in, other than face-to-face consultations, were also not collected.

DISCUSSION

Summary of main findings

This trial provides evidence to suggest that mental health workers may be effective at increasing patients' satisfaction with an episode of care. However, it provides no evidence that practices with mental health workers improved the resolution of mental health symptoms of inpatients compared with the experience of patients in practices without workers.

Strengths and limitations of the study

This study attempted to trial the new role of mental health workers in primary care. The study team worked with predominantly research-naïve practices and aimed to include patients often excluded from trials because of a poor command of written English.

Although adequately powered with a design that minimised selection bias, the study involved only one PCT with an over representation of single-handed practices, nineteen practices and five workers, which limits the generalisability of the findings. There was also an imbalance in the patient sample in terms of ethnicity and language, although practice level differences were conditioned for in the analysis.

The choice of satisfaction with care, rather than change in mental health symptoms, as a primary outcome may seem counterintuitive, however, in addition to the reasons described earlier in the paper, this allowed conclusions to be drawn about both primary and secondary outcomes that might otherwise not have been possible.²⁴

Only 38 patients who were eligible to see the workers actually did so. This may reflect the real world nature of the trial and the fact that the study team emphasised to GPs that they should not alter their usual clinical practice to accommodate the study, but should behave as they would outside of a research context. This study does not estimate the effects of management from a mental health worker at the patient level, but examines the effect of adding a mental health worker to the practice team on patient outcomes. It is important to recognise that workers' roles included a number of other components, such as strengthening the practice primary care mental health infrastructure through audits, registers and in-house training which may have had an impact on the care provided by the wider primary care team. This study's results are therefore supportive of the hypothesis that workers can improve satisfaction among patients with common mental health problems.

As is conventional, costs were calculated for health service use, but did not include use of the voluntary sector. Unlike the other primary and secondary outcomes, costs varied substantially among patients and practices, and this variability is reflected in the width of the confidence intervals reported. It is also important to note that cost-effectiveness was a secondary outcome, and as such, the study lacks adequate power to detect differences in costs.

Comparison with existing literature

Bower's review of models of working in primary care mental health highlighted a number of potential ways of working for mental health workers.⁹ In particular, aspects of befriending²⁵ reflected in the CORE–OM domains and referral facilitator models²⁶ appear to be supported by the findings of this study. This work also echoes recent findings on the value of problem solving by community mental health nurses for anxiety, depression, and life difficulties of patients in primary care.²⁷

Implications for policy and practice

Within a financially limited healthcare system, it is important to evaluate new roles before their widespread introduction. This study suggests that the role of primary care mental health workers may lead to improvements in satisfaction with an episode of care, but provides no evidence of improvement in mental health symptoms. This may be seen as an end in itself within a political climate that emphasises the importance of increasing patient choice.²⁸ Not all PCTs in England currently employ mental health workers. Only 755 of the 1000 workers specified in the NHS plan⁵ were in post in December 2005. There is also evidence that some PCTs are developing the role in ways that fall outside the Best Practice Guidance, siting workers in secondary care settings, or narrowing the focus of their role.¹¹ The results of this study, suggest that workers based in primary care who follow the suggested roles and responsibilities in the Best Practice Guidance⁷ can offer a valued service for the many patients with common mental health problems. The study findings may have increasing relevance to those interested in using workers to help improve access to psychological therapies²⁹ or as part of a collaborative care model of treating people with depression.

Including practice in the model analysing patient costs had a striking effect. There are a number of potential explanations for this, including variations in case mix, variations in clinical practice, and the effect of chance. In view of its modest size and the complex nature of the intervention itself, this study is probably best conceptualised as an exploratory trial³⁰ which may require replication with a larger number of PCTs and practices.

Funding body

This study was funded by the Heart of Birmingham PCT and the Department of Health. The authors declare that they are independent from those funding the study (reference number 1022)

Ethics committee

Ethical approval was granted by the West Birmingham Ethics committee (reference number 02/07/462)

Trial Registry

National Research Register: Pub ID N0138108914

Competing interests

The authors have stated that there are none

Acknowledgements

We would like to acknowledge all the practices involved in this study and the patients who gave their time to fill in the questionnaires. We would also like to acknowledge the help of Hannah Cooper and Charlotte Mann in collecting study data and Siobhan Conroy in co-ordinating much of the data collection and follow up.