Evaluation of a primary care adult mental health service: Year 2

Abstract

Aims This study aimed to examine the effectiveness of a primary care adult mental health service operating within a stepped care model of service delivery.

Methods Supervised by a principal psychologist manager, psychology graduate practitioners provided one-to-one brief cognitive behavioural therapy (CBT) to service users. The Clinical Outcomes in Routine Evaluation-Outcome Measure (CORE-OM) was used to assess service user treatment outcomes. Satisfaction questionnaires were administered to service users and referring general practitioners (GPs).

Results A total of 43 individuals attended for an initial appointment, of whom 19 (44.2%) completed brief CBT treatment. Of the 13 service users who were in the clinical range pre-treatment, 11 (84.6%) achieved clinical and reliably significant improvement. Of the six service users who were in the non-clinical range pre-treatment, three (50%) achieved reliably significant improvement. Both service users and GPs indicated high levels of satisfaction with the service, although service accessibility was highlighted as needing improvement.

Conclusion The service was effective in treating mild to moderate mental health problems in primary care. Stricter adherence to a stepped care model through the provision of low-intensity, high-throughput interventions would be desirable for future service provision.

Introduction

In the Republic of Ireland, approximately 25–33% of primary care attendees have mental health problems.^1,2 As only around 10–20% of these service users will be referred on to specialist mental health services,^1,3 primary care is the only service of contact for most. Treatment in primary care is often inappropriate, with some service users receiving minimal treatment (e.g. empathetic listening), while others are inappropriately provided with pharmacological treatment. Providing the latter for mild to moderate mental health disorders is now considered inconsistent with best practice guidelines.^4,5 In such a context, there appears to be a need to develop primary care mental health services to ensure that service users are provided with accessible and appropriate treatment.

The Improving Access to Psychological Therapy (IAPT) programme (www.iapt.nhs.uk) developed by the National Health Service (NHS) in the UK is one example of a successful initiative that provides accessible primary care mental health treatment to service users. Based on a stepped care model of service delivery,⁶ IAPT aims to provide adults who have mild to moderate mental health problems with interventions appropriate to the severity of their problems. Using systematic monitoring, service users can respectively ‘step up’ or ‘step down’ to more or less intense interventions depending on their clinical progress.⁷ With a reach of approximately 300 000 service users per annum,⁷ recovery rates across IAPT sites have met predetermined targets, ranging from 42% to 55%.⁸

Based on the stepped care model of service delivery (see Figure 1),⁹ a primary care adult mental health service was established in Roscommon, Ireland in late 2009. Similar to IAPT, this service aims to provide accessible, low-intensity psychological interventions for adults with mild to moderate mental health problems. Fulfilling a role similar to that of the IAPT low-intensity workers/psychology wellbeing practitioners,¹⁰ this service was provided by a team of mental health practitioners, each of whom had an academic postgraduate qualification in psychology, who were supervised by a principal psychologist manager.

Figure 1.

Stepped care model of service delivery.⁹

The way in which this service developed during its pilot year has been described in detail elsewhere.¹¹ Significant developments included introducing a ‘walk-in’ self-referral service and a bibliotherapy service (in collaboration with the local libraries), as well as the creation of formal links with GPs and other local mental health services. Given its effectiveness for both depression and anxiety in primary care,¹² the primary mode of intervention was brief CBT. Typically defined as ten sessions or less,¹³ the service provided six sessions of CBT. As brief CBT was offered to all service users regardless of their presentation, an intra-service stepped care model of treatment was not provided. Instead the service attempted to provide a single step of low-intensity treatment below the higher-intensity steps in secondary care.

The aim of the present study was to evaluate the Roscommon Primary Care Adult Mental Health Service in its second year of operation. As with the pilot year, brief CBT was offered to all service users, with the option of engaging in bibliotherapy or psychoeducational groups prior to one-to-one treatment also being available. The specific objectives of the service evaluation were to examine the clinical effectiveness of the brief CBT provided to all service users, and to assess both service user and GP satisfaction.

Methods

Evaluation design

At the initial assessment session, all service users were given the opportunity to provide consent for their data to be used for the purposes of service evaluation or research. All service users were informed that their data would be treated anonymously and stored in line with the Data Protection Act 2003.

The clinical outcomes of service users who were provided with one-to-one brief CBT were examined using a repeated-measures design. Psychological distress was measured using the CORE-OM at pretherapy and post-therapy. A satisfaction questionnaire was administered to service users who completed one-to-one therapy. A separate satisfaction questionnaire was sent to all GPs who made referrals to the service.

Service users

An open referral system including self-referral was employed to facilitate ease of access to the service. Self-referrals could be made through the ‘walk-in clinic’ or ‘call-back service’ provided by the service. Adults suitable for the service were those with mental health problems in the mild to moderate range, and those with mental health problems suitable for treatment in primary care. Exclusion criteria for the service included those individuals with schizophrenia and related disorders, cognitive impairment/dementia, active risk of suicide, violence or self-neglect, eating disorders, substance abuse, personality and behavioural disorders, and chronic depressive and anxiety disorders, and adults who had experienced abuse or neglect during childhood.

Treatment

One-to-one brief CBT was administered to all service users. This treatment consisted of an initial assessment session followed by five sessions of CBT. Based on an initial formulation and service user input, the treatment focused on specific, achievable goals. Treatment strategies commonly implemented included the use of thought diaries, cognitive restructuring, and behavioural experiments (e.g. exposure therapy, behavioural activation). The inter-session period for treatment was approximately 2 weeks.

Training of practitioners

At the start of the service year, practitioners attended workshops run by two clinical psychologists. These focused on the assessment and treatment of adult mental health disorders. The local principal psychologist manager provided group supervision on a fortnightly basis. Located in the same building as the practitioners, he was available for informal consultations when on site, and was contactable by telephone when off site.

Measures

Psychological distress was measured using the 34-item self-report Clinical Outcomes in Routine Evaluation-Outcome Measure (CORE-OM).¹⁴ This measure has four subscales, namely subjective well-being, problems/symptoms, life/social functioning, and risk. As has been recommended by Evans,¹⁵ the 28 non-risk items were used to measure psychological distress.

Service user and GP satisfaction questionnaires were developed for the evaluation. The former profiled satisfaction with waiting times, the number of appointments provided and the overall service. The GP satisfaction questionnaire profiled satisfaction across several service domains, including communication, ease of referral, management of referrals, accessibility, level of care provided to service users, and range of presentations seen.

Data analysis

The clinical outcomes of service users based on their non-risk CORE-OM scores were evaluated using the reliable and clinical change index.^16,17 Clinical change is demonstrated when an individual’s score moves from the clinical range to the non-clinical range (criterion C). Reliable change is demonstrated when an individual’s score shows a statistically significant change. For the current service evaluation, a service user was considered to be ‘recovered’ if they showed both clinical and reliable change. Derived from normative UK data,¹⁴ the cut-off points used for clinical and reliable change were 1.36 for males and 1.5 for females.

Results

Service users

The service received a total of 112 referrals (78 female and 34 male) during the 2010–2011 service year, representing a 35.7% increase in referrals compared with the previous year (n = 72).¹¹ The characteristics of these referrals are presented in Table 1. In total, 65 GPs made 100 referrals to the service. Forty-three referrals (38.4%) attended for an initial appointment. The average waiting time for initial assessment was 9.5 weeks.

Table 1.

Characteristics of referrals (n = 112)

Gender	Male	Female
n (%)	34 (30.4%)	78 (69.6%)
Age (years)	16–20	20–35	35–50	50–65	≥ 65
n (%)	13 (11.6%)	34 (30.4%)	32 (28.6%)	24 (21.4%)	9 (8%)
Referral source	GP	Walk-in	Call-back
n (%)	100 (89.3%)	9 (8%)	3 (2.7%)

Based on GP referral letters, Table 2 indicates the primary mental health problems that led to GP referrals (n = 100). The most common reasons for referral were depression/low mood (34%), anxiety (unspecified; 28%) and stress (10%). A comorbid mental health problem was present for 21% (n = 21) of these referrals.

Table 2.

Proportion of GP referrals (n = 100) with mental health problems

Presenting problem	n (%)	Presenting problem	n (%)
Depression/low mood	34 (34%)	Eating-related issues	2 (2%)
Anxiety (unspecified)	28 (28%)	Interpersonal problems	2 (2%)
Stress	10 (10%)	Rehab support	1 (1%)
Panic disorder	5 (5%)	Bipolar disorder	1 (1%)
Anger	5 (5%)	Generalised anxiety disorder (GAD)	1 (1%)
Bereavement	3 (3%)	Sexual dysfunction	1 (1%)
Low self-esteem	3 (3%)	Psychosomatic problems	1 (1%)
PTSD	3 (3%)

Clinical effectiveness of the brief CBT intervention

Of the 43 service users who attended for an initial appointment, 19 completed the six sessions of brief CBT, representing a completion rate of 44.2%. Based on non-risk CORE-OM scores, the clinical outcomes of these treatment completers are presented in Table 3. Of the 13 service users in the clinical range pretherapy, 11 (84.6%) showed clinically and reliably significant improvement. Of the six service users in the non-clinical range pre-therapy, three (50%) showed reliable improvement. Follow-up data at 3 months were available for eight service users, all of whom had achieved clinical and reliable improvement during treatment. Six (75%) of these service users showed maintenance of clinical and reliable improvement at follow-up.

Table 3.

Number of service users showing reliable and clinical change

	Reliable improvement
Clinical change (criterion C)	Yes	No	Total (%)
Clinically significant change	11	0	11 (57.9%)
Failed to achieve clinically significant change despite high enough initial score	1	1	2 (10.5%)
Started better than criterion for clinically significant change	3	3	6 (31.6%)
Total (%)	15 (78.9%)	4 (21.1%)	19

Service user and GP satisfaction

Of the 19 service users who completed treatment, 16 (84.2%) completed a service user satisfaction questionnaire. Regarding the waiting time for an initial appointment, 37.5% (n = 6) of the service users rated it as ‘very good’, 56.3% (n = 9) rated it as ‘OK’ and 6.3% (n = 1) rated it as ‘bad.’ Most (68.8%; n = 11) were satisfied with the number of appointments provided, while 31.3% (n = 5) indicated that they would have liked more appointments. All of the respondents indicated satisfaction with the overall service, with 75% (n = 12) indicating that they were ‘very satisfied.’ All of the respondents indicated that they would use the service again if they had similar difficulties.

In total, 21 of the 65 GPs (32.3%) who made referrals to the PCAMHS returned completed satisfaction questionnaires. As profiled in Table 4, the majority of GPs rated the service as either ‘good’ or ‘very good’ on all of the domains except for accessibility, and 19 of the 21 GPs (90.5%) indicated that they would be ‘likely’ to refer patients to the service in the future.

Table 4.

GP ratings of service (n = 21)

Service dimension	Respondents (%)	Very poor	Poor	Adequate	Good	Very good
Communication	95.2	0	15%	5%	40%	40%
Ease of referral	100	0	9.5%	19%	38.1%	33.3%
Management of referrals	95.2	0	20%	10%	40%	30%
Accessibility	85.7	5.6%	27.8%	27.8%	16.7%	22.2%
Level of care provided to clients	85.7	0	5.6%	5.6%	47.7%	38.9%
Range of clients seen	76.2	0	0	12.5%	50%	37.5%

Discussion

The current service aimed to provide accessible and effective treatment for adults with mild to moderate mental health problems. The extent to which the service achieved these aims will be discussed, and ways in which the service could be improved and expanded going forward will be considered.

Clinical effectiveness of one-to-one therapy

The 84.6% recovery rate that was observed compares favourably with that reported for CBT interventions within IAPT,¹⁸ and for primary care psychological therapies in the UK.¹⁹ However, due to a number of factors, it is difficult to generalise the findings of this study beyond the sample that was studied. First, the sample was small (n = 19). Future evaluations would benefit from using larger sample sizes, or a meta-analysis could be conducted across successive years. Secondly, the clinical outcomes of the 44.2% of treatment completers may not generalise to the broader group of ‘treatment seekers.’^20,21 This completion rate would be considered low for a CBT intervention,²² and future services need to employ strategies to ensure a higher completion rate, such as the use of motivational interviewing.²³ Thirdly, as a proportion of service users had engaged in bibliotherapy and/or psychoeducational groups prior to brief CBT, it is possible that their clinical outcomes were influenced by an interactional effect (i.e. bibliotherapy and psychoeducational groups increased the effectiveness of brief CBT).

Service user and GP satisfaction

Despite service user and GP satisfaction data indicating high levels of satisfaction with the service, there may have been a positive response bias, given that less than 50% of the total service attendees and GPs responded to the satisfaction questionnaires. In addition, compared with the use of qualitative methodologies, satisfaction questionnaires often fail to detect service users’ criticisms.²⁴ The finding that there were lower levels of satisfaction with the accessibility/waiting time of the service suggests that this area is in need of improvement. This could be achieved by a combination of employing more staff, providing services in more geographically remote clinics, and providing low-intensity, high-throughput interventions as first-line treatments (e.g. group psychoeducational classes, guided self-help, computerised CBT).²⁵

Future service developments

In seeking to establish a more expansive stepped care service, this primary care service needs to develop formal referral and consultation links with local secondary care mental health services. The non-acceptance by the latter of non-GP referrals hindered shared care arrangements and the safe and efficient transfer of service users across the primary–secondary care service interface. Similarly, support from secondary care services would also have empowered the primary care practitioners to deal more appropriately with the sometimes complex referrals received.²⁶

In order for this service model to receive future investment, it will be necessary to demonstrate not only the clinical effectiveness of this service but also its cost-effectiveness, as has been done in the UK with similar services.²⁷ Therefore future evaluations could consider the costs of the service (e.g. practitioner salaries), the economic benefits of treatment (e.g. employment rate of service users), and the costs relative to service user outcomes (i.e. cost utility).

Conclusions

This evaluation demonstrated the effectiveness of this primary care adult mental health service, both in terms of clinical effectiveness, and in terms of service user and GP satisfaction. The results lend support to the proposal that this service model be expanded to more regions in order to meet the existing need for psychological treatment in primary care. To increase both its accessibility and its cost-effectiveness, any such service would benefit from greater adherence to the stepped care model.