Prospective evaluation of mental health training for occupational health practitioners

Abstract

Background

Occupational health (OH) practitioners need to be confident in identifying and managing mental health problems in the workforce.

Aims

To evaluate the effectiveness of a one-day workshop in improving the knowledge, attitude and confidence of OH practitioners in detecting and managing depression, anxiety, suicide risk, alcohol misuse and drug abuse.

Methods

Interactive mental health workshops for 164 OH practitioners held in five regions in England were evaluated by self-administered questionnaire. Data were collected immediately prior to the workshop (T1), immediately after the workshop (T2) and 4 months following the workshop (T3).

Results

At T1, the response rate was 97% (159/164), 90% at T2 and 63% at T3. The mean improvement in participants’ knowledge was 8% (95% CI 6–10) at T2 compared with T1. The biggest improvement was in participants with no previous training in the management of common mental health problems in the workplace, mean improvement 9% (95% CI 6–12). Participants’ confidence improved in all areas assessed at T2, and the improvement in confidence compared with that at baseline was sustained at 4 months (T3). Participants reported using the knowledge gained in clinical practice in all topic areas covered. Use of knowledge gained at the workshop was significantly higher in those who had had previous training in managing common mental health disorders.

Conclusions

This one-day interactive workshop was a feasible and effective method of improving OH professionals’ confidence, knowledge and application of skills in practice in key areas of mental health.

Introduction

Mental health disorders are the leading cause of sickness absence in most industrialized nations [1]. In the UK, mental disorders now account for between 47 and 61% of the total time off work covered by a doctor’s certificate [2] and 37% of all claims for employment support allowance [3]. Moreover, employees with chronic physical symptoms frequently suffer concomitant common mental disorders, for example depression and anxiety, which if untreated lead to a higher risk of them remaining off work [4,5]. Therefore, it is imperative that occupational health (OH) practitioners who advise employers and employees on individuals’ return-to-work programmes are confident and competent in detecting common mental health problems.

In 2008, a national audit assessed how well National Health Service (NHS) OH practitioners screened for and managed depression in staff who had been on sick leave for more than 4 weeks [6]. The audit found a wide variation of practice. OH practitioners attempted to ascertain if employees had symptoms of depression in 58% of cases and asked about suicidal thoughts or thoughts of self-harm in one of three cases with evidence of distress. Furthermore, they asked fewer than 42% of employees about alcohol or drug use. Most OH practitioners who participated in the national audit expressed a wish for further training in assessing and managing depression, suicide risk, and alcohol and drug misuse in their patients. In response to this identified need, a series of one-day interactive workshops were developed specifically for OH practitioners with the aim of improving their ability to recognize and manage common mental disorders. The purpose of this study was to evaluate the effectiveness of the workshops in improving the attendees’ skills and confidence in undertaking a mental health assessment in the workplace and to identify which groups of OH practitioners would benefit the most.

Methods

The workshops were advertised in OH professional society e-newsletters and were held in five regional centres in England (workshop registration forms, and pre- and post-course questionnaires are available as Supplementary data at Occupational Medicine Online). They were funded by NHS Plus (a Department of Health project to improve the quality of OH practice), but a nominal charge was made for attending to discourage non-attendance. They lasted 6 hours, and a maximum of 35 participants attended each workshop. The teaching materials were developed by experienced consultant occupational psychiatrists (SH, MH) and consultant occupational physicians (IM, AH) with input from a senior lecturer in medical education. The workshops were highly interactive, comprising lectures, small group discussion and problem-based cases. These methods were chosen as they have been shown to aid retention and use of knowledge gained [7,8]. The six topics covered were assessment and OH management of depression, anxiety, suicide, alcohol misuse and drug abuse and how to access local psychiatric services. The workshops were delivered by the course developers. Information on accessing local psychiatric services was given by a psychiatrist local to each region. The teaching was supported by an accompanying workshop manual.

The evaluation was designed to assess the first three levels of educational evaluation described by the Kirkpatrick model [9]:

1. satisfaction with the delivery of the workshops (learner satisfaction);

2. confidence, knowledge and attitudes relating to screening for and diagnosing specific conditions (learning outcomes);

3. increased confidence and reported use in clinical practice of the skills learnt in the workshops (performance improvement).

The study size was pragmatic, based on the number of workshop participants. Evaluation data were collected from participants at three points: immediately prior to the workshop (T1), immediately after (T2) and four months following the workshop (T3). Participants acted as their own controls. T1 and T2 questionnaires were administered on the day of the workshop. A hard copy of the T3 questionnaire was sent to participants 4 months after the workshop. After 2 weeks, non-responders were sent a single reminder with an electronic version of the questionnaire to complete. Questionnaire responses were anonymous to the researchers but were linked by an alphanumeric code. Linkage of the questionnaire to the participant’s name and contact details was undertaken only by the course administrator, for the purpose of sending out the follow-up questionnaire. Participation in the study was voluntary, and workshop participants were informed of the evaluation prior to registration.

The 37-item, self-completed T1 questionnaire collected data on the demographic characteristics of the participants, including previous training in and knowledge about common mental health problems and 12 questions on attitudes and confidence in assessing and managing employees with mental health problems or drug and alcohol misuse. The knowledge section contained 20 true/false statements covering the full range of clinical topics presented in the workshop. The knowledge score was measured by calculating the percentage of correct answers given by the participant. The knowledge questions were developed by SH, MH and IM and piloted during previous mental health teaching sessions with OH nurses, to ensure they were understandable and pitched at an appropriate level. The confidence of attendees in managing workers with alcohol problems was tested using the two adequacy questions from the short alcohol problems perception questionnaire [10]. The same two questions were adapted to ask about confidence in managing workers misusing drugs. The confidence of attendees in assessing suicide risk was measured using the two ‘responsibility of the clinician’ questions from the attitude to suicide prevention scale [11]. The T2 questionnaire included seven questions on learner satisfaction (seven-point Likert scale) and the same knowledge, confidence and attitude questions as the T1 questionnaire. The T3 questionnaire contained the same confidence and attitude questions and also included questions on application of the knowledge gained at the workshops using seven-point Likert scale. This included self-perception of change in practice as a result of attending the workshop and direct questions on use of the knowledge gained at the workshop in day-to-day practice.

Data were analysed using STATA version 11.1 [12]. Analysis was by parametric and non-parametric methods for paired data. Baseline (T1) demographic predictors of participation at T2 and T3 were sought using Chi-square tests. The total score for pre- and post-test knowledge was calculated for each participant. The mean pre- and post-test scores were then compared using 95% confidence intervals. Participants were classified as having a positive attitude and being confident if they scored themselves as 5–7 in the seven-point Likert scale for the corresponding questions. Based on this classification, the percentage of participants feeling confident and having a positive attitude concerning a variety of mental health topics were compared at T1, T2 and T3, again using 95% confidence intervals. The impact of the training on practice was measured by summing the seven-point Likert scores in the six implementation questions in the T3 questionnaire. Therefore, the resultant impact score varied between a minimum score of 6 and a maximum of 42. We explored the difference in the impact scores by the demographic characteristics of the attendees using one-way analysis of variance tests. Associations between change in knowledge, confidence and attitude with impact of the training on practice were explored using linear regression modelling.

Results

Of the 164 workshop participants, 159 (97%) completed some of the questionnaires at T1; of whom, 139 (84%) fully completed the sections enquiring about pre-course confidence and attitudes. At T2, 147 (90%) participants completed questionnaires and 103 (63%) returned useable questionnaires at T3. The demographic characteristics of the participants at each of the three points of data collection are shown in Table 1. Responders at follow-up did not differ from non-responders in terms of baseline demographics or previous training (data not shown). The majority of participants had not received prior training in mental health assessment although around 4 in 10 had previously had some training in screening for depression and anxiety. Overall satisfaction with the workshop was very high, with 100% of participants indicating levels of satisfaction of 5–7 on a seven-point Likert scale (a score of 7 being highly satisfied and a score of 1 being highly dissatisfied).

Table 1.

Demographic characteristics of the study participants at the three time points of the data collection

	Pre-training knowledge data, T1 (n = 159)	Post-training knowledge data T2 (n = 147)	Pre-training attitudes and confidence data T1 (n = 139)	Follow-up attitudes and confidence data T3 (n = 103)
	n (%)	n (Response rate in %)	n	n (Response rate in %)
Occupation
Nurse/nurse manager	132 (83)	122 (92)	115	81 (70)
Doctor	18 (11)	16 (89)	16	15 (94)
Other	9 (6)	9 (100)	8	7 (88)
Years of experience in occupational health
<5	31 (20)	31 (100)	25	17 (68)
5–10	52 (33)	47 (90)	47	36 (76)
>10–15	28 (18)	26 (93)	25	18 (72)
>15	48 (30)	43 (90)	42	32 (76)
Agea
<25	25 (16)	23 (92)	22	18 (82)
25–39	109 (69)	103 (95)	97	70 (72)
40–54	24 (15)	20 (83)	19	15 (79)
Sex
Male	22 (14)	21 (96)	21	18 (86)
Female	137 (86)	126 (92)	118	85 (72)
Previous training for screening for depression/anxiety
No/don’t know	96 (60)	87 (91)	84	64 (76)
Yes	63 (40)	60 (95)	55	39 (71)
Previous training in assessing risk of suicidea
No/don’t know	125 (79)	113 (90)	108	79 (73)
Yes	33 (21)	33 (100)	30	24 (80)
Previous training in assessing alcohol use in employees
No/don’t know	120 (76)	112 (93)	104	74 (71)
Yes	39 (25)	35 (90)	35	29 (83)
Previous training in assessing drug use in employees
No/don’t know	127 (80)	119 (94)	111	81 (73)
Yes	32 (20)	28 (88)	28	22 (79)

^aOne missing value.

The training resulted in an improvement in knowledge immediately post-workshop. As illustrated in Table 2, the mean improvement in participants’ knowledge was 8% following the workshop. The largest improvement was in those who had no previous training in the management of common mental health problems although this increase was not significantly different from those who had previous training. Further analysis was undertaken of the change in knowledge pre- and post-training for each question. No clear pattern emerged, and there was no one topic area that was the main driver for change.

Table 2.

Core knowledge pre- and immediately post-training

	n	Mean (SD) pre-training knowledge score, % (95% CI)	Mean (SD) post-training knowledge score, % (95% CI)	Mean improvement in knowledge score between post- and pre-training score, % (95% CI)
All	147	56 (54–57)	64 (62–66)	8 (6–10)
Years in OH
10 years or less	78	55 (52–57)	63 (60–66)	9 (6–11)
More than 10 years	69	57 (55–60)	65 (63–68)	8 (5–12)
Any previous training
No/don’t know	72	54 (52–56)	63 (60–66)	9 (6–12)
Yes	75	58 (55–60)	65 (62–68)	8 (5–10)

Participants’ confidence improved in all areas assessed at T2, and the improvement in confidence compared with that at baseline was sustained at 4 months following the workshops. The largest improvement in confidence was in appropriately advising patients on drugs and alcohol and their effect on work and in assessing suicide risk (Table 3). The training also improved the participants’ attitude to taking responsibility for suicide prevention, when appropriate, in an OH consultation.

Table 3.

Confidence and attitude pre and post training

	Pre-training, T1 % (n = 139)	Post trainingb, T2 % (n = 139)	Follow-up, T3 % (n = 103)	Difference between pre-training and follow-up, % (95% CI)
Confidencea: ‘I feel confident in …’
‘… screening patients for symptoms of a depressive illness’	37	80	73	36 (24–48)
‘… assessing for suicide risk’	10	60	52	42 (30–53)
‘… assessing patients who present with symptoms of anxiety’	46	82	74	28 (16–41)
‘… assessing alcohol use and alcohol related problems’	34	78	67	33 (21–45)
‘… that I can appropriately advise my patients about drinking and its effect on their work’	45	85	85	41 (29–52)
‘… assessing the use of recreational drugs and any drug related effects’	15	54	54	40 (27–52)
‘… in advising my patients about drugs and their effect on work’	22	62	70	48 (36–60)
‘… in my knowledge and underst and ing of local psychiatric services’	24	51	52	27 (15–39)
Attitudea:
‘An occupational health professional could be a useful person to support depressed patients’	78	79	82	4 (–5 to 13)
‘Suicide prevention is my responsibility’	7	28	29	22 (12–33)

^aParticipants were classified as having a positive attitude and being confident if they scored themselves as 5, 6 or 7 in the corresponding 7 scale questions.

^bContains some missing values.

Participants reported use of the knowledge they gained at the workshop in day-to-day clinical practice in all six topic areas covered in the training programme. There were no significant differences between the total impact score by occupational group, years of experience in OH, age and gender. However, the mean impact score was significantly higher in those who had previous training in managing common mental health disorders in the workplace (Table 4). Univariate analysis showed that after adjusting for previous training, the impact on day-to-day practice was associated with an improvement in confidence in dealing with mental health problems but not with an improvement in knowledge or attitude towards mental health disorders (Table 5).

Table 4.

Use of knowledge gained at workshop in day-to-day clinical practice

	Mean (SD) impact score (maximum 42)	P-value
Occupation
Nurse/nurse manager	27.1 (6.7)	NS
Doctor	31.7 (6.0)
Other	28.3 (9.5)
Years of experience in occupational health
<5	30.3 (8.9)	NS
5–10	27.2 (6.7)
>10–15	28.4 (6.6)
>15	26.9 (6.3)
Age
<25	28.2 (6.7)	NS
25–39	27.4 (7.6)
40–54	29.1 (4.5)
Sex
Male	30.4 (6.2)	NS
Female	27.2 (7.1)
Any previous training
No/don’t know	25.9 (7.6)	0.01
Yes	29.6 (5.9)

NS, not significant. P > 0.05).

Table 5.

Univariate associations between impact score at follow-up phase with change in knowledge (T2–T1), confidence (T3–T1) and attitude (T3–T1)

	Unadjusted		Adjusted for previous training (any)
	Regression coefficient (95% CI)	P-value	Regression coefficient (95% CI)	P-value
Change in knowledgea	0.1 (−0.5 to 0.7)	NS	0.1 (−0.5 to 0.7)	NS
Change in confidence	0.2 (0.1 to 0.3)	<0.05	0.2 (0.4 to 0.3)	<0.01
Change in attitude	0.1 (−0.5 to 0.8)	NS	0.1 (−0.5 to 0.7)	NS

^aContains eight missing values.

Discussion

Evaluation of this interactive mental health training workshop for OH professionals has shown it to be effective in improving participants’ knowledge, increasing their confidence in assessing employees with common mental health problems and improving their attitude towards the subject. Furthermore, the increased confidence was sustained at 4 months following the training, and participants reported applying the knowledge that they had gained in their clinical practice.

Despite the novelty of our findings, there are some important limitations that should be considered. Firstly, the participants in this study were self-selected and therefore might have been more interested in the subject and keener to learn than a random sample of OH practitioners. However, the workshops were aimed at OH practitioners who felt that their clinical skills required improvement, and as such the results should therefore be applicable to similar workshops, advertised in a similar manner, as self-perceived needs are a key aspect of ensuring the success of further medical education [13]. While there might have been some attrition bias in our sample, we had a high response rate to the questionnaires and the demographic characteristics of responders and non-responders at 4 months were similar. Finally, we acknowledge that some, but not all, of our outcome measures were based on self-report and utilized measures that have not been subject to wide-scale validation.

Previous studies have shown that short mental health training courses can improve the confidence of general practitioners in their abilities to deal with depression and common mental disorders [14,15]. To the best of our knowledge, this is the first time a mental health training package has been shown to be effective among OH professionals. Although our evaluation was not designed to assess whether the workshop led to an improvement in outcomes for employees who consulted OH practitioners, it is known that mental health training in general practitioners has an impact on patient care [16].

Importantly, our evaluation showed that the workshops led to an improvement in knowledge of common mental health disorders and confidence in dealing with them in practice. This improvement in knowledge and confidence was independent of the occupation or seniority of the participants. This suggests that it is appropriate to continue to deliver workshops to multidisciplinary groups of OH practitioners, regardless of their level of experience or previous training. Not surprisingly, the largest improvement in knowledge following the training was in those with no prior training in common mental health disorders in the workplace, but participants who had received previous training in mental health appeared to be more readily able to implement their training in practice than those with no previous training. This may be the result of refreshing previously acquired knowledge. It is difficult to assess how meaningful the clinical practice impact scores are, but our findings suggest that more than one training session may be needed to change practice. Use of the skills acquired in day-to-day practice was associated with improved confidence of the participants. It may be that using the skills acquired in the workshop led to improved confidence or that those participants who felt most confident were more ready to apply these skills in practice. A striking feature of the workshops was that they resulted in improved confidence that was maintained at 4 months. We would expect participants to become more confident in their skills as they put them to use in practice.

Mental health disorders are becoming an increasingly important part of OH professionals’ caseload. Not only mental disorders are now the most common health reason for work absenteeism but also cases involving mental health disorders are often more complex and time-consuming. It is therefore likely that there will be increasing demand for interventions aimed at better equipping OH professionals to identify and manage common mental health problems. We conclude that this one-day interactive workshop on common mental health disorders was an efficient, feasible and effective method of improving OH professionals’ confidence, knowledge and application of skills in practice in a number of key areas of mental health management.

Key points

• Interactive workshop-style teaching is effective in improving the knowledge, confidence and attitude of occupational health professionals in key areas of identifying and managing common mental health problems in occupational health practice.

• This style of workshop is effective in improving knowledge, confidence and attitude towards mental health issues in participants from a mixture of occupational health professional groups and of varying levels of seniority.

• Participants reported significant changes in their clinical practice 4 months after attending the workshops.

Conflicts of interest

None declared.

Funding

National Health Service Plus; Guy’s and St Thomas’ National Health Service Foundation Trust (to I.M.); National Institute of Health Research Biomedical Research Centre Nucleus (M.H.).