Assessing Mental Health First Aid Skills Using Simulated Patients

Abstract

Objective. To evaluate mental health first aid (MHFA) skills using simulated patients and to compare self-reported confidence in providing MHFA with performance during simulated patient roleplays.

Methods. Pharmacy students self-evaluated their confidence in providing MHFA post-training. Two mental health vignettes and an assessment rubric based on the MHFA Action Plan were developed to assess students’ observed MHFA skills during audio-recorded simulated patient roleplays.

Results. There were 163 students who completed the MHFA training, of which 88% completed self-evaluations. There were 84% to 98% of students who self-reported that they agreed or strongly agreed they were confident providing MHFA. Postnatal depression (PND) and suicide vignettes were randomly assigned to 36 students. More students participating in the PND roleplay took appropriate actions, compared to those participating in the suicide role-play. However, more students participating in the suicide role play assessed alcohol and/or drug use. Ten (71%) participants in the PND roleplay and six (40%) in the suicide roleplay either avoided using suicide-specific terminology completely or used multiple terms rendering their inquiry unclear.

Conclusion. Self-evaluated confidence levels in providing MHFA did not always reflect observed performance. Students had difficulty addressing suicide with only half passing the suicide vignette and many avoiding suicide-specific terminology. This indicates that both self-reported and observed behaviors should be used for post-training assessments.

INTRODUCTION

Mental Health First Aid (MHFA) is defined as “the help provided to a person who is developing a mental health problem, experiencing the worsening of an existing mental health problem or in a mental health crisis…until appropriate professional help is received or the crisis resolves.”¹ Developed in Australia in 2001, the MHFA program is designed for all members of the community, with approximately 1% of the Australian population having completed MHFA training.^1,2 MHFA training is now embedded in various health degrees, offered in over 20 countries and recommended by various organizations, such as the Quality Care Pharmacy Program (QCPP), a community pharmacy quality assurance program in Australia.^3,4 MHFA Australia has developed a tailored course called “Blended MHFA in the Pharmacy” for pharmacists and pharmacy staff, which satisfies QCPP requirements.⁵ In Australia, The Pharmaceutical Society’s Mental Healthcare Framework recognizes pharmacists as primary health care professionals who have an important role to play within mental health care.⁶ Globally, the International Pharmaceutical Federation has urged members to include pharmacists as part of their “human resource development policy” so that “an increase by 20% of service coverage for severe mental disorders can be achieved.”⁷ Hence, it is essential that pharmacists are equipped with the necessary skills to help patients with mental health problems.

MHFA has been shown to have a positive impact on mental health knowledge, attitudes and self-reported behaviors, among various population groups, including pharmacy students.^8-11 Studies have used different self-report methods to assess participants’ behaviors after completing MHFA training, such as confidence, intentions, and help provided.^8-10,12-17 However, research on observed performance is lacking. Psychologists have noted the shift from direct observation to “introspective self-reports, hypothetical scenarios and questionnaire ratings” to measure self-reported behaviors.¹⁸ The increasing use of questionnaires asking people “what they have done, will do or would do” may be due to ethical and feasibility concerns;¹⁸ however, various studies have illustrated discordance between self-reported and observed behaviors. For example, Tamburrino and colleagues found that medical students’ self-reported preferences on an inventory did not reflect their abilities during interviews with simulated outpatients.¹⁹ Jenner and colleagues reported a difference between health care professionals’ observed hand hygiene behaviors and their self-reported practice.²⁰ Simulated patients provide a means of evaluating students’ ability to “show how” they use their knowledge.²¹ This form of assessment assumes that students “know” the content and “know how” to apply the skills they have learnt.²¹ “Showing how” knowledge is applied is the third step in Miller’s framework for clinical assessment.²¹ It is only inferior to the last step in the pyramid entitled “does,” which evaluates the “actions” taken by the student when he/she is working independently.²¹ However, this final step of assessment is often “difficult to measure accurately and reliably.”²¹

In light of this evidence, it is essential to incorporate observed behavioral measures of performance in studies that assess behaviors.¹⁸ Incorporating performance feedback in simulated patient studies is also recommended.²² Hence, this study has two objectives. The first is to evaluate the impact of MHFA training on pharmacy students’ communication and behavioral skills during roleplays with simulated patients, and to provide immediate performance feedback on these skills. The second is to compare students’ self-reported MHFA skills with observed performance during simulated roleplays.

METHODS

Fourth-year bachelor of pharmacy (BPharm) students in the final semester of their degree at The University of Sydney completed the MHFA training course, within their curriculum, delivered by an accredited MHFA instructor and registered pharmacist. Post-training, MHFA Australia emailed students a voluntary self-evaluation form.

Registered pharmacists, mental health researchers and an accredited MHFA instructor developed two depression vignettes based on the Diagnostic and Statistical Manual of Mental Disorders, 5th Edition (DSM-5) criteria.²³ One simulated patient was a first-time mother having difficulty breastfeeding and the other was a person struggling with a partner’s death and having suicidal thoughts (Table 1). These vignettes were enacted by the tutors during the simulated patient roleplays. The simulated roleplay in this study was part of the students’ Professional Practice unit, which required each student to perform eight different skills: handling a request for an over-the-counter medicine, handling a first-aid situation, providing discharge counseling to a patient going home from the hospital, answering a drug information query, dealing with a medication-related ethical dilemma, taking a patient’s medication history and reconciling the medicines list, completing a Home Medicines Review (HMR), and providing part of a Disease Management Service. An HMR “involves the patient, their general practitioner (GP), an accredited pharmacist and regular community pharmacy.”²⁴ During an HMR, “the pharmacist visits the patient at their home, reviews their medicine routine and provides their GP with a report” and a medicine management plan is agreed upon by the GP and the patient.²⁴ On the other hand, “disease management is an approach to patient care that coordinates medical resources for patients across the entire health care delivery system.”^25,26

Table 1.

Key Elements of the Mental Health Vignettes

There were 10 versions of each skill to be performed, rendering a total of 80 scenarios. Two of the first-aid scenarios, described previously in Table 1, were designed to allow students to implement the principles of MHFA. Other first-aid scenarios included asthma first aid, physical first aid, angina, hypoglycemia, wound care or anaphylaxis. Two pharmacists who had completed MHFA training acted as the simulated patients and provided performance feedback.

After MHFA training, the 10 first-aid scenarios were randomly allocated through simulated patient roleplay during weekly tutorials. Students were not informed of their allocations prior to the tutorial. The 30-minute interaction involved an audio-recorded simulated patient roleplay, tutor assessment, self-assessment and on-the-spot performance feedback from an experienced tutor who had completed MHFA training and was a registered pharmacist in Australia.

A purpose-designed assessment rubric was developed based on the MHFA Action Plan (Table 2) and a scoring system developed by MHFA researchers.²⁷ Each item in the rubric was scored from 0 to 2 (0 points = incorrect, inappropriate or missing behaviors, 1 point = partial demonstration of skills, 2 points = full demonstration of appropriate skills).²⁷ Allocation of scores for each item was determined by consensus of all four authors, all of which are registered pharmacists and experienced pharmacy educators, one of which is an accredited MHFA instructor and two of which have completed MHFA training and are mental health researchers. The resulting 12-item rubric mapped to the MHFA Action Plan and was adapted for a community pharmacy context (Table 2). This rubric was used by the simulated patient to assess students’ MHFA skills during the roleplays (audio recorded), to facilitate performance feedback by the simulated patient and for students to self-assess their own performance. To pass the assessment, students had to get at least 12 out of 24 marks and assess for suicidal thoughts. Other actions were also required to pass the assessment, based on MHFA guidelines.⁴ For the suicide vignette, the simulated patient admitted to having suicidal thoughts and a plan in place. Therefore, the student had to take the appropriate action by immediately referring the simulated patient to the appropriate professional help, such as a general practitioner (GP) and ensure that the simulated patient was not left alone (eg, accompany him/her to the GP’s clinic), to pass the assessment. For the PND vignette, it was necessary to inquire about suicidal thoughts; however, the simulated patient did not have current suicidal thoughts. Therefore, to pass the assessment and competently provide MHFA, it was necessary for the student to refer the simulated patient to appropriate professional help (eg, GP).

Table 2.

Assessment Rubric Based on the Mental Health First Aid Action Plan – ALGEE ⁴

The post-training self-evaluations were analyzed using SPSS 22 (IBM, Armonk, NY). After data collection was completed, the same tutor who performed the 30 simulated patient role-plays listened to all 36 audio-recordings and re-scored them all. The results reported in this article are based on the scores of the audio-recordings to ensure consistency among the outcomes. They were also categorized as competent (or not) for key behavioral outcomes by assessing their ability to provide ALGEE, which stands for approach, assess and assist with any crisis; listen non-judgmentally; give support and information; encourage appropriate professional help; encourage other supports.⁴

For any missing or unclear recordings, tutor marks were assumed to be accurate. Any discrepancies or ambiguities were discussed with the research team to reach a decision by consensus. Ethical approval was received from The University of Sydney Human Research Ethics Committee.

RESULTS

Between July and August 2015, 163 students completed MHFA training. Following the training, 143 (87.7%) students completed the self-evaluation form. Students’ responses to the self-evaluation items are displayed in Table 3.

Table 3.

Students’ Self-Reported Confidence in Providing ALGEE (n=143)

Simulated Roleplay

One tutor was the simulated patient in all 18 PND roleplays and 12 suicide roleplays (30 in total). This tutor also re-scored all of the roleplays based on the audio-recordings, upon which the results of this article are based. The second tutor was the simulated patient for six of the suicide roleplays. Thirty-six students were randomly assigned to receive either the PND (n=18) or the suicide (n=18) vignette. During the PND vignette, one assessment was not completed and excluded in the data analysis. Hence, data were only available for 17 of the 18 PND roleplays. Based on these criteria, 76.5% passed the PND vignette and 50.0% passed the suicide vignette. Table 4 shows student performance in applying the ALGEE process, as per the audio-recordings. In general, most students appropriately carried out the ALGEE actions; however, 14 (82%) students assigned to the PND roleplay and 13 (72%) students assigned to the suicide roleplay did not ask the simulated patient about drug and/or alcohol use.

Table 4.

Key Behavioral Outcomes During the Simulated Roleplay Audio-Recordings

During both roleplays, it was often difficult to determine if the participants asked about suicidal thoughts, specifically, or self-injury, more generally. This is because 71.4% of participants in the PND roleplay and 40% in the suicide roleplay either completely avoided using suicide-specific terminology or used specific and non-specific terminology. Terminology that is non-specific to suicide and may indicate assessment of non-suicidal self-injury include terms, such as “hurt” or “harm.” All participants who asked about self-injury, regardless of the terminology used, were given points for asking about suicide. However, to ask about suicide, specifically, Mental Health First Aiders must ask that person directly and must not avoid using the word “suicide.”⁴ To demonstrate appropriate language when referring to suicide, a Mental Health First Aider should use the terms “suicide” or “die by suicide.”⁴ Other appropriate examples of suicide inquiries include “Are you having thoughts of suicide?” and “Are you thinking about killing yourself?”

Terms that were considered to be non-direct included inquiries about harming, injuring, mutilating or hurting oneself or having thoughts of such actions, such as self-harm.²⁸ The term self-harm is not clearly defined and can refer to non-suicidal self-injury.²⁸ During the suicide vignette, six students either used multiple terms or used only non-suicidal terminology to assess suicidal thoughts or ask related questions, such as whether the simulated patient had a plan in place. Similarly, during the PND vignette, when inquiring about suicide, 10 students used multiple terms or only non-suicidal terminology. Hence, it was difficult to determine if these students were using the terms interchangeably to inquire about only suicide attempts or if they were inquiring about non-suicidal self-injury. Therefore, only 28.6% and 60.0% of students during the PND and suicide roleplays, respectively, inquired about suicide using exclusively suicide-specific terminology, such as “suicide,” “killing,” or “taking your own life.”

DISCUSSION

This study presents findings on an important and often overlooked topic in the literature, evaluating observed performance following MHFA training, with the use of simulated patients. The post-training confidence level of participants in applying the MHFA action plan, ALGEE, often differed from their observed performance. Only 28.6% and 60% of students assigned to the PND and suicide role-play, respectively, used non-ambiguous, suicide-specific terminology to directly ask about suicide. Moreover, students’ abilities differed, based on the vignette, whereby a lower percentage of students were able to encourage appropriate professional help (88.2% vs 50%) and other support (88.2% vs 77.8%) for the suicidal simulated patient.

Students both underestimated and overestimated their abilities in providing MHFA. The self-evaluation questionnaire (Table 3) indicated that 84.0% of students agreed or strongly agreed that they felt confident in approaching someone developing a mental health problem or experiencing a crisis. However, during the simulated roleplay, 100.0% and 94.4% of students appropriately approached the patient for the PND and suicide vignette, respectively.

During the self-evaluation of confidence, 95.8% of students either agreed or strongly agreed that they felt confident they could encourage a person to seek appropriate professional help. However, this was an overestimation for the suicide vignette, whereby only 50.0% of students took the appropriate actions, which involved both referring to an appropriate health care professional and not leaving the simulated patient alone. Students also overestimated their ability to provide the simulated patient with access to other support, despite being taught about support such as Australian helplines and websites during MHFA training. Furthermore, participants receive a personal copy of the MHFA manual which contains these details. During the self-reported evaluations, 95.8% and 96.5% of students either agreed or strongly agreed that they felt confident they could encourage access to other supports and offer information and other support, respectively. However, only 88.2% and 77.8% encouraged self-help during the PND and suicide roleplays, respectively. The tendency to over- and underestimate in self-report measures has been demonstrated in other areas, including but not limited to job performance, weight perceptions and video game playing time. ^29-31

During the self-reported evaluation, 88.9% of students reported that they agreed or strongly agreed that they felt confident asking someone about suicidal thoughts. During the roleplay, 82.4% of those allocated to the PND vignette and 83.3% of those allocated to the suicide vignette inquired about suicide. So, it seems as though students did not over- or underestimate their ability to ask about suicidal thoughts. However, upon re-scoring the audio recordings, it became evident that the students’ inquiries were often unclear as they used both suicidal and non-suicidal self-injury terminology. During MHFA, students are taught not to avoid the word “suicide.”⁴ Nonetheless, among those who inquired about suicide, 71.4% and 40.0% of students either avoided suicidal terminology completely or used multiple terms (eg, “kill,” “hurt,” “attempt suicide” and/or “harm”) throughout their suicide assessments for the PND and suicide roleplays, respectively. This made it difficult, at times, to determine whether they were addressing suicide attempts or non-suicidal self-injury. Students learnt about non-suicidal self-injury during MHFA training, hence, it is unclear whether students who asked about “harming” or “hurting” were referring to suicide attempts or non-suicidal self-injury. Furthermore, given the use of both suicide-specific and non-specific terminology, it was unclear if terms, such as “hurting” and “harming” were used interchangeably with suicide-specific terms, such as “suicide” and “killing,” or if they were used to ask about non-suicidal self-injury. Moreover, even though both simulated patients clearly expressed that they were not feeling well and could have potentially had suicidal thoughts, over 15% of students did not ask about suicidal thoughts, at all. Collectively, these findings indicate that some participants did not feel confident or comfortable inquiring about suicidal thoughts.

These findings may indicate that students still felt uncomfortable providing help to suicidal persons post-training. This could be due to the myths and stigma associated with suicide. A recent study among medical students reported that only 49% of participants correctly answered “false” to the statement “people who talk about suicide rarely commit suicide.”³² Furthermore, both postgraduate and undergraduate medical students struggled with recognizing suicidal “signs,” as demonstrated by a mean percentage correct of only 43% and 58%, respectively, on an instrument that measures suicide literacy.³² In this study, students had completed MHFA training, which specifically addresses common suicide myths. MHFA participants are taught that “although some people think that asking about suicide can put the idea in the person’s mind, this is not true.”⁴ Furthermore, MHFA training also addresses myths such as “someone who talks about suicide isn’t really serious” and informs participants that suicidal persons may talk about suicide to express “how badly they are feeling.”⁴ Nonetheless, many students did not ask about suicide or used unclear terminology, indicating lower levels of skill acquisition as they could not “show how” they would apply the information they “knew” and self-reported that they “knew how” to use.²¹

During MHFA training, students receive extensive information on how to apply ALGEE for a person experiencing suicidal thoughts. On the other hand, even though, they do receive similar detailed information on applying ALGEE on a person experiencing depression, they do not receive detailed information on how their approach should be modified when helping a person with PND, specifically. Nonetheless, 88.2% were able to take the appropriate actions and 88.2% were able to encourage self-help during the PND roleplay while only 50% and 77.8% were able to do so for the suicide roleplay, respectively. In addition, 29.4% of students assigned to the PND roleplay asked the simulated patient if she had any intentions of harming the baby, which is not something they learnt during training. Furthermore, even though the postnatal simulated patient did not have suicidal thoughts, 21.4% of students still inquired if she had a plan to attempt suicide in place. It appears that students were more capable and/or confident in assessing the simulated patient and encouraging appropriate actions in the PND roleplay than the suicide roleplay. This could have been due to the “crisis” nature of the suicide roleplay and the stigma surrounding suicide. However, there are various reports regarding the lack of concordance between self-reported and observed behaviors, regardless of the crisis nature of the behavior being examined. For example, a recent systematic review comparing direct and self-report measures of physical activity found self-report measures to be both higher and lower than direct measures, depending on the method used, rendering it difficult to find a “clear trend” among the differences.³³

We found that students tend to under- and overestimate their skills when comparing self-reported behaviors with observed performance during simulated patient roleplays. This sheds light on the need to include direct observations of behaviors to comprehensively assess interventions. Use of simulated patients to evaluate participants’ MHFA skills assesses students’ abilities on a higher level of Miller’s pyramid.²¹ Not only does it ensure that participants know what MHFA is and know how to provide it, but it also evaluates their performance by allowing them to show how they would apply their newly learned skills.²¹ Even after training, participants are often unable to adequately use some of the skills taught. Assessing training programs by allowing participants to show how they would use their knowledge or by observing their actions, if possible, can inform their modification and revision to ensure participants have mastered the skills that are taught.²¹ Despite most students in this study appropriately carrying out the MHFA Action Plan, only 50% of students assigned to the suicide roleplay passed the assessment, while 76.5% of those assigned to the PND roleplay passed. This could indicate that health care students need further support handling patients at risk of suicide. Given that suicide is among the three leading causes of death among people aged 15-44, it is essential that primary health care professionals know how to recognize, approach, assess and assist those at risk of suicide.^4,34 Incorporating these skills into the curricula of health care students may potentially impact suicide rates. However, to ensure primary health care professionals are capable of providing such services, it is necessary to assess their skills after training.

Regardless of their performance, after the roleplay, all students were given immediate performance feedback by the simulated patient. Medical education research has shown that students who receive immediate feedback are more likely to agree that a standardized patient exercise provided them “with further insight into specific interviewing techniques” and “was a safe method…to work through…weaknesses” than those who did not receive immediate feedback.³⁵ Among pharmacists, providing feedback after simulated patient visits is an acceptable way to improve the “quality of over-the-counter consultations” and “for assessing current practice.”³⁶ Furthermore, immediate feedback is appreciated among pharmacy staff as it allows them to recognize “specific areas for potential improvement” as it is focused around “real life practice.”³⁷ Hence, incorporating immediate performance feedback in this study ensured that the simulated patient exercise was not solely a means of assessment, but also a way for students to improve their skills.²²

Given that previous studies documenting improvements in behaviors have usually based their findings on self-reported behaviors, this study illustrates the potential for under- and overestimation of confidence in providing various MHFA actions when using self-report as the sole measure for behaviors. Despite these strengths, this study could have been improved by using a larger sample size of students to improve the generalizability of the findings. Furthermore, given that participants were pharmacy students, it is unclear if the findings are generalizable to Australian pharmacists. However, the students were in the final semester of their degree and were only months away from registering as intern pharmacists with provisional registration in Australia. Nonetheless, further research with varying participant groups is also warranted.

Another potential limitation which may have affected the students’ performance was the use of tutors who were known to the students as simulated patients and that the numbers of scenarios delivered by the two tutors were not even. Hence, future study designs simulating real life environments in a non-academic setting using actors or consumers with a lived experience of mental illness may yield different results.

CONCLUSION

Although assessment of actual behaviors following MHFA training may be difficult to achieve, this study is among the first to evaluate health care professional students’ observed behaviors using simulated patients. This has allowed for a more thorough evaluation of the skills participants acquired and their ability to apply the ALGEE actions after completing MHFA training. Since participants both under- and overestimated their ability to perform some of the key skills, after MHFA training, it is evident that direct observations of behaviors should supplement self-reported evaluations of interventions. By doing so, a higher level of knowledge acquisition and implementation can be ascertained, post-training.