Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2024 Jun 14.
Published in final edited form as: Res Soc Work Pract. 2023 Mar 15;34(2):182–193. doi: 10.1177/10497315231161563

Computerized Suicide Prevention Clinical Training Simulations: A Pilot Study

Lindsay A Bornheimer 1,2, Juliann Li Verdugo 1, Laura Humm 3, Chris Steacy 3, Julie Krasnick 1, Julie Goldstein Grumet 4, James E Aikens 5, Katherine Gold 5, Barbara Hiltz 1, Matthew J Smith 1
PMCID: PMC11178331  NIHMSID: NIHMS1949539  PMID: 38881845

Abstract

Purpose:

Mental health providers are well-positioned to engage in suicide prevention efforts, yet implementation depends on skill acquisition and providers often report feeling underprepared. This pilot study explored the acceptability, feasibility, and preliminary effectiveness of three suicide prevention-focused simulations with virtual clients.

Method:

Students (n=22) were recruited from a MSW program, completed pre- and post-test surveys, and engaged with three simulated trainings: 1) suicide risk assessment, 2) safety planning, and 3) motivating a client to treatment.

Results:

Simulations were reported to be acceptable and feasible, with strong student desire and need for greater suicide prevention training. We observed significant improvements over time in clinical skills via simulated training scores and perceptions of clinical preparedness.

Discussion:

Preliminary findings indicate simulated training with virtual clients is promising and suggest the three suicide prevention simulations may be useful, scalable, and effective in social work training programs and beyond.

Keywords: simulated training, suicide assessment, safety planning, suicide prevention, student trainees

Suicide is a critical public health concern and a leading cause of preventable death worldwide (World Health Organization, 2019; Constanza et al., 2020). Despite increased attention to prevention strategies, rates of suicide continue to rise with suicide remaining the tenth leading cause of premature death among adults in the United States (Martinez-Ales et al., 2020; Henry, 2021). Given suicide prevention strategies are multifaceted in nature and involve multi-level efforts (e.g., population-, institutional-, and individual-level), it is essential that the healthcare field recognizes their critical role in detecting and addressing suicide risk (O’Brien et al., 2019; King et al., 2017). Studies show 83% of individuals who died by suicide visited a healthcare provider within one year of death, nearly 50% were seen by a primary care provider within one month of death, and 20% saw a mental health provider within one month of death (Luoma et al., 2002; Ahmedani et al., 2014; Abed Faghri et al., 2010; O’Brien et al., 2019). Therefore, health and behavioral healthcare providers have a significant opportunity to contribute to suicide prevention efforts.

Comprehensive mental health provider training in suicide risk assessment augments the administration of brief binary self-report suicide screening questions often used by providers. While it is profoundly important to use evidence-based and standardized screening and assessment tools, these tools alone do not capture the nuance of suicide risk. Despite the importance of provider training in suicide assessment, intervention, and prevention efforts, over 50% of behavioral healthcare providers reported having no formal suicide assessment or prevention training in a 2016 survey (n=16,000; Silvia et al., 2016) and approximately 67% of healthcare providers endorsed having limited knowledge of suicide risk warning signs and low levels of confidence in their clinical response to a suicidal client in a 2019 survey (n=15,000; Harmer et al., 2022). Furthermore, studies suggest that many mental health providers, particularly early career clinicians, report a strong desire for greater suicide assessment training (Moscardini et al., 2020; Wakai et al., 2020) and express a need for innovative training techniques to increase their competence and self-efficacy in working with suicidal clients (O’Brien et al., 2019).

Clinical training for providers in healthcare and behavioral healthcare has traditionally involved in-person workshops and seminars (Mallonee et al., 2018). While such trainings provide face-to-face opportunities to practice skills, they also can be costly, difficult to fit into provider workload, and lack standardization across sites and settings (Frank et al., 2020; Morse, Salyers, Rollins, Monroe-DeVita, & Pfahler, 2012). Data show virtual and web-based provider training to be more effective than paper-based manualized training (Sholomskas & Carroll, 2006) and superior to in-person training in one study (Dimeff et al., 2009). Innovative technology-assisted and simulation-based clinical trainings with virtual clients are emerging in the behavioral healthcare field and provide an opportunity to gain and apply skills through a combination of learning didactic content, practicing skills, and receiving both coaching and feedback (Kourgiantakis et al., 2020). In addition to increasing access for trainees to learn evidence-based approaches, simulation-based trainings are available on-demand, self-paced, interactive, individualized, and deliver a standardized training experience (Magill et al., 2022). Beyond training, however, it is also essential for these technologies to evaluate skills for proficiency and fidelity to a suicide prevention strategy; a process that is time-consuming and often unstandardized (Gamarra et al., 2015; Perepletchikova et al., 2009).

The current pilot study aimed to explore the: 1) acceptability (e.g., helpfulness and usefulness), 2) feasibility and usage (e.g., engagement and training adherence), and 3) preliminary effectiveness of three suicide-focused simulated virtual sessions involving both training and evaluation of skills among clinical social work trainees in a Midwestern region of the US. We hypothesized that the simulations would be acceptable, feasible, adhered to, and produce preliminary improvements in clinical skill application and perception over time. In addition, we hypothesized that improvements in clinical skill application would positively associate with skill perception and simulation acceptability.

Method

This pilot study involved a one-group pre-experimental design using quantitative and qualitative data collection before and after mental health trainees engaged with a simulated suicide prevention-focused training and evaluation tool. Participants included 22 Master’s-level graduate students enrolled in an accredited Master of Social Work (MSW) program at the University of Michigan. All students enrolled in the first author’s 1-credit elective suicide prevention course in the University’s MSW program were eligible to participate and were told about this optional study in the first course session. Interested students were given a link to a Qualtrics survey that detailed study information and allowed for consent to be obtained electronically. Students were informed that the course instructor would not know who participated in the study, and their decision to do so would not impact their course grade. This study was reviewed and approved by the University of Michigan Institutional Review Board.

The course delivered suicide prevention content during two weeks in May 2022. Taught by the first author, the course included 45 hours of materials with asynchronous (prerecorded) lectures, readings, treatment video examples, and two 5-hour live class sessions. Course objectives included increasing clinical skills and confidence in suicide risk assessment, formulating suicide risk level, and prevention-focused approaches at the population and community levels (e.g., psychoeducation, screening, assessment, and gatekeeper training) and individual level (e.g., Motivational Interviewing [MI], Cognitive Behavioral Therapy [CBT], Dialectical Behavior Therapy [DBT], and lethal means restriction counseling).

Simulations

Simulations used in this pilot study were developed by SIMmersion (www.simmersion.com) in collaboration with the Institute for Family Health (IFH) and the Educational Development Center (EDC) as part of a National Institute of Mental Health (NIMH)-funded project (R44MH114710). SIMmersion uses a proprietary technology called PeopleSim to increase knowledge acquisition, allow for skill-building practice, and provide integrated feedback required to develop advanced skills in their computer-assisted clinical training sessions with virtual human clients. Prior studies of SIMmersion’s non-suicide-focused simulations have demonstrated that trainees effectively learned to use clinical techniques such as client engagement, mental and behavioral health screening, cognitive behavioral therapy, and motivational interviewing (Huttar & BrintzenhofeSzoc, 2020; Mastroleo et al., 2020; Washburn et al., 2020; Putney et al., 2019a,b; Smith et al., 2021).

The three simulations used in the current pilot study were developed in the Suicide Prevention Role-plays for Interactive Training (SPiRIT) project and involve interactive role-play simulations to train clinicians to talk with virtual clients and assess their suicide risk, complete a safety plan with them, and motivate them to treatment. The PeopleSim technology uses video of pre-recorded client actors, voice recognition, and non-branching logic for trainees to engage in a clinical session with the virtual clients. Trainees choose the direction of their simulated clinical session by selecting statements, responses, and questions that vary in degrees of accuracy for the purpose of each simulated training. The selection of clinical statements (or questions) cumulatively informs the algorithm that adjusts each virtual client’s mood and response to the applied clinical skills of the trainee. The technology includes a coach who provides real-time feedback (i.e., validating effective skills and giving corrective feedback) and a scoring system (i.e., based upon specific clinical skills for each simulation) provides a comprehensive report of performance at the end of each session. The computer-generated performance scoring reflects the application of skills learned and expected in the simulated training experience. In addition, and specifically for the suicide assessment simulation, a knowledge assessment followed the simulated virtual client session (i.e., multiple choice quiz) and responses contributed to performance scoring. Preliminary investigations of the three suicide-focused simulations demonstrated preliminary provider feasibility and acceptability among a sample of physicians, social workers, and nurses (n=20) in healthcare settings (O’Brien et al., 2019), and organizational feasibility and acceptability among a sample of administrators, managers, and educators (n=9) of healthcare settings (O’Brien et al., 2022) in a Northeastern region of the US. The three simulations with fictional characters are described below and presented in Figure 1.

Figure 1.

Figure 1

Open in a new tab

Suicide Prevention Simulated Training Screenshots

Suicide Assessment Simulation with Taye.

The trainee’s goal is framed as developing rapport with Taye, assessing for suicide thoughts and behavior, and determining her level of suicide risk. The simulation randomly selects one of four variations of Taye (4 various suicide risk levels) to present the trainee at the start of each launched session. Taye’s openness and willingness to engage with the trainee also varies by her 4 suicide risk presentations.

Safety Planning Simulation with Henry.

The trainee’s goal is framed as developing rapport with Henry, orienting him to a safety plan, and completing a safety plan with him. The simulation randomly selects one of four variations of Henry (4 various presentations) to present the trainee at the start of each launched session. Henry’s openness and willingness to engage with the trainee in safety planning also varies by his 4 different presentations.

Motivating a Client to Treatment Simulation with Katrina.

The trainee’s goal is framed as developing rapport with Katrina and building her motivation and commitment to begin suicide-focused treatment using motivational interviewing skills. The simulation randomly selects one of two variations of Katrina (2 presentations of motivation and barriers towards seeking treatment) to present the trainee at the start of each launched session. Katrina’s openness and willingness to consider treatment with the trainee also varies by his 4 different motivation and barrier presentations.

Measurement

The pre-test survey was given to participants via Qualtrics on the first day of the course and assessed demographic characteristics, clinical and MSW program experience, degree of comfort in talking with clients about suicide, and clinical skills (preliminary effectiveness). The post-test survey was given to participants via Qualtrics on the last day of the course and included quantitative questions related to simulation acceptability, feasibility, clinical skills (preliminary effectiveness), and three open-ended qualitative questions to share their simulation experiences. Details of quantitative and qualitative questions are described below, and simulation usage and performance data were retrieved from the simulation technology platform.

Acceptability.

Acceptability of simulations was primarily assessed by five questions asking participants to rate the degree to which they felt the simulations were helpful for their training and skills, were engaging as a training experience, were realistic (i.e., characters and conversations), made them want to try again (i.e., keep using the simulations), and the extent to which they would recommend the simulations to other MSW trainees. Responses were on a 5-point Likert scale ranging from 1 (not at all) to 4 (very), with higher scores representing greater acceptability of simulations. We calculated an item-level mean score across these 5 items, which ranged from 1 to 4 with higher scores representing greater acceptability. Secondary acceptability-focused questions were asked in the post-survey with binary response options (agree or disagree) regarding accessibility, desirability, trainee anxiety, and potential harm to a human client.

Feasibility.

Feasibility was assessed by simulation platform data including information on simulation tool engagement and session performance scores (first, second, and third session score means). For simulation engagement, the system auto generated the total number of minutes participants spent talking with virtual clients, the total number of minutes reviewing the eLearning curriculum (including descriptions of how sessions are scored), and the total number of sessions completed. Performance score data showed simulation session scores for all sessions engaged with while on the platform, with scores ranging from 0 to 100 (higher scores are better). We collected the first, second, and third scores, in addition to mean, highest, and lowest scores from the platform data area. The study protocol requested participants to complete three sessions per simulation and achieve at least a 75% score once per simulation. Scores represent clinical skills in various domains (e.g., building rapport, understanding the patient, identifying risk factors, exploring suicidal ideation/behavior, effective safety planning, use of motivational interviewing techniques). The scoring process and criteria are conveyed to the trainee each time they launch a new session.

Preliminary effectiveness.

Preliminary effectiveness was assessed by participant perceptions of preparedness for practicing skills with a client and improvements in simulated session scores. One preparedness question was asked with tailoring for each of the three simulations in the pre- and post-test surveys, such as, “How prepared do you feel to conduct a suicide assessment with a client?” Responses were scored on a 5-point Likert scale ranging from 1 (not at all) to 4 (very), with higher scores representing greater preparedness. In addition, using the performance scores described in the feasibility section above, we calculated change scores from the first to third score with a positive change score indicating improvement over time. Scores represent the evaluation of participants’ skill application in the simulated virtual client session.

Qualitative questions.

Participants answered three qualitative questions at the end of the post-test survey with free-text boxes for typed responses. Questions asked: 1) What did you like most about the simulated training?, 2) What did you like least about the simulated training?, and 3) How did the simulated training compare with other training you’ve received in suicide prevention overall?

Data Analysis

Quantitative data were analyzed using SPSS 28 and included descriptive statistics (e.g., frequency, mean, standard deviation, percentage) to evaluate participant demographic characteristics, acceptability, and feasibility of simulations. Acceptability evaluations involved examinations of the post-test simulation experience survey and feasibility evaluations included examination of simulation tool engagement (total minutes engaged with each virtual client in simulated sessions, total minutes spent engaging with the eLearning content, and total number of virtual client sessions completed), adherence to the study protocol (% of participants who engaged in three session attempts per simulation and achieved the required score of at least 75% once per simulation) and session performance scores (first, second, and third session score means). To assess for preliminary effectiveness of the simulations, we used paired-samples t-tests to evaluate: 1) whether post-test levels of preparedness for practicing skills differed from pre-test levels, and 2) whether the third simulation session score differed from the first session score. Given the small sample size (n=22), we reference significant changes from pre- to post-test as preliminary. In addition, Cohen’s d was calculated with Hedges’ correction to determine the standardized differences between means given our sample size (Hedges & Olkin, 1985). A very small effect is considered to be < 0.2, small is between 0.2 and 0.5, medium is between 0.5 and 0.8, and large is greater than 0.8 (Cohen, 1977). Lastly, we performed Pearson correlations to explore relationships between the change in scores from first to third simulated sessions, post-test mean acceptability scores, and post-test preparedness ratings.

Regarding hypotheses, we predicted the following using the measurement and data analysis approaches described above. First, participants would find the simulations to be acceptable (i.e., somewhat to very acceptable), accessible given its virtual and on-demand format, desirable to use, less anxiety provoking to engage with than a human client, and cause harm than practicing skills with a human client. Second, the simulations would be feasible per participants meeting the 75% minimum score requirement within 3 session attempts. Third, and pertaining to preliminary effectiveness, participants would report significantly higher levels of perceived preparedness after the simulation training in comparison to before the training. In addition, simulation session scores would significantly improve across the three required sessions. Lastly, improvements in simulation session scores would positively associate with greater perceptions of preparedness and simulation acceptability, and similarly, that greater perceptions of preparedness would positively associate with simulation acceptability.

Qualitative data were transcribed and coded independently by two research assistants (RAs) in preparation for codebook development. An open coding technique was used to generate themes across qualitative questions (Saldana, 2016) and grounded theory methods were utilized for analysis (Charmaz, 2014). Preliminary codes of the RAs were evaluated by the first author, who facilitated discussions to establish agreement among both RA coders to achieve inter-coder consistency. Themes were ultimately organized into a final framework by the coding team of three. The following strategies for rigor (Padgett, 2016) were included in the study: 1) triangulation, specifically analytic triangulation with more than one coder, and 2) use of an audit trail. Qualitative findings are presented towards the end of the results section below.

Results

Participant Characteristics

Characteristics of participants are presented in Table 1. Participants (n=22) were M=27.50 years old (SD=8.0), most often identified as female (n=18, 81.1%), non-Latinx/Hispanic (n=21, 95.5%), and White (n=17, 77.3%). Most participants felt their MSW program focuses a little or not at all on suicide prevention in courses (n=16, 72.7%) with all participants desiring more opportunities to learn suicide prevention skills prior to graduation (n=22, 100%). Prior to the course and simulations, participants most often endorsed feeling ‘a little to ‘somewhat comfortable’ talking with clients about suicide (n=16, 72.8%) and ‘not at all’ to ‘a little comfortable’ talking with clients who are suicidal (i.e., clients who endorse ideation, plan, attempt; n= 21, 95.5%). Of the 22 participants, one (0.5%) did not complete the simulated trainings or post-test survey.

Table 1.

Participant characteristics

Continuous Characteristic n M,SD
Age (M,SD) 22 27.50, 8.0
How many hours per week do you currently work with clients who are at risk for suicide, endorse having suicide ideation, or engage in suicide behavior? (M,SD) 22 2.32, 1.04
Categorical Characteristic n %
Gender
 Female 18 81.1
 Male 4 18.2
Ethnicity
 Latinx/Hispanic 1 4.5
 Non-Latinx/Hispanic 21 95.5
Race (select all that apply, percent is over 100)
 African American/Black 2 9.1
 Asian 5 22.7
 White 17 77.3
 Biracial 2 9.1
How much of the MSW program have you completed to date?
 90% (gradating now) 1 4.5
 75% 14 63.6
 50% 5 22.7
 25% 1 4.5
 Less than 25% 1 4.5
Is your pathway/focus in the MSW program Interpersonal Practice/Clinical?
 Yes 15 68.2
 No 7 31.8
Did you take coursework/workshops focused on suicide prevention prior to this class?
 Yes 8 36.4
 No 14 63.6
Did you take coursework/workshops focused on suicide assessment prior to this class?
 Yes 8 36.4
 No 14 63.6
Did you take coursework/workshops where you learned safety planning prior to this class?
 Yes 7 31.8
 No 15 68.2
Rate the degree to which you believe the MSW program focuses on suicide prevention in courses
 It is focused on a lot 1 4.5
 It is somewhat focused on 5 22.7
 It is focused on a little or not at all 16 72.7
Do you wish there were more opportunities to learn suicide prevention skills in the MSW program?
 Yes 22 100
 No 0 0
How comfortable do you feel talking with clients about suicide prior to this class?
 Very comfortable 3 13.6
 Somewhat comfortable 8 36.4
 A little comfortable 8 36.4
 Not at all comfortable 3 13.6
How comfortable do you feel talking with clients who are suicidal prior to this class (i.e., clients who say yes to having ideation, plan, attempt)?
 Very comfortable 1 4.5
 Somewhat comfortable 6 27.3
 A little comfortable 6 27.3
 Not at all comfortable 9 40.9
Open in a new tab

Acceptability

The item-level acceptability means for the suicide assessment, safety planning, and motivating to treatment simulation are presented in Table 2. We hypothesized that participants would find the simulation training to be acceptable (i.e., somewhat to very acceptable), accessible, desirable, less anxiety provoking than with a human client, and cause less potential harm than with a human client. The item-level acceptability mean (ranging from 1 to 4) was 3.33 (SD=0.495) for the suicide assessment simulation, 3.39 (SD= 0.598) for the safety planning simulation, and 2.96 (SD=0.709) for the motivating to treatment simulation; suggesting the simulations were perceived as being somewhat to very acceptable. All participants agreed that practicing suicide prevention skills with a simulated virtual client is less anxiety provoking (n=21, 100%) and causes less potential harm (n=21, 100%) than with a human client. The majority agreed that the simulated training sessions were accessible given the on-demand nature (n=20, 95.2%), complementary to practicing with a real human client (n=18, 85.7%), desirable as a training option given the many disruptions that have occurred in face-to-face clinical training due to the COVID-19 pandemic (n=17, 81%).

Table 2.

Post-Simulation Experience Survey Responses

Survey Questions Specific to Simulations Suicide Assessmenta
(Taye)		 Safety Planninga
(Henry)		 Motivating to Treatmenta
(Katrina)
M SD M SD M SD
Acceptability items
 How helpful was the simulation? 3.57  .507 3.57 .598 3.05 .865
 How engaging was the simulation? 3.47  .512 3.48 .749 3.09 .700
 How realistic was the simulation character? 3.14  .727 3.24 .700 3.14 .854
 How much did you want to try the simulation again? 3.09  .889 3.33 .913 2.71 1.055
 How likely are you to recommend the simulation to others? 3.81  .669 3.33 .796 2.81 .928
Mean acceptability score (mean of below items) 3.33  .495 3.39 .598 2.96 .709
Survey Questions of General Experiences n %
Practicing suicide prevention skills with a simulated client can is less-anxiety provoking that practicing with a human client while a trainee.
 Agree 21 100
 Disagree 0 0
Practicing suicide prevention skills with a simulated client can cause less potential harm than practicing with a human client while a trainee.
 Agree 21 100
 Disagree 0 0
Practicing suicide prevention skills with a simulated client is a realistic training method that is complementary to practicing with a human client.
 Agree 18 85.7
 Disagree 3 14.3
Given the context of COVID-19 and prior experiences with disruptions in face-to-face clinical training, simulated client training in suicide prevention is a desirable option.
 Agree 17 81.0
 Disagree 4 19.0
Given its virtual format, simulated client training in suicide prevention is an accessible option for practicing clinical skills (i.e., the on-demand simulation platform is convenient and feasible to access for practicing skills).
 Agree 20 95.2
 Disagree 1 4.8
Open in a new tab
a

Scores range from 1 (not at all) to 4 (very) with higher scores being better (i.e., more desirable learning outcomes)

Feasibility

Quantitative data of simulation feasibility and usage are presented in Table 3. We hypothesized that participants would engage with the simulation training and meet the 75% minimum score requirement within 3 session attempts. On average, students completed 3.23 (SD=1.02) suicide assessment sessions, 3.32 (SD=0.89) safety planning sessions, and 4.14 (SD=2.32) motivating to treatment sessions. Almost all students completed the 3 required session attempts for each of the three simulations (n=21, 95.5%). All students reached the required threshold of a 75% or greater score in the suicide assessment and motivating to treatment simulations (n=22, 100%), while 95% (n=21) of students obtained the required score in safety planning. The overall simulation session score means were M=81.43 (SD=6.48) for suicide assessment, M=77.95 (SD=10.82) for safety planning, and M=77.09 (SD=7.80) for motivating to treatment. As for time, students spent an average of M=57.95 (SD=38.04) minutes in the suicide assessment sessions, M=107.68 (SD=84.91) minutes in the safety planning sessions, and M=33.65 (SD=26.99) minutes in the motivating to treatment sessions.

Table 3.

Simulation Usage Data

Suicide Assessment
(Taye)		 Safety Planning
(Henry)		 Motivating to Treatment
(Katrina)
M SD M SD M SD
Mean number of sessions completeda 3.23 1.02 3.32 .89 4.14, 2.32
Percentage of students who completed 3 sessions as requested (n, %) 21 95.5 21 95.5 21, 95.5
Percentage of students who scored over 75% at least once as requested (n, %) 22 100 21 95.5 22, 100
Mean scoreb 81.43 6.48 77.95 10.82 77.09 7.80
Highest scoreb 86.68 4.06 86.14 7.67 85.50 4.91
Lowest scoreb 62.55 13.99 62.09 14.95 61.64 15.36
First session scoreb 75.50 9.55 72.45 12.79 71.09 11.35
Second session scoreb 81.14 8.59 78.36 11.43 79.45 6.52
Third session scoreb 86.68 4.06 85.86 8.41 85.50 4.91
Total minutes engaged in eLearning 3.77 5.93 4.14 11.35 2.86 6.78
Total minutes engaged in simulated session 57.95 38.04 107.68 84.91 33.65 26.99
Open in a new tab
a

Three attempts were requested per simulation.

b

Simulation session scores range from 0 to 100 with higher scores representing better performance outcomes.

Preliminary Effectiveness

Participant preparedness and simulation session scores are displayed in Table 4 along with significance testing and effect size. We hypothesized that participants would report significantly higher levels of perceived preparedness after the simulation training in comparison to before the training. In addition, we hypothesized that simulation session scores would significantly improve across the three required sessions. As shown in Table 4, participant perceptions of their preparedness for using skills with clients significantly improved from pre- and post-test for suicide assessment (pre: M=2.00, SD=0.77; post: M=3.14, SD=0.57; t(21)= −5.16, p<.001), safety planning (pre: M=2.14, SD=0.96; post: M=3.33, SD=0.57; t(21)= −5.29, p<.001), and motivating to treatment (pre: M=2.09, SD=0.94; post: M=2.95, SD=0.86; t(21)= −2.99, p<.01). The effect sizes were calculated to be large for suicide assessment (1.11), large for safety planning (1.13), and medium for motivating to treatment (0.64) per Cohen’s d with Hedges’ correction.

Table 4.

Preliminary Effectiveness per Preparedness and Simulation Scores

Pre-Survey Rating Post-Survey Rating Significance Effect Size
M SD M SD
Preparedness for suicide assessmenta 2.04 0.75 3.14 .573 t(21)= −5.16, p<.001** 1.11
Preparedness for suicide safety planninga 2.18 0.95 3.33 .578 t(21)= −5.29, p<.001** 1.13
Preparedness for motivating to treatmenta 2.05 0.95 2.95 .865 t(21)= −2.99, p<.01* 0.64
First Session Score Third Session Score Significance Effect Size
M SD M SD
Suicide assessment with Tayeb 75.50 9.55 86.68 4.06 t(21)= −6.079, p<.001** 1.27
Suicide safety planning with Henryb 72.45 12.79 85.86 8.41 t(21)= −5.438, p<.001** 1.14
Motivating to treatment with Katrinab 71.09 11.35 85.50 4.91 t(21)= −6.961, p<.001** 1.46
Open in a new tab
a

Scores range from 1 (not at all prepared) to 4 (very prepared) with higher scores being better.

b

Simulation session scores range from 0 to 100 with higher scores representing better performance on the simulation.

*

p<.01,

**

p<.001

In addition, simulation performance scores significantly improved from the first to third session for suicide assessment (pre: M=75.50, SD=9.55; post: M=86.68, SD=4.06; t(21)= −6.079, p<.001), safety planning (pre: M=72.45, SD=12.79; post: M=85.86, SD=8.41; t(21)= −5.438, p<.001), and motivating to treatment (pre: M=71.09, SD=11.35; post: M=85.50, SD=4.91; t(21)= −6.961, p<.001). The effect sizes were calculated to be large for suicide assessment (1.27), safety planning (1.14), and motivating to treatment (1.46) per Cohen’s d with Hedges’ correction.

Exploratory Correlations

We hypothesized that improvements in simulation session scores would positively associate with greater perceptions of preparedness and simulation acceptability. In addition, we hypothesized that greater perceptions of preparedness would positively associate with simulation acceptability. Findings demonstrated that participants with greater improved skills over time (i.e., the change in simulation scores) for the suicide assessment simulation endorsed more preparedness to conduct suicide assessments with clients at post-test (r=0.445, p<.05) and perceived the suicide assessment simulation to be more acceptable (r=0.496, p<.05). No significant correlations were found between mean change in performance scores and preparedness or acceptability for the other two simulations (i.e., safety planning and motivating to treatment). Participants who endorsed greater preparedness for each of the three simulation skill areas at post-test also perceived all three simulations to be more acceptable at post-test: suicide assessment (r=0.775, p<.001), safety planning (r=0.647, p<.01), and motivating to treatment (r=0.894, p<.001).

Qualitative Themes

A total of 4 themes emerged from the first question regarding what students liked the most about the simulated training: 1) realistic experience (realistic client, realistic practice experience, realistic conversation, realistic resistance from client, and realistic variation in scenarios and client mood), 2) helpful feedback (helpful summary at the end, helpful overview of performance, helpful coach during session), 3) good opportunity for clinical practice (on-demand ability to practice skills with a client, safe training environment where trainee can do no harm, and encouraging training environment where trainee receives motivation from platform), and 4) challenging practice experience (resistance and indifference of client was challenging, changing scenarios each time with the same client made it challenging, wrong statements or answers for the provider to choose made it challenging). The following is an example quote from a student: “The presence of wrong answers or responses a provider could say was distracting and made it challenging in a helpful way” (theme 4).

A total of 3 themes emerged from the second question regarding what students liked the least about the simulated training: 1) technology glitches (clinician conversation options did not align with topic at all times, client did not respond to correct statement clinician made at times, help coach was not always aligned with the topic, and help coach feedback was not aligned with performance report at end of simulation), 2) desire for greater emotional connection with the client (more empathetic response options are desired for the provider, more validating and affirming statements are desired for the provider, more opportunities for the provider to build rapport with the client are desired), and 3) desire for more biopsychosocial questions (more questions for providers to gain understandings of the client’s background their presenting problem are desired, ability for the provider to ask questions related to sociodemographic information including gender and sexual identity are desired). The following is an example quote from a student: “I wanted more empathetic and reflective options as the provider to build rapport with the client before asking many questions, leading a task, or making a recommendation” (theme 2).

A total of 4 themes emerged from the third question regarding how the simulated training compared to prior suicide prevention training overall: 1) more helpful than other trainings (more effective training, more informative and in-depth content, requires greater critical thinking), 2) more enjoyable than other trainings (enjoyed practicing with a simulated client instead of a clinical trainee or mental health professional, enjoyed the virtual aspect of the training and opportunity to remotely practice clinical skills on-demand), 3) more focused on motivational interviewing (MI) than other trainings, and 4) no prior training for comparison. The following is an example quote from a student: “The training included a lot more motivational interviewing skills than I have had in other workshops, and I think those skills are important in suicide prevention” (theme 3).

Discussion and Applications to Practice

This pilot study aimed to explore the acceptability, feasibility, use, and preliminary effectiveness of three recently developed suicide-focused simulated training sessions among MSW students: suicide assessment, suicide safety planning, and motivating a client at risk for suicide to treatment. Overall, most participants shared that they did not have prior coursework or training experience in suicide assessment, prevention, or safety planning, and all expressed a desire for suicide prevention to have greater presence in the education and training curriculum.

We hypothesized that the simulations would be acceptable, feasible, adhered to, and produce preliminary improvements in clinical skill perception and application over time. Participants reported high acceptability scores for all three simulations. Also as anticipated, participants agreed practicing suicide prevention skills with a simulated client is less anxiety provoking and can cause less potential harm than if practicing with a real human client in the field. Consistent with prior literature (e.g., Kourgiantakis et al. 2020; Magill et al., 2022), the majority of participants agreed the simulated training sessions were accessible, complementary to practicing with a human, and desirable to supplement in-person training and practice; particularly given limitations in clinical training opportunities resulting from COVID-19.

Usage data supports the feasibility of participants engaging with the simulations, given all but one student (95.5%) completed the 3 required session attempts and all but one student (95.5%) obtained the required competence score of 75% or greater across all simulations. With the requirement of 3 session attempts for each simulated training, the average time spent in the training’s simulated client sessions range from 102 to 321 minutes (1 hours and 42 minutes to 3 hours and 32 minutes); which is a realistically feasible duration in comparison to trainee coursework or provider continuing education workshops, and is also consistent with prior research with SIMmersion’s simulations focused on training motivational interviewing and cognitive behavioral therapy skills (Smith et al., 2021).

Preliminary effectiveness and participant perceptions of preparedness findings were also consistent with our hypotheses. Perceived readiness to use skills with a client significantly improved from before engaging with each of the simulated trainings to completing the trainings. Importantly, performance aligned with participant perceptions of preparedness or skill acquisition, with scores significantly improving from the first to third session attempt for all simulated trainings. This is promising to observe since some literature indicates competence perception does not always strongly correlate with skill performance (Dunning et al., 2004). Further supporting the finding that preparedness perception and skill performance relate in this sample, our exploratory analyses showed that participants with greater changes in simulated suicide performance scores endorsed more preparedness to apply such skills with clients in the future in in comparison to participants with smaller changes in performance scores.

Qualitative open-ended question findings reinforce and expand upon the quantitative results summarized above, with the first emerging grouping of themes of the simulated experience being realistic, a good training opportunity, a challenging practice experience, and including helpful feedback. A second grouping of themes suggested areas for improvement, such as technology glitches, a desire for greater emotional connection with the client, and a desire to have more biopsychosocial questions available for provider use. The final grouping of themes suggested the simulations were more helpful, enjoyable, and focused on use of motivational interviewing skills than other trainings attended (for those who had prior training experience). Importantly, the simulation technology continues to be refined and evolves with technological glitches being addressed in real-time and discussions among the development team about expanding provider options for responses that may include more engagement and biopsychosocial assessment overall.

Findings point towards two overarching implications. First, the sample of social work trainees expressed prior to training that they felt mostly uncomfortable and inadequately prepared to ask about suicide and work with suicidal clients. A notable aspect of this was their limited exposure and opportunities to receive suicide prevention training as a Master’s-level graduate student in a MSW program, in their field placement, and in prior behavioral health-focused jobs at the bachelor level. The topic of suicide prevention in mental health clinical training and education programs has been examined over the years, with studies showing overall minimal time allocated to training in suicide specific skills despite the existence of suicidal individuals across all service delivery settings (Mackelprang et al., 2014). One study in particular revealed that only 21% of students received formal suicide prevention training in their MSW program, slightly more than 50% stated that at least one of their courses addressed suicide, and among the students who received any kind of suicide prevention training or coursework, almost 50% reported that 2 hours or less were spent on the topic (Feldman & Freedenthal, 2006). Though the study was conducted over a decade ago, the more recent landscape of MSW programs dedicating time to suicide prevention education and training remains minimal. However, and importantly, even for those who do receive training, often missing are opportunities to practice suicide assessment and treatment skills for skill acquisition and competence (Smith et al., 2021). With an urgency to enhance the preparation of clinical skills among trainees, didactic training is not adequate to facilitate clinical competence (Duron & Giardina, 2018), and therefore simulated trainings such as in the current pilot study provide an opportunity for didactic training, clinical practice, and competence evaluation.

Second, while students may find the simulations to be feasible and acceptable, it is essential to ultimately target the setting and context in which simulations would be utilized. An important aspect of implementing an innovation such as these simulated clinical trainings is determining how it fits within the organizational context of a clinical training program (Jacobs et al., 2015; Kirchner et al., 2020). For example, faculty and instructors could choose to utilize this type of training as a supplement to their courses, or administrators and program directors can choose to recommend, require use of, or establish a distinct aspect of a training program outside of coursework to include such simulated training experiences. In addition to determining where the innovation lives, it is essential that there is buy-in from the individuals leading its implementation (e.g., leadership) and from those engaged in delivering the innovation (e.g., faculty and instructors; Cresswell et al., 2013). In the current pilot study, the School of Social Work and Associate Dean of Educational Programs adopted the school-wide implementation of multiple simulations over the past several years and the three new suicide prevention simulations in this pilot study will soon join the list for faculty to use in courses.

There are important limitations to note. First, participants were students in an elective course who were given opportunity to participate in this pilot study with initial interest in suicide prevention given the course topics and goals. Therefore, student perceptions, performance, and emerging themes from qualitative data may differ for student trainees not enrolled or interested in suicide prevention or this simulated training experience. Within a similar vein and expanding on the first limitation, it was not possible to distinguish skills and self-efficacy gained from the simulated training versus the elective course content given similar suicide-prevention skills were taught. However, the observed correlations offer initial support that practicing with the simulations could influence practical skills and qualitative feedback also reinforce the positive impact of simulations within the context of the course. Second, preliminary effectiveness was explored using simulated training scores and self-reported skill perception. Our future work will involve examinations of skill application with clients via trainee and provider medical chart record documentation. Third, students most often identified as female, White, and non-Hispanic/Latinx, thus the sample is not representative of clinical student trainees the US or Worldwide. Fourth, MSW students may experience greater anxiety regarding suicide risk and assessment training in comparison to MSW graduates and licensed clinicians. Future research will investigate the simulations among other social work and professional groups, such as via a continuing education course. Fifth, SIMmersion’s simulations are commonly used in the MSW program where this study occurred (e.g., simulations often focus on assessment and behavioral health treatment using MI and CBT). All students in the pilot study engaged with non-suicide focused simulations prior to this course in other classes and were familiar with the technology platform and session experience. Some participants experienced the following suicide-focused simulations in one prior course: suicide assessment (n= 9, 41%), safety planning (n=3, 14%), and motivating to suicide treatment (n=3, 14%). Although this prior exposure is limiting, we still observed significant increases in participant scores over time for each simulation. This suggests that the prior exposure may not have limited current improvements in simulation scoring. Lastly, the sample size was small (n=22) and underpowered to detect statistically significant small-to-medium effect sizes, therefore simulation effectiveness was preliminarily explored. Future effectiveness and implementation research will be conducted among larger and more diverse samples including randomization and representation of various roles and disciplines within behavioral health.

In sum, the current pilot study’s findings highlight social work trainee desire and need for greater suicide prevention training and the promising impact of this simulated clinical training technology innovation. These pilot study findings overall are positive and promising, demonstrating a great potential for these simulations to be a useful, scalable, and effective approach to suicide prevention clinical trainings in social work education and beyond. Our future work will aim to investigate the effectiveness of simulations on a larger scale, including comparative investigations of the virtual simulation training versus a comparable in-person training, with a goal of establishing an approach to scaling and disseminating the trainings for other institutions and fields.

Acknowledgments

This work was supported by the National Institute on Mental Health of the National Institutes of Health under Award Number R44MH114710.

JG is employed by Education Development Center, both LH and CS are employed by SIMmersion, LLC, and LH owns shares in SIMmersion, LLC. Both Education Development Center and SIMmersion may benefit from sales of the product once commercialized. MJS, the University of Michigan, and the University of Michigan School of Social Work receive royalties on sales of a SIMmersion LLC product that was not the focus of this study.

Footnotes

The other authors have no conflicts of interest to declare.

References

  1. Ahmedani BK, Simon GE, Stewart C, Beck A, Waitzfelder BE, Rossom R, … & Solberg LI (2014). Health care contacts in the year before suicide death. Journal of General Internal Medicine, 29, 870–877. 10.1007/s11606-014-2767-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Bandealy A, Herrera N, Weissman M, & Scheidt P (2020). Use of lethal means restriction counseling for suicide prevention in pediatric primary care. Preventive Medicine, 130, 105855. 10.1016/j.ypmed.2019.105855 [DOI] [PubMed] [Google Scholar]
  3. Charmaz K (2014). Constructing Grounded Theory. SAGE Publications, Inc. [Google Scholar]
  4. Cohen J (1977). Statistical power analysis for the behavioral sciences. Lawrence Erlbaum Associates. 10.4324/9780203771587 [DOI] [Google Scholar]
  5. Costanza A, Amerio A, Odone A, Baertschi M, Richard-Lepouriel H, Weber K, … & Canuto A (2020). Suicide prevention from a public health perspective. What makes life meaningful? The opinion of some suicidal patients. Acta Bio Medica: Atenei Parmensis, 91(Suppl 3), 128. 10.23750/abm.v91i3-s.9417 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Cresswell KM, Bates DW, & Sheikh A (2013). Ten key considerations for the successful implementation and adoption of large-scale health information technology. Journal of the American Medical Informatics Association, 20(e1), e9–e13. http://dx.doi.org/10.1136%2Famiajnl-2013-001684 [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. DeCou CR, Comtois KA, & Landes SJ (2019). Dialectical behavior therapy is effective for the treatment of suicidal behavior: A meta-analysis. Behavior Therapy, 50(1), 60–72. 10.1016/j.beth.2018.03.009 [DOI] [PubMed] [Google Scholar]
  8. Dimeff LA, Koerner K, Woodcock EA, Beadnell B, Brown MZ, Skutch JM, … & Harned MS (2009). Which training method works best? A randomized controlled trial comparing three methods of training clinicians in dialectical behavior therapy skills. Behaviour research and therapy, 47(11), 921–930. 10.1016/j.brat.2009.07.011 [DOI] [PubMed] [Google Scholar]
  9. Dunning D, Heath C, & Suls JM (2004). Flawed self-assessment: Implications for health, education, and the workplace. Psychological science in the public interest, 5(3), 69–106. 10.1111/j.1529-1006.2004.00018.x [DOI] [PubMed] [Google Scholar]
  10. Duron JF, & Giardina TD (2018). Teaching philosophies and practices in social work education: Do the core competencies influence our consciousness?. Social Work Education, 37(5), 603–616. 10.1080/02615479.2018.1450371 [DOI] [Google Scholar]
  11. Faghri NMA, Boisvert CM, & Faghri S (2010). Understanding the expanding role of primary care physicians (PCPs) to primary psychiatric care physicians (PPCPs): enhancing the assessment and treatment of psychiatric conditions. Mental Health in Family Medicine, 7(1), 17. [PMC free article] [PubMed] [Google Scholar]
  12. Feldman BN, & Freedenthal S (2006). Social work education in suicide intervention and prevention: An unmet need?. Suicide and Life-Threatening Behavior, 36(4), 467–480. 10.1521/suli.2006.36.4.467 [DOI] [PubMed] [Google Scholar]
  13. Frank HE, Becker-Haimes EM, & Kendall PC (2020). Therapist training in evidence‐based interventions for mental health: A systematic review of training approaches and outcomes. Clinical Psychology: Science and Practice, 27(3), 20. 10.1111/cpsp.12330 [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Gamarra JM, Luciano MT, Gradus JL, & Stirman SW (2015). Assessing variability and implementation fidelity of suicide prevention safety planning in a regional VA healthcare system. Crisis. 10.1027/0227-5910/a000345 [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Harmer B, Lee S, Duong TVH, & Saadabadi A (2022). Suicidal ideation. StatPearls. [PubMed] [Google Scholar]
  16. Hedges LV, & Olkin I (1985). Statistical methods for metaanalysis. Academic Press. [Google Scholar]
  17. Henry M (2021). Suicide prevention: A multisectorial public health concern. Preventive Medicine, 152, 106772. 10.1016/j.ypmed.2021.106772 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Huttar CM, & BrintzenhofeSzoc K (2020). Virtual reality and computer simulation in social work education: A systematic review. Journal of Social Work Education, 56(1), 131–141. 10.1080/10437797.2019.1648221 [DOI] [Google Scholar]
  19. Jacobs SR, Weiner BJ, Reeve BB, Hofmann DA, Christian M, & Weinberger M (2015). Determining the predictors of innovation implementation in healthcare: a quantitative analysis of implementation effectiveness. BMC Health Services Research, 15, 1–13. 10.1186/s12913-014-0657-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. King CA, Horwitz A, Czyz E, & Lindsay R (2017). Suicide risk screening in healthcare settings: identifying males and females at risk. Journal of Clinical Psychology in Medical Settings, 24, 8–20. http://dx.doi.org/10.1007%2Fs10880-017-9486-y [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Kirchner JE, Smith JL, Powell BJ, Waltz TJ, & Proctor EK (2020). Getting a clinical innovation into practice: an introduction to implementation strategies. Psychiatry Research, 283, 112467. 10.1016/j.psychres.2019.06.042 [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Kourgiantakis T, Sewell KM, Hu R, Logan J, & Bogo M (2020). Simulation in social work education: A scoping review. Research on Social Work Practice, 30(4), 433–450. 10.1177/1049731519885015 [DOI] [Google Scholar]
  23. Luoma JB, Martin CE, & Pearson JL (2002). Contact with mental health and primary care providers before suicide: a review of the evidence. American Journal of Psychiatry, 159(6), 909–916. 10.1176/appi.ajp.159.6.909 [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Mackelprang JL, Karle J, Reihl KM, & Cash RE (2014). Suicide intervention skills: Graduate training and exposure to suicide among psychology trainees. Training and Education in Professional Psychology, 8(2), 136. 10.1037/tep0000050 [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Magill M, Mastroleo NR, & Martino S (2022). Technology-based methods for training counseling skills in behavioral health: A scoping review. Journal of Technology in Behavioral Science, 7(3), 325–336. 10.1007/s41347-022-00252-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Mallonee S, Phillips J, Holloway K, & Riggs D (2018). Training providers in the use of evidence-based treatments: A comparison of in-person and online delivery modes. Psychology Learning & Teaching, 17(1), 61–72. 10.1177/1475725717744678 [DOI] [Google Scholar]
  27. Martinez-Ales G, Hernandez-Calle D, Khauli N, & Keyes KM (2020). Why are suicide rates increasing in the United States? Towards a multilevel reimagination of suicide prevention. Behavioral Neurobiology of Suicide and Self Harm, 1–23. 10.1007/7854_2020_158 [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Mastroleo NR, Humm L, Williams CM, Kiluk BD, Hoadley A, & Magill M (2020). Initial testing of a computer-based simulation training module to support clinicians’ acquisition of CBT skills for substance use disorder treatment. Journal of Substance Abuse Treatment, 114, 108014. 10.1016/j.jsat.2020.108014 [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Morse G, Salyers MP, Rollins AL, Monroe-DeVita M, & Pfahler C (2012). Burnout in mental health services: A review of the problem and its remediation. Administration and Policy in Mental Health and Mental Health Services Research, 39, 341–352. http://dx.doi.org/10.1007%2Fs10488-011-0352-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Moscardini EH, Hill RM, Dodd CG, Do C, Kaplow JB, & Tucker RP (2020). Suicide safety planning: Clinician training, comfort, and safety plan utilization. International Journal of Environmental Research and Public Health, 17(18), 6444. 10.3390/ijerph17186444 [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. O’Brien KHM, Fuxman S, Humm L, Tirone N, Pires WJ, Cole A, & Grumet JG (2019). Suicide risk assessment training using an online virtual patient simulation. MHealth, 5. 10.21037/mhealth.2019.08.03 [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. O’Brien KHM, Quinlan K, Humm L, Cole A, Pires WJ, Jacobs A, & Grumet JG (2022). A qualitative study of provider feedback on the feasibility and acceptability of virtual patient simulations for suicide prevention training. MHealth, 8. 10.21037/mhealth-22-15 [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Padgett DK (2016). Qualitative methods in social work research (Vol. 36). Sage publications. [Google Scholar]
  34. Perepletchikova F, Hilt LM, Chereji E, & Kazdin AE (2009). Barriers to implementing treatment integrity procedures: survey of treatment outcome researchers. Journal of Consulting and Clinical Psychology, 77(2), 212. 10.1037/a0015232 [DOI] [PubMed] [Google Scholar]
  35. Putney JM, Collin CR, Halmo R, Cadet T, & O’Brien K (2021). Teaching note—Assessing competence in screening and brief intervention using online patient simulation and critical self-reflection. Journal of Social Work Education, 57(3), 598–603. 10.1080/10437797.2019.1671276 [DOI] [Google Scholar]
  36. Saldaña J (2021). The coding manual for qualitative researchers (4th ed.). SAGE Publications, Inc. [Google Scholar]
  37. Saxe Zerden LD, Lombardi BM, & Jones A (2019). Social workers in integrated health care: Improving care throughout the life course. Social Work in Health Care, 58(1), 142–149. 10.1080/00981389.2019.1553934 [DOI] [PubMed] [Google Scholar]
  38. Sholomskas DE, & Carroll KM (2006). One small step for manuals: Computer-assisted training in twelve-step facilitation. Journal of Studies on Alcohol, 67(6), 939–945. 10.15288/jsa.2006.67.939 [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Silva C, Smith AR, Dodd DR, Covington DW, & Joiner TE (2016). Suicide-related knowledge and confidence among behavioral health care staff in seven states. Psychiatric Services, 67(11), 1240–1245. 10.1176/appi.ps.201500271 [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Smith MJ, Bornheimer LA, Li J, Blajeski S, Hiltz B, Fischer DJ, … & Ruffolo M (2021). Computerized clinical training simulations with virtual clients abusing alcohol: Initial feasibility, acceptability, and effectiveness. Clinical Social Work Journal, 49, 184–196. 10.1007/s10615-020-00779-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Stanley B, Brown G, Brent DA, Wells K, Poling K, Curry J, … & Hughes J (2009). Cognitive-behavioral therapy for suicide prevention (CBT-SP): treatment model, feasibility, and acceptability. Journal of the American Academy of Child & Adolescent Psychiatry, 48(10), 1005–1013. 10.1097/CHI.0b013e3181b5dbfe [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Stanley B, Brown GK, Brenner LA, Galfalvy HC, Currier GW, Knox KL, … & Green KL (2018). Comparison of the safety planning intervention with follow-up vs usual care of suicidal patients treated in the emergency department. JAMA Psychiatry, 75(9), 894–900. 10.1001/jamapsychiatry.2018.1776 [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Wakai S, Schilling EA, Aseltine RH Jr, Blair EW, Bourbeau J, Duarte A, … & Welsh A (2020). Suicide prevention skills, confidence and training: Results from the Zero Suicide Workforce Survey of behavioral health care professionals. SAGE Open Medicine, 8, 2050312120933152. 10.1177/2050312120933152 [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Washburn M, Parrish DE, & Bordnick PS (2020). Virtual patient simulations for brief assessment of mental health disorders in integrated care settings. Social Work in Mental Health, 18(2), 121–148. https://www.tandfonline.com/doi/full/10.1080/15332985.2017.1336743 [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. World Health Organization (2021). Suicide worldwide in 2019: global health estimates. License: CC BY-NC-SA 3.0 IGO.
  46. Zalsman G, Hawton K, Wasserman D, van Heeringen K, Arensman E, Sarchiapone M, … & Zohar J (2016). Suicide prevention strategies revisited: 10-year systematic review. The Lancet Psychiatry, 3(7), 646–659. 10.1016/s2215-0366(16)30030-x [DOI] [PubMed] [Google Scholar]

RESOURCES