Abstract
The SARS-CoV-2 pandemic led to the rapid transition of many research studies from in-person to telephone follow-up globally. For mental health research in low-income settings, tele-follow-up raises unique safety concerns due to the potential of identifying suicide risk in participants who cannot be immediately referred to in-person care. We developed and iteratively adapted a telephone-delivered protocol designed to follow a positive suicide risk assessment (SRA) screening. We describe the development and implementation of this SRA protocol during follow-up of a cohort of adults with depression in Malawi enrolled in the Sub-Saharan Africa Regional Partnership for Mental Health Capacity Building (SHARP) randomized control trial during the COVID-19 era. We assess protocol feasibility and performance, describe challenges and lessons learned during protocol development, and discuss how this protocol may function as a model for use in other settings. Transition from in-person to telephone SRAs was feasible and identified participants with suicidal ideation (SI). Follow-up protocol monitoring indicated a 100% resolution rate of SI in cases following the SRA during this period, indicating that this was an effective strategy for monitoring SI virtually. Over 2% of participants monitored by phone screened positive for SI in the first six months of protocol implementation. Most were passive risk (73%). There were no suicides or suicide attempts during the study period. Barriers to implementation included use of a contact person for participants without personal phones, intermittent network problems, and pre-paid phone plans delaying follow-up. Delays in follow-up due to challenges with reaching contact persons, intermittent network problems, and pre-paid phone plans should be considered in future adaptations. Future directions include validation studies for use of this protocol in its existing context. This protocol was successful at identifying suicide risk levels and providing research assistants and participants with structured follow-up and referral plans. The protocol can serve as a model for virtual SRA development and is currently being adapted for use in other contexts.
Introduction
Mental health disorders are among the leading causes of death and disability worldwide, especially in low- and middle-income countries (LMICs). Globally, mental health disorders account for nearly a third of years lived with disability and are the fifth-leading cause of disability-adjusted life years (DALYs), making up 13% of DALYs worldwide [1–3]. Aproximately three quarters of this burden resides in LMICs [4]. In addition to morbidity and disability, untreated mental health disorders are associated with early mortality with an estimated 75% of suicides occurring in LMICs [5]. Yearly prevalence of suicide attempt and suicidal behavior in Malawi is approximately 0.8% and 7.9%, respectively [6]. While effective and low cost treatments are available for the most common mental disorders, the treatment gap is yawning [7, 8]. The proportion of mental health workers in LMICs is as low as 2 per 100,000 population; accordingly, most affected individuals in LMICs do not receive care [9]. The average disparity between those with mental illness in need of care and those who receive services is over 90% in most LMICs [10–14]. The extremely limited access to and availability of national or local suicide hotlines in this context further indicated need for a telephone-based Suicide Risk Assessment (SRA) during the SARS-CoV-2.
The limited availability of mental health services makes identification and management of suicidal thoughts and behavior, often referred to as suicidality, especially challenging. Such behavior can include passive and active suicidal ideation (SI), suicide attempts, and death due to suicide [15, 16]. Passive SI is defined as having thoughts of suicide without the intention to act on suicidal thoughts, while active SI is defined as having suicidal thoughts and intention to act on such thoughts [15, 17]. Passive and active SI typically require different clinical interventions. Conducting mental health research often requires SI measurement, with careful assessment and triage, and is traditionally dependent on in-person assessment with the ability to immediately engage participants if clinical safety measures are needed [18–27].
The SARS-CoV-2 pandemic disrupted and further limited mental health care by precluding many of these face-to-face assessments [28]. The pandemic forced ongoing studies of mental heath in LMICs to quickly pivot from in-person to virtual SI assessments with consideration of how to safely triage and respond to suicide risk by telephone. Increased suicidal risk has previously been associated with epidemics, with ongoing studies suggesting the possibility of increased risk of suicide during the current SARS-CoV-2 pandemic [29–32]. Further, assessment of the feasibility of adapting in-person mental health assessments to virtual assessments in low-resource settings during the pandemic is essential and missing from scientific literature.
In the midst of an ongoing clinical trial integrating depression treatment in non-communicable disease (NCD) settings in Malawi, researchers and participants in the Sub-Saharan Africa Regional Partnership for Mental Health Capacity Building (SHARP) study underwent a rapid change from in-person suicidality assessments to telephone-based assessments by developing a protocol to provide a robust, feasible, and rapidly adaptable protocol to assess suicidality in a timely, safe, quality, affordable, and virtual manner [33–35].
This paper describes the feasibility of development and implementation of this suicidality assessment protocol. We aim to: 1) assess the feasibility (defined as the successful use of the tool in this study’s context) of the protocol for assessing suicidality over telephone among patients who screen positive during the Suicide Risk Assessment (SRA), 2) describe the information we were able to collect and categorize, 3) describe the primary challenges identified and lessons learned during protocol development, and 4) discuss future developments and protocol use in other settings [36, 37].
Methods
The parent, SHARP scale-up study is an implementation science trial comparing the success of a basic versus an enhanced implementation package in achieving integration of depression treatment with NCD treatment at 10 NCD clinics in Malawi [33, 35]. Clinics in the parent study were randomized with a 1:1 ratio to a basic or enhanced implementation strategy to compare implementation and effectiveness outcomes and cost-effectiveness of the two strategies. Patients receiving care at each clinic who meet eligibility criteria described below were invited to enroll in the study to provide clinical outcome measures. In this paper, we describe the feasibility, development, and implementation of the SHARP Safety Response Protocol for Phone Interviews in response to the SARS-CoV-2 pandemic. We do not report results of the parent study trial.
Participants
The cohort of patients (n = 946) consisted of participants meeting the following parent study inclusion criteria: aged 18–65 years and being a patient in care for diabetes or hypertension management at a participating NCD clinic. Of these patients, n = 739 had elevated depressive symptoms (a score ≥5 on the Patient Health Questionnaire-9 [PHQ-9]) at baseline and were eligible for inclusion in this analysis. Exclusion criteria are history of bipolar or psychotic disorder and/or an emergent self-harm threat. One participant was excluded from analysis due to being a prevalent SI case already receiving regular SI follow-up prior to the study period. Among the n = 738 participants eligible for analysis, n = 602 SI screenings were conducted during the study period.
Ethical approval
All study materials and research activities have been approved by the National Health Sciences Research Committee of Malawi (NHSRC; Approval # 17–3110) and the Biomedical Institutional Review Board of the University of North Carolina at Chapel Hill (Approval #17/11/1925). All methods were performed in accordance with the relevant guidelines and regulations. The current manuscript does not report results of the parent study. All participants gave written, informed consent to participate in the parent study. Written informed consent clearly explained the purpose of the research, what will be required of the individual, and the risks and benefits of participation in the participant’s preferred language. All informed consent procedures for the parent study were approved by both ethics committees.
Measures: PHQ-9 and SRA
Participants are asked to complete the PHQ-9, a depression screening tool developed and originally validated in high-income countries but subsequently validated in Malawi and many other low-income countries, as part of baseline and follow-up assessments at 3, 6, and 12 months [38–40]. The final PHQ-9 question asks, “During the past two weeks, how many days have you been bothered by thoughts that you would be better off dead or of hurting yourself in some way?” Participants respond with: “0 days”, “1–7 days”, 8–12 days”, or “13 or 14 days”. Any response other than “0 days” was considered a positive screen for suicidal ideation, requiring a follow-up SRA following this protocol’s schedule. The PHQ-9 questionnaire and SRA were translated into Chichewa and Chitumbuka for use in Malawi [41].
Setting
From May 13, 2019 through March 24, 2020, interview data were collected with in-person interviews at 10 NCD clinics in all three regions of Malawi. However, as a result of the SARS-CoV-2 pandemic and the patient population’s high risk for severe complications of COVID-19, all in-person participant contact was put on hold from March 25, 2020 until October 16, 2020. The research team decided to transition to telephone-based follow-up interview data collection, necessitating the development of virtual study and safety assessment and response protocols.
Protocol development
The SRA protocol guides research team members through a series of risk assessments and the creation of a follow-up plan. The protocol was originally designed to be implemented in-person immediately after a participant screens positive on PHQ-9 Question 9 (any response other than “0 days”) or indicates concerns of suicidal thoughts. The tablet-based data collection software, Open Data Kit (ODK), is set to warn the RA to conduct the SRA when the participant answers affirmatively to PHQ-9 Question 9 [42]. If the patient screens positive, the RA is instructed to not leave the patient alone, complete the telephone protocol form, transfer the patient to clinician care, enter the event into the study log for review by team members, and follow up with the participant as indicated by the protocol (SF 1, SF 2). If the participant indicates SI via a method other than the PHQ-9, RA’s also completed the SRA.
The research team met in March 2020 to discuss methods of adapting the in-person protocol to assess and manage suicide risk virtually during, and beyond, the SARS-CoV-2 pandemic. Given the immediate need of a suicidality protocol without time for a full-validation study of the protocol before implementation due to the pandemic, subject matter experts helped develop the guide. Anticipating that hospitals would have increased resource strain, decreased patient capacity, and increased SARS-CoV-2 transmission risk, the team revised the SRA for telephone delivery with branching logic to guide trained research assistants (RAs) in creating a follow-up plan with participants, reducing participants’ immediate suicide risk, engaging a nearby support person, creating a follow-up plan with participants, and referring participants to in-person care when clinically indicated. Research assistants, with a minimum high school education level and training in SHARP study procedures, were trained by study staff and subject matter experts in suicidality and related health protocols and interventions, including psychiatry, health behavior, and epidemiology experts.
The in-person SRA is an algorithmic questionnaire that is administered by phone in Chichewa or Chitumbuka, and responses are translated into English for data entry. Research staff provide feedback about the event to the research coordinator after each time the protocol is administered. The protocol was implemented in April 2020 after an iterative review process including all research team members with diverse areas of subject matter expertise.
Procedure: Protocol training
The RAs received two trainings for telephone SRA implementation from clinical and research study team members. These trainings consisted of two 1-hour formal trainings via Zoom in which team members discussed the protocol and practiced protocol implementation in simulated scenarios. Team members had a follow-up meeting one week later to answer questions that arose while preparing for protocol implementation after the two training sessions.
Procedure: Protocol implementation of initial assessment
Section 1: Suicide risk assessment (SF 1–2)
The aim of Section 1 is to assess level of SI. Part A (SF 1) assesses active versus passive suicidality and Part B assesses severity of active thoughts, if present. The RA assesses the participant’s current self-harm risk in Part C (SF 1), regardless of the participant’s reporting of active versus passive suicidal thoughts in Part A. Suicide risk level is defined based on participant responses to Parts 1A-C. Importantly, “thoughts of hurting yourself right now” (SF 1, C.1) was added to the telephone protocol, as no clinician (who would have asked this question in-person and provided immediate intervention in the pre-COVID period) was available with this virtual assessment. The addition of this question to the telephone protocol allowed immediate, virtual assessment of active high and acute risk status.
Section 2: Confirm patient location
The purpose of Section 2 (SF 1) is to confirm the patient’s location and identify if other individuals are nearby to help the participant as needed. The RA continues to Section 3 if the participant’s suicide risk is active-high or active-emergent. The RA skips to Section 4 if the participant’s suicide risk is passive, active-low, or active-moderate.
Section 3: Safety plan and next steps for active-high or active-emergent
Section 3 (SF 1) is critical in confirming contact information, on-site support persons, and a strategy to assist the patient in reaching mental health care. The RA confirms the participant’s emergency contact and Health Surveillance Assistant (HSA) information. An HSA is a trained community-based health provider who is able to link community members to various health services. The RA asks the participant to go to the nearest health facility to speak with a mental health professional with the help of a nearby trusted contact. If no trusted contact is available, the RA speaks with an emergency contact in the participant’s locator form regarding the immediate safety concern. If no emergency contact is available, the RA assists the participant in contacting a health professional at the nearest health facility and notifies the study’s clinical coordinator at the participant’s health facility.
Section 4: Safety plan and next steps for passive, active-low, or active-moderate
The overall goal of Section 4 (SF 1) is to create a safety and follow-up plan. The RA asks the participant to inform a trusted contact if the participant feels as though they might harm themselves. If the participant does not agree, the RA returns to Section 3 to proceed with the active-high and active-emergent workflow, including asking the participant if they can speak with a nearby individual. If the participant agrees to inform a trusted contact if the participant feels as though they might harm themselves, the RA continues through Section 4 to assess the presence of potentially dangerous items in the home. If the RA cannot confirm that the participant has handed over potentially dangerous items, the RA completes Section 3. Otherwise, the RA provides the participant with referral information to mental health specialists and asks the participant to seek care if they believe they may harm themselves.
The protocol then defines a follow-up schedule based on suicide risk level (SF 1–5). If the RA cannot reach the participant at the follow-up time, the RA calls the participant’s contact person. If the RA is unable to reach the participant’s contact, the RA informs the study coordinator for further guidance.
Procedure: Protocol implementation of Section 5: Saftey follow-up and Section 6: Summary of follow-up contacts
During the scheduled follow-up contacts, the RA asks how the participant is doing, for updates on the agreed upon action plan from the previous call, and screens for SI with PHQ-9 Question 9 (SF 1). If no SI is present, the RA continues with the prescribed follow-up schedule. If suicidal risk is present, the RA restarts the protocol, returning to Section 1.
Passive risk follow-up (SF 1, SF 3)
Follow-up ends if the participant begins with a passive risk and improves to no risk at the one week follow-up. If the participant maintains a passive risk at weekly follow-up for three follow-ups, the RA asks the participant to contact the RA for the 4th follow-up to confirm participant ability to proactively manage their own care. Follow-up ends when the participant successfully contacts the RA for follow-up.
Active-low follow-up (SF 1, SF 3)
If the participant starts with active-low risk, follow-up is conducted weekly until the participant improves to no suicidal risk (in which case follow-up ends) or the participant improves to passive suicidal risk. If the latter, follow-up is completed according to the passive risk schedule. If the participant remains at passive or active-low risk for 3 follow-ups, the RA requests that the participant initiaties the 4th follow-up contact. If the participant successfully contacts the RA for the 4th weekly follow-up, follow-up can end.
Active-moderate follow-up (SF 1, SF 4)
Follow-up is conducted every 3 days for participants beginning with active-moderate risk. If the participant improves to active-low or passive risk, follow-up transitions to weekly follow-up according to the above guidelines. Follow-up can end if the participant improves to no suicidal risk at two consecutive weekly follow-ups. Follow-up ends without asking the participant to contact the RA for the last follow-up if the participant can confirm their health professional’s contact information and that they will contact their mental health professional if thoughts of suicide return. The RA contacts the study clinical coordinator for further direction if the participant maintains active-moderate risk after 3 contacts.
Active high and active-emergent follow-up (SF 1, SF 5)
If the participant begins with active-high or active-emergent risk, follow-up is conducted every day until the participant improves to active-moderate, active-low, or passive risk (in which case follow-up transitions to the follow-up for the participant’s new risk category). If the participant has no suicidal risk at two consecutive weekly follow-ups, follow up can end. The RA is instructed to contact the study coordinator if the participant continues to have active-high or active emergent suicidal risk after 3 contacts.
In all risk levels, the RA instructs the participant to confirm that the participant has mental health professional contact information and will contact the health professional if thoughts of self-harm return. In all cases, the RA fills out the protocol form, including Section 6: Summary of Follow-up Contacts.
Study analysis and statistics
We describe the feasibility of protocol implementation and discuss results of the implementation of our telephone protocol when participants screened positive for suicide risk during baseline, 3-, 6-, and 12-month surveys during the first 6 months of telephone protocol implementation (March 25, 2020-September 25, 2020). We use frequency distributions to highlight screened population characteristics, including demographic characteristics. Feasibility was measured as the proportion of completed follow-ups out of total number of follow-ups due as indicated by the safety response protocol. We report the number of participants screened, the number of follow-ups completed, and the number of suicide risk cases resolved during telephone protocol use. We apply descriptive analyses to outline protocol implementation, including SRA screening and frequency of positive results. All statistical analyses were conducted using STATA IC (Version 16) software [43].
Results
A total of n = 602 follow-up interviews among n = 738 eligible participants were conducted during the study period. The majority of participants were females aged 50 years or older who were married with children (Table 1). The majority (81%) of participants reported mobile phone access at baseline, with 2% having access to a landline at baseline.
Table 1. Baseline characteristics of SHARP participants*.
N (%) | ||
---|---|---|
Age Group | 738 | |
18–29 | 22 (2.98) | |
30–39 | 84 (11.38) | |
40–49 | 196 (26.56) | |
50+ | 436 (59.08) | |
Missing | 0 (0.00) | |
Gender | Female | 581 (78.73) |
Male | 157 (21.27) | |
Missing | 0 (0.00) | |
Employment | Employed | 700 (94.85) |
Unemployed | 34 (4.61) | |
Missing | 4 (0.54) | |
Marital status | Married | 489 (66.26) |
Separated | 68 (9.21) | |
Divorced | 44 (5.96) | |
Widowed | 120 (16.26) | |
Cohabitating with partner | 3 (0.41) | |
Never married | 13 (1.76) | |
Missing | 1 (0.14) | |
Parity ** | 0–4 | 192 (33.05) |
5–6 | 150 (25.82) | |
7–8 | 142 (24.44) | |
9–14 | 95 (16.35) | |
Missing | 2 (0.34) | |
Phone access | Mobile phone | 599 (81.17) |
Non-mobile phone | 15 (2.03) | |
SRA screening in 6 months prior to telephone protocol implementation *** | Yes | 5 (33.33) |
No | 10 (66.67) |
*Of all participants eligible for current analysis (completed 0, 3, 6, and/or 12 month interviews and had a baseline PHQ-9 score of at least 5)
**n children, of female participants only
***Of those who screened positive in during the telephone protocol implementation period
Feasiblity of protocol implementation and challenges
Study RAs used the protocol to successfully identify suicide risk level of all participants who screened positive for SI, with 100% of cases receiving the SRA being resolved. A total of 602 SI screenings took place in the first six months of telephone protocol implementation (during which no new participants were enrolled because of COVID-19 restrictions) (Table 2). During this period, 13 (2%) participants received a SRA after a positive PHQ-9 screening, with follow-up plans varying based on participant risk-level. Three (<1%) participants recieved a SRA after other indication of SI to study staff. In total, 15 (2%) participants received a SRA for any reason. All patients who screened positive received a SRA.
Table 2. Frequency of factors related to Suicide Risk Assessment (SRA).
SRA telephone protocol period* | ||
---|---|---|
Identification of suicidal ideation (SI) | 15 | |
Positive PHQ-9 Question 9 | 12 (1.99) | |
Otherwise reported SI | 3 (0.50) | |
Missing | 0 (0.00) | |
Self-harm or suicidal thoughts in the last 2 weeks among those who screened positive on PHQ-9 Question 9 | 12 | |
1–7 days | 8 (66.67) | |
8–12 days | 1 (8.33) | |
13–14 days | 3 (25.00) | |
Missing | 0 (0.00) | |
Suicide risk level among those who screened positive on PHQ-9 Question 9 or otherwise reported SI | 15 | |
Passive** | 11 (73.33) | |
Active-Low** | 1 (6.67) | |
Active-Moderate | 1 (6.67) | |
Active-High/Active-Acute | 2 (13.33) | |
Number of protocol follow-ups | ||
Mean | 2.40 | |
Median | 2.00 | |
Min | 1 | |
Max | 5 | |
Duration of protocol follow-up (days) | ||
Mean | 13.86 | |
Median | 13.00 | |
Min | 0 | |
Max | 35 |
*SRA via telephone SRA analysis conducted for 6 months of telephone assessments after the start of the SARS-CoV-2 pandemic between March 25, 2020-September 24, 2020. Includes participants screened for SI during this study period.
**In-person assessment appeared to be more sensitive to active-low vs. passive risk in comparion to telephone assessments
***One contact (day of positive screening) only
Most participants had passive risk (n = 11, 73%), followed by active-high (n = 2, 13%), active-moderate (n = 1, 7%), and active-low (n = 1, 7%) risk. Most participants with positive screenings reported suicidal thoughts 1–7 days in the previous two weeks (Table 3). Nearly one third of patients who screened positive during telephone protocol implementation (n = 5, 33%) screened positive at least once in the 6 months prior to telephone protocol implementation. No participants screened positive more than one time during telephone protocol use. There were no suicide attempts or deaths during the telephone implementation period.
Table 3. PHQ-9 Question 9 participant responses among participants screened during the suicidality protocol implementation period.
Telephone PHQ-9 Q9 N (%) | Depressive severity | N (%) | ||
---|---|---|---|---|
Baseline | 0* | 0* | ||
0 days | 0 (0.00) | No depression | 0 (0.00) | |
1–7 days | 0 (0.00) | Mild | 0 (0.00) | |
8–12 days | 0 (0.00) | Moderate | 0 (0.00) | |
13–14 days | 0 (0.00) | Moderately severe | 0 (0.00) | |
Missing | 0 (0.00) | Severe | 0 (0.00) | |
Missing | 0 (0.00) | |||
3 month follow-up | 83 | 83 | ||
0 days | 80 (96.39) | No depression | 46 (55.42) | |
1–7 days | 2 (2.41) | Mild | 33 (39.76) | |
8–12 days | 1 (1.20) | Moderate | 4 (4.82) | |
13–14 days | 0 (0.00) | Moderately severe | 0 (0.00) | |
Missing | 0 (0.00) | Severe | 0 (0.00) | |
Missing | 0 (0.00) | |||
6 month follow-up | 234 | 234 | ||
0 days | 231 (98.72) | No depression | 133 (56.84) | |
1–7 days | 3 (1.28) | Mild | 73 (31.20) | |
8–12 days | 0 (0.00) | Moderate | 21 (8.97) | |
13–14 days | 0 (0.00) | Moderately severe | 5 (2.14) | |
Missing | 0 (0.00) | Severe | 0 (0.00) | |
Missing | 2 (0.85) | |||
12 month follow-up | 285 | 285 | ||
0 days | 279 (97.89) | No depression | 168 (58.95) | |
1–7 days | 3 (1.05) | Mild | 93 (32.63) | |
8–12 days | 0 (0.00) | Moderate | 13 (4.56) | |
13–14 days | 3 (1.05) | Moderately severe | 5 (1.75) | |
Missing | 0 (0.00) | Severe | 4 (1.40) | |
Missing | 2 (0.70) |
*No new patients were enrolled during the study period due to the pandemic
**n = 2 participants are missing overall PHQ-9 score
The telephone protocol successfully guided the monitoring of all patients with SI until resolution of SI risk, with a 100% resolution rate of SI during this period indicating that this was an effective strategy to monitoring SI virtually. The first case during the telephone protocol period required study staff and the clinical lead reminding the RA to implement the new protocol. Only one patient required multiple contact attempts for assessment by the RA during their protocol follow-up period and all other contact attempts were successful on the first try. One participant conducted follow-up calls using a clinic phone during clinical visits. The mean duration of follow-up during the telephone protocol period was 14 days (range: [0, 35]), with the mean number of follow-ups was 2.4 (range: [0, 5]).
Researchers identified several challenges to telephone protocol implementation. Telephone visits were occasionally limited by cell phone access due to short, fixed-term phone plans and intermittent network problems affecting audio quality. Prepaid phones were able to receive calls from the research team and clinicians, but participants who had exhausted their prepaid minutes were unable to call research team members and clinicians until a new phone plan was purchased. These phone plans also resulted in some participants changing telephone numbers and access daily, risking delay of scheduled follow-up contacts or precluding follow-up contacts entirely. The use of a contact person’s phone for follow-up risked delaying completion of SI assessments when the contact person was unwilling or unable to give the phone to the participant.
RAs reported being unsure if the contact person was hiding information about harmful items in the home. Additionally, participants could not always receive the needed services when referred to the nearest health facility, forcing them to weigh the costs associated with traveling long distances to district health facilities to access mental health care. Transport problems from the home to the referred health facilities were a challenge to the referral process due to costs associated with travel. RAs reported that poor support services received by referred patients negatively affected trust between RAs, patients, and health providers in some cases.
Case review
The study team generated adverse event reports weekly. One research coordinator and a psychiatrist reviewed all SI cases weekly. The coordinator asked each RA on a weekly basis if the SI protocol had been implemented in the last week. If so, the RA was instructed to follow up with the research coordinator individually. Noting that the protocol form itself is not submitted to the research team, but functions as a guide followed during telephone assessments, the RAs submitted a narrative of the telephone protocol event to the research team to review.
The research team met weekly to discuss fidelity to the protocol in each case, and one team member followed-up with RAs individually via WhatsApp messaging when more information on a case is needed or discussion about improvements to protocol fidelity was indicated [44]. The research team closed each protocol case when the weekly review team agreed that all protocol components had been completed with fidelity and that suicide risk was no longer present. In these meetings, the team also discussed if protocol modification was needed to address challenges being faced by the research team (e.g., how to modify the protocol in the event of chronic suicidal ideation).
Protocol implementation example
The following case study demonstrates the application of the telephone-based SRA protocol.
Day 0 –Study RA reported that ODK directed them to complete an SRA with Participant A during a routine, phone-based data collection interview. After completing the SRA, the result was Passive SI. The RA followed the ‘Passive SI’ protocol steps and determined that the participant did not have access to items they could use to harm themselves, established contact with the participant’s spouse (should the RA be unable to reach the participant), and provided mental health referral information should the participant’s suicidal thoughts worsen. The participant mentioned to the RA that they did not feel like they were currently a danger to themselves or others and said they would follow-up with the referral.
Day 7 –Study RA contacted the participant by phone one week later with fidelity, consistent with the protocol’s Passive SI guidelines. During this call the participant endorsed that they no longer had thoughts of harming themself by answering “no” to PHQ-9 question 9.
Day 9 –Study clinical lead met with study coordinator to discuss the case. After noting the RA’s fidelity to the protocol and Participant A’s improvement at follow-up, the case was closed resulting in no further follow-up.
One key point illustrated by this case study is that, although the full protocol is very detailed, its implementation typically was quite efficient in practice. In fact, the protocol was not needed for 98% of study contacts, and when it was needed, safety concerns were appropriately handled with a small amount of follow-up contact. Thus, while a detailed protocol that anticipated all possible eventualities and backup processes was essential, the actual added effort for the research team was quite modest.
Discussion
Adding to the baseline challenges of assessing and managing suicidal behavior, the SARS-CoV-2 pandemic has disrupted and limited mental health care by precluding many face-to-face assessments, which has been especially problematic for the assessment and management of suicidality [25, 28]. Currently, 93% of countries are experiencing disruptions to mental healthcare and 24% of countries reported a disruption in suicide prevention programs due to the pandemic [45]. Increased suicidal risk has previously been associated with epidemics, with possibility of increased risk of suicide during the current SARS-CoV-2 pandemic [29–32]. This paper reports on a feasible strategy to monitor suicide risk by phone when in-person assessments are not possible.
The rapid adaptation of SHARP’s in-person SRAs for telephone use during the SARS-CoV-2 pandemic allowed us to continue monitoring SI in an ongoing RCT in Malawi during the pandemic. The possibility of increasing suicide risk during the pandemic is concerning, and adapting and implementing a feasible, virtual protocol that addresses study participant suicide risk events and appropriately refers participants to care is critical [45]. Similar SRA protocols have been adapted for use in RCTs, but few have been developed in non-hospital, emergency, and time-sensitive contexts in which mental health care is severely disrupted [46–49].
Overall, transition from in-person to telephone-based SRAs was feasible and functioned well for patients and research team members. Providing a structured protocol is critical to providing research team members with the support and guidance needed to assess participant suicide risk [46]. The protocol identified participants at risk of suicide and referred participants to clinical settings when indicated. As well, the protocol provided research team members with feasible, structured, and actionable responses to participant SI.
Very few participants had active-low or active-high suicide risk during the telephone protocol period, contrary to what we anticipated based on current literature, subject matter expertise, and prior in-person SRAs [50, 51]. With the addition of the “Thoughts of hurting yourself right now” response option, the protocol treats acute-high and active-emergent cases as equally serious with the same follow-up intensities. As well, the “Might hurt yourself before seeing your doctor again” response option may be less meaningful during pandemic times if clinical visits are postponed. Future directions include validation studies to address these questions.
Importantly, one third of participants who had a positive SRA screening during the telephone protocol implementation had a positive SRA screening in the 6 months prior to telephone protocol implementation, a reminder of the importance of feasible and well-designed SRA follow-up schedules [52, 53].
Challenges cited by research team members include inability to predict certain challenges faced during the pandemic (e.g., changes in governmental and institutional policies about quarantine, travel, and other factors affecting participants), network and audio difficulties, and risk of participant mobile phone access delaying or precluding follow-up visits [54–56]. As some participants had phone access through close contacts reported in SRA Sections 2 and 3, it was criticial to collect this information during assessments and participant follow-up was occasionally limited by access to the contact person’s phone. Phone minute-related challenges limited participant ability to initiate calls with clinicians and research team members, despite being able to receive calls from the research team. RAs identified challenges to referring participants to health centers, as not all centers could provide the needed services. Despite inability to predict all possible challenges, the research team created a protocol and multi-layered review process robust enough to meet challenges and circumstances faced by participants and research staff through iterative adjustment and adaptation.
This study includes a small sample of adult participants, limiting generalizability. Lack of availability of qualitative fidelity data for all events precluded formal analysis of protocol fidelity. However, all RA protocol-related actions and responses were monitored by the research team through a weekly, iterative review process and real-time correspondence via WhatsApp. Finally, it is possible that participant responses to questions about suicidality differ when asked about suicidality by phone versus in person, and possibly undreport symptoms [19]. But, prior SI scales have performed well under rapid, self-reporting conditions conditions [18, 22, 49]. The protocol performed conservatively, to error on classifying patients as having a higher severity status. As well, in-depth, weekly case-reviews of all suicide risk events and RA responses by the research team ensured that the protocol was implemented accurately and that participants received appropriate and timely follow-up and care when clinically indicated.
Research implications include the need to assess the protocol’s sensitivity, specificity, positive predictive value (PPV), and negative predicative value (NPV) [57, 58]. Implications also include identifying and including variables for implicit suicidal thoughts to more accurately identify active-low compared to active-moderate and active-high participants, as this may result in different clinical action indication [58–60].
This protocol can serve as a model for adaptation of adult SI protocols for telephone or virtual use in times of pandemic or not. Further, this protocol can function as a model for use in other contexts, particularly in adult populations in LMICs.
Conclusions
Mental health care barriers magnified by the SARS-CoV-2 pandemic highlight the increasing need to address mental healthcare through technology. In order to conduct mental health research or health care remotely, researchers and clinicians must have access to standardized, reliable, and accurate suicide risk assessment protocols. The telephone suicide risk assessment protocol developed to assess SHARP participant suicide risk during the SARS-CoV-2 pandemic can serve as a model for development of virtual suicide risk assessment protocols, in times of pandemic or not.
Supporting information
Data Availability
The data repository is located at https://nda.nih.gov/edit_collection.html?id=2822. All data used in this analysis are available free of cost to researchers.
Funding Statement
This study received funding from the National Institute of Mental Health (PI: Brian W. Pence, PhD; U19MH113202-01) (https://www.nimh.nih.gov). KRL received support from supplement U19MH113202-04S1 (https://www.nimh.nih.gov). JD received support from supplement U19MH113202-01S2 and T32AI070114 (https://www.nimh.nih.gov; https://sph.unc.edu/epid/epid-student-central/epid-student-funding/). Neither the funding agency nor funders had any part in data collection, data analysis, or decision-making about publication.
References
- 1.Russ SJ, Herbert J, Cooper P, Gunnar MR, Goodyer I, Croudace T, et al. Cortisol levels in response to starting school in children at increased risk for social phobia. Psychoneuroendocrinology. 2012. Apr 1;37(4):462–74. doi: 10.1016/j.psyneuen.2011.07.014 [DOI] [PubMed] [Google Scholar]
- 2.Whiteford HA, Ferrari AJ, Degenhardt L, Feigin V, Vos T. The Global Burden of Mental, Neurological and Substance Use Disorders: An Analysis from the Global Burden of Disease Study 2010. PLoS One. 2015. Feb;10(2):e0116820. doi: 10.1371/journal.pone.0116820 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Vigo D, Thornicroft G, Atun R. Estimating the true global burden of mental illness. The Lancet Psychiatry. 2016. Feb 1;3(2):171–8. doi: 10.1016/S2215-0366(15)00505-2 [DOI] [PubMed] [Google Scholar]
- 4.Gureje O, Seedat S, Kola L, Appiah-Poku J, Othieno C, Harris B, et al. Partnership for mental health development in Sub-Saharan Africa (PaM-D): a collaborative initiative for research and capacity building. Epidemiology and Psychiatric Sciences. 2019. Aug;28(4):389–96. doi: 10.1017/S2045796018000707 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Bantjes J, Iemmi V, Coast E, Channer K, Leone T, McDaid D, et al. Poverty and suicide research in low- and middle-income countries: systematic mapping of literature published in English and a proposed research agenda. Global Mental Health [Internet]. 2016. ed [cited 2021 Nov 8];3. Available from: http://www.cambridge.org/core/journals/global-mental-health/article/poverty-and-suicide-research-in-low-and-middleincome-countries-systematic-mapping-of-literature-published-in-english-and-a-proposed-research-agenda/320D28C103EC8F27C5111F8D26CA8D33 doi: 10.1017/gmh.2016.27 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Pengpid S, Peltzer K. Prevalence and correlates of suicidal behaviour among adults in Malawi: a nationally representative cross-sectional survey in 2017. International Journal of Mental Health Systems. 2021. Jun 6;15(1):57. doi: 10.1186/s13033-021-00483-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Patel V, Chisholm D, Parikh R, Charlson FJ, Degenhardt L, Dua T, et al. Addressing the burden of mental, neurological, and substance use disorders: key messages from Disease Control Priorities, 3rd edition. The Lancet. 2016. Apr 16;387(10028):1672–85. doi: 10.1016/S0140-6736(15)00390-6 [DOI] [PubMed] [Google Scholar]
- 8.Singla DR, Kohrt BA, Murray LK, Anand A, Chorpita BF, Patel V. Psychological Treatments for the World: Lessons from Low- and Middle-Income Countries. Annu Rev Clin Psychol. 2017. May 8;13(1):149–81. doi: 10.1146/annurev-clinpsy-032816-045217 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Wang ML, Dorer DJ, Fleming MP, Catlin EA. Clinical Outcomes of Near-Term Infants. Pediatrics. 2004. Aug 1;114(2):372–6. doi: 10.1542/peds.114.2.372 [DOI] [PubMed] [Google Scholar]
- 10.Demyttenaere Koen, Bruffaerts Ronny, Jose Posada-Villa Isabelle Gasquet, Kovess Viviane, Lepine Jean Pierre, et al. Prevalence, Severity, and Unmet Need for Treatment of Mental Disorders in the World Health Organization World Mental Health Surveys. JAMA. 2004. Jun 2;291(21):2581. doi: 10.1001/jama.291.21.2581 [DOI] [PubMed] [Google Scholar]
- 11.Degenhardt L, Glantz M, Evans-Lacko S, Sadikova E, Sampson N, Thornicroft G, et al. Estimating treatment coverage for people with substance use disorders: an analysis of data from the World Mental Health Surveys. World Psychiatry. 2017. Oct;16(3):299–307. doi: 10.1002/wps.20457 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Alonso J, Liu Z, Evans-Lacko S, Sadikova E, Sampson N, Chatterji S, et al. Treatment gap for anxiety disorders is global: Results of the World Mental Health Surveys in 21 countries. Depression and Anxiety. 2018;35(3):195–208. doi: 10.1002/da.22711 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Thornicroft G, Somnath C, Evans-Lacko S, Gruber M, Sampson N, Aguilar-Gaxiola S, et al. Undertreatment of people with major depressive disorder in 21 countries. The British Journal of Psychiatry. 2017. Feb;210(2):119–24. doi: 10.1192/bjp.bp.116.188078 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Mental health: massive scale-up of resources needed if global targets are to be met [Internet]. WHO. World Health Organization; 2018 [cited 2021 May 27]. http://www.who.int/mental_health/evidence/atlas/atlas_2017_web_note/en/
- 15.Booniam S, Wongpakaran T, Lerttrakarnnon P, Jiraniramai S, Kuntawong P, Wongpakaran N. Predictors of Passive and Active Suicidal Ideation and Suicide Attempt Among Older People: A Study in Tertiary Care Settings in Thailand. Neuropsychiatr Dis Treat. 2020. Dec 17;16:3135–44. doi: 10.2147/NDT.S283022 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Sheehan DV, Lecrubier Y, Sheehan KH, Amorim P, Janavs J, Weiller E, et al. The Mini-International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J Clin Psychiatry. 1998;59 Suppl 20:22–33;quiz 34–57. [PubMed] [Google Scholar]
- 17.Barry LC, Wakefield DB, Trestman RL, Conwell Y. Active and passive suicidal ideation in older prisoners. Crisis: The Journal of Crisis Intervention and Suicide Prevention. 2016;37(2):88–94. doi: 10.1027/0227-5910/a000350 [DOI] [PubMed] [Google Scholar]
- 18.Harbauer G, Ring M, Schuetz C, Andreae A, Haas S. Suicidality assessment with PRISM-S—Simple, fast, and visual: A brief nonverbal method to assess suicidality in adolescent and adult patients. Crisis: The Journal of Crisis Intervention and Suicide Prevention. 2013;34(2):131–6. doi: 10.1027/0227-5910/a000164 [DOI] [PubMed] [Google Scholar]
- 19.Viguera AC, Milano N, Laurel R, Thompson NR, Griffith SD, Baldessarini RJ, et al. Comparison of Electronic Screening for Suicidal Risk With the Patient Health Questionnaire Item 9 and the Columbia Suicide Severity Rating Scale in an Outpatient Psychiatric Clinic. Psychosomatics. 2015. Sep 1;56(5):460–9. doi: 10.1016/j.psym.2015.04.005 [DOI] [PubMed] [Google Scholar]
- 20.Forkmann T, Spangenberg L, Rath D, Hallensleben N, Hegerl U, Kersting A, et al. Assessing suicidality in real time: A psychometric evaluation of self-report items for the assessment of suicidal ideation and its proximal risk factors using ecological momentary assessments. Journal of Abnormal Psychology. 2018. Nov;127(8):758–69. doi: 10.1037/abn0000381 [DOI] [PubMed] [Google Scholar]
- 21.Ellis TE, Rufino KA, Allen JG. A controlled comparison trial of the Collaborative Assessment and Management of Suicidality (CAMS) in an inpatient setting: Outcomes at discharge and six-month follow-up. Psychiatry Research. 2017. Mar 1;249:252–60. doi: 10.1016/j.psychres.2017.01.032 [DOI] [PubMed] [Google Scholar]
- 22.Garza NDL, Rush AJ, Killian MO, Grannemann BD, Carmody TJ, Trivedi MH. The Concise Health Risk Tracking Self-Report (CHRT-SR) assessment of suicidality in depressed outpatients: A psychometric evaluation. Depression and Anxiety. 2019;36(4):313–20. doi: 10.1002/da.22855 [DOI] [PubMed] [Google Scholar]
- 23.Stefansson J, Nordström P, Jokinen J. Suicide Intent Scale in the prediction of suicide. Journal of Affective Disorders. 2012. Jan;136(1–2):167–71. doi: 10.1016/j.jad.2010.11.016 [DOI] [PubMed] [Google Scholar]
- 24.Freedenthal S. Assessing the Wish to Die: A 30-Year Review of the Suicide Intent Scale. Arch Suicide Res. 2008;12(4):277–98. doi: 10.1080/13811110802324698 [DOI] [PubMed] [Google Scholar]
- 25.Posner K, Oquendo MA, Gould M, Stanley B, Davies M. Columbia Classification Algorithm of Suicide Assessment (C-CASA): Classification of Suicidal Events in the FDA’s Pediatric Suicidal Risk Analysis of Antidepressants. Am J Psychiatry. 2007. Jul;164(7):1035–43. doi: 10.1176/ajp.2007.164.7.1035 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Lindh ÅU, Beckman K, Carlborg A, Waern M, Salander Renberg E, Dahlin M, et al. Predicting suicide: A comparison between clinical suicide risk assessment and the Suicide Intent Scale. Journal of Affective Disorders. 2020. Feb 15;263:445–9. doi: 10.1016/j.jad.2019.11.131 [DOI] [PubMed] [Google Scholar]
- 27.Beck RW, Morris JB, Beck AT. Cross-Validation of the Suicidal Intent Scale. Psychol Rep. 1974. Apr 1;34(2):445–6. doi: 10.2466/pr0.1974.34.2.445 [DOI] [PubMed] [Google Scholar]
- 28.Huang C, Wang Y, Li X, Ren L, Zhao J, Hu Y, et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. The Lancet. 2020. Feb 15;395(10223):497–506. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Reger MA, Stanley IH, Joiner TE. Suicide Mortality and Coronavirus Disease 2019—A Perfect Storm? JAMA Psychiatry. 2020. Nov 1;77(11):1093. doi: 10.1001/jamapsychiatry.2020.1060 [DOI] [PubMed] [Google Scholar]
- 30.Zortea TC, Brenna CTA, Joyce M, McClelland H, Tippett M, Tran MM, et al. The impact of infectious disease-related public health emergencies on suicide, suicidal behavior, and suicidal thoughts: A systematic review. Crisis: The Journal of Crisis Intervention and Suicide Prevention [Internet]. 2020. Oct 16 [cited 2021 Jan 22]; Available from: https://auth.lib.unc.edu/ezproxy_auth.php?url=http://search.ebscohost.com/login.aspx?direct=true&db=pdh&AN=2020-78674-001&site=ehost-live&scope=site [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Cheung YT, Chau PH, Yip PSF. A revisit on older adults suicides and Severe Acute Respiratory Syndrome (SARS) epidemic in Hong Kong. International Journal of Geriatric Psychiatry. 2008;23(12):1231–8. doi: 10.1002/gps.2056 [DOI] [PubMed] [Google Scholar]
- 32.Ammerman BA, Burke TA, Jacobucci R, McClure K. Preliminary investigation of the association between COVID-19 and suicidal thoughts and behaviors in the U.S. Journal of Psychiatric Research. 2021. Feb 1;134:32–8. doi: 10.1016/j.jpsychires.2020.12.037 [DOI] [PubMed] [Google Scholar]
- 33.Akiba CF, Go V, Mwapasa V, Hosseinipour M, Gaynes BN, Amberbir A, et al. The Sub-Saharan Africa Regional Partnership (SHARP) for Mental Health Capacity Building: a program protocol for building implementation science and mental health research and policymaking capacity in Malawi and Tanzania. Int J Ment Health Syst [Internet]. 2019. Nov 9 [cited 2020 Jun 15];13. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6842238/ doi: 10.1186/s13033-019-0327-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Interim Briefing Note Addressing Mental Health and Psychosocial Aspects of COVID-19 Outbreak (developed by the IASC’s Reference Group on Mental Health and Psychosocial Support) | IASC [Internet]. Geneva: Inter-Agency Standing Committee (IASC); 2020. Feb [cited 2021 Jan 20] p. 15. https://interagencystandingcommittee.org/iasc-reference-group-mental-health-and-psychosocial-support-emergency-settings/interim-briefing [Google Scholar]
- 35.Gaynes BN, Akiba CF, Hosseinipour MC, Kulisewa K, Amberbir A, Udedi M, et al. The Sub-Saharan Africa Regional Partnership (SHARP) for Mental Health Capacity-Building Scale-Up Trial: Study Design and Protocol. Psychiatr Serv. 2021. Jul 1;72(7):812–21. doi: 10.1176/appi.ps.202000003 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda. Adm Policy Ment Health. 2011;38(2):65–76. doi: 10.1007/s10488-010-0319-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Karsh B. Beyond usability: designing effective technology implementation systems to promote patient safety. Qual Saf Health Care. 2004. Oct;13(5):388–94. doi: 10.1136/qhc.13.5.388 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Cholera R, Gaynes BN, Pence BW, Bassett J, Qangule N, Macphail C, et al. Validity of the patient health questionnaire-9 to screen for depression in a high-HIV burden primary healthcare clinic in Johannesburg, South Africa. Journal of Affective Disorders. 2014. Oct 1;167:160–6. doi: 10.1016/j.jad.2014.06.003 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Gilbody S, Richards D, Brealey S, Hewitt C. Screening for Depression in Medical Settings with the Patient Health Questionnaire (PHQ): A Diagnostic Meta-Analysis. J GEN INTERN MED. 2007. Nov 1;22(11):1596–602. doi: 10.1007/s11606-007-0333-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Monahan PO, Shacham E, Reece M, Kroenke K, Ong’or WO, Omollo O, et al. Validity/Reliability of PHQ-9 and PHQ-2 Depression Scales Among Adults Living with HIV/AIDS in Western Kenya. J GEN INTERN MED. 2008. Nov 20;24(2):189. doi: 10.1007/s11606-008-0846-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Udedi M, Muula AS, Stewart RC, Pence BW. The validity of the patient health Questionnaire-9 to screen for depression in patients with type-2 diabetes mellitus in non-communicable diseases clinics in Malawi. BMC Psychiatry [Internet]. 2019. [cited 2021 May 28];19. Available from: http://www.proquest.com/docview/2193965614/abstract/C51342FC4AFE499FPQ/1 doi: 10.1186/s12888-019-2062-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Open data kit: Proceedings of the 4th ACM/IEEE International Conference on Information and Communication Technologies and Development [Internet]. [cited 2021 Feb 21]. https://dl.acm.org/doi/10.1145/2369220.2369236
- 43.StataCorp. Stata Statistical Software: Release 16. College Station, TX: StataCorp LLC; 2019. [Google Scholar]
- 44.WhatsApp Messaging [Internet]. Mountain View, California: Facebook, Inc.; 2020. https://www.whatsapp.com/?lang=en
- 45.The impact of COVID-19 on mental, neurological and substance use services [Internet]. [cited 2021 Jan 11]. https://www.who.int/publications-detail-redirect/978924012455
- 46.Dunn SL, DeVon HA, Collins EG, Luong A, Buursma MP, Gutierrez-Kapheim M, et al. Suicide Risk Management Protocol for a Randomized Controlled Trial of Cardiac Patients Reporting Hopelessness. Nursing Research. 2021. Jan;70(1):72–9. doi: 10.1097/NNR.0000000000000474 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Park SW, Lee JH, Lee EK, Song JJ, Park HS, Hwang SY, et al. Development of the Suicide Risk Scale for Medical Inpatients. Journal of Korean Medical Science. 2018. Jan 15;33. doi: 10.3346/jkms.2018.33.e18 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Arias SA, Zhang Z, Hillerns C, Sullivan AF, Boudreaux ED, Miller I, et al. Using Structured Telephone Follow-up Assessments to Improve Suicide-Related Adverse Event Detection. Suicide and Life-Threatening Behavior. 2014;44(5):537–47. doi: 10.1111/sltb.12088 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Herbeck Belnap B, Schulberg HC, He F, Mazumdar S, Reynolds CF, Rollman BL. Electronic protocol for suicide risk management in research participants. Journal of Psychosomatic Research. 2015. Apr 1;78(4):340–5. doi: 10.1016/j.jpsychores.2014.12.012 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Large M, Kaneson M, Myles N, Myles H, Gunaratne P, Ryan C. Meta-Analysis of Longitudinal Cohort Studies of Suicide Risk Assessment among Psychiatric Patients: Heterogeneity in Results and Lack of Improvement over Time. PLoS One. 2016. Jun;11(6):e0156322. doi: 10.1371/journal.pone.0156322 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Park S, Lee Y, Youn T, Kim BS, Park JI, Kim H, et al. Association between level of suicide risk, characteristics of suicide attempts, and mental disorders among suicide attempters. BMC Public Health [Internet]. 2018. Apr 11 [cited 2021 Mar 3];18. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5896087/ doi: 10.1186/s12889-018-5387-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Exbrayat S, Coudrot C, Gourdon X, Gay A, Sevos J, Pellet J, et al. Effect of telephone follow-up on repeated suicide attempt in patients discharged from an emergency psychiatry department: a controlled study. BMC Psychiatry [Internet]. 2017. [cited 2021 Jan 11];17. Available from: http://search.proquest.com/docview/1883038900/abstract/8C34DC3F5C1D4150PQ/1 doi: 10.1186/s12888-017-1258-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.Miller IW, Camargo CA, Arias SA, Sullivan AF, Allen MH, Goldstein AB, et al. Suicide Prevention in an Emergency Department Population: The ED-SAFE Study. JAMA Psychiatry. 2017. Jun 1;74(6):563. doi: 10.1001/jamapsychiatry.2017.0678 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.Aranda-Jan CB, Mohutsiwa-Dibe N, Loukanova S. Systematic review on what works, what does not work and why of implementation of mobile health (mHealth) projects in Africa. BMC Public Health. 2014. Dec;14(1):188. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Haberer JE, Kiwanuka J, Nansera D, Wilson IB, Bangsberg DR. Challenges in Using Mobile Phones for Collection of Antiretroviral Therapy Adherence Data in a Resource-Limited Setting. AIDS Behav. 2010. Dec 1;14(6):1294–301. doi: 10.1007/s10461-010-9720-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 56.Chang LW, Kagaayi J, Arem H, Nakigozi G, Ssempijja V, Serwadda D, et al. Impact of a mHealth Intervention for Peer Health Workers on AIDS Care in Rural Uganda: A Mixed Methods Evaluation of a Cluster-Randomized Trial. AIDS Behav. 2011. Jul 8;15(8):1776. doi: 10.1007/s10461-011-9995-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.Runeson B, Odeberg J, Pettersson A, Edbom T, Adamsson IJ, Waern M. Instruments for the assessment of suicide risk: A systematic review evaluating the certainty of the evidence. PLoS One. 2017. Jul;12(7):e0180292. doi: 10.1371/journal.pone.0180292 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58.Bolton JM, Gunnell D, Turecki G. Suicide risk assessment and intervention in people with mental illness. BMJ. 2015. Nov 9;351:h4978. doi: 10.1136/bmj.h4978 [DOI] [PubMed] [Google Scholar]
- 59.Glenn CR, Millner AJ, Esposito EC, Porter AC, Nock MK. Implicit Identification with Death Predicts Suicidal Thoughts and Behaviors in Adolescents. J Clin Child Adolesc Psychol. 2019;48(2):263–72. doi: 10.1080/15374416.2018.1528548 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 60.Greenwald AG, McGhee DE, Schwartz JLK. Measuring individual differences in implicit cognition: The implicit association test. Journal of Personality and Social Psychology. 1998;74(6):1464–80. doi: 10.1037//0022-3514.74.6.1464 [DOI] [PubMed] [Google Scholar]