Developing a Mobile App to Prevent Suicide Based on a Software Development Life Cycle: Application of Ecological Momentary Assessment and Interventions

ABSTRACT

To improve mental health, diverse mobile applications (apps) have been developed to target those who actively use smartphones and in‐phone apps. In this vein, this study developed a mobile app, eRAPPORT, to prevent suicide using ecological momentary assessment and intervention. A brief literature review and mobile app market search were conducted to determine the app's content and modality. The design and contents of eRAPPORT were developed through consulting with contextual and methodological experts for end‐users' need assessment and technician review. For implementation, an experienced mobile app developer collaborated with the research team to create the app, which includes a safety plan, a feature to track suicidal risk and mental health status, evidence‐based information from international and national authorised organisations, an outreach service with online/offline counselling and pop‐ups displaying emergency contacts. Twenty‐five general adults participated in the feasibility study. After using eRAPPORT for a month, they completed a self‐reported questionnaire, followed by an in‐depth interview of functionality, acceptability and safety. Both quantitative and qualitative evaluations assured the moderate level of usability and acceptability due to some features that should be improved before applying those with a high risk of suicide. No critical adverse event was reported. Thus, this eRAPPORT feasibility study provides fundamental information to describe the patient‐centred processes on how to develop a mobile app for suicide prevention. Further study will be conducted to test the app's effectiveness and determine the application in clinical practice for monitoring and preventing suicide risk by collecting real‐time and longitudinal data.

1. Introduction

1.1. Background

Suicide is a serious health concern globally. The Organisation for Economic Cooperation and Development (OECD) and the World Health Organisation (WHO) continuously monitor suicide rates worldwide to evaluate each country's mental health (OECD ²⁰²³; WHO ²⁰²¹). Based on the 2019 WHO report, there were 700,000 deaths globally that represented approximately 1.3% of all deaths (WHO ²⁰²¹). During the COVID‐19 pandemic, countries expressed concerns about rising suicide rates and implemented robust action plans to address mental health issues and suicide risks linked to depression, anxiety and social isolation (Gunnell et al. 2020). Mental illnesses are well known as risk factors for suicide (Rockett et al. 2023). Thus, the WHO (2012) emphasises comprehensive interventions, including early detection of suicide attempts, continuous risk monitoring and providing timely intervention of mental health services.

Ecological momentary assessment (EMA) and ecological momentary intervention (EMI) are useful methodologies to achieve these strategies. EMA is a data collection method to capture an individual's activities, emotions and thoughts in real time within natural environments (Shiffman, Stone, and Hufford ²⁰⁰⁸). It can provide individualised programs by incorporating the intra‐individual data for designing micro‐interventions that are implemented in a timely manner (McDevitt‐Murphy, Luciano, and Zakarian ²⁰¹⁸). The strengths of EMA and EMI lie in their ability to provide repetitive assessments and interventions in real time, adapt to daily living environments and deliver personalised support tailored to individual needs (Blevins et al. 2021; Smith and Juarascio ²⁰¹⁹). Thus, both methodologies have been widely used for developing mHealth interventions and, more recently, digital therapeutics in mental health research and practices (Lee et al. 2023). Previous EMA and EMI mobile applications (apps) have been extensively developed in mental health areas, such as alcohol use (Blevins et al. 2021), eating disorder (Smith and Juarascio 2019), mood disorders (Gual‐Montolio et al. ²⁰²³), and suicide risk (Barrigon et al. 2022; Morgiève et al. ²⁰²⁰). Systematic reviews and meta‐analyses of suicide preventive interventions confirmed that the EMA and EMI are promising in digital health care because of its individualised approach to the specific person compared to the other programs (Jiménez‐Muñoz et al. ²⁰²²). Furthermore, these strengths would bode well with younger generations, current emerging technologies of better user interface/user experience (UI/UX) and high connectivity among end users, infrastructures and health care systems (Grist, Porter, and Stallard ²⁰¹⁸).

Although several mobile applications are available on the market (American Psychological Association [APA] ²⁰²³), there are concerns due to the significance of suicide and the characteristics of individuals at risk of suicide (Morgiève et al. 2020; Primack et al. ²⁰²²). In current app stores, the users can choose a specific app based on the app's review regarding its appropriateness of purpose and content, user feedback, privacy/security measures and functionality (APA ²⁰²³). However, it is difficult to get detailed information on how to develop suicide‐preventive apps, especially targeting individuals with mental health problems and high suicide risk. Digital health care for the vulnerable population should be carefully designed to ensure safe and reasonable use of the product (Ministry of Food and Drug Safety ²⁰²⁰). For example, individuals at suicide risk often experience low mental and physical energy, which makes it difficult for them to consistently perform self‐monitoring and daily reporting tasks (Morgiève et al. 2020). Such an app should be very easy to use and sensitive to meet their needs when dealing with mental health issues in daily life (Dubov et al. 2021; Weisel et al. ²⁰¹⁹). Therefore, the contents and functionality should also be well designed based on the end‐users' perspectives. However, many mobile apps are developed through producer‐led approaches rather than user‐centred approaches, creating a gap in reflecting end users' perspectives and feedback in the design process.

To consider these limitations, our research team developed the eRAPPORT app (emotional Resources and Prevention Platform of Relieving suicidal Thoughts). To enhance the systematic design and strategic implementation of end‐users' needs, eRAPPORT was developed using the framework of a software development life cycle (SDLC). The SDLC is a stepwise procedure for designing, building and sustaining a software project to achieve overall goals, specific objectives, functional criteria and user requirements of the project (Opralova 2008). It has been used to develop information systems, as well as health‐related apps targeting diverse populations, through the processes of planning, analysis, design, development, implementation and maintenance (Opralova 2008). The SDLC framework was selected for developing eRAPPORT because it could reflect the unmet needs from different end users, such as individuals at risk of suicide, their case managers and those responsible for national suicide preventions. Moreover, the SDLC (1) completes standardised execution with a multidisciplinary research team; (2) maintains quality of the software by step‐by‐step procedures; and (3) continuously modifies contextual and technical features considering current literature, mHealth trends and real‐world feasibility (Ehrler, Lovis, and Blondon ²⁰¹⁹).

2. Methods

2.1. Aims

This study developed and tested the feasibility of eRAPPORT, a mobile app designed to prevent suicide. It also provided information on how to develop the app, with details regarding the utilisation of the SDLC framework and EMA and EMI methodologies. Moreover, using a mixed‐methods approach, a feasibility study of a sample of general adults with low suicide risk was conducted to ensure the app's functionality, acceptability and safety before moving on to the clinical trial with those at greater risk of suicide.

2.2. The SDLC Framework

2.2.1. Analysis

First, we conducted a brief literature review of peer‐reviewed papers published in the last 10 years focusing on the application of EMA or EMI for suicide prevention in adults. The search terms included the keywords ‘suicide’, ‘prevention’ and ‘app’ in either Korean or English compatibly. Table 1 shows the 15 relevant studies identified. Second, two app markets, Google Play and Apple App Store, were explored in relation to mental health and suicide. Although both stores share similarities in presenting keyword search results, differences in displayed apps may arise from factors such as the degree of keyword matching, subcategories and features, inclusion of advertisements and variations in the available apps on each platform. Consequently, the research team conducted searches for apps with the keyword ‘suicide’ in both Korean and English on iOS‐based iPhones and iPads and Android‐based smartphones. Overall, 355 apps were initially identified. These apps were manually screened and sorted using Microsoft Excel based on relevance to mental health and suicide prevention. After a manual review, 23 apps were selected for in‐depth analysis (Figure 1).

TABLE 1.

Characteristics of the 15 studies obtained from the literature search.

Author (Year)	Country	Study design	Participants	Major contents of intervention regarding suicide prevention	Feasibility outcomes
Andreasson et al. (2017)	Denmark	Protocol for RCT	546 participants in the national suicide prevention clinics (intervention = 273, TAU = 273)	1. Safety plan 2. Crisis support 3. Helpline 4. Other (recommendation for restricting access to lethal means)	1. App usage 2. Functionality 3. User satisfaction 4. Technical stability
Bruen et al. (2020)	UK	Qualitative: Feasibility study	80 participants in acute adult mental health wards	1. EMA (daily journaling) 2. Baseline and exit assessment 3. Safety plan	1. Acceptability 2. Engagement 3. Functionality 4. User satisfaction 5. Technological issues
Buus et al. (2019)	Denmark	Instrumental case study	Focus groups: 26 (young or adult users, relatives, clinicians) Workshops: 89 participants (Denmark and Australia)	1. Safety plan 2. Crisis support 3. Helpline 4. Other (recommendation for restricting access to lethal means)	1. Engagement 2. Functionality 3. Technological issues
Buus et al. (2020)	Denmark	Qualitative: Focus group study	26 participants in 4 focus groups, including young or adult users, relatives and clinicians	1. Safety plan 2. Crisis support 3. Helpline 4. Other (recommendation for restricting access to lethal means)	1. Acceptability 2. Engagement 3. Functionality 4. Technological issues
Dubov et al. (2021)	USA	Qualitative: Pilot study	16 transgender individuals	1. Safety plan (mood patterns and suggestions) 2. EMA (mood tracking and diary) 3. Helpline (medical service providers and services)	1. Acceptability 2. Engagement 3. Functionality 4. Privacy and trust 5. Usability 6. User satisfaction
Grist, Porter, and Stallard (2018)	UK	Mixed: Phase 1 trial	40 adolescents aged 12–17 years	1. EMI (mood diary) 2. Safety plan 3. Coping techniques/training tools 4. Others (dialectical behaviour therapy, cognitive behavioural therapy, mindfulness and behavioural activation)	1. Acceptability 2. Engagement 3. Functionality 4. Safety 5. Usability
Kennard et al. (2018)	USA	Pilot study for RCT	66 adolescents (intervention = 34, TAU = 32), 12–18 years	1. Safety plan 2. Coping techniques/training tools	1. Acceptability 2. Engagement 3. Safety 4. Usability 5. User satisfaction
Melvin et al. (2019)	Australia	Quantitative: Open‐label single‐group trial	36 participants aged 16–42 years	1. Safety plan 2. Helpline (Emergency card)	1. App usage 2. App feedback 3. Usability
Morgiève et al. (2020)	France	Quantitative: Case series study	14 patients from emergency departments and mental health departments	1. EMA 2. Safety plan 3. Coping techniques/training tools 4. Helpline 5. Others (emotion regulation module, identification of warning signs)	1. Acceptability 2. Engagement 3. Safety 4. Usability
O'Grady et al. (2020)	Ireland	Mixed: Usability testing	18 Students aged 15–29 years	1. EMA 2. Safety plan 3. Crisis support 4. Coping techniques/training tools	1. Acceptability 2. Engagement 3. Safety 4. Usability
O'Toole, Arendt, and Pedersen (2019)	Denmark	RCT	129 participants (intervention = 60, TAU = 69)	1. EMA (daily record keeping) 2. Psychological education 3. Safety plan 4. Helpline 5. Other (digital hope kit and method)	1. Engagement 2. Safety
Pauwels et al. (2017)	Spain	Quantitative: Usability test	21 end users (individuals with suicidal ideation) and 8 expert panel (experts, psychiatrists, psychologists)	1. Safety plan 2. EMA 3. Coping techniques and training tools 4. Other (symptom/behaviour tracking)	1. Acceptability 2. Engagement 3. Implementation 4. Safety 5. Usability 6. User satisfaction
Rodante et al. (2022)	USA	RCT: A pilot cluster	18 participants (intervention = 9, control = 9) aged 18–65 years	1. Safety plan 2. Helpline 3. Psychological education	1. Acceptability 2. Engagement 3. Usability
Shand et al. (2013)	Australia	Protocol for RCT	150 participants (intervention = 75, control = 75)	1. Safety plan 2. Helpline 3. Others (emotion regulation and mindfulness, values identification and goal setting)	1. Acceptability 2. Engagement 3. Safety 4. Security
Tighe et al. (2017)	Australia	Pilot study for RCT	61 participants (intervention = 30, waitlist = 31)	1. Safety plan 2. Helpline 3. Others (emotion regulation and mindfulness, values identification and goal setting)	1. Engagement 2. Safety

Abbreviations: EMA, ecological momentary assessment; EMI, ecological momentary intervention; RCT, randomised controlled trial; TAU, treatment as usual.

FIGURE 1.

PRISMA flowchart in the selection of mobile apps related to suicide prevention.

2.2.2. Design

To specify the design, our research team received feedback from key personnel while developing eRAPPORT, including patients, health care providers and stakeholder partners. Based on the Patient‐Centred Outcomes Research Engagement principles, these individuals are important when developing healthcare programs and interventions, as their feedback is trustworthy, meaningful and extremely useful (Sheridan et al. 2017). We thus conducted semi‐structured interviews with two high‐risk individuals registered at a community suicide prevention centre to understand their unique experiences and perspectives. Additionally, the healthcare providers were invited to verify the selected contents and suggest a suitable design for eRAPPORT. The clinical experts comprised a registered nurse specialising in psychiatric nursing, a psychiatrist and two social workers at a community mental health centre.

The one‐to‐one interviews focused on exploring ‘what they need for preventing suicide’, ‘what they expect from the mobile app’ and ‘how they would continue using the developing app’. The patients indicated a strong need for (1) personalised content based on ‘my data’, especially regarding EMI, and (2) mHealth training and technical support for using eRAPPORT. Furthermore, the clinical care providers stated the following opinions based on their previous experiences with patients at high risk for suicide. First, the safety planning should be placed on the first page and simplified to enter the data. Second, ‘mood tracking’, ‘resources’ and ‘emergency calls’ should be responsive to the user's needs or mood changes. Third, the overall design should be intuitive and simple to use because high dropout or poor adherence is very common in our target group when they felt tired, helpless or negative about self‐care (Grist, Porter, and Stallard ²⁰¹⁸).

2.2.3. Implementation

The research team collaborated with a mobile app developer (CodeCrain Ltd., Seoul, Republic of Korea). The research team was primarily responsible for constructing the contents based on user‐centred perspectives, whereas the developer implemented the mHealth technologies to visualise eRAPPORT as the following procedures:

1. The research team provided a list of details of functional requirements that eRAPPORT should include. For non‐functional prototyping, the research team and app developer discussed how to visualise the implementable technologies within the time and budget limit.

2. For design visualisation, a design document outlining the app's layout and features was created using Microsoft PowerPoint. An art designer and a UX/UI designer carefully collaborated to decide the main colour and themes. To include visual elements that induced rapport building in users with anxious dispositions, we selected low‐saturation orange as the primary colour of the app to give it a calming and cosy feel while keeping in mind the target end‐user's characteristics (Balcombe and De Leo ²⁰²²). We constructed the screen design using Zeplin (Zeplin Inc., San Francisco, CA, USA), a design collaboration tool, together with UI/UX designers and front‐end developers.

3. We consulted three experts using the Delphi method to evaluate the PowerPoint screen design. The experts comprised a registered nurse specialising in psychiatric nursing, a psychiatrist and a nurse in the community mental health centre. After drafting the design, front‐ and back‐end prototypes were developed and stored on the administrator web page.

4. A test‐version prototype was developed to find and fix technical errors. Internally, the research team members independently verified whether the app was produced according to the initial functional requirements and draft design documents. An internal review was conducted examining the workflow, technical and systematic errors and data storage.

5. The research team invited those who were willing to use eRAPPORT with the various Android and iOS devices. The Android system sent an address through a URL where only people who had been invited could install the app, whereas iOS used the TestFlight program (Apple Inc., Cupertino, CA, USA) to allow only authorised individuals to install the app. Overall, 46 people participated in the field tests to check the app's functionality, identify device models that did not work and provide suggestions for improvements. Finally, the prototype was then modified and finalised.

6. The prototype on Android and iOS were submitted to the Google Play Store and Apple Store, respectively, that were open to the public.

2.2.4. Evaluation

To evaluate the feasibility (functionality, acceptability and safety) of eRAPPORT with general adults, we (1) checked the user's access time, (2) confirmed data storage in Amazon web services and (3) conducted an online self‐reported survey and semi‐structured interviews about how they felt. Descriptive analyses for quantitative data and content analysis for qualitative data were performed to summarise the results.

2.3. Participants

Participants were recruited from the community, including universities and hospitals, and through online homepage postings to assume the role of app users and evaluate eRAPPORT's usability and acceptability among the general population. To be eligible to participate in the study, users had to (1) be at least 19 years old, (2) be able to use a smartphone and the app without assistance from others, (3) have a device capable of running eRAPPORT and (4) be able to read and write Korean. Based on relevant studies (McManama O'Brien et al. ²⁰¹⁷; Primack et al. ²⁰²²), we recruited 25 participants considering the 25% dropout rate (Dubov et al. 2021; Kennard et al. ²⁰¹⁸; Primack et al. ²⁰²²).

2.4. Procedure

This study was approved by the Severance Hospital IRB (Approval no. 4‐2023‐0997), and all participants provided voluntary informed consent. Participants were contacted by the researcher at Weeks 2 and 4 of the study periods. They were given 2 weeks to try out the app with the option of dropping out by deactivating the app or continuing for an additional 2 weeks. At Weeks 2 and 4, participants completed an online questionnaire to assess app usability and acceptability, and at the end of the study, an in‐depth interview was conducted to obtain user evaluation and feedback on the app.

Among the 25 participants, one person was excluded due to incomplete data. To ensure that only those who used the app sufficiently were included, users who accessed the app for > 40% of the time during the study period were selected (Bruen et al. 2020; Pauwels et al. ²⁰¹⁷; Primack et al. ²⁰²²). Another participant was excluded as they stopped using the app after 2 weeks. Thus, 23 participants completed the study for 4 weeks and participated in the questionnaire survey and in‐depth interviews.

2.5. Measures and Analyses

The System Usability Scale (SUS) was used to assess eRAPPORT's usability, and the Client Satisfaction Questionnaire (CSQ‐8) was used to assess the acceptability of interventions in digital mental health. Thus, participants evaluated general characteristics of the app, SUS (Weeks 2 and 4) and CSQ‐8 (Weeks 2 and 4) through questionnaires. The quantitative data were statistically analysed using SPSS/WIN 27.0, and descriptive statistics were performed to analyse the data. The qualitative data was summarised using content analysis.

Users were asked to rate eRAPPORT's usability using 10 items from the SUS (Brooke 1996) on a 5‐point Likert scale (1 = strongly disagree to 5 = strongly agree). The sum of the odd and even items was multiplied by 2.5 to obtain the overall SUS score, which was between 0 (very poor usability) and 100 (very good usability). As recommended by Bangor, Kortum, and Miller (2009), a SUS score of ≥ 68 is considered above average, 63–68 is marginally high, 51–62 is marginally low and ≤ 50 is unacceptable. The Cronbach's α of the SUS was 0.83 and 0.90 at Weeks 2 and 4, respectively.

Additionally, users answered the CSQ‐8 (Attkisson and Zwick 1982) to assess eRAPPORT's acceptability. The CSQ‐8 comprises eight items on a 4‐point Likert scale (1 = low satisfaction to 4 = high satisfaction). However, a transformation of the data proposed in recent guidelines to improve the representation and understanding of CSQ‐8 data multiplies the sum of all items by 3.125 to obtain a CSQ score between 25 (very unsatisfied) and 100 (very satisfied) (Attkisson 2020). An acceptable level of satisfaction was ≥ 62.5 on the linearly transformed scoring. The Cronbach's α of the CSQ was 0.93 and 0.95 at Weeks 2 and 4, respectively.

After using the app, in‐depth interviews were conducted via Zoom meetings or phone calls using a semi‐structured questionnaire to discuss user‐friendliness, app design, usefulness of the app in suicide prevention, any adverse events and errors in the app. This process followed Gale et al.'s (2013) framework, which involved transcription, coding, analytical framework development and application, data charting and interpretation. This process systematically codes and organises significant interview content, enabling structured data interpretation. We asked users who stopped using the app after 2 weeks to participate in an in‐depth interview about why they stopped and how the app could be improved, but they declined to participate.

3. Results

3.1. Review of Literature and Apps on the Market

The literature and app review focused on the main contents of the programs, evaluation criteria and implementation strategies with diverse suicide prevention‐related mHealth technologies. After reviewing 15 relevant studies and 23 apps, the primary contents of eRAPPORT were determined, including ‘safety planning’, ‘mood tracking with EMA’, ‘timely help‐seeking for EMI’ and ‘international information and national resources with evidence‐based practice’. Most of the studies and apps were developed in English that focused on general mental health rather than being specific to suicide prevention. Moreover, most researchers and developers seemed to be Information and Communications Technology companies or app developers; hence, it was unclear how mental health researchers and clinicians were involved in the development of these apps.

Specifically, we found the widespread use of the ‘Safety Plan’ concept when developing suicide prevention apps. Several studies structured app contents using EMA throughout the daily diaries. Additionally, based on the relevant app reviews, the two main features were ‘Helpline’ (17 apps, 73.9%) and ‘Safety Plan’ (16 apps, 69.6%) (Appendix S1). Our research team downloaded and used some of them, such as Stay Alive, Relief Link, Reminder, MY3, Safety Net, PSNE Scotland and others. Specifically, we carefully compared each content in terms of a safety plan, suicide prevention information and the available resources among the downloaded apps. Most apps did not show the evidence sources, and the UX/UI were complex, thereby lowering acceptability. Therefore, to overcome these limitations, we decided an evidence‐based approach with user‐friendly designs should be used to develop eRAPPORT.

3.2. Contents of eRAPPORT

Based on the literature and market review, it is determined that eRAPPORT contains five categories: (1) safety planning, (2) mood tracking, (3) mood calendar, (4) resources and (5) information. Although the original app is developed in Korean (Appendix S2), Figure 2 shows the English‐translated version of eRAPPORT.

FIGURE 2.

Main menu of the eRAPPORT app.

3.2.1. Safety Planning

When users logged in, they filled out the first page of safety planning. Safety planning is an evidence‐based technique for preventing suicide in diverse groups that helps reduce the risk of suicidal thoughts and behaviour by facilitating the development of coping strategies (Dubov et al. 2021; Grist, Porter, and Stallard ²⁰¹⁸; Nuij et al. ²⁰²²). The users had to answer five questions, including listing their warning signs, internal coping strategies, social contacts for distraction, people whom they could ask for help and their plan in case of a suicide crisis (Figure 3). After completing these questions, users could report the use of planned strategies if needed.

FIGURE 3.

Screenshot of warning signs, coping strategies, social contacts and environmental improvement strategies in the safety planning.

3.2.2. Daily and Monthly Mood Tracking

Users could document their daily mood at least thrice a day. The questionnaire assessed the users' feelings of happiness, sadness, anxiety, comfort and stress. Mood levels were gauged using the sliding bar, which evaluated mood as percentages (0%, 25%, 50%, 75% and 100%) on a 5‐point Likert‐type scale (Figure 4). Using EMA, mood tracking can determine the user's risk of suicide in real time. When the users recorded their mood at bedtime, the question of suicide ideations and attempts of that day would come up. If they answered ‘Yes’ to any of these questions, the user would be contacted by helpline calls (Figure 5) as the EMI provided real‐time management for the user. Users could also review the history of the moods recorded. Monthly calendars and weekly graphs indicated the mood trends by using friendly emoticons (Figure 6).

FIGURE 4.

Screenshot of users' feelings of happiness, sadness, anxiety, comfort and stress in mood tracking.

FIGURE 5.

Screenshot of the helpline pop‐up with questions about suicide ideations and attempts.

FIGURE 6.

Mood calendars and mood trends using graphs and statistics.

3.2.3. Information and Resources

This section provided available resources that users could access when they felt depressed or required mental health services. We also indicated all suicide prevention centres and relevant organisations on a map of Korea (Figure 7). Additionally, we provided specific details about (1) public mental health services and (2) a mental counselling app and coupons that users could use if they required online counselling. We aimed to correct misunderstandings about depression and suicide for the general population (Figure 8) and provided evidence‐based information retrieved from the WHO, the Ministry of Health and Welfare and the Korea Foundation for Suicide Prevention. For mental health, recent statistics, effective coping skills and up‐to‐date treatments for depressive disorders were also explained. Further information was provided on maintaining physical health and performing self‐care such as diet, medication, sleep and exercise.

FIGURE 7.

Resource of suicide prevention centres and services available across the country.

FIGURE 8.

Information on coping strategies for suicide prevention.

3.3. eRAPPORT's Design and Operation

We developed eRAPPORT by adopting user‐friendly design, intuitive use and high accessibility. Users could access ‘safety planning’ and ‘mood‐tracking’ pages directly from the main page, which also included emergency calls and contacts. They entered such information in the fourth step of safety planning (Figure 9). With only one click, users can call someone to ask for help if they feel in danger. Furthermore, users can choose from various wallpapers to customise their app.

FIGURE 9.

Main page with safety plan, mood tracking, people whom they could ask for help and emergency calls.

3.4. Evaluation of eRAPPORT

3.4.1. User Evaluation

Table 2 provides an overview of the participant's characteristics. Most participants were women (n = 21; 87.5%), with an average age of 30.63 ± 8.64 years. One participant (4.2%) had a history of mental health problems, and none of the participants reported suicidal thoughts while using the app. The combined mean total score of the SUS was 65.63 ± 14.15 at Week 2 and 62.61 ± 14.76 at Week 4, whereas the combined mean total score of the CSQ‐8 was 67.06 ± 12.87 at Week 2 and 69.16 ± 15.32 at Week 4 (Figure 10).

TABLE 2.

Characteristics of the participants.

Variables		n (%)	Mean (SD)
Gender	Men	3 (12.5)
	Women	21 (87.5)
Age	29 or younger	17 (70.8)	30.63 (8.64)
	30–39 years	4 (16.7)
	40–49 years	2 (8.3)
	≥ 50 years	1 (4.2)
Marital status	Married	5 (20.8)
	Single	19 (79.2)
Educational level	High school or lower	3 (12.5)
	College	20 (83.3)
	Graduate school	1 (4.2)
Have underlying diseases?	Yes	3 (12.6)
	No	21 (87.4)
Have you had mental illness?	Yes	1 (4.2)
	No	23 (95.8)

FIGURE 10.

Scores of the System Usability Scale and Client Satisfaction Questionnaire.

Regarding eRAPPORT's usefulness, most users thought that recognising the pattern of their feelings helped implement a safety plan. They also mentioned that the information on eRAPPORT was easy to understand, helpful in directly accessing an emergency hotline and ready to register the contact information of emergency contact persons. However, eight participants (33.3%) were concerned that depressed individuals at risk of suicide might become more vulnerable because frequent self‐reporting might irritate the person when asked to identify their feelings of depression and wondered if they would take the initiative to use the app to request counselling or use the emergency hotline. Although these emotional records were helpful, the users doubted whether such features were effective interventions for suicide prevention. However, none of the study participants reported an increase in suicide ideations or attempts.

Thus, they suggested that careful monitoring should be provided by mental health personnel when using eRAPPORT. Participants provided some suggestions on improving the app's user‐friendly design. In terms of user‐friendliness, most users found the app easy to use, with a clean interface and easy‐to‐understand content. They also expressed that recording their emotions was cathartic, in that they felt like someone was looking after their feelings. The activity also provided the opportunity for self‐reflection. However, some users found it cumbersome to have to answer the same questions repeatedly and complained about the lack of a reminder to record their emotions thrice a day on some devices (Table 3).

TABLE 3.

Participant's experiences and direct quotes.

Themes	Examples of direct quotes
Easy to use	‘I didn't have much difficulty using it, and when I tried to use it as explained, it was easy to use’. (Participant 9)
Repetitive questions, inconvenient	‘It's a bit inconvenient to [have to] check my emotional state three times’. (Participant 19)
Simplified design	‘I think it's neat and good. It's convenient [and] has only the necessary parts and functions…’. (Participant 5)
	‘The app design was okay. It was cute, but I felt like it was a little too simple, like it only had one function…’. (Participant 6)
Useful to reflect on their feelings	‘When I first [registered the contact information of important] people who protect me, it was nice to be able to look at it and be like, ‘Oh, I have people on my side. I can reach out to this person anytime’. It was excellent to be able to write notes like [in] a diary. That memo can be [saved] there… like a daily [self‐record]’. (Participant 8)
Convenient information	‘In terms of contact information, such as psychological counselling centres, it was actually good that they were directly available on the app without having to search for them online.’ (Participant 15)

4. Discussion

We developed the mobile app eRAPPORT to target suicide prevention by integrating the methodologies of EMA and EMI. Our research team followed the systematic method of the SDLC to emphasise the end‐user's needs in developing the mHealth app. The quantitative and qualitative findings indicated that eRAPPORT had well‐designed functionality, reasonable acceptability and low concern regarding safety at this early stage of evaluation through a feasibility study. eRAPPORT has the potential to facilitate the development of individual‐centred suicide prevention for users in real‐time and within real‐world settings compared with current suicide prevention practices. Like eRAPPORT, several apps for suicide prevention using EMA or EMI have been developed, including Strength Within Me (Bruen et al. 2020), TransLife (Dubov et al. 2021), BlueIce (Grist, Porter, and Stallard ²⁰¹⁸), BackUp (Nuij et al. 2022; Pauwels et al. ²⁰¹⁷), mEMA (Nuij et al. 2022), EMMA (Morgiève et al. 2020), SafePlan (O'Grady et al. ²⁰²⁰) and LifeApp'tite Mobile App (O'Toole, Arendt, and Pedersen ²⁰¹⁹). Each app mainly required recording momentary emotions daily. Compared to these apps, eRAPPORT has the following strengths: (1) it was systemically developed based on a structured framework; (2) it adopted a user‐centred design, including multidisciplinary expert opinion, end‐users' need assessment of the high‐risk groups and feasibility tests on acceptability and safety; and (3) it integrated both EMA and EMI for enhancing digital therapeutics.

Our detailed description of eRAPPORT's development processes is helpful for guiding future researchers to develop the mHealth app in digital health care research. Relatively few studies have reported detailed information on the development processes, although the mHealth technology should be carefully designed for considering the impact of adverse events, such as suicide or self‐harm behaviours (Aljedaani and Babar 2021; Jabangwe, Edison, and Duc ²⁰¹⁸). Systematic procedures based on the frameworks, such as SDLC, enable us to assure clinical validity, acceptability of the end users and safety concerns (Ehrler, Lovis, and Blondon ²⁰¹⁹; Mounir, Youssef, and Zulkifli ²⁰²²). Our detailed information on each step of developing eRAPPORT will encourage future app developers or researchers to enhance other mental health apps for the vulnerable populations.

eRAPPORT was acceptable to our feasibility study participants. Slightly lower SUS scores and above‐average CSQ‐8 scores were obtained compared with the average for desirable usability levels, indicating eRAPPORT's moderate usability and high acceptance. The scores for system usability and consumer satisfaction are similar to those of previous studies (Nuij et al. 2022; Rahmadiana et al. ²⁰²¹). The adherence rate for completing the intervention sessions with eRAPPORT was 95.8%, much higher than 52% (Rahmadiana et al. 2021) and 70.6% (Nuij et al. 2022). Reporting adherence could slightly differ between those without and those with suicidal ideation (Nuij et al. 2022), especially young adults with depression or anxiety (Rahmadiana et al. 2021). In general, a higher number of dropouts results in a lower system usability score. An analysis of the factors influencing the discontinuation of app use is important in improving usability, especially if the app requires adherence. In our study, some participants felt that the repetitive surveys were cumbersome, which was reported in previous studies as fatigue (Nuij et al. 2022). As the use of EMA‐based apps is required multiple times a day, obtaining feedback from dropouts can help increase app quality and overall customer satisfaction.

Using SUS and CSQ‐8 as quantitative evaluations, qualitative interviews are very insightful to learn user experience (Følstad 2017). Therefore, both in‐depth interviews and questionnaires are required to obtain comprehensive feedback for a user‐centred approach. Most studies also qualitatively evaluated usability and feasibility (Bruen et al. 2020; Dubov et al. ²⁰²¹; Grist, Porter, and Stallard ²⁰¹⁸; Morgiève et al. ²⁰²⁰; Nuij et al. ²⁰²²; O'Grady et al. ²⁰²⁰; O'Toole, Arendt, and Pedersen ²⁰¹⁹; Pauwels et al. ²⁰¹⁷). However, eRAPPORT includes several key distinctions. Although other suicide prevention apps tend to rely on generic designs, eRAPPORT is tailored according to the perspectives of high‐risk users and mental healthcare providers. Additionally, the apps developed in English generally focus on more Western contexts, whereas eRAPPORT incorporates culturally relevant content tailored to the Korean context. Finally, eRAPPORT is evidence‐based as it integrates safety planning from current practices in mental healthcare, official statistics from a national organisation and other scientific rationale from the literature. eRAPPORT was designed based on reliable information provided by the Korean government and national suicide prevention centres (Ministry of Health and Welfare ^{2019; 2022}; National Center for Disaster and Trauma ²⁰²¹). Its high UX design provides (1) simple frames and features that prevent user fatigue through concise, intuitive visualisations that consider the potentially low level of physical and emotional energy and (2) integrates rigorously tested instruments, consistent with national and other recently published guidelines on suicide prevention (Martinengo et al. 2019; WHO ²⁰¹²) and evidence‐based practices of safety planning (Jiménez‐Muñoz et al. ²⁰²²; Morgiève et al. ²⁰²⁰).

However, there were several limitations in our study that should be noted. First, as this app is in the early phase of digital health care and being evaluated through a feasibility study rather than a clinical trials, our findings cannot be generalised. Thus, the next phase of clinical trials, due to the exploratory nature of the study, should be conducted with larger groups of participants, including more males, those with significant suicide risk and those living in diverse communities, to determine the appropriate usage, effectiveness and scope of the app. Second, along with our preliminary findings of acceptability and safety, the next study would be essential to further ensure safety, as suicide as an adverse event is critically harmful. Carefully designed step‐wise trials should be conducted to ensure both safety and the effectiveness of interventions. Third, only two adults with suicide risk have participated as the lived experts in semi‐structured interviews; thus, future studies should include more such participants to assure the generalisability of the study findings. Fourth, although the SDLC was useful for developing eRAPPORT, it did not provide an in‐depth understanding of mental health and digital behaviour. Moreover, when examining technology usage and health behaviour, it is important to integrate both the methodological and contextual frameworks. Therefore, future digital suicide prevention and intervention studies should integrate the health belief model (Rosenstock 1966) or suicide‐specific theory (Díaz‐Oliván et al. ²⁰²¹) to improve the feasibility of such interventions.

To proceed to the next phase of the clinical trial, we considered the addition of more real‐time digital interventions to increase accessibility and synchronisation of momentary records. To expand the applications of EMA and EMI in suicide prevention, we must accumulate continuous data on occurrences of suicidal ideation and depressive moods for proactive intervention. However, users prefer a more convenient way of recording moods and logging risk factors, such as on a smartwatch. Such additional features enable greater access to mental health care and simultaneous recording, allowing for better implementation of EMI as quality data accumulate and precise patterns emerge. Through this development process, the EMA information collected through the continuous recording of users can provide timely intervention and support.

5. Conclusions

eRAPPORT was primarily developed to prevent suicide, and it included safety planning, tracking of suicidal thoughts and other mental health assessments, resources and information for Korean suicide prevention, online/offline counselling requests, an emergency contact list and a Q&A section on mental health issues. Based on the current feasibility, eRAPPORT should be tested on more individuals with high suicide risk in terms of its effectiveness and application. Although this study primarily focused on the preliminary stage, it has the potential to move forward as a digital intervention for preventing suicide at the community level. EMA data collected from mobile apps can be combined with a lifelog to develop suicide prevention algorithms and serve as the basis for improving versions of suicide prevention apps. Through EMA and EMI, real‐time and longitudinal data were gathered to monitor mental health and suicide risk among adults living in the community. Thus, detailed documentation from our lived experience has been critically helpful for the design and systemic processes based on frameworks in mental healthcare research.

6. Relevance for Mental Health Practice

Many mental health nurses have developed mHealth interventions or related programs. However, it is difficult to share fundamental information about how to develop and test the feasibility of current research and practices. Although this study primarily tested eRAPPORT's functionality, acceptability and safety, our details can be helpful for developing related programs and interventions. As digital therapeutics for suicide prevention are still in the development phases, using mHealth technology in practice, including eRAPPORT, requires support from mental health practitioners and policymakers combined with participant awareness. Mental health nurses can use the app to monitor mental health at home, connect the patient's data with evidence‐based data, and provide timely interventions in emergencies. Additionally, policymakers keep updating the guidelines to approve, implement and evaluate the mHealth strategies and policies for digital interventions in mental health. Thus, those with mental health care are able to find evidence‐based, safe and more acceptable technologies to improve their health and safety, especially for suicide prevention.

Author Contributions

J.H. designed the study, searched the articles and apps, collected and interviewed, interpreted the data, performed the data analysis and drafted the manuscript. K.H.Y. collected data, performed the data analysis and drafted the manuscript. J.Y. searched and analysed the articles and apps, performed the data analysis and drafted the manuscript. K.H.J. was responsible for study design as a principal investigator, supervision of data analyses and drafting and revising all steps of manuscripts as well as funding acquisition. All authors made substantial contributions to complete the study and approved the final version of the manuscript based on ICMJE authorship criteria.

Ethics Statement

Ethical approval was granted by the Severance Hospital institutional review board (IRB; Approval no. 4–2023‐0997).

Conflicts of Interest

The authors declare no conflicts of interest.

Supporting information

Appendix S1.

Appendix S2.

Data Availability Statement

The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.