Abstract
Background
In the European Union, around 5 million people are affected by psychotic disorders, and approximately 30%-50% of people with schizophrenia have treatment-resistant schizophrenia (TRS). Mobile health (mHealth) interventions may be effective in preventing relapses, increasing treatment adherence, and managing some of the symptoms of schizophrenia. People with schizophrenia seem willing and able to use smartphones to monitor their symptoms and engage in therapeutic interventions. mHealth studies have been performed with other clinical populations but not in populations with TRS.
Objective
The purpose of this study was to present the 3-month prospective results of the m-RESIST intervention. This study aims to assess the feasibility, acceptability, and usability of the m-RESIST intervention and the satisfaction among patients with TRS after using this intervention.
Methods
A prospective multicenter feasibility study without a control group was undertaken with patients with TRS. This study was performed at 3 sites: Sant Pau Hospital (Barcelona, Spain), Semmelweis University (Budapest, Hungary), and Sheba Medical Center and Gertner Institute of Epidemiology and Health Policy Research (Ramat-Gan, Israel). The m-RESIST intervention consisted of a smartwatch, a mobile app, a web-based platform, and a tailored therapeutic program. The m-RESIST intervention was delivered to patients with TRS and assisted by mental health care providers (psychiatrists and psychologists). Feasibility, usability, acceptability, and user satisfaction were measured.
Results
This study was performed with 39 patients with TRS. The dropout rate was 18% (7/39), the main reasons being as follows: loss to follow-up, clinical worsening, physical discomfort of the smartwatch, and social stigma. Patients’ acceptance of m-RESIST ranged from moderate to high. The m-RESIST intervention could provide better control of the illness and appropriate care, together with offering user-friendly and easy-to-use technology. In terms of user experience, patients indicated that m-RESIST enabled easier and quicker communication with clinicians and made them feel more protected and safer. Patients’ satisfaction was generally good: 78% (25/32) considered the quality of service as good or excellent, 84% (27/32) reported that they would use it again, and 94% (30/32) reported that they were mostly satisfied.
Conclusions
The m-RESIST project has provided the basis for a new modular program based on novel technology: the m-RESIST intervention. This program was well-accepted by patients in terms of acceptability, usability, and satisfaction. Our results offer an encouraging starting point regarding mHealth technologies for patients with TRS.
Trial Registration
ClinicalTrials.gov NCT03064776; https://clinicaltrials.gov/ct2/show/record/NCT03064776
International Registered Report Identifier (IRRID)
RR2-10.1136/bmjopen-2017-021346
Keywords: schizophrenia, treatment-resistant, digital mental health, mHealth, mobile health, mental health, mental illness, mental disorder, psychosis, symptom management, adherence, acceptability, usability, feasibility, digital intervention, mobile intervention, mobile phone
Introduction
Schizophrenia is a severe mental disorder that affects approximately 1% of the population worldwide [1]. The results of a meta-analysis showed that only 13.5% of the patients achieve recovery, which includes clinical remission and good social functioning outcome [2]. Despite the proven efficacy of antipsychotic drugs, 30%-50% of patients with schizophrenia obtain a scarce benefit from conventional treatments [3,4]. This inadequate response to pharmacological treatment is known as treatment-resistant schizophrenia (TRS). TRS implicates an important burden at 3 levels, as follows: (1) clinical: negative attitude to medication, drug abuse, and nutritional/physical health problems; (2) economic: hospitalizations and polytherapy; and (3) humanistic: depression and social isolation [5]. As a result, dealing with TRS involves a high emotional burden for patients and their caregivers, affecting their quality of life. The shortcomings of the current health care and social support systems cannot provide adequate and effective solutions to these patients.
During the last decade, the incorporation of information and communication technology (ICT) into health care services (eHealth) has led to developing interventions aimed to improve patients’ quality of life. In the case of TRS, ICT tools could offer a novel opportunity to overcome important barriers by (1) enabling tailored therapeutic processes, (2) improving accessibility and continuous integrated care, and (3) promoting patients’ empowerment and participation of informal caregivers in therapeutic processes. Results from earlier studies on psychosis [6-8] support the idea of including ICT tools in the treatment of patients with TRS. Moreover, smartphone ownership among people with schizophrenia is relatively high and increasing [9-11]. Patients seemed willing and able to use smartphones to monitor their symptoms, engage in therapeutic interventions, and increase physical exercise [12]. Feasibility studies previous to m-RESIST have shown that interventions based on mobile health (mHealth) could promote the empowerment of patients with schizophrenia [9,13]. Further, some mHealth interventions may be effective for increasing treatment adherence, decreasing relapses, and improving symptoms [6,14,15].
To the best of our knowledge, none of the previously published studies on mHealth has focused on TRS [16]. We aimed to present the results of a 3-month prospective multicenter feasibility study performed as part of the m-RESIST project, wherein a new hybrid program (digital interventions along with face-to-face strategies) using wearable computing solutions and offering high modular and flexible functioning was designed and developed.
This study aims to assess the feasibility, acceptability, usability, and satisfaction of patients with TRS after using the m-RESIST intervention. We hypothesized that the m-RESIST intervention would have acceptable rates of willingness to enroll (≥70%) and attrition (nonusage and dropout attrition <15% in both measures) and would be highly accepted by patients with TRS, showing high scores in acceptability, usability, and satisfaction reported by more than 80% of the sample.
Methods
Study Design
A prospective multicenter feasibility study without a control group was performed in patients with a diagnosis of TRS between March and November 2017. Participants were recruited from 3 clinical sites: Sant Pau Hospital (Barcelona, Spain), Semmelweis University (Budapest, Hungary), and Sheba Medical Center and Gertner Institute of Epidemiology and Health Policy Research (Ramat-Gan, Israel). Patients underwent a 3-month-long modular intervention, which was specifically designed for patients with TRS. The intervention was based on the recognition of early warning signs of psychosis in order to improve positive symptoms, treatment adherence, and healthy lifestyle (Multimedia Appendix 1). To deliver the m-RESIST intervention and to promote active participation in the therapeutic processes, 3 mHealth tools were included: a wearable device (smartwatch), a mobile app (m-RESIST app), and a web-based platform. The m-RESIST intervention was created to be provided in 5 languages: Spanish, Catalan, Hungarian, Hebrew, and English. Mental health care providers assisted patients with TRS in using the intervention. The trial was registered in Clinical Trials (NCT03064776), and the study protocol was published earlier [17].
Ethics Approval
Ethics approval for this study was provided by the corresponding clinical ethics committees of the 3 participating clinical sites: Hospital Santa Creu i Sant Committee (IIBSP-RES-2016-51), ETT Tukeb, Semmelweis Ethical Committee (54920-4/2016/EKU), and Sheba Medical Center Ethical Committee (3472-16 SMC). Written informed consent was obtained from the participants before performing any study-related activities. The database generated by the study did not contain any identification of the participants but only a numerical code kept in a separate list; thus, the patients’ identities were protected. The information collected in this study was always treated as grouped data and never as individual or personal data, thereby maintaining anonymity and confidentiality.
Study Participants
Patients with TRS were recruited for this study. Inclusion criteria were patients aged 18-45 years with a diagnosis of schizophrenia according to Diagnostic and Statistical Manual of Mental Disorders, fifth edition criteria; duration of disease less than 15 years to meet the criteria for TRS [18,19]; familiarity with ICT tools and physical capability to use them; and presence of a caregiver. Exclusion criteria were to meet criteria for remission according to the Remission of Schizophrenia Working Group [20]; presence of delusions mainly related to their therapists or with new technologies; presence of vision, hearing, or motor impairment, interfering with operating a smartphone; presence of a caregiver or informal carer who is not used to ICT tools or has physical incapability to use them; and presence of intellectual developmental disability. The caregivers were also recruited as part of the m-RESIST project. Their input about the functioning of the intervention was collected in focus groups. These data were not analyzed in this study.
A structured interview was administered to the patients, including sociodemographic data and ICT level measured with 5 grades (none, basic, average, good, excellent) for 3 questions: “How do you rate your current skills in using technological devices?,” “What devices do you own?,” and “For what purpose have you used them in the last 2 months?”
Measures in This Study
Feasibility
Feasibility was assessed by analyzing patients’ willingness to participate and the rates of dropouts, nonuse, and compliance. For dropouts, the reasons and factors that could have prevented withdrawal were identified. A digital questionnaire (Qualtrics software) was used to present the following 3 questions: “Why have you decided to stop participating in the study? What was the main reason?,” “How could the m-RESIST intervention have been different to enhance your experience?,” and “Is there anything else that we haven’t discussed yet that you would like to mention?”
Acceptance and Usability
A mixed qualitative and quantitative methodological approach was used to analyze the outcomes. This study was performed following the principles of the 2 theoretical frameworks that best fit the evaluation of the m-RESIST intervention for acceptance and usability: Technology Acceptance Model (TAM [21]) and Living Labs or Ecosystem of Open Innovation [22].
The theoretical framework assumed to explore the acceptance of the m-RESIST intervention in patients with TRS was an extension of Davis’s model and based mainly on Chau and Hu’s model of telemedicine acceptance [23], which comprises 3 dimensions: technological (including perceived utility, perceived ease of use, habit), individual (attitude, barriers, intention to use), and organizational contexts (subjective norm, facilitators). All these variables were self-assessed by an adapted version to the field of mental health of the TAM scale [24], administered at the end of the study.
Usability was evaluated online in 2 different ways.
Measure of the degree of usability and experience after using the intervention: perceived ease of use, perceived utility, perceived quality of content, and attitude were assessed by the user experience questionnaire. The questionnaire was presented using the Qualtrics software. It consisted of 4 closed (4-level Likert scale) and 7 open questions. Data were collected in the middle and at the end of the study.
Measure of the continuous experience degree: the aim was to capture the continuous experience with the intervention. Through the app’s messaging system, the following question was sent to patients once a week and on different days and times: “What is it like to use the m-RESIST intervention?”
Satisfaction
User satisfaction was measured using the Client Satisfaction Questionnaire-8 item scale [25] to collect opinions about the service received. This scale contains 8 items with a range of scores from 1 to 4. This measure was self-assessed and collected at the end of the study.
Safety Measures
The presence of adverse events, defined as any clinical change or illness reported, was monitored using patient interviews throughout the study.
Procedures, Stages, and Study Plan
This study was performed over 4 periods (for more details, see Alonso-Solís et al [17]): recruitment, preintervention, intervention, and follow-up. During this time, the last release of m-RESIST prototype was used and the content was frozen; therefore, no major changes were realized. Once patients received explanations about the study and signed the informed consent form (recruitment period), the study smartwatch and smartphone with the m-RESIST app preinstalled were given, and patients were trained to use the devices. Then, key elements of the m-RESIST intervention were performed (eg, treatment plan) (preintervention period). Next, the 3-month intervention was conducted, including 6 sessions with the mental health care providers, delivered every 15 days. Two visits were conducted face-to-face and 4 were held by videoconference. During this period, patients were recommended to use the smartwatch daily and to check the app at least once a day for questionnaires, recommendations, and reminders. At the end of the 3 months, the final visit was performed (follow-up period). The variables of feasibility, acceptability, usability, and satisfaction with the intervention were evaluated.
Data Analysis
Statistical analyses were performed with the SPSS statistics (version 24; IBM Corp). Descriptive statistical analyses were performed to summarize the sociodemographic characteristics of the patients with TRS. Regarding the main variables, descriptive statistical analyses (means [SD]; frequencies and percentages) were performed to determine the feasibility (rates of willingness to participate, dropouts, and nonuse of the intervention), acceptability, usability for quantitative data, and satisfaction with the intervention. Differences in prescores and postscores of usability were analyzed using paired 2-sided t tests. Furthermore, in case of the usability variable, qualitative data were collected from the user experience questionnaire and the interval question. A qualitative thematic/content analysis was performed according to Mayring [26]. This method is a summary technique, where patterns in the text are found to look for themes in the data.
In terms of the operational criteria of willingness to enroll, nonusage, dropout, acceptability, usability, and satisfaction, there was no consensus to determine the cutoff points. For this reason, in this study, they were defined based on the criteria of the feasibility studies concerning the importance of establishing criteria aimed at the success of a randomized clinical trial [27,28]. Thus, high and demanding criteria were defined.
Results
Sample Population Characteristics
The final sample population in this pilot study consisted of 39 patients with TRS: 11 in Sant Pau Hospital, 14 in Semmelweis University, and 14 in Gertner Institute. No statistically significant differences in sociodemographic characteristics were found at baseline between the participants in the 3 sites. See Table 1 for a summary of the sociodemographic data. Regarding the degree of skill with ICT tools, around 75% (28/39) of the total sample of patients was distributed between the 2 intermediate levels (good and average levels).
Table 1.
Baseline sociodemographic characteristics of the patients with treatment-resistant schizophrenia (N=39).
| Characteristics | Values | |
| Age (years), mean (SD) | 33.6 (7.67) | |
| Gender, n (%) | ||
|
|
Women | 17 (44) |
|
|
Men | 22 (56) |
| Ethnicity, n (%) | ||
|
|
Asian | 9 (23) |
|
|
Caucasian | 29 (74) |
|
|
Sub-Saharan Africa | 1 (3) |
| Marital status, n (%) | ||
|
|
Divorced/separated | 2 (5) |
|
|
Married | 4 (10) |
|
|
Single | 30 (77) |
|
|
Stable relationship | 2 (5) |
|
|
Widowed | 1 (3) |
| Cohabitation, n (%) | ||
|
|
Alone | 4 (10) |
|
|
Sheltered accommodation | 1 (3) |
|
|
With children | 1 (3) |
|
|
With parents or relatives | 27 (69) |
|
|
With flat mates (non–sheltered accommodation) | 1 (3) |
|
|
With significant other | 5 (13) |
| Educational level, n (%) | ||
|
|
Primary school | 18 (46) |
|
|
Secondary school | 13 (33) |
|
|
University | 8 (21) |
| Occupation, n (%) | ||
|
|
Full-time job | 10 (26) |
|
|
Part-time job | 9 (23) |
|
|
Pensioner | 6 (15) |
|
|
Sheltered job | 2 (5) |
|
|
Student | 1 (3) |
|
|
Unemployed | 11 (28) |
Feasibility of the m-RESIST Intervention
The variables of willingness to participate, dropouts in the study, nonuse of the intervention, and patient compliance with the app were evaluated. The study participation flowchart based on the CONSORT (Consolidated Standards of Reporting Trials) 2010 statement recommendations [28] is presented in Figure 1. Approximately 52% (42/81) of the patients with TRS identified as candidates were receptive to participating in this study. With this result, the hypothesis was not confirmed (rate ≥70%). The reasons for not participating were lack of interest and fear of being controlled by electronic devices. Of the 39 patients who started the 3-month intervention period, 7 dropped out of the study, which represented a dropout rate of 18%. With this result, the hypothesis was not confirmed (rate <15%). Regarding the reasons for the withdrawal of patients, we grouped them into 4 main categories: (1) loss to follow-up, that is, not attending visits or answering calls; (2) clinical worsening, where hospitalization was required in 2 cases; (3) physical discomfort, mainly associated with the smartwatch, and (4) social stigma, that is, concern about showing technological devices in public that could be related to their diagnosis. All participants who completed the pilot study and participated in the follow-up (n=32) used the 2 m-RESIST tools (smartwatch and app) until the end of the study. The hypothesis of the study was fulfilled (nonuse rate <15%).
Figure 1.
m-RESIST study participation flowchart based on the CONSORT (Consolidated Standards of Reporting Trials) 2010 statement recommendations.
With regard to compliance of the patients, the rate of reading messages was 49.5% (895/1818) and the response rate was 13.6% (246/1818). Regarding the number of questionnaires that the patients received during the study and the degree of compliance with them, the response rate was close to 40%.
Acceptability of the m-RESIST Intervention
The mean score for TAM dimensions on a scale from 1 (totally disagree) to 7 (totally agree) showed positive results in all dimensions (see Table 2).
Table 2.
Results of Technology Acceptance Model dimensions (n=32).
| Dimension | Values, mean (SD) |
| Perceived utility | 5.16 (1.14) |
| Perceived ease of use | 5.39 (1.16) |
| Attitude | 5.02 (1.10) |
| Barriers | 3.84 (1.84) |
| Intention to use | 5.19 (1.38) |
| Subjective norma | 5.34 (1.04) |
| Facilitatorsb | 5.34 (1.08) |
aAssesses the extent to which the potential adopter believes that people who are important to him or her will approve his or her adopting the system.
bRefers to the degree to which the potential adopter believes that an organizational and technical infrastructure exists to support the use of the system.
Over 70% (22/32) of the patients considered that by using the m-RESIST intervention, health care professionals could provide better control of their conditions and they could receive more appropriate care. Over 80% (27/32) of the patients considered m-RESIST intervention as a user-friendly technology and easy-to-use; low perception of barriers was detected (12/32, 38%) and most of the patients (27/32, 84%) agreed with the existence of facilitators (81%-84%). Thus, the acceptability of the m-RESIST intervention was moderate (the tendency of response was near to a frequency of 75%), with 75% (24/32) of the patients showing a moderate-to-high intention to use the m-RESIST intervention. With this result, the hypothesis was not confirmed (>80%).
Usability of the m-RESIST Intervention
Results of Closed Questions
The results of the usability analysis of the m-RESIST intervention were different between the app and the smartwatch. Over 80% (25/32) of the patients perceived the app as easy-to-use. It was considered easy to learn, clear, and understandable, and this perception increased over time (Multimedia Appendix 2). Over 65% of the patients found that the quality of the app’s content was good (eg, fonts, size, overall look, feel of the pages). Finally, over 65% of the patients considered the app to be well-integrated (24/32), recommended the app to others (19/32), and thought it would be good to use it in addition to the traditional treatment methods (22/32).
Regarding the smartwatch, a critical attitude prevailed about its ease of use and its usefulness in managing the disorder (Multimedia Appendix 2). Only over 40% (14/32) of the patients showed a positive attitude. At the end of the pilot study, over 60% (18/32) reported that they would recommend the device to others, but only 31% (10/32) reported that they would use it to manage their condition. As shown in Table 3, patients' experience with the digital tools was similar in both periods of time.
Table 3.
User experience at periods Time 1 (middle pilot) and Time 2 (end of pilot) (n=32).a
|
|
Time 1, mean (SD) | Time 2, mean (SD) | t (df) | P value | |
| App | |||||
|
|
PEUb | 16.81 (2.21) | 16.45 (2.11) | 1.10 (30) | .28 |
|
|
PUc | 12.32 (2.89) | 11.39 (3.87) | 1.30 (30) | .20 |
|
|
PQCd | 17.77(3.55) | 17.45(3.87) | 0.58 (30) | .57 |
|
|
Attitude | 21.26 (3.69) | 20.97 (3.23) | 0.40 (30) | .69 |
| Smartwatch | |||||
|
|
PEU | 17.23 (2.50) | 17.39 (2.38) | –0.32 (30) | .75 |
|
|
PU | 13.39 (3.16) | 13.58 (3.06) | –0.43 (30) | .67 |
|
|
PQC | 22.13 (5.43) | 21.29 (5.77) | 1.24 (30) | .23 |
|
|
Attitude | 21.94 (4.30) | 22.65 (4.18) | –1.01 (30) | .32 |
aScale scores from 1 (strongly agree; excellent) to 5 (strongly disagree; not know/no opinion).
bPEU: perceived ease of use.
cPU: perceived usefulness.
dPQC: perceived quality of content.
The usability of the m-RESIST intervention was medium for the app (the tendency of response was near to a frequency of 65%) and low for the smartwatch (the tendency of response was near to a frequency of 45%). With this result, the hypothesis was not confirmed (>80%).
Results of Open Questions
The first core theme was the feeling of security due to easier and quicker communication with professionals. The second theme was the perception of the m-RESIST intervention as promising and useful, and the step counter and the alarm button were evaluated better than other features. The third theme was the m-RESIST limitations. Patients mentioned technical difficulties with the system, mostly related to problems of synchronization and charge in the smartwatch and the mobile phone. Some patients also had other complaints about the smartwatch, such as the device irritated their skin and that there was lack of choice regarding the type of watch (color, size). Finally, patients in general communicated about being proud of being chosen as a participant for an ICT research project.
Patient Satisfaction With the m-RESIST Intervention
Of the total sample, 78% (25/32) of the participants rated the quality of the service as excellent or good, 75% (24/32) felt they received the services they wanted, and 84% (28/32) reported that they would recommend the intervention program. Moreover, 94% (30/32) of the patients were very or mostly satisfied with the intervention and 84% (27/32) reported that they would use it again. Satisfaction with the m-RESIST intervention was generally good with a mean total score of 25 (SD 4.31) on the Client Satisfaction Questionnaire-8 item scale. With this result, the hypothesis was not confirmed (>80%).
Safety
Regarding the safety of the study, the clinical status of the patient was monitored and evaluated based on the incidence of adverse events. Three serious adverse events were reported during this study that required hospitalization due to worsening of psychotic symptoms. No direct association between them and the m-RESIST intervention or the protocol procedures was found. The most probable causes were the evolution of the clinical pattern in 2 cases and a decrease in the sleep pattern in the other case.
Discussion
Principal Findings
Our novel findings indicate that the m-RESIST intervention is feasible and is an acceptable, satisfactory, and potentially useful tool for a population with TRS who have major clinical challenges with usually poor outcomes and low adherence to treatment.
Feasibility of the m-RESIST Intervention
Most of the patients in our study had a high knowledge of ICT and owned a smartphone with an internet connection. This finding is similar to that reported in previous studies on populations with psychosis. According to a 2015 meta-analysis [9], smartphone ownership rate among patients with psychosis has increased proportionally in recent years (81%), and this rate is similar to that observed in the general population (90%). Moreover, the willingness rate to participate in this study was 52% (42/81), which was in line with that reported in other pilot studies on mobile app–based interventions for schizophrenia: 50% in the ClinTouch study [8], 40% in FOCUS [6], and 58% in PeerFIT [29]. Further, the dropout rate in our study was acceptable and similar to that reported in previous studies such as ClinTouch [8], PeerFIT [29], and CrossCheck [30]. However, some studies had almost no dropouts, for example, FOCUS [6], SleepSight [31], or App4Independence [15].
In future clinical trials, it is necessary to explore the actors and reasons for dropouts to diminish such instances and to adapt the digital intervention to the target populations. Of the pilot studies on psychosis included in 2 systematic reviews [12,32], only 3 examined the reasons for dropouts [6,29,33]. Laine et al [34] obtained high rates of refusal to participate (70%) and abandonment (48%), in contrast with the high compliance of the final sample. These findings suggest that digital technology can be used by patients with psychotic disorders. However, to incorporate digital health in patients’ treatments, apart from a receptive attitude, it is necessary to explore the degree of trust in technology. An interesting fact observed is that the initial reluctance toward using new technologies can be reversed: as the patient’s experience increases, a more receptive attitude is observed. Finally, the nonuse rate of the intervention in this study was observed to be low as in other mobile phone–based interventions, wherein all the patients who underwent the final evaluations in the studies used the devices until the end [8,33,35].
It was not possible to validly evaluate the compliance of the patients with this intervention. Errors in the app (eg, visualization of messages with errors in the characters, part of the message missing, errors in the notifications), mainly during the first middle part of the pilot study, resulted in patients’ inability to access the contents properly. Thus, results on compliance must be evaluated with caution.
Acceptability, Usability, and Satisfaction of the m-RESIST Intervention
The m-RESIST digital intervention achieved a moderate level of acceptability, with 75% (24/32) of the patients showing moderate to high intention of using the intervention. The acceptability results are consistent with data shown in a meta-analysis [12], where the authors reported that around 60% of patients with psychotic disorders were in favor of using smartphones to monitor their mental health status. In general, most studies on patients with schizophrenia have shown their high acceptability toward digital interventions based on smartphones and aimed to (1) monitor variables (clinical symptoms, functioning, and mood) using the ecological momentary assessment methodology or other monitoring systems; (2) implement interventions to improve clinical symptoms and promote healthy lifestyle habits; and (3) facilitate contact with clinical professionals [12,34-36].
Regarding usability, the tendency of response toward the app was positive in 70% (22/32) of the patients. In previous mHealth pilot studies, similar values of usability were obtained [37,38]. It is important to note that most patients considered that the m-RESIST intervention should be used in addition to traditional treatments and not as a replacement. The patients feared losing face-to-face contact with the professional team and “being monitored by a machine.” The study of Baumel et al [39] would be a good example of how to add digitalization to the traditional treatment for schizophrenia. Baumel et al [39] developed a technology-based intervention for schizophrenia, which integrated digital tools and human support, and it achieved good outcomes in terms of retention (93% of the participants finished the program), acceptability, and satisfaction. Baumel et al [39] considered that the coach’s supportive role, providing technological guidance and delivering relapse prevention treatment, was one of the keys to patient adherence to their intervention.
In addition, it should be highlighted that previous mHealth studies aimed at people with schizophrenia used different methodologies to evaluate acceptability and usability. However, none of them have elaborated semistructured questionnaires to assess operational dimensions of acceptability, such as perceived ease of use or barriers. In this sense, it has not yet been possible to “capture the complex nature of acceptability” [40]. Berry et al [40] insist on the need to perform qualitative evaluations of acceptability during the different stages of a study. This same idea would also apply to the usability variable [41]. In the case of the m-RESIST study, semistructured interviews focusing on the evaluation of specific dimensions of acceptability and usability and based on current theoretical approaches (TAM, Living labs) were established.
Regarding satisfaction with the m-RESIST intervention, results showed that 78% (25/32) of the sample was quite to very satisfied with the service provided: 94% (30/32) of the sample was generally satisfied with the m-RESIST intervention and 84% (27/32) would use it again. This percentage is similar to that obtained in other digital interventions tailored for patients with schizophrenia (FOCUS [42], PEAR-004 [43], PeerFIT [44], WellWave [45]).
A noteworthy finding of our study is that patients distinguished the whole intervention from its components, that is, patients had a positive and satisfactory vision about the m-RESIST intervention (digital tools and tailored therapeutic program), but they had doubts about the functioning of the app and the smartwatch because they did not work properly during the pilots. Although a high percentage of patients reported that they would recommend the m-RESIST intervention (digital tools and tailored therapeutic program) and considered it to be a good idea to add the app and smartwatch to the treatment as usual, only 35% (11/32) reported that they would use these specific tools to manage their condition, while practically, the rest were undecided about the usability of the app and smartwatch to treat their own state of health. It is probably because patients appreciate the digital tools’ functionalities (eg, communication, steps counter, alarm button for emergencies) but could not test them properly due to technical problems. Thus, the m-RESIST intervention was valued as promising and it was rated successfully, but in terms of usability, it did not convince completely.
The added value of the m-RESIST intervention is to respond to the need of creating new care possibilities adapted to the specific nature of well-identified clinical subgroups such as patients with TRS [46]. The m-RESIST intervention has been designed to overcome fragmentation in providing care and to promote recovery from TRS. In the former case, the m-RESIST intervention favors continuity and immediacy of care by using digital tools to monitor patients’ state and detect individual early warning signs and ease communication with health care professionals. In the latter case, clinical, social, and personal recovery is promoted by the tailored therapeutic program delivered to patients with TRS (symptoms management-emotion regulation and healthy habits modules; see Multimedia Appendix 1).
Strengths of This Study
This study has important strengths. First, this is the first time that digital technology has been incorporated for TRS intervention. This is important because apart from drugs, there are no specific programs of therapy available for this severe profile of patients. Second, we designed an intervention acceptable and with potential usability for TRS, with capacity for innovation at 2 levels: (1) methodological, created from a model where the patients with TRS and their needs are at the center [47] and (2) technological, intervention based on sensors, app, and web-based platform that would allow personalized follow-up and monitoring of patients based on their risk levels. Third, our study indicates that m-RESIST could be a safe digital intervention for patients with severe schizophrenia. Finally, we also developed an operational evaluation of acceptability and usability, performing interviews throughout different stages of the study.
Limitations of This Study
There are some limitations in this study. First, the sample size was relatively small, although the size is similar to that reported in previous feasibility studies. Second, the presence of errors in the system operation, especially in communication and in questionnaires, made it impossible to assess the compliance of the participants. Third, the patients were not involved in the process of choosing the smartwatch, and the functioning of the smartwatch was also a limitation. An improvement in the whole process would have positively impacted the patient adherence and usability of this device.
Conclusions
To date, no major progression is in sight for the treatment of schizophrenia and TRS [48], but small steps can be achieved, as shown in our study. The m-RESIST pilot study suggests that this intervention is a feasible treatment option for patients with TRS: easy to use, advisable, and with impact on the motivation for using it. The m-RESIST intervention has contributed to TRS treatment, thereby providing the basis of a new therapeutic program, which is based on novel technology, uses wearable computing solutions, and offers high modular and flexible functioning. For further development of integrated care tools, it will be crucial to involve health care providers and organizations. Thus, intervention programs based on ICTs will eventually be incorporated as an addition to the community services already existing in the mental health care routine.
Acknowledgments
This work has been supported by the Horizon 2020 Framework Programme of the European Union (grant 643552) and partly funded by CERCA (Centres de Recerca de Catalunya) Programme/Generalitat de Catalunya. We thank all the patients who participated in this study for their essential contribution to our field of research. We are also thankful to Ona Corporation (IIBSP-AMA2016-99).
Abbreviations
- CONSORT
Consolidated Standards of Reporting Trials
- ICT
information and communication technology
- mHealth
mobile health
- TAM
Technology Acceptance Model
- TRS
treatment-resistant schizophrenia
m-RESIST tailored therapeutic program.
User experience questionnaire: open questions for the app and the smartwatch.
Data Availability
The data sets generated and analyzed during this study are available from the corresponding author on reasonable request.
Footnotes
Authors' Contributions: We are grateful to all the members of the m-RESIST project, who are also the authors of this paper under the name of m-RESIST Group. The m-RESIST Group is composed of the following members: Francisco Alcalde, Enrico d'Amico, Caritat Almazán, Anna Alonso-Solís, Jesús Berdún, István Bitter, Walter Baccinelli, Chiara Bonizzi, María Bugheroni, Johanna Caro Mendivelso, Asaf Caspi, Tanguy Coenen, Anat Cohen, Xavier Constant, Iluminada Corripio, Marisol Escobar, Kinga Farkas, Kata Fazekas, Yoram Feldman, Emmanuel Gimenez, Shenja van der Graaf, Eva Grasa, Marianne Haapea, Levente Herman, Margarita Hospedales, Elena Huerta-Ramos, Matti Isohanni, Erika Jääskeläinen, Charlotte Jewel, Teija Juola, Timo Jämsä, Rachelle Kaye, Alejandro Keymer-Gausset, Panagiotis Kokkinakis, Hannu Koponen, Silvia Marcó, Gregoris Mentzas, Jouko Miettunen, Jani Moilanen, Susana Ochoa, Ilias Papas, Fotis Paraskevopoulos, Elisabeth Reixach, Alexandra Roldán, Katya Rubinstein, Elena Rubio-Abadal, Garifalia Sebú, Annika Seppälä, Jussi Seppälä, Valentina Simonetti, Matthias Stevens, Vittorio Tauro, Anna Triantafillou, Zsolt Szabolcs Unoka, Judith Usall, Vincenzo Vella, David Vermeir and Ilaria de Vita
Conflicts of Interest: None declared.
References
- 1.Leucht S, Busch R, Kissling W, Kane JM. Early prediction of antipsychotic nonresponse among patients with schizophrenia. J Clin Psychiatry. 2007 Mar;68(3):352–60. doi: 10.4088/jcp.v68n0301. [DOI] [PubMed] [Google Scholar]
- 2.Jääskeläinen Erika, Juola P, Hirvonen N, McGrath JJ, Saha S, Isohanni M, Veijola J, Miettunen J. A systematic review and meta-analysis of recovery in schizophrenia. Schizophr Bull. 2013 Nov;39(6):1296–306. doi: 10.1093/schbul/sbs130. https://europepmc.org/abstract/MED/23172003 .sbs130 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Kennedy JL, Altar CA, Taylor DL, Degtiar I, Hornberger JC. The social and economic burden of treatment-resistant schizophrenia: a systematic literature review. Int Clin Psychopharmacol. 2014 Mar;29(2):63–76. doi: 10.1097/YIC.0b013e32836508e6. [DOI] [PubMed] [Google Scholar]
- 4.Mortimer AM, Singh P, Shepherd CJ, Puthiryackal J. Clozapine for treatment-resistant schizophrenia: National Institute of Clinical Excellence (NICE) guidance in the real world. Clin Schizophr Relat Psychoses. 2010 Apr;4(1):49–55. doi: 10.3371/CSRP.4.1.4.T708575317412375 [DOI] [PubMed] [Google Scholar]
- 5.Millier A, Schmidt U, Angermeyer MC, Chauhan D, Murthy V, Toumi M, Cadi-Soussi N. Humanistic burden in schizophrenia: a literature review. J Psychiatr Res. 2014 Jul;54:85–93. doi: 10.1016/j.jpsychires.2014.03.021. https://linkinghub.elsevier.com/retrieve/pii/S0022-3956(14)00103-4 .S0022-3956(14)00103-4 [DOI] [PubMed] [Google Scholar]
- 6.Ben-Zeev D, Brenner CJ, Begale M, Duffecy J, Mohr DC, Mueser KT. Feasibility, acceptability, and preliminary efficacy of a smartphone intervention for schizophrenia. Schizophr Bull. 2014 Nov;40(6):1244–53. doi: 10.1093/schbul/sbu033. https://europepmc.org/abstract/MED/24609454 .sbu033 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Granholm E, Ben-Zeev D, Link PC, Bradshaw KR, Holden JL. Mobile Assessment and Treatment for Schizophrenia (MATS): a pilot trial of an interactive text-messaging intervention for medication adherence, socialization, and auditory hallucinations. Schizophr Bull. 2012 May;38(3):414–25. doi: 10.1093/schbul/sbr155. https://europepmc.org/abstract/MED/22080492 .sbr155 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Palmier-Claus JE, Ainsworth J, Machin M, Barrowclough C, Dunn G, Barkus E, Rogers A, Wykes T, Kapur S, Buchan I, Salter E, Lewis SW. The feasibility and validity of ambulatory self-report of psychotic symptoms using a smartphone software application. BMC Psychiatry. 2012 Oct 17;12:172. doi: 10.1186/1471-244X-12-172. https://bmcpsychiatry.biomedcentral.com/articles/10.1186/1471-244X-12-172 .1471-244X-12-172 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Firth J, Cotter J, Torous J, Bucci S, Firth JA, Yung AR. Mobile Phone Ownership and Endorsement of "mHealth" Among People With Psychosis: A Meta-analysis of Cross-sectional Studies. Schizophr Bull. 2016 Mar;42(2):448–55. doi: 10.1093/schbul/sbv132. https://europepmc.org/abstract/MED/26400871 .sbv132 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Brunette MF, Achtyes E, Pratt S, Stilwell K, Opperman M, Guarino S, Kay-Lambkin F. Use of Smartphones, Computers and Social Media Among People with SMI: Opportunity for Intervention. Community Ment Health J. 2019 Aug;55(6):973–978. doi: 10.1007/s10597-019-00431-7. https://europepmc.org/abstract/MED/31175518 .10.1007/s10597-019-00431-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Young AS, Cohen AN, Niv N, Nowlin-Finch N, Oberman RS, Olmos-Ochoa TT, Goldberg RW, Whelan F. Mobile Phone and Smartphone Use by People With Serious Mental Illness. Psychiatr Serv. 2020 Mar 01;71(3):280–283. doi: 10.1176/appi.ps.201900203. https://europepmc.org/abstract/MED/31744429 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Firth J, Torous J. Smartphone Apps for Schizophrenia: A Systematic Review. JMIR Mhealth Uhealth. 2015 Nov 06;3(4):e102. doi: 10.2196/mhealth.4930. https://mhealth.jmir.org/2015/4/e102/ v3i4e102 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Bonet L, Izquierdo C, Escartí MJ, Sancho JV, Arce D, Blanquer I, Sanjuan J. Use of mobile technologies in patients with psychosis: A systematic review. Revista de Psiquiatría y Salud Mental (English Edition) 2017 Jul;10(3):168–178. doi: 10.1016/j.rpsmen.2017.05.010. [DOI] [PubMed] [Google Scholar]
- 14.Buck B, Scherer E, Brian R, Wang R, Wang W, Campbell A, Choudhury T, Hauser M, Kane JM, Ben-Zeev D. Relationships between smartphone social behavior and relapse in schizophrenia: A preliminary report. Schizophr Res. 2019 Jun;208:167–172. doi: 10.1016/j.schres.2019.03.014. https://europepmc.org/abstract/MED/30940400 .S0920-9964(19)30108-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Kidd SA, Feldcamp L, Adler A, Kaleis L, Wang W, Vichnevetski K, McKenzie K, Voineskos A. Feasibility and outcomes of a multi-function mobile health approach for the schizophrenia spectrum: App4Independence (A4i) PLoS One. 2019;14(7):e0219491. doi: 10.1371/journal.pone.0219491. https://dx.plos.org/10.1371/journal.pone.0219491 .PONE-D-18-20735 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Seppälä Jussi, De Vita I, Jämsä Timo, Miettunen J, Isohanni M, Rubinstein K, Feldman Y, Grasa E, Corripio I, Berdun J, D'Amico E, M-RESIST Group. Bulgheroni M. Mobile Phone and Wearable Sensor-Based mHealth Approaches for Psychiatric Disorders and Symptoms: Systematic Review. JMIR Ment Health. 2019 Feb 20;6(2):e9819. doi: 10.2196/mental.9819. https://mental.jmir.org/2019/2/e9819/ v6i2e9819 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Alonso-Solís Anna, Rubinstein K, Corripio I, Jaaskelainen E, Seppälä Annika, Vella VA, Caro-Mendivelso J, Caspi A, Isohanni M, Unoka Z, Van der Graff S, Farkas K, Huerta-Ramos E, Marcó-García Silvia, Stevens M, Coenen T, Hospedales M, Berdún Jesús, m-Resist group. Grasa EM. Mobile therapeutic attention for treatment-resistant schizophrenia (m-RESIST): a prospective multicentre feasibility study protocol in patients and their caregivers. BMJ Open. 2018 Jul 16;8(7):e021346. doi: 10.1136/bmjopen-2017-021346. https://bmjopen.bmj.com/lookup/pmidlookup?view=long&pmid=30012788 .bmjopen-2017-021346 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Elkis H, Meltzer HY. [Refractory schizophrenia] Braz J Psychiatry. 2007 Oct;29 Suppl 2:S41–7. doi: 10.1590/s1516-44462007000600002.S1516-44462007000600002 [DOI] [PubMed] [Google Scholar]
- 19.Peuskens J. The evolving definition of treatment resistance. J Clin Psychiatry. 1999;60 Suppl 12:4–8. [PubMed] [Google Scholar]
- 20.Andreasen NC, Carpenter WT, Kane JM, Lasser RA, Marder SR, Weinberger DR. Remission in schizophrenia: proposed criteria and rationale for consensus. Am J Psychiatry. 2005 Mar;162(3):441–9. doi: 10.1176/appi.ajp.162.3.441.162/3/441 [DOI] [PubMed] [Google Scholar]
- 21.Davis FD. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly. 1989 Sep;13(3):319. doi: 10.2307/249008. [DOI] [Google Scholar]
- 22.Eriksson M, Niitamo V, Kulkki S. State-of-the-art in utilizing Living Labs approach to user-centric ICT innovation - a European approach. Living Labs. 2005. [2023-05-31]. https://www.semanticscholar.org/paper/State-ofthe-art-in-utilizing-Living-Labs-approach-Eriksson/2edd5e0fef9f7f9fd0262dea937cb997b3ab8d5f .
- 23.Chau PY, Hu PJ. Examining a Model of Information Technology Acceptance by Individual Professionals: An Exploratory Study. Journal of Management Information Systems. 2014 Dec 23;18(4):191–229. doi: 10.1080/07421222.2002.11045699. [DOI] [Google Scholar]
- 24.Gagnon MP, Orruño Estibalitz, Asua J, Abdeljelil AB, Emparanza J. Using a modified technology acceptance model to evaluate healthcare professionals' adoption of a new telemonitoring system. Telemed J E Health. 2012;18(1):54–9. doi: 10.1089/tmj.2011.0066. https://europepmc.org/abstract/MED/22082108 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Roberts RE, Attkisson C. Assessing client satisfaction among Hispanics. Eval Program Plann. 1983;6(3-4):401–13. doi: 10.1016/0149-7189(83)90019-8.0149-7189(83)90019-8 [DOI] [PubMed] [Google Scholar]
- 26.Mayring P. Qualitative Content Analysis: Theoretical Foundation, Basic Procedures and Software Solution. Austria: Klagenfurt; 2014. [Google Scholar]
- 27.Thabane L, Hopewell S, Lancaster GA, Bond CM, Coleman CL, Campbell MJ, Eldridge SM. Methods and processes for development of a CONSORT extension for reporting pilot randomized controlled trials. Pilot Feasibility Stud. 2016;2:25. doi: 10.1186/s40814-016-0065-z. https://pilotfeasibilitystudies.biomedcentral.com/articles/10.1186/s40814-016-0065-z .65 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Eldridge Sandra M, Chan Claire L, Campbell Michael J, Bond Christine M, Hopewell Sally, Thabane Lehana, Lancaster Gillian A, PAFS consensus group CONSORT 2010 statement: extension to randomised pilot and feasibility trials. BMJ. 2016 Oct 24;355:i5239. doi: 10.1136/bmj.i5239. http://www.bmj.com/lookup/pmidlookup?view=long&pmid=27777223 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Aschbrenner KA, Naslund JA, Barre LK, Mueser KT, Kinney A, Bartels SJ. Peer health coaching for overweight and obese individuals with serious mental illness: intervention development and initial feasibility study. Transl Behav Med. 2015 Sep;5(3):277–84. doi: 10.1007/s13142-015-0313-4. https://europepmc.org/abstract/MED/26327933 .313 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Buck B, Hallgren KA, Scherer E, Brian R, Wang R, Wang W, Campbell A, Choudhury T, Hauser M, Kane JM, Ben-Zeev D. Capturing behavioral indicators of persecutory ideation using mobile technology. J Psychiatr Res. 2019 Sep;116:112–117. doi: 10.1016/j.jpsychires.2019.06.002. https://europepmc.org/abstract/MED/31226579 .S0022-3956(19)30023-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Kerz M, Folarin A, Meyer N, Begale M, Maccabe J, Dobson R. SleepSight: A wearables-based relapse prevention system for schizophrenia. UbiComp '16: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct; September 12-16; New York, USA. 2016. https://discovery.ucl.ac.uk/id/eprint/1528435/ [DOI] [Google Scholar]
- 32.Alvarez-Jimenez M, Alcazar-Corcoles MA, González-Blanch C, Bendall S, McGorry PD, Gleeson JF. Online, social media and mobile technologies for psychosis treatment: a systematic review on novel user-led interventions. Schizophr Res. 2014 Jun;156(1):96–106. doi: 10.1016/j.schres.2014.03.021.S0920-9964(14)00147-9 [DOI] [PubMed] [Google Scholar]
- 33.Ainsworth J, Palmier-Claus JE, Machin M, Barrowclough C, Dunn G, Rogers A, Buchan I, Barkus E, Kapur S, Wykes T, Hopkins RS, Lewis S. A comparison of two delivery modalities of a mobile phone-based assessment for serious mental illness: native smartphone application vs text-messaging only implementations. J Med Internet Res. 2013 Apr 05;15(4):e60. doi: 10.2196/jmir.2328. https://www.jmir.org/2013/4/e60/ v15i4e60 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Laine A, Välimäki Maritta, Pekurinen V, Löyttyniemi Eliisa, Marttunen M, Anttila M. Feasibility, Acceptability, and Preliminary Impacts of Web-Based Patient Education on Patients With Schizophrenia Spectrum Disorder: Quasi-Experimental Cluster Study. J Med Internet Res. 2019 Oct 17;21(10):e13073. doi: 10.2196/13073. https://www.jmir.org/2019/10/e13073/ v21i10e13073 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Eisner E, Bucci S, Berry N, Emsley R, Barrowclough C, Drake RJ. Feasibility of using a smartphone app to assess early signs, basic symptoms and psychotic symptoms over six months: A preliminary report. Schizophr Res. 2019 Jun;208:105–113. doi: 10.1016/j.schres.2019.04.003. https://linkinghub.elsevier.com/retrieve/pii/S0920-9964(19)30126-4 .S0920-9964(19)30126-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Kreyenbuhl J, Record EJ, Himelhoch S, Charlotte M, Palmer-Bacon J, Dixon LB, Medoff DR, Li L. Development and Feasibility Testing of a Smartphone Intervention to Improve Adherence to Antipsychotic Medications. Clin Schizophr Relat Psychoses. 2019;12(4):152–167. doi: 10.3371/CSRP.KRRE.070816. https://europepmc.org/abstract/MED/27454213 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Ben-Zeev D, Kaiser SM, Krzos I. Remote "hovering" with individuals with psychotic disorders and substance use: feasibility, engagement, and therapeutic alliance with a text-messaging mobile interventionist. J Dual Diagn. 2014;10(4):197–203. doi: 10.1080/15504263.2014.962336. https://europepmc.org/abstract/MED/25391277 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.de Almeida RS, Couto A, Marques A, Queirós C, Martins C. Mobile Application for Self-Management in Schizophrenia: A Pilot Study. Journal of Technology in Human Services. 2018 Jul 31;36(4):179–190. doi: 10.1080/15228835.2018.1483859. [DOI] [Google Scholar]
- 39.Baumel A, Correll CU, Hauser M, Brunette M, Rotondi A, Ben-Zeev D, Gottlieb JD, Mueser KT, Achtyes ED, Schooler NR, Robinson DG, Gingerich S, Marcy P, Meyer-Kalos P, Kane JM. Health Technology Intervention After Hospitalization for Schizophrenia: Service Utilization and User Satisfaction. Psychiatr Serv. 2016 Sep 01;67(9):1035–8. doi: 10.1176/appi.ps.201500317. [DOI] [PubMed] [Google Scholar]
- 40.Berry N, Lobban F, Emsley R, Bucci S. Acceptability of Interventions Delivered Online and Through Mobile Phones for People Who Experience Severe Mental Health Problems: A Systematic Review. J Med Internet Res. 2016 May 31;18(5):e121. doi: 10.2196/jmir.5250. https://www.jmir.org/2016/5/e121/ v18i5e121 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Chivilgina O, Wangmo T, Elger BS, Heinrich T, Jotterand F. mHealth for schizophrenia spectrum disorders management: A systematic review. Int J Soc Psychiatry. 2020 Nov;66(7):642–665. doi: 10.1177/0020764020933287. [DOI] [PubMed] [Google Scholar]
- 42.Ben-Zeev D, Brian RM, Jonathan G, Razzano L, Pashka N, Carpenter-Song E, Drake RE, Scherer EA. Mobile Health (mHealth) Versus Clinic-Based Group Intervention for People With Serious Mental Illness: A Randomized Controlled Trial. Psychiatr Serv. 2018 Sep 01;69(9):978–985. doi: 10.1176/appi.ps.201800063. [DOI] [PubMed] [Google Scholar]
- 43.Ghaemi SN, Sverdlov O, van Dam J, Campellone T, Gerwien R. A Smartphone-Based Intervention as an Adjunct to Standard-of-Care Treatment for Schizophrenia: Randomized Controlled Trial. JMIR Form Res. 2022 Mar 28;6(3):e29154. doi: 10.2196/29154. https://formative.jmir.org/2022/3/e29154/ v6i3e29154 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Naslund JA, Aschbrenner KA, Barre LK, Bartels SJ. Feasibility of popular m-health technologies for activity tracking among individuals with serious mental illness. Telemed J E Health. 2015 Mar;21(3):213–6. doi: 10.1089/tmj.2014.0105. https://europepmc.org/abstract/MED/25536190 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Macias C, Panch T, Hicks YM, Scolnick JS, Weene DL, Öngür Dost, Cohen BM. Using Smartphone Apps to Promote Psychiatric and Physical Well-Being. Psychiatr Q. 2015 Dec;86(4):505–19. doi: 10.1007/s11126-015-9337-7.10.1007/s11126-015-9337-7 [DOI] [PubMed] [Google Scholar]
- 46.Rotondi AJ, Eack SM, Hanusa BH, Spring MB, Haas GL. Critical design elements of e-health applications for users with severe mental illness: singular focus, simple architecture, prominent contents, explicit navigation, and inclusive hyperlinks. Schizophr Bull. 2015 Mar;41(2):440–8. doi: 10.1093/schbul/sbt194. https://europepmc.org/abstract/MED/24375458 .sbt194 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Huerta-Ramos E, Escobar-Villegas MS, Rubinstein K, Unoka ZS, Grasa E, Hospedales M, Jääskeläinen Erika, Rubio-Abadal E, Caspi A, Bitter I, Berdun J, Seppälä Jussi, Ochoa S, Fazekas K, M-RESIST Group. Corripio I, Usall J. Measuring Users' Receptivity Toward an Integral Intervention Model Based on mHealth Solutions for Patients With Treatment-Resistant Schizophrenia (m-RESIST): A Qualitative Study. JMIR Mhealth Uhealth. 2016 Sep 28;4(3):e112. doi: 10.2196/mhealth.5716. https://mhealth.jmir.org/2016/3/e112/ v4i3e112 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Isohanni M, Miettunen J, Penttilä M. Psychotic Disorders: Comprehensive Conceptualization and Treatments. New York: Oxford University Press; 2020. Life span development of schizophrenia: symptoms, clinical course, and outcomes. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
m-RESIST tailored therapeutic program.
User experience questionnaire: open questions for the app and the smartwatch.
Data Availability Statement
The data sets generated and analyzed during this study are available from the corresponding author on reasonable request.