Skip to main content
NCBI home page
International Journal of Methods in Psychiatric Research logoLink to International Journal of Methods in Psychiatric Research
. 2017 Mar 9;26(1):e1554. doi: 10.1002/mpr.1554

User profiles of an electronic mental health tool for ecological momentary assessment: MEmind

María Luisa Barrigón 1,2, Sofian Berrouiguet 1,3, Juan José Carballo 1,2,4, Covadonga Bonal‐Giménez 1, Pablo Fernández‐Navarro 5,6, Bernadette Pfang 7, David Delgado‐Gómez 8, Philippe Courtet 9, Fuensanta Aroca 10, Jorge Lopez‐Castroman 11, Antonio Artés‐Rodríguez 12,13,14, Enrique Baca‐García 1,2,14,15,16,17,✉; MEmind study group1,15,16,17,18,19
PMCID: PMC6877232  PMID: 28276176

Abstract

Ecological momentary assessment (EMA) is gaining importance in psychiatry. This article assesses the characteristics of patients who used a new electronic EMA tool: the MEmind Wellness Tracker. Over one year, 13811 adult outpatients in our Psychiatry Department were asked to use MEmind. We collected information about socio‐demographic data, psychiatric diagnoses, illness severity, stressful life events and suicidal thoughts/behavior. We compared active users (N = 2838) and non‐active users (N = 10,973) of MEmind and performed a Random Forest analysis to assess which variables could predict its use. Univariate analyses revealed that MEmind‐users were younger (42.2 ± 13.5 years versus 48.5 ± 16.3 years; χ 2 = 18.85; P < 0.001) and more frequently diagnosed with anxiety related disorders (57.9% versus 46.7%; χ 2 = 105.92; P = 0.000) than non‐active users. They were more likely to report thoughts about death and suicide (up to 24% of active users expressed wish for death) and had experienced more stressful life events than non‐active users (57% versus 48.5%; χ 2 = 64.65; P < 0.001). In the Random Forest analysis, 31 variables showed mean decrease accuracy values higher than zero with a 95% confidence interval (CI), including sex, age, suicidal thoughts, life threatening events and several diagnoses. In the light of these results, strategies to improve EMA and e‐Mental Health adherence are discussed.

Keywords: experience sampling methods, health records, internet, Random Forest analysis, smartphone, suicide

1. INTRODUCTION

Three out of four persons in Western societies use the internet regularly (Internet World Stats, 2015). According to the European Survey on Information and Communication Technology, in 2013, 72% of Europeans used the internet at least once a week and 62% of them used it every day or almost every day. The proportion of individuals who have never used the internet decreased from 42% in 2006 to 21% in 2013 (Eurostat, 2013). In Spain, figures are similar. In 2014, 71.2% of individuals aged between 16 and 74 years used the internet at least once a week and 60% used it daily (Instituto Nacional de Estadistica, 2014)

Given its widespread use, it is not surprising that the internet has reached the health sector, and currently electronic health (e‐Health) is an expanding field. Electronic mental health (e‐Mental Health) is an emergent area within e‐Health (Firth, Torous, & Yung, 2016; Marzano et al., 2015). E‐Mental Health delivers or enhances mental health services and information through the internet and related technologies (Christensen & Hickie, 2010; Lal & Adair, 2014; Mental Health Commission of Canada, 2014). It comprises different instruments such as computerized interventions, virtual reality or gaming in the therapeutic field, telehealth and telemedicine, peer support through social media and other technologies, and evaluation of big data or symptoms by means of web tools, smartphone technologies or wearable devices (Mental Health Commission of Canada, 2014).

With regards to evaluation, e‐Mental Health techniques enable clinical and research approaches that focus on capturing real‐world and real‐time data, typically referred to as ecological momentary assessment (EMA) (Stone, Shiffman, Atienza, & Nebeling, 2007). Using EMA, symptoms are captured at the moment they occur or very shortly thereafter, thus reducing retrospective recall biases associated with traditional medical visits. EMA also allows self‐assessment in natural environments, rather than institutional settings, thus maximizing ecological validity (Shiffman, Stone, & Hufford, 2008).

Regarding the validity of online health interviews, evidence supports that people are more forthright on online health questionnaires than during face‐to‐face interviews (Bennett & Glasgow, 2009), especially regarding sensitive areas of importance in psychiatry, such as traumatic events, substance abuse or suicidal thoughts and/or behavior (Barak, 2007; Christensen & Hickie, 2010; Lin et al., 2007)

Since its development, EMA has been used to investigate different psychiatric disorders such as anxiety (Walz, Nauta, & Aan Het Rot, 2014), obsessive compulsive disorder (Tilley & Rees, 2014), affective disorders (Barge‐Schaapveld, Nicolson, Berkhof, & deVries, 1999; Husky, Mazure, Maciejewski, & Swendsen, 2008; Peeters, Berkhof, Delespaul, Rottenberg, & Nicolson, 2006; Putnam & McSweeney, 2008), psychosis (Myin‐Germeys, Birchwood, & Kwapil, 2011; Myin‐Germeys et al., 2003; Swendsen, Ben‐Zeev, & Granholm, 2011), attention deficit hyperactivity disorder (O'Mahony, Florentino‐Liano, Carballo, Baca‐García, & Rodríguez, 2014), eating disorders (Forbush & Hunt, 2014; Hilbert & Tuschen‐Caffier, 2007; Smyth et al., 2007), substance use disorders (Shiffman, 2009), borderline personality disorder (Hasler, Hopwood, Jacob, Brändle, & Schulte‐Vels, 2014; Santangelo, Bohus, & Ebner‐Priemer, 2012) and, most recently, in population at risk for suicide (Husky et al., 2014). EMA instruments provide a framework for ecological assessments in psychiatry but to date their application is heterogeneous and not comprehensive; although there have been close correlates in the field of primary care, such as the “My Own Health Report” which assesses 10 domains of health behaviors and psychosocial issues: eating patterns, physical activity, sleep, smoking, risky drinking, substance abuse, stress, anxiety and depression, overall health status and demographics (Glasgow et al., 2014).

With a substantial growth in its research over the last five years, use of EMA tools has gained broad acceptance in research (Firth et al., 2016), but there is still a lack of knowledge in clinical practice. Moreover, some initiatives that tried to introduce e‐Health into daily life have failed. For instance, only 0.13% of people invited to HealthSpace, a personal electronic health record promoted by the National Health Service (NHS) in England, opened an advanced account (Greenhalgh, Hinder, Stramer, Bratan, & Russell, 2010). These facts highlight the need for a proper characterization of those patients who are prone to use e‐Health apps.

The aim of this study was to determine the socio‐demographic and clinical characteristics of active users of a new EMA tool, MEmind Wellness Tracker. This knowledge may help to step forward the actual shortcomings in the adherence of EMA and e‐Mental Health apps. We hypothesize that active MEmind users will present a specific profile indicating a higher vulnerability to anxiety and affective disorders and suicidal behaviors.

2. MATERIAL AND METHODS

2.1. Materials: MEmind Wellness Tracker

MEmind is a web application that was developed to merge different data sources (including patient and caregiver inputs) and provide summaries of the patient state in clinical practice. The web application is available at http://www.memind.net using internet‐connected devices including Smartphone, Tablet and computers (with any operating system). The MEmind application has two interfaces, one for clinicians: the “electronic health record (EHR) view” and another for patients: the “EMA view” (clinicians can access data registered by patients in subsequent visits and use this information in their treatment routine).

The EHR view was designed to capture data from standard psychiatric assessments, including socio‐demographic, diagnostic and treatment information and nursing annotations (vital signs and anthropometric measurements). Clinicians received training through a video tutorial (see Supporting Information Multimedia Appendix) and could ask for support through the MEmind support web page (web.http://memind.net). Information collected can be seen in screenshots of the web in Pictures 1–6 (see Supporting Information).

The EMA interface consisted of three tabs: (1) How are you today?, with questions on eating and sleeping and the World Health Organization (WHO) (Five) Well‐being Index (WHO, 2015); (2) General Health Questionnaire, with this questionnaire in the 12 items version (Sánchez‐López & Dresch, 2008); (3) Notes, a free‐text area (see Supporting Information Pictures 7–10).

2.2. Setting and participants

Our project was carried out during a one‐year period and included all adult outpatient psychiatric services within the Psychiatry Department of Hospital Fundación Jiménez Díaz in Madrid, Spain. This Department comprises six community mental health centers and is part of the Spanish National Health Service, providing tax‐funded medical care to a catchment area of around 850,000 people.

From May 2014, all clinicians (doctors, clinical psychologists and nurses) of the Psychiatry Department were encouraged to use the MEmind Wellness Tracker systematically in their clinical activity, after receiving specific training. A total of 84 clinicians were invited to recruit patients and 66 (78.6%) participated actively in the recruitment (details on clinician participation in each center can be seen in Supporting Information Table S1).

All patients in follow‐up were eligible for the study. During this first year of use of MEmind, 34,236 patients were assisted in the mental health centers. Inclusion criteria were either male or female outpatients, aged 18 or older, who gave written informed consent. Exclusion criteria were illiteracy, refusal to participate, current imprisonment, being under guardianship and emergency situations in which the patient's state of health did not allow for a written informed consent.

2.3. Ethical considerations and data protection

The study was carried out in compliance with the Declaration of Helsinki and approved by the Local Ethics Committee. All participants gave written informed consent, after a complete description of the study.

Concerning data protection, access to the online user interface was restricted to participating clinicians (MEmind Study Group). User identification was ensured by a username and password. The data provided by the clinician was encrypted by Secure Socket Layer/Transport Layer Security (SSL/TLS) between the investigator's computer and the server. Data was stored in an external server created for research purposes. Only the principal investigator (EBG) had an access code to the server. MEmind used keys when creating encrypted volumes and any snapshots created from the encrypted volumes. Each volume was encrypted with a unique 256‐bit key; any snapshots of this volume and any subsequent volumes created from those snapshots also shared that key. These keys were protected by a key management infrastructure, which implemented strong logical and physical security controls to prevent unauthorized access. Data and associated keys were encrypted using the industry‐standard AES‐256 algorithm. Furthermore, an external auditor guaranteed that security measures met the Organic Law for Data Protection standards at a high protection level.

2.4. Study procedure

Participants were registered in the web tool and received a username and a password. The username remained the same throughout all contacts with psychiatric services within the study area. In this way, any clinician involved in the follow‐up of a patient could access his/her MEmind records. Clinicians collected the following information in the EHR: socio‐demographic data, ICD‐10 (International Classification of Diseases, 10th revision) mental disorder diagnoses (WHO, 1992), illness severity measured by the Clinical Global Impression (CGI) scale (Guy, 1976), medical conditions, stressful life events during the previous six months using the Life Threatening Events (LTE) (Brugha & Cragg, 1990) and suicidal thoughts and behavior: wish for death, desire for self‐harm, thoughts about suicide, suicide plans and suicide attempts.

Subsequently, patients were encouraged to connect to the EMA interface as many times as they wanted to (they were told that they could access the website as often as they wanted and a maximum of one register per day would be created, but no instruction about the frequency of connection was made, similarly no reminder [e.g. e‐mail or sms] was made in this first step of MEmind development). Patients who had accessed the web tool at least once were considered active users; otherwise, they were considered non‐active users.

2.5. Statistical analysis

Characteristics of active and non‐active users were compared using chi‐square test and t‐test as appropriate. The Statistical Package for the Social Sciences (SPSS) version 22.0 was used for the analyses.

To determine which variables would predict the use of MEmind, we performed feature selection using Random Forest (Breiman, 2001) and a 10‐fold Cross Validation (CV) approach for model evaluation. Random Forest is a classification algorithm (Breiman, 2001) that uses an ensemble of classification trees (Breiman, Friedman, Stone, & Olshen, 1984; Hastie, Tibshirani, & Friedman, 2009; Ripley, 2007). Each of the classification trees is built using a bootstrap sample of the data, and at each split the candidate set of variables is a random subset of the variables. Random Forest has demonstrated excellent performance in classification tasks, being comparable to that of support vector machines. It allows selection taking into account possible interactions between variables (Baca‐García et al., 2006; Baca‐García et al., 2007; Dudoit & Fridlyand, 2003; Liaw & Wiener, 2002; Wu et al., 2003). In our study, we performed a 10‐fold CV approach to avoid model overfitting. We randomly divided the original database into 10 databases. Nine of these folds were used as training sets to estimate the best Random Forest “mtry” and “ntree” parameters (those that confer the lower out‐of‐bag (OOB) error model) (Breiman, 2001). Using the Random Forest algorithm with these parameters in the remaining fold denominated “test set” (different for each of the 10 databases), the “mean decrease accuracy” (MDA) was calculated for each variable. The MDA is a measure of the impact of each feature on the accuracy of the model (Breiman, 2001). We selected those variables or features with a mean MDA value higher than zero (mean of 10 estimations in the 10 test sets of the approach for each variable). Finally, we repeated the same approach (10‐fold CV) with the selected variables, to perform a final model evaluation consisting in the estimation of the mean values of accuracy, precision, sensitivity and specificity of the model in the test sets of the 10 databases created.

3. RESULTS

From May 2014 to May 2015, 15,438 patients were registered for MEmind, for 1627 patients there were missing data, thus 13,811 patients were analyzed, of which 2838 (20.5%) accessed the EMA interface on at least one occasion.

3.1. Univariate analysis

Table 1 summarizes the demographic and clinical characteristics of the total sample and compares active users and non‐active users. Participants were mostly women (62%), with a mean age of 47.2 years old and with an active job status (47.6%). Most patients were diagnosed with anxiety related disorders (F4 ICD‐10) (49%) and with mood disorders (F3 ICD‐10) (23.5%). Active users were slightly younger than non‐active users (42.2 ± 13.5 years versus 48.5 ± 16.3 years; χ 2 = 18.85; P < 0.001) and more likely to be women (64.7% versus 61.4%; χ 2 = 10.36; P < 0.001). Regarding diagnoses, among those patients diagnosed with anxiety related disorders there were more active users (57.9% active users versus 46.7% non‐active users; χ 2 = 105.92; P = 0.000), while among those with mental disorders due to physiological conditions, substance use disorders, psychotic disorders and mental retardation there were more non‐active users. In those diagnosed with affective disorders, differences in use of the application were not found. In addition, Tables 2 and 3 show the profile of patients regarding suicidal thoughts and behaviors and experience of stressful life events, respectively. Active users were more likely to have thoughts about death and suicide, and up to 24% of them expressed a wish to die (Table 2). Active users were more likely to experience stressful life events than non‐active users, with 57% of active users suffering at least one stressful life event compared to 48.5% of non‐active users (χ 2 = 64.65; P < 0.001) (Table 3).

Table 1.

Demographic characteristics and clinical features across groups

Demographic and clinical characteristics All participants (N = 13,811) Statistics

Active users

(N = 2838)

Non‐active users

(N = 10,973)

 t/χ 2 df P Value
Age, years (mean ± standard deviation) 47.2 ± 15.9 42.2 ± 13.5 48.5 ± 16.3 18.85 13809 0.000
Sex (% males) 38% (5242) 35.3% (1003) 38.6% (4239) 10.36 1 0.001
Marital status (% married)a 49.1% (6303/12834) 53.6% (1433/2673) 47.9% (4870/10161) 82.57 3 0.000
Job status (% currently employed)b 47.6% (6041/12698) 54.6% (1437/2633) 45.7% (4604/10065) 248.49 5 0.000
CGI‐Severity (% moderately ill and more) 45% (5384/11955) 43.4% (1079/2485) 45.5% (4305/9470) 3.31 1 0.069
ICD‐10 Diagnosis (N = 12753)
(F00–F09) Disorders due to physiological conditions 2.8% (362) 1% (27) 3.3% (335) 40.96 1 0.000
(F10–F19) Substance use disorders 7.7% (987) 4.8% (128) 8.5% (859) 41.17 1 0.000
(F20–F29) Schizophrenia and related disorders 12% (1524) 6.7% (180) 13.3% (1344) 87.32 1 0.000
(F30–F39) Mood disorders 23.5% (2992) 23% (614) 23.6% (2378) 0.44 1 0.508
(F40–F49) Anxiety disorders 49% (6255) 57.9% (1547) 46.7% (4708) 105.92 1 0.000
(F50–F59) Disorders related to physiological disturbances 4.8% (611) 5.6% (149) 4.6% (462) 4.57 1 0.033
(F60–F69) Personality disorders 12.4% (1576) 13.4% (358) 12.1% (1218) 3.38 1 0.066
(F70–F79) Mental retardation 1.4% (182) 0.7% (19) 1.6% (163) 12.32 1 0.000
(F80–F89) Disorders of psychological development 0.3% (38) 0.2% (6) 0.3% (32) 0.631 1 0.434
(F90–F99) Childhood and adolescence onset disorders 20.1% (2564) 19.7% (526) 20.2% (2038) 0.37 1 0.543
Open in a new tab
a

This variable also includes subjects currently living with a partner at least during the last six months.

b

This variable also includes students and housewives.

Table 2.

Passive and active suicidal thoughts and/or behaviors profile across groups

Suicidal thoughts and behaviors

All participants

(N = 13,811)

 Statistics

Active users

(N = 2838)

Non‐active users

(N = 10,973)

 χ 2 df P Value
Wish for Death 17.8% (2463) 24% (682) 16.2% (1781) 93.63 1 0.000
Wish for Self‐Harm 7.2% (999) 8.5% (240) 6.9% (759) 7.97 1 0.005
Thoughts about Suicide 9.8% (1359) 12% (341) 9.2% (1015) 19.48 1 0.000
Suicide Plan 2.9% (395) 3.9% (110) 2.6% (285) 13.27 1 0.000
Suicide Attempt 8.6% (1181) 8.5% (242) 8.5% (939) 0.003 1 0.959
Open in a new tab

Table 3.

Stressful life events experienced across groups

Stressful life events

All participants

(N = 13,811)

 Statistics

Active users

(N = 2838)

Non‐active users

(N = 10,973)

 χ 2 df P Value
1. Serious injury or illness to subject 13.7% (1894) 15.2% (432) 13.3% (1462) 6.87 1 0.004
2. Serious injury or illness to a close relative 13.1% (1812) 15.8% (448) 12.4% (1364) 22.27 1 0.000
3. Death of first‐degree relative including child or spouse 9.1% (1263) 10.8% (306) 8.7% (957) 11.53 1 0.000
4. Death of close family friend or second‐degree relative 5.9% (810) 8.6% (243) 5.2% (567) 47.08 1 0.000
5. Separation due to marital difficulties 4.8% (667) 5.1% (146) 4.7% (521) 0.77 1 0.380
6. Broke off a steady relationship 8.3% (1151) 10.6% (302) 7.7% (849) 24.89 1 0.000
7. Serious problem with a close friend, neighbor or relative 15.3% (2111) 18.7% (530) 14.4% (1581) 31.71 1 0.000
8. Unemployed or seeking work for more than one month 7.8% (1077) 10.1% (288) 7.2% (789) 27.43 1 0.000
9. Subject sacked from work 4.5% (622) 6.2% (175) 4.1% (447) 22.96 1 0.000
10. Major financial crisis 9.2% (1271) 10.2% (290) 8.9% (981) 4.41 1 0.036
11. Problems with police and court appearance 3.3% (455) 4% (115) 3.1% (340) 6.44 1 0.011
12. Something valuable lost or stolen 2% (282) 2.7% (78) 1.9% (204) 8.92 1 0.003
Open in a new tab

3.2. Random Forest analysis

Figure 1 shows the mean MDA of all variables. There were eight features with mean MDA values lower than zero that showed a 95% confidence interval (CI). These variables were living with the father, having attempted suicide since the last visit to a community mental health center and psychiatric diagnoses in the ICD‐10 categories F0, F1, F5, F6, F7 and F8. However, 31 features with mean MDA values higher than zero showed a 95% CI (see Figure 2).

Figure 1.

Figure 1

Open in a new tab

Random Forest of all variables included in analysis. ICD codes; F00–F09: Mental disorders due to known physiological conditions; F10–F19: Mental and behavioral disorders due to psychoactive substance use; F20–F29: Schizophrenia, schizotypal, delusional, and other non‐mood psychotic disorders; F30–F39: Mood [affective] disorders; F40–F49: Anxiety, dissociative, stress‐related, somatoform and other non‐psychotic mental disorders; F50–F59: Behavioral syndromes associated with physiological disturbances and physical factors; F60–F69: Disorders of adult personality and behavior; F70–F79: Mental retardation; F80–F89: Disorders of psychological development; F90–F98: Behavioral and emotional disorders with onset usually occurring in childhood and adolescence. EEAG (GAF: Global Assessment of Functioning); CGI: Clinical Global Impression; ICD F30–F39: Mood [affective] disorders; SLE: Stressful Life Events

Figure 2.

Figure 2

Open in a new tab

Random Forest analysis: final model. ICD codes; F20–F29: Schizophrenia, schizotypal, delusional, and other non‐mood psychotic disorders; F30–F39: Mood [affective] disorders; F40–F49: Anxiety, dissociative, stress‐related, somatoform and other non‐psychotic mental disorders; F50–F59: Behavioral syndromes associated with physiological disturbances and physical factors; F90–F98: Behavioral and emotional disorders with onset usually occurring in childhood and adolescence. EEAG (GAF: Global Assessment of Functioning); CGI: Clinical Global Impression; SLE: Stressful Life Events

The final model evaluation using the 31 selected features showed a mean accuracy value of 79.45% (95% CI: 79.43–79.47%) and mean specificity value of one (1–1). However, both sensibility and precision mean values were zero.

4. DISCUSSION

Compared to non‐active users, patients using MEmind were younger, were more often diagnosed with anxiety or mood disorders but less frequently with psychotic disorders, and had a higher prevalence of wish for death and suicidal ideation.

4.1. Socio‐demographic characteristics

Age was the most important variable related with being an active user of the web tool. This finding is consistent with previous results showing a better acceptance of e‐Mental Health among young people, corresponding with higher technological abilities in this group. (Cunningham, Gulliver, Farrer, Bennett, & Carron‐Arthur, 2014; Lal & Adair, 2014). Nevertheless, in our sample MEmind users were only slightly younger than non‐users. This finding is intriguing since previous findings suggest a greater use of e‐Mental Health among young adults (Cunningham et al., 2014; Lal & Adair, 2014) and a bigger age gap would be expected. A plausible explanation is that we developed this pilot study for adult outpatients and thus young people were not included. Marital status and job status were also predictors of the use of MEmind in the Random Forest analysis, with patients in the “married/living with partner” and “working” categories being active users more frequently than patients in other categories. A previous study in a small sample of 218 Australians did not find differences in demographic variables, including marital status, between people who preferred e‐health and those who did not (Klein & Cook, 2010).

4.2. Use of MEmind according to clinical characteristics

The Random Forest analysis showed that anxiety related disorder and mood disorder diagnoses were among the most important variables predicting to be an active user. Severity of illness was among the top 10 variables predicting to be an active user, with people with less severe illness using MEmind more than those with more severe illness, although this difference was not statistically significant in univariate analysis. This is in line with the promising results obtained by online psychological interventions in mild to moderate depression and anxiety disorders (Cunningham et al., 2014). Moreover, the current NICE (National Institute for Health and Care Excellence) recommendation considers e‐Mental Health treatments as the first line of treatment for mild to moderate depression (Mental Health Commission of Canada, 2014). Similarly, several authors have proposed low‐intensity treatment using computerized therapies for patients with mild intensity mental disorders in a stepped‐care model (Van Straten, Seekles, van't Veer‐Tazelaar, Beekman, & Cuijpers, 2010).

Furthermore, patients with more severe disorders such as those with mental disorders due to physiological conditions and psychotic disorders accessed the website less often, in keeping with previous research, which observed low access to the internet among people with serious mental illness including psychotic disorders (Clayton et al., 2009). In this population, cognitive deficits and delusions may interfere with the use of technological devices (Bell, Grech, Maiden, Halligan, & Ellis, 2005; Rotondi et al., 2007), and other EMA approaches should be considered, for instance inertial measurement units (IMUs) for measuring movement and level of activity (O'Mahony et al., 2014). However, in the field of intervention a recent review suggests that e‐Mental Health services are at least as effective as usual care or non‐technological approaches, opening the door to investigation on e‐Mental Health interventions in psychosis (Van der Krieke, Wunderink, Emerencia, de Jonge, & Sytema, 2014).

4.3. MEmind and self‐injurious thoughts and behavior

In our sample, those patients answering affirmatively to questions regarding suicide in clinical evaluation were most likely to be active users. A recent study which used EMA technology for assessing people at different levels of risk for suicide, with daily assessments during one week, found good acceptability among participants. More importantly, negative and suicidal thoughts were not reactivated during repeated assessment (Husky et al., 2014). These results are consistent with our findings. Patients with suicidal thoughts tended to use MEmind, probably because they felt comfortable with the anonymous nature of the online interface (Barak, 2007; Christensen & Hickie, 2010; Lin et al., 2007). In light of these results it seems plausible to consider that an adequate identification of these subjects at risk could help in the development of specific preventive and/or clinical interventions.

As a related issue, it is worth mentioning a pioneer study seeking to evaluate suicidal behavior using the internet with a large – more than 48,000 people – population‐based sample (de Araújo, Mazzochi, Lara, & Ottoni, 2015). Speaking of this project, Grunebaum stated that “Big Data has reached the field of suicidology” (Grunebaum, 2015), thus highlighting the growing importance of EMA methodology in investigation of the suicide phenomenon. Another interesting project using the internet is the Israeli‐based suicide‐prevention initiative, SAHAR (Barak, 2007). Although in our pilot analysis we did not initially consider developing prevention strategies for those subjects reporting self‐injurious thoughts and behaviors, given the results of our study we believe that the development of a preventive protocol would be a feasible goal in the future.

4.4. Strengths, limitations and future possibilities

The main strength of our study is the development of a powerful novel tool for efficient data collection from a very large sample. MEmind is an EMA tool designed for the comprehensive evaluation of mental conditions, with easy access through any device with internet connection (a mobile app will soon be available). Not only does our web tool have important implications for research in mental health, but also promises to be an effective aid in clinical practice.

Furthermore, MEmind provides patients a space where they can freely express themselves. It is well known that this sense of privacy can facilitate sincerity concerning more sensitive questions (Barak, 2007; Christensen & Hickie, 2010; Lin et al., 2007). In this study we did not analyze the comments of active users, but we observed that a larger proportion of those answering affirmatively to questions regarding suicide asked by the clinician were active users, probably in relation with this sense of security and privacy.

Unlike other studies, our participants were not instructed about a desired frequency of use of the tool and no reminder was made in order to avoid the inconvenience associated with EMA research designs and to minimize the risk of intrusiveness into daily life (Rhee, Miner, Sterling, Halterman, & Fairbanks, 2014).

Our study has also some limitations. First of all, only a small percentage – around 20% – of recruited participants decided to use MEmind, which challenges us to develop strategies to engage patients with our tool. Secondly, at present our tool is designed for evaluation with no option for treatment, a possibility that would be particularly important with regards to suicide, as we have observed people with wish for death and suicidal ideation are willing to use the web tool. A future addition will be a warning message for the clinician when a patient repeatedly provides affirmative information about suicidal ideation. Finally, there are limitations inherent to the use of MEmind in the clinical setting as it is relatively time‐consuming for clinicians during consultation, which complicates optimal data collection.

Most importantly, taking into account our results and as mentioned earlier, our web tool has many future possibilities. As one of the current limitations is the low rate use, gamification (the implementation of mechanics of video games in non‐video game contexts) could be an interesting strategy (Miller, Cafazzo, & Seto, 2014). In addition, personalized questionnaires, giving feedback and permitting patient access to their own registers and symptom charts could promote usability. We have developed an interface for family and caregivers in order to add information, which is particularly useful in child patients and those with severe mental illness. As a final goal, intervention and treatment modules will be developed, which will make MEmind a more useful tool in clinical practice, focusing particularly on the principal users (patients with anxiety related and mood disorders with a mild level of symptoms), which will also optimize clinicians' time.

Our initiative is in line with international government approaches guiding e‐Mental Health at policy and strategy levels (Mental Health Commission of Canada, 2014). Although currently there is no well‐defined European strategy concerning this issue, reference has been made to the important role of information and communication technology regarding affordable health care (Digital Agenda for Europe – European Commission, 2015).

5. CONCLUSION

Despite outstanding issues regarding quality control, access and security, e‐Mental Health is a rising phenomenon with potential to transform and improve health care delivery. In this study, we present the characteristics of active users of MEmind. It would be necessary to develop tools based on implicit data for non‐active users, seemingly older and with more severe illness. Future gamification and feedback strategies could increase acceptability and adherence. MEmind will allow clinicians and researchers to access large populations of patients and caregivers in their own environment in an efficient way.

Supporting information

Multimedia Appendix_MeMind ‐ Tutorial

Click here for additional data file. (156.5MB, mp4)

Supplementary Table S1. Clinicians' participation

Click here for additional data file. (14.7KB, docx)

Picture 1. EHR view_tab1

Click here for additional data file. (96.4KB, png)

Picture 2. EHR view_tab2

Click here for additional data file. (102.6KB, png)

Picture 3. EHR view_tab3

Click here for additional data file. (78.1KB, png)

Picture 4. EHR view_tab4

Click here for additional data file. (71.9KB, png)

Picture 5. EHR view_tab5

Click here for additional data file. (67.1KB, png)

Picture 6. EHR view_tab6

Click here for additional data file. (48.8KB, png)

Picture 7. EMA view_Tab 1_First Questions

Click here for additional data file. (112.9KB, png)

Picture 8. EMA view_Tab 1_WHO 5

Click here for additional data file. (92.4KB, png)

Picture 9. EMA view_Tab 2_GHQ 12

Click here for additional data file. (253.3KB, png)

Picture 10. EMA view_Tab 3_Notes

Click here for additional data file. (42.3KB, png)

ACKNOWLEDGEMENTS

Financial support for this study was provided in part by Instituto de Salud Carlos III co‐founded by Fondo Europeo de Desarrollo Regional – FEDER (PI13/02200 Grant), Plan Nacional de Drogas (project 2015I073), Papiit Grant IN108216 and by the French Embassy (Scientific Coordination Office, Nicolas Urai) in Madrid, Spain. The funding agreement ensured the authors' independence in designing the study, interpreting the data, writing, and publishing the report.

DECLARATION OF INTEREST STATEMENT

The authors have no competing interests.

Barrigón ML, Berrouiguet S, Carballo JJ, et al. User profiles of an electronic mental health tool for ecological momentary assessment: MEmind. Int J Methods Psychiatr Res. 2017;26:e1554 10.1002/mpr.1554.

REFERENCES

  1. Baca‐García, E. , Perez‐Rodriguez, M. M. , Basurte‐Villamor, I. , Saiz‐Ruiz, J. , Leiva‐Murillo, J. M. , de Prado‐Cumplido, M. , … de Leon, J. (2006). Using data mining to explore complex clinical decisions: A study of hospitalization after a suicide attempt. The Journal of Clinical Psychiatry, 67(7), 1124–1132. [DOI] [PubMed] [Google Scholar]
  2. Baca‐García, E. , Perez‐Rodriguez, M. M. , Saiz‐Gonzalez, D. , Basurte‐Villamor, I. , Saiz‐Ruiz, J. , Leiva‐Murillo, J. M. , … de Leon, J. (2007). Variables associated with familial suicide attempts in a sample of suicide attempters. Progress in Neuro‐Psychopharmacology and Biological Psychiatry, 31(6), 1312–1316. [DOI] [PubMed] [Google Scholar]
  3. Barak, A. (2007). Emotional support and suicide prevention through the Internet: A field project report. Computers in Human Behavior, 23(2), 971–984. [Google Scholar]
  4. Barge‐Schaapveld, D. Q. C. M. , Nicolson, N. A. , Berkhof, J. , & deVries, M. W. (1999). Quality of life in depression: Daily life determinants and variability. Psychiatry Research, 88(3), 173–189. [DOI] [PubMed] [Google Scholar]
  5. Bell, V. , Grech, E. , Maiden, C. , Halligan, P. W. , & Ellis, H. D. (2005). ‘Internet delusions’: A case series and theoretical integration. Psychopathology, 38(3), 144–150. [DOI] [PubMed] [Google Scholar]
  6. Bennett, G. G. , & Glasgow, R. E. (2009). The delivery of public health interventions via the internet: Actualizing their potential. Annual Review of Public Health, 30(1), 273–292. [DOI] [PubMed] [Google Scholar]
  7. Breiman, L. (2001). Random Forests. Machine Learning, 45(1), 5–32. [Google Scholar]
  8. Breiman, L. , Friedman, J. , Stone, C. J. , & Olshen, R. A. (1984). Classification and Regression Trees. London: Taylor & Francis. [Google Scholar]
  9. Brugha, T. S. , & Cragg, D. (1990). The list of threatening experiences: The reliability and validity of a brief life events questionnaire. Acta Psychiatrica Scandinavica, 82(1), 77–81. [DOI] [PubMed] [Google Scholar]
  10. Christensen, H. , & Hickie, I. B. (2010). Using e‐health applications to deliver new mental health services. Medical Journal of Australia, 192(11), S53–S56. [DOI] [PubMed] [Google Scholar]
  11. Clayton, S. , Borzekowski, D. , Himelhoch, S. , Dixon, L. , Potts, W. , Medoff, D. , … Balis, D. (2009). Media and internet ownership and use among mental health outpatients with serious mental illness. American Journal of Drug & Alcohol Abuse, 35(5), 364–367. [DOI] [PubMed] [Google Scholar]
  12. Cunningham, J. A. , Gulliver, A. , Farrer, L. , Bennett, K. , & Carron‐Arthur, B. (2014). Internet interventions for mental health and addictions: Current findings and future directions. Current Psychiatry Reports, 16(12), 521. [DOI] [PubMed] [Google Scholar]
  13. de Araújo, R. M. F. , Mazzochi, L. , Lara, D. R. , & Ottoni, G. L. (2015). Thinking about dying and trying and intending to die: Results on suicidal behavior from a large web‐based sample. The Journal of Clinical Psychiatry, 76(3), e359–e365. [DOI] [PubMed] [Google Scholar]
  14. Digital Agenda for Europe – European Commission . (2015). Digital Agenda for Europe. https://ec.europa.eu/digital-agenda/ [6 December 2015].
  15. Dudoit, S. , & Fridlyand, J. (2003). Classification in microarray experiments In Statistical Analysis of Gene Expression Microarray Data. (pp. 93–158). New York: CRC Press. [Google Scholar]
  16. Eurostat . (2013). Internet use statistics – individuals. http://ec.europa.eu/eurostat/statistics-explained/index.php/Internet_use_statistics_-_individuals [29 August 2015].
  17. Firth, J. , Torous, J. , & Yung, A. R. (2016). Ecological momentary assessment and beyond: The rising interest in e‐mental health research. Journal of Psychiatric Research, 80, 3–4. [DOI] [PubMed] [Google Scholar]
  18. Forbush, K. T. , & Hunt, T. K. (2014). Characterization of eating patterns among individuals with eating disorders: What is the state of the plate? Physiology & Behavior, 134, 92–109. [DOI] [PubMed] [Google Scholar]
  19. Glasgow, R. E. , Kessler, R. S. , Ory, M. G. , Roby, D. , Gorin, S. S. , & Krist, A. (2014). Conducting rapid, relevant research: Lessons learned from the my own health report project. American Journal of Preventive Medicine, 47(2), 212–219. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Greenhalgh, T. , Hinder, S. , Stramer, K. , Bratan, T. , & Russell, J. (2010). Adoption, non‐adoption, and abandonment of a personal electronic health record: Case study of HealthSpace. BMJ (Clinical Research Edition), 341, c5814. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Grunebaum, M. F. (2015). Suicidology meets ‘Big Data’. The Journal of Clinical Psychiatry, 76(3), e383–e384. [DOI] [PubMed] [Google Scholar]
  22. Guy, W. (1976). Early Clinical Drug Evaluation (ECDEU) Assessment Manual for Psychopharmacology. Rockville, MD: Department of Health, Education, and Welfare. http://miksa.ils.unc.edu/unc-hit/media/CGI.pdf [6 December 2014].
  23. Hasler, G. , Hopwood, C. J. , Jacob, G. A. , Brändle, L. S. , & Schulte‐Vels, T. (2014). Patient‐reported outcomes in borderline personality disorder. Dialogues in Clinical Neuroscience, 16(2), 255–266. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Hastie, T. , Tibshirani, R. , & Friedman, J. (2009). The Elements of Statistical Learning, Springer Series in Statistics. New York: Springer. [Google Scholar]
  25. Hilbert, A. , & Tuschen‐Caffier, B. (2007). Maintenance of binge eating through negative mood: A naturalistic comparison of binge eating disorder and bulimia nervosa. International Journal of Eating Disorders, 40(6), 521–530. [DOI] [PubMed] [Google Scholar]
  26. Husky, M. M. , Mazure, C. M. , Maciejewski, P. K. , & Swendsen, J. D. (2008). Past depression and gender interact to influence emotional reactivity to daily life stress. Cognitive Therapy and Research, 33(3), 264–271. [Google Scholar]
  27. Husky, M. , Olié, E. , Guillaume, S. , Genty, C. , Swendsen, J. , & Courtet, P. (2014). Feasibility and validity of ecological momentary assessment in the investigation of suicide risk. Psychiatry Research, 220(1–2), 564–570. [DOI] [PubMed] [Google Scholar]
  28. Instituto Nacional de Estadistica . (2014). Encuesta sobre Equipamiento y Uso de Tecnologías de Información y Comunicación en los Hogares. Año 2014. http://www.ine.es/prensa/np864.pdf [29 August 2015].
  29. Internet World Stats . (2015). http://www.internetworldstats.com/stats.htm [13 December 2015].
  30. Klein, B. , & Cook, S. (2010). Preferences for e‐mental health services amongst an online Australian sample? Sensoria: A Journal of Mind, Brain & Culture, 6(1), 28–39. [Google Scholar]
  31. Lal, S. , & Adair, C. E. (2014). E‐Mental health: A rapid review of the literature. Psychiatric Services, 65(1), 24–32. [DOI] [PubMed] [Google Scholar]
  32. Liaw, A. , & Wiener, M. (2002). Classification and regression by randomForest. R news, 2(3), 18–22. [Google Scholar]
  33. Lin, C.‐C. , Bai, Y.‐M. , Liu, C.‐Y. , Hsiao, M.‐C. , Chen, J.‐Y. , Tsai, S.‐J. , … Li, Y. C. (2007). Web‐based tools can be used reliably to detect patients with major depressive disorder and subsyndromal depressive symptoms. BMC Psychiatry, 7, 12. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Marzano, L. , Bardill, A. , Fields, B. , Herd, K. , Veale, D. , Grey, N. , & Moran, P. (2015). The application of mHealth to mental health: Opportunities and challenges. The Lancet. Psychiatry, 2(10), 942–948. [DOI] [PubMed] [Google Scholar]
  35. Mental Health Commission of Canada (2014). E‐Mental Health in Canada: Transforming the Mental Health System Using Technology. Ottawa, ON: http://www.mentalhealthcommission.ca Mental Health Commission of Canada. [Google Scholar]
  36. Miller, A. S. , Cafazzo, J. A. , & Seto, E. (2014). A game plan: Gamification design principles in mHealth applications for chronic disease management. Health Informatics Journal, 22(2), 184–193. [DOI] [PubMed] [Google Scholar]
  37. Myin‐Germeys, I. , Peeters, F. , Havermans, R. , Nicolson, N. A. , DeVries, M. W. , Delespaul, P. , & Van Os, J. (2003). Emotional reactivity to daily life stress in psychosis and affective disorder: An experience sampling study. Acta Psychiatrica Scandinavica, 107(2), 124–131. [DOI] [PubMed] [Google Scholar]
  38. Myin‐Germeys, I. , Birchwood, M. , & Kwapil, T. (2011). From environment to therapy in psychosis: A real‐world momentary assessment approach. Schizophrenia Bulletin, 37(2), 244–247. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. O'Mahony, N. , Florentino‐Liano, B. , Carballo, J. J. , Baca‐García, E. , & Rodríguez, A. A. (2014). Objective diagnosis of ADHD using IMUs. Medical Engineering & Physics, 36(7), 922–926. [DOI] [PubMed] [Google Scholar]
  40. Peeters, F. , Berkhof, J. , Delespaul, P. , Rottenberg, J. , & Nicolson, N. A. (2006). Diurnal mood variation in major depressive disorder. Emotion, 6(3), 383–391. [DOI] [PubMed] [Google Scholar]
  41. Putnam, K. M. , & McSweeney, L. B. (2008). Depressive symptoms and baseline prefrontal EEG alpha activity: A study utilizing Ecological Momentary Assessment. Biological Psychology, 77(2), 237–240. [DOI] [PubMed] [Google Scholar]
  42. Rhee, H. , Miner, S. , Sterling, M. , Halterman, J. S. , & Fairbanks, E. (2014). The development of an automated device for asthma monitoring for adolescents: Methodologic approach and user acceptability. JMIR mHealth and uHealth, 2(2). [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Ripley, B. D. (2007). Pattern Recognition and Neural Networks. Cambridge: Cambridge University Press. [Google Scholar]
  44. Rotondi, A. J. , Sinkule, J. , Haas, G. L. , Spring, M. B. , Litschge, C. M. , Newhill, C. E. , … Anderson, C. M. (2007). Designing websites for persons with cognitive deficits: Design and usability of a psychoeducational intervention for persons with severe mental illness. Psychological Services, 4(3), 202–224. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Sánchez‐López, M. P. S. , & Dresch, V. (2008). The 12‐Item General Health Questionnaire (GHQ‐12): Reliability, external validity and factor structure in the Spanish population. Psicothema, 20(4), 839–843. [PubMed] [Google Scholar]
  46. Santangelo, P. , Bohus, M. , & Ebner‐Priemer, U. W. (2012). Ecological momentary assessment in borderline personality disorder: A review of recent findings and methodological challenges. Journal of Personality Disorders, 28(4), 555–576. [DOI] [PubMed] [Google Scholar]
  47. Shiffman, S. (2009). Ecological momentary assessment (EMA) in studies of substance use. Psychological Assessment, 21(4), 486–497. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Shiffman, S. , Stone, A. A. , & Hufford, M. R. (2008). Ecological momentary assessment. Annual Review of Clinical Psychology, 4(1), 1–32. [DOI] [PubMed] [Google Scholar]
  49. Smyth, J. M. , Wonderlich, S. A. , Heron, K. E. , Sliwinski, M. J. , Crosby, R. D. , Mitchell, J. E. , & Engel, S. G. (2007). Daily and momentary mood and stress are associated with binge eating and vomiting in bulimia nervosa patients in the natural environment. Journal of Consulting and Clinical Psychology, 75(4), 629–638. [DOI] [PubMed] [Google Scholar]
  50. Stone, A. , Shiffman, S. , Atienza, A. , & Nebeling, L. (2007). The Science of Real‐Time Data Capture: Self‐Reports in Health Research. Oxford: Oxford University Press. [Google Scholar]
  51. Swendsen, J. , Ben‐Zeev, D. , & Granholm, E. (2011). Real‐time electronic ambulatory monitoring of substance use and symptom expression in schizophrenia. American Journal of Psychiatry, 168(2), 202–209. [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Tilley, P. J. M. , & Rees, C. S. (2014). A clinical case study of the use of ecological momentary assessment in obsessive compulsive disorder. Frontiers in Psychology, 5, 339. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Van der Krieke, L. , Wunderink, L. , Emerencia, A. C. , de Jonge, P. , & Sytema, S. (2014). E‐Mental health self‐management for psychotic disorders: State of the art and future perspectives. Psychiatric Services, 65(1), 33–49. [DOI] [PubMed] [Google Scholar]
  54. Van Straten, A. , Seekles, W. , van't Veer‐Tazelaar, N. J. , Beekman, A. T. F. , & Cuijpers, P. (2010). Stepped care for depression in primary care: what should be offered and how? Medical Journal of Australia, 192(11 Suppl), S36–S39. [DOI] [PubMed] [Google Scholar]
  55. Walz, L. C. , Nauta, M. H. , & Aan Het Rot, M. (2014). Experience sampling and ecological momentary assessment for studying the daily lives of patients with anxiety disorders: A systematic review. Journal of Anxiety Disorders, 28(8), 925–937. [DOI] [PubMed] [Google Scholar]
  56. World Health Organization (WHO) (1992). Tenth Revision of the International Classification of Diseases and Related Health Problems (ICD‐10). Geneva: WHO. [Google Scholar]
  57. World Health Organization (WHO) (2015). The WHO‐5 website. Retrieved August 29, 2015, from https://www.psykiatri-regionh.dk/who-5/Documents/WHO5_English.pdf
  58. Wu, B. , Abbott, T. , Fishman, D. , McMurray, W. , Mor, G. , Stone, K. , … Zhao, H. (2003). Comparison of statistical methods for classification of ovarian cancer using mass spectrometry data. Bioinformatics, 19(13), 1636–1643. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Multimedia Appendix_MeMind ‐ Tutorial

Click here for additional data file. (156.5MB, mp4)

Supplementary Table S1. Clinicians' participation

Click here for additional data file. (14.7KB, docx)

Picture 1. EHR view_tab1

Click here for additional data file. (96.4KB, png)

Picture 2. EHR view_tab2

Click here for additional data file. (102.6KB, png)

Picture 3. EHR view_tab3

Click here for additional data file. (78.1KB, png)

Picture 4. EHR view_tab4

Click here for additional data file. (71.9KB, png)

Picture 5. EHR view_tab5

Click here for additional data file. (67.1KB, png)

Picture 6. EHR view_tab6

Click here for additional data file. (48.8KB, png)

Picture 7. EMA view_Tab 1_First Questions

Click here for additional data file. (112.9KB, png)

Picture 8. EMA view_Tab 1_WHO 5

Click here for additional data file. (92.4KB, png)

Picture 9. EMA view_Tab 2_GHQ 12

Click here for additional data file. (253.3KB, png)

Picture 10. EMA view_Tab 3_Notes

Click here for additional data file. (42.3KB, png)

Articles from International Journal of Methods in Psychiatric Research are provided here courtesy of Wiley

RESOURCES