Sensing behavioral symptoms of mental health and delivering personalized interventions using mobile technologies

Abstract

Unlike most other health conditions, the treatment of mental illness relies on subjective measurement. In addition, the criteria for diagnosing mental illnesses are based on broad categories of symptoms that do not account for individual deviations from these criteria. The increasing availability of personal digital devices, such as smartphones that are equipped with sensors, offers a new opportunity to continuously and passively measure human behavior in situ. This promises to lead to more precise assessment of human behavior and ultimately individual mental health. More refined modeling of individual mental health and a consideration of individual context, assessed through continuous monitoring, opens the way for more precise and personalized digital interventions that may help increase the number of positive clinical outcomes in mental healthcare. In this paper, we provide a conceptual review of such techniques for measuring, modeling, and treating mental illness and maintaining mental health.

1. INTRODUCTION

Over the last 10 years, technology has become more proximal to human activity. As more and more people adopt today’s technology, healthcare involving technology in some respect is becoming increasingly acceptable. Ownership of smartphones is especially prevalent among underserved minority groups: 47% of black non-Hispanics and 49% of Hispanics own smart phones, compared to 42% of non-Hispanic whites (Pew Research Center, 2013). From the perspective of mental healthcare, mobile technology appears to be a feasible medium for delivering care; for example, a recent community-based survey of over 1,500 people with serious mental illnesses found that over 80% of patients with bipolar disorder (BD) owned and used mobile phones regularly for calling, texting, and the internet (Ben-Zeev, Davis, Kaiser, Krzsos, & Drake, 2013).

The evolution and rapid dissemination of mobile and sensor technology has created unprecedented opportunities for personalized data collection in an extremely granular, unobtrusive, and even affordable way. As a result, there has been a recent and significant increase in research to identify novel methods to measure, understand, and treat mental illness (and improve mental well-being). However, in terms of day-to-day clinical practice technological approaches for measuring and treating mental illness are far behind developments in general health. This shortcoming is particularly important given that the accurate understanding of daily behavior is central to mental health and wellness assessment.

The rise of Personal Informatics, consumer friendly tools that allow individuals to conveniently track a wide range of elements including mood, activity, and sleep, alongside the related Quantified Self movement, which focuses on using these tools to empower individuals, is enabling people to have more control over an ever-increasing amount of personal data. As early as 2006, the Institute of Medicine recognized that these technologies had the potential to transform mental health services by providing more continuous and precise information on patient-specific behavior, symptoms, and medication side effects. To transform this potential into practice requires that we develop methods to transform raw data into actionable knowledge and then use this in turn to support personalized interventions.

In this paper, we provide a broad overview of recent innovations in digital technologies to measure, understand, and intervene in mental healthcare and finish by presenting a three-stage framework that can be used when integrating such technologies into clinical practice.

2. MEASUREMENT: FROM GENERAL CATEGORIES TO PERSONALIZED ASSESSMENT

In practice, mental illnesses are defined by clinical criteria outlined in diagnostic manuals like the Diagnostic and Statistical Manual (DSM).Many of these criteria center on assessing behavior as a direct 2017 Wiley Periodicals, Inc. symptom of a mental illness. For example, decreased physical activity, abnormal sleep patterns, and social interactions are well-documented symptoms of depression (American Psychiatric Association, 2013). Significant changes in the amount of physical and mental activity can be a behavioral indicator of manic and depressive states in BD (Goodwin, & Jamison, 2007). Currently these conditions are measured using subjective questionnaires that have well-identified limitations (Stone, Shiffman, Schwartz, Broderick, & Hufford, 2003). In addition, the diagnostic criteria provide general bins in which to categorize symptoms, yet there may be other symptomatic and prodromal behaviors shared among some individuals or that are individual specific that have not yet been detected because there has been no way to detect them.

Some researchers have called for a move away from traditional methods of diagnosis such as the DSM-5—criteria used by mental health professionals to assign a diagnosis to patient—toward treatments that “convey the reality of the patient’s experience” (Licinio, 2005). Where current clinical criteria for mental illness make it easier for clinicians to identify and treat clusters of common symptoms, in reality each mental illness category—be it anxiety disorder or major depression—are umbrella terms for heterogeneous diseases. There is a growing recognition, evidenced by the creation of the Research Domain Criteria (RDoC), of the potential for neuroscience and genetics to transform the classification of disorders into clusters of pathophysiological symptoms (Insel et al., 2010). Although RDoC is promising for clinical research, it is not clear how long it will be before genetic and neuroscientific classifications of mental illnesses will be ready for clinical use (Kupfer, & Regier, 2011).

Objective behavior sensing via personal digital devices offers the potential to bridge this gap between the classical diagnostic categories and the fundamental neuroscientific classifications. There has been a marked rise in the use of personal technologies to automatically detect behavior, as evidenced by the growing number of commercially available devices for with inbuilt sensors. An increasing number of studies demonstrate the promise of behavior-monitoring like this to mental health (Lane et al., 2010).

There are two main methods for assessing behavior related to mental health:

1. Measurement of real-world behavior through sensors (most commonly using sensors embedded in smartphones).

2. Measurement of behavior mediated through technology (e.g., assessing mental health via social media behavior).

2.1. Real world behavior sensing

An increasingly common approach to measuring mental health is to use the sensors embedded in smartphones to measure human behavior. Commonly available smartphones are equipped with a range of sensors including GPS, accelerometers, gyroscope, light, temperature, and a microphone. Together or alone, these sensors can be used to measure a range of human behavior.

One of the first approaches was to use a similar range of sensors in concert to recognize physical activity and classify it into specific categories such as walking, running, climbing stairs, and so on, a feature that is now commonly available in every smartphone (Lester, Choudhury, & Borriello, 2006). In a study to detect social isolation in older adults, a similar approach to measuring physical activity was shown to significantly correlate with the Yale Physical Activity Survey (YPA), a survey instrument to assess physical activity (Rabbi, Ali, Choudhury, & Berke, 2011). In this same study, social activity was assessed using an on the body microphone. Again there was a strong correlation between the amount of human speech detected and the CES-D, a brief self-report scale to measure depression.

2.2. Technology-mediated behavior

A second area of focus in behavior sensing involves looking at technology-mediated behavior. This can either explore whether aspects of technology use correlate with behavior or mental states. For example, a recent study validated a lightweight algorithm in a student population for predicting sleep timing and duration that only assesses whether a person’s smartphone screen was on or off. In this 100-day study with nine participants, this algorithm achieved an accuracy of 23.8 min (CI +11 min; P <.05) compared to self-reports (Abdullah et al., 2014). These findings were replicated in a larger and more diverse sample of 400 participants (Cut-tone et al., 2016). A refined version of this algorithm that incorporated social media app use and Facebook posts among other variables improved accuracy and outperformed more complex algorithms based on multiple environmental factors such as light, movement, and sound as well as phone locking and charging events (Murnane, Abdullah, Matthews, Choudhury, & Gay, 2015).

An increasing amount of work focuses on analysis of online behavior to assess mental health. For example, de Choudhury, Gamon, Counts, & Horvitz, 2013 have found social media data can provide signals for detecting the onset of depression. Social media datasets and other digitized signals of behavior (such as bank and e-mail records) very often comprise significant personal datastores that span multiple years. These datasets offer considerable potential for retrospective analyses that explore linguistic content and temporal factors in order to characterize mental health.

3. THE AUTOMATIC INFERENCE OF MENTAL HEALTH

In the wider area of modeling human behavior from mobile sensing, there is often the need to bridge the gap between the physically sensed data streams and outcomes that are informative to health practitioners (e.g., mental health assessment scores). A common method used for this purpose is supervised machine learning (Lane et al., 2010). In essence, this approach takes a set of data points acquired from sensing (training set) with known associated outcomes (labels). These data are then used for learning based on a variety of optimization algorithms designed to minimize the error margins between model output and the labels. After the model outputs have been evaluated and reach an acceptable level of accuracy (in practice this is often viewed as levels significantly greater than chance), the model can then be used in practice to map new unseen sensed data to clinically meaningful scores. Although this can also be achieved using more traditional methods such as statistical modeling, it is often the case that with human behavior the necessary mappings are complex, high dimensional, and require flexible approaches (Breiman, 2000). Consequently, the use of machine learning has become increasingly popular in recent years.

That said, there remain two critical dependencies when applying supervised machine learning for mobile behavioral modeling. The first is the prior determination of input variables (features) that are genuinely descriptive of the outcome of interest. For example, one such feature could be the amount of time spent at home for someone with depression manifesting in social withdrawal; this value can readily be inferred from geo-locations traces (GPS) from the smartphone. The second is the truthfulness and reliability of the labels that are assigned to each of the instances in the training set. Only when both conditions are satisfactory can an effective model be generated. However, in mobile sensing there is currently no standard method to automatically engineer features from unprocessed sensor data. As such is it often the case that researchers need to rely on expert domain knowledge from clinical researchers and practitioners, intuition, or other related prior experimentation to successfully determine descriptive features. For some inference tasks this can easily be done when the outcome of interest has a clear causal relationship to the measured features (e.g., the inference of a walking motion using an accelerometer signal). But, in the case of outcomes with greater levels of complexity, abstraction, and subjectivity, such as a person’s mental health state, this feature-engineering task is considerably more challenging. However, recent research has begun to demonstrate that this too is feasible. In the following, we will highlight examples of inferences related to mental health were created using passively sensed smartphone data.

3.1. Inferring the Patient Health Questionnaire score for depression

Recent studies have demonstrated that depression can also be inferred using passively acquired smartphone data. These include Canzian and Musolesi (2015), where the authors use a wide range of descriptive features derived from GPS data alone. The features were designed to capture various characteristics within an individual’s geo-locational movement pattern. Distance, maximum distance traveled between two tracked points, number of different places visited, among other features, were shown to be descriptive of two stratified depression states defined by PHQ-8 (where PHQ is Patient Health Questionnaire) scores that were higher or lower than the study average. The authors demonstrate effective prediction using these features calculated over varying time. In a similar study by Saeb et al., 2015, the authors showed significant correlations exists between the PHQ-9 score and various functions of geo-locational movement and phone use. These include movement trace regularity within a 24-hr period that they call circa-dian movement, the normalized location entropy (uniformity of presence between routinely visited locations), and location variance (GPS mobility independent of location) as well as phone use duration and frequency. Then using a threshold logistical regression model that distinguished participants with depressive symptoms (PHQ-9 score ≥5) from those without (PHQ-9 score < 5), they achieved accuracies up to 86.5%.

3.2. Inferring the Social Rhythm Metric for bipolar disorder

The authors in Abdullah, Matthews et al. (2016) sought to infer the Social Rhythm Metric (SRM; Monk, Frank, Potts, & Kupfer, 2002) solely from passively acquired smartphone data. The SRM is a clinically validated self-report measure of stability and rhythmicity, used most commonly to assess the routines of individuals with bipolar disorder (BD). It is a patient-completed self-report scale completed as part of Interpersonal and Social Rhythm Therapy (Frank 2007), an empirically validated treatment for BD that has been an empirically validated treatment for BD and has been shown to be associated with more rapid remission of bipolar depression, longer symptom-free periods, and improved occupational functioning (Frank et al., 2005, 2008). The standard procedure to calculate this metric is a five-item self-report scale measuring five anchor events during the day: these being the time of: getting out of bed, first contact with another person, start of work or school, dinner and going to bed. The number of people interacted with at each event is also reported as well as overall valence of mood during each day. A resultant SRM score (0–7) is calculated, which quantifies the temporal regularity of these anchor events over a week.

In Abdullah, Matthews et al. (2016), the authors hypothesized that patterns in the digital footprints leveraged from the smartphone is another indicator of behavioral regularity. GPS-related geo-locational information, travel distance, number of conversations (using in-phone human voice detection, and nonstationary activity, inferred from accelerometer signals) were calculated per hour. The rationale being that movement and voice characterizes the behavioral patterns most relevant to the SRM survey. These inputs were then used as features to train a model on self-reported SRM scores used as labels. The model used in this study is the popular classifier support vector machine (SVM) that was used to discriminate between scores higher or lower than 3.5, which is deemed to represent stable rhythm or unstable social rhythms, respectively. The authors achieved a high predictive performance with a precision of 0.85 and recall of 0.86. This SVM classifier can then be deployed to new patients and give a high or low rhythm stability inference without the need for any self-reporting. Such unobtrusive methods that do not require the user to complete self-report surveys are temporally sustainable as long as the patient uses a smartphone.

3.3. Future directions in inference

We have highlighted studies that have principally focused on outward physical sensing streams to monitor behavior. However, further to this it is possible to leverage soft sensing that uses data from the interaction pattern between the user and the smartphone. We have mentioned above phone locked/unlocked state as well as measures on the types of apps being used. These simple and coarse examples of soft sensing haven proven to be very effective and has led a nascent research direction in this field. As we have mentioned, the phone lock state has been leveraged to infer sleep (Abdullah et al., 2014), but more recent advances on this (Abdullah, Murnane et al., 2016) also made use of various patterns within the lock/unlock states to infer the alertness level of users. This inference was accurate within 84 ms of the reaction times from the Psychomotor Vigilance Test (PVT). Patterns in the use of apps as well as logs from cellular communication and text messaging are also an area of increasing investigation with respect to psychological outcomes (Murnane et al., 2016). Specifically for mental health and wellness, groundwork is being made from the generation of tailored apps such as the IntelliCare suite for depression and anxiety (Mohr et al., 2017) from which the app-use pattern can be leveraged as features for passive inference.

All of the aforementioned methods have hitherto relied on the successful determination of descriptive features for each application. However, in machine learning research, methods within the area of deep learning has demonstrated a capacity to automatically learn descriptive features without the need for prior crafting. To date, this has mostly been demonstrated with large unimodal datasets in other domains that have reliable labels such as image classification. The use of deep learning methods is less well understood for multimodal data with abstract labels such as mobile sensing for mental health. That said, Martínez & Yannakakis (2014) proposed a Convolutional Neural Network based framework designed to fuse combinations of continuous and physiological signals to successfully infer six affective states (anxiety, frustration, fun, relaxation, challenge, and excitement). Such work shows the feasibility of using deep learning in mental health that could ultimately be more accurate than standard methods as they would algorithmically learn important patterns, including patterns that we are currently unaware of.

4. PERSONALIZED MENTAL HEALTH INTERVENTIONS

Generally, within mobile health systems (mHealth) the form and strategy for providing feedback to users or human experts can be broadly categorized into three main approaches: (1) the aggregation of collected data into summary statistics; this is often augmented with aesthetic and intuitive visualizations. For example, UbiFit (Consolvo et al., 2008) or BeWell (Lane et al., 2011) that uses background wallpaper to show aggregated physical activity, social interaction, and sleep. This strategy is often used for goal setting or priming toward a prescribed goal; (2) the display of a wide range of data for self-exploration and reflection, again often with use of visual aids; and (3) the direct issue of recommendations or prescribed instructions. Though these feedback strategies have been broadly implemented, it is notable that they are elementary and have a heavy reliance on users’ willingness to take the right post-feedback actions, not to mention to possess the required skills to understand the feedback. For example, for strategy (3), most recommendations used in mHealth systems are generic or only tailored to suit a specific demographic or clinical population as a whole. Also, none of these methods make any in-depth analysis to fully exploit the nuances and complexities within person-specific data to improve the impact and persuasiveness of the feedback. In the following, we discuss some recent experiments that have explored further strategies that try to address this and take advantage of the increasing ability to infer user context and state. The following studies have shown early indications of greater persuasiveness when compared to control conditions.

4.1. Leveraging preferences to generate more persuasive feedback

The use of logged data from an individual’s past actions to hone future feedback in non-mobile systems is not new. This is widely used in a variety of domains, for example, the interfaces of media streaming services such as NetFlix display films that are similar to those that a subscriber has previously watched or targeted advertising based on a person’s past purchase or browsing history. That said, there has been little use of such methods with tracked mobile data for behavioral feedback. To our knowledge, the first notable case of this is Rabbi, Aung, Zhang, & Choudhury, (2015), where the authors proposed MyBehavior: a historical data-driven recommendation engine to promote more exercise and lower calorie food intake. This system uses both active and passive data from the smartphone to characterize a user’s habits and choices. Levels of physical activity can be passively inferred using an in-phone mobility state classifier (e.g., walking or stationary). Also the user can manually log instances of significant exercise and the durations as well as instances of food intake. The overarching objective of this system is to persuade users to adopt a healthier lifestyle that is quantified by calorie loss. The system is underpinned by an online machine learning algorithm that seeks to rank more highly the food and exercise suggestions that are both healthy (maximizing calorie loss) but are also similar to actions that have been logged in the past. This is based on behavior theory in that new suggestions that are perceived to be familiar are also perceived to be low-effort which in turn has a greater chance of actualization. Results over 10-week user trials showed significant and sustained increases in the adoption of the habit-driven suggestions when compared to randomly generated suggestions with no basis on an individual’s past behavior.

4.2. Subliminal feedback

So far we have discussed strategies that explicitly presents feedback to the user. However, this approach ultimately relies on conscious decision points leading to the adherence of feedback or lack thereof. However, recently there has been an interesting line of research in the field of mobile technology that attempts to use subtle forms of feedback to elicit responses upon which the user does not have a full awareness of sometimes termed as nudging (Kahneman, 2011). The authors of the position paper demonstrate this in a ubiquitous computing setting which they call mindless computing, described as “a mobile or ubiquitous, persuasive technology designed to subtly influence the behavior of the user without requiring their conscious awareness” (Adams, Costa, Jung, & Choudhury, 2015). These authors demonstrate two proofs of concept studies that exploit subtle feedback in two sensory modes: audio and visual. The first study showed that people can be influenced to speak in a lowered pitch by hearing themselves speak in real time but fed back with a lower pitch than their natural pitch. In the second study, the authors exploit the Delboeuf optical illusion and created plates that were lit underneath with an array of LEDs to change color. This was done to set different levels of contrast with various colored food placed upon it in a concentric shape. Results showed that the amount of food placed on a highly contrasting plate was perceived to be less than when placed on a low contrast plate, even though the actual amount of food remained the same. These are two simple yet effective examples of real time sensing and feedback to influence perception and behavior in a subconscious way. A later study in wearable technology (Costa, Adams, Jung, Guimbetière, & Choudhury, 2016) again demonstrated this mindless computing principle but using deliberate false feedback. Here, the authors create a vibrating wrist worn device from which the user believes the rate of vibration is a direct mirror of their own heart rate. It was found that providing deliberate low frequency pulse rates levels resulted in significantly lower levels anxiety change when compared to control conditions in which (1) the participants did not associate the slow vibration with their own heart rate or (2) the vibration was not false and did reflect their true heat rate.

5. FRAMEWORK FOR INTEGRATING BEHAVIORAL SENSING INTO MENTAL HEALTH

We have described approaches used in research today to collect, analyze, and act upon data collected passively through the smartphone. Although these processes are common across many areas in health-care, and beyond, we have demonstrated how they are having an impact in mental health and promise to change how we understand and treat mental illness in the future. To conclude, we present a three-part framework (see Fig. 1) that can be used to employ behavioral sensing in mental health.

FIGURE 1.

Framework for using behavioral sensing in mental health

First, is the measurement stage. It is recommended to collect a broad range of behavioral data through the smartphone over an extended time. Generally, the range of measures collected through the smartphone is relatively consistent regardless of condition. This is in part because much of current research is exploring the rela tionships between these new signals in order to identify new digital biomarkers. Hence, it makes sense to cast a wide net. Another reason for this is because many of these data streams underpin behaviors that are known to be relevant across mental illnesses (i.e., accelerometer data as a measure of physical activity). Machine learning techniques can be used to find novel connections or structures in the data.

Second, the raw data collected at the measurement stage is processed to create new features with a goal to create clinically meaningful inferences. This process can be guided by current clinical thresholds (e.g., for depression), observations from clinical practice, or novel features (such as various mobility measures made possible by GPS). The output of this phase can be the identification of novel digital biomarkers—behavioral patterns derived from mobile sensing—that are significant of or predictive of various mental states of interest. For example, various features collected via the smartphone such as a decrease in locations visited might be strongly related to psychomotor retardation. Furthermore, this phase can generate models that can then be used on their own as a real-time monitor of mental health.

The final phase focuses on how to integrate these inferences into the management of the target condition. This could range from using the inferences alongside a traditional clinical intervention such as psychotherapy or pharmacotherapy to create highly personalized digital interventions such as psychoeducation modules or chrono-therapeutic interventions like light therapy. If at the measurement phase, commonly used measures of mental health such as the PHQ are used, then it may be possible to create models that effectively can translate inferences into outputs that practitioners are already familiar with (i.e., PHQ scores). The identification of biomarkers for depression could help personalize clinical decisions identifying treatments that have a greater chance of success for each individual and then automatically monitoring individual progress when on that prescribed treatment.

6. CONCLUSION

Although there exists no objective test for mental illness, the paper and pencil tools currently used to assess mental health have known and considerable weaknesses. In addition, most existing online treatments are based on prior clinical practice with a heavy emphasis on non-dynamic content and general behavioral suggestions. Yet, despite a significant increase in the availability and use of smartphones and online technology, clinical mental health has made very little use of the clear potential to provide personalized assessment and interventions for mental health through these media. There is a considerable opportunity to harness the findings from recent innovations to transform the measurement, modeling, and treatment of mental health. This is, of course, easier said than done. There are considerable challenges such as the need to protect patient privacy and the need for new scalable inference methods that will make it easier to make more rapid progress and broadly validate these approaches. However, through a combination of moment-to-moment granular behavioral sensing, novel machine learning methods to create personal models and novel intervention strategies that harness these personalized models, there is now the potential to radically transform both our understanding of mental health and our ability to personalize the treatment of mental illness to each person in order to increase positive outcomes.