Systematic review of smartphone-based passive sensing for health and wellbeing

Abstract

Objective

To review published empirical literature on the use of smartphone-based passive sensing for health and wellbeing.

Material and Methods

A systematic review of the English language literature was performed following PRISMA guidelines. Papers indexed in computing, technology, and medical databases were included if they were empirical, focused on health and/or wellbeing, involved the collection of data via smartphones, and described the utilized technology as passive or requiring minimal user interaction.

Results

Thirty-five papers were included in the review. Studies were performed around the world, with samples of up to 171 (median n=15) representing individuals with bipolar disorder, schizophrenia, depression, older adults, and the general population. The majority of studies used Android operating system and an array of smartphone sensors, most frequently capturing accelerometry, location, audio, and usage data. Captured data were usually sent to a remote server for processing but were shared with participants in only 40% of studies. Reported benefits of passive sensing included accurately detecting changes in status, behavior change through feedback, and increased accountability in participants. Studies reported facing technical, methodological, and privacy challenges.

Discussion

Studies in the nascent area of smartphone-based passive sensing for health and wellbeing demonstrate promise and invite continued research and investment. Existing studies suffer from weaknesses in research design, lack of feedback and clinical integration, and inadequate attention to privacy issues. Key recommendations relate to develop passive sensing strategies matching the problem at hand, using personalized interventions, and addressing methodological and privacy challenges.

Conclusion

As evolving passive sensing technology presents new possibilities for health and wellbeing, additional research must address methodological, clinical integration, and privacy issues. Doing so depends on interdisciplinary collaboration between informatics and clinical experts.

Graphical abstract

1. INTRODUCTION

Patients’ disease management and preventive health behaviors benefit from the collection and tracking of health-related data, from daily weights to calorie counts to pain scores [1, 2]. Clinicians, too, are increasingly interested in capturing patient-reported outcomes including current status, symptoms and adverse events such as falls [3]. Patient, clinician, and collaborative use of data to make decisions is the hallmark of an emerging era of personal or precision medicine, ushered in by decades of advocacy [4] and a recent $215 million US investment in precision medicine funding [5].

These trends are accompanied by the proliferation of personal health information systems such as personal health records (PHR) systems [2], wearable consumer devices (e.g., activity trackers [6]), and smartphone applications, which aid in capturing, storing, managing, transmitting, interpreting, and acting on large volumes of patient data [7].

The 1998 American College of Medical Informatics (ACMI) Summit presciently identified wearable computing systems as a way to achieve the “audacious goal” of empowering individuals via biomedical informatics [8]. Wearable, portable, or mobile computing permits continual passive sensing: the capture of data about a person without extra effort on their part. The concept of passive sensing comes from extensive research conducted in the field of ubiquitous computing, where it is also called ‘context-aware computing’ [9]. Two main advantages of passive sensing over traditional data collection methods are that it is less intrusive and enables just-in-time adaptive interventions based on data captured and processed in situ [10]. Passive sensing for health and wellbeing refers to various methods to collect data from patients or lay users in situ without requiring their direct interaction with any artifact or person (see Appendix A1 for definition of this and related terms). Users may be able to turn sensing on and off, but need not make any input to produce data collection. The combined unobtrusiveness and pervasiveness of passive sensing makes it possible to gather data at any time, longitudinally, and with little stigma or additional burden on patients’ awareness, memory, or behavior. Such benefits are especially useful in the domains of mental health and mental illness, including dementia, schizophrenia, and mood disorders, where data may be sensitive, stigmatized, and subject to distortion. Indeed, passive sensing has been argued by mental health researchers as a promising component in ambulatory assessment [11].

Passive sensing is not new but the related technology has evolved: for instance, physical activity, sleep, and cardiovascular disease research has employed passive sensing for decades, using an evolving suite of technologies from pedometers, polysomnography, and cardiovascular implantable electronic devices to commercial wristband activity trackers, smartwatches, and smartphones [12–15]. Mobile health technologies that can passively collect information have been promoted in the medical literature as a way to reduce burden and improve care for healthcare consumers [16].

Smartphones, in particular, are a novel technology for passive sensing described in the literature but not systematically reviewed [17, 18]. Smartphones are unique because of their increasing computational power and pervasiveness. As of 2015, 68% of US adults owned smartphones, approaching the rate of desktop or laptop computer ownership (73%) [19]. Even among older adults, smartphone ownership has doubled from 18% to 42% between 2013 and 2016 [20]. Smartphones are used for various activities, including for health-related purposes, by the majority of owners across all age groups [21]. Because a smartphone is ubiquitous in the daily life of so many in the US and globally, sensing via smartphone may be less obtrusive—though perhaps no less intrusive—than specialized wearable medical or fitness devices.

Smartphones are of further interest for passive sensing because they combine multiple sensors (Apple’s iPhone 7 has six [22], while the Samsung Galaxy S8 has eleven [23]). They also capture behavioral data such as call, texting, or social media activity; have advanced Internet, storage, and processing capabilities; and permit the creation of personal profiles and personalized, just-in-time visualizations and alerts to users and their support network [24]. Smartphones can be used to passively capture data such as speech characteristics, location, and activity, which can be interpreted to assess depression, sleep, or loneliness. These smartphone sensors have been used in multiple commercial applications, ranging from car navigation to fitness tracking applications (see Appendix A2 for a fuller list of smartphone sensors and examples of related commercial applications).

Although several reviews have examined the use of portable activity sensing devices [6] and the use of smartphones generally for health and wellbeing [25–27], to our knowledge the growing body of studies of smartphone-based passive sensing has not been systematically reviewed. The goal of this study was to address this gap in the biomedical informatics literature.

2. OBJECTIVES

The main study objective was to review published literature on smartphone-based passive sensing for health and wellbeing. Specific research questions were:

• To which health-related domains and populations has passive sensing via smartphone been applied?

• What data collection approaches have been used for passive sensing via smartphones?

• How were sensed data processed and used after acquisition?

• What are the benefits of passive sensing via smartphone?

• What are the challenges, such as privacy issues, of passive sensing via smartphones?

3. METHODS

We followed Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [28] to perform a systematic review of the literature on smartphone-based passive sensing for health and wellbeing.

3.1 Type of Studies

Studies were included if they: 1) were empirical; 2) primarily focused on health and/or wellbeing of participants; 3) involved the collection of data via smartphones; and 4) described the utilized technology as passive or requiring minimal user interaction.

We included health-related studies of people with or without diseases. “Smartphone” was defined as any phone equipped with a mobile operating system—Android, Apple iOS, Symbian OS, Windows Mobile—on which applications can be installed to capture data from the phone’s sensors. Passive was defined as data being collected without user input beyond starting the application, apart from any data actively collected by the study for validation purposes.

Studies were excluded if they used wearable devices paired with a phone because these did not use the smartphone’s sensors. Studies that required participants to attach the smartphone to their body, clothing, or a permanent fixture (e.g., furniture) were also excluded because they did not use the device’s primary telecommunication, display, or input functions; for example, most gait-tracking applications were excluded as they often used the phone as a pure sensor device affixed to the waistline.

We included English-language studies published any time through January 2017, the last month studied. Peer-reviewed journal papers and conference proceedings papers were included; extended abstracts were excluded.

3.2 Search Strategy for the Identification of Studies

We performed two searches in domain-specific databases representing computing and technology (ACM) and medicine (MEDLINE), followed by cross-domain database searches in Web of Science. This was followed by a cited reference search, whose findings were duplicated in the database search. Queries were tailored to each database (Table 1).

Table 1.

Queries performed in four research databases, results returned, and papers retained.

Database	Query	Results returned	Unique papers retained
ACM Digital Library - Association for Computing Machinery	+(health* wellbeing medicine hospital clinic nursing) +(mobile smartphone iphone android) +(detect* sensing sensor GPS Accelerometer microphone “global positioning system”)	1008	11
MEDLINE (PubMed)	(detector OR detection OR sensing OR sensor OR GPS OR Accelerometer OR microphone OR “global positioning system”) and (smartphone or “Mobile phone” or iphone OR android OR “mobile sensor”)	1366	14
Web of Science	((detector OR detection OR sensing OR sensor OR GPS OR Accelerometer OR microphone OR “global positioning system”) and (smartphone or “Mobile phone” or iphone OR android OR “mobile sensor”) AND (health\* OR wellbeing OR medicine OR hospital OR clinic OR nursing))	1318	10

4. RESULTS

We included in the full review a total of 35 publications [29–63], summarized in Tables 2–5. These were selected from 3,246 returned results (Figure 1), with the majority of references discarded for irrelevance (e.g., chemistry research), absence of sensor data (e.g., proof of concept papers), and use of wearable devices. Several studies were excluded because they collected data only under controlled laboratory conditions, for example, requiring participants to sit and stand repeatedly to test a motion sensor.

Table 2.

Summary of mental health studies, ordered by condition then year of publication.

Condition	Author (Year) Location	Study Purpose	Principal Findings	Sensors Used	Sample Size & Type	Study Length (Days)
Bipolar disorder	Osmani et al. (2013) [53] Austria	Correlate physical activity with symptoms of bipolar disorder	Significant correlations between activity levels and bipolar states for some individual patients but not for others.	• Accelerometer	5 Patients with bipolar disorder	90
Bipolar disorder	Grünerbl et al. (2014) [45] Austria	Detect state and state change for people with bipolar disorder	Detection of state change with 96% precision and 94% recall; recognition of state with 80% accuracy.	• Accelerometer • GPS	12 Patients with bipolar disorder	84
Bipolar disorder	Grünerbl et al. (2015) [44] Austria	Detect state and state change for people with bipolar disorder	Detection of state change with 97% precision and 97% recall; recognition of state with a 76% accuracy.	• Accelerometer • Call logs • GPS • Microphone	10 Patients with bipolar disorder	84
Bipolar disorder	Abdullah et al. (2016) [29] USA	Predict scores on the social rhythm metric (SRM) scale among people with bipolar disorder using generalized and personalized models	Prediction of states with 85% precision and 86% recall. Social rhythm metric score inferred with 0.92 root-mean-square error for personalized models and 1.40 for the generalized model.	• Accelerometer • Call logs • Light sensor • SMS patterns	7 Patients with bipolar disorder	28
Bipolar disorder	Beiwinkel et al. (2016) [34] Germany	Detect features to be used for the monitoring of bipolar disorder	Significant correlations between subset smartphone sensor data on one hand and depressive and manic symptoms on the other, but none above clinical thresholds.	• Accelerometer • Antenna • Call logs • Device activity • GPS • SMS patterns	13 Patients with bipolar disorder	356
Depression	Burns et al. (2011) [38] USA	Reduce depressive symptoms among people with major depressive disorder	Prediction of depression from sensor data no better than chance.	• Accelerometer • Bluetooth • GPS • Light sensor	8 People with major depressive disorder	56
Depression	Canzian et al. (2015) [39] UK	Predict depressive symptoms from location data	• Prediction of depression with > 75% sensitivity and specificity, using a support vector machine classifier on a personalized model with a time span of 8 days or more. • Prediction of depression with > 60% sensitivity and specificity, using a support vector machine classifier on a generalized model with a time span of 8 days or more.	• Antenna • GPS	28 General sample	71*
Depression	Saeb et al. (2015) [57] USA	Predict depressive symptoms from location and phone usage data	Prediction of depression with 86% accuracy for the best feature, using a logistic regression classifier.	• Device activity • GPS	28 General sample	14
Depression	Saeb et al. (2016) [56] USA	Correlate location data with depression symptoms	• Significant negative correlations between GPS features (location variance, entropy, circadian movement) and depression. • Relation between GPS features and depression more evident on weekends, when participants are not constrained by work or school schedule.	• GPS	48 University students	70
Depression	Wahle et al. (2016) [61] Switzerland	Predict depression from smartphone data, using an application delivering context- sensitive cognitive behavioral therapy-based micro-interventions	• Prediction of depression with 61% accuracy using a support vector machine classifier. • Prediction of depression with 59% accuracy using a random forest classifier.	• Accelerometer • Calendar • Call logs • Device activity • GPS • SMS patterns	36 General sample	> 14
Schizophrenia	Ben-Zeev et al. (2016) [36] USA	Examine the feasibility and acceptance of passive sensing among people with schizophrenia	People with schizophrenia open to sensing, a third expressed concern about privacy, two-thirds expressed interest in receiving feedback.	• Accelerometer • Bluetooth • GP S • Light sensor • Microphone	20 Inpatients and outpatients with schizophrenia	10.15*
Schizophrenia	Difrancesco et al. (2016) [41] UK	Detect out-of-home activities among people with schizophrenia in order to infer social functioning	Detection of out-of-home activity with precision between 72% and 95%, and recall between 69% and 77% for the best method.	• GPS	5 Patients with schizophrenia	5
Schizophrenia	Wang et al. (2016) [62] USA	Correlate smartphone data with schizophrenia	Significant correlation between ground truth and predicted mental health status scores, using random forest regression.	• Accelerometer • Application usage • Call logs • Device activity • GPS • Light sensor • Microphone • SMS patterns	21 Patients with schizophrenia	133.76*
General mental health	Ma et al. (2014) [49] China	Predict mood from smartphone data	Prediction of mood with an accuracy of 70% for the model comprised of sensor and social features, using Markov-Chain Monte Carlo methods	• Accelerometer • Activity • Call logs • SMS patterns	15 University students and non-students	30
General mental health	Wang et al. (2014) [63] USA	Correlate smartphone data with depressive symptoms among college students	• Significant negative correlation between sleep and depression. • Significant negative correlations between conversation frequency and duration and depression.	• Accelerometer • Application usage • Bluetooth • Call logs • GPS • Light sensor • Microphone • SMS patterns	48 University students	70
General mental health	Ben-Zeev et al. (2015) [35] USA	Evaluate the prediction of daily stress levels, mental health status from smartphone data	• Sleep duration and mobility associated with daily stress levels. • Speech duration, geospatial activity, sleep duration, kinesthetic activity associated with mental health status.	• Accelerometer • Device activity • GPS • Light sensor • Microphone	47 University students	70
General mental health	Asselbergs et al. (2016) [32] Netherlands	Predict mood from smartphone data	Prediction of 55% to 76% of mood scores using personalized linear regression.	• Accelerometer • Application usage • Call logs • Device activity • SMS patterns	27 University students	35.5*
General mental health	Huang et al. (2016) [46] USA	Correlate places visited by university students with their social anxiety	Significant negative correlation between time spent at religious locations and reported social anxiety	• GPS	16 University students	10

GPS: Global Positioning System; SMS: Short Message Service;

^*average duration of subject participation; precision refers to positive predictive value; recall refers to sensitivity, or hit rate. Patients: participants receiving professional care.

Table 5.

Summary of studies in other domains, ordered by condition then year of publication.

Condition	Author (Year)	Study Purpose	Principal Findings	Sensors Used		Sample Size & Type	Study Length (Days)
Addiction	Lee et al. (2014) [48] Korea	Correlate application use with smartphone addiction	Significant positive correlation between smartphone daily use time and the Korean smartphone addiction scale.	• Application usage • GPS		14 General sample	> 7
Addiction	Naughton et al. (2016) [52] UK	Evaluate a just-in-time intervention using location data to send timely messages to smokers	Feasible but some non-compliance in reporting smoking.	• GPS		13 Smokers	34*
Chronic heart failure	Aranki et al. (2016) [31] USA	Sense physical activity among people with chronic heart failure for transmission to doctors	Feasible despite technological and usability challenges.	• Accelerometer • Call logs • GPS • Proximity sensor		15 People with chronic heart failure	< 90
Geriatrics	Vathsangam et al. (2014) [60] USA	Evaluate the detection of physical activity from accelerometer data in order to encourage older adults to exercise	Participants appreciated the utility of the application but would like more feedback	• Accelerometer		8 Older adults	21
Geriatrics	Sanchez et al. (2015) [58] Mexico	Predict loneliness in older adults to send them positive messages	Correct classification of family loneliness and spousal loneliness for > 80% of participants, with the average time spent out of home and total of times out of home found to be the most important attributes	• Call logs • GPS • SMS patterns		12 Older adults	7
Stress	Stutz et al. (2015) [59] Austria	Correlate smartphone data with stress	Significant correlations between perceived stress scores (daily and weekly averages) and the sensed features, with noisiness (positive correlation), number of time device is powered on (positive), and changes in light (positive) among the most significant features		• Accelerometer • Application usage • Call logs • Device activity • Light sensor • Microphone • SMS patterns	15 University students	14
Stress	Garcia-Ceja et al. (2016) [43] Italy	Detect and predict stress from accelerometer data	• Prediction of stress with 95% accuracy using the best similar-user model (decision tree) • Prediction of stress with 95% accuracy using the best similar-user models (decision tree and Naïve Bayes). • Prediction of stress with 87% accuracy using the best general model (decision tree).		• Accelerometer	30 Company employees	40

GPS: Global Positioning System; SMS: Short Message Service;

^*average enrollment length among participants, Older adults: people 60 years old or older.

Figure 1.

PRISMA Diagram of the Literature Review Process

Seventeen studies (49%) were performed by US research teams and 14 (40%) by Europeans. Other studies originated in China [33, 49], Korea [48], and Mexico [58].

Mental health was the most common application domain for studies using passive sensing on smartphones, with 18 (51%) studies on mental health: five (14%) on bipolar disorder; five (14%) on depression; and three (9%) on schizophrenia. Other domains included sleep (6; 17%) and general health (4; 11%) (see Figure 2).

Figure 2.

Domains of the reviewed papers.

Seven studies integrated passive sensing in behavior change interventions [38, 52, 54, 55, 58, 60, 61], such as personalized feedback to promote exercise and healthy eating [55]. Other studies used passive sensing to demonstrate the ability to capture or monitor data related to health and wellbeing.

Study sample sizes ranged from 5 to 171, with a mean of 23.1±27.9 participants and a median of 15. Three studies had open enrollment, meaning that participants downloaded an application from an application portal (e.g., Apple AppStore, Google Play Store) [39, 47, 61]; these studies were characterized by high dropout rates. Twenty-four studies reported a fixed study length, ranging from five days to a year, with a mean of 53.5±71 days and a median of 30 [29, 32–35, 37, 38, 40, 41, 43–46, 49, 50, 53–60, 63]. Eleven others reported variable between-subject study durations [30–32, 39, 42, 47, 48, 51, 52, 61, 62], citing reasons such as rolling enrollment, participant dropout, and having no defined study length.

Nine studies included participants with a clinically-diagnosed mental health condition [29, 34, 36, 38, 41, 44, 45, 53, 62], two studied adults over 60 years old [58, 60], one enrolled people with chronic heart failure [31], and one studied smokers [52]. Nine studies enrolled university students [30, 32, 35, 40, 42, 46, 56, 59, 63] and another three recruited participants on university campuses [49, 54, 55]. Other studies included participants from various backgrounds [37, 39, 43, 47, 48, 50, 51, 57, 61].

Thirty (86%) of the reviewed studies were conducted between 2014 and January 2017 (cf. Figure 3). During each of these three years, mental health studies made up more than 40% of the publications.

Figure 3.

Reviewed papers by year of publication (Note: January 2017 is merged with 2016).

4.2 Sensors Used

As seen in Table 6, studies captured data from a variety of smartphone physical sensors and device analytics. The most used physical sensors were the accelerometer (25 studies), Global Positioning System sensor (GPS; 22 studies), light sensor (10 studies), and microphone (9 studies). Studies also collected data on device analytics, including call logs (14 studies), device activity (defined as screen on/off and device on/off; 11 studies), and Short Message Service (SMS) patterns (frequency and/or recipients; 11 studies).

Table 6.

Sensors used in reviewed studies.

Physical Sensor	Papers	Device Analytics	Papers
Accelerometer	[29, 31–38, 40, 43–45, 47, 49–51, 53–55, 59–63]	Call logs	[29–34, 42, 44, 49, 58, 59, 61–63]
GPS	[31, 33–36, 38, 39, 41, 42, 44–46, 48, 52, 54–58, 61–63]	Device activity	[30–32, 34, 35, 40, 50, 57, 59, 61, 62]
Light sensor	[29, 33, 35, 36, 38, 40, 50, 59, 62, 63]	SMS patterns	[29, 30, 32–34, 49, 58, 59, 61–63]
Microphone	[33, 35, 36, 40, 44, 50, 59, 62, 63]	Application usage	[30, 32, 42, 48, 50, 59, 62, 63]
Bluetooth	[36, 38, 42, 63]	Browser history	[30]
Antenna	[34, 39]	Calendar	[61]
Proximity sensor	[31, 50]

Most studies combined multiple sensors, an emerging strategy as phones have become more energy efficient and the overhead of capturing data has diminished. Eleven studies recorded input from five or more sensors [30, 32–36, 50, 59, 61–63], among which seven were mental health studies. Studies with more than three sensors usually relied on machine learning prediction models to process and interpret data; for example, one study combined accelerometer as a proxy of physical activity and sleep, microphone as a proxy of social activity, and GPS for location changes to infer daily stress levels [35]. Ten studies recorded data from only one sensor, either the accelerometer or GPS [37, 41, 43, 46, 47, 51–53, 56, 60].

4.3 Operating systems

Thirty-one studies (89%) used the Android operating system (OS), compared to two using Apple iOS [37, 51], and one using the now-defunct Symbian OS [38]. This could be explained by the access granted on Android phones, making it easier for data capture, communication, and processing tasks to run in the background. In contrast, Apple’s iOS made it harder for applications to access data from other applications without explicit user permission. The operating system could not be ascertained for one study [46].

4.4 Validation Measures

To validate the interpretation of sensed data, studies employed various traditional measures or other assessments of “ground truth,” hereafter referred to as validation measures. Most studies then reported the correlation between validation measures and the interpretation derived from processing sensor data. Studies of depression used the PHQ-8 or PHQ-9 self-report instruments. Studies of bipolar disorder primarily used clinician assessments based on a battery of scales [34, 44, 45, 53], although one used a self-report questionnaire [29]. For sleep studies, smartphone sensor-based results were compared to those from a medical activity tracker [51], a popular consumer activity tracker [40], laboratory-based polysomnography [37], and self-report questionnaires or sleep diaries [30, 33, 50]. Other studies used instruments relevant to their application domain, including questionnaires, ecological momentary assessment (EMA), and professional assessments (e.g., for bipolar disorder [44, 45, 53]). Studies differed in the timing of validation measures, from one-time measures to seven measures per day (e.g., [59]) or pre-post assessments.

4.5 Data Processing and Use

The software application used in most studies (21; 60%) communicated with a remote server to save sensed data to a database for processing and, at times, within-study feedback to participants. In eight studies, data were scrambled for privacy on the phone (via hashing or anonymization of audio data) before being transmitted to the server [29, 30, 34–36, 57, 62, 63].

Server communication was not used in 10 studies (29%) [35, 37, 44, 45, 51, 53–56, 60]. Five studies produced feedback locally [37, 47, 54, 55, 60], without any server communication; for example, health status was processed directly on the phone in one study on predicting health status from accelerometry [47]. Three studies performed complex calculations—data classification or prediction modeling—directly on the smartphone [37, 54, 55]; for example, sensed geographical locations were processed on the device to cluster physical activities [54, 55]. In four studies (11%) describing post-study processing, we could not determine whether a remote server was used [30, 40, 43, 61].

Feedback to Participants

Fourteen studies (40%) reported providing some sort of feedback to study participants [29, 31, 33, 37, 38, 40, 47–49, 52, 54, 55, 58, 61]. The applications in five studies displayed graphs representing mental health status [29, 38], sleep data [37], physical activity [47], and the mobile applications participants used the most [48]. Two studies provided prepared motivational messages to participants based on collected data [31, 58] and three displayed tailored messages [52, 54, 55], e.g., “25% of the time you smoke [is when] you are working” [52]. Three studies showed participants text descriptions of their sensed data and/or sensor-predicted status, without encouraging behavior change [33, 40, 49]. As an example of presenting both data and data-driven interventions, one study displayed depression data as text and delivered micro cognitive behavioral therapy modules based on the data [61]. A study published in 2011 only provided a text string depicting predicted depression status on the smartphone, with more detailed graphical feedback available on a companion website [38]. Two studies allowed clinicians to view their patients’ data through a separate web portal [31, 48]. Five studies computed the data locally [37, 47, 54, 55, 60] and provided feedback on the phone, whereas the rest required server communication to provide feedback to participants.

Correlation with Validation Measures

In the vast majority of studies, data were processed and correlated to validation measures, to test the validity of interpretations or predictions made through passive sensing. In seven studies, the correlation was performed while the study was ongoing [31, 37, 49, 54, 55, 60, 61] and after study completion in 23 studies. Data processing used different families of algorithms for interpreting or predicting the participant’s status. The most popular were Support Vector Machine [29, 31, 39, 47, 58, 61], naïve Bayes classifiers [43–45, 58], decision trees [38, 43, 50, 62], random forests [59, 61], and linear regression [30, 46, 57, 59]. Other prediction methods include Bayesian networks [50] and logistic regression [57]. Five studies compared several machine learning methods to predict participant status [43, 50, 58, 59, 61]. Some studies just performed correlation analyses without prediction of the participant’s status, i.e. they did not establish a mathematical relationship between the sensor data and the validation measures [e.g., 39, 48, 53, 56, 63].

4.6 Benefits of passive sensing and related findings

Nearly all studies demonstrated or otherwise reported benefits of passive sensing using smartphones. In mental health studies, findings included significant correlations with validation measures and successful prediction models for some or all the studied variables [29, 34, 44, 45, 53, 56, 57, 61–63]. For example, two bipolar disorder studies reported precision and recall (or hit rate) over 94% for bipolar state change detection [44, 45], and one study predicted bipolar states with precision and recall over 85% [29]. Sleep studies reported sufficient precision, defined as the detection of sleep duration within a one-hour margin [30, 40]. These results illustrate smartphone capability to deliver usable information that can be integrated into behavior change interventions for health and wellbeing.

Seven studies demonstrated individualized or similar-user models as better for predicting participant status compared to generalized models [39, 43–45, 54, 55, 61]. Two other studies argued for using personal models on the basis that the relationship between sensed data and behavior is individual-specific [35, 49].

Six studies conducted interviews or usability testing with their participants [36, 38, 40, 52, 55, 60]. Participants appreciated the ease of use of the system [36, 60] and that it did not interfere with their everyday life [36, 40]. Participants valued receiving feedback [38, 52, 60] as long as it was understandable [i.e., reported in a way target users could understand;40, 60], timely [52], and relevant to their lifestyle [55].

Studies also highlighted the objectivity of smartphone sensor measurements [31, 34, 36, 39, 41, 42, 44, 45, 49, 53], the ability to take frequent measurements [29, 34, 37, 38, 41, 55, 57], the possibility of performing just-in-time and adaptive interventions [52, 55, 61], and reduced burden for patients [29–31, 35, 53]. Authors also mentioned the ubiquity of smartphones, the affordability of the interventions, and non-invasiveness.

4.7 Challenges of passive sensing

The apparent ease of deploying passive sensing campaigns for health and wellbeing was counterbalanced by several reported challenges. Although not systematically reported across studies, these challenges could be divided into three categories: technological, methodological, and privacy issues.

Technological challenges

In two studies, authors reported battery drainage concerns [31, 38]. Five studies mentioned the lack of sensor precision [38, 40, 41, 52, 60]; for example, location data were sometimes inaccurate, leading to participant frustration [52]. Three studies reported not being able to access application data that would have been useful in their prediction model [42, 48, 49].

Methodological challenges

Eleven studies noted concerns about generalizability due to low sample size [44, 45, 56–59], possible sample bias [32, 35, 46, 48], and variability in the study data sample [34, 35]. Seven studies reported a limited or null relationship between passively sensed data and validation measures [34, 38, 42, 46, 49, 50, 61]. Problems encountered include low variability of symptoms in the sample [34, 38] (e.g., few manic episodes occurring among bipolar participants during the study period [34]), noisy sensor data [38], technical problems leading to unusable data [38, 42], trying to predict personal phenomena with generalized models (e.g., for mood [49]), difficulty assessing “ground truth” [50], and biased samples [46]. Some studies called for more data labeling from participants, for example by having participants answer more frequent depression questionnaires [38, 56], to better train the prediction models. Studies also reported participants disabling the phone’s sensing capabilities [53] and not carrying their phones [36, 41, 53].

Privacy issues

Privacy issues were mentioned in 20 papers. Most papers did not thoroughly discuss privacy issues, but merely described their methods for protecting data privacy, which included the following:

• secure communication with external servers [34–36, 38, 39, 57, 62, 63],

• anonymization of data [30, 34, 44, 45, 57, 59, 62, 63],

• scrambling audio [29, 35, 36, 44],

• local storage/processing of data as opposed to sending data to an outside server [44, 45, 54].

In one instance, study participants mentioned that they would not grant access to as much information if the passive sensing application were a commercial product rather than coming from a university [52].

Fifteen studies made no explicit mention of privacy or a plan for privacy protection [33, 37, 41, 43, 46–49, 51, 53, 55, 56, 58, 60, 61].

5. DISCUSSION

The reviewed studies illustrate the potential of passive sensing via smartphones in the domain of health and wellbeing. Indeed, this review reveals the broad use of smartphone-based passive sensing across application domains, with a particular representation of mental health and sleep, two areas where passive sensing may be useful as a way to replace or supplement self-report. A number of passive sensing strategies for data collection, processing, and use were demonstrated, offering informaticians and healthcare researchers several options for future passive sensing projects, including interesting emerging methods such as machine learning or just-in-time processing and feedback. The reviewed studies generally demonstrated feasibility and validity of smartphone-based passive sensing, the latter evidenced by significant associations between traditional and sensing-based assessments. Studies also concluded that passive sensing was more accurate and less intrusive compared to self-report measures. However, additional work remains in several areas, including evaluating the health benefits of interventions using smartphone-based passive sensing, integrating passive sensing in clinical care programs, and addressing important implementation issues such as privacy and technology acceptance.

Using mobile phones for passive sensing is encouraging not only because of the potential power of continual monitoring and feedback of health-related data but also because of the non-intrusiveness of passive sensing. A smartphone-based passive sensing approach for health and wellbeing is well aligned with the concept of minimally disruptive medicine, defined as “a patient-centered approach to care that focuses on achieving patient goals for life and health while imposing the smallest possible treatment burden on patients’ lives” [64–66]. Passive sensing can ease—or, minimally, not add to—“work that is delegated to patients and their families” [67], by facilitating or automating difficult tasks such as self-monitoring or daily logging [68]. It can also positively affect health outcomes when used as a component of behavioral intervention technologies [69]. Although passive data collection raises other ethical issues, it is less likely to disrupt a person’s thoughts and activities than diaries, paper questionnaires, telephonic or electronic prompts for data, and similar methods [70, 71]. Mobile phones, in particular, may be less disruptive because they are often already embedded in people’s routines and have broader market penetration than wearable activity trackers or medical devices (e.g., Holter monitors).

Smartphones are also useful as a means for capturing passive data because they capture user-specific social and personal user data, collected when users make calls, write and send texts, manage contacts, or are simply present in an environment. They contain a multitude of sensors, which can be used simultaneously, provided sufficient battery power. Smartphones have other advantages such as their many functionalities (calling, data service, settings control), Internet connectivity, advanced processors, and high-resolution display. However, research needs to be done to test the hypotheses that, compared to other measurement approaches, smartphone-based passive sensing is less disruptive, more effective, more efficient, and more likely to be accepted and used over time.

5.1 Strengths and weaknesses of reviewed studies

The 35 reviewed studies applied passive sensing across domains of health and wellness, demonstrating a degree of generalizability. Multiple studies in the area of mental health showed it was feasible to use passive sensing, including ones capturing sensitive data such as location [35, 56], in a domain surrounded by ethical issues related to privacy, consent, and self-awareness. However, while people appear to accept sharing personal data for research, they may be more reserved when commercial interests are present [52, 72]. At the same time, not all domains were covered in the reviewed studies, raising questions about the applicability of smartphone-based passive sensing for other diseases, multiple comorbid conditions, and populations of older, cognitively impaired, rural-dwelling, or vulnerable individuals. Overall, few studies reported participants’ views on passive sensing and privacy, raising concerns about acceptance outside academic research studies, especially when sensitive sensors—microphone, GPS—are used [73]. The concern is especially high for research among ethnic minorities, for whom privacy is an important but perhaps underappreciated concern [74].

The sample size of most studies was acceptable for feasibility assessment but not to demonstrate clinical value, as others have noted about innovative health informatics research [27, 75]. For example, Fiordelli et al.’s [75] systematic literature review of mobile health (mHealth) research between 2002 and 2012 found that the average sample size decreased over the years, although the variety of study designs has increased as more clinical studies have been performed over time. The majority of the studies reviewed here were able to manage the technological challenges related to sensors, data processing, and security, although in many cases this was easier to accomplish when studies were performed outside of routine clinical care or with healthy volunteers, for example, university students enrolled in a class [63].

Overall, although the studies were innovative, as a whole they did not demonstrate the use of passive sensing in actual clinical contexts and did not measure or report changes in health outcomes, as most studies were not interventional by nature. Studies generally dealt with human-computer interaction (HCI) and technological issues rather than addressing questions of clinical integration or scalability. Notably, only 18 papers (51%) were published in healthcare venues. This may explain why issues such as privacy or health outcomes were not comprehensively addressed and sometimes ignored.

In terms of study reporting, technical elements of the studies were usually sufficiently reported. While older studies often had missing or inadequate information about settings and implementation, recent studies tend to be more rigorous on these aspects—following a global phenomenon in mHealth studies [76]—but for the most part fail to systematically report challenges, especially ethics- and privacy-related ones. Systematically reporting technological and methodological challenges, as well as the views of participants on ethics and privacy, would benefit the planning and execution of future studies using passive sensing on smartphones.

5.2 Recommendations

Choosing the right passive sensing strategy

Our review showed many different ways to configure the data collection, processing, and use of a smartphone-based passive sensing system. For example, studies differed in the number and type of sensors used, location and timing of data processing, and the nature of feedback to users.

Interestingly, the number of sensors used in research studies has been relatively stable over the years; the average sensor count across studies was between 2.5 and 4 for any given year. As sensors have become more energy-efficient and smartphone makers have added dedicated chips to process sensor data, it has become more practical to capture data from as many sensors as possible, for subsequent processing as needed. However, as more data streams are captured, it is important to derive new features—i.e., features that can be deduced from raw sensor data, from simple mathematical calculations to the number of speakers in a room—to facilitate machine learning [77]. These computed features should match the problem at hand, such as speech detection for people with schizophrenia, an indicator of social functioning [35].

An important distinction between studies was the nature of the input from participants. In a few cases, the approach required little to no input from study participants, using unsupervised machine learning algorithm classes, e.g., clustering. This can be used to learn the correspondence between sensed data and an interpretation, such as how geographical coordinates inform a lack of mobility [55]. In most cases, however, participants were required to label sensed data in the study’s initial stages, for example by tapping a button each time a cigarette was smoked [52]. These labeled data points are especially helpful for identifying outliers but may be less practical than completely passive strategies.

In general, given the many possible strategies for passive sensing, we recommend choosing a combination of data collection, processing, and use that is based on project- and population-specific needs: a mix-and-match or configural approach.

Personalized and Similar-User Models

A few of the studies reported null or weak correspondence between sensed data and a phenomenon of interest. For example, in one study the prediction of depression from sensor data yielded 60% accuracy [61]. However, some have pointed out that what might be misconstrued as inaccurate sensor data could be more valuable by applying personal rather than population-based prediction models [55]. A particular pattern in one’s data may reveal something characteristic of that user [78]: “different people will have different behavioral indicators of mental health difficulties” [35]. The use of personal sensing mirrors n-of-1 clinical trials and indeed, some have suggested the use of sensing devices for n-of-1 trials [79].

An alternative to strictly individualized models is using “similar user” models, or models grouping similar users to increase the volume of data to be used by machine learning algorithms (e.g., [43]). While these models may have lower accuracy than personalized models, they are more generalizable and do not rely on as much user-labeled data.

Next Steps for Passive Sensing

The advent of deep learning systems, combined with increasing mobile computing power, suggest a future direction for passive sensing for smartphones [80]. Initiatives such as Google’s TensorFlow and Apple’s Core ML enable developers to train and use neural networks directly on smartphones in order to perform data processing that formerly required a remote server, for example, offline language translation [81–83]. These emerging technologies may ultimately permit rapid and context-sensitive passive sensing, machine learning, and just-in-time personalized intervention delivery, especially if integrated within existing frameworks for behavior change technologies (e.g., [84]).

Future work must also better address privacy, both conceptually and practically. Most studies addressed data security via secure transmission or encryption, but future studies must also tackle other privacy issues, for example, those related to the third-party use of personal data or storage of data in databanks not controlled by device users [85]. Judging from the major barriers to personal health records adoption [86], concerns about privacy may also deter widespread adoption of passive sensing. Much like any new and spreading technology, future studies must critically and comprehensively assess the acceptance and longitudinal use of passive sensing systems [87] as well as any adverse consequences.

A major general recommendation to address some of the above issues is for technology specialists (e.g., informaticists, computer scientists) to partner more effectively with clinical experts to identify and address problems amenable to passive sensing [69, 88, 89]. Only through these kinds of partnerships can novel technologies be designed and assessed for practical value, scalability, and sustainability. This partnership is especially important in specialty fields such as mental health, where passive sensing is promising but has not reached its full potential [26, 69, 88].

Recommendations for future research on passive sensing for health are compiled in Table 7.

Table 7.

Research opportunities and related informatics methods.

Health and Wellbeing

• Extension of smartphone-based passive sensing to new health and wellbeing domains, such as caregiving (e.g., a notification sent when somebody wakes up).

• Testing the integration of passive sensing into clinical care, care coordination, and telehealth.

• Studies of passive sensing for population health management and public health.

• Studies of passive sensing in the context of precision medicine.

• Controlled trials of efficacy and comparative effectiveness of passive sensing-enabled interventions on health outcomes.

Policy and Privacy

• Understanding privacy and data ownership concerns and preferences among potential end-users of smartphone-based passive sensing. Specific technology topics for research on privacy include cross-application communication, cross-device communication, and health data aggregators (e.g., Apple Health).

• Development and testing of new privacy and security protocols as well as strategies for users to set custom privacy and security settings.

• Implementation of a legal framework to address privacy and data ownership in passive sensing on smartphones, especially for sensitive health domains such as mental health.

• Discussion of a legal framework to address failures in data protection strategies (e.g., data leak), taking into account consumers, clinicians, and researchers.

• Research on the effect of concerns about privacy on the acceptance and use of passive sensing technologies.

Analytic Models

• Comparison of personalized and similar-user models with general models across several measured phenomena to assess the relative fitness of each model.

• Comparison of the same models between devices to see if significant differences exist.

• Focus on higher-level data and clinical interpretations (e.g., bipolar cycles) as the detection of lower-level data (e.g., sleep duration) matures.

Human-Computer Interaction

• Analysis of cost effectiveness and efficacy of passive sensing on smartphones vs passive sensing with wearables and traditional methods such as paper- based logging.

• Replication of studies with larger and more diverse samples.

• Combination of passive sensing technologies and other data sources for multiple conditions, using various strategies including pulling composite data from a third party, such as the operating system or middleware (e.g., [90]).

• Integration with electronic health record (EHR) and personal health record (PHR) products in the contexts of personal health information management and clinical use of patient-generated data [91].

• Development and testing of clinician-facing interfaces to efficiently and effectively utilize passively-acquired data.

• Longitudinal research on the acceptance and use of passive sensing technology for health, over time (months, years, decades).

6. LIMITATIONS

Because of the topic of the review and the infancy of the field, papers may not have been captured in our search, despite the use of broad terminology and brand names (e.g., Android, iPhone) in the search queries. This review was unique in focusing on mobile phone systems, because of the advantages described above, but consequently did not incorporate the broader literature on passive sensing using wearable devices such as activity trackers [75] or data collection from social networks [17, 18]. Given the small and heterogeneous set of reviewed papers, we were unable to apply a systematic quality evaluation system or draw conclusions about effect sizes using quantitative meta-analysis.

7. CONCLUSION

As demonstrated by the present systematic review, the field of passive sensing for health and wellbeing shows early promise, despite ongoing maturation. Several stakeholders may benefit from future application of smartphone-based passive sensing: 1) users, who may in the future be able to receive just-in-time or scheduled feedback on data without much additional burden; 2) healthcare professionals, who may be able to receive more accurate and timelier reports about their clients; and 3) researchers, who may gain access to rich datasets with validated data concerning participants’ behavior. The use of data that are patient-specific, accurate, and minimally burdensome may power future models of health and healthcare that are smarter, more connected, and more personalized. However, there remain multiple gaps between this vision and the present state of the art. In particular, additional research is needed to address major issues such as clinical efficacy, integration of newer analytic approaches including artificial intelligence (AI), privacy issues, and implementation of passive sensing into actual clinical care. Addressing these issues will require advances in both technology and in the composition of research teams towards interdisciplinary collaborations of experts on technology, human-computer interaction, and clinical care.

Table 3.

Summary of sleep studies, ordered by year of publication.

Author (Year) Location	Study Purpose	Principal Findings	Sensors Used	Sample Size & Type	Study Length (Days)
Bai et al. (2012) [33] China	Predict sleep quality from smartphone data collected during the day	Prediction of sleep quality with 78% accuracy, using a factor graph model.	• Accelerometer • Call logs • GPS • Light sensor • Microphone • SMS patterns	15 Not specified	30
Natale et al. (2012) [51] Italy	Compare commercial sleep monitoring device data with three smartphone accelerometer algorithms for assessing sleep	• No statistical difference for total sleep time between the best performing algorithm and the commercial monitoring device. • Agreement rate on sleep-wake discrimination of 90% between the best performing algorithm and the commercial monitoring device.	• Accelerometer	13 General sample	4.8*
Chen et al. (2013) [40] USA	Compare an application detecting sleep from smartphone data with three other sleep detection methods	• Better user experience and lower perceived intrusiveness for the passive sensing application than for the other sleep detection methods • Greatest error in the estimation of sleep duration (+/− 43 min) compared to the other sleep detection methods.	• Accelerometer • Device activity • Light sensor • Microphone	8 University students	7
Abdullah et al. (2014) [30] USA	Predict sleep time, duration, and deprivation from smartphone data	Social jetlag, sleep inertia, and sleep debt can be estimated from sensor data.	• Application usage • Browser history • Call logs • Device activity • SMS patterns	9 University students	91.1*
Min et al. (2014) [50] USA	Detect sleep and sleep quality in natural settings	• Detection of sleep with 94% accuracy using a Bayesian network with feature selection model for the individual model. • Detection of sleep quality with 84% accuracy using a Bayesian network/feature selection individual model, and 81% accuracy using a Bayesian network/feature selection global model.	• Accelerometer • Application usage • Device activity • Light sensor • Microphone • Proximity sensor	27 General sample	30
Bhat et al. (2015) [37] USA	Compare a commercial phone sleep monitoring application for iPhone with in-laboratory polysomnography	Detection of sleep by the application with 90% sensitivity and 50% specificity.	• Accelerometer	20 General sample	6

GPS: Global Positioning System; SMS: Short Message Service;

^*average duration of subject participation.

Table 4.

Summary of general health and wellbeing studies, ordered by year of publication.

Author (Year)	Study Purpose	Principal Findings	Sensors Used	Sample Size & Type	Study Length (Days)
Rabbi et al. (2015a) [55] USA	Evaluate the generation of recommendations for calorie loss from accelerometer and location data, using personalized vs. generic recommendations	Group with personalized suggestions performed better on physical activity and dietary behavior than control.	• Accelerometer • GPS	17 University students and non-students	21
Rabbi et al. (2015b) [54] USA	Evaluate the generation of recommendations for calorie loss from accelerometer and location data	Significant increase in physical activity and decrease in calorie consumption when participants received personalized recommendations.	• Accelerometer • GPS	16 University students and non-students	98
Eskes et al. (2016) [42] Netherlands	Predict sociability from smartphone data	Weak relationship between smartphone use and overall sociability assessments.	• Application usage • Bluetooth • Call logs • GPS	10 University students	11.4*
Kelly et al. (2017) [47] UK	Predict health status from accelerometer data	Prediction of health status with a mean absolute error of 11.7, using a support vector machine classifier.	• Accelerometer	171 General sample	4.8*

GPS: Global Positioning System;

^*average duration of subject participation.

APPENDICES

A.1 Definition of Terms Related to Passive Sensing

Term	Definition
Ecological Momentary Assessment	“Repeated sampling of subject’s current behaviors and experiences in real time, in subjects’ natural environments.” [11]
mHealth (mobile Health)	Mobile technologies for health or healthcare. This term includes technologies used by health professionals or nonprofessionals [75]
Mobile Sensing	Term encompassing all portable technologies (phones, wearables, etc.) relying on sensors. Mobile sensing is not limited to the individual but can be used to capture crowd phenomena, as well as environmental phenomena. May require user input to capture data.
Internet of Things	Communication of traditional physical objects (e.g., body weight scale, fridge) with other objects and systems (e.g., electronic health records) via the Internet [92].
Passive Sensing	Technique utilizing technologies capturing personal, crowd, or environmental data with little to no user input or effort during data collection. Passive sensing can be mobile but can also be embedded in the environment (e.g., thermal sensors).
Pervasive/Ubiquitous Technology	Computing devices that are present in the environment rather than as specific machines [93]; their interfaces become “invisible, natural and everywhere” for the user [94].
Smartphone	Cellular phones capable of performing advanced computing tasks whose features can be extended through applications downloaded from the Internet [95].

A.2 Summary of Main External Smartphone Sensors Used in Passive Sensing

Term	Function	Commercial Application Examples
Accelerometer & Gyroscope	Determining the speed of movement in space as well as speed of rotation of the device.	Pedometer application. Activity tracking (e.g., Google Fit)
Antenna	Detecting nearby cellular towers and relaying the signal to the broadband processor for voice/SMS/data communication.	Contextual messages when entering a certain area (e.g., text messages received when roaming in another country)
Bluetooth	Detecting and communicating with other Bluetooth-enabled devices.	Wireless audio. Transmission of files between phones.
Global Positioning System (GPS)	Receiving information of four or more GPS satellites to calculate the position of the device.	Car navigation (e.g., Google Maps Navigation)
Light sensor	Determining the amount of light reaching the device.	Automatic screen brightness adjustment.
Microphone	Capturing external sounds onto the device to for recording, processing, or transmission [96]	Audio recorder. Phone calls.
Proximity sensor	Detecting the proximity between the front of the phone and any obstacle, such as a human face.	Turning off the phone screen during calls.