Smartphone and Wearable Device-Based Digital Phenotyping to Understand Substance use and its Syndemics

Abstract

Digital phenotyping is a process that allows researchers to leverage smartphone and wearable data to explore how technology use relates to behavioral health outcomes. In this Research Concepts article, we provide background on prior research that has employed digital phenotyping; the fundamentals of how digital phenotyping works, using examples from participant data; the application of digital phenotyping in the context of substance use and its syndemics; and the ethical, legal and social implications of digital phenotyping. We discuss applications for digital phenotyping in medical toxicology, as well as potential uses for digital phenotyping in future research. We also highlight the importance of obtaining ground truth annotation in order to identify and establish digital phenotypes of key behaviors of interest. Finally, there are many potential roles for medical toxicologists to leverage digital phenotyping both in research and in the future as a clinical tool to better understand the contextual features associated with drug poisoning and overdose. This article demonstrates how medical toxicologists and researchers can progress through phases of a research trajectory using digital phenotyping to better understand behavior and its association with smartphone usage.

Introduction

In recent decades, mobile phones have revolutionized the ways in which individuals access the Internet, communicate, and record activities during their daily lives. Advances in these technologies have resulted in a generation of smartphones that not only facilitate increasing interconnectedness among individuals, but provide new sources of data through which health behaviors can be queried, contextualized, and eventually modified. As of 2021, 97% of Americans owned a cell phone and 85% owned a smartphone [1]. Internationally, smartphone use is also widespread, but disparities in prevalence exist. Among high-, middle-, and low-income countries surveyed in 2022, overall smartphone prevalence was 85% [2], but high-income countries had 31% greater smartphone prevalence [3]. In parallel to the increasing ubiquity of smartphones is the rise of paired wearable devices, such as smartwatches and other fitness trackers. Despite only being introduced to the public during the 2010s, by 2020, 30% of the US population already owned and used wearable devices [4]. Of those who reported ownership of a wearable device, 82% reported willingness to share data from these devices with their healthcare providers [4]. Although not all individuals worldwide own a smartphone or a wearable device, these devices are ubiquitous and no longer confined to individuals with higher socioeconomic status.

The development of smartphones and wearables has been accompanied by progressive device miniaturization and integration of sensors needed to ensure the daily operation of these devices. For example, most smartphones today have integrated gyroscopic sensors that allow individuals to reorient screens in landscape and portrait modes [5], Bluetooth modules that facilitate low energy linkage to an Internet of Things (IoT) [6], and global positioning system (GPS) chips that enable individuals to map directions [7]. While these sensors increase the practical usability of smartphones, they also collect various types of passively generated data that provides insights into the context of daily life. Increasingly, data from these sensors have been collected to discover patterns that might reflect behaviors of phone and wearable device use. Applications can range from geolocating individuals in their day-to-day lives, to helping advertisers promote products for purchase, to providing safety data to those affected by natural disasters. Other forms of actively collected digital data (e.g., text messages, images, social media posts) also exist and require their own sets of methods, which are beyond the scope of this Research Concept article.

Based on successes in leveraging smartphone use patterns to personalize commercial advertising, researchers have piloted similar techniques to discovering health behaviors, a technique called digital phenotyping. Digital phenotyping seeks to identify patterns of smartphone or wearable device use and understand their relationships to specific health behaviors (Fig. 1) [8]. Digital phenotyping has been used to predict the risk of diagnosis, exacerbation, or relapse, and identify triggers in various mental and physical health problems, such as: predicting relapse in schizophrenia [9], asthma treatment [10], defining profiles of suicidal thinking [11], sleep-related cardiovascular disease risk [12], and substance use treatment [13]. As smartphones and wearables become increasingly integrated into daily activities, their use can be associated with real-time, subclinical changes in health behaviors that previously went undetected until clinically manifested.

Fig. 1.

Digital phenotyping collects passive data from smartphones or wearables with a target biobehavioral outcome in mind. Large amounts of data are collected from devices, and then analyzed using various machine learning techniques to develop algorithms than can be piloted among populations of interest

Despite the promise of digital phenotyping, however, much exploration is still needed to contextualize smartphone data and understand the willingness of individuals to participate in phenotyping efforts in health-related contexts. Additionally, most digital phenotyping work seeks to understand the relationship between smartphone-based sensor activity and physiologic changes induced by disease (for example, decreased respiratory rate suggestive of opioid overdose). Because individuals carry smartphones with them during the course of life, it is plausible that similar data gathering strategies can be used to uncover phenotypes associated with syndemic conditions. In this research concepts paper, we review the basic operational characteristics of digital phenotyping both on smartphones and wearable devices, describe its application in substance use disorders and its syndemic problems, and provide future recommendations for the field in relation to medical toxicology. Broadly defined, syndemic problems are a set of co-occurring psychosocial, structural, and physical health problems that are hypothesized to interact synergistically to produce worse health outcomes [14]. One example of syndemic problems is a set of mutually enhancing problems of substance use, depression, post-traumatic stress, and food insecurity that interrelated to negatively impact HIV outcomes [15]. Another example of syndemic problems are obesity, undernutrition, and climate change, which are thought to interact and exacerbate one another [16].

Fundamentals of Digital Phenotyping

Digital phenotyping is the practice of passively collecting operational and physiologic data through smartphones or wearable devices to understand patterns of changing health behaviors. In order to leverage digital phenotyping, we assume individuals own and use the phone or system that will be used to collect data. Strategies to verify this may include asking individuals to log into the phone, or for wearable devices, biometrically verifying the user through cardiac markers including R-R intervals [17]. In different countries, sharing of devices including smartphones may be common— in these instances, recognizing these cultural uses of smartphones and providing an interval of data assessment to understand variations in use that may correspond to different users may help parse data for analysis. While there are many commercially available wearable devices and smartphones that rapidly evolve over time, many have common sensors that may allow specific data streams to be harmonized across devices (Table 1). These sensors typically collect large volumes of passive data that are used to run background smartphone functions. A log of background sensor data can be obtained using commercially available or custom-designed apps that run in the background of the smartphone system when installed. This facilitates collection of large and variable datasets, which can be later used to deduce smartphone or device use. The frequency of sampling, volume of data, and strategies to transmit data to researchers (either via wireless connection or using a system’s existing cellular signal), are tunable to the ultimate objective of the research question.

Table 1.

Common sensors integrated into contemporary smartphones and wearable devices

Data Type	Smartphone Sensors Only	Both Smartphone and Wearable Sensors	Wearable Device Sensors Only
Accelerometer		X
Altimeter			X
Barometer		X
Battery life	X
Bioimpedance sensor (respiratory rate, sleep, etc.)			X
Compass			X
Device motion	X
Electrodermal activity sensor (EDA)			X
Gesture sensors (wrist motion)			X
Global positioning system (GPS)		X
Gravity	X
Gyroscope		X
Heart rate			X
Keyboard	X
Light exposure/screen functionality		X
Low energy Bluetooth	X
Magnetometer		X
Microphone			X
MQ Telemetry Transport (MQTT)	X
Optical and electrical heart sensors			X
Orientation			X
Power status	X
Proximity		X
Reachability	X
Speaker			X
SPO2 monitor			X
Temperature	X
Temperature (skin)			X
UV sensor			X
Vibration motor			X
WiFi usage	X

Due to the large volume of data that is collected via digital phenotyping, obtaining authoritative ground truth annotation of an individual’s actions and behavior is vital to developing initial algorithms that might identify and potentially respond to these behaviors. Most research studies will therefore utilize ecological momentary assessments (EMA), consisting of brief text or in-app surveys to collect self-report data on daily behaviors. It is important to note that EMA is not the same as digital biomarkers [18], and in digital phenotyping, EMA can be used to assist researchers in identifying, labeling, and understanding phenotypic data. For example, researchers interested in developing a digital phenotype of substance use will need participants to self-report past instances of substance use in order to train their algorithms to detect future substance use based on digital phenotyping data. Other strategies that should be employed by researchers include obtaining key ground-truth information from participants during enrollment – such as a home address and frequently visited locations of interest – that will be used to ground GPS data during analysis. Finally, automated annotation strategies using ingestible sensors, wearable devices, or geofenced locations may allow research participants to contribute more contextual information alongside phenotyping data linked to events or places of interest.

While the concepts of digital phenotyping may apply to both wearable and smartphone based systems, our current work has leveraged smartphone systems given their ubiquity, and unintrusiveness. While this limits data collection to existing smartphone sensors and limits the ability to collect physiologic data, smartphone based phenotyping may be useful to better understand general patterns of phone use and their relationship to pervasive structural syndemic issues that may affect daily life, especially among individuals with substance use disorder. This leads to research questions surrounding how individuals experience and cope with syndemic related problems rather than questions of physiological adaptations (e.g. Changes in resting heart rate or heart rate variability or electrodermal activity in response to daily stress). Additionally data curation varies between the two phenotyping strategies— while one can take advantage of native data connectivity or wireless linkage in smartphones to transmit real time sensor data, most wearable devices will rely on connection to a second device like a smartphone to relay information to a study team. As we describe potential applications of digital phenotyping in this research concepts piece, we focus on how smartphones and their inherent fallacies may be leveraged to understand syndemic conditions and their intersection with substance use.

Application of Digital Phenotyping in Substance use and its Syndemics

Digital phenotyping is particularly applicable in the context of identifying specific health behaviors. Many health behaviors occur in unique contexts, with identifiable antecedents that can be detected based on smartphone usage. When individuals provide detailed annotations of these behaviors, digital phenotypes can be derived. This presents a means by which to identify and potentially intervene on these behaviors as, or before, they occur. Substance use is one such behavior that is highly contextual, has readily identifiable antecedents, directly affects health, and is amenable to real-time assessment and potentially intervention.

Other similar factors may also be related to substance use. For example, some individuals may obtain substances from specific locations prior to use, which can identified through GPS data. Researchers have used GPS [19] In the time preceding substance use, some individuals’ keyboard activity may rapidly increase as they attempt to obtain substances. Immediately before substance use occurs, some individuals’ accelerometry, battery life, light exposure, reachability, and Wi-Fi usage may change. Additionally, given the relationship between substance use and other mental health comorbidities, frequency of phone use, battery life, screen time and other parameters may grossly change in the days preceding or immediately post substance use. There may also be data from wearable devices that can be associated with obtaining substances, preparing to use substances, and/or subsequent substance use – such as changes in heart rate, bioimpedance and electrodermal activity, SPO2, skin temperature, and gesture sensors [20–23]. While much phenotyping data seeks to detect real time changes associated with substance use, there is also value in understanding longitudinal trends as just in time, personalized interventions may not be readily available, and in many clinical settings, digital phenotyping data may be implemented to help inform existing clinical assessments among these patients.

Digital phenotyping also has applications in problems related to substance use. One way to conceptualize the interrelation of substance use with other problems is via syndemic theory. Synergistic epidemics, or syndemics, are posited to be biopsychosociostructural problems that are interrelated and mutually reinforcing [14, 24]. When comorbid epidemics are present, they are thought to synergistically relate to one another such that worsening of one problem leads to worsening of other comorbid problems in what can be conceptualized as a positive feedback loop within a network of syndemic problems. For example, increases in depressive symptoms within an individual may lead to substance use as a form of coping. Much of the research on syndemic problems has been conducted among people with HIV or at increased risk of HIV acquisition. Substance use is a commonly studied syndemic problem [14, 25–28], which has a number of comorbid syndemic problems, such as HIV treatment or HIV prevention non-adherence, Depression, anxiety disorders, post-traumatic stress disorder, attention-deficit hyperactivity disorder, schizophrenia, and food insecurity [15, 29–31]. Similarly to substance use, these problems can also be examined and better understood using digital phenotyping methods. From a syndemic perspective, we may also see changes in their smartphone usage due to depressive symptoms, which could be detected and intervened on [32].

Once digitally phenotyped, syndemic problems can also be intervened upon in real-time to reduce negative effects on health outcomes.

The techniques of digital phenotyping can therefore be employed to better understand smartphone and wearable device data collected in the context of substance use and its syndemic problems, through annotations of such data provided by participants via EMA. Specifically, participants can provide information on when and where they last obtained and used substances. It is important to note the importance of collecting ground truth annotations via EMA in an ecologically valid way. The closer the EMA is to the event of interest, the more reliable these data will be, and the more beneficial they will be for digital phenotyping efforts. Limitations of EMA exist, including engagement and timing. In the absence of valid and useful EMA data, the large amount of digital phenotyping data is difficult to interpret and utilize [19]. Despite research suggesting that individuals with substance use disorder are willing to provide EMA data [32], among a sample of individuals with opioid use disorder engaging in a digital phenotyping study, EMA response rates declined over time [32]. It may be beneficial to adopt research designs that incentivize EMA completion (e.g., an additional payment for a certain percentage of EMA survey completion).

Once digital phenotypes of substance use are identified, algorithms can be trained to detect substance use in real-time – or even before it occurs, potentially using the phenotypes of its syndemic problems as a base for detection. In applying a syndemic lens to substance use, enhancing our ability to create digital phenotypes of substance use will also require the ability to digitally phenotype comorbid problems. Therefore, understanding digital phenotypes of problems that are syndemic with substance use will subsequently improve our ability to create digital phenotypes of substance use and provide intervention. Once detected, a network of interventions can be delivered to reduce or prevent substance use. These interventions may be empiric behavioral (e.g., psychoeducation, motivational interviewing, behavioral activation) or non-behavioral (e.g., suggesting dosage of pharmacological therapy, linkage to services), and can serve to help individuals achieve their substance use cessation or harm reduction goals.

Applications of Digital Phenotyping in Understanding Contextual Patterns - a Case Report

Of all the sensor streams available in digital phenotyping, one of the most promising modalities of interest may be GPS. The best way to understand how useful GPS data might be in creating a digital phenotyping model is to look at an example of data that has been collected over time. As an exemplar case we provide preliminary GPS data from an ongoing investigation that leverages digital phenotyping to discover contexts in patterns of daily activity that may correspond to HIV pre-exposure chemoprophylaxis (PrEP) adherence (Fig. 2) [33]. By utilizing density-based clustering algorithms, we can begin to understand what spaces are important when attempting to identify specific behaviors. We can see that there are distinct clusters of activity taking place over time. Furthermore, we can also utilize the center-most points of these clusters (indicated by stars) and supplement them with ground-truth information about specific behavior of interest (which can be collected through EMA, surveys, interviews, etc.) to determine what the clusters may mean. For example, GPS clusters can indicate that an individual is at home, versus at work or other locations of interest. Deviation from a regular routine and expected locations could indicate a change in schedule – or, in the context of individuals with substance use disorders, such data could suggest potential active use. Algorithms can then be triggered to provide support to individuals in the event that GPS data indicates schedule or location changes that may indicate health-related risks.

Fig. 2.

Clustering of GPS data into geographic regions. Here the stars indicate the center-most points of each individual cluster. Each color shown on the map below represents the centroid that cluster belongs to and is labeled with the corresponding predicted activity. These GPS data have been anonymized for publication

While clustering algorithms are a popular method to remove noise and extraneous data points from the raw GPS data available, one can imagine that they can be utilized to create geofences that can indicate when a participant may be entering an area that’s determined to be a predictor of specific behavior – such as substance use. These strategies must be predicated on appropriate ground-truth information when substance use (or any other behavior of interest) has happened in the past. For example, if ground truth annotation provides information on past substance use locations, future GPS data could be indicative of potentially imminent substance use. Outside of clustering methods, other strategies like movement features – such as location variance, time spent at home, time transitioning between locations, total distance traveled, diurnal movement, and the total number of unique location clusters visited – could indicate changes in behavior that may lead to behavioral outcomes including substance use or medication nonadherence. Learning algorithms, such as support vector machines, linear regression, and convolutional neural networks (CNNs) have all shown promising results when applied to similar predictive and retrospective tasks [34–37]. The ultimate model of choice will need to be grounded in the types and quality of data as well as the presence of available ground-truth data.

GPS data can be correlated with other phenotypes to further understand user behavior. For instance, Fig. 3 shows one week of screen state information that was captured concurrently with GPS data from Fig. 2. Screen states are identified as being on (screen on and phone actively being used), off (screen off and no interaction with phone operating system), or in “doze,” where the operating system has gone idle. By correlating the screen state activities to the GPS clusters described above, we can begin to understand additional behavior patterns that occur during certain activities not available for interpretation via GPS data, such as sleep patterns [38].

Fig. 3.

Identification of screen states captured through user interactions with a phone. Here the different markers indicate what screen state was captured and the lines are colored with the corresponding location the user was at when the activity was captured

Ethical, Legal and Social Implications of Digital Phenotyping

There are potentially wide-ranging ethical, legal, and social implications associated with digital phenotyping.

First, there are concerns regarding ethical principles like consent, autonomy and preventive misconception. Second, there could be potential legal implications of phenotyping data, particularly in association with stigmatizing or sensitive health conditions. Finally, there may be social implications for individuals engaging with digital phenotyping in the event of the inadvertent discovery of sensitive activities that may be linked to health behaviors.

Ethical Implications

Given that the concept of digital phenotyping is likely to be foreign to potential research participants, and the fact that the strategy captures a broad range of sensor-based data, substantial explanations and detailed discussions during the informed consent process will be warranted. This will be particularly important when enrolling individuals in studies that utilize digital phenotyping in an effort to understand stigmatized conditions like substance use or HIV. Strategies for managing ethical implications of using digital phenotyping in the research context this may include providing sample data outputs from digital phenotyping studies to help ground the research, and providing participants with detailed information around the types of data that will be collected during the study (e.g., the length of their text messages and time sent/received) – and, critically, the types of data that will not be collected (e.g., the content of their phone calls and text messages).

During the course of a study, digital phenotyping data may also uncover other behaviors that, while related, may not be the primary aim of the research. For example, a study measuring phenotypes of substance use may inadvertently uncover patterns of device usage correlated to worsening depression. The specific responsibilities of the investigator and research team remain unclear in these contexts, but the informed consent process should certainly describe in detail the scope of phenotyping data obtained and how it might potentially be used. As digital phenotyping procedures and technology continue to advance, additional guidelines and public policy may be helpful in providing guidance to research teams who conduct digital phenotyping research. As analytic procedures continue to improve, previously collected digital phenotyping data will be able to be analyzed in new ways that may discover new phenotypes of various syndemic problems. It will be important for researchers to have guidance on how best to analyze digital phenotyping data in secondary data analyses as well as to guide data collection that is cognizant of these potential future advances.

Another key ethical concern is preventive misconception—the idea that enrolling in a study may actually have therapeutic efficacy. Individuals may perceive enhanced vigilance surrounding certain activities by research team members, while in reality phenotyping data may be queried only periodically. For example, participants may feel more comfortable engaging in behaviors that could be dangerous to their health because they incorrectly believe someone is monitoring their data continuously and will intervene if needed. Ensuring that participants clearly understand the frequency of both data collection and data review by a research team – including the principal investigator – and the intended uses for collected data, represent critical elements of the informed consent process.

Legal and Social Implications

There may also be legal and social implications surrounding certain elements of digital phenotyping data. Key location data, especially if associated with illicit activities like substance use, may have implications within the criminal justice system as police and prosecutors may wish to use digital phenotyping data in cases involving illegal behavior. It will be critically important to ensure digital phenotyping data are confidential and participants considering whether to consent to provide such data are well-informed of any potential risks. Discovery of mental health outcomes, engagement with devices during specific times of the day, or correlates with other activities may additionally have social implications. For example, discovery of different locations during a time when a participant is supposed to be at school or work could impact job or classroom performance. These risks should be described to participants and strategies to mitigate legal implications should be explored. These may include obtaining a Certificate of Confidentiality to protect participants and their data, or anonymizing data through a randomized offset (introducing fuzziness to data) to obscure precise data outside the context of a research study [39], or combining data by cohort and analyze and report on data at a group level rather than an individual level.

Future Directions and Applications

Digital phenotyping provides several opportunities to address the complex interplay between substance use disorders and their syndemic conditions, specifically socioeconomic, mental health, and structural barriers that impact access to care and recovery. These contextual cues may assist addiction medicine clinicians and medical toxicologists in understanding key peripheral information that may personalize the treatment of substance use disorder. While the traditional model of substance use treatment involves providing both behavioral and pharmacologic therapy, understanding the contexts in which individuals with SUD face daily challenges to medication adherence, experience craving, and confront challenges that may result in substance use, this traditional treatment paradigm remains reactionary with interventions occurring at scheduled clinical visits. Digital phenotyping may be able to advance the delivery of interventions that support key SUD treatment options like medication adherence or help annotate periods of high risk where just-in-time counseling or behavioral coaching may mitigate craving. For the medical toxicologist, context information, particularly GPS based data that suggests substance use may be leveraged to help inform public health interventions such as deciding on placement of harm reduction outreach, expanding drug testing in the community and in hospitals, or when provided in aggregate, used in bedside counseling for patients seen as toxicology consults surrounding the likely circumstances of substance use and strategies to avoid geographic triggers. To enable these future applications of digital phenotyping, the discovery of key phenotypes surrounding substance use and syndemic conditions should be defined for a population of interest. Early discovery studies may consist of single-arm, observational trials involving a structured effort to obtain ground truth information for comparison against collected digital phenotyping data. These observational trials form the basis of algorithm development and define the specificity of data streams necessary to develop a digital phenotype. These trials may additionally inform new data analytic algorithms and strategies to account for missing data and noise that is inherent from individual smartphone use. Additionally, with improving extraction techniques and appropriate ethical parameters, natural language processing algorithms may be used to infuse phenotyping data with even more psychosocial contexts based on prosody, mood, and other parameters to enhance the ecological validity of future interventions.

One major challenge in the discovery of such phenotypes lies in optimally annotating data streams to assist in data analysis. Potential solutions include conducting baseline qualitative interviews to understand patterns of daily routine, which can be used to contextualize phenotyping data obtained later during the study; structured ecological momentary assessments to understand daily events of interest; or using adjunctive wearable or ingestible systems to automatically annotate key time periods of interest within a day. When specific phenotypes that may suggest substance use and its syndemics are clarified and identified, future research should investigate strategies to leverage such phenotyping data to provide just-in-time behavioral support or to inform existing counseling strategies around substance use. Studies may randomize individuals to a digital phenotyping-driven intervention for substance use compared to standard of care, or could provide specific geofenced interventions around substance use mitigation based on locations that may trigger substance use (Fig. 4).

Fig. 4.

Digital phenotyping research trajectory flow diagram

Finally digital phenotyping can be used to address many public health problems in the field of medical toxicology. For example, the use of wearable based phenotyping may help medical toxicologists to understand trajectories of substance use among patients in substance use recovery, which would allow for more personalized guidance for these individuals in the context of their syndemic presentation [40]. Specific digital phenotypes could also indicate changes in movement and/or patterns of device use that could be leveraged by medical toxicologists to detect adverse drug reactions. Finally, on a population level, as digital phenotyping techniques continue to evolve, poison centers may be able to utilize digital phenotyping information to understand the location and timing of overdose events or ingestions, and potentially even the circumstances in which these ingestions occur, to facilitate timely intervention and referrals to hospitals.

In sum, digital phenotyping represents an exciting new frontier in biobehavioral research. As smartphones and wearable devices become an increasingly integral aspect of everyday life, the use of digital phenotyping to identify and provide an opportunity to deliver real-time interventions as, or before, deleterious health behaviors occur or worsen is appealing. However, multiple ethical, legal and social implications exist, and additional research on digital phenotyping, associated interventions, and the mitigation of such implications is needed.

Funding

PRC funded by DP2DA056107, the Kenneth H Mayer Fellowship at Fenway Health and R25DA058490.