Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2021 Sep 1.
Published in final edited form as: Curr Addict Rep. 2020 Jun 27;7(3):280–290. doi: 10.1007/s40429-020-00322-y

Developments in Mobile Health Just-in-Time Adaptive Interventions for Addiction Science

Stephanie M Carpenter 1,*, Marianne Menictas 2,*, Inbal Nahum-Shani 1, David W Wetter 3, Susan A Murphy 2
PMCID: PMC7968352  NIHMSID: NIHMS1607660  PMID: 33747711

Abstract

Purpose of Review

Addiction is a serious and prevalent problem across the globe. An important challenge facing intervention science is how to support addiction treatment and recovery while mitigating the associated cost and stigma. A promising solution is the use of mobile health (mHealth) just-in-time adaptive interventions (JITAIs), in which intervention options are delivered in situ via a mobile device when individuals are most in need.

Recent findings

The present review describes the use of mHealth JITAIs to support addiction treatment and recovery, and provides guidance on when and how the micro-randomized trial (MRT) can be used to optimize a JITAI. We describe the design of five mHealth JITAIs in addiction and three MRT studies, and discuss challenges and future directions.

Summary

This review aims to provide guidance for constructing effective JITAIs to support addiction treatment and recovery.

Keywords: Micro-Randomized Trial, Just-In-Time Adaptive Intervention, Mobile Health, Addiction

1. Introduction

Addiction is a serious and prevalent problem across the globe, spanning tobacco, alcohol, opioids, other drugs, and gambling behaviors. Developing new approaches to support formal treatment and recovery for addiction is critical given prominent barriers to seeking and receiving help, such as perceived stigma, cost, burden, and limited treatment availability [1]. Further, many triggers that lead to relapse occur outside of standard treatment settings and increased risk for lapse/relapse may occur frequently and rapidly [26]. Mobile health (mHealth), defined as the use of mobile and wireless technologies for health promotion [7], offers a promising approach for addressing these barriers.

mHealth tools, such as smartphone apps, text messaging, and interactive voice response are effective approaches for extending addiction treatment outside of the clinic [811]. Indeed, mobile apps hold promise as smartphone use has become increasingly ubiquitous, including among individuals with limited access to treatment as a way to reduce substance use [1214]. A key advantage of mHealth interventions is the potential to deliver efficacious strategies in response to rapid changes in an individual’s circumstances by identifying when, for whom, and to what extent an intervention is needed [15, 16]. Despite this promise, it can be challenging to administer mHealth interventions in the wild when individuals have limited time and attention [4, 5]. It is critical for scientists to develop interventions that support individuals in moments when they are most vulnerable to lapse (e.g., at the highest risk), as well as most receptive to (e.g., willing and able to use) an intervention [5, 17].

An emerging intervention design, the just-in-time adaptive intervention (JITAI), holds enormous potential for promoting health behavior change in real-time when individuals need it most. JITAIs leverage mobile technology to deliver the right type of support, at the right time based on ongoing information about the individual’s internal state and context. JITAIs are designed to address rapidly emerging vulnerabilities (e.g., high likelihood of drug use/lapse) and/or windows of opportunity for positive changes (e.g., convenient times to learn a new skill and/or build resilience), while attempting to minimize participant burden, disruptions and habituation.

Constructing JITAIs requires study designs and analytic methods that capitalize on the rich, fine-grained, temporal data that can be collected with mobile and wireless technology. To optimize JITAIs, researchers [18, 19] introduced the micro-randomized trial (MRT) design, in which each individual is randomized to intervention options at each of the many times when it might be effective to deliver an intervention. This trial design allows investigators to address scientific questions regarding: (1) the causal effect on near-time, proximal, outcomes of providing an intervention option, compared to an alternative (Table 1), and (2) whether this effect varies depending on an individual’s internal state and context. Data from an MRT can help researchers determine how and under what conditions intervention options should be delivered to optimize intervention effectiveness (Table 1).

Table 1:

Definition of JITAI terms [cf. 5] and components and outcomes of example JITAI

Definition of JITAI terms
Term Definition
Components Four components play an important role in the design of JITAIs: (1) decision points, (2) intervention options, (3) tailoring variables, and (4) decision rules [5].
Decision points Points in time in which an intervention decision can be made.
Intervention options Possible treatments/actions that could be delivered at any given decision point (e.g., coping strategies, advice, no intervention).
Tailoring variables Information about the individual's internal state (e.g., stress, affect) or context (e.g., location) that is used to adapt the intervention (i.e., decide when and how to intervene).
Decision rules Used to operationalize the adaptation of a JITAI by specifying which intervention option to offer, for whom, and when (e.g., under which internal states and contexts). The decision rules link the intervention options and tailoring variables in a systematic way.
Optimize The construction of an effective, efficient and scalable intervention [20]
Proximal Outcome The near-time, short-term goals of an intervention (e.g., reductions in daily substance use or stress, intervention engagement). Particularly relevant when conducting a study (e.g., MRT) to optimize a JITAI.
Distal Outcome The ultimate, longer-term goals of an intervention. Usually a primary clinical outcome (e.g., long-term abstinence).
Components and outcomes of example JITAI
Component/Outcome Description
Decision rule If (risk for lapse is high):
 Send prompt to engage in a brief self-regulatory activity
Else:
 No prompt
Decision points Every 2 hours
Intervention options Messages to encourage engagement in self-regulatory activities; no message
Tailoring variables Risk for lapse (e.g., may include a combination of emotions, cravings, stressors, cigarette availability, HRV, and location)
Proximal outcome Engagement in a self-regulatory activity following the decision point
Distal outcome Smoking abstinence

Table 2:

Components, outcomes, and results or study stage of each JITAI and MRT described in this review.

JITAI/MRT name Addiction Components / Outcomes Results/Analysis Stage

Just-In-Time-Adaptive-Interventions (JITAIs)

Q Sense Tobacco Decision Points: May occur every minute.
Tailoring variables: Entering or lingering in a high-risk location.
Intervention options: Delivery of coping strategies vs. no delivery.
Distal outcome: Smoking abstinence.
This feasibility pilot study (N = 15) revealed that both the application (56% received geofence-triggered support; 50% of the geofence-triggered support message notifications were tapped on within 30 minutes; 78.2% of delivered messages were rated using the 5-star rating system); and use of geolocation (collected by the context-aware Q Sense system in 97% of smoking reports with a mean accuracy of 31.6 (SD = 16.8) meters) were feasible. Participants also indicated that they saw value in the geofence-triggered support and did not indicate privacy concerns [30]

Smart-T Tobacco Decision Points: Followed self-monitoring assessment (i.e., random EMAs were delivered 4x a day; participants could also self-initiate EMAs when they felt an urge, were about to lapse, or had lapsed).
Tailoring variables: Risk level for smoking lapse (e.g., low risk, high risk, lapsed). Risk level was determined by and tailored to EMA responses to the following four lapse triggers: negative affect/stress, smoking urge, easy access to cigarettes, and motivation to quit. High risk level was determined by and tailored to the highest rated of these lapse triggers.
Intervention options: 1 week pre-quit: motivational messages. 2 weeks post-quit: Low risk – maintaining abstinence motivation and general cessation advice; High risk – ways to cope, tailored on the highest rated of the lapse triggers; Already lapsed – motivational messages to return to abstinence.
Distal outcome: Smoking abstinence.
Participants (N = 59) in the feasibility & acceptability study responded to 87% of the intervention assessments, 83% used the on-demand application features, and 20% remained abstinent twelve weeks post-quit. Thus, the study was found to be both feasible and acceptable [31].
Smart-T2 Participants (N = 81) were randomized to one of three arms in a pilot randomized controlled trial (RCT): (i) Smart-T2, (ii) NCI QuitGuide, and (iii) usual care. The NCI QuitGuide is a smartphone application that adheres to clinical practice guidelines and includes motivational messages and detailed medication information. Usual care was the standard in-person care provided by a cessation clinic. This pilot RCT lasted for 13 weeks (1 week pre-quit and 12 weeks post-quit). Smart-T2 performed as well as NCI QuitGuide and usual care. Abstinence rates were: Smart-T2 = 6/27, 22%; NCI QuitGuide = 4/27, 15%; and usual care = 4/27, 15%. The Smart-T2 group rated the app positively and indicated they believed that the app could help them stay quit [32].

A-CHESS Alcohol Decision points: May occur every minute.
Tailoring variable: Proximity to a high-risk location.
Intervention options: Delivery of alert vs. no alert.
Distal outcome: Alcohol abstinence.
Participants (N = 349) who met the criteria for DSM-IV alcohol dependence at five residential treatment programs were enrolled in the RCT and randomized to either a control (N = 179; treatment as usual) or A-CHESS (N = 170; treatment as usual + A-CHESS) group. Treatment as usual varied across the five residential treatment programs and none offered coordinated care after discharge. A-CHESS provided location tracking, self-monitoring, on-demand support information and services, and options for communication with peers and counselors. Surveys were administered at month 4, 8, and 12 asking participants to self-report their number of risky drinking days in the previous 30 days. Results indicated that during the 8 months of intervention and 4 months of follow-up, the A-CHESS intervention group self-reported significantly fewer risky drinking days than the control group (M = 1.39 vs. 2.75, respectively; p = .003) [9].

PHIT for duty Alcohol Decision Points: Biweekly.
Tailoring variables: Psychometric and psychophysiological data indicative of mild or moderate health risk.
Intervention options: Recommended activities to reduce symptoms and prevent disease (e.g., stress reduction, mindfulness meditation, progressive muscle relaxation, behavior therapy, health messaging, behavioral education in sleep quality and alcohol use, and links to professional care).
Distal outcome: Improved mental health, including reduced PTSD symptoms.
Although the PHIT framework is built to flexibly target substance use [34, 35], the PHIT for duty research described here focused on alcohol use [36]. Participants (N = 31) rated usability on a 1 (very hard) to 5 (very easy) scale and also completed the System Usability Scale (SUS) questionnaire (N = 9). Results indicated (mean ± SD) high overall usability of the PHIT (4.5 ± 0.6) application and relatively high usability of the pulse sensor (3.7 ± 1.2) and sleep monitor (4.4 ± 0.7), suggesting the application and sensors are relatively feasible and acceptable. Further, a comparison of PHIT-based mHealth assessments to traditional paper forms demonstrated a high correlation (r = 0.87) [36].

Micro-Randomized Trials (MRTs)

Sense2Stop Tobacco Decision Points: The minute at the peak of each stressed episode during a 12-hour day.
Intervention options: A smartphone prompt to engage in a digital stress-management intervention or to receive no prompt.
Proximal outcome: Subsequent two-hour likelihood of stress.
Distal outcome: Time to smoking relapse.
Study (N = 75) data analyses are currently underway [43].

SARA Alcohol & Drug Use Decision Points: Two per day (4pm and 6pm).
Intervention options: 4pm notification with an inspirational message vs. no message. 6pm reminder notification to complete survey (simple reminder vs. persuasive reminder) and two active tasks.
Proximal outcome: Whether the daily evening assessment is completed.
Distal outcome: Long-term adherence to assessment completion.
Study (N = 74) data analyses are currently underway [47]

MARS (orienting example) Tobacco Decision Points: 6 times per day (approx. two hours between each one).
Intervention options: A prompt recommending a low or high effort self-regulatory activity vs. no prompt.
Proximal outcome: Engagement in self-regulatory activities in the subsequent hour following a decision point.
Distal outcome: Smoking abstinence.
Data collection is about to begin [40].

The present review focuses on the use of mHealth JITAIs to support addiction treatment and recovery in the domains of tobacco, alcohol, and other drug use. We begin with an example JITAI and a review of existing JITAIs developed to treat these addictions. We then discuss when the use of mHealth MRT designs is and is not necessary, and describe examples of MRT designs that answer scientific questions about how and under what conditions intervention options should be delivered to optimize a JITAI.

2. JITAIs to Support Addiction Recovery

2.1. What is a JITAI?

JITAIs can address events and conditions that change rapidly and represent risk for adverse outcomes (e.g., stress, location). Because these conditions change outside of standard treatment settings, where multiple demands compete for an individual’s effort and attention, JITAIs should minimize disruptions to the daily lives and routines of individuals [4, 5]. JITAIs adapt intervention delivery to an individual’s internal state and context via decision rules (Table 1) that input the individual’s state and output whether and what type of intervention option to deliver. The individual’s current state includes data describing their real-time internal state (e.g., stress, motivation), real-world external context (e.g., weather, current location), and information about their past behavior and intervention response. At each of many predetermined times (decision points, such as every minute; Table 1), the individual’s state is observed and if conditions warrant (e.g., the individual is entering a high risk location) an intervention option is delivered (e.g., recommendation to leave the high risk location). Since these events and conditions are rapidly changing, ongoing monitoring of the individual is required to identify at which of the many possible times support is needed [15, 16].

For illustrative purposes, consider a relatively simple example of a JITAI that aims to address vulnerability to a lapse among smokers attempting to quit by increasing engagement with self-regulatory activities in real-time. Here, self-regulatory activities (e.g., breathing exercises, distraction) aid an individual in modulating their emotions, cravings, and behaviors [21, 22] in the service of a quit-attempt. This example JITAI is motivated by evidence suggesting that although self-regulatory activities can be useful in reducing negative emotions, cravings, and lapse rate [2327], individuals often fail to utilize these strategies when needed, in real time [28]. To help individuals better address real-time experiences and events that increase risk for lapse, during the 10-days post-quit participants are prompted 6 times per day to self-report key factors (e.g., emotions, craving, stressors, cigarette availability) via an ecological momentary assessment (EMA; i.e., brief survey). In addition, Heart Rate Variability (HRV) and location are measured throughout the day via wearable sensors and GPS. If this combined information indicates a moment of high risk for lapse (tailoring variable), then a prompt (intervention option) is sent via a mobile device, encouraging an individual to engage in a self-regulatory activity (e.g., consider going for a walk [29]). Otherwise, no prompt is delivered. The point in time when a decision is made about whether to deliver or not deliver a prompt is the decision point. The decision rule and other components outlined in Table 1 can be used to describe this JITAI.

We will now turn our discussion to a sample of existing JITAIs already under development.

2.2. Existing JITAIs

We provide example JITAIs in the following critical domains of addiction treatment: (1) tobacco use; (2) alcohol and drug use. Here, we combine alcohol and drug use because several, although not all, existing JITAIs target comorbid use or both alcohol and drug use separately. Please note that this review is not comprehensive, and other JITAIs exist that support addiction treatment. Additionally, because most JITAIs in the addiction field are in early stages of development, as is the case with these examples, not all have been evaluated for efficacy. Table 2 contains additional details about the components, outcomes, and results of the JITAIs described here.

2.2.1. Tobacco Use

Q Sense [30] is a mobile phone application for tobacco smoking cessation that senses an individual’s location and delivers behavioral support (coping strategies) triggered by and tailored to contextual features to prevent smoking lapse during a quit attempt. The first stage of this intervention entails a pre-quit assessment period that begins when the individual completes a demographic and smoking survey and indicates their desired quit date in Q Sense. Preliminary research has suggested an average of 19.8 days for the pre-quit assessment stage [30], at which time a high-risk location is identified (i.e., a place where the individual reports smoking more than four times during the pre-quit period). Each time an individual reports smoking, Q Sense administers an EMA to gather information on mood, stress, urge, current context (e.g., home, working, socializing), and whether others are present and also smoking. Identification of a high-risk location triggers the system to create a virtual perimeter (geofence) around that area. The combined EMA and location data trigger delivery of a tailored support or feedback message (e.g., “Based on 12 reports, 25% of the times you smoke you are working.” [30]). That is, the delivery of support is adapted to the individual’s current location and the content of the support is tailored based on the individual’s EMA responses (including self-reports of their current context). Individuals transition to the second stage on their desired quit date. During the second, 28-day post-quit period of the intervention, the application passively monitors the individual’s context throughout each day to detect whether they enter a high-risk location. If an individual enters and/or lingers for five minutes in a high-risk location, then Q Sense sends a notification to their mobile device. Tapping on this geofence-triggered notification delivers a support message (e.g., coping strategy) tailored to the information collected during the pre-quit assessment stage. For example, if an individual self-reports high stress levels when smoking at a specific high-risk location (e.g., home) during the pre-quit stage, then support messages with coping strategies for moderate-to-high stress will be delivered during the post-quit stage when the individual is at home. No message is delivered if the individual is not in a high-risk location. Further support messages are delivered every 3 hours that the individual remains in the high-risk location. During these first two stages, a tailored quitting preparation message or smoking fact is delivered each morning. Following the delivery of any support message, individuals can rate the message using a 5-star scale. At the end of each day, individuals fill out a short app-based survey on the number of daily cigarettes smoked, the strength and frequency of smoking urges, and abstinence self-efficacy. The third stage is a passive monitoring period without proactive support delivery. The distal outcome of this intervention is smoking abstinence. The intervention self-report assessments and geofencing were found to be feasible, acceptable, and reasonably reliable (Table 2).

Smart-T [31] is a mHealth cessation application developed for economically disadvantaged tobacco smokers during a quit attempt. Post-quit, this intervention utilizes four randomly delivered EMAs per day to collect information on urges, affect, stress, and cigarette availability and an end-of-day diary. Individuals can also initiate an EMA if they are experiencing an urge, are about to lapse, or have already lapsed. If the EMA responses indicate one of three specific levels of risk to lapse (e.g., low, high, or lapsed), then the person receives a tailored message promoting a specific strategy. Strategies include cessation advice and tips to those identified at either a low or high risk to lapse, and messages targeting abstinence motivation to those who have already lapsed. The Smart-T app also provides on-demand content, including direct access to a tobacco quitline and quit tips with smoking cessation advice, coping strategies, medication recommendations, and benefits of successfully quitting [31]. The distal outcome is smoking abstinence. Initial pilot testing indicated that use of the Smart-T application was feasible and acceptable [31]. The latest version of the application, Smart-T2 [32], was recently compared to NCI’s free QuitGuide application (containing motivational messages and information about medication) and standard of treatment (usual care: 6-weeks of in-person counseling) in a three-arm pilot randomized controlled trial (Table 2).

2.2.2. Alcohol and Drug Use

The Addiction-Comprehensive Health Enhancement Support System (A-CHESS), is a mHealth application designed to improve continuing care for alcohol use disorders [9]. A-CHESS includes both on-demand content (e.g., an individual can select an audio-guided relaxation when desired), as well as JITAI options. For instance, a global positioning system (GPS) technology tracks when an individual approaches a location that they pre-specified as a place where they regularly obtained and/or consumed alcohol in the past (e.g., a bar they frequented). If the individual approaches the high-risk location, then the GPS initiates the delivery of an A-CHESS alert to the individual asking if they want to be in that location; otherwise no alert is delivered [9]. Individuals about to relapse can also press a panic button to contact social support. A reduced Brief Addiction Monitoring (BAM) Index is administered each week to assess lifestyle balance, sleep quality, negative affect, and recent substance use and BAM feedback is displayed graphically in A-CHESS. Individuals can permit counselors to view check-in data and receive A-CHESS notifications if BAM completion is below threshold. Here [9], the distal outcome is alcohol abstinence. Results of a randomized controlled trial comparing patients enrolled in a control (treatment as usual) program to those in an A-CHESS (treatment as usual + A-CHESS) program suggested that A-CHESS is efficacious in reducing alcohol use (Table 2). Research is currently underway to extend A-CHESS for use in improving outcomes for drug use, including opioid dependence [33].

Based on the Personal Health Intervention Toolkit (PHIT) [34], a mobile app framework for personalized health intervention studies, the goal of PHIT for duty [35] is to build resilience in healthy U.S. military troops and prevent substance use and psychological health problems in high-risk personnel. PHIT for duty is delivered using a smartphone or tablet and incorporates optional nonintrusive physiological (e.g., pulse sensor for assessment of HRV) and behavioral (e.g., body motion/actigraphy, sleep monitor) sensors for health status monitoring. Biweekly psychometric health risk assessments monitor stress, depression, anger, anxiety, alcohol use, and sleep quality. This JITAI is an intelligent virtual advisor that analyzes the psychometric and physiological data using evidence-based rules and scripted processes that create a feedback loop tailoring the application to the individual. If the analysis indicates mild to moderate health risk, then the virtual advisor recommends tailored intervention options that include mindfulness-based activities and behavioral education (Table 2). The distal outcome of this intervention is improved mental health. Evaluations of usability and health assessment accuracy have shown promise for use in mHealth research [36] (Table 2).

The aforementioned JITAIs highlight ongoing developments in mobile sensing technology (e.g., geosensing in A-CHESS [9] and Q Sense [30]) that have made it possible to detect, in real-time, some moments when and where an intervention might need to be delivered. While sensors provide essential contextual information and are low-burden, other additional evidence (e.g., information on when to deliver an intervention based on an individual’s emotions, cravings, psychological states, and cognitive states) that might be used to increase JITAI effectiveness is still most reliably measured with self-report assessments.

Further, existing behavioral theories are often static (i.e., assume individual stability over time [5]) and this limits the extent to which these theories can guide the development of real-time, dynamic JITAIs [37, 38]. More experimentation may be needed to answer scientific questions about under which conditions it is optimal to deliver each intervention option as well as to further develop dynamic behavioral theory [5].

To answer scientific questions about how best to optimize JITAIs, as well as to provide data to further develop behavioral theory, the micro-randomized trial (MRT) [18, 19] was developed. We describe the MRT experimental design below.

3. MRTs for Optimization of a JITAI

An MRT is an increasingly popular experimental design in which participants may be randomized 100s or 1000s of times to different intervention options. Schematics of a selection of MRTs are available to guide future research [see 39]. Returning to the self-regulatory intervention introduced during the discussion of JITAIs, an MRT design can be used to inform the decision rules in constructing this JITAI. This is the case for the Mobile Assistance for Regulating Smoking (MARS) MRT [40].

MARS is a 10-day MRT designed to optimize the delivery of prompts recommending that individuals engage in self-regulatory activities. Participants are randomized 6 times per day, approximately every two hours, to one of three intervention options: (1) a prompt recommending a relatively high-effort self-regulatory activity (e.g., 3-minute meditation exercise), (2) a prompt recommending a relatively low effort self-regulatory activity (e.g., simple instruction for substitution activity), or (3) no message. Randomization probabilities are set so that an average of 3 prompts will be delivered each day (1.5 times the prompt will recommend a relatively high effort activity, and 1.5 times the prompt will recommend a relatively low effort activity). Lapse, Heart Rate Variability (HRV), and location will be measured via wearable sensors and GPS. In addition, one hour following each randomization, participants will receive an EMA to report engagement in self-regulatory activities in the last hour (primary proximal outcome). The EMA will also include questions about other key factors associated with lapse risk (e.g., emotions, cravings, cigarette availability, stressors). These factors will be used to investigate the conditions in which subsequent recommendations to engage in a self-regulatory activity increase proximal engagement in self-regulatory activities, as well as whether engagement is associated with changes in risk factors.

As in MARS, MRTs [cf. 18, 19] are designed to facilitate optimizing JITAIs. The following types of scientific questions are often considered when researchers are interested in optimizing a JITAI [cf. 41]:

  1. Is it worthwhile to deliver an intervention option? That is, does delivery of a prompt recommending a self-regulatory activity increase engagement in self-regulatory activities, as compared to no message, on average across all individual states and circumstances?

  2. Under what conditions is the delivery of an intervention option most beneficial? For instance, a prompt to engage in a self-regulatory activity may be especially helpful during times of craving, as compared to no message.

  3. Which type of intervention option is most beneficial? It may be that prompts recommending a low-effort self-regulatory activity are on average more likely to increase engagement than those recommending a high-effort activity, but that high-effort activities are particularly valuable for building resiliency.

  4. Under what conditions is one type of intervention option more beneficial than another? When considering the effectiveness of different prompts, it is important to consider their timing (e.g., what time of day is best to deliver high- versus low-effort self-regulatory recommendations?) and the context in which the intervention options are delivered (e.g., are smokers more likely to engage in low-effort than high-effort self-regulatory activities while at work?).

Once there is sufficient empirical and/or theoretical evidence to support the optimal delivery of intervention component(s) (see Table 1 for definition), then a researcher can evaluate the efficacy of this JITAI package in a randomized controlled trial comparing the JITAI to a different intervention package (e.g., standard of care; as in A-CHESS [9]). See [42] for further information on principles underlying optimization of bio-behavioral interventions.

3.1. Existing MRT Studies

We will now describe two MRTs that have already been conducted. This review is not comprehensive and additional MRTs exist in addiction science and other fields; see [39] for a subset of MRTs in use across other health fields. The selected MRTs are within the same areas of addiction described above: (1) tobacco use; (2) alcohol and drug use.

3.1.1. Tobacco Use

Sense2Stop [43] evaluated the feasibility of a JITAI aiming to decrease stress in tobacco smokers during a quit attempt. Participants were instructed to prepare to quit smoking on days 1–3 of the study, quit smoking on day 4, and attempt to abstain from smoking thereafter. Throughout the 10-day post-quit period, a machine learning algorithm [44, 45] randomly assigned a participant to either receive a smartphone prompt to engage in a digital stress-management intervention, or to receive no prompt. A second machine learning algorithm [46] provided real-time classifications of physiological stress based on physiological measurements (HRV, respiration) from sensors. Together the two algorithms were used to construct randomization probabilities. If a participant had not yet lapsed, then during episodes classified as stressed they received an average of 1.5 stress-management prompts per day. If a participant had already lapsed, then during episodes classified as stressed they received an average of 1 stress-management prompt per day. Here, the proximal outcome was the likelihood of stress in the subsequent two-hours, and the distal outcome was time to smoking relapse. Analysis addressing the primary aim for this MRT is currently underway (Table 2).

3.1.2. Alcohol and Drug Use

Substance Abuse Research Assistant (SARA) [47] is a mobile application aiming to increase and sustain engagement in an evening assessment of alcohol and marijuana use-related behaviors. The evening assessment involved one survey and two active tasks (to assess reaction time and spatial memory). The survey included questions related to that day’s mood, stress, loneliness, and hopefulness. This one-month MRT study consisted of participants aged 14–24 years, who reported binge drinking or marijuana use in the previous month. SARA provided a variety of engagement strategies to incentivize participants to complete the assessment. A base strategy involved a virtual aquarium that became richer and more complex as the participant completed assessments. A variety of other engagement strategies were randomized, including: (1) At 4 pm, participants were randomized at a 0.5 probability to either receive a push notification with an inspirational youth-focused message from a contemporary celebrity, or to receive no message, and (2) At 6 pm, participants were randomized at a 0.5 probability to receive one of two types of reminders (simple or persuasive reminder) to complete the assessment. The proximal outcome was completion of evening assessments, and the distal outcome was long-term adherence to study assessment completion. The primary analysis is currently underway (Table 2).

4. When do JITAIs require optimization via an MRT?

Not all JITAIs require optimization via an MRT. A researcher might not conduct an MRT study when empirical and/or theoretical evidence already exists to fully construct the JITAI decision rules. In other words, if existing evidence is sufficient to identify the decision points, tailoring variables and intervention options that would form an effective JITAI, then an MRT study would not be necessary. Another setting in which a researcher would not use an MRT is if the times at which support is needed are very rare, such as in the case of suicide attempts [cf. 48]. While suicide attempts are a significant problem among individuals seeking treatment for addictions, an MRT is unlikely to provide sufficient data to assess what are the best intervention options at times of suicide attempts. However, suicidal ideation tends to occur at a higher rate [cf. 48], and so it would be possible for an MRT to optimize a JITAI component for intervening during moments of suicidal ideation. Thus, researchers should decide if existing theoretical and empirical evidence is sufficient to develop an efficacious JITAI to support their population, whether the event of interest is suitable for an MRT design, and/or whether further empirical evidence is needed.

5. Challenges & Future Directions

Research on the development of JITAIs is in its early stages and more work is needed to understand how and under what conditions individuals are most likely to engage with recommended interventions. Multiple important challenges and future directions remain.

5.1. Challenges

It is critical to note that for all the advancements made in mobile data collection technologies (e.g., smartphones, wearables sensors), many existing devices are still maturing and currently lack the precision necessary to provide useful information about the conditions under which it is best to offer a specific intervention option. For example, many mobile technologies have technological limitations that can be misleading (e.g., indication of stress when the participant is exercising) or lead to data loss. Geolocation technologies can fail to provide accurate location information due to network connectively issues. Sensors developed to assist in the collection of smoking cessation, such as puffMarker [49], may record a puff when the hand gestures upward, regardless of whether or not the person is actually smoking. Further, mobile phone applications may freeze or crash, and all mHealth studies are susceptible to data loss if participants shut off or fail to charge their mobile devices and sensors. Data loss may also occur if participants find wearable sensors to be uncomfortable or burdensome and consequently stop wearing the devices. There are also limitations in the use of sensors, even when paired with machine learning, in the sensitivity and prediction of the presence of internal states like craving or stress [50]. Indeed, the detection rate of internal states by passive inputs is below 50%, and the prediction accuracy for future states is approximately 70–80% [50]. Thus, an important challenge is to provide meaningful interventions despite technological limitations.

A second challenge involves the selection and integration of empirical, theoretical, and practical evidence into dynamic scientific models to inform the construction of JITAIs that treat addiction [see 4]. Many existing behavioral models are static and developed to be tested within laboratory settings. Although the development and testing of theoretical models in laboratory settings is critical to the advancement of addiction science, to develop interventions that are efficacious in the real-world, scientific models must account for the dynamic, real-time factors that influence individuals in the field. This includes real-time emotions, cravings/urges, psychological states (e.g., motivations), availability of cognitive resources, burden, the availability of a substance, or contexts/people that remind an individual of the use of a particular substance. Development of these dynamic, theoretical models requires real-time assessment.

A third challenge involves intervention engagement. Even rigorously tested and theoretically grounded JITAIs for addiction will be ineffective if people do not engage in the intervention. Individuals who are not engaged may also fail to charge or consistently wear sensors, further contributing to the data loss challenges described above. One promising approach to increasing intervention engagement is to integrate strategies from behavioral science (e.g., persuasion, reward) that are likely, based on the extant theoretical and empirical evidence, to promote real-time engagement in interventions.

A fourth challenge involves ethical considerations such as privacy, confidentiality and safety of the individual. Many of the strengths of mHealth research (i.e., the ability to provide support to large and broad samples and continually collect streaming data on a range of sensitive behaviors) drive privacy and security concerns [5157]. Privacy is particularly critical when the data collected and stored on a mobile device involve information concerning illicit drug use or related activities. Researchers must find ways to encrypt data, password-protect software and devices, and provide individuals with protections from legal and other detrimental consequences [51, 53, 56]. mHealth data also contain rich geolocation, user activity, and sensor information that is difficult to de-identify due to unique fingerprints that devices leave in the data [51, 57]. As such, there must be specific plans in place to manage and summarize data in a way that maintains confidentiality. The ethics of intervening in a real-life setting may also lead to the decision to not provide support (even when support may be needed; e.g., a stress episode is detected while an individual is driving), or to modify the type of support provided [4, 58] (e.g., counselor calls an individual who reports suicidal intent) to protect individual safety. Due to the overall ethical challenge, it is critical that researchers develop JITAIs customized to address the unique privacy, confidentiality and safety considerations specific to their population.

5.2. Future Directions

Several promising future directions exist for the optimization of JITAIs for addiction treatment and recovery:

  1. The development of future dynamic models that will guide scientific questions about real-time, real-world addiction treatment and recovery to inform the design of MRTs and the construction of future JITAIs.

  2. The integration of engagement strategies that promote the use and efficacy of interventions.

  3. The development of new analytic methods that capitalize on the rich, fine-grained, temporal data available via mHealth technologies.

  4. More precision in the ability of JITAIs to detect when individuals are able, as well as unable, to benefit from the intervention (e.g., when they are most vulnerable to lapse, or highly motivated and able to maintain their addiction recovery without support from the intervention), and adapt to send more or fewer intervention strategies, depending on the individual’s current needs. Specifically, future JITAIs should be able to harness algorithms that accurately predict, in real-time, when an individual’s external contexts and internal physiological and psychological states signal vulnerability to (re)lapse so that delivery of the intervention can increase or decrease in intensity accordingly.

6. Conclusions

This review has discussed the promise of mHealth JITAIs for addiction treatment and recovery from tobacco, alcohol, and drug use. Although the use of JITAIs in addiction science is in its infancy, MRT experimental designs can help answer scientific questions that support the optimization of JITAIs. Through outlining an example JITAI and reviewing several already existing JITAIs and MRTs, we have illustrated the variety of topics and ways in which JITAIs can be optimized.

Although the current review has focused primarily on JITAIs designed to treat substance use, JITAIs are also promising for the treatment of other addictions, such as food addictions and gambling. Further research is needed to fully realize the potential of JITAIs to support treatment and recovery from a wide range of addictive behaviors. The present review aims to generate enthusiasm and provide guidance for constructing effective JITAIs for addiction treatment and recovery.

Acknowledgements

This work was supported in part by U.S National Institutes of Health grants R01AA23187 (NIH/NIAAA), P50DA039838 (NIH/NIDA), U01CA229437 (NIH/NCI), R01DA039901 (NIH/NIDA), P30CA042014 (NIH/NCI), UL1TR002538 (NIH/NCATS), and the Huntsman Cancer Foundation.

Footnotes

Conflict of Interest Stephanie M. Carpenter, Marianne Menictas, Inbal Nahum-Shani, David W. Wetter, and Susan A. Murphy declare that they have no conflicts of interest.

Compliance with Ethical Standards

Human and Animal Rights and Informed Consent All reported studies/experiments with human or animal subjects performed by the authors have been previously published and complied with all applicable ethical standards (including the Helsinki declaration and its amendments, institutional/national research committee standards, and international/national/institutional guidelines).

7 References

Papers of particular interest, published recently, have been highlighted as:

* Of importance

** Of major importance

  • 1.Duncan LG, Mendoza S, Hansen H. Buprenorphine maintenance for opioid dependence in public sector healthcare: benefits and barriers. Journal of addiction medicine and therapeutic science. 2015;1(2):31. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Gwaltney CJ, Shiffman S, Balabanis MH, Paty JA. Dynamic self-efficacy and outcome expectancies: prediction of smoking lapse and relapse. Journal of abnormal psychology. 2005. November;114(4):661. [DOI] [PubMed] [Google Scholar]
  • 3.Mezinskis JP, Honos-Webb L, Kropp F, Somoza E. The measurement of craving. Journal of addictive diseases. 2001. October 10;20(3):67–85. [DOI] [PubMed] [Google Scholar]
  • 4.Nahum-Shani I, Hekler EB, Spruijt-Metz D. Building health behavior models to guide the development of just-in-time adaptive interventions: a pragmatic framework. Health Psychology. 2015. December;34(S):1209. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Nahum-Shani I, Smith SN, Spring BJ, Collins LM, Witkiewitz K, Tewari A, Murphy SA. Just-in-time adaptive interventions (JITAIs) in mobile health: key components and design principles for ongoing health behavior support. Annals of Behavioral Medicine. 2018. June;52(6):446–62.** This is an important article as it lays out the framework that we use for JITAIs in mHealth to provide a better understanding of the implications of providing timely and ecologically sound support for intervention adherence and retention.
  • 6.Pickens RW, Johanson CE. Craving: consensus of status and agenda for future research. Drug and Alcohol Dependence. 1992. June 1;30(2):127–31. [DOI] [PubMed] [Google Scholar]
  • 7.World Health Organization. mHealth. Use of appropriate digital technologies for public health: report by Director-General. 71st World Health Assembly provisional agenda item. 2017. m 12: A71.* This important article provides us with our definition of mHealth.
  • 8.Whittaker R, McRobbie H, Bullen C, Rodgers A, Gu Y. Mobile phone-based interventions for smoking cessation. Cochrane Database of Systematic Reviews. 2016(4). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Gustafson DH, McTavish FM, Chih MY, Atwood AK, Johnson RA, Boyle MG, Levy MS, Driscoll H, Chisholm SM, Dillenburg L, Isham A. A smartphone application to support recovery from alcoholism: a randomized clinical trial. JAMA psychiatry. 2014. May 1;71(5):566–72. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Brendryen H, Kraft P. Happy Ending: a randomized controlled trial of a digital multi-media smoking cessation intervention. Addiction. 2008. Mar;103(3):478–84. [DOI] [PubMed] [Google Scholar]
  • 11.Tofighi B, Nicholson JM, McNeely J, Muench F, Lee JD. Mobile phone messaging for illicit drug and alcohol dependence: a systematic review of the literature. Drug and alcohol review. 2017. July;36(4):477–91. [DOI] [PubMed] [Google Scholar]
  • 12.Penzenstadler L, Chatton A, Van Singer M, Khazaal Y. Quality of smartphone apps related to alcohol use disorder. European addiction research. 2016;22(6):329–38. [DOI] [PubMed] [Google Scholar]
  • 13.Weaver ER, Horyniak DR, Jenkinson R, Dietze P, Lim MS. “Let’s get wasted!” and other apps: characteristics, acceptability, and use of alcohol-related smartphone applications. JMIR mHealth and uHealth. 2013;1(1):e9.. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Tofighi B, Leonard N, Greco P, Hadavand A, Acosta MC, Lee JD. Technology use patterns among patients enrolled in inpatient detoxification treatment. Journal of addiction medicine. 2019. July;13(4):279. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Furnari M, Epstein DH, Phillips KA, Jobes ML, Kowalczyk WJ, Vahabzadeh M, Lin JL, Preston KL. Some of the people, some of the time: field evidence for associations and dissociations between stress and drug use. Psychopharmacology. 2015. October 1;232(19):3529–37. [DOI] [PubMed] [Google Scholar]
  • 16.Epstein DH. Let’s agree to agree: a comment on Hogarth (2020), with a plea for not-so-competing theories of addiction. Neuropsychopharmacology. 2020. April;45(5):715–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Spruijt-Metz D, Wen CK, O’Reilly G, Li M, Lee S, Emken BA, Mitra U, Annavaram M, Ragusa G, Narayanan S. Innovations in the use of interactive technology to support weight management. Current obesity reports. 2015. December 1;4(4):510–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Klasnja P, Hekler EB, Shiffman S, Boruvka A, Almirall D, Tewari A, Murphy SA. Microrandomized trials: An experimental design for developing just-in-time adaptive interventions. Health Psychology. 2015. December;34(S):1220.** This important article provides justification for the use of MRTs in mHealth.
  • 19.Liao P, Klasnja P, Tewari A, Murphy SA. Sample size calculations for micro-randomized trials in mHealth. Statistics in medicine. 2016. May 30;35(12):1944–71. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Collins LM. Optimization of behavioral, biobehavioral, and biomedical interventions: the multiphase optimization strategy (MOST). Springer; 2018. February 8. [Google Scholar]
  • 21.Carver CS, Scheier MF. On the self-regulation of behavior. Cambridge University Press; 2001. May 7. [Google Scholar]
  • 22.Bandura A Social cognitive theory of self-regulation. Organizational behavior and human decision processes. 1991. December 1;50(2):248–87. [Google Scholar]
  • 23.Oikonomou MT, Arvanitis M, Sokolove RL. Mindfulness training for smoking cessation: a meta-analysis of randomized-controlled trials. Journal of health psychology. 2017. December;22(14):1841–50. [DOI] [PubMed] [Google Scholar]
  • 24.Davis JM, Fleming MF, Bonus KA, Baker TB. A pilot study on mindfulness based stress reduction for smokers. BMC complementary and alternative medicine. 2007. December;7(1):2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Maglione MA, Maher AR, Ewing B, Colaiaco B, Newberry S, Kandrack R, Shanman RM, Sorbero ME, Hempel S. Efficacy of mindfulness meditation for smoking cessation: A systematic review and meta-analysis. Addictive behaviors. 2017. June 1;69:27–34. [DOI] [PubMed] [Google Scholar]
  • 26.Cropley M, Ussher M, Charitou E. Acute effects of a guided relaxation routine (body scan) on tobacco withdrawal symptoms and cravings in abstinent smokers. Addiction. 2007. June;102(6):989–93. [DOI] [PubMed] [Google Scholar]
  • 27.Black DS. Mindfulness-based interventions: an antidote to suffering in the context of substance use, misuse, and addiction. Substance use & misuse. 2014. April 16;49(5):487–91. [DOI] [PubMed] [Google Scholar]
  • 28.Vettese LC, Toneatto T, Stea JN, Nguyen L, Wang JJ. Do mindfulness meditation participants do their homework? And does it make a difference? A review of the empirical evidence. Journal of Cognitive Psychotherapy. 2009. August 1;23(3):198–225. [Google Scholar]
  • 29.Thayer RE, Newman JR, McClain TM. Self-regulation of mood: strategies for changing a bad mood, raising energy, and reducing tension. Journal of Personality and Social Psychology, 1994;67(5), 910. [DOI] [PubMed] [Google Scholar]
  • 30.Naughton F, Hopewell S, Lathia N, Schalbroeck R, Brown C, Mascolo C, McEwen A, Sutton S. A context-sensing mobile phone app (Q sense) for smoking cessation: a mixed-methods study. JMIR mHealth and uHealth. 2016;4(3):e106. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Businelle MS, Ma P, Kendzor DE, Frank SG, Vidrine DJ, Wetter DW. An ecological momentary intervention for smoking cessation: evaluation of feasibility and effectiveness. Journal of medical Internet research. 2016;18(12):e321. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Hébert ET, Ra CK, Alexander AC, Helt A, Moisiuc R, Kendzor DE, Vidrine DJ, Funk-Lawler RK, Businelle MS. A Mobile Just-in-Time Adaptive Intervention for Smoking Cessation: Pilot Randomized Controlled Trial. Journal of Medical Internet Research. 2020;22(3):e16907.* This important article provides detail on a pilot RCT for evaluation of the Smart-T2 JITAI package. The results of this pilot RCT suggest that smartphone-based smoking cessation JITAIs may be capable of providing similar outcomes to traditional, in-person counselling.
  • 33.Gustafson DH, Landucci G, McTavish F, Kornfield R, Johnson RA, Mares ML, Westergaard RP, Quanbeck A, Alagoz E, Pe-Romashko K and Thomas C, 2016. The effect of bundling medication-assisted treatment for opioid addiction with mHealth: study protocol for a randomized clinical trial. Trials, 17(1), p.592. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Eckhoff RP, Kizakevich PN, Bakalov V, Zhang Y, Bryant SP, Hobbs MA. A platform to build mobile health apps: the Personal Health Intervention Toolkit (PHIT). JMIR mHealth and uHealth. 2015;3(2):e46. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Kizakevich PN, Hubal RC, Brown J, Lyden J, Spira JL, Eckhoff R, Zhang Y, Bryant S, Munoz G. PHIT for duty, a mobile approach for psychological health intervention. Annual Review of Cybertherapy and Telemedicine. 2012. September 14;181:268–72. [PubMed] [Google Scholar]
  • 36.Kizakevich PN, Eckhoff R, Brown J, Tueller SJ, Weimer B, Bell S, Weeks A, Hourani LL, Spira JL, King LA. PHIT for duty, a mobile application for stress reduction, sleep improvement, and alcohol moderation. Military medicine. 2018. March 1;183(suppl_1):353–63.* This important article provides details on the evaluations of usability and health assessment accuracy for the PHIT for Duty JITAI package.
  • 37.Riley WT, Rivera DE, Atienza AA, Nilsen W, Allison SM, Mermelstein R. Health behavior models in the age of mobile interventions: are our theories up to the task?. Translational behavioral medicine. 2011. March 1;1(1):53–71. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Spruijt-Metz D, Nilsen W. Dynamic models of behavior for just-in-time adaptive interventions. IEEE Pervasive Computing. 2014. July 9;13(3):13–7. [Google Scholar]
  • 39.Murphy SA, Harvard University. Micro-Randomized Trials. Available from: http://people.seas.harvard.edu/~samurphy/JITAI_MRT/mrts4.html [30 March 2020]** This important website gives examples of MRT schematics.
  • 40.Nahum-Shani B, Wetter DW Novel use of mHealth data to identify states of vulnerability and receptivity to JITAIs. 2018. Grant Funded by NIH/NCI U01 CA229437.** This funded grant provides important detail on MARS, our orienting example for MRTs.
  • 41.Walton A, Nahum-Shani I, Crosby L, Klasnja P, Murphy S. Optimizing digital integrated care via micro-randomized trials. Clinical Pharmacology & Therapeutics. 2018. July;104(1):53–8.** This important article provides justification for the use of MRTs in mHealth.
  • 42.Collins LM. Optimization of behavioral, biobehavioral, and biomedical interventions: the multiphase optimization strategy (MOST). Springer; 2018. February 8. [Google Scholar]
  • 43.Spring B (2017). Sense2stop: Mobile sensor data to knowledge. In: ClinicalTrials.gov [Internet]. Bethesda (MD): National Library of Medicine (US). 2000- [cited 2019 May 10]. Available at https://clinicaltrials.gov/ct2/show/nct03184389.* This important website provides detail on the Sense2Stop clinical trial, a phase of which includes the Sense2Stop MRT described above.
  • 44.Dempsey W, Liao P, Kumar S, Murphy SA. The stratified micro-randomized trial design: sample size considerations for testing nested causal effects of time-varying treatments. arXiv preprint arXiv:1711.03587. 2017. November 9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Liao P, Dempsey W, Sarker H, Hossain SM, Al’Absi M, Klasnja P, Murphy S. Just-in-time but not too much: Determining treatment timing in mobile health. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies. 2018. December 27;2(4):1–21. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Hovsepian K, Al’Absi M, Ertin E, Kamarck T, Nakajima M, Kumar S. cStress: towards a gold standard for continuous stress assessment in the mobile environment. In Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing 2015 Sep 7 (pp. 493–504). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Rabbi M, Philyaw-Kotov M, Lee J, Mansour A, Dent L, Wang X, Cunningham R, Bonar E, Nahum-Shani I, Klasnja P, Walton M. SARA: a mobile app to engage users in health data collection. InProceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers 2017 Sep 11 (pp. 781–789).* This important article provides further detail on the Alcohol and Drug Use MRT, SARA, described above.
  • 48.Yuodelis-Flores C, Ries RK. Addiction and suicide: a review. The American journal on addictions. 2015. March;24(2):98–104. [DOI] [PubMed] [Google Scholar]
  • 49.Saleheen N, Ali AA, Hossain SM, Sarker H, Chatterjee S, Marlin B, Ertin E, Al’Absi M, Kumar S. puffMarker: a multi-sensor approach for pinpointing the timing of first lapse in smoking cessation. InProceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing 2015 Sep 7 (pp. 999–1010). [PMC free article] [PubMed] [Google Scholar]
  • 50.Epstein DH, Tyburski M, Kowalczyk WJ, Burgess-Hull AJ, Phillips KA, Curtis BL, Preston KL. Prediction of stress and drug craving ninety minutes in the future with passively collected GPS data. NPJ Digital Medicine. 2020. March 4;3(1):1–2.* This important article discusses current limitations to the use of sensor technologies.
  • 51.Arellano AM, Dai W, Wang S, Jiang X, Ohno-Machado L. Privacy Policy and Technology in Biomedical Data Science. Annual review of biomedical data science. 2018. July 20;1:115–29. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 52.Dehling T, Gao F, Schneider S, Sunyaev A. Exploring the far side of mobile health: information security and privacy of mobile health apps on iOS and Android. JMIR Mhealth Uhealth. 2015; 3 (1): e8. doi: 10.2196/mhealth.3672. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 53.Papageorgiou A, Strigkos M, Politou E, Alepis E, Solanas A, Patsakis C. Security and privacy analysis of mobile health applications: the alarming state of practice. IEEE Access. 2018. January 29;6:9390–403. [Google Scholar]
  • 54.SageBionetworks. Privacy Toolkit for Mobile Health, 2019. https://designmanual.sagebionetworks.org/privacy_toolkit.html (accessed on 09/07/2019).
  • 55.Huckvale K, Prieto JT, Tilney M, Benghozi PJ, Car J. Unaddressed privacy risks in accredited health and wellness apps: a cross-sectional systematic assessment. BMC medicine. 2015. December;13(1):214. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 56.Martínez-Pérez B, De La Torre-Díez I, López-Coronado M. Privacy and security in mobile health apps: a review and recommendations. Journal of medical systems. 2015. January 1;39(1):181. [DOI] [PubMed] [Google Scholar]
  • 57.Bojinov H, Michalevsky Y, Nakibly G, Boneh D. Mobile device identification via sensor fingerprinting. arXiv preprint arXiv:1408.1416. 2014. August 6. [Google Scholar]
  • 58.Kjeldskov J, Skov MB, Als BS, Høegh RT. Mobile Human-Computer Interaction-Mobile HCI 2004. Berlin: Springer Heidelberg; 2004. Is it worth the hassle? Exploring the added value of evaluating the usability of context-aware mobile systems in the field; pp. 61–73. [Google Scholar]

RESOURCES