Practical considerations when designing an online learning algorithm for an app-based mHealth intervention

Abstract

The ubiquitous nature of mobile health (mHealth) technology has expanded opportunities for the integration of reinforcement learning into traditional clinical trial designs, allowing researchers to learn individualized treatment policies during the study. LOWSALT4LIFE 2 (LS4L2) is a recent trial aimed at reducing sodium intake among hypertensive individuals through an app-based intervention. A reinforcement learning algorithm, which was deployed in one of the trial arms, was designed to send reminder notifications to promote app engagement in contexts where the notification would be effective, i.e., when a participant was likely to open the app in the next 30-minutes and not when prior data suggested reduced effectiveness. Such an algorithm can improve app-based mHealth interventions by reducing participant burden and more effectively promoting behavior change. We encountered various challenges during the implementation of the learning algorithm, which we present as a template to solving challenges in future trials that deploy reinforcement learning algorithms. We provide template solutions based on LS4L2 for solving the key challenges of (i) defining a relevant reward, (ii) determining a meaningful timescale for optimization, (iii) specifying a robust statistical model that allows for automation, (iv) balancing model flexibility with computational cost, and (v) addressing missing values in gradually collected data.

1. Introduction

The ubiquitous nature of mobile health (mHealth) technology has enabled the integration of digital tools into clinical trials in a wide variety of domains, from substance use and mental health studies to cardiovascular disease and physical activity [7, 12, 17]. These technologies not only enable high-frequency, low-burden data collection, but also help overcome obstacles in the delivery of care, making it possible to deliver behavioral interventions and provide accessible support to individuals anytime, anywhere. The primary goal of many digital intervention trials [11] is to assess whether and when it is useful to trigger delivery of behavioral interventions. These trials allow clinicians and behavioral scientists to build just-in-time adaptive interventions (JITAIs) that trigger behavioral interventions in the right moment and at the right time when individuals are most in need of support [16].

To determine when to deliver interventions, early digital intervention trials relied on treatment rules defined at baseline using relevant clinical knowledge [3, 4]. While such digital intervention trials can lead to more effective intervention packages for future patients, more recent digital intervention trials [1, 14] have gone beyond rule-based protocols and focused on benefiting patients enrolled in the trial by learning personalized intervention policies, i.e., the chosen intervention at a given time may depend on the individual’s past responses in contexts similar to their current context. Building personalized intervention policies requires online reinforcement learning (RL) algorithms [20] that take as input streaming, real-time data and output a sequence of intervention rules that update throughout the trial. These algorithms typically seek to optimize a measured outcome called a reward and try to balance exploration—the goal of exploring potential interventions that give high reward in a given context—and exploitation—the goal of choosing the optimal intervention that maximizes the reward in a given context.

The most common online RL learning algorithms deployed in digital intervention trials are contextual bandit algorithms [13]. These are an example of a myopic online RL algorithm, which takes as input the current context and outputs an intervention from the set of possible intervention options. Canonical bandit algorithms include ϵ-greedy, Upper Confidence Band, and Thompson Sampling [13]. Thompson Sampling is the most common approach in digital intervention trials due to its stochastic nature. Thompson Sampling maintains a probability distribution for the expected reward of each intervention option and, at each step, samples from these distributions to select the intervention with the highest sampled value. The probabilistic algorithm underlying Thompson Sampling (with minor modifications) ensures clinicians and behavioral scientists can perform statistical inference post-study [10, 22], allowing assessment of whether and when it was useful to trigger delivery of behavioral interventions.

In this paper, we present a digital intervention trial that uses a contextual bandit algorithm to optimize app-based notifications aimed at addressing high dietary sodium intake among individuals with hypertension. When designing the statistical algorithm embedded in the digital intervention trial, the study team had to balance statistical complexity and computational cost—a task that required making many decisions motivated by practical concerns. We present these challenges and our solutions as a template to solving challenges in future trials that deploy online RL algorithms.

1.1. Related work

In tandem with the growing application of contextual bandits in mHealth, researchers have considered best practices for designing and implementing trials that incorporate such components into their protocols. We mention a few works providing case-studies and guidance here. Figueroa et al. [6] addressed challenges that emerged throughout the execution of the DIAMANTE study, an randomized controlled trial (RCT) that aims to increase physical activity in participants through text messaging interventions. One arm of the three-arm trial employs RL techniques to personalize message delivery. Our current work builds on their list of challenges by focusing on statistical concerns in designing online RL algorithms. A more general discussion of frameworks for RL algorithm development in mHealth is provided by Trella et al.[21]. Their approach has three pillars: personalization, the degree to which an algorithm can learn to take the best action for each user; computability, the ability of an algorithm to run efficiently online; and stability, the capacity of an algorithm to operate mostly autonomously and in a wide variety of situations. Our current work builds on their general framework by diving into the technical tradeoffs of these three pillars.

1.2. Outline

Researchers attempting to implement a contextual bandit in a digital intervention trial will encounter various challenges, which we categorize into five general areas: (i) how to define a relevant reward, (ii) how to determine the timescale of optimization, (iii) how to specify a robust statistical model that allows for automation, (iv) how to balance model flexibility with computational cost, and (v) how to address missing values present in data that are collected gradually. Here, we describe important factors to consider when developing solutions to these challenges and outline pragmatic solutions to these general practical problems using LOWSALT4LIFE 2 as a running example. This paper aims to serve as a template for thinking about these potential challenges, which we believe will be encountered in similar studies involving contextual bandit algorithms in mHealth. We present the motivating trial in Section 2, describe commonly encountered practical issues in Section 3, illustrate our pragmatic solution in Section 4 through simulation, and conclude with a discussion in Section 5.

2. Motivating trial design: LOWSALT4LIFE 2

The primary objective of the LOWSALT4LIFE 2 (LS4L2) trial is to evaluate whether the inclusion of a JITAI within a mobile application (App + JITAI) reduces dietary sodium intake more than the mobile application alone (App Alone) among hypertension patients. This trial is a two-arm RCT in which adults currently being treated for hypertension are randomized with equal probability to receive or not receive a JITAI through a mobile app. The JITAI is triggered based on arrival at a grocery store or restaurant and takes as input the current context of the participant. During the first 2 months in the study, participants are then randomized to receive an app-notification with probability 80%, and not receive an app-notification with probability 20%. The notifications aim to increase proximal app engagement. At 2 months, participants in the JITAI arm are randomized again with equal probability to either continue with the current JITAI or transition to a personalized JITAI (pJITAI). A diagram of the trial is given in Figure 1.

Figure 1:

LS4L2 is a two-arm (App Alone vs App + JITAI) trial. After two months, participants in the App + JITAI arm undergo a second randomization to either continue with the current JITAI or transition to a personalized JITAI (pJITAI). The App + pJITAI sub-arm includes a contextual bandit algorithm to optimize delivery of an app-based notification. Within this sub-arm, participants are randomized to potentially receive a mobile notification from the study’s app each time they arrive at a grocery store or restaurant. The randomization probabilities are based on context covariates, all of which are collected passively.

Details of the LS4L2 protocol have been published previously [5]. Enrollment and follow-up for all 410 study participants in the LS4L2 trial is complete and the study is currently undergoing primary analysis. Of the 410 participants, 206 were assigned to the LS4L App + JITAI study arm. For the purposes of this paper, we focus on the development of the contextual bandit algorithm used only in the App + pJITAI arm of the trial in which participants receive the pJITAI.

Since the US population tends to consume most of its dietary sodium from processed foods and foods eaten at restaurants [2], the JITAI was designed to target participants when they arrive at a grocery store or restaurant by using these events as decision points within the App + JITAI trial arm. Arrival at either location is passively-sensed using geo-fencing and GPS. We refer to these times as decision times. At decision times, the mobile phone provides the following passively-collected context variables: the time of the week (e.g., weekend, weekday, holiday), time of the day (e.g., morning, afternoon, etc.), current situation as coded by the mobile system (e.g., shopping, social, working out, etc.), weather (e.g., very cold, cold, warm, etc.), and past app engagement (1 if the participant has opened the app in the past week and 0 otherwise). In LS4L2, all measured context variables are categorical. Within the App+pJITAI sub-arm, the Thompson sampling algorithm consists of two components. First, at every decision time t and for every participant i in the study, there is a known probabilistic mapping π_i,t : 𝒳 → [0, 1] that takes as input the current context X_i,t ∈ 𝒳 for that participant (i.e., set of current categorical variables) and outputs a probability of sending an app-based notification, π_i,t(X_i,t). Second, there is a learning algorithm that takes the current collected data as input and learns the probabilistic mapping. The goal of the Thompson sampling algorithm is to optimize delivery of the notifications to be at times when the participant is more likely than not to engage with the study app. In LS4L2, we use a type of logistic regression to learn the mapping from a given context to the context-specific probability of sending a notification. We will discuss each subcomponent of the algorithm in detail in the sections below.

3. Commonly encountered practical issues

In this section, we present practical issues encountered when designing the contextual bandit algorithm used to optimize the JITAI within the App+pJITAI arm in the LS4L2 trial.

3.1. Constructing a relevant reward for the learning algorithm

RL algorithms, such as contextual bandit algorithms, require a pre-specified reward. These algorithms then use this reward as a measurable signal to decide in which contexts it is appropriate to provide an intervention or stimulus (e.g., app-based notification) and in which contexts it is not. From the perspective of designing interventions for behavior change, the scientific goal is to use a stimulus (e.g., notification from the app) to encourage participants to engage in a certain behavior (e.g., thinking about dietary sodium consumption and choosing low sodium alternatives). In mHealth, measuring the desired behavior moments after the Thompson Sampling algorithm makes a decision is difficult. While tools for real-time measurements such as ecological momentary assessments (EMAs) [19] have been used to collect in-the-moment data, the ability to passively detect the desired behaviors is often either impossible or not yet available due to current limitations in sensing technologies. In LS4L2, the scientific team decided that EMAs had three disadvantages: (1) self-report was too high burden, (2) EMA could result in a high degree of missingness, and (3) self-reported behavior may not be a good measure of underlying behavior. In LS4L2, the scientific team decided EMA was too burdensome and inaccurate, while passive measurements were not sufficiently reliable.

The inability to measure the task of interest is a common problem in digital intervention trials. A common issue in the development of online RL algorithms for behavior change is the conflation of the algorithm’s reward and the behavioral task of interest. We solve this practical challenge by operationalizing a relevant, passively measurable reward that serves as a suitable proxy for the desired behavior. By viewing the reward as a proxy, the study team can assess its suitability for learning good policies aimed at getting participants to engage in the behavior of interest. In LS4L2, the scientific team decided that a binary outcome based on whether or not the user engages with the app in the 30-minute window following the decision time is a suitable proxy for our proximal behavioral outcome of true interest—whether or not someone makes low sodium choices. Interacting with the app may include searching for low-salt foods at a given restaurant or looking up alternative low-salt options when purchasing food at a grocery store. The choice of a 30-minute window attempts to strike a balance between signal and noise. That is, we aim to allow enough time to detect the participant’s possible interaction with the app after arriving at a grocery store or restaurant (i.e., after the decision point) while minimizing potential noise included in our measurement of the reward if we were to wait an extended period of time to observe a potential app interaction.

It is critical for scientific teams designing bandit algorithms to think about whether the reward is a suitable proxy for the behavior of interest. In the first few months after study enrollment, thinking of and choosing low sodium alternatives was not thought to be part of the participant’s routine when entering grocery stores or restaurants. Therefore, proximal app engagement was seen to be a good proxy for whether the participant practiced the behavior of interest. In the final month of the study, if the intervention was successful at helping a participant achieve behavior change, that participant may be less likely to use the app and may successfully practice the behavior without app support. In this case, the notification may still serve as a reminder and be helpful in maintaining the desired behavior; however, proximal app engagement will no longer be a good proxy. For this reason, this choice of reward may lead the RL algorithm to send fewer messages than desired in the later stages of the study. Due to habituation and burden, the study team found this a suitable tradeoff. After LS4L2 is completed, a critical analysis will try to better understand whether the proxy reward was suitable and how to improve our choice for future studies. Our solution to the challenge of identifying a relevant reward by disentangling the reward from the stimulus/behavior framework makes such analyses possible. It should be noted that many behaviors have important, hypothesized downstream impacts on measurable intermediate and distal outcomes. We recommend the scientific team assess the impact of the bandit algorithm on these outcomes to assess suitability for future deployments.

3.2. Determining a meaningful timescale for optimization

Because participants are enrolled in the study sequentially, the amount of potential information available to the learning algorithm gradually increases as the study progresses. As a result, little information will be available to the learning algorithm for the first few participants enrolled in the study, but a larger information base will be available for participants enrolled later in the study. A key consideration in designing a contextual bandit then becomes the appropriate timescale for the learning algorithm. In other words, how often is it appropriate to update our mapping π from the contextual environment to the randomization probability for each study participant? Our process for making these determinations is described below.

In LS4L2, we identified two considerations that guided this choice. First, the reward and decision point timescales can inform the appropriate learning timescale. Based on historical data, we expected only a few (2–4) decision points every week per individual. Since the reward was binary, we wanted to ensure sufficient data so that the learning algorithm would construct probabilistic maps that were different from the previous ones. Based on this, we decided that monthly updates were the maximum update rate. In other settings, daily or weekly updates may be more appropriate when the reward is continuous and the number of decision points more numerous. In these settings, it is worth using historical data to determine the accrual rate of information which will inform the minimum learning timescale. Second, we needed to consider the expected monetary cost of computation per participant in the study. Fitting the hierarchical Bayesian logistic regression model that operates behind the scenes in our learning algorithm can require substantial computing power, so we must consider what the total cost will be if we must fit the model multiple times to each participant. Although we might expect a significant behavior change at the month-by-month level, doing so would be too expensive given the total number of participants in the study. Conducting the first update after two months facilitates exploration and accumulation of data about a binary behavior (i.e., opening the study app) observed at relatively infrequent decision points (i.e., a few times per week). The scientific team decided that two months is a sufficient time period in which we expect to observe any potential meaningful behavior changes, such as habituation to app notifications. Thus, to balance the gradual accumulation of potential information useful for optimization with the computational cost of carrying out an update to the delivery rules, the learning timescale was set to every 2 months so that the learning algorithm was run at 2 and 4 months after recruitment for every participant to update their probabilistic maps. This choice of learning timescale in LowSalt4Life is fast enough to ensure rapid convergence to a good intervention policy per participant but slow enough to ensure computational costs were not exorbitant. In general, this depends heavily on the underlying algorithms and their performance. In more recent work, we have adopted approximate hierarchical Bayesian Thompson Sampling algorithms that are faster [10]; however, partially pooling causes large computational costs even for approximate methods. Thus, even when learning rates could be daily, in larger trials with partial pooling weekly updates are often necessary to balance computational costs.

3.3. Building a robust, automated learning algorithm

Contextual bandit algorithms enable personalized probabilistic maps to be built over time during the course of the study. In clinical trials, it is imperative that the learning algorithm is feasibly deployed in practice, so that repeated updates of the learning algorithm (in months 2 and 4 since recruitment for every participant) neither “break” (i.e., fail to construct a new probabilistic map) nor require monitoring/maintenance that was not part of the study protocol. In LS4L2, the learning algorithm consists of fitting a hierarchical Bayesian logistic regression model with both population-level and individual-level parameters and reporting summaries based on the resulting posterior distributions. In LS4L2, we define a model as “breaking” if the Bayesian regression either fails to run (i.e., any error resulting in no posterior draws) or does not produce reasonable posterior distributions (i.e., issues with mixing of MCMC chains).

To make our algorithm robust when we have both small amounts of data (i.e., at the start of the study, since data are collected sequentially) and higher order interaction terms (which may be important predictors of response to a notification), we use a combination of partial pooling and shrinkage priors. Specifically, our Bayesian logistic regression model is designed to be hierarchical (i.e., includes patient-specific parameters), enabling partial pooling of information. Partial pooling allows the probabilistic map for a participant to start at a population-level mapping when there are few participant-level data and become more personalized through the inclusion of participant-specific parameters as data accumulate. Even with partial pooling, a common issue for Bayesian logistic regression is separation, which occurs when a context variable perfectly predicts the binary reward. To ensure model stability and avoid issues of separation, we employed weakly informative default priors [9] that are designed to have higher degrees of shrinkage as the complexity of the term increases. Specifically, the prior on higher order interaction terms are designed to have a significantly larger amount of shrinkage towards zero than main effects. In LS4L2, the same shrinkage priors are used for both population-level and individual-level model parameters. When deploying automated bandit algorithms, it is also important to also implement failure monitoring systems and statistical monitoring systems. We discuss such systems in the context of LS4L2 in Appendix Section A.1.

3.4. Balancing model flexibility and computational cost

In the contextual bandit literature, it is common to assume correct model specification. In practice, the model form is not known, which leads to the natural question of how to balance the complexity of the model with the amount of information currently available. One model may be too simple and ignore contextual moderators; however, another model may be too complex and lead to slower learning of important contextual moderators. Complex models may also suffer from instability early in the study when little data has been collected. To directly address this trade-off, the LS4L2 study team decided that model specification would take into account observed variation in the categorical variables while respecting an upper limit on the number of interaction terms in the model. Figure 2 visualizes this process for the context variables, showing how the study team balanced model complexity and computational cost with the amount of information available in the data.

Figure 2:

Determining model complexity for the single participant model. Assume that the data for the current participant contains p covariates (excluding intervention), which we index by j. We apply these rules to determine inclusion of fixed effects. Note that the number of “unknown” observations for each covariate are not counted towards the minimum cell size. For inclusion of random effect, a similar set of rules is used by only main effects and first-order interactions between intervention and covariates are considered (no three-way interactions). Different rules are applied for inclusion of baseline covariates.

In addition to intervention personalization with respect to context variables, LS4L2 accounts for potential differences in responsiveness to notifications over the course of the study (e.g., habituation) by incorporating an indicator variable for each study period. These study period binary indicators are denoted S₁ and S₂, and “turn on” after the 2- and 4-month updates respectively (i.e, S₁ = I(daysInStudy > 56) and S₂ = I(daysInStudy > 112)). In addition to Figure 2, the LS4L2 team defined rules for including study period-specific interaction terms using the rules as described above but within each study period. That is, the first round applies the rules for including model terms to all the data, the second round applies the rules for including model terms to only data collected after the 2 month optimization point (i.e., when S₁ = 1)), and (3) the third round applies the rules for including model terms to only data collected after the 4 month optimization point (i.e., when S₂ = 1)). All terms added to the model in the second and third rounds are interacted with S₁ and S₂, respectively. For baseline categorical covariates (i.,e., age and gender), we apply a slightly different set of rules to determine their inclusion. These rules—for including both main effects and interaction terms—are based on the variability in patient age and gender (See Appendix section A.2 for more details).

3.5. Addressing missing values in gradually collected data

Missing data are commonplace in clinical trials but present an additional challenge when designing online RL algorithms [15]. Distinct challenges arise in both subroutines of the RL algorithm (i.e., probabilistic mapping and learning algorithm). Recall that the learning algorithm requires fitting a Bayesian hierarchical logistic regression model. In LS4L2, the reward and intervention are always observed, but certain categorical contextual variables may be missing. For contextual variables with missing data, the LS4L2 team decided to augment the existing categories with a new level termed “Unknown”. Model fitting via MCMC is then performed with this strategy. For the probabilistic mappings, a key challenge that arises with categorical context variables is that there will be contexts that have never been observed for which we may need to generate a probabilistic mapping. In this case, the in-the-moment decision making algorithm may request a notification probability for a context where the probabilistic map cannot be built using the regression model fit by the learning algorithm. In LS4L2, we break scenarios into two categories:

• Scenario 1 (missing data scenario) is when the context variable is measured as “unknown”, but the level “unknown” for that variable is not included in the data used to fit the model.

• Scenario 2 (new context prediction scenario) is when the context variable is measured as any level except “unknown”, but the measured level is not included in the data used to fit Bayesian hierarchical logistic regression.

Both settings require rules for propagating the updated intervention probabilities from the fitted Bayesian hierarchical logistic regression to the scenarios described in Appendix Section A.3.

4. Simulation Study

4.1. Simulation Design

We study our LS4L2 algorithm and its performance relative to two alternative contextual bandit learning algorithms that were considered when designing the LS4L2 clinical trial. Both alternatives utilize Bayesian logistic regression models to update the probabilistic mapping but differ in their degree of model complexity. As Let X = (X₁,…,X_p) denote the set of context variables and let A be a binary variable that indicates whether or not a participant received the app-based intervention. The first algorithm, referred to as “Simple”, relies on the model shown in Equation (1) with main effects for the intervention indicator and context covariates only. The second, referred to as “Complicated”, estimates the probability that the app is opened using the maximal model in the LS4L2 learning algorithm (see Equation (2)).

Equation 1:

Logistic regression model for learning the probability of opening the study app at each decision point when the study participant did/did not receive the intervention in a specific context in the “Simple” learning algorithm.

$$\begin{gathered} \text{logit}\left(p_i\mid A_i,X_i\right)={\beta }_0+{\beta }_1{A}_i+\sum _{j=1}^p{\beta }_{2j}{X}_{ij} \\ +\left[{\beta }_0^{\left(1\right)}+{\beta }_1^{\left(1\right)}{A}_i+\sum _{j=1}^p{\beta }_{2j}^{\left(1\right)}{X}_{ij}\right]\times S_{1i} \\ +\left[{\beta }_0^{\left(2\right)}+{\beta }_1^{\left(2\right)}{A}_i+\sum _{j=1}^p{\beta }_{2j}^{\left(2\right)}{X}_{ij}\right]\times S_{2i} \end{gathered}$$ (1)

Equation 2:

Logistic regression model for learning the probability of opening the study app at each decision point when the study participant did/did not receive the intervention in a specific context in the “Complicated” learning algorithm.

$$\begin{gathered} \mathit{\text{logit}}\left(p_i\mid A_i,X_i\right)={\beta }_0+{\beta }_1{A}_i+\sum _{j=1}^p{\beta }_{2j}{X}_{ij}+\sum _{j=1}^p\sum _{j^{\prime }=j+1}^p{\beta }_{3j,j^{\prime }}{X}_{ij}{X}_{i{j}^{\prime }}+\sum _{j=1}^p{\beta }_{4j}{A}_i{X}_{ij}+\sum _{j=1}^p\sum _{j^{\prime }=j+1}^p{\beta }_{5j,j^{\prime }}{A}_i{X}_{ij}{X}_{i{j}^{\prime }} \\ +\left[{\beta }_0^{\left(1\right)}+{\beta }_1^{\left(1\right)}{A}_i+\sum _{j=1}^p{\beta }_{2j}^{\left(1\right)}{X}_{ij}+\sum _{j=1}^p\sum _{j^{\prime }=j+1}^p{\beta }_{3j,j^{\prime }}^{\left(1\right)}{X}_{ij}{X}_{i{j}^{\prime }}+\sum _{j=1}^p{\beta }_{4j}^{\left(1\right)}{A}_i{X}_{ij}+\sum _{j=1}^p\sum _{j^{\prime }=j+1}^p{\beta }_{5j,j^{\prime }}^{\left(1\right)}{A}_i{X}_{ij}{X}_{i{j}^{\prime }}\right]\times S_{1i} \\ +\left[{\beta }_0^{\left(2\right)}+{\beta }_1^{\left(2\right)}{A}_i+\sum _{j=1}^p{\beta }_{2j}^{\left(2\right)}{X}_{ij}+\sum _{j=1}^p\sum _{j^{\prime }=j+1}^p{\beta }_{3j,j^{\prime }}^{\left(2\right)}{X}_{ij}{X}_{i{j}^{\prime }}+\sum _{j=1}^p{\beta }_{3j}^{\left(2\right)}{A}_i{X}_{ij}+\sum _{j=1}^p\sum _{j^{\prime }=j+1}^p{\beta }_{5j,j^{\prime }}^{\left(2\right)}{A}_i{X}_{ij}{X}_{i{j}^{\prime }}\right]\times S_{2i} \end{gathered}$$ (2)

We consider two settings for the true distribution of the binary reward given the context covariates. In setting 1, we assume that the true effect of the intervention is not moderated by context covariates. In setting 2, we assume the reward distribution is characterized by a high degree of intervention effect heterogeneity. Detailed specification of each generative model is provided in Appendix A.4. The following characteristics of the simulated datasets are informed by real LS4L2 study data: (i) participant enrollment and the learning timescale, (ii) timing of decision points for each participant, and (iii) distribution of context covariates. We do not consider age and gender in the simulation.

We compare the performance of the three algorithms by averaging cumulative regret across the synthetic datasets. Informally, regret at a single decision point is the difference in the probability that the app was opened under the optimal and realized actions. Mathematically, we define cumulative regret at decision point T by $R_T≔\sum _{t=1}^T\left[.95*P\left(Y=1\mid A^{*},X\right)+.05*P\left(Y=1\mid 1-A^{*},X\right)\right]-P\left(Y=1\mid {\hat{A}}_t,X\right)$ where A* is the action that maximizes P(Y = 1|X), Â_t is the action selected by the learning algorithm at decision point T, and Y is the binary reward. Our definition of cumulative regret reflects that the randomization probability in LS4L2 is bounded above by 0.95 to allow for continued exploration and ensure the study team can assess time-varying causal moderation with the LS4L2 data.

The LS4L2 and “Simple” algorithms were deployed on each of the 100 generated synthetic datasets (50 in each of the two settings). Due to computational constraints, the “Complicated” algorithm was deployed on 5 synthetic datasets in each setting. Average cumulative regret across the synthetic datasets was calculated for each algorithm in each setting. We additionally calculated pointwise interquartile bands for the cumulative regret for the LS4L2 and “Simple” algorithms. All simulations were carried out in Python, with Bayesian logistic regression models fit using the pymc3 package [18] that uses Markov-Chain-Monte-Carlo (MCMC) using the no-u-turn-sampler (NUTS).

4.2. Simulation Results

Figure 3 presents the average cumulative regret for both settings. In setting 1 (no contextual moderators), we find the “Simple” algorithm led to the lowest average cumulative regret. Our proposed LS4L2 algorithm is a compromise between the “Simple” and “Complicated” approaches (Figure 3A). The superior performance of the Simple algorithm was anticipated, as it involves estimating significantly fewer parameters than the alternatives. In setting 2 (many contextual moderators), the LS4L2 and “Complicated” algorithms yield similar average cumulative regret (Figure 3B), while the “Simple” algorithm falls behind significantly due to its inability to capture the treatment effect heterogeneity. The gap in average cumulative regret among the three algorithms is much smaller in setting 1 relative to the gaps we see in setting 2. Simulation results provide empirical support for the study team’s algorithmic design choices. Specifically, the LS4L2 algorithm was designed to retain acceptable performance across these two settings while also accounting for computational costs.

Figure 3:

Average cumulative regret across simulated datasets for each algorithm and in each setting are presented, along with pointwise interquartile bands for the cumulative regret for the LS4L2 and Simple algorithms. Due to computational constraints, the Complicated algorithm was only deployed on 5 simulated datasets in each setting. The cumulative regret curves for each of these simulations are shown as dashed lines.

5. Discussion

In this paper, we discuss challenges in the deployment of an online RL algorithm in clinical trials and their solutions. Our work contributes to the growing literature on frameworks for RL algorithm development in mHealth by diving into the technical tradeoffs among personalization, computation, and stability. Specifically, we discuss how model flexibility leads to a sharp trade-off between personalization, learning, computation, and robustness. In LS4L2, our flexible model specification and use of weakly informative default priors ensured we could personalize with a computationally tractable model and reduced the risk that this model would break during the study period. Moreover, we emphasize the distinction between the reward for the RL algorithm and behavioral task of interest. In LS4L2, this distinction helped us think about when the reward and task were well aligned and when they were not. This will inform future studies as we seek better rewards that align with the behavioral task of interest.

Future work can be categorized into three directions. First, we think the issue of computation and online data storage will become even more important as study size grows. Scalable, approximate Bayesian methods should be considered as well as federated learning techniques where parameters are shared across local models to avoid fitting the computationally expensive, hierarchical model. Second, contextual bandit algorithms are myopic, focusing on improving the proximal outcome with no weight on delayed effects. In mHealth, we expect most of the delayed effects to be due to habituation and/or burden. In this case, we may consider adding a penalty (cost) to sending a message to account for these delayed effects as in [21]. Alternatively, we may consider using discounted reward or long-term average reward Markov Decision Process approaches [20]. However, typically these approaches require a large amount of data to learn effective policies. Finally, often there is an intermediate outcome that is more aligned with the behavioral task. In this case, it may be appropriate to consider the intermediate outcome and consider an episodic RL approach [8].

Appendix

A.1. Algorithm Monitoring: A new component of study monitoring for digital intervention trials

Study monitoring is a critical component of clinical trials, as it ensures the integrity, quality, and accuracy of the data collected. Protocols for monitoring data collection are essential because they provide a detailed roadmap for how data will be collected, managed, and monitored throughout the study. Adhering to a robust monitoring plan helps to identify and resolve issues early, prevent protocol deviations, and ultimately ensure the reliability of the study’s findings.

When deploying bandit algorithms, it is important to also draft a study protocol for monitoring of the algorithm in addition to standard trial protocols. These algorithms require two types of monitoring systems: (1) a failure monitoring system to monitor failure of the learning algorithm to update and failure of the probabilistic mappings to work properly, and (2) a statistical monitoring system to monitor whether the probabilistic mappings are working as intended.

In LS4L2, the failure monitoring system consists of checks on the output of the learning algorithm and whether the per-participant updates were running at the specified times (2 and 4 months since recruitment). The study team downloads the output of the learning algorithm from the cloud and runs sanity checks by comparing to locally fit versions of the model.

This allows the study team to identify potential software bugs or issues with the collected data. Any bug/issue is (1) dated, (2) documented as a protocol deviation, and (3) the software changes are recorded in a log file. Our failure monitoring was able to identify a key issue in data collection due to software updates that was remedied within days. The issue was fixed and documented as a protocol deviation. Without such failure monitoring systems, mHealth clinical trials will only identify these failure modes after the trial is completed.

In addition, the study team decided on two statistical checks to ensure fidelity to the algorithm protocol. First, we assessed whether the MCMC chains stored on the cloud matched closely with local runs of the same updates. Automated reports were generated for every participant at each time that the learning algorithm was run and reviewed by a study team member. A data analyst reviewed these documents for potential deviations/bugs. In studies with more frequent updates, such reports may not be able to be manually reviewed and an automated review should be protocolized.

Second, we assessed the statistical calibration of our deployed contextual bandit. Figure A.1 summarizes our approach. We categorized all decision points by the personalized and context-specific randomization probabilities. In each bin of width 0.05, we then calculated the empirical probability with which notifications were actually sent and a 95% asymptotically normal confidence interval. If our implementation is well calibrated then we would expect the confidence intervals to cover the midpoint of each bin. In Figure A.1, LS4L2 is well calibrated with three bins with statistically significant deviation which the study team deemed acceptable.

Figure A.1:

As an additional layer of study monitoring, we examined the calibration of our algorithm. We calculated the empirical probability with which notifications were sent to study participants in bins corresponding to the individualized decision rules learned by the learning algorithm. Uncertainty was quantified with asymptotically normal 95% CIs for the observed probabilities.

A.2. Rules for including age and gender in the model

For baseline covariates of age and gender, we apply a slightly different set of rules to determine their inclusion. If the data contain at least two subjects with different ages, then age is included as a main effect. If the data contain at least four subjects with different ages, then an interaction term between age and treatment is also included in the model. The same rules are applied to gender. To determine if a three-way interaction term between age, gender, and treatment should be included, we check to see if at least one cell in the two-by-two table for age and gender has at least four observations (where age is temporarily converted to a binary variable by partitioning values at the mean for purposes of creating the contingency table).

A.3. Accounting for Missing Data in the Probabilistic Mapping

Recall that as a consequence of having categorical covariates, sometimes previously unobserved contexts will arise for which the algorithm requires a probabilistic mapping. In this situation, the usual map can not be constructed using the regression model fit by the learning algorithm. There is a missing data scenario where the context variable is measured as “unknown”, but the level “unknown” for that variable is not included in the data used to fit the model. We also encounter a new context prediction scenario when the context variable is measured as any level except “unknown”, but the measured level is not included in the data used to fit Bayesian hierarchical logistic regression. We describe how each scenario is handled in LS4L2 below.

In LS4L2, the study team considers only categorical context variables. This meant that all potential contexts can be enumerated in a large table (each row representing a different context combination) and those combinations falling under Scenario 1 and 2 can be flagged. For rows that are not flagged, the probabilistic map can be constructed using the Bayesian hierarchical logistic regression and a column is added to the table to represent the probability of sending a notification in this context. For rows that are flagged, we implement a procedure for building the probability map that dependeds on the scenario. In the missing data scenario, we impute a value for the updated intervention probability by taking an average across all available levels of the covariate in the small table (the subset of the large table with only unflagged rows). In the new context prediction scenario, we impute a value for the updated intervention probability by either averaging across all available levels of the context variable or linking the unavailable level of the context variable to the closest value for which an intervention probability is available. For example, if a probability is unavailable for “cold” temperatures but it is available for “cool” temperatures, then temporarily set all levels of “cold” to “cool”.

A.4. Simulation Design Details

A.4.1. Setting 1: No Contextual Moderators

In setting 1, the effect of treatment is constant across levels of the context covariates. The model governing the true probability of opening the app in a given context, X = (X₁,…,X_p) and under a specific treatment, A ∈ {0, 1}, where S₁ = I(daysInStudy > 56) and S₂ = I(daysInStudy > 112):

$$\begin{gathered} \mathit{\text{logit}}\left(p_i\mid A_i,X_i\right)={\beta }_0+{\beta }_1{A}_i+\sum _{j=1}^p{\beta }_{2j}{X}_{ij}+\sum _{j=1}^p\sum _{j^{\prime }=j+1}^p{\beta }_{3j,j^{\prime }}{X}_{ij}{X}_{i{j}^{\prime }} \\ +\left[{\beta }_0^{\left(1\right)}+\sum _{j=1}^p{\beta }_{2j}^{\left(1\right)}{X}_{ij}+\sum _{j=1}^p\sum _{j^{\prime }=j+1}^p{\beta }_{3j,j^{\prime }}^{\left(1\right)}{X}_{ij}{X}_{i{j}^{\prime }}\right]\times S_{1i} \\ +\left[{\beta }_0^{\left(2\right)}+\sum _{j=1}^p{\beta }_{2j}^{\left(2\right)}{X}_{ij}+\sum _{j=1}^p\sum _{j^{\prime }=j+1}^p{\beta }_{3j,j^{\prime }}^{\left(2\right)}{X}_{ij}{X}_{i{j}^{\prime }}\right]\times S_{2i} \end{gathered}$$ (A.1)

To reflect our belief that sending a notification results in participants being about 1.5 times more likely to open the smartphone app compared to those who did not receive a notification, we sampled the true treatment effect from a normal distribution with mean 0.4 and standard deviation 1/8.

The remaining true coefficients were sampled from priors according to Table A.1.

The exact values of each coefficient in setting 1 can be found in setting1_coefficients.csv.

Table A.1:

Prior specifications by interaction order.

Interaction Order	Prior Distribution	Mean	SD
1	Normal	0	1.000
2	Normal	0	0.250
3	Normal	0	0.0625
4	Normal	0	0.0156

A.4.2. Setting 2: Many Contextual Moderators

In setting 2, the effect of treatment is free to vary across levels of the context covariates. The mathematical model that determines the true probability that the app is opened under a given treatment, A, in a given context, X is given below in equation A.2.

$$\begin{gathered} \mathit{\text{logit}}\left(p_i\mid A_i,X_i\right)={\beta }_0+{\beta }_1{A}_i+\sum _{j=1}^p{\beta }_{2j}{X}_{ij}+\sum _{j=1}^p\sum _{j^{\prime }=j+1}^p{\beta }_{3j,j^{\prime }}{X}_{ij}{X}_{i{j}^{\prime }}+\sum _{j=1}^p{\beta }_{4j}{A}_i{X}_{ij}+\sum _{j=1}^p\sum _{j^{\prime }=j+1}^p{\beta }_{5j,j^{\prime }}{A}_i{X}_{ij}{X}_{i{j}^{\prime }} \\ +\left[{\beta }_0^{\left(1\right)}+{\beta }_1^{\left(1\right)}{A}_i+\sum _{j=1}^p{\beta }_{2j}^{\left(1\right)}{X}_{ij}+\sum _{j=1}^p\sum _{j^{\prime }=j+1}^p{\beta }_{3j,j^{\prime }}^{\left(1\right)}{X}_{ij}{X}_{i{j}^{\prime }}+\sum _{j=1}^p{\beta }_{4j}^{\left(1\right)}{A}_i{X}_{ij}+\sum _{j=1}^p\sum _{j^{\prime }=j+1}^p{\beta }_{5j,j^{\prime }}^{\left(1\right)}{A}_i{X}_{ij}{X}_{i{j}^{\prime }}\right]\times S_{1i} \\ +\left[{\beta }_0^{\left(2\right)}+{\beta }_1^{\left(2\right)}{A}_i+\sum _{j=1}^p{\beta }_{2j}^{\left(2\right)}{X}_{ij}+\sum _{j=1}^p\sum _{j^{\prime }=j+1}^p{\beta }_{3j,j^{\prime }}^{\left(2\right)}{X}_{ij}{X}_{i{j}^{\prime }}+\sum _{j=1}^p{\beta }_{3j}^{\left(2\right)}{A}_i{X}_{ij}+\sum _{j=1}^p\sum _{j^{\prime }=j+1}^p{\beta }_{5j,j^{\prime }}^{\left(2\right)}{A}_i{X}_{ij}{X}_{i{j}^{\prime }}\right]\times S_{2i} \end{gathered}$$ (A.2)

Setting 2 is designed so that time of the week, time of the day, and engagement level are strong moderators of treatment effect. The following specification was chosen so that the optimal treatment (i.e. the decision to send a notification) varies according to the values of contextual variables at a given decision point. During the week, daytime hours, and when engagement is low, we assume that the effect of treatment is normally distributed with mean −0.4 and standard deviation 1/8. We also assume that the difference in treatment effect when engagement is high, at night, or when it is the weekend has a normal distribution with mean .5 and standard deviation 1/16. Further, we sampled the study-period interaction with treatment from a normal distribution with mean −0.1 and standard deviation 1/32 to reflect our strong belief that treatment effect diminishes over the course of the study. As in setting 1, the remaining coefficients were sampled from priors according to Table A.1

The exact values of each coefficient in setting 2 can be found in setting1_coefficients.csv.