Designing Adaptive Intensive Interventions Using Methods from Engineering

Abstract

Objective

Adaptive intensive interventions are introduced and new methods from the field of control engineering for use in their design are illustrated.

Method

A detailed step-by-step explanation of how control engineering methods can be used with intensive longitudinal data to design an adaptive intensive intervention is provided. The methods are evaluated via simulation.

Results

Simulation results illustrate how the designed adaptive intensive intervention can result in improved outcomes with less treatment by providing treatment only when it is needed. Furthermore, the methods are robust to model misspecification as well as the influence of unobserved causes.

Conclusions

These new methods can be used to design adaptive interventions that are effective yet reduce participant burden.

One of the main challenges clinical psychologists and psychiatrists increasingly face today is the complexity of the decision process when treating individuals for chronic problems such as drug addiction, binge eating disorders, and non-suicidal self-injury. Decisions may include the best pharmaceutical intervention at the most appropriate dosage, the most promising therapeutic intervention, and the optimal duration of various treatments. In addition, clinical and behavioral scientists are increasingly collecting intensive longitudinal data (ILD) and—more recently—providing a variety of behavioral interventions intensively, often via web- or smart phone-based interventions (e.g., Heron & Smyth, 2010; Gustafson et al., 2011). With this unparalleled opportunity for health behavior change comes the important new responsibility of deciding how best to adapt intensive behavioral interventions to the person.

Adaptive interventions operationalize the adaptation and re-adaptation of behavioral interventions; they are sequences of treatments that are adapted and re-adapted to individual circumstances and behaviors in order to achieve and maintain health behavior change. They adapt to the individual using decision rules, which input individual information such as symptoms, adherence to prior treatment, and biological measures, and output treatment recommendations, such as the most promising intervention or the appropriate dose of the intervention. An adaptive intensive intervention uses the intensive longitudinal data collected on an individual to deliver the appropriate intervention when the individual is in need. An example is provided by Gustafson and colleagues (Gustafson et al., 2011), who developed Alcohol-CHESS (ACHESS), which is adaptive in that it provides recovering individuals access to social support when the GPS on their smartphone detects that they are near a high-risk location (e.g., an alcohol retail outlet). In this case the decision rule inputs the geographical location, checks whether this location is within a certain distance of established high-risk locations, and then alerts the individual that they may be at increased risk, giving them an opportunity to call someone for support. Individuals can also set up ACHESS to automatically alert a family member, peer, or professional when they are in a high-risk location for immediate social support. Thus this intervention provides treatment at the time when the individual most needs support. This adaptive intensive intervention provides a low-cost, context-sensitive way to augment more traditional treatment (e.g., in-person psychotherapy).

Scientists developing adaptive interventions typically use behavioral theory, literature reviews, and clinical experience to fully form decision rules specifying how the intervention will be adapted to the person. However this approach to adaptive intervention design is difficult in an ILD context because the intensive nature of the intervention makes the decision process highly complex. Furthermore because the intervention may be provided repeatedly over time, it is crucial to provide treatment only when needed to avoid overburdening participants. Even when there is no monetary cost associated with additional treatment, more treatment may have other hidden costs, such as treatment fatigue. Identifying the intervention frequency that optimally balances effectiveness with participant burden can be difficult; further, the ideal frequency is likely to differ across individuals.

In this study we describe how tools from control engineering can provide a systematic way to design adaptive intensive interventions using ILD. In these approaches algorithms for adapting treatment are developed using existing observational ILD data. Then at each time point during the implementation of the intervention for a new individual, the developed algorithm uses the most recent data collected on the individual to recommend an intervention for this individual at this time. As we demonstrate, when treatment fatigue is an issue, this approach is superior to full treatment in which every individual is treated at every instance. The adaptive intensive intervention will use less total treatment than a standard protocol, yet result in stronger effects because it uses treatment only when it is needed.

To make the ideas and tools concrete we illustrate the design of an adaptive intensive intervention for smoking-cessation. Combined treatments involving both a pharmacological and a behavioral component have been most successful in achieving smoking cessation (Fiore Jorenby, Piasecki, & Baker, 2000). Yet even with effective smoking cessation aids, less than 30 percent of smokers achieve long-term abstinence in a given quit attempt (Fiore, et al., 2000). Thus we focus on designing an adaptive intensive behavioral intervention for use in conjunction with a pharmacological treatment.

Smoking lapse is highly associated with intra-individual phenomena such as craving (e.g., Shiffman et al., 1997). Higher self-efficacy has been associated with lower self-reported nicotine craving among smokers (e.g., Shadel & Cervone, 2006), and there is extensive evidence that negative affect is closely linked to craving (e.g., Baker, Piper, McCarthy, Majeskie, & Fiore, 2004; Piasecki, Fiore, McCarthy, & Baker, 2002). Both self-efficacy and negative affect have been the focus of previous smoking cessation treatments (e.g., Mermelstein, Hedeker, & Wong, 2003). Therefore, our hypothetical intervention is designed to improve self-efficacy and decrease negative affect, and thus decrease smoking urge. In the developed intervention the smartphone will collect self-report data three times per day (approximately morning, midday and evening); following each data collection, the smartphone will either not prompt or prompt the participant to access a smartphone behavioral application. This hypothetical smartphone application will contain a variety of intervention components such as messages designed to maintain motivation, improve coping skills, provide positive reinforcement from the participant's primary care doctor, and remind the participant why he/she wants to quit smoking.

For illustrative purposes we assume we have ILD from a study, n = 150 in which participants are prompted by their smartphone to provide data in the morning, midday and in the evening for 50 days. This results in data from time points t=1 to 150 for each participant. At each time t, participants report their momentary self-efficacy (SE(t)), negative affect (NA(t)), and smoking urge (SU(t)), each on a continuous scale. Furthermore, participants may receive treatment by accessing the smartphone behavioral application following each point of data collection (i.e., the participant may access the behavioral application up to three times per day). Whether or not the participant accessed the behavioral application at a particular time is indicated by T(t) (coded 1 if the participant receives the treatment and coded 0 otherwise).

Methods

Designing an Adaptive Intensive Intervention Using Control Engineering

In this section, we provide a step-by-step explanation of control engineering methods that can inform the development of an adaptive intensive intervention using ILD. The methods are illustrated by considering the design of the adaptive intensive, smoking-cessation intervention discussed in the Introduction. First we fit a dynamical model, which in this case is a system of regression equations, to the data. We then provide an example of how the scientific team can quantify the treatment goal. We then describe how to use the dynamical model with estimated regression coefficients and the treatment goal to develop the decision rules comprising the adaptive intensive intervention. Finally, we describe how to implement the adaptive intensive intervention with new individuals.

The first step is to fit a dynamical model to the data, that is, a model for the variables at each time point t. Recall that three continuous dependent variables, smoking urge, negative affect, and self-efficacy (SU(t), NA(t), SE(t)), are assessed at each time t for each participant. Here we use linear models in which each variable at time t depends linearly on both the variables and treatments at times t−1 and t−2. For example the linear models might be

$$\mathit{\text{SU}}(t)=a_0+a_1\mathit{\text{SU}}(t-1)+a_2\mathit{\text{SU}}(t-2)+b_1\mathit{\text{NA}}(t-1)+b_2\mathit{\text{NA}}(t-2)+c_1\mathit{\text{SE}}(t-1)+c_2\mathit{\text{SE}}(t-2)+\epsilon (t),$$ (1)

$$\mathit{\text{NA}}(t)=d_0+d_1\mathit{\text{SU}}(t-1)+d_2\mathit{\text{SU}}(t-2)+e_1\mathit{\text{NA}}(t-1)+e_2\mathit{\text{NA}}(t-2)+f_1\mathit{\text{SE}}(t-1)+f_2\mathit{\text{SE}}(t-2)+q_{1}T(t-1)+q_{2}T(t-2)+{\epsilon }^{\mathit{\text{na}}}(t),\text{and}$$ (2)

$$\mathit{\text{SE}}(t)=g_0+g_1\mathit{\text{SU}}(t-1)+g_2\mathit{\text{SU}}(t-2)+h_1\mathit{\text{NA}}(t-1)+h_2\mathit{\text{NA}}(t-2)+r_1\mathit{\text{SE}}(t-1)+r_2\mathit{\text{SE}}(t-2)+s_{1}T(t-1)+s_{2}T(t-2)+{\epsilon }^{\mathit{\text{se}}}(t),$$ (3)

where Var(ε(t)) = σ², $\mathit{\text{Var}}({\epsilon }^{\mathit{\text{na}}}(t))={\sigma }_{\mathit{\text{na}}}^2$, $\mathit{\text{Var}}({\epsilon }^{\mathit{\text{se}}}(t))={\sigma }_{\mathit{\text{se}}}^2$ for t=1, …, 150 and {ε(1), ε^na(1), ε^se(1), ε(2), ε^na(2), ε^se(2), …, ε(150), ε^na(150), ε^se(150)} are independent, normal, mean zero errors¹. Note the model for SU(t) (see Equation 1) does not include treatment, as our hypothetical behavioral intervention is assumed to directly impact self-efficacy and negative affect. Treatment could be included in this model as an independent variable if it is expected to impact smoking urge directly. Furthermore, (2) and (3) permit treatment fatigue; for example, if in model (2) the regression coefficients of the terms involving treatment are of opposite sign, the effect of treatment T(t−1)=1 will be washed out if treatment has been applied at time t-2 ; i.e., if T(t−2)=1. In all of these models data from even earlier time points, say time t-3, may be included. Because the error terms are assumed independent, three separate standard regression models can be fit, one each for smoking urge, negative affect, and self-efficacy, corresponding to Equations 1-3 above. Appendix C (available at http://personal.psu.edu/cml18/jccp) provides hypothetical data from 150 individuals to demonstrate how to structure the data. Together, fitting these three models will provide estimates of all regression coefficients and error variances, σ̂², ${\widehat{\sigma }}_{\mathit{\text{na}}}^2$, and ${\widehat{\sigma }}_{\mathit{\text{se}}}^2$.

The second step is to quantify the treatment goal for a new individual. In our example, the treatment goal is to reduce smoking urge. We translate this treatment goal into an “objective function” that will be used to determine the treatment—if any—that will be provided at each time point for each individual. An objective function that reflects this goal for this individual at each time point t is the sum of squares of smoking urge

$$\sum _{k\ge t+2}{[\mathit{\text{SU}}(k)]}^2.$$ (4)

We want to provide an adaptive intervention so as to minimize this individual's sum of squares of smoking urge. The third step is to construct the decision rules that underlie the adaptive intervention. We illustrate the use of a promising and widely used method from engineering: model predictive control (MPC; e.g., Wang & Rivera, 2008; Wang, Rivera, & Kempf, 2007). MPC is based on the fact that Equation 4 is implicitly a function of treatment at time t. This dependence on treatment at time t can be seen by considering only the first term in the sum, namely [SU(t + 2)]². We “unpack” [SU(t + 2)]² as follows. First consider Equation 1 with t+2 in place of t. We see that SU(t + 2) and thus [SU(t + 2)]² is a function of SU(t + 1), SU(t), SE(t + 1), SE(t) and NA(t + 1), NA(t). Next consider Equations 2, 3 with t+1 in place of t. Unpack [SU(t + 2)]² further by substituting SE(t + 1), NA(t + 1) with their respective formulae from Equations 2, 3. After making these substitutions, we see that [SU(t + 2)]² is actually a function of treatment at time t, T(t), as well as smoking urge at time t and time t-1, SU(t), SU(t − 1), self-efficacy at time t and time t-1, SE(t), SE(t − 1), negative affect at time t and time t-1, NA(t), NA(t − 1) and past treatment, T(t − 1). Thus the Equation 4 is a function of treatment at time t, present and past smoking urge, present and past self-efficacy, present and past negative affect and past treatment. Suppose we have observations of present and past smoking urge, present and past self-efficacy, present and past negative affect and past treatment on this new individual. Intuitively the MPC algorithm compares the value of Equation 4 when T(t) is set to 1 to the value of Equation 4 when T(t) is set to 0. If T(t) = 1 leads to a smaller value of Equation 4, then the optimal decision from the MPC algorithm is to provide treatment to this new individual at time t. Now Equation 4 contains more than just the first term,[SU(t + 2)]², resulting in a more complex objective function. None-the-less, once unpacked, Equation 4 is a function of treatment at time t, present and past smoking urge, present and past self-efficacy, present and past negative affect and past treatment.

There are three aspects of Equation 4 that require further explanation. First, if two new individuals have the same observed values of present and past smoking urge, present and past self-efficacy, present and past negative affect and past treatment then the results of the MPC algorithm will be the same (i.e., the two individuals will be recommended the same treatment at that time). Second, because Equation 4 includes a sum over all time points beginning at t+2, this objective function accounts for the presence of delayed effects of time t treatment. Thus, for a treatment at time t to be recommended, the treatment must not only be predicted to reduce smoking urges at time t+2, it must also be predicted to lead to a reduction in smoking urge at later time points, at least as compared to no treatment at time t. Third, each term in Equation 4 squares the smoking urge, SU(k); the square acts to prevent treatment at time t if treatment is predicted to lead to too many future occasions of very high smoking urge. In particular this means that if no treatment at time t is likely to lead to a constant moderate smoking urge in the future, but treatment at time t is likely to lead occasionally to small smoking urges but often to high smoking urge, then no treatment is better. For further details concerning the objective function and the MPC algorithm see Appendix A at http://personal.psu.edu/cml18/jccp.

In practice, prior to providing any treatment for a new individual, we can run the MPC algorithm for each possible value of smoking urge, negative affect, and self-efficacy at past and present time (times t−1, t respectively) and the treatment indicator at the past time (time t -1; e.g., all values of SE(t), NA(t), SU(t), T(t − 1), SE(t − 1), NA(t − 1) and SU(t − 1)) to obtain the best values for the treatment T(t). We could use this to produce a large decision rule table in which each row corresponds to a different value of SE(t), NA(t), SU(t), T(t − 1), SE(t − 1), NA(t − 1) and SU(t − 1) and the last column of the table provides the best treatment at the present time t. Table 1 provides a small subset of the rows that would be found in the full table. The full decision rule table provides the decision rules for the adaptive intensive behavioral intervention.

Table 1. A Partial Decision Rule Table.

Description of Individual Observations	Input							Output Treatment Recommendation
	Self-Efficacy		Negative Affect		Smoking Urge		Treated
	(t-1)	(t)	(t-1)	(t)	(t-1)	(t)	(t-1)	(t)
Low, improving self-efficacy; High, improving negative affect; High, improving smoking urge; No prior treatment	1.1	2.1	1.9	1.6	5.0	3.5	No	No
Low, improving self-efficacy; High, improving negative affect; High, improving smoking urge; No prior treatment (more extreme than the above)	0.9	1.4	2.5	1.9	6.8	5.7	No	Yes
Low, stable self-efficacy, High deteriorating negative affect, High, deteriorating smoking urge; Prior Treatment	2.0	1.8	1.8	2.2	5.8	6.4	Yes	No
Moderate, stable self-efficacy; Moderate, stable negative affect; High, stable smoking urge; Prior treatment	3.1	3.0	0.1	0.08	3.9	4.0	Yes	No
High, improving self-efficacy; High, stable negative affect; High, deteriorating smoking urge; No prior treatment	3.8	4.5	1.8	2.2	5.2	6.0	No	No
High, deteriorating self-efficacy; Low, stable negative affect; High, stable smoking urges; Prior treatment	4.9	4.3	-1.7	-1.8	5.3	5.1	Yes	No
Low deteriorating self-efficacy; Low, deteriorating negative affect; Moderate stable smoking urge; Prior treatment	1.69	0.80	1.14	1.88	2.3	2	Yes	Yes
High deteriorating self-efficacy; Low, stable negative affect; Low deteriorating smoking urge; No prior treatment	3.47	2.97	0.31	0.28	0	0.75	No	Yes

Note. t refers to current time for which treatment recommendation is being made, in this case, t=75. In this population, the mean self-efficacy is 3.2 with a standard deviation of 1.0, the mean smoking urge is 2.8 with a standard deviation of 1.8 and the mean negative affect is -0.08 with a standard deviation of 1.1.

To implement the adaptive intervention on a new individual we proceed as follows. On the morning of day 1, the individual provides information on their smoking urge, self-efficacy and negative affect; then the smartphone provides the behavioral intervention. At midday we begin to use the adaptive intensive intervention. The individual provides information on their level of smoking urge, self-efficacy and negative affect. Treatment T(1) has the value 1 (treatment was provided in the morning). We find the row in the decision rule table corresponding to the individual's values of SE(1), NA(1), SU(1), T(1), SE(2), NA(2), and SU(2) or, equivalently, run the MPC algorithm, to obtain the optimal treatment for the individual and set T(2) to the optimal treatment. That is, if the optimal treatment is 1, we provide treatment at midday; otherwise we do not. In the evening we repeat this process, using the individual's values on SE(2), NA(2), SU(2), T(2), SE(3), NA(3), and SU(3) to use the decision rule table to obtain T(3). We repeat this process for the duration of treatment.

An Illustrative Example of the Development and Implementation of an Adaptive Intensive Intervention using MPC

In this section, we illustrate how the above steps for designing an adaptive intensive intervention can be carried out. We do this using a simulated intensive longitudinal data set that loosely resembles the data from a smoking cessation study (Shiffman et al., 2000; Shiffman, Hickcox, et al., 1996; Shiffman, Paty, Gnys, Kassel, & Hickcox, 1996). The simulated data set contains 150 participants from a hypothetical smoking cessation study conducted over a period of 50 days; measurements were collected three times per day. At each occasion, treatment was provided with probability .5. See Appendix B for the model used to generate the simulated data; this data set is available at http://personal.psu.edu/cml18/jccp.

The first step is to fit Equations 1-3 to the data. This results in the fitted regression models

$$\mathit{\text{SU}}(t)=1.12+0.48\mathit{\text{SU}}(t-1)+0.34\mathit{\text{SU}}(t-2)+0.016\mathit{\text{NA}}(t-1)-0.015\mathit{\text{NA}}(t-2)-0.1\mathit{\text{SE}}(t-1)-0.088\mathit{\text{SE}}(t-2)+\epsilon (t)$$ (5)

$$\mathit{\text{NA}}(t)=0.14+0.003\mathit{\text{SU}}(t-1)+0.0009\mathit{\text{SU}}(t-2)+0.49\mathit{\text{NA}}(t-1)+0.25\mathit{\text{NA}}(t-2)-0.05\mathit{\text{SE}}(t-1)+0.005\mathit{\text{SE}}(t-2)-0.22T(t-1)+0.15T(t-2)+{\epsilon }^{\mathit{\text{na}}}(t)$$

$$\mathit{\text{SE}}(t)=0.54-0.008\mathit{\text{SU}}(t-1)-0.004\mathit{\text{SU}}(t-2)-0.018\mathit{\text{NA}}(t-1)+0.001\mathit{\text{NA}}(t-2)+0.49\mathit{\text{SE}}(t-1)+0.36\mathit{\text{SE}}(t-2)+0.18T(t-1)-0.15T(t-2)+{\epsilon }^{\mathit{\text{se}}}(t),$$

with estimated error variances σ̂² = 1.1, ${\widehat{\sigma }}_{\mathit{\text{na}}}^2=0.06$, and ${\widehat{\sigma }}_{\mathit{\text{se}}}^2=0.3$. Equation 5 indicates that smoking urge at any time t is positively correlated with past smoking urge, and negatively correlated with past self-efficacy. Furthermore Equation 5 shows a mixed relationship between smoking urge at time t and past negative affect, depending on the time lag.

The next step is to design an adaptive intensive intervention algorithm using the above fitted regression models. We use the MPC algorithm as described above. Data from the 150 participants are used to fit the dynamical models; there is no further use of the data from the 150 participants. The dynamical models are then used to specify the objective function as described earlier. Use of the MPC algorithm results in decision rules which, based on observations of past and present self-efficacy, past and present negative affect and past treatment, provide the best treatment at the present time t. The (partial) decision rule table provided in Table 1 was constructed using the MPC algorithm.

Consider row one. This row corresponds to observations indicating poor self-efficacy, negative affect, high smoking urge and no treatment at prior time, yet improvement in each of these variables from time t-1 to time t; the MPC algorithm recommends no treatment for an individual with these observations. The observations in row two are similar to row one, but with an excessively large observed smoking urge; the algorithm recommends treatment. In row three the observations correspond to stable self-efficacy but poor levels of negative affect and smoking urge and treatment at the prior time. Here the MPC algorithm does not recommends treatment because treatment had been given at the previous time.

In Figure 1 we illustrate the implementation of the above adaptive intensive intervention (corresponding to the full decision rule table) on a simulated individual. This figure shows the evolution of the individual's smoking urge for two different interventions: in blue we depict the response to the adaptive intensive intervention (using the decision rule table) and in green we show the response to full treatment intervention (treatment three times per day). Moreover, the red “stars” indicate when treatment is provided by the adaptive intervention. As one can see, full treatment works better in the beginning, when the individual has not yet experienced treatment fatigue. However, the proposed adaptive intervention is more effective in the long term. In particular, when the model forecasts that smoking urge is expected to be excessive, then treatment is provided. As a result treatment is only provided as needed, with the decision based on present and past information about the patient. Otherwise, treatment is not provided. As shown in Figure 1, treatment is provided when an increasing trend in smoking urge occurs.

Figure 1.

Difference in smoking urge under the adaptive intensive intervention and full treatment for one hypothetical individual.

A Simulation Study Evaluating the MPC Approach

To evaluate the MPC approach to developing an adaptive intensive intervention we conducted a simulation study. Consider two different populations of individuals, labeled A and B, defined below. For each population, 500 data sets are simulated, each containing 150 virtual participants. Each data set represents a smoking cessation study conducted over a period of 50 days; as before, measurements are collected three times per day. At each measurement occasion treatment is provided with probability 0.5. The models used to generate the data are

$$\mathit{\text{SU}}(t)=1.1+.50\mathit{\text{SU}}(t-1)+.35\mathit{\text{SU}}(t-2)+.01\mathit{\text{NA}}(t-1)-.12\mathit{\text{SE}}(t-1)-.01\mathit{\text{SE}}(t-2)+w(t)+\epsilon (t)$$ (6)

$$\mathit{\text{NA}}(t)=.15+.01\mathit{\text{SU}}(t-1)+.49\mathit{\text{NA}}(t-1)+.24\mathit{\text{NA}}(t-2)-.05\mathit{\text{SE}}(t-1)+{\mathit{\text{treat}}}^{\mathit{\text{NA}}}\left(\overline{T}(t)\right)+{\epsilon }^{\mathit{\text{na}}}(t)$$ (7)

$$\mathit{\text{SE}}(t)=.52-.01\mathit{\text{SU}}(t-1)-.02\mathit{\text{NA}}(t-1)+.49\mathit{\text{SE}}(t-1)+.36\mathit{\text{SE}}(t-2)+{\mathit{\text{treat}}}^{\mathit{\text{SE}}}\left(\overline{T}(t)\right)+{\epsilon }^{\mathit{\text{se}}}(t)$$ (8)

where Var(ε(j)) = 1.2, Var(ε^na(j)) = 0.6, Var(ε^se(j)) = 0.3 for j=1, …, 150 and {ε(1), ε^na(1), ε^se(1), ε(2), ε^na(2), ε^se(2), …, e(n), ε^na(n), ε^se(n)} are independent, normal, mean zero errors. The term w(t) in Equation (6) reflects random disturbances (i.e., unobserved momentary events that affect urge to smoke at that moment, e.g., family stress, health issues²).

From Equation 6 we see that higher smoking urge, more negative affect, and lower self-efficacy at the previous time predicts higher smoking urge at time t. In Equation 7 we see that higher smoking urge, more negative affect, and lower self-efficacy at past time points predicts more negative affect at time t. Finally, in Equation 8, lower smoking urge, less negative affect, and higher self-efficacy at past time points predicts higher self-efficacy at time t.

The treatment terms treat^NA (T̅(t)) in Equation 7 and treat^SE (T̅(t)) in Equation 8 are given by

$${\mathit{\text{treat}}}^{\mathit{\text{NA}}}\left(\overline{T}(t)\right)=-0.2T(t-1)+0.2\mathit{\text{logistic}}({\sum }_{u=t-44}^{u-2}T(u))T(t-2)$$ (9)

and

$${\mathit{\text{treat}}}^{\mathit{\text{SE}}}\left(\overline{T}(t)\right)=0.2T(t-1)-0.2\mathit{\text{logistic}}({\sum }_{u=t-44}^{t-2}T(u))T(t-2).$$ (10)

The Appendices provide the form of the logistic function; the value of this logistic function is close to 1 if the total amount of treatment, ${\sum }_{u=t-44}^{t-2}T(u)$, is large; it is close to 0 if the total amount of past treatment is close to 0. The coefficients of T(t − 1) imply that providing treatment at time t-1 reduced negative affect and increased self-efficacy. However, providing treatment many times during the last 2 weeks (e.g., ${\sum }_{u=t-44}^{t-2}T(u)$ is large) produces treatment fatigue, to the extent that providing treatment at time t-2 greatly reduces the effect of treatment at time t-1 on both negative affect and self-efficacy.

The term w(t) reflects unmeasured momentary events that affect urge to smoke. In virtual population A, w(t) is set to 0 (i.e., there were no unmeasured momentary events). In virtual population B, w(t) is zero for 75% of randomly selected time points; for the remaining time points it is uniformly distributed across a range of momentary positive and negative effects between -2.5 and 2.5. Thus the impact of the momentary events is sporadic as it is present at only 25% of the time points.

Results

The dynamical model in Equations 1-3 was fit to each of the 500 data sets from each of the two populations. For each population, the MPC algorithm was applied using each of the 500 fitted models resulting in 500 decision rule tables (500 adaptive intensive interventions). Next, for each of the 500 adaptive interventions, we recorded the smoking urge, negative affect and self-efficacy of a simulated individual under three scenarios: (1) if provided the adaptive intensive intervention, (2) if provided a full treatment intervention (treatment three times per day for all days) as a comparison, and (3) if not provided treatment. Results are below.

Population A

The results are summarized in Table 2 (upper panel) and Figures 2 and 3. First note from Table 2 that using the adaptive intensive intervention resulted in the lowest average level of smoking urge, lowest average negative affect and highest average self-efficacy. Also note that when the adaptive intervention was used, treatment was provided on average only once every two days (mean treatments per day = 0.54). Furthermore, the mean differences between smoking urge under no treatment and smoking urge under the adaptive intervention or full treatment intervention were .28 (SE=.004) and .20 (SE=.002), respectively. Both the full treatment intervention and the adaptive intervention reduced smoking urge, but the adaptive intervention was more effective and used less treatment. Figure 2 provides a more complete picture of the differences in average smoking urge between no treatment, adaptive treatment and full treatment. Note that high values (i.e., greater improvement) are preferred in this Figure.

Table 2. Descriptive Statistics for the 500 Evaluations.

Variable	No Treatment		Full Treatment		Adaptive Intensive Treatment
	Mean (SE)	SD	Mean (SE)	SD	Mean (SE)	SD
Population A
Smoking Urge	3.04 (0.027)	1.64	2.84 (0.026)	1.61	2.76 (0.023)	1.58
Negative Affect	0.09(0.012)	1.03	-0.02(0.012)	1.04	-0.08(0.011)	1.04
Self-Efficacy	3.01(0.014)	0.94	3.18 (0.014)	0.93	3.23 (0.013)	0.94
Treatments Per Day	0	0	3	0	0.54 (0.007)	0.69
Population B
Smoking Urge	3.2 (0.028)	2.01	3.03 (0.027)	1.97	2.93 (0.025)	1.94
Negative Affect	0.1 (0.011)	1.03	-0.02 (0.011)	1.04	-0.1 (0.010)	1.03
Self-Efficacy	3.03 (0.013)	0.92	3.19 (0.013)	0.91	3.3 (0.012)	0.92
Treatments Per Day	0	0	3	0	0.6 (0.007)	0.72

Figure 2.

Difference in mean smoking urge as compared to no treatment for all individuals in Population A.

Figure 3.

Difference in mean smoking urge as compared to no treatment for smokers with higher than average smoking urge in Population A.

An adaptive intensive intervention should provide treatment only when needed—when, in the absence of treatment, the smoking urge is expected to be large at subsequent times. To see this, consider 233 of the 500 individuals with a mean smoking urge above 3.04 in the absence of treatment. For this subgroup the mean differences between smoking urge for an individual under no treatment and an individual's smoking urge under the adaptive intervention or full treatment intervention are .34 (SE= 005) and .21 (SE= 002), respectively. In other words, the adaptive intervention performs better than the full treatment intervention among these higher-risk individuals. Also, treatment was provided, on average, about twice every three days (.65/day)—more frequently than for the whole population, which received the treatment, on average, once per two days. Figure 3 provides the histograms of the mean difference in smoking urge between no treatment and (a) adaptive intervention and (b) full treatment for this subgroup.

Population B

Recall that the virtual participants in population B are similar to those in population A, except that they experience unmeasured momentary events that affect urge to smoke (these are the w(t)s). Further recall that we fit the dynamical models given in Equations 1-3 which do not contain w(t)s; that is, the data analyst is unaware of these momentary events.

The results of this simulation are summarized in Table 2 (lower panel). As before, the adaptive intensive intervention resulted in the lowest level of mean smoking urge, lowest mean negative affect and highest mean self-efficacy. Thus even though the data analyst is unaware of the momentary events, the intervention algorithm produces improvements in smoking urge with less treatment. The average number of treatments in the adaptive intervention was slightly higher than in the scenario with no unmeasured momentary events, as can be seen by comparing the lower and upper panels of Table 2. Of course the algorithm cannot completely adjust for the unobserved momentary events, as can be seen by comparing the mean smoking urge reported in the lower panel of Table 2 to that in the upper panel.

In this setting, for a new individual, the mean differences between smoking urge under no treatment and smoking urge under the adaptive intervention or full treatment intervention are .27 (SE=.004) and 0.17 (SE=.002), respectively, which is very similar to those in population A. That is, both the full treatment intervention and the adaptive intervention reduce smoking urge; however, the adaptive intervention is more effective and uses less treatment. The histograms of the mean difference in smoking urge between no treatment and adaptive treatment or the full treatment intervention are quite similar to that for population A and thus are not shown.

Discussion

Adaptive intensive interventions are of increasing interest across a variety of health behavior change settings. They are particularly promising in settings in which behavior change is challenging and/or difficult to maintain. This is because the treatment can be adapted to individual progress, provided over longer periods of time, and strategically provided so as to prevent negative effects of participant burden. The possible applications across the behavioral intervention field are numerous, including momentary interventions to encourage meditation practice or increase self-awareness (e.g., of stress) and using momentary GPS data to detect locations associated with greater risk (e.g., of alcohol use relapse) indicating immediate/increased need for contact from a trained professional.

Tools from engineering have the potential to transform how many behavioral interventions are designed and to maximize their effectiveness. Here we illustrated how a collaborative team of behavioral scientists and a control engineer can use ILD to design an adaptive intensive intervention. Our simulation study showed that when the dynamical model incorporates the main characteristics of the behavior, the resulting adaptive intervention performs well even in the presence of substantial uncertainties. That is, the control engineering approach is able to accommodate settings in which patient behavior and treatment response cannot be precisely modeled; rather, this approach takes into account a degree of (perhaps large) uncertainty in the design of an adaptive intensive intervention. This is often the case in settings in which control engineering is used (Khalil, 2002). One reason for the robust performance is the adaptive nature of the intervention. At each time point, the adaptive intervention algorithm uses both the individual's current status and a prediction of the individual's future status to recommend treatment. Moreover, again by looking at the current behavior of the individual, this algorithm detects when participants deviate from model predictions and adjusts recommendations accordingly.

Our simulated example concerned when to provide treatment and how to adapt these decisions to each individual as they progress; adaptive intensive interventions can also adapt the specific type of treatment to the individual. To assist in achieving the promise of adaptive intensive interventions, methods that can use ILD to inform the design of high-quality, adaptive intensive interventions are needed. The control engineering approach described here provides a principled approach to designing adaptive interventions based on available ILD.

From a clinical perspective, several challenges remain for implementing these engineering methods to design effective adaptive intensive interventions. First, dynamical models for modeling behavior change are required; work in this direction is underway (e.g., Thomas et al., 2012; Riley et al., 2011; Navarro-Barrientos, Rivera & Collins, 2011). These models must allow for discrete and semi-continuous variables, both prevalent across many areas of behavioral change. Second, an objective function as in Equation 4 must be specified. There are many possible choices for the objective function; further research is needed to determine which objective functions are best suited for use in the development of adaptive intensive interventions. Third, although exposure to treatment need not be randomized in the data, the approach used in this study assumes that there are no confounders (i.e., third variables) that explain an observed association between treatment and the outcome. Thus, as is always the case, researchers who wish to estimate treatment effects using observational data need to strive to collect important confounders. Fourth, in the example, data was collected at equidistant time instants, and the example had no missing data. It is crucial to eliminate these assumptions. For example, inconsistent spacing of measurement times can be handled in the error terms: one can round the time labels and use them as if they were true time; the error this creates can be incorporated in the error terms of the model, since the approach described here is robust with respect to the errors. Multiple imputation (Schafer, 1997) is one well-known approach for accommodating missing data; how best to combine multiple imputation with control engineering methods requires further research.

Lastly, the effectiveness of an adaptive intensive intervention developed via the use of a control systems approach should be empirically tested in a randomized controlled clinical trial. Depending on the setting, a comparison group might be assigned to decision rules developed using more traditional approaches or a comparison group might be assigned only self-report data collection or standard care.

Through collaborations with engineers, clinicians and behavioral scientists can provide alternative or supplemental interventions that are principled, cost-effective, and more efficacious than standard treatment, ultimately improving mental and physical health of patients. Such work will advance clinical and behavioral interventions by, for example, reducing variability across time in symptoms by providing treatment when needed, adapting treatments based on patients' previous and current states, and compensating for uncertainty that will always exist in real-world settings. These methods will lead to improved operationalization of performance criteria and then to intervention designs that target these criteria (Rivera, Pew, Collins, & Murphy, 2005). In sum, the new methods described here hold the potential to transform how behavioral interventions are conceived of, designed, and conducted so that treatments can be given intensively over time in a way that maximizes effectiveness while minimizing usage of resources and participant burden.

Appendix A

Technical Details Concerning the Objective Function and MPC

In practice, three tuning parameters need to be specified (see below for discussion of how to set the values of these tuning parameters). First, often a tuning parameter θ is subtracted from each smoking urge; θ can be interpreted as the average level of smoking urge achievable by particular treatment(s). Second, when there are many time points per individual, minimizing Equation 4 becomes computationally intractable. Usually one only considers a sum over K time points,

$$\sum _{k=t+2}^{k=t+K}{[\mathit{\text{SU}}(k)-\theta ]}^2.$$ (A1)

This introduces a second tuning parameter, K. Third, the total amount of treatment provided over a time period including recent past and the “future time window” of size K is usually constrained, say by M units; M is the third tuning parameter. The rationale for adding this constraint is to reflect the concern that participants may experience treatment fatigue above the level of fatigue that can be represented by the models in Equations 2 and 3. Imposing a constraint on the total amount of treatment allows us to accommodate long-term treatment fatigue without making these models more complicated (e.g., including terms for time lags greater than 2). Next note that for any k in Equation A1, SU(k) (see Equation 1) depends on the error ε(k). Moreover, it also depends on SE, NA and, given equations 2 and 3, this implies that SU(k) also depends on errors ε^na(u), ε^se(u) for time points u<k. These errors are unobservable. A robust approach is to first maximize Equation A1 over a grid of values for these normal error terms and then minimize the result to obtain the optimal level of treatment for individual i at time t.

The adaptive intervention design algorithm is completely specified by Equation A1 and values for the tuning parameters, θ, K, and M. Values for the tuning parameters should be selected based on simulation studies completed prior to providing treatment; the values of the tuning parameters that produce the best adaptive intervention in simulation are then used in implementation. The guidelines that we used in our simulations to choose tuning parameter values were the following: (1) θ was specified to reflect our goal of significantly reducing smoking urge. More precisely, our objective was approximately a 30 percent improvement over mean smoking urge without treatment (this translated to θ=2 since mean smoking urge without treatment is approximately 3; see table 2 for details). We did not choose θ=0 because this is not realistic for the population in this study. Careful analysis of the data showed that a zero smoking urge can only be achieved for short periods of time under full treatment and it is not sustainable in the long term. (2) We specified K=20 time points; the choice of K is usually limited by computational resources: larger values lead to more computationally complex optimization problems. (3) M=26/3 (treatment approximately once per day) was chosen based on data analyses to determine the threshold for the number of consecutive treatments that led to a significant drop in effectiveness. Recall that, in the simulation study here, one has the possibility of providing treatment 3 times per day. The number M=26/3 then means that we look at the number of treatments provided in the last two days (6 instances of possible treatment) and at the window of future treatments that we are considering (K=20) and restrict the treatment schedule in such a way as to avoid providing treatment more than one third of the time; i.e., in average, over this time interval we only provide one treatment per day. In some cases, previous research or theoretical knowledge can guide selection of the tuning parameters, for example by specifying the maximum amount of treatment that participants are expected to tolerate. Further details of the algorithm can be found in Appendix C available at http://personal.psu.edu/cml18/jccp.

Appendix B

Simulated Data for 150 Participants

The simulation models used to generate each of the simulated participants are

$$\mathit{\text{SU}}(t)=1.1+.50\mathit{\text{SU}}(t-1)+.35\mathit{\text{SU}}(t-2)+.01\mathit{\text{NA}}(t-1)-.12\mathit{\text{SE}}(t-1)-.01\mathit{\text{SE}}(t-2)+w(t)+\epsilon (t)$$ (B1)

$$\mathit{\text{NA}}(t)=.15+.01\mathit{\text{SU}}(t-1)+.49\mathit{\text{NA}}(t-1)+.24\mathit{\text{NA}}(t-2)-.05\mathit{\text{SE}}(t-1)+{\mathit{\text{treat}}}^{\mathit{\text{NA}}}\left(\overline{T}(t)\right)+{\epsilon }^{\mathit{\text{na}}}(t)$$ (B2)

$$\mathit{\text{SE}}(t)=.52-.01\mathit{\text{SU}}(t-1)-.02\mathit{\text{NA}}(t-1)+.49\mathit{\text{SE}}(t-1)+.36\mathit{\text{SE}}(t-2)+{\mathit{\text{treat}}}^{\mathit{\text{SE}}}\left(\overline{T}(t)\right)+{\epsilon }^{\mathit{\text{se}}}(t)$$ (B3)

where Var(ε(t)) = 1.2, Var(ε^na(t)) = 0.6, Var(ε^na(t)) = 0.3 for t=1, …, 150 and {ε(1), ε^na(1), ε^se(1), ε(2), ε^na(2), ε^se(2), …, ε(150), ε^na(150), ε^se(150)} are independent, normal, mean zero errors. The term w(t) reflects random disturbances (i.e., unobserved momentary events that affect urge to smoke at that moment, e.g., family stress, health issues³).

From Equation B1 we see that higher smoking urge, more negative affect, and lower self-efficacy at the previous time predicts higher smoking urge at time t. In Equation B2 we see that higher smoking urge, more negative affect, and lower self-efficacy at past time points predicts more negative affect at time t. Finally, in Equation B3, lower smoking urge, less negative affect, and higher self-efficacy at past time points predicts higher self-efficacy at time t.

The treatment terms treat^NA (T̅(t)) in Equation B2 and treat^SE (T̅(t)) in Equation B3 are given by

$${\mathit{\text{treat}}}^{\mathit{\text{NA}}}\left(\overline{T}(t)\right)=-0.2T(t-1)+0.2\mathit{\text{logistic}}({\sum }_{u=t-44}^{u-2}T(u))T(t-2)$$ (B4)

and

$${\mathit{\text{treat}}}^{\mathit{\text{SE}}}\left(\overline{T}(t)\right)=0.2T(t-1)-0.2\mathit{\text{logistic}}({\sum }_{u=t-44}^{t-2}T(u))T(t-2).$$ (B5)

See Figure B1 for the form of the logistic function; the value of this logistic function is close to 1 if the total amount of treatment, ${\sum }_{u=t-44}^{t-2}T(u)$, is large; it is close to 0 if the total amount of past treatment is close to 0. The coefficients of T(t − 1) imply that providing treatment at time t-1 reduces negative affect and increases self-efficacy. However, providing treatment many times during the last 2 weeks (e.g., ${\sum }_{u=t-44}^{t-2}T(u)$ is large) produces treatment fatigue, to the extent that providing treatment at time t-2 greatly reduces the effect of treatment at time t-1 on both negative affect and self-efficacy.

Figure B1.

The function logistic is a logistic function of the form $\mathit{\text{logistic}}(x)=\frac{1}{1+e^{-(x-17)}}$.