Dynamic Treatment Regimes for Managing Chronic Health Conditions: A Statistical Perspective

Abstract

Dynamic treatment regimes are an emerging and important methodological area in health research, particularly in the management of chronic health conditions. This paradigm encompasses the ideological shift in research from the acute care model to the chronic care model. It allows individualization of treatment (type, dosage, timing) at each stage of intervention. Constructing evidence-based dynamic treatment regimes requires implementation of cutting-edge design and analysis tools. Here I briefly discuss some of these modern tools, namely the sequential multiple assignment randomized trial (SMART) design and a regression-based analysis approach called Q-learning.

Chronic disorders are one of today's most pressing public health issues in both the American¹ and global² arenas. For example, widely prevailing conditions such as hypertension, obesity, diabetes, nicotine addiction, alcohol and drug abuse, HIV infection, and depression are all chronic. In many cases, effective long-term care of patients with these chronic conditions requires ongoing medical intervention following the chronic care model,^1,3 rather than the more traditional acute care model. Some of the key features of health care emphasized by the chronic care model are individualization of care according to patient needs, optimization of patient outcomes through a series of interventions, and health services based on evidence (as opposed to expert opinion only).

First, in this paradigm clinicians treat patients in multiple stages, individualizing treatment type or dosage according to ongoing measures of patient response, adherence, burden, side effects, and preference. Second, instead of determining a single course of treatment (static treatment), clinicians sequentially make decisions about what to do next to optimize patient outcomes given their case history (dynamic treatment). The primary motivations for considering sequences of treatments are high interpatient variability in response to treatment, probability of relapse, presence or emergence of comorbid conditions, time-varying severity of side effects, and reduction of costs and burden when intensive treatment is unnecessary.⁴

Third, although there exist traditional practice guidelines for clinicians that are primarily based on expert opinions, the chronic care model advocates that these regimes be more objective and evidence based. In fact, Wagner et al. described the chronic care model as “a synthesis of evidence-based system changes intended as a guide to quality improvement and disease management activities.”^3(p69)

In this context, dynamic treatment regimes (DTRs) offer a way to operationalize the sequential decision-making process involved in adaptive clinical practice and thereby a potential way to improve it. Formally, a DTR is a sequence of decision rules, 1 per stage of intervention. Each decision rule takes a patient's individual characteristics and treatment history observed up to a given stage as input and offers a recommended treatment at that stage (recommendations can include treatment type, dosage, and timing). Conceptually, a DTR can be viewed as a decision support system, which is 1 of the 6 elements of the chronic care model.³

DTRs are developed to define the sequence of treatments that will result in the most favorable clinical outcome possible. A DTR is optimal if it optimizes the mean long-term outcome (e.g., the outcome observed at the end of the final stage of treatment). A concrete example of a DTR, originally described by Murphy,⁵ can serve as an illustration.

ADDICTION MANAGEMENT EXAMPLE

Consider a simple addiction management study involving alcohol-dependent participants, with only 2 stages of decision: choosing the initial treatment and choosing the secondary treatment. Initially the clinician may prescribe either an opiate antagonist (naltrexone) or cognitive–behavioral therapy (CBT). Participants are classified as treatment responders or nonresponders according to their level of heavy drinking in the subsequent 2 months while they are on initial treatment. If a participant is a nonresponder to naltrexone, the clinician must decide whether to switch to CBT or augment naltrexone with CBT and an enhanced motivational program (i.e., enhanced motivation + CBT + naltrexone). If a participant is a nonresponder to CBT, the clinician must decide whether to switch to naltrexone or augment CBT with naltrexone and an enhanced motivational program (enhanced motivation + CBT + naltrexone).

Responders to the initial treatment can be assigned either to telephone monitoring or to telephone counseling and monitoring. In this study setup, researchers can formulate a DTR that results in the highest percentage of days abstinent over a 1-year period. This DTR would consist of 2 decision rules: in the first rule, pretreatment information (e.g., level of addiction) is used to select the initial treatment, and in the second rule intermediate outcomes (e.g., adherence to initial treatment, self-management skill level, number of heavy drinking days while on initial treatment) can be used to choose the secondary treatment.

A natural question that arises at this point is how to actually develop evidence-based DTRs. One simple yet rigorous approach is to initially specify the decision rules in terms of certain unknown parameters and then to use patient data to estimate them. For example, in the addiction management study example, a sensible stage 1 decision rule is to prescribe naltrexone if the participant's baseline level of addiction (e.g., number of heavy drinking days in a prespecified period) is greater than a threshold value (ψ), and to prescribe CBT otherwise (ψ is an unknown parameter). If the decision rule is to be truly evidence based, ψ needs to be estimated from data in a principled way (rather than specified by expert opinion).

The data needed to develop an optimal DTR (e.g., to estimate ψ) can be obtained from either observational studies or randomized studies. It is well known that estimates based on observational data are often subject to selection or confounding bias, and hence randomized data are preferable, allowing more accurate estimations and stronger statistical inferences. This is especially important when dealing with DTRs, given that these biases can compound over stages.

One crucial point to note here is that developing DTRs is an exploratory (developmental) procedure rather than a confirmatory procedure. Randomized controlled trials are typically the gold standard for evaluating or confirming the efficacy of a newly developed intervention but not for developing the intervention per se. Thus, generating meaningful data for developing optimal DTRs is beyond the scope of the usual confirmatory randomized controlled trial; special design considerations are required. A special class of designs, sequential multiple assignment randomized trial (SMART)^5–7 designs, are well suited for developing optimal DTRs.

SEQUENTIAL MULTIPLE ASSIGNMENT RANDOMIZED TRIAL DESIGNS

SMART designs involve an initial randomization of patients to possible treatment options, followed by rerandomizations at each subsequent stage of some or all of the patients to another treatment available at that stage. The rerandomizations at each subsequent stage may depend on information collected after previous treatments but prior to new treatment assignment (e.g., how well the patient responded to the previous treatment). Thus, even though a participant is randomized more than once, ethical constraints are not violated.

Examples of the use of the SMART design (or its precursors) include the Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) for treatment of Alzheimer's disease,⁸ the Sequenced Treatment Alternatives to Relieve Depression (STAR*D) trial,⁹ 2-stage cancer trials,^10,11 2-stage smoking cessation trials,^12,13 and a 2-stage trial designed to reduce mood and neurovegetative symptoms among patients with malignant melanoma.¹⁴ SMART designs attempt to conform to the clinical practice procedures used in treating chronic disorders but retain the well-known virtues of randomization (compared with observational studies). Figure 1 presents a schematic of a SMART design for the addiction management example.

FIGURE 1.

Sequential multiple assignment randomized trial (SMART) design schematic for the addiction management example.

Note. CBT = cognitive−behavioral therapy; EM = enhanced motivation; NTX = naltrexone; R = randomization; TM = telephone monitoring; TMC = telephone counseling and monitoring.

A competing approach to determining an optimal DTR could be to conduct separate randomized controlled trials for separate stages, to find the optimal treatment at each stage based on the data from these trials, and then to combine these optimal treatments from individual stages to create a DTR. However, this design strategy is myopic and may often result in a suboptimal DTR.⁶ Many treatments have effects, such as improving the impact of a future treatment or alleviating long-term side effects that prevent a patient from being able to use an alternative useful treatment in the future, that do not occur until after the intermediate outcome (e.g., response to initial treatment) has been measured. SMART designs are capable of addressing these issues.

This point can be further elucidated with the addiction management example. Suppose telephone counseling and monitoring is more effective than telephone monitoring alone among CBT responders (i.e., the participant learns to use counseling during CBT and thus is able to take advantage of the counseling offered). Individuals who received naltrexone during the initial treatment would not have learned to use counseling, and thus among responders to naltrexone the addition of counseling would not improve abstinence levels relative to monitoring alone. If an individual is a CBT responder, it is best to offer telephone counseling and monitoring as the secondary treatment. If the individual is a naltrexone responder, however, it is best to offer the less expensive telephone monitoring as the secondary treatment.

In summary, even if CBT and naltrexone result in the same proportion of responders (or even if CBT appears less effective at the initial stage), CBT may be the best initial treatment as part of the DTR owing to the enhanced effect of telephone counseling and monitoring when it is preceded by CBT. This issue, often referred to as a delayed effect, is important to consider when determining an optimal DTR. Furthermore, even though the results of the initial stage may indicate that 1 treatment is less effective than another, the former treatment may elicit diagnostic information that allows the investigator to better match the subsequent treatment to each participant and thus improve the primary outcome. Also, participants who enroll and remain in single-stage trials may be different from those who enroll and remain in a SMART study (such cohort effects have been discussed by Murphy et al.¹⁵).

As is the case with any study, sample size calculation is a crucial part of SMART design. In a SMART study, one can investigate multiple research questions, both concerning entire DTRs and concerning certain components thereof. In the case of powering a SMART design, however, the investigator needs to choose a primary research question (primary hypothesis) and calculate sample size on the basis of that question. In addition, 1 or more secondary research questions can be investigated. Although SMART provides unbiased answers (free from confounding) to these secondary questions by virtue of randomization, it does not necessarily have the desired power to address such secondary hypotheses.

Hypotheses concerning components of a DTR in the addiction management study example are as follows: after control for secondary treatments, the initial naltrexone treatment will result in the same mean outcome (number of days abstinent) as the initial CBT treatment, and among nonresponders to the initial treatment (naltrexone or CBT), a switch to the other treatment (CBT or naltrexone, as the case may be) will be as effective as treatment augmentation (enhanced motivation + CBT + naltrexone; Figure 1). For the first primary hypothesis, the sample size formula is the same as that for a 2-group comparison. For the second primary hypothesis, the sample size formula is the same as that for a 2-group comparison of nonresponders, and it is thus a function of the nonresponse rate to initial treatment.

An example hypothesis concerning the entire DTR is that 2 DTRs differing in initial treatment will have the same mean outcome. In this case, the formula for the estimated mean outcome of a DTR given by Murphy⁵ is used in calculating the sample size. Oetting et al.¹⁶ provided sample size formulas for each of these choices of primary hypothesis under different working assumptions; Feng and Wahed¹⁷ and Li and Murphy¹⁸ provided sample size formulas for time-to-event (survival) outcomes. However, there are still open questions relating to sample size issues in a SMART design that warrant further research.

ANALYSIS OF SEQUENTIAL MULTIPLE ASSIGNMENT RANDOMIZED TRIAL DATA

For clarity, I consider SMARTs with only 2 stages of treatment, as in the addiction management example (however, the analysis method described subsequently can be generalized to more stages as well). Longitudinal data on a single patient participating in a 2-stage SMART study are given by the trajectory (O₁,A₁,O₂,A₂,Y), where O₁ denotes all of the pretreatment information (possibly a vector) at the beginning of stage 1, A₁ is the treatment assigned at stage 1, O₂ denotes all of the intermediate observations (possibly a vector) made on the patient prior to treatment at the beginning of stage 2, A₂ is the treatment assigned at stage 2, and Y is the primary outcome (either an end-of-the-study observation or a summary of observations made throughout the study).

Assume throughout that a higher value of Y is favorable. For example, in the addiction management study, O₁ could be baseline level of addiction, A₁ could be either naltrexone or CBT, A₂ could be responder–nonresponder status (based on number of heavy drinking days during treatment), and O₂ could be 1 of the 2 treatment possibilities available depending on A₁ and O₂ (Figure 1). Finally, the primary outcome Y could be the percentage of days abstinent over the study period.

For simplicity, consider binary treatments at each stage, coded −1 or 1, and continuous Y. Suppose that data are available for n patients, where the data trajectory for the ith patient is given by

To develop an analysis strategy for a 2-stage SMART design, one can initially conceptualize a simple, single-stage study with data trajectory (O₁,A₁,Y). To make treatment recommendations for different subgroups of patients (i.e., patients with different baseline information O₁), one needs to understand the behavior of the conditional mean outcome E(Y |O₁,A₁) as a function of O₁ and A₁. A reasonable approach to doing so is to consider a regression model of Y on O₁ and A₁, keeping an interaction term between O₁ and A₁ in the model:

The superscript T in this expression denotes the vector transpose. The first part of the model (i.e., ), although relevant for predicting the outcome Y, is not relevant for making a treatment decision (because this part remains the same for either choice of A₁). It is the second part, , that governs the treatment decision. The parameters in this model can be estimated via the usual least squares method, with , and denoting the estimates. Then the estimated optimal treatment decision rule recommends the treatment that, for a given value of O₁, maximizes the estimated conditional mean outcome. In other words, this rule recommends the treatment A₁ = 1 if the quantity and recommends the treatment A₁ = −1 otherwise. If O₁ is a single variable (in which case ), such as the number of heavy drinking days in a prespecified period prior to receipt of treatment, the rule amounts to prescribing A₁ = 1 if and A₁ = −1 otherwise. In the addiction management example (considering only a single stage of intervention of naltrexone vs CBT), if naltrexone is denoted by A₁ = 1 and CBT by A₁ = −1, then the estimated decision rule prescribes naltrexone if the number of heavy drinking days is greater than the threshold value and prescribes CBT otherwise. Thus, using the regression approach, one can specify a very intuitive decision rule (as discussed in the addiction management example).

With this background, one can think about the analysis strategy for the 2-stage SMART with a patient's data trajectory given by (O₁,A₁,O₂,A₂,Y). A natural extension of the approach just outlined would be to model the conditional mean outcome E(Y |O₁,A₁,O₂,A₂) and run an all-at-once regression analysis. Unfortunately, this is not a good option because of the possibility of bias in the estimation of the stage 1 treatment effect arising as a consequence of what is known as Berkson's paradox.^5,19 This phenomenon can be explained with the help of the addiction management example.

Suppose that there is an unobserved variable (U) that affects a patient's ability to respond to treatment. For simplicity, conceptualize U as the stability in a patient's life (U = 1 if stable and U = 0 otherwise). U can be expected to be positively correlated with the intermediate outcome responder–nonresponder status (O₂) as well as with Y, percentage of days abstinent. Suppose that the initial treatments have differing effects on responder–nonresponder status (O₂). Because the initial treatment assignment is randomized, U and A₁ should be uncorrelated. However, there will be a conditional correlation between U and A₁, given nonresponse to the initial treatment (i.e., given O₂).

Intuitively, a nonresponder who received the better initial treatment is more likely to have an unstable life (U = 0). This phenomenon is called Berkson's paradox. Note that when running a regression of Y on (O₁,A₁,O₂,A₂), one conditions on O₂. Conditionally on O₂, the unobserved variable U and A₁ will be correlated. This correlation, coupled with the correlation between U and Y, will induce a spurious (noncausal) correlation between A₁ and Y (even though A₁ is randomized). As a consequence, the stage 1 treatment effect will be estimated with bias.

Figure 2 (where O₁ and A₂ are excluded to simplify the diagram) may help provide a visual understanding of the situation. The direct (solid) arrow from A₁ to Y is the true stage 1 treatment effect that the researcher needs to estimate. However, because U is unobserved and thus not included in the regression, the spurious effect arising from Berkson's paradox, represented by the dotted path from A₁ to Y via O₂ and U, contaminates the true effect of A₁. Thus, combining the stage 1 and stage 2 variables in a single regression is potentially problematic. However, any method that proceeds in a stage-by-stage manner does not suffer from this problem. One such method is Q-learning, originally developed in computer science^20,21 and later adapted to statistics.^12,22

FIGURE 2.

Diagram displaying the spurious effect between A₁ and Y as a consequence of Berkson's paradox.

Note. A₁ = treatment assigned at stage 1; O₂ = observations at beginning of stage 2; U = unobserved variable; Y = primary outcome.

Q-LEARNING

A useful first step is to formally define the case history of a patient at any given stage of intervention. At stage 1, the history (H₁) simply consists of the baseline information (O₁). At stage 2, the history (H₂) consists of the baseline information (O₁), the stage 1 treatment (A₁), and stage 2 pretreatment variables (intermediate outcomes, O₂). In general, a patient's case history consists of the collection of all previous and current observations and all previous treatments (for purposes of analysis, one can work with a lower-dimensional summary statistic of the entire history). The next step is to formally define what are called the Q-functions (a term coined in the computer science literature, where Q denotes “quality of treatment”) for the 2 stages as follows:

Note that although Q₂ is the conditional mean function as in the simple single-stage study, the definition of Q₁ is slightly more involved. It represents the conditional expectation of an unobserved pseudo-outcome given stage 1 history and treatment. The pseudo-outcome, , represents a patient's best possible mean outcome if she or he is given the best treatment at stage 2.

Also note that Q₁ is defined as a function of Q₂, which means that Q₂ needs to be specified first. In other words, the definitions of Q-functions move backward in time. Recall that in managing chronic disorders, the goal is to optimize the long-term mean outcome rather than any immediate (intermediate) outcome so that delayed effects can be addressed. Thus, unless long-term outcomes are modeled first (and thereby optimal future treatments are determined), one cannot make optimal decisions in the current stage. This intuition justifies the backward movement of the definition of Q-functions. This general approach is known as backward induction or dynamic programming.²³ If the 2 Q-functions were known (e.g., as when the true multivariate distribution of the data is known), then the optimal DTR, for example d = (d₁, d₂), would be given by

In this equation, d_j is a decision rule at the jth stage; for a patient with history H_j, this decision rule recommends the treatment that maximizes the Q-function at the jth stage (j = 1,2). In practice, however, the true Q-functions are rarely known and hence must be “learned” (estimated) from the data. This process is called Q-learning. Note that Q-functions are conditional expectations, and hence they can be estimated via a regression approach. Consider linear regression models for the Q-functions. The Q-function at stage j (j = 1,2) can be modeled as

where the vector H_j includes the scalar “1” so that the model has an intercept term. The first term on the right side, , does not change with treatment. The second term, , denotes the interaction between history and treatment and is crucial for decision making. The Q-learning algorithm proceeds as follows:

1. Estimate the stage 2 parameters (β₂, ψ₂) by regressing Y_i on (H_2i, A_2i), i = 1, …, n, using the model for Q₂ via least squares; label the estimates .

2. Construct (the sample version of) the stage 1 pseudo-outcome: .

3. Estimate the stage 1 parameters (β₁, ψ₁) by regressing on (H_1i, A_1i), i = 1, …, n using the model for Q₁ via least squares; label the estimates .

Once the 2 Q-functions have been estimated, the estimated optimal treatment decision rules are given by

That is, for a patient with history H_j, the estimated optimal decision rules recommend the treatments that maximize the estimated Q-functions. Because A_j is binary (coded −1 or 1) and Q_j is specified by the model in equation 6, it follows that A_j = 1 maximizes Q_j if , and A_j = −1 maximizes Q_j otherwise.

A rule such as that just described is quite realistic and, again, can be illustrated with the addiction management example. It was noted in the addiction management example how a stage 1 rule, such as to prescribe naltrexone if the number of heavy drinking days is greater than a threshold and prescribe CBT otherwise, falls under this framework. Now focus on stage 2. O₂ = 1 denotes responders to initial treatment, and O₂ = 0 denotes nonresponders. For any patient, there are only 2 treatment options at stage 2 that depend on case history. More precisely, for responders to initial treatment (i.e., for patients with O₂ = 1), A₂ is either telephone monitoring (1) or telephone counseling and monitoring (−1). For nonresponders to naltrexone (i.e., for patients with A₁ = 1, O₂ = 0), A₂ is either CBT (1) or enhanced motivation + CBT + naltrexone (−1). For nonresponders to CBT (i.e., for patients with A₁ = −1, O₂ = 0), A₂ is either naltrexone (1) or enhanced motivation + CBT + naltrexone (−1; Figure 1).

Now suppose that the optimal stage 2 treatments are telephone monitoring (A₂ = 1) for responders to naltrexone (A₁ = 1, O₂ = 1), telephone counseling and monitoring (A₂ = −1) for responders to CBT (A₁ = −1, O₂ = 1), and enhanced motivation + CBT + naltrexone (A₂ = −1) for all nonresponders (O₂ = 0). Then the optimal rule can be compactly stated as “prescribe A₂ = 1 if A₁O₂ > 0, and prescribe A₂ = −1 otherwise.” Thus, the interaction term A₁O₂ serves as the only relevant history for decision making at stage 2 (i.e., H₂ = A₁O₂), and the parameter is simply equal to 1. Applying Q-learning can lead investigators to such a rule.

In summary, Q-learning involves 2 separate regression analyses, 1 per stage, instead of an all-at-once regression of Y on (O₁,A₁,O₂,A₂). As a result, any spurious path between the outcome Y and the stage 1 treatment A₁ via the unmeasured confounding variables is broken, and thus Berkson's paradox is avoided. This method can easily be generalized to more than 2 stages (i.e., to studies in which patients are randomized to treatments more than twice). Implementation of the algorithm is relatively easy as well; standard software can be used. Chakraborty et al.¹² provided an application of Q-learning (the basic version discussed here as well as an improved version) to analyze a 2-stage smoking cessation trial¹³ with the goal of determining the optimal DTR (2-stage behavioral intervention).

DISCUSSION AND CONCLUSIONS

Dynamic treatment regimes offer an important methodological framework for developing evidence-based interventions to manage chronic disorders. As mentioned earlier, this framework encompasses the ideological shift in research from the acute care model to the chronic care model. Furthermore, instead of adhering to the old-school “one-size-fits-all” principle of health intervention, it allows individualization of treatment type, dosage, and timing at each intervention stage. The methodology for constructing optimal evidence-based DTRs has developed relatively recently at the interface of statistics and computer science. Hence, most public health researchers are unfamiliar with these new developments. I have attempted to briefly present the key concepts and tools needed to construct an optimal DTR.

This article is in no way intended to provide a complete review of all of the methods available for analyzing data from SMART designs (and longitudinal observational studies) and thereby constructing optimal DTRs. There exist other methods, of course, such as the likelihood-based methods developed by Thall et al.^10,24,25 and the semiparametric methods developed by Murphy²⁶ and Robins.²⁷ Moodie et al.²⁸ offered a useful discussion of DTRs, including the connection between different semiparametric methods of estimation and their application to observational data. Nonetheless, Q-learning is probably the simplest method and one that I hope will appeal to a wider readership. I believe that the DTR paradigm has great promise for application in a variety of medical and public health research projects, in particular the management of chronic diseases.