Estimating tree-based dynamic treatment regimes using observational data with restricted treatment sequences

Abstract

A dynamic treatment regime (DTR) is a sequence of decision rules that provide guidance on how to treat individuals based on their static and time-varying status. Existing observational data are often used to generate hypotheses about effective DTRs. A common challenge with observational data, however, is the need for analysts to consider “restrictions” on the treatment sequences. Such restrictions may be necessary for settings where (1) one or more treatment sequences that were offered to individuals when the data were collected are no longer considered viable in practice, (2) specific treatment sequences are no longer available, or (3) the scientific focus of the analysis concerns a specific type of treatment sequences (eg, “stepped-up” treatments). To address this challenge, we propose a restricted tree–based reinforcement learning (RT-RL) method that searches for an interpretable DTR with the maximum expected outcome, given a (set of) user-specified restriction(s), which specifies treatment options (at each stage) that ought not to be considered as part of the estimated tree-based DTR. In simulations, we evaluate the performance of RT-RL versus the standard approach of ignoring the partial data for individuals not following the (set of) restriction(s). The method is illustrated using an observational data set to estimate a two-stage stepped-up DTR for guiding the level of care placement for adolescents with substance use disorder.

1 |. INTRODUCTION

A dynamic treatment regime (DTR) is a sequence of decision rules that provide guidance on how to treat individuals based on their static and time-varying status (Murphy, 2003). In a DTR, a patient’s observed past and current disease status and medical history, including treatments offered in the past or adherence to those treatments, are used to decide what treatment to offer next.

Various methods have been developed for estimating an optimal DTR from observational data, where optimal is defined as maximizing the expected value of a desired health outcome. Methods such as marginal structural models with inverse probability weighting (IPW) (Murphy et al., 2001; Wang et al., 2012), Q- and A-learning methods (Murphy, 2003; Nahum-Shani et al., 2012; Schulte et al., 2014; Sutton & Barto, 1998; Watkins & Dayan, 1992), and dynamical system models (Rivera et al., 2007) stand out as the most commonly used methods for evaluating an optimal DTR.

New approaches for estimating optimal “tree-based DTRs” have been proposed more recently. Laber and Zhao (2015) introduced tree-based learning for estimating a single-stage treatment decision rule; they employed a novel inverse probability weighted purity measure to guide the tree construction. Tao et al. (2018) generalized the method with augmented AIPW for use with multi-stage multi-treatment observational data to estimate an optimal tree-based DTR (ie, multistage treatment regime). Sun and Wang (2021) proposed a stochastic tree search method. In a tree-based DTR, the decision rule at each stage is a collection of nested binary splits. Each split is on a value (ie, a threshold) along the range of one of the possible inputs for the decision rule. As a result, tree-based DTRs are potentially more easily interpretable than other types of DTRs because the thresholds are values on already interpretable metrics and scales. To better appreciate this distinction, consider Figure 1, which describes two hypothetical examples of decision rules to guide the level of care (LOC) placement for adolescents with substance use disorder, based loosely on our study’s motivating example. The general pattern is the same for both decision rules: recommend a higher LOC to individuals with severe conditions at program entry. However, the decision rule in panel (A) (tree-based DTR) is arguably easier to interpret than panel (B) (DTR that is not tree based). For panel (B), practitioners require some intuition about the meaning of the linear combination and relative “weights” of the two tailoring variables, in addition to the threshold value 3. In panel (A), on the other hand, the thresholds are more easy to understand; for individuals with mental health severity $\left(X_{1,1}\right)$ in the lower tercile of its range or with substance use severity $\left(X_{1,2}\right)$ in the lower 2 terciles of its range, the recommended treatment is outpatient care $\left(A_1=1\right)$ while otherwise, the recommendation is inpatient care $\left(A_1=-1\right)$.

FIGURE 1.

Compare a tree-based DTR (A) with a DTR that is not tree-based (B). To guide the level of care (LOC) placement for adolescents with substance use disorder, we consider two continuous candidate tailoring variables on the range [0,3] (from less to more severe), where $X_{1,1}$ = mental health severity at program entry and $X_{1,2}$ = substance use severity at program entry. Suppose the first-stage treatment recommendation $\left(A_1\right)$ can be outpatient LOC $\left(A_1=1\right)$ or inpatient LOC $\left(A_1=-1\right)$. The decision rule in panel (b) requires extra understanding of the linear combination $2X_{1,1}+X_{1,2}$ including weights (2 for $X_{1,1}$ and 1 for $X_{1,2}$) and its cutoff 3, whereas the decision rule in panel (A) can be easily understood

In this paper, we develop a further extension of methods for estimating tree-based DTRs with observational data. The extension proposes a solution to a common challenge: the need to consider “restrictions” to the class of DTRs offered to individuals when the observational data were collected. There are at least two reasons why a domain scientist or analyst would be interested in considering a restricted class of DTRs in analyses with observational data. First, there may be concerns about the current viability or availability of some treatment sequences that were offered to individuals when the observational data were collected. For example, if a drug is withdrawn completely from the market, such as with the weight-loss drug lorcaserin (US Food and Drug Administration, 2020), then analyses of observational data that include this drug (Fidler et al., 2011; O’neil et al., 2012; Smith et al., 2010) may consider restricting to the class of DTRs that does not include lorcaserin. Similarly, if a drug is recalled by the Food and Drug Administration (FDA) due to problems in manufacturing and distribution, such as with lidocaine (US Food and Drug Administration, 2021), then analyses of observational data that include this drug (Hall et al., 2016) may consider restricting to the class of DTRs that does not include lidocaine. In both examples, although in the data, some individuals are observed to be offered the drug that is no longer considered available or viable, respectively, when estimating an optimal tree-based DTR for future use, one needs to exclude such treatments at one or more stages. A second reason is if the scientific focus of the analysis concerns a specific type of DTR, but when the data were collected, some individuals were observed under a broader set of DTRs. This is the case with our motivating example, which seeks to estimate a two-stage DTR guiding the LOC placement for adolescents with substance use disorder. In the data, it was observed that adolescents receiving inpatient treatment in stage 1 could have treatment discontinued in stage 2 (ie, a “stepped down” strategy). However, the domain scientists we are working with were interested solely in the class of “stepped up” DTRs (Sobell & Sobell, 2000), whereby more resource-intensive treatments (such as inpatient treatment) are reserved as a stage 2 treatment for individuals who do not benefit from a less resource-intensive stage 1 treatment.

To address this challenge, we propose an restricted tree–based reinforcement learning (RT-RL) method that searches for a tree-based DTR with the maximum expected outcome, given a (set of) user-specified restriction(s) on treatment options at one or more stages that ought not to be considered as part of the estimated tree-based DTR. For additional protection against time-varying confounding bias and to potentially improve the method’s statistical efficiency, the proposed method employs a doubly robust augmented inverse probability weighted (AIPW) estimator as part of the algorithm.

The rest of the paper is organized as follows: In Section 2, we formalize the restricted optimization problem, describe a proposed solution for handling the patient history and treatment options, and present the RT-RL estimating procedure. In Section 3, using simulation studies, we compare the performance of the proposed method with a naive method where the observed data along the restricted sequence(s) are censored from the analysis. In Section 4, we apply the RT-RL approach to the Global Appraisal for Individual Needs (GAIN) data to estimate a two-stage stepped-up DTR to guide the LOC placement for adolescents with substance use disorder. The results of the proposed method are compared with those using the regular T-RL and with nondynamic regimes. Finally, in Section 5, we summarize and discuss the findings.

2 |. METHOD

2.1 |. Notation and setup

We consider an observational study with $n$ patients and $T$ treatment stages where there are $K_j\left(K_j\ge 2\right)$ potential treatment options at the $j\text{th}$ treatment stage, $j=1,\dots ,T$. Patients are observed to follow one of the treatments available at each stage. For brevity, we suppress the patient index $i(i=1,\dots ,n)$ when no confusion exists. Let ${\mathbf{X}}_j$ denote patient’s covariates observed during the 𝑗th treatment stage and $A_j$ denote the treatment at the $j\text{th}$ treatment stage, where $A_j$ may take a value $a_j$, a specific value for the $j\text{th}$ treatment that belongs to the stage-specific treatment space ${\mathcal{A}}_j$ containing $K_j$ treatment options. As a general notation in the following, we use an overline “-” and a subscript $j$ on a variable to denote the history of this variable up to the $j\text{th}$ stage. Thus, ${\overline{\mathbf{A}}}_{j-1}\equiv \left(A_1,\dots ,A_{j-1}\right)$ denotes a random vector of past treatments prior to stage $j$ and ${\overline{\mathbf{a}}}_T\equiv \left(a_1,\dots ,a_T\right)$ denotes a specific treatment route. We denote the clinical outcome observed following $A_j$ with $R_j$, also known as rewards, which depends on the precedent patient characteristics ${\overline{\mathbf{X}}}_j$ and previous treatments ${\overline{\mathbf{A}}}_j$. We consider the overall outcome of interest $Y$ as some functional of the reward history such that $Y\equiv f\left(R_1,\dots ,R_T\right)$, where $f(.)$ is a prespecified function (eg, sum), assuming $Y$ is bounded with larger values preferred. To recommend personalized optimal treatment for future patients at stage $j$, we would like to use the observational data to infer an optimal decision rule depending on the patient history ${\mathbf{H}}_j={\left({\overline{\mathbf{A}}}_{j-1},{\overline{\mathbf{X}}}_j^T\right)}^T\in {\mathcal{H}}_j$ at stage $j$.

However, some of the observed treatment sequences are not viable for future patients when certain “restrictions” to the class of DTRs are required as introduced in Section 1. We call such treatment routes as nonviable treatment arms, which should be excluded from the domain of the DTRs of interest during estimation. Suppose there are $M_j$ nonviable arms at stage $j$, and denote them as ${\overline{\mathbf{b}}}_{m,j}=\left(b_{1,m,j},\dots ,b_{j,m,j}\right)$ where $m=1,\dots ,M_j$ and $b_{j^{\prime },m,j}\in {\mathcal{A}}_{j^{\prime }}$ for $j^{\prime }=1,\dots ,j$. Excluding nonviable treatment arms from the observed treatment stages, we obtain a set of viable treatment sequences from stage 1 to stage $j$ denoted as ${\overline{\mathcal{A}}}_j^{res}\equiv {\overline{\mathcal{A}}}_j/\left\{{\overline{\mathbf{b}}}_{m,j},m=1,\dots ,M_j\right\}$. Correspondingly, the domain of viable patient history is ${\mathcal{H}}_j^{res}=\left\{{\mathbf{H}}_j:{\overline{\mathbf{A}}}_{j-1}\in {\overline{\mathcal{A}}}_{j-1}^{res}\right\}$. In the rest of the paper, we denote the random variable of viable patient history as ${\mathbf{H}}_j^{res}$, and the value of ${\mathbf{H}}_j^{res}$ is in ${\mathcal{H}}_j^{res}$.

We use ${\mathbf{g}}^{res}=\left(g_1^{res},\dots ,g_T^{res}\right)$ to denote a sequence of restricted viable rules for personalized treatment decisions across the different treatment stages, that is, a viable restricted DTR, where $g_j^{res}$ is a meaningful mapping function from the domain of restricted patient history ${\mathcal{H}}_j^{res}$ to ${\mathcal{A}}_j^{res}$, the range of viable treatment options at $j$, given the past. Denote the collection of such restricted meaningful mappings as ${\mathcal{G}}^{\text{res}}=\left({\mathcal{G}}_1^{\text{res}},\dots ,{\mathcal{G}}_T^{\text{res}}\right)$. Specifically, $g_j^{\mathit{\text{res}}}\in {\mathcal{G}}_j^{\text{res}}$ maps from viable patient histories to applicable stage $j$ treatments conditional on ${\overline{\mathbf{A}}}_{j-1}$ such that $\left({\overline{\mathbf{A}}}_{j-1},g_j^{res}\left({\mathbf{H}}_j^{res}\right)\right)\in {\overline{\mathcal{A}}}_j^{res}$.

To identify the optimal restricted DTR among ${\mathcal{G}}^{res}$, we consider the counterfactual framework for causal inference defined in Robins (1986). At stage $T$, let $Y^{*}\left(A_1,\dots ,A_{T-1},a_T\right)$ or $Y^{*}\left(a_T\right)$ denote the counterfactual outcome for patients had all received treatment $a_T$ at stage $T$ conditional on previous treatments. Our goal is to find the optimal one among the restricted DTRs in ${\mathcal{G}}_T^{\text{res}}$ that maximizes the expected counterfactual outcome, that is,

$$g_T^{\text{res,opt}}=\underset{g_T^{res}\left({\mathbf{H}}_T^{res}\right)\in {\mathcal{G}}_T^{res}}{argmax}E\left[\sum _{a_T\in {\mathcal{A}}_T^{res}}{Y}^{*}\left(a_T\right)I\left\{g_T^{res}\left({\mathbf{H}}_T^{res}\right)=a_T\right\}\right].$$

For any stage $j$ before $T$, we seek the best treatment decision rule by maximizing the expected counterfactual outcome with all future treatments optimized to avoid confounding. This is because the causal impact of current treatment on the expected counterfactual outcome can only be examined when future treatments are optimized and past history is controlled. We denote the counterfactual outcome at stage $j$ given ${\overline{\mathbf{A}}}_{j-1}$ and future optimal treatments $g_{j+1}^{\text{res, opt}},\dots ,g_T^{\text{res, opt}}$ as $Y^{*}\left({\overline{\mathbf{A}}}_{j-1},a_j,g_{j+1}^{\text{res, opt}},\dots ,g_T^{\text{res, opt}}\right)$. Sincesuchcounterfactual outcomes cannot be observed, we estimate the stage-wise pseudo-outcome with observed data considering viable restricted treatment space by

$$P{O}_j^{res}={\widehat{E}}^{res}\left\{Y^{*}\left(A_1,\dots ,A_j,g_{j+1}^{\text{res, opt}},\dots ,g_T^{\text{res, opt}}\right)\right\},$$

at stage $j$ for $j=1,\dots ,T-1$, where ${\widehat{E}}^{res}$ denotes the estimated expectation considering only the viable patient history and treatment routes space, $g_j^{res,opt}\in {\mathcal{G}}_j^{res}$ and $\left(A_1,\dots ,A_j\right)\in {\overline{\mathcal{A}}}_j^{res}$, which is equivalent to the recursive form

$$P{O}_j^{res}=\widehat{E}\left\{P{O}_{j+1}^{res}\mid A_{j+1}=g_{j+1}^{\text{res,opt}}\left({\mathbf{H}}_{j+1}^{res}\right),{\mathbf{H}}_{j+1}^{res}\right\}.$$

At the final stage, we define $P{O}_T^{res}=Y$. Let ${\text{PO}}_j^{res,*}\left({\overline{\mathbf{A}}}_{j-1},\right.\left.a_j\right)$, or ${\text{PO}}_j^{res,*}\left(a_j\right)$ for brevity, denote the counterfactual pseudo-outcome for a patient treated with $a_j\in {\mathcal{A}}_j^{res}$ and viable past treatments $\left(A_1,\dots ,A_{j-1}\right)\in {\overline{\mathcal{A}}}_{j-1}^{res}$. We have

$${\text{PO}}_j^{res,*}\left(g_j^{res}\right)=\sum _{a_j\in {\mathcal{A}}_j^{res}}{\text{PO}}_j^{res,*}\left(a_j\right)I\left\{g_j^{res}\left({\mathbf{H}}_j^{res}\right)=a_j\right\}.$$

Our optimization problem at stage $j(j<T)$ among the meaningful mappings becomes:

$$g_j^{\text{res,opt}}=\underset{g_j\left({\mathbf{H}}_j^{res}\right)\in {\mathcal{G}}_j^{res}}{argmax}E\left[{\text{PO}}_j^{res,*}\left(g_j^{res}\right)\right].$$

2.2 |. Constrained optimization procedure

We make the following three assumptions considering only the applicable treatment routes (Murphy et al., 2001; Robins & Hernán, 2009):

1. Consistency. The observed outcome would be the same as the counterfactual outcome that would have been observed given the same treatments, which indicates $Y={\sum }_{a_T\in {\mathcal{A}}_T^{res}}{Y}^{\ast }\left(a_T\right)I\left(A_T=a_T\right)$ for stage $T$, where $A_T\in {\mathcal{A}}_T^{res}$. Similarly, the pseudo-outcome agrees with the counterfactual pseudo-outcome $P{O}_j^{res}=\sum _{a_j\in {\mathcal{A}}_j^{res}}{\text{PO}}_j^{res,*}\left(a_j\right)I\left\{A_j=\right.\left.a_j\right\}$ for stage $j<T$ and $A_j\in {\mathcal{A}}_j^{res}$.

2. No unmeasured confounding. For any treatment sequence ${\overline{\mathbf{a}}}_T=\left(a_1,\dots ,a_T\right)$, treatment $A_j$ is independent of future outcomes (rewards), given ${\mathbf{H}}_j^{res}\in {\mathcal{H}}_j^{res}$. Specifically, $A_j\perp \left(R_j\left({\overline{\mathbf{a}}}_j\right),\dots ,R_T\left({\overline{\mathbf{a}}}_T\right)\right)\mid {\mathbf{H}}_j^{res},\forall j=1,\dots ,T$.

3. Positivity. There exist constants $0<c_0<c_1<1$ such that with probability 1 the propensity score ${\pi }_{a_j}\left({\mathbf{H}}_j^{res}\right)=Pr\left(A_j=a_j\mid {\mathbf{H}}_j^{res}\right)\in \left(c_0,c_1\right)$, for $A_j\in {\mathcal{A}}_j^{res}$.

With the three assumptions, we can bridge the pseudo-outcome derived from the observational data with the expected counterfactual pseudo-outcome for a specific $g_j^{res}\in {\mathcal{G}}_j^{res}$ at any stage $j<T$. Conditional on ${\mathbf{H}}_j^{res}$, we have

$$\begin{array}{l}E\left\{{\text{PO}}_j^{res,*}\left(g_j^{res}\right)\right\}\\ =E_{{\mathbf{H}}_j^{res}}\left[\sum _{a_j\in {\mathcal{A}}_j^{res}}E\left\{{\text{PO}}_j^{res,*}\left(a_j\right)\mid {\mathbf{H}}_j^{res}\right\}I\left\{g_j^{res}\left({\mathbf{H}}_j^{res}\right)=a_j\right\}\right].\end{array}$$

With no unmeasured confounder assumption, it is easy to show that

$$\begin{array}{l}E\left\{{\text{PO}}_j^{res,*}\left(g_j^{res}\right)\right\}\\ =E_{{\mathbf{H}}_j^{res}}\left[\sum _{a_j\in {\mathcal{A}}_j^{res}}E\left\{{\text{PO}}_j^{res,*}\left(a_j\right)\mid A_j=a_j,{\mathbf{H}}_j^{res}\right\}I\left\{g_j^{res}\left({\mathbf{H}}_j^{res}\right)=a_j\right\}\right].\end{array}$$

Then, using consistency assumption and positivity assumption, we can link

$$\begin{array}{l}E\left\{{\text{PO}}_j^{res,*}\left(g_j^{res}\right)\right\}\\ =E_{{\mathbf{H}}_j^{res}}\left[\sum _{a_j\in {\mathcal{A}}_j^{res}}E\left\{{\text{PO}}_j^{res}\mid A_j=a_j,{\mathbf{H}}_j^{res}\right\}I\left\{g_j^{res}\left({\mathbf{H}}_j^{res}\right)=a_j\right\}\right].\end{array}$$

Let ${\mu }_{j,a_j}^{res}\left({\mathbf{H}}_j^{res}\right)=E\left(P{O}_j^{res}\mid A_j=a_j,{\mathbf{H}}_j^{res}\right)$, then our goal is to find

$$g_j^{\text{res,opt}}=\underset{g_j^{res}\in {\mathcal{G}}_j^{res}}{argmax}{E}_{{\mathbf{H}}_j^{res}}\left[\sum _{a_j\in {\mathcal{A}}_j^{res}}{\mu }_{j,a_j}^{res}\left({\mathbf{H}}_j^{res}\right)I\left\{g_j^{res}\left({\mathbf{H}}_j^{res}\right)=a_j\right\}\right],$$

at stage $j$ in the space of applicable treatment options. Likewise, we have

$$g_T^{\text{res,opt}}=\underset{g_T^{res}\in {\mathcal{G}}_T^{\text{res}}}{argmax}{E}_{{\mathbf{H}}_T^{res}}\left[\sum _{a_T\in {\mathcal{A}}_T^{res}}{\mu }_{T,a_T}^{res}\left({\mathbf{H}}_T^{res}\right)I\left\{g_T^{res}\left({\mathbf{H}}_T^{res}\right)=a_T\right\}\right]$$

at stage $T$, with ${\mu }_{T,a_T}^{res}\left({\mathbf{H}}_T^{res}\right)=E\left(Y\mid A_T=a_T,{\mathbf{H}}_T^{res}\right)$.

The proposed method RT-RL utilizes the backward induction technique (Bather, 2000) to estimate the restricted DTR. We estimate the counterfactual mean outcome using an AIPW estimator similar to Tao and Wang (2017). Specifically, we modify the estimation of $E\left\{Y^{*}\left(a_T\right)\right\}$ for $a_T\in {\mathcal{A}}_T^{res}$ under the viable patient history space ${\mathcal{H}}_T^{res}$ at final stage $T$. Suppose that we have $n$ patients with $n^{res,T}$ patients received applicable treatment combinations until stage $T$. We propose to estimate $E\left\{Y^{*}\left(a_T\right)\right\}$ with ${\mathbb{P}}_{n^{res,T}}\left\{{\widehat{\mu }}_{T,a_T}^{\text{res,AIPW}}\left({\mathbf{H}}_T^{res}\right)\right\}$, where ${\widehat{\mu }}_{T,a_T}^{\text{res,AIPW}}\left({\mathbf{H}}_T^{res}\right)=\frac{I\left(A_T=a_T\right)}{{\widehat{\pi }}_{T,a_T}\left({\mathbf{H}}_T^{res}\right)}Y+\left\{1-\frac{I\left(A_T=a_T\right)}{{\widehat{\pi }}_{T,a_T}\left({\mathbf{H}}_T^{res}\right)}\right\}{\widehat{\mu }}_{T,a_T}^{res}\left({\mathbf{H}}_T^{res}\right)$, ${\widehat{\pi }}_{T,a_T}\left({\mathbf{H}}_T^{res}\right)$ is the estimated propensity score, and ${\widehat{\mu }}_{T,a_T}^{res}\left({\mathbf{H}}_T^{res}\right)$ denotes the estimated conditional mean.

Proposition 1 (Double Robustness).

Suppose patient observations ${\left\{{\overline{\mathbf{X}}}_{i,T},{\overline{\mathbf{A}}}_{i,(T-1)},A_{i,T},Y_i\right\}}_{i=1}^n$ are independent and identically distributed. $A$ subset of $n^{res,T}$ patients have viable past treatment routes until stage $T$. Without loss of generality, we define viable patient observations as ${\left\{{\mathbf{H}}_{i,T}^{res},A_{i,T},Y_i\right\}}_{i=1}^{n^{res,T}}\equiv {\left\{{\overline{\mathbf{X}}}_{i,T},{\overline{\mathbf{A}}}_{i,(T-1)},A_{i,T},Y_i\right\}}_{i=1}^{n^{res,T}}$ such that ${\overline{\mathbf{A}}}_{i,(T-1)}\in {\overline{\mathcal{A}}}_{T-1}^{res}$. ${\mathbb{P}}_{n^{\mathit{\text{res}},T}}\left\{{\widehat{\mu }}_{T,a_T}^{res,\mathit{\text{AIPW}}}\left({\mathbf{H}}_T^{res}\right)\right\}$ is a consistent estimator of $E\left\{Y^{*}\left(a_T\right)\right\}$ if either the propensity score model of ${\pi }_{T,a_T}\left({\mathbf{H}}_T^{res}\right)$ or the conditional mean model of ${\mu }_{T,a_T}^{res}\left({\mathbf{H}}_T^{res}\right)$ is correctly specified (the proof for Proposition 1 is provided in Supporting Information A).

For the final stage $T$, we propose to estimate the optimal regime with

$$\begin{array}{l}{\widehat{g}}_T^{\text{res,opt}}=\underset{g_T^{res}\in {\mathcal{G}}_T^{\text{res}}}{argmax}{\mathbb{P}}_{n^{res,T}}\left[\sum _{a_T\in {\mathcal{A}}_T^{res}}{\widehat{\mu }}_{T,a_T}^{res,AIPW}\left({\mathbf{H}}_T^{res}\right)I\left\{g_T^{res}\left({\mathbf{H}}_T^{res}\right)=a_T\right\}\right]\\ =\underset{g_T^{res}\in {\mathcal{G}}_T^{\text{res}}}{argmax}\frac{1}{n^{res,T}}\sum _{i=1}^{n^{res,T}}\left[\frac{I\left(A_{i,T}=g_T^{res}\left({\mathbf{H}}_{i,T}^{res}\right)\right)}{{\widehat{\pi }}_{T,g_T^{res}}\left({\mathbf{H}}_{i,T}^{res}\right)}{Y}_i\right.\\ \left.+\left\{1-\frac{I\left(A_{i,T}=g_T^{res}\left({\mathbf{H}}_{i,T}^{res}\right)\right)}{{\widehat{\pi }}_{T,g_T^{res}}\left({\mathbf{H}}_{i,T}^{res}\right)}\right\}{\widehat{\mu }}_{T,g_T^{res}}^{res}\left({\mathbf{H}}_{i,T}^{res}\right)\right].\end{array}$$

Similarly, for stage $j(1\le j<T)$, our proposed estimator ${\widehat{g}}_j^{\text{res,opt}}$ is

$$\begin{array}{l}\underset{g_j^{res}\in {\mathcal{G}}_j^{\text{res}}}{argmax}\frac{1}{n^{res,j}}\sum _{i=1}^{n^{res,j}}\left[\frac{I\left(A_{i,j}=g_j^{res}\left({\mathbf{H}}_{i,j}^{res}\right)\right)}{{\widehat{\pi }}_{j,g_j^{res}}\left({\mathbf{H}}_{i,j}^{res}\right)}P{O}_{i,j}^{res}\right.\\ \left.+\left\{1-\frac{I\left(A_{i,j}=g_j^{\mathit{\text{res}}}\left({\mathbf{H}}_{i,j}^{res}\right)\right)}{{\widehat{\pi }}_{j,g_j^{\mathit{\text{res}}}}\left({\mathbf{H}}_{i,j}^{res}\right)}\right\}{\widehat{\mu }}_{j,g_j^{\mathit{\text{res}}}}^{\mathit{\text{res}}}\left({\mathbf{H}}_{i,j}^{res}\right)\right],\end{array}$$

given a propensity score model and a conditional mean model.

We utilize tree-based reinforcement learning (T-RL) to search for the optimal treatment regime that closely follows the procedure proposed by Tao et al. (2018), but only considering the viable restricted space. We propose to use the AIPW estimator ${\mathbb{P}}_{n^{res,j}}\left\{{\widehat{\mu }}_{j,a_j}^{\text{res},\text{AIPW}}\left({\mathbf{H}}_j^{res}\right)\right\}$ in the purity measure for tree model construction. The tree model divides patients into subgroups with similar histories and recommends the corresponding optimal treatment to each subgroup, maximizing the estimated group average stage-wise pseudo-outcome. The proposed procedure provides decision rules at each stage to decide on viable individualized treatment tailored by patient characteristics recorded in ${\mathbf{H}}_j^{res}$. While considering a partition over node $\text{Ω}$ that split patients into two groups $\omega$ and ${\omega }^c$, we define ${\mathcal{P}}_j(\text{Ω},\varphi )$, the purity measure before the partition, as:

$$\begin{array}{l}\underset{a_j\in {\mathcal{A}}_j^{res}}{\max }{\mathbb{P}}_{n^{res,j}}\left[\left\{\frac{I\left(A_j=a_j\right)}{{\widehat{\pi }}_{j,a_j}\left({\mathbf{H}}_j^{res}\right)}{\text{PO}}_j^{res}+\left.\left\{1-\frac{I\left(A_j=a_j\right)}{{\widehat{\pi }}_{j,a_j}\left({\mathbf{H}}_j^{res}\right)}\right\}{\widehat{\mu }}_{j,a_j}^{res}\left({\mathbf{H}}_j^{res}\right)\right\}\right\}\right.\\ \left.\times I\left({\mathbf{H}}_j^{res}\in \text{Ω}\right)\right],\end{array}$$

for $j=1,\dots ,T$. We compare ${\mathcal{P}}_j(\text{Ω},\varphi )$ with ${\mathcal{P}}_j(\text{Ω},\omega )$, the purity measure after the partition: ${\max }_{a_1,a_2\in {\mathcal{A}}_j^{res}}{\mathbb{P}}_{n^{res,j}}\left\{\left.\left.\frac{I\left(A_j=a_j\right)}{{\widehat{\pi }}_{j,a_j}\left({\mathbf{H}}_j^{res}\right)}{\text{PO}}_j^{res}+\left\{1-\frac{I\left(A_j=a_j\right)}{{\widehat{\pi }}_{j,a_j}\left({\mathbf{H}}_j^{res}\right)}\right\}{\widehat{\mu }}_{j,a_j}^{res}\left({\mathbf{H}}_j^{res}\right)\right\}\right\}\times I\left\{g_{j,\omega ,a_1,a_2}^{res}\right.\right.\left.\left.\left({\mathbf{H}}_j^{res}\right)=a_j\right\}I\left({\mathbf{H}}_j^{res}\in \text{Ω}\right)\right]$, where $g_{j,\omega ,a_1,a_2}^{res}$ is the treatment rule that assigns $a_1$ to patient subgroup $\omega$ and assigns $a_2$ to patient subgroup ${\omega }^c$. We chose the best split $\omega$ based on the largest improvement in purity measure ${\mathcal{P}}_j(\text{Ω},\omega )-{\mathcal{P}}_j(\text{Ω},\varphi )$.

2.3 |. Implementing RT-RL

We implement RT-RL by restricting and allocating patients based on past treatment sequences during the optimization procedure. This implies that only patientswith viable treatment combinations up to stage $j$, that is, ${\overline{\mathbf{A}}}_j\in {\overline{\mathcal{A}}}_j^{res}$, can contribute to the treatment regime estimation at current stage $j$ for $j=1,\dots ,T$. Among patients with viable treatment sequences up to stage $j-1$ but at risk of receiving nonviable sequences at stage $j$, we fit separate T-RL models based on past treatment sequences. In each model, we search the best treatment among $\left\{a_j\in {\mathcal{A}}_j^{res}:\left({\overline{\mathbf{A}}}_{j-1},a_j\right)\in \right.\left.{\overline{\mathcal{A}}}_j^{res}\right\}$. An additional T-RL is fitted using the remaining patient records without restrictions on treatment options.

Moreover, when the observed treatments of a patient satisfy ${\overline{\mathbf{A}}}_j\in {\overline{\mathcal{A}}}_j^{res}$ and ${\overline{\mathbf{A}}}_{j^{*}}\notin {\overline{\mathcal{A}}}_{j^{*}}^{res}$, $\forall j^{*}>j$ with $j<T$, this patient can only contribute to the optimal regime estimation for the first $j$ stages. To use the information in this patient record, we need to estimate what is the best viable treatment within the restricted set and adjust for any suboptimal loss by calculating $P{O}_j^{\mathit{\text{res}}}$ using data from similar patients with applicable treatments until stage $j+1$.

When fitting tree models, we adapt the stopping criteria proposed by Tao et al. (2018). Node size $\left(n_0\right)$ controls the minimal number of subjects in each final node, and the depth of the tree $\text{(}d\text{)}$ avoids complicated regimes. Moreover, a positive constant $\lambda$ for minimal purity improvement is used for examining if the current best split ${\widehat{\omega }}^{\text{opt}}=argma{x}_{\omega }\left[{\mathcal{P}}_j(\text{Ω},\omega )\right]$ satisfies ${\mathcal{P}}_j(\text{Ω},\omega )-{\mathcal{P}}_j(\text{Ω},\varphi )>\lambda$, which guarantees user-specified meaningful splits. We set values for $n_0$, $d$, and $\lambda$ for pruning based on specific applications. A detailed RT-RL algorithm for $T\ge 2$ can be found in Supporting Information B.

For demonstration purposes, from now on, we consider a two-stage problem $(T=2)$ with $M$ nonviable treatment routes at stage 2 denoted as ${\overline{\mathbf{b}}}_{m,2}=\left(b_{1,m},b_{2,m}\right)$, $m=1,\dots ,M$, where $b_{1,m}$ and $b_{2,m}$ denote treatments for stage 1 and stage 2, respectively. The optimal DTR should consider only viable treatment routes that have $A_1\notin \left\{b_{1,1},\dots ,b_{1,M}\right\}$ or $A_1=b_{1,m}$ and $A_2\ne b_{2,m}$, $\forall m$. In stage 2, we estimate the restricted treatment decision rule by allocating patients based on if $A_1$ agrees with any nonviable sequences. First, we construct an overall conditional mean model ${\widehat{\mu }}_{2,a_2}^{res}$ with patients who have received viable treatment routes. Then, we create $M+1$ groups so that patients with $A_1=b_{1,m}$ are allocated to the $m\text{th}$ group for $m=1,2,\dots ,M$, and the rest are in the 0th group. We define the restricted patient history, treatment set, and treatment rules as follows. For the 0th group: ${\mathcal{H}}_{0,2}^{\text{res}}={\mathcal{H}}_2\setminus \left\{\left(b_{1,m},{\overline{\mathbf{X}}}_2^T\right):m=1,\dots ,M\right\}$, ${\mathcal{A}}_{0,2}^{\text{res}}={\mathcal{A}}_2$, and ${\mathcal{G}}_{0,2}^{\text{res}}=\left\{g_2^{res}:{\mathcal{H}}_{0,2}^{\text{res}}\mapsto {\mathcal{A}}_{0,2}^{\text{res}}\right\}$. For the $m\text{th}$ group: ${\mathcal{H}}_{\text{m},2}^{\text{res}}=\left\{\left(b_{1,m},{\overline{\mathbf{X}}}_2^T\right)\right.$, ${\mathcal{A}}_{\text{m},2}^{\text{res}}={\mathcal{A}}_2\setminus \left\{b_{2,m}\right\}$, and ${\mathcal{G}}_{\text{m},2}^{\text{res}}=\left\{g_2^{res}:{\mathcal{H}}_{\text{m},2}^{\text{res}}\mapsto {\mathcal{A}}_{\text{m},2}^{\text{res}}\right\}$, where $m=1,\dots ,M$.

Group-specific propensity scores are estimated using data from patients who received applicable treatment routes

$${\widehat{\pi }}_{2,A_2}\left({\mathbf{H}}_2^{res}\right)=\left\{\begin{array}{c}{\widehat{\pi }}_{2,A_2}^0\left({\mathbf{H}}_2^{res}\right),{\mathbf{H}}_2^{res}\in {\mathcal{H}}_{0,2}^{res}\\ {\widehat{\pi }}_{2,A_2}^m\left({\mathbf{H}}_2^{res}\right),A_2\in {\mathcal{A}}_{m,2}^{res}\text{and}{\mathbf{H}}_2^{res}\in {\mathcal{H}}_{\text{m},2}^{res},m=1,\dots ,M.\end{array}\right.$$

Then, separate T-RL models are fitted using patient observations in each restricted group that have ${\overline{\mathbf{a}}}_2=\left(b_{1,m},c_{2,m}\right)$, where $c_{2,m}\ne b_{2,m}$. We use all patients in the 0th group to fit the remaining T-RL model since all observed treatment sequences in this group are compatible with those of our interest.

Then the restricted optimal decision rule ${\widehat{g}}_2^{res,opt}$ at stage 2 is estimated by:

$$\left\{\begin{array}{ll}\underset{g_2^{\mathit{\text{res}}}\in {\mathcal{G}}_{m,2}^{\text{res}}}{argmax}{\mathbb{P}}_{n^{\mathit{\text{res}},m,2}}\left[\sum _{a_2\in {\mathcal{A}}_{m,2}^{\mathit{\text{res}}}}{\mu }_{2,a_2}^{res,AIPW}\left({\mathbf{H}}_2^{res}\right)I\left\{g_2^{res}\left({\mathbf{H}}_2^{res}\right)=a_2\right\}\right]\hfill & {\mathbf{H}}_2^{res}\in {\mathcal{H}}_{\text{m},2}^{res}\hfill \\ \underset{g_2^{\mathit{\text{res}}}\in {\mathcal{G}}_{0,2}^{\text{res}}}{argmax}{\mathbb{P}}_{n^{\mathit{\text{res}},0,2}}\left[\sum _{a_2\in {\mathcal{A}}_2}{\mu }_{2,a_2}^{res,AIPW}\left({\mathbf{H}}_2^{res}\right)I\left\{g_2^{res}\left({\mathbf{H}}_2^{res}\right)=a_2\right\}\right]\hfill & {\mathbf{H}}_2^{res}\in {\mathcal{H}}_{0,2}^{res}\hfill \end{array},\right.$$

where $m=1,\dots ,M$, $n^{\mathit{\text{res}},m,2}$ is the number of patients in the $m\text{th}$ group with $A_2\ne b_{2,m}$ while $n^{res,0,2}$ is the number of patients in the 0th group. Notice that ${\mathcal{H}}_2^{\text{res}}={\mathcal{H}}_{0,2}^{\text{res}}\cup {\mathcal{H}}_{1,2}^{\text{res}}\cup \dots \cup {\mathcal{H}}_{M,2}^{\text{res}}$.

In stage 1, we perform backward induction to estimate the optimal decision rule. The pseudo-outcome $P{O}_1^{\text{res}}$ is derived as $\widehat{E}\left\{Y\mid A_2={\widehat{g}}_2^{res,opt}\left({\mathbf{H}}_2^{res}\right),{\mathbf{H}}_2^{res}\right\}\stackrel{\Delta }{=}{\widehat{\mu }}_{2,{\widehat{g}}_2^{res,opt}}\left({\mathbf{H}}_2^{res}\right)$, which is the estimated expectation of the potential outcome one would have observed if the future treatment at stage 2 is already optimized by ${\widehat{g}}_2^{res,opt}$. At stage 1, we do not have any constrains on ${\mathcal{A}}_1$, ${\mathcal{H}}_1$, and ${\mathcal{G}}_1$. Hence, we have ${\mathbf{H}}_1^{res}={\mathbf{H}}_1$. Let ${\widehat{\mu }}_{1,a_1}^{\text{res,AIPW}}\left({\mathbf{H}}_1\right)$ denote $\widehat{E}\left(P{O}_1^{res}\mid A_1=a_1,{\mathbf{H}}_1\right)$. The optimal treatment decision rule at stage 1 is estimated as:

$${\widehat{g}}_1^{res,opt}=\underset{g_1^{res}\in {\mathcal{G}}_1}{argmax}{\mathbb{P}}_n\left[\sum _{a_1\in {\mathcal{A}}_1}{\widehat{\mu }}_{1,a_1}^{res,AIPW}\left({\mathbf{H}}_1\right)I\left\{g_1^{res}\left({\mathbf{H}}_1\right)=a_1\right\}\right].$$

In practice, we use a modified pseudo-outcome ${\widetilde{\text{PO}}}_j^{\text{res}}$ (Huangetal., 2015) to reduce the bias due to possiblemodel misspecification from the conditional model:

$${\widetilde{\text{PO}}}_j^{\text{res}}={\widetilde{\text{PO}}}_{j+1}^{\text{res}}+{\widehat{\mu }}_{j,g_j^{res,opt}}^{\text{res}}\left({\mathbf{H}}_j^{res}\right)-{\widehat{\mu }}_{j,A_j}^{res}\left({\mathbf{H}}_j\right)$$

for $j<T$. With backward induction and defining ${\widetilde{\text{PO}}}_T^{\mathit{\text{res}}}=Y$, it is easy to show that:

$${\widetilde{\text{PO}}}_j^{\text{res}}=Y+\sum _{t=j+1}^T\left\{{\widehat{\mu }}_{t,g_t^{res,opt}}^{\text{res}}\left({\mathbf{H}}_t^{res}\right)-{\widehat{\mu }}_{t,A_t}^{\text{res}}\left({\mathbf{H}}_t\right)\right\}$$

for $j<T$. Thus, for stage 1, since ${\mathbf{H}}_1^{res}={\mathbf{H}}_1$ we have:

$${\widehat{\mu }}_{1,a_1}^{\text{res},\text{AIPW}}\left({\mathbf{H}}_1\right)=\frac{I\left(A_1=a_1\right)}{{\widehat{\pi }}_{1,a_1}\left({\mathbf{H}}_1\right)}{\widetilde{\text{PO}}}_1^{\text{res}}+\left\{1-\frac{I\left(A_1=a_1\right)}{{\widehat{\pi }}_{1,a_1}\left({\mathbf{H}}_1\right)}\right\}{\widehat{\mu }}_{1,a_1}\left({\mathbf{H}}_1\right),$$

where ${\widehat{\pi }}_{1,a_1}\left({\mathbf{H}}_1\right)$ and ${\widehat{\mu }}_{1,a_1}\left({\mathbf{H}}_1\right)$ are estimated using all records.

To give an illustrative example, Figure 2 shows a two-stage three-treatment RT-RL patient allocation procedure with a nonviable arm $A_1=1$, $A_2=2$. In the second stage, patients with $A_1=1$ are allocated to the first group for being at risk of receiving the nonviable arm. We use patient records with $A_1=1$, $A_2\ne 2$ to fit the T-RL that seeks personalized optimal treatment among $A_2=0$ or $A_2=1$ for patients in group 1. Then, we “correct” the outcomes for patients in the nonviable arm with the pseudo-outcomes ${\widetilde{\text{PO}}}_1^{\text{res}}$ so that they can contribute to the first-stage optimal treatment decision rule estimation. This procedure optimizes the restricted patients’ second-stage treatment by only considering viable treatment combinations. Note that their best viable treatment is estimated by patients with similar characteristics who received viable treatments. The rest of the patients are allocated to the 0th group, where a normal T-RL is fitted with no restrictions. The treatment regimes estimated by RT-RL provide viable treatment recommendations to future patients based on past treatment sequences.

FIGURE 2.

Patient allocation for a two-stage three-treatment RT-RL estimation with nonviable route $A_1=1$, $A_2=2$. Group 0 contains patients with no risk of receiving the nonviable route. Group 1 contains patients with $A_1=1$ and hence are at risk of receiving the nonviable route. Patients with the nonviable treatment route only contribute to stage 1 decision rule estimation. Their stage 2 rewards will be “corrected” according to similar patients with $A_1=1$, $A_2\ne 2$ to create pseudo-outcomes in stage 1

Through this process, we use the observed data of the patients in the nonviable route before their last stage to provide better estimates of $g_1^{res,opt}$ and avoid selection bias in the study population. In the meanwhile, the above process accommodates the restrictions during the optimization procedure and guarantees the estimated optimal regime is interpretable and viable in practice.

3 |. SIMULATION STUDIES

We conduct simulation studies to evaluate the performances of the proposed RT-RL compared to a naive method, T-RL with data censoring, that is, using the regular T-RL and ignoring partial patient records in the nonviable arm. The simulation studies have two treatment stages $(T=2)$, and three treatments $(K=3)$ per stage ($A_j\in \{0,1,2\}$ for $j=1,2$) with one nonviable treatment route ($A_1=1$ and $A_2=2$). Specifically, in our setup, patients in the nonviable treatment route have healthier baseline conditions and better outcomes. Thus, the traditional methods using the observed compatible data would still try to recommend the nonviable route to future patients in order to maximize the health outcomes.

We generate three continuous covariates $\left(X_1,X_2,X_3\right)$ independently from $N(0,1)$. The clinical outcomes are generated such that a higher value of the sum of rewards at each stage $\left(Y=Y_1+Y_2\right)$ is desired. Treatment $A_1$ is generated from a multinomial distribution $\left.P\left(A_1=\right.k\right)={\pi }_{k1}$, $k=0,1,2$ with ${\pi }_{0,1}=\frac{p_{1,1}}{p_{1,1}+p_{2,1}+1}$, ${\pi }_{1,1}=\frac{p_{2,1}}{p_{1,1}+p_{2,1}+1}$, and ${\pi }_{2,1}=1-{\pi }_{0,1}-{\pi }_{1,1}$, where $p_{1,1}=\exp \left\{-0.2X_1+0.3X_2-0.2I\left(X_3>-0.5\right)+0.5\right\}$, $p_{2,1}=\exp \left\{0.3X_2+1.5I\left(X_3>-0.5\right)+0.5\right\}$. The underlying optimal treatments for stage 1 follow:

$$g_1^{\text{opt}}(H)=\left\{\begin{array}{cc}0& X_1<0,X_2<0\\ 1& X_2\ge 0\\ 2& X_1\ge 0,X_2<0.\end{array}\right.$$

The reward at stage 1 follows: $Y_1=\exp (1+0.05\ast \left.X_2-2*\left|A_1-A_{\text{opt}}\right|\right)+ϵ$, where $ϵ~N(0,0.5)$. To mimic a typical setting, where some observed treatments with good targeted clinical outcomes are not viable for future patients under certain “restrictions,” we allow a large portion of patients who received the nonviable treatment sequence to have better clinical outcomes than viable sequences. We assign $A_2$ conditional on $A_1$ such that patients have a greater possibility of receiving the nonviable arm when $A_1=1$. $A_2$ follows a three-level multinomial distribution with corresponding probabilities $\left({\pi }_{0,2},{\pi }_{1,2},{\pi }_{2,2}\right)=\left(\frac{p_{1,2}}{p_{1,2}+p_{2,2}+1},\frac{p_{2,2}}{p_{1,2}+p_{2,2}+1},\frac{1}{p_{1,2}+p_{2,2}+1}\right)$, where $p_{1,2}=\exp \left\{0.05Y_1-0.05X_1-0.2I\left(X_3>-0.5\right)-1\right\}$ and $p_{2,2}=\exp \left\{0.08Y_1-0.2X_1-0.1I\left(X_3>-0.5\right)-1.6\right\}$. On the other hand, when $A_1\ne 1$, $\left({\pi }_{0,2}^{\prime },{\pi }_{1,2}^{\prime },{\pi }_{2,2}^{\prime }\right)=\left(\frac{p_{1,2}^{\prime }}{p_{1,2}^{\prime }+p_{2,2}^{\prime }+10},\frac{p_{2,2}^{\prime }}{p_{1,2}^{\prime }+p_{2,2}^{\prime }+10},\frac{10}{p_{1,2}^{\prime }+p_{2,2}^{\prime }+10}\right)$, where $p_{1,2}^{\prime }=\exp \left(-0.2Y_1-0.05X_1+4\right)$, $p_{2,2}^{\prime }=\exp \left(-0.08Y_1+0.6X_1+3\right)$. We define the stage 2 optimal decision rule as:

$$g_2^{\text{opt}}\left(H\mid A_1=1\right)=\left\{\begin{array}{cc}0& Y_1\le 0.5\\ 1& 0.5<Y_1\le 1.5\\ 2& Y_1>1.5\end{array},g_2^{\text{opt}}\left(H\mid A_1\ne 1\right)=\left\{\begin{array}{cc}0& X_2>-0.5\\ 1& X_2\le -0.5\end{array},\right.\right.$$

and the reward as:

$$Y_2=\exp \left\{1+I\left(X_3>-0.5\right)*I\left(A_1=1\right)-6*\exp \left(\left|A_2-g_2^{\text{opt}}(H)\right|\right)\right\}+ϵ,$$

where $ϵ~N(0,0.5)$. At stage 2, the optimal rewards for patients in the nonviable route are $E\left[Y_2\mid A_1=1,A_2=\right.\left.g_2^{\text{opt}}(H)=2\right]=e^2$, which is higher than that of the patients in the viable routes $E\left[Y_2\mid A_1\ne 1,A_2=g_2^{\text{opt}}(H)\right]=e^1$. However, we would like to restrict $\left(A_1=1,A_2=2\right)$ out of our estimated treatment decision domain.

We compare our approach with a naive method that rules out the partial data of patients in the nonviable arm for stage 2 estimations but only includes them in stage 1 estimation when their treatments are still viable. For both methods, we utilize the AIPW version of the purity measure to allow robust estimation. We set the minimum size of the terminal nodes as 20, specify $\lambda$ as 5% in the splitting criteria, and the maximal depth as 3. One hundred replications are generated under each scenario considering different training sample sizes ($n=3000,5000$) and different propensity score model specifications (correct or incorrect). The two methods are evaluated on the agreement between the optimal treatments recommended by the estimated DTR and the true constrained optimal treatments of a randomly generated external test data set with $n=1000$.

Table 1 summarizes the simulation results. Opt % shows the empirical mean of the percentage of subjects correctly classified to their constrained optimal treatments. Both methods perform well in the first stage, correctly specifying the true optimal treatment for over 99% of patients. The naive approach using T-RL with data censoring is slightly more stable than the proposed method, having a smaller standard deviation for opt %. However, for the second stage and overall, RT-RL has a much higher opt % than the naive method. Although censoring patients’ data with nonviable treatment sequences, the naive method still recommends the “problematic” treatment sequence to 36%–42% of patients in the evaluation set. Both estimated regimes improve patient outcomes, while the proposed algorithm improves less, which is not surprising as it rules out the nonviable arm for future patients. As demonstrated in the simulation results, the proposed and naive methods are doubly robust as similar results are observed under the correctly or incorrectly specified $\pi$ model.

TABLE 1.

Comparing simulation results between RT-RL and T-RL with data censoring. Standard errors are in parenthesis. Opt % records the average percent of subjects in the evaluation set being recommended the true optimal treatment route. $\widehat{E}\left(Y\mid {\widehat{g}}^{\text{opt}}\right)-\widehat{E}\left(Y\mid g^{\text{obs}}\right)$ is the improvement of the estimated outcomes when following the estimated DTR versus the observed DTR. “% of Recommendation with Nonviable Arm” shows the percent of subjects in the evaluation set being recommended the nonviable treatment sequence

	Correct $\pi$		Incorrect $\pi$
Method	RT-RL	Naive T-RL with Data Censoring	RT-RL	Naive T-RL with data censoring
Training sample	n = 3000
Opt % stage 1	99.37 (0.44)	99.43 (0.34)	98.69 (4.91)	99.43 (0.34)
Opt % stage 2	83.73 (12.25)	31.58 (17.58)	80.89 (16.11)	31.56 (17.53)
Opt % overall	83.42 (12.24)	31.31 (17.57)	80.28 (16.40)	31.29 (17.53)
$\widehat{E}\left(Y{\widehat{g}}^{opt}\right)-\widehat{E}\left(Y{g}^{obs}\right)$	1.28 (1.54)	1.38 (1.50)	1.27 (1.55)	1.38 (1.50)
% of recommendation with nonviable arm	0	36.35	0	36.98
Training sample	n = 5000
Opt % stage 1	99.56 (0.33)	99.58 (0.30)	99.39 (0.44)	99.59 (0.29)
Opt % Sstage 2	90.84 (9.17)	33.28 (12.33)	85.16 (17.55)	33.21 (11.34)
Opt % overall	90.52 (9.12)	33.06 (12.36)	84.85 (17.54)	32.99 (11.37)
$\widehat{E}\left(Y\mid {\widehat{g}}^{opt}\right)-\widehat{E}\left(Y\mid g^{obs}\right)$	1.30 (1.54)	1.40 (1.49)	1.30 (1.55)	1.40 (1.49)
% of recommendation with nonviable arm	0	41.71	0	42.39

4 |. APPLICATION TO LOC PLACEMENT FOR ADOLESCENT SUBSTANCE USERS

We use the proposed RT-RL method to estimate a viable optimal two-stage DTR to guide the LOC placement for adolescent substance users over 0–3 months (stage 1) and 3–6 months (stage 2) to minimize substance use at month 12 after intake, based on a longitudinal observational data set ($n=\mathrm{10,131}$), known as the GAIN (Dennis et al., 2003). The observed possible LOC treatment options at both stages include inpatient $(A=1)$ and outpatient $(A=2)$, while stage 2 also allows no treatment $(A=3)$. These treatment options represent different levels of patient care during each 3-month period. Inpatient or residential is the most intensive treatment, where youth are admitted for at least one night to a residential, inpatient, or hospital program. Outpatient refers to a regular (1–8 h per week) or intensive outpatient (more than 8 h per week) program. The outcome of interest for this study is the Substance Frequency Scale (SFS) at 12 months after intake, which is coded as $Y=-1\times SFS$ to ensure that a higher value is preferable.

According to National Institutes of Health (NIH), since relapses often occur for adolescents with substance abuse, more than one episode of treatment may be necessary (Godley et al., 2007). Also, inpatient treatment is recommendedforaseverelevelofaddiction.Stayingintreatment for an adequate period of time is especially important for patients with $A_1=1$. However, in the collected data, we observed a group of patients who started with inpatient treatment $\left(A_1=1\right)$ and followed with no treatment $\left(A_2=3\right)$. Thus, given the collected data and NIH recommendations, we consider constraining the possible treatment regimes over $A_2\le 2$ given $A_1=1$ by restricting to “stepped up” DTRs that do not allow treatment discontinuation for adolescents starting with inpatient treatment. Note that an alternative way to address the relapse issue is recording the frequency of relapse during the data collection phase and optimizing a combined outcome of SFS and the number of relapses. However, it is challenging to track patients after they leave the healthcare facility, and extra efforts are required to determine the weights or offsets in the combined outcome. The advantage of the proposed methodology is using the collected data and prespecified value function related to SFS to search for an optimal treatment strategy, which is demonstrated below.

Specifically, we randomly split the data 3:1 for training and evaluation. We split the data multiple times using different random seeds and the results were similar, hence only one example is shown here. The training data contain 7599 randomly selected records with the remaining 2532 records in the evaluation data. This split facilitates the comparison of the RT-RL with a naive T-RL method (ignoring partial records for patients in the nonviable arm) and nondynamic regimes such as $A_1=A_2=1$ and $A_1=A_2=2$. The naive T-RL method censors 404 patients’ data who received the nonviable treatment route $A_1=1$, $A_2=3$. The viability and interpretability of ${\widehat{g}}^{\text{opt}}$ provided by the restricted and naive T-RL methods can be assessed with the probability of recommending the nonviable route to patients in the evaluation set. Moreover, we compare $\widehat{E}\left(Y\mid {\widehat{g}}^{\text{opt}}\right)-\widehat{E}\left(Y\mid g^{\text{obs}}\right)$ and the percent of patients who have $\widehat{E}\left(Y\mid {\widehat{g}}^{\text{opt}}\right)>\widehat{E}\left(Y\mid g^{\text{obs}}\right)$ to examine the methods’ ability to suggest optimal treatments.

According to Figure 3, RT-RL provides a viable DTR based on SFS at intake (sfs8p_0), living risk index at (lri7_0) intake, and first-stage treatment $\left(A_1\right)$. It recommends $A_1=A_2=1$ to 947 (37.4%) patients and $A_1=A_2=2$ to 1585 (62.6%) patients in the evaluation data. Although training set patients’ data were censored in the nonviable arm when using the naive T-RL, it still allows “stepped down” treatments in the evaluation set for patients who greatly improved their SFS at month 6, which leads to 674 (26.6%) patients being recommended for the nonviable treatment sequence.

FIGURE 3.

Estimated optimal DTR from RT-RL and Naive T-RL. sfs8p_t: −1* SFS at t months post intake, higher value preferred; lri7_0: living risk index at intake, lower value preferred

Table 2 shows that RT-RL provides an optimal DTR in the desired class with the greatest improvement (0.7 points) in the expected outcome of interest (negative SFS) at 12 months post intake. The scale of the improvement can be standardized using pooled standard deviation. Given that the pooled standard deviation of the expected counterfactual outcomes under the observed DTR and the RT-RL estimated optimal DTR is $\sqrt{\frac{var\left\{\widehat{E}\left(Y\mid {\widehat{g}}^{opt}\right)\right\}+var\left\{\widehat{E}\left(Y\mid g^{obs}\right)\right\}}{2}}=4.11$ in the evaluation set, the improvement of 0.7 using RT-RL is equivalent to 0.17 SD. When comparing our method with the naive T-RL with data censoring, the improvement in the expected outcome of interest is 0.22 SD. Similarly, compared with the non-DTRs $A_1=A_2=1$ and $A_1=A_2=2$, the improvements are 0.21 SD and 0.13 SD, respectively. Moreover, RT-RL is expected to help 56% of patients improve their potential mean substance use outcomes compared to the observed treatment in the evaluation set. In contrast, the naive T-RL cannot provide a viable and well-performing regime due to the lack of cross-stagerestrictionsand thebias introducedbyignoring partial records for patients in the nonviable arm.

TABLE 2.

Treatment regime performance on evaluation data. % improved refers to the percent of patients who are expected to benefit from the estimated regime compared to the observed treatment

	RT-RL	Naive T-RL	$A_1=1$, $A_2=1$	$A_1=2$, $A_2=2$
$\widehat{E}\left(Y\mid {\widehat{g}}^{\text{opt}}\right)-\widehat{E}\left(Y\mid g^{\text{obs}}\right)$ (SD)	0.70 (1.4)	−0.08 (1.9)	−0.03 (1.9)	0.14 (1.1)
% improved	56.0	38.6	39.8	42.3
SD of $\widehat{E}\left(Y\mid {\widehat{g}}^{\text{opt}}\right)$	3.78	3.34	3.11	4.60

5 |. DISCUSSION

The proposed method RT-RL searches for viable and interpretable DTR within clinically meaningful treatment trajectories. It allows user-specified restrictions on the observed treatment routes. According to simulation and GAIN data application results, RT-RL avoids selection bias when using partial information from patients in the nonviable arm to estimate optimal treatment decision rules for stages before their first “problematic” treatment. We achieve this by “correcting” the counterfactual rewards of the patients in the nonviable arms using information from similar patients who received viable treatments. The idea of stratifying based on patient history and truncating treatment options is not limited to tree-based methods. Such a framework for DTR estimation considering suboptimal correction can be applied to other methods when appropriate.

The naive T-RL method suffers from information loss and possible selection bias yet still recommends nonviable treatment routes to new patients. This happens because the rules are estimated stage wise, and the naive method considers all observed treatments in the current stage as possible treatments. Without restrictions based on previous treatment sequences and constraints on the current stage treatment space, it is not guaranteed that the recommended treatment is different from the nonviable treatment $b_{j,m,j}$ when the previous recommendations are ${\overline{\mathbf{a}}}_{j-1}=\left(b_{1,m,j},\mathrm{..},b_{j-1,m,j}\right)$.

In this study, we focused on constraints with nonviable treatment sequences. This approach provides a solution for many observational studies with relatively large sample sizes. However, when we have little observational data in any of the 0th to $M_j\text{th}$ subgroups, the restricted DTR is difficult to estimate. Similarly, the sample size of each subgroup will decrease with an increased number of nonviable arms. This small sample size problem may affect the reliability of the resulting DTR. Thus, we require the positivity assumption to hold for estimating a restricted optimal DTR, where we observe a group of similar patients treated with each potential treatment sequence. Even when the positivity assumption holds, the decision rules’ uncertainties may differ based on the sample size of each subgroup. Practically, to quantify the uncertainty in a subgroup, we can estimate the optimal DTRs using different bootstrapped training samples and examine the consistency of the suggested optimal treatments in one test sample. When significant uncertainty is observed, we may consider using ensemble methods, including Bayes optimal classifier, bagging, and boosting, to provide a stabler rule (Mitchell et al., 1997; Freund & Schapire, 1997), each of which has a unique subsampling method. Nevertheless, one disadvantage of the ensemble methods is that they can be challenging to interpret. The resulting set of decision rules might be too complicated for clinical practice as the interpretations for each rule, and the voting weights are unexplainable.

Moreover, the treatment sequence constraints can also come from patient characteristics or stage-wise rewards $\left(R_j\right)$. For example, if future treatments must consider whether a patient is a responder to current treatment at stage $j$, all DTR ignoring the responder status at stage $j$ should not be applicable. Further studies are necessary to solve this issue.

Supplementary Material

Web Appendices referenced in Sections 2 and R code for implementing the proposed methods are available with this paper at the Biometrics website on Wiley Online Library. The R code is also on Github at https://github.com/Team-Wang-Lab/Restricted-TRL.

DATA AVAILABILITY STATEMENT

The application study data in this paper are not publicly available due to patient privacy and confidentiality issues.