Fusing Individualized Treatment Rules Using Secondary Outcomes

Abstract

An individualized treatment rule (ITR) is a decision rule that recommends treatments for patients based on their individual feature variables. In many practices, the ideal ITR for the primary outcome is also expected to cause minimal harm to other secondary outcomes. Therefore, our objective is to learn an ITR that not only maximizes the value function for the primary outcome, but also approximates the optimal rule for the secondary outcomes as closely as possible. To achieve this goal, we introduce a fusion penalty to encourage the ITRs based on different outcomes to yield similar recommendations. Two algorithms are proposed to estimate the ITR using surrogate loss functions. We prove that the agreement rate between the estimated ITR of the primary outcome and the optimal ITRs of the secondary outcomes converges to the true agreement rate faster than if the secondary outcomes are not taken into consideration. Furthermore, we derive the non-asymptotic properties of the value function and misclassification rate for the proposed method. Finally, simulation studies and a real data example are used to demonstrate the finite-sample performance of the proposed method.

1. INTRODUCTION

An individualized treatment rule (ITR) is a decision rule that recommends treatments to patients based on their pre-treatment covariates, such as demographics, medical history, and health status. Many methods have been proposed to estimate the optimal ITR using data from a randomized controlled trial or an observational study. The goal is to find the ITR that would yield the maximal primary outcome if patients follow the treatment recommendations. Regression-based approaches start with estimating Q-functions, which are the expected outcomes associated with each treatment option, and the optimal treatment is determined by maximizing the Q-functions. Exemplary methods include Q-learning (Qian and Murphy, 2011, Ma et al., 2022) and A-learning (Murphy, 2003; Shi et al., 2018). Alternative methods directly search for the ITR that maximizes the mean reward, also called the value function, using either an inverse probability weighted estimator (Zhao et al., 2012, Chen et al., 2016, Zhang et al. 2020; Gao et al., 2022; Ma et al., 2023) or a doubly robust estimator (Zhang et al., 2012: Zhou et al. 2017, Liu et al. 2018, Zhao et al., 2019). However, all these methods focus on one primary outcome when estimating the ITRs.

In practical settings, it is important to consider certain secondary outcomes. Non-favorable secondary outcomes potentially indicate worsened overall health or an increased risk of treatment non-compliance. Therefore, when estimating the optimal ITR for the primary outcome, it is crucial to ensure that the ITR also optimizes the secondary outcomes to the greatest extent. For example, for major depressive disorder (MDD; Trivedi et al. 2016), one common outcome to measure depressive symptoms is the Quick Inventory of Depressive Symptomatology (QIDS) score, a rating scale of the patient’s emotional state over the past seven days. Besides, another important outcome is the Clinical Global Improvement (CGI) scale, which is often used to assess a patient’s symptoms, behavior, and the impact on the patient’s ability to function. It is an indicator of overall clinical improvement. Although the primary goal in many studies is to identify the optimal treatment strategy for improving the QIDS score, it is equally important for such a strategy to be effective for the CGI scale.

In this work, we have a twofold objective, which is to estimate the optimal ITR for the primary outcome and to ensure that the resulting treatment decision closely aligns with the optimal treatment for the secondary outcomes. Specifically, we aim to fuse the treatment rules for different outcomes to provide a unified ITR that performs optimally for the primary outcome and effectively for the secondary outcomes, even if it may not be optimal for the latter. To achieve this goal, we introduce a novel fused learning framework to estimate the optimal ITR, namely the fused individualized treatment rule (FITR). Under this framework, we maximize the value function for the primary outcome. At the same time, we incorporate a fusion penalty to encourage agreement between the estimated ITR and the optimal ITRs for the secondary outcomes. The latter are assumed to be obtained apriori using external data or the data from the same study. The fusion penalty is designed as a weighted sum of the disagreement rates between the treatment rules, with weights determined based on the similarity of treatment effect on these outcomes. Therefore, this penalty aims to encourage consistency between the ITRs for the different outcomes.

Related Work.

Several approaches have been proposed to learn ITRs that can handle multiple outcomes simultaneously. Some of these approaches (Wang et al. 2018, Laber et al. 2018, Fang et al. 2022) estimated the optimal ITRs for the primary outcome while restricting the secondary outcome to be no less (or no larger if the outcome is a risk outcome) than a threshold. However, these approaches require the threshold to be pre-specified and only ensure that the average value of the secondary outcome is controlled, but not for any specific individuals. Another class of methods (Thall et al., 2002, Luckett et al., 2021) constructed expert-derived or data-driven patient-specific composite outcomes for learning the ITRs. However, their ITRs may not be optimal for the primary outcome of interest. In addition, Laber et al. (2014) and Lizotte and Laber (2016) proposed constructing a set-valued regime for competing outcomes to generate non-inferior outcome vectors but does not recommend a single treatment, limiting their use for practice. Our goal is to learn an optimal ITR for the primary outcome while ensuring closeness to the optimal secondary outcomes, which is distinct from all existing approaches for learning multi-outcome ITRs.

Our problem is also fundamentally different from the literature that combines multiple studies in analysis. Meta-analysis (Haidich, 2010, Lin and Zeng, 2010; Claggett et al., 2014 Liu et al., 2015) combines the estimators across multiple studies efficiently. Integrative data analysis (Curran and Hussong, 2009, Brown et al. 2018) considers multiple data sources or summary statistics using shared parameter models. Transfer learning (Li et al., 2021, Cai and Wei, 2021, Tian and Feng, 2022) uses existing knowledge from one task to assist in learning a new task. However, these existing methods often require individual-level data or assume specific models or distributions for each study to achieve integration, so are not applicable to combining different treatment rules.

Main Contributions.

Our paper introduces several significant contributions. (1) We propose a fusion penalty to encourage the primary outcome ITR to closely agree with the optimal secondary outcome ITRs. The proposed method relies on the secondary outcome only through its treatment rules. We do not require individual-level data for the secondary outcome when such rules are already available. In addition, there is no restriction on the form of the decision function since the fusion penalty is directly applied to the disagreement rate. The proposed method is not affected by the magnitude of the decision function or the function space in which the secondary outcome ITRs are estimated. (2) To efficiently solve for the FITR, we propose two algorithms that use surrogate losses to substitute the noncontinuous, nonconvex value function and fusion penalty in the objective function. (3) Theoretically, we prove that the agreement rate between the estimated FITR learned with the surrogate loss will converge to the true agreement rate, and the convergence rate is faster than the approach without considering the secondary outcome. We also obtain the convergence rate for the value function and misclassification rate of FITR. (4) In the numeric experiments, we show that the agreement rate indeed converges faster to the true rate. Moreover, when the true optimal ITRs of different outcomes are closely aligned, the learned FITR actually yields higher value function and accuracy for the primary outcome compared to traditional methods. By leveraging the secondary outcomes as useful side information, the proposed method has the capability to improve the treatment decisions for multiple outcomes simultaneously.

2. METHODOLOGY

The vector $\mathit{X}\in 𝒳\subset {\mathbb{R}}^d$ represents pre-treatment covariates for tailoring treatment decisions, where $d$ is the dimension of $\mathit{X}$. The treatment $A\in 𝒜=\{1,-1\}$ is assumed to be binary. The primary outcome is denoted by $R_1$. Without loss of generality, we assume that a higher value of this outcome indicates a better health condition. Denote $R_1\left(a\right)$ as the potential outcome under the treatment $a$. For the primary outcome $R_1$, an ITR ${𝒟}_1:𝒳\to 𝒜$ maps a patient’s covariates to a treatment suggestion. It can also be expressed as ${𝒟}_1\left(\mathit{X}\right)=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\left\{f_1\left(\mathit{X}\right)\right\}$ for some decision function $f_1:𝒳\to \mathbb{R}$. Define ${𝒱}_1\left(f_1\right):=\mathbb{E}\left[R_1\left\{{𝒟}_1\left(\mathit{X}\right)\right\}\right]$ as the value function associated with $f_1$, which is the average potential outcome when the treatments follow the ITR ${𝒟}_1=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\left\{f_1\right\}$ for all $\mathit{X}$. Our goal is to estimate the optimal ITR that maximizes this value function. The optimal ITR ${𝒟}_1^{\mathrm{*}}$ is given as $\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\{f_1^{\mathrm{*}}\}$, where $f_1^{\mathrm{*}}(\mathit{x})=\mathbb{E}\left[R_1(1)\mid \mathit{X}=\mathit{x}\right]-\mathbb{E}\left[R_1(-1)\mid \mathit{X}=\mathit{x}\right]$.

We make standard assumptions in causal inference, including ignorability, consistency, and positivity (see Section B.1, so that the optimal ITR can be estimated from a sample of $n$ i.i.d data of $\left(\mathit{X},A,R_1\right)$. Under these conditions, according to Qian and Murphy (2011), we have ${𝒱}_1\left(f_1\right)=\mathbb{E}\left[R_1\mathbb{1}\left\{A{f}_1(\mathit{X})>0\right\}/\pi (A;\mathit{X})\right]$, where $\pi (A;\mathit{X})$ is the probability of taking treatment $A$ given covariates $\mathit{X}$ in the data. Thus, the optimal ITR can be estimated by solving

$$\underset{f_1\in \mathscr{H}}{\text{min}}\frac{1}{n}\sum _{i=1}^n\frac{R_{i1}\mathbb{1}\left\{A_i{f}_1\left({\mathit{X}}_i\right)<0\right\}}{\pi \left(A_i;{\mathit{X}}_i\right)}+{\lambda }_{1n}{‖f_1‖}^2,$$

where $\mathscr{H}$ is some function class, $∥f_1∥$ is the semi-norm for $f_1$ in its function space, and ${\lambda }_{1n}$ is a tuning parameter that depends on the sample size $n$.

Separate Learning (SepL)

Since minimizing a loss function with 0–1 loss is computationally challenging, it can be substituted by some convex surrogate loss (Bartlett et al. 2006), denoted by $\varphi (x)$. For example, Zhao et al. (2012) proposed using the hinge loss where $\varphi (x)=\mathrm{m}\mathrm{a}\mathrm{x}(1-x,0)$. In our implementation, we propose using the logistic loss, $\varphi (t)=\mathrm{l}\mathrm{o}\mathrm{g}\left(1+e^{-t}\right)$, due to its differentiability and computational stability. To further reduce the variability of the value function estimator, we replace $R_{i1}$ by $R_{i1}-\mathbb{E}\left(R_{i1}\mid {\mathit{X}}_i\right)$, which does not change the optimal ITR. The conditional expectation $\mathbb{E}\left(R_{i1}\mid {\mathit{X}}_i\right)$ can be estimated by fitting $R_{i1}$ against ${\mathit{X}}_i$ with any regression methods, e.g. simple linear regression. Moreover, we simultaneously flip the sign of $R_{i1}-\mathbb{E}\left(R_{i1}\mid {\mathit{X}}_i\right)$ and $A$ when $R_{i1}-\mathbb{E}\left(R_{i1}\mid {\mathit{X}}_i\right)$ is negative (c.f., Liu et al. 2018). This allows us to formulate the problem as a convex optimization problem

$$\begin{gathered} {\tilde{f}}_{1n}=\underset{f_1\in \mathscr{H}}{\text{arg}\text{min}}\frac{1}{n}\sum _{i=1}^n\frac{\left|R_{i1}-\mathbb{E}\left(R_{i1}\mid {\mathit{X}}_i\right)\right|}{\pi \left(A_i;{\mathit{X}}_i\right)}. \\ \varphi \left[A_i\text{sign}\left\{R_{i1}-\mathbb{E}\left(R_{i1}\mid {\mathit{X}}_i\right)\right\}{f}_1\left({\mathit{X}}_i\right)\right]+{\lambda }_{1n}{‖f_1‖}^2. \end{gathered}$$

We refer to this method as separate learning $(SepL)$.

2.1. Learning Fused ITR Using Optimal Rules for Secondary Outcomes

The secondary outcomes are denoted as $R_2,\dots ,R_K$ for $K\ge 2$. Suppose we have obtained the ITRs ${\tilde{f}}_2,\dots ,{\tilde{f}}_K$ for the secondary outcomes either from external data or the same study. Our goal is to estimate the optimal ITR for the primary outcome while encouraging it to be consistent as much as possible with these secondary outcome ITRs. To this end, we propose a fusion penalty on the disagreement rates between $f_1$ and ${\tilde{f}}_2,\dots ,{\tilde{f}}_K$. Specifically, we estimate the fused individualized treatment rule (FITR) $f_1$ by minimizing

$$\begin{gathered} \frac{1}{n}\sum _{i=1}^n\frac{R_{i1}}{\pi \left(A_i;{\mathit{X}}_i\right)}\mathbb{1}\left\{A_i{f}_1\left({\mathit{X}}_i\right)<0\right\}+{\lambda }_{1n}{‖f_1‖}^2 \\ +\frac{{\mu }_{1n}}{n}\sum _{i=1}^n\sum _{k=2}^K{\text{Ω}}_{1k}\mathbb{1}\left\{f_1\left({\mathit{X}}_i\right){\tilde{f}}_k\left({\mathit{X}}_i\right)<0\right\}, \end{gathered}$$ (1)

where ${\mathrm{\Omega }}_{1k}$ is a nonnegative pre-specified constant that reflects the prior knowledge of the similarity between the primary outcome and each secondary outcome $R_k$ For example, ${\mathrm{\Omega }}_{1k}$ can be expert-specified or defined as the correlation between $R_1$ and $R_k$. If ${\mathrm{\Omega }}_{1k}$ is negative, we can simultaneously flip the sign of ${\mathrm{\Omega }}_{1k}$ and ${\tilde{f}}_k$ to encourage the consistency between $f_1$ and $-{\tilde{f}}_k$. The tuning parameter ${\mu }_{1n}$ of the fusion penalty, which reweights ${\mathrm{\Omega }}_{1k}$ for all $k$, is determined data-adaptively using cross-validation. It is selected to maximize the estimated value function of $f_1$, thus automatically quantifying the similarity between the outcomes.

The fusion penalty is related to the Laplacian penalty (Huang et al. 2011), which is used to learn multiple models simultaneously while encouraging similarity among them. With $\mathbf{L}$ as the Laplacian matrix and $\mathit{f}:=\left\{f_1,\dots ,f_K\right\}$ as the vector of decision functions for each outcome, the Laplacian penalty can be defined as $\frac{1}{n}{\sum }_{i=1}^n[\mathbb{1}\left\{\mathit{f}\left({\mathit{X}}_i\right)>0\right\}\cdot \mathbf{L}\cdot \mathbb{1}{\left\{\mathit{f}\left({\mathit{X}}_i\right)>0\right\}}^T+\mathbb{1}\left\{\mathit{f}\left({\mathit{X}}_i\right)<0\right\}\cdot \mathbf{L}\cdot \mathbb{1}{\left\{\mathit{f}\left({\mathit{X}}_i\right)<0\right\}}^T]$. It prompts $f_k\left({\mathit{X}}_i\right)$ and $f_j\left({\mathit{X}}_i\right)$ to have the same sign when $L_{kj}<0$ for $k,j=1,\dots ,K,k\ne j$. This is equivalent to (1) if we use the same data to learn FITRs for all outcomes simultaneously.

FITR-Ramp.

The optimization problem in (1) is known to be NP-hard. To tackle this problem, we substitute the 0–1 losses with a smooth loss function for optimization purposes. The 0–1 loss in the first part of the expression is substituted by a logistic loss as in SepL. For the 0–1 loss in the fusion penalty, we use a ramp loss ${\psi }_{\kappa }(t)=\mathrm{m}\mathrm{i}\mathrm{n}\{1,\mathrm{m}\mathrm{a}\mathrm{x}\{0,1-t/\kappa \left\}\right\}$, where $\kappa$ is a tuning parameter, for approximation since the latter converges to the 0–1 loss when $\kappa$ decreases to zero. With the same variance reduction and negative reward handling trick, we propose the FITR-Ramp method to estimate $f_1$ by

$$\begin{gathered} {\widehat{f}}_{1n}=\underset{f_1\in \mathscr{H}}{\text{arg}\text{min}}\frac{1}{n}\sum _{i=1}^n\frac{\left|R_{i1}-\mathbb{E}\left(R_{i1}\mid {\mathit{X}}_i\right)\right|}{\pi \left(A_i;{\mathit{X}}_i\right)}. \\ \varphi \left[A_i\text{sign}\left\{R_{i1}-\mathbb{E}\left(R_{i1}\mid {\mathit{X}}_i\right)\right\}{f}_1\left({\mathit{X}}_i\right)\right]+{\lambda }_{1n}{‖f_1‖}^2 \\ +\frac{{\mu }_{1n}}{n}\sum _{i=1}^n\sum _{k=2}^K{\text{Ω}}_{1k}{\psi }_{{\kappa }_{1n}}\left[f_1\left({\mathit{X}}_i\right){\tilde{f}}_k\left({\mathit{X}}_i\right)\right]. \end{gathered}$$ (2)

We allow ${\kappa }_{1n}$ to depend on the sample size $n$.

FITR-IntL.

Alternatively, we can solve the optimization problem in (1) using the following procedure. Notice that the disagreement between $f_1$ and ${\tilde{f}}_k$ can be expressed with the disagreement between $A_i$ and $f_1$ since

$$\begin{gathered} \mathbb{1}\left\{f_1\left({\mathit{X}}_i\right){\tilde{f}}_k\left({\mathit{X}}_i\right)<0\right\} \\ =\mathbb{1}\left\{A_i{f}_1\left({\mathit{X}}_i\right)<0\right\}\mathbb{1}\left\{A_i{\tilde{f}}_k\left({\mathit{X}}_i\right)>0\right\} \\ +\left[1-\mathbb{1}\left\{A_i{f}_1\left({\mathit{X}}_i\right)<0\right\}\right]\mathbb{1}\left\{A_i{\tilde{f}}_k\left({\mathit{X}}_i\right)<0\right\} \\ =\mathbb{1}\left\{A_i{f}_1\left({\mathit{X}}_i\right)<0\right\}\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\left\{A_i{\tilde{f}}_k\left({\mathit{X}}_i\right)\right\}+\mathbb{1}\left\{A_i{\tilde{f}}_k\left({\mathit{X}}_i\right)<0\right\}. \end{gathered}$$

The last term does not depend on $f_1$ given observed data. Therefore, the problem in (1) is equivalent to ${\widehat{f}}_{1n}=\mathrm{a}\mathrm{r}\mathrm{g}{\mathrm{m}\mathrm{i}\mathrm{n}}_{f_1\in \mathscr{H}}\frac{1}{n}{\sum }_{i=1}^n\frac{{\tilde{R}}_{i1}}{\pi \left(A_i;{\mathit{X}}_i\right)}\mathbb{1}\left\{A_i{f}_1\left({\mathit{X}}_i\right)<0\right\}+{\lambda }_{1n}{∥f_1∥}^2$, where

$${\tilde{R}}_{i1}=R_{i1}+{\mu }_{1n}\pi \left(A_i;{\mathit{X}}_i\right)\sum _{k=2}^K{\mathrm{\Omega }}_{1k}\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\left\{A_i{\tilde{f}}_k\left({\mathit{X}}_i\right)\right\}$$

is the pseudo outcome. We then substitute the indicator function by the logistic loss and estimate ${\widehat{f}}_{1n}$ by

$$\begin{gathered} {\widehat{f}}_{1n}=\underset{f_1\in \mathscr{H}}{\text{arg}\text{min}}\frac{1}{n}\sum _{i=1}^n\frac{\left|{\tilde{R}}_{i1}-\mathbb{E}\left({\tilde{R}}_{i1}\mid {\mathit{X}}_i\right)\right|}{\pi \left(A_i;{\mathit{X}}_i\right)}. \\ \varphi \left[A_i\text{sign}\left\{{\tilde{R}}_{i1}-\mathbb{E}\left({\tilde{R}}_{i1}\mid {\mathit{X}}_i\right)\right\}{f}_1\left({\mathit{X}}_i\right)\right]+{\lambda }_{1n}{‖f_1‖}^2 \end{gathered}$$ (3)

with the same variance reduction and negative reward handling trick. We refer to this optimization method as FITR-IntL. It is worth noting that a similar procedure was originally proposed in Qiu et al. (in press) and Qiu (2018) to integrate treatment rules from multiple studies for the same outcome without theoretical justifications, whereas our focus is on integrating multiple outcomes.

Implementation.

In SepL, FITR-Ramp and FITR-IntL, the semi-norm for $f_1$ is usually chosen from a reproducing kernel Hilbert space (RKHS) associated with a real-valued kernel function $k:𝒳\times 𝒳\to \mathbb{R}$. The choice of $k$ can be the linear kernel $k\left(\mathit{x},{\mathit{x}}^{\prime }\right)={\mathit{x}}^T{\mathit{x}}^{\prime }$ which yields a linear decision function for $f_1$, or the Gaussian kernel $k\left(\mathit{x},{\mathit{x}}^{\prime }\right)=\mathrm{e}\mathrm{x}\mathrm{p}(-{\sigma }_{1n}^2{∥\mathit{x}-{\mathit{x}}^{\prime }∥}_2^2)$ which yield a nonlinear decision function, where ${\sigma }_{1n}$ is a parameter depending on $n$. By the representer theorem, the minimizer for $f_1$ takes the form $f(\mathit{X})={\sum }_{i=1}^n{\alpha }_{i}k\left(\mathit{X},{\mathit{X}}_i\right)$, so the optimization problems can be restricted to class $\mathscr{H}:=\left\{f:f(\mathit{X})={\sum }_{i=1}^n{\alpha }_{i}k\left(\mathit{X},{\mathit{X}}_i\right),\left({\alpha }_1,\dots ,{\alpha }_n\right)\in \right.\left.{\mathbb{R}}^n\right\}$. The function class $\mathscr{H}$ is nonparametric when the RKHS with Gaussian kernel is used. When ${\mu }_{1n},{\kappa }_{1n}\to 0,{\sigma }_{1n}^2\to \mathrm{\infty }$ as $n\to \mathrm{\infty }$, this nonparametric method is asymptotically equivalent to unconstrained problems without penalties. Therefore, the learned FITR is asymptotically optimal even when the non-constant shift $E(R\mid \mathit{X})$ and the fusion penalty are introduced.

For computation, FITR-Ramp can be solved by the difference of convex algorithm (DCA) (Le Thi and Pham Dinh, 2018) or the Powell algorithm (Powell, 1964 Press et al., 2007). DCA re-expresses the objective function as the difference between two convex functions, whereas the Powell algorithm is suitable for non-differentiable objective functions. In contrast, FITR-IntL has a differentiable convex objective function so that it can be easily solved by gradient-based algorithms including variations of the gradient descent algorithm (Ruder, 2016) or the BFGS algorithm (Fletcher, 1987). The tuning parameters ${\lambda }_{1n},{\mu }_{1n},{\kappa }_{1n}$ can be selected by cross-validation. The computation time of one replication when $n=200$ is about 0.003 seconds for SepL, 0.003 seconds for FITR-IntL, 0.086 seconds for FITR-Ramp when using the linear kernel, and about 0.457 seconds for SepL, 0.356 seconds for FITR-IntL, 2.744 seconds for FITR-Ramp when using the Gaussian kernel.

3. THEORETICAL RESULTS

In this section, we provide theoretical results about the agreement rate between ${\hat{f}}_{1n}$ and $f_2^{\mathrm{*}}$ in FITR-Ramp. See Section B. 1 in the supplementary material for the convergence rates of the value function ${𝒱}_1({\hat{f}}_{1n})$ and the misclassification rate $\mathbb{P}({\widehat{f}}_{1n}{f}_1^{\mathrm{*}}<0)$.

Without loss of generality, we assume that ${\tilde{f}}_k:𝒳\to \{1,-1\}$ is a binary mapping learned from a dataset of size $N_k$ for all $k=2,\dots ,K$, since every decision function $f$ can be transformed into a binary function by taking its sign. In this section, we assume that the conditional expectation of $R_1$ has already been removed and that the signs have been flipped if the remaining reward is negative. Then the problem becomes

$$\begin{gathered} \underset{f_1\in \mathscr{H}}{\mathrm{m}\mathrm{i}\mathrm{n}}\frac{1}{n}\sum _{i=1}^n\frac{R_{i1}}{\pi \left(A_i;{\mathit{X}}_i\right)}\varphi \left[A_i{f}_1\left({\mathit{X}}_i\right)\right]+{\lambda }_{1n}{∥f_1∥}^2 \\ +\frac{{\mu }_{1n}}{n}\sum _{i=1}^n\sum _{k=2}^K{\mathrm{\Omega }}_{1k}{\psi }_{{\kappa }_{1n}}\left[f_1\left({\mathit{X}}_i\right){\tilde{f}}_k\left({\mathit{X}}_i\right)\right]. \end{gathered}$$

We introduce the following assumptions to derive the convergence rate of the agreement rate.

Assumption 1.

Suppose that $0\le R_1\le c_1$ for all $R_1$ and for some constant $c_1>0$.

Assumption 1 assumes that the primary outcome is nonnegative and bounded. Furthermore, with $r_1^{(a)}(\mathit{X}):=\mathbb{E}\left(R_1\mid A=a,\mathit{X}\right)\ge 0$, we define

$$\eta (\mathit{X}):=\left\{\begin{array}{ll}\frac{r_1^{(1)}(\mathit{X})}{{\sum }_{a\in 𝒜}{r}_1^{(a)}(\mathit{X})},& \text{if}{\sum }_{a\in 𝒜}{r}_1^{(a)}(\mathit{X})>0,\\ \frac{1}{2},& \text{if}{r}_1^{(a)}(\mathit{X})=0\text{for}\text{all}a\in 𝒜.\end{array}\right.$$

The definition of $\eta (\mathit{X})$ aligns with the probability $P(y=1\mid x)$ in classification literature, where $x$ is the predictor, and $y$ is the label. Denote the regions ${𝒳}_{-1}:=\{\mathit{x}\in 𝒳:\eta (\mathit{x})<1/2\}$, ${𝒳}_1:=\{\mathit{x}\in 𝒳:\eta (\mathit{x})>1/2\}$ and ${𝒳}_0:=\{\mathit{x}\in 𝒳:\eta (\mathit{x})=1/2\}$. Finally, we define a distance function $\mathit{x}\mapsto {\omega }_{\mathit{x}}$ as

$${\omega }_{\mathit{x}}:=\left\{\begin{array}{ll}d\left(\mathit{x},{𝒳}_0\cup {𝒳}_1\right),& \text{if}\mathit{x}\in {𝒳}_{-1},\\ d\left(\mathit{x},{𝒳}_0\cup {𝒳}_{-1}\right),& \text{if}\mathit{x}\in {𝒳}_1,\\ 0,& \text{otherwise,}\end{array}\right.$$

where $d(\mathit{x},𝒮)$ denotes the distance of $\mathit{x}$ to a set $𝒮$ with respect to the Euclidean norm.

Assumption 2.

Assume that there exists a constant $q>0$ for the distribution of $\mathit{X}$ such that

$${\int }_{𝒳}\mathrm{e}\mathrm{x}\mathrm{p}\left(-\frac{{\omega }_{\mathit{x}}^2}{t}\right){\mathbb{P}}_{𝒳}(d\mathit{x})\le C{t}^{qd/2}$$ (4)

for all $t>0$ and some constant $C>0$. We say $q=\mathrm{\infty }$ if the inequality holds for all $q>0$.

Note that Assumption 2 is used to bound the approximation error when we estimate ITRs in an RKHS, and the logistic loss is used. It is stronger than the geometric noise assumption (Steinwart and Scovel, 2007) in the sense that $|2\eta (\mathit{x})-1|$ is not included in the left-hand side of (4) and $|2\eta (\mathit{x})-1|\le 1$. However, when $t\to 0$, we can still ensure that the left-hand side of (4) goes to zero.

Assumption 3.

For any $k=2,\dots ,K$, assume that the estimator of the secondary outcome ITR ${\tilde{f}}_k$ converges to $f_k^{\mathrm{*}}$ in the sense that $\mathbb{P}({\tilde{f}}_k{f}_k^{\mathrm{*}}<0)\le {\tilde{\delta }}_{k{N}_k}(\tau )$ with probability at least $1-e^{\tau }$, where $N_k$ is the sample size of the dataset for learning ${\tilde{f}}_k$ and ${\tilde{\delta }}_{k{N}_k}(\tau )=o(1)$ as $N_k\to \mathrm{\infty }$.

Assumption 3 is used to quantify the agreement rate between ${\tilde{f}}_k$ and its corresponding optimal ITR $f_k^{\mathrm{*}}$. It is a weak assumption that only requires ${\tilde{f}}_k$ to converge to its true optimal ITR and can be satisfied by general methods for learning ITRs. We provide the convergence rate ${\tilde{\delta }}_{k{N}_k}(\tau )$ in the remark of Corollary B.2 when ${\tilde{f}}_k$ is learned by SepL.

We first focus on the case when $K=2$. That is, there exists only one secondary treatment rule ${\tilde{f}}_2$, which is already estimated from an external or the same dataset. Define the first and second part in the loss function as

$$\begin{gathered} {\ell }_1\circ f_1(\mathit{Z}):=\frac{R_1}{\pi (A\mid \mathit{X})}\varphi \left[A{f}_1(\mathit{X})\right]\text{and} \\ {\ell }_2\circ f_1(\mathit{Z}):={\mu }_{1n}{\mathrm{\Omega }}_{12}{\psi }_{{\kappa }_{1n}}\left[f_1(\mathit{X}){\tilde{f}}_2(\mathit{X})\right]. \end{gathered}$$

Let the risk based on the surrogate losses be

$$ℛ\left(f_1\right):=\mathbb{E}\left[{\ell }_1\circ f_1+{\ell }_2\circ f_1\right].$$

In the following inequalities, “ $\lesssim$ “ indicates that the left-hand side is no larger than the right-hand side for all $n$ up to a universal constant.

Lemma 3.1.

Under Assumptions B.1 B.3 and 1–2, when ${\mu }_{1n},{\kappa }_{1n}\to 0,{\sigma }_{1n}^2\to \mathrm{\infty }$, and ${\kappa }_n\le 1$ for all $n$, for any constant $\delta >0,0<\nu <2$ and for all $\tau \ge 1$, we have $\mathbb{P}(ℛ({\hat{f}}_{1n})-ℛ(f_1^{\mathrm{*}})\lesssim {\delta }_{1n}(\tau ))\ge 1-e^{-\tau }$, where

$$\begin{gathered} {\delta }_{1n}(\tau ):={\lambda }_{1n}^{-\frac{1}{2}}{n}^{-\frac{1}{2}}\left[\sqrt{\tau }+{\sigma }_{1n}^{(1-\nu /2)(1+\delta )d}{\gamma }_{1n}^{\nu }\right] \\ +{\lambda }_{1n}{\sigma }_{1n}^d+{\gamma }_{1n}(2d{)}^{\frac{qd}{2}}{\sigma }_{1n}^{-qd}, \end{gathered}$$ (5)

and ${\gamma }_{1n}:=1+{\mu }_{1n}{\kappa }_{1n}^{-1}$.

The first term on the right-hand side of (5) is the estimation error, and the sum of the last two terms is the approximation error. A larger class $\mathscr{H}$ generally leads to a larger estimation error and a smaller approximation error, which corresponds to small penalty parameters ${\lambda }_{1n}$ and ${\mu }_{1n}$ with less restriction on the norm of the space or the agreement with the secondary outcome ITR. The optimal rate that balances the estimation error and approximation error is

$${\delta }_{1n}\left(\tau \right)={\gamma }_{1n}^{\frac{2q\nu +2\mathrm{\Delta }+1}{3q+2\mathrm{\Delta }+1}}{n}^{-\frac{q}{3q+2\mathrm{\Delta }+1}},$$ (6)

where $\mathrm{\Delta }=(1-\nu /2)(1+\delta )$. Note that when ${\mu }_{1n}=0$, FITR degenerates to SepL learned with the logistic loss. In this case, the optimal convergence rate is

$${\delta }_{1n}^{(0)}(\tau )=n^{-\frac{q}{3q+2\mathrm{\Delta }+1}}.$$ (7)

The optimal tuning parameters are provided in Section B.2.

Then we can present the convergence rates of the agreement rate $\mathbb{P}({\widehat{f}}_{1n}{f}_2^{\mathrm{*}}>0)$.

Theorem 3.2.

Under Assumptions B.1 B.3 and 1–3, the agreement rate between ${\hat{f}}_{1n}$ and ${\hat{f}}_2^{\mathrm{*}}$ satisfies

$$\mathbb{P}(f_1^{\mathrm{*}}{f}_2^{\mathrm{*}}>0)-\mathbb{P}({\widehat{f}}_{1n}{f}_2^{\mathrm{*}}>0)\lesssim \frac{{\delta }_{1n}(\tau )}{{\mu }_{1n}}+{\tilde{\delta }}_{2N_2}(\tau )$$ (8)

with probability at least $1-2e^{\tau }$.

Now we compare the agreement rate with or without the fusion penalty. Note that for SepL,

$$\begin{gathered} \mathbb{P}(f_1^{\mathrm{*}}{f}_2^{\mathrm{*}}>0)-\mathbb{P}({\widehat{f}}_{1n}{f}_2^{\mathrm{*}}>0)\le \mathbb{P}({\widehat{f}}_{1n}{f}_1^{\mathrm{*}}<0) \\ \lesssim n^{-\frac{\alpha }{2-\alpha }\frac{q}{3q+2\mathrm{\Delta }+1}} \end{gathered}$$ (9)

with high probability under additional Assumptions B. 4 and B.5 (see details in Section B.2). On the other hand, for FITR-Ramp, (8) and (6) implies

$$\begin{gathered} \mathbb{P}(f_1^{\mathrm{*}}{f}_2^{\mathrm{*}}>0)-\mathbb{P}({\widehat{f}}_{1n}{f}_2^{\mathrm{*}}>0) \\ \lesssim {\mu }_{1n}^{-1}{\gamma }_{1n}^{\frac{2q\nu +2\mathrm{\Delta }+1}{3q+2\mathrm{\Delta }+1}}{n}^{-\frac{q}{3q+2\mathrm{\Delta }+1}}+{\tilde{\delta }}_{2N_2}(\tau ) \end{gathered}$$ (10)

with high probability. The inequalities (9) and (10) suggest that SepL can learn the disagreement rate faster than FITR-Ramp when the data are fully separated by the decision boundary, i.e. when $\alpha =1$. However, FITR-Ramp has a faster convergence rate when

$$\left\{\begin{array}{ll}{\mu }_{1n}^{3q-2q\nu }{\kappa }_{1n}^{2q\nu +2\mathrm{\Delta }+1}\gtrsim n^{-\frac{2-2\alpha }{2-\alpha }q},& \text{if}{\mu }_{1n}\gtrsim {\kappa }_{1n},\\ {\mu }_{1n}\gtrsim n^{-\frac{2-2\alpha }{2-\alpha }\frac{q}{3q+2\mathrm{\Delta }+1}},& \text{otherwise,}\end{array}\right.$$ (11)

and ${\tilde{f}}_2$ is learned using SepL and $N_2\gg n$. The requirement of the sample size $N_2$ is consistent with the conclusion in transfer learning literature (Li et al. 2021. Tian and Feng, 2022), which also requires the sample size of auxiliary data to be much larger than the sample size of target data. The relationship (11) holds when, for example, $\alpha =1/2,\nu =3/2,\delta =1,\mathrm{\Delta }=1/2,\mathrm{m}\mathrm{i}\mathrm{n}\left\{{\mu }_{1n},{\kappa }_{1n}\right\}\gtrsim n^{-2q/(9q+6)}$, and the sample size $N_2\gtrsim n^3\mathrm{m}\mathrm{i}\mathrm{n}{\left\{{\mu }_{1n},{\kappa }_{1n}\right\}}^{(9q+6)/q}\gtrsim n$. That is, the data are not fully separated, and the tuning parameters ${\mu }_{1n},{\kappa }_{1n}$ are not decreasing too fast. Besides, (10) does not rely on Assumptions B.4 and B.5. The former is Tsybakov’s noise assumption, and the latter assumes that for each patient, at least one treatment can yield a positive mean reward.

When $K\ge 2$, Theorem 3.2 can be generalized as follows.

Theorem 3.3.

Under Assumptions B.1 B.3 and 1–3. the agreement rate between ${\widehat{f}}_{1n}$ and any $f_k^{\mathrm{*}}$ for $k\ge 2$ satisfies

$$\mathbb{P}(f_1^{\mathrm{*}}{f}_k^{\mathrm{*}}>0)-\mathbb{P}({\widehat{f}}_{1n}{f}_k^{\mathrm{*}}>0)\lesssim \frac{{\delta }_{1n}(\tau )}{{\mu }_{1n}}+\sum _{k=2}^K{\tilde{\delta }}_{k{N}_k}(\tau )$$

with probability at least $1-K{e}^{\tau }$.

4. SIMULATION STUDY

In the simulation study, we learn FITR for the primary outcome using the proposed methods FITR-Ramp and FITR-IntL, comparing them with the baseline method SepL. Suppose the ITR ${\tilde{f}}_{k{N}_k}$ for the secondary outcome $R_k$ is estimated with SepL using an external dataset of sample size $N_k$. Even though the parameter dimension of the Gaussian kernel in RKHS depends on the sample size, the values of $n$ and $N_k$ can be different for ${\widehat{f}}_{1n}$ and ${\tilde{f}}_{k{N}_k}$ since the fusion penalty relies on secondary outcome ITRs only through the treatment suggestions. The weight ${\mathrm{\Omega }}_{1k}$ is chosen to be the Pearson correlation between $R_1$ and $R_k$. Our experiments show that Spearman’s rank correlation (Zar, 2014) yields similar results. Further implementation details can be found in Section A. 1

Simulation Settings.

We assume the data are collected from a randomized controlled trial and $\pi \left(1;{\mathit{X}}_i\right)=\pi \left(-1;{\mathit{X}}_i\right)=0.5$ for all $i=1,\dots ,n$. The $k$ th outcome is defined as $R_{ik}=m_k\left({\mathit{X}}_i\right)+T_k\left({\mathit{X}}_i,A_i\right)+{ϵ}_k\left({\mathit{X}}_i,A_i\right)$, where $m_k$ is the main effect, $T_k$ is the interaction effect between the covariates and the treatment, and ${ϵ}_k$ is the noise term. By choosing different $T_k$, we allow both linear and nonlinear treatment rules. We let the dimension of covariates be $d=10$ (see details in Section A.2). The simulation process is repeated 400 times under each scenario.

The sample size of the external dataset $N_k$ is set to $r\cdot n$ for $k\ge 2$, with $r$ taking values from the set $\{0,1,2,4,8,\mathrm{\infty }\}$. In the case of $r=0$, we do not estimate ${\tilde{f}}_{k{N}_k}$, and FITR degenerates to SepL. When $r=\mathrm{\infty }$, we essentially have ${\tilde{f}}_{k{N}_k}=f_k^{\mathrm{*}}$, which represents the scenario where the true ITR $f_k^{\mathrm{*}}$ is known.

We first assess performance with $K=2$ and $n=200$. Consider the following four scenarios. In all scenarios, the main effects are defined to be nonlinear functions of $\mathit{X}$,

$$m_1\left(\mathit{X}\right)=1+2X_1+X_2^2+X_1{X}_2,$$

$$m_2\left(\mathit{X}\right)=1+2X_1^2+1.5X_2+0.5X_1{X}_2.$$

The interaction terms are defined as follows:

• In linear scenarios $\mathrm{S}1\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{S}2$,

  $$T_1\left(\mathit{X},A\right)=0.5A\left(0.2-X_1-2X_2\right),$$

  $$T_2\left(\mathit{X},A\right)=0.8A\left(0.2-X_1-{\gamma }_1{X}_2\right),$$

  where ${\gamma }_1=1.8$ in S1 and ${\gamma }_1=1.4$ in S2.
• In nonlinear scenarios S3 and S4,

  $$T_1\left(\mathit{X},A\right)=1.0A\left(-2.2-e^{X_1}-e^{X_2}\right),$$

  $$T_2\left(\mathit{X},A\right)=1.5A\left(-{\gamma }_2-e^{X_1}-e^{X_2}\right),$$

  where ${\gamma }_2=2.3$ in S3 and ${\gamma }_2=2.4$ in S4.

The true agreement rate $\mathbb{P}(f_1^{\mathrm{*}}{f}_2^{\mathrm{*}}>0)$ is 98.6% for S1, 94.5% for S2, 96.4% for S3, and 92.8% for S4. Table A. 1 presents the true optimal value functions for each scenario.

Simulation Results.

Our experiments show that the linear kernel performs better in linear scenarios, while the Gaussian kernel is more effective in nonlinear scenarios. We present results using only the most suitable kernel for each scenario.

The average disagreement rate $\mathbb{P}({\hat{f}}_{1n}{f}_2^{\mathrm{*}}<0)$ across all replications, along with its standard deviation, is plotted in Figure 1 for different $N_k$. As expected, the disagreement rate remains unchanged as $N_k$ increases if ${\widehat{f}}_{1n}$ is learned by SepL. In contrast, the fusion penalty’s effect on reducing the disagreement rate becomes more significant with increasing $N_k$. Specifically, FITR-Ramp is better in linear scenarios, while FITR-IntL outperforms it in nonlinear scenarios. This distinction may be attributed to the fact that FITR-IntL imposes a heavier penalty on the disagreement rate when the parameter dimension is high. The improvement in the agreement rate slows down after $N_k/n\ge 2$, indicating that a moderately small external dataset is sufficient for learning an FITR with comparable performance to using the true $f_2^{\mathrm{*}}$. As the difference between outcomes increases, the disagreement rate increases for all methods, but an improvement with FITR is still observed.

Figure 1:

The mean and standard deviation of disagreement rate between FITR ${\hat{f}}_{1n}$ learned using SepL, FITR-Ramp or FITR-IntL, and the true secondary outcome ITR $f_2^{\mathrm{*}}$. The dashed line represents the true disagreement rate $\mathbb{P}\left({\hat{f}}_{1n}{f}_2^{\mathrm{*}}>0\right)$.

In Figure 2, we evaluate FITR’s ability to generate treatment suggestions for the primary outcome. We present two metrics: (a) the root mean square error (RMSE) of the value function ${𝒱}_1({\widehat{f}}_{1n})$ in comparison to the optimal value function ${𝒱}_1(f_1^{\mathrm{*}})$, and (b) the average misclassification rate $\mathbb{P}({\widehat{f}}_{1n}{f}_1^{\mathrm{*}}<0)$ along with its standard deviation. As an additional advantage of the proposed method, the primary outcome is actually improved by the FITR with the help of the secondary outcome. Similarly, FITR-Ramp in S1, S2 and FITR-IntL in S3, S4 observe the minimum RMSE and misclassification rate among all methods. When the discrepancy between outcomes increases, the performance of SepL remains unaffected, but the improvement by FITR-Ramp and FITR-IntL is reduced.

Figure 2:

(a) The RMSE of value functions and (b) the mean and standard deviation of misclassification rate.

Additional results for different values of $K$ and $n$ are available in Section A.3. When $K=2$ and $n=100$ in scenarios S3 and S4, and the Gaussian kernel is used, FITR-Ramp instead of FITR-IntL yields the minimum RMSE of the value function. This indicates that FITR-IntL may be unstable when the sample size is small, and the parameter dimension is high. Furthermore, in cases where $K=3$, the disagreement rate, RMSE, and misclassification rate further decrease. The FITR performs better for the primary outcome as long as either $R_2$ or $R_3$ is close to $R_1$, even if $f_2^{\mathrm{*}}$ and $f_3^{\mathrm{*}}$ differ from $f_1^{\mathrm{*}}$ in opposite directions, suggesting the advantages of incorporating multiple secondary outcomes. In practice, cross-validation can be used to choose the best model and kernel.

Sensitivity Analysis.

To demonstrate the influence of the similarity between outcomes on the estimated FITRs, we fix the primary outcome and vary the secondary outcome when $K=2$. Specifically, we use the same main effect and noise term as in S1 and let the interaction effect be

$$T_1\left(\mathit{X},A\right)=0.5A\left(0.2-X_1-2X_2\right),$$

$$T_2\left(\mathit{X},A\right)=0.8A\left(0.2-X_1-2\rho X_2\right),$$

where $\rho$ controls the similarity between $R_1$ and $R_2$. In Table A. 3 we summarize different values of $\rho$ and their corresponding true agreement rates. The performance metrics, including the disagreement rate, RMSE, and misclassification rate, are presented when $n=200$ and $N_2=4n$. The table shows that the fusion penalty consistently increases the agreement rate between ${\widehat{f}}_{1n}$ and $f_2^{\mathrm{*}}$. However, the value function and accuracy of FITRs exceed SepL only when $\rho \ge 0.5$. Although ${\mu }_{1n}$ tuned by cross-validation should ideally be zero when the outcomes are substantially different, practical constraints such as a small sample size may result in ${\mu }_{1n}>0$, leading to decreased value functions. This suggests that the true agreement rate should be no less than 87% in this scenario for the fusion penalty to have a positive impact on the treatment suggestions for the primary outcome.

5. REAL DATA ANALYSIS

We apply our proposed methods to a study of major depressive disorder (MDD) (Trivedi et al., 2016) which randomized MDD patients to serotonin selective reuptake inhibitor sertraline (SERT) or placebo (PBO).

Description of the Dataset.

The metric for depressive symptoms is the Quick Inventory of Depressive Symptomatology (QIDS) score, with a lower score indicating better relief of symptoms. Seven covariates are used for learning ITRs, including the NEO-Five Factor Inventory score (NEO), Flanker Interference Accuracy score (Flanker), sex, age, education years, Edinburgh Handedness Inventory (EHI) score, and the QIDS score at the baseline (see details in Section A.5).

The primary outcome is the reduction in QIDS at the end of the treatment phase compared to the baseline (QIDS-change), with a larger positive value indicating a more desirable improvement in symptoms. In addition to the primary outcome, we examine two secondary outcomes. One is the clinician-assessed Clinical Global Improvement scale (CGI) as an overall assessment of treatment effect, and a smaller score suggests better improvement (Busner and Targum, 2007). The other is the Social Adjustment Scale (SAS), which evaluates the impact of a person’s mental health difficulties, with a higher score indicating greater impairment. We consider the QIDS-change and the negative values of CGI and SAS as the three outcomes. The dataset contains 186 patients with complete information. Further details can be found in Section A. 5

Analysis Results.

The weights ${\mathrm{\Omega }}_{1k}$ for $k=2,3$ are set by the Pearson correlation between the primary and secondary outcomes, which equals 0.72 and 0.19 for CGI and SAS, respectively. Our experiments show that the linear kernel outperforms the Gaussian kernel on this dataset, so we present the results of the linear kernel. We examine the agreement rates between the ITRs of the primary outcome and secondary outcomes in the upper panel of Table 1. Since we do not have access to the true optimal secondary outcome ITR, the agreement rate is calculated based on the ITRs ${\tilde{f}}_{\text{CGI}}$ and ${\tilde{f}}_{\text{SAS}}$ estimated with SepL. The table suggests that FITR-IntL or FITR-Ramp increases the agreement rate compared to SepL, especially between QIDS-change and CGI.

Table 1:

The upper panel shows the agreement rates between the FITR ${\tilde{f}}_{\text{QIDS}}$ of the primary outcome QIDS-change, learned from SepL, FITR-IntL or FITR-Ramp, and the secondary outcome ITRs ${\tilde{f}}_{\text{CGI}}$, ${\tilde{f}}_{\text{SAS}}$. The lower panel shows the estimated value functions of the OSFA strategy, SepL, FITR-IntL, and FITR-Ramp for the primary outcome.

Agreement rates between ITRs of primary and secondary outcomes
Secondary ITR			${\tilde{f}}_{\text{CGI}}$	${\tilde{f}}_{\text{SAS}}$
Primary ITR learned from		SepL	0.807	0.694
		FITR-IntL	0.903	0.737
		FITR-Ramp	0.887	0.720
Value functions from 400 replicates
All SERT	All PBO	SepL	FITR-IntL	FITR-Ramp
8.000	6.438	8.869 (0.286)	9.082 (0.317)	9.015 (0.327)

We also assess the value functions of the primary outcome for FITR-IntL, FITR-Ramp, SepL, and the one-size-for-all (OSFA) strategy (see the lower panel of Table 1). The value function ${𝒱}_1\left(f_1\right)$ is estimated with the normalized inverse probability weighted estimator

$$\frac{\sum _{i=1}^n{R}_{i1}\mathbb{1}\left[A_i=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\left\{f_k\left({\mathit{X}}_i\right)\right\}\right]/{\pi }_i\left(A_i;{\mathit{X}}_i\right)}{\sum _{i=1}^n\mathbb{1}\left[A_i=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\left\{f_k\left({\mathit{X}}_i\right)\right\}\right]/{\pi }_i\left(A_i;{\mathit{X}}_i\right)},$$ (12)

which is a consistent estimator. The mean and standard deviation of the value functions across 400 replications are reported. Due to the limited sample size, we use 5-fold cross-validation in each replication, averaging the estimated value function across 5 validation sets to obtain the estimate for the current replication. For OSFA, we directly calculate the mean reward on the subpopulation with the desired treatment, which is equivalent to 12 since ${\pi }_i=0.5$ for all $i$. The tuning parameters are selected using 4-fold cross-validation. The results suggest that compared to OSFA and SepL, FITR-IntL and FITR-Ramp can improve the value functions for QIDS-change when assisted by the other secondary outcomes. FITR-IntL has the best performance, and it increases the value function of SepL by approximately one standard deviation.

To compare the learned ITR using different methods, we present the coefficients of each variable in Table A.4. We observe that ${\widehat{f}}_{\text{QIDS}}$ and ${\tilde{f}}_{\text{CGI}}$, when both estimated using SepL without the fusion penalty, share the same signs of the coefficients. This indicates that QIDS-change and negative CGI indeed need similar treatments. In addition, for QIDS-change, the signs of the coefficients in SepL, FITR-Ramp, and FITR-IntL are all consistent, suggesting that the fusion penalty does not change the ITR significantly from SepL.

6. DISCUSSION

In this work, we have proposed a method to optimize the primary outcome while also approximating the optimal secondary outcome ITRs as closely as possible. A fusion penalty is introduced to encourage the ITRs to generate similar treatment recommendations. We demonstrate theoretically and numerically that the agreement rate between the FITR for the primary outcome and the ITR for the secondary outcome converges faster to its true value than SepL. Moreover, the simulation and real data studies show that the ITR learned by FITR-Ramp and FITR-IntL have a higher value function and accuracy than SepL when the true optimal ITRs are closely aligned.

The proposed fusion penalty is a general method that can be easily extended to any variant of outcome-weighted learning (Zhao et al., 2015, Chen et al., 2016. Gao et al. 2022, Ma et al. (2023), a classification-based method to estimate an ITR as the sign of some decision function. There are several promising directions that are worth further research. A possible extension is to let the tuning parameters $\lambda$ and $\mu$ depend on the covariates $\mathit{X}$ since the fusion level should vary if the optimal ITRs for the primary and secondary outcomes differ substantially for specific patients. Another interesting direction is to explore a minimax bound for the convergence rate of the agreement rate. Such a bound might be tighter than the current results presented in Section 3. Nevertheless, the upper bounds for SepL and FITR are currently derived under the same conditions, allowing for a meaningful comparison between them.