Transfer learning of individualized treatment rules from experimental to real-world data

Abstract

Individualized treatment effect lies at the heart of precision medicine. Interpretable individualized treatment rules (ITRs) are desirable for clinicians or policymakers due to their intuitive appeal and transparency. The gold-standard approach to estimating the ITRs is randomized experiments, where subjects are randomized to different treatment groups and the confounding bias is minimized to the extent possible. However, experimental studies are limited in external validity because of their selection restrictions, and therefore the underlying study population is not representative of the target real-world population. Conventional learning methods of optimal interpretable ITRs for a target population based only on experimental data are biased. On the other hand, real-world data (RWD) are becoming popular and provide a representative sample of the target population. To learn the generalizable optimal interpretable ITRs, we propose an integrative transfer learning method based on weighting schemes to calibrate the covariate distribution of the experiment to that of the RWD. Theoretically, we establish the risk consistency for the proposed ITR estimator. Empirically, we evaluate the finite-sample performance of the transfer learner through simulations and apply it to a real data application of a job training program.

1. Introduction

A personalized recommendation, tailored to individual characteristics, is becoming increasingly popular in real life such as healthcare, education and e-commerce. An optimal individualized treatment rule (ITR) is defined as maximizing the mean value functionof an outcome of interest over the target population had all individuals in the whole population follow the given rule. Many machine learning approaches to estimating the optimal ITR are available, such as outcome-weighted learning with support vector machine (Zhao et al. 2012; Zhou et al. 2017), regression-based methods with Adaboost (Kang et al. 2014), regularized linear basis function (Qian and Murphy 2011), K-nearest neighbor (Zhou and Kosorok 2017) and generalized additive model (Moodie et al. 2014). However, the ITRs derived by machine learning methods can be too complex to extract practically meaningful insights for policymakers. For example, in healthcare, clinicians need to scrutinize the estimated ITRs for scientific validity, but the black-box rules conceal the clear relationship between patient characteristics and treatment recommendations. In these cases, parsimonious and interpretable ITRs are desirable. Interpretable ITRs usually refer to the ones where clinicians can explicitly obtain the information of how the decision rules recommend a certain treatment/intervention given a patient’s covariate information instead of some complex black-box approaches such as random forest and neural network. We consider the simple forms: linear rules, which can help clinicians to see how each covariate leads to the final treatment recommendations. Other examples of interpretable decision rules include the decision trees (Laber and Zhao 2015; Tao et al. 2018) and “if-then-else” decision lists (Gail and Simon 1985; Zhang et al. 2015; Lakkaraju and Rudin 2017).

It is well known that randomized experiments are the gold standard for learning the optimal ITRs because randomization of treatments in the design stage can ensure no unmeasured confounding (Greenland 1990). However, randomized experiments might lack external validity or representativeness (Rothwell 2005) because of the selective criteria for individual eligibility in the experiments. Consequently, the distribution of the covariates in the randomized experiments might differ from that in the target population. Thus, a parsimonious ITR constructed from randomized experiments cannot directly generalize to a target population (Zhao et al. 2019) due to the covariate shift. Alternatively, real-world data (RWD) such as population-based survey data usually involve a large and representative sample of the target population. Figure 1 presents a simple example to illustrate the bias of the estimated optimal linear ITR based only on the experiment. A large body of literature in statistics has focused on transporting or generalizing the average treatment effects from the experiments to a target population or a different study population (e.g., Cole and Stuart 2010; Hartman et al. 2015). One exception for the ITRs is Zhao et al. (2019) and Mo et al. (2020) which estimates the optimal linear ITR from the experimental data by optimizing the worst-case quality assessment among all covariate distributions in the target population satisfying some moment conditions or distributional closeness, while we address the problem in a different situation where we already have a representative sample at hand.

Fig. 1.

Linear scores from the population, experiment, and transfer-learned optimal linear ITRs. The population $\mathit{X}$ follows Unif^[0,1] (the x-axis), and the conditional average treatment effect is $\tau (\mathit{x})=\mathit{x}(\mathit{x}-3/4)$ (the solid black line). The experimental sample of size 6227 (red dots) is selected based on a Bernoulli trial with selection probability $\exp (-11\mathit{x})/\{1+\exp (-11\mathit{x})\}$, and the RWD sample is a random sample of 8000 from the population. The plot shows that the linear score of the optimal linear ITR based only on the experiment (the dash black line) is far from the population counterpart (the solid blue line); while the one from the transfer learner (the dot green line) becomes closer by utilizing the representativeness of the RWD sample (Section 3).

In this article, we propose a general framework for estimating a generalizable optimal interpretable ITR from randomized experiments to a target population represented by the RWD. We propose transfer weighting that shifts the covariate distribution in the randomized experiments to that in the RWD. To correctly estimate the population optimal ITR, transfer weights are used in two places. First, the empirical estimator of the population value function is biased when considering only the experimental sample, and hence the transfer weights are used to correct the bias of the estimator of the value function. Second, the nuisance functions in the value function such as the propensity score and the outcome mean also require transfer weighting to estimate the population counterparts. Then we can obtain the weighted interpretable optimal ITR by reformulating the problem of maximizing the weighted value to that of minimizing the weighted classification error with flexible loss function choices. We consider different parametric and nonparametric approaches to learning transfer weights. Moreover, a practical issue arises regarding the choice of transfer weights. Weighting can correct for the bias but also reduce the effective sample size and increase the variance of the weighted value estimator. To address the bias-variance tradeoff, we propose a cross-validation procedure that maximizes the estimated population value function to choose among the weighting schemes. Continuing with the simple example in Figure 1, the optimal linear ITR from the proposed transfer learner becomes closer to the population optimal linear ITR than the biased experimental estimator. Lastly, we provide the theoretical guarantee of the risk consistency of the proposed transfer weighted ITRs.

The remaining of the article is organized as follows. We introduce our proposed method in Section 3. In Section 4, we study the asymptotic properties of the proposed estimator to guarantee its performance in the target population theoretically. In Section 5, we use various simulation settings to illustrate how this approach works. In Section 6, we apply the proposed method to the National Supported Work program (LaLonde 1986) for improving the decisions of whether to recommend the job training program based on the individual characteristics of the target population. We conclude the article in Section 7.

2. Basic Setup

2.1. Notation

Let $\mathit{X}\in 𝒳\subseteq {ℝ}^p$ be a vector of pre-treatment covariates, $A\in 𝒜=\{0,1\}$ the binary treatment, and $Y\in ℝ$ the outcome of interest. Under the potential outcomes framework (Rubin 1974), let $Y^{\ast }(a)$ denote the potential outcome had the individual received treatment $a\in 𝒜$. Define a treatment regime as a mapping $d:𝒳\to 𝒜$ : from the space of individual characteristic to the treatment. Then we define the potential outcome under regime d as $Y^{\ast }(d)=\sum _{a\in 𝒜}{Y}^{\ast }(a)\mathbf{1}\{d(\mathit{X})=a\}$, the value of regime $d$ as $V(d)=\mathbb{E}\left\{Y^{\ast }(d)\right\}$, where the expectation is taken over the distribution in the target population, and the optimal regime $d^{\text{opt}}$ as the one satisfying $V\left(d^{\text{opt}}\right)\ge V(d),\forall d$. As discussed in the introduction, in many applications especially the high-stake ones such as supporting clinicians and judges to make decisions (Lakkaraju and Rudin 2017), interpretable decision rules are desirable since they might bring more domain insights and be more trustworthy to decision-makers compared with the complex black-box regimes. Thus, it is valuable to learn an optimal interpretable regime. Specifically, we focus on the regimes with linear forms, i.e, $d(\mathit{X};\eta )=\mathbf{1}\left({\eta }_0+{\eta }_1^{\top }\mathit{X}>0\right)$, denoted by $d_{\eta }$ for simplicity.

We aim at learning ITRs which lead to the highest benefits for the target population with size $N$, ${\left\{{\mathit{X}}_i,Y_i^{\ast }(0),Y_i^{\ast }(1)\right\}}_{i=1}^N$. Suppose we have access to two independent data sources: the experiment data with ${\left\{\left({\mathit{X}}_i,A_i,Y_i\right)\right\}}_{i\in {𝓘}_n}$ and the RWD with ${\left\{{\mathit{X}}_i\right\}}_{i\in {𝓘}_m}$, where, ${𝓘}_n,{𝓘}_m\subseteq \{1,\dots ,N\}$ are samples’ index sets with size $n$, $m$, respectively. In the experimental data, individuals are selected based on certain criteria and then are randomized to receive treatments. The experimental data might induce selection bias which leads to a different distribution of $\mathit{X}$ in experiments from that in the target population. On the other hand, we assume that individuals in the RWD are a random sample from the target population, thus the distribution of the covariates in the RWD can characterize that of the target population. Let $S_i$ be the binary indicator of the $i$th individual participates in the randomized experiment: $S_i=1$ if $i\in {𝓘}_n$ and 0 if $i\in {𝓘}_m$. We denote ${\pi }_A(\mathit{X})$ as the propensity score of receiving the active treatment $P(A=1\mid \mathit{X}),Q(\mathit{X},a)$ as the conditional expectation of potential outcomes given covariates $\mathbb{E}\left\{Y^{\ast }(a)\mid \mathit{X}\right\}$, and $\tau (\mathit{X})$ as the contrast function $Q(\mathit{X},1)-Q(\mathit{X},0)$.

We make the following assumptions, which are standard and well-studied assumptions in the causal inference and treatment regime literature (Chakraborty 2013).

Assumption 1.

a. $A\perp \perp \left\{Y^{\ast }(0),Y^{\ast }(1)\right\}\mid (\mathit{X},S=1)$. b: $Y=Y^{\ast }(A)$. c: There exist $c_1,c_2\in (0,1)$ such that ${\pi }_A(\mathit{X})\in \left[c_1,c_2\right]$ for all $\mathit{X}\in 𝒳$.

Assumption 1.a holds by the design of a completely randomized experiment, where the treatment is independent of the potential outcomes and covariates. It also holds by the design of a conditional randomized experiment, where the treatment is independent of the potential outcomes within each level of $\mathit{X}$. Besides, it also holds in observational studies without unmeasured confounders. In general, our framework is applicable to generalizing the findings from the observational studies to the target population.

2.2. Existing learning methods for ITRs

Identifying the optimal individualized treatment rules has been studied in a large body of statistical literature. One category is regression-based by directly estimating the conditional expectation of the outcome given patients’ characteristics and treatments received (Murphy 2005). Both parametric and nonparametric models can be used to approximate $Q(\mathit{X},A)$. To encourage interpretability, one strategy is to posit a parametric model as $Q(\mathit{X},A;\beta )$, and the estimated optimal ITR within the class indexed by $\beta$ is defined as $\mathbf{1}\{\tau (\mathit{X};\widehat{\beta })>0\}$.

Another category is value-based by first estimating the value of an ITR and then searching the ITR that renders the highest value. One strategy is to estimate ${\pi }_A(\mathit{X})$ first, where one common way is to posit a parametric model such as logistic regression or a constant, and then use the inverse probability weighting (IPW; Horvitz and Thompson 1952) to estimate the value of an ITR indexed by $\eta$. This estimator is consistent to the true value only if ${\pi }_A\left({\mathit{X}}_i\right)$ is correctly specified and can be unstable if the estimate ${\widehat{\pi }}_A\left({\mathit{X}}_i\right)$ is close to zero. The augmented inverse probability weighting (AIPW; Zhang, Tsiatis, Laber and Davidian (2012)) approach is introduced to combine the IPW and outcome regression estimator $\widehat{Q}\left({\mathit{X}}_i,A\right)$ to estimate the value

$${\widehat{V}}_{\text{aw}}\left(d_{\eta };{𝓘}_n\right)=\frac{1}{n}\sum _{i\in {𝓘}_n}\left(\left[\frac{A_{i}d\left({\mathit{X}}_i;\eta \right)}{{\widehat{\pi }}_A\left({\mathit{X}}_i\right)}+\frac{\left(1-A_i\right)\left\{1-d\left({\mathit{X}}_i;\eta \right)\right\}}{1-{\widehat{\pi }}_A\left({\mathit{X}}_i\right)}\right]\left[Y_i-\widehat{Q}\left\{{\mathit{X}}_i,d\left({\mathit{X}}_i;\eta \right)\right\}\right]-\widehat{Q}\left\{{\mathit{X}}_i,d\left({\mathit{X}}_i;\eta \right)\right\}\right).$$ (1)

The AIPW estimator is doubly robust in the sense that it will be consistent to the value of the ITR $d_{\eta }$ if either the model for ${\pi }_A(\mathit{X})$ or $Q(\mathit{X},A)$ is correctly specified, but not necessarily both.

Learning the optimal ITR by optimizing the value can also be formulated as a classification problem (Zhang, Tsiatis, Davidian, Zhang and Laber 2012). The classification perspective has equivalent forms to all the regression-based and value-based methods above, and can also bring us additional optimization advantages over directly value searching.

Lemma 1.

The optimal treatment rule by maximizing $V\left(d_{\eta }\right)$ in $\eta$ is equivalent to that by minimizing the risk

$$\mathbb{E}\left(|\tau (\mathit{X})|{[\mathbf{1}\{\tau (\mathit{X})>0\}-d(\mathit{X};\eta )]}^2\right)$$ (2)

in $\eta$ (Zhang, Tsiatis, Davidian, Zhang and Laber 2012).

Lemma 1 reformulates the problem of estimating an optimal treatment rule as a weighted classification problem, where $|\tau (\mathit{X})|$ is regarded as the weight, $\mathbf{1}\{\tau (\mathit{X})>0\}$ the binary label, and $d(\mathit{X};\eta )$ the classification rule of interest. Let $l_{0-1}$ be a 0–1 loss, i.e., $l_{0-1}(u)=\mathbf{1}(u\le 0)$. The objective function (2) can also be rewritten as

$${𝓡}_{𝓕}\left(\tau ;\eta ,l_{0-1}\right)=\mathbb{E}\left\{{𝓛}_{\left(\tau ;\eta ,l_{0-1}\right)\},}{𝓛}_{\left(\tau ;\eta ,l_{0-1}\right)}=|\tau (\mathit{X})|l_{0-1}\{[2\mathbf{1}\{\tau (\mathit{X})>0\}-1]f(\mathit{X};\eta )\right\},$$ (3)

Where $f(\mathit{X};\eta )={\eta }^{\top }\mathit{X},d(\mathit{X};\eta )=\mathbf{1}\{f(\mathit{X};\eta )>0\}$, $𝓕$ is a certain class covering the parameters of the ITRs, i.e., $\eta \in 𝓕$.

Let ${\widehat{\tau }}_i$ be a predictor of $\tau \left({\mathit{X}}_i\right)$. From Lemma 1, to estimate the optimal ITR, one can minimize the following empirical objective function:

$$\begin{array}{l}{n}^{-1}\sum _{i\in {𝓘}_n}\left|{\widehat{\tau }}_i\right|{\left\{\mathbf{1}\left({\widehat{\tau }}_i>0\right)-d\left({\mathit{X}}_i;\eta \right)\right\}}^2=n^{-1}\sum _{i\in {𝓘}_n}\left|{\widehat{\tau }}_i\right|\mathbf{1}\left\{\mathbf{1}\left({\widehat{\tau }}_i>0\right)\ne d\left({\mathit{X}}_i;\eta \right)\right\}\\ =n^{-1}\sum _{i\in {𝓘}_n}\left|{\widehat{\tau }}_i\right|l_{0-1}\left[\left\{2\mathbf{1}\left({\widehat{\tau }}_i>0\right)-1\right\}f\left({\mathit{X}}_i;\eta \right)\right]\\ =n^{-1}\sum _{i\in {𝓘}_n}𝓛\left({\widehat{\tau }}_i;\eta ,l_{0-1}\right).\end{array}$$ (4)

In this classification framework, we can also consider the three different approaches to estimating $\tau \left({\mathit{X}}_i\right)$, which have the equivalent forms and enjoy the same properties with the regression- and value-based estimator. For example, the AIPW-type predictor of $\tau \left({\mathit{X}}_i\right)$ is

$${\widehat{\tau }}_{\text{aw},i}=\left[\frac{\mathbf{1}\left(A_i=1\right)\left\{Y_i-\widehat{Q}\left({\mathit{X}}_i,1\right)\right\}}{{\widehat{\pi }}_A\left({\mathit{X}}_i\right)}+\widehat{Q}\left({\mathit{X}}_i,1\right)\right]-\left[\frac{\mathbf{1}\left(A_i=0\right)\left\{Y_i-\widehat{Q}\left({\mathit{X}}_i,0\right)\right\}}{1-{\widehat{\pi }}_A\left({\mathit{X}}_i\right)}+\widehat{Q}\left({\mathit{X}}_i,0\right)\right].$$

The solution obtained by minimizing (4) in $\eta$ with replacing ${\widehat{\tau }}_i$ with ${\widehat{\tau }}_{\text{aw},i}$ is equivalent to that obtained by maximizing the AIPW value estimator ${\widehat{V}}_{\text{aw}}\left(d_{\eta };{𝓘}_n\right)$ in (1) and enjoys the doubly robust property.

When the experimental sample is representative of the target population, the above ITR learning methods can estimate the population optimal ITRs that lead to the highest value. To correct for the selection bias of the experimental sample, we propose an integrative strategy that leverages the randomization in the experimental sample and the representativeness of the RWD. As pointed out by Zhao et al. (2019), the optimal rule obtained from the experimental sample is also asymptotically optimal for the target population if no restriction to the class of $d$, but such optimal rules are too complex. Therefore, we focus on learning interpretable and parsimonious ITRs, and thus integrative methods are required to correct for the selection bias of the experimental sample.

3. Integrative transfer learning of ITRs

3.1. Assumptions for identifying population optimal ITRs

To identify population optimal ITRs, we assume the covariates $\mathit{X}$ can fully explain the selection mechanism of inclusion into the randomized experiment and the covariate distribution in the RWD is representative to the target population.

Assumption 2.

a: $\left\{Y^{\ast }(0),Y^{\ast }(1)\right\}\perp \perp S\mid \mathit{X}$. b: $f(\mathit{X}\mid S=0)=f(\mathit{X})$. c: There exist $c_3,c_4\in (0,1)$ such that ${\pi }_S(\mathit{X})\in \left[c_3,c_4\right]$ for all $\mathit{X}\in 𝒳$.

Assumption 2.a is commonly assumed in the generalizability literature (see Stuart et al. (2011); Buchanan et al. (2018)), which requires that the observed covariates $\mathit{X}$ are rich enough such that it can decide whether or not the subject is selected into the experimental data. Weaker versions of the assumption include the mean exchangeability, i.e., $\mathbb{E}\left\{Y^{\ast }(a)\mid \mathit{X},S=0\right\}=\mathbb{E}\left\{Y^{\ast }(a)\mid \mathit{X},S=1\right\}$, $a=0,1$ (Zhao et al. 2019; Dahabreh et al. 2019).

Assumption 2.b requires that covariate distribution in the RWD is the same as that in the target population so that we can leverage the representativeness of the RWD. As a result, it implies that $m^{-1}\sum _{i\in {𝓘}_m}g\left({\mathit{X}}_i\right)$ would be an unbiased estimator of $\mathbb{E}\{g(\mathit{X})\}$, the covariate expectation in the target population for any $g$. Assumption 2.a and Assumption 2.b are also adopted in the covariate shift problem in machine learning, a sub-area in domain adaptation, where standard classifiers do not perform well when the independent variables in the training data have a different covariate distribution from that in the testing data. Re-weighting individuals to correct the over- or under-representation is one way to cope with the covariate shift in classification problems (Kouw and Loog 2018).

Assumption 2.c means that all subjects have nonzero probabilities of being selected into the experiment which require no existence of covariates that prohibit to participate the experiment, which is used to make $1/{\pi }_S(\mathit{X})$ well-defined. If Assumption 2.c is violated, the support of covariates in two populations has limited overlap, so generalization can only be made to a restricted population without extrapolation (Yang and Ding 2018). This is also a common assumption in covariate shift literature (Uehara et al. 2020).

The following proposition identifies the population value function using the inverse of sampling score as the transfer weight, and leads to an equivalent weighted classification loss function to identify the population optimal ITR.

Proposition 1.

Let the transfer weight be $w={\pi }_S^{-1}(\mathit{X})$. Under Assumption 1.b, Assumption 2.a and Assumption 2.c, the population value of an ITR $d_{\eta }$ is identified by $V\left(d_{\eta }\right)=\mathbb{E}\left[\omega S{Y}^{\ast }\left\{d_{\eta }(\mathit{X})\right\}\right]$. Then the population optimal ITR can be obtained by minimizing

$$\mathbb{E}\left(wS|\tau (\mathit{X})|{\left[\mathbf{1}\{\tau (\mathit{X})>0\}-d_{\eta }(\mathit{X})\right]}^2\right)\text{in}\eta .$$

Two challenges arise for using Proposition 1 to estimate the optimal ITR: (i) the weights $\omega$ are unknown and (ii) the estimators of $\tau (\mathit{X})$ discussed so far such as ${\widehat{\tau }}_{\text{aw}}$ are biased based only on the experimental sample. In the following subsections, we present the estimation of the transfer weights and unbiased estimators of $\tau (\mathit{X})$.

3.2. Estimation of transfer weights

We consider various parametric and nonparametric methods to estimate the transfer weights $\omega$. Each method has its own benefits: parametric methods are easy to implement and efficient if the models are correctly specified, while nonparametric methods are more robust and less sensitive to model misspecification.

3.2.1. Parametric approach

For parametric approaches, we assume the target population size $N$ is known. Similar to modeling propensity score ${\pi }_A(\mathit{X})$, we can posit a logistic regression model for ${\pi }_S(\mathit{X})$; i.e., $\text{logit}\left\{{\pi }_s(\mathit{X};\alpha )\right\}={\alpha }_0+{\alpha }_1^{\top }\mathit{X}$, where $\alpha ={\left({\alpha }_0,{\alpha }_1^{\top }\right)}^{\top }\in {ℝ}^{p+1}$. The typical maximum likelihood estimator of $\alpha$ can be obtained by

$$\begin{array}{l}\widehat{\alpha }=\underset{\alpha }{\arg \max }\frac{1}{N}\sum _{i=1}^N\left[S_i\text{log}{\pi }_s\left({\mathit{X}}_i;\alpha \right)+\left(1-S_i\right)\text{log}\left\{1-{\pi }_s\left({\mathit{X}}_i;\alpha \right)\right\}\right]\\ =\underset{\alpha }{\arg \max }\frac{1}{N}\sum _{i=1}^N\left[S_i\left({\alpha }_0+{\alpha }_1^{\top }{\mathit{X}}_i\right)-\text{log}\left\{1+\exp \left({\alpha }_0+{\alpha }_1^{\top }{\mathit{X}}_i\right)\right\}\right].\end{array}$$ (5)

However, the second term $N^{-1}\sum _{i=1}^N\text{log}\left\{1+\exp \left({\alpha }_0+{\alpha }_1^{\top }{\mathit{X}}_i\right)\right\}$ in (5) is not feasible to calculate because we can not observe $\mathit{X}$ for all individuals in the population. The key insight is that this term can be estimated by $m^{-1}\sum _{i\in {𝓘}_m}\text{log}\left\{1+\exp \left({\alpha }_0+{\alpha }_1^{\top }{\mathit{X}}_i\right)\right\}$ based on the RWD. This strategy leads to our modified maximum likelihood estimator

$$\widehat{\alpha }=\underset{\alpha }{\arg \max }\left[\frac{1}{N}\sum _{i=1}^N{S}_i\left({\alpha }_0+{\alpha }_1^{\top }{\mathit{X}}_i\right)-\frac{1}{m}\sum _{i=1}^m\text{log}\left\{1+\exp \left({\alpha }_0+{\alpha }_1^{\top }{\mathit{X}}_i\right)\right\}\right].$$

Alternatively, Assumption 2.b leads to unbiased estimating equations of $\alpha$

$$\mathbb{E}\left\{\frac{Sg(\mathit{X})}{{\pi }_S(\mathit{X};\alpha )}\right\}=\mathbb{E}\{g(\mathit{X})\},$$

for any $g$. Therefore, we can use the following estimating equations

$$\frac{1}{N}\sum _{i=1}^N\frac{S_{i}g\left({\mathit{X}}_i\right)}{{\pi }_S\left({\mathit{X}}_i;\alpha \right)}=\frac{1}{m}\sum _{i\in {𝓘}_m}g\left({\mathit{X}}_i\right)$$

for $\alpha$, where $g(\mathit{X})\in {ℝ}^{p+1}$. For simplicity, one can take $g(\mathit{X})$ as $\partial \text{logit}\left\{{\pi }_S(\mathit{X};\alpha )\right\}/\partial \alpha$.

3.2.2. Nonparametric approach

The above parametric approaches require ${\pi }_S(\mathit{X};\alpha )$ to be correctly specified and the population size $N$ to be known. These requirements may be stringent, because the sampling mechanism into the randomized experiment is unknown and the population size is difficult to obtain. We now consider the constrained optimization algorithm of Wang and Zubizarreta (2020) to estimate the transfer weights by $\underset{w}{\min }\sum _{i=1}^N{S}_i{w}_i\text{log}{w}_i$ subject to

$$\left|\sum _{i=1}^N{w}_i{S}_i{g}_k\left({\mathit{X}}_i\right)-\frac{1}{m}\sum _{i\in {𝓘}_m}{g}_k\left({\mathit{X}}_i\right)\right|\le {\sigma }_k(k=1,\dots ,K),$$ (6)

Where ${\sigma }_k\ge 0$; $w_i\in [0,1]$, $i\in {𝓘}_n$, $\sum _{i\in {𝓘}_n}{w}_i=1$, and $\left\{g_1,\dots ,g_K\right\}$ can be chosen as the first-, second- and higher order moments of the covariate distribution. The constants ${\sigma }_k$ are the tolerate limits of the imbalances in $g_K$. When all the ${\sigma }_k$’s are taken 0, (6) becomes the exact balance, and the solution $w$ becomes the entropy balancing weights (Hainmueller 2012). The choice of ${\sigma }_k$ is related to the standard bias-variance tradeoff. On the one hand, if ${\sigma }_k$ is too small, the weight distribution can have large variability in order to satisfy the stringent constraints, and in some extreme scenarios, the weights may not exist. On the other hand, if ${\sigma }_k$ is too large, the covariate imbalances remain and therefore the resulting weights are not sufficient to correct for the selection bias. Following Wang and Zubizarreta (2020), we choose ${\sigma }_k$ from a pre-specified set by the tuning algorithm described in Supplementary Materials S1.

3.3. Estimation of $\tau (\mathit{X})$

The transfer weights are also required to estimate $\tau (\mathit{X})$ from the experimental data. In parallel to Section 2.2, we can obtain the weighted estimators of $\tau (\mathit{X})$. For example, the transfer AIPW-type predictor of $\tau \left({\mathit{X}}_i\right)$

$${\widehat{\tau }}_{\text{aw},i}^w=\left[\frac{\mathbf{1}\left(A_i=1\right)\left\{Y_i-{\widehat{Q}}_w\left({\mathit{X}}_i,1\right)\right\}}{{\widehat{\pi }}_A^w\left({\mathit{X}}_i\right)}+{\widehat{Q}}_w\left({\mathit{X}}_i,1\right)\right]-\left[\frac{\mathbf{1}\left(A_i=0\right)\left\{Y_i-{\widehat{Q}}_w\left({\mathit{X}}_i,0\right)\right\}}{1-{\widehat{\pi }}_A^w\left({\mathit{X}}_i\right)}+{\widehat{Q}}_w\left({\mathit{X}}_i,0\right)\right],$$

where ${\widehat{Q}}_w(\cdot ,a)$ can be estimated by weighted least square for linear $Q$ functions with the estimated transfer weights, and ${\widehat{\pi }}_A^w\left({\mathit{X}}_i\right)={\widehat{P}}_w\left(A_i=1\mid {\mathit{X}}_i\right)$ is the weighted propensity score which can be obtained by the weighted regression model with the estimated transfer weights using the experimental data.

Let ${\widehat{\tau }}_i^w$ be a generic weighted predictor of $\tau (\mathit{X})$. Proposition 1 implies an empirical risk function to estimate $\eta$

$$\frac{1}{n}\sum _{i\in {𝓘}_n}{\widehat{w}}_i\left|{\widehat{\tau }}_i^w\right|{\left\{\mathbf{1}\left({\widehat{\tau }}_i^w>0\right)-d\left({\mathit{X}}_i;\eta \right)\right\}}^2=\frac{1}{n}\sum _{i\in {𝓘}_n}{\widehat{w}}_i\left|{\widehat{\tau }}_i^w\right|l_{0-1}\left[\left\{2\mathbf{1}\left({\widehat{\tau }}_i^w>0\right)-1\right\}f\left({\mathit{X}}_i;\eta \right)\right].$$ (7)

It is challenging to optimize the objective function (7) since it involves a non-smooth non-convex objective function of $\eta$. One way is to optimize it directly using grid search or the genetic algorithm in Zhang, Tsiatis, Laber and Davidian (2012) for learning linear decision rules (implemented with rgenoud in R), but grid search becomes untenable for a higher dimension of $\eta$, and the genetic algorithm cannot guarantee a unique solution. Alternatively, one can replace $l_{0-1}$ with a surrogate loss function, such as a hinge loss used in the support vector machine (Vapnik 1998) and outcome weighted learning (Zhao et al. 2012), or a ramp loss (Collobert et al. 2006) which truncates the unbounded hinge loss by trading convexity for robustness to outliers (Wu and Liu 2007). We adopt the smoothed ramp loss function proposed in Zhou et al. (2017)

$$l(u)=0\text{if}u\ge 1,{(1-u)}^2\text{if}0\le u<1,2-{(1+u)}^2\text{if}-1\le u<0,2\text{if}u\le -1.$$ (8)

which retains the robustness and gains computational advantages for being smooth everywhere. $\mathrm{l}(u)$ is the difference of two smooth convex functions, $l(u)=l_1(u)-l_0(u)$, where

$$l_s(u)=0\text{if}u\ge s;{(s-u)}^2\text{if}s-1\le u<s;2s-2u-1\text{if}u<s-1.$$ (9)

Similarly to Zhou et al. (2017), we can apply the d.c. algorithm (Le Thi Hoai and Tao 1997) to solve the non-convex minimization problem by first decomposing the original objective function into the summation of a convex component and a concave component and then minimizing a sequence of convex subproblems. Applying (9) to (7), our empirical risk objective becomes

$$\underset{\text{a}\text{convex}\text{function}\text{of}\eta }{\underbrace{\frac{1}{n}\sum _{i\in {𝓘}_n}{\widehat{w}}_i\left|{\widehat{\tau }}_i^w\right|l_1\left(u_i\right)}}+\underset{\text{a}\text{concave}\text{function}\text{of}\eta }{\underbrace{\frac{1}{n}\sum _{i\in {𝓘}_n}\left\{-{\widehat{w}}_i\left|{\widehat{\tau }}_i^w\right|l_0\left(u_i\right)\right\}}},$$ (10)

where $u_i=\left\{2\mathbf{1}\left({\widehat{\tau }}_i^w>0\right)-1\right\}f\left({\mathit{X}}_i;\eta \right)$. Let $C_i^{\text{cav}}=-{\widehat{w}}_i\left|{\widehat{\tau }}_i^w\right|l_0\left(u_i\right)$ and

$${\xi }_i=\frac{\partial C_i^{\text{cav}}}{\partial u_i}=-{\widehat{w}}_i\left|{\widehat{\tau }}_i^w\right|\frac{\partial l_0\left(u_i\right)}{\partial u_i},$$ (11)

for $i\in {𝓘}_n$. Therefore, the convex subproblem at $(t+1)$ th iteration in the d.c. algorithm is to

$$\underset{\eta }{\min }\frac{1}{n}\sum _{i\in {𝓘}_n}{\widehat{w}}_i\left|{\widehat{\tau }}_i^w\right|l_1\left(u_i\right)+\frac{1}{n}\sum _{i\in {𝓘}_n}{\xi }_i^{(t)}{u}_i,$$ (12)

which is a smooth unconstrained optimization problem and can be solved with many efficient algorithms such as L-BFGS (Nocedal 1980). Algorithm 2 in the Supplementary Material summarizes the proposed procedure to obtain the minimizer $\widehat{\eta }$.

3.4. Cross-validation

We have discussed various approaches for the estimation of transfer weights and the individual contrast function. Although transfer weights can correct the selection bias of the experimental data, they also increase the variance of the estimator compared to the unweighted counterpart. The effect of weighting on the precision of estimators can be quantified by the effective sample size originated in survey sampling (Kish and Stat 1992). If the true contrast function can be approximated well by linear models, the estimated ITR with transfer weights may not outperform the one without weighting. As a result, for a given application, it is critical to developing a data-adaptive procedure to choose the best method among the aforementioned weighted estimators and whether or not to use transfer weights.

Toward this end, we propose a cross-validation procedure. To proceed, we split the sample into two parts, one for learning the ITRs and the other for evaluating the estimated ITRs. We also adopt multiple sample splits to increase the robustness since a single split might have dramatic varying results due to randomness. For evaluation of an ITR $d_{\eta }$, we estimate the value for the given ITR by the weighted AIPW estimator in (13) with the estimated nonparametric transfer weights $\widehat{w}$,

$$\widehat{V}\left(d_{\eta };\widehat{w},{𝓘}_n\right)=\sum _{i\in {𝓘}_n}{\widehat{w}}_i\left(\left[\frac{A_{i}d\left({\mathit{X}}_i;\eta \right)}{{\widehat{\pi }}_A^w\left({\mathit{X}}_i\right)}+\frac{\left(1-A_i\right)\left\{1-d\left({\mathit{X}}_i;\eta \right)\right\}}{1-{\widehat{\pi }}_A^w\left({\mathit{X}}_i\right)}\right]\left[Y_i-{\widehat{Q}}_w\left\{{\mathit{X}}_i,d\left({\mathit{X}}_i;\eta \right)\right\}\right]-{\widehat{Q}}_w\left\{{\mathit{X}}_i,d\left({\mathit{X}}_i;\eta \right)\right\}\right).$$ (13)

Since we are more interested in the comparisons among different weighting and unweighting methods instead of different contrast function estimation methods, thus we fix an estimation method for the contrast function at first, and then use cross-validation to choose among the set of candidate methods $𝒢$. This $𝒢$ contains the proposed learning method with nonparametric transfer weights described in Section 3.2.2, and unweighted estimation method. If the target population size $N$ is available, we can also include the weighted estimators with the two parametric approaches introduced in Section 3.2.1 into $𝒢$. Then we can get the value estimates averaged over all the sample splits for each learning method in $𝒢$, and return the method with the highest value estimates. The more explicit description of the procedure is presented in Algorithm 3 in the Supplementary Material.

4. Asymptotic Properties

In this section, we provide a theoretical guarantee that the estimated ITR within a certain class $𝓕$ is consistent with the true population optimal ITR within $𝓕$. Toward this end, we show that the expected risk of the estimated ITR is consistent with that of the true optimal ITR within $𝓕$. In particular, we consider the ITRs learned by the nonparametric transfer weights, weighted AIPW estimator ${\widehat{\tau }}_{\text{aw}}^w$ with the logistic regression model ${\pi }_A(\mathit{X};\gamma )$ and the linear $Q$ function. The extensions to other cases with parametric transfer weights and regression and IPW estimators are straightforward.

We introduce more notations. Let $\theta ={\left({\gamma }^{\top },{\beta }^{\top }\right)}^{\top }$ and let the contrast function

$${\tau }_{\text{aw}}^{\theta }(\mathit{X})=\frac{\mathbf{1}(A=1)\{Y-Q(\mathit{X},1;\beta )\}}{{\pi }_A(\mathit{X};\gamma )}+Q(\mathit{X},1;\beta )-\frac{\mathbf{1}(A=0)\{Y-Q(\mathit{X},0;\beta )\}}{1-{\pi }_A(\mathit{X};\gamma )}-Q(\mathit{X},0;\beta ).$$ (14)

Denote the general smooth ramp loss function $l^{\zeta }(u)=l\left({\zeta }_{1}u\right)/{\zeta }_2$, where ${\zeta }_1,{\zeta }_2>0$, $l$ is defined in (8), so it is easy to see that when $\zeta =\left({\zeta }_1,{\zeta }_2\right)\to {\zeta }^{\ast }\triangleq (+\infty ,2)$, the general smooth ramp loss function will converge to the 0–1 loss function, i.e., $l^{\zeta }(u)\to l_{0-1}(u),\forall u$. Note that the general smooth ramp loss function can also be written as the difference of two convex functions, $l^{\zeta }(u)=l_1\left({\zeta }_{1}u\right)/{\zeta }_2-l_0\left({\zeta }_{1}u\right)/{\zeta }_2$, thus the d.c. algorithm in Algorithm 2 can be easily generalized to $l^{\zeta }(u)$. Recalling the loss function $𝓛\left(\tau ;\eta ,l_{0-1}\right)$ and the corresponding $𝓕$-risk ${𝓡}_{𝓕}\left(\tau ;\eta ,l_{0-1}\right)$ defined in (3), we replace $\tau$ with ${\tau }_{\text{aw}}^{\theta }$ and obtain:

$${𝓡}_{𝓕}\left({\tau }_{\text{aw}}^{\theta };\eta ,l_{0-1}\right)=\mathbb{P}\left\{𝓛\left({\tau }_{\text{aw}}^{\theta };\eta ,l_{0-1}\right)\right\},𝓛\left({\tau }_{\text{aw}}^{\theta };\eta ,l_{0-1}\right)=\left|{\tau }_{\text{aw}}^{\theta }(\mathit{X})\right|l_{0-1}\left(\left[2\mathbf{1}\left\{{\tau }_{\text{aw}}^{\theta }(\mathit{X})>0\right\}-1\right]f(\mathit{X};\eta )\right),\forall \eta \in 𝓕,$$

denoted as ${𝓡}_{𝓕}(\eta )$ and $𝓛(\eta ,\theta )$ respectively for simplicity. We define the minimal $𝓕$-risk as ${𝓡}_{𝓕}^{\ast }=\underset{\eta \in 𝓕}{\inf }{𝓡}_{𝓕}(\eta )$. Similarly, replacing $l_{0-1}$ in ${𝓡}_{𝓕}(\eta )$ and $𝓛(\eta ,\theta )$ with the general smooth ramp loss $l^{\zeta }$, we can obtain ${𝓡}_{𝓕}\left({\tau }_{\text{aw}}^{\theta };\eta ,l^{\zeta }\right)$ and $𝓛\left({\tau }_{\text{aw}}^{\theta };\eta ,l^{\zeta }\right)$, denoted as ${𝓡}_{{\zeta }^{𝓕}}(\eta )$ and $L^{\zeta }(\eta ,\theta )$, respectively. Likewise, the minimal $𝓕$-risk with $l^{\zeta }$ is defined as ${𝓡}_{{\zeta }^{𝓕}}^{\ast }=\underset{\eta {\in }_{𝓕}}{\inf }{𝓡}_{{\zeta }^{𝓕}}(\eta )$.

The goal is to show that the decision rule is $𝓕$-consistency (Bartlett et al. 2006), i.e., $\underset{N\to \infty }{\lim }{𝓡}_{𝓕}(\widehat{\eta })={𝓡}_{𝓕}^{\ast }$ Where $\widehat{\eta }={\text{argmin}}_{\eta \in 𝓕}\sum _{i=1}^N{S}_i{\widehat{w}}_i{L}^{\zeta }\left(\eta ,{\widehat{\theta }}_w\right)$ with ${\widehat{\theta }}_w={\left({\widehat{\gamma }}_w^{\top },{\widehat{\beta }}_w^{\top }\right)}^{\top }$. In particular, ${\widehat{\gamma }}_w=\arg {\max }_{\gamma }\sum _{i=1}^N{\widehat{w}}_i{S}_i\left[A_i\text{log}{\pi }_A\left({\mathit{X}}_i;\gamma \right)+\left(1-A_i\right)\text{log}\left\{1-{\pi }_A\left({\mathit{X}}_i;\gamma \right)\right\}\right]$ converges in probability to $\arg {\max }_{\gamma }\mathbb{E}\left[A\text{log}{\pi }_A(\mathit{X};\gamma )+\left(1-A_i\right)\text{log}\left\{1-{\pi }_A(\mathit{X};\gamma )\right\}\right]\stackrel{\text{Δ}}{=}{\gamma }^{\ast },{\widehat{\beta }}_{w,a}={\text{argmin}}_{\beta }\sum _{\left\{i:A_i=a\mid S_i=1\right\}}{\widehat{w}}_i{S}_i{\left(Y_i-{\beta }^{\top }{\mathit{X}}_i\right)}^2$ converges in probability to ${\left\{\mathbb{E}\left(\mathit{X}{\mathit{X}}^{\top }\right)\right\}}^{-1}\times \mathbb{E}\left\{\mathit{X}{Y}^{\ast }(a)\right\}\triangleq {\beta }_a^{\ast }$, for $a=0,1$, and thus ${\widehat{\beta }}_w={\left({\widehat{\beta }}_{w,1}^{\top },{\widehat{\beta }}_{w,0}^{\top }\right)}^{\top }$ converges to ${\beta }^{\ast }={\left({\beta }_1^{\ast \text{T}},{\beta }_0^{\ast \text{T}}\right)}^{\text{T}}$. The convergences of ${\widehat{\gamma }}_w$ and ${\widehat{\beta }}_w$ are implied from the weak law of large numbers and the result ${\max }_i\left|N{\widehat{w}}_i-1/{\pi }_S\left({\mathit{X}}_i\right)\right|=o_p(1)$ proved in Theorem 2 in Wang and Zubizarreta (2020). Let ${\theta }^{\ast }={\left({\gamma }^{\ast \top },{\beta }^{\ast \top }\right)}^{\top }$, thus we have

$${\widehat{\theta }}_w\stackrel{p}{\to }{\theta }^{\ast }.$$ (15)

It is easy to see that ${𝓡}_{{\zeta }^{𝓕}}(\eta )$ will converge to ${𝓡}_{𝓕}(\eta )$ when $\zeta \to {\zeta }^{\ast }$. Therefore, to obtain $𝓕$-consistency, it suffices to show that $\underset{N\to \infty }{\lim }{𝓡}_{{\zeta }^{𝓕}}(\widehat{\eta })={𝓡}_{{\zeta }^{𝓕}}^{\ast }$ which is stated in the theorem under the regularity conditions below. Define $m^{\ast }(\mathit{x})={\left({\rho }^{\prime }\right)}^{-1}[1/\{N\pi (\mathit{x})\}],h(\mathit{x})=\mathbb{E}\left\{L_i^{\zeta }\left(\eta ,{\theta }^{\ast }\right)\mid \mathit{x}\right\}$ and denote ${\pi }_s\left({\mathit{X}}_i\right)$ as ${\pi }_i$ for simplicity. The following regularity conditions follow from Wang and Zubizarreta (2020) to ensure the asymptotic properties of nonparametric weights.

Assumption 3.

a: There exists the limit of $\widehat{\eta }$, denoted as ${\eta }^{\ast }$. b: ${\left|\partial L_i^{\zeta }(\eta ,\theta )/\partial \theta \right|}_{\theta ={\theta }^{\ast }}\left|\le M_1<\infty ,\right|\partial L_i^{\zeta }(\eta ,\theta )/{\partial \eta |}_{\eta ={\eta }^{\ast }}\mid \le M_2<\infty$. c: $\mathbb{P}{L}^{\zeta }\left(\eta ,{\theta }^{\ast }\right)$ is continuous in $\eta$. d: $\mathbb{E}\left|L_i^{\zeta }\left(\eta ,{\theta }^{\ast }\right)-\mathbb{E}\left\{L_i^{\zeta }\left(\eta ,{\theta }^{\ast }\right)\mid \mathit{X}\right\}\right|<\infty$. e: ${\rho }^{\prime }(\cdot )$ is bounded away from zero and Lipschitz. f: $m^{\ast }(\cdot )\in 𝓜$ and $h(\cdot )\in 𝓗$, where $𝓜$ and $𝓗$ are two sets of smooth functions satisfying $\text{log}{N}_{[]}\left\{ϵ,𝓜,L_2(P)\right\}\le c{(1/ϵ)}^{1/k_1}$ and $\text{log}{N}_{[]}\left\{ϵ,𝓗,L_2(P)\right\}\le c{\left(1{/}^{ϵ}\right)}^{1/k_2}$ for a positive constant c and $k_1$, $k_2>1/2$, with $N_{[]}\left\{ϵ,𝒮,L_2(P)\right\}$ denoting the covering number of the set $𝒮$ by $ϵ$-brackets. g: There exist $r_y>1/2$ and ${\lambda }_2^{\ast }$ such that $h(\mathit{X})$ satisfies $\underset{\mathit{X}\in 𝒳}{\sup }\left|h(\mathit{X})-B{(\mathit{X})}^{\top }{\lambda }_2^{\ast }\right|=O\left(K^{-r_y}\right)$. h: $K^{-r_{\pi }-r_y+1/2}=o(1)$. i: ${\Vert \sigma \Vert }^2=o_{\mathbb{P}}(1)$. j: $m^{-1}\sum _{i\in {𝓘}_m}B\left({\mathit{X}}_i\right)-N^{-1}\sum _{i=1}^{N}B\left({\mathit{X}}_i\right)=o_{\mathbb{P}}(1)$.

Theorem 1.

Under Assumptions 2–3, we have $\underset{N\to \infty }{\lim }{𝓡}_{{\zeta }^{𝓕}}(\widehat{\eta })={𝓡}_{{\zeta }^{𝓕}}^{\ast }$.

Our asymptotic results require $n$ and $m$ both to go to infinity. If $m\nrightarrow \infty ,m^{-1}\sum _{i\in {𝓘}_m}g\left({\mathit{X}}_i\right)$ for any function $g(\cdot )$ will not converge to $\mathbb{E}\{g(\mathit{X})\}$. If $n\nrightarrow \infty$, the estimated transferring weights will not converge to the true ones. In either case, the final estimated optimal ITRs will not be consistent for the ones of the target population.

5. Simulation study

We evaluate the finite-sample performances of the proposed estimators with a set of simulation studies. We first generate a target population of size $N={10}^6$. The covariates are generated by ${\mathit{X}}_i={\left(X_{i1},X_{i2}\right)}^{\top }\sim N\left({\mathbf{1}}_{2\times 1},I_{2\times 2}\right)$, and the potential outcome is generated by $Y_i^{\ast }(a)\mid {\mathit{X}}_i=1+2X_{i1}+3X_{i2}+a\tau \left({\mathit{X}}_i\right)+{ϵ}_i(a)$, where ${ϵ}_i(a)$ i.i.d. follows $N\left(0,{0.5}^2\right)$ for $a=0,1$. We consider three contrast functions: (I). $\tau \left({\mathit{X}}_i\right)=\arctan \left\{\exp \left(1+X_{i1}\right)-3X_{i2}-5\right\}$; (II). $\tau \left({\mathit{X}}_i\right)=\cos (1)+\cos \left(X_{i1}\right)+\cos \left(X_{i2}\right)-3/2$; (III). $\tau \left({\mathit{X}}_i\right)=1+2X_{i1}+3X_{i2}$.

To evaluate the necessity and performance of weighting methods, we consider two categories of data-generating models. One is the scenarios where the true optimal policies are not in linear forms, i.e., the settings (I) and (II), while the other one is considering the true optimal policy is in a linear form as the setting (III). We start with the first two scenarios. From the data visualization, we observed that the covariates shifted between the experimental data and RWD to a varying extent in the two generative models: the covariates generated by (I) have similar distributions between the experimental data and RWD, while those generated by (II) have very different ones. Then we generate the RWD by random sample $m=5000$ subjects from the target population ${\left\{{\mathit{X}}_i\right\}}_{i=1}^N$, denoted as ${\left\{{\mathit{X}}_i\right\}}_{i\in {𝓘}_m}$. To form the experimental sample, we generate the indicator of selection according to $S_i\mid {\mathit{X}}_i\sim \text{Bernoul1i}\left[\exp \left({\alpha }_0+{\alpha }_1^{\top }{\mathit{X}}_i\right)/\left\{1+\exp \left({\alpha }_0+{\alpha }_1^{\top }{\mathit{X}}_i\right)\right\}\right]$, where ${\alpha }_1={(1,-2)}^{\top }$ and ${\alpha }_0=-8$ which gives the average sample sizes of the experimental data over the simulation replicates round 1386, respectively. In the experimental data denoted as ${𝓘}_n$, the treatments are randomly assigned according to $A_i\sim \text{Bernoulli}(0.5)$, and then the actual observed outcomes $Y_i=Y_i^{\ast }\left(A_i\right),i\in {𝓘}_n$.

For each setting, we compare the following estimators:

1. “ $W_{\mathit{1}}$ ”: our proposed weighted estimator with parametric weights learned by the modified maximum likelihood introduced in Section 3.2.1.

2. “ $W_{\mathit{2}}$ ”: our proposed weighted estimator with parametric weights learned by the estimating equations introduced in Section 3.2.1.

3. “cv”: our proposed weighted estimator using the cross-validation procedure introduced in Section 3.4, and we set the number of sample splits as 10 in all the simulation studies.

4. “np”: our proposed weighted estimator with nonparametric weights.

5. “unweight”: only using the experimental data to learn the optimal linear ITRs.

6. “bm”: using the weighted estimator but we replace the estimated transfer weights ${\widehat{w}}_i$ with the normalized true weights $w_i={\left\{{\pi }_S\left({\mathit{X}}_i\right)\right\}}^{-1}/\sum _{i\in {𝓘}_n}{\left\{{\pi }_S\left({\mathit{X}}_i\right)\right\}}^{-1}$, and replace the estimated contract functions ${\widehat{\tau }}_i$ with the true contrast values $Y_i^{\ast }(1)-Y_i^{\ast }(0)$.

7. “Imai”: the method proposed in Imai et al. (2013). They construct the transfer weights by fitting a Bayesian additive regression trees (BART) model using covariates as predictors and whether to be in the experimental data as outcomes, and then use a variant of SVM to estimate the conditional treatment effect (implemented in the R package FindIt). Then given $\mathit{X}$, we can assign treatment 1 if its estimated conditional treatment effect is positive, and 0 otherwise.

8. 8. “DBN”: the method proposed in Mo et al. (2020), where they propose to use distributionally robust ITRs, that is, value functions are evaluated under all testing covariate distributions that are ”close” to the training distribution, and the worse-case takes a minimal one, and then the ITRs are learned by maximizing the defined worst-case value function.

For the estimators 1. – 5. in the above, we estimate the contrast function using ${\tau }_{\text{aw}}$ with linear $Q$ functions and constant propensity scores estimated via the proportion of $A=1$ in the experimental data, $n^{-1}\sum _{i\in {𝓘}_n}{A}_i$.

To study the impact of model misspecification of ${\pi }_S(\mathit{X})$, we consider to estimate $\alpha$ by fitting logistic regression wrongly: ${\pi }_s(\mathit{X};\alpha )=\exp \left({\alpha }_0+{\alpha }_1^{\top }\mathit{X}\right)/\left\{1+\exp \left({\alpha }_0+{\alpha }_1^{\top }\mathit{X}\right)\right\}$, where $\mathit{X}=\left({\mathit{X}}_1^2,{\mathit{X}}_2^2\right)$. And as described in the proposed methods, only estimators 1. – 3. need to specify the sampling model. To measure the performance of the different approaches, we use the mean value MSE defined as $N^{-1}\sum _{i=1}^N{\left(Y_i^{\ast }\left\{d_{\eta }\left({\mathit{X}}_i\right)\right\}-Y_i^{\ast }\left[\mathbf{1}\left\{\tau \left({\mathit{X}}_i\right)>0\right\}\right]\right)}^2$. Thus the lower the MSE, the better performance the approach has.

The results are summarized in Figure 2. For the generative model (I) shown in Figure 2, unweighted and DBN rules perform worst in all the cases. The second parametric weighting method is relatively robust compared with the first parametric one when the sampling model is misspecified; this is reasonable since the first one uses maximum likelihood estimation which is more reliable on the correctness of the parametric model specification. When the training data and testing data are similarly distributed, the performances of the parametric, nonparametric weighting methods and the cross-validation procedure are similar; in this case, sampling model misspecification does not have much influence on the rule learning.

Fig. 2.

Value MSE boxplots of different methods for the generative model (I) and (II). The red point in each boxplot is the mean value MSE averaged over 200 Monte Carlo replicates.

For the generative model (II) shown in Figure 2, among the performances of weighted and unweighted estimators, we can see the weighted estimators outperform the unweighted estimators when the model correctly specified; the nonparametric weighted estimator outperforms other estimators and the performance of the parametric weighted estimators has dramatically decreased when the sampling model was misspecified. These findings make sense, because when the more different the training and testing data are, the more crucial the bias correction is. Thus sampling model misspecification might have a huge influence on the learned rules.

From both generative models, we can see that the nonparametric weighting and cross-validation procedure are more robust when the sampling model is unknown and recommended in practice use. As for DBN estimator, it performs badly in terms of MSE magnitude or variability, one possible reason might be that the tuning parameters when measuring the ”closeness” of two distributions need to be chosen more carefully in practice (Mo et al. 2020). As for the comparison Imai also has relatively good performance among all estimators and has relatively small variability.

As for setting (III), we consider the correctly specified sample score model. As shown in Figure 3, the unweighted method performs the best, close to the benchmark. This demonstrates that when the true optimal policies are in linear forms, then learning with only the experimental data performs the best. Thus weighting is not a necessity, and it even will induce the variance of the estimators. The other competitive estimator is using cross-validation. Therefore, when we have no idea of the true data generating models, using the CV is a good choice in practice.

Fig. 3.

Value MSE boxplots of different methods for the generative model (III). The red point in each boxplot is the mean value MSE averaged over 200 Monte Carlo replicates.

6. Real Data Application

We apply the proposed methods to personalized recommendations of a job training program on post-market earning. Our analysis is based on two data sources (downloaded from https://users.nber.org/~rdehejia/data/.nswdata2.html): an experimental dataset from the National Supported Work (NSW) program (LaLonde 1986) and a non-experimental dataset from the Current Population Survey (CPS; LaLonde (1986)). The NSW dataset includes 185 individuals who received the job training program $(A=1)$ and 260 individuals who did not receive the job training program $(A=0)$, and the CPS dataset includes 15992 individuals without the job training program. Let $\mathit{X}$ be a vector including an intercept 1 and the pre-intervention covariates: age, education, Black (1 if black, 0 otherwise), Hispanic (1 if Hispanic, 0 otherwise), married (1 if married, 0 otherwise), whether having high school degree (1 if no degree, 0 otherwise), earning in 1974, and earning in 1975. The outcome variable is the earning in 1978. We transfer the earnings in the year 1974, 1975, and 1978 by taking the logarithm of the earnings plus one, denoted as “log.Re74”, “log.Re75” and “log.Re78”. We aim to learn a linear rule $d(\mathit{X};\eta )=\mathbf{1}\left({\mathit{X}}^{\top }\eta >0\right)$ tailoring the recommendation of the job training program to an individual with characteristics $\mathit{X}$.

The CPS sample is generally representative of the target population; while the participants in the NSW program may not represent the target population. Table 1 contrasts sample averages of each pre-intervention covariate in the NSW and CPS samples and shows that the covariates are highly imbalanced between the two samples. On average, the NSW sample is younger, less educated, more black people, fewer married, more without high school degree, fewer earnings in 1974 and 1975. Because of imbalance, ITRs cannot be learned based only on the NSW sample, thus we are motivated to use the proposed weighted transfer learning approaches to learn optimal linear ITRs which benefit the target population most.

Table 1.

Sample averages of pre-intervention variables in the NSW and CPS samples and results from the weighted transfer and unweighted learning methods: The first two rows are sample averages of each pre-intervention covariate in the NSW and CPS samples, respectively. The last two rows are estimated linear coefficients of each corresponding covariate in the linear decision rule from the weighted transfer and unweighted learning methods after normalizing the covariates, respectively.

	intercept	Age	Educ	Black	Hisp	Married	Nodegr	log.Re74	log.Re75
Experiment NSW	1	25.37	10.20	0.83	0.09	0.17	0.78	2.27	2.70
Non-experiment CPS	1	33.23	12.03	0.07	0.07	0.71	0.30	8.23	8.29
	${\eta }_0$	${\eta }_1$	${\eta }_2$	${\eta }_3$	${\eta }_4$	${\eta }_5$	${\eta }_6$	${\eta }_7$	${\eta }_8$
Unweighted learning ${\widehat{\eta }}_{\text{u}}$	1	0.11	0.16	−0.01	0.04	0.67	−0.67	−3.20	2.53
Weighted transfer learning ${\widehat{\eta }}_{\text{w}}$	1	−0.62	−0.21	0.13	0.10	0.52	−0.85	−5.14	4.02

We consider the unweighted and weighted transfer learning methods. For the weighted transfer learning method, we consider the nonparametric weighting approach, because the parametric approach requires the target population size $N$ which is unknown in this application, and use the AIPW estimator ${\widehat{\tau }}_{\text{aw}}$ with linear Q functions. We first normalize all the covariates in the combined two data sets for a better learning process. We also implement the cross-validation procedure to choose between the unweighted and weighted transfer learning methods.

Table 1 presents the estimated linear coefficients of each corresponding variable in the linear decision rule from the weighted transfer and unweighted learning methods, respectively. Moreover, from the cross-validation procedure, the weighted transfer learning method produces a smaller population risk than the unweighted learning method and therefore is chosen as the final result. Compared with the learned decision rule with only the experiment data NSW which oversamples the low-income individuals, younger, less educational or black individuals are more likely to have higher outcomes if offered the job training program in the target population.

To assess the performances of the learned rules for the target population, following Kallus (2017), we create a test sample that is representative of the target population. We sample 100 individuals at random from the CPS sample to form a control group and then find their closest matches in the NSW treated sample to form the treatment group (use pairmatch in R package optmatch). The test sample thus contains the 100 matched pairs, mimicking a experimental sample where binary actions are randomly assigned with equal probabilities. We use random forest on the test data to estimate $Q$ function $\mathbb{E}(Y\mid \mathit{X},a)$, denoted the estimator as ${\widehat{Q}}_{\text{test}}(\cdot ,a)$, $a=0,1$, then we can use $\sum _{i\in \text{CPS}}{\widehat{Q}}_{\text{test}}\left\{{\mathit{X}}_i,d\left({\mathit{X}}_i;\eta \right)\right\}/|\text{CPS}|$ to estimate the value of an ITR $d(\cdot ;\eta )$, where $\left|\text{CPS}\right|$ is the sample size of CPS. We replicate this sampling test data procedure 100 times, estimate values of the three ITRs for each replicate, and get the averaged estimated values (standard error) as follows: 6.73 (0.01) and 6.39 (0.01) for $d\left(\cdot ;{\widehat{\eta }}_w\right)$ and $d\left(\cdot ;{\widehat{\eta }}_u\right)$, thus the transfer weighted rule has the higher value estimate. Although this evaluation method needs assumption of no unmeasured confounders for the non-experimental data CPS, which is not needed in our learning procedure, it can still bring us confidence, to some degree, that the weighted linear rule has better generalizability to the target population.

7. Discussion

We develop a general framework to learn the optimal interpretable ITRs for the target population by leveraging no unmeasured confounding in the experimental data and the representativeness of covariates in the RWD. The proposed procedure is easy to implement. By constructing transfer weights to correct the distribution of covariates in the experimental data, we can learn the optimal interpretable ITRs for the target population. Besides, we provide a data-adaptive procedure to choose among different weighting methods by modified cross-validation. Moreover, we establish the theoretical guarantee for the consistency of the proposed estimator of ITRs.

For data integration problems, there are generally two study designs: nested and non-nested ones (Dahabreh et al. 2019). For the nested designs, the experimental data is a subsample from the RWD (e.g., comprehensive cohort studies or when the study is nested in a larger healthcare system that forms the target population), while in the non-nested designs, the experimental data and the RWD are separate (e.g., different geographic locations) (see Dahabreh et al. 2021; Steingrimsson et al. 2021; Lee et al. 2021, for details). Although our simulation and data application consider non-nested studies, our proposed method is equally applicable to nested studies. The key assumption is that the domain of covariates in the target and experimental populations is the same. In the nested designs, the target population becomes the population that the RWD represents, which can be different from the experimental population.

Some future directions worth considering. One is to quantify the uncertainty of the estimated ITRs and therefore to conduct statistical inference. Moreover, in some scenarios, beside the covariate information, comparable treatment and outcome information are also available in the RWD. Unlike the randomized experiment, the treatment assignment in the RWD is determined by the preference of physicians and patients. In this case, an interesting topic is to develop integrative methods that can leverage the strengths of both data sources for efficient estimation of the ITRs. Besides, we only consider binary treatments in this paper to illustrate the proposed framework. However, it can be extended to the case with multiple treatments. To do this, we can still use the same transferring weights and apply them to the value function for the multiple treatments (Zhang et al.; 2012). The additional difficulty to overcome is how to optimize this nonsmooth objective function. One possible direction is to reformulate the problem as a weighted multi-classification problem, which can be an interesting future research topic. Finally, Assumption 2.a is plausible if $\mathit{X}$ captures all variables that related to the experiment participation and outcome (Buchanan et al. 2018) or the sampling mechanisms of the study samples depend only on $\mathit{X}$. Similar to the no unmeasured confounders assumption in causal inference from observational studies, this assumption is generally not testable empirically but needs domain knowledge to evaluate its plausibility. In the future, we will develop sensitivity analysis to assess the robustness of the study conclusion to the unverifiable assumption.