Estimation and Optimization of Composite Outcomes

Abstract

There is tremendous interest in precision medicine as a means to improve patient outcomes by tailoring treatment to individual characteristics. An individualized treatment rule formalizes precision medicine as a map from patient information to a recommended treatment. A treatment rule is defined to be optimal if it maximizes the mean of a scalar outcome in a population of interest, e.g., symptom reduction. However, clinical and intervention scientists often seek to balance multiple and possibly competing outcomes, e.g., symptom reduction and the risk of an adverse event. One approach to precision medicine in this setting is to elicit a composite outcome which balances all competing outcomes; unfortunately, eliciting a composite outcome directly from patients is difficult without a high-quality instrument, and an expert-derived composite outcome may not account for heterogeneity in patient preferences. We propose a new paradigm for the study of precision medicine using observational data that relies solely on the assumption that clinicians are approximately (i.e., imperfectly) making decisions to maximize individual patient utility. Estimated composite outcomes are subsequently used to construct an estimator of an individualized treatment rule which maximizes the mean of patient-specific composite outcomes. The estimated composite outcomes and estimated optimal individualized treatment rule provide new insights into patient preference heterogeneity, clinician behavior, and the value of precision medicine in a given domain. We derive inference procedures for the proposed estimators under mild conditions and demonstrate their finite sample performance through a suite of simulation experiments and an illustrative application to data from a study of bipolar depression.

1. Introduction

Precision medicine is an approach to healthcare that involves tailoring treatment based on individual patient characteristics (Hamburg and Collins, 2010; Collins and Varmus, 2015). Accounting for heterogeneity by tailoring treatment has the potential to improve patient outcomes in many therapeutic areas. An individualized treatment rule formalizes precision medicine as a map from the space of patient covariates into the space of allowable treatments (Murphy, 2003; Robins, 2004). Almost all methods for estimating individualized treatment rules have been designed to optimize a scalar outcome (exceptions will be discussed shortly). However, in practice, clinical decision making often requires balancing trade-offs between multiple outcomes. For example, clinicians treating patients with bipolar disorder must manage both depression and mania. Antidepressants may help correct depressive episodes but may also induce manic episodes (Sachs et al., 2007; Ghaemi, 2008; Goldberg, 2008; Wu et al., 2015). We propose a novel framework for using observational data to estimate a composite outcome and the corresponding optimal individualized treatment rule.

The estimation of optimal individualized treatment rules has been studied extensively, leading to a wide range of estimators designed to suit an array of data structures and data-generating processes (Kosorok and Laber, 2019; Tsiatis et al., 2020). These estimators include: regression-based methods like Q-learning (Murphy, 2005; Qian and Murphy, 2011; Schulte et al., 2014; Laber et al., 2014a), A-learning (Murphy, 2003; Robins, 2004; Blatt et al., 2004; Moodie et al., 2007; Wallace and Moodie, 2015), and regret regression (Henderson et al., 2010); direct-search methods (Rubin and van der Laan, 2012; Zhang et al., 2012b; Zhao et al., 2012; Zhang et al., 2013; Zhou et al., 2017) based on inverse probability weighting (Robins, 1999; Murphy et al., 2001; van der Laan and Petersen, 2007; Robins et al., 2008); and hybrid methods (Taylor et al., 2015; Zhang et al., 2018). The preceding methods require specification of a single scalar outcome that will be used to define an optimal regime; were individual patient utilities known, then they could be used as the outcome in any of these methods. However, in general such utilities are not known though they can be elicited provided a high-quality instrument is available (Butler et al., 2018); in the absence of such an instrument, preference elicitation is difficult to apply. A method for constructing a composite utility that is best predicted using a non-parametric machine learning model is proposed by (Benkeser et al., 2020); howeer, they do not consider heterogeneous utilities or the construction of precision medicine strategies.¹

We propose a new paradigm for estimating optimal individualized treatment rules from observational data without eliciting patient preferences. The key premise is that clinicians are attempting to act optimally with respect to each patient’s utility and thus the observed treatment decisions contain information about individual patient utilities. This idea is similar to that introduced by Wallace et al. (2018) (see also Wallace et al., 2016); however, we provide an estimator for the probability that a patient is treated optimally, rather than assuming that all patients are treated optimally. We construct estimators of individual patient utilities which do not require that clinicians are acting optimally, only that they approximately follow an optimal policy. This approach allows us to describe the goals of the decision maker and how these goals vary across patients, determine what makes a patient more or less likely to be treated optimally under standard care, and estimate the decision rule which optimizes patient-specific composite outcomes. We develop this approach in the context of a single-stage, binary decision in the presence of two outcomes. An extension to the setting with more than two outcomes is discussed in the Appendix.

Other methods for estimating optimal treatment rules in presence of multiple outcomes include using an expert-derived composite outcome for all patients (Thall et al., 2002, 2007; Murray et al., 2016; Moser et al., 2020). However, this does not account for differences in the utility function across patients and in some cases it may not be possible to elicit a high-quality composite outcome from an expert. Alternatively, multiple outcomes can be incorporated using set-valued treatment regimes (Laber et al., 2014b; Lizotte and Laber, 2016; Wu, 2016), constrained optimization (Linn et al., 2015; Laber et al., 2018; Wang et al., 2018), or inverse preference elicitation (Lizotte et al., 2012). Schnell et al. (2017) extend methods for estimating the benefiting subgroup to the case of multiple outcomes using the concept of admissibility (see also Schnell et al., 2016). However, none of these approaches provide a method for estimating an individual patient’s utility.

This work is closely related to inverse reinforcement learning (Kalman, 1964; Ng et al., 2000; Abbeel and Ng, 2004; Ratliff et al., 2006), which involves studying decisions made by an expert and constructing the utility function that is optimized by the expert’s decisions. Inverse reinforcement learning has been successfully applied in navigation (Ziebart et al., 2008) and human locomotion (Mombaur et al., 2009). Inverse reinforcement learning methods assume that decisions are made in a single environment. However, in the context of precision medicine, both the utility function and the probability of optimal treatment may vary across patients. Our approach is a version of inverse reinforcement learning with multiple environments.

This work is also related to the notion of stated and revealed preferences in the health economics literature.² Viewed through this lens, our work might be characterized as using clinical decisions as a kind of surrogate for patient revealed preferences thereby avoiding the need for the elicitation of stated preferences using specialized instruments. This is advantageous as the construction of high-quality instruments is difficult and collection of preference information is not routine in many areas (Carlsson and Martinsson, 2003; Ryan et al., 2007; de Bekker-Grob et al., 2012; Soekhai et al., 2019); though see Butler et al. (2018) for an illustrative application when such an instrument is available. Challenges associated with preference elicitation for precision medicine are discussed in Laber et al. (2014b), Lizotte and Laber (2016).

In Section 2, we introduce a pseudo-likelihood method to estimate patient utility functions from observational data. In Section 3, we state a number of theoretical results pertaining to the proposed method, including consistency and inference for the maximum pseudo-likelihood estimators. Section 4 presents a series of simulation experiments used to evaluate the finite sample performance of the proposed methods. Section 5 presents an illustrative application using data from the STEP-BD bipolar disorder study. Conclusions and a discussion of future research are given in Section 6. Proofs are given in the appendix along with additional simulation results and a discussion of an extension to more than two outcomes.

2. Pseudo-likelihood Estimation of Utility Functions

Assume the available data are (X_i, A_i, Y_i, Z_i), i = 1, …, n, which comprise n independent and identically distributed copies of (X, A, Y, Z), where $X\in \mathcal{X}\subseteq {ℝ}^p$ are patient covariates, $A\in \mathcal{A}=\{-1,1\}$ is a binary treatment, and Y and Z are two real-valued outcomes for which higher values are more desirable. The extension to scenarios with more than two outcomes is discussed in the Appendix. An individualized treatment rule is a function $d:\mathcal{X}\to \mathcal{A}$ such that, under d, a patient presenting with covariates X = x will be assigned to treatment d(x). Let Y*(a) denote the potential outcome under treatment $a\in \mathcal{A}$, and for any regime d, define $Y^{*}(d)={\sum }_{a\in \mathcal{A}}{Y}^{*}(a)1\{d(X)=a\}$. An optimal regime for the outcome Y, say $d_Y^{\text{opt}}$, satisfies $\mathbb{E}{Y}^{*}\left(d_Y^{\text{opt }}\right)\ge \mathbb{E}{Y}^{*}(d)$ for any other regime d. The optimal regime for the outcome Z, say $d_Z^{\text{opt}}$, is defined analogously. In order to identify these optimal regimes, and subsequently to identify the optimal regime across the class of utility functions introduced below, we make the following assumptions.

Assumption 1 Consistency, Y = Y*(A) and Z = Z*(A).

Assumption 2 Positivity, Pr(A = a|X = x) ≥ c > 0 for some constant c and all pairs $(x,a)\in \mathcal{X}\times \mathcal{A}$.

Assumption 3 Ignorability, {Y*(−1), Y*(1)}⊥A|X and {Z*(−1), Z*(1)}⊥A|X.

In addition we assume that there is no interference between units nor are the multiple versions of treatment (Rubin, 1980). These assumptions are standard in causal inference (Robins, 2004; Hernan and Robins, 2010). Assumption 3 is not empirically verifiable in observational studies (Rosenbaum and Rubin, 1983; Rosenbaum, 1984).

Define $Q_Y(x,a)=\mathbb{E}(Y|X=x,A=a)$. Then, under the preceding assumptions, it can be shown that $d_Y^{\text{opt}}(x)=\text{arg\hspace{0.17em}}{\text{max}}_{a\in \mathcal{A}}{Q}_Y(x,a)$ (Zhang et al., 2012b; Qian and Murphy, 2011). Similarly, it follows that $d_Z^{\text{opt }}(x)=\text{arg\hspace{0.17em}}{\text{max}}_{a\in \mathcal{A}}{Q}_Z(x,a)$ where $Q_Z(x,a)=\mathbb{E}(Z|X=x,A=a)$. In general, $d_Y^{\text{opt}}(x)$ need not equal $d_Z^{\text{opt}}(x)$; therefore, if both Y and Z are clinically relevant, neither $d_Y^{\text{opt}}$ nor $d_Z^{\text{opt}}$ may be acceptable. We assume that there exists an unknown and possibly covariate-dependent utility U = u(Y, Z), where $u:{ℝ}^2\to ℝ$ measures the “goodness” of the outcome pair (y, z). The optimal regime with respect to U, say $d_U^{\text{opt}}$, satisfies $\mathbb{E}{U}^{*}\left(d_U^{\text{opt}}\right)=\mathbb{E}u\left\{Y^{*}\left(d_U^{\text{opt}}\right),Z^{*}\left(d_U^{\text{opt}}\right)\right\}\ge \mathbb{E}u\left\{Y^{*}(d),Z^{*}(d)\right\}=\mathbb{E}{U}^{*}(d)$ for any other regime d. The goal is to use the observed data to estimate the utility and subsequently $d_U^{\text{opt }}$. Define $Q_U(x,a)=\mathbb{E}(U|X=x,A=a)$. For the class of utility functions we consider below, Q_U(x, a) is a (possibly covariate-dependent) convex combination of Q_Y (x, a) and Q_Z(x, a) and is therefore identifiable under the stated causal assumptions and furthermore $d_U^{\text{opt}}(\mathit{x})=\text{arg\hspace{0.17em}}{\text{max}}_{a\in \mathcal{A}}{Q}_U(\mathit{x},a)$.

We assume that clinicians act with the goal of optimizing each patient’s utility and that their success in identifying the optimal treatment depends on individual patient characteristics. Therefore, we assume that the clinicians are approximately, i.e., imperfectly, assigning treatment according to $d_U^{\text{opt}}(x)$. If the clinician were always able to correctly identify the optimal treatment and assign $A=d_U^{\text{opt}}(X)$ for each patient, there would be no need to estimate the optimal treatment policy (Wallace et al., 2016). Instead, we assume that the decisions of the clinician are imperfect and that $\text{Pr}\left\{A=d_U^{\text{opt}}(\mathit{x})|X=\mathit{x}\right\}=\text{expit}\left(x^{\top }\beta \right)$ where β is an unknown parameter. We show in Section 2.2 that the model is identifiable under mild conditions; e.g., these exclude the possibility of a malevolent clinician that is systematically assigning poor treatments. We implicitly assume throughout that X may contain higher order terms, interactions, or basis functions constructed from patient covariates.

2.1. Fixed Utility

We begin by assuming that the utility function is constant across patients and takes the form u(y, z; ω) = ωy + (1 − ω)z for some ω ∈ [0, 1]. Lemma 1 of Butler et al. (2018) states that, for a broad class of utility functions, the optimal individualized treatment rule is equivalent to the optimal rule for a utility function of this form. Define Q_ω(x, a) = ωQ_Y (x, a) + (1 − ω)Q_Z(x, a) and define $d_{\omega }^{\text{opt}}(x)=\text{arg\hspace{0.17em}}{\text{max}}_{a\in \mathcal{A}}{Q}_{\omega }(x,a)$. Let ${\widehat{Q}}_{Y,n}$ and ${\widehat{Q}}_{Z,n}$ denote estimators of Q_Y and Q_Z obtained from regression models fit to the observed data (Qian and Murphy, 2011). For a fixed value of ω, let ${\widehat{Q}}_{\omega ,n}(x,a)=\omega {\widehat{Q}}_{Y,n}(x,a)+(1-\omega ){\widehat{Q}}_{Z,n}(x,a)$ and subsequently let ${\widehat{d}}_{\omega ,n}(x)=\text{arg\hspace{0.17em}}{\text{max}}_{a\in \mathcal{A}}{\widehat{Q}}_{\omega ,n}(x,a)$ be the plug-in estimator of $d_{\omega }^{\text{opt}}(x)$. Given ${\widehat{Q}}_{Y,n}$ and ${\widehat{Q}}_{Z,n}$, ${\widehat{d}}_{\omega ,n}(x)$ can be computed for each ω ∈ [0, 1].

The joint distribution of (X, A, Y, Z) is

$$\begin{gathered} f(\mathit{X},A,Y,Z)=f(Y,Z|\mathit{X},A)f(A|\mathit{X})f(\mathit{X}) \\ =f(Y,Z|\mathit{X},A)f(\mathit{X})\frac{\text{exp}\left[{\mathit{X}}^{\top }\beta 1\left\{A=d_{\omega }^{\text{opt}}(\mathit{X})\right\}\right]}{1+\text{exp}\left({\mathit{X}}^{\top }\beta \right)}. \end{gathered}$$

Assuming that f(Y, Z|X, A) and f(X) do not depend on ω or β, the likelihood for (ω, β) is

$${\mathcal{L}}_n(\omega ,\beta )\propto \prod _{i=1}^n\frac{\text{exp}\left[X_i^{\top }\beta 1\left\{A_i=d_{\omega }^{\text{opt}}\left(X_i\right)\right\}\right]}{1+\text{exp}\left(X_i^{\top }\beta \right)},$$ (1)

which depends on the unknown function $d_{\omega }^{\text{opt }}$. Plugging in ${\widehat{d}}_{\omega ,n}$ for $d_{\omega }^{\text{opt }}$ into (7) yields the pseudo-likelihood

$${\widehat{\mathcal{L}}}_n(\omega ,\beta )\propto \prod _{i=1}^n\frac{\text{exp}\left[X_i^{\top }\beta 1\left\{A_i={\widehat{d}}_{\omega ,n}\left(X_i\right)\right\}\right]}{1+\text{exp}\left(X_i^{\top }\beta \right)}.$$ (2)

If we let ${\widehat{\omega }}_n$ and ${\widehat{\beta }}_n$ denote the maximum pseudo-likelihood estimators obtained by maximizing (2), then an estimator of the utility function is ${\widehat{u}}_n(y,z)=u\left(y,z;{\widehat{\omega }}_n\right)={\widehat{\omega }}_{n}y+\left(1-{\widehat{\omega }}_n\right)z$ and expit $\left(x^{\top }{\widehat{\beta }}_n\right)$ is an estimator of the probability that a patient presenting with covariates x would be treated optimally under standard care. An estimator of the optimal policy at x is ${\widehat{d}}_{{\widehat{\omega }}_n,n}(x)=\text{arg\hspace{0.17em}}{\text{max}}_{a\in \mathcal{A}}{\widehat{Q}}_{{\widehat{\omega }}_n,n}(x,a)$.

Because the pseudo-likelihood given in (2) is non-smooth in ω, standard gradient-based optimization algorithms cannot be used. However, for a given ω, it is straightforward to compute the profile estimator ${\widehat{\beta }}_n(\omega )=\text{arg\hspace{0.17em}}{\text{max}}_{\beta \in {ℝ}^p}{\widehat{\mathcal{L}}}_n(\omega ,\beta )$. We can compute the profile pseudo-likelihood estimator over a grid of values for ω and select the point on the grid yielding the largest pseudo-likelihood. The algorithm to construct (${\widehat{\omega }}_n$, ${\widehat{\beta }}_n$) is given in Algorithm 1 below. Step (3) can be accomplished using logistic regression. The theoretical properties of this estimator are discussed in Section 3.

2.2. Patient-specific Utility

Outcome preferences can vary widely across patients in some application domains, including schizophrenia (Kinter, 2009; Strauss et al., 2010) and pain management (Gan et al., 2004). To accommodate this setting, we assume that the utility function takes the form u(y, z; x, ω) = ω(x)y + {1 − ω(x)}z where $\omega :\mathcal{X}\to [0,1]$ is a smooth function. For illustration, we let ω(x; θ) = expit(x^⊤θ) where θ is an unknown parameter. Misspecified utility models are discussed in the Appendix. Define Q_θ(x, a) = ω(x; θ)Q_Y (x, a) + {1 − ω(x; θ)}Q_Z(x, a) and define $d_{\theta }^{\text{opt}}(x)=\text{arg\hspace{0.17em}}{\text{max}}_{a\in \mathcal{A}}{Q}_{\theta }(x,a)$. Let ${\widehat{Q}}_{Y,n}$ and ${\widehat{Q}}_{Z,n}$ denote estimators of Q_Y and Q_Z obtained from regression models fit to the observed data. For a fixed value of θ, let ${\widehat{Q}}_{\theta ,n}(x,a)=\omega (x;\theta ){\widehat{Q}}_{Y,n}(x,a)+\{1-\omega (x;\theta )\}{\widehat{Q}}_{Z,n}(x,a)$ and subsequently let ${\widehat{d}}_{\theta ,n}(x)=\text{arg\hspace{0.17em}}{\text{max}}_{a\in \mathcal{A}}{\widehat{Q}}_{\theta ,n}(x,a)$ be the plug-in estimator of $d_{\theta }^{\text{opt}}(x)$. Assume that decisions are made according to the model $\text{Pr}\left\{A=d_{\theta }^{\text{opt}}(x)|X=x\right\}=\text{expit}\left(x^{\top }\beta \right)$. We compute the estimators (${\widehat{\theta }}_n$, ${\widehat{\beta }}_n$) of (θ, β) by maximizing the pseudo-likelihood

$${\widehat{\mathcal{L}}}_n(\theta ,\beta )\propto \prod _{i=1}^n\frac{\text{exp}\left[X_i^{\top }\beta 1\left\{A_i={\widehat{d}}_{\theta ,n}\left(X_i\right)\right\}\right]}{1+\text{exp}\left(X_i^{\top }\beta \right)}.$$ (3)

An estimator for the utility function is ${\widehat{u}}_n(y,z;x)=\omega \left(x;{\widehat{\theta }}_n\right)y+\left\{1-\omega \left(x;{\widehat{\theta }}_n\right)\right\}z$ and an estimator for the optimal decision function is ${\widehat{d}}_{{\widehat{\theta }}_n,n}$. The model, as stated is not identifiable. However, we show below that it is identifiable under the following conditions.

Assumption 4 The following conditions hold.

1. $\beta \in \mathcal{B}\subset {ℝ}^p$ and $\theta \in \Theta \subset {ℝ}^q$, where $\mathcal{B}$ and Θ are compact.

2. β₀ ≠ 0.

3. $\mathcal{X}$ is bounded ($\mathit{X}\in \mathcal{X}\subset {ℝ}^p$ a.s.).

4. Let ${\mathcal{X}}_S$ be the collection of subsets of $\mathcal{X}$ consisting of sets of the form $\left\{\mathit{x}\in \mathcal{X}:d_{\theta }\left(x\right)\ne d_{{\theta }_0}\left(x\right)\right\}$ for θ ∈ Θ \ {θ₀}, together with the complements of these sets. Then:

   1. For all $X_S\in {\mathcal{X}}_S$, 0 < Pr(X ∈ X_S) < 1, and
   2. E (XX^T/X ∈ X_S) is full rank $\forall X_S\in {\mathcal{X}}_S$.

Theorem 5 (Identifiability) Under Assumption 4, (θ₀, β₀) is uniquely identified under the model given by ${\mathcal{L}}_n(\theta ,\beta )$.

Remark 6 A less technical but sufficient condition is to assume that (β₀, θ₀) satisfies $\text{Pr}\left\{A=d_{{\theta }_0}\underset{\_}{(}X)|X\right\}>1/2$ almost surely, i.e., that clinical decisions are always better than a coin toss. A proof of sufficiency is given in the Appendix.

As before, the pseudo-likelihood given in (3) is non-smooth in θ and standard gradient-based optimization methods cannot be used. It is again straightforward to compute the profile pseudo-likelihood estimator ${\widehat{\beta }}_n(\theta )=\text{arg\hspace{0.17em}}{\text{max}}_{\beta \in {ℝ}^p}{\widehat{\mathcal{L}}}_n(\theta ,\beta )$ for any $\theta \in {ℝ}^p$. However, because it is computationally infeasible to compute ${\widehat{\beta }}_n(\theta )$ for all θ on a grid if θ is of moderate dimension, we generate a random walk through the parameter space using the Metropolis algorithm as implemented in the metrop function in the R package mcmc (Geyer and Johnson, 2017) and compute the profile pseudo-likelihood for each θ on the random walk. Let ${\tilde{\mathcal{L}}}_n(\theta )={\text{max}}_{\beta \in {ℝ}^p}{\widehat{\mathcal{L}}}_n(\theta ,\beta )$. We can compute ${\tilde{\mathcal{L}}}_n(\theta )={\widehat{\mathcal{L}}}_n\left\{\theta ,{\widehat{\beta }}_n(\theta )\right\}$ by estimating ${\widehat{\beta }}_n(\theta )$ using logistic regression as described in Section 2.1. The algorithm to construct a random walk through the parameter space is given in Algorithm 2 below. After generating a chain (θ¹, …, θ^B), we select the θ^k that leads to the largest value of ${\tilde{\mathcal{L}}}_n\left({\theta }^k\right)$ as the maximum pseudo-likelihood estimator. Standard practice is to choose the variance of the proposal distribution, σ², so that the acceptance proportion is between 0.25 and 0.5 (Geyer and Johnson, 2017).

3. Theoretical Results

Here we state a number of theoretical results pertaining to the proposed pseudo-likelihood estimation method for utility functions. We state results for a patient-specific utility function; the setting where the utility function is fixed is a special case. All proofs are deferred to the Appendix.

We assume that $\text{Pr}\left\{A=d_U^{\text{opt}}(x)|X=x\right\}=\text{expit}\left(x^{\top }{\beta }_0\right)$ and that the true utility function is u(y, z; x, θ₀) = ω (X; θ₀)y + {1 − ω (X; θ₀)}z, where ω(X; θ) has bounded continuous derivative on compact sets and $d_{{\theta }_0}^{\text{opt}}(X)=d_{\theta }^{\text{opt}}(X)$ almost surely implies θ = θ₀, i.e., the model introduced in Section 2.2 is well-defined and correctly specified with true parameters ${\beta }_0\in {ℝ}^p$ and ${\theta }_0\in {ℝ}^d$. We further assume that the estimators ${\widehat{Q}}_{Y,n}(x,a)$ and ${\widehat{Q}}_{Z,n}(x,a)$ are pointwise consistent for all ordered pairs (x, a). Along with Assumptions 1–3, we implicitly assume that the densities f(Y, Z|X, A) and f(X) exist. The following result states the consistency of the maximum pseudo-likelihood estimators for the utility function and the probability of optimal treatment. The proof involves verifying the conditions of Theorem 2.12 of Kosorok (2008).

Theorem 7 (Consistency with patient-specific utility) Let the maximum pseudo- likelihood estimators be as in Section 2.2, $\left({\widehat{\theta }}_n,{\widehat{\beta }}_n\right)=\text{arg\hspace{0.17em}}{\text{max}}_{\theta \in {ℝ}^p,\beta \in \mathcal{B}}{\widehat{\mathcal{L}}}_n(\theta ,\beta )$. Assume that $\mathcal{B}$ is a compact set with ${\beta }_0\in \mathcal{B}$ and that $\Vert \mathbb{E}X\Vert <\infty$ Then, $\Vert {\widehat{\theta }}_n-{\theta }_0\Vert \stackrel{P}{\to }0$ and $\Vert {\widehat{\beta }}_n-{\beta }_0\Vert \stackrel{P}{\to }0 as n\to \infty$, where ∥ · ∥ is the Euclidean norm.

Let $V_{\theta }(d)=\mathbb{E}\{u(Y,Z;X,\theta )|A=d(X)\}$ be the mean composite outcome in a population where decisions are made according to d. The following result establishes the consistency of the value of the estimated optimal policy. The proof uses general theory developed by Qian and Murphy (2011).

Theorem 8 (Value consistency with patient-specific utility) Let ${\widehat{\theta }}_n$ be the maximum pseudo-likelihood estimator for θ and let ${\widehat{d}}_{{\widehat{\theta }}_n,n}$ be the associated estimated optimal policy. Then, under the given assumptions, $\left|V_{{\theta }_0}\left({\widehat{d}}_{{\widehat{\theta }}_n,n}\right)-V_{{\theta }_0}\left(d_{{\theta }_0}^{\text{opt}}\right)\right|\stackrel{P}{\to }0 as n\to \infty$.

Next, we derive the convergence rate and asymptotic distribution of (${\widehat{\theta }}_n$, ${\widehat{\beta }}_n$). Assume that $\mathcal{X}$ is a bounded subset of ${ℝ}^p$ and let $\Vert \cdot {\Vert }_{\mathcal{X}}$ be the sup norm over $\mathcal{X}$, i.e., for $f:\mathcal{X}\to ℝ$, $\Vert f{\Vert }_{\mathcal{X}}={\text{sup}}_{x\in \mathcal{X}}|f(x)|$. Let ${\dot{\omega }}_{\theta }(x)=(\partial /\partial \theta )\omega (x;\theta )$. Assume that $\Vert \Vert {\dot{\omega }}_{{\theta }_0}(x)\Vert {\Vert }_{\mathcal{X}}<\infty$ and that ${\text{lim}}_{\theta \to {\theta }_0}\Vert \Vert {\dot{\omega }}_{\theta }(x)-{\dot{\omega }}_{{\theta }_0}(x)\Vert {\Vert }_{\mathcal{X}}=0$. Define R_Y (x) = Q_Y (x, 1) − Q_Y (x, −1) to be the treatment contrast for outcome Y at patient covariates X = x; define R_Z(x) = Q_Z(x, 1) − Q_Z(x, −1) analogously. Let R₀(x) = R_Y (x) − R_Z(x) denote the difference in the treatment contrasts across the two outcomes. Similarly, define ${\widehat{R}}_{Y,n}(x)={\widehat{Q}}_{Y,n}(x,1)-{\widehat{Q}}_{Y,n}(x,-1)$, ${\widehat{R}}_{Z,n}(x)={\widehat{Q}}_{Z,n}(x,1)-{\widehat{Q}}_{Z,n}(x,-1)$, and ${\widehat{R}}_{0,n}(x)={\widehat{R}}_{Y,n}(x)-{\widehat{R}}_{Z,n}(x)$. Define D_θ(x) = ω(x; θ)R_Y (x) + {1 − ω(x; θ)}R_Z(x) to be the convex combination of treatment contrasts dictated by ω(x; θ) and let ${\widehat{D}}_{\theta ,n}(x)=\omega (x;\theta ){\widehat{R}}_{Y,n}(x)+\{1-\omega (x;\theta )\}{\widehat{R}}_{Z,n}(x)$. Note that $d_{\theta }^{\text{opt}}(x)=\text{sign}\left\{D_{\theta }(x)\right\}$ and ${\widehat{d}}_{\theta ,n}(x)=\text{sign}\left\{{\widehat{D}}_{\theta ,n}(x)\right\}$. Further define

$$P_{\beta }(x)=\text{expit}\left(x^{\top }\beta \right),$$

$${\psi }_{i,A}=\left[1\left\{A_i=d_{{\theta }_0}^{\text{opt}}\left(X_i\right)\right\}-P_{{\beta }_0}\left(X_i\right)\right]X_i,$$

$$I_n(\beta )={\mathbb{E}}_n\left[P_{\beta }(X)\left\{1-P_{\beta }(X)\right\}X{X}^{\top }\right],$$

$$I_0=\mathbb{E}\left[P_{{\beta }_0}(X)\left\{1-P_{{\beta }_0}(X)\right\}X{X}^{\top }\right].$$

We use the following regularity conditions.

Assumption 9 There exist independent and identically distributed influence vectors ${\psi }_{1,Y},{\psi }_{2,Y},\dots \in {ℝ}^{q_1}$, and ${\psi }_{1,Z},{\psi }_{2,Z},\dots \in {ℝ}^{q_2}$, and vector basis functions ϕ_Y (x) and ϕ_Z(x) such that both

$${\Vert \sqrt{n}\left\{{\widehat{R}}_{Y,n}(x)-R_Y(x)\right\}-{\varphi }_Y{(x)}^{\top }{n}^{-1/2}\sum _{i=1}^n{\psi }_{i,Y}\Vert }_{\mathcal{X}}=o_P(1)$$

and

$${\Vert \sqrt{n}\left\{{\widehat{R}}_{Z,n}(x)-R_Z(x)\right\}-{\varphi }_Z{(x)}^{\top }{n}^{-1/2}\sum _{i=1}^n{\psi }_{i,Z}\Vert }_{\mathcal{X}}=o_P(1).$$

Let $Z_{Y,n}=n^{-1/2}{\sum }_{i=1}^n{\psi }_{i,Y}$, $Z_{Z,n}=n^{-1/2}{\sum }_{i=1}^n{\psi }_{i,Z}$, $Z_{A,n}=n^{-1/2}{\sum }_{i=1}^n{\psi }_{i,A}$ and q = q₁ + q₂. Furthermore, assume that ${\Vert R_Y(x)\Vert }_{\mathcal{X}}$, ${\Vert R_Z(x)\Vert }_{\mathcal{X}}$, $\Vert \Vert {\varphi }_Y(x)\Vert {\Vert }_{\mathcal{X}}$ and $\Vert \Vert {\varphi }_Z(x)\Vert {\Vert }_{\mathcal{X}}$ are bounded by some M < ∞. Let ${\Sigma }_0=\mathbb{E}\left[{\left\{{\left({\psi }_{1,Y}^{\top },{\psi }_{1,Z}^{\top },{\psi }_{1,A}^{\top }\right)}^{\top }\right\}}^{\otimes 2}\right]$ be positive definite and finite, where u^⊗2 = uu^⊤.

Assumption 10 The following conditions hold.

1. The random variable $D_{{\theta }_0}(X)$ has a continuous density function f in a neighborhood of 0 with f₀ = f(0) ∈ (0, ∞);

2. The conditional distribution of X given that $\left|D_{{\theta }_0}(X)\right|\le ϵ$ converges to a non-degenerate distribution as ϵ ↓ 0;

3. There exist δ₁, δ₂ > 0 such that

   $$\underset{ϵ\downarrow 0}{\text{lim}}\underset{t\in S^d}{\text{inf}}\text{Pr}\left[\left|X^{\top }{\beta }_0\right|\ge {\delta }_1,\left|\left\{R_Y(X)-R_Z(X)\right\}{\dot{\omega }}_{{\theta }_0}{(X)}^{\top }t\right|\ge {\delta }_1\left|\right|D_{{\theta }_0}(X)|\le ϵ\right]\ge {\delta }_2,$$

   where S^d is the d-dimensional unit sphere.

Assumption 11 Define, for $Z_Y\in {ℝ}^{q_1}$, $Z_Z\in {ℝ}^{q_2}$, and $U\in {ℝ}^d$,

$$\left(Z_Y,Z_Z,U\right)\mapsto k_0\left(Z_Y,Z_Z,U\right)=\mathbb{E}[X\left\{2P_{{\beta }_0}(X)-1\right\}\cdot |\omega \left(X;{\theta }_0\right)R_Y(X){\varphi }_Y{(X)}^{\top }{Z}_Y+\left\{1-\omega \left(X;{\theta }_0\right)\right\}{R}_Z(X){\varphi }_Z{(X)}^{\top }{Z}_Z+R_0(X){\dot{\omega }}_{{\theta }_0}{(X)}^{\top }U\left|\right|D_{{\theta }_0}(X)=0].$$ (4)

Assume that $U\mapsto {\beta }_0^{\top }{k}_0\left(Z_Y,Z_Z,U\right)$ has a unique, finite minimum over ${ℝ}^d$ for all ${\left(Z_Y^{\top },Z_Z^{\top }\right)}^{\top }\in {ℝ}^q$.

Remark 12 Assumption 9 establishes a rate of convergence for the estimated Q-functions and is automatically satisfied if the Q-functions are estimated using linear or generalized linear models with or without interactions or higher order terms. Assumption 10 is needed to ensure that there is positive probability of patients with x values near the boundary between where each treatment is optimal. Assumption 11 is standard in M-estimation.

Let (${\widehat{\theta }}_n$, ${\widehat{\beta }}_n$) be the maximum pseudo-likelihood estimators given in Section 2.2. The following theorem states the asymptotic distribution of (${\widehat{\theta }}_n$, ${\widehat{\beta }}_n$).

Theorem 13 (Asymptotic distribution) Under the given regularity conditions

$$\sqrt{n}\left(\begin{array}{c}{\widehat{\theta }}_n-{\theta }_0\\ {\widehat{\beta }}_n-{\beta }_0\end{array}\right)⇝\left(\begin{array}{c}U\\ I_0^{-1}\left\{Z_A-k_0\left(Z_Y,Z_Z,U\right)\right\}\end{array}\right)\equiv \left(\begin{array}{l}U\hfill \\ B\hfill \end{array}\right),$$ (5)

where ${\left(Z_Y^{\top },Z_Z^{\top },Z_A^{\top }\right)}^{\top }~N\left(0,{\sum }_0\right)$, and $U=\text{arg\hspace{0.17em}}{\text{min}}_{u\in {ℝ}^d}{\beta }_0^{\top }{k}_0\left(Z_Y,Z_Z,u\right)$.

Let $\underset{Z*}{\overset{P}{⇝}}$ denote convergence in probability over Z*, as defined in Section 2.2.3 and Chapter 10 of Kosorok (2008). Theorem 14 below establishes the validity of a parametric bootstrap procedure for approximating the sampling distribution of (${\widehat{\theta }}_n$, ${\widehat{\beta }}_n$).

Theorem 14 (Parametric bootstrap) Assume ${\widehat{\Sigma }}_n={\Sigma }_0+o_P(1)$ and $h_n={\widehat{v}}_n{n}^{-1/5}$, where ${\widehat{v}}_n\stackrel{P}{\to }{v}_0\in (0,\infty )$ and v₀ is the standard error of $D_{{\theta }_0}(X)$. Assume the regularity conditions given above hold. Let $Z^{*}~N\left(0,I^{r\times r}\right)$, where I^r×r is an r × r identity matrix and r = q + p. Let ${\tilde{Z}}_n={\widehat{\Sigma }}_n^{1/2}{Z}^{*}={\left({\tilde{Z}}_Y^{\top },{\tilde{Z}}_Z^{\top },{\tilde{Z}}_A^{\top }\right)}^{\top }$, where

$${\widehat{\Sigma }}_n^{1/2}=\left[\begin{array}{cc}{\widehat{\Sigma }}_1^{1/2}& 0\\ {\widehat{\Sigma }}_2{\widehat{\Sigma }}_1^{-1/2}& {\left({\widehat{\Sigma }}_2-{\widehat{\Sigma }}_{21}{\widehat{\Sigma }}_1^{-1}{\widehat{\Sigma }}_{12}\right)}^{1/2}\end{array}\right],$$

${\widehat{\Sigma }}_1$ is the top left q × q block of ${\widehat{\Sigma }}_n$ (corresponding to Z_Y and Z_Z), ${\widehat{\Sigma }}_2$ is the lower right p × p block, ${\widehat{\Sigma }}_{21}$. is the upper right q × p block, ${\widehat{\Sigma }}_{12}={\widehat{\Sigma }}_{21}^{\top }$, and the matrix square roots are the symmetric square roots obtained from the associated Eigenvalue decompositions. Let

$${\tilde{T}}_n\left(\mathit{X},Z_Y,Z_Z\right)=\omega \left(\mathit{X};{\widehat{\theta }}_n\right){\widehat{R}}_{Y,n}(\mathit{X}){\varphi }_Y{(\mathit{X})}^{\top }{Z}_Y+\left\{1-\omega \left(\mathit{X};{\widehat{\theta }}_n\right)\right\}{\widehat{R}}_{Z,n}(\mathit{X}){\varphi }_Z{(\mathit{X})}^{\top }{Z}_Z$$

and define

$${\tilde{k}}_n\left(Z_Y,Z_Z,U\right)={\mathbb{E}}_n[X\left\{2P_{{\widehat{\beta }}_n}(X)-1\right\}\cdot |{\tilde{T}}_n\left(X,Z_Y,Z_Z\right)+\left\{{\widehat{R}}_{Y,n}(X)-{\widehat{R}}_{Z,n}(X)\right\}{\dot{\omega }}_{{\widehat{\theta }}_n}{(X)}^{\top }U|\cdot h_n^{-1}{\varphi }_0\left\{{\widehat{D}}_{{\widehat{\theta }}_n,n}(X)/h_n\right\}]\times {\left\{{\mathbb{E}}_n\left[h_n^{-1}{\varphi }_0\left\{{\widehat{D}}_{{\widehat{\theta }}_n,n}(X)/h_n\right\}\right]\right\}}^{-1},$$

where ϕ₀ is the standard normal density. Define ${\tilde{U}}_n=\text{arg\hspace{0.17em}}{\text{min}}_{u\in {ℝ}^d}{\widehat{\beta }}_n^{\top }{\tilde{k}}_n\left({\tilde{Z}}_Y,{\tilde{Z}}_Z,u\right)$ and ${\tilde{B}}_n=I_n{\left({\widehat{\beta }}_n\right)}^{-1}\left\{{\tilde{Z}}_A-{\tilde{k}}_n\left({\tilde{Z}}_Y,{\tilde{Z}}_Z,{\tilde{U}}_n\right)\right\}$. Then,

$$\left(\begin{array}{c}{\tilde{U}}_n\\ {\tilde{B}}_n\end{array}\right)\underset{{\tilde{Z}}^{*}}{\overset{P}{⇝}}\left(\begin{array}{l}U\hfill \\ B\hfill \end{array}\right),$$ (6)

where (U^⊤, B^⊤)^⊤ is as defined in Theorem 13.

If we fix a large number of bootstrap replications, B, then $\left({\tilde{U}}_{n,b},{\tilde{B}}_{n,b}\right),b=1,\dots ,B$ will provide an approximation to the sampling distribution of the maximum pseudo-likelihood estimators. In Sections 4 and 5, we demonstrate the use of the bootstrap to test for heterogeneity of patient preferences.

Remark 15 In Theorem 13, it can be seen that when β₀ only involves an intercept, there is no relationship between β₀ and U, as the argmax of an objective function does not change under multiplication by a positive scalar. This relationship is more complex when β₀ includes covariate effects. Theorem 13 also indicates that the asymptotic behaviors of $\widehat{\theta }$ and $\widehat{\beta }$ are driven largely by what happens at the boundary where $D_{{\theta }_0}(X)=0$.

4. Simulation Experiments

4.1. Fixed Utility Simulations

To examine the finite sample performance of the proposed methods, we begin with the following simple generative model. Let X = (X₁, …, X₅)^⊤ be a vector of independent normal random variables with mean 0 and standard deviation 0.5. Let treatment be assigned according to $\text{Pr}\left\{A=d_{\omega }^{\text{opt}}(x)|X=x\right\}=\rho$, i.e., the probability that the clinician correctly identifies the optimal treatment is constant across patients. Let ϵ_Y and ϵ_Z be independent normal random variables with mean 0 and standard deviation 0.5 and let Y = A (4X₁ − 2X₂ + 2) + ϵ_Y and Z = A (2X₁ − 4X₂ − 2) + ϵ_Z. We estimated Q_Y and Q_Z using linear models, implemented the proposed method for a variety of n, ω, and ρ values, and examined ${\widehat{\omega }}_n$, ${\widehat{\rho }}_n$, and ${\widehat{d}}_{{\widehat{\omega }}_n,n}$, across 500 Monte Carlo replications per scenario.

Table 1 contains mean estimates of ω and ρ across replications along with the associated standard deviation across replications, and estimated error rate defined as the proportion of subjects to whom the estimated optimal policy does not recommend the true optimal treatment; to better characterize sampling variability in the estimated error rate the last column displays the median along with the first and third quartiles of the sampling distribution of the estimated error rate.

Table 1:

Estimation results for simulations where utility and probability of optimal treatment are fixed.

n	ω	ρ	${\widehat{\omega }}_n$	${\widehat{\rho }}_n$	Error rate	Median(25th, 75th)
100	0.25	0.60	0.34 (0.24)	0.61 (0.08)	0.12 (0.13)	0.07 (0.03, 0.13)
			0.25 (0.05)	0.80 (0.04)	0.03 (0.02)	0.02 (0.01, 0.03)
	0.75	0.60	0.66 (0.24)	0.61 (0.07)	0.12 (0.13)	0.07 (0.03, 0.14)
			0.75 (0.05)	0.80 (0.04)	0.03 (0.02)	0.02 (0.01, 0.03)
200	0.25	0.60	0.28 (0.16)	0.61 (0.04)	0.07 (0.08)	0.04 (0.02, 0.10)
			0.25 (0.02)	0.80 (0.03)	0.01 (0.01)	0.01 (0.01, 0.02)
	0.75	0.60	0.72 (0.16)	0.61 (0.04)	0.07 (0.09)	0.03 (0.01, 0.08)
			0.75 (0.03)	0.80 (0.03)	0.01 (0.01)	0.01 (0.01, 0.02)
300	0.25	0.60	0.26 (0.11)	0.61 (0.03)	0.05 (0.06)	0.03 (0.01, 0.06)
			0.25 (0.02)	0.80 (0.02)	0.01 (0.01)	0.01 (0.00, 0.01)
	0.75	0.60	0.74 (0.13)	0.61 (0.03)	0.06 (0.07)	0.03 (0.01, 0.08)
			0.75 (0.02)	0.80 (0.02)	0.01 (0.01)	0.01 (0.01, 0.01)
500	0.25	0.60	0.25 (0.08)	0.61 (0.02)	0.04 (0.04)	0.02 (0.01, 0.04)
			0.25 (0.01)	0.80 (0.02)	0.01 (0.01)	0.01 (0.00, 0.01)
	0.75	0.60	0.75 (0.08)	0.61 (0.02)	0.04 (0.04)	0.02 (0.01, 0.05)
			0.75 (0.01)	0.80 (0.02)	0.01 (0.01)	0.01 (0.00, 0.01)

The pseudo-likelihood method performs well at estimating both ω and ρ, with estimation improving with larger sample sizes as expected. Table 2 contains estimated values of the true optimal policy, a policy where the utility function is estimated (the proposed method), policies estimated to maximize the two outcomes individually (corresponding to fixing ω = 1 and ω = 0), and the standard of care. The value of the standard of care is the mean composite outcome under the generative model. For each policy, the value is estimated by generating a testing sample of size 500 with treatment assigned according to the policy and averaging utilities (calculated using the true ω) in the testing set. The standard deviation across replications is included in parentheses.

Table 2:

Value results for simulations where utility and probability of optimal treatment are fixed.

n	ω	ρ	Optimal	Estimated ω	Y only	Z only	Standard of care
100	0.25	0.60	1.90 (0.07)	1.70 (0.38)	0.38 (0.11)	1.76 (0.08)	0.37 (0.24)
			1.90 (0.07)	1.89 (0.07)	0.39 (0.12)	1.76 (0.08)	1.14 (0.21)
	0.75	0.60	1.90 (0.06)	1.71 (0.37)	1.76 (0.08)	0.39 (0.12)	0.37 (0.23)
			1.90 (0.06)	1.89 (0.07)	1.76 (0.08)	0.39 (0.12)	1.14 (0.21)
200	0.25	0.60	1.90 (0.07)	1.82 (0.21)	0.39 (0.11)	1.76 (0.07)	0.39 (0.16)
			1.90 (0.07)	1.90 (0.06)	0.39 (0.11)	1.76 (0.07)	1.14 (0.14)
	0.75	0.60	1.90 (0.07)	1.82 (0.22)	1.76 (0.07)	0.38 (0.11)	0.39 (0.17)
			1.90 (0.07)	1.90 (0.06)	1.76 (0.07)	0.38 (0.11)	1.14 (0.15)
300	0.25	0.60	1.90 (0.07)	1.86 (0.13)	0.39 (0.11)	1.76 (0.07)	0.38 (0.14)
			1.90 (0.07)	1.89 (0.06)	0.39 (0.11)	1.76 (0.07)	1.14 (0.12)
	0.75	0.60	1.90 (0.06)	1.85 (0.17)	1.77 (0.07)	0.38 (0.11)	0.38 (0.14)
			1.90 (0.06)	1.90 (0.07)	1.77 (0.07)	0.38 (0.11)	1.13 (0.12)
500	0.25	0.60	1.90 (0.06)	1.88 (0.10)	0.39 (0.10)	1.76 (0.07)	0.38 (0.10)
			1.90 (0.06)	1.90 (0.07)	0.39 (0.11)	1.76 (0.07)	1.14 (0.09)
	0.75	0.60	1.89 (0.07)	1.88 (0.08)	1.77 (0.07)	0.39 (0.11)	0.38 (0.11)
			1.89 (0.07)	1.90 (0.07)	1.77 (0.07)	0.39 (0.11)	1.14 (0.09)

The column labeled “estimated ω” refers to the proposed method. We see that the proposed method produces values which increase with n and generally come close to the true optimal policy. In all settings, the proposed method offers significant improvement over the standard of care. The proposed method also offers improvement over policies to maximize each individual outcome.

To further examine the performance of the proposed method, we allow the probability of optimal treatment to depend on patient covariates. Let $\text{Pr}\left\{A=d_{\omega }^{\text{opt}}(X)\right\}=\text{expit}\left(0.5+X_1\right)$. This corresponds to the case where β = (0.5, 1, 0, …, 0)^⊤, where the first element of β is an intercept. Let X, Y, and Z be generated as described above. In this generative model, the probability that a patient is treated optimally in standard care is larger for patients with positive values of X₁ and smaller for patients with negative values of X₁. We applied the proposed method to 500 replications of this generative model for various n and ω. Table 3 contains mean estimates of ω, root mean squared error (RMSE) of ${\widehat{\beta }}_n$, and the error rate along with its standard error and quartiles.

Table 3:

Estimation results for simulations where utility is fixed and probability of optimal treatment is variable.

n	ω	${\widehat{\omega }}_n$	RMSE of ${\widehat{\beta }}_n$	Error rate	Median(25th, 75th)
100	0.25	0.34 (0.23)	1.32 (0.50)	0.10 (0.14)	0.04 (0.02, 0.10)
	0.75	0.71 (0.21)	1.37 (0.48)	0.10 (0.11)	0.06 (0.03, 0.12)
200	0.25	0.27 (0.13)	0.81 (0.30)	0.04 (0.08)	0.02 (0.01, 0.03)
	0.75	0.75 (0.15)	0.85 (0.29)	0.07 (0.07)	0.04 (0.02, 0.10)
300	0.25	0.26 (0.09)	0.60 (0.21)	0.03 (0.05)	0.01 (0.01, 0.02)
	0.75	0.75 (0.10)	0.63 (0.22)	0.04 (0.05)	0.03 (0.01, 0.07)
500	0.25	0.25 (0.03)	0.44 (0.14)	0.01 (0.01)	0.01 (0.00, 0.01)
	0.75	0.76 (0.07)	0.46 (0.16)	0.03 (0.04)	0.02 (0.01, 0.04)

Estimation of the observational policy (as defined by β) improves with larger sample sizes. The probability that the estimated policy assigns the optimal treatment also increases with the sample size. The true value of ω does not affect estimation of ω or β.

Table 4 contains estimated values of the true optimal policy, a policy where the utility function is estimated (the proposed method), policies estimated to maximize each outcome individually, and the standard of care. Values are estimated from independent testing sets of size 500 as described above. The value under the standard of care is the mean composite outcome under the generative model.

Table 4:

Value results for simulations where utility is fixed and probability of optimal treatment is variable.

n	ω	Optimal	Estimated ω	Y only	Z only	Standard of care
100	0.25	1.90 (0.07)	1.72 (0.40)	0.39 (0.12)	1.76 (0.07)	0.33 (0.23)
	0.75	1.90 (0.06)	1.75 (0.30)	1.76 (0.08)	0.39 (0.12)	0.56 (0.23)
200	0.25	1.90 (0.06)	1.85 (0.23)	0.37 (0.11)	1.76 (0.07)	0.34 (0.16)
	0.75	1.89 (0.06)	1.83 (0.18)	1.76 (0.07)	0.38 (0.11)	0.58 (0.16)
300	0.25	1.90 (0.06)	1.88 (0.16)	0.39 (0.10)	1.77 (0.07)	0.33 (0.14)
	0.75	1.90 (0.06)	1.87 (0.08)	1.76 (0.07)	0.40 (0.11)	0.57 (0.13)
500	0.25	1.90 (0.07)	1.89 (0.06)	0.39 (0.10)	1.76 (0.07)	0.33 (0.11)
	0.75	1.90 (0.07)	1.88 (0.07)	1.77 (0.07)	0.39 (0.11)	0.58 (0.10)

The proposed method (found in the column labeled “estimated ω”) produces values that are close to the true optimal policy in large samples and a significant improvement over standard of care in small to moderate samples. We note that value under the standard of care differs across ω. When ω is close to 1, the composite outcome places more weight on Y, for which the magnitude of the association with X₁ is larger. Because patients with larger values of X₁ are more likely to be treated optimally in this generative model, the standard of care produces larger composite outcomes when ω is closer to 1. Likewise, the mean composite outcome under policies to maximize each individual outcome varies with the true value of ω.

4.2. Patient-specific Utility Simulations

Next, we examine the case where the utility function is allowed to vary across patients. Let X, Y, and Z be generated as above. Again, assume that $\text{Pr}\left\{A=d_{\theta }^{\text{opt}}(X)\right\}=\text{expit}\left(0.5+X_1\right)$, i.e., β = (0.5, 1, 0, …, 0)^⊤. Consider the composite outcome U = ω(X; θ)Y + {1 − ω(X, θ)}Z, where ω(X; θ) = expit(1 − 0.5X₁), i.e., θ = (1, −0.5, 0, …, 0)^⊤, where the first element of θ is an intercept. We implemented the proposed method for various n and examined estimation of θ and β across 500 replications. Each replication is based on a simulated Markov chain of length 10,000 as described in Section 2.2. Results are summarized in Table 5.

Table 5:

Estimation results for simulations where both utility and probability of optimal treatment are variable.

n	RMSE of ${\widehat{\theta }}_n$	RMSE of ${\widehat{\beta }}_n$	Error rate	Median(25th, 75th)
100	0.85 (0.45)	1.28 (0.46)	0.10 (0.08)	0.06 (0.04, 0.13)
200	0.72 (0.31)	0.81 (0.27)	0.07 (0.06)	0.05 (0.04, 0.08)
300	0.63 (0.16)	0.63 (0.21)	0.05 (0.04)	0.04 (0.03, 0.06)
500	0.60 (0.09)	0.46 (0.15)	0.05 (0.02)	0.04 (0.03, 0.05)

Larger sample sizes produce marginal decreases in the RMSE of ${\widehat{\theta }}_n$. The estimated policy assigns the true optimal treatment more than 80% of the time for all sample sizes and the error rate decreases as the sample size increases. Table 6 contains estimated values of the true optimal policy, the policy estimated using the proposed method, policies estimated to maximize each outcome individually, and standard of care.

Table 6:

Value results for simulations where both utility and probability of optimal treatment are variable.

n	Optimal	Estimated ω	Y only	Z only	Standard of care
100	1.74 (0.06)	1.65 (0.14)	1.66 (0.06)	1.41 (0.08)	0.50 (0.21)
200	1.74 (0.06)	1.69 (0.11)	1.67 (0.06)	1.41 (0.08)	0.49 (0.15)
300	1.74 (0.06)	1.71 (0.07)	1.66 (0.06)	1.41 (0.07)	0.50 (0.13)
500	1.74 (0.06)	1.71 (0.07)	1.66 (0.06)	1.41 (0.08)	0.50 (0.11)

The proposed method produces policies that achieve significant improvement over the standard of care across sample sizes.

Finally, we examine the performance of the parametric bootstrap as described in Section 3. Let X be a bivariate vector of normal random variables with mean 0, standard deviation 0.5, and correlation zero. Let Y and Z be generated as above and let β = (2.5, 1, 0)^⊤ where the first element of β is an intercept. Let θ₍₁₎ be the vector θ with the first element removed. We are interested in testing the null hypothesis H₁ : ∥θ₍₁₎∥ = 0, which corresponds to a test for heterogeneity of patient preferences. The table below contains estimated power across 500 Monte Carlo replications under the null hypothesis, where the true value is θ = (1, 0, 0)^⊤, and two alternative hypotheses: H₁ : θ = (1, 4, 3)^⊤, and H₂ : θ = (1, 6, 6)^⊤. All tests were conducted at level α = 0.05 and based on 1000 bootstrap samples. The last column in Table 7 shows the average agreement between the bootstrap and estimated optimal decision rule when θ₀ = (1, 0, 0); the results suggest the decision rule is stable at sample sizes and generative models considered.

Table 7:

Power of bootstrap test for homogeneity of utility function

n	Type 1 error	Power against H1	Power against H2	Stability
100	0.002	0.238	0.264	0.805
200	0.004	0.790	0.782	0.831
300	0.004	0.958	0.948	0.833
500	0.002	0.998	0.990	0.829

The proposed bootstrap procedure produces type I error rates near nominal levels under the null and moderate power in large samples under alternative hypotheses.

5. Case Study: The STEP-BD Standard Care Pathway

The Systematic Treatment Enhancement Program for Bipolar Disorder (STEP-BD) was a landmark study of the effects of antidepressants in patients with bipolar disorder (Sachs et al., 2007). In addition to a randomized trial assessing outcomes for patients given an antidepressant or placebo, the STEP-BD study also included a large-scale observational study, the standard care pathway. As our method requires observational data on clinical decision making, we apply the proposed method to the observational data from the STEP-BD standard care pathway to estimate decision rules for the use of antidepressants in patients with bipolar disorder. (Clearly, as clinicians are not generally assigning treatment according to their best clinical judgment in a randomized clinical trial, the proposed method is not applicable to the randomized pathway of STEP-BD.)

Although bipolar disorder is characterized by alternating episodes of depression and mania, recurrent depression is the leading cause of impairment among patients with bipolar disorder (Judd et al., 2002). However, the use of antidepressants has not become standard care in bipolar disorder due to the risk of antidepressants inducing manic episodes in certain patients (Ghaemi, 2008; Goldberg, 2008). Thus, the clinical decision in the treatment of bipolar disorder is whether to prescribe antidepressants to a specific patient in order to balance trade-offs between symptoms of depression, symptoms of mania, and other side effects of treatment.

We use the SUM-D score for depression symptoms and the SUM-M score for mania symptoms as outcomes. We consider a patient treated if they took any one of ten antidepressants that appear in the STEP-BD standard care pathway (Deseryl, Serzone, Citalopram, Escitalopram Oxalate, Prozac, Fluvoxamine, Paroxetine, Zoloft, Venlafaxine, or Bupropion). To generate candidate predictors for our model we made use of a complimentary randomized pathway in the STEP-BD trial. In this pathway, the patients are drawn from the same population, and the same variables are measured; however, treatment is randomly assigned so that there is no unmeasured confounding. Using step-wise variable selection to construct an outcome model from these data identified the following variables: mood elevation, anxiety, irritability, baseline SUM-M, and baseline SUM-D. We also used a step-wise logistic regression for the propensity score in the observational pathway to identify any additional potential confounders (Moodie et al., 2012). In addition to the variables in the outcome model, the logistic regression model identified race, insurance status, age, and substance abuse. The union of variables identified in through the randomized pathway and the propensity score were used in our models of the Q-functions and as tailoring variables in our treatment rules. Figure 1 contains box plots of SUM-D scores on the log scale by substance abuse and treatment. Figure 2 contains box plots of SUM-M scores on the log scale by substance abuse and treatment. For both outcomes, lower scores are more desirable. Figure 1 indicates that those without a history of substance abuse benefit from treatment with antidepressants. However, among those with a history of substance abuse, patients treated with antidepressants appear to have worse symptoms of depression. Figure 2 indicates that treatment has no effect on symptoms of mania among those without a history of substance abuse. However, among those with a history of substance abuse, it appears that treatment may be inducing manic episodes. Thus, a sensible treatment policy would be one that tends to prescribe antidepressants only to patients without a history of substance abuse.

Figure 1:

Box plots of log SUM-D by substance abuse and treatment.

Figure 2:

Box plots of log SUM-M by substance abuse and treatment.

We analyzed these data using the proposed method for optimizing composite outcomes. Results are summarized in Table 8 below. We estimated policies where both utility and probability of optimal treatment are fixed (fixed-fixed), where utility is fixed but probability of optimal treatment is assumed to vary between patients (fixed-variable), and where both utility and probability of optimal treatment are assumed to vary between patients (variable-variable). For both the fixed-variable policy and the variable-variable policy, we report ${\mathbb{E}}_n\left\{\text{expit}\left(X^{\top }{\widehat{\beta }}_n\right)\right\}$ in place of ${\widehat{\rho }}_n$ and for the variable-variable policy, we report ${\mathbb{E}}_n\left\{\text{expit}\left(X^{\top }{\widehat{\theta }}_n\right)\right\}$ in place of ${\widehat{\omega }}_n$. Thus, for parameters that are assumed to vary across patients, Table 8 contains the mean estimate in the sample. To evaluate each estimated policy, we used five-fold cross-validation of the inverse probability weighted estimator (IPWE) of the value for each outcome; i.e., for each fold, we used the training portion to estimate the optimal policy and propensity score, and we used the testing portion to compute the IPWE of the value; taking the average of the IPWE value estimates across folds yields the reported values. For both SUM-D and SUM-M, lower scores are preferred. Value is reported as the percent improvement over standard of care, calculated using the estimated utility function. Large percent improvements in value are preferred.

Table 8:

Results of analysis of STEP-BD data for SUM-D and SUM-M.

Policy	SUM-D	SUM-M	Value (% improvement)	${\widehat{\omega }}_n$	${\widehat{\rho }}_n$
fixed-fixed	2.336	0.857	1.8%	0.039	0.431
fixed-variable	2.324	0.838	3.9%	0.039	0.440
variable-variable	2.321	0.804	8.3%	0.334	0.448
standard of care	2.480	0.868	0.0%	·	·

All estimated policies produce more desirable SUM-D scores and SUM-M scores compared to standard of care and improved value according to the estimated utility. Allowing the probability of optimal treatment to vary between patients leads to further improvements in value, as does allowing the utility function to vary between patients. All policies produce similar estimates for the probability of optimal treatment averaged across patients.

The resulting decision rules can be written as the sign of a linear combination of the covariates. As an example, the fixed-fixed policy assigns treatment with antidepressants when 0.032 − 0.001(age) − 0.646(substance abuse) − 0.007(mood elevation) + 0.007(medical insurance) + 0.129(white) is non-negative. The negative coefficient for substance abuse means that a history of substance abuse indicates that a patient should not be prescribed antidepressants. Prior research has shown that patients with a history of substance abuse are more likely to abuse antidepressants (Evans and Sullivan, 2014). This may contribute to the poor outcomes experienced by STEP-BD patients with a history of substance abuse who were treated with antidepressants. Table 9 displays estimates and standard errors of the components of ${\widehat{\theta }}_n$ in the variable-variable policy. A test for preference heterogeneity based on 1000 bootstrap samples generated according to Theorem 14 yielded a p-value < 0.001.

Table 9:

Estimates of ${\widehat{\theta }}_n$ in the variable-variable policy

	Intercept	Age	Substance	Mood elevation	Insurance	Race
Estimate	2.427	−.177	−1.666	−2.632	4.263	1.078
Standard error	3.039	0.953	2.746	2.238	3.241	3.274

As a secondary analysis, we use the SUM-D score and a side effect score as the outcomes. Eight side effects were recorded in the STEP-BD standard care pathway (tremors, dry mouth, sedation, constipation, diarrhea, headache, poor memory, sexual dysfunction, and increased appetite). Patients rated the severity of each side effect from 0 to 4 with larger values indicating more severe side effects. We took the mean score across side effects as the second outcome. Results are summarized in Table 10, reported analogously to those in Table 8.

Table 10:

Results of analysis of STEP-BD data for SUM-D and Side effect score.

Policy	SUM-D	Side effect score	Value (% improvement)	${\widehat{\omega }}_n$	${\widehat{\rho }}_n$
fixed-fixed	2.377	0.156	5.2%	0.601	0.462
fixed-variable	2.384	0.159	5.7%	0.100	0.472
variable-variable	2.430	0.161	6.1%	0.378	0.487
standard of care	2.480	0.172	0.0%	·	·

Each estimated policy produces improved SUM-D scores and improved side effect scores compared to the standard of care. Each policy also produces improvement in value according to the estimated utility function. Again, allowing the utility function to vary between patients results in further improvements in value. Each policy produces similar estimates of the probability that patients are treated optimally in standard care. The variable-variable policy places more weight on SUM-D scores on average compared to the other policies. Table 11 displays estimates and standard errors of coefficients in ${\widehat{\theta }}_n$ in the variable-variable policy. The bootstrap procedure for testing the null hypothesis that patient preferences are homogeneous based on 1000 bootstrap samples yielded a p-value < 0.001.

Table 11:

Estimates of ${\widehat{\theta }}_n$ in the variable-variable policy

	Intercept	Age	Substance	Mood	Irritable	Anxiety	Insurance	Race
Estimate	−3.125	−4.614	2.094	−0.609	2.594	0.332	−3.493	−2.563
Standard error	2.257	2.300	2.395	2.603	2.610	2.538	2.599	2.449

6. Discussion

The estimation of individualized treatment rules has been well-studied in the statistical literature. Existing methods have typically defined the optimal treatment rule as optimizing the mean of a fixed scalar outcome. However, clinical practice often requires consideration of multiple outcomes. Thus, there is a disconnect between existing statistical methods and current clinical practice. It is reasonable to assume that clinicians make treatment decisions for each patient with the goal of maximizing that patient’s utility. Therefore, it is natural to use observational data to estimate patient utilities from observed clinician decisions. This represents a new paradigm for the use of observational data in the context of precision medicine in that clinical decisions are viewed as a (noisy) surrogate for patient preferences and leveraged to improve the quality of a learned treatment rule and to generate new insights into heterogeneity in patient preferences.

The proposed methodology offers many opportunities for future research. In the present manuscript, we have considered only the simplest case— that of one decision time, two outcomes, and two possible treatments. Scenarios with more than two outcomes are discussed in the Appendix, and the simulation results there demonstrate that the proposed method performs well with three outcomes. Extensions to more than two treatments or multiple time points represent potential areas for future research. The proposed method requires positing a parametric model for the utility function. Model misspecification is discussed in the Appendix, and the simulation results there demonstrate that the proposed method performs reasonably well when important covariates are omitted from the model for the utility function. However, the use of semi- or non-parametric models is an important extension. A more technical direction for future work is a more nuanced study of the affect of boundary conditions on the resulting rate of convergence (see Assumption 10). Finally, while we have proposed our utility function estimator inside the framework of one-stage Q-learning, the pseudo-likelihood utility function estimator could be used alongside other existing one-stage optimal treatment policy estimators based on (augmented) inverse probability weighting (e.g., Zhao et al., 2012; Zhang et al., 2012a). There is a great future for the development of methods for optimizing composite outcomes in precision medicine and application of these methods in clinical studies.

Appendix A: Proofs

Proof [Proof of Theorem 5] Consider $(\beta ,\theta )\in \mathcal{B}\times \Theta$ and suppose

$$P_{\beta }{(X)}^{1\left\{Y=d_{\theta }(X)\right\}}(1-P_{\beta }{(X)}^{1-1\left\{Y=d_{\theta }(X)\right\}}=P_{{\beta }_0}{(X)}^{1\left\{Y=d_{{\theta }_0}(X)\right\}}(1-P_{{\beta }_0}{(X)}^{1-1\left\{Y=d_{{\theta }_0}(X)\right\}}\forall (X,Y)\text{ a}.\text{s}.$$ (7)

Let $X_S=\left\{X:d_{\theta }(X)=d_{{\theta }_0}(X)\right\}$. For all X ∈ X_S, we have that $P_{\beta }(X)=P_{{\beta }_0}(X)\Rightarrow \beta ={\beta }_0$ by 4 in Assumption 4 which implies identifiability of P_β(X) on X_S.

Now suppose θ ≠ θ₀, then since β = β₀, we have by (7), that $\forall X\in X_S^c$, $P_{\beta }(X)=1-P_{{\beta }_0}(X)$ which by applying 4 in Assumption 4 again, we obtain that β = −β₀. However, this is impossible by 2 in Assumption 4. Thus θ = θ₀, and we obtain that (β, θ) = (β₀, θ₀). ■

Proof [Proof of Theorem 7] The log of the pseudo-likelihood is given by

$${\widehat{\mathcal{l}}}_n(\theta ,\beta )={\mathbb{E}}_n\left[{\mathit{X}}^{\top }\beta 1\left\{A={\widehat{d}}_{\theta ,n}(\mathit{X})\right\}-\text{log}\left\{1+\text{exp}\left({\mathit{X}}^{\top }\beta \right)\right\}\right].$$

Let $\widehat{m}(\cdot ,\cdot ;\theta ,\beta ):\mathcal{X}\times \mathcal{A}\to ℝ$ be defined by $\widehat{m}(X,A;\theta ,\beta )=X^{\top }\beta 1\left\{A={\widehat{d}}_{\theta ,n}(X)\right\}-\text{log}\left\{1+\text{exp}\left(X^{\top }\beta \right)\right\}$ and consider the class of functions $\left\{\widehat{m}(\cdot ,\cdot ;\theta ,\beta ):\theta \in {ℝ}^p,\beta \in \mathcal{B}\right\}$. The class $\left\{\text{log}\left\{1+\text{exp}\left(X^{\top }\beta \right)\right\}:\beta \in \mathcal{B}\right\}$ is contained in a VC class by Lemma 9.9 (viii) and (v) of Kosorok (2008). By Theorem 9.3 of Kosorok (2008), this is also a Glivenko–Cantelli (GC) class.

Let $u(X,A;\theta )=\omega (X;\theta )\left\{{\widehat{Q}}_{Y,n}(X,A)-{\widehat{Q}}_{Z,n}(X,A)\right\}+{\widehat{Q}}_{Z,n}(X,A)$, which lies in a VC class indexed by $\theta \in {ℝ}^p$ by Lemma 9.6 and Lemma 9.9 (viii), (vi), and (v) of Kosorok (2008). We have that

$$1\left\{A={\widehat{d}}_{\theta ,n}\left(X\right)\right\}=1\left(A=1\right)1\left\{u\left(X,1;\theta \right)-u\left(X,-1,\theta \right)\ge 0\right\}+1(A=-1)1\{u(X,1;\theta )-u(X,-1,\theta )<0\}$$

and it follows that $1\left\{A={\widehat{d}}_{\theta ,n}(X)\right\}$ is contained in a GC class indexed by $\theta \in {ℝ}^p$. From Corollary 9.27 (ii) of Kosorok (2008) it follows that $X^{\top }\beta 1\left\{A={\widehat{d}}_{\theta ,n}(X)\right\}$ lies in a GC class indexed by $(\theta ,\beta )\in {ℝ}^p\times \mathcal{B}$ as long as X^⊤β is uniformly bounded by a function with finite mean, which holds as long as $\mathcal{B}$ is compact and $\Vert \mathbb{E}X\Vert <\infty$. It follows that

$$\underset{(\theta ,\beta )\in {ℝ}^p\times \mathcal{B}}{\text{sup}}\left|\left({\mathbb{E}}_n-\mathbb{E}\right)\left[X^{\top }\beta 1\left\{A={\widehat{d}}_{\theta ,n}(X)\right\}-\text{log}\left\{1+\text{exp}\left(X^{\top }\beta \right)\right\}\right]\right|\stackrel{P}{\to }0.$$

Next, define

$$\widehat{M}(\theta ,\beta )=\mathbb{E}\{\widehat{m}(X,A;\theta ,\beta )\}=\mathbb{E}\left(X^{\top }\beta \mathbb{E}\left[1\left\{A={\widehat{d}}_{\theta ,n}(X)\right\}|X\right]\right)-\mathbb{E}\text{log}\left\{1+\text{exp}\left(X^{\top }\beta \right)\right\}$$

and note that $\widehat{M}(\theta ,\beta )$ is continuous in β. The inside expectation of the first piece is

$$\mathbb{E}\left[1\left\{A={\widehat{d}}_{\theta ,n}(X)\right\}|X\right]=\text{expit}\left(X^{\top }{\beta }_0\right)1\left\{{\widehat{d}}_{\theta ,n}(X)=d_{{\theta }_0}^{\text{opt}}(X)\right\}+\left\{1-\text{expit}\left(X^{\top }{\beta }_0\right)\right\}1\left\{{\widehat{d}}_{\theta ,n}(X)\ne d_{{\theta }_0}^{\text{opt}}(X)\right\},$$

using the fact that $\text{Pr}\left\{A=d_{{\theta }_0}^{\text{opt}}(X)\right\}=\text{expit}\left(X^{\top }{\beta }_0\right)$. Define a(X) = Q_Y (X, 1)−Q_Y (X, −1)−Q_Z(X, 1) + Q_Z(X, −1) and b(X) = Q_Z(X, 1) − Q_Z(X, −1). Similarly, define $\widehat{a}(X)={\widehat{Q}}_{Y,n}(X,1)-{\widehat{Q}}_{Y,n}(X,-1)-{\widehat{Q}}_{Z,n}(X,1)+{\widehat{Q}}_{Z,n}(X,-1)$ and $\widehat{b}(X)={\widehat{Q}}_{Z,n}(X,1)-{\widehat{Q}}_{Z,n}(X,-1)$. Then,

$$\begin{gathered} 1\left\{{\widehat{d}}_{\theta ,n}(X)=d_{{\theta }_0}^{\text{opt}}(X)\right\}=1[\{\omega (X;\theta )\widehat{a}(X)+\widehat{b}(X)\left\}\right\{\omega (X;\theta )a(X)+b(X)\}\ge 0] \\ =1[\omega (X;\theta )\{\omega (X;\theta )a(X)\widehat{a}(X)+\widehat{a}(X)b(X)\}+\omega (X;\theta )a(X)\widehat{b}(X)+\widehat{b}(X)b(X)\ge 0], \end{gathered}$$

and thus $\mathbb{E}\left[1\left\{A={\widehat{d}}_{\theta ,n}(X)\right\}|X\right]$ is continuous in θ.

Let $m(X,A;\theta ,\beta )=X^{\top }\beta 1\left\{A=d_{\theta }^{\text{opt}}(X)\right\}-\text{log}\left\{1+\text{exp}\left(X^{\top }\beta \right)\right\}$. Because the model is identifiable and ${\mathcal{L}}_n(\theta ,\beta )$ is a parametric log-likelihood, $\mathbb{E}m(\mathit{X},A;\theta ,\beta )$ has unique maximizers at θ₀ and β₀. Let ${\tilde{\theta }}_n$ and ${\tilde{\beta }}_n$ be the maximizers of $\mathbb{E}\widehat{m}(X,A;\theta ,\beta )$. Because $\mathbb{E}\left\{\left|{\widehat{d}}_{\theta ,n}(X)-d_{\theta }^{\text{opt}}(X)\right|\right\}\to 0$ in probability, for any $\theta \in {ℝ}^d$, $\mathbb{E}\left[1\left\{A={\widehat{d}}_{\theta ,n}(X)\right\}|X\right]-\mathbb{E}\left[1\left\{A=d_{\theta }^{\text{opt}}(X)\right\}|X\right]\to 0$ in probability, uniformly in θ over compact subsets of ${ℝ}^d$, which implies that both ${\tilde{\theta }}_n\to {\theta }_0$ and $\tilde{\beta }\to {\beta }_0$ in probability. The claim now follows from Lemma 14.3 and Theorem 2.12 of Kosorok (2008).■

Proof [Proof of Theorem 8] Define $Q_{{\theta }_0}(x,a)$ and $Q_{{\widehat{\theta }}_n}(x,a)$ as defined in Section 2. Let u(Y, Z; A, X, θ) = ω(X; θ)Q_Y (X, A) + {1 − ω(X; θ)}Q_Z(X, A). Under the given assumptions, for some constant 0 < c < ∞,

$$\left|V\left({\widehat{d}}_{{\widehat{\theta }}_n,n}\right)-V\left(d_{{\theta }_0}^{\text{opt}}\right)\right|\le c{\left|\mathbb{E}{\left\{u\left(Y,Z;A,X,{\theta }_0\right)-{\widehat{Q}}_{{\widehat{\theta }}_n,n}(X,A)\right\}}^2-\mathbb{E}{\left\{u\left(Y,Z;A,X,{\theta }_0\right)-Q_{{\theta }_0}(X,A)\right\}}^2\right|}^{1/2}$$ (8)

by equation (3.1) of Qian and Murphy (2011) (see also Murphy, 2005). The right hand side of (8) converges in probability to 0 by the consistency of ${\widehat{\theta }}_n$, consistency of ${\widehat{Q}}_{Y,n}$ and ${\widehat{Q}}_{Z,n}$, and the continuous mapping theorem. The result follows.■

Proof [Proof of Theorem 13] By definition of (${\widehat{\theta }}_n$, ${\widehat{\beta }}_n$),

$$\begin{gathered} 0\le {\widehat{\mathcal{l}}}_n\left({\widehat{\theta }}_n,{\widehat{\beta }}_n\right)-{\widehat{\mathcal{l}}}_n\left({\theta }_0,{\beta }_0\right) \\ =\sum _{i=1}^n[X_i^{\top }{\widehat{\beta }}_{n}1\left\{A_i={\widehat{d}}_{{\widehat{\theta }}_n,n}\left(X_i\right)\right\}-X_i^{\top }{\beta }_{0}1\left\{A_i={\widehat{d}}_{{\theta }_0,n}\left(X_i\right)\right\}-{\left({\widehat{\beta }}_n-{\beta }_0\right)}^{\top }{X}_i{P}_{{\beta }_0}\left(X_i\right)]-\frac{1}{2}\sqrt{n}{\left({\widehat{\beta }}_n-{\beta }_0\right)}^{\top }{I}_n\left({\beta }_{*}\right)\sqrt{n}\left({\widehat{\beta }}_n-{\beta }_0\right) \\ =\sqrt{n}{\left({\widehat{\beta }}_n-{\beta }_0\right)}^{\top }{n}^{-1/2}\sum _{i=1}^n{X}_i\left[1\left\{A_i={\widehat{d}}_{{\widehat{\theta }}_n,n}\left(X_i\right)\right\}-P_{{\beta }_0}\left(X_i\right)\right]-\frac{1}{2}\sqrt{n}{\left({\widehat{\beta }}_n-{\beta }_0\right)}^{\top }{I}_n\left({\beta }_{*}\right)\sqrt{n}\left({\widehat{\beta }}_n-{\beta }_0\right)+\sum _{i=1}^n{X}_i^{\top }{\beta }_0\left[1\left\{A_i={\widehat{d}}_{{\widehat{\theta }}_n,n}\left(X_i\right)\right\}-1\left\{A_i={\widehat{d}}_{{\theta }_0,n}\left(X_i\right)\right\}\right], \end{gathered}$$

where β_* is a point between ${\widehat{\beta }}_n$ and β₀. Using the definition of a maximizer and letting ${\widehat{u}}_n(\theta )=n^{-1/2}{\sum }_{i=1}^n{X}_i\left[1\left\{A_i={\widehat{d}}_{\theta ,n}\left(X_i\right)\right\}-P_{{\beta }_0}\left(X_i\right)\right]$, we have that $\sqrt{n}\left({\widehat{\beta }}_n-{\beta }_0\right)=I_n{\left({\beta }_{*}\right)}^{-1}{\widehat{u}}_n\left({\widehat{\theta }}_n\right)$ by setting ${\frac{\partial }{\partial \beta }{\widehat{l}}_n(\widehat{\theta },\beta )|}_{\widehat{\beta }}=0$, since $I_n\left({\beta }_{*}\right)\stackrel{P}{\to }{I}_0$ and I₀ is positive definite. Next, note that

$$\begin{gathered} {\widehat{u}}_n\left({\widehat{\theta }}_n\right)=n^{-1/2}\sum _{i=1}^n{X}_i\left[1\left\{A_i={\widehat{d}}_{{\widehat{\theta }}_n,n}\left(X_i\right)\right\}-1\left\{A_i=d_{{\theta }_0}^{\text{opt}}\left(X_i\right)\right\}\right]+Z_{A,n} \\ ={\mathbb{G}}_n\left(X\left[1\left\{A={\widehat{d}}_{{\widehat{\theta }}_n,n}(X)\right\}-1\left\{A=d_{{\theta }_0}^{\text{opt}}(X)\right\}\right]\right)+Z_{A,n}+\sqrt{n}\mathbb{E}\left(X\left[1\left\{A={\widehat{d}}_{{\widehat{\theta }}_n,n}(X)\right\}-1\left\{A=d_{{\theta }_0}^{\text{opt}}(X)\right\}\right]\right) \\ =Z_{A,n}+\sqrt{n}\mathbb{E}\left(X\left[1\left\{A={\widehat{d}}_{{\widehat{\theta }}_n,n}(X)\right\}-1\left\{A=d_{{\theta }_0}^{\text{opt}}(X)\right\}\right]\right)\left\{1+o_P(1)\right\}+o_P(1), \end{gathered}$$

where ${\mathbb{G}}_{n}f=n^{1/2}\left({\mathbb{E}}_n-\mathbb{E}\right)f(X)$ and $Z_{A,n}=n^{-1/2}{\sum }_{i=1}^n\left[1\left\{A_i=d_{{\theta }_0}^{\text{opt}}\left(X_i\right)\right\}-P_{{\beta }_0}\left(X_i\right)\right]$. We also have $1\left\{A={\widehat{d}}_{{\widehat{\theta }}_n,n}(X)\right\}-1\left\{A=d_{{\theta }_0}^{\text{opt}}(X)\right\}=-\left[2\cdot 1\left\{A=d_{{\theta }_0}^{\text{opt}}(X)\right\}-1\right]$, which implies that

$$\begin{gathered} \sqrt{n}\mathbb{E}\left(X\left[1\left\{A={\widehat{d}}_{{\widehat{\theta }}_n,n}(X)\right\}-1\left\{A=d_{{\theta }_0}^{\text{opt}}(X)\right\}\right]\right) \\ =-\sqrt{n}\mathbb{E}\left[X\left\{2P_{{\beta }_0}(X)-1\right\}1\left\{{\widehat{d}}_{{\widehat{\theta }}_n,n}(X)\ne d_{{\theta }_0}^{\text{opt}}(X)\right\}\right] \\ =-\sqrt{n}\mathbb{E}\{X\{2P_{{\beta }_0}(X)-1\}(1\left[0\le D_{{\theta }_0}(X)<-\left\{{\widehat{D}}_{{\widehat{\theta }}_n,n}(X)-D_{{\theta }_0}(X)\right\}\right] \\ =-\sqrt{n}\mathbb{E}\{X\{2P_{{\beta }_0}(X)-1\}(1\left[0\le D_{{\theta }_0}(X)<-\left\{{\widehat{D}}_{{\widehat{\theta }}_n,n}(X)-D_{{\theta }_0}(X)\right\}\right]+1\left[-\left\{{\widehat{D}}_{{\widehat{\theta }}_n,n}(X)-D_{{\theta }_0}(X)\right\}\le D_{{\theta }_0}(X)<0\right])\} \end{gathered}$$

$$\begin{gathered} =-\sqrt{n}\mathbb{E}\left[X\left\{2P_{{\beta }_0}(X)-1\right\}\left({\int }_0^{-\left({\widehat{D}}_{{\widehat{\theta }}_n}-D_{{\theta }_0}\right)}{f}_0(u)du+{\int }_{-\left({\widehat{D}}_{{\widehat{\theta }}_n}-D_{{\theta }_0}\right)}^0{f}_0(u)du\right)\Vert D_{{\theta }_0}(X)|\le {ϵ}_n\right]\times \left(1+o_P(1)\right) \\ =-\mathbb{E}\left[X\left\{2P_{{\beta }_0}(X)-1\right\}\cdot \left|\sqrt{n}\left\{{\widehat{D}}_{{\widehat{\theta }}_n,n}(X)-D_{{\theta }_0}(X)\right\}\right||D_{{\theta }_0}(X)=0\right]f_0+o_P(1), \end{gathered}$$

for some sequence ϵ_n converging down to zero slowly enough. The second to last equality holds by the fact that ${\Vert {\widehat{D}}_{{\widehat{\theta }}_n,n}(x)-D_{{\theta }_0}(x)\Vert }_{\mathcal{X}}=o_P(1)$ and Assumption 10 yields that the distribution of X converges to a random variable independent of $D_{{\theta }_0}(X)$ conditional on $\left|D_{{\theta }_0}(X)\right|\le {ϵ}_n$, as n goes to ∞.

Note that

$$\begin{gathered} \sqrt{n}\left\{{\widehat{D}}_{{\widehat{\theta }}_n,n}(X)-D_{{\theta }_0}(X)\right\}=\sqrt{n}[\omega \left(X;{\widehat{\theta }}_n\right)\left\{{\widehat{R}}_{Y,n}(X)-R_Y(X)\right\}+\left\{1-\omega \left(X;{\widehat{\theta }}_n\right)\right\}\left\{{\widehat{R}}_{Z,n}(X)-R_Z(X)\right\}]+\sqrt{n}\left\{\omega \left(X;{\widehat{\theta }}_n\right)-\omega \left(X;{\theta }_0\right)\right\}\left\{R_Y(X)-R_Z(X)\right\} \\ =\omega \left(X;{\theta }_0\right){\varphi }_Y{\left(X\right)}^{\top }{Z}_{Y,n}+\left\{1-\omega \left(X;{\theta }_0\right)\right\}{\varphi }_Z{\left(X\right)}^{\top }{Z}_{Z,n}+{\dot{\omega }}_{{\theta }_0}(X)\left\{R_Y(X)-R_Z(X)\right\}\sqrt{n}\left({\widehat{\theta }}_n-{\theta }_0\right)\left\{1+o_P(1)\right\} \\ =O_P\left(1+\sqrt{n}\Vert {\widehat{\theta }}_n-{\theta }_0\Vert \right), \end{gathered}$$

thus, $\Vert {\widehat{u}}_n\left({\widehat{\theta }}_n\right)\Vert =O_P\left(1+\sqrt{n}\Vert {\widehat{\theta }}_n-{\theta }_0\Vert \right)$. Letting $v_n\left({\widehat{\theta }}_n,{\beta }_{*}\right)=n^{-1/2}{\widehat{u}}_n{\left({\widehat{\theta }}_n\right)}^{\top }{I}_n^{-1}\left({\beta }_{*}\right)\cdot {\widehat{u}}_n\left({\widehat{\theta }}_n\right)$,

$$\begin{gathered} 0\le n^{-1/2}\left\{{\widehat{\mathcal{l}}}_n\left({\widehat{\theta }}_n,{\widehat{\beta }}_n\right)-{\widehat{\mathcal{l}}}_n\left({\theta }_0,{\beta }_0\right)\right\} \\ =n^{-1/2}\sum _{i=1}^n{X}_i^{\top }{\beta }_0\left[1\left\{A_i={\widehat{d}}_{{\widehat{\theta }}_n,n}(X)\right\}-1\left\{A_i={\widehat{d}}_{{\theta }_0,n}(X)\right\}\right]+v_n\left({\widehat{\theta }}_n,{\beta }_{\ast }\right)/2 \\ =n^{1/2}\mathbb{E}\left(X^{\top }{\beta }_0\left[1\left\{A={\widehat{d}}_{{\widehat{\theta }}_n,n}(X)\right\}-1\left\{A={\widehat{d}}_{{\theta }_0,n}(X)\right\}\right]\right)+o_P\left(1+\sqrt{n}\Vert {\widehat{\theta }}_n-{\theta }_0\Vert \right) \\ =n^{1/2}\mathbb{E}\left(X^{\top }{\beta }_0\left[1\left\{A={\widehat{d}}_{{\widehat{\theta }}_n,n}(X)\right\}-1\left\{A=d_{{\theta }_0}^{\text{opt}}(X)\right\}\right]\right)-n^{1/2}\mathbb{E}\left(X^{\top }{\beta }_0\left[1\left\{A={\widehat{d}}_{{\theta }_0,n}(X)\right\}-1\left\{A=d_{{\theta }_0}^{\text{opt}}(X)\right\}\right]\right)+o_P\left(1+\sqrt{n}\Vert {\widehat{\theta }}_n-{\theta }_0\Vert \right) \\ =-\mathbb{E}\left[X^{\top }{\beta }_0\left\{2P_{{\beta }_0}(X)-1\right\}\cdot \left|\sqrt{n}\left\{{\widehat{D}}_{{\widehat{\theta }}_n,n}(X)-D_{{\theta }_0}(X)\right\}\right||D_{{\theta }_0}(X)=0\right]f_0{r}_n+O_P(1)+o_P\left(1+\sqrt{n}\Vert {\widehat{\theta }}_n-{\theta }_0\Vert \right) \\ \le -\mathbb{E}\left[X^{\top }{\beta }_0\left\{2P_{{\beta }_0}(X)-1\right\}\cdot \left|\left\{R_Y(X)-R_Z(X)\right\}{\dot{\omega }}_{{\theta }_0}{(X)}^{\top }\sqrt{n}\left({\widehat{\theta }}_n-{\theta }_0\right)\right||D_{{\theta }_0}(X)=0\right]f_0{r}_n+O_P(1)+o_P\left(1+\sqrt{n}\Vert {\widehat{\theta }}_n-{\theta }_0\Vert \right) \\ \le -{\delta }_1^2\left(\frac{\exp \left({\delta }_1\right)-1}{\exp \left({\delta }_1\right)+1}\right)\underset{t\in S^d}{\inf }\left\{\mathrm{Pr}\left[\left|X^{\top }{\beta }_0\right|\ge {\delta }_1,\left|\left\{R_Y(X)-R_Z(X)\right\}{\dot{\omega }}_{{\theta }_0}{(X)}^{\top }t\right|\ge {\delta }_1|D_{{\theta }_0}(X)=0\right]\right\}\times \sqrt{n}\Vert {\widehat{\theta }}_n-{\theta }_0\Vert f_0{r}_n+O_P(1)+o_P\left(1+\sqrt{n}\Vert {\widehat{\theta }}_n-{\theta }_0\Vert \right) \\ \le -{\delta }_2{\delta }_1^2\left(\frac{\exp \left({\delta }_1\right)-1}{\exp \left({\delta }_1\right)+1}\right)\sqrt{n}\Vert {\widehat{\theta }}_n-{\theta }_0\Vert \left\{1+o_P(1)\right\}+O_P(1)+o_P\left(1+\sqrt{n}\Vert {\widehat{\theta }}_n-{\theta }_0\Vert \right), \end{gathered}$$

where r_n = 1 + o_P (1). This implies that $\sqrt{n}\Vert {\widehat{\theta }}_n-{\theta }_0\Vert =O_P(1)$. From the above calculations, and the fact that the arg min or arg max of a function t ↦ g(t) does not change if we add a term that is invariant in t, we obtain that ${\widehat{\theta }}_n=\text{arg\hspace{0.17em}}{\text{min}}_{t\in {ℝ}^d}{M}_n(t)$, where

$$M_n(t)=n^{-1/2}\sum _{i=1}^n{X}_i^{\top }{\beta }_0\left[1\left\{A_i={\widehat{d}}_{t,n}(X)\right\}-1\left\{A_i=d_{{\theta }_0}^{\text{opt}}(X)\right\}\right]+v_n\left(t,{\beta }_{*}\right)/2.$$

Let $M(u)={\beta }_0^{\top }{k}_0\left(Z_Y,Z_Z,u\right)$ and $U=\text{arg\hspace{0.17em}}{\text{min}}_{u\in {ℝ}^d}M(u)$. We will show that $M_n({\theta }_0+u/\sqrt{n})⇝M(u)$ in ℓ^∞(K) for any compact subset K of ${ℝ}^d$. Then, it will follow from the argmax Theorem (See chapter 14 of Kosorok, 2008) that ${\tilde{U}}_n⇝U$, where ${\tilde{U}}_n=\text{arg\hspace{0.17em}}{\text{min}}_{u\in {ℝ}^d}{M}_n\left({\theta }_0+u/\sqrt{n}\right)$. Let $h_n(u)={\theta }_0+u/\sqrt{n}$. Similar arguments along with Assumptions 11 and 10, yield that, for any compact $K\subset {ℝ}^d$,

$$\begin{gathered} \underset{u\in K}{\text{arg\hspace{0.17em}min}}{M}_n\left\{h_n(u)\right\} \\ =\underset{u\in K}{\text{arg\hspace{0.17em}min}}{n}^{1/2}\mathbb{E}\left(X^{\top }{\beta }_0\left[1\left\{A={\widehat{d}}_{h_n(u),n}(X)\right\}-1\left\{A=d_{{\theta }_0}^{\text{opt}}(X)\right\}\right]\right)+o_P(1) \\ =\underset{u\in K}{\text{arg\hspace{0.17em}min}}{n}^{1/2}\mathbb{E}\{X^{\top }{\beta }_0\{2P_{{\beta }_0}(X)-1\}(1[-\left\{{\widehat{D}}_{h_n(u),n}(X)-D_{{\theta }_0}(X)\right\}\le D_{{\theta }_0}(X)<0]+1\left[0\le D_{{\theta }_0}(X)<-\left\{{\widehat{D}}_{h_n(u),n}(X)-D_{{\theta }_0}(X)\right\}\right])\}+o_P(1) \\ =\underset{u\in K}{\text{arg\hspace{0.17em}min}}\mathbb{E}[X^{\top }{\beta }_0\left\{2P_{{\beta }_0}(X)-1\right\}\left|\sqrt{n}\left\{{\widehat{D}}_{h_n(u),n}(X)-D_{{\theta }_0}(X)\right\}\right||D_{{\theta }_0}(X)=0]f_0+o_P(1), \end{gathered}$$

However,

$$\sqrt{n}\left\{{\widehat{D}}_{h_n(u),n}(X)-D_{{\theta }_0}(X)\right\}⇝\omega \left(X;{\theta }_0\right){\varphi }_Y{(X)}^{\top }{Z}_Y+\left\{1-\omega \left(X;{\theta }_0\right)\right\}{\varphi }_Z{(X)}^{\top }{Z}_Z+R_0(X){\dot{\omega }}_{{\theta }_0}{(X)}^{\top }u$$

uniformly over $\mathcal{X}$ when X has its conditional distribution given $D_{{\theta }_0}(X)=0$. By applying continuous mapping theorem, this implies that $M_n(\theta +u/\sqrt{n})⇝M(u)$ in ℓ^∞(K) as desired and thus ${\tilde{U}}_n⇝U$. It is straightforward to verify the remaining conclusions of the theorem using previous arguments.■

Proof [Proof of Theorem 14] Using the assumptions, the fact that both $\sqrt{n}\left({\widehat{\theta }}_n-{\theta }_0\right)=O_P(1)$ and $\sqrt{n}\left({\widehat{\beta }}_n-{\beta }_0\right)=O_P(1)$, and standard arguments, we obtain that, for any compact $K_1\subset {ℝ}^q$, ${\text{sup}}_{{\left(Z_Y^{\top },Z_Z^{\top }\right)}^{\top }\in K_1}{\Vert {\tilde{T}}_n\left\{x,{\tilde{Z}}_Y\left(Z_Y,Z_Z\right),{\tilde{Z}}_Z\left(Z_Y,Z_Z\right)\right\}-T_0\left(x,Z_Y,Z_Z\right)\Vert }_{\mathcal{X}}=O_P(1)$, where

$$\left(\begin{array}{c}{\tilde{Z}}_Y\left(Z_Y,Z_Z\right)\\ {\tilde{Z}}_Z\left(Z_Y,Z_Z\right)\end{array}\right)={\widehat{\Sigma }}_1^{1/2}{\Sigma }_1^{-1/2}\left(\begin{array}{c}{Z}_Y\\ Z_Z\end{array}\right),$$

Σ₁ is the upper left q × q block of Σ₀, and also T₀(x, Z_Y, Z_Z) = ω(x; θ₀)ϕ_Y (x)^⊤Z_Y + {1 − ω(x; θ₀)}ϕ_Z(x)^⊤Z_Z. Furthermore,

$$\begin{gathered} {\Vert \left\{{\widehat{R}}_{Y,n}(x)-{\widehat{R}}_{Z,n}(x)\right\}{\dot{\omega }}_{{\widehat{\theta }}_n}{(x)}^{\top }u-\left\{R_Y(x)-R_Z(x)\right\}{\dot{\omega }}_{{\theta }_0}{(x)}^{\top }u\Vert }_{\mathcal{X}} \\ \le {\Vert \Vert \left\{{\widehat{R}}_{Y,n}(x)-{\widehat{R}}_{Z,n}(x)\right\}{\dot{\omega }}_{{\widehat{\theta }}_n}(x)-\left\{R_Y(x)-R_Z(x)\right\}{\dot{\omega }}_{{\theta }_0}(x)\Vert \Vert }_{\mathcal{X}}\cdot \Vert u\Vert \\ =O_P\left(n^{-1/2}\right)\Vert u\Vert \end{gathered}$$

${\Vert {\widehat{D}}_{{\widehat{\theta }}_n,n}(x)-D_{{\theta }_0}(x)\Vert }_{\mathcal{X}}=O_P\left(n^{-1/2}\right)$, and ${\Vert P_{{\widehat{\beta }}_n}(x)-P_{{\beta }_0}(x)\Vert }_{\mathcal{X}}=O_P\left(n^{-1/2}\right)$. Thus,

$$\underset{{\left(Z_Y^{\top },Z_Z^{\top }\right)}^{\top }\in K_1}{\text{sup}}{\mathbb{E}}_n\left[\Vert X\Vert \cdot \left|\left\{2P_{{\widehat{\beta }}_n}(X)-1\right\}{\tilde{T}}_n\left\{X,{\tilde{Z}}_Y\left(Z_Y,Z_Z\right),{\tilde{Z}}_Z\left(Z_Y,Z_Z\right)\right\}\right|\frac{1}{h_n}{\varphi }_0\left\{\frac{{\widehat{D}}_{{\widehat{\theta }}_n,n}(X)}{h_n}\right\}\right]\le O_P(1){\mathbb{E}}_n\left[\frac{1}{h_n}{\varphi }_0\left\{\frac{{\widehat{D}}_{{\widehat{\theta }}_n,n}(X)}{h_n}\right\}\right].$$ (9)

However,

$$\begin{gathered} {\mathbb{E}}_n\left(\frac{1}{h_n}\left[{\varphi }_0\left\{\frac{{\widehat{D}}_{{\widehat{\theta }}_n,n}(X)}{h_n}\right\}-{\varphi }_0\left\{\frac{D_{{\theta }_0}(X)}{h_n}\right\}\right]\right) \\ ={\mathbb{E}}_n\left[\frac{1}{h_n^3}{\int }_0^1\left\{(1-s)D_{{\theta }_0}(X)+s{\widehat{D}}_{{\widehat{\theta }}_n,n}(X)\right\}{\varphi }_0\left\{\frac{(1-s)D_{{\theta }_0}(X)+s{\widehat{D}}_{{\widehat{\theta }}_n,n}(X)}{h_n}\right\}\cdot \text{d}s\left\{{\widehat{D}}_{{\widehat{\theta }}_n,n}(X)-D_{{\theta }_0}(X)\right\}\right] \\ =O_P\left(\frac{1}{h_n^3{n}^{1/2}}\right)\cdot {\mathbb{E}}_n\left[{\int }_0^1\left\{(1-s)D_{{\theta }_0}(X)+s{\widehat{D}}_{{\widehat{\theta }}_n,n}(X)\right\}{\varphi }_0\left\{\frac{(1-s)D_{{\theta }_0}(X)+s{\widehat{D}}_{{\widehat{\theta }}_n,n}(X)}{h_n}\right\}\text{d}s\right] \\ =O_P\left(\frac{1}{h_n^3{n}^{1/2}}\right)O_P\left(h_n\right)=O_P\left(\frac{1}{h_n^2{n}^{1/2}}\right)=o_P(1), \end{gathered}$$

since $\left|u{\varphi }_0(u)\right|\le {(2\pi )}^{-1/2}{e}^{-1}<\infty$. Now, since $\mathbb{E}\left[h_n^{-1}{\varphi }_0\left\{D_{{\theta }_0}(X)/h_n\right\}\right]\stackrel{P}{\to }{f}_0$, we have that (9) is equal to O_P (1). Thus, if $\Vert u_n\Vert \to \infty$,

$${\widehat{\beta }}_n^{\top }{\tilde{k}}_n\left({\tilde{Z}}_Y,{\tilde{Z}}_Z,u_n\right)\ge {\mathbb{E}}_n\left[{\widehat{\beta }}_n^{\top }X\left\{2P_{{\widehat{\beta }}_n}(X)-1\right\}\cdot \left|\left\{{\widehat{R}}_{Y,n}(X)-{\widehat{R}}_{Z,n}(X)\right\}{\dot{\omega }}_{{\widehat{\theta }}_n}{(X)}^{\top }{u}_n\right|\cdot \frac{1}{h_n}{\varphi }_0\left\{\frac{{\widehat{D}}_{{\widehat{\theta }}_n,n}(X)}{h_n}\right\}\right]-O_P(1),$$ (10)

where the O_P (1) is uniform over K₁. Thus, up to the O_P (1) added on the right-hand side,

$$\begin{gathered} (10)\ge \Vert u_n\Vert \cdot \underset{t\in S^d}{\text{inf}}{\mathbb{E}}_n\left[{\widehat{\beta }}_n^{\top }X\left\{2P_{{\widehat{\beta }}_n}(X)-1\right\}\left|\left\{{\widehat{R}}_{Y,n}(X)-{\widehat{R}}_{Z,n}(X)\right\}{\dot{\omega }}_{{\widehat{\theta }}_n}{(X)}^{\top }t\right|\frac{1}{h_n}{\varphi }_0\left\{\frac{{\widehat{D}}_{{\widehat{\theta }}_n,n}(X)}{h_n}\right\}\right] \\ \ge u_n\left(o_P(1)+\underset{t\in S^d}{\text{inf}}\mathbb{E}\left[{\beta }_0^{\top }X\left\{2P_{{\beta }_0}(X)-1\right\}\left|\left\{R_Y(X)-R_Z(X)\right\}{\dot{\omega }}_{{\theta }_0}{(X)}^{\top }t\right|\frac{1}{h_n}{\varphi }_0\left\{\frac{D_{{\theta }_0}(X)}{h_n}\right\}\right]\right) \\ =\Vert u_n\Vert \cdot \left(o_P(1)+\underset{t\in S^d}{\text{inf}}\mathbb{E}\left[{\beta }_0^{\top }X\left\{2P_{{\beta }_0}(X)-1\right\}\left|\left\{R_Y(X)-R_Z(X)\right\}{\dot{\omega }}_{{\theta }_0}{(X)}^{\top }t\right||D_{{\theta }_0}(X)=0\right]f_0\right) \\ \ge \Vert u_n\Vert \left[o_P(1)+{\delta }_2{\delta }_1^2\left\{\frac{\text{exp}\left({\delta }_1\right)-1}{\text{exp}\left({\delta }_1\right)+1}\right\}\right], \end{gathered}$$

with the expectation over X. Let ${\widehat{U}}_n\left(Z_Y,Z_Z\right)=\text{arg\hspace{0.17em}}{\text{min}}_{u\in {ℝ}^d}{\widehat{\beta }}_n^{\top }{\tilde{k}}_n\left\{{\tilde{Z}}_Y\left(Z_Y,Z_Z\right),{\tilde{Z}}_Z\left(Z_Y,Z_Z\right),u\right\}$, where, if the arg min set has more than one element, one can be chosen randomly or algorithmically. Since the O_P (1) above is uniform over K₁, we conclude that

$$\underset{{\left(Z_Y^{\top },Z_Z^{\top }\right)}^{\top }\in K_1}{\text{sup}}\Vert {\widehat{U}}_n\left(Z_Y,Z_Z\right)\Vert =O_P(1).$$ (11)

Now, let K₂ be any compact subset of ${ℝ}^d$. Previous and standard arguments give us that ${\text{sup}}_{{\left(Z_Y^{\top },Z_Z^{\top }\right)}^{\top }\in K_1}{\text{sup}}_{u\in K_2}\Vert {\tilde{k}}_n\left\{{\tilde{Z}}_Y\left(Z_Y,Z_Z\right),{\tilde{Z}}_Z\left(Z_Y,Z_Z\right),u\right\}-k_0\left(Z_Y,Z_Z,u\right)\Vert =o_P(1)$. Thus, we also have that

$$\underset{{\left(Z_Y^{\top },Z_Z^{\top }\right)}^{\top }\in K_1}{\text{sup}}\underset{u\in K_2}{\text{sup}}\Vert {\widehat{\beta }}_n^{\top }{\tilde{k}}_n\left\{{\tilde{Z}}_Y\left(Z_Y,Z_Z\right),{\tilde{Z}}_Z\left(Z_Y,Z_Z\right),u\right\}-{\beta }_0^{\top }{k}_0\left(Z_Y,Z_Z,u\right)\Vert =o_P(1).$$ (12)

Define $U_0\left(Z_Y,Z_Z\right)=\text{arg\hspace{0.17em}}{\text{max}}_{u\in {ℝ}^d}{\beta }_0^{\top }{k}_0\left(Z_Y,Z_Z,u\right)$. Previous arguments yield that

$$\underset{{\left(Z_Y^{\top },Z_Z^{\top }\right)}^{\top }\in K_1}{\text{sup}}\Vert U_0\left(Z_Y,Z_Z\right)\Vert =O(1).$$ (13)

By Assumption 11, the arg min for each ${\left(Z_Y^{\top },Z_Z^{\top }\right)}^{\top }\in K_1$ is unique. Fix ϵ > 0. By (11), there exists an m₂ < ∞ such that $\text{Pr}\left({\text{sup}}_{{\left(Z_Y^{\top },Z_Z^{\top }\right)}^{\top }\in K_1}\Vert {\widehat{U}}_n\left(Z_Y,Z_Z\right)\Vert <m_2\right)\ge 1-ϵ$ for all n large enough. By (13), we can enlarge m₂ such that

$${\text{sup}}_{{\left(Z_Y^{\top },Z_Z^{\top }\right)}^{\top }\in K_1}\Vert U_0\left(Z_Y,Z_Z\right)\Vert <m_2<\infty .$$

We can also find an m₁ < ∞ such that $K_1\subset K_{m_1}^q$ as defined in Corollary 17. It is straightforward to show that (1) and (3) in Corollary 17 are satisfied by $f(Z,u)={\beta }_0^{\top }{k}_0\left(Z_Y,Z_Z,u\right)$, where $Z={\left(Z_Y^{\top },Z_Z^{\top }\right)}^{\top }$. Let $f_n(Z,u)={\widehat{\beta }}_n^{\top }{\tilde{k}}_n\left(Z_Y,Z_Z,u\right)$. Standard arguments and the given assumptions yield that there exists a w₁ < ∞ such that ${\text{sup}}_{Z\in K_{m_1}^q}{\text{sup}}_{u\in K_{m_2}^d}\left|f_n(Z,u)\right|<w_1$ almost surely and

$$\underset{Z_1,Z_2\in K_{m1}^q:\Vert Z_1-Z_2\Vert <\delta }{\text{sup}}{\Vert f_n\left(Z_1,u\right)-f_n\left(Z_2,u\right)\Vert }_{K_{m_2}^d}<w_1\delta$$

for all δ > 0 and all n ≥ 1 almost surely. Every subsequence in (12) has a further subsequence n″ on which the convergence in probability to zero can be replaced with almost sure convergence. Thus, (2) and (4) of Corollary 17 apply, using the fact that minimizing is equivalent to maximizing after a change in sign. Setting ${\widehat{U}}_n^{*}\left(Z_Y,Z_Z\right)=\text{arg\hspace{0.17em}}{\text{min}}_{u\in K_{m_2}^d}{\widehat{\beta }}_n^{\top }{\tilde{k}}_n\left\{{\tilde{Z}}_Y\left(Z_Y,Z_Z\right),{\tilde{Z}}_Z\left(Z_Y,Z_Z\right),u\right\}$, Corollary 17 now yields that

$$\underset{{\left(Z_Y^{\top },Z_Z^{\top }\right)}^{\top }\in K_{m_1}^q}{\text{sup}}\Vert {\widehat{U}}_{n^{″}}^{*}\left(Z_Y,Z_Z\right)-U_0\left(Z_Y,Z_Z\right)\Vert \to 0$$

almost surely. Since this is true for every subsequence, we have that

$$\underset{{\left(Z_Y^{\top },Z_Z^{\top }\right)}^{\top }\in K_{m_1}^q}{\text{sup}}\Vert {\widehat{U}}_n^{*}\left(Z_Y,Z_Z\right)-U_0\left(Z_Y,Z_Z\right)\Vert \stackrel{P}{\to }0$$

as n → ∞. Note that, on K₂, ${\widehat{U}}_n^{*}\left(Z_Y,Z_Z\right)={\widehat{U}}_n\left(Z_Y,Z_Z\right)$ for all ${\left(Z_Y^{\top },Z_Z^{\top }\right)}^{\top }\in K_{m_1}^q$. Hence,

$$\begin{gathered} \underset{n\to \infty }{\text{lim\hspace{0.17em}sup}}\text{Pr\hspace{0.17em}}\left\{\underset{{\left(Z_Y^{\top },Z_Z^{\top }\right)}^{\top }\in K_{m_1}^q}{\text{sup}}\Vert {\widehat{U}}_n\left(Z_Y,Z_Z\right)-U_0\left(Z_Y,Z_Z\right)\Vert >ϵ\right\} \\ \le \underset{n\to \infty }{\text{lim\hspace{0.17em}sup\hspace{0.17em}}}\left[\text{Pr\hspace{0.17em}}\left\{{\widehat{U}}_n\left(Z_Y,Z_Z\right)\in K_2,\underset{{\left(Z_Y^{\top },Z_Z^{\top }\right)}^{\top }\in K_{m_1}^q}{\text{sup}}\Vert {\widehat{U}}_n^{*}\left(Z_Y,Z_Z\right)-U_0\left(Z_Y,Z_Z\right)\Vert \ge ϵ\right\}+\text{Pr}\left\{{\widehat{U}}_n\left(Z_Y,Z_Z\right)\in K_2^c\right\}\right] \\ \le ϵ. \end{gathered}$$

Since ϵ was arbitrary, we obtain that

$$\underset{{\left(Z_Y^{\top },Z_Z^{\top }\right)}^{\top }\in K_{m_1}^q}{\text{sup}}\Vert {\widehat{U}}_n\left(Z_Y,Z_Z\right)-U_0\left(Z_Y,Z_Z\right)\Vert =o_P(1).$$

Let $B_L(\mathbb{B})$ be the space of all Lipschitz continuous functions mapping $\mathbb{B}\to ℝ$ which are bounded by 1 and which have Lipschitz constant 1. Let ${\mathbb{E}}_Z$ be expectation with respect to $Z_0^{*}={\left(Z_Y^{*},Z_Z^{*},Z_A^{*\top }\right)}^{\top }~N\left(0,{\Sigma }_0\right)$. Let $B_0\left(Z_0^{*}\right)=I_0^{-1}\left[Z_A^{*}-k_0\left\{Z_Y^{*},Z_Z^{*},{\tilde{U}}_n\left(Z_0^{*}\right)\right\}\right]$ and let $f\in B_L\left({ℝ}^{d+p}\right)$. Then, using ${\tilde{U}}_n$ and ${\tilde{B}}_n$ as defined in the statement of the theorem,

$$\begin{gathered} \left|{\mathbb{E}}_Z\left[f\left\{{\tilde{U}}_n\left(Z_0^{*}\right),{\tilde{B}}_n\left(Z_0^{*}\right)\right\}-f\left\{U_0\left(Z_0^{*}\right),B_0\left(Z_0^{*}\right)\right\}\right]\right| \\ \le \left|{\mathbb{E}}_Z\left[f\left\{{\tilde{U}}_n\left(Z_0^{*}\right),{\tilde{B}}_n\left(Z_0^{*}\right)\right\}-f\left\{U_0\left(Z_0^{*}\right),{\tilde{B}}_n\left(Z_0^{*}\right)\right\}\right]\right|+\left|{\mathbb{E}}_Z\left[f\left\{U_0\left(Z_0^{*}\right),{\tilde{B}}_n\left(Z_0^{*}\right)\right\}-f\left\{U_0\left(Z_0^{*}\right),B_0\left(Z_0^{*}\right)\right\}\right]\right| \\ =\left|{\mathbb{E}}_Z\left[f_1\left\{{\tilde{U}}_n\left(Z_0^{*}\right)\right\}-f_1\left\{U_0\left(Z_0^{*}\right)\right\}\right]\right|+\left|{\mathbb{E}}_Z\left[f_2\left\{{\tilde{B}}_n\left(Z_0^{*}\right)\right\}-f_2\left\{B_0\left(Z_0^{*}\right)\right\}\right]\right| \end{gathered}$$

for some other $f_1\in B_L\left({ℝ}^d\right)$ and $f_2\in B_L\left({ℝ}^p\right)$. Hence,

$$\begin{gathered} \underset{f\in B_L\left({ℝ}^{d+p}\right)}{\text{sup}}\left|{\mathbb{E}}_{Z}f\left\{{\tilde{U}}_n\left(Z_0^{*}\right),{\tilde{B}}_n\left(Z_0^{*}\right)\right\}-{\mathbb{E}}_{Z}f\left\{U_0\left(Z_0^{*}\right),B_0\left(Z_0^{*}\right)\right\}\right| \\ \le \underset{f\in B_L\left({ℝ}^d\right)}{\text{sup}}\left|{\mathbb{E}}_{Z}f\left\{{\tilde{U}}_n\left(Z_0^{*}\right)\right\}-{\mathbb{E}}_{Z}f\left\{U_0\left(Z_0^{*}\right)\right\}\right|+\underset{f\in B_L\left({ℝ}^p\right)}{\text{sup}}\left|{\mathbb{E}}_{Z}f\left\{{\tilde{B}}_n\left(Z_0^{*}\right)\right\}-{\mathbb{E}}_{Z}f\left\{B_0\left(Z_0^{*}\right)\right\}\right| \\ =A_n+B_n, \end{gathered}$$

where we define both $A_n={\text{sup}}_{f\in B_L\left({ℝ}^d\right)}\left|{\mathbb{E}}_{Z}f\left\{{\tilde{U}}_n\left(Z_0^{*}\right)\right\}-{\mathbb{E}}_{Z}f\left\{U_0\left(Z_0^{*}\right)\right\}\right|$ and

$$B_n={\text{sup}}_{f\in B_L\left({ℝ}^p\right)}\left|{\mathbb{E}}_{Z}f\left\{{\tilde{B}}_n\left(Z_0^{*}\right)\right\}-{\mathbb{E}}_{Z}f\left\{B_0\left(Z_0^{*}\right)\right\}\right|.$$

Fix some compact $K_1\subset {ℝ}^q$ such that $\text{Pr}\left\{{\left(Z_Y^{*\top },Z_Z^{*}\right)}^{\top }\in K_1\right\}\ge 1-ϵ$. Then,

$$\begin{gathered} \underset{f\in B_L\left({ℝ}^d\right)}{\text{sup}}\left|{\mathbb{E}}_Z\left[f\left\{{\tilde{U}}_n\left(Z_0^{*}\right)\right\}-{\mathbb{E}}_{Z}f\left\{U_0\left(Z_0^{*}\right)\right\}\right]\right|\le \underset{f\in B_L\left({ℝ}^d\right)}{\text{sup}}\left|{\mathbb{E}}_{Z}1\left(Z_0^{*}\in K_1\right)f\left\{{\tilde{U}}_n\left(Z_0^{*}\right)\right\}-{\mathbb{E}}_{Z}1\left(Z_0^{*}\in K_1\right)f\left\{U_0\left(Z_0^{*}\right)\right\}\right|+2{\mathbb{E}}_{Z}1\left(Z_0^{*}\in K_1\right) \\ =o_P(1)+2ϵ, \end{gathered}$$

which implies that A_n = o_P (1) since was arbitrary. For $K_2\subset {ℝ}^{q+p}$ such that $\text{Pr}\left(Z_0^{*}\in K_2\right)\ge 1-ϵ$, previous arguments yield that

$${\text{sup}}_{Z_0^{*}\in K_2}\Vert {\tilde{B}}_n\left(Z_0^{*}\right)-B_0\left(Z_0^{*}\right)\Vert =o_P(1).$$

As before, we can argue that B_n = o_P (1) + 2ϵ, which implies that B_n = o_P (1) since ϵ was arbitrary. The result follows.■

Theorem 16 Let H be a compact set in a metric space with metric d and let $\mathcal{F}$ be a compact subset of C[H] with respect to the uniform norm, ∥ · ∥_H. For each $f\in \mathcal{F}$, let $u(f)=\text{arg\hspace{0.17em}}{\text{max}}_{h\in H}f(h)$, where, when the arg max is not unique, we select one element of the arg max set either randomly or algorithmically. Suppose also that there exists a closed ${\mathcal{F}}_1\subset \mathcal{F}$ for which each $f\in {\mathcal{F}}_1$ has a unique maximum. Then,

$$\underset{\delta \downarrow 0}{\text{lim}}\underset{f\in {\mathcal{F}}_1}{\text{sup}}\underset{g\in \mathcal{F}:\Vert f-g{\Vert }_H<\delta }{\text{sup}}d\{u(f),u(g)\}=0.$$

Proof [Proof of Theorem 16] Fix ϵ > 0. For each $f\in {\mathcal{F}}_1$, there exists δ_f > 0 such that

$$\underset{h\in H\cap B_{ϵ}{\{u(f)\}}^c}{\text{sup}}f(h)<f\{u(f)\}-2{\delta }_f,$$

where B_ϵ(u) is the open d-ball of radius ϵ around u. This follows since the compactness of $\mathcal{F}$ ensures that all $f\in \mathcal{F}$ are continuous. Let $g\in \mathcal{F}$ be such that ∥f – g∥_H < δ_f. Then, f {u(g)} > g {u(g)} − δ_f ≥ g {u(f)} − δ_f > f {u(f)} − 2δ_f, which implies that d{u(g), u(f)} < ϵ. We have that ${\cup }_{f\in {\mathcal{F}}_1}\left\{g\in \mathcal{F}:\Vert g-f{\Vert }_H<{\delta }_f\right\}$ is an open cover of ${\mathcal{F}}_1$. Since ${\mathcal{F}}_1$ is compact, there exists a set ${\mathcal{F}}_1^{ϵ}$ such that ${\mathcal{F}}_1^{ϵ}$ is finite and

$${\cup }_{f\in {\mathcal{F}}_1^{ϵ}}\left\{g\in \mathcal{F}:\Vert g-f{\Vert }_H<{\delta }_f\right\}$$

still covers ${\mathcal{F}}_1$. Let $\left\{f_n\right\}\in \mathcal{F}$ and $\left\{g_n\right\}\in \mathcal{F}$ be sequences such that ∥f_n − g_n∥ → 0. By compactness, every subsequence has a further subsequence n″ such that $f_{n^{\prime \prime }}\to f_0\in {\mathcal{F}}_1$ and $g_{n^{\prime \prime }}\to g_0\in \mathcal{F}$ so that both f₀ and g₀ are in a set of the form $\left\{g\in \mathcal{F}:\Vert g-f{\Vert }_H<{\delta }_f\right\}$ for some $f\in {\mathcal{F}}_1^{ϵ}$. This implies that d{u(g₀), u(f₀)} < ϵ. Since the subsequence was arbitrary, we have that lim sup_n→∞d{u(g_n), u(f_n)} ≤ ϵ. Since ϵ was arbitrary, we now have that lim sup_n→∞d{u(g_n), u(f_n)} = 0 for any sequences $\left\{f_n\right\}\in {\mathcal{F}}_1$ and $\left\{g_n\right\}\in \mathcal{F}$ such that ∥f_n − g_n∥ → 0. This proves the result.■

Corollary 17 For m₁ < ∞, let $K_{m_1}^q=\left\{z\in {ℝ}^q:\Vert z\Vert \le m_1\right\}$. Let (z, u) ↦ f(z, u) and (z, u) ↦ f_n(z, u) be a fixed function and a sequence of functions, respectively, from $K_{m_1}^q\times {ℝ}^d$ to $ℝ$. Suppose there exists m₂ < ∞ such that for each $z\in K_{m_1}^q$, $u(z)=\text{arg\hspace{0.17em}}{\text{max}}_{u\in {ℝ}^d}f(z,u)<m_2$ and is uniquely defined. Suppose also that $u_n(z)=\text{arg\hspace{0.17em}}{\text{max}}_{u\in {ℝ}^d}{f}_n(z,u)<m_2$ for all n large enough, where we allow the arg max to be non-unique, but we randomly or algorithmically select one element from the arg max set. Define $K_{m_2}^d$ similarly to $K_{m_1}^q$ and assume that

1. ${\text{sup}}_{z\in K_{m_1}^q}{\text{sup}}_{u\in K_{m_2}^d}|f(z,u)|<\infty$

2. ${\text{lim\hspace{0.17em}sup}}_{n\to \infty }{\text{sup}}_{z\in K_{m_1}^q}{\text{sup}}_{u\in K_{m_2}^d}\left|f_n(z,u)\right|<\infty$

3. ${\text{lim}}_{\delta \downarrow 0}{\text{sup}}_{z_1,z_2\in K_{m_1}^q:\Vert z_1-z_2\Vert <\delta }{\Vert f\left(z_1,\cdot \right)-f\left(z_2,\cdot \right)\Vert }_{K_{m_2}^d}=0$

4. ${\text{lim}}_{\delta \downarrow 0}{\text{sup}}_{z_1,z_2\in K_{m_1}^q:\Vert z_1-z_2\Vert <\delta }{\Vert f_n\left(z_1,\cdot \right)-f_n\left(z_2,\cdot \right)\Vert }_{K_{m_2}^d}=0$

for all n large enough. Then, provided ${\text{sup}}_{z\in K_{m_1}^q}{\Vert f_n(z,\cdot )-f(z,\cdot )\Vert }_{K_{m_2}^d}\to 0$,

$$\underset{z\in K_{m_1}^q}{\text{sup}}\Vert u_n(z)-u(z)\Vert \to 0,$$

as n → ∞.

Proof [Corollary 17] By the Arzelà–Ascoli Theorem, there exists a compact K ⊂ C[H] for $H=K_{m_2}^d$, such that both f(z,·) ∈ K and f_n(z,·) ∈ K, for all $z\in K_{m_1}^q$ and all n large enough. If we let ${\mathcal{F}}_1=\left\{f(z,\cdot ):z\in K_{m_1}^q\right\}$, we can directly apply Theorem 16 to obtain that

$$\underset{\delta \downarrow 0}{\text{lim}}\underset{z\in K_{m_1}^q}{\text{sup}}\underset{g\in K:\Vert g-f(z,\cdot ){\Vert }_H<\delta }{\text{sup}}\Vert u(g)-u\{f(z,\cdot )\}{\Vert }_H=0.$$

Because ${\text{sup}}_{z\in K_{m_1}^q}{\Vert f_n(z,\cdot )-f(z,\cdot )\Vert }_{K_{m_2}^d}<\delta$ for all n large enough, the result follows.■

Proof [Remark 6.] We require estimation be restricted to parameters (β, θ) which satisfy P_β {A = d_θ(X)|X} > 1/2 with probability one. Suppose towards a contradiction that such a set of parameters also satisfies

$$P_{\beta }{\left(X\right)}^1\left\{A=d_{\theta }(X)\right\}{\left\{1-P_{\beta }(X)\right\}}^{1-1}\left\{A=d_{\theta }(X)\right\}=P_{{\beta }_0}{\left(X\right)}^{1\left\{A=d_{{\theta }_0}(X)\right\}}{\left\{1-P_{{\beta }_0}(X)\right\}}^{1-1\left\{A=d_{{\theta }_0}(X)\right\}}\text{ a}.\text{s}.,$$

where P_β(x) = expit(x^⊤β). For any (x, a) such that $a=d_{\theta }(x)\ne d_{{\theta }_0}(x)$ and it follows that $P_{\beta }(x)=P_{-{\beta }_0}(x)<P_{{\beta }_0}(x)=P_{-\beta }(x)$ which contradicts the restriction that the probability of recommending an optimal treatment exceeds 1/2. Thus, it follows that $d_{\theta }(x)=d_{{\theta }_0}(x)$ for almost all x. This in turn implies that $P\beta (x)=P_{{\beta }_0}(x)$ for almost all x from which it follows that β = β₀ provided $\mathbb{E}X{X}^{\top }$ is full rank.■

Appendix B: Three or More Outcomes

Assume the available data consist of (X_i, A_i, Y_1,i, …, Y_K,i), i = 1, …, n, which comprise n independent and identically distributed copies of (X, A, Y₁, …, Y_K), where X and A are as defined previously, and Y₁, …, Y_K are outcomes, with each outcome coded so that higher values are better. Assume there exists an unknown utility function U = u(Y₁, …, Y_K), where $u:{ℝ}^K\to ℝ$, such that u(y₁, …, y_K) quantifies the “goodness” of the outcome vector (y₁, …, y_K). As before, let U*(d) be the potential utility under a treatment regime d. Let $d_U^{\text{opt}}$ be the optimal regime for the utility defined by u, i.e., $\mathbb{E}{U}^{*}\left(d_U^{\text{opt}}\right)\ge \mathbb{E}{U}^{*}(d)$ for any regime d. The goal is to estimate the utility function and the associated optimal regime in the presence of more than two outcomes.

To begin, we assume that the utility function is constant across patients and takes the form $u\left(y_1,\dots ,y_K;\omega \right)={\sum }_{k=1}^{K-1}{\omega }_k{y}_k+\left(1-{\sum }_{k=1}^{K-1}{\omega }_k\right)y_K$, where ω = (ω₁, …, ω_K−1) is a vector of parameters with ${\sum }_{k=1}^{K-1}{\omega }_k\le 1$ and ω_k ≥ 0 for k = 1, …, K − 1. Thus, we assume that the utility function is a convex combination of the set of outcomes. Let $d_{\omega }^{\text{opt}}$ be the optimal regime for the utility defined by ω. Assume that there exists a true utility function defined by some ω₀ = (ω_1,0, …, ω_K−1,0) such that observed decisions were made with the intent to maximize U = u(y₁, …, y_K; ω₀). Further assume that treatment decisions in the observed data follow $\text{Pr}\left\{A=d_{{\omega }_0}^{\text{opt}}(x)|X=x\right\}=\text{expit}\left(x^{\top }\beta \right)$, where β is an unknown parameter.

Define $Q_{Y_k}(x,a)=\mathbb{E}\left(Y_k|X=x,A=a\right)$, for k = 1, …, K. Define also

$$Q_{\omega }(x,a)=\mathbb{E}\left\{u\left(Y_1,\dots ,Y_K;\omega \right)|X=x,A=a\right\}$$

and note that $Q_{\omega }(x,a)={\sum }_{k=1}^{K-1}{\omega }_k{Q}_{Y_k}(x,a)+\left(1-{\sum }_{k=1}^{K-1}{\omega }_k\right)Q_{Y_K}(x,a)$. The Q-functions for each outcome can be estimated from the observed data using regression models. Let ${\widehat{Q}}_{Y_k,n}(x,a)$ denote an estimator for $Q_{Y_k}(x,a)$. Then, an estimator for Q_ω(x, a) is ${\widehat{Q}}_{\omega ,n}(x,a)={\sum }_{k=1}^{K-1}{\omega }_k{\widehat{Q}}_{Y_k,n}(x,a)+\left(1-{\sum }_{k=1}^{K-1}{\omega }_k\right){\widehat{Q}}_{Y_K,n}(x,a)$. For any fixed ω, we can compute an estimator for $d_{\omega }^{\text{opt}}$ as ${\widehat{d}}_{\omega ,n}(x)=\text{arg\hspace{0.17em}}{\text{max}}_{a\in \mathcal{A}}{\widehat{Q}}_{\omega ,n}(x,a)$. The pseudo-likelihood is

$${\widehat{\mathcal{L}}}_n(\omega ,\beta )\propto \prod _{i=1}^n\frac{\text{exp}\left[X_i^{\top }\beta 1\left\{A_i={\widehat{d}}_{\omega ,n}\left(X_i\right)\right\}\right]}{1+\text{exp}\left(X_i^{\top }\beta \right)},$$ (14)

for a vector β and a vector ω = (ω₁, …, ω_K−1). For K = 2, this reduces to the formulation in Section 2. Estimators for β and ω₁, …, ω_K−1 can be obtained by maximizing the pseudo-likelihood in (14). Letting ${\widehat{\omega }}_n=({\widehat{\omega }}_{1,n},\dots ,{\widehat{\omega }}_{K-1,n}$ denote the maximum pseudo-likelihood estimator for ω, an estimator for the optimal regime is ${\widehat{d}}_{{\widehat{\omega }}_n,n}(x)=\text{arg\hspace{0.17em}}{\text{max}}_{a\in \mathcal{A}}{\widehat{Q}}_{{\widehat{\omega }}_n,n}(x,a)$.

When the number of outcomes is large, maximizing (2) using the grid search proposed in Section 2.1 is infeasible. However, we can use the Metropolis algorithm similar to that proposed for a patient-specific utility function. A patient-specific utility function can be accommodated by setting $u\left(y_1,\dots ,y_K;x,\theta \right)={\sum }_{k=1}^{K-1}\text{expit}\left(x^{\top }{\theta }_k\right)y_1+\left\{1-{\sum }_{k=1}^{K-1}\text{expit}\left(x^{\top }{\theta }_k\right)\right\}{y}_K$ for a vector $\theta ={\left({\theta }_1^{\top },\dots ,{\theta }_{K-1}^{\top }\right)}^{\top }$ and maximizing the pseudo-likelihood using the Metropolis algorithm.

To examine the performance of the proposed method in the presence of more than two outcomes, we use the following generative model. As before, let X = (X₁, …, X₅)^⊤ be independent normal random variables with mean 0 and standard deviation 0.5. Let ϵ₁, ϵ₂ and ϵ₃ be independent normal random variables with mean 0 and standard deviation 0.5. Given treatment assignment, outcomes are generated according to Y₁ = A(4X₁ − 2X₂ + 2) + ϵ₁, Y₂ = A(2X₁ − 4X₂ − 2) + ϵ₂, and Y₃ = 1 + A(X₁ + X₂ + 1)ϵ₃. For a fixed ω = (ω₁, ω₂) and fixed ρ ∈ [0, 1], treatment assignment is made according to $\text{Pr}\left\{A=d_{\omega }^{\text{opt}}(x)|X=x\right\}=\rho$.

We set ω₁ = 0.2, ω₂ = 0.4, and ρ = 0.6, 0.8. Table 12 contains parameter estimates averaged across 500 replications along with standard deviations (in parentheses) across replications. The error rate is the proportion of samples in a testing set that were assigned the optimal treatment by the estimated policy. Table 13 contains estimated values (calculated by generating an independent testing set following the same generative model but with decisions made according to each policy) of the optimal policy, a policy where the utility function is estimated (the proposed method), policies estimated to maximize each outcome individually, and standard of care. The proposed method results in values close to the true optimal in large samples and larger than maximizing each individual outcome across sample sizes.

Table 12:

Estimation results for simulations where utility and probability of optimal treatment are fixed, with three outcomes.

n	ρ	${\widehat{\omega }}_{1,n}$	${\widehat{\omega }}_{2,n}$	${\widehat{\rho }}_n$	Error rate
100	0.60	0.21 (0.16)	0.34 (0.20)	0.63 (0.07)	0.15 (0.11)
	0.80	0.21 (0.07)	0.42 (0.09)	0.81 (0.04)	0.04 (0.03)
200	0.60	0.21 (0.13)	0.40 (0.17)	0.62 (0.04)	0.11 (0.09)
	0.80	0.21 (0.04)	0.41 (0.06)	0.80 (0.03)	0.03 (0.02)
300	0.60	0.21 (0.12)	0.39 (0.16)	0.62 (0.03)	0.09 (0.08)
	0.80	0.20 (0.03)	0.41 (0.04)	0.80 (0.02)	0.02 (0.01)
500	0.60	0.21 (0.09)	0.41 (0.12)	0.61 (0.02)	0.06 (0.05)
	0.80	0.20 (0.02)	0.40 (0.03)	0.80 (0.02)	0.01 (0.01)

Table 13:

Value results for simulations where utility and probability of optimal treatment are fixed, with three outcomes.

n	ρ	Optimal	Estimated utility	Y1 only	Y2 only	Y3 only	Standard of care
100	0.60	1.38 (0.04)	1.28 (0.15)	1.09 (0.06)	1.00 (0.06)	0.62 (0.10)	0.59 (0.14)
	0.80	1.39 (0.04)	1.37 (0.06)	1.09 (0.06)	1.00 (0.06)	0.62 (0.11)	0.99 (0.13)
200	0.60	1.38 (0.04)	1.32 (0.12)	1.09 (0.06)	1.00 (0.06)	0.62 (0.08)	0.60 (0.10)
	0.80	1.39 (0.04)	1.38 (0.05)	1.09 (0.05)	1.00 (0.06)	0.62 (0.09)	0.98 (0.09)
300	0.60	1.38 (0.04)	1.34 (0.10)	1.09 (0.06)	1.00 (0.06)	0.63 (0.08)	0.60 (0.08)
	0.80	1.38 (0.04)	1.39 (0.05)	1.10 (0.06)	1.00 (0.06)	0.63 (0.08)	0.99 (0.07)
500	0.60	1.39 (0.04)	1.36 (0.07)	1.09 (0.05)	1.00 (0.06)	0.62 (0.07)	0.60 (0.06)
	0.80	1.39 (0.04)	1.38 (0.05)	1.10 (0.06)	1.00 (0.06)	0.63 (0.07)	0.99 (0.06)

Appendix C: Misspecified Model for the Utility Function

In this section, we demonstrate that the proposed method achieves reasonable performance even in the presence of a misspecified model for the utility function. Let X, Y, and Z be generated as above. Let the true underlying utility function be u(y, z; x, θ) = ω(x; θ)y + {1 − ω(x; θ)}z, where $\omega (x;\theta )=\text{expit}\left(1+x_1^2+x^{\top }{\theta }_0\right)$ with θ₀ = (−0. 5, 0, 0, 1, 0.5)^⊤. The misspecified model fit to estimate the utility function contained only an intercept, X₁, X₂, X₃, and X₄. Therefore, these simulations represent the setting where one important covariate and a squared term for one covariate are omitted from the model for the utility function. Treatment was assigned according to $\text{Pr}\left\{A=d_{\omega }^{\text{opt}}(x)|X=x\right\}=\text{expit}\left(0.5+x_1\right)$. Table 14 contains the estimated value when the model for the utility function is correctly specified and when the model is incorrectly specified, along with the value of the true optimal policy and the standard of care. The proposed method produces comparable results regardless of whether the utility function is misspecified or not.

Table 14:

First simulation results with a misspecified model for the utility function.

n	Optimal	Correct	Misspecified	Standard of Care
100	1.86 (0.07)	1.61 (0.21)	1.64 (0.20)	0.59 (0.23)
200	1.85 (0.07)	1.68 (0.16)	1.69 (0.17)	0.57 (0.16)
300	1.86 (0.07)	1.72 (0.13)	1.74 (0.13)	0.57 (0.13)
500	1.86 (0.07)	1.77 (0.10)	1.76 (0.11)	0.58 (0.10)

Table 15 contains results for the same model misspecification, but where the true utility function is $\omega (x;\theta )=\text{expit}\left(1+4x_1^2+x^{\top }{\theta }_0\right)$ with θ₀ = (−0.5, 0, 0, 1, 4)^⊤, i.e., the coefficients for the components that are left out of the misspecified model are larger. When the coefficients of the components left out of the utility function model are larger, the proposed method produces better results when the model is correctly specified. However, even in the presence of model misspecification, the proposed method produces results that improve upon the standard of care.

Table 15:

Second simulation results with a misspecified model for the utility function.

n	Optimal	Correct	Misspecified	Standard of Care
100	2.11 (0.08)	1.63 (0.28)	1.64 (0.23)	0.69 (0.26)
200	2.11 (0.08)	1.76 (0.22)	1.68 (0.18)	0.67 (0.18)
300	2.10 (0.07)	1.84 (0.19)	1.70 (0.16)	0.68 (0.15)
500	2.10 (0.08)	1.88 (0.16)	1.73 (0.15)	0.67 (0.12)

Appendix D: Misspecified Model for the Probability of Assigning the Optimal Treatment

Similar to the model with the misspecified utility function, the model with the misspecified probability of the optimal treatment resulted in the values that are comparable to the correct model. X, Y and Z were generated as above. Let $\omega (x;\theta )=\text{expit}\left(1+4x_1^2+x^T{\theta }_0\right)$, with θ₀ = (−0.5, 0, 0, 1, 0.5) and $\text{Pr}\left\{A=d_{\omega }^{\text{opt}}|X=x\right\}=\text{expit}\left(0.5+4x_1^2+x_1\right)$. For this simulation, the misspecified model for the probability of assigning the optimal treatment is assumed to not include $4X_1^2$. In Table 16, the estimated value of the model with the misspecified probability of assigning the optimal treatment is confirmed to be similar to the estimated value of the correct model.

Table 16:

Value results with a misspecified model for the probability of assigning the optimal treatment.

n	Optimal	Correct	Misspecified	Standard of Care
100	2.01 (0.07)	1.90 (0.14)	1.89 (0.20)	1.29 (0.23)
200	2.01 (0.08)	1.96 (0.09)	1.95 (0.12)	1.29 (0.16)
300	2.01 (0.08)	1.97 (0.09)	1.97 (0.10)	1.29 (0.13)
500	2.00 (0.08)	1.98 (0.08)	1.98 (0.08)	1.30 (0.10)

In the next simulation, let ω(x; θ) = expit(1 + x^Tθ₀) with θ₀ = (−0.5, 0, 0, 1, 0.5) and the true probability of assigning the optimal treatment is $\text{Pr}\left\{A=d_{\omega }^{\text{opt}}(x)|X=x\right\}=\text{expit}\left(0.5+x_1\right)$ when the misspecified model is assumed as a constant. In Table 17, it is noticeable that the difference between the two estimated values is similar as in Table 16. For all settings above, the estimated value of the correct model is similar to the estimated value of the models with both the misspecified utility functions and the misspecified probability of optimal treatment.

Table 17:

Value results with a misspecified model for the utility function.

n	Optimal	Correct	Misspecified	Standard of Care
100	1.76 (0.06)	1.66 (0.19)	1.67 (0.19)	0.51 (0.22)
200	1.76 (0.06)	1.71 (0.12)	1.71 (0.13)	0.49 (0.15)
300	1.76 (0.06)	1.72 (0.11)	1.74 (0.08)	0.50 (0.13)
500	1.76 (0.06)	1.75 (0.07)	1.75 (0.06)	0.50 (0.10)

Appendix E: Relationship between the probability of assigning the optimal treatment and the variance of the estimator of the utility

Table 18:

Estimates of ω across different ρs

n	ω╲ρ	0.2	0.3	0.35	0.4
100	0.25	0.25 (0.05)	0.25 (0.09)	0.27 (0.15)	0.33 (0.24)
	0.75	0.75 (0.05)	0.75 (0.10)	0.74 (0.14)	0.66 (0.24)
200	0.25	0.25 (0.03)	0.25 (0.05)	0.25 (0.08)	0.28 (0.16)
	0.75	0.75 (0.03)	0.75 (0.05)	0.75 (0.09)	0.73 (0.13)
300	0.25	0.25 (0.02)	0.25 (0.04)	0.25 (0.07)	0.26 (0.12)
	0.75	0.75 (0.02)	0.75 (0.04)	0.75 (0.07)	0.73 (0.13)
500	0.25	0.25 (0.01)	0.25 (0.02)	0.25 (0.04)	0.25 (0.08)
	0.75	0.75 (0.01)	0.75 (0.03)	0.75 (0.05)	0.76 (0.08)

Table 19:

Estimates of ω across different ρs

n	ω╲ρ	0.6	0.65	0.7	0.8
100	0.25	0.34 (0.24)	0.27 (0.14)	0.25 (0.10)	0.25 (0.05)
	0.75	0.66 (0.24)	0.73 (0.15)	0.74 (0.10)	0.75 (0.05)
200	0.25	0.28 (0.16)	0.25 (0.08)	0.25 (0.05)	0.25 (0.02)
	0.75	0.72 (0.16)	0.75 (0.08)	0.75 (0.05)	0.75 (0.03)
300	0.25	0.26 (0.11)	0.25 (0.07)	0.25 (0.04)	0.25 (0.02)
	0.75	0.74 (0.13)	0.75 (0.07)	0.75 (0.04)	0.75 (0.02)
500	0.25	0.25 (0.08)	0.25 (0.04)	0.25 (0.03)	0.25 (0.01)
	0.75	0.75 (0.08)	0.76 (0.05)	0.75 (0.02)	0.75 (0.01)

In this section, we explore the relationship between the probability of assigning the optimal treatment and the variance of the estimator of the utility, which was mentioned in Remark 15. We empirically examine how the standard error of the estimator of the utility changes as $\text{Pr}\left\{A=d_{\omega }^{\text{opt}}(x)|X=x\right\}$ varies. For simplicity, we focus on the fixed utility setting described in 4.1. Let X, Y and Z be generated as in 4.1. For each ρs, we estimate $\widehat{\omega }$ when ω = 0.25 and ω = 0.75.

Table 18 and Table 19 display $\widehat{\omega }$ and $s.e.(\widehat{\omega })$ across values of n, ω, and ρ. As ρ deviates from 0.5 (so that β is not close to 0), $\widehat{\omega }$ is closer to the true ω, and the standard error of $\widehat{\omega }$ decreases. Moreover, if ω₁ and ω₂ satisfy ω₁ = 1 − ω₂, their estimates and standard errors are similar as logit(ω₁) = −logit(ω₂).

Appendix F: Performance of Metropolis optimizer in the algorithm with the patient-specific utility.

Table 20:

Absolute difference and the Euclidean distance of the estimates of the patient utility and θ₀

n	Absolute diffrence-MH	Absolute difference-Unit ball	Distance-MH	Distance-Unit ball
100	1.49 (0.71)	1.66 (0.30)	0.85 (0.45)	0.86 (0.12)
200	1.30 (0.49)	1.60 (0.31)	0.72 (0.31)	0.84 (0.13)
300	1.19 (0.20)	1.63 (0.29)	0.63 (0.16)	0.86 (0.13)
500	1.15 (0.12)	1.63 (0.32)	0.60 (0.09)	0.85 (0.13)

In this section, we examine the performance of the Metropolis optimizer that is used in 2.2. We randomly select 10000 points in the unit ball around θ₀, and use the point that yields the best likelihood as the reference for the comparison. We define $\widehat{\theta }$ by Metropolis algorithm as ${\widehat{\theta }}_{\text{MH}}$, and a best point from a unit ball around θ₀ as ${\widehat{\theta }}_{\text{Uball }}$. We obtain the distance between θ₀ and ${\widehat{\theta }}_{\text{MH}}$ and the distance between θ₀ and ${\widehat{\theta }}_{\text{Uball}}$ for each of 500 replications.

In Table 20, absolute difference-MH was calculated as the mean of $\left|{\widehat{\theta }}_{\text{MH}}-{\theta }_0\right|$, and absolute difference-Unit ball was calculated as the mean of $\left|{\widehat{\theta }}_{\text{Uball }}-{\theta }_0\right|$ for 500 replications. Similarly, Distance-MH was calculated as the mean of $\sqrt{{\sum }_{j=1}^6{\left({\widehat{\theta }}_{\text{MH},j}-{\theta }_{0,j}\right)}^2}$, and Distance-Unit ball was calculated as the mean of $\sqrt{{\sum }_{i=1}^6{\left({\widehat{\theta }}_{\text{Uball },j}-{\theta }_{0,j}\right)}^2}$ for 500 replications. The standard errors of these distances are in the parentheses.

By Table 20, it is noticeable that both absolute difference and the Euclidean distance of ${\widehat{\theta }}_{\text{MH}}$ are smaller, and we could conclude that it is reasonable to use Metropolis algorithm when estimating a patient-specific utility.