Post-Contextual-Bandit Inference

Abstract

Contextual bandit algorithms are increasingly replacing non-adaptive A/B tests in e-commerce, healthcare, and policymaking because they can both improve outcomes for study participants and increase the chance of identifying good or even best policies. To support credible inference on novel interventions at the end of the study, nonetheless, we still want to construct valid confidence intervals on average treatment effects, subgroup effects, or value of new policies. The adaptive nature of the data collected by contextual bandit algorithms, however, makes this difficult: standard estimators are no longer asymptotically normally distributed and classic confidence intervals fail to provide correct coverage. While this has been addressed in non-contextual settings by using stabilized estimators, the contextual setting poses unique challenges that we tackle for the first time in this paper. We propose the Contextual Adaptive Doubly Robust (CADR) estimator, the first estimator for policy value that is asymptotically normal under contextual adaptive data collection. The main technical challenge in constructing CADR is designing adaptive and consistent conditional standard deviation estimators for stabilization. Extensive numerical experiments using 57 OpenML datasets demonstrate that confidence intervals based on CADR uniquely provide correct coverage.

1. Introduction

Contextual bandits, where personalized decisions are made sequentially and simultaneously with data collection, are increasingly used to address important decision-making problems where data is limited and/or expensive to collect, with applications in product recommendation [Li et al., 2010], revenue management [Kallus and Udell, 2020, Qiang and Bayati, 2016], and personalized medicine [Tewari and Murphy, 2017]. Adaptive experiments, whether based on bandit algorithms or Bayesian optimization, are increasingly being considered in place of classic randomized trials in order to improve both the outcomes for study participants and the chance of identifying the best treatment allocations [Athey et al., 2018, Quinn et al., 2019, Kasy and Sautmann, 2021, Bakshy et al., 2018].

But, at the end of the study, we still want to construct valid confidence intervals on average treatment effects, subgroup effects, or the value of new personalized interventions. Such confidence intervals are, for example, crucial for enabling credible inference on the presence or absence of improvement of novel policies. However, due to the adaptive nature of the data collection, unlike classic randomized trials, standard estimates and their confidence intervals actually fail to provide correct coverage, that is, contain the true parameter with the desired confidence probability (e.g., 95%). A variety of recent work has recognized this and offered remedies [Hadad et al., 2019, Luedtke and van der Laan, 2016], but only for the case of non-contextual adaptive data collection. Like classic confidence intervals, when data comes from a contextual bandit – or any other context-dependent adaptive data collection – these intervals also fail to provide correct coverage. In this paper, we propose the first asymptotically normal estimator for the value of a (possibly contextual) policy from context-dependent adaptively collected data. This asymptotic normality leads directly to the construction of valid confidence intervals.

Our estimator takes the form of a stabilized doubly robust estimator, that is, a weighted time average of an estimate of the so-called canonical gradient using plug in estimators for the outcome model, where each time point is inversely weighted by its estimated conditional standard deviation given the past. We term this the Contextual Adaptive Doubly Robust (CADR) estimator. We show that, given consistent conditional variance estimates which at each time point only depend on previous data, the CADR estimator is asymptotically normal, and as a result we can easily construct asymptotically valid confidence intervals. This normality is in fact robust to misspecifying the outcome model. A significant technical challenge is actually constructing such variance estimators. We resolve this using an adaptive variance estimator based on the importance-sampling ratio of current to past (adaptive) policies at each time point. We also show that we can reliably estimate outcome models from the adaptively-collected data so that we can plug them in. Extensive experiments using 57 OpenML datasets demonstrate the failure of previous approaches and the success of ours at constructing confidence intervals with correct coverage.

1.1. Problem Statement and Notation

The data.

Our data consists of a sequence of observations indexed t = 1, …, T comprising of context $X(t)\in \mathcal{X}$, action $A(t)\in \mathcal{A}$, and outcome $Y(t)\in \mathcal{Y}\subset ℝ$ generated by an adaptive experiment, such as a contextual bandit algorithm. Roughly, at each round t = 1, 2, …, T, an agent formed a contextual policy g_t(a | x) based on all past observations, then observed an independently drawn context vector X(t) ~ Q_0,X, carried out an action A(t) drawn from its current policy g_t(· | X(t)), and observed an outcome Y (t) ~ Q_0,Y (· | A(t), X(t)) depending only on the present context and action. The action and context measurable spaces $\mathcal{X}$, $\mathcal{A}$ are arbitrary, e.g., finite or continuous.

More formally, we let O(t) ≔ (X(t), A(t), Y (t)) and make the following assumptions about the sequence O(1), …, O(T) comprising our dataset. First, we assume X(t) is independent of all else given A(t) and has a time-independent marginal distribution that we denote by Q_0,X. Second, we assume A(t) is independent of all else given O(1), …, O(t − 1), X(t) and we set g_t(· | X(t)) to its (random) conditional distribution given O(1), …, O(t − 1), X(t). Third, we assume Y (t) is independent of all else given X(t), A(t) and has a time-independent conditional distribution given X(t) = x, A(t) = a that is denoted by Q_0,Y (· | A, X). The distributions Q_0,X and Q_0,Y are unknown, while the policies g_t(a | x) are known, as would be the case when running an adaptive experiment. To simplify presentation we endow $\mathcal{A}$ with a base measure ${\mu }_{\mathcal{A}}$ (e.g., counting for finite actions or Lebesgue for continuous actions) and identify policies g_t with conditional densities with respect to (w.r.t.) ${\mu }_{\mathcal{A}}$. In the case of K < ∞ actions, policies are maps from $\mathcal{X}$ to the K-simplex.

Note that, as the agent updates its policy based on already collected observations, g_t is a random O(1), …, O(t − 1)-measurable object. This is the major departure from the setting considered in other literature on off-policy evaluation, which only consider a fixed logging policy, g_t = g, that is independent of the data. See Section 1.2.

The target parameter.

We are interested in inference on a generalized average causal effect expressed as a functional of the unknown distributions above, Ψ₀ = Ψ(Q_0,X, Q_0,Y), where for any distributions Q_X, Q_Y, we define

$$\Psi \left(Q_X,Q_Y\right)≔\int y{Q}_X(dx)g(a\mid x)d{\mu }_{\mathcal{A}}(a)Q_Y(dy\mid a,x),$$

where $g^{*}(a\mid x):\mathcal{A}\times \mathcal{X}\to [-G,G]$ is a given fixed, bounded function. Two examples are: (a) when g* is a policy (conditional density), then Ψ₀ is its value; (b) when g* is the difference between two policies then Ψ₀ is the difference between their values. A prominent example of the latter is when $\mathcal{A}=\{+1,-1\}$ and g*(a | x) = a, which is known as the average treatment effect. If we include an indicator for x being in some set, then we get the subgroup effect.

Defining the conditional mean outcome,

$${\overline{Q}}_0(a,x)≔E_{Q_{0,Y}(\cdot \mid x,a)}[Y]=\int y{Q}_{0,Y}(dy\mid a,x),$$

we note that the target parameter only depends on Q_0,Y via ${\overline{Q}}_0$, so we also overload notation and write $\Psi \left(Q_X,\overline{Q}\right)=\int \overline{Q}(a,x)Q_X(dx)g(a\mid x)d{\mu }_{\mathcal{A}}(a)$ for any function $\overline{Q}:\mathcal{A}\times \mathcal{X}\to \mathcal{Y}$. Note that when $|\mathcal{A}|<\infty$ and ${\mu }_{\mathcal{A}}$ is the counting measure, the integral over a is a simple sum.

Canonical gradient.

We will make repeated use of the following function: for any conditional density (a, x) ↦ g(a | x), any probability distribution Q_X over the context space $\mathcal{X}$, and any function $\overline{Q}:\mathcal{A}\times \mathcal{X}$, we define the function $D^{\prime }(g,\overline{Q}):\mathcal{O}\to ℝ$ by

$$D^{\prime }(g,\overline{Q})(x,a,y)≔\frac{g^{*}(a\mid x)}{g(a\mid x)}(y-\overline{Q}(a,x))+\int \overline{Q}\left(a^{\prime },x\right)g^{*}\left(a^{\prime }\mid x\right)d{\mu }_{\mathcal{A}}\left(a^{\prime }\right).$$

Further, define $D\left(g,Q_X,\overline{Q}\right)=D^{\prime }\left(g,Q_X,\overline{Q}\right)-\Psi \left(Q_X,\overline{Q}\right)$, which coincides with the so-called canonical gradient of the target parameter Ψ w.r.t. the usual nonparametric statistical model comprising all joint distributions over $\mathcal{O}$ [van der Vaart, 2000, van der Laan and Robins, 2003].

Integration operator notation.

For any policy g and distributions Q_X, Q_Y, denote by P_Q,g the induced distribution on $\mathcal{O}$. For any function $f:\mathcal{O}\to ℝ$, we use the integration operator notation

$$P_{Q,g}f=\int f(x,a,y)Q_X(dx)g(a\mid x)d{\mu }_{\mathcal{A}}(a)Q_Y(dy\mid a,x),$$

that is, the expectation w.r.t. P_Q,g alone. Then, for example, for any O(1), …, O(s − 1)-measurable random function $f:\mathcal{O}\to ℝ$, we have that $P_{Q_0,g_s}f=E_{Q_0,g_s}[f(O(s))\mid O(1),\dots ,O(s-1)]$.

1.2. Related Literature and Challenges for Post-Contextual-Bandit Inference

Off-policy evaluation.

In non-adaptive settings, where g_t = g is fixed and does not depend on previous observations, common off-the shelf estimators for the mean outcome under g* include the Inverse Propensity Scoring (IPS) estimator [Beygelzimer and Langford, 2009, Li et al., 2011] and and the Doubly Robust (DR) estimator [Dudík et al., 2011, Robins et al., 1994]:

$${\widehat{\Psi }}^{\text{IPS}}≔\frac{1}{T}\sum _{t=1}^T{D}^{\prime }(g,0),{\widehat{\Psi }}^{\text{DR}}≔\frac{1}{T}\sum _{t=1}^T{D}^{\prime }(g,\widehat{\overline{Q}})$$

where $\widehat{\overline{Q}}$ is an estimator of the outcome model ${\overline{Q}}_0(a,x)$. If we use cross-fitting to estimate $\widehat{\overline{Q}}$ [Chernozhukov et al., 2018], then both the IPS and DR estimators are unbiased and asymptotically normal, permitting straightforward inference using Wald confidence intervals (i.e., ±1.96 of the estimated standard error). There also exist many variants of the IPS and DR estimators that, rather than plugging in the importance sampling (IS) ratios (g*/g_t)(A(t) | X(t)) and/or outcome-model estimators, instead choose them directly with the aim to minimize error [e.g. Kallus, 2018, Farajtabar et al., 2018, Thomas and Brunskill, 2016, Wang et al., 2017, Kallus and Uehara, 2019b].

Inference challenges in adaptive settings.

In the adaptive setting, it is easy to see that, if in the tth term for DR we use an outcome model ${\widehat{\overline{Q}}}_{t-1}$ fit using only the observations O(1), …, O(t − 1), then both the IPS and DR estimators both remain unbiased. However, neither generally converges to a normal distribution. One key difference between the non-adaptive and adaptive settings is that the IS ratios (g*/g_t)(A(t) | X(t)) can both diverge to infinity or converge to zero. As a result of this, the above two estimators may either be dominated by their first terms or their last terms. At a more theoretical level, this violates the classical condition of martingale central limit theorems that the conditional variance of the terms given previous observations stabilizes asymptotically.

Stabilized DR estimators in non-contextual settings.

The issue for inference due to instability of the DR estimator terms was recognized by Luedtke and van der Laan [2016] in another setting. They work in the non-adaptive setting but consider the problem of inferring the maximum mean outcome over all policies when the optimal policy is non-unique. Their proposal is a so-called stabilized estimator, in which each term is inversely weighted by an estimate of its conditional standard deviation given the previous terms. This stabilization trick has been also been reused for off-policy inference from non-contextual bandit data by Hadad et al. [2019], as the stabilized estimator remains asymptotically normal, permitting inference. In their non-contextual setting, an estimate of the conditional standard deviation of the terms can easily be obtained by the inverse square root propensities. In contrast, in our contextual setting, obtaining valid stabilization weights is more challenging and requires a construction involving adaptive training on past data.

1.3. Contributions

In this paper, we construct and analyze a stabilized estimator for policy evaluation from context-dependent adaptively collected data, such as the result of running a contextual bandit algorithm. This then immediately enables inference. After constructing a generic extension of the stabilization trick, the main technical challenge is to construct a sequence of estimators ${\widehat{\sigma }}_1,\dots ,{\widehat{\sigma }}_T$ of the conditional standard deviations that are both consistent and such that for each t, ${\widehat{\sigma }}_t$ only uses the previous data points O(1), …, O(t − 1). We show in extensive experiments across a large set of contextual bandit environments that our confidence intervals uniquely achieve close to nominal coverage.

2. Construction and Analysis of the Generic Contextual Stabilized Estimator

In this section, we give a generic construction of a stabilized estimator in our contextual and adaptive setting. That is, given generic plug-ins for outcome model and conditional standard deviation. We then provide conditions under which the estimator is asymptotically normal, as desired. To develop CADR, we will then proceed to construct appropriate plug in estimators in the proceeding sections.

2.1. Construction of the Estimator

Outcome and variance estimators.

Our estimator uses a sequence ${\left({\widehat{\overline{Q}}}_t\right)}_{t\ge 1}$ of estimators of the outcome model ${\overline{Q}}_0$, such that, for every t, ${\widehat{\overline{Q}}}_t$ is O(1), …, O(t)-measurable, that is, is trained using only the data up to time t. A key part of our estimator are the conditional variance estimators.

Additionally, we require estimates of the conditional standard deviation of the canonical gradient. Define

$${\sigma }_{0,t}≔{\sigma }_{0,t}\left(g_t\right)$$

where ${\sigma }_{0,t}^2(g)≔{\text{Var}}_{Q_0,g}\left(D^{\prime }\left(g,{\widehat{\overline{Q}}}_{t-1}\right)(O(t))\mid O(1),\dots ,O(t-1)\right)$.

Let ${\left({\widehat{\sigma }}_t\right)}_{t\ge 1}$ be a given sequence of estimates of σ_0,t such that ${\widehat{\sigma }}_t$ is O(1), …, O(t − 1)-measurable, that is, is estimated using only the data up to time t.

The generic form of the estimator.

The generic contextual stabilized estimator is then defined as:

$${\widehat{\Psi }}_T≔{\left(\frac{1}{T}\sum _{t=1}^T{\widehat{\sigma }}_t^{-1}\right)}^{-1}\frac{1}{T}\sum _{t=1}^T{\widehat{\sigma }}_t^{-1}{D}^{\prime }\left(g,{\widehat{\overline{Q}}}_{t-1}\right).$$ (1)

2.2. Asymptotic normality guarantees

We next characterize the asymptotic distribution of ${\widehat{\Psi }}_T$ under some assumptions.

Assumption 1 (Non degenerate efficiency bound). ${\text{inf}}_g{P}_{Q_0,g}{D}^2\left(g,{\overline{Q}}_0,Q_{0,X}\right)>0$.

Assumption 1 states that there is no fixed logging policy g such that the efficiency bound for estimation of $\Psi \left({\overline{Q}}_0,Q_{0,X}\right)$ in the nonparametric model, from i.i.d. draws of $P_{Q_0,g}$, is zero. If assumption 1 does not hold, there exists a logging policy g such that, if $O=(X,A,Y)~P_{Q_0,g}$, then (g*(A | X)/g(A | X))Y equals $\Psi \left({\overline{Q}}_0,Q_{0,X}\right)$ with probability 1. In other words, if assumption 1 does not hold, there exists a logging policy g such that $\Psi \left({\overline{Q}}_0,Q_{0,X}\right)$ can be estimated with no error with probability 1 from a single draw of $P_{Q_0,g}$. Thus, it is very lax. An easy sufficient condition for Assumption 1 is that the outcome model has nontrivial variance in that ${\text{Var}}_{Q_{0,X}}\left(\int \overline{Q}(a,X)g^{*}(a\mid x)d{\mu }_{\mathcal{A}}(a)\right)>0$.

Assumption 2 (Consistent standard deviation estimators.). ${\widehat{\sigma }}_t-{\sigma }_{0,t}\stackrel{t\to \infty }{\to }0$ almost surely.

In the next section we will proceed to construct specific estimators ${\widehat{\sigma }}_t$ that satisfy Assumption 2, leading to our proposed CADR estimator and confidence intervals.

Assumption 3 (Exploration rate). For any t ≥ 1, we have that ${\text{inf}}_{a\in \mathcal{A},x\in \mathcal{X}}{g}_t(a\mid x)\gtrsim t^{-1/2}$ almost surely.

Here, a_t ≳ b_t means that for some constant c > 0, we have a_t ≥ cb_t for all t ≥ 1. Assumption 3 requires that the exploration rate of the adaptive experiment does not decay too quickly.

Based on these assumptions, we have the following asymptotic normality result:

Theorem 1. Denote ${\Gamma }_T≔{\left(T^{-1}{\sum }_{t=1}^T{\widehat{\sigma }}_t^{-1}\right)}^{-1}$. Under Assumptions 1 to 3, it holds that ${\Gamma }_T^{-1}\sqrt{T}\left({\widehat{\Psi }}_T-{\Psi }_0\right)\stackrel{d}{\to }\mathcal{N}(0,1)$.

Remark 1. Theorem 1 does not require the outcome model estimator to converge at all. As we will see in Section 3, our conditional variance estimator does require that the outcome model converges to a fixed limit ${\overline{Q}}_1$, but this limit does not have to be the true outcome model ${\overline{Q}}_0$. In other words, consistency of the outcome model is not required at any point of our analysis.

3. Construction of the Conditional Variance Estimator and CADR

We now tackle the construction of ${\widehat{\sigma }}_t$ satisfying our assumptions; namely, they must be adaptively trained only on past data at each t and they must be consistent. Observe that ${\sigma }_{0,t}^2={\sigma }_0^2\left(g_t,{\widehat{\overline{Q}}}_{t-1}\right)$, where we define

$$\begin{gathered} {\sigma }_0^2(g,\overline{Q})≔{\Phi }_{0,1}(g,\overline{Q})-{\left({\Phi }_{0,2}(g,\overline{Q})\right)}^2, \\ {\Phi }_{0,i}(g,\overline{Q})≔P_{Q_0,g}{\left(D^{\prime }\right)}^i(g,\overline{Q}),i=1,2. \end{gathered}$$

Designing an O(1), …, O(t − 1)-measurable estimator of ${\sigma }_{0,t}^2$ presents several challenges. First, while we can only use observations O(1), …, O(t − 1) to estimate it, ${\sigma }_{0,t}^2$ is defined as a function of integrals w.r.t. $P_{Q_0,g_t}$, from which we have only one observation, namely O(t). Second, our estimation target ${\sigma }_{0,t}^2={\sigma }_0\left(g_t,{\widehat{\overline{Q}}}_{t-1}\right)$ is random as it depends on g_t and ${\widehat{\overline{Q}}}_t$. Third, g_t, ${\widehat{\overline{Q}}}_t$ depend on the same observations O(1), …, O(t − 1) that we have at our disposal to estimate ${\sigma }_{0,t}^2$.

Representation via importance sampling.

We can overcome the first difficulty via importance sampling, which allows us to write ${\Phi }_{0,i}(g,\overline{Q})$, i = 1, 2 as integrals w.r.t. $P_{Q_0,g_s}$, s = 1, …, t − 1, i.e., the conditional distributions of observations O(s), s = 1, …, t − 1 given their respective past. Namely, for any s ≥ 1, i = 1, 2, we have that

$${\Phi }_{0,i}(g,\overline{Q})=P_{Q_0,g_s}\frac{g}{g_s}{\left(D^{\prime }\right)}^i(g,\overline{Q}).$$ (2)

Dealing with the randomness of the estimation target.

We now turn to second challenge. Since ${\sigma }_{0,t}^2$ can be written in terms of ${\Phi }_{0,i}\left(g_t,{\widehat{\overline{Q}}}_{t-1}\right)$ for i = 1, 2, Eq. (2) suggests perhaps an approach based on sample averages of $\left(g_t/g_s\right){\left(D^{\prime }\right)}^i\left(g_t,{\widehat{\overline{Q}}}_{t-1}\right)$ over s. However, whenever s < t, the latter is an O(1), …, O(t − 1)-measurable function due to the dependence on g_t and ${\widehat{\overline{Q}}}_t$. Namely, $P_{Q_0,g_s}\left\{\left(g_t/g_s\right){\left(D^{\prime }\right)}^i\left(g_t,{\widehat{\overline{Q}}}_{t-1}\right)\right\}$ does not coincide in general with the conditional expectation $E_{Q_0,g_s}\left[\left(\left(g_t/g_s\right){\left(D^{\prime }\right)}^i\left(g_t,{\widehat{\overline{Q}}}_{t-1}\right)\right)(O(s))\mid \overline{O}(s-1)\right]$, as would arise from a sample average. We now look at solutions to overcome this difficulty, considering first ${\widehat{\overline{Q}}}_{t-1}$ and then g_t.

Dealing with the randomness of ${\widehat{\overline{Q}}}_{t-1}$.

We propose an estimator of ${\sigma }_0^2\left(g,{\widehat{\overline{Q}}}_{t-1}\right)$ for any fixed g. While requiring that ${\widehat{\overline{Q}}}_{t-1}$ converges to the true outcome regression function ${\overline{Q}}_0$ is a strong requirement, most reasonable estimators will at least converge to some fixed limit ${\overline{Q}}_1$. As a result, under an appropriate stochastic convergence condition on $\left({\widehat{\overline{Q}}}_{t-1}\right)t\ge 1$, ${\Phi }_{0,i}\left(g,{\widehat{\overline{Q}}}_{t-1}\right)$ can be reasonably approximated by the corresponding Cesaro averages, defined for i = 1, 2 as

$${\overline{\Phi }}_{0,i,t}(g)≔\frac{1}{t-1}\sum _{s=1}^{t-1}{\Phi }_{0,i}\left(g,{\widehat{\overline{Q}}}_{s-1}\right)=\frac{1}{t-1}\sum _{s=1}^t{E}_{Q_0,g_s}\left[\left(\left(g/g_s\right){\left(D^{\prime }\right)}^i\left(g,{\widehat{\overline{Q}}}_{s-1}\right)\right)(O(s))\mid \overline{O}(s-1)\right].$$

These are easy to estimate from the corresponding sample averages, defined for i = 1, 2 as

$${\widehat{\Phi }}_{i,t}(g)≔\frac{1}{t-1}\sum _{s=1}^t\left(\left(g/g_s\right){\left(D^{\prime }\right)}^i\left(g,{\widehat{\overline{Q}}}_{s-1}\right)\right)(O(s)),$$

since for each i = 1, 2, the difference ${\widehat{\Phi }}_{i,t}(g)-{\overline{\Phi }}_{0,i,t}(g)$ is the average of a martingale difference sequence (MDS). We then define our estimator of ${\sigma }_0^2\left(g,{\widehat{\overline{Q}}}_{t-1}\right)$ as

$${\widehat{\sigma }}_t(g)≔{\widehat{\Phi }}_{1,t}(g)-{\left({\widehat{\Phi }}_{2,t}(g)\right)}^2.$$ (3)

From fixed g to random g_t.

So far, we have proposed and justified the construction of ${\widehat{\sigma }}_t(g)$ as an estimator of ${\sigma }_{0,t}\left(g,{\widehat{\overline{Q}}}_{t-1}\right)$ for a fixed g. We now discuss conditions under which ${\widehat{\sigma }}_t\left(g_t\right)$ is valid estimator of ${\sigma }_{0,t}\left(g_t,{\widehat{\overline{Q}}}_{t-1}\right)$. When g is fixed, for each i = 1, 2, the error ${\widehat{\Phi }}_{i,t}(g)-{\Phi }_{0,i,t}\left(g,{\widehat{\overline{Q}}}_{t-1}\right)$ decomposes as the sum of the MDS average ${\widehat{\Phi }}_{i,t}(g)-{\overline{\Phi }}_{0,i,t}(g)$ and of the Cesaro approximation error ${\overline{\Phi }}_{0,i,t}(g)-{\Phi }_{0,i}\left(g,{\widehat{\overline{Q}}}_{t-1}\right)$. Both differences are straightforward to bound. For a random g_t, the term ${\widehat{\Phi }}_{i,t}\left(g_t\right)-{\overline{\Phi }}_{0,i,t}\left(g_t\right)$ is no longer an MDS average. Fortunately, under a complexity condition on the logging policy class $\mathcal{G}$, we can bound the supremum of the martingale empirical processes $\left\{\left|{\widehat{\Phi }}_{i,t}(g)-{\overline{\Phi }}_{0,i,t}(g)\right|:g\in \mathcal{G}\right\}$, which in turn gives us a bound on $\left|{\widehat{\Phi }}_{i,t}\left(g_t\right)-{\overline{\Phi }}_{0,i,t}\left(g_t\right)\right|$.

Consistency guarantee for ${\widehat{\sigma }}_t^2$.

Our formal consistency result relies on the following assumptions.

Assumption 4 (Outcome regression estimator convergence). There exists β > 0, and a fixed function ${\overline{Q}}_1:\mathcal{A}\times \mathcal{X}\to ℝ$ such that ${\Vert {\widehat{\overline{Q}}}_t-{\overline{Q}}_1\Vert }_{1,Q_{0,X},g^{*}}=O\left(t^{-\beta }\right)$ almost surely.

The next assumption is a bound on the bracketing entropy (see, e.g., [van der Vaart and Wellner, 1996] for definition) of the logging policy class.

Assumption 5 (Complexity of the logging policy class). There exists a class of conditional densities $\mathcal{G}$ such that $g_t\in \mathcal{G}\forall t\ge 1$ almost surely, there exists G > 0 such that ${\text{sup}}_{g\in \mathcal{G}}{\Vert g/g^{\text{ref}}\Vert }_{\infty }\le G$, and for some p > 0

$$\text{log}{N}_{[]}\left(ϵ,\mathcal{G}/g^{\text{ref}},\parallel \cdot {\parallel }_{2,Q_{0,X},g^{\text{ref}}}\right)\lesssim {ϵ}^{-p},$$

where $\mathcal{G}/g^{\text{ref}}≔\left\{g/g^{\text{ref}}:g\in \mathcal{G}\right\}$.

Next, we require a condition on the exploration rate that is stronger than Assumption 3.

Assumption 6 (Exploration rate (stronger)). For ant t ≥ 1, we have that ${\text{inf}}_{a\in \mathcal{A},x\in \mathcal{X}}{g}_t(a\mid x)/g^{\text{ref}}(a\mid x)\gtrsim t^{-\alpha (\beta ,p)}$ almost surely, where α(β, p) ≔ min(1/((3 + p)), 1/(1 + 2p), β).

Theorem 2. Suppose that Assumptions 4 to 6 hold. Then, ${\widehat{\sigma }}_t^2-{\sigma }_{0,t}^2=o(1)$ almost surely.

Remark 2. While we theoretically require the existence of a logging policy class $\mathcal{G}$ with controlled complexity, we do not actually need to know $\mathcal{G}$ to construct our estimator. Moreover, while we require a bound on the bracketing entropy of the logging policy class $\mathcal{G}$, we impose no restriction on the outcome regression model complexity, permitting us to use flexible black-box regression methods.

Remark 3. Assumption 4 requires $\left({\widehat{\overline{Q}}}_t\right)$ to be a sequence of regression estimator, such that for every t ≥ 1, ${\widehat{\overline{Q}}}_t$ is fitted on O(1), …, O(t) and for which we can guarantee a rate of convergence to some fixed limit ${\overline{Q}}_1$. Note that this can at first glance pose a challenge since observations O(1), …, O(t) are adaptively collected. In the appendix, we give guarantees for outcome regression estimation over a nonparametric model using an importance sampling weighted empirical risk minimization.

CADR asymptotics.

Our proposed CADR estimator is now given by plugging our estimates ${\widehat{\sigma }}_t$ from Eq. (3) into Eq. (1), as summarized in Algorithm 1 As an immediate corollary of Theorems 1 and 2 we have our main guarantee for this final estimator, showing CADR is asymptotically normal, whence we immediately obtain asymptotically valid confidence intervals.

Corollary 1 (CADR Asymptotics and Inference). Suppose that Assumptions 1 and 4 to 6 hold. Let ${\widehat{\sigma }}_t$ be given as in Eq. (3). Denote ${\Gamma }_T≔{\left(T^{-1}{\sum }_{t=1}^T{\widehat{\sigma }}_t^{-1}\right)}^{-1}$. Then, ${\Gamma }_T^{-1}\sqrt{T}\left({\widehat{\Psi }}_T-{\Psi }_0\right)\stackrel{d}{\to }\mathcal{N}(0,1).$

Moreover, letting ζ_α denote the α-quantile of the standard normal distribution,

$$\text{Pr}\left[\Psi \left(Q_{0,X},{\overline{Q}}_0\right)\in \left[{\widehat{\Psi }}_T\pm {\zeta }_{1-\alpha /2}{\Gamma }_T/\sqrt{T}\right]\right]\stackrel{T\to \infty }{\to }1-\alpha .$$

4. Empirical Evaluation

We next present computational results on public datasets that demonstrate the robustness of CADR confidence intervals using contextual bandit data with comparison to several baselines. Our experiments focus on the case of finitely-many actions, $\mathcal{A}=\left\{1,\dots ,K\right\}$.

4.1. Baseline Estimators

We compare CADR to several benchmarks. All take the following form for a choice of w_t, ω_t, ${\widehat{\overline{Q}}}_t$:

$${\widehat{\Psi }}_T={\left(\frac{1}{T}{w}_t\right)}^{-1}\frac{1}{T}\sum _{t=1}^T{w}_t{\tilde{D}}_t^{\prime },{\text{CI}}_{\alpha }=\left[{\widehat{\Psi }}_T\pm {\zeta }_{1-\alpha /2}\sqrt{\frac{{\sum }_{t=1}^T{w}_t^2{\left({\tilde{D}}_t^{\prime }-\widehat{\Psi }\right)}^2}{{\left({\sum }_{t=1}^T{w}_t\right)}^2}}\right],$$

where ${\tilde{D}}_t^{\prime }={\omega }_t\left(Y(t)-{\widehat{\overline{Q}}}_{t-1}(A(t),X(t))\right)+\sum _{a=1}^K{\widehat{\overline{Q}}}_{t-1}(a,X(t))g^{*}(a\mid X(t))$.

The Direct Method (DM) sets w_t = 1, ω_t = 0 and fits ${\widehat{\overline{Q}}}_{t-1}(a,\cdot )$ by running some regression method for each a on the data {(X(s), Y (s)) : 1 ≤ s ≤ t − 1, A(s) = a}. We will use either linear regression or decision-tree regression, both using default sklearn parameters. Note that even in non-contextual settings, where ${\widehat{\overline{Q}}}_{t-1}$ is a simple per-arm sample average, ${\widehat{\overline{Q}}}_{t-1}$ may be biased due to adaptive data collection [Xu et al., 2013, Luedtke and van der Laan, 2016, Bowden and Trippa, 2017, Nie et al., 2018, Hadad et al., 2019, Shin et al., 2019]. Inverse Propensity Score Weighting (IPW) sets w_t = 1, ω_t = (g*/g_t)(A(t) | X(t)), ${\widehat{\overline{Q}}}_t=0$. Doubly Robust (DR) sets w_t = 1, ω_t = (g*/g_t)(A(t) | X(t)) and fits ${\widehat{\overline{Q}}}_{t-1}$ as in DM. More Robust Doubly Robust (MRDR) [Farajtabar et al., 2018] is the same as DR but when fitting ${\widehat{\overline{Q}}}_{t-1}$ we reweight each data point by $\frac{g^{*}(A(s)\mid X(s))\left(1-g_s(A(s)\mid X(s))\right)}{g_s{(A(s)\mid X(s))}^2}$. None of the above are generally asymptotically normal under adaptive data collection [Hadad et al., 2019]. Adaptive Doubly Robust (ADR; a.k.a. stabilized onestep estimator for multi-armed bandit data) [Luedtke and van der Laan, 2016, Hadad et al., 2019] is the same as DR but sets $w_t=g_t^{-1/2}(A(t)\mid X(t))$. ADR is unbiased and asymptotically normal for multi-armed bandit logging policies but is biased for context-measurable adaptive logging policies, which is the focus of this paper. Finally, note that our proposal CADR takes the same form as DR but with $w_t={\widehat{\sigma }}_t^{-1}$ using our adaptive conditional standard deviation estimators ${\widehat{\sigma }}_t$ in Eq. (3).

4.2. Contextual Bandit Data from Multiclass Classification Data

To construct our data, we turn K-class classification tasks into a K-armed contextual bandit problems [Dudík et al., 2014, Dimakopoulou et al., 2017, Su et al., 2019], which has the benefits of reproducibility using public datasets and being able to make uncontroversial comparisons using actual ground truth data with counterfactuals. We use the public OpenML Curated Classification benchmarking suite 2018 (OpenML-CC18; BSD 3-Clause license) [Bischl et al., 2017], which has datasets that vary in domain, number of observations, number of classes and number of features. Among these, we select the classification datasets which have less than 100 features. This results in 57 classification datasets from OpenML-CC18 used for evaluation and Table 1 summarizes the characteristics of these datasets.

Table 1:

Characteristics of the 57 OpenML-CC18 datasets used for evaluation.

Samples	Count		Classes	Count		Features	Count
< 1000	17		= 2	31		≥ 2 and < 10	14
≥ 1000 and < 10000	30		> 2 and < 10	17		≥ 10 and < 50	34
≥ 10000	10		≥ 10	9		≥ 50 and ≤ 100	9

Each dataset is a collection of pairs of covariates X and labels L ∈ {1, …, K}. We transform each dataset to the contextual bandit problem as follows. At each round, we draw X(t), L(t) uniformly at random with replacement from the dataset. We reveal the context X(t) to the agent, and given an arm pull A(t), we draw and return the reward $Y(t)~\mathcal{N}(1\{A(t)=L(t)\},1)$. To generate our data, we set T = 10000 and use the following ϵ-greedy procedure. We pull arms uniformly at random until each arm has been pulled at least once. Then at each subsequent round t, we fit ${\widehat{\overline{Q}}}_{t-1}$ using the data up to that time in the same fashion as used for the DM estimator above using decision-tree regressions. We set ${\tilde{A}}_x(t)=\text{arg}{\text{max}}_{a=1,\dots ,K}{\widehat{\overline{Q}}}_{t-1}(a,X(t))$ and ϵ_t = 0.01 · t^−1/3. We then let g_t(a | x) = ϵ_t/K for $a\ne {\tilde{A}}_x(t)$ and $g_t\left({\tilde{A}}_x(t)\mid x\right)=1-{ϵ}_t+{ϵ}_t/K$. That is, with probability ϵ_t we pull a random arm, and otherwise we pull ${\tilde{A}}_{X(t)}(t)$.

We then consider four candidate policies to evaluate: (1) “arm 1 non-contextual”: g*(1 | x) = 1 and otherwise g*(a | x) = 0 (note that the meaning of label “1” changes by dataset), (2) “arm 2 non-contextual”: g*(2 | x) = 1 and otherwise g*(a | x) = 0, (3) “linear contextual”: we sample a new dataset of size T using a uniform exploration policy, then fit ${\widehat{\overline{Q}}}_T$ as above using linear regression, fix $a^{*}=\text{arg}{\text{max}}_{a\in \{1,\dots ,K\}}{\widehat{\overline{Q}}}_T(a,x)$, and set g*(a* | x) = 1 and otherwise g*(a | x) = 0, (4) “tree contextual”: same as “linear contextual” but fit ${\widehat{\overline{Q}}}_T$ using decision-tree regression.

4.3. Results

Figure 1 shows the comparison of CADR estimator against DM, IPW, DR, ADR, and MRDR w.r.t. coverage, that is, the frequency over 64 replications of the 95% confidence interval covering the true Ψ₀, for each of the 57 OpenML-CC18 datasets and 4 target policies. In each subfigure, each dot represents a dataset, the y-axis corresponds to the coverage of the CADR estimator and the x-axis corresponds to the coverage of one of the baseline estimators. The lines represent one standard error over the 64 replications. The dot is depicted in blue if for that dataset CADR has significantly better coverage than the baseline estimator, in red if it has significantly worse coverage, and in black if the difference in coverage of both estimators is within one standard error. In Fig. 1, outcome models for CADR, DM, DR, ADR, and MRDR are fit using linear regression (with default sklearn parameters). In the appendix, we provide additional empirical results where we use decision-tree regressions, or where we use the MRDR outcome model for CADR, or where we use cross-fold estimation across time.

Figure 1:

Comparison of CADR estimator against DM, IPW, DR, ADR and MRDR w.r.t. 95% confidence interval coverage on 57 OpenML-CC18 datasets and 4 target policies.

Across all of our experiments, we observe that the confidence interval of CADR has better coverage of the ground truth than any other baseline, which can be attributed to its asymptotic normality. The second best estimator in terms of coverage is DR. The advantages of CADR over DR are most pronounced when either (a) there is a mismatch between the logging policy and the target policy (e.g., compare the 1st and 2nd rows in Fig. 1; the tree target policy is most similar to the logging policy, which also uses trees) or (b) when the outcome model is bad (either due to model misspecification such as with a linear model on real data or due to small sample size).

5. Conclusions

Adaptive experiments hold great promise for better, more efficient, and even more ethical experiments. However, they complicate post-experiment inference, which is a cornerstone of drawing credible conclusions from controlled experiments. We provided here the first asymptotically normal estimator for policy value and causal effects when data were generated from a contextual adaptive experiment, such as a contextual bandit algorithm. This led to simple and effective confidence intervals given by adding and subtracting multiples of the standard error, making contextual adaptive experiments a more viable option for experimentation in practice.

6. Societal Impact and Limitations

Adaptive experiments hold particular promise in settings where experimentation is costly and/or dangerous, such as in medicine and policymaking. By adapting treatment allocation, harmful interventions can be avoided, outcomes for study participants improved, and smaller studies enabled. Being able to draw credible conclusions from such experiments make them viable replacements for classic randomized trials. Our confidence intervals offer one way to do so. At the same time, and especially subject to our assumption of vanishing but nonzero exploration, these experiments must be subject to the same ethical guidelines as classic randomized experiments. Additionally, the usual caveats of frequentist confidence intervals hold here, such as its interpretation only as a guarantee over data collection, this guarantee only being approximate in finite samples when we rely on asymptotic normality, and the risks of multiple comparisons and of p-hacking. Finally, we note that our inference focused on an average quantity, as such it focuses on social welfare and need not capture the risk to individuals or groups. Subgroup analyses may therefore be helpful in complementing the analysis; these can be conducted by setting g*(a | x) to zero for some x’s. Future work may be necessary to further extend our results to conducting inference on risk metrics such as quantiles of outcomes.