Online learning in bandits with predicted context

Abstract

We consider the contextual bandit problem where at each time, the agent only has access to a noisy version of the context and the error variance (or an estimator of this variance). This setting is motivated by a wide range of applications where the true context for decision-making is unobserved, and only a prediction of the context by a potentially complex machine learning algorithm is available. When the context error is non-vanishing, classical bandit algorithms fail to achieve sublinear regret. We propose the first online algorithm in this setting with sublinear regret guarantees under mild conditions. The key idea is to extend the measurement error model in classical statistics to the online decision-making setting, which is nontrivial due to the policy being dependent on the noisy context observations. We further demonstrate the benefits of the proposed approach in simulation environments based on synthetic and real digital intervention datasets.

1. Introduction

Contextual bandits (Auer, 2002; Langford and Zhang, 2007) represent a classical sequential decision-making problem where an agent aims to maximize cumulative reward based on context information. At each round $t$, the agent observes a context and must choose one of $K$ available actions based on both the current context and previous observations. Once the agent selects an action, she observes the associated reward, which is then used to refine future decision-making. Contextual bandits are typical examples of reinforcement learning problems where a balance between exploring new actions and exploiting previously acquired information is necessary to achieve optimal long-term rewards. It has numerous real-world applications including personalized recommendation systems (Li et al., 2010; Bouneffouf et al., 2012), healthcare (Yom-Tov et al., 2017; Liao et al., 2020), and online education (Liu et al., 2014; Shaikh et al., 2019).

Despite the extensive existing literature on contextual bandits, in many real-world applications, the agent never observes the context exactly. One common reason is that the true context for decision-making can only be detected or learned approximately from observable auxiliary data. For instance, consider the Sense2Stop mobile health study, in which the context is whether the individual is currently physiologically stressed (Battalio et al., 2021). A complex predictor of current stress was constructed/validated based on multiple prior studies (Cohen et al., 1985; Sarker et al., 2016, 2017). This predictor was then tuned to each user in Sense2Stop prior to their quit-smoke attempt and then following the user’s attempt to quit smoking, at each minute, the predictor inputs high-dimensional sensor data on the user and outputs a continuous likelihood of stress for use by the decision-making algorithm. In many such applications in health interventions, models using validated predictions as contexts are preferred to raw sensor data because of the high noise in these settings, and that the decision rules are interpretable so they can be critiqued by domain experts. The second reason why the context is not observed exactly is because of measurement error. Contextual variables, such as user preferences in online advertising, content attributes in recommendation systems, and patient conditions in clinical trials, are prone to noisy measurement. This introduces an additional level of uncertainty that must be accounted for when making decisions.

Motivated by the above, we consider the linear contextual bandit problem where at each round, the agent only has access to a noisy observation of the true context. Moreover, the agent has limited knowledge about the underlying distribution of this noisy observation, as in many practical applications (e.g. the above-mentioned ones). This is especially the case when this ‘observation’ is the output of a complex machine learning algorithm. We only assume that the noisy observation is unbiased, its variance is known or can be estimated, and we put no other essential restrictions on its distribution. In healthcare applications, the estimated error variance for context variables can often be derived from data in prior studies (e.g. pilot studies). This setting is intrinsically difficult for two main reasons: First, when estimating the reward model, the agent needs to take into account the misalignment between the noisy context observation and the reward which depends on the true context. Second, even if the reward model is known, the agent may suffer from making bad decisions at each round because of the inaccurate context.

Our contributions.

We present the first online algorithm MEB (Measurement Error Bandit) with sublinear regret in this setting under mild conditions. MEB achieves $\widetilde{𝒪}\left(T^{2/3}\right)$ regret compare to a standard benchmark and $\widetilde{𝒪}\left(T^{1/2}\right)$ regret compare to a clipped benchmark with minimum exploration probability which is common in many applications (Yang et al., 2020a; Yao et al., 2021). MEB is based on a novel approach to model estimation which removes the systematic bias caused by the noisy context observation. The estimator is inspired by the measurement error literature in statistics (Carroll et al., 1995; Fuller, 2009): We extend this classical method with additional tools in the online decision-making setting due to the policy being dependent on the measurement error.

1.1. Related work

Our work complements several lines of literature in contextual bandits, as listed below.

Latent contextual bandit.

In the latent contextual bandit literature (Zhou and Brunskill, 2016; Sen et al., 2017; Hong et al., 2020a,b; Xu et al., 2021; Nelson et al., 2022; Galozy and Nowaczyk, 2023), the reward is typically modeled as jointly depending on the latent state, the context, and the action. Several works (Zhou and Brunskill, 2016; Hong et al., 2020a,b; Galozy and Nowaczyk, 2023) assume no direct relation between the latent state and the context while setting a parametric reward model. For example, Hong et al. (2020a) assumes the latent state $s\in {𝒮}_l$ is unknown but constant over time. Hong et al. (2020b) assume that the latent state evolves through a Markov chain. Xu et al. (2021) sets a specific context as well as the latent feature for each action, and models the reward depending on them through a generalized linear model. Different from the aforementioned studies, we specify that the observed context is a noisy version of the latent context (which aligns with the applications we are addressing), and then leverage this structure to design the online algorithm.

In another line of work, Sen et al. (2017); Nelson et al. (2022) consider contextual bandit with latent confounder, where the observed context influences the reward through a latent confounder variable, which ranges within a small discrete set. Our setting is distinct from these works in that the latent context can span an infinite (or even continuous) space.

Bandit with noisy context.

In bandit with noisy context (Yun et al., 2017; Kirschner and Krause, 2019; Yang et al., 2020b; Lin and Moothedath, 2022; Lin et al., 2022), the agent has access to a noisy version of the true context and/or some knowledge of the distribution of the noisy observation. Yun et al. (2017); Park and Faradonbeh (2022); Jose and Moothedath (2024) consider settings where the joint distribution of the true context and the noisy observed context is known (up to a parameter). Other works, such as Kirschner and Krause (2019); Yang et al. (2020b); Lin et al. (2022), assume that the agent knows the exact distribution of the context each time, and observes no context. By assuming a linear reward model, Kirschner and Krause (2019) transforms the problem into a linear contextual bandit, and obtains $\widetilde{𝒪}\left(d\sqrt{T}\right)$ regret compared to the policy maximizing the expected reward over the context distribution. Yang et al. (2020b); Lin et al. (2022); Lin and Moothedath (2022) consider variants of the problem such as multiple linear feedbacks, multiple agents, and delayed observation of the exact context. Compared to these works, we consider a practical but more challenging setting where besides an unbiased noisy observation(prediction) for each context, the agent only knows the second-moment information about the distribution. This does not transform into a standard linear contextual bandit as in Kirschner and Krause (2019).

Bandit with inaccurate/corrupted context.

These works consider the setting where the context is simply inaccurate (without randomness), or is corrupted and cannot be recovered. In Yang and Ren (2021a,b), at each round, only an inaccurate context is available to the decision-maker, and the exact context is revealed after the action is taken. In Bouneffouf (2020); Galozy et al. (2020), each context ${\mathbf{x}}_t$ is completely corrupted with some probability and the corruption cannot be recovered. In Ding et al. (2022), the context is attacked by an adversarial agent. Because these works focus more on adversarial settings for the context observations, the application of their regret bounds to our setting generally results in a linear regret. For example, in Yang and Ren (2021a), the regret of Thompson sampling is $\widetilde{𝒪}\left(d\sqrt{T}+\sqrt{d}{\sum }_{t\in \left[T\right]}{∥{\widehat{\mathbf{x}}}_t-{\mathbf{x}}_t∥}_2\right)$, where ${∥{\widehat{\mathbf{x}}}_t-{\mathbf{x}}_t∥}_2$ is the error of the inaccurate context. As is typical in our setting, ${∥{\widehat{\mathbf{x}}}_t-{\mathbf{x}}_t∥}_2$ will be non-vanishing through time, so the second term is linear in $T$. Given the applications we consider, we can exploit the stochastic nature of the noisy context observations in our algorithm to achieve improved performance.

1.2. Notations

Throughout this paper, we use $\left[n\right]$ to represent the set $\{\mathrm{1,2},\dots ,n\}$ for $n\in {\mathbb{N}}^{+}$. For $a,b\in \mathbb{R}$, let $a\wedge b$ denote the minimum of $a$ and $b$. Given $d\in {\mathbb{N}}^{+},{\mathit{I}}_d$ denotes the $d$-by-$d$ identity matrix, and $1_d$ denotes the $d$-dimensional vector with 1 in each entry. For a vector $\mathit{v}\in {\mathbb{R}}^d$, denote $\parallel \mathit{v}{\parallel }_2$ as its ${\ell }_2$ norm. For a matrix $\mathit{M}\in {\mathbb{R}}^{m\times n}$, denote $\parallel \mathit{M}{\parallel }_2$ as its operator norm. The notation $𝒪\left(X\right)$ refers to a quantity that is upper bounded by $X$ up to constant multiplicative factors, while $\widetilde{𝒪}\left(X\right)$ refers to a quantity that is upper bounded by $X$ up to poly-log factors.

2. Measurement error adjustment to bandit with noisy context

2.1. Problem setting

We consider a linear contextual bandit with context space $𝒳\subset {\mathbb{R}}^d$ and binary action space $𝒜=\left\{\mathrm{0,1}\right\}$¹. Let $T$ be the time horizon. As discussed above, we consider the setting where at each time $t\in \left[T\right]$, the agent only observes a noisy version of the context ${\tilde{\mathbf{x}}}_t$ instead of the true underlying context ${\mathbf{x}}_t$. Thus, at time $t$, the observation $o_t$ only contains $\left({\tilde{\mathbf{x}}}_t,a_t,r_t\right)$, where $a_t$ is the action and $r_t$ is the corresponding reward. We further assume that ${\tilde{\mathbf{x}}}_t={\mathbf{x}}_t+{\mathit{ϵ}}_t$, where the error ${\mathit{ϵ}}_t$ is independent of the history ${\mathscr{H}}_{t-1}:={\left\{o_{\tau }\right\}}_{\tau \le t-1},\mathbb{E}{\mathit{ϵ}}_t=0,\text{Var}\left({\mathit{ϵ}}_t\right)={\mathbf{\Sigma }}_{e,t}$. Here, ‘e’ in the subscript means ‘error’. Initially we assume that ${\left({\mathbf{\Sigma }}_{e,t}\right)}_{t\ge 1}$ is known. In Section 2.4, we consider the setting where only estimators of ${\left({\mathbf{\Sigma }}_{e,t}\right)}_{t\ge 1}$ are available. There is no restriction that the distribution of ${\left({\mathit{ϵ}}_t\right)}_{t\ge 1}$ belongs to any known (parametric) family. The reward $r_t=⟨{\mathit{\theta }}_{a_t}^{*},{\mathbf{x}}_t⟩+{\eta }_t$, where $\mathbb{E}\left[{\eta }_t\mid {\mathscr{H}}_{t-1},{\mathit{ϵ}}_t,a_t\right]=0$ and ${\left({\mathit{\theta }}_a^{*}\right)}_{a\in 𝒜}$ are the unknown parameters. It’s worth noting that besides the policy, all the randomness here comes from the reward noise ${\eta }_t$ and the context error ${ϵ}_t$. We treat ${\left({\mathbf{x}}_t\right)}_{t\ge 1}$ as fixed throughout but unknown to the algorithm (Unlike Yun et al. (2017), we don’t assume ${\mathbf{x}}_t$ are i.i.d.). Our goal at each time $t$ is to design policy ${\pi }_t\left(\cdot \mid {\mathscr{H}}_{t-1},{\tilde{\mathbf{x}}}_t\right)\in \Delta \left(𝒜\right)$ given past history ${\mathscr{H}}_{t-1}$ and current observed noisy context ${\tilde{\mathbf{x}}}_t$, so that the agent can maximize the reward by taking action $a_t~{\pi }_t\left(\cdot \mid {\mathscr{H}}_{t-1},{\tilde{\mathbf{x}}}_t\right)$.

If ${\mathbf{\Sigma }}_{e,t}$ is non-vanishing, standard contextual bandit algorithms are generally sub-optimal. To see this, notice that $r_t=⟨{\mathit{\theta }}_{a_t}^{*},{\mathbf{x}}_t⟩+{\eta }_t=⟨{\mathit{\theta }}_{a_t}^{*},{\tilde{\mathbf{x}}}_t⟩+\left({\eta }_t-⟨{\mathit{\theta }}_{a_t}^{*},{\mathit{ϵ}}_t⟩\right)$. This means the error in the reward $r_t$ after observing the noisy context is ${\eta }_t-⟨{\mathit{\theta }}_{a_t}^{*},{\mathit{ϵ}}_t⟩$, where ${\mathit{ϵ}}_t$ and ${\tilde{\mathbf{x}}}_t$ are dependent. Thus, $\mathbb{E}\left[r_t\mid {\tilde{\mathbf{x}}}_t,a_t\right]\ne ⟨{\mathit{\theta }}_{a_t}^{*},{\tilde{\mathbf{x}}}_t⟩$. This is in contrast to the standard linear bandit setting, where given the true context ${\mathbf{x}}_t,\mathbb{E}\left[r_t\mid {\mathbf{x}}_t,a_t\right]=⟨{\mathit{\theta }}_{a_t}^{*},{\mathbf{x}}_t⟩$, which ensures the sublinear regret of classical bandit algorithms such as UCB and Thompson sampling. Therefore, it is necessary to design an online algorithm that adjusts for the errors ${\left({\mathit{ϵ}}_t\right)}_{t\ge 1}$. We assume that the context, parameters and the reward are bounded, as below.

Assumption 2.1 (Boundedness). $\forall t\in \left[T\right],{∥{\tilde{\mathbf{x}}}_t∥}_2\le 1$; There exists a positive constant $R_{\theta }$ such that $\forall a\in \left\{\mathrm{0,1}\right\},{∥{\mathit{\theta }}_a^{*}∥}_2\le R_{\theta }$; There exists a positive constant $R$ such that $\forall t\in \left[T\right],\left|r_t\right|\le R$.

For any policy $\pi ={\left({\pi }_t\right)}_t$, we define the (standard) cumulative regret as

$$\mathrm{R}\mathrm{e}\mathrm{g}\mathrm{r}\mathrm{e}\mathrm{t}\left(T;{\pi }^{*}\right)=\sum _{t\in \left[T\right]}\left[{\mathbb{E}}_{a~{\pi }_t^{*}}⟨{\mathit{\theta }}_a^{*},{\mathbf{x}}_t⟩-{\mathbb{E}}_{a~{\pi }_t}⟨{\mathit{\theta }}_a^{*},{\mathbf{x}}_t⟩\right],$$ (2.1)

where

$${\pi }_t^{*}\left(a\right)=\left\{\begin{array}{ll}1,& \text{if }a=a_t^{*}:={\text{argmax}}_a⟨{\mathit{\theta }}_a^{*},{\mathbf{x}}_t⟩,\\ 0,& \text{otherwise.}\end{array}\right.$$ (2.2)

We denote the standard benchmark policy ${\pi }^{*}={\left({\pi }_t^{*}\right)}_t$. This is summarized in the setting below.

Setting 1. (Standard setting) We aim to minimize Regret $\left(T;{\pi }^{*}\right)$ among the class of all policies.

In many applications including clinical trials, it’s desirable to design the policy under the constraint that each action is sampled with a minimum probability $p_0$. One reason for maintaining exploration is that we can update and re-optimize the policy for future users to allow for potential non-stationarity (Yang et al., 2020a). Additionally, keeping the exploration is also important for after-study analysis (Yao et al., 2021), especially when the goal of the analysis is not pre-specified prior to collecting data with the online algorithm. In these situations, it is desirable to consider only the policies that always maintain an exploration probability of $p_0>0$ for each arm, and compare the performance to the clipped benchmark policy $\left({\bar{\pi }}_t^{*}\right)$:

$${\bar{\pi }}_t^{*}\left(a\right)=\left\{\begin{array}{ll}1-p_0,& \text{if }a=a_{t,}^{*}\\ p_0,& \text{otherwise.}\end{array}\right.$$ (2.3)

This is summarized in the setting below.

Setting 2. (Clipped policy setting) We minimize

$$\text{Regret}\left(T;{\bar{\pi }}^{*}\right)=\sum _{t\in \left[T\right]}\left[{\mathbb{E}}_{a~{\bar{\pi }}_t^{*}}⟨{\mathit{\theta }}_a^{*},{\mathbf{x}}_t⟩-{\mathbb{E}}_{a~{\pi }_t}⟨{\mathit{\theta }}_a^{*},{\mathbf{x}}_t⟩\right]$$

among the class of policies that explore any action with probability at least $p_0$.

In this work, we will provide policies with sublinear regret guarantees in both settings.

2.2. Estimation using weighted measurement error adjustment

In this section, we focus on learning the reward model parameters ${\left({\mathit{\theta }}_a^{*}\right)}_{a\in 𝒜}$ with data ${\mathscr{H}}_t$ after a policy ${\left({\pi }_{\tau }\left(\cdot \mid {\tilde{\mathbf{x}}}_{\tau },{\mathscr{H}}_{\tau -1}\right)\right)}_{\tau \in \left[t\right]}$ has been executed up to time $t\in \left[T\right]$. Learning a consistent model is important in many bandit algorithms for achieving low regret (Abbasi-Yadkori et al., 2011; Agrawal and Goyal, 2013). As we shall see in Section 2.3, consistent estimation of ${\left({\mathit{\theta }}_a^{*}\right)}_{a\in 𝒜}$ plays an essential role in controlling the regret of our proposed algorithm.

Inconsistency of the regularized least-squares (RLS) estimator.

UCB and Thompson sampling, the two classical bandit algorithms, both achieve sublinear regret based on the consistency of the estimator ${\widehat{\mathit{\theta }}}_{a,RLS}^{\left(t\right)}={\left(\lambda I+{\sum }_{\tau \in \left[t\right]}{1}_{\left\{a_{\tau }=a\right\}}{\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }\right)}^{-1}\left({\sum }_{\tau \in \left[t\right]}{1}_{\left\{a_{\tau }=a\right\}}{\mathbf{x}}_{\tau }{r}_{\tau }\right)$ under certain norms. When ${\tilde{\mathbf{x}}}_{\tau }={\mathbf{x}}_{\tau }+{\mathit{ϵ}}_{\tau }$ is observed instead of ${\mathbf{x}}_{\tau }$, the RLS estimator becomes ${\widehat{\mathit{\theta }}}_{a,\mathit{\text{RLSCE}}}^{\left(t\right)}={\left(\lambda I+{\sum }_{\tau \in \left[t\right]}{1}_{\left\{a_{\tau }=a\right\}}{\tilde{\mathbf{x}}}_{\tau }{\tilde{\mathbf{x}}}_{\tau }^{\top }\right)}^{-1}\left({\sum }_{\tau \in \left[t\right]}{1}_{\left\{a_{\tau }=a\right\}}{\tilde{\mathbf{x}}}_{\tau }{r}_{\tau }\right)$. Here ‘RLSCE’ means the RLS estimator with contextual error. However, when ${\mathbf{\Sigma }}_{e,\tau }=\text{Var}\left({\mathit{ϵ}}_{\tau }\right)$ is non-vanishing, ${\widehat{\mathit{\theta }}}_{a,\mathit{\text{RLSCE}}}^{\left(t\right)}$ is generally no longer consistent, which may lead to bad decision-making (see Appendix B for details). In the simple case where ${\left({\mathbf{x}}_{\tau },{\mathit{ϵ}}_{\tau },{\eta }_{\tau }\right)}_{\tau \in \left[t\right]}$ are i.i.d. and there is no action (i.e. set $a_{\tau }\equiv 0$), the inconsistency of ${\widehat{\mathit{\theta }}}_{a,\mathit{\text{RLSCE}}}^{\left(t\right)}$ is studied in the measurement error model literature in statistics (Fuller, 2009; Carroll et al., 1995), and is known as attenuation.

A naive measurement error adjustment.

A measurement error model is a type of regression model designed to accommodate inaccuracies in the measurement of regressors (i.e., instead of observing ${\mathbf{x}}_t$, we observe ${\mathbf{x}}_t+{\mathit{ϵ}}_t$ where ${\mathit{ϵ}}_t$ is a noise term with zero mean). As conventional regression techniques yield inconsistent estimators, measurement error models rectify this issue with adjustments to the estimator that consider these errors. In the current context when we want to estimate ${\mathit{\theta }}_a^{*}$ from history ${\mathscr{H}}_t,{\left({\tilde{\mathbf{x}}}_{\tau }\right)}_{\tau \in \left[t\right]}$ can be viewed as regressors ‘measured with error’, while ${\left(r_{\tau }\right)}_{\tau \in \left[t\right]}$ are dependent variables. If ${\left({\mathit{ϵ}}_{\tau },{\eta }_{\tau }\right)}_{\tau \in \left[t\right]}$ are i.i.d., ${\mathbf{\Sigma }}_{e,\tau }\equiv {\mathbf{\Sigma }}_e$, and there is no action (i.e. set $\left.a_{\tau }\equiv 0\right),{\widehat{\mathit{\theta }}}_{0,me}^{\left(t\right)}:={\left(\frac{1}{t}{\sum }_{\tau \in \left[t\right]}{\tilde{\mathbf{x}}}_{\tau }{\tilde{\mathbf{x}}}_{\tau }^{\top }-{\mathbf{\Sigma }}_e\right)}^{-1}\left(\frac{1}{t}{\sum }_{\tau \in \left[t\right]}{\tilde{\mathbf{x}}}_{\tau }{r}_{\tau }\right)$ is a consistent estimator for ${\mathit{\theta }}_0^{*}$. When multiple actions are present, a naive generalization of the above estimator, ${\widehat{\mathit{\theta }}}_{a,me}^{\left(t\right)}$, is

$${\left(\sum _{\tau \in \left[t\right]}{1}_{\left\{a_{\tau }=a\right\}}\left({\tilde{\mathbf{x}}}_{\tau }{\tilde{\mathbf{x}}}_{\tau }^{\top }-{\mathbf{\Sigma }}_e\right)\right)}^{-1}\cdot \left(\sum _{\tau \in \left[t\right]}{1}_{\left\{a_{\tau }=a\right\}}{\tilde{\mathbf{x}}}_{\tau }{r}_{\tau }\right).$$ (2.4)

Unfortunately, ${\widehat{\mathit{\theta }}}_{a,me}^{\left(t\right)}$ is inconsistent in the multiple-action setting, even if the policy ${\left({\pi }_{\tau }\right)}_{\tau \in \left[t\right]}$ is stationary and not adaptive to history. This difference is essentially due to the interaction between the policy and the measurement error: In the classical measurement error (no action) setting, $\mathbb{E}\left({\tilde{\mathbf{x}}}_{\tau }{\tilde{\mathbf{x}}}_{\tau }^{\top }\right)={\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }+{\mathbf{\Sigma }}_e$, and $\frac{1}{t}{\sum }_{\tau \in \left[t\right]}{\tilde{\mathbf{x}}}_{\tau }{\tilde{\mathbf{x}}}_{\tau }^{\top }-{\mathbf{\Sigma }}_e$ concentrates around its expectation $\frac{1}{t}{\sum }_{\tau \in \left[t\right]}{\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }$. Likewise, $\frac{1}{t}{\sum }_{\tau \in \left[t\right]}{\tilde{\mathbf{x}}}_{\tau }{r}_{\tau }$ concentrates around $\left(\frac{1}{t}{\sum }_{\tau \in \left[t\right]}{\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }\right){\mathit{\theta }}_0^{*}$. Combining the above parts thus yields a consistent estimator of ${\mathit{\theta }}_0^{*}$. In our setting with multiple actions, however, the agent picks the action $a_{\tau }$ based on ${\tilde{\mathbf{x}}}_{\tau }$, so only certain values of ${\tilde{\mathbf{x}}}_{\tau }$ lead to $a_{\tau }=a$. Therefore, for those $\tau \in \left[t\right]$ when we pick $a_{\tau }=a$, it’s more likely that ${\tilde{\mathbf{x}}}_{\tau }$ falls in certain regions depending on the policy, and we shouldn’t expect $\mathbb{E}\left({\tilde{\mathbf{x}}}_{\tau }{\tilde{\mathbf{x}}}_{\tau }^{\top }\right)={\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }+{\mathbf{\Sigma }}_e$ anymore. In other words, the policy creates a complicated dependence between ${\tilde{\mathbf{x}}}_{\tau }{\tilde{\mathbf{x}}}_{\tau }^{\top }$ and $1_{\left\{a_{\tau }=0\right\}}$ for each $\tau$, which changes the limit of $\frac{1}{t}{\sum }_{\tau \in \left[t\right]}{1}_{\left\{a_{\tau }=a\right\}}\left({\tilde{\mathbf{x}}}_{\tau }{\tilde{\mathbf{x}}}_{\tau }^{\top }-{\mathbf{\Sigma }}_e\right)$ (and similarly $\left.\frac{1}{t}{\sum }_{\tau \in \left[t\right]}{1}_{\left\{a_{\tau }=a\right\}}{\tilde{\mathbf{x}}}_{\tau }{r}_{\tau }\right)$. This leads to the inconsistency of the naive estimator (See Appendix B for a concrete example). In Section 3, we provide examples to show that (2.4) not only deviates from the true parameters, but also leads to suboptimal decision-making.

Our proposed estimator.

Inspired by the above observations, for ${\pi }_{\tau }\left(a\mid {\tilde{\mathbf{x}}}_{\tau },{\mathscr{H}}_{\tau -1}\right)$ positive, we construct the following estimator for ${\mathit{\theta }}_a^{*}$ given ${\mathscr{H}}_t$, which corrects (2.4) using importance weights:

$${\widehat{\mathit{\theta }}}_a^{\left(t\right)}:={\left({\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},a}^{\left(t\right)}-\frac{1}{t}{\mathbf{\Sigma }}_{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}\left(a\right){\mathbf{\Sigma }}_{e,\tau }\right)}^{-1}\cdot {\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},r,a}^{\left(t\right)},$$ (2.5)

where

$${\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},a}^{\left(t\right)} =\frac{1}{t}\sum _{\tau \in \left[t\right]}\frac{{\pi }_{\tau }^{nd}\left(a_{\tau }\right)}{{\pi }_{\tau }\left(a_{\tau }\mid {\tilde{\mathbf{x}}}_{\tau },{\mathscr{H}}_{\tau -1}\right)}{1}_{\left\{a_{\tau }=a\right\}}{\tilde{\mathbf{x}}}_{\tau }{\tilde{\mathbf{x}}}_{\tau }^{\top },$$

$${\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},r,a}^{\left(t\right)} =\frac{1}{t}\sum _{\tau \in \left[t\right]}\frac{{\pi }_{\tau }^{nd}\left(a_{\tau }\right)}{{\pi }_{\tau }\left(a_{\tau }\mid {\tilde{\mathbf{x}}}_{\tau },{\mathscr{H}}_{\tau -1}\right)}{1}_{\left\{a_{\tau }=a\right\}}{\tilde{\mathbf{x}}}_{\tau }{r}_{\tau }.$$

Here, ${\left({\pi }_{\tau }^{nd}(\cdot )\right)}_{\tau \in \left[t\right]}$ is a pre-specified policy (doesn’t depend on ${\left({\tilde{\mathbf{x}}}_{\tau }\right)}_{\tau }$ or ${\mathscr{H}}_{\tau -1}$) that can be chosen by the algorithm. We only require the following:

Assumption 2.2. Let ${\mathit{w}}_t=\sqrt{d}\cdot {\mathbf{x}}_t,{\tilde{\mathit{w}}}_t=\sqrt{d}\cdot {\tilde{\mathbf{x}}}_t$ be scaled vectors of ${\mathbf{x}}_t$ and ${\tilde{\mathbf{x}}}_t$. Then there exist positive constants $\xi ,\rho ,{\lambda }_0$ s.t. (i) $\forall \mathit{u}\in {\mathbb{S}}^{d-1},\mathbb{E}{\left({\mathit{u}}^{\top }{\tilde{\mathit{w}}}_t\right)}^4\le \xi$, (ii) $\mathbb{E}{\tilde{\mathit{w}}}_t{\tilde{\mathit{w}}}_t^{\top }⪯\nu {\mathit{I}}_d$, and (iii) $\forall t\ge \sqrt{T,}a\in \left\{\mathrm{0,1}\right\},\frac{1}{t}{\sum }_{\tau \le t}{\pi }_{\tau }^{nd}\left(a\right){\mathit{w}}_{\tau }{\mathit{w}}_{\tau }^{\top }⪰{\lambda }_0{\mathit{I}}_d$.

Remark 2.1. In Assumption 2.2, (i) and (ii) are standard moment assumptions. (iii) is mild. Even restricted to the choice of ${\pi }_{\tau }^{nd}\left(a\right)\equiv 1/2$, under mild conditions, the assumption can be satisfied with deterministic ${\left({\mathbf{x}}_{\tau }\right)}_{\tau \ge 1}$ or stochastic ${\left({\mathbf{x}}_{\tau }\right)}_{\tau \ge 1}$ such as an i.i.d. sequence, a weakly dependent stationary time series (e.g. multivariate ARMA process (Fan and Yao, 2017)), or a sequence with periodicity/seasonality with high probability (See Appendix D for details).

The theorem below gives a high-probability upper bound on ${∥{\widehat{\mathit{\theta }}}_a^{\left(t\right)}-{\mathit{\theta }}_a^{*}∥}_2$ (proof in Appendix D).

Theorem 2.1. For any $t\in \left[T\right]$, denote $q_t:={\mathrm{i}\mathrm{n}\mathrm{f}}_{\tau \le t,a\in \left\{\mathrm{0,1}\right\}}{\pi }_{\tau }\left(a\mid {\tilde{\mathbf{x}}}_{\tau },{\mathscr{H}}_{\tau -1}\right)$. Then under Assumptions 2.1 and 2.2, there exist absolute constants $C,C_1$, such that as long as $q_t\ge C_1\text{max}\left\{\frac{d(d+\mathrm{l}\mathrm{o}\mathrm{g}t)}{{\lambda }_{0}t},\frac{\xi (d+\mathrm{l}\mathrm{o}\mathrm{g}t)}{{\lambda }_0^{2}t}\right\}$, with probability at least $1-\frac{8}{t^2},\forall a\in \left\{\mathrm{0,1}\right\},{∥{\widehat{\mathit{\theta }}}_a^{\left(t\right)}-{\mathit{\theta }}_a^{*}∥}_2$is upper bounded by

$$\frac{C\left(R+R_{\theta }\right)d}{{\lambda }_0}\text{max}\left\{\frac{d+\mathrm{l}\mathrm{o}\mathrm{g}t}{q_{t}t},\frac{\sqrt{\nu }+\sqrt{\xi }}{\sqrt{d}}\sqrt{\frac{d+\mathrm{l}\mathrm{o}\mathrm{g}t}{q_{t}t}}\right\}.$$ (2.6)

Unlike the existing literature on off-policy learning in contextual bandits (e.g. Wang et al. (2017); Zhan et al. (2021); Zhang et al. (2021); Bibaut et al. (2021)), the role of the importance weights here is to correct the dependence of a policy on the observed noisy context with error. The proof idea can be generalized to a large class of off-policy method-of-moment estimators, which might be of independent interest (see Appendix D).

2.3. MEB: Online bandit algorithm with measurement error adjustment

We propose MEB (Measurement Error Bandit), an online bandit algorithm with measurement error adjustment based on the estimator (2.5). The algorithm is presented in Algorithm 1 and is designed for the binary-action setting, although it can be generalized to the case with multiple actions (see Appendix C). For $t\le T_0$, the algorithm is in a warm-up stage and can pick any policy such that there is a minimum sampling probability $p_0^{\left(t\right)}$ for each action (Here $\left.p_0^{\left(t\right)}\in \left(0,\frac{1}{2}\right]\right)$. For instance, the algorithm can do pure exploration with ${\pi }_t\left(a\mid {\tilde{\mathbf{x}}}_t,{\mathscr{H}}_{t-1}\right)\equiv \frac{1}{2}$. For $t>T_0$, given the noisy context ${\tilde{\mathbf{x}}}_t$, the algorithm computes the best action ${\tilde{a}}_t$ according to ${\left({\widehat{\mathit{\theta }}}_a^{(t-1)}\right)}_{a\in \left\{\mathrm{0,1}\right\}}$ calculated from (2.5). Then, it samples ${\tilde{a}}_t$ with probability $1-p_0^{\left(t\right)}$ and keeps an exploration probability of $p_0^{\left(t\right)}$ to sample the other action. In practice, we can often set $p_0^{\left(t\right)}$ to be monotonically decreasing in $t$, in which case $q_t={\mathrm{i}\mathrm{n}\mathrm{f}}_{\tau \le t,a\in \left\{\mathrm{0,1}\right\}}{\pi }_{\tau }\left(a\mid {\tilde{\mathbf{x}}}_{\tau },{\mathscr{H}}_{\tau -1}\right)=p_0^{\left(t\right)}$ for all $t\in \left[T\right]$.

Before presenting the regret analysis, we should first note that our problem is harder than a standard contextual bandit: ${\mathbf{x}}_t$ is unknown, and only ${\tilde{\mathbf{x}}}_t$ is observed. Thus, even if ${\left({\mathit{\theta }}_a^{*}\right)}_{a\in \left\{\mathrm{0,1}\right\}}$ is known, we may still perform suboptimally if ${\tilde{\mathbf{x}}}_t$ is too far from ${\mathbf{x}}_t$ so that it leads to a different optimal action. Example 2.1 below shows that in general, we cannot avoid a linear regret.

Example 2.1. Let $d=1,\left({\mathit{\theta }}_1^{*},{\mathit{\theta }}_0^{*}\right)=(1,-1).{\left({\mathbf{x}}_t\right)}_{t\in \left[T\right]}$ are drawn i.i.d. from $\{\pm 0.2\}$ with equal probability. $\mathbb{P}\left({\tilde{\mathbf{x}}}_t=1\mid {\mathbf{x}}_t=0.2\right)=0.6,\mathbb{P}\left({\tilde{\mathbf{x}}}_t=-1\mid {\mathbf{x}}_t=0.2\right)=0.4;\mathbb{P}\left({\tilde{\mathbf{x}}}_t=1\mid {\mathbf{x}}_t=-0.2\right)=0.4,\mathbb{P}\left({\tilde{\mathbf{x}}}_t=-1\mid {\mathbf{x}}_t=-0.2\right)=0.6$. Intuitively, even if we know ${\left({\mathit{\theta }}_a^{*}\right)}_{a\in \left\{\mathrm{0,1}\right\}}$, there is still a constant probability at each time $t$ that we cannot make the right choice due to ${\tilde{\mathbf{x}}}_t$ and ${\mathbf{x}}_t$ having different signs, and ${\mathbf{x}}_t$ is never known (details in Appendix E). This results in a $\Omega \left(T\right)$ regret.

Fortunately, in practice, we expect that the errors ${\left({ϵ}_t\right)}_{t\in \left[T\right]}$ are relatively ‘small’ in the sense that the optimal action (given $\left.{\left({\mathit{\theta }}_a^{*}\right)}_{a\in \left\{\mathrm{0,1}\right\}}\right)$ is not affected. Specifically, we assume the following:

Assumption 2.3. There exist a constant $\rho \in \left(\mathrm{0,1}\right)$ such that $\forall t\in \left[T\right],\left|⟨{\mathit{\delta }}_{\theta },{\mathit{ϵ}}_t⟩\right|\le \rho \left|⟨{\mathit{\delta }}_{\theta },{\mathbf{x}}_t⟩\right|$ almost surely. Here ${\mathit{\delta }}_{\theta }:={\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*}$.

Assumption 2.3 ensures that the perturbation to the suboptimality gap between the two arms caused by ${\mathit{ϵ}}_t$ is controlled by the true suboptimality gap. In this way, given ${\left({\mathit{\theta }}_a^{*}\right)}_{a\in \left\{\mathrm{0,1}\right\}}$, the optimal action based on ${\tilde{\mathbf{x}}}_t$ will not deviate too much from that based on ${\mathbf{x}}_t$. As a special case, this assumption is satisfied with if $\forall t,\left|⟨{\mathit{\delta }}_{\theta },{\mathbf{x}}_t⟩\right|\ge B_{e,t}{∥{\mathit{\delta }}_{\theta }∥}_2/\rho$. Here $B_{e,t}$ is an upper bound of ${∥{\mathit{ϵ}}_t∥}_2$. Assumption 2.3 can be further weakened to the inequalities holding with high probability (see Appendix E). Note that Assumption 2.3 only guarantees the optimal action is not affected by ${\left({\mathit{ϵ}}_t\right)}_{t\in \left[T\right]}$ given ${\left({\mathit{\theta }}_a^{*}\right)}_{a\in \left\{\mathrm{0,1}\right\}}$. To achieve sublinear regret, ${\left({\mathit{\theta }}_a^{*}\right)}_{a\in \left\{\mathrm{0,1}\right\}}$ still needs to be well-estimated. Thus, even with Assumption 2.3, classical bandit algorithms such as UCB may still suffer from linear regret because of the inconsistent estimator ${\widehat{\mathit{\theta }}}_{a,\mathit{\text{RLSCE}}}^{\left(t\right)}$ (see Appendix B for a concrete example).

We first prove the following theorem, which states that the regret of MEB can be directly controlled by the estimation error. In fact, this theorem holds regardless of the form or quality of the estimation procedure (i.e. in line 7 of Algorithm 1). The proof is in Appendix E.

Theorem 2.2. Let Assumption 2.1 and 2.3 hold.

1. For the standard setting, Algorithm 1 outputs a policy with $\mathrm{R}\mathrm{e}\mathrm{g}\mathrm{r}\mathrm{e}\mathrm{t}\left(T;{\pi }^{*}\right)$ no more than

   $$2T_0{R}_{\theta }+\frac{2}{1-\rho }\cdot \sum _{t=T_0+1}^T\left(p_0^{\left(t\right)}{R}_{\theta }+\underset{a\in \left\{\mathrm{0,1}\right\}}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{\left(t-1\right)}-{\mathit{\theta }}_a^{*}∥}_2\right).$$
2. For the clipped policy setting, Algorithm 1 with the choice of $p_0^{\left(t\right)}\equiv p_0$ outputs a policy with $\mathrm{R}\mathrm{e}\mathrm{g}\mathrm{r}\mathrm{e}\mathrm{t}\left(T;{\bar{\pi }}^{*}\right)$ no more than

   $$2T_0{R}_{\theta }+\frac{2\left(1-2p_0\right)}{1-\rho }\cdot \sum _{t=T_0+1}^T\underset{a\in \left\{\mathrm{0,1}\right\}}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{\left(t-1\right)}-{\mathit{\theta }}_a^{*}∥}_2.$$

The following corollary provides regret guarantees of MEB by combining Theorem 2.1 and 2.2 (proof in Appendix E).

Corollary 2.1. Let Assumption 2.1 to 2.3 hold. There exist universal constants $C,C\prime$ such that:

1. For the standard setting, $\forall T\ge C\prime \text{max}\left\{\left(1+1/{\lambda }_0^{\frac{9}{4}}\right)(d+\mathrm{l}\mathrm{o}\mathrm{g}T{)}^3,{\left(\xi /{\lambda }_0\right)}^{\frac{9}{4}}(d+\mathrm{l}\mathrm{o}\mathrm{g}T{)}^{\frac{4}{3}}\right\}$, with probability at least $1-\frac{16}{\sqrt{T}}$, Algorithm 1 with the choice of $T_0=⌈2d{T}^{\frac{2}{3}}⌉,p_0^{\left(t\right)}=\mathrm{m}\mathrm{i}\mathrm{n}\left\{\frac{1}{2},t^{-\frac{1}{3}}\right\}$ outputs a policy with $\mathrm{R}\mathrm{e}\mathrm{g}\mathrm{r}\mathrm{e}\mathrm{t}\left(T;{\pi }^{*}\right)$ no more than

   $$Cd{T}^{\frac{2}{3}}\left\{\frac{R_{\theta }}{1-\rho }+\frac{(\sqrt{\nu }+\sqrt{\xi }+1)\left(R+R_{\theta }\right)}{(1-\rho ){\lambda }_0}\sqrt{1+\frac{\mathrm{l}\mathrm{o}\mathrm{g}T}{d}}\right\}.$$
2. For the clipped policy setting, $\forall T\ge C\prime \text{max}\left\{(d+\mathrm{l}\mathrm{o}\mathrm{g}T{)}^2/{\left({\lambda }_0{p}_0\right)}^2,{\xi }^2/{\lambda }_0^4(1+\mathrm{l}\mathrm{o}\mathrm{g}T/d{)}^2\right\}$, with probability at least $1-\frac{16}{\sqrt{T}}$, Algorithm 1 with the choice of $T_0=\lceil 2d\sqrt{T}\rceil$ and $p_0^{\left(t\right)}\equiv p_0$ outputs a policy with $\mathrm{R}\mathrm{e}\mathrm{g}\mathrm{r}\mathrm{e}\mathrm{t}\left(T;{\bar{\pi }}^{*}\right)$ no more than

   $$Cd{T}^{\frac{1}{2}}\left\{R_{\theta }+\frac{(\sqrt{\nu }+\sqrt{\xi }+1)\left(1-2p_0\right)\left(R+R_{\theta }\right)}{\sqrt{p_0}(1-\rho ){\lambda }_0}\sqrt{1+\frac{\mathrm{l}\mathrm{o}\mathrm{g}T}{d}}\right\}.$$

Ignoring other factors, the regret upper bound is of order $\widetilde{𝒪}\left(d{T}^{2/3}\right)$ for the standard setting, and $\widetilde{𝒪}\left(d\sqrt{T}\right)$ for the clipped policy setting, depending on the horizon $T$ and dimension $d$.

In certain scenarios (e.g. when $d$ is large), it is desirable to save computational resources by updating the estimates of ${\left({\mathit{\theta }}_a^{*}\right)}_{a\in \left\{\mathrm{0,1}\right\}}$ less frequently in Algorithm 1. Fortunately, low regret guarantees can still be achieved: Suppose at each time $t$, the agent only updates the estimators according to (2.5) at selected time points $t\in 𝒮\subseteq \left[T\right]$ (in line 7); Otherwise, the agent simply makes decisions based on the most recently updated estimators. In Appendix E, we show that time points to perform the updates can be very infrequent, such as ${\left(n^k\right)}_{k\in {\mathbb{N}}^{+}}\left(n\ge 2,n\in {\mathbb{N}}^{+}\right)$, while still achieving the same rate of regret upper bound as in Corollary 2.1.

2.4. MEB given estimated error variance

In practice, the agent might not have perfect knowledge about ${\mathbf{\Sigma }}_{e,t}$, the variance of the error ${\mathit{ϵ}}_t$. In this section, we discuss the situation where at each time $t$, the agent does not know ${\mathbf{\Sigma }}_{e,t}$, and only has a (potentially adaptive) estimator ${\widehat{\mathbf{\Sigma }}}_{e,t}$ for ${\mathbf{\Sigma }}_{e,t}$. This estimator may be derived from auxiliary data or outside knowledge. In this case, in Algorithm 1, we need to replace the estimator (2.5) with the following estimator for decision-making (i.e. in line 7 of Algorithm 1):

$${\tilde{\mathit{\theta }}}_a^{\left(t\right)}:={\left({\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},a}^{\left(t\right)}-\frac{1}{t}{\mathbf{\Sigma }}_{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}(a){\widehat{\mathbf{\Sigma }}}_{e,\tau }\right)}^{-1}\cdot {\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},r,a}^{\left(t\right)}.$$ (2.7)

In Appendix F, we show that with this modification, the additional regret of Algorithm 1 is controlled by

$$d\cdot {\mathbf{\Sigma }}_{t=T_0+1}^T{\text{max}}_{a\in \left\{\mathrm{0,1}\right\}}{∥{\Delta }_t\left(a\right)∥}_2$$

up to a constant depending on the assumptions. Here, for each $t,{\mathbf{\Delta }}_t\left(a\right):=\frac{1}{t}{\sum }_{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}\left(a\right)\left({\widehat{\mathbf{\Sigma }}}_{e,\tau }-\right. \left.{\mathbf{\Sigma }}_{e,\tau }\right)$ is the weighted average of the estimation errors ${\left({\widehat{\mathbf{\Sigma }}}_{e,\tau }-{\mathbf{\Sigma }}_{e,\tau }\right)}_{\tau \in \left[t\right]}$. In practice, it is reasonable to assume that ${\mathbf{\Delta }}_t\left(a\right)$ is small so as not to significantly affect the overall regret: For example, suppose the agent gathers more auxiliary data over time so that ${∥d\left({\widehat{\mathbf{\Sigma }}}_{e,t}-{\mathbf{\Sigma }}_{e,t}\right)∥}_2\lesssim \sqrt{d/t}$, then the additional regret term will be $𝒪\left(\sqrt{dT}\right)$ up to a constant depending on the assumptions.

3. Simulation results

In this section, we complement our theoretical analyses with simulation results on a synthetic environment with artificial noise and reward models as well as a simulation environment based on the real dataset, HeartStep V1 (Klasnja et al., 2018).

Compared algorithms.

In both simulation environments, we compare the following algorithms: Thompson sampling (TS) with normal priors (Russo et al., 2018), Linear Upper Confidence Bound (UCB) approach (Chu et al., 2011), MEB (Algorithm 1), and MEB-naive (MEB plugged in with the naive measurement error estimator (2.4) instead of (2.5)). See Appendix A for a detailed description of the algorithms.

3.1. Synthetic environment

We first test our algorithms on a synthetic environment. We consider a contextual bandit environment with $d=5,T=50000$. In the reward model, we set ${\mathit{\theta }}_0^{*}=(\mathrm{5,6},\mathrm{4,6},4),{\mathit{\theta }}_1^{*}=(\mathrm{6,5},\mathrm{5,5},5)$, and ${\eta }_t$ drawn i.i.d. from $𝒩\left(0,{\sigma }_{\eta }^2\right)$. Let ${\left({\mathbf{x}}_t\right)}_{t\in \left[T\right]}$ be independently sampled from $𝒩\left({\mathit{\mu }}_x,{\mathit{I}}_d\right)$, where ${\mathit{\mu }}_x=1_d$. We further set ${\mathbf{\Sigma }}_{e,t}\equiv {\mathbf{\Sigma }}_e:={\mathit{I}}_d/4$ and consider independent ${\left({\mathit{ϵ}}_t\right)}_{t\in \left[T\right]}$ with Normal distributionwith covariance ${\mathbf{\Sigma }}_e$. We independently generate bandit data for $n_{\mathit{\text{exp}}}=100$ times, and compare among the candidate algorithms in terms of estimation quality and cumulative regret with a moderate exploration probability $p_0=0.2$.

3.2. HeartStep V1 simulation environment

We also construct a simulation environment with HeartSteps dataset. HeartSteps is a physical activity mobile health application, whose primary goal is to help the user prevent negative health outcomes and adopt and maintain healthy behaviors, for example, higher physical activity level. HeartSteps V1 is a 42-day mobile health trial (Dempsey et al., 2015; Klasnja et al., 2015; Liao et al., 2016), where participants are provided a Fitbit tracker and a mobile phone application. One of the intervention components is a contextually-tailored physical activity suggestion that may be delivered at any of the five user-specified times during each day. The delivery times are roughly separated by 2.5 hours.

Construction of the simulated environment.

We follow the simulation setups in Liao et al. (2020). The true context at the time $t$ is denoted by ${\mathbf{x}}_t$ with three main components ${\mathbf{x}}_t=\left(I_t,Z_t,B_t\right)$. Here, $I_t$ is an indicator variable of whether an intervention $\left(A_t=1\right)$ is feasible (e.g. $I_t$ is 0 when the participant is driving a car, a situation where the suggestion should not be sent). $Z_t$ contains some features at time $t.B_t$ is the true treatment burden, which is a function of the participant’s treatment history². Specifically, $B_{t+1}=\lambda B_t+1_{\left\{A_t=1\right\}}$. We assume that ${\left(I_t\right)}_{t\in \left[T\right]}$ and ${\left(Z_t\right)}_{t\in \left[T\right]}$ are sampled i.i.d with the empirical distribution from the Heartstep V1 dataset, and ${\left(B_t\right)}_{t\in \left[T\right]}$ is given by the aforementioned transition model.

The reward model is $r_t(\mathbf{x},a;\mathit{\theta })={\mathbf{x}}^{\top }\mathit{\alpha }+af(\mathbf{x}{)}^{\top }\mathit{\beta }+{\eta }_t$, where $\mathbf{x}$ is the full context, $f\left(\mathbf{x}\right)$ is a subset of $\mathbf{x}$ that is considered to have an impact on the treatment effects, and $\mathit{\theta }={\left({\mathit{\alpha }}^{\top },{\mathit{\beta }}^{\top }\right)}^{\top }\in {\mathbb{R}}^9$. Here ${\eta }_t$ is the Gaussian noise on the reward observation, whose variance ${\sigma }_{\eta }^2$ is chosen to be 0.1, 1.0, and 5.0 respectively (Liao et al., 2016). For a detailed list of variables in the context, see Table 2 in Appendix A.

The true parameters ${\left({\mathit{\theta }}_a^{*}\right)}_{a\in \left\{\mathrm{0,1}\right\}}$ is estimated from GEE (Generalized Estimating Equations) with rewards being the log-transformed step count collected 30 minutes after the decision time.

In light of the measurement error setting in this paper, we consider an observation noise on $B_t$ for the following reasons: 1) The burden $B_t$ can be understood as a prediction of the burden level of the participant, which is particularly crucial in mobile health studies; 2) Other variables are normally believed to have low or no observation noise. Thus, we assume that the agent only observes ${\tilde{\mathbf{x}}}_t=\left(I_t,Z_t,{\tilde{B}}_t\right)$, where ${\tilde{B}}_t=B_t+{ϵ}_t$ and ${ϵ}_t$ is drawn i.i.d. from normal distribution with mean zero and variance ${\sigma }_{ϵ}^2$.

3.3. Results

Table 1 (a) and (b) shows the average regret (cumulative regret divided by $T$) in both the synthetic environment and the real-data environment based on HeartStep V1. We use the same set of ${\sigma }_{ϵ}^2\in \{0.1,1.0,2.0\}$, while different ${\sigma }_{\eta }^2$ reflect the change of absolute values in coefficients in two different environments $\left({\sigma }_{\eta }^2=5.0\right.$ is the level of reward noise in HeartStep V1). MEB shows significantly smaller average regret compared to other baseline methods under most combinations of ${\sigma }_{\eta }^2$ and ${\sigma }_{ϵ}^2$. 1. In certain instances, MEB-naive exhibits performance comparable to MEB. This is attributed to its ability to reduce the variance of model estimation while incurring some bias compared to MEB, rendering it a feasible alternative in practical contexts. Notably, in two extreme scenarios, UCB surpasses both MEB and MEB-naive. This is as expected, since when contextual noise is sufficiently negligible, traditional bandit algorithms are anticipated to outperform the proposed algorithms. An estimation error plot can be found in Appendix A, which also demonstrates that MEB has a lower estimation error.

Table 1:

Average regret for both synthetic environment and real-data environment under different combinations of ${\sigma }_{\eta }^2$ and ${\sigma }_{ϵ}^2$. The results are averages over 100 independent runs and the standard deviations are reported in the full table in Appendix A.

(a) Average regret in the synthetic environment over 50000 steps with clipping probability p = 0.2.
${\mathit{\sigma }}_{\mathit{\eta }}^2$	${\mathit{\sigma }}_{\mathit{ϵ}}^2$	TS	UCB	MEB	MEB-naive
0.01	0.1	0.047	0.046	0.027	0.038
0.1	0.1	0.047	0.047	0.026	0.039
1.0	0.1	0.048	0.048	0.027	0.038
0.01	1.0	0.757	0.647	0.198	0.371
0.1	1.0	0.769	0.721	0.205	0.392
1.0	1.0	0.714	0.697	0.218	0.404
0.01	2.0	1.492	1.504	0.358	0.616
0.1	2.0	1.195	1.333	0.368	0.584
1.0	2.0	1.299	1.476	0.416	0.625
(b) Average regret in tde real-data environment over 2500 steps witd clipping probability p = 0.2.
${\mathit{\sigma }}_{\mathit{\eta }}^2$	${\mathit{\sigma }}_{\mathit{ϵ}}^2$	TS	UCB	MEB	MEB-naive
0.05	0.1	0.027	0.027	0.022	0.024
0.1	0.1	0.026	0.024	0.020	0.020
5.0	0.1	1.030	0.743	0.831	1.173
0.05	1.0	0.412	0.408	0.117	0.112
0.1	1.0	0.309	0.316	0.085	0.087
5.0	1.0	1.321	0.918	1.458	1.322
0.05	2.0	0.660	0.634	0.144	0.148
0.1	2.0	0.740	0.704	0.151	0.155
5.0	2.0	1.585	2.415	1.577	1.436

4. Discussion and conclusions

We propose a new algorithm, MEB, which is the first algorithm with sublinear regret guarantees in contextual bandits with noisy context, where we have limited knowledge of the noise distribution. This setting is common in practice, especially where only predictions for unobserved context are available. MEB leverages the novel estimator (2.5), which extends the conventional measurement error adjustment techniques by considering the interplay between the policy and the measurement error.

Limitations and future directions.

Several questions remain for future investigation. First, is $\widetilde{𝒪}\left(T^{2/3}\right)$ the optimal rate of regret compared to the standard benchmark policy (2.3), as in some other bandits with semi-parametric reward model (e.g. Xu and Wang (2022))? Providing lower bounds on the regret helps us understand the limit of improvement in the online algorithm. Second, we assume that the agent has an unbiased prediction of the true context. It is important to understand how biased predictions affect the results. Last but not least, it’s interesting to see our method can be extended to more complicated decision-making settings (e.g. Markov decision processes).

A. Additional details for simulation studies

A. 1. Compared algorithms

In both simulation environments, we compare the following algorithms: Thompson sampling (TS, see details in Algorithm 2) given normal priors (Russo et al., 2018), Linear Upper Confidence Bound (UCB, see details in Algorithm 3) approach (Chu et al., 2011), MEB (Algorithm 1), and MEB-naive (MEB plugged in with the naive measurement error estimator (2.4) instead of (2.5)). To make a fair comparison between algorithms, we use the same regularization parameter $l=1$ for all algorithms. The hyper-parameter $\rho ,C$ is set to be ${\sigma }_{\eta }^2$ for all results for TS and UCB.

We further compare with robust linear UCB (Algorithm 1 in Ding et al. (2022)) that is shown to achieve minimax rate for adversarial linear bandit.

A. 2. Additional details for HeartStep V1 study

Table 2 presents the list of variables to include in the reward model and in the feature construction for algorithms. Recall that our reward model is $r_t(\mathit{x},a,\mathit{\theta })={\mathit{x}}^{\top }\mathit{\alpha }+af(\mathit{x}{)}^{\top }\mathit{\beta }+{\eta }_t$. All the variables are included in $\mathit{x}$ while only those considered to have an impact on treatment effect will be included in $f\left(\mathit{x}\right)$.

Table 2:

List of variables in HeartSteps V1 study.

Variable	Type	Treatment?
Availability (It)	Discrete	No
Prior 30-minute step count	Continuous	No
Yesterday’s step count	Continuous	No
Prior 30-minute step count	Continuous	No
Location	Discrete	Yes
Current temperature	Continuous	No
Step variation level	Discrete	Yes
Burden variable (Bt)	Continuous	Yes

A.3. Additional results on estimation error

Figure 1:

Log-scaled L2 norm of ${\widehat{\mathit{\theta }}}_1-{\mathit{\theta }}_1^{*}$ of four algorithms in the synthetic environment over 50000 steps under ${\sigma }_{ϵ}^2\in \{0.1,1.0,2.0\}$ and ${\sigma }_{\eta }^2\in \{0.01,0.1,1.0\}$.

Figure 2:

Log-scaled L2 norm of ${\widehat{\mathit{\theta }}}_1-{\mathit{\theta }}_1^{*}$ of four algorithms in the real-data environment based on HeartStep V1 over 2500 steps under ${\sigma }_{ϵ}^2\in \{0.1,1.0,2.0\}$ and ${\sigma }_{\eta }^2\in \{0.05,0.1,5.0\}$.

A. 4. Average regret with standard deviation

Table 3:

Average regret for both synthetic environment and real-data environment under different combinations of ${\sigma }_{\eta }^2$ and ${\sigma }_{ϵ}^2$. The numbers in parentheses are the standard deviations calculated from 100 independent runs.

(a) Average regret in the synthetic environment over 50000 steps with clipping probability p = 0.2.
${\mathit{\sigma }}_{\mathit{\eta }}^2$	${\mathit{\sigma }}_{\mathit{ϵ}}^2$	TS	UCB	MEB	MEB-naive	RobustUCB
0.01	0.1	0.047 (0.0015)	0.046 (0.0015)	0.027 (0.0011)	0.038 (0.0013)	0.050 (0.0051)
0.1	0.1	0.047 (0.0015)	0.047 (0.0015)	0.026 (0.0011)	0.039 (0.0013)	0.049 (0.0048)
1.0	0.1	0.048 (0.0015)	0.048 (0.0015)	0.027 (0.0011)	0.038 (0.0013)	0.044 (0.0047)
0.01	1.0	0.757 (0.0164)	0.647 (0.0145)	0.198 (0.0079)	0.371 (0.0107)	0.652 (0.0050)
0.1	1.0	0.769 (0.0160)	0.721 (0.0156)	0.205 (0.0080)	0.392 (0.0110)	0.753 (0.0056)
1.0	1.0	0.714 (0.0155)	0.697 (0.0150)	0.218 (0.0083)	0.404 (0.0112)	0.589 (0.0047)
0.01	2.0	1.492 (0.0281)	1.504 (0.0283)	0.358 (0.0129)	0.616 (0.0169)	1.608 (0.0102)
0.1	2.0	1.195 (0.0244)	1.333 (0.0260)	0.368 (0.0131)	0.584 (0.0164)	1.064 (0.0079)
1.0	2.0	1.299 (0.0257)	1.476 (0.0277)	0.416 (0.0139)	0.625 (0.0170)	1.881 (0.0114)
(b) Average regret in tde real-data environment over 2500 steps witd clipping probability p = 0.2.
${\mathit{\sigma }}_{\mathit{\eta }}^2$	${\mathit{\sigma }}_{\mathit{ϵ}}^2$	TS	UCB	MEB	MEB-naive	RobustUCB
0.05	0.1	0.027 (0.0067)	0.027 (0.0070)	0.022 (0.0057)	0.024 (0.0058)	0.025 (0.0079)
0.1	0.1	0.026 (0.0057)	0.024 (0.0053)	0.020 (0.0046)	0.020 (0.0046)	0.028 (0.0079)
5.0	0.1	1.030 (0.0287)	0.743 (0.0262)	0.831 (0.0267)	1.173 (0.0343)	1.400 (0.0447)
0.05	1.0	0.412 (0.0355)	0.408 (0.0351)	0.117 (0.0148)	0.112 (0.0143)	0.226 (0.0020)
0.1	1.0	0.309 (0.0293)	0.316 (0.0299)	0.085 (0.0112)	0.087 (0.0116)	0.206 (0.0125)
5.0	1.0	1.321 (0.0417)	0.918 (0.0304)	1.458 (0.0422)	1.322 (0.0388)	1.065 (0.0400)
0.05	2.0	0.660 (0.0343)	0.634 (0.0322)	0.144 (0.0129)	0.148 (0.0133)	0.304 (0.0241)
0.1	2.0	0.740 (0.0505)	0.704 (0.0489)	0.151 (0.0145)	0.155 (0.0149)	0.432 (0.0386)
5.0	2.0	1.585 (0.0454)	2.415 (0.0816)	1.577 (0.0508)	1.436 (0.0462)	1.345 (0.0423)

B. Additional explanations on the regularized least-squares (RLS) estimator the naive estimator (2.4) under noisy context

B. 1. Inconsistency of the RLS estimator

Measurement error model and attenuation.

As briefly mentioned in the main text, a measurement error model is a regression model designed to accommodate inaccuracies in the measurement of regressors. Suppose that there is no action (i.e. set $\left.a_{\tau }\equiv 0\right)$, and ${\left({\mathbf{x}}_{\tau },{\mathit{ϵ}}_{\tau },{\eta }_{\tau }\right)}_{\tau \in \left[t\right]}$ are i.i.d., then the measurement error model is a useful tool to learn ${\mathit{\theta }}_0^{*}$ from ${\mathscr{H}}_t$ collected as follows:

$$\left\{\begin{array}{l}{r}_{\tau }=⟨{\mathit{\theta }}_0^{*},{\mathbf{x}}_{\tau }⟩+{\eta }_{\tau },\\ {\tilde{\mathbf{x}}}_{\tau }={\mathbf{x}}_{\tau }+{\mathit{ϵ}}_{\tau }.\end{array}\right.$$ (B.1)

Here in the measurement error model’s perspective, ${\left({\tilde{\mathbf{x}}}_{\tau }\right)}_{\tau \in \left[t\right]}$ are regressors ‘measured with error’, and ${\left(r_{\tau }\right)}_{\tau \in \left[t\right]}$ are dependent variables.

Regression attenuation, a phenomenon intrinsic to measurement error models, refers to the observation that when the predictors are subject to measurement errors, the Ordinary Least Squares (OLS) estimators of regression coefficients become biased (see, for instance, Carroll et al. (1995)). Specifically, in simple linear regression, the OLS estimator for the slope tends to be biased towards zero. Intuitively, this is because the measurement errors effectively ‘dilute’ the true relationship between variables, making it appear weaker than it actually is.

Inconsistency of the RLS estimator.

Before presenting a concrete numerical example to show the inconsistency of the RLS estimator and that it leads to bad decision-making, below we first apply the theory of measurement error model to give a heuristic argument of why the RLS estimator is inconsistent even in the simplified situation where there is no action (i.e. set $a_{\tau }\equiv 0$), and ${\left({\mathbf{x}}_{\tau },{\mathit{ϵ}}_{\tau },{\eta }_{\tau }\right)}_{\tau \in \left[t\right]}$ are i.i.d..

From Section 3.3.2 in Carroll et al. (1995), given data ${\left({\tilde{\mathbf{x}}}_{\tau },r_{\tau }\right)}_{\tau \in \left[t\right]}$ from (B.1) with multiple covariates (in the simplified case with no action as described above), the OLS estimator for ${\mathit{\theta }}_0^{*}$, denoted as ${\widehat{\mathit{\theta }}}_0^{OLS,t}$, consistently estimates not ${\mathit{\theta }}_0^{*}$ but

$${\tilde{\mathit{\theta }}}_0^{*}={\left({\mathbf{\Sigma }}_{\mathbf{x}}+{\mathbf{\Sigma }}_e\right)}^{-1}{\mathbf{\Sigma }}_{\mathbf{x}}{\mathit{\theta }}_0^{*},$$ (B.2)

where ${\mathbf{\Sigma }}_{\mathbf{x}}=\text{Var}\left({\mathbf{x}}_{\tau }\right),{\mathbf{\Sigma }}_e=\text{Var}\left({\mathit{ϵ}}_{\tau }\right)$. In addition, given fixed $\lambda$, the regularized least squares (RLS) estimator

$${\widehat{\mathit{\theta }}}_0^{RLS,t}\left(\lambda \right)={\left(\frac{\lambda }{t}I+\frac{1}{t}\sum _{\tau \in \left[t\right]}{\tilde{\mathbf{x}}}_{\tau }{\tilde{\mathbf{x}}}_{\tau }^{\top }\right)}^{-1}\left(\frac{1}{t}\sum _{\tau \in \left[t\right]}{\tilde{\mathbf{x}}}_{\tau }{r}_{\tau }\right)={\left(\frac{\lambda }{t}I+\frac{1}{t}\sum _{\tau \in \left[t\right]}{\tilde{\mathbf{x}}}_{\tau }{\tilde{\mathbf{x}}}_{\tau }^{\top }\right)}^{-1}\left(\frac{1}{t}\sum _{\tau \in \left[t\right]}{\tilde{\mathbf{x}}}_{\tau }{\tilde{\mathbf{x}}}_{\tau }^{\top }\right){\widehat{\mathit{\theta }}}_0^{OLS,t},$$

where $\frac{1}{t}{\sum }_{\tau \in \left[t\right]}{\tilde{\mathbf{x}}}_{\tau }{\tilde{\mathbf{x}}}_{\tau }^{\top }\to \text{Var}\left({\tilde{\mathbf{x}}}_{\tau }\right)=\text{Var}\left({\mathbf{x}}_{\tau }\right)+\text{Var}\left({\mathit{ϵ}}_{\tau }\right)$, and $\lambda /t\to 0$ as $t\to \infty$. This means that as $t\to \infty ,{\widehat{\mathit{\theta }}}_0^{RLS,t}\left(\lambda \right)$ and ${\widehat{\mathit{\theta }}}_0^{OLS,t}$ converges to the same limit, which is ${\tilde{\mathit{\theta }}}_0^{*}$. Thus, for fixed $\lambda$, as $t$ grows, ${∥{\widehat{\mathit{\theta }}}_0^{RLS,t}\left(\lambda \right)-{\mathit{\theta }}_0^{*}∥}_2$ converges to ${∥{\tilde{\mathit{\theta }}}_0^{*}-{\mathit{\theta }}_0^{*}∥}_2$ and does not converge to zero in general.

Finally, recall that in classical bandit algorithms such as UCB, the sublinear regret relies on the key property that with high probability, for all $t,{∥{\widehat{\mathit{\theta }}}_0^{RLS,t}\left(\lambda \right)-{\mathit{\theta }}_0^{*}∥}_{V_t}\le \beta$, where $V_t=\lambda I+{\sum }_{\tau \in \left[t\right]}{\tilde{\mathbf{x}}}_{\tau }{\tilde{\mathbf{x}}}_{\tau }^{\top },\beta =\widetilde{𝒪}\left(\sqrt{d}\right)$. Here for a vector $\mathit{v}\in {\mathbb{R}}^d$ and positive definite matrix $\mathbf{M}\in {\mathbb{R}}^{d\times d},\parallel \mathit{v}{\parallel }_{\mathbf{M}}:=\sqrt{{\mathit{v}}^{\top }\mathbf{M}\mathit{v}}$. We argue that this requirement generally no longer holds in the setting with measurement error ${\left({\mathit{ϵ}}_{\tau }\right)}_{\tau \ge 1}$. In fact, notice that since ${\tilde{\mathbf{x}}}_{\tau }$ is i.i.d. in this simplified setting, and $V_t=\lambda I+{\sum }_{\tau \in \left[t\right]}{\tilde{\mathbf{x}}}_{\tau }{\tilde{\mathbf{x}}}_{\tau }^{\top }$, we expect that $\frac{1}{t}{V}_t$ concentrates around $\text{Var}\left({\tilde{\mathbf{x}}}_t\right)$. As long as ${\lambda }_{\mathrm{m}\mathrm{i}\mathrm{n}}\left(\text{Var}\left({\tilde{\mathbf{x}}}_t\right)\right)>0$, with high probability, for all $t,\frac{1}{t}{V}_t\succ c$ for some constant $c$. If this holds,

$${∥{\widehat{\mathit{\theta }}}_0^{RLS,t}\left(\lambda \right)-{\mathit{\theta }}_0^{*}∥}_{V_t}\ge \sqrt{ct}{∥{\widehat{\mathit{\theta }}}_0^{RLS,t}\left(\lambda \right)-{\mathit{\theta }}_0^{*}∥}_2,$$

while the last term ${∥{\widehat{\mathit{\theta }}}_0^{RLS,t}\left(\lambda \right)-{\mathit{\theta }}_0^{*}∥}_2$ converges to a nonzero limit ${∥{\tilde{\mathit{\theta }}}_0^{*}-{\mathit{\theta }}_0^{*}∥}_2$. This indicates that in general, ${∥{\widehat{\mathit{\theta }}}_0^{RLS,t}\left(\lambda \right)-{\mathit{\theta }}_0^{*}∥}_{V_t}$ scales with a rate of at least $\sqrt{t}$, and will not be uniformly bounded by $\widetilde{𝒪}\left(\sqrt{d}\right)$ for all $t$.

An example. The following is an example where given the errors ${\left({\mathit{ϵ}}_{\tau }\right)}_{\tau \in \left[t\right]}$, the RLS estimator in the classical bandit algorithms inconsistently estimates the true reward model, and in addition, it leads to bad decision-making (linear regret) in the classic bandit algorithms.

Example B.1. Consider the standard setting described in Section 2.1. Let $=10000,d=2,{\mathit{\theta }}_1^{*}=(\mathrm{1,0}{)}^{\top },{\mathit{\theta }}_0^{*}=(-\mathrm{1,0}{)}^{\top }$. Let ${\mathbf{x}}_t$ sampled i.i.d. from $\mathrm{U}\mathrm{n}\mathrm{i}\mathrm{f}\left(𝒮\right)$, where $𝒮=\left\{(\mathrm{1,3}{)}^{\top },(-\mathrm{3,1}{)}^{\top },(-1,-3{)}^{\top },(3,-1{)}^{\top }\right\}$. Condition on ${\mathbf{x}}_t,{\mathit{ϵ}}_t$ is uniformly sampled from $\left({\left({\rho }_0{\mathbf{x}}_t^{\left[1\right]},{\rho }_0{\mathbf{x}}_t^{\left[1\right]}\right)}^{\top },{\left(-{\rho }_0{\mathbf{x}}_t^{\left[1\right]},-{\rho }_0{\mathbf{x}}_t^{\left[1\right]}\right)}^{\top }\right)$, independent from any other variable in the history. Here ${\rho }_0=0.9,{\mathbf{x}}_t^{\left[1\right]}$ denotes the first entry in ${\mathbf{x}}_t$. We also let ${\eta }_t$ be i.i.d. drawn from $N\left(\mathrm{0,0.01}\right)$.

We conduct 100 independent experiments. For each experiment, we generate data according to the above, and test the performance of UCB (Algorithm 1 in Chu et al. (2011)) and Thompson sampling with normal priors (Russo et al., 2018) using noisy context ${\tilde{\mathbf{x}}}_t$ instead of the true context. We choose the regularization parameter $\lambda =1$ in the RLS estimator. Additionally, in UCB (Chu et al., 2011), we choose the parameter $\alpha =1$. Figure 3 summarizes the estimation error of the RLS estimator and the cumulative regret of both algorithms with respect to time $t$, showing both the average and standard error across the random experiments. We see that the RLS estimator is unable to estimate the true reward model well. Moreover, it is clear that the regret of both UCB and Thompson sampling is linear in the time horizon. Intuitively, this is because the direction of ${\tilde{\mathit{\theta }}}_0^{*}$ in (B.2) is twisted compared to ${\mathit{\theta }}_0^{*}$, which not only leads to inconsistent estimators, but also the optimal action altered.

Finally, we note that in this setup, Assumption 2.3 is satisfied. This demonstrates that even if the errors ${\left({\mathit{ϵ}}_t\right)}_{t\ge 1}$ do not affect the optimal action given ${\left({\mathit{\theta }}_a^{*}\right)}_{a\in \left\{\mathrm{0,1}\right\}}$, the poor performance of the RLS estimator may still lead to linear regret in classical bandit algorithms.

B. 2. Inconsistency of the naive measurement error adjusted estimator (2.4)

Example B.2. Let $d=1,{\mathbf{x}}_{\tau }\equiv 1$ for all $\tau$, and ${\mathit{ϵ}}_{\tau }~\mathrm{U}\mathrm{n}\mathrm{i}\mathrm{f}(-\mathrm{2,2})$ sampled independently. For the reward model, let ${\mathit{\theta }}_0^{*}=-1,{\mathit{\theta }}_1^{*}=1,{\eta }_{\tau }~\mathrm{U}\mathrm{n}\mathrm{i}\mathrm{f}(-\mathrm{0.1,0.1})$ sampled independently. So in order to maximize expected reward, we should choose action 1 if ${\mathbf{x}}_{\tau }$ is positive and action 0 otherwise.

Suppose the agent takes the following policy that is stationary and non-adaptive to history:

$${\pi }_{\tau }\left(A\right)=\left\{\begin{array}{ll}\frac{2}{3}{1}_{\{A=1\}}+\frac{1}{3}{1}_{\{A=0\}},& \text{ if }{\tilde{\mathbf{x}}}_{\tau }>\rho \\ \frac{1}{3}{1}_{\{A=1\}}+\frac{2}{3}{1}_{\{A=0\}},& \text{ otherwise. }\end{array}\right.$$

Here, $\rho$ is a pre-specified constant. Figure 4 (a) plots the mean and standard deviation of ${\widehat{\mathit{\theta }}}_{0,\text{me}}^{\left(t\right)}$ (as in (2.4)) given 100 independent experiments for each $t=1,\dots ,10000$, where $\rho =-\mathrm{0.5,0},0.5$. Observe that as $t$ grows, ${\widehat{\mathit{\theta }}}_{0,me}^{\left(t\right)}$ converges to different limits for different policies. In general, the limit is not equal to ${\mathit{\theta }}_0^{*}=-1$.

Figure 3:

Estimation error of the RLS estimator and cumulative regret of UCB (Chu et al., 2011) and Thompson sampling (Russo et al., 2018) under contextual error in Example B.1. The red and pink line corresponds to Thompson sampling and UCB respectively. The solid lines indicate the mean values, while the shaded bands represent the standard error across the independent experiments.

In contrast, Figure 4 (b) shows the mean and standard deviation of ${\widehat{\mathit{\theta }}}_0^{\left(t\right)}$ (as in (2.5)) given 100 independent experiments under the same setting with the same three policies as in Figure 4 (a). Unlike the naive estimator (2.4), the proposed estimator (2.5) quickly converges around the true value −1 for all three candidate policies.

C. Generalization to $\mathit{K}\ge 2$ actions

In this section, we assume that $𝒜=\{\mathrm{1,2},\dots ,K\}$ instead of $\left\{\mathrm{0,1}\right\}$. The standard and clipped benchmark become

$${\pi }_t^{*}\left(a\right)=\left\{\begin{array}{ll}1,& \text{if }a=a_t^{*}={\mathrm{a}\mathrm{r}\mathrm{g}\text{max}}_a⟨{\mathit{\theta }}_a^{*},{\mathbf{x}}_t⟩\\ 0,& \text{otherwise},\end{array}\right.$$ (C.1)

and

$${\bar{\pi }}_t^{*}\left(a\right)=\left\{\begin{array}{ll}1-(K-1)p_0,& \text{if }a=a_t^{*}\\ p_0,& \text{otherwise.}\end{array}\right.$$ (C.2)

Figure 4:

Estimated value of ${\mathit{\theta }}_0^{*}$ given the naive estimator (2.4) in (a) and our proposed estimator (2.5) in (b) under different policies under 100 independent experiments. The green, blue, and red line corresponds to the policy with parameter $\rho =-0.5,0$, and 0.5 respectively. The solid lines indicate the mean values, while the shaded bands represent the standard deviation across the independent experiments.

In the $K$-arm setting, we can still estimate ${\mathit{\theta }}_a^{*}$ using (2.5) for each $a\in 𝒜$. Using the same proof ideas as Theorem 2.1, we get the following guarantees for estimation error (proof omitted):

Theorem C.1. For any $t\in \left[T\right]$, let $q_t={\mathrm{i}\mathrm{n}\mathrm{f}}_{\tau \in \left[t\right],a\in 𝒜}{\pi }_{\tau }\left(a\mid {\tilde{\mathbf{x}}}_{\tau },{\mathscr{H}}_{\tau -1}\right)$. Then under Assumption 2.1 and 2.2, there exist constants $C$ and $C_1$ such that as long as $q_t\ge C_1\text{max}\left\{\frac{d(d+\mathrm{l}\mathrm{o}\mathrm{g}t)}{{\lambda }_{0}t},\frac{\xi (d+\mathrm{l}\mathrm{o}\mathrm{g}t)}{{\lambda }_0^{2}t}\right\}$, with probability at least $1-\frac{4K}{t^2}$,

$${∥{\widehat{\mathit{\theta }}}_a^{\left(t\right)}-{\mathit{\theta }}_a^{*}∥}_2\le \frac{C\left(R+R_{\theta }\right)d}{{\lambda }_0}\text{max}\left\{\frac{d+\mathrm{l}\mathrm{o}\mathrm{g}t}{q_{t}t},\frac{\sqrt{\nu }+\sqrt{\xi }}{\sqrt{d}}\sqrt{\frac{d+\mathrm{l}\mathrm{o}\mathrm{g}t}{q_{t}t}}\right\},\hspace{1em}\forall a\in 𝒜.$$

MEB with $K$ actions is shown in Algorithm 4. As in Theorem 2.2, we can control the regret of Algorithm 4 by the estimation error of (2.5). Here, Assumption 2.3 needs to be generalized to the following to adapt to multiple actions:

Assumption C.1. There exists a constant $\rho \in \left(\mathrm{0,1}\right)$ such that $\forall t\in \left[T\right],\forall a_1,a_2\in 𝒜,\mid ⟨{\mathit{\theta }}_{a_1}^{*}-{\mathit{\theta }}_{a_2}^{*},{\mathit{ϵ}}_t⟩|\le \rho |⟨{\mathit{\theta }}_{a_1}^{*}-{\mathit{\theta }}_{a_2}^{*},{\mathbf{x}}_t⟩\mid$ almost surely.

The theorem below is a generalization of Theorem 2.2 to multiple actions (The proof is only slightly different from Theorem 2.2; We briefly discuss the difference in Appendix G.1).

Theorem C.2. Let Assumption 2.1 and C. 1 hold.

1. For the standard setting, Algorithm 4 outputs a policy with $\mathrm{R}\mathrm{e}\mathrm{g}\mathrm{r}\mathrm{e}\mathrm{t}\left(T;{\pi }^{*}\right)$ no more than

   $$2T_0{R}_{\theta }+\frac{2}{1-\rho }\cdot \sum _{t=T_0+1}^T\left[\left(K-1\right)p_0^{\left(t\right)}{R}_{\theta }+\underset{a\in 𝒜}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{\left(t-1\right)}-{\mathit{\theta }}_a^{*}∥}_2\right].$$
2. For the clipped policy setting, Algorithm 4 with the choice of $p_0^{\left(t\right)}\equiv p_0$ outputs a policy with $\text{Regret}\left(T;{\bar{\pi }}^{*}\right)$ no more than

   $$2T_0{R}_{\theta }+\frac{2\left(1-K{p}_0\right)}{1-\rho }\cdot \sum _{t=T_0+1}^T\underset{a\in 𝒜}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{\left(t-1\right)}-{\mathit{\theta }}_a^{*}∥}_2.$$

Combining Theorems C. 1 and C.2, we obtain the following corollary.

Corollary C.1. Let Assumption 2.1, 2.2 and C. 1 hold. There exist universal constants $C,C\prime$ such that:

1. For the standard setting, $\forall T\ge C\prime \text{max}\left\{\left(1+1/{\lambda }_0^{\frac{9}{4}}\right)(d+\mathrm{l}\mathrm{o}\mathrm{g}T{)}^3,{\left(\xi /{\lambda }_0\right)}^{\frac{9}{4}}(d+\mathrm{l}\mathrm{o}\mathrm{g}T{)}^{\frac{4}{3}}\right\}$, with probability at least $1-\frac{8K}{\sqrt{T}}$, Algorithm 4 with the choice of $T_0=⌈2d{T}^{\frac{2}{3}}⌉,p_0^{\left(t\right)}=\mathrm{m}\mathrm{i}\mathrm{n}\left\{\frac{1}{2},t^{-\frac{1}{3}}\right\}$ outputs a policy with $\mathrm{R}\mathrm{e}\mathrm{g}\mathrm{r}\mathrm{e}\mathrm{t}\left(T;{\pi }^{*}\right)$ no more than

   $$Cd{T}^{\frac{2}{3}}\left\{\frac{(K-1)R_{\theta }}{1-\rho }+\frac{(\sqrt{\nu }+\sqrt{\xi }+1)\left(R+R_{\theta }\right)}{(1-\rho ){\lambda }_0}\sqrt{1+\frac{\mathrm{l}\mathrm{o}\mathrm{g}T}{d}}\right\}.$$
2. For the clipped policy setting, $\forall T\ge C\prime \text{max}\left\{(d+\mathrm{l}\mathrm{o}\mathrm{g}T{)}^2/{\left({\lambda }_0{p}_0\right)}^2,{\xi }^2/{\lambda }_0^4(1+\mathrm{l}\mathrm{o}\mathrm{g}T/d{)}^2\right\}$, with probability at least $1-\frac{8K}{\sqrt{T}}$, Algorithm 4 with the choice of $T_0=\lceil 2d\sqrt{T}\rceil$ and $p_0^{\left(t\right)}\equiv p_0$ outputs a policy with $\mathrm{R}\mathrm{e}\mathrm{g}\mathrm{r}\mathrm{e}\mathrm{t}\left(T;{\bar{\pi }}^{*}\right)$ no more than

   $$Cd{T}^{\frac{1}{2}}\left\{R_{\theta }+\frac{(\sqrt{\nu }+\sqrt{\xi }+1)\left(1-K{p}_0\right)\left(R+R_{\theta }\right)}{\sqrt{p_0}(1-\rho ){\lambda }_0}\sqrt{1+\frac{\mathrm{l}\mathrm{o}\mathrm{g}T}{d}}\right\}.$$

D. Analysis of the proposed estimator (2.5)

D. 1. Proof of Theorem 2.1

We fix some $t\in \left[T\right]$, and control ${∥{\widehat{\mathit{\theta }}}_a^{\left(t\right)}-{\mathit{\theta }}_a^{*}∥}_2$ for $a\in \left\{\mathrm{0,1}\right\}$. Towards this goal, we combine analysis of the two random terms ${\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},a}^{\left(t\right)}$ and ${\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},r,a}^{\left(t\right)}$ in the lemma below.

Lemma D.1. Under the same assumptions of Theorem 2.1, there exists an absolute constant $C$ such that with probability at least $1-4/t^2$, both of the followings hold:

$${∥{\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},a}^{\left(t\right)}-\frac{1}{t}\sum _{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}(a)\left({\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }+{\mathbf{\Sigma }}_{e,\tau }\right)∥}_2\le C\text{max}\left\{\frac{d+\text{log}t}{q_{t}t},\frac{\sqrt{\xi }}{d}\cdot \sqrt{\frac{d+\text{log}t}{q_{t}t}}\right\},$$ (D.1)

$${∥{\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},r,a}^{\left(t\right)}-\left(\frac{1}{t}\sum _{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}(a){\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }\right){\mathit{\theta }}_a^{*}∥}_2\le CR\text{max}\left\{\frac{d+\text{log}t}{q_{t}t},\sqrt{\frac{\nu }{d}}\cdot \sqrt{\frac{d+\text{log}t}{q_{t}t}}\right\}.$$ (D.2)

Proof of Lemma D.1 is in Section G.2.

Denote ${\mathbf{\Delta }}_1={\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},a}^{\left(t\right)}-\frac{1}{t}\sum _{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}\left(a\right)\left({\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }+{\mathbf{\Sigma }}_{e,\tau }\right),{\mathbf{\Delta }}_2={\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},r,a}^{\left(t\right)}-\left(\frac{1}{t}\sum _{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}\left(a\right){\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }\right){\mathit{\theta }}_a^{*}$. Then

$$\begin{gathered} {\widehat{\mathit{\theta }}}_a^{\left(t\right)} ={\left({\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},a}^{\left(t\right)}-\frac{1}{t}\sum _{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}(a){\mathbf{\Sigma }}_{e,\tau }\right)}^{-1}\cdot {\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},r,a}^{\left(t\right)} \\ ={\left(\frac{1}{t}\sum _{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}\left(a\right){\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }+{\mathbf{\Delta }}_1\right)}^{-1}\cdot \left[\left(\frac{1}{t}\sum _{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}\left(a\right){\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }\right){\mathit{\theta }}_a^{*}+{\mathbf{\Delta }}_2\right] \\ ={\mathit{\theta }}_a^{*}-{\mathit{J}}_1+{\mathit{J}}_2, \end{gathered}$$

where

$${\mathit{J}}_1:={\left[\frac{1}{t}\sum _{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}(a){\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }+{\mathbf{\Delta }}_1\right]}^{-1}\cdot {\mathbf{\Delta }}_1{\mathit{\theta }}_a^{*},{\mathit{J}}_2:={\left[\frac{1}{t}\sum _{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}\left(a\right){\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }+{\mathbf{\Delta }}_1\right]}^{-1}\cdot {\mathbf{\Delta }}_2.$$

It’s easy to verify that under the event where both (D.1) and (D.2) hold, whenever

$$C\text{max}\left\{\frac{d+\text{log}t}{q_{t}t},\frac{\sqrt{\xi }}{d}\cdot \sqrt{\frac{d+\text{log}t}{q_{t}t}}\right\}\le \frac{{\lambda }_0}{2d},$$ (D.3)

we have

$${∥{\mathit{J}}_1∥}_2\le \frac{2C{R}_{\mathit{\theta }}d}{{\lambda }_0}\text{max}\left\{\frac{d+\text{log}t}{q_{t}t},\frac{\sqrt{\xi }}{d}\cdot \sqrt{\frac{d+\text{log}t}{q_{t}t}}\right\}$$

and

$${∥{\mathit{J}}_2∥}_2\le \frac{2CRd}{{\lambda }_0}\text{max}\left\{\frac{d+\text{log}t}{q_{t}t},\sqrt{\frac{\nu }{d}}\cdot \sqrt{\frac{d+\text{log}t}{q_{t}t}}\right\}.$$

(D.3) can be ensured by $t\ge C_1\text{max}\left\{\frac{d(d+\text{log}t)}{{\lambda }_0{q}_t},\frac{\xi (d+\text{log}t)}{{\lambda }_0^2{q}_t}\right\}$, where $C_1=\text{max}\left\{2C,4C^2\right\}$. Given these guarantees, we have with probability at least $1-4/t^2$,

$${∥{\widehat{\mathit{\theta }}}_a^{\left(t\right)}-{\mathit{\theta }}_a^{*}∥}_2\le {∥{\mathit{J}}_1∥}_2+{∥{\mathit{J}}_2∥}_2\le \frac{2C\left(R+R_{\theta }\right)d}{{\lambda }_0}\text{max}\left\{\frac{d+\text{log}t}{q_{t}t},\frac{\sqrt{\nu }+\sqrt{\xi }}{\sqrt{d}}\cdot \sqrt{\frac{d+\text{log}t}{q_{t}t}}\right\}.$$

Thus we conclude the proof.

D. 2. Additional comments on Assumption 2.2

Following Remark 2.1, all the following examples of $\left\{{\mathbf{x}}_{\tau }\right\}$ allow the existence of ${\lambda }_0$ with ${\pi }_{\tau }^{nd}\left(a\right)\equiv 1/2$ given any reasonably big $t$ with high probability:

• $\left\{\sqrt{d}{\mathbf{x}}_{\tau }\right\}$ is an i.i.d. sequence satisfying $d\mathbb{E}{\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }⪰{\lambda }_1{I}_d,{\lambda }_1>0$;

• $\left\{\sqrt{d}{\mathbf{x}}_{\tau }\right\}$ is a weakly-dependent stationary time series (a common example is the multivariate ARMA process under regularity conditions, see e.g. (Banna et al., 2016)). The stationary distribution $P$ satisfies $d{\mathbb{E}}_{\mathbf{x}~P}\mathbf{x}{\mathbf{x}}^{\top }⪰{\lambda }_2{I}_d,{\lambda }_2>0$;

• $\left\{\sqrt{d}{\mathbf{x}}_{\tau }\right\}$ is a periodic time series such that there exists $t_0\in {\mathbb{N}}^{+}$which satisfies $\frac{d}{t_0}{\sum }_{\tau \in \left(k{t}_0+1,(k+1)t_0\right]}{\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }⪰{\lambda }_3{I}_d$ a.s. $\forall k\in \mathbb{N}$.

D. 3. Generalization to off-policy method-of-moment estimation

(2.5) can be generalized to a class of method-of-moment estimators for off-policy learning. In this section, we delve into the general framework of off-policy method-of-moment estimation. This framework proves valuable in scenarios where a fully parametric model class for the reward is unavailable, yet there is a desire to estimate certain model parameters using offline batched bandit data.

For simplicity, we assume that ${\left(X_t,Y_t\left(a\right):a\in 𝒜\right)}_{t\in \left[T\right]}$ are drawn i.i.d. from an unknown distribution $𝒫$. At each time $t\in \left[T\right]$, the action $A_t$ is drawn from a policy ${\pi }_t\left(\cdot \mid X_t,{\mathscr{H}}_{t-1}\right)$, and the agent observes only $o_t=\left(X_t,A_t,Y_t\left(A_t\right)\right)$ together with the action selection probabilities ${\pi }_t$. Define the history up to time $t$ as ${\mathscr{H}}_t={\left\{o_{\tau }\right\}}_{\tau \le t}$. For $a_0\in 𝒜$, we’re interested in estimating ${\mathit{\theta }}_{a_0}^{*}$, a $d$-dimensional parameter in ${𝒫}^{\left(a_0\right)}$, which is the joint distribution of $\left\{X_t,Y_t\left(a_0\right)\right\}$.

Remark D.1. When the context is i.i.d., the problem of estimating ${\left\{{\mathit{\theta }}_a^{*}\right\}}_{a\in 𝒜}$ in Section 2.2 is a special case of this setup by taking ${\tilde{\mathbf{x}}}_t$ as $X_t$ and $r_t$ as $Y_t$.

The traditional method-of-moment estimator looks for functions $f_1,\dots ,f_d$ as well as a mapping $\varphi :{\mathbb{R}}^d\to {\mathbb{R}}^d$, such that

$${\mathit{\theta }}_{a_0}^{*}=\mathit{\varphi }\left({\mathbb{E}}_{(X,Y)~{𝒫}^{\left(a_0\right)}}{f}_1(X,Y),\dots ,{\mathbb{E}}_{(X,Y)~{𝒫}^{\left(a_0\right)}}{f}_d(X,Y)\right).$$

Then, if given i.i.d. samples ${\left(U_t,V_t\right)}_{t\in \left[n\right]}$ from ${𝒫}^{\left(a_0\right)}$, the estimator takes the form

$${\widehat{\mathit{\theta }}}_{a_0}=\varphi \left(\frac{1}{T}\sum _{t\in \left[T\right]}{f}_1\left(U_t,V_t\right),\dots ,\frac{1}{T}\sum _{t\in \left[T\right]}{f}_d\left(U_t,V_t\right)\right).$$

In fact, the naive estimator (2.4) is of this form. It is clear that we cannot use this estimator for offline batched data ${\mathscr{H}}_T$: There are no i.i.d. samples from ${𝒫}^{\left(a_0\right)}$ because of the policy ${\left\{{\pi }_t\right\}}_{t\in \left[T\right]}$. Instead, we propose the following estimator:

$${\widehat{\mathit{\theta }}}_{a_0}=\varphi \left(\frac{1}{T}\sum _{t\in \left[T\right]}{W}_t{f}_1\left(X_t,Y_t\right),\dots ,\frac{1}{T}\sum _{t\in \left[T\right]}{W}_t{f}_d\left(X_t,Y_t\right)\right),$$

where $W_t=1_{\left\{A_t=a_0\right\}}\frac{{\pi }^{nd}\left(A_t\right)}{{\pi }_t\left(A_t\mid X_t,{\mathscr{H}}_{t-1}\right)}$ for a data-independent probability distribution ${\pi }^{nd}$ on $𝒜$. Similar to the proof of Theorem 2.1, it’s not difficult to see that ${\widehat{\mathit{\theta }}}_{a_0}$ is consistent under mild conditions. In fact, (2.5) is a special case of this estimator when ${\pi }_{\tau }^{nd}$ does not depend on $\tau$. A more detailed analysis is left for future work.

E Analysis of MEB

E. 1. Additional comments on Example 2.1

In example 2.1, the joint distribution of $\left({\mathbf{x}}_t,{\tilde{\mathbf{x}}}_t\right)$ is:

$$\begin{gathered} \mathbb{P}\left(\left({\mathbf{x}}_t,{\tilde{\mathbf{x}}}_t\right)=\left(\mathrm{0.2,1}\right)\right)=0.3,\mathbb{P}\left(\left({\mathbf{x}}_t,{\tilde{\mathbf{x}}}_t\right)=(0.2,-1)\right)=0.2, \\ \mathbb{P}\left(\left({\mathbf{x}}_t,{\tilde{\mathbf{x}}}_t\right)=(-\mathrm{0.2,1})\right)=0.2,\mathbb{P}\left(\left({\mathbf{x}}_t,{\tilde{\mathbf{x}}}_t\right)=(-0.2,-1)\right)=0.3. \end{gathered}$$

The optimal action given ${\left({\mathit{\theta }}_a^{*}\right)}_{a\in \left\{\mathrm{0,1}\right\}}$ and ${\mathbf{x}}_t$ is

$$a_t^{*}\left({\mathbf{x}}_t\right)=\underset{a}{\mathrm{a}\mathrm{r}\mathrm{g}\text{max}}⟨{\mathit{\theta }}_a^{*},{\mathbf{x}}_t⟩=\left\{\begin{array}{ll}1,& \text{ if }{\mathbf{x}}_t=0.2,\\ 0,& \text{ if }{\mathbf{x}}_t=-0.2.\end{array}\right.$$

In the standard bandit setting, the benchmark policy is

$${\pi }_t^{*}\left(a\right)=\left\{\begin{array}{ll}1,& \text{if }a=a_t^{*}\left({\mathbf{x}}_t\right),\\ 0,& \text{otherwise.}\end{array}\right.$$

For any policy $\pi ={\left({\pi }_t\right)}_{t\ge 1}$, the instantaneous regret at time $t$ is

$${\mathrm{R}\mathrm{e}\mathrm{g}\mathrm{r}\mathrm{e}\mathrm{t}}_t\left({\pi }_t,{\pi }_t^{*}\right)=0.4{\mathbb{E}}_{a_t~{\pi }_t\left(\cdot \mid {\tilde{\mathbf{x}}}_t,{\mathscr{H}}_{t-1}\right)}{1}_{\left\{a_t\ne a_t^{*}\left({\mathbf{x}}_t\right)\right\}}.$$

Even if both ${\left({\mathit{\theta }}_a^{*}\right)}_{a\in \left\{\mathrm{0,1}\right\}}$ and the joint distribution of $\left({\mathbf{x}}_t,{\tilde{\mathbf{x}}}_t\right)$ are known, ${\pi }_t$ can only depend on ${\tilde{\mathbf{x}}}_t$ and history, and cannot be based on ${\mathbf{x}}_t$. Thus, there is always a (constant) positive probability that the action $a_t$ sampled from ${\pi }_t$ does not match $a_t^{*}\left({\mathbf{x}}_t\right)$ (otherwise, $a_t$ sampled from ${\pi }_t\left(\cdot \mid {\tilde{\mathbf{x}}}_t,{\mathscr{H}}_{t-1}\right)$ should be equal to $a_t^{*}\left({\mathbf{x}}_t\right)$ a.s.). Thus, the standard cumulative regret will be linear in the time horizon.

E. 2. Proof of Theorem 2.2

We first prove the lemma below. Its proof is in Appendix G.3.

Lemma E.1. Under Assumption 2.3, we have $a_t^{*}=a_t^{†}:={\mathrm{a}\mathrm{r}\mathrm{g}\text{max}}_a⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩$. Consequently, ${\pi }_t^{*}={\pi }_t^{†},{\bar{\pi }}_t^{*}={\bar{\pi }}_t^{†}$ (given a fixed minimum action selection probability $p_0$), where

$${\pi }_t^{†}\left(a\right)=\left\{\begin{array}{ll}1,& \text{if }a=a_t^{†},\\ 0,& \text{otherwise}\end{array}{\bar{\pi }}_t^{†}(a)=\left\{\begin{array}{ll}1-p_0,& \text{if }a=a_t^{†},\\ p_0,& \text{otherwise.}\end{array}\right.\right.$$

In the below, we define

$$\begin{gathered} {\widehat{\text{Regret}}}_t\left(\pi ,{\pi }^{*}\right):={\mathbb{E}}_{a~{\pi }_t^{*}}⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩-{\mathbb{E}}_{a~{\pi }_t}⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩, \\ {\widehat{\text{Regret}}}_t\left(\pi ,{\bar{\pi }}^{*}\right):={\mathbb{E}}_{a~{\bar{\pi }}_t^{*}}⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩-{\mathbb{E}}_{a~{\pi }_t}⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩. \end{gathered}$$

Standard setting. In the standard setting, we give the lemma below (proof in Appendix G.4).

Lemma E.2. Under the assumptions of Theorem 2.2, at any time $t>T_0$,

$${\widehat{\text{Regret}}}_t\left(\pi ,{\pi }^{*}\right)\le 2p_0^{\left(t\right)}{R}_{\theta }+2\underset{a\in \left\{\mathrm{0,1}\right\}}{\text{max} }{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2\text{. }$$

Note that for any time $t>T_0$, the instantaneous regret at time $t:{\text{Regret}}_t\left(\pi ,{\pi }^{*}\right)=\left({\pi }_t^{*}\left(1\right)-\right.\left.{\pi }_t\left(1\right)\right)⟨{\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\mathbf{x}}_t⟩$, and that ${\widehat{\text{Regret}}}_t\left(\pi ,{\pi }^{*}\right)=\left({\pi }_t^{*}\left(1\right)-{\pi }_t\left(1\right)\right)⟨{\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\tilde{\mathbf{x}}}_t⟩$. Moreover,

$$\begin{gathered} \left|\langle {\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\tilde{\text{x}}}_t\rangle \right|=\left|\langle {\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\mathbf{\text{x}}}_t\rangle +\langle {\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\mathit{ϵ}}_t\rangle \right| \\ \ge \left|\langle {\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\mathbf{\text{x}}}_t\rangle \right|-\left|\langle {\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\mathit{ϵ}}_t\rangle \right| \\ \ge \frac{1}{\rho }\left|\langle {\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\mathit{ϵ}}_t\rangle \right|-\left|\langle {\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\mathit{ϵ}}_t\rangle \right| \\ =\frac{1-\rho }{\rho }\left|\langle {\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\mathit{ϵ}}_t\rangle \right|. \end{gathered}$$

Here we used Assumption 2.3. Thus we have

$$\begin{gathered} {\text{Regret}}_t\left(\pi ,{\pi }^{*}\right) ={\widehat{\text{Regret}}}_t\left(\pi ,{\pi }^{*}\right)-\left({\pi }_t^{*}\left(1\right)-{\pi }_t\left(1\right)\right)⟨{\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\mathit{ϵ}}_t⟩ \le {\widehat{\text{Regret}}}_t\left(\pi ,{\pi }^{*}\right)+\left|{\pi }_t^{*}\left(1\right)-{\pi }_t\left(1\right)\right|\cdot \frac{\rho }{1-\rho }\left|⟨{\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\tilde{\mathbf{x}}}_t⟩\right| \\ =\frac{1}{1-\rho }{\widehat{\text{Regret}}}_t\left(\pi ,{\pi }^{*}\right). \end{gathered}$$

Combining Lemma E.2, we obtain that for any $t>T_0$,

$${\text{Regret}}_t\left(\pi ,{\pi }^{*}\right)\le \frac{2}{1-\rho }\left(p_0^{\left(t\right)}{R}_{\theta }+\underset{a\in \left\{\mathrm{0,1}\right\}}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{\left(t-1\right)}-{\mathit{\theta }}_a^{*}∥}_2\right).$$

Finally, when $t\le T_0$, since $∥{\mathbf{x}}_t∥\le 1,∥{\mathit{\theta }}_a^{*}∥\le R_{\theta }$, the instantaneous regret ${\text{Regret}}_t\left(\pi ,{\pi }^{*}\right)\le 2R_{\theta }$. We conclude the proof by summing up all the instantaneous regret terms.

Clipped policy setting.In the clipped policy setting, we give the lemma below (proof in Appendix G.5).

Lemma E.3. Under the assumptions of Theorem 2.2, at any time $t>T_0$,

$${\widehat{\text{Regret}}}_t\left(\pi ,{\bar{\pi }}^{*}\right)\le 2\left(1-2p_0\right)\underset{a\in \left\{\mathrm{0,1}\right\}}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2\text{. }$$

Note that the instantaneous regret at time $t:{\text{Regret}}_t\left(\pi ,{\bar{\pi }}^{*}\right)=\left({\bar{\pi }}_t^{*}\left(1\right)-{\pi }_t\left(1\right)\right)⟨{\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\mathbf{x}}_t⟩$, and that ${\widehat{\text{Regret}}}_t\left(\pi ,{\bar{\pi }}^{*}\right)=\left({\bar{\pi }}_t^{*}\left(1\right)-{\pi }_t\left(1\right)\right)⟨{\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\tilde{\mathbf{x}}}_t⟩$. Similar to the standard setting, for $t>T_0$, under Assumption 2.3, we have

$${\text{Regret}}_t\left(\pi ,{\bar{\pi }}^{*}\right)\le \frac{1}{1-\rho }{\widehat{\text{Regret}}}_t\left(\pi ,{\bar{\pi }}^{*}\right)\le \frac{2\left(1-2p_0\right)}{1-\rho }\underset{a\in \left\{\mathrm{0,1}\right\}}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{\left(t-1\right)}-{\mathit{\theta }}_a^{*}∥}_2.$$

We conclude the proof by summing up all the instantaneous regret terms, and noticing that for $t\le T_0,{\text{Regret}}_t\left(\pi ,{\bar{\pi }}^{*}\right)\le 2R_{\theta }$.

Results with a high-probability version of Assumption 2.3.As briefly mentioned in the main paper, Assumption 2.3 can be weakened to the inequalities holding with high probability. Now instead of Assumption 2.3, we assume the following:

Assumption E.1. There exist constants $\rho \in (0,1),c_e\in \left[\mathrm{0,1}\right]$ such that ${\sum }_{t=1}^T\mathbb{P}\left(A_t^c\right)\le c_e$. Here $A_t$ denotes the event $\left\{\left|⟨{\mathit{\delta }}_{\theta },{\mathit{ϵ}}_t⟩\right|\le \rho \left|⟨{\mathit{\delta }}_{\theta },{\mathbf{x}}_t⟩\right|\right\}$, and ${\mathit{\delta }}_{\theta }={\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*}$.

It’s easy to see that the result of Lemma E. 1 hold at time $t$ under the event $A_t$. Further, following the same arguments, we obtain that under Assumption 2.1, the results for either the standard setting or the clipped policy setting hold under the event ${\cap }_{t=T_0+1}^T{A}_t$. Therefore, with Assumption 2.1 and E.1, in either setting, the results in Theorem 2.2 hold with probability at least $1-c_e$.

E. 3. Proof of Corollary 2.1

Standard setting. First, notice that $q_t={\mathrm{m}\mathrm{i}\mathrm{n}}_{\tau \le t,a\in \left\{\mathrm{0,1}\right\}}{\pi }_{\tau }\left(a\mid {\tilde{\mathbf{x}}}_t,{\mathscr{H}}_{\tau -1}\right)=p_0^{\left(t\right)}$, since $p_0^{\left(t\right)}$ is monotonically decreasing in $t$. Theorem 2.1 indicates that, as long as $\forall t>T_0$,

$$p_0^{\left(t\right)}\ge C_1\text{max}\left\{\frac{d\left(d+\text{log}t\right)}{{\lambda }_{0}t},\frac{\xi \left(d+\text{log}t\right)}{{\lambda }_0^{2}t}\right\},$$ (E.1)

then with probability at least $1-\frac{8}{t^2},\forall a\in \left\{\mathrm{0,1}\right\}$,

$${∥{\widehat{\mathit{\theta }}}_a^{\left(t\right)}-{\mathit{\theta }}_a^{*}∥}_2\le \frac{C\left(R+R_{\theta }\right)d}{{\lambda }_0}\text{max}\left\{\frac{d+\text{log}t}{q_{t}t},\frac{\sqrt{\nu }+\sqrt{\xi }}{\sqrt{d}}\sqrt{\frac{d+\text{log}t}{q_{t}t}}\right\}$$ (E.2)

Plug it into Theorem 2.2, we have that with high probability,

$$\text{Regret}\left(T;{\pi }^{*}\right)\le 2R_{\theta }\cdot ⌈2d{T}^{2/3}⌉+\frac{2}{1-\rho }{I}_1,$$

where

$$\begin{gathered} I_1 =\sum _{t=T_0+1}^T\left(p_0^{\left(t\right)}{R}_{\theta }+\underset{a}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2\right) \\ \le \sum _{t=T_0+1}^T{t}^{-\frac{1}{3}}{R}_{\theta }+\sum _{t=T_0}^{T-1}\frac{C\left(R+R_{\theta }\right)d}{{\lambda }_0}\text{max}\left\{\frac{d+\text{log}t}{t^{\frac{2}{3}}},\frac{\sqrt{\nu }+\sqrt{\xi }}{\sqrt{d}}\sqrt{\frac{d+\text{log}t}{t^{\frac{2}{3}}}}\right\} \\ \le 2R_{\theta }{T}^{\frac{2}{3}}+\frac{C\left(R+R_{\theta }\right)d}{{\lambda }_0}\cdot \sum _{t=T_0}^{T-1}\frac{d+\text{log}t}{t^{\frac{2}{3}}}+\frac{C\left(R+R_{\theta }\right)d}{{\lambda }_0}\cdot \sum _{t=T_0}^{T-1}\frac{\sqrt{\nu }+\sqrt{\xi }}{\sqrt{d}}\sqrt{\frac{d+\text{log}t}{t^{\frac{2}{3}}}} \\ \le 2R_{\theta }{T}^{\frac{2}{3}}+\frac{3C\left(R+R_{\theta }\right)}{{\lambda }_0}d(d+\text{log}T)T^{\frac{1}{3}}+\frac{3C(\sqrt{\nu }+\sqrt{\xi })\left(R+R_{\theta }\right)}{2{\lambda }_0}\sqrt{d(d+\text{log}T)}{T}^{\frac{2}{3}} \\ \le 2R_{\theta }{T}^{\frac{2}{3}}+\frac{C\prime }{{\lambda }_0}(\sqrt{\nu }+\sqrt{\xi }+1)\left(R+R_{\theta }\right)\sqrt{d(d+\text{log}T)}{T}^{\frac{2}{3}}, \end{gathered}$$

for a universal constant $C\prime$, where the last inequality holds if in addition, $T\ge \left[d\right(d+\text{log}T){]}^{\frac{3}{2}}$.

The proof is concluded by combining the above requirement for $T$ as well as (E.1).

Clipped policy setting.

Similar to the standard setting, according to Theorem 2.1, as long as $\forall t>T_0$,

$$p_0\ge C_1\text{max}\left\{\frac{d(d+\text{log}t)}{{\lambda }_{0}t},\frac{\xi (d+\text{log}t)}{{\lambda }_0^{2}t}\right\},$$ (E.3)

then with probability at least $1-\frac{8}{t^2},\forall a\in \left\{\mathrm{0,1}\right\}$,

$${∥{\widehat{\mathit{\theta }}}_a^{\left(t\right)}-{\mathit{\theta }}_a^{*}∥}_2\le \frac{C\left(R+R_{\theta }\right)d}{{\lambda }_0}\text{max}\left\{\frac{d+\text{log}t}{p_{0}t},\frac{\sqrt{\nu }+\sqrt{\xi }}{\sqrt{d}}\sqrt{\frac{d+\text{log}t}{p_{0}t}}\right\}$$ (E.4)

Plug it into Theorem 2.2, we have that with high probability,

$$\text{Regret}\left(T;{\bar{\pi }}^{*}\right)\le 2R_{\theta }\cdot ⌈2d{T}^{1/2}⌉+\frac{2\left(1-2p_0\right)}{1-\rho }{I}_2\text{, }$$

where

$$\begin{gathered} I_2 =\sum _{t=T_0+1}^T\underset{a}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2 \\ \le \sum _{t=T_0}^{T-1}\frac{C\left(R+R_{\theta }\right)d}{{\lambda }_0}\text{max}\left\{\frac{d+\text{log}t}{p_{0}t},\frac{\sqrt{\nu }+\sqrt{\xi }}{\sqrt{d}}\sqrt{\frac{d+\text{log}t}{p_{0}t}}\right\} \\ \le \frac{C\left(R+R_{\theta }\right)d}{{\lambda }_0}\sum _{t=T_0}^{T-1}\frac{d+\text{log}t}{p_{0}t}+\frac{C\left(R+R_{\theta }\right)d}{{\lambda }_0}\sum _{t=T_0}^{T-1}\frac{\sqrt{\nu }+\sqrt{\xi }}{\sqrt{d}}\sqrt{\frac{d+\text{log}t}{p_{0}t}} \\ \le \frac{2C\left(R+R_{\theta }\right)}{{\lambda }_0}d(d+\text{log}T)\text{log}T+\frac{2C\left(R+R_{\theta }\right)}{{\lambda }_0}\frac{\sqrt{\nu }+\sqrt{\xi }}{\sqrt{p_0}}\sqrt{d(d+\text{log}T)}\sqrt{T} \\ \le \frac{2C\left(R+R_{\theta }\right)(\sqrt{\nu }+\sqrt{\xi }+1)}{{\lambda }_0\sqrt{p_0}}\sqrt{d(d+\text{log}T)}\sqrt{T,} \end{gathered}$$

where the last inequality holds if in addition, $T\ge d(d+\text{log}T){\text{log}}^{2}T$.

The proof is concluded by plugging the above into the regret upper bound formula and combining the requirements for $T$.

Results with a high-probability version of Assumption 2.3. Recall that Assumption E. 1 is a weakened version of Assumption 2.3 with a high-probability statement. Given Assumptions 2.1, 2.2, and E. 1 (instead of 2.3), in the standard setting, the results of Theorem 2.1 hold as long as (E.2) or (E.4) for all $t>T_0,a\in \left\{\mathrm{0,1}\right\}$ and under the event ${\cap }_{t=T_0+1}^T{A}_t$. Thus, we deduce that the regret upper bound in (i) hold with probability at least $1-16/\sqrt{T}-c_e$. Similarly, given Assumptions 2.1, 2.2, and E.1, in the clipped benchmark setting, the regret upper bound in (ii) hold with probability at least $1-16/\sqrt{T}-c_e$.

E. 4. MEB with infrequent model update

As mentioned at the end of Section 2.3, in certain scenarios (e.g. when $d$ is large), we can save computational resources by updating the estimates of ${\left({\mathit{\theta }}_a^{*}\right)}_{a\in \left\{\mathrm{0,1}\right\}}$ less frequently. In Algorithm 5, we propose a variant of Algorithm 1. At each time $t$, given the noisy context ${\tilde{\mathbf{x}}}_t$, the algorithm computes the best action ${\tilde{a}}_t$ according to the most recently updated estimators of ${\left({\mathit{\theta }}_a^{*}\right)}_{a\in \left\{\mathrm{0,1}\right\}}$. Then, it samples ${\tilde{a}}_t$ with probability $1-p_0^{\left(t\right)}$ and keeps an exploration probability of $p_0^{\left(t\right)}$ to sample the other action. In the meantime, the agent only has to update the estimate of ${\left({\mathit{\theta }}_a^{*}\right)}_{a\in \left\{\mathrm{0,1}\right\}}$ once in a while to save computation power: The algorithm specifies a subset $𝒮\subset \left[T\right]$ and updates the estimators according to (2.5) only when $t\in 𝒮$.

Under mild conditions, Algorithm 5 achieves the same order of regret upper bound as Algorithm 1, as seen from Theorem E. 1 and Corollary E. 1 below. They are modified versions of Theorem 2.2 and Corollary 2.1.

Theorem E.1. Let $s_{\text{min}}:={\mathrm{m}\mathrm{i}\mathrm{n}}_{s\in 𝒮}s$ be the first time Algorithm 5 updates the model. Suppose Assumption 2.1 and 2.3 hold.

1. For the standard setting, for any $T_0\ge s_{\mathrm{m}\mathrm{i}\mathrm{n}}$, Algorithm 5 outputs a policy such that

   $$\text{Regret}\left(T;{\pi }^{*}\right)\le 2T_0{R}_{\theta }+\frac{2}{1-\rho }\sum _{t\in \left(T_0,T\right]}\left(p_0^{\left(t\right)}{R}_{\theta }+\underset{a\in \left\{\mathrm{0,1}\right\}}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{\left(s_t\right)}-{\mathit{\theta }}_a^{*}∥}_2\right).$$
2. For the clipped policy setting, for any $T_0\ge s_{\mathrm{m}\mathrm{i}\mathrm{n}}$, Algorithm 5 with the choice of $p_0^{\left(t\right)}\equiv p_0$ outputs a policy such that

   $$\text{Regret}\left(T;{\bar{\pi }}^{*}\right)\le 2T_0{R}_{\theta }+\frac{2\left(1-2p_0\right)}{1-\rho }{\mathbf{\Sigma }}_{t\in \left(T_0,T\right]}\underset{a\in \left\{\mathrm{0,1}\right\}}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{\left(s_t\right)}-{\mathit{\theta }}_a^{*}∥}_2.$$

Here for any $t\in \left[T\right],s_t:=\text{max}\{s\in 𝒮:s<t\}$.

The proof of Theorem E. 1 is very similar to that of Theorem 2.2, and is thus omitted.

Corollary E.1. Let Assumption 2.1 to 2.3 hold. There exist constants $C,C\prime$ such that:

1. In the standard setting, as long as the set of model update times $𝒮$ satisfies: (a) $s_{\mathrm{m}\mathrm{i}\mathrm{n}}\le d{T}^{2/3}$; (b) $\forall t\in \left(d{T}^{2/3},T\right],s_t=\text{max}\{s\in 𝒮:s<t\}\ge \alpha t$ for some constant $\alpha \in \left(e^{-d},1\right)$, then $\forall T\ge C\prime /\alpha \cdot \text{max}\left\{\left(1+1/{\lambda }_0^{\frac{9}{4}}\right)(d+\text{log}T{)}^3,{\left(\xi /{\lambda }_0\right)}^{\frac{9}{4}}(d+\text{log}T{)}^{\frac{4}{3}}\right\}$, with probability at least $1-\frac{16}{\sqrt{T}}$, Algorithm 5 with the choice of $p_0^{\left(t\right)}=\mathrm{m}\mathrm{i}\mathrm{n}\left\{\frac{1}{2},t^{-1/3}\right\}$ achieves $\text{Regret}\left(T;{\pi }^{*}\right)\le Cd{T}^{\frac{2}{3}}\left\{\frac{R_{\theta }}{\alpha }+\frac{R_{\theta }}{1-\rho }+\frac{(\sqrt{\nu }+\sqrt{\xi }+1)\left(R+R_{\theta }\right)}{{\alpha }^{1/3}{\lambda }_0(1-\rho )}\sqrt{1+\frac{\text{log}T}{d}}\right\}$.

2. In the clipped policy setting, as long as the set of model update times $𝒮$ satisfies: (a) $s_{\text{min}}\le d\sqrt{T}$; (b) $\forall t\in (d\sqrt{T},T],s_t=\text{max}\{s\in 𝒮:s<t\}\ge \alpha t$ for some constant $\alpha \in \left(e^{-d},1\right)$, then for any $T$ s.t. $T\ge C\prime /\alpha \cdot \text{max}\left\{(d+\text{log}T{)}^2/{\left({\lambda }_0{p}_0\right)}^2,{\xi }^2/{\lambda }_0^4(1+\text{log}T/d{)}^2\right\}$, with probability at least $1-\frac{16}{\sqrt{T}}$, Algorithm 5 with the choice of $p_0^{\left(t\right)}\equiv p_0$ achieves: $\text{Regret}\left(T;{\bar{\pi }}^{*}\right)\le Cd\sqrt{T}\left\{\frac{R_{\theta }}{\alpha }+\frac{(\sqrt{\nu }+\sqrt{\xi }+1)\left(1-2p_0\right)\left(R+R_{\theta }\right)}{{\lambda }_0(1-\rho )\sqrt{\alpha p_0}}\sqrt{1+\frac{\text{log}T}{d}}\right\}$.

The proof of Corollary E. 1 can be directly obtained by combining Theorem 2.1 and E. 1 with $T_0=d{T}^{2/3}/\alpha$ in the standard setting and $T_0=d{T}^{1/2}/\alpha$ in the clipped benchmark setting. Thus, the proof is omitted here.

In Corollary E.1, condition (a) and (b) essentially requires Algorithm 5 not to start learning the model too late, and to keep updating the learned model at least at time points with a ‘geometric’ growth rate. This covers a wide range of choices of $𝒮$ in practice. Two typical examples of $𝒮$ could be: (1) $𝒮=\left\{t\in \left[T\right]:t=k{t}_0\right.$ for some $\left.k\in {\mathbb{N}}^{+}\right\}$(the model is learned every $t_0$ time points routinely, where $t_0$ is a constant integer); (2) If $1/\alpha \in {\mathbb{N}}^{+},𝒮=\left\{t\in \left[T\right]:t=(1/\alpha {)}^k\right.$ for some $\left.k\in {\mathbb{N}}^{+}\right\}$(the model only needs to be learned $𝒪\left(\text{log}T\right)$ times to save computation).

F Analysis with estimated error variance

We consider the setting where at each time $t$, the agent does not have access to ${\mathbf{\Sigma }}_{e,t}$, but has a (potentially adaptive) estimator ${\widehat{\mathbf{\Sigma }}}_{e,t}$. In this setting, we estimate the model using (2.7) instead of (2.5) and plug into Algorithm 1. The following theorem controls the estimation error of ${\tilde{\mathit{\theta }}}_a^{\left(t\right)}$. Note that compared to Theorem 2.1, the additional error caused by inaccuracty of ${\widehat{\mathbf{\Sigma }}}_{e,t}$ can be controlled by ${\mathbf{\Delta }}_t\left(a\right):=\frac{1}{t}{\sum }_{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}\left(a\right)\left({\widehat{\mathbf{\Sigma }}}_{e,\tau }-{\mathbf{\Sigma }}_{e,\tau }\right)$, the weighted average of the estimation errors ${\left({\widehat{\mathbf{\Sigma }}}_{e,\tau }-{\mathbf{\Sigma }}_{e,\tau }\right)}_{\tau \in \left[t\right]}$.

Theorem F.1. Recall that $q_t:={\mathrm{i}\mathrm{n}\mathrm{f}}_{\tau \le t,a\in \left\{\mathrm{0,1}\right\}}{\pi }_{\tau }\left(a\mid {\tilde{\mathbf{x}}}_{\tau },{\mathscr{H}}_{\tau -1}\right)$. Then under Assumptions 2.1 and 2.2, there exist constants $C$ and $C_1^{\prime }$ such that as long as $q_t\ge C_1^{\prime }\text{max}\left\{\frac{d(d+\text{log}t)}{{\lambda }_{0}t},\frac{\xi (d+\text{log}t)}{{\lambda }_0^{2}t}\right\}$ and ${\text{max}}_{a\in \left\{\mathrm{0,1}\right\}}{∥{\Delta }_t\left(a\right)∥}_2\le \frac{{\lambda }_0}{4d}$, with probability at least $1-\frac{8}{t^2}$,

$${∥{\tilde{\mathit{\theta }}}_a^{\left(t\right)}-{\mathit{\theta }}_a^{*}∥}_2\le \frac{C\left(R+R_{\mathit{\theta }}\right)d}{{\lambda }_0}\text{max}\left\{\frac{d+\text{log}t}{q_{t}t},\frac{\sqrt{\xi }+\sqrt{\nu }}{\sqrt{d}}\cdot \sqrt{\frac{d+\text{log}t}{q_{t}t}}\right\}+\frac{2R_{\theta }d}{{\lambda }_0}{∥{\mathbf{\Delta }}_t\left(a\right)∥}_2,\hspace{1em}\forall a\in \left\{\mathrm{0,1}\right\}.$$ (F.1)

The proof of Theorem F. 1 is in Appendix G.6.

By combining Theorem F. 1 and 2.2 (with (2.7) instead of (2.5)), we obtain the following regret bounds for Algorithm 1 with (2.7) as the plug-in estimator.

Corollary F.1. Suppose Assumption 2.1 to 2.3 hold. Then there exist universal constants $C,C^{\prime \prime }$ such that:

1. In the standard setting, if ${\text{max}}_{t\in \left[d{T}^{2/3},T\right]}{\text{max}}_{a\in \left\{\mathrm{0,1}\right\}}{∥{\Delta }_t\left(a\right)∥}_2\le \frac{{\lambda }_0}{4d},T\ge C^{\prime \prime }\text{max}\left\{\left(1+1/{\lambda }_0^{\frac{9}{4}}\right)(d+\right.\left.\text{log}T{)}^3,{\left(\xi /{\lambda }_0\right)}^{\frac{9}{4}}(d+\text{log}T{)}^{\frac{4}{3}}\right\}$, by choosing $T_0=⌈2d{T}^{\frac{2}{3}}⌉,p_0^{\left(t\right)}=\mathrm{m}\mathrm{i}\mathrm{n}\left\{\frac{1}{2},t^{-\frac{1}{3}}\right\}$, and (2.7) instead of (2.5) in Algorithm 1, with probability at least $1-\frac{16}{\sqrt{T}}$,

   $$\text{Regret}\left(T;{\pi }^{*}\right)\le Cd{T}^{\frac{2}{3}}\left\{\frac{R_{\theta }}{1-\rho }+\frac{(\sqrt{\nu }+\sqrt{\xi }+1)\left(R+R_{\theta }\right)}{(1-\rho ){\lambda }_0}\sqrt{1+\frac{\text{log}T}{d}}\right\}+\frac{4R_{\theta }d}{(1-\rho ){\lambda }_0}\sum _{t=T_0-1}^T\underset{a\in \left\{\mathrm{0,1}\right\}}{\text{max}}{∥{\Delta }_t\left(a\right)∥}_2.$$
2. In the clipped policy setting, as long as ${\text{max}}_{t\in [d\sqrt{T},T]}{\text{max}}_{a\in \left\{\mathrm{0,1}\right\}}{∥{\Delta }_t\left(a\right)∥}_2\le \frac{{\lambda }_0}{4d},T\ge C^{\prime \prime }\text{max}\left\{\right(d+\left.\text{log}T{)}^2/{\left({\lambda }_0{p}_0\right)}^2,{\xi }^2/{\lambda }_0^4(1+\text{log}T/d{)}^2\right\}$, by choosing $T_0=\lceil 2d\sqrt{T}\rceil ,p_0^{\left(t\right)}\equiv p_0$, and (2.7) instead of (2.5) in Algorithm 1, with probability at least $1-\frac{16}{\sqrt{T}}$,

   $$\text{Regret}\left(T;{\bar{\pi }}^{*}\right)\le Cd{T}^{\frac{1}{2}}\left\{R_{\theta }+\frac{(\sqrt{\nu }+\sqrt{\xi }+1)\left(1-2p_0\right)\left(R+R_{\theta }\right)}{\sqrt{p_0}(1-\rho ){\lambda }_0}\sqrt{1+\frac{\text{log}T}{d}}\right\}+\frac{4\left(1-2p_0\right)R_{\theta }d}{(1-\rho ){\lambda }_0}\sum _{t=T_0-1}^T\underset{a\in \left\{\mathrm{0,1}\right\}}{\text{max}}{∥{\mathbf{\Delta }}_t\left(a\right)∥}_2.$$

The proof is in Appendix G.7.

G Additional proofs

G. 1. Proof of Theorem C. 2

The proof of Theorem C. 2 is very similar to Theorem 2.2, and we only need to note the difference in Lemma E. 2 and E. 3 (for the standard setting and the clipped policy setting respectively), as stated below. Recall that

$$\begin{gathered} {\widehat{\text{Regret}}}_t\left(\pi ,{\pi }^{*}\right):={\mathbb{E}}_{a~{\pi }_t^{*}}⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩-{\mathbb{E}}_{a~{\pi }_t}⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩, \\ {\widehat{\text{Regret}}}_t\left(\pi ,{\bar{\pi }}^{*}\right):={\mathbb{E}}_{a~{\bar{\pi }}_t^{*}}⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩-{\mathbb{E}}_{a~{\pi }_t}⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩. \end{gathered}$$

Standard setting. At any time $t>T_0$, we have

$$\begin{gathered} {\widehat{\text{Regret}}}_t\left(\pi ,{\pi }^{*}\right) ={\mathbb{E}}_{a~{\pi }_t^{*}}⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩-{\mathbb{E}}_{a~{\pi }_t}⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩ \\ =⟨{\mathit{\theta }}_{a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩-\left[\left(1-(K-1)p_0^{\left(t\right)}\right)⟨{\mathit{\theta }}_{{\tilde{a}}_t}^{*},{\tilde{\mathbf{x}}}_t⟩+\sum _{a\ne {\tilde{a}}_t}{p}_0^{\left(t\right)}⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩\right] \\ =p_0^{\left(t\right)}\sum _{a\ne a_t^{*}}⟨{\mathit{\theta }}_{a_t^{*}}^{*}-{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩+1_{\left\{a_t^{*}\ne {\tilde{a}}_t\right\}}\left(1-K{p}_0^{\left(t\right)}\right)⟨{\mathit{\theta }}_{a_t^{*}}^{*}-{\mathit{\theta }}_{{\tilde{a}}_t}^{*},{\tilde{\mathbf{x}}}_t⟩ \\ \le 2(K-1)p_0^{\left(t\right)}{R}_{\theta }+1_{\left\{a_t^{*}\ne {\tilde{a}}_t\right\}}⟨{\mathit{\theta }}_{a_t^{*}}^{*}-{\mathit{\theta }}_{{\tilde{a}}_t}^{*},{\tilde{\mathbf{x}}}_t⟩. \end{gathered}$$ (G.1)

Here note that Lemma E. 1 still holds under Assumption C.1, so $a_t^{*}={\text{argmax}}_a⟨{\mathit{\theta }}_a^{*},{\mathbf{x}}_t⟩={\text{argmax}}_a⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩,{\tilde{a}}_t={\text{argmax}}_a⟨{\widehat{\mathit{\theta }}}_a^{(t-1)},{\tilde{\mathbf{x}}}_t⟩$.

Note that $a_t^{*}\ne {\tilde{a}}_t$ implies that

$$\left\{\begin{array}{l}⟨{\mathit{\theta }}_{a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩\ge ⟨{\mathit{\theta }}_{{\tilde{a}}_t}^{*},{\tilde{\mathbf{x}}}_t⟩\\ ⟨{\widehat{\mathit{\theta }}}_{a_t^{*}}^{(t-1)},{\tilde{\mathbf{x}}}_t⟩\le ⟨{\widehat{\mathit{\theta }}}_{{\tilde{a}}_t}^{(t-1)},{\tilde{\mathbf{x}}}_t⟩\end{array}\right.$$

which leads to

$$\begin{gathered} ⟨{\mathit{\theta }}_{a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩ \ge ⟨{\mathit{\theta }}_{{\tilde{a}}_t}^{*},{\tilde{\mathbf{x}}}_t⟩\ge ⟨{\widehat{\mathit{\theta }}}_{{\tilde{a}}_t}^{(t-1)},{\tilde{\mathbf{x}}}_t⟩-\underset{a}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2 \\ \ge ⟨{\widehat{\mathit{\theta }}}_{a_t^{*}}^{(t-1)},{\tilde{\mathbf{x}}}_t⟩-\underset{a}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2\ge ⟨{\mathit{\theta }}_{a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩-2\underset{a}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2, \end{gathered}$$

and further implies $\left|⟨{\mathit{\theta }}_{a_t^{*}}^{*}-{\mathit{\theta }}_{{\tilde{a}}_t}^{*},{\tilde{\mathbf{x}}}_t⟩\right|\le 2{\text{max}}_a{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2$. Plugging in the above to (G.1) leads to

$${\widehat{\text{Regret}}}_t\left(\pi ,{\pi }^{*}\right)\le 2(K-1)p_0^{\left(t\right)}{R}_{\theta }+2\underset{a}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2\text{. }$$

The rest of the proof can be done in the same way as the proof of Theorem 2.2.

Clipped policy setting. At any time $t>T_0$, we have

$$\begin{gathered} {\widehat{\text{Regret}}}_t\left(\pi ,{\bar{\pi }}^{*}\right) ={\mathbb{E}}_{a~{\pi }_t^{*}}⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩-{\mathbb{E}}_{a~{\pi }_t}⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩ \\ \le \left(1-K{p}_0\right)1_{\left\{a_t^{*}\ne {\tilde{a}}_t\right\}}\left|⟨{\mathit{\theta }}_{a_t^{*}}^{*}-{\mathit{\theta }}_{{\tilde{a}}_t}^{*},{\tilde{\mathbf{x}}}_t⟩\right|. \end{gathered}$$ (G.2)

Here recall that $a_t^{*}={\text{argmax}}_a⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩,{\tilde{a}}_t:={\mathrm{a}\mathrm{r}\mathrm{g}\text{max}}_a⟨{\widehat{\mathit{\theta }}}_a^{(t-1)},{\tilde{\mathbf{x}}}_t⟩$.

Note that $a_t^{*}\ne {\tilde{a}}_t$ implies that

$$\left\{\begin{array}{l}⟨{\mathit{\theta }}_{a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩\ge ⟨{\mathit{\theta }}_{{\tilde{a}}_t}^{*},{\tilde{\mathbf{x}}}_t⟩\\ ⟨{\widehat{\mathit{\theta }}}_{a_t^{*}}^{(t-1)},{\tilde{\mathbf{x}}}_t⟩\le ⟨{\widehat{\mathit{\theta }}}_{{\tilde{a}}_t}^{(t-1)},{\tilde{\mathbf{x}}}_t⟩\end{array}\right.$$

which leads to

$$\begin{gathered} ⟨{\mathit{\theta }}_{a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩ \ge ⟨{\mathit{\theta }}_{a_t}^{*},{\tilde{\mathbf{x}}}_t⟩\ge ⟨{\widehat{\mathit{\theta }}}_{{\tilde{a}}_t}^{(t-1)},{\tilde{\mathbf{x}}}_t⟩-\underset{a}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2 \\ \ge ⟨{\widehat{\mathit{\theta }}}_{a_t^{*}}^{(t-1)},{\tilde{\mathbf{x}}}_t⟩-\underset{a}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2\ge ⟨{\mathit{\theta }}_{a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩-2\underset{a}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2, \end{gathered}$$

and further implies $\left|⟨{\mathit{\theta }}_{a_t^{*}}^{*}-{\mathit{\theta }}_{{\tilde{a}}_t}^{*},{\tilde{\mathbf{x}}}_t⟩\right|\le 2{\text{max}}_a{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2$.

Plugging in the above to (G.2) leads to

$${\widehat{\text{Regret}}}_t\left(\pi ,{\bar{\pi }}^{*}\right)\le 2\left(1-K{p}_0\right)\underset{a}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2$$

for $t>T_0$. The rest of the proof can be done in the same way as the proof of Theorem 2.2.

G. 2. Proof of Lemma D. 1

We first analyze ${\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},a}^{\left(t\right)}$. Notice that ${\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},a}^{\left(t\right)}=\frac{1}{t}{\sum }_{\tau \in \left[t\right]}{\mathit{V}}_{\tau ,a}$, where ${\mathit{V}}_{\tau ,a}=\frac{{\pi }_{\tau }^{nd}\left(A_{\tau }\right)}{{\pi }_{\tau }\left(A_{\tau }\mid {\tilde{\mathbf{x}}}_{\tau },{\mathscr{H}}_{\tau -1}\right)}{1}_{\left\{A_{\tau }=a\right\}}{\tilde{\mathbf{x}}}_{\tau }{\tilde{\mathbf{x}}}_{\tau }^{\top }$. For any fixed $\mathit{u}\in {\mathbb{S}}^{d-1}:=\left\{{\mathit{u}}^{\prime }\in {\mathbb{R}}^d:{∥{\mathit{u}}^{\prime }∥}_2=1\right\},{\left(v_{\mathit{u},\tau ,a}:={\mathit{u}}^{\tau }\left[{\mathit{V}}_{\tau ,a}-\mathbb{E}\left[{\mathit{V}}_{\tau ,a}\mid {\mathscr{H}}_{\tau -1}\right]\right]\mathit{u}\right)}_{\tau }$ is a martingale difference sequence. Moreover, we can verify that $\left|v_{\mathit{u},\tau ,a}\right|\le \frac{2}{q_t}$ and

$$\begin{gathered} \text{Var}\left(v_{\mathit{u},\tau ,a}\mid {\mathscr{H}}_{\tau -1}\right) \le \mathbb{E}\left[{\left({\mathit{u}}^{\top }{\mathit{V}}_{\tau ,a}\mathit{u}\right)}^2\mid {\mathscr{H}}_{\tau -1}\right]=\mathbb{E}\left[\left.{\left(\frac{{\pi }_{\tau }^{nd}\left(A_{\tau }\right)}{{\pi }_{\tau }\left(A_{\tau }\mid {\tilde{\mathbf{x}}}_{\tau },{\mathscr{H}}_{\tau -1}\right)}\right)}^2{1}_{\left\{A_{\tau }=a\right\}}{\left({\mathit{u}}^{\top }{\tilde{\mathbf{x}}}_{\tau }{\tilde{\mathbf{x}}}_{\tau }^{\top }\mathit{u}\right)}^2\right|{\mathscr{H}}_{\tau -1}\right] \\ ={\mathbb{E}}_{{\mathit{ϵ}}_{\tau },A_{\tau }~{\pi }_{\tau }^{nd}(\cdot )}\left[\left.\left(\frac{{\pi }_{\tau }^{nd}\left(A_{\tau }\right)}{{\pi }_{\tau }\left(A_{\tau }\mid {\tilde{\mathbf{x}}}_{\tau },{\mathscr{H}}_{\tau -1}\right)}\right)1_{\left\{A_{\tau }=a\right\}}{\left({\mathit{u}}^{\top }{\tilde{\mathbf{x}}}_{\tau }{\tilde{\mathbf{x}}}_{\tau }^{\top }\mathit{u}\right)}^2\right|{\mathscr{H}}_{\tau -1}\right]\le \frac{1}{q_t}\mathbb{E}{\left({\mathit{u}}^{\top }{\tilde{\mathbf{x}}}_{\tau }\right)}^4\le \frac{\xi }{d^2{q}_t}. \end{gathered}$$

According to Freedman’s Inequality (Freedman, 1975), for any ${\gamma }_1,{\gamma }_2>0$,

$$\mathbb{P}\left(\sum _{\tau \in \left[t\right]}{v}_{\mathit{u},\tau ,a}\ge {\gamma }_1,\sum _{\tau \in \left[t\right]}\text{Var}\left(v_{\mathit{u},\tau ,a}\mid {\mathscr{H}}_{\tau -1}\right)\le {\gamma }_2\right)\le e^{-\frac{{\gamma }_1^2}{2\left(\frac{2}{q_t}{\gamma }_1+{\gamma }_2\right)}}.$$

Set ${\gamma }_2=\frac{\xi t}{d^2{q}_t}$, and we obtain $\mathbb{P}\left({\sum }_{\tau \in \left[t\right]}{v}_{\mathit{u},\tau ,a}\ge {\gamma }_1\right)\le e^{-\frac{d^2{q}_t{\gamma }_1^2}{2\left(2d^2{\gamma }_1+\xi t\right)}}$. Applying the same analysis to ${\left(-v_{\mathit{u},\tau ,a}\right)}_{\tau }$ and combining the results gives $\mathbb{P}\left(\left|{\sum }_{\tau \in \left[t\right]}{v}_{\mathit{u},\tau ,a}\right|\ge {\gamma }_1\right)\le 2e^{-\frac{d^2{q}_t{\gamma }_1^2}{2\left(2d^2{\gamma }_1+\xi t\right)}}$.

Denote ${\mathit{M}}_t=\frac{1}{t}{\sum }_{\tau \in \left[t\right]}\left({\mathit{V}}_{\tau ,a}-\mathbb{E}\left[{\mathit{V}}_{\tau ,a}\mid {\mathscr{H}}_{\tau -1}\right]\right)$, then the above means that $\forall \mathit{u}\in {\mathbb{S}}^{d-1}$,

$$\mathbb{P}\left(\left|{\mathit{u}}^{\top }{\mathit{M}}_t\mathit{u}\right|\ge \frac{{\gamma }_1}{t}\right)\le 2e^{-\frac{d^2{q}_t{\gamma }_1^2}{2\left(2d^2{\gamma }_1+\xi t\right)}}.$$ (G.3)

Let $𝒩$ be a $\frac{1}{4}$-net of ${\mathbb{S}}^{d-1},\left|𝒩\right|\le 9^d.\forall \mathit{u}\in {\mathbb{S}}^{d-1}$, find ${\mathit{u}}^{\prime }\in 𝒩$ s.t. ${∥\mathit{u}-{\mathit{u}}^{\prime }∥}_2\le \frac{1}{4}$, and we have

$$\left|{\mathit{u}}^{\top }{\mathit{M}}_t\mathit{u}-{\mathit{u}}^{\prime \top }{\mathit{M}}_t{\mathit{u}}^{\prime }\right|\le \left|{\mathit{u}}^{\top }{\mathit{M}}_t\left(\mathit{u}-{\mathit{u}}^{\prime }\right)\right|+\left|{\mathit{u}}^{\prime \top }{\mathit{M}}_t\left(\mathit{u}-{\mathit{u}}^{\prime }\right)\right|\le \frac{1}{2}{∥{\mathit{M}}_t∥}_2.$$

This implies that

$${∥{\mathit{M}}_t∥}_2=\underset{\mathit{u}\in {\mathbb{S}}^{d-1}}{\mathrm{s}\mathrm{u}\mathrm{p}}\left|{\mathit{u}}^{\top }{\mathit{M}}_t\mathit{u}\right|\le \underset{{\mathit{u}}^{\prime }\in 𝒩}{\mathrm{s}\mathrm{u}\mathrm{p}}\left|{\mathit{u}}^{\prime \top }{\mathit{M}}_t{\mathit{u}}^{\prime }\right|+\frac{1}{2}{∥{\mathit{M}}_t∥}_2,$$

and thus ${\mathrm{s}\mathrm{u}\mathrm{p}}_{\mathit{u}\in 𝒩}\left|{\mathit{u}}^{\top }{\mathit{M}}_t\mathit{u}\right|\ge \frac{1}{2}{∥{\mathit{M}}_t∥}_2$. Combining the above and (G.3), we obtain that for any ${\gamma }_1>0$,

$$\mathbb{P}\left({∥{\mathit{M}}_t∥}_2\ge \frac{2{\gamma }_1}{t}\right)\le \mathbb{P}\left(\underset{\mathit{u}\in 𝒩}{\mathrm{s}\mathrm{u}\mathrm{p}}\left|{\mathit{u}}^{\top }{\mathit{M}}_t\mathit{u}\right|\ge \frac{{\gamma }_1}{t}\right)\le 9^d\cdot \mathbb{P}\left(\left|{\mathit{u}}^{\top }{\mathit{M}}_t\mathit{u}\right|\ge \frac{{\gamma }_1}{t}\right)=2\cdot 9^d\cdot e^{-\frac{d^2{q}_t{\gamma }_1^2}{2\left(2d^2{\gamma }_1+\xi t\right)}}.$$

By choosing ${\gamma }_1=24\text{max}\left\{\frac{d+\text{log}t}{q_t},\frac{\sqrt{\xi }}{d}\cdot \sqrt{\frac{t(d+\text{log}t)}{q_t}}\right\}$, and noticing that ${\mathit{M}}_t={\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},a}^{\left(t\right)}-\frac{1}{t}{\sum }_{\tau \in \left[t\right]}\mathbb{E}\left[{\mathit{V}}_{\tau ,a}\mid {\mathscr{H}}_{\tau -1}\right]$, we have

$$\mathbb{P}\left({∥{\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},a}^{\left(t\right)}-\frac{1}{t}\sum _{\tau \in \left[t\right]}\mathbb{E}\left[{\mathit{V}}_{\tau ,a}\mid {\mathscr{H}}_{\tau -1}\right]∥}_2\ge 48\text{max}\left\{\frac{d+\text{log}t}{q_{t}t},\frac{\sqrt{\xi }}{d}\cdot \sqrt{\frac{d+\text{log}t}{q_{t}t}}\right\}\right)\le \frac{2}{t^2}.$$ (G.4)

At the same time, we have

$$\begin{gathered} \mathbb{E}\left[{\mathit{V}}_{\tau ,a}\mid {\mathscr{H}}_{\tau -1}\right] ={\mathbb{E}}_{{\mathit{ϵ}}_{\tau }}\left[\left.{\mathbb{E}}_{A_{\tau }~{\pi }_{\tau }\left(\cdot \mid {\tilde{\mathbf{x}}}_{\tau },{\mathscr{H}}_{\tau -1}\right)}\left[\left.\frac{{\pi }_{\tau }^{nd}\left(A_{\tau }\right)}{{\pi }_{\tau }\left(A_{\tau }\mid {\tilde{\mathbf{x}}}_{\tau },{\mathscr{H}}_{\tau -1}\right)}{1}_{\left\{A_{\tau }=a\right\}}{\tilde{\mathbf{x}}}_{\tau }{\tilde{\mathbf{x}}}_{\tau }^{\top }\right|{\mathit{ϵ}}_{\tau },{\mathscr{H}}_{\tau -1}\right]\right|{\mathscr{H}}_{\tau -1}\right] \\ ={\mathbb{E}}_{{\mathit{ϵ}}_{\tau }}\left[{\mathbb{E}}_{A_{\tau }~{\pi }_{\tau }^{nd}(\cdot )}\left[1_{\left\{A_{\tau }=a\right\}}{\tilde{\mathbf{x}}}_{\tau }{\tilde{\mathbf{x}}}_{\tau }^{\top }\mid {\mathit{ϵ}}_{\tau },{\mathscr{H}}_{\tau -1}\right]\mid {\mathscr{H}}_{\tau -1}\right] \\ ={\mathbb{E}}_{{\mathit{ϵ}}_{\tau }}\left[{\pi }_{\tau }^{nd}\left(a\right)\cdot {\tilde{\mathbf{x}}}_{\tau }{\tilde{\mathbf{x}}}_{\tau }^{\top }\mid {\mathscr{H}}_{\tau -1}\right] \\ ={\pi }_{\tau }^{nd}\left(a\right)\left({\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }+{\mathbf{\Sigma }}_{e,\tau }\right). \end{gathered}$$ (G.5)

Here we’ve used the facts that (i) ${\left\{{\pi }_{\tau }^{nd}\right\}}_{\tau }$ is data-independent; (ii) $\mathbb{E}\left[{\mathit{ϵ}}_{\tau }\mid {\mathscr{H}}_{\tau -1}\right]=0,\text{Var}\left[{\mathit{ϵ}}_{\tau }\mid {\mathscr{H}}_{\tau -1}\right]={\mathbf{\Sigma }}_{e,\tau }$. Plug (G.5) into (G.4), and we get

$$\mathbb{P}\left({∥{\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},a}^{\left(t\right)}-\frac{1}{t}\sum _{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}\left(a\right)\left({\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }+{\mathbf{\Sigma }}_{e,\tau }\right)∥}_2\ge 48\text{max}\left\{\frac{d+\text{log}t}{q_{t}t},\frac{\sqrt{\xi }}{d}\cdot \sqrt{\frac{d+\text{log}t}{q_{t}t}}\right\}\right)\le \frac{2}{t^2}.$$ (G.6)

The analysis for ${\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},r,a}^{\left(t\right)}$ is similar. Write ${\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},r,a}^{\left(t\right)}=\frac{1}{t}{\sum }_{\tau \in \left[t\right]}{\mathit{Z}}_{\tau ,a},{\mathit{Z}}_{\tau ,a}:=\frac{{\pi }_{\tau }^{nd}\left(A_{\tau }\right)}{{\pi }_{\tau }\left(A_{\tau }{\tilde{\mathbf{x}}}_{\tau },{\mathscr{H}}_{\tau -1}\right)}{1}_{\left\{A_{\tau }=a\right\}}{\tilde{\mathbf{X}}}_{\tau }{r}_{\tau }$. Then for any $\mathit{u}\in {𝒮}^{d-1}$, it’s easy to verify that $\left|{\left({\mathit{Z}}_{\tau ,a}-\mathbb{E}\left[{\mathit{Z}}_{\tau ,a}\mid {\mathscr{H}}_{\tau -1}\right]\right)}^{\top }\mathit{u}\right|\le \frac{2R}{q_t}$, and $\text{Var}\left(\left({\mathit{Z}}_{\tau ,a}-\right.\right.\left.{\left.\mathbb{E}\left[{\mathit{Z}}_{\tau ,a}\mid {\mathscr{H}}_{\tau -1}\right]\right)}^{\top }\mathit{u}\mid {\mathscr{H}}_{\tau -1}\right)\le \mathbb{E}\left[{\left({\mathit{Z}}_{\tau ,a}^{\top }\mathit{u}\right)}^2\mid {\mathscr{H}}_{\tau -1}\right]\le \frac{\nu R^2}{q_{t}d}$. Applying Freedman’s Inequality leads to

$$\mathbb{P}\left(\left|{\mathit{z}}_t^{\top }\mathit{u}\right|\ge \frac{{\gamma }_1}{t}\right)\le 2e^{-\frac{d{q}_t{\gamma }_1^2}{4Rd{\gamma }_1+2R^2\nu t}},$$ (G.7)

where ${\mathit{z}}_t:=\frac{1}{t}{\sum }_{\tau \in \left[t\right]}\left({\mathit{Z}}_{\tau ,a}-\mathbb{E}\left[{\mathit{Z}}_{\tau ,a}\mid {\mathscr{H}}_{\tau -1}\right]\right)$.

Recall that $𝒩$ is a $\frac{1}{4}$-net of ${\mathbb{S}}^{d-1},\left|𝒩\right|\le 9^d.\forall \mathit{u}\in {\mathbb{S}}^{d-1}$, find ${\mathit{u}}^{\prime }\in {\mathbb{S}}^{d-1}$ s.t. $∥\mathit{u}-{\mathit{u}}^{\prime }∥\le 1/4$, then $\left|{\mathit{z}}_t^{\top }\mathit{u}-{\mathit{z}}_t^{\top }{\mathit{u}}^{\prime }\right|\le \frac{1}{4}{∥{\mathit{z}}_t∥}_2$, and thus

$${∥{\mathit{z}}_t∥}_2=\underset{\mathit{u}\in {\mathbb{S}}^{d-1}}{\mathrm{s}\mathrm{u}\mathrm{p}}\left|{\mathit{z}}_t^{\top }\mathit{u}\right|\le \underset{{\mathit{u}}^{\prime }\in 𝒩}{\mathrm{s}\mathrm{u}\mathrm{p}}\left|{\mathit{z}}_t^{\top }{\mathit{u}}^{\prime }\right|+\frac{1}{4}{∥{\mathit{z}}_t∥}_2$$

which implies that ${\mathrm{s}\mathrm{u}\mathrm{p}}_{\mathit{u}\in 𝒩}\left|{\mathit{z}}_t^{\top }\mathit{u}\right|\ge \frac{3}{4}{∥{\mathit{z}}_t∥}_2$. Taking this and (G.7) into account, we derive that

$$\mathbb{P}\left({∥{\mathit{z}}_t∥}_2\ge \frac{4}{3}\frac{{\gamma }_1}{t}\right)\le \mathbb{P}\left(\underset{\mathit{u}\in 𝒩}{\mathrm{s}\mathrm{u}\mathrm{p}}\left|{\mathit{z}}_t^{\top }\mathit{u}\right|\ge \frac{{\gamma }_1}{t}\right)\le 9^d\cdot 2e^{-\frac{d{q}_t{\gamma }_1^2}{4Rd{\gamma }_1+2R^2\nu t}}$$

By choosing ${\gamma }_1=24R \text{max}\left\{\frac{d+\text{log}t}{q_t},\sqrt{\nu }\cdot \sqrt{\frac{(d+\text{log}t)t}{d{q}_t}}\right\}$ and noticing that ${\mathit{z}}_t={\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},r,a}^{\left(t\right)}-\frac{1}{t}{\sum }_{\tau \in \left[t\right]}\mathbb{E}\left[{\mathit{Z}}_{\tau ,a}\mid {\mathscr{H}}_{\tau -1}\right]$, we obtain

$$\mathbb{P}\left({∥{\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},r,a}^{\left(t\right)}-\frac{1}{t}\sum _{\tau \in \left[t\right]}\mathbb{E}\left[{\mathit{Z}}_{\tau ,a}\mid {\mathscr{H}}_{\tau -1}\right]∥}_2\ge 32R \text{max}\left\{\frac{d+\text{log}t}{q_{t}t},\sqrt{\frac{\nu }{d}}\cdot \sqrt{\frac{d+\text{log}t}{q_{t}t}}\right\}\right)\le \frac{2}{t^2}.$$ (G.8)

Finally, because

$$\begin{gathered} \mathbb{E}\left[{\mathit{Z}}_{\tau ,a}\mid {\mathscr{H}}_{\tau -1}\right] ={\mathbb{E}}_{{\mathit{ϵ}}_{\tau }}\left[\left.{\mathbb{E}}_{A_{\tau }~{\pi }_{\tau }\left(\cdot \mid {\tilde{\mathbf{x}}}_{\tau },{\mathscr{H}}_{\tau -1}\right),{\eta }_{\tau }}\left[\left.\frac{{\pi }_{\tau }^{nd}\left(A_{\tau }\right)}{{\pi }_{\tau }\left(A_{\tau }\mid {\tilde{\mathbf{x}}}_{\tau },{\mathscr{H}}_{\tau -1}\right)}{1}_{\left\{A_{\tau }=a\right\}}{\tilde{\mathbf{x}}}_{\tau }{r}_{\tau }\right|{\mathscr{H}}_{\tau -1},{\mathit{ϵ}}_{\tau }\right]\right|{\mathscr{H}}_{\tau -1}\right] \\ ={\mathbb{E}}_{{\mathit{ϵ}}_{\tau }}\left[{\mathbb{E}}_{A_{\tau }~{\pi }_{\tau }^{nd}(\cdot ),{\eta }_{\tau }}\left[1_{\left\{A_{\tau }=a\right\}}{\tilde{\mathbf{x}}}_{\tau }{r}_{\tau }\mid {\mathscr{H}}_{\tau -1},{\mathit{ϵ}}_{\tau }\right]\mid {\mathscr{H}}_{\tau -1}\right] \\ ={\mathbb{E}}_{{\mathit{ϵ}}_{\tau }}\left[{\mathbb{E}}_{A_{\tau }~{\pi }_{\tau }^{nd}(\cdot ),{\eta }_{\tau }}\left[1_{\left\{A_{\tau }=a\right\}}{\tilde{\mathbf{x}}}_{\tau }\left({\mathbf{x}}_{\tau }^{\top }{\mathit{\theta }}_a^{*}+{\eta }_{\tau }\right)\mid {\mathscr{H}}_{\tau -1},{\mathit{ϵ}}_{\tau }\right]\mid {\mathscr{H}}_{\tau -1}\right] \\ ={\mathbb{E}}_{{\mathit{ϵ}}_{\tau }}\left[{\pi }_{\tau }^{nd}\left(a\right){\mathbb{E}}_{{\eta }_{\tau }}\left[{\tilde{\mathbf{x}}}_{\tau }\left({\mathbf{x}}_{\tau }^{\top }{\mathit{\theta }}_a^{*}+{\eta }_{\tau }\right)\mid {\mathscr{H}}_{\tau -1},{\mathit{ϵ}}_{\tau }\right]\mid {\mathscr{H}}_{\tau -1}\right] \\ ={\pi }_{\tau }^{nd}\left(a\right){\mathbb{E}}_{{\mathit{ϵ}}_{\tau }}\left[{\tilde{\mathbf{x}}}_{\tau }{\mathbf{x}}_{\tau }^{\top }{\mathit{\theta }}_a^{*}\mid {\mathscr{H}}_{\tau -1}\right]={\pi }_{\tau }^{nd}\left(a\right)\left({\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }\right){\mathit{\theta }}_a^{*}, \end{gathered}$$

Plug in (G.8), and we obtain

$$\mathbb{P}\left({∥{\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},r,a}^{\left(t\right)}-\left[\frac{1}{t}\sum _{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}(a){\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }\right]{\mathit{\theta }}_a^{*}∥}_2\ge 32R \text{max}\left\{\frac{d+\text{log}t}{q_{t}t},\sqrt{\frac{\nu }{d}}\cdot \sqrt{\frac{d+\text{log}t}{q_{t}t}}\right\}\right)\le \frac{2}{t^2}.$$ (G.9)

Combining (G.9) and (G.6), we conclude the proof.

G. 3. Proof of Lemma E.1

We only need to prove

$$\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\left(⟨{\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\mathbf{x}}_t⟩\right)=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\left(⟨{\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\tilde{\mathbf{x}}}_t⟩\right).$$ (G.10)

If $⟨{\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\mathbf{x}}_t⟩=0,(\mathrm{G}.10)$ is a direct consequence of Assumption 2.3. If $⟨{\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\mathbf{x}}_t⟩\ne 0$, without loss of generality, suppose $⟨{\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\mathbf{x}}_t⟩>0$. Then according to Assumption 2.3,

$$⟨{\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\tilde{\mathbf{x}}}_t⟩=⟨{\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\mathbf{x}}_t⟩+⟨{\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\mathit{ϵ}}_t⟩\ge (1-\rho )⟨{\mathit{\theta }}_1^{*}-{\mathit{\theta }}_0^{*},{\mathbf{x}}_t⟩>0.$$

Thus (G.10) is true.

G. 4. Proof of Lemma E.2

We have

$$\begin{gathered} {\widehat{\text{Regret}}}_t\left(\pi ,{\pi }^{*}\right) ={\mathbb{E}}_{a~{\pi }_t^{*}}⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩-{\mathbb{E}}_{a~{\pi }_t}⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩ \\ =⟨{\mathit{\theta }}_{a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩-\left[\left(1-p_0^{\left(t\right)}\right)⟨{\mathit{\theta }}_{{\tilde{a}}_t}^{*},{\tilde{\mathbf{x}}}_t⟩+p_0^{\left(t\right)}⟨{\mathit{\theta }}_{1-{\tilde{a}}_t}^{*},{\tilde{\mathbf{x}}}_t⟩\right] \\ =1_{\left\{a_t^{*}={\tilde{a}}_t\right\}}{p}_0^{\left(t\right)}⟨{\mathit{\theta }}_{a_t^{*}}^{*}-{\mathit{\theta }}_{1-a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩+1_{\left\{a_t^{*}\ne {\tilde{a}}_t\right\}}\left(1-p_0^{\left(t\right)}\right)⟨{\mathit{\theta }}_{a_t^{*}}^{*}-{\mathit{\theta }}_{1-a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩ \\ \le 2p_0^{\left(t\right)}{R}_{\theta }+1_{\left\{a_t^{*}\ne {\tilde{a}}_t\right\}}⟨{\mathit{\theta }}_{a_t^{*}}^{*}-{\mathit{\theta }}_{1-a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩. \end{gathered}$$ (G.11)

Here recall that $a_t^{*}={\mathrm{a}\mathrm{r}\mathrm{g}\text{max}}_a⟨{\mathit{\theta }}_a^{*},{\mathbf{x}}_t⟩={\mathrm{a}\mathrm{r}\mathrm{g}\text{max}}_a⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩,{\tilde{a}}_t:={\mathrm{a}\mathrm{r}\mathrm{g}\text{max}}_{a\in \left\{\mathrm{0,1}\right\}}⟨{\widehat{\mathit{\theta }}}_a^{(t-1)},{\tilde{\mathbf{x}}}_t⟩$.

Note that $a_t^{*}\ne {\tilde{a}}_t$ implies that

$$\left\{\begin{array}{l}⟨{\mathit{\theta }}_{a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩\ge ⟨{\mathit{\theta }}_{1-a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩\\ ⟨{\widehat{\mathit{\theta }}}_{a_t^{*}}^{(t-1)},{\tilde{\mathbf{x}}}_t⟩\le ⟨{\widehat{\mathit{\theta }}}_{1-a_t^{*}}^{(t-1)},{\tilde{\mathbf{x}}}_t⟩\end{array}\right.$$

which leads to

$$⟨{\mathit{\theta }}_{a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩ \ge ⟨{\mathit{\theta }}_{1-a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩\ge ⟨{\widehat{\mathit{\theta }}}_{1-a_t^{*}}^{(t-1)},{\tilde{\mathbf{x}}}_t⟩-\underset{a}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2 \ge ⟨{\widehat{\mathit{\theta }}}_{a_t^{*}}^{(t-1)},{\tilde{\mathbf{x}}}_t⟩-\underset{a}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2\ge ⟨{\mathit{\theta }}_{a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩-2\underset{a}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2,$$

and further implies $\left|⟨{\mathit{\theta }}_{a_t^{*}}^{*}-{\mathit{\theta }}_{1-a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩\right|\le 2{\text{max}}_a{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2$.

Plugging in the above to (G.11) leads to

$${\widehat{\text{Regret}}}_t\left(\pi ,{\pi }^{*}\right)\le 2p_0^{\left(t\right)}{R}_{\theta }+2\underset{a}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2\text{. }$$

G. 5. Proof of Lemma E.3

At any time $t>T_0$, we have

$$\begin{gathered} {\widehat{\text{Regret}}}_t\left(\pi ,{\bar{\pi }}^{*}\right) ={\mathbb{E}}_{a~{\bar{\pi }}_t^{*}}⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩-{\mathbb{E}}_{a~{\pi }_t}⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩ \\ =\left[\left(1-p_0\right)⟨{\mathit{\theta }}_{a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩+p_0⟨{\mathit{\theta }}_{1-a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩\right]-\left[\left(1-p_0\right)⟨{\mathit{\theta }}_{{\tilde{a}}_t}^{*},{\tilde{\mathbf{x}}}_t⟩+p_0⟨{\mathit{\theta }}_{1-{\tilde{a}}_t}^{*},{\tilde{\mathbf{x}}}_t⟩\right] \\ \le \left(1-2p_0\right)1_{\left\{a_t^{*}\ne {\tilde{a}}_t\right\}}\left|⟨{\mathit{\theta }}_{a_t^{*}}^{*}-{\mathit{\theta }}_{1-a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩\right|. \end{gathered}$$ (G.12)

Here recall that $a_t^{*}={\mathrm{a}\mathrm{r}\mathrm{g}\text{max}}_a⟨{\mathit{\theta }}_a^{*},{\mathbf{x}}_t⟩={\mathrm{a}\mathrm{r}\mathrm{g}\text{max}}_a⟨{\mathit{\theta }}_a^{*},{\tilde{\mathbf{x}}}_t⟩,{\tilde{a}}_t:={\mathrm{a}\mathrm{r}\mathrm{g}\text{max}}_{a\in \left\{\mathrm{0,1}\right\}}⟨{\widehat{\mathit{\theta }}}_a^{(t-1)},{\tilde{\mathbf{x}}}_t⟩$.

Note that $a_t^{*}\ne {\tilde{a}}_t$ implies that

$$\left\{\begin{array}{l}⟨{\mathit{\theta }}_{a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩\ge ⟨{\mathit{\theta }}_{1-a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩\\ ⟨{\widehat{\mathit{\theta }}}_{a_t^{*}}^{(t-1)},{\tilde{\mathbf{x}}}_t⟩\le ⟨{\widehat{\mathit{\theta }}}_{1-a_t^{*}}^{(t-1)},{\tilde{\mathbf{x}}}_t⟩\end{array}\right.$$

which leads to

$$\begin{gathered} ⟨{\mathit{\theta }}_{a_t^{*}}^{*},{\tilde{\mathit{x}}}_t⟩ \ge ⟨{\mathit{\theta }}_{1-a_t^{*}}^{*},{\tilde{\mathit{x}}}_t⟩\ge ⟨{\widehat{\mathit{\theta }}}_{1-a_t^{*}}^{(t-1)},{\tilde{\mathit{x}}}_t⟩-\underset{a}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2 \\ \ge ⟨{\widehat{\mathit{\theta }}}_{a_t^{*}}^{(t-1)},{\tilde{\mathit{x}}}_t⟩-\underset{a}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2\ge ⟨{\mathit{\theta }}_{a_t^{*}}^{*},{\tilde{\mathit{x}}}_t⟩-2\underset{a}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2, \end{gathered}$$

and further implies $\left|⟨{\mathit{\theta }}_{a_t^{*}}^{*}-{\mathit{\theta }}_{1-a_t^{*}}^{*},{\tilde{\mathbf{x}}}_t⟩\right|\le 2{\text{max}}_a{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2$.

Plugging in the above to (G.12) leads to

$${\widehat{\text{Regret}}}_t\left(\pi ,{\bar{\pi }}^{*}\right)\le 2\left(1-2p_0\right)\underset{a}{\text{max}}{∥{\widehat{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2$$

G. 6. Proof of Theorem F.1

Fix $t\in \left[T\right]$ such that the conditions of Theorem 2.3 hold. Fix $a\in \left\{\mathrm{0,1}\right\}$. As in Appendix D, define ${\mathbf{\Delta }}_1:={\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},a}^{\left(t\right)}-\frac{1}{t}{\sum }_{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}\left(a\right)\left({\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }+{\mathbf{\Sigma }}_{e,\tau }\right),{\mathbf{\Delta }}_2:={\widehat{\mathbf{\Sigma }}}_{\tilde{\mathbf{x}},r,a}^{\left(t\right)}-\left(\frac{1}{t}{\sum }_{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}\left(a\right){\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }\right){\mathit{\theta }}_a^{*}$. We also let ${\mathbf{\Delta }}_3:=-{\mathbf{\Delta }}_t\left(a\right)=-\frac{1}{t}{\sum }_{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}\left(a\right)\left({\widehat{\mathbf{\Sigma }}}_{e,\tau }-{\mathbf{\Sigma }}_{e,\tau }\right)$. Recall Lemma D.1: with probability at least $1-4/t^2$,

$${∥{\mathbf{\Delta }}_1∥}_2\le C \text{max}\left\{\frac{d+\text{log}t}{q_{t}t},\frac{\sqrt{\xi }}{d}\sqrt{\frac{d+\text{log}t}{q_{t}t}}\right\},{∥{\mathbf{\Delta }}_2∥}_2\le CR \text{max}\left\{\frac{d+\text{log}t}{q_{t}t},\left.\sqrt{\frac{\nu }{d}}\sqrt{\frac{d+\text{log}t}{q_{t}t}}\right\}.\right.$$

Meanwhile,

$$\begin{gathered} {\tilde{\theta }}_a^{\left(t\right)}={\left({\widehat{\Sigma }}_{\tilde{x},a}^{\left(t\right)}-\frac{1}{t}\sum _{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}\left(a\right){\widehat{\Sigma }}_{e,\tau }\right)}^{-1}\cdot {\widehat{\Sigma }}_{\tilde{x},r,a}^{\left(t\right)} \\ ={\left(\frac{1}{t}\sum _{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}\left(a\right)x_{\tau }{x}_{\tau }^{\top }+{\Delta }_1+{\Delta }_3\right)}^{-1}\cdot \left[\left(\frac{1}{t}\sum _{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}\left(a\right)x_{\tau }{x}_{\tau }^{\top }\right){\theta }_a^{*}+{\Delta }_2\right] \\ ={\theta }_a^{*}-J_1^{\prime }+J_2^{\prime }, \end{gathered}$$

where

$${\mathit{J}}_1^{\prime }:={\left[\frac{1}{t}\sum _{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}(a){\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }+{\mathbf{\Delta }}_1+{\mathbf{\Delta }}_3\right]}^{-1}\left({\mathbf{\Delta }}_1+{\mathbf{\Delta }}_3\right){\mathit{\theta }}_a^{*},{\mathit{J}}_2^{\prime }:={\left[\frac{1}{t}\sum _{\tau \in \left[t\right]}{\pi }_{\tau }^{nd}(a){\mathbf{x}}_{\tau }{\mathbf{x}}_{\tau }^{\top }+{\mathbf{\Delta }}_1+{\mathbf{\Delta }}_3\right]}^{-1}{\mathbf{\Delta }}_2.$$

Under the events where both (D.1) and (D.2) hold, whenever

$$C \text{max}\left\{\frac{d+\text{log}t}{q_{t}t},\frac{\sqrt{\xi }}{d}\sqrt{\frac{d+\text{log}t}{q_{t}t}}\right\}\le \frac{{\lambda }_0}{4d}$$ (G.13)

and ${∥{\mathbf{\Delta }}_3∥}_2\le \frac{{\lambda }_0}{4d}$, we have

$${∥{\mathit{J}}_1^{\prime }∥}_2\le \frac{2d{R}_{\mathit{\theta }}}{{\lambda }_0}\left(C \text{max}\left\{\frac{d+\text{log}t}{q_{t}t},\frac{\sqrt{\xi }}{d}\sqrt{\frac{d+\text{log}t}{q_{t}t}}\right\}+{∥{\mathbf{\Delta }}_3∥}_2\right)$$

and

$${∥{\mathit{J}}_2^{\prime }∥}_2\le \frac{2CdR}{{\lambda }_0}\text{max}\left\{\frac{d+\text{log}t}{q_{t}t},\sqrt{\frac{\nu }{d}}\sqrt{\frac{d+\text{log}t}{q_{t}t}}\right\}.$$

(G.13) can be ensured by $t\ge C_1^{\prime }\text{max}\left\{\frac{d(d+\text{log}t)}{{\lambda }_0{q}_t},\frac{\xi (d+\text{log}t)}{{\lambda }_0^2{q}_t}\right\}$, where $C_1^{\prime }=\text{max}\left\{4C,16C^2\right\}$. Given these guarantees, we have with probability at least $1-4/t^2$,

$$∥{\tilde{\mathit{\theta }}}_a^{\left(t\right)}-{\mathit{\theta }}_a^{*}∥\le {∥{\mathit{J}}_1^{\prime }∥}_2+{∥{\mathit{J}}_2^{\prime }∥}_2\le \frac{2C\left(R+R_{\mathit{\theta }}\right)d}{{\lambda }_0}\text{max}\left\{\frac{d+\text{log}t}{q_{t}t},\frac{\sqrt{\xi }+\sqrt{\nu }}{\sqrt{d}}\sqrt{\frac{d+\text{log}t}{q_{t}t}}\right\}+\frac{2R_{\theta }d}{{\lambda }_0}{∥{\mathbf{\Delta }}_t\left(a\right)∥}_2.$$

Thus we conclude the proof.

G.7. Proof of Corollary F.1

Standard setting. Notice that $p_0^{\left(t\right)}$ is monotonically decreasing in $t$. Theorem F.1 indicates that, as long as $\forall t>T_0$,

$$p_0^{\left(t\right)}\ge C\prime \text{max}\left\{\frac{d(d+\text{log}t)}{{\lambda }_{0}t},\frac{\xi (d+\text{log}t)}{{\lambda }_0^{2}t}\right\},$$ (G.14)

then with probability at least $1-\frac{8}{t^2},\forall a\in \left\{\mathrm{0,1}\right\}$,

$${∥{\widehat{\mathit{\theta }}}_a^{\left(t\right)}-{\mathit{\theta }}_a^{*}∥}_2\le \frac{C\left(R+R_{\theta }\right)d}{{\lambda }_0}\text{max}\left\{\frac{d+\text{log}t}{t^{\frac{2}{3}}},\frac{\sqrt{\nu }+\sqrt{\xi }}{\sqrt{d}}\sqrt{\frac{d+\text{log}t}{t^{\frac{2}{3}}}}\right\}+\frac{2R_{\theta }d}{{\lambda }_0}{∥{\mathbf{\Delta }}_t\left(a\right)∥}_2$$ (G.15)

Plug it into Theorem 2.2, we have that with high probability,

$$\text{Regret}\left(T;{\pi }^{*}\right)\le 2R_{\theta }\cdot ⌈2d{T}^{2/3}⌉+\frac{2}{1-\rho }{I}_1^{\prime },$$

where

$$\begin{gathered} I_1 =\sum _{t=T_0+1}^T\left(p_0^{\left(t\right)}{R}_{\theta }+\underset{a}{\text{max}}{∥{\tilde{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2\right) \\ \le \sum _{t=T_0+1}^T{t}^{-\frac{1}{3}}{R}_{\theta }+\sum _{t=T_0}^{T-1}\left[\frac{C\left(R+R_{\theta }\right)d}{{\lambda }_0}\text{max}\left\{\frac{d+\text{log}t}{t^{\frac{2}{3}}},\frac{\sqrt{\nu }+\sqrt{\xi }}{\sqrt{d}}\sqrt{\frac{d+\text{log}t}{t^{\frac{2}{3}}}}\right\}+\underset{a}{\text{max}}\frac{2R_{\theta }d}{{\lambda }_0}{∥{\mathbf{\Delta }}_t\left(a\right)∥}_2\right] \\ \le 2R_{\theta }{T}^{\frac{2}{3}}+\frac{C\prime }{{\lambda }_0}(\sqrt{\nu }+\sqrt{\xi }+1)\left(R+R_{\theta }\right)\sqrt{d(d+\text{log}T)}{T}^{\frac{2}{3}}+\frac{2R_{\theta }d}{{\lambda }_0}\sum _{t\ge T_0}{∥{\mathbf{\Delta }}_t\left(a\right)∥}_2, \end{gathered}$$

for a universal constant $C\prime$, where the last inequality holds if in addition, $T\ge \left[d\right(d+\text{log}T){]}^{\frac{3}{2}}$.

The proof is concluded by combining the above requirement for $T$ as well as (G.14).

Clipped policy setting.

Similar to the standard setting, according to Theorem F.1, as long as $\forall t>T_0$,

$$p_0\ge C_1\text{max}\left\{\frac{d\left(d+\text{log}t\right)}{{\lambda }_{0}t},\frac{\xi \left(d+\text{log}t\right)}{{\lambda }_0^{2}t}\right\},$$ (G.16)

then with probability at least $1-\frac{8}{t^2},\forall a\in \left\{\mathrm{0,1}\right\}$,

$${∥{\tilde{\mathit{\theta }}}_a^{\left(t\right)}-{\mathit{\theta }}_a^{*}∥}_2\le \frac{C\left(R+R_{\theta }\right)d}{{\lambda }_0}\text{max}\left\{\frac{d+\text{log}t}{p_{0}t},\frac{\sqrt{\nu }+\sqrt{\xi }}{\sqrt{d}}\sqrt{\frac{d+\text{log}t}{p_{0}t}}\right\}+\frac{2R_{\theta }d}{{\lambda }_0}{∥{\Delta }_t\left(a\right)∥}_2$$ (G.17)

Plug it into Theorem 2.2, we have that with high probability,

$$\text{Regret}\left(T;{\bar{\pi }}^{*}\right)\le 2R_{\theta }\cdot ⌈2d{T}^{1/2}⌉+\frac{2\left(1-2p_0\right)}{1-\rho }{I}_2^{\prime },$$

where

$$\begin{gathered} I_2^{\prime }=\sum _{t=T_0+1}^T\underset{a}{\text{max}}{∥{\tilde{\mathit{\theta }}}_a^{(t-1)}-{\mathit{\theta }}_a^{*}∥}_2 \le \sum _{t=T_0}^{T-1}\left[\frac{C\left(R+R_{\theta }\right)d}{{\lambda }_0}\text{max}\left\{\frac{d+\text{log}t}{p_{0}t},\frac{\sqrt{\nu }+\sqrt{\xi }}{\sqrt{d}}\sqrt{\frac{d+\text{log}t}{p_{0}t}}\right\}+\underset{a}{\text{max}}\frac{2R_{\theta }d}{{\lambda }_0}{∥{\Delta }_t\left(a\right)∥}_2\right] \\ \le \frac{2C\left(R+R_{\theta }\right)\left(\sqrt{\nu }+\sqrt{\xi }+1\right)}{{\lambda }_0\sqrt{p_0}}\sqrt{d\left(d+\text{log}T\right)}\sqrt{T}+\frac{2R_{\theta }d}{{\lambda }_0}\sum _{t=T_0}^{T-1}\underset{a}{\text{max}}{∥{\Delta }_t\left(a\right)∥}_2, \end{gathered}$$

where the last inequality holds if in addition, $T\ge d(d+\text{log}T){\text{log}}^{2}T$.

The proof is concluded by plugging the above into the regret upper bound formula and combining the requirements for $T$.