Algorithm 2 Upper confidence bound (UCB)

1:for $i=1,2,\dots ,K$do 2:    Apply treatment i            ▹ apply each treatment once 3:for $i=K+1,K+2,\dots ,$do 4:    for $u\in \{1,\dots ,K\}$ do 5:        Estimate $\widehat{{\theta }_u}=\left[\frac{{\alpha }_u}{{\alpha }_u+{\beta }_u}\right]$          ▹ estimate mean rewards 6:        $n_{u,i}\leftarrow$ # of times treatment u has been applied so far 7:    $U_i={arg\; max}_u\left[\widehat{{\theta }_u}+\sqrt{\frac{\ln i}{n_{u,i}}}\right]$        ▹ select and apply action 8:    Apply $U_i$ and observe $C_i$ 9:    $({\alpha }_{U_i},{\beta }_{U_i})\leftarrow ({\alpha }_{U_i},{\beta }_{U_i})+(C_i,1-C_i)$     ▹ update distribution