Algorithm 3 Contextual bandit for clinical trial optimization

1:for $m=1,2,\dots ,2^D$do              ▹ initialize all context-free bandits 2:    $MAB\left(m\right)\leftarrow$initialize context-free bandit() 3:for$i=1,2,\dots ,$do 4:    $X_i\leftarrow$observe context(patient i) 5:    $MAB★=MAB\left(X_i\right)$             ▹ bandit associated with context $X_i$ 6:    $U_i\leftarrow$select treatment($MAB★$)   ▹ select a treatment based on priors in $MAB★$ 7:    Apply $U_i$ and observe $C_i$ 8:    update prior($U_i$, $MAB★$)          ▹ update the prior of $U_i$ in $MAB★$