Algorithm 1 DQN-CNN with Experience Replay

Initialize the memory stored in the experience of replay D, the number of iterations M

Randomly initialize the Q-value function

for iteration number = 1, M do

randomly initialize the first action $a_1$

initialize the first state $s_1$

for $t$ = 1, T do

if the probability is ϵ, select a random action $a_t$

otherwise select $a_t={}_{a^{\prime }}{}^{max}\mathrm{Q}^{*}\left(s_{\mathrm{t}},a;\mathsf{\theta }\right)$

input $a_t$, $s_t$ into $CNN$, get classification ${\mathrm{c}}_t=CNN\left(a_t,s_t\right)$

if $c_t==\mathrm{label}$ then

reward $r_t=1$

if $t$ < T then

reward $r_t=2$

else $r_t=0$

execute $a_t$, get $r_t$ and next state ${\mathrm{s}}_{t+1}$

stored ($s_t,a_t,r_t,s_{t+1}$) in D

using a gradient descending of random small batches to get sample ($s_j,a_j,r_j,s_{j+1}$) $$y_{\mathrm{j}}=\{\begin{array}{c}{r}_j{s}_{j+1}\ne \mathrm{t}\mathrm{e}\mathrm{r}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}\\ r_j+\gamma {}_{a^{\prime }}{}^{max}\mathrm{Q}\left(s_{j+1},a^{\prime };\theta \right)s_{j+1}=\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}\end{array}$$

Calculate the gradient of ${\left({\mathrm{y}}_j-\mathrm{Q}\left(s_j,a_i;\theta \right)\right)}^2$ to update $\theta$

end if

end for