Algorithm 4 Approximate State Matching Q-learning Algorithm

Input: Environment $E:S=S_u;A=A_u;T=T_u;R=R_u$ Output: Learned Q table   1: Initialize Q table   2: while$i\le$ the number of iterations do   3:    Return s to the initial state   4:    while s is not terminal state do   5:        if s is not in Q table then   6:           Add s to the Q table and initialize s   7:        end if   8:        if s is a Guided State then   9:           Choose a from A using policy derived from $Q\left(s\right)$ 10:        else 11:           Select the similar set $S_{ene}$ of s using Algorithm 2 12:           $Q(s,a,temporary)=\frac{{\displaystyle \sum _{ϵ\in S_{ene}}}Q(ϵ,a)}{\parallel S_{ene}\parallel }$ for all $a\in A$ 13:           Choose a from A using policy derived from $Q(s,temporary)$ 14:           if $a\ne e$ then 15:               Select the similar set $S_{vp}\cap S_{hp}$ of s using Algorithm 3 16:               $Q(s,a,temporary)=\frac{{\displaystyle \sum _{(ϵ\in S_{vp}\cap S_{hp})\wedge Q(ϵ,a)\ge 0}}Q(ϵ,a)}{\parallel S_{vp}\cap S_{hp}\parallel }$ for all $a\in A$ 17:               Choose a from A using policy derived from $Q(s,temporary)$ 18:           end if 19:        end if 20:        Take action a, observe $s^{{}^{\prime }}=T_u(s,a)$ and $r=R_u(s,a,s^{\prime })$ 21:        $Q(s,a)\leftarrow Q(s,a)+\alpha [r+\gamma ma{x}_{a^{{}^{\prime }}}Q(s^{{}^{\prime }},a^{{}^{\prime }})-Q(s,a)]$ 22:        $s\leftarrow s^{{}^{\prime }}$ 23:    end while 24: end while 25: Return Q table