Algorithm 1 : Q-learning extended Kalman filter

1: ${\widehat{\mathit{x}}}_{\mathrm{P},0}^{\mathrm{B}}={\widehat{\mathit{x}}}_{\mathrm{P},0}^{\mathrm{E}}={\widehat{\mathit{x}}}_{\mathrm{P},0}$, ${\mathit{P}}_{\mathrm{P},0}^{\mathrm{B}}={\mathit{P}}_{\mathrm{P},0}^{\mathrm{E}}={\mathit{P}}_{\mathrm{P},0}$                                                                    $⊳$ Initialization

2: k $\leftarrow$ 0

3: $\mathsf{\Theta }\leftarrow \mathbf{0}$

4: for each period, do

5:      for all $a\in \mathit{A}$, do

6:          $C^{\mathrm{B}}\leftarrow \mathbf{0}$, $C^{\mathrm{E}}\leftarrow \mathbf{0}$

7:          for $t=1,2,\cdots ,T$, do

8:              $k\leftarrow k+1$

9:              $\left[{\widehat{\mathit{x}}}_{\mathrm{P},\mathrm{k}}^{\mathrm{B}},{\mathit{P}}_{\mathrm{P},\mathrm{k}}^{\mathrm{B}},{\tilde{\mathit{y}}}_{\mathrm{P},\mathrm{k}}^{\mathrm{B}}\right]\leftarrow \mathrm{EKF}\left({\widehat{\mathit{x}}}_{\mathrm{P},\mathrm{k}-1}^{\mathrm{B}},{\mathit{P}}_{\mathrm{P},\mathrm{k}-1}^{\mathrm{B}},{\mathit{y}}_{\mathrm{P},\mathrm{k}},{\mathit{Q}}_{\mathrm{P},\mathrm{k}},{\mathit{R}}_{\mathrm{P},\mathrm{k}}\right)$                  $⊳$ Benchmark filter

10:            $\left[{\widehat{\mathit{x}}}_{\mathrm{P},\mathrm{k}}^{\mathrm{E}},{\mathit{P}}_{\mathrm{P},\mathrm{k}}^{\mathrm{E}},{\tilde{\mathit{y}}}_{\mathrm{P},\mathrm{k}}^{\mathrm{E}}\right]\leftarrow \mathrm{EKF}\left({\widehat{\mathit{x}}}_{\mathrm{P},\mathrm{k}-1}^{\mathrm{E}},{\mathit{P}}_{\mathrm{P},\mathrm{k}-1}^{\mathrm{E}},{\mathit{y}}_{\mathrm{P},\mathrm{k}},{\widehat{\mathit{Q}}}_{\mathrm{P},\mathrm{k}}^{\left(s,a\right)},{\mathit{R}}_{\mathrm{P},\mathrm{k}}\right)$                $⊳$ Exploring filter

11:            $\left[{\widehat{\mathit{x}}}_{\mathrm{P},\mathrm{k}},{\mathit{P}}_{\mathrm{P},\mathrm{k}},{\tilde{\mathit{y}}}_{\mathrm{P},\mathrm{k}}\right]\leftarrow \mathrm{EKF}\left({\widehat{\mathit{x}}}_{\mathrm{P},\mathrm{k}-1},{\mathit{P}}_{\mathrm{P},\mathrm{k}-1},{\mathit{y}}_{\mathrm{P},\mathrm{k}},{\widehat{\mathit{Q}}}_{\mathrm{P},\mathrm{k}}^{\left(s,a_{\max }\right)},{\mathit{R}}_{\mathrm{P},\mathrm{k}}\right)$            $⊳$ Main filter

12:              $C^{\mathrm{B}}\leftarrow C^{\mathrm{B}}+\frac{1}{T-1}\left[{\left({\tilde{\mathit{y}}}_{\mathrm{k}}^B\right)}^{\mathrm{T}}{\mathit{R}}_{\mathrm{k}}^{-1}{\tilde{\mathit{y}}}_{\mathrm{k}}^{\mathrm{B}}-C^{\mathrm{B}}\right]$

13:              $C^{\mathrm{E}}\leftarrow C^{\mathrm{E}}+\frac{1}{T-1}\left[{\left({\tilde{\mathit{y}}}_{\mathrm{k}}^{\mathrm{E}}\right)}^{\mathrm{T}}{\mathit{R}}_{\mathrm{k}}^{-1}{\tilde{\mathit{y}}}_{\mathrm{k}}^{\mathrm{E}}-C^{\mathrm{E}}\right]$

14:            end for

15:            $R\left(s,a\right)\leftarrow C^{\mathrm{B}}-C^{\mathrm{E}}$                                                                            $⊳$ Calculation of Reward

16:            $\mathsf{\Theta }\leftarrow \mathsf{\Theta }+\alpha \left[R\left(s,a\right)+\gamma \underset{a^{\prime }}{\max }\left({\mathsf{\Phi }}^{\mathrm{T}}\left(s’,a’\right)\mathsf{\Theta }\right)-{\mathsf{\Phi }}^{\mathrm{T}}\left(s,a\right)\mathsf{\Theta }\right]\mathsf{\Phi }\left(s,a\right)$     $⊳$ Update of weight

17:            ${\widehat{\mathbf{x}}}_{\mathrm{P},\mathrm{k}}^{\mathrm{E}}\leftarrow {\widehat{\mathbf{x}}}_{\mathrm{P},\mathrm{k}}^{\mathrm{B}}$, ${\mathbf{P}}_{\mathrm{P},\mathrm{k}}^{\mathrm{E}}\leftarrow {\mathbf{P}}_{\mathrm{P},\mathrm{k}}^{\mathrm{B}}$                                                                   $⊳$ Resetting of exploring filter

18:        end for

19:        $a_{\max }\leftarrow \arg \underset{{\mathrm{a}}^{\prime }}{\max }\left({\mathsf{\Phi }}^{\mathrm{T}}\left(s,a\right)\mathsf{\Theta }\right)$                                                             $⊳$ Selection of the best action

20: end for

21: return $\left\{{\widehat{\mathit{x}}}_{\mathrm{P},\mathrm{k}}\right\}$ and $\left\{{\mathit{P}}_{\mathrm{P},\mathrm{k}}\right\}$                                                                                $⊳$ Result of state estimation