Table 2.

The structure of Q-table

State	Action
	$a_{11}$	$a_{12}$	$a_{21}$	$\cdots$	$a_{D2}$
$(0,0)$	$Q(s_{00},a_{11})$	$Q(s_{00},a_{12})$	$Q(s_{00},a_{21})$	$Q(s_{00},\cdots )$	$Q(s_{00},a_{D1})$
$(1,0)$	$Q(s_{10},a_{11})$	$Q(s_{10},a_{12})$	$Q(s_{10},a_{21})$	$Q(s_{10},\cdots )$	$Q(s_{10},a_{D1})$
$\cdots$	$Q(\cdots ,a_{11})$	$Q(\cdots ,a_{12})$	$Q(\cdots ,a_{21})$	$Q(\cdots ,\cdots )$	$Q(\cdots ,a_{D2})$
$(M,0)$	$Q(s_{M0},a_{11})$	$Q(s_{M0},a_{12})$	$Q(s_{M0},a_{21})$	$Q(s_{M0},\cdots )$	$Q(s_{M0},a_{D2})$
$(M,1)$	$Q(s_{M0},a_{11})$	$Q(s_{M0},a_{12})$	$Q(s_{M0},a_{21})$	$Q(s_{M0},\cdots )$	$Q(s_{M1},a_{D2})$
$\cdots$	$Q(\cdots ,a_{11})$	$Q(\cdots ,a_{12})$	$Q(\cdots ,a_{21})$	$Q(\cdots ,\cdots )$	$Q(\cdots ,a_{D2})$
$(M,N)$	$Q(s_{\mathit{MN}},a_{11})$	$Q(s_{\mathit{MN}},a_{12})$	$Q(s_{\mathit{MN}},a_{21})$	$Q(s_{\mathit{MN}},\cdots )$	$Q(s_{\mathit{MN}},a_{D2})$