Algorithm 1 The hierarchical feedback learning control of network component i

Inputs:

The measured state $s_{i,t}$,

learning rate $\alpha$,

balanced factor $\beta$, and discount factor $\lambda$.

Initialize:

deep reinforcement learning models $W_{i,t}$

# Obtain next action and execute it.

$a_{i,t}\leftarrow {\pi }_{i,t}$

# Suppose $a_{i,t}$ is establishing the TCP flow for component j.

# Calculate reward function with Measured RTT.

$R_{s_{i,t},s_{i,t+1}}^{a_{i,t}}$ = $\beta \frac{S{T}_j^i}{R{T}_j^i\sqrt{Loss}}$

# Re-calculate reward function with the feedback reward.

$R_{s_{i,t},s_{i,t+1}}^{a_{i,t}}$ += $(1-\beta )r_j$

# Calculate TD-error

$L\left(W_{i,t}\right)=R_{s_{i,t},s_{i,t+1}}^{a_{i,t}}+\gamma {max}_{a}Q(s_{i,t+1},A_i,W_{i,t})-Q(s_{i,t},a_{i,t},W_{i,t})$

# Update $W_{i,t}$ to $W_{i,t+1}$

$W_{i,t+1}\leftarrow W_{i,t}-\alpha \frac{dL}{d{W}_{i,t}}{\left\{L\left(W_{i,t}\right)\right\}}^2$

$t\leftarrow t+1$