Algorithm 1. DRL for MASS Autonomous Navigation Decision-making

● Input: Start sampling from random state $s_0$ and randomly select action. Sampling is terminated at $T$ cycles or the MASS collides. The resulting sample set is $S$. Each input in $S$ must be included: (1) Current states $s_t$, (2) action $a$, (3) return $r$, (4) the next state after the action $s_{t+1}$, and (5) the termination condition ● Output: weights parameter ${\omega }^{\ast }$ for DRL Require: $\omega$: a small positive number representing the allowed smallest convergence tolerance; $S$: the state set; $P(s^{\prime },r|s,a)$: the transition probability from current state and action to next state and reward; $\gamma$: the discount factor; 1: Initialize the optimal value function $Q(s)$, $\forall s\in S$ arbitrarily 2: For episode = 1, M do 3:    For t = 1, T do 4:    repeat 5:    $\omega \leftarrow 0$ 6:    for $s\in S$ do 7:      target $q\leftarrow Q(s)$ 8:      $Q(s)\leftarrow {\max }_a\left[r+\gamma {\displaystyle \sum s^{\prime }},rP(s^{\prime },r|s,a)Q(s)\right]$ 9:      $\omega \leftarrow \max (\omega ,|q-Q(s)|)$ 10:   until $\omega <0$ 11:   end for 12: end for 13: ${\pi }^{\ast }(s)\approx argma{x}_a\left[r+\gamma {\displaystyle \sum s^{\prime }},rP(s^{\prime },r|s,a)Q(s)\right]$