Algorithm 1 Pseudocode of the DDPG-HP Algorithm.

Input: State observation s, sampling time Ts = 0.02 s, total duration Tf = 20 s Random initialization: Critic network Q(s, a|θc) and Target-Critic network Q′(s, a|θc′) Actor network μ(s |θμ) and Target-Actor network μ′(s|θμ′) Experience replay pool D with capacity = 106, minibatch = 128 OU noise: initial variance = 1.2, variance decay rate = 1 × 10−6 Training loop: For episode = 1 to Nmax do:     Initialize environment state s = [xcar, ycar, vcar, …, xp, yp, vp]     For t = 1 to ceil (Tf/Ts) do:         Select action at based on current policy and OU noise         Execute action at = [throttle/brake, steeR]T         Obtain reward rt and next state st+1         Store experience (st, at, rt, st+1) into replay pool D         If number of samples in D ≥ 128 then:             Randomly sample 128 experiences as minibatch (si, ai, ri, si+1)             Compute TD error δi;             Update Critic network parameters: ${\theta }^c\leftarrow {\theta }^c+{\alpha }_c{\mathsf{\nabla }}_{{\theta }^c}L({\theta }^c)$             Update Actor network parameters: ${\theta }^{\mu }\leftarrow {\theta }^{\mu }+{\alpha }_a{\mathsf{\nabla }}_{{\theta }^{\mu }}J({\theta }^{\mu })$             Soft update target networks: $$\begin{gathered} {\theta }^{c^{\prime }}\leftarrow \tau {\theta }^c+(1-\tau ){\theta }^{c^{\prime }} \\ {\theta }^{{\mu }^{\prime }}\leftarrow \tau {\theta }^{\mu }+(1-\tau ){\theta }^{{\mu }^{\prime }} \end{gathered}$$             End         End         If convergence criterion is met then             Stop training         End End Return: Trained Actor network μ(s|θμ)