Skip to main content
. 2026 Nov 4;26(21):6982. doi: 10.3390/s26216982
Algorithm 1 DDPG-PID Auto-Tuning Control Strategy
Initialize Simulink model DDPG-PID Auto-Tuning.
Define a reinforcement learning environment, including a state space and an action space, where the action space consists of the gain parameters Kp, Ki, and Kd of the PID controller.
Initialize two Critic networks ϕ1 and ϕ2, Actor network μ(sθμ), and their corresponding target networks μ′, ϕ1′, and ϕ2′.
Initialize the prioritized experience replay buffer D.
for episode = 1, M performs the following steps
  Initialize the random process for action exploration N
  Receive initial observation state s1
  for t = 1, T performs the following steps
  Select the action according to the current policy:
at=μ(stθμ)+Nt (14)
  where at represents PID parameters Kp, Ki, and Kd.
  Execute action at, which means using the current PID gain Kp, Ki, and Kd control model.
  Observe the reward rt, next state st+1, and the termination signal is_done.
  Store experience (st,at,rt,st+1,is_done) in the replay buffer D.
  Calculate the TD error δt and priority pt, and sample N experiences according to the prioritized sampling probability p(i) using Equations (21)–(24).
  Calculate the two target values ytcritic1 and ytcritic2 using Equations (28) and (29).
  Update Critic networks ϕ1 and ϕ2 according to Equations (30) and (31).
  Update the Actor network according to the deterministic policy gradient in Equation (27).
  Update the priorities of the sampled experiences using the new TD errors according to Equations (19) and (20).
  Soft-update the target Actor μ′ and the target Critic networks ϕ1′ and ϕ2′.
  Set st←st+1.
  if is_done = 1, break.
  End the current round and continue to the next round.
  After training, simulation verification is performed.
  The trained agent was used for simulation to verify the effectiveness of the PID parameter self-tuning controller. During the verification process, the agent optimized the control system response by adjusting the PID gains Kp, Ki, and Kd.
 end
end