| Algorithm 1 DDPG-PID Auto-Tuning Control Strategy | ||
|
Initialize Simulink model DDPG-PID Auto-Tuning. Define a reinforcement learning environment, including a state space and an action space, where the action space consists of the gain parameters , , and of the PID controller. Initialize two Critic networks and , Actor network , and their corresponding target networks , , and . Initialize the prioritized experience replay buffer D. for episode = 1, M performs the following steps Initialize the random process for action exploration N Receive initial observation state for t = 1, T performs the following steps Select the action according to the current policy:
Execute action , which means using the current PID gain , , and control model. Observe the reward , next state , and the termination signal is_done. Store experience in the replay buffer D. Calculate the TD error and priority , and sample N experiences according to the prioritized sampling probability using Equations (21)–(24). Calculate the two target values and using Equations (28) and (29). Update Critic networks and according to Equations (30) and (31). Update the Actor network according to the deterministic policy gradient in Equation (27). Update the priorities of the sampled experiences using the new TD errors according to Equations (19) and (20). Soft-update the target Actor and the target Critic networks and . Set . if is_done = 1, break. End the current round and continue to the next round. After training, simulation verification is performed. The trained agent was used for simulation to verify the effectiveness of the PID parameter self-tuning controller. During the verification process, the agent optimized the control system response by adjusting the PID gains , , and . end end |