Adaptive PID-based deep reinforcement learning for load frequency control in islanded microgrids with heterogeneous resources and energy storage

Abstract

Load frequency control (LFC) is an essential measure in maintaining stability in power systems in islanded microgrids that include heterogeneous generation sources and energy storage systems. Island microgrid systems (IMGs) that integrate renewable energy sources widely use proportional–integral–derivative (PID) controllers for LFC. However, the overall control performance is highly sensitive to the accurate tuning of PID controller parameters. To address this problem, this paper proposes an adaptive PID tuning approach that studies the individual evaluations of the Deep Deterministic Policy Gradient (DDPG) and Twin Delayed Deep Deterministic Policy Gradient (TD3) reinforcement learning algorithms. The proposed reinforcement learning (RL)-based adaptive PID tuning approach is used to adaptively regulate PID gains under the inherent uncertainties of the IMGs environment. The proposed RL-PID controller approach employs an agent that is trained offline through repeated interactions with the IMGs model, where a suitable reward function guides the learning process toward an optimal control policy. Once trained, the agent is implemented online to continuously update the PID gains for coordinated control of the Wind Turbine Generator (WTG), Solar Photovoltaic (SPV), Fuel Cell (FC) units, Electric Vehicle (EV), and Biogas Turbine Generator (BTG), ensuring effective load demand tracking. Simulations of an IMGs demonstrate that both DDPG-based PID controllers and TD3-based PID controllers outperform conventional PID controllers in dynamic response, settling time, and robustness to disturbances. Furthermore, the TD3-PID shows better stability and reduced oscillations compared to the DDPG-PID, which can be explained by the fact that it improves the policy update mechanism, leading to more effective adjustments in response to system changes. Although studied PID tuning approaches incorporate intelligent mechanisms for gain adjustment, the results indicate that the RL-based adaptive PID controller provides improved overall performance.

Introduction

Due to the growing demand, several unconventional sources have been integrated into power systems, increasing the complexity and unpredictability of power grids¹.Renewable energy sources offer notable environmental and economic benefits to modern power systems but also present operational challenges. integration with conventional generators, stable voltage and frequency regulation, and advanced control strategies for both grid-connected and islanded modes are essential considerations in microgrids². Currently, distributed generation accounts for a relatively small share of total power production, but its role in power systems is rapidly expanding³. Variations in wind speed and solar irradiance lead to significant fluctuations in power output and system frequency when combining diesel, solar, and wind generation. These fluctuations, along with reduced system inertia, make it difficult to maintain frequency stability during disturbances⁴. A microgrid is defined as a small-scale electrical grid with defined boundaries that can function as a single controllable entity. The microgrids, which can operate both in grid-connected and islanded modes, are increasingly important in smart cities for integrating distributed energy resources^1,5,6. When disconnected from the main grid the connected is called islanded microgrids (IMGs) must independently regulate voltage, frequency, and system stability to serve local loads⁶. However, low inertia, high renewable penetration, and load uncertainty pose ongoing frequency stability challenges in IMGs, making load frequency control (LFC) a key area of extensive research⁴.

LFCs have been extensively investigated to ensure reliable and stable microgrid operation. Conventional and advanced LFC approaches for microgrids and smart grids include classical, robust, and intelligent approaches^7,8. Early studies of the LFC on the IMGs focused on rule-based and fuzzy controllers, often enhanced with heuristic tuning methods such as particle swarm optimization (PSO) to improve dynamic performance^9,10. Other studies proposed constrained fuzzy-receding horizon controllers for renewable integration¹¹.More recently, metaheuristic algorithms like Coati Optimization Algorithm (COA)¹², grasshopper optimization (GOA)², whale optimization algorithm (WOA)¹³, and moth-flame optimization (MOFO)¹⁴ have been used to tune PID controllers. The COA has been applied to improve PID‑based controllers for robust LFC in urban IMG systems, demonstrating improved performance and robustness compared to other metaheuristic and classical approaches¹⁵. These heuristic algorithms have demonstrated their capacity and resilience in the enhanced LFC in a microgrid. In addition, the conventional PID controller parameters are often determined using trial-and-error tuning criteria, although such approaches may not guarantee optimal performance under varying operating conditions.

Over the past few decades, control strategies have explored various intelligent and robust approaches for managing nonlinear and active system dynamics^16–18. For example, the sliding-mode control strategy has been used on dual-inverter photovoltaic systems to make them more robust and better at rejecting outside interference¹⁶. Although the intelligent control strategies have shown considerable advantages, especially in system scenarios subject to multiple disturbance signals, they mostly depend on model-based formulations, which include some limitations in real-time implementation when used in dynamic and unpredictable traffic obstacles where conditions can change rapidly, such as sudden changes or unexpected obstacles in traffic. In contrast, the proposed adaptive controller approach motivations on a data-driven RL framework for adaptive PID tuning, enabling real-time optimization and improving adaptability for LFC in the standalone microgrid systems.

However, most of these techniques depend on offline tuning for the PID controller and lack real-time adaptability under rapidly changing conditions. Power system applications increasingly utilize Machine Learning (ML) and Deep Learning (DL) approaches due to their advanced data analysis and predictive capabilities¹⁹.These techniques have demonstrated significant effectiveness in complex tasks such as power demand forecasting, frequency analysis, and system control. Reinforcement learning (RL) is a branch of ML that focuses on continuous interaction trained agent with environment.

Nowadays microgrids are increasingly applying RL architectures to manage the challenges posed by the uncertainties introduced by renewable sources such as solar and wind²⁰. Different RL architectures have been explored to address limitations of the conventional LFC methods^21–27. A deep reinforcement learning (DRL)-fuzzy controller is developed to manage the LFC in a hybrid microgrid²¹. In the study²², an RL-based distributed energy management framework is proposed, and in²⁸, a DRL-based approach was developed for LFC to maintain system frequency stability and power balance. In the same direction, a graph-network-based DRL framework is applied for coordinated frequency regulation²³ , and the DRL-based secondary control is designed for IMGs²⁴ , With flexible demand-side management, a DRL controller is integrated²⁵,while DRL-based frequency control is proposed for AC microgrids²⁶, and DRL with model predictive secondary frequency control is combined for coordinated photovoltaics (PVs), energy storage system (ESS), and electric vehicles (EVs) operation in PV–ESS–EV microgrids, demonstrating improved frequency regulation under renewable and EV-induced uncertainties²⁷.

Although DRL approaches make systems more flexible and stronger, there are still problems with training complexity, convergence speed, and scalability, which can hinder their practical implementation in real-time applications, particularly in environments with high variability in renewable energy sources and demand patterns²⁵.This paper proposes that we need a computationally efficient, adaptive LFC approach that can keep the frequency deviations of the system within baseline limits even when there is a lot of renewable energy entering the IMGs. RL-based frequency regulation is a suitable fit for microgrids containing renewable energy because it doesn’t rely on models and can change over time.

Traditional LFC approaches don’t work as well when there are nonlinearities and power swings. In RL, the reward function usually penalises big changes in frequencies, oscillations, overshoots, and extended settlement times. This guides the system toward stable and efficient performance⁵.

Deep Deterministic Policy Gradient (DDPG) is a well-known DRL technique that uses Q-learning principles to merge policy-based and value-based approaches through the Actor–Critic architecture²⁹. DDPG has problems like overestimation bias, being sensitive to hyperparameters, and being unstable during training, especially in very random environments. The Twin Delayed Deep Deterministic Policy Gradient (TD3) is a better version of DDPG that fixes these problems by making three important changes: twin critic networks to reduce overestimation bias, delayed policy updates to stabilize learning, and target policy smoothing to make it more robust^30,31.

Two RL-based PID tuning approaches, TD3–PID and DDPG–PID, are suggested and implemented for LFC in an IMG. The proposed LFC frameworks are rigorously assessed regarding frequency control performance, convergence behavior, resilience to disturbances, and sensitivity to variations in system parameters, thereby elucidating the relative advantages and disadvantages of each LFC approach in IMGs. Here are the main contributions of this work:

• A complete mathematical model of the IMG is created that includes different types of generating sources, such as renewable and conventional distributed generators (DGs) and energy storage devices. This model has the transfer functions of the electrical and control subsystems, which makes it a good and accurate platform for LFC research.

• The suggested RL-PID controllers use RL algorithms, and the DDPG and TD3 RL algorithms are used to automatically adjust the gains of the PID controllers. This RL-based PID control framework lets you change parameters in real time, making the system more flexible and stronger when the load and generating conditions change.

• The proposed adaptive-RL-PID controllers utilize for LFC of the IMG to successfully reduce frequency deviations and control effort when there are disruptions and uncertainties in renewable generation.

• A comprehensive comparison examination of the DDPG–PID and TD3–PID controllers is performed utilizing MATLAB-Simulink simulation environments. The results indicate that both RL-based PID controllers exist, with the PID controller utilizing TD3 surpassing the DDPG controller in transient response, settling time, and robustness.

• The proposed RL-PID controllers were assessed across different scenarios and conditions. These selected operations demonstrate effective and adaptable solutions for LFC in the IMGs featuring varied energy generation and storage systems.

This paper is structured as follows: Section II talks about how the IMG is built mathematically. Section "Simulation Implementation Block Diagram"provides a comprehensive explanation of the RL algorithms, including the TD3 and DDPG algorithms. Section IV presents the simulation results, while Section V compares the proposed approach with recent studies on RL-based LFC in power systems. Finally, Section VI concludes the paper.

Mathematical modelling of the islanded microgrid

An IMG typically consists of the DGs, an ESS, consumer load demand, controllers, and power converters⁶. DGs, which can be conventional or renewable and generally have low ratings, produce variable output dependent on location and weather. Interfacing DGs and ESSs through power electronic converters enhance system reliability, while residential loads introduce stochastic demand variations arising from household consumption patterns. Figure 1 shows the studied schematic of the microgrid, featuring DGs such as wind turbine generators (WTG), solar photovoltaic (SPV) panels, diesel engine generators (DEG), biogas turbine generators (BGTG) and fuel cells (FC), as well as ESSs like battery energy storage systems (BESS), flywheel energy storage systems (FESS), and EV. Generated power meets load demand, while excess energy is stored in ESS. Power converters ensure appropriate energy delivery.

Fig. 1.

General schematic of the studied IMG along with DG- units and energy resources via DC/AC interface inverters.

The following section provides the generalized transfer function and mathematical modeling of DGs and ESS.

Diesel engine generator (DEG) model

The DEG serves as a compact and efficient power generation unit capable of rapid startup and responsive output control. It ensures a continuous and high-quality power supply to critical loads in the IMG. The operating principle of the DEG has been explained in principle³², emphasizing its dynamic response characteristics under varying load conditions. The dynamic behavior of DEG can be represented by a first-order transfer function¹⁰, as given in (1):

1

In (1) and represent the governor gain and time constant, respectively, whereas and corresponds to the diesel generator gain and time constant, respectively. Figure 2 shows the mathematical representation of distributed energy generator.

Fig. 2.

Mathematical representation of a DEG.

As illustrated in Fig. 2, and represent the frequency deviation and the command signal generated by the low pass filter output controller, respectively. The term indicates the governor valve speed control coefficient of the DEG, commonly represented by . The variation and the ramp-rate limits of the output can be denoted by , and , respectively. The represents the variation in the power output of the DEG. A balance between generation and demand is reached, which is represented by , and no control measure is needed.

when is positive, it means that the demand is more than the generated power and the opposite is true. Further information on the DEG model is discussed in^10,32.

Solar photovoltaic (SPV) model

The SPV panel transforms solar energy into electrical energy. The generated output power is contingent upon the surface temperature and the intensity of radiation incident on the panel³³.

A first-order transfer function can represent the dynamic behavior of the solar PV system, as shown in Eq. (2):

2

where and represent the gain and time constant of the solar PV system, respectively. Figure 3 shows the mathematical representations of a practical PV model.

Fig. 3.

Mathematical representations of a practical PV model.

Wind turbine generator (WTG) model

The WTG generates electrical power by harnessing wind speed^33,34.The dynamic response of the WTG is represented by a first-order transfer function, as shown in (3):

3

The gain and the time constant of WTG are represented by the variables and _, respectively, Fig. 4 presents the mathematical formulation for WTG system.

Fig. 4.

Mathematical formulations for a practical WTG system.

Biogas turbine generator (BGTG) model

Biogas is made from biodegradable trash and animal waste³⁵. Electricity is generated from the biogas produced¹⁴. The transfer function of BGTG can be represented as follows:

4

In (4), , , , , , and refer to lead time, lag time, actuator valve delay, biogas delay, and discharge time constant of the BGTG, respectively³⁶.

Fuel cell (FC) model

The FC-units can effectively produce electricity, heat, and water with hydrogen and air as fuel. The development of fuel cells with ionic membranes improves system performance and facilitates the use of hydrogen energy. This work integrates frequency controllers with stationary energy storage systems to alleviate instability resulting from the intermittency of (WTG and SPV) sources.

The FC block is succeeded by an inverter and a connected device. The transfer function of the FC can be expressed as a first-order lag, as shown in Eq. (5)^33,37:

5

where and stand for the approximated model of FC gain and time constant, respectively.

Battery energy storage (BES) model

The mathematical model for ESS is derived from the BES model, which captures the dynamic behavior of the battery during charging and discharging cycles³³. The first-order transfer function can be represented as in (6)³⁸.

6

where and stand for the approximated model of BESS gain and time constant, respectively. The operational constraints of the battery are:

7

where is the state-of-charge deviation from the nominal value, constrained by . The SOC range reflects the allowable fluctuation around the nominal operating point, ensuring that the battery neither exceeds its upper charge limit nor falls below its minimum discharge threshold, and is ESS power injection (positive for charging, negative for discharging).

Flywheel energy storage (FESS) model

The FESS stores excess energy as kinetic energy during off-peak periods and discharges it during peak demand intervals³³. The first-order transfer function can be represented in Eq. (8).

8

where and denotes for approximated model of the FESS gain and time constant, respectively.

Electric vehicle (EV) model

The EVs can be used as storage systems to supply electricity and maintain system stability during emergencies³⁴. The first-order transfer function for EV is provided in Eq. (9).

9

where and stand for the approximated model of EV gain and time constant, respectively.

Dynamics of the generator model

The dynamics of a synchronous generator in an IMG can be approximated using a first-order transfer function³⁹. The generator transfer function can be written as:

10

where is the inertia constant and is the damping constant. The generator is governed by the swing equation^36,39.

11

Here, is the power that is generated, and is the power that is needed. This simplified generator model accurately represents the fundamental dynamics necessary for LFC analysis and controller design in IMGs⁷.

Block diagram of IMG

The studded IMG system has two main control layers: the primary control loop and the secondary control loop. Figure 5 depicts a block diagram of the studied IMG-based RL-PID controller shown in Fig. 1. The studied block diagram of the IMG system based on the RL-PID controller shows the interaction between the microgrid model, RL agent, PID control structure, and feedback signals ( and its derived features).

Fig. 5.

Block diagram for the studied IMG-based RL-based adaptive PID control integrated with controllable DGs, ESS, and interconnected loads.

The ESS is the main control loop, which quickly adjusts the frequency and voltage in a small area. The controlled DGs are the second control loop, which makes sure that the frequency is restored and power is shared across the whole system. Uncontrollable DGs, such as wind and solar units, are treated as disturbances due to their intermittent power output. The mathematical models of the DGs and ESS are represented in Eqs. (1, 2, 3, 4, 5, 6, 8, and 10). Table 1 shows the first-order transfer functions and nominal values of each DG and ESS used in the model. Renewable sources like WTG and PV units provide stochastic, weather-dependent outputs, while controllable DGs (DEG, BGTG, and FC units) regulate power to balance generation and demand.

Table 1.

Equivalent first order transfer functions of DGUs and ESS^10,36,40.

DGUs/ESS	Equation number	Parameters	Nominal transfer function
DEG	1	TDEG = 0.08, KDEG = 1 TT = 0.4, KT = 1
SPV	2	Tspv = 0.8, Kspv = 1
WTG	3	TWT = 1.5, KWT = 1
BGTG	4	Xd = 0.6, Yd = 1, dv = 0.05, Tcri = 0.01, Tbgd = 0.23,
FC	5	Tfc = 0.26, Kfc = 1
BESS	6	Tbes = 0.1, Kbes = 1
FESS	8	Tfess = 0.1, Kfess = 1
EV	9	Tev = 0.9, Kev = 0.7
DG	10	D = 0.0015, M = 0.1667 R = 3

The proposed RL-adaptive PID controller employs an actor–critic structure to dynamically tune PID parameters based on environment states and reward feedback. The RL-adaptive PID controller regulates the dynamics of controllable distributed generation units (DGUs), modeled through transfer functions representing BGTG units, turbine dynamics, and other subsystems. The system is also connected to ESS and EV loads, which are both modeled as first-order dynamic models.

The ESSs offer temporary backup assistance, with primary control governed by adjustments in ESS power and secondary control achieved through the regulation of controllable DGUs. The primary control loop depends on the fast dynamic response of the interconnected load-side units, namely the BES, FESS, and EV models, represented by first-order transfer functions with gains , ,and . These sources provide immediate stabilization of frequency and voltage following disturbances and operate independently of the PID controller. In addition, the primary control loop is typically decentralized and does not depend on communication infrastructure, thereby enhancing robustness and reliability.

On the other hand, the secondary control loop includes the RL-based supervisory layer integrated with the PID controller. This loop operates more slowly and functions on a system-wide scale, coordinating efforts across DG units and interconnected loads.

Fundamentals of RL- algorithms

DDPG algorithm

The DDPG algorithm is composed of two main components: the actor neural network and the critic neural network, both implemented as deep neural networks. Figure 6 shows the general topology of these networks; in actor and critic topology, is the system state, denotes the action, and corresponds to the Q-function of the critic network⁴¹.

Fig. 6.

Actor and critic networks.

The actor network correlates system states with continuous control actions to optimize long-term rewards, while the critic assesses these actions using Q-value estimation and offers feedback for policy enhancement. This actor–critic framework adeptly manages continuous action spaces, rendering it appropriate for adaptive PID tuning in the load–frequency regulation of IMGs.

The critic network estimates the value functions of states or state–action pairs in the environment. The value function represents the expected cumulative reward for starting in a given state and performing a specific action. Q-learning is a common way to find the value function. In this approaches, a deep neural network approximates the Q-function by linking state-action pairs to the expected future reward⁴². In this context, the critic evaluates the actions proposed by the actor network and provides feedback on their effectiveness in achieving the desired objectives or maximizing rewards. Table 2 summarizes the standard notations used in the DDPG algorithms.

Table 2.

Description of primary notations in DDPG algorithm.

Notation	RL component
	State of the environment at time step
	State of the environment for next state at time step
	Action executed at time step
	Action executed at the next time step
	Reward obtained in state
	Q-function (critic network)
	Policy mapping from states to actions (actor network)
	Temporal-difference (TD) target
	Set of all admissible actions
	Set of all possible environment states

The DDPG algorithm is an actor–critic method that simultaneously learns optimal estimates of both the policy and value functions, combining the advantages of policy-based and value-based approaches⁴³.

From a theoretical perspective, integration of action constraints can be analyzed through the lens of constrained Markov decision processes, where the constraint of feasibility and safety is added at the cost of strict Bellman optimality to obtain solutions that are optimal in the constrained admissible policy space. The policy is updated using the Q-function learned from off-policy data in the Bellman equation⁴⁴.

If the optimal Q-function is known, the optimal action is obtained as:

12

where is the action that maximizes the expected cumulative reward for a given state .

The Bellman equation represents an initial step in the learning process to find the best Q-function ⁴⁵:

13

Q-learning approximates this using the TD update:

14

And,

where is the TD error, is discount factor , ,and is the learning rate. For continuous state and action spaces, neural networks are used to approximate Q-values and policies. The critic network , parameterized by wight , and the actor network , parameterized by , are modified in (15) as follows:

15

Here, the target is defined as:

16

where and are the target critic and target actor networks, respectively. Using a mini-batch of samples, the target for each sample is:

17

The loss function for the critic network is the mean square error (MSE) between the target and predicted Q-values⁴⁶:

18

In the same context, the actor network is updated by the policy gradient⁴⁷:

19

Finally, soft updates (Polyak averaging) are applied to the target networks⁴⁶:

20

where is the soft update coefficient τ ∈ [0, 1], τ <  < 1. This framework enables stable training of RL model in continuous action-space control, making it suitable for adaptive PID tuning in IMG LFC. The main steps of the DDPG algorithm are summarized in the pseudocode shown in Algorithm 1⁴⁶.

Algorithm 1: Summary of the pseudocode for the DDPG algorithm.

TD3-based RL for continuous control

The TD3 is an off-policy actor–critic RL algorithm developed for continuous action spaces. It extends DDPG by addressing overestimation bias and training instability through algorithmic enhancements⁴⁸. Since TD3 is derived from DDPG, DDPG framework is first presented as a baseline. RL enables an agent to learn optimal behaviour through interactions with an environment by maximizing cumulative rewards⁴⁶.

In RL, an agent perceives the state, chooses actions via a policy, and obtains incentives to enhance performance; the primary components include the agent, environment, state, action, reward, policy, and value function., as illustrated in Fig. 7.

Fig. 7.

General scheme of RL architecture.

Figure 8 shows the overall architecture of the TD3 algorithm, which employs six neural networks: one actor responsible for selecting actions, and its corresponding target actor network, two critics, and two target critic networks³¹.

Fig. 8.

General structure of TD3 algorithm.

During interaction with the environment, the agent collects experience tuples (state, action, reward, and next state) defined as parameters , which are stored in a replay buffer. Instead of being used immediately, these experiences are randomly sampled to update the networks.

This mechanism, shared with DDPG, breaks temporal correlations between consecutive samples and improves learning stability in continuous action spaces. The replay buffer has a fixed capacity and overwrites older samples as new ones are added.

A major limitation of the DDPG is overestimation bias, where Q-values are systematically overestimated due to function approximation and maximization errors, leading to numerical instability.TD3 is proposed to address these issues by introducing three key improvements within the actor–critic framework: (i) clipped double Q-learning, (ii) target policy smoothing, and (iii) delayed policy and target updates³¹.

Clipped double Q-learning reduces overestimation by maintaining two critic networks, and , and using the minimum of their estimates to compute the target value, as shown in Fig. 9. This prevents excessively optimistic Q-value updates. Both critics are updated using the same TD target, while a single actor policy is retained. Target policy smoothing further improves stability by adding small clipped Gaussian noise to the target action, ensuring smoother value estimates and preventing abrupt policy changes. The noise and resulting action are both clipped to remain within valid bounds. Delayed policy and target updates form the third enhancement. Unlike DDPG, TD3 updates the actor and target networks less frequently than the critics.

Fig. 9.

Data flow over the target network to determine the TD target computation using clipped double Q-learning.

This delay reduces variance in policy updates and provides more stable TD targets. Target networks are updated using Polyak averaging with a small update coefficient , similar to DDPG. The entire TD3 procedure is summarised in Algorithm 2⁴³.

Algorithm 2: Summary of the pseudocode for the TD3 algorithm

Neural actor-critic algorithm

The neural actor-critic denotes the integration of two artificial neural networks: the actor network and the critic network .

The proposed technique involves a critic network that supervises the algorithm, monitoring faults from the actor’s interactions with the environment at the state , in accordance with the established policy. The network adjusts them to achieve an ideal assessment of actor action , which estimates the maximum potential reward r⁴⁸. The primary advantage of RL algorithms is in their reliance on interaction with the environment⁴⁸. In other terms, when an agent executes an action, it expects either a reward of + r or a penalty of − r. The control mechanism can be concisely articulated as follows. The quantified frequency deviation produces the control error, which acts as input for the RL agent. The calculated error concurrently advances to calculate reward step to either reward or penalize the neural RL-agent. Table 3 refers to the Architecture of the Critic and Actor Networks.

Table 3.

Architecture of the critic and actor networks.

Network	Layer type	Layer details / parameters	Dimension	Activation
Critic network	SP	FIL		–
		FC	500 units	ReLU
		FC	500 units	ReLU
	AP	FIL		–
		FC	500 units	ReLU
	CP	AL (SP + AP)	500	ReLU
		FC	250	ReLU
	Output	FC		1
Actor network	SP	FIL		–
	3— HL {1,2,3}	FC	512	ReLU
	2—HL {4,5}	FC	256	ReLU
	HL 6	FC	128	ReLU
	HL 7	FC	64	ReLU
	Output	FC	Action vector
		OS	outputs actions:

FIL is Feature Input Layer, SP is State Path, AP is Action Path, CP is Common Path, HL is Hidden Layer, Fc is Fully Connected Layer, AL is Addition Layer, and OS is Output Scaling.

In this context, represents the quantity of samples in the current batch (where =1 for a single sample), and is the number of features in the state (observation).The sigmoid function is an often employed activation function in statistics, neural networks, and machine learning⁴⁹. Its principal role is to convert any real number into a value within the range of 0 to 1. The sigmoid is represented mathematically as follows⁵⁰:

21

In this case, x is the input , and is the sigmoid function’s output . A scaling sigmoid function is created to rescale the actor network output in order to guarantee equitable tuning of the controller gains in this investigation.

The network output is linearly mapped by this function to the appropriate PID gain limits, which fall within the range . The mathematical representation of linearly rescaled sigmoid functions is equal to⁵⁰.

22

In this instance, is the output of the scaling sigmoid function, constrained to maximum and minimum values and , respectively, so that the heuristic only chooses values in the range [0.1,2]. Figure 10 shows the difference between and , which are used in the actor network. The scaled sigmoid changes the output of the actor network linearly to the action range [0.1, 2], while the normal sigmoid changes inputs to the range [0, 1]. Both functions keep the "S"-shaped curve, but the scaled version makes sure that the actor’s outputs meet the needs of the environment while still being easy to differentiate for gradient-based optimization⁴⁹.

Fig. 10.

The conventional sigmoid function and the scaled sigmoid function are compared.

Reward functions

In RL-based control, reward functions are typically formulated to penalise frequency deviations and generation costs, thereby promoting efficient power system operations. Additional penalty terms are introduced during training to correct large frequency regulation errors and discourage undesirable behaviours that may arise during exploration. The IMG controller system in this work imposes additional costs on negative outcomes to steer the learning process towards more reliable and cost-effective judgements while reducing the probability of unstable behaviour²². The reward function is defined as:

23

The penalty term is defined as

24

where is the control input generated by the actor neural network, ,and is a weighting coefficient, and r represent the reward function.

The control design problem is therefore modeled as follows as a constrained optimization problem:

25

where represents the set of PID controller parameters. corresponds to the expected long-term reward defined in (24). Maximizing is equivalent to minimizing the long-term average of the squared frequency deviation and control effort with explicit penalization of frequency excursions beyond 0.06 Hz.

This work applies DDPG and TD3 deep RL algorithms to adaptively tune PID controllers for IMG frequency regulation. By treating PID gains as continuous actions, the controller gains are regulated in real time based on , improving stability under variable demands and renewable generation stochastics.

A well-defined observation space and reward function further support reliable convergence and effective frequency control, making the TD3-based approach particularly suitable for uncertain microgrid environments. A standard PID-type controller is used, and it is characterized by the following input–output connection⁵¹.

25

Here, is the input of control rule function, and is the controller gain, and are the parameters of PID controller.

Figure 11 shows the proposed control structure with the PID controller parameters tuned using the TD3 algorithm. A TD3 agent employs an actor representation to decide actions based on observations.

Fig. 11.

Illustration TD3-based PID tuning approach.

where  is the system error,  is the system output, and  is the reference signal.The weights of the actor are equal to the gains of the PID controller, and hence the PID controller could be modeled as a neural network with a single fully connected layer that takes the error, error integral and error derivative as inputs (the system state). The output of the actor neural network is the command signal (activity).

The combination of DDPG-PID controller and the TD3-PID controller-based tuning approaches allows the controller to react dynamically to the system uncertainties and nonlinearities without having to have an accurate mathematical model of the IMG. The appropriate design of the observation space and reward function significantly influences the efficacy and dependability of an RL-based PID controller. Table 4 shows the common observation variables.

Table 4.

Common observation variables.

Symbol	Description	Purpose
	Error in instantaneous control	Quantifies the departure from the reference
	Rate of error change	Captures fleeting behavior
	Integral of error	Reflects the total offset
	System output	Assists in determining steady-state performance
	Change of control input	Promotes efficient control action

Simulation implementation block diagram

In order to enhance the clarity of the proposed RL-agent integrated with a PID controller for LFC in the IMG system, a detailed simulation block diagram has been developed in Simulink environment, as illustrated in Fig. 12. The block diagram describes the implementation framework of the PID controller integrated with an RL agent, where the RL agent adaptively realizes control parameters to achieve effective  regulation in the IMG system.

Fig. 12.

An overall simulation block diagram of the closed-loop RL agent integrated with a PID controller for intelligent IMG.

The implementation is structured into seven main subsystems: (1) IMG (plant), (2) Observation Generation, (3) Reward Calculation, (4) RL Agent, (5) Stop Criteria, (6) Stochastic Model of WTG, and (7) Stochastic Model of PV panels. The stochastic energy subsystem models of WTG and PV include random inputs and transfer functions to emulate environmental variability, producing energy of and , respectively. The IMG sub-block (plant) is the nonlinear dynamics of DGUs and ESS, which are described in terms of equivalent first-order transfer functions listed in Table 1 and a block diagram demonstrated in Fig. 5.

The PV and WTG subsystems, illustrated in Fig. 12, emulate the stochastic and dynamic behaviour of the renewable energy sources incorporated in IMG. Both PV and WTG inputs are simulated with stochastic uniform random signals to describe the natural variability of the solar irradiance and wind speed profile, respectively. These signals are biased and scaled to ensure the generated profile remains within a defined operating range. In PV panel, there is an adjustment of the input with of 0.1 and 0.08 in the WTG. In both systems, a constant bias term, denoted to as , is included to ensure that the input stays within an acceptable range. The processed inputs are then fed into dynamic blocks that have two parameters: a direct gain and indirect gain , which together represent a first-order transfer function for the dynamic response of renewable generation systems. The outputs of these components are added to a constant bias term , which ensures proper scaling of the generated power signal.

Following this step, the combined signals are scaled using multiplication blocks. The two systems have a similar scaling factor, , and the final adjustment is done by different parameters: in the PV system and in the WTG system. The choice of these parameters is to normalize output power and represent the relative contribution from each source. The final output of this process represents the wind power generation , which is then supplied to the IMG system as part of the total generation input for frequency regulation analysis. Similarly, the output of the PV subsystem is also used as an input to the IMG system for frequency regulation analysis.

The IMG is the environment in the implementation RL block diagram, which produces error signal of the and control effort signal as a response to variations in the load and generation. These signals are processed in the Generate Observations block, where transfer functions are applied to extract relevant state variables. The output observation vector is provided to the RL agent, serving as the system state.

The calculate Reward block is used to assess the performance of the system by summing squared error, control effort, and violation of bounds. The reward mechanism helps to punish large variations in error and superfluous control behaviour of the agent, therefore steering the agent to stable and efficient performance. The stop-criteria block supervises the simulation, terminating the process when error surpasses predetermined thresholds (e.g., ).

The RL agent uses the TD3 or DDPG algorithms to turn the observation and reward signals into the adaptive PID parameters . The actor network generates deterministic control actions, while the critic network evaluates their quality.

Simulation results

The proposed adaptive RL–PID controllers (DDPG–PID and TD3–PID) were evaluated for LFC of an IMG incorporating heterogeneous generation sources, including; DEG, SPV, WTG, BGTG, FC, BES, FESS, EV storage model. Simulations were conducted using native simulation functions in MATLAB/Simulink R2022b on a Windows 10 PC with an Intel® Core™ i7-14600 K CPU (1.6 GHz) and 16 GB RAM. These RL– based controllers automatically adjust the PID gains in real time to keep the frequency stable even when the load and generation conditions change. Table 5 summarizes the training parameters for TD3 and DDPG algorithms. Collectively, these parameters support efficient convergence and reliable performance during reinforcement learning.

Table 5.

Training parameters for TD3 and DDPG RL algorithms.

Parameter	Value
Mini-batch size	128
Max-episodes	1000
Replay buffer size	5e6
Sample time	0.25 s
Simulation time per episode	90 s
Reward averaging window	10
Early stopping threshold	345
learning rate: [actor, critic]	[1e-4,1e-5]
Discount factor	0.99
Target-smooth factor	1e-3
DDPG Noise (variance)	0.2
DDPG NOISE DECAY	1e-5
TD3 noise std	0.1
TD3 noise decay	1e-5

Figure 13 shows the training reward curves for the TD3 and DDPG algorithms. rapid convergence is demonstrated in terms of average reward index, with rewards stabilizing near the upper bound (approximately 3599.96) during first 10 episodes. Initially, the TD3 algorithm has slightly higher and more stable average rewards than the DDPG algorithm. The DDPG algorithm, on the other hand, has a steep initial increase before levelling off.

Fig. 13.

Training Reward Curve for TD3-PID and DDPG-PID Algorithms.

Figure 14 illustrates the training frequency response for PID controllers designed with TD3 and DDPG algorithms. The frequency response training analyses demonstrate that both RL-agent PID controllers are effective in regulating frequencies, as they promptly return the system to its nominal value after an initial deviation and maintain stability despite the presence of stochastic demand. Minor oscillations can be observed within acceptable limits, which means the performance of damping is satisfactory for and lines. The power requirement standard compares stochastic demand to generated power, showing that both TD3-PID and DDPG-PID controllers can follow the load variations that vary rapidly. The almost ideal match of the demand and the generated power curve depict the approach of TD3 and DDPG algorithms, which are adaptive and accurate in real time.

Fig. 14.

Frequency response during the training of RL agent-based PID controllers using TD3 and DDPG algorithms.

Figure 0.15 shows the dynamic performance response of the RL-PID controllers (TD3-PID and DDPG-PID) in comparison with the uncontrolled scenario, focusing on both frequency deviation (), and power deviation ().

Both TD3-PID and DDPG-PID exhibit fast convergence to a nominal frequency of 50 Hz and effectively track the reference signal and maintain minimal steady-state error. The transient behaviour shows that TD3-PID and DDPG-PID effectively suppress oscillations within a few seconds. Among the two controllers, the TD3-PID exhibits slightly better damping characteristics than the DDPG-PID, showcasing reduced overshoot and a more gradual convergence. In contrast, in the case of an uncontrolled loop, which is primary control loop without a controller, the response reveals significant oscillations and a pronounced steady-state deviation below the nominal frequency. This behaviour clearly indicates that the IMG is unstable when there isn’t a controller in the primary control loop. A similar trend is observed in terms of demeaned deviation (). In the case of using RL-agent-based PID controllers, both controllers accurately track demand (0.02 pu) with negligible steady-state error and rapid settling.

The initial transient oscillations, as shown in Fig. 15, are rapidly attenuated, which demonstrates strong damping capability. In comparison, TD3-PID shows slightly better performance, indicated by smaller-amplitude oscillations and faster stabilization. Conversely, uncontrollable exhibits display pronounced oscillation behaviour with a slower decay, suggesting poor damping and a susceptibility to sustained disturbances.

Fig. 15.

Dynamic responses of and in the LFC-based IMG using TD3-PID and DDPG-PID controllers compared with the IMG without a controller in the secondary control loop.

Evaluation of efficiency

The performance of the system was measured using time-domain performance metrics, namely the Integral of Squared Error (ISE), Integral of Absolute Error (IAE), Integral of Time-Weighted Squared Error (ITSE), and Integral of Time-Weighted Absolute Error (ITAE). The assessment system was based on four separate case studies which were aimed to measure dynamic performance as well as robustness.

• Efficiency (Cases I–III): assessing the system’s reaction to manage load demand uncertainties across various load patterns.

• Robustness (Case IV): This means doing a sensitivity analysis to see how the system works when there are parametric uncertainties.

Case I: constant load demand

The case analyzes the behaviour of proposed controllers in the presence of sudden load changes. To determine the stability of the IMG system and dynamic response characteristics, a constant-load scenario is used. Particularly, the IMG system is subjected to a step increase in load demand of 0.02 pu to test how this system responds to a sudden change in demand.

Figures 16, 17, 18 and 19 depicts a comprehensive performance evaluation of the TD3-PID and DDPG-PID controllers in maintaining stability under this disturbance condition.

Fig. 16.

Case I: Output power changes for controllable sources.

Fig. 17.

Sharing power for storage sources case I.

Fig. 18.

Changes in the output power of Wind and Solar units– Case I.

Fig. 19.

frequency response of the TD3–PID controller and DDPG-PID Case I.

Figure 16 illustrates how the primary controllable sources in IMG affect this demand case. As can be seen for both controllers, power is immediately sent from the diesel generator (), fuel cell (), and biogas turbine () to address the load change issue. However, the TD3-PID controller transitions with fewer ripples and leads to fewer fluctuations in the output power of sources.

Figure 17 shows the working principles of BESS, FESS, and EV units as forms of energy storage elements, which are indicated by , , and , respectively. This source significantly impacts the rapid changes in power that occur after a rise in load demand, thereby helping to establish balance in the IMG system and relieve strain on traditional generation resources.

As shown in Fig. 17, TD3-PID and DDPG-PID controllers can effectively manage these storage resources. However, the TD3-PID controller seems to create a more consistent power-sharing profile. Figure 18 demonstrates the stochastic power output from wind generators and PV panels, which are denoted by and , respectively. Even though the output power from these renewable sources is naturally unpredictable when delivered to IMG, both RL-PID controllers indicate robust operation, effectively mitigating these variations and maintaining overall system stability under stochastic conditions, as shown in Fig. 19 for and stochastic power deviation .

As shown in Fig. 19, after the initial sudden load change, the frequency response for TD3-PID and DDGP-PID controllers restores the frequency to its nominal value with minimal overshoot and undershoot. The frequency remains stable throughout the simulation period, which is additional supported by the low values observed for the performance indices ISE, IAE, ITAE, and ITSE.

Table 6 lists a comparative time-domain evaluation of the TD3-PID and DDPG-PID controllers for Case I. The results indicate that both controllers achieve rapid settling times, with TD3-PID and DDPG-PID reaching steady-state in approximately 0.87 s and 0.88 s, respectively.

Table 6.

Comparison of time-domain metrics for TD3-PID and DDPG-PID controllers in Case I.

Metric	Controller
	TD3-PID	DDPG-PID
Undershoot	0.25696	0.23815
Max overshoot	0.43069	0.4589
Settling Time	0.87*10–1	0.88*10–1
ISE	7.2938*10–5	5.3673*105
IAE	2.6575*10–2	2.2582*10–2
ITAE	5.0196*10–1	4.9536*10–1
ITSE	2.5095*10–4	1.7590*10–4

Similarly, the TD3-PID controller has a slightly smaller maximum overshoot of approximately 0.43, compared to the DDPG-PID controller, which reaches around 0.46; this implies that the TD3-PID controller dampens transient peaks more effectively. In terms of undershoot, both controllers perform similarly, with a slightly lower value in DDPG-PID.

The performance indices also enable the determination of the effectiveness of each control approach. Thus, the TD-PID controller has a lower ISE, implying a smaller cumulative squared error. However, the DDPG-PID controller shows better performance regarding the key goals of reducing IAE, ITAE, and ITSE, indicating its capability to reduce the absolute and time-weighted errors.

Case II: random load demand variations

This case evaluates the performance of the proposed controllers under stochastic and unpredictable load variations, reflecting realistic operating conditions. In order to increase the practical relevance of analysis, a sequence of random load demand ranging as follows pu is used. Figures 20, 21 and 22 illustrate the dynamic response of the IMG under these stochastic demanding conditions, characterized by multiple stochastic step changes in load demand.

Fig. 20.

Power sharing for controllable sources case II.

Fig. 21.

Power sharing for storage sources case II.

Fig. 22.

Frequency response of various RL controllers Case II.

Analyze the reaction of controllable power sources in case II:

Figure 20 shows the controllable generation sources: DG, FC-units, and BGTG. These sources all change the amount of power they produce based on how demand is changing.

It appears in Fig. 20 that each generator is able to ramp up or down as needed, contributing to a coordinated effort that keeps the power balance intact.

Figure 21 shows how different energy storage units share power. The BESS, FESS, and EV are shown as , and units., respectively. Each storage unit responds rapidly to load disturbances, highlighting its critical role in providing immediate power support to maintain system stability. These sudden changes in storage output are accompanied by shifts in load steps, which reveal their efficiency to reduce short-term fluctuations and the burden on conventional generation units.

Frequency response analysis of case II:

Figure 22 shows the frequency response for both RL-based controllers. Every load change causes a brief , but both TD3-PID and DDPG-PID controllers quickly restore the frequency to its nominal value with minimal overshoot and fast settling times. In addition, the time-domain performance metrics for both controllers are summarized in Table 7. The TD3-PID and DDPG-PID controllers are under the multiple-step load demand scenario.

Table 7.

Performance metrics of TD3-PID and DDPG-PID controllers during multiple-step load demand case II.

Metric	Controller
	TD3-PID	DDPG-PID
Undershoot	0.607	0.567
Max overshoot	0.809	0.8107
Settling Time	87*10–1	88*10–1
ISE	2.596*10–4	2.015*10–4
IAE	7.384*10–2	6.306*10–2
ITAE	3.1845*10–1	2.8391*10–1
ITSE	9.815*10–3	7.862*10–3

The TD3-PID and DDPG-PID controllers achieve similar settling times, with the system stabilizing in under one second after each loading event. The maximum overshoot values are nearly identical to both controllers, which indicate that TD3-PID and DDPG-PID controllers provide effective damping and avoid excessive frequency excursions during transients. The undershot values are also close, reflecting their comparable ability to minimize negative deviations from the nominal frequency.

The performance indices, including ISE, IAE, ITAE, and ITSE, indicate lower values for the DDPG-PID controller, which implies a slight performance advantage. These lower values mean that the IMG system is better at reducing cumulative and time-weighted errors. In this context, the results indicate that the DDPG-PID controller is slightly more efficient in minimizing the total deviation of the nominal operating point and smoother system recovery, especially in the conditions of frequent and considerable changes in loads.

Case III: pulsed load fluctuations

The case evaluates the robustness of the proposed RL-agent PID controllers for high-frequency, repetitive load changes, reflecting industrial cyclic processes and EV charging/discharging processes. A load profile with peaks of 0.018 pu at 50 Hz frequency is driven, which is overlaid by stochastic noise to recreate realistic operating conditions.

Figures 23, 24, and 25 demonstrate the dynamical reaction of the RL-PID controller based on the IMG system to stochastic load changes, where controllable generation units, as well as the ESS, must be capable of responding to sudden and dramatic shifts in demand.

Fig. 23.

A Case III situation for sharing power between controlled sources.

Fig. 24.

Changes in the output power for storage source units—Case III.

Fig. 25.

Frequency response of various controllers based on the TD3-PID and DDPG-PID controller Case III.

Figure 23 shows the performance of controllable sources such as the DG, FC, and BGTG. The results indicate that, under both RL–PID controllers, these units effectively track load variations in both increasing and decreasing demand scenarios while maintaining frequency deviations within nominal limits.

The transitions are well-coordinated, resulting in minimal oscillations and preventing sustained steady-state deviations. Figure 24 shows the contribution of the ESS, such as BES, FESS, and EV, to support the operation of the system. The coordinated control strategy allows rapid and effective responses to load disturbances, which means both controllers can distribute the power demand among all the available assets.

Analysis of the frequency response for case III:

Figure 25 illustrates the frequency response to RL-agent-based PID controllers for Case III. Following each pulsed load disturbance, the IMG frequency exhibits transient deviations from its nominal value; however, it is rapidly restored. The TD3-PID and DDPG-PID controllers demonstrate fast dynamic recovery with minimal overshoot and undershoot, indicating effective damping performance under high-frequency load variations.

Both the TD3-PID and DDPG-PID controllers show fast recovery and little overshoot and undershoot. In addition to that, the generated power closely tracks the stochastic demand profile, ensuring continuous power balance between supply and demand throughout the simulation period.

Table 8 summarizes the time-domain evaluation metrics for pulsed demand fluctuations. Both RL-agent-based PID controllers achieve similar settling times, with the IMG stabilizing in just under one second. The maximum overshoot and undershoot values are very near to each other, which means that they work equally well at regulating temporary frequency changes.

Table 8.

Performance metrics of TD3-PID and DDPG-PID controllers during pulsed load demand Case III.

Metric	Controller
	TD3-PID	DDPG-PID
Undershoot	0.5477	0.5908
Max Overshoot	0.7936	0.7975
Settling Time	996*10–1	996*10–1
ISE	2.596*10–4	2.015*10–4
IAE	7.384*10–2	6.306*10–2
ITAE	3.1845*10–1	3.1626*10–1
ITSE	9.815*10–3	7.862*10–3

It appears that the TD3-PID controller has a slightly lower undershoot, but the DDPG-PID controller has slightly lower error indices (ISE, IAE, ITAE, and ITSE). This result indicated that it is slightly better at reducing cumulative and time-weighted errors.

Case IV: Gaussian load variation

In order to further evaluate the effectiveness of the proposed TD3-PID and DDPG-PID controllers, additional analyses were conducted by incorporating significant system nonlinearities and stochastic load demand. The load demand is randomly distributed according to a Gaussian distribution, which introduces parametric uncertainty into the system and provides valuable insights into the performance of the TD3-PID and DDPG-PID controllers.

Gaussian load change is the use of Gaussian Processes (GP) as a statistical method to model and capture the natural variability and uncertainty in microgrid load patterns⁵².

Figures 26 and 27 show both RL-based PID controllers are successful at coordinating generation and storage sources to respond to highly variable demand. The controllable generators, such as the DG units, FC units, and BGTG units, have output patterns that stay the same even when parameters change, as shown in Fig. 26.

Fig. 26.

A Case IV scenario for power sharing among controllable sources.

Fig. 27.

Changes in the output power for storage source units – Case IV.

Similarly, Fig. 27 shows the performance of storage sources as the BESS, FESS, and EV units adapt quickly to buffer the system from random load fluctuations. The outputs of the storage sources show that demand is random, but transitions are still smooth and controlled.

Frequency response analysis of case IV:

Figure 28 shows the frequency response and the ability of the generated power to track the randomly varying demand. Both the DDPG-PID and TD3-PID archives and keep frequency deviations minimal, restoring nominal frequency swiftly after each disturbance.

Fig. 28.

Frequency response of various controllers based on the TD3-PID and DDPG-PID controller Case IV.

The generated power curve closely follows the random profile of the demand, illustrating effective real-time balancing. The performance indices presented in Table 9 support these observations, showing comparable error values for both TD3-PID and DDPG-PID controllers. DDPG-PID achieves slightly lower maximum overshoot and undershoot, while both controllers maintain nearly identical settling times and very close error metrics.

Table 9.

Comparison of TD3-PID and DDPG-PID Performance under Randomly Distributed Gaussian Load.

Metric	Controller
	TD3-PID	DDPG-PID
Undershoot	0.550	0.515
Max Overshoot	0.419	0.392
Settling Time	999*10–1	999*10–1
ISE	9.849*10–5	8.8.092*10–5
IAE	6.499*10–2	6.188*10–2
ITAE	2.860*10–1	2.886*10–1
ITSE	1.936*10–3	1.933*10–3

These results, conducted under Case IV, underscore the robustness of TD3-PID and DDPG-PID approaches in coping with system nonlinearities, as well as their effectiveness in sustaining frequency stability under unpredictable operating conditions.

Sensitivity analysis to parameter changes

In order to thoroughly evaluate the sensitivity of the IMG to a broad range of parameter uncertainties, uniaxial and multiaxial sensitivity analysis is conducted. In the multiaxial evaluation approach, all previously identified uncertain parameters are varied simultaneously within the IMG model, while all other settings are kept consistent with those in case I to enable direct comparison with the baseline case. In the uniaxial evaluation approach, only one parameter varied in the IMG model, and other settings are kept the same as in case I.

In particular, the inertia constant (), load damping coefficient (,) generator time constant (), and fuel cell time constant () are changed by ± 25% for single-parameter and multiple-parameter variations of uncertainties scenarios, as shown in Table 10.

Table 10.

Time domain evaluation indexes unverified uncertain system parameters—Case I.

Parameter	Change rate (%)	TD3-PID					DDPG-PID
		Settling time (S)	Undershoot (%)	Overshoot (%)	ITAE	ITSE	Settling time (S)	Undershoot (%)	Overshoot (%)	ITAE	ITSE
	−25(Uniaxial)	0.854	0.513	0.383	0.502	9.51*10 −10	0.854	0.477	0.531	0.495	2.64*10 −9
	−25 (Multiaxial)	0.853	0.515	0.511	0.502	2.64*10 −9	0.854	0.477	0.531	0.495	2.64*10 −9
	Normal	0.854	0.854	0.431	0.502	1.49*10 −9	0.855	0.476	0.459	0.495	1.49*10 −9
	+ 25(Multiaxial)	0.854	0.513	0.383	0.502	9.51*10 −10	0.855	0.471	0.404	0.498	9.51*10 −10
	+ 25 (Uniaxial)	0.854	0.513	0.383	0.502	9.51*10 −10	0.855	0.471	0.404	0.498	9.51*10 −10
	−25(Uniaxial)	0.854	0.514	0.431	0.502	1.49*10 −9	0.855	0.476	0.459	0.495	1.49*10 −9
	−25 (Multiaxial)	0.854	0.514	0.431	0.502	1.49*10 −9	0.855	0.476	0.459	0.495	1.49E-09
	Normal	0.854	0.514	0.431	0.502	1.49*10 9	0.855	0.476	0.459	0.495	1.49*10 −9
	+ 25(Multiaxial)	0.854	0.514	0.431	0.502	1.49*10 −9	0.855	0.476	0.459	0.495	1.49*10 −9
	+ 25 (Uniaxial)	0.854	0.514	0.431	0.502	1.49*10 −9	0.855	0.476	0.459	0.495	1.49*10 −9
	−25(Uniaxial)	0.855	0.474	0.415	0.494	1.49*10 −9	0.855	0.513	0.396	0.500	1.49*10 −9
	−25 (Multiaxial)	0.855	0.513	0.396	0.500	1.49*10 −9	0.855	0.474	0.415	0.494	1.49*10 −9
	Normal	0.855	0.476	0.459	0.495	1.49*10 9	0.854	0.514	0.431	0.502	1.49*10 −9
	+ 25(Multiaxial)	0.853	0.514	0.460	0.505	1.49*10 −9	0.854	0.469	0.486	0.500	1.49*10 −9
	+ 25 (Uniaxial)	0.854	0.469	0.486	0.500	1.49*10 −9	0.853	0.514	0.460	0.505	1.49*10 −9
	−25(Uniaxial)	0.854	0.514	0.443	0.502	1.49*10 −9	0.854	0.481	0.442	0.495	1.49*10 −9
	−25 (Multiaxial)	0.854	0.514	0.412	0.502	1.49*10 −9	0.854	0.481	0.442	0.495	1.49*10 −9
	Normal	0.854	0.514	0.431	0.502	1.49*10 −9	0.855	0.476	0.459	0.495	1.49*10 −9
	+ 25(Multiaxial)	0.854	0.514	0.443	0.502	1.49*10 −9	0.855	0.468	0.470	0.496	1.49*10 −9
	+ 25 (Uniaxial)	0.854	0.514	0.412	0.502	1.49*10 −9	0.855	0.468	0.470	0.496	1.49*10 −9

The IMG frequency response, which demonstrates controllability, is illustrated in Fig. 29 (TD3–PID and DDPG to PID) under a uniaxial and multiaxial scenarios. In addition, corresponding quantitative time-domain performance indices are reported in Table 10.

Fig. 29.

The frequency response of the TD3-PID and DDPG-PID controllers under ± 25% uncertainties in uniaxial and multiaxial parameters.

It is observed that both RL-agent-based PID controllers preserve closed-loop stability under all uncertainty scenarios, confirming the robustness of the proposed approach against both uniaxial and multiaxial variations. The settling time remains nearly constant for both proposed RL-agent-based PID controllers s), indicating that neither controller is significantly affected by the considered uncertainties in terms of response speed.

For variations in the coefficient (), TD3-PID dependably demonstrations lower overshoot during a negative perturbation than DDPG-PID, even with a negative perturbation (e.g., 0.383% vs. 0.531% in the case of a -25% uniaxial perturbation).

Although DDPG-PID shows a minor decrease in undershoot, this gain is accompanied by a more significant increase in overshoot, resulting in a transient response that is less balanced. The ITAE and ITSE indices for DDPG-PID and TD3-PID are similar, but TD3-PID usually gives lower or similar ITSE values, which means that the error signal is stronger.

In the inertia () variations, RL-agent-based PID controllers demonstrate negligible sensitivity, as reflected by nearly identical performance indices across all perturbations. Nevertheless, TD3-PID maintains more consistent transient characteristics, with almost invariant overshoot and ITAE values, whereas DDPG-PID shows minor fluctuations. This indicates enhanced robustness of TD3-PID to inertia uncertainty.

In the case of the variations, the differences observed are more noticeable. TD3-PID has a lower ITAE in some cases (e.g., 0.494 at 0.25 carefulness during uniaxial change), indicating the best performance regarding overall error.

Moreover, the responses in Fig. 29 show that the TD3-PID provides faster damping and reduces oscillatory behaviour in the transient region. Although DDPG-PID occasionally yields slightly lower overshoot under specific multiaxial conditions, its performance is less consistent across different scenarios.

Similarly, for () variations, both Rl-agent based PID maintain stable operation with minimal variation in settling time and ITAE. The zoomed-in plots show that TD3-PID generally has smoother responses and slightly lower oscillation amplitudes. In addition, the DDPG-PID shows marginally higher deviations under multiaxial perturbations.

The graphical results in Fig. 29 corroborate the numerical results in Table 10. The results confirmed that the TD3-PID controller makes trajectories smoother by reducing oscillations and improving damping in all cases of uncertainty. On the other hand, DDPG-PID indicates stability and tends to exhibit relatively more overshoot and less consistent transient behaviour.

Comparison with recent research

Recent advancements in RL-agent methodologies have attracted considerable attention for LFC in power systems, particularly in IMGs, which are characterized by considerable uncertainty and nonlinear dynamics. DRL- controllers have been effectively employed for frequency regulation in²⁶ and²³, demonstrating significant advantages in reducing minimizing frequency deviation, settling time, and curtailing overshoot. The integration of graph neural networks with deep RL has greatly improved system-wide coordination by capturing the structural dependencies between different energy resources²³.

The capacity of RL-based controllers to maintain accurate frequency tracking in dynamic operating settings has been analyzed in deep RL model employing an actor-critic architecture, as detailed in²⁴, where tracking error served as the primary performance metric. The research delineated in²⁵ employed DRL for the optimization of energy scheduling, utilizing cost and profile error as assessment metrics, with a focus on microgrid energy management rather than direct frequency regulation.

These strategies are good for running a business on a budget, but they don’t expressly address transient frequency stability. Furthermore, as noted in²⁶, DDPG-based control strategies have been extensively adopted in multi-area power systems, markedly decreasing overshoot and settling time relative to traditional LFC approach (Table 11) .

Table 11.

A comparison of recent studies for RL-based LFC in the power system.

References no	Year	Microgrid structure	Approach	Metrics	Control
23	2024	IMG	Graph neural net + Deep–RL	Overshoot, settling	LFC
24	2022	IMG	Actor–critic Deep–RL	Tracking error	LFC
25	2021	On-Gird- Microgrid	Deep–RL optimization	Energy scheduling	Cost, profile error
26	2022	IMG	Deep–RL: DDPG TD3	Overshoot, settling time, Peak time	LFC
Current work	–	IMG	Deep–RL: TD3-PID DDPG-PID	ISE, IAE, ITAE, ITSE, , overshoot, settling time	LFC

In addition to the indirect comparison with recent studies, a direct comparison is also conducted to observe the performance of the proposed RL-based PID controllers. Figure 30 shows the convergence curves of the WOA-PID¹³ and COA-PID¹² algorithms, both configured with a search size of 25 and 30 iterations. The WOA algorithm curve shows a sharper decrease in the ITAE objective function in the earlier iterations and a lower final value compared to COA, which shows a superior convergence behaviour and optimization effectiveness.

Fig. 30.

Convergence of learning-based PID parameters via optimization algorithms.

These results indicate that WOA is more efficient in identifying optimal PID parameters for LFC in the IMG. Table 12 lists the resultant optimum controller gains that were found using both optimization algorithms and the classical criteria.

Table 12.

Optimal controller parameters using WOA-PID, COA-PID, and non-learning tuning approaches.

Controller
WOA-PID	3.84	10.00	2.63
COA-PID	3.00	7.32	2.57
PID	3.71	1.39	0.33

Figure 31 compares the behaviour of the tuned PID parameters under different load demands. This comparison includes learning and non-learning algorithms evaluated over 500 episodes. It appears from , , and behaviour that the COA-driven PID, WOA-driven PID, and conventional PID controllers maintain fixed parameter values throughout, as expected.

Fig. 31.

Comparison of PID Parameters Using Learning and Non-Learning Algorithms Under Varying Load demands.

On the other hand, TD3 driven PID was shown to be more robust and adjustable to changing requests, whereas DDPG-driven PID approach need further development to also be as efficient in dynamic settings, especially regarding the ability to adjust hyperparameters and evolve exploration policies.

The performance metrics in Table 13 and the performance shown in Fig. 31 for the multiple-step load-demand scenario clearly show how much better learning-based RL-PID controllers are than both the fixed-parameter and baseline parameter-tuning PID controller approaches. The curves in Fig. 32 indicate that TD3-PID and DDPG-PID controllers recover the system response faster under load disturbance conditions and have fewer frequency excursions than both fixed-parameter learning-based (WOA-PID, COA-PID) and baseline PID controller approaches.

Table 13.

Comparative performance metrics for learning-based and non-learning-based approaches for tuning PID controllers.

Metric	PID tuning approach
	Learning-based fixed-parameters		Non-learning baseline	Learning-based adaptive parameters
	WOA-PID	COA-PID	Conventional -PID	TD3-PID	DDPG-PID
Undershoot	0.4727	0.4748	1.1026	0.607	0.567
Max Overshoot	0.7031	0.7167	1.673	0.809	0.8107
Settling Time	100.19	100.89	103.42	8.7	8.8
ISE	2.90 × 10−5	3.76 × 10−5	4.01 × 10−4	2.60 × 10−4	2.02 × 10−4
IAE	0.0159	0.0205	0.0557	0.0738	0.0631
ITAE	0.7976	0.9713	2.3287	0.3185	0.2839
ITSE	0.00158	0.00186	0.00914	0.00982	0.00786

Fig. 32.

Compares the proposed RL-based adaptive PID with WOA-PID, COA-PID, and a non-learning baseline approach.

The adaptive RL-based PID approach reduce response metrics like overshoot and undershoot and quickly return to baseline frequency while exhibiting superior dynamic performance and robustness to disturbance rejection. These observations align with the quantitative results presented in Table 13 and reinforce the efficacy of adaptive, learning-based PID controllers for frequency regulation in dynamic power systems.

Furthermore, both proposed TD3-PID and DDPG-PID demonstrate superior error minimization, as evidenced by lower ISE, IAE, ITAE, and ITSE values when compared to WOA-PID and COA-PID. In terms of error-based indices, the TD3-PID and DDPG-PID show the lowest values for ISE, IAE, ITAE, and ITSE, better reflecting enhanced tracking accuracy and improved disturbance rejection.

Remarks on the proposed approach

This paper presented deep RL-adaptive PID controllers employing TD3 and DDPG algorithms for LFC in IMGs to mitigate the performance constraints of the conventional PID tuning approach in which adjustments to disturbances and parameter uncertainties occur. The system performance is assessed through a comprehensive array of time-domain and frequency-domain metrics, including ISE, IAE, ITAE, ITSE, deviation of , overshoot, and settling time, while simultaneously adjusting PID parameters in real-time. This integrated framework provides a more efficient approach for LFC in IMGs with significant renewable energy source penetration, characterized by improved dynamic performance, robustness, and adaptability.

In order to comprehensively evaluate the performance of the TD3-PID and DDPG-PID controllers, three different load disturbance scenarios are considered. In Case I, the IMG system is loaded with a constant load demand of 0.02 pu. Case II: The system is exposed to a sequence of step load changes in magnitudes applied at successive time instants during the simulation horizon, representing random and rapidly changing demand conditions.

The load demand in Case IV is modeled using a Gaussian random stochastic process to capture continuous random fluctuations and parametric uncertainty in the system. The dynamic responses in the two scenarios are represented in Figs. 20, 21, 22, 23, 24, 25, 26 and 27. Both the TD3-PID and DDPG-PID controllers perform well in reducing the frequency variations and stabilizing the system during deterministic step changes and stochastic Gaussian changes in the load demand model.

The TD3-PID shows better robustness and transient performance, despite appear slightly less stable and less acceptable under parameter uncertainties, compared to RL-agent-based PID controllers. It provides reduced overshoots, a less dynamic response, and predictive error indices, as the graph illustrated in Fig. 29 corroborates the numerical results in Table 10. These results show that the TD3-based PID tuning approach is better at reducing uncertain and linked parameter changes than the DDPG-based PID approach.

As can be seen in Fig. 32 and results in Table 13, the proposed RL‑PID controllers outperform the conventional non‑learning approach by adaptively tuning gains in real time, thereby improving robustness, settling time, and frequency stability without requiring precise system models.

Compared with existing learning‑based approaches such as heuristic optimization, including COA-PID and WOA-PID, which often rely on offline tuning, the proposed DDPG‑ and TD3‑based controllers achieve online adaptation through direct interaction with the microgrid environment. TD3‑PID further enhances stability and reduces oscillations by mitigating overestimation bias in policy updates.

These merits highlight the advantages of our approach in achieving adaptive, robust, and efficient LFC in heterogeneous IMGs.

Conclusion

This paper significantly advances the field of load frequency control in IMGs by proposing a DRL-based PID controller that adapts the PID controller gains in response to dynamic operating conditions. A cosmopolitan control framework was developed in which the PID parameters are optimally adjusted using an RL learning-based algorithm. The presented design of an adaptive PID controller for LFC in an IMG system leverages the performance of two RL frameworks, the TD3 and DDPG. The IMG under consideration incorporates renewable and distributed energy resources such as WTG, SPV units, FC units, EVs, and BGTGs under varying local consumer demand. In the proposed controller approach, an intelligent agent interacts with the IMG environment to change the PID controller gains in real time. To improve the performance prior to real-time deployment, the agent is initially trained offline. This training phase involves the careful formulation of reward functions, state observations, and termination criteria while accounting for a range of frequency deviations. The effectiveness of the proposed RL-based PID controller is evaluated through multiple simulation scenarios and benchmarked against a conventional baseline PID controller, a COA PID controller, and a WOA-PID controller.

The simulation results from different operating scenarios and load disturbances show that the proposed TD3-PID and DDPG-PID controllers have better transient performance, as exposed by big drops in overshoot, undershoot, response time, and settling time.

In one of the scenarios, a disturbance in the consumer load was assumed (this issue was not included in the agent’s offline training), and the proposed RL-based controller significantly outperformed both the COA-PID controller and the WOA-PID controller in both transient and steady states. This performance can be attributed to the fact that the proposed RL agents can learn from the environment, enabling better tuning of the gains of the PID controllers.

Furthermore, these results support the fact that TD3-PID and DDPG-PID approaches improve the dynamic stability of isolated microgrids and provide high-quality frequency regulation, especially in the system parameter sensitivity scenario. The scenario of robustness analysis, conducted under ± 25% variations in key system parameters, reveals a high level of adaptability and resilience to parameter uncertainties, underscoring the suitability of the proposed approaches for real-world IMGs operations.

In addition to the DDPG and TD3 algorithms, future work can be extended to investigate the performance of the Soft Actor-Critic (SAC) algorithm in an RL-adaptive PID framework and study the ability of SAC to further improve learning efficiency and overall traffic signal control.

Future work can also include further analysis in terms of system stability in the frequency domain, i.e., the gain and phase margins. Moreover, the adaptability of RL-based approaches can be improved by hyperparameter tuning and exploration algorithms, particularly in the DDPG-PID-based approach, which can lead to improved performance in dynamic settings and more robust control solutions, such as faster convergence and reduced chances of overfitting under varying conditions.

Author contributions

Dana R. S. Suliman: Conceptualization, Methodology, Investigation, Formal analysis, Writing—original draft, Visualization. Ali Nasser Hussain: Software, Data curation, Validation, Formal analysis, Writing—review & editing. Ahmed K. Ali: Conceptualization, Methodology, Supervision, Writing—review & editing, Correspondence. Mohammed Alruwaili: Resources, Funding acquisition, Formal analysis, Writing—review & editing. Moustafa Ahmed Ibrahim: Supervision, Project administration, Methodology, Validation, Writing—review & editing, Correspondence. Hossam Kotb: Conceptualization, Formal analysis, Supervision, Writing—review & editing.

Funding

This research was funded by the Deanship of Scientific Research at Northern Border University, Arar, KSA, for funding this research (through the project number NBU-FFR-2026-2124-03).

Data availability

The last updated simulation files and code associated with this work will be made publicly available on the following repository: https://github.com/Ahmedalrubayi.

Declarations

Competing interests

The authors declare no competing interests.