Multi-robot hierarchical safe reinforcement learning autonomous decision-making strategy based on uniformly ultimate boundedness constraints

Abstract

Deep reinforcement learning has exhibited exceptional capabilities in a variety of sequential decision-making problems, providing a standardized learning paradigm for the development of intelligent multi-robot systems. Nevertheless, when confronted with dynamic and unstructured environments, the security of decision-making strategies encounters serious challenges. The absence of security will leave multi-robot susceptible to unknown risks and potential physical damage. To tackle the safety challenges in autonomous decision-making of multi-robot systems, this manuscripts concentrates on a uniformly ultimately bounded constrained hierarchical safety reinforcement learning strategy (UBSRL). Initially, the approach innovatively proposes an event-triggered hierarchical safety reinforcement learning framework based on the constrained Markov decision process. The integrated framework achieves a harmonious advancement in both decision-making security and efficiency, facilitated by the seamless collaboration between the upper-tier evolutionary network and the lower-tier restoration network. Subsequently, by incorporating supplementary Lyapunov safety cost networks, a comprehensive strategy optimization mechanism that includes multiple safety cost constraints is devised, and the Lagrange multiplier principle is employed to address the challenge of identifying the optimal strategy. Finally, leveraging the principles of uniformly ultimate boundedness, the stability of the autonomous decision-making system is scrutinized. This analysis reveals that the action trajectories of multiple robots can be reverted to a safe space within a finite time frame from any perilous state, thereby theoretically substantiating the efficacy of the safety constraints embedded within the proposed strategy. Subsequent to exhaustive training and meticulous evaluation within a multitude of standardized scenarios, the outcomes indicate that the UBSRL strategy can effectively restricts the safety indicators to remain below the threshold, markedly enhancing the stability and task completion rate of the motion strategy.

Introduction

The rapid evolution of machine learning technology has seen deep reinforcement learning (DRL) become a cornerstone in the development of multi-robot systems, endowing them with the capacity for autonomous control, decision-making and advanced intelligent services^1–3. It is broadly accepted that DRL constitutes a methodology within the machine learning domain, focused on discerning optimal strategies through recursive interactions with dynamic environments, employing a trial-and-error framework. Despite continuous endeavors to implement and enhance reinforcement learning (RL) methodologies, it is apparent that numerous current approaches do not sufficiently address the problems of risk management or decision-making safety^4–6. Certain methodologies even incorporate random exploration into their processes, seeking to enhance approximations without due regard for the inherent risks. In these instances, the implementation of unsafe motion strategies can lead not only to the system entering hazardous conditions but also to potential damage to the robot’s physical structure^7–9. Additionally, it has been noted by researchers that contemporary DRL motion control strategies are not well-suited to managing dangerous tasks that may be encounterd during operation^10,11.

Reinforcement learning autonomous decision-making aim to select the optimal strategy to maximize rewards. Through the exploration of diverse states, the robot ultimately discerns the sequence of states and actions that yield the highest reward¹². While this learning approach can yield an optimal strategy, it may also lead the robot to diverge from the anticipated behavioral trajectory, potentially resulting in unsafe outcomes^13,14. Consequently, incorporating safety considerations into the robot’s motion control process and implementing measures to restrict and circumvent these unsafe scenarios has emerged as a pivotal research focus within the field of deep reinforcement learning for robotics^15–18.

To tackle the challenge of inadequate decision-making security in the realm of deep reinforcement learning, a class of methods known as safe reinforcement learning has been introduced^19–22. This approach, while adhering to safety constraints, employs constrained Markov decision processes to model motion and aims to identify optimal strategies by maximizing expected returns^23–25. Instead of directly imposing constraints on the action space^26–28, it integrates multi-dimensional restrictions derived from action value updates, effectively partitioning the state-action space into viable and non-viable zones^20,29. In practice, an agent’s operation is confined to state-action sequences that meet these constraints, ensuring only permissible behaviors are executed. This methodology safeguards against agents performing dangerous actions that could result in avoidable harm³⁰. Within the robotics sector, especially in multi-robot industries that require real-time control, such as logistics distribution, intelligent transportation³¹, and smart warehousing, safe reinforcement learning systems have found extensive application^32–35.

Safety reinforcement learning autonomous decision-making methods are categorized into two primary groups: risk shielding and safety decision-making. The risk shielding safety reinforcement learning approach directly identifies perilous actions and excludes them from the viable action set, ensuring that the agent’s decisions consistently adhere to safety criteria. Hou et al. delved into the fundamental workings of action shielding in robots and introduced the Action-Level Masking method³⁶. Angelopoulos et al. put forth a technique that incorporates human preferences into reinforcement learning by employing a shielding mechanism³⁷. The shielding mechanism examines all possible actions in the current state and eliminates those that contravene safety protocols, thereby establishing a subset of safe actions. Safety decision-making approaches based on danger shielding demonstrate robust control over the exclusion of hazardous actions, enhancing the efficiency of reinforcement learning decisions while maintaining safety. Nonetheless, issues such as global action screening strategies and interpretable shielding mechanisms need to be explored in future studies.

In contrast to risk shielding, reinforcement learning methodologies that incorporate safety decision-making expand upon the shielding mechanism by introducing a suite of safety action options and dynamically calibrate the range of these actions. This strategy circumvents the limitation of singular action choices and bolsters the safety and resilience of the reinforcement learning system within unpredictable settings. Ji et al. introduced a multi-agent deep Q-network framework, grounded in safety decision-making, aimed at the secure management of adaptable robotic arms, which fine-tunes the action set boundaries to safeguard against unsafe actions³⁸. Duan et al. integrated offline and online training methodologies and implemented a “safety shield” correction mechanism to adeptly rectify unsafe decisions³⁹.

Reinforcement learning (RL) approaches that incorporate safety considerations have the potential to yield secure alternatives while circumventing unsafe actions, thereby significantly enhancing the robustness and reliability of RL systems. Nonetheless, the challenge persists in devising an optimal set of safety-constrained behaviors that are adaptable across various states and capable of dynamically adjusting to the evolving environmental conditions.

Although the concept of safe reinforcement learning has been introduced, its practical application still confronts several significant challenges, as elaborated in^40,41. Primarily, agents integrated with safety constraints frequently prioritize safety excessively, compromising their ability to explore optimal strategies, which ultimately results in suboptimal decision-making efficiency within robotic systems^42–44. Secondly, Markov processes incorporating multiple constraints grapple with difficulties in model resolution and intricate computational procedures, impeding the realization of seamless end-to-end control^40,41. Lastly, these constraints lack the capability to adaptively adjust in response to varying environmental conditions, especially in the realm of multi-robot systems, where ensuring the safety of each robot’s decision-making process poses an even greater challenge⁴².

To tackle potential security concerns that may emerge during the sequential decision-making process, a novel a uniformly ultimately bounded constrained hierarchical safety reinforcement learning strategy (UBSRL) approach is introduced for multi-robot systems collaborative decision-making tasks.

The primary contributions of this manuscript can be articulated as follows:

• Proposes an event-triggered hierarchical safety reinforcement learning framework based on a constrained Markov decision process, which achieves a harmonious advancement in both decision-making safety and efficiency, facilitated by the seamless collaboration between the upper-tier evolutionary networks and the lower-tier restoration networks.

• By incorporating supplementary Lyapunov safety cost networks, a comprehensive strategy optimization mechanism that includes multiple safety cost constraints is devised, and the Lagrange multiplier method is employed to address the challenge of identifying the optimal strategy.

• The stability of the autonomous decision-making system has been established through the application of the uniformly ultimately boundedness theory. This analysis demonstrates that the action trajectories of multiple robots can be reverted to a safe space within a finite timeframe, regardless of their initial perilous state.

• Subsequent to exhaustive training and meticulous evaluation within a multitude of standardized scenarios, the outcomes indicate that the UBSRL methodology can effectively restricts the safety indicators to remain below the threshold, markedly enhancing the stability and security of the motion strategy.

The rest of this paper is organized as follows. Section 2 presents the work related to Markov modeling for safe reinforcement learning methods under constraints. Section 3 describes the design process of the UBSRL approach. Section 4 analyzes the stability of the system using the theory of uniformly ultimate boundedness. Then, we evaluate the performance of our method with the existing algorithms in Sect. 5. Section 6 concludes the paper.

Problem description

Markov decision making with security constraints

The conventional reinforcement learning approach identifies the optimal policy by leveraging the Markov decision process. In contrast, the secure reinforcement learning method, which incorporates safety constraints, determines the optimal policy through a restricted Markov decision process. The decision-making process of this approach is illustrated in Fig. 1.

Fig. 1.

Markov process with safety constraints.

A Markov process constrained by safety considerations can be characterized as follows: . S is a state vector with finite dimensions, and the state vector for all agents can be represented as: . A is the action space of robot:. P is the state transition probability; R stands for immediate reward, and the reward value can be defined according to the robot task environment. C denotes the safety constraint function, which is evaluated in a similar way as the reward function. I represents the total number of robots.

As shown in Fig. 1, the interaction between the robot and the environment can be considered as a discrete-time sequence of a Markov process. Based on the initially perceived environment s₀, the robot makes a corresponding decision action a₀. After the action a₀ is executed by the robot, the environment transitions to a new state s₁, with the probability of the environmental transition probability P. At this stage, the environment immediately assigns a reward r₁ and a cost c₁ to the robot. Subsequently, the robot executes an action a₁ based on the current state s₁, causing the environmental state to transition to s₂ and again providing the robot with a reward r₁ and a cost c₁. This interactive process exhibits a trend of infinite continuation, thereby forming a Markov chain: s₀, a₀, s₁, r₁, a₁, …, s_t−1, r_t−1, a_t−1, s_t, r_t,…. In accordance with the theoretical framework of the Markov chain, the safe reinforcement learning strategy will continuously undergo iterative optimization until convergence is achieved.

Safe reinforcement learning

The Markov decision process with security constraints can be formulated as the process of solving the optimal security policy while satisfying the security constraints as follows.

1

where, denotes the security policies with the security constraints, is the objective function.

The goal of the agent is to maximize the system reward. The reward can be written as the cumulative discounted expected value of all future rewards:

2

where, T is the final time, t is the current time, and s₀ is the initial state of all agents.

In order to consider the effect of the action vector on the state-value function . The target reward in the multi-agent system can be further represented by the action-value function :

3

where, denotes the system state at the next moment, and is the immediate reward of the current robot.

In addition to action-value function , safe reinforcement learning methods require another evaluation function to evaluate the safety of policies. That is, the nonnegative safety constraint function . The constraint function is used to evaluate the security of the action.

Safe reinforcement learning mainly focuses on the safety constraint function under long-term cumulative expectations. For a given trajectory, the cumulative discounted constraint function is as follows.

4

where, represents the immediate cost, denotes the action-state trajectory.

Robot safety constraints can be broadly categorized into two main types: direct constraints on the action space and indirect constraints. The mechanism of direct constraints can be articulated as follows:

5

In which, and represent the upper and lower thresholds of robot actions.

In this scenario, the robot’s action space is frequently constrained, which inherently limits its flexibility. Consequently, our UBSRL approach eschews this restrictive mechanism, opting instead for an indirect constrained safe reinforcement learning strategy.

Safe reinforcement learning strategies impose constraints not directly on the robot’s action space, but rather on its cost function, representing an indirect constraint approach. This indirect constraint approach can be further categorized into two types: hard constraints and soft constraints.

When undertaking decision-making tasks that demand stringent security measures, the hard constraint method is typically implemented, ensuring that the single-step constraint is consistently enforced. Consequently, the set of security policies can be articulated as follows:

6

The rigid constraints of single-step decision-making impose overly stringent limitations on the range of actions available, typically used in model-based reinforcement learning methodologies for autonomous decision-making.

In contrast, most model-free reinforcement learning tasks favor soft constraints, which are governed by the cumulative expected discounted value of the cost function. Consequently, the safe policy can be articulated as follows:

7

Consequently, the UBSRL strategy employs a more adaptable indirect constraint approach. The robot’s action space remains unrestricted; the objective is solely to guide the robot in steering clear of hazardous zones while encouraging exploration of safer areas.

In the robot decision-making process, if the action at each step satisfies condition , the current action can complies with the safety constraint. Should the robot inadvertently enter a hazardous state, it maintains the capability to recover from the unsafe condition within a finite time frame. This soft constraint methodology has demonstrated impressive results in tasks such as multi-robot autonomous decision-making tasks.

In multi-agent safe reinforcement learning, the goal of safe Markov decision problem is to maximize the cumulative rewards r while ensuring that the expected safety constraint is less than the constraint threshold m. The solution process of the optimal policy can be formulated as a constrained optimization problem as follows.

8

where, m is the threshold of the safety constraint and is the expected safety constraint: .

In the autonomous decision-making process, the agent is awarded a reward and incurs a cost at each step. Our goal is to maximize the cumulative reward over time, while ensuring that the cost threshold is not breached. This presents a soft constrained optimization challenge. The optimal policy can be derived by optimizing the objective function that balances rewards with safety constraints.

Method design

Event-triggered layered safety policy

Decision safety is a crucial aspect to contemplate in multi-robot system. Nevertheless, agents frequently disregard potential hazards in pursuit of high value, inadvertently leading multi-robot into perilous zones. To mitigate this issue, we introduce an event-triggered hierarchical security policy in this section. This policy encompasses two tiers: the evolution strategy and the security recovery strategy, as depicted in Fig. 2.

Fig. 2.

Event-triggered hierarchical security policy.

The evolution strategy boasts swift learning and updating capabilities, enabling it to promptly converge on the optimal solution. Conversely, the safety policy emphasizes ensuring that the robot makes prudent decisions when safety constraints are breached, guiding it back to a secure state. Upon the driving event’s output exceeding a predetermined threshold, the safety trigger is activated, prompting the system to transition from the evolution strategy to the security recovery strategy. This transition ensures the safety of the robot’s action outputs. The event-triggered state of the robot is defined as follows:

Definition 1

If the robot undertakes the current action and subsequently collides, rolls over, or falls into a trap, such that the trigger event value is greater than or equal to the threshold value, i.e.: , the robot will be deemed to have entered a dangerous space. Consequently, the subsequent action of the robot will be constrained, and the autonomous decision-making strategy will be switched to a safe recovery strategy.

Definition 2

If the robot executes the current action, and the trigger event value fails to reach the trigger threshold m, i.e.: , the robot’s actions will remain unconstrained. Consequently, the robot will proceed with the evolution strategy, or transition from the previous safety recovery strategy to the evolution strategy.

Then, the experience buffer is constructed separately according to the different policy: the non-safe experience buffer and the safe experience buffer.

When the conditions of Definition 1 are satisfied, the agent executes the safe recovery strategy, and the dangerous state experience samples will be stored in the non-safe experience buffer .

When the conditions of Definition 2 are satisfied, the agent executes the When the conditions of Definition 1 are satisfied, the agent executes the safe recovery strategy, and the state experience samples will be stored in the non-safe experience buffer .

• (A)Evolution strategy.

In the realm of the event-triggered hierarchical security strategy, the evolutionary approach adopts an intricate multi-agent parallel structure, which is illustrated in Fig. 3. Each individual agent within this framework possesses a dedicated Actor-Critic network, enhancing its learning capabilities. Additionally, to mitigate the potential overestimation of Q values, these agents employ a dual Critic network. Notably, the agents leverage global information to augment their learning processes and utilize local information for making informed decisions. The cumulative joint strategy, emanating from the collective actions of all agents, can be succinctly articulated as:, and its policy parameters can be expressed as:.

Fig. 3.

Multi-agent distributed network structure of the evolution strategy.

In the evolutionary strategy, each actor agent is responsible for generating the robot action , and the Critic network is responsible for evaluating the current action value. After each interaction with the environment, an experience sample will be produced and stored in a safe sample buffer .

The cumulative expected reward for the i th agent can be used as an objective function of strategy optimization:

9

where, is the global observation vector, represents the observation of the agent.

Then, the objective function can be optimized through the method of solving the strategy gradient.

10

where, is the deterministic action strategy function, represents the network parameter of the deterministic action strategy function, anddenotes the state-action value n.

The evaluation network employs a centralized update mechanism and utilizes Temporal Difference (TD) error to refine the network’s updates. The loss function for the Critic network’s updates can be articulated as:

11

where, represents the target action value function of the i-th agent, denotes the current Critic value network parameter, and represents the target Critic network parameter.

The value of the can be easily overestimated due to the approximate error inherent in the value function. When evaluating the strategy generated by the Actor using an overestimated action value function, this leads to an overestimation of the current action’s value. To address this issue, the evolutionary network incorporates an additional set of Critic networks. These multiple Critic networks evaluate the action value simultaneously and choose the smallest value for updating, thereby mitigating the impact of overestimation.

When calculating the Temporal Difference (TD) error, the updated target action value can be expressed as:

12

Furthermore, as the deterministic strategy is prone to the approximation error of the function, the evolutionary strategy imposes regularization on the target strategy network. The target action is approximated within a narrow range by introducing a micro-noise that adheres to a normal distribution.

13

where, is the micronoise, b is the boundary value of the noise.

By smoothing the target action value on similar actions, regularization can solve the possible incorrect spikes in the deterministic strategy gradient and avoid fragile behaviors in policy networks.

• (B)Safe recovery strategy.

The safety of actions is a crucial consideration for multi-robot systems. Traditional evolutionary strategies incorporate only reward information regarding actions, yet they fail to account for safety considerations. This deficiency often leads to overly aggressive strategies that can place multi-robot in perilous situations. To address this issue, this section proposes the construction of a secure recovery network that incorporates safety constraints. As depicted in Fig. 4, the secure recovery network integrates a Lyapunov evaluation network into the multi-agent distributed Actor-Critic architecture. This addition is designed to evaluate the expected cost value of each robot action, thereby determining the associated risk level.

Fig. 4.

Multi-agent network architecture with security constraints.

The Lyapunov evaluation function is a conceptual function whose value depends on the corresponding action-state vector. The Lyapunov state function can be obtained from the Lyapunov action evaluation function:

14

The value of Lyapunovevaluation function is calculated in a similar way to the action-states value . The Belman equation is still satisfied between the current generation values and the future generation values:

15

where, is the immediate non-negative security constraint function. denotes the target Lyapunov evaluation function.

The target Lyapunov evaluation function also serves to reduce bias that may arise from the overestimation of evaluation values. Instead of iteratively updating the network parameters, the target network merely replicates a subset of weights recently updated by the current network, a process referred to as delayed updates.

Network architecture optimization

• (A)Policy network optimization.

To update the policy network, the objective function needs to be maximized. Based on the boundary security constraint, the objective function to maximize can be expressed as follows:

16

where, m is the safety constraint threshold.

To streamline the resolution of Eq. (16), the policy optimization subject to security constraints is recast as a multi-objective optimization problem, leveraging the Lagrange multiplier technique. When the agent is in a non-secure state:, a new objective function is built for the policy network. The function includes the action value function and the security cost function. The specific objective function is as follows:

17

where, is the Lagrange multiplier, k is the safety factor parameter, and is the non-safe experience buffer.

The updating process of the policy network can be viewed as a process of minimizing the objective function of the policy network. The gradient descent method is used to optimize the policy network, and the optimization result of the policy network Actor is obtained as follows.

18

where, represents in the current strategy.

Based on the appropriate step size, the policy parameter is updated along the gradient direction of the policy network, so as to ensure that the policy at the next time is not worse than the previous time until convergence.

Then, the objective function for the Lagrange multiplier is constructed as follows.

19

The Lagrange multiplier utilizes the gradient ascent methodology to continuously calibrate in tandem with the network’s iterative updates, ensuring the robustness of the strategic framework.

• (B)Critic network update optimization.

The Critic network refines the policy network’s weights by minimizing the loss function via gradient descent. Consequently, a parameter is identified that maximizes the expected value function.

The Critic network retrieves samples from the global experience replay buffer and optimizes them in accordance with the Temporal Difference (TD) error of the state-action function. Its loss function is expressed as follows:

20

where, represents the number of samples drawn by each batch from the safety memory buffer, is the target value of the action value .

• (C)Lyapunov safe network optimization.

The Lyapunov safe network undergoes optimization through the application of the gradient descent method. The loss function for the Lyapunov network is articulated as follows:

21

where, U_D represents the set of dangerous states, denotes the Lyapunov cost function, and is the target value of the Lyapunov cost function. indicates the number of samples drawn from the dangerous states per batch.

The target value of the Lyapunov cost function is calculated from the current cost and the Lyapunov cost function of the next state :

22

The Critic network and the Lyapunov safe network undergo parallel training and synchronous updates. However, they differ in their sampling methods: the Critic network selects samples from the secure states subset, whereas the Lyapunov safe network selects from the non-secure states subset.

During training, the target network parameters for the Actor, Critic, and Lyapunov networks are not immediately updated. Instead, they experience delayed updates. The weight parameters are replicated from their respective current networks, and only a fraction of the change values are updated in each instance:

23

, , represent the soft update factor of the weight parameter. A smaller update factor can optimize the target network delay and prevent the unstable bias caused by the overestimation.

Uniformly ultimate boundedness stability analysis

System stability is a critical prerequisite for robot motion control. Classical control methods often utilize Lyapunov theory to demonstrate the stability of the control system. However, for data-driven reinforcement learning (RL) motion control systems, the traditional Lyapunov stability analysis method is not applicable. Among various stability analysis methods, uniformly ultimate boundedness (UUB) has been proven effective⁴¹. A UUB-stable autonomous decision-making strategy ensures that action vectors approach an equilibrium state within a limited number of steps. Uniform ultimate bounded stability (UUB) and safety reinforcement learning constraints are integrated. Multi-robot systems must learn a reinforcement learning controller with a UUB guarantee to tackle autonomous decision-making tasks involving safety constraints. Each update of the controller meets the UUB criteria and incrementally enhances the cumulative reward; The Lyapunov cost function continuously diminishes, eventually dropping below a threshold. When faced with potential danger, the robot proactively moves towards a safe area, enabling the robot to recover from the hazard within a limited timeframe.

UUB stability definition for safe reinforcement learning

This section broadens the classical definition of uniformly ultimately bounded (UUB) stability to encompass a more general scenario. We reevaluate the concept of UUB stability within the context of safe reinforcement learning, specifically to tackle the challenges associated with multi-robot motion control tasks.

The definition of the UUB stability theorem is as follows:

Theorem 1

If a control system has positive constants b, m and , there exists , such that ,, Then, this system is said to be uniformly ultimate bounded about .

Leveraging the stability assurance provided by UUB, the trajectory will enter and remain within the stability threshold range after a finite period of time. Subsequently, we will apply the uniformly ultimate boundedness method to verify the security of decision-making strategy under the constraints of Lyapunov safety. The stability of the Lyapunov safety constraint function is defined based on Theorem 1.^41,42.

Theorem 2

If there is a function a Lyapunov function and positive constants , such that:

24

Then, the motor control system is uniformly ultimately bounded with boundary .

Moreover, the expected value is bounded during the N steps, and also satisfies:, .

Remark 1

Represents the Lyapunov cost function, which guarantees the stability of the control strategy. The Lyapunov cost function can be calculated from the Lyapunov action-value function :

25

In Eq. (24), means the average distribution of s in N steps.

26

where, N represents the maximum moment when the probability of a hazardous sample in the sample set is greater than zero, represents the state with the N time step.

The is used to determine whether the current state s belongs to the unsafe set:

27

where,

UUB stability proof for safe reinforcement learning

First, it is imperative to demonstrate conclusively that N is a finite constant. We employ the counterexample method to postulate that N is infinite^39,41,42, with the intention of deriving a contradiction, such that:

28

The first term on the left of the inequality in Eq. (24) can be rewritten as:

29

Since is finite, it is obtained when N tends to an infinite value.

30

According to the known condition , the Eq. (29) can be arranged as:

31

Combination Eqs. (30) and (31), it can be obtained:

32

The second term on the left of the inequality in Eq. (24) can be rewritten as:

33

By comparing Eqs. (32) and (33), it can be concluded that if both inequalities are satisfied, the result will be contrary to the reality of Eq. (24). Therefore, our assumption about with is a pseudo-proposition. So it can be concluded that there exists a finite N such that for any .

The proof that the expectation of is bounded after N time steps will be continued next. Since it is a finite value of N has been proved, it can be obtained:

34

Collating Eq. (34) gives:

35

After the decomposition of the left term of the inequality Eq. (35), there exists an arbitrary moment of time such that.

36

By definition in Theorem 2, Eq. (36) can be expressed as:

37

The probabilistic choice of samples for each space in the defined sample space can be described as:

38

The expected value of the cost function can be expressed as:

39

Combining Eqs. (38) and (39), the expected value of the cost function can be further expressed as:

40

According to inequality (40), the expectation value of the cost function is bounded in a finite number of steps N and satisfies, . Thus, the reinforcement learning motion control system has been demonstrated to be uniformly ultimately bounded (UUB) stable.

Upon transitioning into an unsafe condition, the robot possesses the capability to navigate out of peril within a specified and finite sequence of N steps, thereby ensuring that the cost function value descends beneath the established safety threshold.

Experiment

Initialization setting

In this section, we present the performance evaluation of the proposed algorithm across two distinct environments. The experimental setup encompasses a standardized scenario known as Safety-Gymnasium and a custom-built scenario. Safety-Gymnasium is a highly modular, readable, and customizable benchmark environment library, constructed upon the MuJoCo physics engine. It serves primarily as a platform for evaluating the performance of safe reinforcement learning algorithms. Conversely, the custom-built scenario is developed using CoppeliaSim, a professional robot physical simulator, and is designed to facilitate the training and testing of multi-robot cooperative safe reinforcement learning algorithms within unfamiliar settings. Throughout the experimentation, the network hyperparameters for the UBSRL algorithm, as introduced in this paper, are specified in Table 1.

Table 1.

Network parameters for the UBSRL algorithm.

Parameters	Instruction	Values
l a	Learning rate of actor	0.0001
l c	Learning rate of critic	0.0001
l s	Learning rate of Lyapunov	0.0001
Batch_size	Sample size	512
γ	Discount factor	0.99
τ1, τ2, τ3	Soft update factor	0.01
UD, Us	Experience buffer capacity	10^6

Experiments in standardized scenarios

The Safety Gymnasium⁴³ encompasses three standardized scenarios: Circle, Goal and Button, as depicted in Fig. 5.

Fig. 5.

Safety-Gymnasium safe reinforcement learning scenarios.

In the Circle scenario, the robot is tasked with navigating in a circular path around the central region while ensuring it remains within the safe range. The green zone represents this safe area, and the robot’s starting point is randomly determined within the circular zone for each training session. In the Goal scenario, the robot must navigate to a target position while avoiding numerous obstacles scattered throughout the scene. Upon reaching the target, the goal is randomly relocated to a new position, though the overall layout of the scene remains constant. The Button scenario presents multiple goals, and the robot’s objective is to reach each goal sequentially until it has visited them all.

In the standardized experimental scenarios, a comprehensive series of experiments were conducted using the proposed UBSRL method, as well as two other secure reinforcement learning methods, CPO and L-PPO. The evaluation criteria included episode reward values and round cost values. The training goal was to maximize the reward while ensuring the cost remained at a minimum. Within the bounds of the security constraints, the higher the reward value, the better the performance. The fluctuations in reward values and security cost values for each method across the three scenarios are depicted in Figs. 6 and 7.

Fig. 6.

Cost values comparison in standard scenarios.

Fig. 7.

Reward values comparison in standard scenarios.

Figure 6 depicts the evolution of cost values for the three methods under consideration. On the other hand, Fig. 7 depicts the variation in the reward curves of these methods. Observing the figures, it becomes evident that the proposed UBSRL method satisfies the safety constraints in all three scenarios. Furthermore, the safety cost value has diminished beneath the safety threshold, denoted by the dotted line, after 400 episodes of training. This outcome is attributed to the significant role played by the Lyapunov safety constraint network during the training process, signifying that the robot’s action decisions comply with safety performance requirements. Turning our attention to Fig. 7, we can discern that the proposed method’s episode reward value exhibits clear superiority over the two comparative methods. After 450 episodes of training, it tends to stabilize near the peak value, with the maximum reward being markedly higher than the other methods, and with minimal fluctuations. In conclusion, through lateral comparison, it becomes apparent that the two comparative methods, CPO and L-PPO, fail to confine stable actions within the safety threshold. Their strategies oscillate above this threshold, leading to lower reward values and unstable convergence.

To further elucidate the rationality and efficacy of the proposed network architecture, we endeavored to streamline the evaluation network of UBSRL by substituting the dual evaluation network with a singular one, and subsequently termed this refined approach Simplified-UBSRL. Building upon this, additional experiments were executed within the Safety-Gymnasium safety reinforcement learning environment, where we compared the algorithm’s loss function and task failure rate variations. The outcomes are depicted in Fig. 8; Table 2.

Fig. 8.

Comparison for the loss function.

Table 2.

Comparison of Task failure rates in different scenarios.

	Circle	Goal	Button
UBSRL	1.5%	2.1%	3.6%
Simplified-UBSRL	3.5%	4.8%	5.5%

The research results indicate that in three distinct experimental scenarios, network simplification resulted in a substantial rise in the volatility of the loss function, hindering convergence. This could potentially compromise the robot’s capacity to make prompt and accurate decisions. A deeper analysis reveals that when contrasting the task failure rates across various scenarios, the probability of the robot colliding or encountering danger escalated notably, with the task failure rate surging by over 2%.

Consequently, it is concluded that network simplification did not effectively improve the robot’s motion control efficiency. The proposed method demonstrates a more stable security performance in the context of standardized secure reinforcement learning scenarios.

Experiments in multi-robot scenarios

In this section, we aim to evaluate the safety performance and the efficacy of cooperative decision-making among multi-robot. To this end, we have constructed several representative motion scenarios using the robot physics simulator CoppeliaSim, as depicted in Fig. 9.

Fig. 9.

Safe reinforcement learning environments for multi-robot.

These scenarios encompass a range of robots maneuvering through both dynamic and static obstacles. Following extensive experimentation, we compare and analyze the security performance of the collaborative capabilities of our proposed UBSRL method against those of other reinforcement learning approaches. As depicted in Fig. 9, scenario 1 encompasses a multi-robot formation navigation undertaking, characterized by a circular domain designated as a safe area. Each robotic entity is tasked with maintaining a secure distance from its peers and traversing the circular perimeter in a uniform clockwise fashion.

The cumulative reward value for the entire multi-robot system can be articulated as the aggregate sum of the individual rewards attained by each constituent robot.

41

The reward for a single robot can be specifically expressed as follows:

42

where, is the regional center position, is the current position of the ith robot, and represents the angle deviation.

The cost of a robot includes two parts: movement state cost and environmental cost, which can be specifically represented as:

43

where, represents the action space of the robot, is the movement state cost, denotes the environmental cost.

Scenario 2 involves a cooperative target search task for a team of robots. These robots must work together to gather the maximum number of green targets, all the while steering clear of red danger zones and obstacles, until they have achieved full coverage of all targets. A robot is rewarded for reaching the green zone and incurs a penalty upon entering the red zone. The reward that a robot earns can be articulated as follows.

44

where, is the immediate reward, denotes the environment reward, and is the goal arrival reward.

The calculation method for robot safety costs is similar to scenario 1, covering both motion state costs and environmental costs.

45

In Scenario 3, several robots are required to navigate to the target area while maintaining a specific formation. This scenario is riddled with obstacles and hazardous zones, and the robots’ goal positions are subject to real-time changes. Throughout the training process, the unmanned robots’ rewards can be articulated as follows.

46

where, is the formation reward.

In order to ensure that the multi-robot systems maintain a certain formation, the formation reward is set the as:

47

where, are the adjustment coefficient, are the real-time distance of the formation and is the preset distance of the formation.

The safety cost includes three parts: motion state cost, environmental cost, and collision cost. The specific expressions are as follows:

48

where, denotes the movement state cost, signifies the environmental cost, and represents the formation deviation cost. When the robot strays from the intended formation sequence,

Multi-robot autonomous decision-making experiments were conducted across three distinct scenarios, utilizing the UBSRL, CPO, and SAC methodologies respectively. Each episode was allotted a maximum of 500 training time steps. The evolution of the average episode reward and the average episode cost function across these scenarios is depicted in Figs. 10 and 11.

Fig. 10.

Reward values in a multi-robot scenarios.

Fig. 11.

Cost values in a multi-robot scenarios.

As illustrated in Fig. 10, the methodology presented in this paper demonstrates consistent stability within scenario 1. Up until the 3000th iteration, the reward value surged rapidly, surpassing those of the other two methodologies. Following the 3000th episode, the reward value experienced a slight decline as the robot started to exit the zone of highest reward and opted for a more conservative strategy, influenced by the safety Lyapunov function’s actions. A similar pattern is observed with the CPO methods, which can be attributed to the constraining impact of the cost function. The fluctuation of reward and substitution values in scenarios 2 and 3 mirrors that of scenario 1. The UBSRL method achieves a state of convergence in reward value after 2000 to 3000 training iterations, outpacing the CPO security reinforcement learning method, yet marginally falling short of the SAC method. Despite the SAC algorithm achieving the highest reward value, it suffers from the lowest success rate due to insufficient safety constraints. Figure 11 reveals that the cost value of the UBSRL method proposed herein is lower than that of the CPO method, suggesting that UBSRL exhibits superior safety constraint performance compared to the CPO method.

In scenario 3, the robots must continuously adapt their postures to maintain a stable formation. Consequently, to evaluate the efficacy of the proposed method, UBSRL, in enabling autonomous formation of multi-robot, the alterations in the average speed and inter-vehicle distances were experimentally examined, as depicted in Fig. 12.

Fig. 12.

Motion performance indicators of multi-robot scenarios.

Figure 11 illustrates the comparison of multi-robot formation states, including (a) the variation of formation velocity among multi-robot systems, and (b) the variation of formation distance among multi-robot systems. As depicted in Fig. 12, the velocity of the robots exhibits significant fluctuations in the initial stages, struggling to stabilize around the desired speed and distance. However, as training continues, these velocity fluctuations start to diminish. By approximately 4000 episodes, the robots’ velocity becomes essentially stable, maintaining the set speed. A similar pattern is observed in the distance variation.

The experimental statistics, which summarize the task success rates of three algorithms, are presented in Fig. 12c. The success rate is a crucial metric for evaluating the effectiveness of decision-making system. Figure 12c reveals that the proposed method achieves the highest task success rate, whereas the SAC method yields the lowest. When comparing scenarios, due to the high task complexity across the three scenarios, scenario 3 has the lowest task success rate, influenced by the combined effects of safety constraints.

To conduct a more thorough comparison of the performance efficacy of the method introduced in this paper, we incorporated new safety reinforcement learning methods as benchmarks, including: CPO⁴⁸, CVPO⁴⁹, and STR⁵⁰. Initially, we compiled the distribution of task failure rates across diverse scenarios, as illustrated in Table 3.

Table 3.

Failure rate comparison in different scenarios.

Scenario	Failure rate			Risk rate			Timeout rate
	1	2	3	1	2	3	1	2	3
UBSRL	0.5%	1.2%	1.5%	0.4%	0.5%	0.8%	0.1%	0.7%	0.7%
CPO	2.2%	3.0%	4.5%	1.8%	1.4%	2.8	0.4	1.8	1.7
CVPO	2.5	3.2%	4.7%	1.9%	1.6	3.2	0.6	1.6	1.5
STR	1.8%	2.6%	3.2%	1.4	1.1	2.1	0.4	1.5	1.1

The task failure rate comprises two components: the danger rate and the timeout rate. The danger rate pertains to the likelihood of the robot encountering obstacles or venturing into hazardous zones. The timeout rate signifies the probability of the robot failing to achieve the designated goal within the prescribed timeframe. The outcomes indicate that UBSRL exhibits the lowest failure rate, danger rate, and timeout rate across all scenarios, demonstrating the utmost stability in performance. In contrast, CPO and CVPO display relatively high failure rates, particularly in scenario 3, where CVPO attains the peak failure rate of 4.7%. The performance of STR lies between UBSRL and CPO/CVPO, with both its failure rate and danger rate being relatively low. However, in scenario 2, its timeout rate is marginally higher than UBSRL. These results can verify that our proposed UBSRL performance performs better than the contrast algorithm.

In addition, the task success rate and average time of the robot were calculated, as shown in Figs. 13 and 14. As can be seen from the results, our task success rate was the highest among the three scenarios. In terms of the average time consuming to complete a task, it can be seen that scene one is relatively simple and the task takes the least time. Scenario 2 requires more tasks to explore and takes the most time. By comparing with other methods, it can be found that the average time consumption of our proposed method is the least, indicating that the closer the selected strategy is to the optimal, and the performance of the self-strategy is better than other methods.

Fig. 13.

Comparison of success rates.

Fig. 14.

Comparison of average time.

Furthermore, the task success rates and average time of the robots were computed and are depicted in Figs. 13 and 14. The outcomes reveal that the task success rate of UBSRL was the highest across the three scenarios. Regarding the average time required per episode, it is evident that scenario 1 is the simplest, resulting in the shortest task duration.In scenario 2, the robot must fulfill a multitude of exploration goals, thus necessitating the greatest time expenditure. Upon comparison with alternative methods, it becomes evident that our proposed method exhibits the lowest average time consumption, and the performance of the strategy surpasses that of other methods.

To ascertain the efficacy and resilience of the UBSRL, supplementary experiments were executed within more intricate and dynamic scenarios. The experimental paradigm encompasses two types dynamic scenarios of multi-robot collaborative interaction. As depicted in Fig. 15a illustrates a multi-robot cooperative search scenario, which incorporates a variety of obstacles and pedestrian. The robots are tasked with identifying the closest target and navigating through the impediments to attain the objective. Should a robot encounter an obstacle or fail to reach the target within the stipulated timeframe, the mission is deemed unsuccessful. Figure 15b portrays a multi-robot cooperative pursuit scenario involving 3 pursuers and 1 evader. The pursuer robots are required to diligently encircle the evader while circumventing all dynamic and static obstacles.

Fig. 15.

Multi-robot collaborative interaction scenarios.

Upon completing 10,000 episodes of training experiments within dynamic environments, a comparative analysis of reward and cost values was conducted for various safety-focused reinforcement learning strategies. The outcomes are depicted in Figs. 16 and 17.

Fig. 16.

Comparison of reward values in different scenarios.

Fig. 17.

Comparison of cost values in different scenarios.

As illustrated in Fig. 17, the initial phase witnessed a rapid learning efficiency across all algorithms, with a swift increase in reward values. This indicates that all methodologies swiftly adapt and learn effective strategies during the early stages of training. UBSRL exhibited a marginally superior trend at this juncture, demonstrating its efficacy in the early learning phase. The UBSRL sustained high reward values in subsequent stages, showcasing commendable long-term performance and stability. The STR also maintained a relatively consistent performance in later stages, albeit with reward values slightly below those of UBSRL. In contrast, the reward values for CPO and CVPO experienced significant fluctuations in the later stages and exhibited lower overall levels compared to UBSRL and STR, suggesting that both algorithms may be somewhat deficient in stability and efficiency when dealing with long-term learning tasks.

Figure 17 presents the cost curves of the four methods varying with the number of training Episodes. During the nascent stages of training, the cost values for all methods are notably elevated, suggesting that the preliminary learning phase is characterized by exploratory behavior and the quest for effective strategy acquisition. The UBSRL methods exhibits the most precipitous decline in cost during this initial phase, underscoring its superior efficiency in the early learning stages. Furthermore, UBSRL sustains the lowest cost metric in subsequent phases, signifying its robustness and efficiency in enduring learning processes. The STR algorithm also demonstrates a tendency towards stabilization in later stages, albeit at a marginally elevated cost level compared to UBSRL. Conversely, CPO and CVPO algorithms experience more pronounced cost fluctuations in the later stages, and their overall cost levels are higher than those observed for UBSRL and STR. In the context of complex dynamic collaborative scenarios, the UBSRL algorithm demonstrates superior performance relative to the other three algorithms, particularly in the early and long-term learning phases.

Upon analyzing the task success rates, reward values, and cost values of the methodologies in variety of scenarios, it becomes evident that the UBSRL method presented in this paper demonstrates superior security and stability performance within complex, unsafe environments. The UBSRL method showcases commendable adaptability across various scenarios.

Conclusion

This paper introduces a novel hierarchical safe reinforcement learning strategy, grounded in uniformly ultimately bounded constraints, that effectively addresses the issue of insufficient safety in autonomous decision-making processes. This strategy is constructed on the foundation of an event-triggered distributed deep deterministic policy gradient framework, and offers a safety guarantee for the multi-robot systems. Furthermore, by integrating the Lyapunov security cost network, an optimization objective for with multiple conditional constraints has been developed, and the calculation process has been streamlined through the application of the Lagrange multiplier technique. Ultimately, the stability of the proposed strategy is analyzed employing the uniformly ultimately bounded theory, ensuring that the state trajectories of robots can be restored to safe states within a finite time frame. Experimental results in multi-robot scenarios demonstrate that the ultimately bounded constrained hierarchical safety reinforcement learning strategy (UBSRL) outperforms existing safety reinforcement learning approaches in terms of task execution efficiency and decision security, providing a theoretical foundation for multi-robot collaborative decision-making systems.

In the future, we aim to extend our UBSRL method to flexible DRL network architectures based on dynamically evolving networks in more complex and variable task environments, and investigate how to enhance the adaptability and robustness of multi-agent reinforcement learning Autonomous decision-making strategy.

Author contributions

H.S., H.J., L.Z., C.W. and S. Q. conceived and designed research; H.S. and H.J. performed experiments; L.Z. and C.W. analyzed the data; All authors interpreted the results; S. Q. helped with methodology and prepared figures; H.S. drafted manuscript; All authors edited and revised manuscript; All the authors agreed with the results and conclusions of the manuscript.

Data availability

The datasets used and analysed during the current study available from the corresponding author on reasonable request.

Declarations

Competing interests

The authors declare no competing interests.