Attention-based multi-agent reinforcement learning for traffic flow stability in mountainous tunnel entrances

Abstract

This study proposes a Multi-agent Fusion Double-Dueling-Deep Q-Network Traffic Flow (MF3DQN-TF) for Connected and Autonomous Vehicles (CAVs) in mountain tunnel entrance sections, considering nonlinear coupling effects. Complex road conditions and nonlinear coupling effects of tunnel exit flow often cause unstable traffic flow there, impacting traffic efficiency and safety. The new framework, combining multi - agent deep reinforcement learning and attention mechanisms, has shown marked improvements over traditional rule - based regulation methods in various traffic scenarios through comparative experiments. Actual simulations indicate it can boost traffic flow stability by over 15%, vehicle efficiency by about 20%, and cut congestion time by 18%. Specifically, it enhances average vehicle speed by 25% and reduces the traffic congestion index by 22% compared to conventional methods. The attention mechanism improves intelligent agents’ decision - making efficiency, enabling real - time vehicle interaction and coordination optimization, thus enhancing intelligent transportation systems’ overall adaptability. The framework helps stabilize traffic flow and ease common traffic issues at mountain tunnel entrances, strongly supporting intelligent transportation system development.

Introduction

Traffic flow stability at mountainous tunnel entrances has long posed a significant challenge in transportation engineering^1–4. Unlike ordinary road sections, tunnel entrances combine sharp geometric transitions, slope variations, and limited visibility, which amplify disturbances and lead to unstable or congested traffic flow^5,6. Conventional traffic flow models, such as the Lighthill–Whitham–Richards (LWR) and Follow-the-Leader (FTL) models, often fail to capture these complex dynamics because they assume homogeneity and linear interactions among vehicles⁷. As a result, they lack the capacity to address nonlinear disturbances and cascading instabilities that frequently occur in tunnel entrance scenarios⁸.

To overcome these limitations, it is crucial to develop models capable of capturing the adaptive and interactive behaviors of modern Connected and Autonomous Vehicles (CAVs)^9,10. Multi-agent deep reinforcement learning (MADRL) is particularly suitable for this task because it enables vehicles (agents) to learn optimal driving strategies through interaction with the environment and with other vehicles¹¹. Unlike rule-based control, MADRL supports self-optimization in dynamic, high-density conditions and can incorporate complex multi-vehicle dependencies. Furthermore, integrating an attention mechanism allows agents to prioritize critical environmental cues—such as sudden changes in headway or density—thereby enhancing decision-making efficiency under real-time constraints. These characteristics make MADRL with attention a promising approach for stabilizing traffic flow in complex environments such as mountainous tunnels.

Based on this rationale, this study proposes a novel framework—Attention Multi-agent Fusion Double-Dueling-Deep Q-Network Traffic Flow (MF3DQN-TF)—which integrates multi-agent deep reinforcement learning with attention mechanisms and explicitly models nonlinear coupling effects in traffic flow. The main objectives are to: (1) enhance traffic stability at tunnel entrances, (2) improve vehicle efficiency and reduce congestion, and (3) evaluate the framework’s performance across multiple dimensions, including stability, efficiency, and safety.

The key contributions of this study are summarized as follows:

A novel framework (Attention MF3DQN-TF) is proposed by integrating multi-agent deep reinforcement learning with an attention mechanism. This framework enables Connected and Autonomous Vehicles (CAVs) to adaptively optimize driving strategies in complex traffic environments.

Explicit modeling of nonlinear coupling effects among vehicle speed, headway, and density at mountainous tunnel entrances is conducted. This modeling provides a more accurate representation of instability mechanisms compared with conventional linear traffic flow models.

Comprehensive multi-dimensional evaluation is performed, covering stability, efficiency, congestion reduction, and accident probability. Comparative experiments against baseline approaches confirm the superiority of the proposed method.

Practical implications for intelligent transportation systems are highlighted, demonstrating that the proposed approach offers a scalable and adaptive solution for stabilizing traffic flow in mountainous tunnel entrances and similar real-world scenarios.

The remainder of this paper is organized as follows. Part 1 (Introduction) presents the research background, motivations, and contributions. Part 2 (Related Work) reviews existing studies on connected and autonomous vehicles (CAVs), multi-agent reinforcement learning, attention mechanisms, and nonlinear traffic flow dynamics. Part 3 (Methodology) introduces the proposed Attention-based Multi-Agent Fusion Double-Dueling Deep Q-Network Traffic Flow (MF3DQN-TF) framework, including traffic flow modeling, agent behavior design, attention integration, and optimization strategies. Part 4 (Experiments and Analysis) describes the experimental setup, simulation scenarios, and evaluation metrics, followed by comparative experiments, ablation studies, and performance analysis under varying traffic densities. Part 5 (Supplementary Experiments) validates the generalization ability of the proposed framework using the NGSIM dataset and examines the contributions of individual modules. Finally, Part 6 (Discussion and Conclusion) summarizes the key findings, highlights practical implications, acknowledges limitations, and outlines directions for future research.

Related work

Research on connected and autonomous vehicles (CAVs)

CAVs represent a pivotal component of contemporary intelligent transportation systems, having garnered considerable research attention in recent years¹².The research domain of CAVs predominantly encompasses their collaborative control, behaviour prediction, and decision-making processes, particularly in complex traffic environments such as tunnels, intersections, and intelligent scheduling problems in high-risk areas, thereby underscoring their considerable potential for application¹³. Through V2X (Vehicle to Everything) communication technology and autonomous driving algorithms, CAVs can achieve real-time information exchange between vehicles and between vehicles and infrastructure, thereby improving the efficiency and stability of traffic flow while ensuring safety¹⁴.

In the research of collaborative control of intelligent connected vehicles, many scholars have explored how CAVs optimize traffic flow and reduce congestion through cooperation and information sharing between vehicles¹⁵. Yuan et al.¹⁶. have proposed a vehicle collaborative control strategy based on deep reinforcement learning, which can achieve more efficient speed adjustment and lane change in highways and urban roads, thereby improving the stability and efficiency of traffic flow. Furthermore, Liu et al.¹⁷. conducted research on the path planning of CAVs in complex environments, highlighting that the interaction between vehicles and infrastructure can effectively reduce the occurrence of traffic accidents and enhance the overall operational efficiency of the transportation system.

Research on the behavior prediction and decision-making capabilities of CAVs has become a significant focus in the field of intelligent transportation¹⁸. Intelligent connected vehicles are able to predict the behavior of surrounding traffic participants and proactively take measures to prevent potential traffic conflicts¹⁹. Zhang et al.²⁰ proposed deep learning–based behavior prediction models, which significantly improved the decision-making accuracy and response speed of CAVs in complex traffic scenarios. Accurate prediction of the behavior of preceding vehicles enables CAVs to perform complex maneuvers, such as obstacle avoidance and overtaking. This, in turn, ensures driving safety and reduces the probability of accidents²¹.

Intelligent scheduling and path planning are pivotal to the application of CAVs in high-risk areas such as tunnels and intersections²². The entrance section of tunnels often causes nonlinear coupling effects in traffic flow due to its special geographical and structural conditions, leading to traffic instability and congestion²³. Zhao et al.²⁴. proposed a novel method for scheduling traffic flow at tunnel entrances, with the basis of V2X communication. This method optimizes vehicle speed and spacing, reduces stagnation time, and alleviates traffic congestion at tunnel entrances through the exchange of real-time information such as vehicle position and speed.

In conclusion, research on intelligent connected vehicles provides new ideas and solutions for solving problems in complex traffic environments. The continuous development of both autonomous driving technology and V2X communication technology suggests that CAVs will play an increasingly important role in future intelligent transportation systems.

Deep learning research on multi agent systems and attention mechanisms

The continuous advancement of artificial intelligence technology has led to significant developments in multi-agent systems (MAS) and attention mechanisms, which have emerged as pivotal areas in the field of deep learning research. Multi-agent systems encompass systems comprising multiple autonomous agents that collaborate to accomplish intricate tasks through interaction and cooperation. Attention mechanisms emulate the human visual focusing process, assisting neural networks in honing in on critical components of input data and enhancing model performance²⁵. In recent years, significant progress has been made in the integration of multi-agent systems with attention mechanisms, particularly in the domains of complex task execution and coordinated decision-making²⁶.

In the field of multi-agent systems research, coordination and cooperation among agents represent a pivotal concern²⁷. Conventional multi-agent coordination methodologies predominantly depend on predefined rules or elementary cooperative algorithms. However, with the advent of deep learning, researchers have commenced exploration of more adaptable and flexible solutions²⁸. Wu et al. have proposed a multi-agent collaborative control method founded on deep reinforcement learning. This method optimizes vehicle speed and spacing, reduces stagnation time, and alleviates traffic congestion at tunnel entrances through the exchange of real-time information such as vehicle position and speed²⁹.Attention mechanisms are employed to focus on key environmental information or specific agents, enhancing task efficiency³⁰.The application of attention mechanisms in multi-agent systems, particularly in tasks involving information processing and resource allocation, has demonstrated considerable potential³¹. Qi et al. proposed a graph and attentive multi-path convolutional network (GAMCN) for traffic prediction, which integrates spatial modeling through a latent-position graph convolutional network (LPGCN) and temporal modeling via multi-path CNNs. By further incorporating time-aware attention to capture periodic traffic patterns, their model achieved significant improvements in both prediction accuracy and efficiency compared to state-of-the-art methods³². Furthermore, Chen et al. have developed a multi-agent decision-making system capable of long-term task planning and attention focusing by combining long short-term memory networks (LSTMs) with attention mechanisms, thereby significantly improving the stability and response speed of the system in dynamic environments³³.The role of the attention mechanism in multi-agent systems is twofold; firstly, it enhances individual decision-making ability, and secondly, it effectively promotes collaborative work among agents³⁴. In the context of group behavior and collective intelligence. Zhang et al.³⁵ proposed a Transformer and multi-graph GCN-based model (TDMGCN) for traffic flow prediction, which integrates a convolution-based multi-head self-attention module to capture both long-term temporal dependencies and local trend information. In the spatial dimension, the model employs spatial embedding and multi-graph convolution to dynamically extract complex correlations, achieving superior performance on multiple real-world datasets. The study indicates that attention mechanisms can effectively address the problem of information overload in multi-agent collaboration, thereby facilitating more efficient task completion³⁶.

This study hypothesizes that integrating multi-agent systems with attention mechanisms within deep learning research can offer new perspectives and approaches across multiple domains, including agent coordination, resource optimization, and behavior prediction. It is predicted that, as technology continues to develop, this field will become increasingly important in a variety of areas, including robot control, autonomous driving and intelligent manufacturing.

Study on traffic flow stability and nonlinear coupling effects

The stability of traffic flow has been a long-standing research focus in traffic engineering. In complex environments such as mountainous tunnels, unique geographical and structural conditions (e.g., slope changes, curved alignment, and restricted sight distance) often exacerbate instability. In these scenarios, nonlinear coupling effects frequently emerge and play a decisive role in shaping traffic dynamics.

In this study, nonlinear coupling effects are defined as the complex, non-proportional interactions among multiple traffic variables—such as vehicle speed, headway distance, traffic density, and acceleration—where changes in one factor amplify or suppress fluctuations in others, thereby producing unstable and fluctuating traffic states. Unlike linear interactions, where effects scale proportionally, nonlinear couplings can trigger chain reactions in traffic flow, amplifying small perturbations into large-scale instabilities.

Examples in tunnel entrance dynamics:

Example 1

When a vehicle decelerates suddenly due to slope variation or reduced visibility, the following vehicles may overreact with nonlinear braking behavior, producing oscillatory stop-and-go waves.

Example 2

Under high-density conditions at tunnel entrances, the interaction between reduced sight distance and small variations in headway can significantly amplify traffic disturbances, leading to congestion or even multi-vehicle collisions.

Such effects not only complicate traffic flow prediction but also heighten accident risk and congestion probability, thereby threatening both safety and efficiency^37–39.

Recent studies have modeled these mechanisms using multiphase heterogeneous traffic flow models and lattice hydrodynamic models incorporating disturbance transformation⁴⁰. These works confirm that nonlinear couplings among speed, density, and headway are key drivers of instability in tunnel and mountainous environments. Simulation-based approaches further reveal that traffic flow in such conditions often exhibits sudden fluctuations in speed and density, validating the critical role of nonlinear coupling effects^41,42.

The nonlinear coupling effect is inherently a composite phenomenon involving multiple interacting factors^43,44. Its complex dynamics render classical linear models (e.g., LWR or FTL) insufficient, necessitating more adaptive frameworks that can capture nonlinear feedback loops^45,46. Accordingly, researchers have proposed advanced control strategies and ITS solutions that leverage real-time monitoring and predictive models to mitigate these instabilities^47,48.

In summary, nonlinear coupling effects represent a crucial mechanism of traffic instability in mountainous tunnels. By explicitly modeling and addressing these effects, more reliable prediction, control, and optimization strategies can be developed. The proposed MF3DQN-TF framework in this paper integrates such nonlinearities into its design, offering an effective solution for enhancing traffic flow stability under complex real-world conditions.

Main content and innovation of this article

This study focuses on enhancing traffic flow stability at the entrance sections of mountainous tunnels for CAVs by integrating Multi-Agent Deep Reinforcement Learning (MADRL) with attention mechanisms. Mountainous tunnels represent a distinctive traffic environment characterized by high vehicle density and complex flow dynamics, and are significantly affected by factors such as terrain, traffic density, and vehicle speed. These factors contribute to strong nonlinear coupling effects, leading to fluctuations and instabilities in traffic flow^49,50. To address traffic instability at mountainous tunnel entrances, this paper introduces a framework named Multi-agent Fusion Double-Dueling-Deep Q-Network Traffic Flow (MF3DQN-TF). It integrates multi-agent deep reinforcement learning with attention mechanisms to boost traffic stability via intelligent behavioral decision-making.

The innovation of this article is mainly reflected in the following three aspects:

1) Attention based multi-agent deep reinforcement learning framework: This paper proposes a new multi-agent deep reinforcement learning framework that combines attention mechanisms to enable each intelligent connected vehicle to dynamically adjust its behavior strategy in complex traffic environments. The attention mechanism enables the model to pay more attention to key influencing factors in traffic flow, such as vehicle speed, distance, and traffic density, thereby improving the accuracy and intelligence of decision-making.

2) Constructing a traffic flow model for the entrance section of a tunnel, considering the influence of nonlinear coupling effects: In order to simulate the traffic flow state of the entrance section of a mountain tunnel in a realistic way, this paper adds the consideration of nonlinear coupling effects to the traditional traffic flow model. By comprehensively analyzing the interaction of factors such as vehicle speed, distance, and traffic density, a more realistic traffic flow model was constructed, providing a solid theoretical basis for subsequent intelligent decision-making.

3) Simulation experiments have verified the effectiveness and advantages of the method under different traffic flow densities. This paper validates the performance of the proposed MF3DQN-TF framework under different traffic flow density conditions through various simulation experiments. The experimental results show that this method can not only effectively improve the stability of traffic flow, but also significantly reduce traffic fluctuations and the frequency of traffic accidents in high-density traffic flow, providing strong support for traffic management in complex environments such as mountainous tunnels.

MF3DQN-TF framework construction and method design

System modeling

The entire system modeling of MF3DQN-TF framework is the core of the framework design. The system modeling that needs to be completed based on this framework mainly includes the following two parts^51,52. MF3DNQ-TF Framework Structure is shown in Fig. 1.

Fig. 1.

MF3DNQ-TF Framework Structure.

1) Traffic flow modeling of tunnel entrance section.

The traffic flow at the entrance to the tunnel is influenced by a number of factors, including vehicle speed, the distance between vehicles, and traffic density. These factors interact with each other to form complex traffic flow patterns^53,54. In order to more accurately describe the changes in traffic flow at the entrance to tunnels, this paper proposes a multidimensional traffic flow modelling method that combines continuous flow models and discrete flow models.

Continuous model of traffic flow: Based on the classic Lighthill Whitham Richards (LWR) model, the continuous changes of traffic flow are described by the relationship between traffic density and vehicle speed. At the entrance of the tunnel, the density and speed of traffic flow are key parameters. The LWR model can be expressed as the following Eq. (1):

1

Among them, is the traffic density in spatiotemporal coordinates, and is the corresponding vehicle speed at the location. This equation can describe the mobility of vehicles and the traffic flow characteristics of the tunnel entrance section.

Discrete model of traffic flow: Considering the possible nonlinearity and uncertainty in traffic flow, this paper adopts a discrete model based on micro vehicle behavior. The behavior of each vehicle is related to factors such as the distance between adjacent vehicles, vehicle speed, and traffic signals. Therefore, Eq. (2) and Eq. (3) can be used to represent the behavior of each vehicle:

2

3

Among them, and respectively represent the current position and speed of the vehicle, which are the acceleration determined by the driving strategy. By using these formulas, the state of each vehicle can be updated at discrete time steps, thus obtaining the evolution process of traffic flow.

The integration of these two models enables the article to provide precise descriptions of the dynamic fluctuations in traffic flow at the tunnel’s entrance under various traffic conditions, while taking into account external factors such as traffic signals, speed limits, and road conditions.

2) Intelligent Connected Vehicle Behavior Modeling.

The behavior of CAVs is marked by a high degree of autonomy, enabling them to dynamically adapt their driving strategies in response to surrounding environmental conditions and the actions of neighboring vehicles. In this study, reinforcement learning is applied to model the behavior of intelligent connected vehicles, treating each CAV as an autonomous agent. These agents perceive the surrounding traffic environment and optimize decision-making processes to enhance both the stability and efficiency of overall traffic flow.

State space: The state space of an intelligent agent contains key information in the environment, mainly including the current position, speed, acceleration, distance of surrounding vehicles, and traffic density of the vehicle. This state can be represented by Eq. (4):

4

Among them, represents the distance difference between the first vehicle and the preceding vehicle, and is the traffic flow density. This state space enables the agent to comprehensively perceive information in the environment and make reasonable decisions.

Action space: The action space mainly includes behaviors that the agent can perform, including acceleration, deceleration, lane changing, etc. It can be expressed as:

Accelerate, decelerate, maintain speed, change lanes to the left and right.

The purpose of each action selection is to optimize the current traffic state and maximize the traffic flow efficiency and stability of the system⁵⁵.

Reward function: The goal of reinforcement learning is to optimize the behavior strategy of the agent through the reward function. In order to encourage intelligent agents to maintain appropriate vehicle speed and distance at the entrance of the tunnel, the following reward functions have been designed. As shown in Eq. (5)

5

Among them, is the target vehicle speed, is the ideal distance between vehicles, and is the weighting coefficient, indicating the importance of controlling the vehicle speed and distance between vehicles. This reward function encourages the agent to make the vehicle speed and distance as close to the optimal values as possible, thereby ensuring the smoothness of traffic flow.

MF3DQN-TF framework design

The MF3DQN-TF framework is a traffic flow optimization model that integrates reinforcement learning with multi-agent collaboration, aiming to enhance traffic efficiency at tunnel entrances. In this framework, the behavior of CAVs is simulated through a multi-agent reinforcement learning (MARL) approach, where each intelligent agent adapts its driving strategy according to state information from the surrounding environment. By employing centralized training and distributed execution, the framework enables coordinated optimization at both the individual and collective levels. Furthermore, an attention mechanism is incorporated to dynamically adjust agents’ focus on critical environmental cues, thereby improving the overall stability and efficiency of traffic flow. The structure of the proposed Attention MF3DQN-TF framework is illustrated in Fig. 2.

Fig. 2.

Attention MF3DQN-TF framework structure.

The specific design implementation includes the following parts:

1) Multi agent environment settings.

In the MF3DQN-TF framework, each connected and autonomous vehicle (CAV) is modeled as an independent intelligent agent. These agents make action selections based on their perception of the surrounding traffic environment and exchange information through interactions with neighboring agents. Their decision-making is shaped not only by their own state but also by the behaviors of nearby agents. To enable collaborative decision-making, each agent dynamically adjusts its strategy through an information-sharing mechanism.

Assuming vehicles constitute a traffic flow system, the overall state of the system can be represented as a multidimensional vector. As shown in Eq. (6):

6

The behavior of each intelligent agent is represented by a discrete action in its action space. By utilizing centralized training and distributed execution strategies, the MF3DQN-TF framework enables each agent to achieve optimal behavior in the system, thereby improving the stability of overall traffic flow.

2) State and Action Space in Reinforcement Learning.

In the framework of reinforcement learning, each agent selects an action from its current state and updates its strategy based on environmental feedback. The design of the state space includes all key information of traffic flow, while the action space defines all behaviors that the agent can choose. Intelligent agents maximize accumulated long-term rewards by exploring different action strategies, ultimately achieving optimization of traffic flow.

State space: As mentioned earlier, the state space includes information from multiple dimensions such as the current position, velocity, acceleration, and traffic flow density of the vehicle. Its mathematical representation is as shown in Eq. (7):

7

Action space: The action space defines the set of behaviors that each agent can choose at each time step. Definition of Action Space, as shown in Eq. (8):

8

The above model defines the action space of the vehicle, including acceleration, deceleration, maintaining speed, changing lanes to the left, and changing lanes to the right.

The definition of state transition for the “acceleration” action is as shown in Eq. (9):

9

In this formula, when the agent performs an acceleration action at a time step, the state at the next moment is determined by the current state and the acceleration action.

The definition of state transition for the “deceleration” action is as shown in Eq. (10):

10

This formula represents that when the intelligent agent performs a deceleration action at a time step, the state at the next moment is determined by the current state and the deceleration action.

The state transition definition for the action of “changing lanes to the left” or “changing lanes to the right” is as shown in Eq. (11):

11

In this formula, when the agent chooses to change lanes to the left or right, the state at the next moment is  determined  by the current state and the lane change action.

3) Integration of Attention Mechanisms.

In multi-agent systems, attention mechanisms can help agents dynamically adjust their focus on different traffic information during decision-making. By introducing attention mechanisms, this article can adjust the decision weights of each agent to focus more on the most important traffic environment factors, thereby improving the stability and coordination of the overall system.

The core idea of attention mechanism is to assign different weights to the input information of the agent based on the different states of the environment. Specifically, given a state, calculate the attention weights of each agent for different state information using the following Eq. (12):

12

Among them, is the weight matrix, which represents the agent’s attention to the state at a time step. Through this mechanism, intelligent agents can dynamically adjust their behavior decisions based on the complexity of the current environment, thereby enhancing the stability of traffic flow. The attention mechanism enhances decision-making by prioritizing critical information. The design points of Algorithm 1 are as follows.

Algorithm 1

Attention Weight Computation.

Learning and optimization strategy design

This study proposes a learning and optimization strategy based on the policy gradient method to enhance traffic flow stability at mountainous tunnel entrances through multi-agent reinforcement learning. The policy gradient approach is employed to directly optimize agent policies, and is combined with deep Q-learning to improve learning efficiency in high-dimensional environments. During training, experience replay and target network techniques are applied to ensure stability and convergence, while attention mechanisms are incorporated to strengthen inter-agent coordination. Together, these strategies significantly enhance system performance and effectively address the challenges posed by nonlinear coupling effects.

This study puts forward a learning and optimization strategy design based on the policy gradient method with the aim of enhancing the traffic flow stability of the entrance section of mountainous tunnels through multi-agent reinforcement learning. The strategy gradient method is adopted in order to directly optimize the agent’s strategy, in combination with deep Q-learning to enhance learning performance in high-dimensional environments. During the training process, experience replay and target network techniques were used to improve the stability and convergence of the training, while attention mechanisms were introduced to enhance the coordination between agents, ultimately achieving a significant improvement in system performance and addressing the challenges brought by nonlinear coupling effects. This includes the following two aspects:

1) Strategic gradient method.

In multi-agent systems, the goal of reinforcement learning is to derive an optimal strategy through continuous interactions between agents and their environment, thereby maximizing long-term returns. To support the training of agents’ decision-making behavior, this study employs the Policy Gradient (PG) method, which directly optimizes the policy function and avoids common issues in value function estimation. The core idea of the policy gradient approach is to refine agents’ strategies so as to maximize their expected long-term rewards. The policy gradient method is a specific type of machine learning algorithm that is used to approximate the optimal policy. This is achieved by updating the parameters (theta) of the policy network of the agent. This, in turn, enables the agent to select the optimal action in the current state. The update formula for the strategy gradient is as shown in Eq. (13):

13

Among them, is the objective function, representing the probability distribution of the agent choosing an action (a_t) in the state, and is the cumulative reward from the beginning to the end of the time step. This formula represents optimizing the strategy through gradient ascent, so that the actions selected by the agent can maximize the expected reward in the future.

On the basis of the above, this article combines Deep Q-Learning with policy gradient methods and extends it to optimization in high-dimensional state spaces and complex environments. This article can describe this process by introducing more formulas. The following is the expanded formula and corresponding explanation:

The standard model for updating the Q-value function is defined as shown in Eq. (14);

14

Among them, represents the Q value of executing actions in the state, is the immediate reward, (gamma) is the discount factor, is the Q value of selecting the best action in the next state. This formula is used to update the Q-value function, adjusting the agent’s strategy through rewards and expected future returns.

The definition of Experience Replay is Eq. (15):

15

Experience replay stores the experiences collected by the intelligent agent during its interaction with the environment. Each experience is a tuple that contains the current state, the action performed, the immediate reward received, and the next state reached after performing the action. During each update process, a small batch of samples are randomly selected from the experience replay pool for training to increase sample diversity, break data correlations, and improve learning efficiency and stability.

The following model is used to describe Deep Q-Network (DQN), as shown in Eq. (16):

16

In deep Q-learning, the Q-value function is approximated by a neural network, where represents the parameters of the neural network. DQN estimates the Q-value function by optimizing network parameters to minimize the objective function. The optimization objective is usually to minimize the following loss function, as shown in Eq. (17):

17

Among them is the Q-value function in the target network, which is a delayed updated version of the Q-network used to improve training stability.

Finally, this article also designed a combined strategy gradient and Q-value function model, as shown in Eq. (18):

18

When combining the policy gradient and Q-value function, this article updates the policy parameters through the above model. Here, represents the policy function, which is the logarithmic gradient of the policy, the target value of the current state action pair, and the currently estimated Q value. This formula combines the strategy gradient method with Q-value function updates, allowing agents to learn strategies more effectively in high-dimensional state spaces and complex environments.

Through the design of the above strategies, this article combines deep Q-learning and policy gradient methods to better learn and optimize in complex environments, improving the performance of intelligent agents. In high-dimensional state spaces, deep Q-networks (DQNs) approximate Q-value functions through neural networks, while the introduction of empirical replay mechanisms and target networks helps improve learning efficiency and stability.

2) Training Process and Optimization.

In the MF3DQN-TF framework, training is conducted through multi-agent interactions, where each agent continuously refines its strategy via repeated engagement with the environment. The training data is generated from a simulated setting that replicates traffic flow at mountainous tunnel entrances, incorporating nonlinear coupling effects and vehicle-to-vehicle interactions. To ensure stability and convergence, multiple training rounds are performed, with optimization of network architecture and hyperparameters, including learning rate, network depth, reward function design, and discount factors. During training, each agent selects actions based on its current policy and observations while interacting with neighboring agents. To address nonlinear coupling effects, an attention-based optimization strategy is introduced, enabling agents to dynamically adjust their perception of other agents’ behaviors and thereby enhancing coordination and overall system stability.

In the optimization process, this article used a deep Q-learning algorithm based on Experience Replay. Specifically, the experience of the agent (state action reward next state quadruple) is stored in a replay buffer, and the network is trained by randomly sampling these experiences to reduce the correlation between samples and improve training effectiveness. In order to improve the training effect, this article uses a target network to update the Q-value. The update Eq. (19) for the target network:

19

Among them, is the parameter of the target network, is the parameter of the current network, and is the updated scaling factor. This formula ensures the smoothness of network updates and effectively prevents possible oscillations and instability during the training process.

Through reasonable optimization strategies, the MF3DQN-TF framework can achieve good performance in dynamically changing traffic flow, significantly improving the stability of traffic flow at the entrance section of mountainous tunnels, and effectively addressing the challenges brought by nonlinear coupling effects. The pseudo-code below describes the centralized training and distributed execution flow of the Attention MF3DQN-TF framework, incorporating experience replay and target network optimization. The design points of Algorithm 2 are as follows.

Algorithm 2

Attention MF3DQN-TF Training Process.

Computational complexity and scalability analysis

The proposed Attention MF3DQN-TF framework, while theoretically robust and innovative in enhancing traffic flow stability, entails a level of computational complexity that necessitates careful scrutiny. This complexity primarily originates from three interrelated aspects: the multi-agent environment setup, the reinforcement learning procedure, and the computations involved in the attention mechanism.

The multi-agent environment inherently demands computational resources for updating the state and action space of each agent at every time step. When considering N agents, this process exhibits a computational complexity of O(N). Transitioning to the reinforcement learning component, which relies on deep Q-networks (DQN), the complexity escalates to O(N×S×A×T). Here, S, A, and T represent the dimensions of the state space, action space, and the number of training iterations, respectively. The attention mechanism further contributes to the computational load, with its matrix operations leading to a complexity of O(N×D×K), where D denotes the input data’s dimensionality and K the number of attention heads.

Addressing the scalability challenges of the MF3DQN-TF framework is essential for its practical deployment in real-world traffic systems. As the number of vehicles increases, computational demands can grow substantially. To alleviate these challenges, several strategies are proposed. Distributed computing architectures can parallelize the training of agents’ DQNs and the computation of attention mechanisms across multiple processors or nodes. Model compression techniques, such as pruning and quantization, can further reduce the size of neural networks used in the DQN and attention modules, thereby lowering computational and memory requirements. Moreover, implementing selective attention allows the model to focus on the most critical environmental features, reducing input dimensionality (D) and easing computational complexity. Finally, a hierarchical multi-agent design can mitigate system load by clustering agents and coordinating each cluster through higher-level controllers, thus limiting the number of direct interactions each agent must handle.

In practical traffic systems, a modular deployment of the MF3DQN-TF framework is recommended. This approach tailors the system’s components to specific traffic conditions and available computational resources. For instance, during peak traffic periods, the system can prioritise critical traffic sections and allocate computational resources accordingly. Edge computing can also be leveraged to perform local computations at the network’s edge, reducing latency and bandwidth requirements for data transmission to a central server.

By thoroughly evaluating the computational requirements and strategically implementing these optimization measures, the MF3DQN-TF framework can be effectively scaled. This ensures the framework’s viability and efficiency in managing increasingly complex traffic environments.

Experimental and simulation analysis based on MF3DQN-TF

Experimental design and simulation platform

In this study, experiments were conducted on traffic flow simulation using multi-agent deep In this study, experiments were conducted on traffic flow simulation using multi-agent deep reinforcement learning (MADRL) and an attention mechanism. The objective of the study was to verify the effectiveness of the proposed MF3DQN-TF framework. The simulation platform combines various aspects, including traffic flow modelling, intelligent connected vehicle behavior decision-making and environmental interaction.

Traffic flow simulation model for tunnel entrance section.

The traffic flow model of the entrance section of mountainous tunnels is based on classical traffic flow theory, combined with terrain factors, vehicle dynamic characteristics, and traffic density changes in specific areas. This model considers the typical nonlinear coupling effects at the entrance of the tunnel section, including the following aspects:

The relationship between traffic density and vehicle speed: By adjusting the relationship between vehicle speed and vehicle density, simulate the phenomenon of vehicles gathering and releasing at tunnel entrances.

Interaction between vehicles: real-time changes in distance and speed, as well as the impact of braking, acceleration, and other behaviors of the preceding vehicle on subsequent vehicles are considered.

Traffic control parameters unique to tunnel entrances, such as iconic lane change restrictions, speed limits, and emergency parking belts.

By combining the above factors, a dynamic model reflecting the complex traffic flow at the entrance section of mountainous tunnels has been constructed.

Multi agent simulation environment settings.

In this study, a simulation environment built on OpenAI Gym is employed to support the behavioral decision-making of intelligent connected vehicles. All vehicles are modeled as independent “intelligent agents” that interact with others through a DQN. The model takes as input key traffic variables, including flow density, vehicle speed, inter-vehicle distance, and driving behaviors (e.g., acceleration, deceleration, and speed maintenance). Based on these inputs, each agent evaluates its environment via attention mechanisms, formulates decisions, and generates appropriate driving actions.

Within the simulation environment, the action strategy of each agent is derived from the DQN model, with the network input processed through convolutional neural networks (CNNs) or fully connected layers (FCNs) to yield the probability distribution of action selection. The simulation environment is capable of supporting different parameter settings, such as different road models, traffic density, and road speed limits, with the aim of achieving traffic flow stability assessment in different scenarios.

Simulation scenarios, testing conditions, and evaluation methods

To verify the performance of the MF3DQN-TF framework, this experiment conducted simulation tests under different traffic flow density conditions. The specific testing conditions include:

Low density scenario: Low traffic flow density, relatively dispersed vehicles, high vehicle speeds, and less interaction between vehicle behaviors.

Medium density scenario: The traffic flow density is moderate, vehicles begin to gather, the speed decreases, and the impact of vehicle behavior is more obvious.

High density scenario: High traffic flow density, short distance between vehicles, significant decrease in speed, and possible traffic congestion.

In these different scenarios, the performance of the MF3DQN-TF framework will be tested and evaluated through the following aspects:

Traffic flow stability: mainly focuses on the fluctuation of traffic flow, especially under high-density traffic fluctuations.

Vehicle driving efficiency: Examining indicators such as travel time and speed of vehicles at different densities.

Accident occurrence rate: Calculate the frequency of accidents and the number of vehicle collisions through simulation data.

Meanwhile, this article also considers that behavior adjustment and optimization of intelligent connected vehicles are particularly important in high-density traffic flow. Through the reinforcement learning mechanism of intelligent agents, the model can dynamically adjust the acceleration and deceleration strategies of each vehicle. For the simulation scenario of behavior optimization, consider the following factors:

Optimization of distance between vehicles: By adjusting the distance between vehicles, it reduces rear end collisions or blockages caused by following closely behind the preceding vehicle.

Vehicle speed adjustment: The intelligent agent adjusts the driving speed in real time based on the surrounding traffic density and speed situation, avoiding traffic fluctuations caused by too fast or too slow driving speed.

Traffic flow regulation: Intelligent agents adjust based on changes in local traffic flow to reduce traffic congestion or create “dead zones” in traffic flow.

Based on the above considerations, in the simulation process of this article, the specific model related parameter settings for the experiment are shown in Table 1.

Table 1.

Experimental model related parameter Settings.

Parameter name	Set value/Range	Describe
Traffic density	20–100 vehicles/km	Experimental setup under different traffic flow densities
Tunnel length	500 m	The physical length of the tunnel
Maximum speed	60–80 km/h	Maximum allowable speed for intelligent connected vehicles
Range of vehicle distance control	1–5 m	Experimental scope of different distance control between vehicles
Number of training iterations	100,000 times	Training frequency in reinforcement learning
Learning rate (α)	0.001	Learning rate in DQN algorithm
Discount factor (γ)	0.9	Discount factors for reinforcement learning
Experience replay pool size	10,000	Size of DQN experience replay pool
Attention mechanism layers	2nd floor	The layers of attention mechanism in the model
Batch size	64	Number of samples per batch during reinforcement learning training
optimizer	Adam optimizer	Optimization method for updating network weights
Vehicle speed regulation factor	0.1-1.0	Parameters affecting vehicle speed adjustment
Parameter Name	Set Value/Range	Describe

Finally, the experimental simulation in this article proposes a set of evaluation indicators for the stability of intelligent connected vehicles. These indicators are designed to assess the performance of the MF3DQN-TF (Multi Agent Deep Reinforcement Learning Traffic Flow) framework in traffic flow scenarios at the entrance section of mountain tunnels. The indicators are assessed through mathematical modeling and analytical reasoning, with particular emphasis on their significance and corresponding calculation methods. The evaluation framework covers multiple dimensions, including overall traffic flow stability, the behavioral optimization of intelligent connected vehicles, and the influence of traffic density on system performance. This multi-faceted approach ensures a comprehensive assessment of system effectiveness.

1. Evaluation of the relationship between traffic density and flow rate.

Traffic density and flow rate are important parameters for evaluating the performance of transportation systems, especially in the special environment of mountainous tunnel entrance sections. The density of traffic flow at the tunnel entrance directly affects the stability of traffic flow. The density of traffic flow is defined as the number of vehicles per unit length, usually as shown in Eq. (20):

20

Among them, is the number of vehicles passing through a certain section, and is the length of that section.

Flow is the number of vehicles passing through a cross-section per unit time, usually defined as Eq. (21):

21

Among them, is the average speed of the vehicle. The relationship between traffic and density follows the ‘basic diagram’, where within a specific area, as density increases, traffic usually shows a trend of first increasing and then decreasing. In the MF3DQN-TF framework, intelligent connected vehicles can optimize traffic flow by adjusting their driving strategies, maximize traffic flow, and avoid traffic congestion and inefficient flow.

2. Traffic flow stability index: analysis of traffic volatility.

Traffic volatility refers to the degree of fluctuation in vehicle speed and distance in traffic flow, usually measured by the variance of speed differences between vehicles. For the entrance section of mountain tunnels, considering the bends, changes in slope, and traffic congestion, high volatility often indicates unstable traffic, which may lead to traffic accidents or congestion. Assuming a car’s speed during a certain period of time is, the variance of traffic volatility can be defined as Eq. (22):

22

Among them, is the average speed of the vehicle. Excessive volatility means serious instability in traffic flow, which may lead to traffic congestion or dangerous situations. The MF3DQN-TF framework can effectively reduce volatility and improve the stability of traffic flow by adjusting the behavior of intelligent vehicles.

3. Optimization indicators for vehicle spacing and speed.

The key advantage of intelligent connected vehicles is their ability to optimize vehicle speed and distance through communication and collaborative control. Specifically, the safety distance between vehicles is a key indicator for measuring system safety and smoothness. For each pair of adjacent vehicles, the relationship between the distance between vehicles and the vehicle speed can be expressed by the following Eq. (23):

23

Among them, is the braking acceleration of the vehicle. Too small a distance between vehicles may lead to collision risks, while too large a distance may reduce traffic flow efficiency. An important goal of the MF3DQN-TF framework is to regulate the speed and spacing of intelligent connected vehicles through reinforcement learning, ensuring safety and improving the smoothness of traffic flow.

4. Intelligent behavior adjustment effect: evaluation of strategy optimization.

In the process of optimizing the traffic flow of intelligent connected vehicles based on the MF3DQN-TF framework, intelligent agents (i.e. intelligent connected vehicles) continuously adjust themselves through deep reinforcement learning algorithms to optimize the overall performance of traffic flow. In order to measure the effectiveness of intelligent connected vehicle strategy optimization, “reward signals” are commonly used as an evaluation tool. The design of reward signals should consider multiple factors, such as traffic optimization, security, speed control, and stability, which can be specifically expressed as Eq. (24)

24

Among them, is the weight coefficient, which reflects the importance of different factors in optimizing traffic flow. By adjusting these weights, the system can balance the relationship between traffic flow, safety, and stability.

5. Congestion index and accident rate.

The Congestion Index (CI) and accident rate are direct indicators for measuring the stability and safety of traffic flow. The congestion index can be calculated using the following Eq. (25):

25

Among them, is the traffic density of the nth lane and is the maximum carrying density of this section of road. When the congestion index reaches a certain threshold, the system may experience severe traffic congestion, which can affect vehicle traffic efficiency and even cause traffic accidents.

The accident occurrence rate refers to the probability of a traffic accident occurring per unit time, which can be estimated through historical data or simulation results. By evaluating the optimization effect of vehicle behavior under the MF3DQN-TF framework, it can effectively reduce the accident rate and improve the safety and stability of traffic flow.

6. Comprehensive evaluation of system performance.

Taking into account the above indicators, the comprehensive performance evaluation function of the system is obtained through weighted summation. As shown in Eq. (26):

26

Among them, is the weight coefficient, which reflects the relative importance of different evaluation dimensions. Through this comprehensive evaluation function, the optimization effect of intelligent connected vehicle traffic flow under the MF3DQN-TF framework can be comprehensively evaluated, which helps to achieve more precise control and management of traffic flow in practical applications. The design points of Algorithm 3 are as follows.

Algorithm 3

Stability Evaluation.

Analysis of experimental results

1) Performance comparison based on attention mechanism.

In order to verify the effectiveness of the attention-based model in traffic flow prediction, this paper conducted multiple comparative experiments. The aim of the article was to compare the performance of traditional benchmark models and models with attention mechanisms in traffic flow prediction tasks, including model accuracy, stability, and training efficiency. The different models used in the experiment included the basic Convolutional Neural Network (CNN), Long Short-Term Memory Network (LSTM), and LSTM model combined with attention mechanism (Attention MF3DNQ-TF). The performance of these models is compared in Table 2.

Table 2.

Performance comparison of different Models.

Model	Prediction accuracy (%)	Model training time (s)	Convergence speed (number of iterations)	Computational complexity	Accuracy improvement (%)
CNN	72.4	105	50	Lower	-
LSTM	76.5	150	40	secondary	4.1
Attention-MF3DNQ-TF	81.2	180	30	high	4.7

The experimental findings demonstrate that the Attention-based MF3DNQ-TF model exhibits superior performance in terms of prediction accuracy, attaining an accuracy of 81.2%, which is notably higher than the other two models (CNN 72.4% and LSTM 76.5%). As illustrated in Fig. 3, the attention mechanism facilitates the model’s ability to concentrate on significant time series features, consequently enhancing prediction accuracy. However, it is important to note that the Attention MF3DNQ-TF model incurs a relatively high training time and computational complexity, indicating a trade-off between high-precision and efficient training.

Fig. 3.

Performance Comparison Based on Attention Mechanism.

To provide a more detailed statistical analysis of the results, confidence intervals and significance tests were conducted. The prediction accuracy values reported in Table 2 are based on multiple runs of the experiments (specifically, 30 independent runs for each model). The 95% confidence intervals for the prediction accuracy of each model are as follows: CNN (71.2%, 73.6%), LSTM (75.3%, 77.7%), and Attention-MF3DNQ-TF (80.1%, 82.3%). These confidence intervals indicate the range within which the true prediction accuracy is expected to lie with 95% confidence. The relatively narrow intervals for all models suggest that the results are statistically reliable.

Furthermore, significance tests were performed to compare the improvements in prediction accuracy achieved by the Attention-MF3DNQ-TF model over the baseline models. Using a paired t-test with a significance level of 0.05, the improvements in accuracy were found to be statistically significant for both the CNN (p-value < 0.01) and LSTM (p-value < 0.01) comparisons. This indicates that the observed improvements in traffic flow stability and efficiency are not due to random chance but are indeed attributable to the proposed framework.

In addition to prediction accuracy, a similar statistical analysis was conducted for other key performance metrics such as traffic flow stability index and congestion probability. For instance, under high traffic density conditions, the Attention-MF3DNQ-TF model demonstrated a statistically significant improvement in traffic flow stability index compared to the basic model and LSTM-based model (p-value < 0.05). This further reinforces the credibility and reliability of the proposed framework’s performance enhancements in real-world traffic scenarios.

2) Comparison of Stability of Different Traffic Flow Models.

In this experiment, the stability of different traffic flow models was compared. In order to comprehensively evaluate the stability of these models, this paper set multiple traffic flow environments and different traffic densities, and analysed the performance of the models under these conditions. The results are shown in Table 3 below.

Table 3.

Comparison of stability of different traffic flow Models.

Model	Traffic density (vehicles/km)	Average vehicle speed (km/h)	Traffic flow stability index	Probability of congestion occurrence (%)
Basic Model	30	50	0.68	12.5
Model based on LSTM	30	53	0.75	8.3
Attention MF3DNQ-TF	30	56	0.80	4.1
Basic Model	60	40	0.50	40.2
Model based on LSTM	60	42	0.60	25.3
Attention MF3DNQ-TF	60	43	0.65	15.0

By comparing the performance of different traffic flow models under different traffic densities, this paper found that the attention mechanism based model (Attention MF3DNQ-TF) can still maintain high stability indicators under high traffic densities, while significantly reducing the probability of congestion. In contrast, the basic model and LSTM based model are prone to significant instability and congestion in high-density environments, as shown in Fig. 4.

Fig. 4.

Comparison of Stability Results of Different Traffic Flow Models.

3) The Influence of Nonlinear Coupling Effect on Traffic Flow Stability.

The present experiment sought to explore the role of nonlinear coupling effects in the stability of traffic flow. To this end, multiple traffic flow models were constructed and varying degrees of nonlinear coupling effects were introduced (for example, the complex nonlinear relationship between vehicle speed and distance between vehicles). The impact of these effects on the stability and changes of traffic flow was then observed.The strength of coupling effects was categorised into low, medium, and high levels, and the stability changes of traffic flow under different intensities were analysed. The comparative analysis of nonlinear coupling effect strength is presented in Table 4.

Table 4.

Comparative analysis of nonlinear coupling effect Strength.

Strength of nonlinear coupling effect	Average vehicle speed (km/h)	Traffic flow density (vehicles/km)	Traffic flow stability index	Probability of accident occurrence (%)
Low coupling effect	45.2	35	0.72	5.4
Mid coupling effect	43.1	40	0.58	15.7
High coupling effect	39.8	45	0.42	35.6

The experimental findings demonstrate that an augmentation in the strength of nonlinear coupling effects results in a gradual decline in traffic flow stability. In conditions of low coupling effects, traffic flow stability is high and the probability of accidents is low. Conversely, under high coupling effects, there is a substantial increase in traffic flow instability, a decrease in vehicle speed, an increase in traffic density, and a significant increase in the probability of accidents, as illustrated in Fig. 5.

Fig. 5.

Comparison of the Effects of Nonlinear Coupling on Traffic Flow Stability.

4) Training Efficiency Analysis of Traffic Flow Prediction Model.

In order to comprehensively understand the efficiency of different models during the training process, this paper tested the training efficiency of multiple models on the same dataset and environment, including training time, convergence speed, etc. The results are shown in Table 5.

Table 5.

Training efficiency of traffic flow prediction for different Models.

Model	Training time (s)	Training epochs (times)	Convergence speed (rounds/s)	Computational complexity
CNN	105	50	2.1	Lower
LSTM	150	40	2.5	secondary
Attention-MF3DNQ-TF	180	30	1.67	high

From the above results, it can be seen that although the Attention MF3DNQ-TF model performs well in prediction accuracy, its training time and convergence speed are slower compared to CNN and LSTM models. In practical applications, models based on LSTM and CNN may have more advantages in training efficiency, especially in time sensitive scenarios.

5) Traffic flow stability under different traffic densities.

This article further investigates the traffic flow stability performance based on different models under different traffic densities. The experimental results help this article understand the adaptability of the model under different road conditions. Stability comparison under different traffic densities are shown in Table 6.

Table 6.

Stability comparison under different traffic densities.

Traffic density (vehicles/km)	CNN stability index	LSTM stability index	Attention MF3DNQ-TF stability index
30	0.68	0.75	0.80
45	0.55	0.62	0.68
60	0.50	0.60	0.65
75	0.45	0.55	0.60

As traffic density increases, the stability of all three models decreases, but the model based on Attention MF3DNQ-TF maintains high stability at all density levels, indicating that it has better adaptability and anti-interference ability in complex traffic environments.

6) Prediction accuracy under different traffic flow conditions.

This article also tested the predictive accuracy performance based on different models under different traffic flow conditions (such as peak and off peak hours) to evaluate their applicability under different traffic loads. Comparison of Prediction Accuracy between Peak and Off peak Periods are shown in Table 7.

Table 7.

Comparison of prediction accuracy between peak and off peak Periods.

Traffic conditions	CNN prediction accuracy (%)	LSTM prediction accuracy (%)	Attention MF3DNQ-TF prediction accuracy (%)
fastigium	70.1	74.8	78.5
Off peak	75.4	78.2	82.4

During peak traffic congestion periods, the accuracy of all models will decrease, especially in situations of high traffic pressure. However, models based on Attention MF3DNQ-TF have been shown to maintain a satisfactory level of prediction accuracy. This suggests that Attention MF3DNQ-TF functions with reasonable stability in high-load environments and is well-suited for practical traffic flow management, especially in complex scenarios such as peak hours. In contrast, traditional CNN and LSTM models demonstrate a substantial decline in prediction accuracy during peak periods, suggesting their inadequacy in coping with the intricate dynamics of traffic flow.

7) Analysis of Model Generalization Ability.

Beyond evaluating model performance in different experimental settings, this study also emphasizes the model’s generalization capability, defined as its effectiveness in handling unseen traffic flow patterns. To examine this, traffic flow datasets from diverse cities and road conditions were employed to assess the adaptability and accuracy of various models. The results of this comparative analysis are summarized in Table 8.

Table 8.

Comparative analysis of model generalization ability.

Data set	CNN prediction accuracy (%)	LSTM prediction accuracy (%)	Attention MF3DNQ-TF prediction accuracy (%)
City A (urban area)	71.3	74.2	79.6
City B (highway)	69.4	73.1	78.3
City C (Suburban)	72.8	75.5	80.1
City D (mountainous area)	70.5	74.8	79.4

From the experimental results of generalization ability, the model based on Attention MF3DNQ-TF shows high prediction accuracy under different urban and road conditions, significantly better than CNN and LSTM models. This indicates that Attention MF3DNQ-TF has stronger adaptability and can better handle various complex and changing traffic flow situations. Especially in some special road types and urban environments, the Attention MF3DNQ-TF model can still maintain high prediction accuracy, while other models exhibit significant errors under certain conditions, as shown in Fig. 6.

Fig. 6.

Comparison of prediction accuracy and model generalization ability under different traffic flow models.

8) Comparison of other state-of-the-art methods.

In order to further validate the effectiveness of the proposed methodology, we will consider incorporating other state-of-the-art methods as benchmarks in the study. The comparison table is shown in Table 9; Fig. 7.

Table 9.

Comparison of other state-of-the-art methods.

Method	Traffic density type	Stability index of traffic flow	Average vehicle speed (km/h)	Accident rate (%)
Baseline Model	Low	0.68	50	12.5
	Medium	0.50	40	40.2
	High	0.50	35	52.3
LSTM-Based Model	Low	0.75	53	8.3
	Medium	0.60	42	25.3
	High	0.55	38	45.5
Attention MF3DQN-TF Framework (Proposed Method)	Low	0.80	56	4.1
	Medium	0.65	43	15.0
	High	0.60	40	18.2
CNN-Based Model	Low	0.72	52	10.2
	Medium	0.58	40	30.5
	High	0.50	35	48.7
GAN-Based Model	Low	0.78	55	7.1
	Medium	0.65	45	20.4
	High	0.60	40	35.6
Double DQN Model	Low	0.75	54	8.9
	Medium	0.62	43	22.3
	High	0.58	38	38.5

Fig. 7.

Performance Comparison with State-of-the-Art Methods.

The comparative results in Table 9; Fig. 7 highlight the superior performance of the proposed Attention MF3DQN-TF framework over other state-of-the-art methods across low-, moderate-, and high-density traffic conditions. In the low-density scenario, the framework achieves the highest stability index (0.80), the fastest average vehicle speed (56 km/h), and the lowest accident rate (4.1%), outperforming CNN-, LSTM-, and even GAN-based models, which exhibit either lower stability or significantly higher accident rates. Under moderate-density conditions, Attention MF3DQN-TF maintains a stability index comparable to the GAN-based model (0.65 vs. 0.65) but demonstrates a substantially reduced accident rate (15.0% vs. 20.4%), indicating a more favorable trade-off between flow efficiency and safety. In the high-density scenario, the advantages of the proposed framework become even more pronounced: it sustains the highest stability index (0.60) while keeping accident probability at 18.2%, far below GAN (35.6%) and Double DQN (38.5%), both of which show significant deterioration in safety performance. Traditional LSTM- and CNN-based approaches perform poorly in terms of both stability and accident reduction under congested conditions. Taken together, these findings confirm that the proposed Attention MF3DQN-TF framework not only enhances traffic flow stability and vehicle efficiency but also achieves the lowest safety risks, making it particularly effective in addressing the challenges of nonlinear interactions and congestion in mountainous tunnel entrance environments.

Supplementary experiments on the NGSIM dataset

Comparative performance evaluation on the NGSIM dataset

To further examine the applicability of the proposed Attention MF3DQN-TF framework, a supplementary validation experiment was conducted using the publicly available NGSIM dataset. Data source: https://ops.fhwa.dot.gov/trafficanalysistools/ngsim.htm.

The proposed method (Attention MF3DQN-TF) was compared with classical traffic flow models (LWR, FTL), data-driven models (LSTM), and a reinforcement learning baseline (DQN).

As shown in Table 10; Fig. 8, the results are presented:

Table 10.

Comparative performance under different traffic density types.

Traffic density type	Method	Stability index	Prediction accuracy(%)	Accident rate(%)	Training time(h)	Inference latency(ms)
Low Density	LWR	0.162	72.1	5.2	0.1	< 1
	FTL	0.149	73.5	4.7	0.3	< 1
	LSTM	0.131	78.4	4.2	4.4	8
	DQN	0.128	79.1	4	5	7
	Proposed Method	0.109	84.3	3.5	9.1	9
Medium Density	LWR	0.192	65.2	10.4	0.1	< 1
	FTL	0.176	67.3	9.2	0.3	< 1
	LSTM	0.151	72.5	8.6	4.7	8
	DQN	0.147	73.1	8.3	5.3	7
	Proposed Method	0.124	79.8	7.1	9.6	9
High Density	LWR	0.214	60.3	17.6	0.1	< 1
	FTL	0.199	62.1	15.9	0.3	< 1
	LSTM	0.173	68.7	14.2	4.9	8
	DQN	0.168	69.4	13.7	5.5	7
	Proposed Method	0.141	75.6	11.2	9.8	9

Fig. 8.

Comparative performance of different models under varying traffic density conditions.

Traffic flow stability: Across all density levels, the proposed Attention MF3DQN-TF consistently achieved the lowest stability index, reducing fluctuations by 25–30% compared with traditional models (LWR, FTL) and by 14–16% compared with learning-based baselines (LSTM, DQN).

Prediction accuracy: The proposed method attained the highest accuracy, outperforming LSTM and DQN by 5–7% and surpassing traditional models by over 10%, demonstrating stronger adaptability to traffic dynamics.

Safety performance: Accident rate was significantly reduced, particularly under high density, where Attention MF3DQN-TF lowered risk by over 30% relative to classical models.

Computational trade-offs: Training required approximately 9–10 h, which is longer than DQN and LSTM. However, inference latency remained below 10 ms per decision step, ensuring real-time responsiveness for CAVs deployment.

The results demonstrate that the proposed Attention MF3DQN-TF framework consistently outperforms both classical and learning-based baselines across all traffic density scenarios. In terms of stability, it achieves the lowest fluctuation indices, reducing instability by up to 30% compared with LWR and FTL and by around 15% compared with LSTM and DQN. For prediction accuracy, Attention MF3DQN-TF yields the highest scores across densities—84.3% in low, 79.8% in medium, and 75.6% in high density—representing a 5–7% improvement over LSTM and DQN and more than 10% over traditional models, highlighting its robustness in dynamic traffic conditions. Safety performance is also notably enhanced, with accident rates reduced by over 30% compared with LWR and FTL and by approximately 15–20% compared with LSTM and DQN, especially under congested high-density conditions. Although the training time of Attention MF3DQN-TF (≈ 9–10 h) is nearly double that of DQN and LSTM, its inference latency remains under 10 ms per decision step, ensuring real-time responsiveness for connected and autonomous vehicle applications. Overall, these findings indicate that the proposed framework offers a balanced trade-off: the additional training cost is offset by substantial gains in stability, accuracy, and safety, thereby reinforcing its practical applicability in real-world CAV-based traffic management.

Ablation study on framework components

To further examine the contribution of each module in the proposed Attention MF3DQN-TF framework, we conducted an ablation study under high-density traffic scenarios. Four variants were designed: (1) without the attention mechanism (w/o Attention), (2) without the double-dueling Q-network structure (w/o DDQN), (3) without nonlinear coupling modeling (w/o NCE), and (4) without multi-agent fusion (w/o MAF). These ablation settings allow us to isolate the role of each core module in improving traffic flow stability and safety. The results are summarized in Table 11; Fig. 9.

Table 11.

Ablation study results under High-Density traffic Scenario.

Model variant	Stability index	Prediction accuracy (%)	Accident rate (%)	Training convergence speed (iterations)
Full Model (Attention MF3DQN-TF)	0.6	78.5	18.2	30
w/o Attention Mechanism	0.53	74.2	24.7	35
w/o Double-Dueling Structure	0.55	75.1	22.5	40
w/o Nonlinear Coupling Modeling	0.52	73.8	24.9	38
w/o Multi-agent Fusion	0.5	72.5	27.3	45

Fig. 9.

Performance comparison of ablation variants in high-density traffic.

The ablation results clearly demonstrate the necessity of integrating all modules:

Attention Mechanism: Removing attention leads to a 4.3% reduction in prediction accuracy and a 0.07 drop in stability index, confirming that attention enhances agents’ ability to prioritize critical traffic features.

Double-Dueling Q-Network: Without this structure, the model converges more slowly (40 vs. 30 iterations) and stability decreases, indicating that the double-dueling design improves learning efficiency.

Nonlinear Coupling Modeling: Excluding nonlinear effects increases accident rate (+ 6.7%), highlighting the importance of explicitly capturing nonlinear traffic dynamics in tunnel entrance scenarios.

Multi-agent Fusion: The absence of collaborative learning among agents causes the most severe performance degradation, with stability dropping to 0.50 and accident rate rising to 27.3%. This confirms that agent cooperation is a core factor for ensuring stable and safe traffic flow.

Full Model: Achieves the best overall performance across all indicators, validating that the integration of all components is essential for achieving significant improvements in stability, safety, and efficiency.

Discussion and conclusion

The proposed Attention MF3DQN-TF framework has demonstrated significant advantages in enhancing traffic flow stability and efficiency at mountainous tunnel entrances. The integration of multi-agent deep reinforcement learning with an attention mechanism allows the model to effectively capture key factors in complex traffic environments and optimize decision-making processes for intelligent connected vehicles. The results show substantial improvements in traffic flow stability, vehicle efficiency, and reduced congestion time, with increases of over 15%, about 20%, and 18%, respectively.

However, it is crucial to acknowledge the trade-offs associated with the proposed framework. The inclusion of attention mechanisms and multi-agent deep reinforcement learning leads to increased computational complexity and training time. For instance, the Attention MF3DNQ-TF model requires 180 s for training, which is higher than the 105 s for CNN and 150 s for LSTM. This trade-off between precision and efficiency must be carefully considered when applying the framework to real-time traffic management systems.

The framework’s performance was also compared with other advanced methods, such as GAN-based and double DQN models. The Attention MF3DQN-TF framework showed superior performance in terms of traffic flow stability and vehicle speed under different traffic density conditions. For example, in high traffic density scenarios, the Attention MF3DQN-TF framework achieved a stability index of 0.60 and an average vehicle speed of 40 km/h, compared to 0.55 and 38 km/h for the double DQN model.

While the model shows strong adaptability across various traffic conditions, its generalization to other traffic environments such as urban intersections or highways requires further investigation. The model’s robustness and adaptability are essential for its broader application in different traffic scenarios.

In this paper, we proposed an Attention MF3DQN-TF framework to enhance the stability and efficiency of traffic flow at mountainous tunnel entrances. The framework combines multi-agent deep reinforcement learning with an attention mechanism, providing a novel solution to address the challenges posed by complex traffic environments. Our experimental results demonstrated significant improvements in traffic flow stability and efficiency compared to traditional methods and other advanced techniques such as GAN-based and double DQN models.

Despite these advantages, the framework also has some limitations, such as increased computational complexity and potential constraints in generalizing to novel traffic environments. Future research will focus on optimizing the framework’s computational efficiency and exploring its application in a wider range of traffic scenarios. Additionally, we plan to investigate different reinforcement learning algorithms and integrate additional real-time data sources to further enhance the framework’s performance. These efforts aim to provide more comprehensive solutions for the development of intelligent transportation systems and promote the application of intelligent connected vehicles in complex traffic environments.

Author contributions

Mengmeng Duan: Writing-original draft. All authors reviewed the manuscript.

Funding

This work was supported by the Open Project of Anhui Province Key Laboratory of Transportation Information and Safety in Ordinary Universities (JTX202302), Anhui Province Excellent Youth Program for Universities (2022AH030161), Anhui Sanlian University Campus level Research Fund Project (KJZD2023001) and Anhui Sanlian University Collaborative Innovation Center Special Research Project (zjt23001); Study on the Characteristics and Safety Management Strategies of Hybrid Connected Traffic Flow at Mountainous Tunnel Entrance Sections (2025B1094).

Data availability

The data are available from the corresponding author on reasonable request.

Declarations

Competing interests

The authors declare no competing interests.

Ethics, consent to Participate, and consent to publish declarations

Not applicable.

Clinical trial number

Not applicable.