Application of deep reinforcement learning in parameter optimization and refinement of turbulence models

Abstract

In the field of computational fluid dynamics, the accuracy of turbulence models is crucial. The aim of this study is to improve the accuracy of simulations by optimizing turbulence model parameters, in order to address the cost and time limitations of traditional wind tunnel tests and on-site measurements. Based on the SST (Shear Stress Transport) k-ω turbulence model, this article proposed a parameter optimization method for turbulence models based on DDPG (Deep Deterministic Policy Gradient). Using wind pressure coefficient (WPC) simulation as an example. Numerical simulation of complex building wind fields was achieved using OpenFOAM software, and sensitivity analysis of model parameters was conducted. Key parameters that significantly affected simulation results were identified, and GPR (Gaussian Process Regression) was established as a surrogate model to fit the initial CFD (Computational Fluid Dynamics) simulation data. The DDPG algorithm was used for parameter optimization, and model parameters were gradually adjusted to minimize wind pressure simulation errors. The experimental results showed that in a single wind direction angle, the average, Root Mean Square (RMS), maximum, and minimum values of the DDPG optimized WPC were closer to the actual WPC. In the set wind direction angle of 0°-50°, the simulated WPC after DDPG optimization was closer to the actual WPC than before optimization. The DDPG optimization method significantly reduced the MAE (Mean Absolute Error) and RMSE (Root Mean Square Error) of the WPC, and its optimization effect was significantly better than the GA (Genetic Algorithm) and PSO (Particle Swarm Optimization) methods.

Introduction

In the field of building wind engineering^1,2, the accuracy of wind loads is very important for the safety of buildings. With the continuous emergence of super tall buildings and complex structures, accurate calculation of wind loads has become an important issue to ensure structural safety and service life. Although traditional wind tunnel tests can provide high-precision experimental data, their expensive cost and long experimental cycles limit their widespread application. Although traditional wind tunnel tests^3,4 and on-site measurements have high accuracy, their high cost and time limit their widespread application. In contrast, computational fluid dynamics numerical simulation has gradually become an important auxiliary tool in the design phase due to its advantages of low cost and high efficiency. The commonly used Reynolds-averaged Navier-Stokes turbulence model^5,6 still has significant limitations in its application in complex flow fields, especially in simulating fluid separation and reflux phenomena, with poor accuracy. In CFD simulations^7,8, the RANS turbulence model is widely used due to its high computational efficiency, but its limitations in complex flow simulations are also becoming increasingly apparent. Especially the SST k-ω turbulence model^9,10, although it has certain advantages in simulating wall attached flow and positive wind pressure regions, the accuracy of simulation results significantly decreases when dealing with complex phenomena such as fluid separation and reflux, leading to significant errors in the negative pressure region. In order to improve the simulation accuracy of the Reynolds-Averaged Navier-Stokes (RANS) model, research in recent years has gradually shifted towards improving model performance by optimizing model parameters. Especially the SST k-ω turbulence model, due to its advantages in simulating flow characteristics near the wall, has become a focus of research. Deep reinforcement learning^11,12, as an emerging optimization technique, can automatically adjust model parameters to optimize the objective function through interactive learning between agents and the environment. This method can not only identify and correct uncertainties in model parameters, but also improve the accuracy of simulation results under complex flow conditions.

The optimization of turbulence model parameters¹³ has always been an important research direction in fluid dynamics and engineering applications, aiming to improve the prediction accuracy and applicability of turbulence models. Turbulence model parameters refer to the key values that describe the fluid motion, energy transfer and dissipation mechanism in turbulence, including constants, coefficients and functional relationships. These parameters determine the accuracy of the model’s response to flow field characteristics and instability. Their calibration and optimization are the basis for improving the accuracy and scope of turbulence prediction and are the criteria for engineering applications. Model parameters refer to the key variables used to describe fluid behavior in turbulence models. These parameters determine the accuracy and applicability of the model. By optimizing these parameters, the degree of agreement between the simulation results and the actual flow characteristics can be improved, thereby enhancing the predictive ability of the model. Traditional turbulence models such as the Reynolds-averaged Navier-Stokes equation model¹⁴ and the large eddy simulation model¹⁵ describe turbulence phenomena through varying degrees of approximation and are widely used in fields such as aerospace, civil engineering, and marine engineering. Early research on parameter optimization of turbulence models mainly relied on experience and experimental methods¹⁶. Through a large amount of experimental data and engineering experience, researchers have determined the calibration values of some commonly used turbulence model parameters. However, this method is both time-consuming and laborious, and it is difficult to ensure generalization ability in different application scenarios¹⁷. For example, in aerospace engineering, models are used to predict aerodynamic performance under varying flight conditions, while in civil engineering, they simulate wind loads on structures such as high-rise buildings and bridges. Additionally, turbulence models play a vital role in automotive design, industrial pipeline flow analysis, and environmental studies, such as atmospheric dispersion and oceanic currents. Each scenario involves distinct flow behaviors and boundary conditions, making it challenging to develop a universally applicable parameter optimization method that maintains high predictive accuracy across all these diverse contexts. Specific application scenarios include aircraft aerodynamic analysis, industrial pipeline fluid transportation, environmental wind field simulation, etc. In these scenarios, turbulence models need to adapt to different flow characteristics to ensure the accuracy and reliability of predictions. Due to developments in computer and numerical techniques, numerical optimization has become more important in the optimization of turbulence model parameters¹⁸. Stochastic approaches like differential evolution, genetic algorithms, and particle swarm optimization have advanced significantly. Particle swarm optimization¹⁹ uses swarm intelligence to gradually grow, while differential evolution²⁰ uses selection and mutation to increase global search and mutation. These methods have proven to be extremely effective in resolving high-dimensional, multimodal, and non-convex problems, making them essential for tuning turbulence models. Despite advances, precise modeling of complex flow fields remains challenging.

Especially for turbulence models, deep reinforcement learning (DRL) offers improved model parameter optimization. DRL^21,22 excels in high-dimensional, complicated issues by combining deep learning and reinforcement learning to optimize solutions through environmental interaction. Unlike conventional techniques, DRL^23,24 explores and optimizes parameters fast through continuous interaction by using reward signals for learning without the need for explicit gradient information. Compared with traditional back-propagation and evolutionary algorithms, although DRL has a high training cost, it does not require gradient information, can flexibly explore high-dimensional non-convex parameter space, quickly find optimization strategies, and effectively improve the adaptability of the model. Traditional optimization algorithms such as gradient descent and genetic algorithms usually rely on explicit mathematical models or gradient information, and have limited efficiency when dealing with complex nonlinear problems. Deep reinforcement learning directly optimizes through reward signals without explicit gradients or models, and can more flexibly cope with the high-dimensional nonlinear characteristics of turbulent parameter optimization. For complicated flows, including fluid separation, DRL has been demonstrated to increase simulation accuracy by Novati Guido et al.²⁵ and the DDPG method²⁶, which adjusted the turbulence model parameters to match experimental data. DDPG²⁷ and Soft Actor-Critic²⁸ are two techniques that perform well in continuous high-dimensional optimization. Methods based on value functions, such as Deep Q-Network²⁹, are more applied to the optimization of discrete parameter spaces. The study by Zhang Li et al.³⁰ found that combining surrogate models and DRL algorithms can improve optimization efficiency by constructing low-cost approximate models. The framework uses the MLMF-DNN proxy model and combines multi-credibility data to improve the optimization efficiency of high-dimensional problems. The proxy model is used to approximate high-precision simulation, reduce computational costs, and enhance optimization stability and generalization capabilities. Gaussian process regression is used as a surrogate model to replace real turbulence simulations with a low-cost approximation. The agent model is responsible for quickly evaluating parameter effects in the framework, significantly reducing computational costs and improving the optimization efficiency of deep reinforcement learning algorithms. The application of deep reinforcement learning in parameter optimization³¹ of turbulence models provides new methods and tools for engineering simulation³². Through autonomous learning and optimization of intelligent agents, the simulation accuracy of turbulence models can be improved; computational costs can be significantly reduced; the efficiency and reliability of engineering design can be enhanced.

Using OpenFOAM for numerical simulation of complex building wind fields, sensitivity analysis is performed by changing the default values of parameters, and key parameters that significantly affect the simulation results are identified as and . Through sensitivity analysis, the study found that the two parameters and in the turbulence model have a significant impact on the simulation results. When the values of these parameters change, the calculated wind pressure coefficient will be significantly changed, which needs to be paid attention to and adjusted during the optimization process. This article uses GPR and Support Vector Machine (SVM) to construct surrogate models, and finally selects GPR as the surrogate model after data fitting. In this study, “surrogate model” refers to a proxy model built using machine learning algorithms, which can quickly approximate the output of the original complex CFD model at a lower computational cost, thereby replacing expensive simulation calculations in the parameter optimization process; and “after data fitting” refers to the process of building a proxy model, by using a large amount of CFD simulation data as a training set, the model parameters are tuned to minimize the difference between the model output and the real data, thereby obtaining a model with good prediction accuracy and generalization ability. After data fitting, the constructed GPR model performed well in accurately capturing the relationship between input and output, so it was finally selected as the proxy model for this study. The DDPG algorithm is used for parameter optimization, and model parameters are gradually adjusted to minimize wind pressure simulation errors. The experimental results show that in a single wind direction angle, the optimized WPC of DDPG is closer to the actual measured values in terms of average, root mean square, maximum, and minimum values. Compared with other studies, the innovation of this paper lies in combining deep reinforcement learning with the proxy model to optimize the parameters of the turbulence model, overcoming the computational complexity of traditional optimization methods. Through sensitivity analysis, key parameters are clarified, and the GPR proxy model is used to reduce the computational cost, which has strong practicality and broad application prospects.

Methods for optimizing turbulence model parameters

Framework

The research framework of this article is shown in Fig. 1.

Fig. 1.

Research framework of this article.

The OpenFOAM is used to implement SST k-ω turbulence model, and boundary conditions, mesh type, quantity, etc., are set. Using the SST k-ω turbulence model, a structured hexahedral mesh was created using blockMesh, with approximately 150,000 cells. The boundary conditions were set as follows: a fixed velocity of 10 m/s at the inlet, zero gradient at the outlet, and no-slip conditions on the wall. The turbulence model is implemented in OpenFOAM, with the inlet velocity set to 10 m/s, the outlet pressure set to zero gradient, and the wall using the noSlip boundary condition. The blockMesh type is used for meshing to ensure calculation accuracy. Sensitivity analysis is conducted on the parameters of the SST k-ω turbulence model to identify key parameters that have a significant impact on the simulation results. A Gaussian process regression model and SVM are established to select the optimal surrogate model. The surrogate model is trained using initial CFD simulation data to approximate the output of the turbulence model.

Analysis of blunt body flow problem: the wind direction angle is set to θ = 0°, and the DDPG algorithm is used for parameter optimization. The optimized parameters are substituted into the SST k-ω turbulence model and validated using wind tunnel test data^33,34. Through the exploration and utilization of intelligent agents in simulated environments, model parameters are gradually adjusted to minimize wind pressure simulation errors.

Parameter optimization adaptability analysis: the range of different wind direction angles θ is set within the range of 0° to 50°, and the wind direction angle is adjusted every 5° interval. The optimized parameters are substituted into the SST k-ω turbulence model, and the simulated WPCs are output through the surrogate model. The optimization effect is verified using wind tunnel test data. “Optimization effect” refers to the improvement in the accuracy of the optimized SST k-ω turbulence model in predicting the wind pressure coefficient, including the reduction in MSE, the reduction in RMSE, and the enhanced fit between the optimized model calculation results and the wind tunnel test data. The WPCs under different wind direction angles are compared to evaluate the adaptability and consistency of parameter optimization at each wind direction angle.

In order to ensure data quality, the collected data is cleaned, and abnormal data points are removed. Missing data items are filled in through interpolation to ensure the continuity of the data in time and space. Data cleaning refers to the preprocessing of the collected raw data to improve data quality and reliability. In this study, the identification of abnormal data points is based on statistical analysis methods, mainly using the standard deviation method, that is, calculating the deviation of the data point relative to the mean. If the deviation exceeds three times the standard deviation, it is regarded as an outlier and eliminated. Data cleaning also includes the detection of duplicate data and physically unreasonable data, including negative wind pressure coefficients or flow velocities beyond a reasonable range. For missing data, linear interpolation is used to fill in to ensure the continuity of the data in time and space, thereby improving the calculation accuracy and stability of the turbulence model. Outliers are calculated by comparing the deviation of the data point from the mean to the multiple of the standard deviation, and are considered outliers when the multiple is greater than 3. Missing data is processed by mean filling to ensure the continuity and accuracy of the data and avoid negative impact on model performance.

The deviation between data points and the mean is calculated as a multiple of the standard deviation, using the equation:

1

In Eq. (1), represents the mean and represents the standard deviation. When , it indicates that the data point is an outlier. The mean here refers to the arithmetic average of all observations in the data set and is used to measure the central tendency of the data. The standardized deviation evaluates the relative position of the data point in the overall distribution by calculating the deviation of the data point from the mean and dividing it by the standard deviation to identify outliers.

The mean of the data is used to fill in missing values, and the equation for the mean of the data is:

2

The “mean of the data” refers to the arithmetic average of all valid observations in the data set, that is, the sum of all data points divided by the number of data points. The mean is used to fill in missing values to maintain data consistency, reduce information loss, and ensure the stability and accuracy of statistical analysis or model training.

Construction of numerical simulation environment

The inlet velocity is set according to the wind speed in the wind tunnel test, and the outlet pressure is set to atmospheric pressure. The wall conditions need to consider the roughness and non slip conditions of the wind tunnel wall. Through these settings, it is ensured that CFD simulation can accurately reflect the flow field characteristics in wind tunnel tests, thereby improving the reliability and accuracy of simulation results.

The wind tunnel test data used in this article is sourced from the Wind Tunnel Laboratory of Tokyo Institute of Technology in Japan^35,36. The building wind load model is shown in Fig. 2.

Fig. 2.

Building wind load model.

OpenFOAM is used to build building wind load models. In OpenFOAM^37,38, the length, height, and width of the model are set to 0.1 m, and the scale of the model modeling is 1:400. The exposure factor is set to . The start time is set to 0s; the end time is set to 32s; the sampling frequency is set to 1000 Hz. The network type is set through blockMesh; the windward and leeward faces are set as patch; the other faces are set as wall. 25 measurement points are arranged on each of the four surfaces of the building. When the wind exerts pressure on the building surface, the value is positive, and when the wind exerts suction on the building surface, the value is negative.

The boundary conditions set by OpenFOAM are shown in Table 1.

Table 1.

Boundary conditions set by openfoam.

Configuration information	Windward	R-sideward	Leeward	L-sideward
U	fixedValue	noSlip	zeroGradient	noSlip
P	zeroGradient	zeroGradient	fixedValue	zeroGradient
omega	fixedValue	omegaWallFunction	zeroGradient	omegaWallFunction
k	fixedValue	kqRWallFunction	zeroGradient	kqRWallFunction
nut	calculated	nutkWallFunction	calculated	nutkWallFunction

In OpenFOAM, nut is a variable representing turbulent viscosity and is used to describe the effects of turbulence. The turbulent kinetic energy equation is expressed as:

3

In Eq. (3), k represents turbulent kinetic energy; is the velocity component; is the turbulent dissipation coefficient; is the turbulence generation term; represents the specific dissipation rate; is the coefficient of motion viscosity; is the turbulent viscosity coefficient; is the turbulent Prandtl number in the turbulent kinetic energy equation.

The specific dissipation rate equation is expressed as:

4

In Eq. (4), is the coefficient of the turbulence model, which takes different values in different regions and can be represented as , , , and . is the coefficient of the turbulence model, which takes different values in different regions and can be represented as and . and represent the turbulent Prandtl number in the specific dissipation rate equation and the turbulent Prandtl number in the mixing function, respectively.

Production item is represented as:

5

6

The turbulent viscosity coefficient is expressed as:

7

In Eq. (7), is a parameter in the turbulent viscosity coefficient.

S is the modulus of the strain rate tensor:

8

The mixed function F is used to achieve a smooth transition between the k-ω model and the k- model:

9

10

In Eq. (9), is the derivative of turbulent kinetic energy; y is the distance from the wall surface; is the turbulent Prandtl number in the dissipation rate equation.

The default values of each parameter in the SST k-ω turbulence model are shown in Table 2.

Table 2.

Default parameter values for SST k-ω turbulence model.

Parameter	Default value	Parameter	Default value
	0.850		0.083
	1.000		0.090
	0.500		0.310
	0.856		1.000
	0.075		0.500

Parameters , , and affect the turbulent energy dissipation rate and are crucial for predicting the accuracy of flow separation and reattachment regions. is used to adjust the behavior of the model in different flow regions to ensure reasonable results can be obtained in both near wall and far wall regions. By optimizing these parameters, the performance of the model in complex flow fields is improved, making it more accurate in simulating fluid separation, reattachment, and vortex phenomena.

The sensitivity analysis was carried out by adjusting the key parameters of the SSTk-ω model (multiplying by 0.6, 1.0, and 1.4), comparing the simulation output errors, and using the Sobol index and grid independence test to quantitatively evaluate the parameter impact, thereby identifying the factors that affect the simulation results and ensuring that the simulation results are accurate and reliable.

The 10 parameters mentioned in Table 2 are subjected to parameter changes, with values of 0.6 and 1.4 times the original parameter values. When the Sobol method is used to evaluate the key parameters of the turbulence model, multiple combinations of parameter space are generated through sampling methods, and then the model output corresponding to each combination is calculated. Using the Sobol index decomposition method, the total variance is decomposed into the main effect and interaction effect of each parameter to quantify the contribution of each parameter to the output variance. By performing sensitivity analysis on multiple turbulence model parameters, the key parameters that have a greater impact on the model prediction accuracy can be identified. The reason why the parameter variation range is chosen to be 0.6 and 1.4 is to simulate the common variation range of turbulence model parameters in actual engineering. This variation range can effectively cover possible errors and uncertainties without deviating too much from the original value, ensuring that the sensitivity analysis can reflect the impact of parameter changes on the simulation results. The average WPCs of each measurement point under different parameters are shown in Fig. 3. Sensitivity analysis evaluates the impact of each parameter on the simulation results by changing the values ​​of the model parameters (0.6 times and 1.4 times). By observing the changes in the average wind pressure coefficient under different parameter settings, key parameters that have a significant impact on the simulation results are identified.

Fig. 3.

Average WPCs at different measurement points under different parameters. (A) ; (B) ; (C) ; (D) ; (E) ; (F) ; (G) ; (H) ; (I) ; (J)

Figure 3 shows the average WPCs of each measurement point under 10 parameter changes in sequence. The significance of the parameter changes on the simulation results can be determined by observing the numerical curve changes of the average WPCs. It can be seen that in the range of measurement points 1–5, 21–25, 41–45, 61–65, 81–85, the measurement points are located on the windward side, and the average WPC is relatively large. On the basis of the default parameter values, 0.6 times the parameter value and 1.4 times the parameter value are also set. By comparing the three curves under each parameter, it can be found that the curves in Fig. 3G and H show the most significant differences between the 0.6 times parameter and the default parameter, as well as between the 1.4 times parameter and the default parameter. Therefore, and are key parameters that have a significant impact on the simulation results.

Establishment of surrogate model

The surrogate model can quickly approximate complex turbulence phenomena and significantly reduce computational costs by using a small amount of high-precision computational fluid dynamics simulation data for training. Surrogate models can efficiently explore the parameter space, optimize turbulence model parameters, and improve simulation accuracy and reliability. By refining simulation results, accurately predicting and analyzing turbulence behavior can improve computational efficiency in turbulence research and facilitate subsequent parameter optimization analysis. In order to solve the problems of overfitting and generalization of the proxy model, cross-validation and Bayesian optimization can be used to adjust hyperparameters. In addition, the fusion of multi-credibility proxy models improves adaptability, and combined with the online update mechanism, the model dynamically adapts to different turbulence simulation scenarios, improving robustness and generalization ability.

Surrogate models are used for regression tasks. They approximate complex physical processes at low computational cost by learning the mapping relationship between input variables and output results. In turbulence simulation, surrogate models can replace high-cost CFD simulations to achieve efficient parameter optimization.

This article uses Gaussian process regression^39,40 and SVM^41,42 to establish surrogate models, and compares their data fitting effects to select the optimal surrogate model. GPR and SVM were selected for surrogate model analysis based on their respective advantages: GPR can provide uncertainty quantification, is suitable for small samples and high-dimensional problems, has good generalization ability, and is suitable for parameter optimization in turbulence simulation; SVM performs well in processing high-dimensional nonlinear data, can effectively avoid overfitting, and improve model robustness.

SVM and GPR were selected as surrogate models for comparison, mainly based on their complementary characteristics in regression tasks. GPR has the ability to quantify uncertainty and is suitable for small sample high-dimensional problems, while SVM performs well in dealing with nonlinear relationships and high-dimensional features and can effectively avoid overfitting. By comparing the fitting accuracy of the two, the selection of the optimal surrogate model is verified.

The mean function in Gaussian process regression is represented as:

11

The covariance function is expressed as:

12

During the training process, the logarithmic marginal likelihood function is maximized to optimize the parameters of the kernel function. The logarithmic marginal likelihood function is:

13

The kernel function used in SVM is the radial basis function, and the equation is:

14

In Eq. (14), is the kernel parameter.

SVM adopts an insensitive loss function to ignore the deviation of errors within the -range:

15

The comparison of data fitting effects between GPR and SVM is shown in Fig. 4. SVM and GPR are chosen for comparison because they perform well in dealing with regression problems. SVM can effectively handle high-dimensional data and avoid overfitting, while GPR provides good uncertainty quantification and is suitable for complex nonlinear relationships. Both are often used to balance accuracy and computational efficiency in surrogate model construction, so they are ideal choices for comparison.

Fig. 4.

Data fitting performance of GPR and SVM. (A) GPR data fitting; (B) SVM data fitting.

Using and for parameter input, 1000 data simulations are conducted in OpenFOAM. The average WPC is used as the corresponding output data. GPR and SVM are respectively used for data fitting. Comparing Fig. 4A and B, it can be intuitively analyzed that the data fitted by GPR is very close to the generated data in OpenFOAM, while the data fitting effect of SVM is relatively poor. Surrogate models are used for regression tasks and make predictions by learning the relationship between input and output. By approximating the calculation process of the actual model, the computational cost is reduced. Surrogate models perform well in optimization problems and can provide fast approximate solutions for complex systems.

The equation for MSE (Mean Square Error) is:

16

The equation for the average absolute error is:

17

In Eqs. (16) and (17), and represent the predicted and actual average WPCs, respectively.

The fitting error results of GPR and SVM models under 10 fold cross validation are shown in Table 3.

Table 3.

Fitting error results of GPR and SVM models.

Fold number	MSE		MAE
	SVM	GPR	SVM	GPR
1	0.0101	0.0034	0.0467	0.0482
2	0.038	0.0037	0.0686	0.0508
3	0.0068	0.0036	0.0415	0.0503
4	0.0046	0.0036	0.0363	0.0492
5	0.0813	0.0032	0.0728	0.0469
6	0.0077	0.0032	0.0484	0.0467
7	0.0077	0.0036	0.052	0.0493
8	0.0241	0.0037	0.0597	0.0516
9	0.003	0.0034	0.0279	0.0478
10	0.0127	0.0031	0.0644	0.0475
Average value	0.0196	0.00345	0.05183	0.04883

The data fitted by GPR and SVM models, as well as the average WPC data output by SST k-ω turbulence model, are analyzed by MSE and MAE. By comparing four sets of data, it can be found that the MSE and MAE of GPR are lower than SVM, with an average MSE of 0.00345 and an average MAE of 0.04883 for GPR; the average MSE of SVM is 0.0196 and the average MAE is 0.05183. Based on the results in Fig. 4; Table 3, GPR can better approximate the output of the SST k-ω turbulence model. Therefore, this article chooses GPR as the surrogate model. Surrogate models can accelerate parameter optimization analysis by approximating complex actual models. Different surrogate models have their own advantages and disadvantages. They may suffer from overfitting problems, resulting in inaccurate predictions of new data. In addition, surrogate models are usually designed for specific situations. When the model settings or parameters change, the original surrogate model may not be able to generalize effectively, reducing its applicability.

The advantage of using the GPR proxy model is that it can efficiently approximate CFD simulation results, significantly reduce computational costs, and provide uncertainty quantification to enhance prediction stability. Compared with direct CFD simulation, GPR can use a small amount of high-precision CFD data for training, reducing computing resource consumption and making parameter optimization more efficient. In addition, GPR has good generalization capabilities and can still accurately fit complex flow field characteristics under small sample conditions, thereby achieving a better balance between computational efficiency and accuracy when optimizing turbulence model parameters.

Application of DDPG algorithm

Reinforcement learning is a machine learning method that learns by interacting with the environment and based on reward signals. Its main components include agents, environments, states, actions, and rewards. Agents interact with the environment by selecting actions to obtain rewards or penalties, thereby optimizing decision-making strategies. In this paper, reinforcement learning is used to optimize the parameters of the turbulence model, and the reward signal is used to guide the model to select the optimal parameters to improve simulation accuracy and computational efficiency.

Reinforcement learning is a self-learning method in which an intelligent agent interacts with the environment and optimizes its strategy based on reward signals. Its key components include state, action, reward, strategy, and value function. This study uses the DDPG algorithm in deep reinforcement learning to optimize the parameters of the SST k-ω turbulence model, where the state represents the initial parameters of the turbulence model, the action corresponds to the parameter adjustment, the reward function aims to minimize the error of the wind pressure coefficient, and the policy network is used to learn the optimal parameter update rules. Compared with traditional optimization methods (such as GA and PSO), DDPG can efficiently search in continuous high-dimensional space, and combine with the GPR agent model to accelerate the optimization process, thereby improving computational efficiency, reducing simulation costs, and improving the applicability and generalization ability of the model.

The action space represents the adjustment values of parameters, and the reward function is defined as a negative simulation error, that is, the optimization objective is to minimize the error and obtain the highest reward.

The Actor network^43,44 accepts the current parameter value and outputs an action, which is the parameter adjustment value; the Critic network calculates the reward value obtained after taking this action. By continuously updating the parameters of these two networks, the DDPG algorithm can gradually approach the optimal strategy. The DDPG optimization framework based on the GPR proxy model is shown in Fig. 5.

Fig. 5.

DDPG optimization framework based on the GPR proxy model.

DDPG is adopted for single wind direction angle optimization, and a fixed sized experience replay pool is initialized to store the interactive data generated during the optimization process. In each training step, a batch of data samples are randomly selected from the experience replay pool for training to reduce bias caused by data correlation. By executing actions (adjusting parameters and ), new states and rewards are obtained. Under a single wind direction angle, by adjusting the parameters, the simulation error of the WPC in the SST k-ω turbulence model is minimized. After each execution of the action, the simulation error is calculated and used as feedback as a reward to guide the next step of parameter adjustment.

The optimization objective is to minimize the simulation error of the WPC in the SST k-ω turbulence model, measured by mean square error. The optimization objective function equation is:

18

In Eq. 18, and represent the actual and simulated WPCs for the i-th data point, respectively.

The equation for the reward function is:

19

During the optimization process, GPR surrogate models are used to replace costly CFD simulations. The GPR model can quickly predict the simulation error after parameter adjustment and improve optimization efficiency by training the initial CFD simulation data. In each training step, the GPR surrogate model is used to predict simulation errors under different parameter combinations, thereby accelerating the optimization process.

The Adam optimizer dynamically adjusts the learning rate by calculating the moving average of gradients, enabling the network parameters to converge quickly and stably. The Adam optimizer updates the parameters of the Actor and Critic networks based on the samples extracted from the experience replay pool to continuously improve the quality of the strategy.

DDPG is adopted for multi-wind direction angle optimization, with the range of wind direction angle θ set within the range of 0° to 50°, and wind direction angle adjustment conducted every 5° interval. By using data from multiple wind directions, the dataset is expanded to analyze the practical application effects of DDPG optimization.

The calculation of WPC is:

20

Software version information

• Visio: Microsoft Visio 2019, https://www.microsoft.com/zh-cn/microsoft-365/visio/flowchart-software.

• OpenFOAM: OpenFOAM v2306, https://www.openfoam.com/.

• Python: Python 3.8, https://www.python.org/.

• NumPy 1.26.4, https://numpy.org/.

• Pandas 2.2.2, https://pandas.pydata.org/.

• Matplotlib 3.8.4, https://matplotlib.org/.

• SciPy 1.12.0, https://scipy.org/.

• Scikit-learn 1.4.2, https://scikit-learn.org/.

• TensorFlow 2.16.1, https://www.tensorflow.org/.

• PyTorch 2.3.1, https://pytorch.org/.

• PyFoam 2023.10, https://github.com/PyFoam/PyFoam.

• Xarray 2024.3.0, https://docs.xarray.dev/.

• Plotly 5.22.0, https://plotly.com/python/.

Results

WPC of unidirectional angle

Average WPC

When the wind direction angle is 0°, the actual average WPC, the average WPC of the SST k-ω turbulence model before optimization, and the average WPC after DDPG optimization are compared, and the differences among the three sets of data on four surfaces are compared. The comparison results of the average WPC are shown in Fig. 6.

Fig. 6.

Average WPC. (A) Actual average WPC; (B) Average WPC before optimization; (C) Average optimized WPC.

From Fig. 6A, it can be seen that the actual WPC shows a significant positive value on the windward side, especially in the middle and lower areas of the building, which is related to the direct impact of the airflow on the windward side. The WPCs on the right and left sides show symmetrical changes, gradually decreasing from top to bottom, with a minimum value close to − 1.2. On the leeward side, the overall WPC is negative, indicating that the surface is in the suction zone and is greatly affected by airflow separation. Figure 6B shows the average WPC of the optimized SST k-ω turbulence model. Compared with actual data, the predicted WPC on the windward side has a smoother distribution in the central region, indicating that the model has certain errors when dealing with the high-pressure area on the windward side. There is also a certain deviation in the predicted values of the WPCs on the right and left sides, indicating that the model has inaccuracies in predicting the lateral area. There are significant differences between high-value areas such as the leeward side and the actual area. Figure 6C shows the average WPC of the SST k-ω turbulence model optimized by DDPG. It can be seen that the distribution of the WPC on the windward side is closer to the actual value, especially in the middle and lower regions, reflecting a significant improvement in the predictive ability of the optimized model in the high-pressure area on the windward side. The SST k-ω turbulence model optimized by the DDPG algorithm performs more accurately in dealing with the high pressure region on the windward side. This improvement is attributed to the effective exploration and utilization strategy of the DDPG algorithm in the parameter optimization process, which enables the model parameters and to better fit the actual WPC distribution.

RMS value of WPC

The actual WPC RMS values at a single wind angle, the SST k-ω turbulence model WPC RMS values before optimization, and the WPC RMS values after DDPG optimization are compared. The comparison results of the RMS values of the WPC are shown in Fig. 7.

Fig. 7.

RMS value of WPC. (A) Actual RMS value of WPC; (B) RMS value of WPC before optimization; (C) RMS value of optimized WPC.

From Fig. 7A, it can be seen that the actual RMS value of the WPC shows a significant positive value on the windward side, especially in the central area of the building. The RMS value of the WPC reaches the highest value of about 0.8, which is related to the direct impact of the airflow on the windward side. The RMS values of the WPCs on the right and left sides show symmetrical changes, while the overall RMS value of the WPC on the leeward side is positive, indicating that the surface is in the reattachment zone of the airflow and is greatly affected by airflow disturbances. Combining Fig. 7B and C, it can be observed that the RMS values of the WPC of the optimized SST k-ω turbulence model with DDPG are closer to the actual data in the right, left, leeward, and contour regions. On the windward side, the optimized RMS value of the WPC is closer to the actual data, but the contour area in the middle and lower parts differs significantly from the actual area. By analyzing the data from various aspects, it can be found that the optimized model performs more accurately in handling high RMS value areas on the windward side, lateral RMS value areas, and reattachment areas on the leeward side.

Maximum WPC

The comparison results of the maximum WPC are shown in Fig. 8.

Fig. 8.

Maximum value of WPC. (A) Actual maximum WPC; (B) Maximum WPC before optimization; (C) Maximum WPC after optimization.

Combining Fig. 8A–C, it can be seen that on the windward side, the maximum actual WPC is around 2.8; the maximum value before optimization is around 3.2; the maximum value after optimization is also around 2.8. Moreover, on the windward side, the contour area with the maximum optimized WPC is very close to the actual contour area. On the left and right sides, as well as on the leeward side, the contour area with the maximum optimized WPC is closer to the actual contour area. The maximum values of the actual, pre-optimized, and post-optimized WPCs on the right side are at the levels of 0.5, 0.9, and 0.6, respectively. The maximum values of the actual, pre-optimized, and post-optimized WPCs on the left side are at the levels of 0.8, 1.1, and 0.7, respectively. The actual, pre-optimized, and post-optimized maximum WPCs on the leeward side are at the levels of 0.1, 0.6, and 0.2, respectively.

Minimum WPC

The comparison results of the minimum WPC are shown in Fig. 9.

Fig. 9.

Minimum WPC. (A) Actual minimum WPC; (B) Minimum WPC before optimization; (C) Minimum WPC after optimization.

On both sides and the leeward side, the contour area of the optimized minimum WPC is significantly closer to the actual contour area of the minimum WPC, and the optimized minimum WPC is closer to the actual value on all sides. The actual, pre-optimized, and post-optimized minimum WPCs on the windward side are at the levels of -0.5, -0.9, and − 0.5, respectively. The minimum values of the actual, pre-optimized, and post-optimized WPCs on the right side are at the levels of − 3.5, − 3.8, and − 3.6, respectively.

WPC at different wind directions

WPC of windward

By changing the wind direction angle, the adaptability and consistency of parameter optimization are analyzed. The comparison results of Windward WPCs for different wind directions are shown in Fig. 10.

Fig. 10.

WPC of Windward. (A) Actual WPC of Windward; (B) WPC of Windward before optimization; (C) WPC of Windward after optimization.

In Fig. 10, the horizontal axis represents 11 wind directions between 0° and 50°, and the vertical axis represents the WPC. The average, RMS, positive and negative values of the WPC are displayed. From Fig. 10A, it can be seen that the actual average WPC on the Windward surface remains stable between wind angles of 0° and 50°, with a slight decrease, and the RMS value remains stable as well. The positive electrode value fluctuates slightly as the wind direction angle increases, but overall remains around 2.8. The negative value significantly decreases after a wind direction angle of 40°, from approximately − 1.0 to − 2.2. At higher wind angles, the extreme negative WPC on the Windward surface significantly decreases. Figure 10B shows the prediction results of the optimized SST k-ω turbulence model. It can be seen that the average and RMS values of the WPC predicted by the model are relatively close to the actual values. However, there is a significant difference between the positive electrode value at 0° and 5° and the actual positive electrode value of the WPC. There is a significant difference between the negative value between 20° and 35° and the actual negative value of the WPC. Figure 10C shows the prediction results of the SST k-ω turbulence model optimized by DDPG. The optimized model shows higher accuracy in predicting the average WPC, RMS value, positive pole value, and negative extreme value, and the overall data trend is very close to the actual data trend. Overall, the model optimized by DDPG predicts the average, RMS, positive and negative values of Windward WPC more accurately than before at different wind directions. The optimized model improves the accuracy and reliability of WPC prediction, enhances the adaptability and consistency of the model under different wind directions, and verifies the effectiveness of the DDPG algorithm in optimizing turbulence model parameters.

WPC of R-Sideward

The comparison results of R-Sideward WPCs for different wind directions are shown in Fig. 11.

Fig. 11.

WPC of R-Sideward. (A) Actual WPC of R-Sideward; (B) WPC of R-Sideward before optimization; (C) WPC of R-Sideward after optimization.

By comparing Fig. 11A–C, it can be seen that the actual, pre-optimized, and post-optimized R-Sideward WPC average values and RMS values are relatively close. The negative value of the WPC before optimization is -4 at 0°, but the actual negative value of the WPC is larger than − 4. The negative value of the WPC before optimization is less than − 2 at 50°, while the actual negative value of the WPC is greater than − 2. The positive pole value of the WPC before optimization should be greater than 2 between 25° -50°, but the actual positive pole value is significantly smaller than the positive pole value before optimization. The positive and negative values optimized by the DDPG algorithm are very close to the actual values under various wind direction conditions. Therefore, parameter optimization through DDPG significantly improves the positive and negative pole simulation effects of R-Sideward WPC.

WPCs of Leeward

The comparison results of Leeward WPCs for different wind directions are shown in Fig. 12.

Fig. 12.

WPCs of Leeward. (A) Actual WPC of Leeward; (B) WPC of Leeward before optimization; (C) WPC for Leeward after optimization.

The data trends of the actual, pre-optimized, and post-optimized average WPCs of Leeward are very close, indicating that the simulation effect of the average WPC before and after optimization is similar. However, in terms of RMS, positive electrode value, and negative electrode value, the pre-optimized data is clearly not as close to the actual data as the optimized data. Observing the RMS and positive electrode values in actual and optimized data, it can be found that there is a certain gap between the RMS and positive electrode values, and the data gap shows an overall increasing trend towards the increase of wind direction angle. However, the RMS and positive electrode values before optimization are very closely related. At 5°, 10°, and 15°, the actual and optimized negative electrode values are approximately at the − 1.5 level, but the pre-optimized negative electrode values are significantly lower than − 1.5. The average values of actual, pre-optimized, and post-optimized WPCs in Leeward are relatively close, but DDPG can effectively improve the simulation effect on the RMS, positive value, and negative extreme value of WPCs through parameter optimization.

WPC of L-Sideward

The comparison results of L-Sideward WPCs for different wind directions are shown in Fig. 13.

Fig. 13.

WPC of L-Sideward. (A) Actual WPC of L-Sideward; (B) WPC of L-Sideward before optimization; (C) WPC of L-Sideward after optimization.

At L-Sideward, the average value, negative value, and RMS data curve of the actual, pre-optimized, and post-optimized WPCs are relatively close. However, the data on the positive electrode value indicates that the optimized positive electrode value is closer to the actual positive electrode value. The actual and optimized positive electrode values are both between 0 and 1, and the data trend of positive electrode values is very close.

Comparison results of WPC simulation accuracy

The MAE and RMSE of the WPC and the actual WPC before optimization, DDPG optimization, GA optimization, and PSO optimization of the SST k-ω turbulence model parameters are calculated separately to analyze the simulation accuracy of the WPC. The simulation accuracy of WPC is shown in Fig. 14.

Fig. 14.

Simulation accuracy of WPC. (A) MAE of WPC simulation; (B) RMSE of WPC simulation.

From the MAE results in Fig. 14A, it can be seen that the unoptimized SST k-ω turbulence model maintains a MAE of about 0.296 at all wind angles, indicating that the model has a certain degree of error in simulating WPCs and the error is relatively stable. In contrast, the DDPG optimized model significantly reduces MAE, with an average MAE value of about 0.065 for all wind directions, indicating that DDPG optimization has a significant effect on improving simulation accuracy. The performance of GA and PSO optimized models in MAE is slightly inferior to DDPG, but still significantly better than the unoptimized model. The MAE optimized by GA is about 0.175 at each wind direction angle, while the MAE optimized by PSO is about 0.200 at each wind direction angle, which is significantly lower than the unoptimized model but higher than the DDPG optimized model. The RMSE findings in Fig. 14B show that the unoptimized SST k-ω turbulence model maintains an RMSE of around 0.343 at all wind angles, which is comparable to the MAE values. This implies that in simulating WPC, the model has some mistakes, mostly stable faults. The DDPG optimized model also significantly reduces RMSE, with an average RMSE value of about 0.076 for all wind directions, proving the effectiveness of DDPG optimization in improving simulation accuracy.

Mesh sensitivity analysis

Three grids with different densities (coarse, medium, and fine) were selected, and the WPC fitting effect was measured by MSE. The results are shown in Table 4.

Table 4.

Grid sensitivity analysis.

Mesh density	Cell number	DDPG	GA	PSO
Coarse mesh	50,000	0.085	0.120	0.135
Medium mesh	150,000	0.065	0.095	0.110
Fine mesh	300,000	0.055	0.085	0.100

In this study, the wind pressure coefficient was analyzed for sensitivity by dividing the computational domain grid into coarse grid, medium grid and fine grid, corresponding to 50,000, 150,000 and 300,000 cells respectively, and three optimization algorithms, DDPG, GA and PSO, were used with MSE as the evaluation index. The experimental results show that with the increase of grid density, each algorithm shows a significant trend of error reduction. The MSE value of the DDPG algorithm gradually decreases from 0.085 for coarse grid to 0.065 for medium grid and then to 0.055 for fine grid, while the MSE values ​​of the GA algorithm are 0.120, 0.095 and 0.085 respectively, and those of the PSO algorithm are 0.135, 0.110 and 0.100. This result shows that the refinement of the grid significantly improves the simulation accuracy. At the same time, DDPG shows better performance and stronger robustness in parameter optimization, and the prediction of the wind pressure coefficient is closer to the actual value, which verifies its application potential in complex flow field simulation. This study provides a quantitative basis for wind pressure simulation and points out the development direction for introducing adaptive grids and multi-objective methods for engineering optimization.

Conclusions

This article not only demonstrated the effectiveness and applicability of DDPG in refining turbulence model parameters, but also provided a more inexpensive and precise method for simulating and estimating WPC in engineering practice. The improved turbulence model reduced economic losses and deaths caused by wind damage by increasing the reliability and safety of building wind resistance design and by more accurately capturing fluctuations in wind pressure. Nevertheless, there are specific research limitations, such as the need for additional testing regarding the applicability and resilience of optimization algorithms in handling increasingly complicated real-world working settings. Subsequent investigations may investigate the amalgamation of alternative machine learning methods and optimization strategies to enhance the precision and robustness of the model. Simultaneously, the model may be made more applicable to various wind situations and building types, providing more thorough technical assistance for urban planning and architectural design.

Author contributions

Zhan Zhang was solely responsible for the conception and design of the study, the acquisition, analysis, and interpretation of data, and drafting and revising the manuscript. All aspects of this work were carried out by Zhan Zhang.

Data availability

Data is available in this paper.

Declarations

Competing interests

The authors declare no competing interests.