Forecasting small-scale dynamics of fluid turbulence using deep neural networks

Significance

In recent decades, direct numerical simulations (DNS) of the Navier–Stokes equations have become a prominent tool for studying turbulent flows ubiquitously encountered in nature and technology. However, both practical and theoretical requirements for increasing the Reynolds number present a challenge for the DNS: As the Reynolds number increases, cutting-edge simulations need to be larger and longer, rendering them prohibitively expensive and unfeasible. Alternatively, can one leverage deep neural networks to advantageously learn from existing simulations at lower Reynolds numbers to cheaply and accurately predict small-scale properties at higher unseen Reynolds numbers? This study represents a significant step forward in addressing this fundamental and crucial question at hand.

Abstract

Turbulence in fluid flows is characterized by a wide range of interacting scales. Since the scale range increases as some power of the flow Reynolds number, a faithful simulation of the entire scale range is prohibitively expensive at high Reynolds numbers. The most expensive aspect concerns the small-scale motions; thus, major emphasis is placed on understanding and modeling them, taking advantage of their putative universality. In this work, using physics-informed deep learning methods, we present a modeling framework to capture and predict the small-scale dynamics of turbulence, via the velocity gradient tensor. The model is based on obtaining functional closures for the pressure Hessian and viscous Laplacian contributions as functions of velocity gradient tensor. This task is accomplished using deep neural networks that are consistent with physical constraints and explicitly incorporate Reynolds number dependence to account for small-scale intermittency. We then utilize a massive direct numerical simulation database, spanning two orders of magnitude in the large-scale Reynolds number, for training and validation. The model learns from low to moderate Reynolds numbers and successfully predicts velocity gradient statistics at both seen and higher (unseen) Reynolds numbers. The success of our present approach demonstrates the viability of deep learning over traditional modeling approaches in capturing and predicting small-scale features of turbulence.

Turbulent fluid flows, ubiquitous in nature and technology, are characterized by strong and chaotic fluctuations across a wide range of interacting scales in space and time. Such multiscale interactions are highly nonlinear, leading to mathematical intractability of the governing equations. Consequently, turbulence has defied an adequate framework despite a sustained effort in physics, mathematics and engineering, and our present understanding remains incomplete, often relying on phenomenological approaches (1–3). An essential notion in this regard is that of small-scale universality (4), which forms the backbone of turbulence theories and models. It stipulates that, while the large scales are nonuniversal because of their dependence on flow geometry and energy injection mechanisms, such dependencies become progressively weaker as energy cascades to smaller scales, ultimately endowing them with some form of universality that depends only on a few parameters of the flow. This, of course, is not guaranteed to be true.

From this perspective, this putative universality requires sufficiently large separation between the scales at which the energy is injected and those at which it is dissipated into molecular motion. This scale separation is determined by the Reynolds number, Re (1); thus, investigating universality requires data at high Re. While such high Reynolds numbers are attainable in some laboratory flows and all geophysical flows, most quantities pertaining to small scales are still very difficult to measure (5). Alternatively, direct numerical simulations (DNS) of the governing equations, where the entire range of scales is resolved on a computational mesh (6), provide information (at least in principle) on every quantity desired. However, DNS is extremely expensive, with recent studies showing that its cost scales even faster than the traditional estimate of Re³ (7–9). Thus, despite the rapid advances in high-performance computing, high-Re DNS, representative of natural and engineering flows, remains unlikely for the foreseeable future.

Motivated by these considerations, we devise here an alternative approach based on machine learning techniques to characterize the small scales of turbulence. In recent years, the use of machine learning, especially deep learning, has ushered in a new paradigm in various scientific fields (10). The field of turbulence is no different, and there has been a flurry of machine learning methods to improve turbulence modeling (11, 12). A vast majority of them utilize the framework of supervised learning (13), where neural networks are trained on input data against labeled output data, although other paradigms have also be used (14, 15). The learning is also often “physics informed,” i.e., neural networks are designed to satisfy some physical constraints, enabling efficient learning including significantly improved accuracy and stability (16). The approach utilized here follows a broadly similar paradigm, but in a framework specifically designed for small scales of turbulence. In particular, we capture the small-scale dynamics of turbulence by training deep neural networks on existing DNS data at low and moderate Re and demonstrate the capability for predicting their dynamics at both seen Re and higher unseen Re, with important consequences for turbulence simulations.

The small scales of turbulence can be conveniently studied via the velocity gradient tensor A = ∇u, where u is the turbulent velocity field. The tensor A encodes various structural and statistical properties of turbulence, which are known to be universal to various degrees. The non-Gaussianity of its fluctuations and the associated extreme events (8, 17–20), the negative skewness of the longitudinal (or diagonal) components associated with the energy cascade (from large to small scales) (21, 22), the preferential alignment of vorticity with the intermediate strain eigenvector (23, 24), are a few notable examples. By taking the gradient of the incompressible Navier–Stokes equations, the evolution equation for A becomes

$$\begin{array}{c}\hfill \frac{D\mathbf{A}}{\mathit{Dt}}=-{\mathbf{A}}^2-\mathbf{H}+\nu {\mathrm{\nabla }}^2\mathbf{A},\end{array}$$ [1]

where D/Dt is the material (or Lagrangian) derivative, H = ∇∇P is the Hessian tensor of the kinematic pressure P and ν the kinematic viscosity. The above equation, studied by a number of authors with different perspectives, dictates that the velocity gradient tensor changes along a fluid element according to quadratic nonlinearity, the pressure effects, and viscous diffusion. Since the trace Tr(A)=0 by incompressibility, it follows that

$$\begin{array}{c}\hfill {\mathrm{\nabla }}^{2}P=\mathrm{Tr}(\mathbf{H})=-\mathrm{Tr}({\mathbf{A}}^2).\end{array}$$ [2]

That is, the pressure field is related to A through a Poisson equation, implying that the pressure Hessian is nonlocal, essentially coupling all scales of the flow.

In DNS, Eq. 1 is numerically solved on a large computational mesh by resolving all dynamically relevant scales (6); whereas other simulation paradigms resolve only a range of scales. For instance, in large-eddy simulation (LES), the large-scales are resolved on a mesh, and the effects of small scales are directly modeled. In contrast, our approach here is to directly develop a reduced-order closure model for A. This can be accomplished by modeling the pressure Hessian and the viscous Laplacian terms explicitly in terms of A, leading to a fully local description (25), i.e., the dynamics of A can be modeled by an ordinary differential equation (ODE), whereby statistical quantities of interest can be obtained, for example, by running Monte Carlo simulations of the ODE with arbitrary initial conditions. Note that this approach models the Lagrangian dynamics of A, as opposed to Eulerian results obtained on a computational mesh. The main benefit of this approach is that it allows us to directly obtain the statistical properties of A, which otherwise can be obtained only from DNS.

Several attempts of this sort—see, for example, refs. 26–32—have been made over the years, including the use of neural networks more recently (33, 34). While these models have enjoyed reasonable success, they still struggle to capture various crucial aspects of velocity gradient dynamics. In particular, they have not been able to capture the Re dependencies of velocity gradient statistics, which is a crucial aspect of small-scale intermittency (2, 3). To rectify such shortcomings is a primary motivation of the current work. In the following section, we first provide the general framework for developing the closure model and describe the deep learning tools required for our purposes.

Modeling Using Tensor Representation Theory

To obtain a functional closure, the pressure Hessian and viscous Laplacian terms in Eq. 1 need to be specified as tensor functions of A. This can be most generally achieved by using tensor representation theory (35, 36), which has often been used in various modeling contexts (32–34, 37, 38). Below, we briefly summarize the basic theory and the changes introduced in this work to better capture the dynamics of A.

Basic Framework.

Tensor representation theory allows us to express any desired (second-order) tensor as a function of A. This is achieved by expressing the desired tensor as a linear combination of tensors in an appropriate tensor basis constructed from A, with the coefficients that are functions of the scalar basis of A (35, 36). To obtain the tensor and scalar bases, the first step is to decompose A into its symmetric and skew-symmetric parts, which are the strain-rate and rotation rate tensors, respectively:

$$\begin{array}{c}\hfill \mathbf{S}=\frac{1}{2}(\mathbf{A}+{\mathbf{A}}^{\mathrm{T}}),\mathbf{R}=\frac{1}{2}(\mathbf{A}-{\mathbf{A}}^{\mathrm{T}}).\end{array}$$ [3]

$$\begin{array}{cc}\hfill {\mathbf{T}}^{(1)}& =\mathbf{S},{\mathbf{T}}^{(2)}=\mathbf{S}\mathbf{R}-\mathbf{R}\mathbf{S},{\mathbf{T}}^{(3)}={\mathbf{S}}^2-\frac{1}{3}\mathrm{Tr}({\mathbf{S}}^2)\mathbf{I}{\mathbf{T}}^{(4)}={\mathbf{R}}^2-\frac{1}{3}\mathrm{Tr}({\mathbf{R}}^2)\mathbf{I},{\mathbf{T}}^{(5)}=\mathbf{R}{\mathbf{S}}^2-{\mathbf{S}}^2\mathbf{R},\hfill \\ \hfill {\mathbf{T}}^{(6)}& =\mathbf{S}{\mathbf{R}}^2+{\mathbf{R}}^2\mathbf{S}-\frac{2}{3}\mathrm{Tr}(\mathbf{S}{\mathbf{R}}^2)\mathbf{I},{\mathbf{T}}^{(7)}=\mathbf{R}\mathbf{S}{\mathbf{R}}^2-{\mathbf{R}}^2\mathbf{S}\mathbf{R},{\mathbf{T}}^{(8)}=\mathbf{S}\mathbf{R}{\mathbf{S}}^2-{\mathbf{S}}^2\mathbf{R}\mathbf{S},\hfill \\ \hfill {\mathbf{T}}^{(9)}& ={\mathbf{R}}^2{\mathbf{S}}^2+{\mathbf{S}}^2{\mathbf{R}}^2-\frac{2}{3}\mathrm{Tr}({\mathbf{S}}^2{\mathbf{R}}^2)\mathbf{I},{\mathbf{T}}^{(10)}=\mathbf{R}{\mathbf{S}}^2{\mathbf{R}}^2-{\mathbf{R}}^2{\mathbf{S}}^2\mathbf{R}.\hfill \end{array}$$ [4]

$$\begin{array}{cc}\hfill {\mathbf{B}}^{(1)}& =\mathbf{R},{\mathbf{B}}^{(2)}=\mathbf{S}\mathbf{R}+\mathbf{R}\mathbf{S},{\mathbf{B}}^{(3)}={\mathbf{S}}^2\mathbf{R}+\mathbf{R}{\mathbf{S}}^2,{\mathbf{B}}^{(4)}={\mathbf{R}}^2\mathbf{S}-\mathbf{S}{\mathbf{R}}^2,\hfill \\ \hfill {\mathbf{B}}^{(5)}& ={\mathbf{R}}^2{\mathbf{S}}^2-{\mathbf{S}}^2{\mathbf{R}}^2,{\mathbf{B}}^{(6)}=\mathbf{S}{\mathbf{R}}^2{\mathbf{S}}^2-{\mathbf{S}}^2{\mathbf{R}}^2\mathbf{S}.\hfill \end{array}$$ [5]

$$\begin{array}{cc}\hfill {\lambda }_1& =\mathrm{Tr}({\mathbf{S}}^2),{\lambda }_2=\mathrm{Tr}({\mathbf{R}}^2),{\lambda }_3=\mathrm{Tr}({\mathbf{S}}^3),{\lambda }_4=\mathrm{Tr}({\mathbf{R}}^2\mathbf{S}),{\lambda }_5=\mathrm{Tr}({\mathbf{S}}^2{\mathbf{R}}^2).\hfill \end{array}$$ [6]

Using S and R, we can construct general bases for tensors and scalars given in Eqs. 4–6. The ten T⁽ⁱ⁾ in Eq. 4 form the basis for symmetric tensors and the six B⁽ⁱ⁾ in Eq. 5 for skew-symmetric tensors; λ_i in Eq. 6 is the basis of scalar invariants required to determine the necessary coefficients. Since incompressibility gives $Tr(\mathbf{S})=0$ (and Tr(R)=0 trivially), it is easy to show that Tr(T⁽ⁱ⁾)=0. While the symmetric tensor basis T⁽ⁱ⁾ is formulated to be trace-free owing to incompressibility, a symmetric tensor basis does not in general have to be trace-free; indeed, such a basis will be somewhat different from T⁽ⁱ⁾ (36). Likewise, there are six scalar invariants for a general tensor basis (36), but incompressibility reduces the number to five, given in Eq. 3 (37).

Using the above framework, we can functionally model the pressure Hessian and viscous Laplacian tensors. However, note that while the pressure Hessian tensor is symmetric, it has a nonzero trace given by Eq. 2. Thus, we have to model the deviatoric part of the pressure Hessian tensor H_d as

$$\begin{array}{c}\hfill {\mathbf{H}}_{\mathrm{d}}\equiv \mathbf{H}-\frac{1}{3}\mathrm{Tr}(\mathbf{H})\mathbf{I}={\displaystyle \sum _{i=1}^{10}}{c}_1^{(i)}({\lambda }_1,...,{\lambda }_5){\mathbf{T}}^{(i)},\end{array}$$ [7]

where the ten coefficients c₁⁽ⁱ⁾ have to be determined as functions of the scalar invariants. The term Tr(H) (and hence the isotropic part of H) does not pose a closure problem, since it can be written exactly in terms of A as in Eq. 2. The viscous Laplacian can be decomposed simply into symmetric and skew-symmetric contributions as ∇²A = ν∇²S + ν∇²R, which can be modeled using the respective tensor bases:

$$\begin{array}{c}\hfill {\mathrm{\nabla }}^2\mathbf{A}={\displaystyle \sum _{i=1}^{10}}{c}_2^{(i)}({\lambda }_1,...,{\lambda }_5){\mathbf{T}}^{(i)}+{\displaystyle \sum _{i=1}^6}{c}_3^{(i)}({\lambda }_1,...,{\lambda }_5){\mathbf{B}}^{(i)}.\end{array}$$ [8]

Here, we need to evaluate the 16 coefficients c₂⁽ⁱ⁾ and c₃⁽ⁱ⁾ as functions of the scalar invariants.

This brief description nominally captures all previous attempts to model the velocity gradient dynamics. For example, the pressure Hessian models developed in refs. 29–32 retain up to second-order terms in Eq. 4, whereas the models for viscous Laplacian retain (in the same works) just the first-order term. In refs. 33 and 34, the pressure Hessian tensor was modeled in the same fashion as here. (However, no previous model accounted for the Re dependence of velocity gradient statistics, as we shall discuss in the next subsection.) In all these methods, the required scalar coefficients c₁⁽ⁱ⁾ to c₃⁽ⁱ⁾ were obtained as nonlinear scalar functions of A, satisfying a small set of the physical constraints. However, one can more generally utilize the power of deep learning to directly obtain the coefficients c₁⁽ⁱ⁾ to c₃⁽ⁱ⁾, thus, in principle, learning all the necessary physical constraints from the data itself (33, 34).

Nondimensionalization and Reynolds Number Dependence.

To utilize the above framework in conjunction with neural networks, it is important to nondimensionalize all quantities. There are several reasons. First, it is simply more convenient to work with nondimensional quantities in numerics. Second, it facilitates efficient learning (of network weights and biases) since the tensors in the bases are otherwise of different orders in A. For instance, while the pressure Hessian is second order in A, the symmetric tensor basis spans first to fifth order in A, implying that the coefficients c₁⁽ⁱ⁾ vary from order 1 to −3 in A. Appropriate nondimensionalization renders all coefficients to be of the same order, which leads to better and faster learning (13). Finally, nondimensionalization also allows us to appropriately introduce Re as a parameter, whereby the model system can be run at any chosen Re to obtain desired (nondimensional) statistics of velocity gradients.

However, given the multiscale nature of turbulence, the choice of variables for nondimensionalization is not unique. Since velocity gradients characterize small scales, a natural choice is to utilize the Kolmogorov time and length scales given as

$$\begin{array}{c}\hfill {\tau }_K={(\nu /⟨ϵ⟩)}^{1/2},{\eta }_K={({\nu }^3/⟨ϵ⟩)}^{1/4}.\end{array}$$ [9]

Here, ϵ = 2νS_ijS_ij is the energy dissipation rate and ⟨ ⋅ ⟩ denotes averaging over space and time. In homogeneous turbulence, we have ⟨S_ijS_ij⟩=⟨R_ijR_ij⟩=⟨A_ijA_ij⟩/2. Thus, ⟨ϵ⟩=ν⟨A_ijA_ij⟩, giving ⟨A_ijA_ij⟩τ_K² = 1, implying that 1/τ_K quantifies the rms amplitude of A. This justifies the choice of Kolmogorov variables to nondimensionalize A; in fact, the above relations between the mean quantities and τ_K allow us to impose convenient constraints while running the model system (as described in Materials and Methods). However, the choice is not obvious if one considers the extreme events and hence higher-order statistics of A, and also of pressure Hessian and viscous Laplacian terms, since, due to intermittency, they cannot be expected to scale on Kolmogorov variables (7, 8). We will persist with Kolmogorov normalization but introduce a phenomenological procedure to account for intermittency.

In summary, the following nondimensionalization is used: t^* = t/τ_K, x^* = x/η_K (i.e., ∇^* = η_K∇), A^* = Aτ_K, H^* = Hτ_K² (and H_d^* = H_dτ_K²). We then obtain the following equation for A^*:

$$\begin{array}{c}\hfill \frac{D{\mathbf{A}}^{\ast }}{D{t}^{\ast }}=-({{\mathbf{A}}^{\ast }}^2-\frac{1}{3}\mathrm{Tr}({{\mathbf{A}}^{\ast }}^2)\mathbf{I})-{\mathbf{H}}_{\mathrm{d}}^{\ast }+{{\mathrm{\nabla }}^{\ast }}^2{\mathbf{A}}^{\ast }.\end{array}$$ [10]

The terms H_d^* and ∇^*2A^* can now be modeled in terms of A^* using the previously described framework, i.e., utilizing Eqs. 7 and 8, where the tensor bases are appropriately replaced by their nondimensional counterparts, i.e., T^*(i) and B^*(i), and the coefficients c₁⁽ⁱ⁾, c₂⁽ⁱ⁾, c₃⁽ⁱ⁾ are dimensionless. It can be seen immediately that the above system does not have any Reynolds number dependence. This is not surprising since we utilized Kolmogorov scales for nondimensionalization, temporarily choosing to ignore intermittency. Thus, the Reynolds number dependence has to be reintroduced by hand (and validated a posteriori). Previous modeling attempts do not recognize this aspect and have consequently not captured the Reynolds number dependence; see, for example, refs. 25, 31, and 34.

It is worth stressing that there is no foolproof way of introducing the Re dependence into the system. For instance, one could use the large scales, say L for length and U for velocity (and L/U for time) for nondimensionalization, but it would be an incorrect choice for many aspects of the gradients. Instead, we devise the following pragmatic way to introduce the Reynolds number dependence and intermittency effects. We maintain the nondimensionalization by Kolmogorov variables and rescale the tensor bases as

$$\begin{array}{cc}\hfill {\mathbf{H}}_{\mathrm{d}}^{\ast }& ={\displaystyle \sum _{i=1}^{10}}{c}_1^{(i)}{R}_{\lambda }^{-{\beta }_1^{(i)}}{\mathbf{T}}^{\ast (i)},\hfill \end{array}$$ [11]

$$\begin{array}{cc}\hfill {{\mathrm{\nabla }}^{\ast }}^2{\mathbf{A}}^{\ast }& ={\displaystyle \sum _{i=1}^{10}}{c}_2^{(i)}{R}_{\lambda }^{-{\beta }_3^{(i)}}{\mathbf{T}}^{\ast (i)}+{\displaystyle \sum _{i=1}^6}{c}_3^{(i)}{R}_{\lambda }^{-{\beta }_3^{(i)}}{\mathbf{B}}^{\ast (i)}.\hfill \end{array}$$ [12]

Here, R_λ is the Reynolds number based on Taylor length scale [note that R_λ ∼ Re^1/2 (1)] and the exponents β₁⁽ⁱ⁾, β₂⁽ⁱ⁾, and β₃⁽ⁱ⁾ are additional model parameters which will be determined. This choice is motivated by two main reasons. First, the well-known multifractal description of turbulence suggests that velocity gradient statistics scale as power laws (or combinations of power laws) in R_λ (1–3). Second, the tensors in the bases span various orders of A, all of which feel the intermittency effects differently. Thus, the Reynolds number factors can rescale them to the same order, allowing for more efficient learning, essentially acting as additional physics-informed constraints to accommodate intermittency. If our physical understanding improves, it may well be possible to improve upon our present formulation.

Reynolds-Number-Scaled Tensor-Based Neural Network (ReS-TBNN).

We now consider the neural network architecture utilized to model the unclosed terms. The tensor-based neural network (TBNN), utilizing only the symmetric basis from Eq. 4, was first proposed by Ling et al. (38) for turbulence modeling of the Reynolds stress tensor. More recently, it was extended to modeling pressure Hessian in refs. 33 and 34, while continuing to ignore Reynolds number effects. Unlike traditional neural networks, TBNN utilizes two input layers. The architecture to model the pressure Hessian is shown in Fig. 1. The first input layer uses the scalar basis λ_i, which are then fed forward to multiple hidden layers to obtain the scalar coefficients c₁⁽ⁱ⁾ in the first output layer. The essence of this step is to model the scalar coefficients as strongly nonlinear functions of the scalar basis— an exercise traditionally performed by humans (29–31). This is precisely the step where deep neural networks are advantageous. The second input layer uses the rescaled tensor basis as input, for instance, R_λ^−β₁(i)T⁽ⁱ⁾, for the pressure Hessian. This second input layer is contracted with the first output layer to obtain the predicted pressure Hessian tensor in the final output layer, in accordance with Eq. 11. We reiterate that only the deviatoric part of pressure Hessian needs to be modeled. The architecture for viscous Laplacian is essentially identical to that shown in Fig. 1, with the difference that the first output layer and the second input layer have both 16 nodes, corresponding to the coefficients c₂⁽ⁱ⁾ and c₃⁽ⁱ⁾ and the tensors T⁽ⁱ⁾ and B⁽ⁱ⁾, with appropriate prefactors corresponding to the Reynolds number scaling.

Fig. 1.

Reynolds-number-scaled tensor-based neural network (ReS-TBNN) architecture utilized for modeling the deviatoric pressure Hessian, based on Eq. 11. The first output layer and the second input layer both have 10 nodes. Note that the exponents β⁽ⁱ⁾s are not fixed inputs but obtained from network training using standard backpropagation. A similar network is utilized for the viscous Laplacian, utilizing both the symmetric and skew-symmetric tensor bases, as mentioned in Eq. 12. The first output layer and second input layer have both 16 nodes in this case.

Training and Validation of the ReS-TBNN Model

DNS Data.

To train the ReS-TBNN model, the “ground truth” data are obtained from a massive DNS database corresponding to forced stationary isotropic turbulence in a periodic domain (39). The simulations were performed using Fourier pseudospectral methods (40), allowing us to obtain the data with the highest accuracy practicable. A key aspect of our data is that we have simultaneously achieved a wide range of Reynolds numbers and the necessary small-scale resolution to accurately resolve extreme events (41, 42). Both of these conditions are indispensable for successful model development. The Taylor-scale-based Reynolds number R_λ of our database ranges from 140 to 1,300. The data have been since utilized and validated in several recent studies (43–47). A brief account of DNS and the database is provided in Materials and Methods, and more details can be found in the references just mentioned. In order to train our network, only the data for R_λ= 140 to 650 are utilized; subsequently, we will demonstrate that the trained network can predict with reasonable success the statistics at higher (unseen) R_λ= 1,300. Though this R_λ is only twice as large as the largest one used in the training, its usefulness should be assessed in the context of the computational expense of DNS, which would be easily 100 times larger. This is because the cost of DNS increases at least as strongly as R_λ⁶, going up to R_λ⁸ in the limit of large R_λ, to accurately resolve the smallest scales (7, 9).

We point out that the model can also be trained over the range R_λ= 140 to 390 and used to predict results at R_λ = 650 and 1,300. However, learning from this smaller range of R_λ is suboptimal for capturing trends that can be extrapolated to significantly higher R_λ. Additionally, learning from low R_λ alone is not helpful because many features of turbulence are not fully developed for those conditions. For the present, learning from the range R_λ= 140 to 650 and predicting the performance at R_λ= 1,300 allows an optimal situation. As remarked in the previous paragraph, the corresponding DNS effort would be much more expensive. Clearly, prediction over a wider range would be desirable. One can also use the full range of available of R_λ= 140 to 1,300 for learning and predicting at a higher unseen R_λ. Since the DNS data at higher R_λ are not yet available, the predictions would be unverifiable, so we leave this task for the future.

Training of the ReS-TBNN Model.

The training of the ReS-TBNN model is implemented in FORTRAN using a massively parallel in-house deep-learning library. We utilize the distributed training paradigm with data parallelism, i.e., the training data are split across many processors, with each processor having access to the same model. The model parameters are synchronized via interprocessor communication after each training epoch (executed using MPI collective communication calls). To update the parameters of the neural network, the quadratic loss function is minimized using the standard backpropagation algorithm (13). For example, for the pressure Hessian tensor, the loss function is given by

$$\begin{array}{c}\hfill \mathcal{L}=\frac{1}{2N_{\mathrm{data}}}{\displaystyle \sum _{m=1}^{N_{\mathrm{data}}}}||{\widehat{\mathbf{H}}}_{\mathrm{d}}^{(m)}-{\mathbf{H}}_{\mathrm{d}}^{(m)}{||}_F^2.\end{array}$$ [13]

where ${\widehat{\mathbf{H}}}_d$ is the model output and ||⋅||_F denotes the Frobenius norm. A similar loss function can also be written for the viscous Laplacian term. The network weights and biases, as well as the exponents β⁽ⁱ⁾ in Eqs. 11 and 12, are simply updated using gradient descent: x = x − α(∂ℒ/∂x), where x is the variable being updated and α is the learning rate.

The training data are compiled from DNS runs corresponding to R_λ= 140 to 650. About one billion data points are utilized for training, split evenly across all R_λ in the range. The training is performed on about a thousand processors (with each processor handling one million data points). Note that each data point corresponds to a combination of the tensors A, H_d, and ∇²A and the particular value of R_λ. As discussed earlier, all variables are nondimensionalized by the Kolmogorov scales. The training is performed for many thousands of epochs, until the loss function reaches a plateau (Materials and Methods). The learning rate is always kept low at 10⁻⁶. As one might expect, the choice of the hyperparameters, such as the number of hidden layers and the number of nodes per layer, plays a crucial role in obtaining the best model. To select the optimal network configuration, we do not utilize a validation dataset, but directly compare the velocity gradient statistics with DNS results. We found that a network with about 25 layers, each with about 50 nodes, provides optimal results; increasing the number of layers and the number of nodes per layer results in only marginal improvement [and can also lead to overfitting (13)].

Comparison of the ReS-TBNN Model with DNS

The effectiveness of the trained ReS-TBNN model will now be evaluated by comparing its outcome with DNS results. We first focus on the Reynolds number trend of velocity gradient statistics (this being a key contribution of the model). We particularly consider the PDFs that display increasingly non-Gaussian tails with an increasing Reynolds number because of intermittency. All components of A exhibit intermittency, but it is convenient to consider scalar quantities of direct physical significance, such as the energy dissipation rate, whose mean value is the net energy flux from large to small scales. As is well known, the instantaneous energy transfers are highly intermittent, leading to extreme dissipation events (48).

Fig. 2 shows comparisons of the PDFs of the energy dissipation rate, normalized by its mean value, from DNS and the model. Panels A–C illustrate the comparison on log–log scales at R_λ = 140, 650, and 1,300, respectively, showing excellent agreement between the two results. The ReS-TBNN model has been trained only up to R_λ = 650 and has not seen any data for R_λ= 1,300. For a closer inspection, Panels D and E show the same comparisons on linear-log scales for all R_λ available. The model captures the intermittent tails qualitatively well, though the extreme events are overpredicted (see below). We believe that this overprediction occurs because of the Reynolds number scaling of the tensor bases; essentially, the rescaling serves to normalize the tensor and the extreme events have slightly stronger influence on the weights and biases. Note that, similar to the dissipation rate, one can also consider other scalar measures derived from A, such as enstrophy Ω = ω_iω_i, where ω_i = ϵ_ijkA_jk is the vorticity vector (with ϵ_ijk being the Levi-Civita symbol). Although not shown here, the agreement observed for enstrophy is similar.

Fig. 2.

Comparisons of probability density functions (PDFs) of the energy dissipation rate, nondimensionalized by the mean value, as obtained from network model and DNS. Panels A–C show the comparison at R_λ = 140, 650, and 1,300, respectively, on log–log scales. Panel D shows the PDFs from DNS for various R_λ on lin–log scales highlighting the intermittency of PDF tails. Panel E shows the PDFs obtained from the network model for the same set of R_λ as panel D, showing the effectiveness of the model in predicting the PDF tails. We reiterate that R_λ= 1,300 is never seen by the model.

The overprediction by the model occurs principally for events with a probability less than about 10⁻⁹. Such events are obviously very important for high-order moments, but the reliability of such very large moments is not quite assured for the DNS data itself. For example, suppose we compute the sixth moment of the energy dissipation rate. This is equivalent to obtaining the 12th-order moment of velocity gradients, which would be stretching one’s credulity even for the large size of the present database. To better understand how well the model and the DNS agree, we directly compare some moments from the PDFs. In Table 1, we list the second- and fourth-order moments of dissipation and enstrophy from both the model and the DNS. We also compare the third- and fourth-order moments of individual components of A. The results for skewness of A₁₂ are not shown since it is zero (within statistical error) for both the DNS and the model. Clearly, the results from the model are very satisfactory for the second order of dissipation and enstrophy but less so for the fourth.

Table 1.

Comparison of various statistics from DNS and the model

R λ	140	240	390	650	1,300
⟨ϵ2⟩/⟨ϵ⟩2
DNS	2.75	3.28	3.86	4.67	6.27
model	2.49	2.78	3.74	5.16	7.72
⟨Ω2⟩/⟨Ω⟩2
DNS	4.89	6.22	7.49	9.26	12.6
model	4.48	5.50	7.25	9.80	14.4
⟨ϵ4⟩/⟨ϵ2⟩2
DNS	37.8	83.8	171	387	1,298
model	101	260	434	628	2,355
⟨Ω4⟩/⟨Ω2⟩2
DNS	150	345	741	1,636	6,917
model	316	812	1,666	3,101	13,210
skewness A11
DNS	-0.52	-0.55	-0.59	-0.63	-0.70
model	-0.49	-0.54	-0.60	-0.66	-0.75
flatness A11
DNS	5.73	6.82	8.02	9.87	13.1
model	5.20	5.80	7.80	10.9	15.2
flatness A12
DNS	8.71	10.9	13.2	16.5	22.2
model	8.07	9.90	13.0	17.8	25.8

Shown are the second- and fourth-order moments of dissipation (ϵ) and enstrophy (Ω), and the skewness and flatness factors of longitudinal (A₁₁) and transverse (A₁₂) velocity gradient components. Note that the skewness of A₁₂ is not shown, since it is zero (within statistical uncertainty), from both DNS and the model.

It is worth noting that the second-order moments of dissipation and enstrophy can be directly related to fourth-order moments of velocity gradient components in the following manner (49):

$$\begin{array}{c}\hfill ⟨{ϵ}^2⟩/{⟨ϵ⟩}^2=\frac{15}{7}F(A_{11}),⟨{\mathrm{\Omega }}^2⟩/{⟨\mathrm{\Omega }⟩}^2=\frac{9}{5}F(A_{12}).\end{array}$$ [14]

Here, F(A₁₁) and F(A₁₂) are the flatness factors of the components A₁₁ and A₁₂, respectively. These results are exact for isotropic turbulence at any Reynolds number and are nominally satisfied far from solid boundaries in other turbulent flows at high Reynolds numbers, i.e., when local isotropy holds (2, 4). It can be seen from Table 1 that both these relations are well satisfied in the model and DNS results. In fact, many such isotropic relations exist for various moment orders of velocity gradients. For instance, for the second-order moments, they are

$$\begin{array}{cc}& ⟨A_{\alpha \alpha }^2⟩=⟨A_{\beta \beta }^2⟩,\text{for}\alpha =\beta \hfill \\ \hfill & ⟨A_{\alpha \beta }^2⟩=2⟨A_{\alpha \alpha }^2⟩,\text{for}\alpha \ne \beta \hfill \\ \hfill & ⟨A_{\alpha \alpha }{A}_{\beta \beta }⟩=⟨A_{\alpha \beta }{A}_{\beta \alpha }⟩=-⟨A_{\alpha \alpha }^2⟩/2,\text{for}\alpha \ne \beta \hfill \end{array}$$ [15]

where repeated indices in α and β do not imply summation. Essentially, all second-order moments can be simply described by the moment ⟨A₁₁²⟩. From the above, it also follows

$$\begin{array}{cc}\hfill ⟨ϵ⟩/\nu & =2⟨S_{\mathit{ij}}{S}_{\mathit{ij}}⟩=15⟨A_{11}^2⟩,\hfill \\ \hfill ⟨\mathrm{\Omega }⟩& =2⟨R_{\mathit{ij}}{R}_{\mathit{ij}}⟩=15⟨A_{11}^2⟩.\hfill \end{array}$$ [16]

Although not shown explicitly, we note that the above relations are all satisfied in our model results (and, of course, DNS) at all Reynolds numbers.

The comparisons in Fig. 2 and Table 1 predominantly focused on Reynolds number scaling of various individual statistics. However, it is equally important to capture the structure of velocity gradient tensor. We examine two well-known universal results to this end, the first being the alignment of vorticity vector with the eigenvectors of strain tensor, shown in Fig. 3. Panel a shows the PDFs of the cosines of the alignment from DNS. Consistent with the well-known result from the literature (23), vorticity preferentially aligns with the second eigenvector of strain and is weakly orthogonal to the third eigenvector, whereas there is no preferential alignment with the first eigenvector. There is virtually no R_λ dependence of these PDFs (as noted in ref. 24). In Fig. 3B, the corresponding result is shown from the ReS-TBNN model. The model captures the trends very well, with slight enhancement of the respective alignments. This trend is consistent with the result in Fig. 2 where the model slightly overpredicts extreme events [note that the alignments are enhanced when considering extreme events (24)]. We also note that the model shows only very weak Reynolds number dependence of the alignment properties, which is inconsequential for all practical purposes.

Fig. 3.

Comparison of the PDFs of the cosine of the angles between the vorticity unit vector $\widehat{\mathit{\omega }}$ and the eigenvectors of the strain tensor e_i, corresponding to eigenvalues λ_i, where λ₁ ≥ λ₂ ≥ λ₃. Panel A shows the result from DNS corresponding to R_λ=1,300 in solid lines and R_λ = 140 in dashed lines. Panel B shows the result from the network model corresponding to same R_λ values in solid and dashed lines. The alignment PDFs have almost no R_λ dependence in DNS; that shown by the model is negligible also.

The second structural aspect concerns the local critical point analysis of A, which identifies the flow topology using the second and third invariants of A (50): Q = −Tr(A²)/2 and R = −Tr(A³)/3. (Note that the first invariant of A, i.e., its trace, is zero from incompressibility.) The joint PDF of these two invariants is known to exhibit a universal tear-drop shape (25, 51). For a final comparison between DNS and model, we compare the joint PDF in Fig. 4 A and B obtained from DNS and the model, respectively, at R_λ = 140; the same results for R_λ=1,300 are shown in Fig. 4 C and D. In both cases, the model predicts the joint PDF quite well in all four quadrants. It is worth noting that the joint PDFs also exhibit strong intermittency with increasing Reynolds number; this aspect is again well captured by the model, similar to the result in Fig. 2.

Fig. 4.

Comparison of joint PDFs of the invariants of the velocity gradient tensor, defined as $Q^{\ast }=-\frac{1}{2}Tr({\mathbf{A}}^2){\tau }_K^2$ and $R^{\ast }=-\frac{1}{3}Tr({\mathbf{A}}^3){\tau }_K^3$. Panel A shows the result from DNS at R_λ = 140, and Panel B shows the corresponding result from the network model. Panel C shows the DNS result at R_λ= 1,300, and Panel D shows the corresponding result from the model.

As a final remark, we note that one can compare many other quantities to evaluate the performance of the model (with respect to DNS results). For instance, one can consider the temporal dynamics of velocity gradients projected onto the Q − R plane (29, 31, 34) or how they influence trajectories of particles in turbulence (32). Such comprehensive studies, including detailed comparisons of our model with the previous ones, will be presented in a subsequent paper. In the current study, we have highlighted the most important contribution of our model, which is to capture intermittency and Reynolds numbers trends of velocity gradient statistics.

Discussion

In studying the dynamics of turbulence, the DNS of Navier–Stokes equations on massive supercomputers is now an established area for gaining a fuller understanding of flow physics, leading to more reliable predictions. However, both theoretical and practical needs demand ever-increasing size of computations, so fluid turbulence will always remain, for the foreseeable future, as one of the frontier computational problems, no matter how large the supercomputers become.

In this regard, the major bottleneck is the need to simulate small scales of turbulence with high fidelity (adequate resolution in space and time, convergence, etc.). To make progress on real problems, one needs to model small scales well, for instance, in LES where large scales are resolved, but small scales are modeled assuming a degree of universality. This modeling approach has been largely guided by “human learning,” often resulting in ad hoc considerations depending on the flow. As modern methods of deep machine learning have expanded, it appears possible for them to aid in modeling by directly learning from a vast amount of high-fidelity data that are already available over some range of Reynolds numbers. In this scenario, deep neural networks are allowed to do the fitting at a deeper level of instantaneous data, in the process satisfying a substantially larger set of constraints than possible by “human learning.” If this attempt succeeds, we will have a powerful tool in assimilating the lower Reynolds number data for predicting flow properties at higher unseen Reynolds numbers. This is a difficult problem given the nature of turbulence.

In this paper, we have made a ground-level attempt toward our stated goal. We have demonstrated that the small-scale dynamics of turbulence, as captured by velocity gradients, can be modeled reasonably well using deep neural networks. The deep neural networks are set up to functionally model the nonlocal pressure and viscous contributions to velocity gradient dynamics. The networks are then trained on a range of Reynolds numbers available from DNS, and the training is leveraged to predict results at higher Reynolds numbers, whose properties the network does not know in advance. The effort is very encouraging not only in predicting the intermittency of velocity gradients with increasing Reynolds number but also various signature topological properties of velocity gradient tensor, such as alignment of vorticity with strain rate eigenvectors, and the joint PDFs of invariants displaying a tear-drop shape. Overall, the modeling effort developed here provides a substantial improvement upon prior work, especially with respect to the robustness of the local functional modeling of pressure and viscous terms across a range of Reynolds numbers.

There are certain shortcomings of the trained model when considering truly extreme events; fortunately, they contribute significantly only to higher-order moments. It should be possible to further improve this aspect by incorporating the current deep learning approach in alternative frameworks for velocity gradient dynamics, which incorporate Reynolds number dependencies more naturally; see, for example, refs. 52 and 53. Likewise, it would be worth utilizing recurrent neural networks (13) to enable direct learning of both spatial and temporal correlations in the data. While the current model already captures many temporal correlations well (although they are not reported here), the training is performed purely on spatial data. Note that in stationary homogeneous turbulence explored here, one-point and one-time statistics are identical. Nevertheless, we note that the incorporation of both spatial and temporal learning could greatly improve the effectiveness of the current framework.

Finally, it would also be worth expanding the current effort in a more concerted way to other modeling paradigms such as LES (54)—allowing one to tackle more complex turbulent flows at Reynolds numbers of practical interest in nature and engineering. Such an extension can be accomplished, for instance, by considering filtered velocity gradient tensor, which would be amenable to the same tensor framework as utilized here (34, 38). Likewise, the framework developed here can also be extended to study the dynamics of scalar gradients in turbulent mixing problems (55), especially in the high Schmidt number regime; though these conditions are even more challenging for DNS (56), recent efforts have led to generation of high-fidelity data at reasonably high Reynolds numbers (57, 58). Efforts in these directions are under way and will be reported as future work.

Materials and Methods

Direct Numerical Simulations.

The data utilized here are obtained by the DNS of incompressible Navier–Stokes equations

$$\begin{array}{c}\hfill \partial \mathbf{u}/\partial t+\mathbf{u}·\mathrm{\nabla }\mathbf{u}=-\mathrm{\nabla }P+\nu {\mathrm{\nabla }}^2\mathbf{u}+\mathbf{f},\end{array}$$ [17]

where u is the velocity, satisfying ∇ ⋅ u = 0, P is the kinematic pressure, and ν is the kinematic viscosity. The term f corresponds to large-scale forcing required to maintain a statistically stationary state. The simulations correspond to the canonical setup of isotropic turbulence with periodic boundary conditions in a cubic domain of side length L₀ = 2π. It is well known that such a setup allows one to reach the highest Reynolds numbers in DNS and is ideal for studying small scales (39). Taking the gradient of Eq. 17 leads to Eq. 1, without the forcing term. When performing the Monte-Carlo simulations of the ReS-TBNN model, a forcing term is reintroduced to mimic this effect and achieve stationary statistics (as described in the subsection after the next).

The DNS domain consists of N³ grid points with uniform grid spacing Δx = L₀/N in each direction. The equations are solved using a massively parallelized version of the well-known Fourier pseudospectral algorithm of Rogallo (40); the resulting aliasing errors are controlled by a combination of grid shifting and spherical truncation (59). For time integration, the second-order Runge–Kutta method is used, with the time step Δt subject to the Courant number C constraint for numerical stability: Δt = CΔx/||u||_∞ (where ||⋅||_∞ is the L^∞ norm). An important consideration in studying velocity gradients and associated extreme events is that of spatial resolution, captured by the ratio Δx/η_K (η_K being the Kolmogorov length scale, defined earlier in Eq. 9). For pseudospectral DNS, spatial resolution is also prescribed by the parameter k_maxη_K, where $k_{\max }=\sqrt{2}N/3$ is the maximum resolved wavenumber. It can be easily shown that Δx/η ≈ 3/k_maxη_K. All our runs correspond to high spatial resolution, going up to k_maxη_K ≈ 6 to accurately resolve extreme events. The DNS database, along with various simulation parameters, are summarized in Table 2.

Table 2.

Various simulation parameters for the DNS runs utilized here: the Taylor-scale Reynolds number (R_λ), the number of grid points (N³), spatial resolution (k_maxη), ratio of large-eddy turnover time (T_E) to Kolmogorov time scale (τ_K), and length of simulation (T_sim) in statistically stationary state

R λ	N 3	k max η	TE/τK	T sim
140	1,0243	5.82	16.0	6.5TE
240	2,0483	5.70	30.3	6.0TE
390	4,0963	5.81	48.4	4.0TE
650	8,1923	5.65	74.4	2.0TE
1,300	12,2883	2.95	147.4	20τK

We note that one can also utilize Lagrangian data, as obtained from following fluid particle trajectories along with the Eulerian DNS (60), to train and validate the deep learning network (34). However, since the network relies on obtaining a local functional closure, it does not make a difference whether Eulerian or Lagrangian data are utilized, provided they both are statistically stationary. Note that Lagrangian data are obtained from Eulerian data using spline interpolation (61, 62) and thus are less accurate, especially for higher-order moments (46, 60). Even with this caveat, it would be desirable to construct recurrent neural networks (13), to enable direct learning of both spatial and temporal dependencies in the data, possibly leading to improved predictive capabilities.

ReS-TBNN Loss Function.

Fig. 5 shows the behavior of the loss functions, for training of both pressure Hessian and viscous Laplacian terms, versus the number of epochs lapsed as the training is performed. Evidently, they become flat beyond a certain point. An early stopping criterion, as marked in Fig. 5, is utilized for training of the networks to avoid overfitting (63).

Fig. 5.

Decay of the loss function during training of the ReS-TBNN for pressure Hessian and viscous Laplacian terms. One epoch corresponds to the entire set of available training data spanning R_λ= 140 to 650.

Monte-Carlo Simulations of the ReS-TBNN Model.

Once the pressure Hessian and viscous Laplacian terms are modeled as functions of A, we can obtain a closed system in A, which can be solved for arbitrary initial conditions. However, it is also necessary to add a forcing term to the model (26), mimicking the effects of large-scale forcing, to achieve stationary statistics. Additionally, the forcing term also acts to reproduce some effects of nonlocality (of pressure Hessian), which are lost due to a local functional closure (28, 29, 31). Thus, the closed system is posed in the form of a stochastic ODE, given as

$$\begin{array}{c}\hfill d{\mathbf{A}}^{\ast }=\mathcal{F}({\mathbf{A}}^{\ast })d{t}^{\ast }+d{\mathbf{F}}^{\ast }.\end{array}$$ [18]

Here, ℱ(A^*) is the deterministic tensor function, obtained from Eqs. 10–12 and given as

$$\begin{array}{cc}\hfill \mathcal{F}({\mathbf{A}}^{\ast })=& -({{\mathbf{A}}^{\ast }}^2-\frac{1}{3}\mathrm{Tr}({{\mathbf{A}}^{\ast }}^2)\mathbf{I})-{\displaystyle \sum _{i=1}^{10}}{c}_1^{(i)}{R}_{\lambda }^{-{\beta }_1^{(i)}}{\mathbf{T}}^{\ast (i)}\hfill \\ \hfill & +{\displaystyle \sum _{i=1}^{10}}{c}_2^{(i)}{R}_{\lambda }^{-{\beta }_2^{(i)}}{\mathbf{T}}^{\ast (i)}+{\displaystyle \sum _{i=1}^6}{c}_3^{(i)}{R}_{\lambda }^{-{\beta }_3^{(i)}}{\mathbf{B}}^{\ast (i)}.\hfill \end{array}$$ [19]

and dF^* is the stochastic forcing term

$$\begin{array}{c}\hfill d{F}_{\mathit{ij}}^{\ast }=b_{\mathit{ijkl}}d{W}_{\mathit{kl}}.\end{array}$$ [20]

built on the tensorial Wiener process, i.e., ⟨dW_ij⟩=0 and ⟨dW_ijdW_kl⟩=δ_ikδ_jldt^* with diffusion tensor D_ijkl = b_ijmnb_klmn.

For the drift term b_ijkl, we utilize the result of ref. 31

$$\begin{array}{c}\hfill b_{\mathit{ijkl}}=-\frac{1}{3}{D}_S{\delta }_{\mathit{ij}}{\delta }_{\mathit{kl}}+\frac{1}{2}\left(D_S+D_R\right){\delta }_{\mathit{ik}}{\delta }_{\mathit{jl}}+\frac{1}{2}\left(D_S-D_R\right){\delta }_{\mathit{il}}{\delta }_{\mathit{jk}}.\end{array}$$ [21]

where D_S and D_R are free parameters that can be tuned to appropriately force the symmetric and skew-symmetric parts of A^*, i.e., S^* and R^*, respectively, allowing us to impose consistency conditions for stationarity: ⟨S_ij^*S_ij^*⟩=1/2 and ⟨R_ij^*R_ij^*⟩=1/2. Note that the former follows directly from the definition of Kolmogorov time scale, whereas the latter follows from statistical homogeneity. It also readily follows that ⟨A_ij^*A_ij^*⟩=1. The Monte-Carlo simulations are performed starting from random Gaussian initial conditions of A^*, until a stationary state as prescribed by the above conditions is reached. Thereafter, the simulations are extended for a desired duration to obtain converged statistics.

We note that given the complex functional form of the deep learning closure, we encounter some rogue trajectories (since the deep learning–based closure does not guarantee stability). The encounter rate is only about one in a million. Such trajectories are simply discarded from the ensemble but, if desired, can be regularized as described in ref. 32.

Data, Materials, and Software Availability

All study data are included in the article.