A Newton–Krylov method with an approximate analytical Jacobian for implicit solution of Navier–Stokes equations on staggered overset-curvilinear grids with immersed boundaries

Abstract

The explicit and semi-implicit schemes in flow simulations involving complex geometries and moving boundaries suffer from time-step size restriction and low convergence rates. Implicit schemes can be used to overcome these restrictions, but implementing them to solve the Navier-Stokes equations is not straightforward due to their non-linearity. Among the implicit schemes for nonlinear equations, Newton-based techniques are preferred over fixed-point techniques because of their high convergence rate but each Newton iteration is more expensive than a fixed-point iteration. Krylov subspace methods are one of the most advanced iterative methods that can be combined with Newton methods, i.e., Newton-Krylov Methods (NKMs) to solve non-linear systems of equations. The success of NKMs vastly depends on the scheme for forming the Jacobian, e.g., automatic differentiation is very expensive, and matrix-free methods without a preconditioner slow down as the mesh is refined. A novel, computationally inexpensive analytical Jacobian for NKM is developed to solve unsteady incompressible Navier-Stokes momentum equations on staggered overset-curvilinear grids with immersed boundaries. Moreover, the analytical Jacobian is used to form preconditioner for matrix-free method in order to improve its performance. The NKM with the analytical Jacobian was validated and verified against Taylor-Green vortex, inline oscillations of a cylinder in a fluid initially at rest, and pulsatile flow in a 90 degree bend. The capability of the method in handling complex geometries with multiple overset grids and immersed boundaries is shown by simulating an intracranial aneurysm. It was shown that the NKM with an analytical Jacobian is 1.17 to 14.77 times faster than the fixed-point Runge-Kutta method, and 1.74 to 152.3 times (excluding an intensively stretched grid) faster than automatic differentiation depending on the grid (size) and the flow problem. In addition, it was shown that using only the diagonal of the Jacobian further improves the performance by 42 – 74% compared to the full Jacobian. The NKM with an analytical Jacobian showed better performance than the fixed point Runge-Kutta because it converged with higher time steps and in approximately 30% less iterations even when the grid was stretched and the Reynold number was increased. In fact, stretching the grid decreased the performance of all methods, but the fixed-point Runge-Kutta performance decreased 4.57 and 2.26 times more than NKM with a diagonal Jacobian when the stretching factor was increased, respectively. The NKM with a diagonal analytical Jacobian and matrix-free method with an analytical preconditioner are the fastest methods and the superiority of one to another depends on the flow problem. Furthermore, the implemented methods are fully parallelized with parallel efficiency of 80–90% on the problems tested. The NKM with the analytical Jacobian can guide building preconditioners for other techniques to improve their performance in the future.

1. Introduction

Simulating flows involving arbitrarily moving complex geometries are of interest in many areas, such as flow inside the vascular systems [1, 2, 3, 4, 5, 6, 7], swimming/flying in nature [8, 9, 10, 11, 12, 13], flows with suspended particles [14, 15, 16], etc. Currently, one of the main challenges in simulating such flows is the small time-step size restriction, e.g., Griffith [17] discussed the severe restriction on the time-step size imposed by his explicit scheme in simulations of an aortic heart valve using the immersed boundary method. Mittal et al. [18] used an implicit Crank-Nicolson scheme for the diffusion terms to eliminate the numerical instabilities, while the non-linear convective terms were treated explicitly in their immersed boundary method. The simulations of Mangual et al. [19], Domenichini and Pedrizzetti [20], de Tullio et al. [21], and Tytell et al. [22] all used the immersed boundary method, which sets small time-step restrictions due to their explicit nature. Robust implicit time integrations are needed to reduce the time-step limitations of such simulations, which will lower the computational cost and allow for higher spatial resolution. However, developing fully implicit methods for Navier-Stokes equations is not straightforward due to their non-linearity. In these cases, the methods for solving non-linear systems of equations need to be employed. There are two main categories of implicit solvers for non-linear systems of equations: 1) fixed-point methods; and 2) Newton-based methods. The applied mathematics community has emphasized on Newton-based methods for boundary value problems in which the steady-state solution of non-linear equations is desired [23]. However, the computational physics community has focused on fixed-point type methods for initial value problems, where dynamic evolution of the equations is required [23].

The first category is interchangeably named as fixed-point, Picard linearization, and successive substitution method [24]. The fixed-point method for solving the system of non-linear algebraic equations F(U⃗) = 0 can be written as:

$$F({\overrightarrow{U}}^{(k)})={\overrightarrow{U}}^{(k+1)}-{\overrightarrow{U}}^{(k)}=\mathrm{\Delta }{\overrightarrow{U}}^{(k)}$$ (1)

where k is the fixed-point iteration index. Dual time-stepping methods can be viewed as a fixed-point iteration as well [25, 26]. Fixed-point methods are implemented in conjunction with the pressure-velocity coupling algorithms such as artificial compressibility [26], SIMPLE family [27], PISO [28], and fractional-step [29] for solving incompressible Navier-Stokes equation. Fractional-step requires smaller time-step size in comparison to SIMPLE and PISO, while it requires less correction iterations than SIMPLE and PISO [30].

The second category is the Newton method, which is a classical algorithm for finding a solution to the system of non-linear algebraic equations F(U⃗) = 0 [31, 32]. Given an initial guess U⃗⁽⁰⁾, a sequence of steps ΔU⃗^(k) is computed as follows:

$$\begin{array}{r}A({\overrightarrow{U}}^{(k)})\mathrm{\Delta }{\overrightarrow{U}}^{(k)}=-F({\overrightarrow{U}}^{(k)})\\ {\overrightarrow{U}}^{(k+1)}={\overrightarrow{U}}^{(k)}+\mathrm{\Delta }{\overrightarrow{U}}^{(k)}\end{array}$$ (2)

where k is Newton iteration index and $A=\frac{d\overrightarrow{F}}{d\overrightarrow{U}}$ is the associated Jacobian matrix [31, 33]. The Newton Eq. (2) is a system of linear algebraic equations which must be solved at each Newton iteration (k). Newton-based methods have been previously implemented in conjunction with the pressure-velocity coupling algorithms such as velocity-vorticity method [34, 35], as a multigrid smoother in a SIMPLE method [36], and fractional-step [37] for solving incompressible Navier-Stokes equation. The main advantage of Newton methods over fixed-point type methods is the ability to take large time-steps and super-linear convergence rates [38]. On the other hand, requiring a sufficiently good initial condition is the Newton methods’ drawback [39, 38]. Considering the advantages of Newton methods, they are implemented in this framework and are systematically compared against a fixed-point method.

The Newton equation at each k can be solved by either direct or iterative methods. Because A is typically approximated (the approximation of A does not affect the solution as long as the Newton iterations converge) and the initial condition is typically far from the solution, the exact solution at each iteration (k) may not be justified. Therefore, an iterative method is typically used for solving Newton equations at each iteration. In this case, the exact solution of Eq. (2) is not obtained at each iteration k, which is referred to as the inexact Newton method [40]. Therefore, a strict quadratic convergence is typically not achieved [23]. Some of the most advanced iterative methods are the Krylov subspace methods [41, 42, 43, 44] including the generalized minimal residual method (GMRES) [45] and preconditioned GMRES [46]. Krylov subspace methods were introduced by Hestenes et al. [47] and Reid [48] as iterative methods to solve large linear systems of equations.

For solving Newton equations using a Krylov subspace method either the Jacobian (A) or the Jacobian-vector product (AΔU⃗) should be formed by: (a) a matrix-free (MF); (b) an automatic differentiation (AD); or (c) an analytical Jacobian method. In MF, the Jacobian matrix is not explicitly formed because Krylov subspace methods only require the Jacobian-vector product to carry out the iterations (not the individual elements of the Jacobian) [23]. However, successful application of MF to any given problem depends on the appropriate preconditioner, which might be challenging to obtain [23]. In AD, the Jacobian (not the Jacobian-vector product) is approximated using automatic differentiation [49]. This method can accelerate convergence by more than an order of magnitude relative to an explicit solver [50]. AD was reported to improve the performance and robustness of a finite volume scheme by using a set of techniques for computing accurate derivatives of non-linear functions compared to an approximate one [51]. Also, the vertex-elimination, automatic differentiation approach was used to calculate the Jacobians of functions as efficient as hand-coded Jacobian [52]. However, forming the complete Jacobian for a large system of equations is extremely expensive because it requires many evaluations of the discretized equations to approximate the Jacobian’s derivatives [23]. Finally, the Jacobian is formed analytically in analytical Jacobian. Briley-McDonald [53] and Beam-Warming [54, 55] pioneered applying analytical Jacobian to solve the compressible Navier-Stokes equations in curvilinear coordinates (partial transformation, i.e., the dependent variables such as velocities are in Cartesian coordinates but the derivatives are in curvilinear coordinates) on non-staggered grids. An application of Beam-Warming method in two-dimensions (2D) was first presented by Steger [56] and in 3D by Pulliam and Steger [57]. Pulliam et al. introduced a diagonalization of the Jacobian blocks for decreasing the computational cost [58]. In fact, the Jacobian of the Navier-Stokes in Cartesian or partial transformation into curvilinear coordinates on non-staggered grids is known [59, 30], which has applied to implicitly solve compressible and incompressible flows. However, there is no analytical Jacobian, to the best of the authors’ knowledge, derived or used for fully transformed Navier-Stokes equations on staggered grids. In fact, the derivation of an analytical Jacobian for advanced schemes, such as upwind methods, can become very complex as recognized by several researchers [60, 51, 61].

We are interested in discretizing the Navier-Stokes equations on staggered grids because using non-staggered grids requires adding a fourth-order dissipation to the pressure-Poisson equation to avoid odd-even decoupling and obtain a smooth pressure field, which can produce large errors near moving boundaries that typically create large pressure gradients [62]. The fourth-order dissipation does not allow the discrete continuity equation to be satisfied to machine zero. However, the discretization on staggered grids does not require adding the dissipation term. Therefore, if staggered grids are used, the discrete continuity equation on the control volume can be satisfied to machine zero given that the pressure-Poisson equation converges to machine zero. To clarify the difficulties in the derivation of an analytical Jacobian on a staggered grid, for example, Fig. 1 presents the stored location of variables U¹ and U² and non-linear functions F¹ and F² on a 2D staggered grid. Statement (3) is the part of the Jacobian for the cell plotted in Fig. 1, which is multiplied by the corresponding $\left[\begin{array}{l}\mathrm{\Delta }{U}_{(i+\frac{1}{2},j)}^1\hfill \\ \mathrm{\Delta }{U}_{(i,j+\frac{1}{2})}^2\hfill \end{array}\right]$

Figure 1.

The stored location of variables U¹ and U² and non-linear functions F¹ and F² on a 2D staggered grid. The (i, j) is at the center of the control volume and the variables are stored on the faces of the control volume.

$$\left[\begin{array}{ll}\frac{\partial F_{(i+\frac{1}{2},j)}^1}{\partial U_{(i+\frac{1}{2},j)}^1}\hfill & \frac{\partial F_{(i+\frac{1}{2},j)}^1}{\partial U_{(i,j+\frac{1}{2})}^2}\hfill \\ \frac{\partial F_{(i,j+\frac{1}{2})}^2}{\partial U_{(i+\frac{1}{2},j)}^1}\hfill & \frac{\partial F_{(i,j+\frac{1}{2})}^2}{\partial U_{(i,j+\frac{1}{2})}^2}\hfill \end{array}\right]$$ (3)

The major difficulty here is to compute $\frac{\partial F_{(i+\frac{1}{2},j)}^1}{\partial U_{(i,j+\frac{1}{2})}^2}$ and $\frac{\partial F_{(i,j+\frac{1}{2})}^2}{\partial U_{(i+\frac{1}{2},j)}^1}$ terms, in which the numerator and denominator of the derivatives are not stored at the same location (Fig. 1). This adds to the difficulties in obtaining the Jacobian for full transformation of Navier-Stokes in curvilinear coordinates, which would involve Christoffel symbols. We overcome these difficulties through the hybrid staggered/non-staggered approach [63] and taking a set of appropriate discretizations and interpolations, explained thoroughly in Section 2.3.1. Note that we do not form the Jacobian by taking average of the variables from the staggered locations and compute them at the center point using the non-staggered formulation for the Jacobian. This will not solve the problem for these derivatives because, for example, $(\frac{\partial F_{(i,j)}^1}{\partial U_{(i,j)}^2}+\frac{\partial F_{(i+1,j)}^1}{\partial U_{(i+1,j)}^2})/2$ is not equal to $\frac{\partial F_{(i+\frac{1}{2},j)}^1}{\partial U_{(i,j+\frac{1}{2})}^2}$. Furthermore, there are several choices for the discretizations and interpolations. We have tested different discretizations and interpolations for forming the analytical Jacobian to reach to the one that is closest to the numerical Jacobian obtained by the automatic differentiation scheme.

The goal here is to improve upon the current implicit MF and the semi-implicit, fixed-point Runge-Kutta scheme (FPRK) [63, 37], and achieve a faster method for solving flows involving complex geometries. Therefore, the developed methods must work with our immersed boundary and also overset grid techniques [64]. Combination of immersed boundary and overset grid techniques enable us to efficiently study flows in highly complicated geometries without wasted computational nodes located outside of the region of interest [64]. We have derived an analytical Jacobian, to implicitly solve Navier-Stokes equation on staggered grids using Newton-Krylov method, which also works with our immersed boundary and overset grid techniques. We have used the derived analytical Jacobian as a preconditioner to improve the convergence rate of MF. Furthermore, AD is added to the developed methods to guide the development of the analytical Jacobian and to systematically compare all of the Jacobian formation schemes. There have been a few studies on numerical analysis of MF [65, 66, 67] and FPRK [68, 69], but studies that compare different methods on a realistic problems do not exist. This paper provides insights on different numerical aspects of widely used NKMs (MF, AD, and analytical Jacobian) and FPRK by comparing the results of convergence rate, optimum CFL, and the required iterations and also Grid size, topology, Re, and time-step effect on the computational time of aforementioned methods. Furthermore, we comprehensively compare the computational time of the NKM with a simplified analytical Jacobian (diagonal analytical Jacobian) and the full Jacobian. There is no agreement on the superiority of diagonal or full Jacobian over the other in terms of computational time, e.g., some state diagonal [70] whereas others state full Jacobian [71, 72] is faster. Our comparisons reveal the superior Jacobian for implicit solution of the incompressible Navier-Stokes equations.

This paper is organized as follows: In the numerical method Section 2, we present the governing equations in curvilinear coordinates. Then, we provide a brief overview of AD and MF, followed by a detailed explanation of the derivation of the analytical Jacobian. Finally, we describe the modifications required for this method to work with immersed boundaries and overset grids. In the validations and verifications Section 3, we present several cases to show the accuracy and capabilities of our developed methods. The Taylor-Green vortex problem verifies the 2^nd order accuracy of the method on curvilinear grids. The comparison of numerical results from the steady and pulsatile flow through the 90° bend against experimental data further validates the method and its consistency on overset grids. In addition, the comparison of numerical results from the inline oscillation of a cylinder against experimental data validates the method for flows with moving immersed boundaries. The inline oscillation of a cylinder problem verifies the 2^nd order accuracy of the method with moving immersed boundaries. In computational performance Section 4, these numerical examples are used to explore different aspects of the developed method such as parallel efficiency, effect of grid size, topology, Re, and time-step on the computational time and verify the improvements of computational time of analytical Jacobian against FPRK, MF, and AD. At the end of this section, the performance of the developed method for a realistic application is explored. At the end, the findings of this study and the future directions are summarized in Section 5.

2. Numerical method

We use the following notations throughout this paper: superscript with parenthesis (e.g., U^(*)) for the time level, and subscript with parenthesis for location (e.g., $U_{(i+\frac{1}{2},j,k)}$). Also, subscript without parenthesis indicates direction in Cartesian coordinates (e.g., u_q, and x_q) and, superscript without parenthesis indicates direction in curvilinear co-ordinates (e.g., U^m, and ξ^m). Vectors and matrices are represented by → and bold symbols (e.g., A), respectively. Component of matrix at m^th row and n^th column is presented by A_mn. Einstein’s tensor notation, where repeated indices imply summation, is used unless otherwise indicated (k, d, and q = 1,2,3).

2.1. The governing equations

The governing equations are the 3D, unsteady incompressible Navier-Stokes equations for a Newtonian fluid in curvilinear coordinates. A generalized curvilinear coordinate mapping is employed to transform equations from Cartesian (x₁, x₂, x₃) to curvilinear (ξ¹, ξ², ξ³) coordinates, where ξ^m = ξ^m(x₁, x₂, x₃). The curvilinear formulation is based on the hybrid staggered/non-staggered approach [37], which has all the advantages of a pure staggered grid formulation (i.e., satisfies the discrete continuity exactly) and eliminates the need for computing the Christoffel symbols. The governing equations read as follows [37]:

$$\frac{\partial }{\partial {\xi }^q}(\frac{U^q}{J})=0$$ (4)

$$F^m(U,P)=\frac{1}{J}\frac{\partial }{\partial t}(U^m)+({\xi }_{x_q}^m)\left\{\frac{\partial }{\partial {\xi }^k}(\frac{1}{J}{U}^k{u}_q)+\frac{\partial }{\partial {\xi }^k}(\frac{1}{J}P{\xi }_{x_q}^k)-\frac{1}{Re}\frac{{\partial }^2}{\partial {\xi }^k\partial {\xi }^d}(\frac{1}{J}{g}^{kd}{u}_q)\right\}=0$$ (5)

where u_q, P, and t are the non-dimensional Cartesian velocity, pressure, and physical time, respectively, and Re is the Reynolds number of the flow based on characteristic length and velocity scales. In Eq. (4–5), J is the Jacobian of the geometric transformation, J = ∂( ξ¹, ξ², ξ³)/∂(x₁, x₂, x₃), g^kd is the contravariant metric tensor, $g^{kd}={\xi }_{x_q}^k{\xi }_{x_q}^d$, and U^q are the contravariant velocity components, which are correlated with the Cartesian velocity components as follows:

$$U^k=u_q{\xi }_{x_q}^k$$ (6)

The Cartesian velocity is related with the contravariant velocity components as follows:

$$u_q=U^k{x}_q^{{\xi }^k}$$ (7)

where ${\xi }_{x_q}^k$ and $x_q^{{\xi }^k}$ symbols indicate $\frac{\partial {\xi }^k}{\partial x_q}$ and $\frac{\partial x_q}{\partial {\xi }^k}$, respectively. In our hybrid staggered/non-staggered approach, the pressure and the Cartesian velocities are stored at the center of the control volume (i, j,k), and the contravariant velocity components are stored at the center of the control volume’s surfaces ( $i+\frac{1}{2}$, j, k), (i, $j+\frac{1}{2}$, k), and (i, j, $k+\frac{1}{2}$) for U¹, U², and U³, respectively [37].

2.2. Overview of the overset-CURVIB

The curvilinear/immersed boundary (CURVIB) and overset grid methods are extensively described and validated previously [64, 37, 63], but we provide a brief overview of these methods in this section for completeness. The governing Eq. (4) and (5) are advanced in time using a fractional step method on a curvilinear grid [37]. The momentum Eqs. (5) are discretized in time in a fully implicit manner using a second-order backward difference:

$$\overrightarrow{F}({\overrightarrow{U}}^{(★)},P^{(n)})=\frac{3{\overrightarrow{U}}^{(★)}-4{\overrightarrow{U}}^{(n)}+{\overrightarrow{U}}^{(n-1)}}{2\mathrm{\Delta }t}-\overrightarrow{\mathit{RHS}}({\overrightarrow{U}}^{(★)},P^{(n)})=0$$ (8)

where n denotes the physical time level and ${\mathit{RHS}}^m(\overrightarrow{U},P)=-({\xi }_{x_q}^m)\left\{\frac{\partial }{\partial {\xi }^k}\frac{1}{J}(U^k{u}_q)+\frac{\partial }{\partial {\xi }^k}\frac{1}{J}(P{\xi }_{x_q}^k)-\frac{1}{Re}\frac{{\partial }^2}{\partial {\xi }^k\partial {\xi }^d}(\frac{1}{J}{g}^{kd}{u}_q)\right\}$. The convection term and the viscous term in the RHS are discretized using QUICK and three-point central finite difference schemes, respectively. These terms are calculated on the center nodes of a cell similar to other non-staggered methods, then interpolated to cell faces to form the RHS for the contravariant velocities which are stored at the staggered locations [37]. Eq. (8) is solved implicitly using an NKM (discussed in Section 2.3 and 2.3.1) or FPRK to obtain intermediate fluxes U⃗^(*). The intermediate fluxes are not divergence-free and need to be corrected to satisfy the continuity Eq. (4). This is accomplished by solving the following Poisson equation for the pressure correction ϕ = P⁽ⁿ⁺¹⁾ − P⁽ⁿ⁾:

$$-\frac{\partial }{\partial {\xi }^r}({\xi }_{x_q}^r{G}^q(\varphi ))=\frac{3}{2\mathrm{\Delta }t}\frac{\partial }{\partial {\xi }^r}(\frac{1}{J}{U}^{r(★)})$$ (9)

where $G^q(\varphi )=-\frac{\partial }{\partial {\xi }^k}(\frac{1}{J}\varphi {\xi }_{x_q}^k)$ is the gradient operator. The Poisson equation Eq. (9) is solved using GMRES with multigrid as a preconditioner [37]. Obtaining the solution of the Poisson equation, the pressure and contravariant velocities are updated as follows:

$$P^{(n+1)}=P^{(n)}+\varphi$$ (10)

$$U^{r(n+1)}=U^{r(★)}+\frac{2\mathrm{\Delta }t}{3}J{\xi }_{x_q}^r{G}^q(\varphi )$$ (11)

2.3. Solving the momentum equation

The system of non-linear algebraic equations that results from implicit discretization of the Navier-Stokes, i.e., Eq. (8) gives:

$$\overrightarrow{F}({\overrightarrow{U}}^{(★)})=0$$ (12)

This equation can be solved by either a Newton method or a dual time-stepping method. In our framework, we have implemented both to compare and contrast them on realistic problems. The dual time-stepping method, which can be viewed as fixed-point iteration (Eq. 1), is formulated as follows:

$$\frac{\mathrm{\Delta }{\overrightarrow{U}}^{(k)}}{J}=-\mathrm{\Delta }\tau \overrightarrow{F}({\overrightarrow{U}}^{(k)})$$ (13)

where k is the dual time iteration index and Δτ is the dual time-step. A sequence of steps in the dual time-stepping method is computed by ΔU⃗^(k) = U⃗^(k+1) − U⃗^(k). The Newton method can be written as:

$$\mathbf{A}({\overrightarrow{U}}^{(k)}){\overrightarrow{\mathrm{\Delta }U}}^{(k)}=-\overrightarrow{F}({\overrightarrow{U}}^{(k)})$$ (14)

where $\mathbf{A}=\frac{d\overrightarrow{F}}{d\overrightarrow{U}}$ is the Jacobian matrix and ΔU⃗^(k) = U⃗^(k+1) − U⃗^(k) is a sequence of steps in the Newton equations. Given U⃗⁽⁰⁾ = U⃗⁽ⁿ⁾, the Newton iteration continues until F⃗(U⃗^(k)) → 0, at which U⃗^(k+1) U⃗^(*) (k = 0,1,2,…). A in the Newton Eq. (14) can be approximated because as long as ${\overrightarrow{\mathrm{\Delta }U}}^{(k)}\to 0$, then $\mathbf{A}{\overrightarrow{\mathrm{\Delta }U}}^{(k)}\to 0$, i.e, F⃗(U⃗^(k)) → 0 converges to the desired accuracy. Therefore, an approximate Jacobian can be used instead of the exact Jacobian. Three most commonly used techniques to form the approximate Jacobian are: (1) the approximate analytical; (2) automatic differentiation; and (3) matrix-free method.

The Jacobian can be formed by taking the numerical derivative of F with respect to U⃗, which is recognized as the automatic differentiation (AD) technique [50] as follows:

$$\mathbf{A}=\frac{\overrightarrow{F}(\overrightarrow{U}+\overrightarrow{\mathrm{\Delta }U})-\overrightarrow{F}(\overrightarrow{U})}{\overrightarrow{\mathrm{\Delta }U}}$$ (15)

Another approach is the matrix-free method, which does not require forming the Jacobian explicitly, if a Krylov subspace method is used for solving Eq. (14). Krylov methods only require the product of the Jacobian (A) and the vector (U⃗), which be approximated by [31]:

$$\mathbf{A}\overrightarrow{V}=\frac{\overrightarrow{F}(\overrightarrow{U}+\sigma \overrightarrow{V})-\overrightarrow{F}(\overrightarrow{U})}{\sigma }$$ (16)

where U⃗ is the current approximation to the root of the system of equations, V⃗ is the orthonormal basis of Krylov subspace, and σ is a scalar [31].

Finally, A can be formed taking analytical derivatives of the system of equations with respect to U⃗. The analytical Jacobian has been previously derived and used on non-staggered grids [57, 54]. However, to the best of the authors’ knowledge, it has not been derived for staggered grids as discussed in the introduction (Fig. 1). We explain the formation of the analytical Jacobian in Section 2.3.1.

We have implemented all of the above technique in approximating the Jacobian, i.e., AD, MF, and analytical Jacobian in our CURVIB framework. After approximating the Jacobian, we need to solve Eq. (14). If the number of unknowns is large, computing the exact solution using a direct method for solving Eq. (14) at each Newton iteration (k) can be expensive and may not be justified when initial condition is far from the solution. Therefore, it is reasonable to use an iterative method to solve this equation within the required accuracy [40]. We use flexible GMRES (FGM-RES) [46] in AD, MF, analytical Jacobian for solving Eq. (14) at each k as the Krylov-subspace iterative method using PETSc libraries [73]. The restarting iteration number for FGMRES algorithm is 30, and the relative residual, which is defined as $\frac{{\Vert \mathbf{A}({\overrightarrow{U}}^{(l)}){\overrightarrow{\mathrm{\Delta }U}}^{(l)}+\overrightarrow{F}({\overrightarrow{U}}^{(l)})\Vert }_2}{{\Vert \mathbf{A}({\overrightarrow{U}}^{(0)}){\overrightarrow{\mathrm{\Delta }U}}^{(0)}+\overrightarrow{F}({\overrightarrow{U}}^{(0)})\Vert }_2}$ (|| ||₂ indicates the 2 norm), is fixed to 10⁻³ (l = 0, 1, 2,…). Also, we use the Jacobi preconditioner [74] for the FGMRES in all Newton methods.

2.3.1. The approximate analytical Jacobian

The major difficulties of forming an analytical Jacobian for full transformation of Navier-Stokes in curvilinear coordinates on staggered grids are: (1) the numerator and denominator of the derivatives in Jacobian are not stored at the same location in staggered grids; and (2) obtaining the Jacobian for full transformation of Navier-Stokes in curvilinear coordinates involves Christoffel symbols. Christoffel symbols were avoided in the full transformation of RHS in Eq. 8 (not the Jacobian) by using indirect transformation in the hybrid staggered/non-staggered approach [37]. In this approach, first the partially transformed RHS is calculated based on the Cartesian variables stored at non-staggered locations, and then it is interpolated to the staggered locations and multiplied by the appropriate metrics of transformation to form the final RHS for the contravariant velocities on the staggered locations. Because in this indirect transformation the RHS is written in terms of the Cartesian velocities and not the contravariant ones and the RHS calculation involves interpolations, deriving the Jacobian by linearizing the discrete RHS is not the best choice. Instead, we start from the analytical RHS and apply appropriate discretizations, interpolations, and simplifications required to resolve theses difficulties, which are explained in this section.

We derive the Jacobian A on staggered grids from the momentum Eq. (5), which is written for the contravariant velocities at the staggered locations:

$$F(U^{m(★)},P^{(n)})=\frac{1}{J}\frac{\partial }{\partial t}{[U^m]}^{(★)}+({\xi }_{x_q}^m)\left\{\frac{\partial }{\partial {\xi }^k}[\frac{1}{J}{(U^k{u}_q)}^{(★)}]+\frac{\partial }{\partial {\xi }^k}[\frac{1}{J}{(P{\xi }_{x_q}^k)}^{(n)}]-\frac{1}{Re}\frac{\partial }{\partial {\xi }^k}[\frac{1}{J}{g}^{kd}\frac{\partial }{\partial {\xi }^d}(u_q^{(★)})]\right\}=0$$ (17)

Applying a multivariate Taylor expansion about a U⁽ⁿ⁾, and removing higher order terms, and Eq. (17) at current time-step n yields:

$$A_{mp}\mathrm{\Delta }{U}^p=\frac{1}{J}{\delta }_{mp}\frac{\partial }{\partial t}[\mathrm{\Delta }{U}^p]+({\xi }_{x_q}^m)\frac{\partial }{\partial {\xi }^k}\frac{1}{J}\left[{\delta }_{kp}{u}_q\mathrm{\Delta }{U}^p+U^k{x}_q^{{\xi }^p}\mathrm{\Delta }{U}^p\right]-({\xi }_{x_q}^m)\frac{1}{Re}\frac{\partial }{\partial {\xi }^k}\left[\frac{1}{J}{g}^{kd}\frac{\partial }{\partial {\xi }^d}(x_q^{{\xi }^p}\mathrm{\Delta }{U}^p)\right]$$ (18)

where δ_kp is the Kronecker delta. In order to derive the Jacobian, it is necessary to make some simplifications. These simplifications are acceptable as long as the Newton equations converge to the desired accuracy because the right-hand side of Eq. (14) is not affected by these simplifications. It is assumed that the metrics of transformation ${\xi }_{x_q}^m$, g^kd, and J are locally constant, e.g., $\partial {\xi }_{x_q}^m/\partial {\xi }^k=0$, to avoid dealing with their derivative, i.e., the Christoffel symbols, which are expensive to calculate and put extra constraints on the smoothness of the curvilinear grid. Off-diagonal components of the contravariant metric tensor g^kd are negligible relative to the diagonal components. Therefore, g^kd is approximated by only the diagonal part. Applying these simplifications on Eq. (18) leads to:

$$A_{mp}\mathrm{\Delta }{U}^p=\underset{\text{Unsteady}\text{acceleration}\text{term}}{\underbrace{\frac{1}{J}{\delta }_{mp}\frac{3}{2\mathrm{\Delta }t}\mathrm{\Delta }{U}^p}}+\underset{\text{Convective}\text{acceleration}\text{term}}{\underbrace{\frac{1}{J}\frac{\partial }{\partial {\xi }^k}\left[{\delta }_{kp}{U}^m\mathrm{\Delta }{U}^p+U^k{\delta }_{mp}\mathrm{\Delta }{U}^p\right]}}\underset{\text{Viscous}\text{term}}{\underbrace{-\frac{1}{Re}\frac{1}{J}{g}^{kk}{\delta }_{kd}\frac{{\partial }^2}{\partial {\xi }^d\partial {\xi }^d}\left[{\delta }_{mp}\mathrm{\Delta }{U}^p\right]}}$$ (19)

Note that $\frac{3}{2\mathrm{\Delta }t}\mathrm{\Delta }{U}_l^p$ is obtained by discretizing the unsteady acceleration term with second-order backward difference with respect to time similar to Eq. (8). Furthermore, ${\xi }_{x_q}^m{x}_q^{{\xi }^p}={\delta }_{mp}$ is used for obtaining Eq. (19).

The major challenge here is determining the terms in Eq. (19) such as $\frac{\partial }{\partial {\xi }^k}(\mathrm{\Delta }{U}^m)$ or $\frac{\partial }{\partial {\xi }^k}(U^m)$, in which k ≠ m. The reason is that U^m for m = 1,2,3 are not computed and stored at all surfaces (see Fig. 1) and only stored on the corresponding staggered locations. Therefore, some interpolations are required to determine these terms. Our experience shows that the convergence rate of the Newton equations Eq. (14) directly depends on such interpolations. To illustrate how these interpolations are chosen, Eq. (19) is written in 2D, and m = 1 for clarity without the loss of generality (the implementation in 3D is provided in the Appendix A). When m = 1, all the terms at the right-hand side of Eq. (19) are stored at ( $i+\frac{1}{2}$, j).

$$\begin{array}{l}\sum _{c=-3}^5\sum _{s=-3}^5[{(A_{11}\mathrm{\Delta }{U}^1)}_{(i+\frac{s}{2},j+c)}+{(A_{12}\mathrm{\Delta }{U}^2)}_{(i+c,j+\frac{s}{2})}]=\underset{\text{Unsteady}\text{term}}{\underbrace{\frac{1}{J}\frac{3}{2\mathrm{\Delta }t}\mathrm{\Delta }{U}_{(i+\frac{1}{2},j)}^1}}+\underset{\text{Convection}\text{term}1}{\underbrace{\frac{2}{J}\frac{\partial }{\partial {\xi }^1}{\left[U^1\mathrm{\Delta }{U}^1\right]}_{(i+\frac{1}{2},j)}}}+\underset{\text{Convection}\text{term}2}{\underbrace{\frac{1}{J}\frac{\partial }{\partial {\xi }^2}{\left[U^1\mathrm{\Delta }{U}^2\right]}_{(i+\frac{1}{2},j)}}}\\ +\underset{\text{Convection}\text{term}3}{\underbrace{\frac{1}{J}\frac{\partial }{\partial {\xi }^2}{\left[U^2\mathrm{\Delta }{U}^1\right]}_{(i+\frac{1}{2},j)}}}\underset{\text{Viscous}\text{term}}{\underbrace{-\frac{1}{Re}\frac{1}{J}\left[g_{(i+\frac{1}{2},j)}^{11}\frac{{\partial }^2}{\partial {\xi }^1\partial {\xi }^1}\left(\mathrm{\Delta }{U}_{(i+\frac{1}{2},j)}^1\right)+g_{(i+\frac{1}{2},j)}^{22}\frac{{\partial }^2}{\partial {\xi }^2\partial {\xi }^2}\left(\mathrm{\Delta }{U}_{(i+\frac{1}{2},j)}^1\right)\right]}}\end{array}$$ (20)

By discretizing the derivatives of the right-hand side of Eq. (20), all the resulting terms must be formed such that they contain ΔU¹ and ΔU² at the corresponding surface centers (i.e., ( $i+\frac{s}{2}$, j + c) and (i + c, $j+\frac{s}{2}$), respectively). s and c are the integer numbers (s ≠ 0) that determine the location of the surfaces. Note that the limit of s and c are determined based on the discretization and interpolations of the right-hand side of Eq. (20). The Jacobian matrix components, A₁₁ and A₁₂, are found by deriving the coefficients of ΔU^m in the right-hand side of Eq. (20), which must include U¹,U²,g¹¹, and g²² at the corresponding surface center. For clarification in Fig. 2, the assembled Jacobian matrix is plotted schematically for a 2D grid. Red and green rectangles correspond to coefficients for row 2 ×( j × I M + i) and 2 ×( j × I M + i) + 1 (I M and JM are the number of nodes in ξ¹ and ξ² direction, respectively) of the Jacobian, respectively, and the black rectangle shows the sub-Jacobian matrix, which is multiplied by the corresponding increment of contravariant velocities (blue rectangle). The Jacobian matrix contains I M × JM number of these sub-Jacobians.

Figure 2.

Schematic of assembled Jacobian matrix in two dimensional equations.

The terms of the right-hand side of Eq. (20) are labeled in order to expand and illustrate them in detail. The unsteady term can be placed in the Jacobian matrix without any modification because ΔU¹ is available at ( $i+\frac{1}{2}$, j). Fourth-order central difference with respect to ξ¹ is applied to the Convection term 1. QUICK scheme incorporates either two nodes before or after the discretization node. By using a fourth order difference, which incorporates two nodes after and before a given node, we can reach to the closest term produced by QUICK without involving the flow direction in the Jacobian, based on our tests comparing the AD Jacobian with the analytical Jacobian. Since all of the resulting terms are at the appropriate locations, there is no need for any interpolation.

$$\frac{2}{J}\frac{\partial }{\partial {\xi }^1}{\left[U^1\mathrm{\Delta }{U}^1\right]}_{(i+\frac{1}{2},j)}=\frac{2}{J}\frac{1}{12}\left[{(U^1\mathrm{\Delta }{U}^1)}_{(i-\frac{3}{2},j)}-8{(U^1\mathrm{\Delta }{U}^1)}_{(i-\frac{1}{2},j)}+8{(U^1\mathrm{\Delta }{U}^1)}_{(i+\frac{3}{2},j)}-{(U^1\mathrm{\Delta }{U}^1)}_{(i+\frac{5}{2},j)}\right]$$ (21)

Note that Δξⁱ in the curvilinear transformation is uniform and equal to one. Convection term 2 is first substituted with the average of the neighboring nodes because ΔU² is not available at ( $i+\frac{1}{2}$, j) location. Then, the derivative on the resulting term is discretized with the central difference.

$$\begin{array}{l}\frac{1}{J}\frac{\partial }{\partial {\xi }^2}{[U^1\mathrm{\Delta }{U}^2]}_{(i+\frac{1}{2},j)}=\frac{1}{2}\frac{1}{J}\frac{\partial }{\partial {\xi }^2}\left[{(U^1\mathrm{\Delta }{U}^2)}_{(i,j)}+{(U^1\mathrm{\Delta }{U}^2)}_{(i+1,j)}\right]=\\ \frac{1}{2}\frac{1}{J}\left[{({\stackrel{\sim }{U}}^1\mathrm{\Delta }{U}^2)}_{(i,j+\frac{1}{2})}-{({\stackrel{\sim }{U}}^1\mathrm{\Delta }{U}^2)}_{(i,j-\frac{1}{2})}+{({\stackrel{\sim }{U}}^1\mathrm{\Delta }{U}^2)}_{(i+1,j+\frac{1}{2})}-{({\stackrel{\sim }{U}}^1\mathrm{\Delta }{U}^2)}_{(i+1,j-\frac{1}{2})}\right]\end{array}$$ (22)

U¹ is shown by Ũ¹ to emphasize the need for another interpolation to determine its value at (i, $j\pm \frac{1}{2}$) surfaces. As an example, Ũ¹ at (i, $j+\frac{1}{2}$) surface is interpolated from the neighboring nodes as follows:

$${\stackrel{\sim }{U}}_{(i,j+\frac{1}{2})}^1=\frac{1}{4}(U_{(i-\frac{1}{2},j)}^1+U_{(i-\frac{1}{2},j+1)}^1+U_{(i+\frac{1}{2},j)}^1+U_{(i+\frac{1}{2},j+1)}^1)$$ (23)

Eq. (23) can be written for 3D grid as:

$${\stackrel{\sim }{U}}_{(i,j+\frac{1}{2},k)}^1=\frac{1}{4}(U_{(i-\frac{1}{2},j,k)}^1+U_{(i-\frac{1}{2},j+1,k)}^1+U_{(i+\frac{1}{2},j,k)}^1+U_{(i+\frac{1}{2},j+1,k)}^1)$$ (24)

The same interpolation as Convection term 2 is required for Convection term 3 (Eq. (22))

$$\begin{array}{l}\frac{1}{J}\frac{\partial }{\partial {\xi }^2}{\left[U^2\mathrm{\Delta }{U}^1\right]}_{(i+\frac{1}{2},j)}=\frac{1}{2}\frac{1}{J}\frac{\partial }{\partial {\xi }^2}\left[{(U^2\mathrm{\Delta }{U}^1)}_{(i,j)}+{(U^2\mathrm{\Delta }{U}^1)}_{(i+1,j)}\right]=\\ \frac{1}{2}\frac{1}{J}\left[{(U^2\mathrm{\Delta }{\stackrel{\sim }{U}}^1)}_{(i,j+\frac{1}{2})}-{(U^2\mathrm{\Delta }{\stackrel{\sim }{U}}^1)}_{(i,j-\frac{1}{2})}+{(U^2\mathrm{\Delta }{\stackrel{\sim }{U}}^1)}_{(i+1,j+\frac{1}{2})}-{(U^2\mathrm{\Delta }{\stackrel{\sim }{U}}^1)}_{(i+1,j-\frac{1}{2})}\right]\end{array}$$ (25)

where ΔŨ¹ at (i, $j+\frac{1}{2}$) is interpolated similar to Ũ¹ (Eq. (23)). We found such interpolations to be sufficient to get good approximations of the analytical Jacobian on different grids by comparing the values of the analytical Jacobian with that obtained by automatic differentiation (AD).

Finally, the central difference is used to discretize the viscous term in the Jacobian:

$$\begin{array}{l}-\frac{1}{Re}\frac{1}{J}\left[g_{(i+\frac{1}{2},j)}^{11}\frac{{\partial }^2}{\partial {\xi }^1\partial {\xi }^1}\left(\mathrm{\Delta }{U}_{(i+\frac{1}{2},j)}^1\right)+g_{(i+\frac{1}{2},j)}^{22}\frac{{\partial }^2}{\partial {\xi }^2\partial {\xi }^2}\left(\mathrm{\Delta }{U}_{(i+\frac{1}{2},j)}^1\right)\right]=\\ -\frac{1}{Re}\frac{1}{J}\left[g_{(i+\frac{1}{2},j)}^{11}\left(\mathrm{\Delta }{U}_{(i+\frac{3}{2},j)}^1-2\mathrm{\Delta }{U}_{(i+\frac{1}{2},j)}^1+\mathrm{\Delta }{U}_{(i-\frac{1}{2},j)}^1\right)+{\stackrel{\sim }{g}}_{(i+\frac{1}{2},j)}^{22}\left(\mathrm{\Delta }{U}_{(i+\frac{1}{2},j+1)}^1-2\mathrm{\Delta }{U}_{(i+\frac{1}{2},j)}^1+\mathrm{\Delta }{U}_{(i+\frac{1}{2},j-1)}^1\right)\right]\end{array}$$ (26)

where ${\stackrel{\sim }{g}}_{(i+\frac{1}{2},j)}^{22}$ is evaluated similar to Ũ¹ from Eq. (23). By substituting Eq. (21), (22), (25), and (26) into Eq. (20), the Jacobian matrix components A₁₁ and A₁₂ can be evaluated, which are reported in Table 2 and Table 3, respectively.

Table 2.

$J_{(i+\frac{1}{2},j)}{A}_{11({\xi }^1,{\xi }^2)}$ components of the analytical Jacobian for 2D Navier-Stokes equations, where J is the Jacobian of the geometric transformation.

location at ξ1 direction	location at ξ2 direction	$J_{(i+\frac{1}{2},j)}{A}_{11({\xi }^1,{\xi }^2)}$
$i-\frac{3}{2}$	j	$\frac{1}{6}{U}_{(i-\frac{3}{2},j)}^1$
$i-\frac{1}{2}$	j − 1	$-\frac{1}{8}{U}_{(i,j-\frac{1}{2})}^2$
$i-\frac{1}{2}$	j	$-\frac{1}{Re}{g}_{(i+\frac{1}{2},j)}^{11}-\frac{8}{6}{U}_{(i-\frac{1}{2},j)}^1+\frac{1}{8}(U_{(i,j+\frac{1}{2})}^2-U_{(i,j-\frac{1}{2})}^2)$
$i-\frac{1}{2}$	j + 1	$\frac{1}{8}{U}_{(i,j+\frac{1}{2})}^2$
$i+\frac{1}{2}$	j − 1	$-\frac{1}{Re}{\stackrel{\sim }{g}}_{(i+\frac{1}{2},j)}^{22}-\frac{1}{8}(U_{(i,j-\frac{1}{2})}^2+U_{(i+1,j-\frac{1}{2})}^2)$
$i+\frac{1}{2}$	j	$\frac{3}{2}\mathrm{\Delta }t+\frac{2}{Re}(g_{(i+\frac{1}{2},j)}^{11}+{\stackrel{\sim }{g}}_{(i+\frac{1}{2},j)}^{22})+\frac{1}{8}(U_{(i,j+\frac{1}{2})}^2-U_{(i,j-\frac{1}{2})}^2+U_{(i+1,j+\frac{1}{2})}^2-U_{(i+1,j-\frac{1}{2})}^2)$
$i+\frac{1}{2}$	j + 1	$-\frac{1}{Re}{\stackrel{\sim }{g}}_{(i+\frac{1}{2},j)}^{22}+\frac{1}{8}(U_{(i,j+\frac{1}{2})}^2+U_{(i+1,j+\frac{1}{2})}^2)$
$i+\frac{3}{2}$	j − 1	$-\frac{1}{8}{U}_{(i+1,j-\frac{1}{2})}^2$
$i+\frac{3}{2}$	j	$-\frac{1}{Re}{g}_{(i+\frac{1}{2},j)}^{11}+\frac{8}{6}{U}_{(i+\frac{3}{2},j)}^1+\frac{1}{8}(U_{(i+1,j+\frac{1}{2})}^2-U_{(i+1,j-\frac{1}{2})}^2)$
$i+\frac{3}{2}$	j + 1	$\frac{1}{8}{U}_{(i+1,j+\frac{1}{2})}^2$
$i+\frac{5}{2}$	j	$-\frac{1}{6}{U}_{(i+\frac{5}{2},j)}^1$

Table 3.

$J_{(i+\frac{1}{2},j)}{A}_{12({\xi }^1,{\xi }^2)}$ components of the analytical Jacobian for 2D Navier-Stokes equations, where J is the Jacobian of the geometric transformation.

location at ξ1 direction	location at ξ2 direction	$J_{(i+\frac{1}{2},j)}{A}_{12({\xi }^1,{\xi }^2)}$
i	$j-\frac{1}{2}$	$-\frac{1}{2}{\stackrel{\sim }{U}}_{(i,j-\frac{1}{2})}^1$
i + 1	$j-\frac{1}{2}$	$-\frac{1}{2}{\stackrel{\sim }{U}}_{(i+1,j-\frac{1}{2})}^1$
i	$j+\frac{1}{2}$	$\frac{1}{2}{\stackrel{\sim }{U}}_{(i,j+\frac{1}{2})}^1$
i + 1	$j+\frac{1}{2}$	$\frac{1}{2}{\stackrel{\sim }{U}}_{(i+1,j+\frac{1}{2})}^1$

These tables present the coefficients of the single line of Jacobian, which are multiplied by the corresponding component of variables (e.g., in Table 3, the coefficient indicated for i and $j+\frac{1}{2}$ are multiplied by $\mathrm{\Delta }{U}_{(i,j+\frac{1}{2})}^2$). The analytical Jacobian components in 3D for all directions are reported in Appendix A.

The diagonal analytical Jacobian (DJ) is obtained by neglecting off-diagonal components of the assembled Jacobian, i.e., we only consider diagonal component, e.g., $A_{11(i+\frac{1}{2},j)}$ and $A_{22(i,j+\frac{1}{2})}$ in Fig. 2. It is important to consider the contribution of convective and viscous terms from surfaces of the neighboring control volumes to the diagonal of the Jacobian, e.g., from Table 2, $A_{11(i+\frac{1}{2},j)}$ includes terms of $\frac{1}{2Re}(g_{(i,j+\frac{1}{2})}^{22}+g_{(i+1,j+\frac{1}{2})}^{22}+g_{(i,j-\frac{1}{2})}^{22}+g_{(i+1,j-\frac{1}{2})}^{22})+\frac{1}{8}(U_{(i,j+\frac{1}{2})}^2-U_{(i,j-\frac{1}{2})}^2+U_{(i+1,j+\frac{1}{2})}^2-U_{(i+1,j-\frac{1}{2})}^2)$ from surfaces of the neighboring control volumes. Without considering the aforementioned terms, based on our experience, the performance of DJ in terms of the computational time dramatically decreases. Therefore, the derivation of the full Jacobian to obtain the diagonal Jacobian is necessary. We discuss the required modification on the Jacobian for immersed boundaries and overset grids in sections 2.3.2 and 2.3.3, respectively.

2.3.2. Treatment of the domain and immersed boundaries

A sharp-interface immersed boundary method is used to handle the 3D, arbitrary complex boundaries inside the domain [62]. The method blanks out the nodes that are outside the flow domain and do not affect the solution. These nodes are identified using an efficient ray-tracing algorithm [63]. The boundary conditions are reconstructed on the fluid nodes in the immediate vicinity of the immersed boundary along the normal to the boundary as shown in Fig 3 [62]. The equations are solved on the rest of the grid nodes (fluid nodes). The method has been shown to be second-order accurate for a variety of flows [62, 75]. Since contravariant velocities at the immersed boundaries are reconstructed from interior nodes, the equation 8 is not solved on these boundaries and ΔU^m must be set to zero. We treat the domain boundaries (e.g., inflow boundary) similar to the immersed boundaries. Since removing the boundary variables from the system of non-linear equations are expensive (requires forming a new data structure), at the boundaries ΔU^m = 0 must be enforced. To do so, for all boundary nodes we set the diagonal to one and off-diagonals to zero in the rows and columns that correspond to these nodes in the Jacobian matrix. In addition, we set the function value to zero in Eq. 14 for all the boundary nodes. For example, if $U_{(i+\frac{1}{2},j)}^1$ is located at a boundary, the diagonal corresponding to this node is set to one and the other components in the row and column of this diagonal are set to zero in the Jacobian matrix and the corresponding function value ( $F_{(i+\frac{1}{2},j)}^1$) is also set to zero as follows:

$$\left[\begin{array}{cccccccccc}& & & & 0& & & & & \\ & & & & & & & & & \\ & & & & ⋮& & & & & \\ & & & & 0& & & & & \\ 0& & \cdots & 0& 1& 0& & \cdots & & 0\\ \cdots & {(A_{21})}_{(i-\frac{3}{2},j-1)}& {(A_{22})}_{(i-2,j-\frac{1}{2})}& \cdots & 0& {(A_{22})}_{(i,j+\frac{1}{2})}& \cdots & {(A_{21})}_{(i+\frac{5}{2},j+1)}& {(A_{22})}_{(i+2,j+\frac{3}{2})}& \cdots \\ & & & & & & & & & \\ & & & & ⋮& & & & & \\ & & & & & & & & & \\ & & & & 0& & & & & \end{array}\right]\left[\begin{array}{c}⋮\\ \mathrm{\Delta }{U}_{(i-\frac{3}{2},j-1)}^1\\ \mathrm{\Delta }{U}_{(i-2,j-\frac{1}{2})}^2\\ ⋮\\ \mathrm{\Delta }{U}_{(i+\frac{1}{2},j)}^1\\ \mathrm{\Delta }{U}_{(i,j+\frac{1}{2})}^2\\ ⋮\\ \mathrm{\Delta }{U}_{(i+\frac{5}{2},j+1)}^1\\ \mathrm{\Delta }{U}_{(i+2,j+\frac{3}{2})}^2\\ ⋮\end{array}\right]=\left[\begin{array}{c}⋮\\ F_{(i-\frac{3}{2},j-1)}^1\\ F_{(i-2,j-\frac{1}{2})}^2\\ ⋮\\ 0\\ F_{(i,j+\frac{1}{2})}^2\\ ⋮\\ F_{(i+\frac{5}{2},j+1)}^1\\ F_{(i+2,j+\frac{3}{2})}^2\\ ⋮\end{array}\right]$$ (27)

Figure 3.

Shematic of the sharp-interface (Γ) immersed boundary method. The velocity at the immersed boundary nodes (u_b) are reconstructed along the normal direction using a quadratic interpolation between velocity of wall and fluid nodes, u_a and u_c, respectively.

The solution of Eq. (27) with these modifications will enforce ΔU^m = 0 on the boundaries. Note that the value of U^m is updated on all the boundary nodes after each function evaluation during each iteration to reach the implicit solution on these nodes as well.

2.3.3. Treatment of overset grid

In order to reduce wasted nodes in a domain, caused by using immersed boundary method, we use the overset grid approach. In this approach, several sub-grids are arbitrary overlapped to discretize a complex, multi-connected flow domain. To solve the governing equations at each sub-grid, boundary conditions at the interfaces are constructed by interpolation from host sub-grid (i.e., sub-grid that contains the interface). The details of the overset-CURVIB method can be found in the recent publication by Borazjani et al. [64]. To avoid iterations in this method, all the subdomains must be solved simultaneously. Therefore, the Jacobian matrices, right-hand side vectors, and solution vectors of all subdomains are packed and solved together as follows:

$$\left[\begin{array}{ccccc}\mathbf{A}(1)& 0& 0& 0& 0\\ 0& \ddots & 0& 0& 0\\ 0& 0& \mathbf{A}(b)& 0& 0\\ 0& 0& 0& \ddots & 0\\ 0& 0& 0& 0& \mathbf{A}(r)\end{array}\right]\left[\begin{array}{c}\overrightarrow{\mathrm{\Delta }U}(1)\\ ⋮\\ \overrightarrow{\mathrm{\Delta }U}(b)\\ ⋮\\ \overrightarrow{\mathrm{\Delta }U}(r)\end{array}\right]=-\left[\begin{array}{c}\overrightarrow{F}(1)\\ ⋮\\ \overrightarrow{F}(b)\\ ⋮\\ \overrightarrow{F}(r)\end{array}\right]$$ (28)

where A(b), F⃗(b), and $\overrightarrow{\mathrm{\Delta }U}(b)$ are the Jacobian matrix, right-hand side vector, and the corrections of contravariant velocity vector for subdomain b (b = 1,…,r, where r is the number of the subdomains), respectively. Note that in order to improve the computational time, the zeroes shown in Eq. (28) in the packed Jacobian matrix are not stored.

3. Validations and verifications

In this section, first we apply the NKM with analytical Jacobian to several numerical experiments to verify the accuracy and validate the developed solver such as: (1) Taylor-Green vortex with a distorted grid; (2) Inline oscillations of a cylinder in a fluid initially at rest; (3) The steady flow through a 90° bend; (4) The pulsatile flow through a 90° bend.

3.1. Taylor-Green vortex flow

Taylor-Green vortex is an unsteady flow of a decaying vortex with periodic boundary conditions in two directions. It has an exact analytical solution that satisfies the 2D incompressible Navier-Stokes equations [76, 77], which makes it suitable to verify the precision of the computational results. The boundary conditions are periodic boundary and the initial condition is computed based on the analytical solution at t = 0. The distorted curvilinear grid is chosen to verify the performance of the proposed methods within curvilinear coordinates. The domain size is 2π × 2π with 101 ×101 nodes. A quarter of the domain with size of π ×π is selected as a sub-domain. The lower and left sides are stretched with δ = 1.1 in x and y directions, respectively, while the upper and right sides are stretched with the similar ratio but in reverse directions. After applying a grid smoother, the sub-domain is mirrored in respect to y axis and the resulted sud-domain is mirrored in respect to x axis to obtain a periodic domain with 2π × 2π size. The geometric stretching factor is defined as follows:

$$\delta =\frac{x_{i+\frac{3}{2}}-x_{i+\frac{1}{2}}}{x_{i+\frac{1}{2}}-x_{i-\frac{1}{2}}}$$ (29)

Fig. 4(a) shows the distorted curvilinear grid and the contours of the out-of-plane vorticity at a cross section of the computational domain at t = 4 (Δt = 0.01) in a simulation with Reynolds number (Re) equal to 10. Fig. 4(b) shows the out-of-plane vorticity at the mid-line parallel to the horizontal axis at t = 4. It can be observed that the numerical result (solid line) is in excellent agreement with the exact solution (circles). To quantify the order of accuracy of the solver in space and time, the error in the numerical result is calculated relative to the analytical solution for three successively finer grids and time-steps (i.e., 101 × 101 with Δt = 0.004, 251 × 251 with Δt = 0.002, and 501 × 501 with Δt = 0.001) in the distorted curvilinear mesh as follows:

$$\mathit{Standard}\mathit{Error}=\frac{1}{\mathit{NxNy}}\sqrt{\sum _{i=1}^{Nx}\sum _{j=1}^{Ny}[{(u_{(i,j)}-u_{(i,j)}^{\mathit{exact}})}^2+{(v_{(i,j)}-v_{(i,j)}^{\mathit{exact}})}^2]}$$ (30)

where Nx and Ny are the number of grid points in each i and j direction, respectively, u_(i,j) and v_(i,j) are the numerical solutions of velocities on the (i, j) grid node, and $u_{(i,j)}^{\mathit{exact}}$ and $v_{(i,j)}^{\mathit{exact}}$ are the analytical solutions. Fig. 5 plots the error against the grid spacing and time step for Taylor-Green vortex problem at t = 1 and Re = 10 in log-log scale. Comparison between numerical results (circles) and a line with slope of two (solid) indicates the 2^nd order accuracy of the solver.

Figure 4.

Taylor-Green vortex problem at t = 4 and Re = 10: (a) the out-of-plane vorticity contours and curvilinear grid for 2π × 2π domain; (b) the numerical result comparison of the out-of-plane vorticity at the mid-line parallel to the horizontal axis with its exact solution.

Figure 5.

Standard error for Taylor-Green vortex flow as function of the grid spacing/time-step in log-log scale at t = 1 and Re = 10. CFL is kept the same in all test cases, i.e., $\frac{\mathrm{\Delta }t}{\mathrm{\Delta }{t}_{\mathit{max}}}=\frac{\mathrm{\Delta }x}{\mathrm{\Delta }{x}_{\mathit{max}}}$.

3.2. Inline oscillations of a cylinder in a fluid initially at rest

The developed method is validated for flows with moving immersed boundaries against an experimental benchmark [78], in which a circular cylinder starts to oscillate in the horizontal direction in a fluid initially at rest. The translational motion of the cylinder is given by a harmonic oscillation:

$$x_1(t)=-A_m\mathit{sin}(2ft)$$ (31)

where x₁ is the location of the center of the cylinder in horizontal direction, f is the oscillation frequency, and A_m is the oscillation amplitude. This problem is governed by two non-dimensional parameters, i.e., Reynolds number Re and Keulegan-Carpenter number KC defined as:

$$Re=\frac{U_{m}D}{\nu },KC=\frac{U_m}{fD}$$ (32)

where U_m is the maximum oscillation velocity, D is the cylinder diameter, and ν is the fluid kinematics viscosity. The computations are conducted at Re = 100 and KC = 5, for which both experimental and numerical results have been reported [78]. The domain size is 100D ×100D with 361 ×241 nodes in the inline (oscillatory) and transverse directions, respectively. 150 ×50 nodes are distributed uniformly in a 3D ×D box, which contains the cylinder during the oscillations. The non-dimensional time-step of Δt = 0.0167 is used for this simulation. The Neumann boundary condition is applied on the domain boundaries. Fig. 6 compares the inline velocity profiles at x₁ = 0.6D for three different phase angles (ϕ = 2πf ) calculated by our method and measured by Dutsch et al. [78]. As can be observed in Fig. 6, the velocity profiles are in excellent agreement with the experimental measurements. To quantify the order of accuracy of the solver in space and time, the error of numerical results for two grids and time-steps (i.e., 361 ×241 with Δt = 0.0167, and 721 × 477 with Δt = 0.00835) are calculated relative to the numerical result of 1441 × 969 grid (finest grid) and Δt = 0.004175 based on equation 30. Fig. 6(b) plots the error against the grid spacing and time step for inline oscillations of a cylinder problem at f = 180° in log-log scale. Comparison between numerical results (circles) and a line with slope of two (solid) indicates the 2^nd order accuracy of the solver.

Figure 6.

(a) The calculated inline velocity component ( $\frac{u_1}{U_m}$) profile at x₁ = 0.6D for three different phase angles (ϕ = 2πf ) are compared with experimental results (circles) [78] (b) Standard error for velocity components as function of the grid spacing/time-step in log-log scale at f = 180° calculated based on equation 30, where u^exact, v^exact are velocity components of numerical results obtained using 1441 ×969 grid. CFL is kept the same in all test cases, i.e., $\frac{\mathrm{\Delta }t}{\mathrm{\Delta }{t}_{\mathit{max}}}=\frac{\mathrm{\Delta }x}{\mathrm{\Delta }{x}_{\mathit{max}}}$.

3.3. Pulsatile flow through a 90° bend

We simulate the 3D pulsatile flow through a strongly curved 90° pipe bend to validate the developed method for unsteady flows and overset grids. This test case was studied experimentally by Rindt et al. [79]. In order to validate the method with multiple overset grids, we simulate a test case with three subdomains. The geometry of the test case is shown in Fig. 7(a). Note that the geometry of the single domain is the same as the overset grid, but the whole geometry is meshed by a single curvilinear grid.

Figure 7.

(a) The geometry and grid for the 90° bend with three curvilinear overlapping grids: inlet subdomain (blue), bend subdomain (white), and outlet subdomain (orange). (b) Incoming flow waveform. Solidline: analytical non-dimensional bulk velocity waveform prescribed at the inlet of the flow domain obtained by solving the Womersley equation; Circles: the measurements of non-dimensional inlet bulk velocity from experiments [79].

As shown in Fig. 7(a), the radius of the curvature of the bend is three times of the pipe diameter (D). The inlet is placed 5D before the bend and the outlet is placed 7D after the bend. A gear pump providing a steady flow with the Reynolds number equal to 500 ( $Re=\frac{UD}{\vartheta }$, where U is the bulk velocity, D is the pipe diameter, and ϑ is the kinematic viscosity) in conjunction with a piston pump generating sinusoidal flow waveform with Re = −300 to Re = 300 were used to generate the pulsatile flow waveform in the experiments [79]. The resulting Womersley number of the flow in the experiments were 7.8.

To generate a waveform that matches the experiment’s waveform, the inlet boundary condition is the velocity profile set from the Womersley solution of a fully developed pulsatile flow in a circular pipe as follows [37]:

$$u_{\mathit{inlet}}=2[1-{(\frac{r}{R})}^2]-i\frac{k}{\omega }{e}^{i\omega t}[1-\frac{J_0(r\sqrt{\frac{-i\omega }{v}})}{J_0(R\sqrt{\frac{-i\omega }{v}})}]$$ (33)

where J₀ is the zero order Bessel function of the first kind, R is the radius of the pipe, r is the radial distance from the center of the pipe, ω = 13.31 rad/s is the angular frequency of the flow oscillation, and v is the flow viscosity. The constant k is selected to be 0.375 to generate a sinusoidal flow waveform, which matches the experiment’s waveform [37, 79]. The above equation is solved using MATLAB, and the resulting solutions are stored and fed into the solver to specify the time-varying inlet flow (Fig. 7(b)). As shown in Fig. 7(b), the computed inlet Re waveform is in reasonably good overall agreement with the experimental inlet Re. The outlet boundary condition is the Neumann boundary condition. The time period of inflow oscillations, which is non-dimensionalized based on D and U, is T = 12.3. The non-dimensional time-step Δt = 0.0123 is used for the simulations. The stramwise velocity equal to one is used as the initial condition.

Grid-independency investigation is carried out by simulating pulsatile flow through the 90° bend on three different single domain grids with 25 ×25 ×91, 33 ×33 ×121, and 41 ×41 ×151 grid nodes. The streamwise velocity profiles at the plane of symmetry for different radial positions r̄ = (r – r_o)/(r_i – r_o) (r_o and r_i are the outer and inner bend radius, respectively) at 0.5T time instant and θ = 45° are plotted in Fig. 8 for the three different grids. It indicates that the grid node 33 × 33 × 121 is enough to obtain a grid independent result because the streamwise velocity profile at the plane of symmetry for 33 × 33 × 121 and 41 × 41 × 151 grids are almost identical. Therefore, this grid resolution is used for comparing the numerical results to the experimental data.

Figure 8.

The streamwise velocity profiles at the symmetry plane for pulsatile flow in the 90° bend at t = 0.5T and θ = 45° for three different grids (25 × 25 × 91, 33 × 33 × 121, and 41 × 41 × 151).

The simulated flow in the 90° bend for a single grid and overset grids with the pulsatile inlet are compared against the experimental results [79] in Fig. 9. The streamwise velocity profiles at the symmetry plane are plotted at five different locations (i.e., θ = 0°, 22.5°, 45°, 67.5°, and 90°) for four different time instants during the cycle (t = 0, 0.25T, 0.5T, and 0.75T). Note that to test the overset method, three overlapping grids as shown in Fig. 7(a) (33 × 33 × 24, 33 × 33 × 78, and 33 × 33 × 38 grids for the inlet, middle, and the outlet subdomains, respectively) are used. As observed from Fig. 9, the computational results are in good overall agreement with the experimental results. The maximum discrepancy is at t = 0.75T, where the largest deviation from experimental inlet waveform from Womersly inlet waveform exists (Fig. 7(b)). The solutions on the overset grid are in excellent agreement with the single grid results.

Figure 9.

Pulsatile flow through a 90° bend. The calculated streamwise velocity profiles at the plane of symmetry for a single grid (solid) and overset grid (dashed) are compared with experimental results (circles) at different time instants. r̄ = (r – r_o)/(r_i – r_o) (r_o and r_i are the outer and inner bend radius, respectively), U_θ is the streamwise velocity at the symmetry plane, and U_m is the maximum streamwise velocity at the symmetry plane at t/T = 0 and θ° = 0

3.4. Steady flow through a 90° bend

We simulate 3D steady flow through a strongly curved 90° pipe bend to validate and investigate the computational performance of the developed method. This test case was studied experimentally by Bovendeerd et al. [80]. Note that the geometry and grid resolution used for comparing the results against the experimental data, for both the overset and the single grid, are the same as those in pulsatile flow through a 90° bend (Fig. 7(a)). In this test case, the inlet $Re=\frac{UD}{\vartheta }=700$, where U is the bulk velocity, D is the pipe diameter, and ϑ is the kinematic viscosity [80] with a fully developed (parabolic) velocity profile boundary condition at the inlet, and the Neumann boundary condition at the outlet. The non-dimensional time-step of Δt = 0.02 is used for these simulations. The stramwise velocity equal to one is used as the initial condition.

The simulated flow in the 90° bend with the steady inlet, for both single and overset grids, is validated against the experimental data [80]. The streamwise velocity profiles at the plane of symmetry for numerical and experimental results are plotted at different angles (θ = 0°, 23.4°, 58.5°, and 90°) in the bend in Fig. 10. As observed, the calculated velocity profiles for both single and overset grids are in an excellent agreement with the experimental measurements.

Figure 10.

Steady flow through a 90° bend. The calculated streamwise velocity profiles at the plane of symmetry for a single grid (solid) and overset grid (dashed) are compared with experimental results (circles). r̄ = (r – r_o)/(r_i – r_o) (r_o and r_i are the outer and inner bend radius, respectively), U_θ is the streamwise velocity at the symmetry plane, and U_m is the maximum streamwise velocity at the symmetry plane at θ° = 0.

4. Computational performance

The test cases in section 3 are used to investigate different numerical aspects of the developed methods including comparing the computational time and convergence rate of the analytical Jacobian with AD, MF, and FPRK, grid topology and Reynolds number effect on computational time, as well as the parallel computing efficiency. We investigate the analytical Jacobian in two forms: First, the full form of the analytical Jacobian (FJ); Second, only the diagonal part of the analytical Jacobian (DJ).

4.1. Computational time and convergence rate

The steady flow through a 90° bend (3.4) is used to investigate the computational time and convergence rate among DJ, FJ, AD, MF, and FPRK. For each method, at each iteration, the residual is computed based on the l²-norm of the right-hand side of Eq. (14). The residual at iteration k is normalized by the residual of the first iteration, which is referred to as the relative residual hereafter. The convergence criteria is set based on the relative residual less than 10⁻³. This convergence criteria was found to be sufficient to obtain excellent agreement against experimental and numerical data for different benchmarks such as Taylor-Green vortex and the pulsatile/steady flow in the 90° bend. It should be noted that the problem reaches to the steady state at t = 1.23, i.e., all presented results are calculated in the transient regime. MF without preconditioner only converged on the coarsest grid. This is because MF requires Krylov subspace method as an iterative technique to solve the Newton equation at each iteration (k). While the convergence of Krylov methods depends on the condition number [23], i.e., the MF convergence degrades for Jacobians with large condition numbers. The Jacobian condition number is defined as the absolute value of maximum eigenvalue of the Jacobian matrix divided by the absolute value of minimum eigenvalue, which can be approximately extracted by solving the linear Newton equation by Krylov subspace method. The approach is to compute the eigenvalues of the orthogonal projection of the Jacobian onto the Krylov subspace in the Arnoldi algorithm [81]. The Jacobian condition number increases accordingly by increasing the grid size [23], as shown in Fig. 11 which plots the Jacobian condition number normalized relative to the smallest one for different grid sizes at initial time. Because of the high condition number on fine grids (Fig. 11), the MF without preconditioner converges only on the coarsest grid. To resolve this issue, a preconditioner is formed for MF. We used analytical Jacobian to form a preconditioner for MF because forming a preconditioner from the Jacobian computed numerically is quite expensive (similar to AD) and defeats the purpose of the matrix-free. The Jacobi preconditioner is used in the NKMs.

Figure 11.

The normalized Jacobian condition number a function of normalized grid size for simulation of steady flow in the 90° bend at initial time instant.

Fig. 12 plots the relative residual at non-dimensional t = 0.5 versus the number of iterations for different methods and grid sizes. As observed in Fig. 12(a) and (b), on coarse grids, there is no significant difference in the decrease of the relative residual among FJ, DJ, MF, and AD. MF has a slightly sharper relative convergence slope than FJ. AD’s relative residual slope is noticeably sharper than FJ for finer mesh cases (Fig. 12(c) and (d)). The primary reason for this trend is that the Jacobian in AD is exact, i.e., without any simplifications. Consequently, solving the Newton equations with an almost exact Jacobian at each iteration results in a more accurate estimation of the Newton equations solution. Thus, the number of the Newton iterations needed for convergence (k_max) at each time-step decreases. To illustrate this point, Fig. 13 presents the average iterations required for convergence on different grids. The average iterations is the average of k_max in all time-steps from non-dimensional t = 0 until t = 1. It can be observed that the average iterations for AD is about 18% less than MF, which is about 19% less than FJ, which is about 30% less than DJ in all grid sizes, except on the coarsest grid on which the values are quite close. The number of iterations of the FPRK is higher than all NKMs, except on the finest grid on which it is similar to DJ. Nevertheless, the number of iterations for convergence is not the only parameter that determines the computational time.

Figure 12.

Convergence of the Newton Eq. (14) at non-dimensional t = 0.5 for simulation of steady flow in the 90° bend for different methods and grid sizes. The relative residual of the right-hand side of the Newton Eq. (14) is plotted for each iteration. The iterations are considered converged when the relative residual becomes less than 10⁻³. The residual of Newton methods at the first iteration is 0.235, 0.188, 0.185, and 0.178 in (a), (b), (c), and (d), respectively and the residual of FPRK at the first iteration is 0.112, 0.105, 0.098, and 0.095 in (a), (b), (c), and (d), respectively.

Figure 13.

The average of k_max (the iterations required for convergence at each time-step) in all time-steps from non-dimensional t = 0 until t = 1 for steady flow in the 90° bend.

The performance of the developed methods in terms of the computational time is investigated by comparing FJ, DJ, AD, MF, and FPRK. The computational time is defined as the sum of all the time spent on solving the momentum equation (not the pressure Poisson Eq. (9)) at each time-step until non-dimensional t = 1 is reached. Therefore, the computational time depends on (1) the number of the time-steps to reach t = 1; and (2) the CPU time of each time-step. The number of time-steps is determined by the optimum CFL number ( $\mathit{CFL}=\frac{U\mathrm{\Delta }t}{h_{\mathit{min}}}$, where U is the inlet average velocity, Δt is the optimum time-step defined in Appendix B, and h_min is the minimum grid size of all directions of the curvilinear mesh), whereas the CPU time of each time-step depends on the number and cost of the Newton or fixed-point iterations to converge at each time-step.

Fig. 14 presents the computational time of the stated methods on four different grids (i.e., 21 × 21 × 101, 81 × 81 × 161, 145 × 145 × 301, and 185 × 185 × 351). Note that grid sizes are normalized relative to the minimum grid size, which is 21 × 21 ×101, and the computational time is normalized by the minimum computational time. To be consistent, the flow for a given grid is simulated on a similar number of CPUs for all methods. The simulations were carried out on the parallel computing cluster, Nami, with 28 nodes, each node containing two 8-core (Magny-Cours) AMD 2.0 GHz with 2GB RAM per core and QDR infiniband. All the simulations in this study were carried out on this parallel computing cluster (Nami).

Figure 14.

Normalized computational time as function of the method type from initial time instant until t = 1 for steady flow in the 90° bend for different grids.

It can be observed in Fig. 14 that AD spends 13.8 to 40.09 times more computational time than MF, which spends 2.72 to 4.24 times more computational time than FJ, which spends 1.72 to 2.15 times more computational time than DJ depending on the grid (size) for solving the momentum equation. Considering that the time-step size (consequently, the number of time-steps to reach t = 1) for AD, FJ, and DJ is similar on a specific grid, the question is why the computational time of DJ is smaller than FJ, and both of them smaller than AD, which converges in less iterations than all the other methods (Fig. 12). There are two main reasons for this trend: First, forming the numerical Jacobian in AD involves many function evaluations which consumes more computational time; Second, more components in the Jacobian (A) results in more expensive solutions of the Newton Eq. (14) at each iteration (k). Our experience shows that for this particular example, AD, FJ, and DJ need about 10, 3, and 1 FGMRES iterations to converge at each Newton iteration (k), respectively. The main reason for higher computational time of MF relative to FJ and DJ is the higher FGMRES iterations to converge at each Newton iteration, similar to AD (approximately 6 for this particular example).

Regardless of the mesh size, FJ and DJ require less computational time than FPRK, especially on finer grids (Fig. 14). There are two main reasons for this behavior: First, FJ and DJ generally need less iterations to converge than FPRK on the same grid (Fig. 13); Second, FJ and DJ converge at a larger time-step (larger optimum CFL) than fixed-point methods because of its semi-implicit nature, i.e., can reach t = 1 with lower number of time-steps. Fig. 15 presents the optimum CFL used in the simulations for each method and each grid size. It can be observed in Fig. 15 that the optimum CFL number decreases as grid size increases for FPRK, while it stays almost the same for DJ, FJ, and AD. Note that there is a trade-off between the time-step size and the number of the Newton iterations needed for convergence. By increasing the time-step size, the number of Newton iterations required for convergence increases because the initial condition for the Newton iteration, i.e., previous time-step solution, moves away from the solution as the time-step increases for the results presented in this section. We selected the optimum time-step size to minimize the computational time. The effect of time-step (CFL number) is further investigated in Section 4.4.

Figure 15.

Optimum CFL number as function of normalized grid size for steady flow in the 90° bend for different method types.

4.2. Grid topology effect on computational time

In the derivation of the analytical Jacobian, we made the simplifying assumption that the grid properties are locally constant to avoid Christoffel symbols and simplify the formulation (Section 2.3.1). This assumption is exactly satisfied on uniform grids, but creates deviations between the analytical Jacobian and the exact Jacobian on non-uniform grids. To investigate the effect of grid topology and non-uniformity on the computational time, the Taylor-Green vortex (Section 3.1) with Re = 10 is investigated. A quarter of the domain with size of π×π is selected as a sub-domain. The left side of the sub-domain is uniformly discretized and the lower side of the sub-domain is divided into four segments and stretched at each segment individually in order to keep minimum and maximum grid size in a reasonable range while enabling us to produce high grid stretching. It should be noted that each segment is stretched in reverse direction of that in neighbor segment(s). After applying a grid smoother, the sub-domain is mirrored in respect to y axis and the resulted sud-domain is mirrored in respect to x axis to obtain a periodic domain with 2π × 2π size. The geometric stretching factor is calculated from equation 29. Fig. 16 shows maximum difference between FJ and the AD Jacobian with different grid topologies (uniform, δ = 1.1, 1.2, and 1.5, where δ is the geometric stretching factor) and the same grid size (101 × 101) at Re = 10 from initial time instant until t = 1. The difference is calculated based on the infinity norm of the subtraction of the AD Jacobian from FJ, denoted as Δ_∞, normalized by the infinity norm of the AD Jacobian, denoted as ||A||_∞. As can be observed from Fig. 16, the normalized difference is negligible in uniform and even δ = 1.1 grids (less than 1%). By increasing the stretching factor, the normalized difference between FJ and AD Jacobian increases. However, the stretching factor should be bounded for maintaining the order of accuracy, e.g., the stretching error is highly increased from δ = 1.0083 to 1.0456 in [82]. It should be noted that the normalized difference between DJ and the diagonal of AD Jacobian is negligible for all grid topologies, i.e., the difference in Fig. 16 is mainly because of the off diagonal elements of the Jacobians.

Figure 16.

Maximum difference between FJ and the AD Jacobian in Taylor-Green vortex flow with different grid topologies and the same grid size (101 × 101) at Re = 10 from initial time instant until t = 1. The difference is calculated based on the infinity norm of the subtraction of the AD Jacobian from FJ and normalized by the infinity norm of the AD Jacobian.

The computational time is calculated as the summation of all the time spent on solving the momentum equation at each time-step until the non-dimensional t = 1 is reached. Computational time increases with δ in all methods, e.g., the ratio of computational time with δ = 1.1, 1.2, and 1.5 to that with uniform mesh is 1.44, 1.83, and 5.23, respectively in MF. However, the amount of increase is not the same between methods. To compare the computational time between methods as delta increases, Fig. 17(a) presents the normalized computational time of DJ, FJ, MF, FPRK, and AD methods on three different grid topologies (i.e., δ = 1, 1.1, 1.2, and 1.5) with the same grid size (101 × 101). The computational time is normalized by the minimum computational time obtained among methods on each grid topology. The computational time of MF is less than all other methods for all mesh topologies except the uniform mesh, which MF and DJ’s computational time are approximately equal. The ratio of DJ’s computational time to MF’s computational time increases from 0.94 to 2.77, 3.94, and 38.79 by increasing the stretching factor from 1 to 1.1, 1.2, and 1.5, receptively. Regardless of the mesh topology in Fig. 17(a), FJ spends more computational time than DJ for solving the momentum equation in all mesh topologies. The computational time ratio of AD to that of DJ decreases by increasing stretching factor, i.e., AD’s performance in comparison to the DJ, FJ, and FPRK degrades more slowly as the stretching factor is increased. However, because of high expense of full Jacobian formation and solving it, it still takes more computational time than DJ. The computational time of FPRK is higher in comparison to DJ in all mesh topologies. The ratio of FPRK’s computational time to DJ’s computational time increases from 2.5 to 11.15, 11.54, and 12.62 by increasing the stretching factor from 1 to 1.1, 1.2, and 1.5, receptively.

Figure 17.

Taylor-Green vortex flow with different grid topologies and same grid size (101 × 101) at Re = 10 for different method types: (a) computational time from initial time instant until t = 1 normalized by the fastest of them on each grid; and (b) optimum CFL number.

The main reason for the above trends is the CFL number. Fig. 17(b) presents the optimum CFL used in the simulations for each method and mesh topology. The degrade of the optimal time-step because of the rapid change in grid for MF (Δt = 0.1 and 0.05 for δ = 1 and 1.5, respectively) is much smaller than FJ and DJ (Δt = 0.1 and 0.0015 for δ = 1 and 1.5, respectively). Therefore, the rapid change in grid does not decrease, but increases the optimum CFL for MF, in contrast to DJ and FJ. This is the consequence of the fact that by increasing the stretching factor, the normalized difference between numerical and analytical Jacobian increases as well. The similar trend can be observed for AD. However, AD requires considerably more computational time to compute the product of Jacobian vector in comparison to MF since AD forms the Jacobian and then multiply it by vector, in contrast to MF. The CFL number decrease also cause faster degrade in FPRK’s computational performance in comparison to DJ. As observed, NKM’s optimum CFL is larger than FPRK regardless of mesh topology, which indicates lower number of time-steps (lower cost) to reach t = 1. In fact, for FPRK, in contrast to NKMs, CFL drastically decreases from 0.8 on a uniform grid to 0.000025 in δ = 1.5.

4.3. Reynolds number effect on computational time

The inline oscillations of a cylinder in a fluid initially at rest (Section 3.2) in three-dimension is used to investigate the Reynolds number effect on computational time among FJ, DJ, AD, and FPRK. The domain size is 100D×100D×D with 361×241×51 nodes in the inline (oscillatory), transverse, and cylinder’s height directions, respectively. Fig. 18(a) presents the normalized computational time of the stated methods for Re = 100, 1000, and 10000. The computational time is calculated as the summation of all the time spent on solving the momentum equation at each time-step until non-dimensional t = 1 is reached and is normalized by the minimum computational time at each Re. It can be observed in Fig. 18(a) that AD’s computational time is 36.52 to 52.65 times higher than FJ, which is 2.33 to 2.55 times higher than DJ, which is 1.38 to 9.2 times less than MF depending on the Reynolds number for solving the momentum equation. Computational time correlates with Re in DJ, FJ, and AD, e.g., the ratio of computational time with Re = 1000 and Re = 10000 to that with Re = 100 is 1.69 and 2.47, respectively in DJ. This is probably because increasing Re decreases the diagonal dominance of the Jacobian because the viscous terms in the diagonal are proportional to 1/Re (Appendix A). Diagonally dominant matrices are well-conditioned and converge faster [83]. In contrast, the computational time in MF decreases by increasing Re, i.e., the ratio of computational time with Re = 100 and 1000 to that with Re = 10000 is 2.7 and 1.05, respectively. This is because the normalized condition number of Jacobian, calculated similar to section 3.4, decreases by increasing Re, i.e., the normalized condition number of Re = 100 and 1000 to that of Re = 10000 is approximately 2.46 and 1.05, respectively, which leads to the less number of Newton iterations needed for convergence in higher Re (k_max ≈ 9, 3, and 3 for Re = 100, 1000, and 10000, respectively). This does not apply for DJ and FJ, e.g., k_max ≈ 6, 7, and 10 for Re = 100, 1000, and 10000, respectively. Although AD benefits from the low number of Newton iterations at each time steps in Re = 1000 and Re = 10000(k ≈ 3) similar to MF, the high computational cost of Jacobian formation in AD defeats this advantages, in contrast to MF.

Figure 18.

Inline oscillations of a cylinder in a fluid initially at rest in three-dimension at Re = 100, 1000, and 10000 for different method types: (a) computational time from initial time instant until t = 1 normalized by the fastest of them at each Re; and (b) optimum CFL number.

The computational time of FPRK increases by rising Re form 100 to 1000 and remains constant by rising Re form 1000 to 10000. FPRK’s computational time is higher than DJ at all Reynolds numbers while the ratio of FPRK computational time to DJ computational time decreases by increasing Re from 7 to 3.1. The main reason for the observed trends in Fig. 18(a) is the CFL number. Fig. 18(b) presents the optimum CFL used in the simulations for each method and Re. It can be observed that DJ’s optimum CFL is larger than FPRK (Fig. 18(b)). However by increasing Re the optimum CFL in DJ decreases, in contrast to FPRK which stays constant.

4.4. Convergence region: time-step effect on computational time

If the initial solution is far from the exact solution, the Newton method might fail to converge [39, 38]. The initial condition for the Newton method in here is the solution at the previous time instant. Therefore, the smaller the time-step, the closer the initial solution to the exact solution. The pulsatile flow through a 90° bend (Section 3.3) is used to investigate the time-step effect on computational time DJ, MF, FJ, and AD. The computational time is the sum of all the time spent on solving the momentum equation at each time-step until non-dimensional t = 1 is reached. The computational time is normalized by the minimum computational time for each method. Increasing the Δt will reduce the number of time-step to reach t = 1 but will require more Newton iterations because the initial solution of the Newton method has move away from the exact solution, i.e., there is a trade-off between the time-step size and the number of the Newton iterations needed for convergence. There is an optimum time-step size at which the computational time is minimum. It is noted that Δt is restricted with implemented pressure-velocity coupling algorithms, i.e., fractional-step, to conserve the accuracy of method and practically cannot be increased without bounds. Fig. 19 presents the normalized computational time of the stated methods for 145 × 145 × 301 grid on 160 CPUs using different time-step sizes. It is observed in Fig. 19 that the optimum time-step for DJ, FJ, and MF is similar (Δt = 0.01) and that for AD is Δt = 0.005. The optimum time-step size in FPRK is the largest Δt at which FPRK converges and in this case it is 0.001 (CFL=0.185). The increase in the time-step did not increase the computational time required for solving the Poisson equation.

Figure 19.

Computational time as function of time-step from initial time instant until t = 1 for pulsatile flow in the 90° bend for different grids. The computational time is normalized by the minimum computational time for each method.

4.5. Parallel performance

For further examination of the proposed methods, steady flow in the 90° bend (3.4) is used to demonstrate parallel performance of the analytical Jacobian for NKM. Parallel efficiency and speed up for strong-scaling based on the computational time on the smallest CPU number (N_min) is defined as follows:

$$\mathit{parallel}\mathit{efficiency}=\frac{T(N_{\mathit{min}})\times N_{\mathit{min}}}{T(N)\times N}\mathit{Speed}up=\frac{T(N)}{T(N_{\mathit{min}})}$$ (34)

where T(N) denotes computational time on N CPUs. N_min are 32 and 64 for DJ and FJ, respectively. We cannot run simulations on less than N_min number of CPUs because of memory requirements. The computational time is computed by the sum of all the time spent on solving the momentum equation for 10 time-steps from initial time instant, in which each time-step includes 10 Newton iterations (100 Newton iterations in total). Because of the memory requirements associated with the forming Jacobian containing more non-zero components in FJ, DJ requires less memory than FJ. Fig. 20 plots parallel efficiency and speed-up for FJ and DJ on the single grid (217 × 217 × 437, which is about 20 million nodes) on a different number of CPUs (32, 64, 128, 256, and 448). As shown in Fig. 20, the developed methods present excellent parallel efficiency and scale well with increasing CPU. The cache effect [84] can be observed on 64 and 128 number of CPUs for DJ and FJ, respectively, which increases the efficiency to slightly over one.

Figure 20.

Strong-scaling parallel performance investigation: (a) Speed up; (b) Efficiency for steady flow in the 90° bend on 217 × 217 × 437 grid dimension.

The same example is used for weak-scaling (grid size per CPU remains almost constant) investigation of speed up ( $\frac{T(16)}{T(N)}$) as shown in Fig. 21. The 16, 144, 240, and 448 number of CPUs are used for solving the same problem on 73 × 73 × 137, 149 × 149 × 297, 173 × 173 × 368, and 217 × 217 × 437 grid dimensions, respectively. Note that speed up is calculated based on the computational time of the numerical example on the coarsest grid (73 × 73 × 137) on the smallest CPU number (16). As shown in Fig. 21, the developed methods scale well with increasing CPU and grid size.

Figure 21.

Weak-scaling parallel speed up investigation for steady flow in the 90° bend.

4.6. Performance on a problem with both immersed boundary and overset grids

An intracranial aneurysm is simulated to show the ability of the developed methods to handle complex geometries with multiple overset grids and immersed boundaries. This geometry is reconstructed from three-dimensional rotational angiography of a human subject [85]. It is meshed by three body-fitted curvilinear meshes for inlet (with 61 × 61 × 293 grid nodes) and two outlets (with 33 × 33 × 145 and 45 × 45 × 73 grid nodes) and one fine uniform mesh for the dome (with 201 × 201 × 201 grid nodes) as shown in Fig. 22(a). The geometry of the dome is placed as an immersed boundary in the domain with the uniform grid. The Neumann boundary condition is applied on the outlet boundaries, and the flux of each outlet is determined based on the principle of optimal work [86, 87]. The inlet boundary condition is a pulsatile velocity waveform (Fig. 22(b)) [88]. The non-dimensional time period (T) of the inflow waveform is equal to 16.67, which is determined based on the 75 beats per minute heart rate (0.8s), the aneurysm inlet diameter, and the bulk velocity. The non-dimensional time-step Δt = 0.004 is used for this problem, and the Reynolds number is 145 ( $Re=\frac{UD}{\vartheta }$, where U is the bulk inlet velocity, D is the aneurysm inlet diameter, and ϑ is the blood kinematic viscosity). The NKM with DJ and FJ could successfully converge in approximately 5 Newton iterations at each time-step. Fig. 23 presents the evolution of the non-dimensional out-of-plane vorticity at a plane in the dome of the intracranial aneurysm at various time instants during a cycle. This numerical simulation has been carried out for a cycle on 96 CPUs. DJ is 1.24, 1.53, and 3.36 times faster than MF, FJ, and FPRK, respectively, in solving the momentum equation. The method to solve the momentum equation does not affect the required to solve the Poisson equation, i.e., the computational time to solve the Poisson equation is the same for all DJ, FJ, and FPRK. Therefore, DJ is only 1.15, 1.34, and 2.51 times faster than MF, FJ, and FPRK, respectively, in the overall solution (including both momentum and Poisson equations). As mentioned before, DJ is 9 to 25 times faster than FPRK for the simulations of the flow in the 90° bend. This value reduces to 3.36 in an intracranial aneurysm simulation because the time-step size is the same (non-dimensional Δt = 0.004) in all methods, i.e., DJ, FJ and FPRK. FPRK could converges with a Δt as large as the Newton method on the uniform grid. Therefore, DJ only takes advantage of converging in less iterations rather than both iterations and time-steps. In this simulation, similar to the flow in the 90° bend, DJ is faster than FJ. There is a trade-off between completeness of the analytical Jacobian and the Newton iterations required for convergence (i.e, the Newton equations with more complete analytical Jacobian converge in less Newton iterations, although it increases the computational cost because of increasing the FGMRES iterations and forming analytical Jacobian with more components). This simulation clearly show that using analytical Jacobian decreases the computational cost relative to fixed point and other NKMs such as AD and MF.

Figure 22.

(a) An intracranial aneurysm geometry consists of three curvilinear meshes for inlet, outlet1, outlet2, and a single immersed boundary at dome. (b) Pulsatile inlet flow for an Aneurysm where T is the non-dimensional time period of inflow oscillations and U is the median velocity.

Figure 23.

Evolution of the non-dimensional out-of-plane vorticity at a plane in the dome of the intracranial aneurysm at various time instants during a cycle; (a) t/T = 0.03, (b) t/T = 0.15, (c) t/T = 0.25, (d) t/T = 0.3, and (f) t/T = 0.75. The schematic of the flow direction is shown in (a).

5. Discussions and conclusions

Explicit and semi-implicit methods, e.g., FPRK, are computationally expensive to simulate flows involving complex geometries with moving boundaries mainly because of their severe time-step restriction and low convergence rates. We have developed a novel analytical Jacobian for the Newton-type implicit method in combination with Krylov subspace methods in order to enhance both the time-step size and the convergence rate. These methods were added to the previous framework [63, 64, 75] in which the Navier-Stokes equations were solved using FPRK and MF without a preconditioner. The analytical Jacobians were modified to work with the immersed boundary and overset grid methods to be able to handle complex geometries such as an intracranial aneurysm. Analytical Jacobian were validated and verified against well-known benchmarks with overset grids, moving immersed boundaries, pulsatile inlet flows, and steady flow on a curvilinear coordinates. Furthermore, analytical Jacobian presents the 2^nd order accuracy in space and time and excellent parallel efficiency (more than 80–90%). Finally, the analytical Jacobian was used as a preconditioner for the MF to improve its performance.

We build the Jacobian with four techniques: (1) MF, in which no Jacobian-matrix is formed; (2) AD, in which the Jacobian is formed completely; (3) FJ, in which some simplifications are applied to derive it analytically; and (4) DJ, in which FJ is approximated by its diagonal part. We have developed DJ to investigate if we can further reduce the computational time in comparison to FJ, considering that there is no agreement in literature on the superiority of one over the other in term of the computational time. Wright et al. [71] implemented lower-upper symmetric Gauss-Seidel method for the simulation of viscous flows and concluded that, for different Reynolds numbers, grid topologies, and grid sizes, full Jacobian achieves convergence faster than the diagonal Jacobian in terms of computational time. The diagonal Jacobian in aforementioned study is simply formed by a spectral approximation. The full and/or diagonal Jacobian (formed by the spectral approximation) are used as preconditioner for Krylov methods to solve Navier-Stokes equations [89]. Ekici et al. [89] stated that for time-accurate computations, hybrid of diagonal and full Jacobian method produces the best performance. Similarly Radhakrishnan and Hindmarsh [72] found that diagonal Jacobian (computed by neglecting all off-diagonal elements of the full Jacobian formed by automatic differentiation scheme) for Newton iteration for solving ordinary differential equations is not as fast as the full Jacobian. In contrast, Waziri et al. [70] stated that the diagonal Jacobian for nonlinear equations is faster than the full Jacobian (computed by automatic differentiation method). Therefore, the superiority of one over the other in terms of the computational time depends on the set of equation and the solver. For Navier-Stokes equations on staggered grids, based on our results, DJ is always faster than FJ.

We have investigated effect of Re, grid topology, and grid size on their performance in terms of computational time. The computational time ratio of FJ over DJ decreases by the increase of grid size (from 2.15 on coarse grid to 1.72 on fine grid) for steady flow in the 90° bend, increases by the increase of stretching factor (from 2.13 when δ = 1 to 3.12 when δ = 1.5) for Taylor-Green vortex problem, and slightly decreases by the increase of Re (from 2.55 with Re = 100 to 2.33 with Re = 10000) for the inline oscillations of a cylinder.

AD is the most expensive method among the developed methods in most of the simulations. There are two main reasons for high expense of AD despite its high convergence rate (especially on fine grids): First, the numerical Jacobian formation involves many function evaluations which consumes more computational time; Second, the Jacobian with more components leads to more expensive solutions of the Newton Eq. (14) at each iteration (k). However, AD’s performance improves in term of computational time when there is a rapid change in the grid because of higher optimum CFL in comparison to FPRK, FJ, and DJ (Fig. 17(b)).

To the best of our knowledge, this is the first study that different Newton methods are compared with fixed point method in terms of computational time on a real problem. DJ and FJ are faster than FPRK in all our simulations. There are two main reasons for this behavior: First, DJ and FJ generally need less Newton iterations to converge than FPRK on the same grid; Second, FJ and DJ’s optimum CFL is larger than fixed-point methods because of its semi-implicit nature. The computational time ratio of FPRK over DJ increases by the increase of grid size (from 9.37 on coarse grid to 25.4 on fine grid) for steady flow in the 90° bend, increases by the increase of stretching factor (from 2.5 when δ = 1 to 12.6 when δ = 1.5) for Taylor-Green vortex problem, and decreases by the increase of Re (from 7.01 with Re = 100 to 3.1 with Re = 10000) for the inline oscillations of a cylinder.

It is observed that MF without preconditioner converges only on the coarsest grid for steady flow in the 90° bend because the condition number of the Jacobian increases as the grid is refined (Fig. 11). Preconditioners for MF is formed using the analytical Jacobian since forming the Jacobian numerically is quite expensive (similar to AD). MF is the only method which can compete with performance of DJ in terms of the computational time on solving the momentum equation among all other methods (MF, FJ, AD, and FPRK). In fact, the computational time of DJ is 2.77, 3.94, and 38.79 times higher than MF in Taylor-Green vortex problem with stretching factor of 1.1, 1.2, and 1.5, receptively. In addition, the computational time of MF is slightly higher than DJ (1.38 times) in the inline oscillations of a cylinder with Re = 10000. However, MF’s computational time is 3.9 to 9.22 times higher than DJ’s for steady flow in the 90° bend depending on grid sizes and this ratio is 9.22 and 2.11 for the inline oscillations of a cylinder with Re = 100 and 1000, respectively. The computational time ratio of MF over DJ decreases by the increase of grid size (from 8.16 on coarse grid to 3.99 on fine grid) for steady flow in the 90° bend, decreases by the increase of stretching factor (from 1.05 when δ = 1 to 0.026 when δ = 1.5) for Taylor-Green vortex problem, and decreases by the increase of Re (from 9.22 with Re = 100 to 1.38 with Re = 10000) for the inline oscillations of a cylinder.

It was shown herein that using a simplified analytical Jacobian in combination with Krylov subspace method can lead to a computationally efficient implicit method for solving Navier-Stokes equations. The results of this study can be used to improve the performance of other numerical techniques for solving Navier-Stokes equations. It should be noted that the computational times calculated in this study are restricted to our implementations of the different methods.

Nomenclature

NKM

Newton-Krylov method

AD

Automatic differentiation, i.e., NKM, in which Jacobian is formed by automatic differentiation method

FJ

Full Jacobian, i.e., analytical Jacobian, in which whole components are considered

DJ

Diagonal Jacobian, i.e., analytical Jacobian, in which diagonal components are considered

MF

Matrix-free NKM

FPRK

Fixed-point Runge-Kutta method

Appendix A. The components of the analytical Jacobian for 3D Navier-Stokes equation on a staggered grid

In the following we present the components of the analytical Jacobian for 3D Navier-Stokes equation on a staggered grid. For assembling these components to derive the analytical Jacobian see Fig. 2. For simplicity, all the components, presented following, are factorized by $\frac{1}{J}$ at corresponding locations.

Appendix B. Optimal computational time

The steady flow through a 90° bend (Section 3.4) is used to investigate the time-step effect on computational time DJ, FJ, MF, and AD. The computational time is the sum of all the time spent on solving the momentum equation at each time-step until non-dimensional t = 1 is reached. The computational time is normalized by the minimum computational time for each method. Increasing the Δt will reduce the number of time-step to reach t = 1 but will require more Newton iterations because the initial solution of the Newton method has move away from the exact solution, i.e., there is a trade-off between the time-step size and the number of the Newton iterations needed for convergence. There is an optimum time-step size at which the computational time is minimum. Fig. B.24 presents the normalized computational time of the stated methods for 145 × 145 × 301 grid on 160 CPUs using different time-step sizes. It is observed from Fig. B.24 that the optimum time-step for DJ, FJ, AD is similar (Δt = 0.01) and that for MF is Δt = 0.005. Based on our simulations, these methods can converge at very large time-step size (e.g., Δt = 1 equivalent to CFL = 185). However, this causes considerably small convergence rate, e.g., DJ requires 2673 Newton iterations to decrease the relative residual by three orders of magnitude. The optimum time-step size in FPRK is the largest Δt at which FPRK converges and in this case it is 0.001 (CFL=0.185).

Figure B.24.

Computational time as function of time-step from initial time instant until t = 1 for steady flow in the 90° bend for different grids. The computational time is normalized by the minimum computational time for each method.

Table A.4.

$J_{(i+\frac{1}{2},j,k)}{A}_{11({\xi }^1,{\xi }^2,{\xi }^3)}$ components of the analytical Jacobian for 3D Navier-Stokes equations, where J is the Jacobian of the geometric transformation.

ξ1	ξ2	ξ3	$J_{(i+\frac{1}{2},j,k)}{A}_{11({\xi }^1,{\xi }^2,{\xi }^3)}$
$i-\frac{3}{2}$	j	k	$\frac{1}{6}{U}_{(i-\frac{3}{2},j,k)}^1$
$i-\frac{1}{2}$	j − 1	k	$-\frac{1}{8}{U}_{(i,j-\frac{1}{2},k)}^2$
$i-\frac{1}{2}$	j	k − 1	$-\frac{1}{8}{U}_{(i,j,k-\frac{1}{2})}^3$
$i-\frac{1}{2}$	j	k	$-\frac{1}{Re}{g}_{(i+\frac{1}{2},j,k)}^{11}-\frac{8}{6}{U}_{(i-\frac{1}{2},j,k)}^1+\frac{1}{8}(U_{(i,j+\frac{1}{2},k)}^2-U_{(i,j-\frac{1}{2},k)}^2+U_{(i,j,k+\frac{1}{2})}^3-U_{(i,j,k-\frac{1}{2})}^3)$
$i-\frac{1}{2}$	j	k + 1	$\frac{1}{8}{U}_{(i,j,k+\frac{1}{2})}^3$
$i-\frac{1}{2}$	j + 1	k	$\frac{1}{8}{U}_{(i,j+\frac{1}{2},k)}^2$
$i+\frac{1}{2}$	j − 1	k	$-\frac{1}{Re}{\stackrel{\sim }{g}}_{(i+\frac{1}{2},j,k)}^{22}-\frac{1}{8}(U_{(i,j-\frac{1}{2},k)}^2+U_{(i+1,j-\frac{1}{2},k)}^2)$
$i+\frac{1}{2}$	j	k − 1	$-\frac{1}{Re}{\stackrel{\sim }{g}}_{(i+\frac{1}{2},j,k)}^{33}-\frac{1}{8}(U_{(i,j,k-\frac{1}{2})}^3+U_{(i+1,j,k-\frac{1}{2})}^3)$
$i+\frac{1}{2}$	j	k	$$\begin{gathered} \begin{array}{c}\frac{3}{2}\mathrm{\Delta }t+\frac{2}{Re}(g_{(i+\frac{1}{2},j,k)}^{11},{\stackrel{\sim }{g}}_{(i+\frac{1}{2},j,k)}^{22}+{\stackrel{\sim }{g}}_{(i+\frac{1}{2},j,k)}^{33})\\ +\frac{1}{8}(U_{(i,j+\frac{1}{2},k)}^2-U_{(i,j-\frac{1}{2},k)}^2+U_{(i+1,j+\frac{1}{2},k)}^2-U_{(i+1,j-\frac{1}{2},k)}^2+U_{(i,j,k+\frac{1}{2})}^3-U_{(i,j,k-\frac{1}{2})}^3+U_{(i+1,j,k+\frac{1}{2})}^3 \\ -U_{(i+1,j,k-\frac{1}{2})}^3)\end{array} \end{gathered}$$
$i+\frac{1}{2}$	j	k + 1	$-\frac{1}{Re}{\stackrel{\sim }{g}}_{(i+\frac{1}{2},j,k)}^{33}+\frac{1}{8}(U_{(i,j,k+\frac{1}{2})}^3+U_{(i+1,j,k+\frac{1}{2})}^3)$
$i+\frac{1}{2}$	j + 1	k	$-\frac{1}{Re}{\stackrel{\sim }{g}}_{(i+\frac{1}{2},j,k)}^{22}+\frac{1}{8}(U_{(i,j+\frac{1}{2},k)}^2+U_{(i+1,j+\frac{1}{2},k)}^2)$
$i+\frac{3}{2}$	j − 1	k	$\frac{1}{8}{U}_{(i+1,j-\frac{1}{2},k)}^2$
$i+\frac{3}{2}$	j	k − 1	$\frac{1}{8}{U}_{(i+1,j,k-\frac{1}{2})}^3$
$i+\frac{3}{2}$	j	k	$-\frac{1}{Re}{g}_{(i+\frac{1}{2},jk)}^{11}+\frac{8}{6}{U}_{(i+\frac{3}{2},j,k)}^1+\frac{1}{8}(U_{(i+1,j+\frac{1}{2},k)}^2-U_{(i+1,j-\frac{1}{2},k)}^2+U_{(i+1,j,k+\frac{1}{2})}^3-U_{(i+1,j,k-\frac{1}{2})}^3)$
$i+\frac{3}{2}$	j	k + 1	$\frac{1}{8}{U}_{(i+1,j,k+\frac{1}{2})}^3$
$i+\frac{3}{2}$	j + 1	k	$\frac{1}{8}{U}_{(i+1,j+\frac{1}{2},k)}^2$
$i+\frac{5}{2}$	j	k	$-\frac{1}{6}{U}_{(i+\frac{5}{2},j,k)}^1$

Table A.5.

$J_{(i+\frac{1}{2},j,k)}{A}_{12({\xi }^1,{\xi }^2,{\xi }^3)}$ components of the analytical Jacobian for 3D Navier-Stokes equations, where J is the Jacobian of the geometric transformation.

ξ1	ξ2	ξ3	$J_{(i+\frac{1}{2},j,k)}{A}_{12({\xi }^1,{\xi }^2,{\xi }^3)}$
i	$j-\frac{1}{2}$	k	$-\frac{1}{2}{\stackrel{\sim }{U}}_{(i,j-\frac{1}{2},k)}^1$
i + 1	$j-\frac{1}{2}$	k	$-\frac{1}{2}{\stackrel{\sim }{U}}_{(i+1,j-\frac{1}{2},k)}^1$
i	$j+\frac{1}{2}$	k	$\frac{1}{2}{\stackrel{\sim }{U}}_{(i,j+\frac{1}{2},k)}^1$
i + 1	$j+\frac{1}{2}$	k	$\frac{1}{2}{\stackrel{\sim }{U}}_{(i+1,j+\frac{1}{2},k)}^1$

Table A.6.

$J_{(i+\frac{1}{2},j,k)}{A}_{13({\xi }^1,{\xi }^2,{\xi }^3)}$ components of the analytical Jacobian for 3D Navier-Stokes equations, where J is the Jacobian of the geometric transformation.

ξ1	ξ2	ξ3	$J_{(i+\frac{1}{2},j,k)}{A}_{13({\xi }^1,{\xi }^2,{\xi }^3)}$
i	j	$k-\frac{1}{2}$	$-\frac{1}{2}{\stackrel{\sim }{U}}_{(i,j,k-\frac{1}{2})}^1$
i + 1	j	$k-\frac{1}{2}$	$-\frac{1}{2}{\stackrel{\sim }{U}}_{(i+1,j,k-\frac{1}{2})}^1$
i	j	$k+\frac{1}{2}$	$\frac{1}{2}{\stackrel{\sim }{U}}_{(i,j,k+\frac{1}{2})}^1$
i + 1	j	$k+\frac{1}{2}$	$\frac{1}{2}{\stackrel{\sim }{U}}_{(i+1,j,k+\frac{1}{2})}^1$

Table A.7.

$J_{(i,j+\frac{1}{2},k)}{A}_{21({\xi }^1,{\xi }^2,{\xi }^3)}$ components of the analytical Jacobian for 3D Navier-Stokes equations, where J is the Jacobian of the geometric transformation.

ξ1	ξ2	ξ3	$J_{(i,j+\frac{1}{2},k)}{A}_{21({\xi }^1,{\xi }^2,{\xi }^3)}$
$i-\frac{1}{2}$	j	k	$-\frac{1}{2}{\stackrel{\sim }{U}}_{(i-\frac{1}{2},j,k)}^2$
$i-\frac{1}{2}$	j + 1	k	$-\frac{1}{2}{\stackrel{\sim }{U}}_{(i-\frac{1}{2},j+1,k)}^2$
$i+\frac{1}{2}$	j	k	$\frac{1}{2}{\stackrel{\sim }{U}}_{(i+\frac{1}{2},j,k)}^2$
$i+\frac{1}{2}$	j + 1	k	$\frac{1}{2}{\stackrel{\sim }{U}}_{(i+\frac{1}{2},j+1,k)}^2$

Table A.8.

$J_{(i,j+\frac{1}{2},k)}{A}_{22({\xi }^1,{\xi }^2,{\xi }^3)}$ components of the analytical Jacobian for 3D Navier-Stokes equations, where J is the Jacobian of the geometric transformation.

ξ1	ξ2	ξ3	$J_{(i,j+\frac{1}{2},k)}{A}_{22({\xi }^1,{\xi }^2,{\xi }^3)}$
i	$j-\frac{3}{2}$	k	$\frac{1}{6}{U}_{(i,j-\frac{3}{2},k)}^2$
i − 1	$j-\frac{1}{2}$	k	$-\frac{1}{8}{U}_{(i-\frac{1}{2},j,k)}^2$
i	$j-\frac{1}{2}$	k − 1	$-\frac{1}{8}{U}_{(i,j,k-\frac{1}{2})}^3$
i	$j-\frac{1}{2}$	k	$-\frac{1}{Re}{g}_{(i,j+\frac{1}{2},k)}^{22}-\frac{8}{6}{U}_{(i,j-\frac{1}{2},k)}^2+\frac{1}{8}(U_{(i+\frac{1}{2},j,k)}^1-U_{(i-\frac{1}{2},j,k)}^1+U_{(i,j,k+\frac{1}{2})}^3-U_{(i,j,k-\frac{1}{2})}^3)$
i	$j-\frac{1}{2}$	k + 1	$\frac{1}{8}{U}_{(i,j,k+\frac{1}{2})}^3$
i + 1	$j-\frac{1}{2}$	k	$\frac{1}{8}{U}_{(i+\frac{1}{2},j,k)}^1$
i − 1	$j+\frac{1}{2}$	k	$-\frac{1}{Re}{\stackrel{\sim }{g}}_{(i,j+\frac{1}{2},k)}^{11}-\frac{1}{8}(U_{(i-\frac{1}{2},j,k)}^1+U_{(i-\frac{1}{2},j+1,k)}^1)$
i	$j+\frac{1}{2}$	k − 1	$-\frac{1}{Re}{\stackrel{\sim }{g}}_{(i,j+\frac{1}{2},k)}^{33}-\frac{1}{8}(U_{(i,j,k-\frac{1}{2})}^3+U_{(i,j+1,k-\frac{1}{2})}^3)$
i	$j+\frac{1}{2}$	k	$$\begin{gathered} \begin{array}{c}\frac{3}{2}\mathrm{\Delta }t+\frac{2}{Re}({\stackrel{\sim }{g}}_{(i,j+\frac{1}{2},k)}^{11}+g_{(i,j+\frac{1}{2},k)}^{22}+{\stackrel{\sim }{g}}_{(i,j+\frac{1}{2},k)}^{33})\\ +\frac{1}{8}(U_{(i+\frac{1}{2},j,k)}^1-U_{(i-\frac{1}{2},j,k)}^1+U_{(i+\frac{1}{2},j+1,k)}^1-U_{(i-\frac{1}{2},j+1,k)}^1+U_{(i,j,k+\frac{1}{2})}^3-U_{(i,j,k-\frac{1}{2})}^3+U_{(i,j+1,k+\frac{1}{2})}^3 \\ -U_{(i,j+1,k-\frac{1}{2})}^3)\end{array} \end{gathered}$$
i	$j+\frac{1}{2}$	k + 1	$-\frac{1}{Re}{\stackrel{\sim }{g}}_{(i,j+\frac{1}{2},k)}^{33}+\frac{1}{8}(U_{(i,j,k+\frac{1}{2})}^3+U_{(i,j+1,k+\frac{1}{2})}^3)$
i + 1	$j+\frac{1}{2}$	k	$-\frac{1}{Re}{\stackrel{\sim }{g}}_{(i,j+\frac{1}{2},k)}^{11}+\frac{1}{8}(U_{(i+\frac{1}{2},j,k)}^1+U_{(i+\frac{1}{2},j+1,k)}^1)$
i − 1	$j+\frac{3}{2}$	k	$-\frac{1}{8}{U}_{(i-\frac{1}{2},j+1,k)}^1$
i	$j+\frac{3}{2}$	k − 1	$-\frac{1}{8}{U}_{(i,j+1,k-\frac{1}{2})}^3$
i	$j+\frac{3}{2}$	k	$$\begin{gathered} -\frac{1}{Re}{g}_{(i,j+\frac{1}{2},k)}^{22}+\frac{8}{6}{U}_{(i,j+\frac{3}{2},k)}^2+\frac{1}{8}(U_{(i+\frac{1}{2},j+1,k)}^1-U_{(i-\frac{1}{2},,j+1,k)}^1+U_{(i,j+1,k+\frac{1}{2})}^3 \\ -U_{(i,j+1,k-\frac{1}{2})}^3) \end{gathered}$$
i	$j+\frac{3}{2}$	k + 1	$\frac{1}{8}{U}_{(i,j+1,k+\frac{1}{2})}^3$
i + 1	$j+\frac{3}{2}$	k	$\frac{1}{8}{U}_{(i+\frac{1}{2},j+1,k)}^1$
i	$j+\frac{5}{2}$	k	$-\frac{1}{6}{U}_{(i,j+\frac{5}{2},k)}^2$

Table A.9.

$J_{(i,j+\frac{1}{2},k)}{A}_{23({\xi }^1,{\xi }^2,{\xi }^3)}$ components of the analytical Jacobian for 3D Navier-Stokes equations, where J is the Jacobian of the geometric transformation.

ξ1	ξ2	ξ3	$J_{(i,j+\frac{1}{2},k)}{A}_{23({\xi }^1,{\xi }^2,{\xi }^3)}$
i	j	$k-\frac{1}{2}$	$-\frac{1}{2}{\stackrel{\sim }{U}}_{(i,j,k-\frac{1}{2})}^2$
i	j + 1	$k-\frac{1}{2}$	$-\frac{1}{2}{\stackrel{\sim }{U}}_{(i,j+1,k-\frac{1}{2})}^2$
i	j	$k+\frac{1}{2}$	$\frac{1}{2}{\stackrel{\sim }{U}}_{(i,j,k+\frac{1}{2})}^2$
i	j + 1	$k+\frac{1}{2}$	$\frac{1}{2}{\stackrel{\sim }{U}}_{(i,j+1,k+\frac{1}{2})}^2$

Table A.10.

$J_{(i,j,k+\frac{1}{2})}{A}_{31({\xi }^1,{\xi }^2,{\xi }^3)}$ components of the analytical Jacobian for 3D Navier-Stokes equations, where J is the Jacobian of the geometric transformation.

ξ1	ξ2	ξ3	$J_{(i,j,k+\frac{1}{2})}{A}_{31({\xi }^1,{\xi }^2,{\xi }^3)}$
$i-\frac{1}{2}$	j	k	$-\frac{1}{2}{\stackrel{\sim }{U}}_{(i-\frac{1}{2},j,k)}^3$
$i-\frac{1}{2}$	j	k + 1	$-\frac{1}{2}{\stackrel{\sim }{U}}_{(i-\frac{1}{2},j,k+1)}^3$
$i+\frac{1}{2}$	j	k	$\frac{1}{2}{\stackrel{\sim }{U}}_{(i+\frac{1}{2},j,k)}^3$
$i+\frac{1}{2}$	j	k + 1	$\frac{1}{2}{\stackrel{\sim }{U}}_{(i+\frac{1}{2},j,k+1)}^3$

Table A.11.

$J_{(i,j,k+\frac{1}{2})}{A}_{32({\xi }^1,{\xi }^2,{\xi }^3)}$ components of the analytical Jacobian for 3D Navier-Stokes equations, where J is the Jacobian of the geometric transformation.

ξ1	ξ2	ξ3	$J_{(i,j,k+\frac{1}{2})}{A}_{32({\xi }^1,{\xi }^2,{\xi }^3)}$
i	$j-\frac{1}{2}$	k	$-\frac{1}{2}{\stackrel{\sim }{U}}_{(i,j-\frac{1}{2},k)}^3$
i	$j-\frac{1}{2}$	k + 1	$-\frac{1}{2}{\stackrel{\sim }{U}}_{(i,j-\frac{1}{2},k+1)}^3$
i	$j+\frac{1}{2}$	k	$\frac{1}{2}{\stackrel{\sim }{U}}_{(i,j+\frac{1}{2},k)}^3$
i	$j+\frac{1}{2}$	k + 1	$\frac{1}{2}{\stackrel{\sim }{U}}_{(i,j+\frac{1}{2},k+1)}^3$

Table A.12.

$J_{(i,j,k+\frac{1}{2})}{A}_{33({\xi }^1,{\xi }^2,{\xi }^3)}$ components of the analytical Jacobian for 3D Navier-Stokes equations, where J is the Jacobian of the geometric transformation.

ξ1	ξ2	ξ3	$J_{(i,j,k+\frac{1}{2})}{A}_{33({\xi }^1,{\xi }^2,{\xi }^3)}$
i	j	$k-\frac{3}{2}$	$\frac{1}{6}{U}_{(i,j,k-\frac{3}{2})}^3$
i	j − 1	$k-\frac{1}{2}$	$-\frac{1}{8}{U}_{(i,j-\frac{1}{2},k)}^2$
i − 1	j	$k-\frac{1}{2}$	$-\frac{1}{8}{U}_{(i-\frac{1}{2},j,k)}^1$
i	j	$k-\frac{1}{2}$	$-\frac{1}{Re}{g}_{(i,j,k+\frac{1}{2})}^{33}-\frac{8}{6}{U}_{(i,j,k-\frac{1}{2})}^3+\frac{1}{8}(U_{(i+\frac{1}{2},j,k)}^1-U_{(i-\frac{1}{2},j,k)}^1+U_{(i,j+\frac{1}{2},k)}^2-U_{(i,j-\frac{1}{2},k)}^2)$
i + 1	j	$k-\frac{1}{2}$	$\frac{1}{8}{U}_{(i+\frac{1}{2},j,k)}^1$
i	j + 1	$k-\frac{1}{2}$	$\frac{1}{8}{U}_{(i,j+\frac{1}{2},k)}^2$
i	j − 1	$k+\frac{1}{2}$	$-\frac{1}{Re}{\stackrel{\sim }{g}}_{(i,j,k+\frac{1}{2})}^{22}-\frac{1}{8}(U_{(i,j-\frac{1}{2},k)}^2+U_{(i,j-\frac{1}{2},k+1)}^2)$
i − 1	j	$k+\frac{1}{2}$	$-\frac{1}{Re}{\stackrel{\sim }{g}}_{(i,j,k+\frac{1}{2})}^{11}-\frac{1}{8}(U_{(i-\frac{1}{2},j,k)}^1+U_{(i-\frac{1}{2},j,k+1)}^1)$
i	j	$k+\frac{1}{2}$	$$\begin{gathered} \begin{array}{c}\frac{3}{2}\mathrm{\Delta }t+\frac{2}{Re}({\stackrel{\sim }{g}}_{(i,j,k+\frac{1}{2})}^{11}+{\stackrel{\sim }{g}}_{(i,j,k+\frac{1}{2})}^{22}+g_{(i,j,k+\frac{1}{2})}^{33})\\ +\frac{1}{8}(U_{(i+\frac{1}{2},j,k)}^1-U_{(i-\frac{1}{2},j,k)}^1+U_{(i+\frac{1}{2},j,k+1)}^1-U_{(i-\frac{1}{2},j,k+1)}^1+U_{(i,j+\frac{1}{2},k)}^2-U_{(i,j-\frac{1}{2},k)}^2+U_{(i,j+\frac{1}{2},k+1)}^2 \\ -U_{(i,j-\frac{1}{2},k+1)}^2)\end{array} \end{gathered}$$
i + 1	j	$k+\frac{1}{2}$	$-\frac{1}{Re}{\stackrel{\sim }{g}}_{(i,j,k+\frac{1}{2})}^{11}+\frac{1}{8}(U_{(i+\frac{1}{2},j,k)}^1+U_{(i+\frac{1}{2},j,k+1)}^1)$
i	j + 1	$k+\frac{1}{2}$	$-\frac{1}{Re}{\stackrel{\sim }{g}}_{(i,j,k+\frac{1}{2})}^{22}+\frac{1}{8}(U_{(i,j+\frac{1}{2},k)}^2+U_{(i,j+\frac{1}{2},k+1)}^2)$
i	j − 1	$k+\frac{3}{2}$	$-\frac{1}{8}{U}_{(i,j-\frac{1}{2},k+1)}^2$
i − 1	j	$k+\frac{3}{2}$	$-\frac{1}{8}{U}_{(i-\frac{1}{2},j,k+1)}^1$
i	j	$k+\frac{3}{2}$	$-\frac{1}{Re}{g}_{(i,j,k+\frac{1}{2})}^{33}+\frac{8}{6}{U}_{(i,j,k+\frac{3}{2})}^3+\frac{1}{8}(U_{(i+\frac{1}{2},j,k+1)}^1-U_{(i-\frac{1}{2},j,k+1)}^1+U_{(i,j+\frac{1}{2},k+1)}^2-U_{(i,j-\frac{1}{2},k+1)}^2)$
i + 1	j	$k+\frac{3}{2}$	$\frac{1}{8}{U}_{(i+\frac{1}{2},j,k+1)}^1$
i	j + 1	$k+\frac{3}{2}$	$\frac{1}{8}{U}_{(i,j+\frac{1}{2},k+1)}^2$
i	j	$k+\frac{5}{2}$	$-\frac{1}{6}{U}_{(i,j,k+\frac{5}{2})}^3$