Anatomically accurate high resolution modeling of human whole heart electromechanics: A strongly scalable algebraic multigrid solver method for nonlinear deformation

Abstract

Electromechanical (EM) models of the heart have been used successfully to study fundamental mechanisms underlying a heart beat in health and disease. However, in all modeling studies reported so far numerous simplifications were made in terms of representing biophysical details of cellular function and its heterogeneity, gross anatomy and tissue microstructure, as well as the bidirectional coupling between electrophysiology (EP) and tissue distension. One limiting factor is the employed spatial discretization methods which are not sufficiently flexible to accommodate complex geometries or resolve heterogeneities, but, even more importantly, the limited efficiency of the prevailing solver techniques which are not sufficiently scalable to deal with the incurring increase in degrees of freedom (DOF) when modeling cardiac electromechanics at high spatio-temporal resolution.

This study reports on the development of a novel methodology for solving the nonlinear equation of finite elasticity using human whole organ models of cardiac electromechanics, discretized at a high para-cellular resolution. Three patient-specific, anatomically accurate, whole heart EM models were reconstructed from magnetic resonance (MR) scans at resolutions of 220 μm, 440 μm and 880 μm, yielding meshes of approximately 184.6, 24.4 and 3.7 million tetrahedral elements and 95.9, 13.2 and 2.1 million displacement DOF, respectively. The same mesh was used for discretizing the governing equations of both electrophysiology (EP) and nonlinear elasticity. A novel algebraic multigrid (AMG) preconditioner for an iterative Krylov solver was developed to deal with the resulting computational load. The AMG preconditioner was designed under the primary objective of achieving favorable strong scaling characteristics for both setup and solution runtimes, as this is key for exploiting current high performance computing hardware.

Benchmark results using the 220 μm, 440 μm and 880 μm meshes demonstrate efficient scaling up to 1024, 4096 and 8192 compute cores which allowed the simulation of a single heart beat in 44.3, 87.8 and 235.3 minutes, respectively. The efficiency of the method allows fast simulation cycles without compromising anatomical or biophysical detail.

1. Introduction

Electromechanical function of the heart emerges from a complex cascade of processes, which interact across a broad range of spatial and temporal scales. Despite the large body of experimental and theoretical research accumulated, our current understanding of how alterations in subcellular function affect whole organ pumping performance or, conversely, how changes at the systemic level influence subcellular function, remains incomplete. Computational modeling bears high promise as a methodology for integrating experimental data into a mechanistic framework which enables the quantitative observation of complex cause-effect relationships across a broad range of spatio-temporal scales. While the potential gains from modeling approaches are significant, the challenges to be addressed are daunting. Historically, the development of EP and mechanical models of cardiac function proceeded rather independently than in tandem. The vast majority of EP modeling studies ignored any effects due to mechanical deformation, and, vice versa, most mechanical modeling studies did not represent explicitly any feedback of deformation on EP. While such a simplified EM coupling has proven suitable for addressing a variety of questions [85, 53, 60], the assumption of a unidirectional coupling between EP and mechanical deformation is inaccurate as there is strong evidence that EP is modulated by tissue distension via mechano-electric feedback (MEF) mechanisms [58, 62, 63]. However, from a model development point of view, it is attractive to assume that MEF can be neglected, as this allows to develop EP and mechanical models independently which reduces the complexity of implementation. This effective split into two separate sequentially executed solution steps is reflected in a notable divergence in the employed methodologies between EP and mechanics modeling communities.

State of the art organ scale EP finite element or finite volume models are discretized at a high spatio-temporal resolution using tetrahedral [113] or hybrid elements [101]. High resolutions are not only necessary to capture the fast transients and steep depolarization wavefronts, but also to resolve fine microscopic scale structural detail [94, 123] as well as functional heterogeneities [78, 18, 57]. In contrast, mechanical models build on the assumption that cardiac deformation is governed by smoother spatial and slower temporal scales, suggesting that the use of much coarser spatio-temporal discretizations may yield sufficient accuracy. Thus the use of higher order cubic Hermite elements became popular [24] as they allow tessellation of stylized ventricular anatomy using a very small number of elements and they avoid volumetric locking, however, their major drawback is the limited capacity to accommodate complex geometries. This issue is being addressed now with the use of tetrahedral P₁ – P₀, P₂ – P₀ or mixed formulation P₂ – P₁ elements, which have been introduced only very recently for modeling cardiac mechanics [43, 42, 32]. In general, finite element discretizations with tetrahedral and hexahedral elements of the same order showed similar results in terms of precision and efficiency [19].

Unsurprisingly, as a consequence the cost of model execution has been addressed quite differently as well. In EP modeling, due to a large number of DOF, a pressing need for strongly scalable iterative solvers has emerged [82, 98], whereas for the lower dimensional mechanical models this has not been the case. While high resolution models and corresponding scalable solver algorithms have been developed for vascular applications [61, 3], for cardiac mechanics direct solvers, executed on a very small number of compute cores [32], prevail.

The use of higher spatial resolutions and more flexible tetrahedral or hybrid finite element meshes is also driven by the needs of clinical modeling applications. Generation pipelines for individualized EM models rely increasingly on reconstruction from tomographic imaging [112] which provide an ever increasing level of anatomical detail [68]. Multimodal clinical imaging also facilitates more accurate tissue classification. Clinically important delineation of viable myocardium, fat deposits, cleft spaces and vascularization, as well as substrate abnormalities such as infarct scars [104] or the presence of fibrosis [76, 77] is becoming feasible. These are known to be important factors influencing cardiac deformation, however, the spatial resolutions and methodology currently used impede their elucidation in modeling studies.

Spatial resolution also becomes an important factor when considering bidirectionally coupled, anatomically accurate EM models. A bidirectional link between EP and mechanics demands information being passed back and forth between EP and mechanics solver components. With tomographically reconstructed models the bidirectional projection of data may constitute a significant technical challenge when markedly different spatio-temporal resolutions are used. Either one resorts to using simplified, stylized anatomical representations for both physics to ensure an exact overlap of both domains, a refined electrical mesh is derived from spatial refinement of finite elements forming the mechanical mesh [15], or one opts to use the same high resolution mesh for both physics and accepts the increase in computational load. The advantage of the latter approach is that convergence studies with bidirectionally coupled EM models are feasible as discretization errors are governed by the spatio-temporal discretization of the EP problem. When attempting to build human whole organ EM models the use of high resolution meshes becomes particularly challenging due to the larger size of a human heart compared to other species used in many previous EM modeling studies [83], and the overall increase in DOF needed for discretizing both ventricles as well as the thin-walled atria [11].

This study reports on the development of a novel methodology for modeling of human whole organ cardiac electromechanics at a high para-cellular resolution. A clinical magnetic resonance (MR) scan was segmented to generate a patient-specific, anatomically accurate whole heart EM model. Tetrahedral finite element meshes were generated from the segmented image stack at resolutions of 220 μm, 440 μm and 880 μm, resulting in meshes with approximately 184.6, 24.4 and 3.7 million tetrahedral elements and 95.9, 13.2 and 2.1 million displacement DOF, respectively. The same mesh was used for discretizing the governing equations of both electrophysiology (EP) and nonlinear elasticity. A novel algebraic multigrid (AMG) preconditioner for an iterative Krylov solver was developed to deal with the resulting computational load. The AMG preconditioner was designed under the primary objective of achieving favorable strong scaling characteristics for both setup and solution runtimes, as this is key for exploiting current high performance computing hardware.

Strong scaling benchmarks using the 220 μm, 440 μm and 880 μm meshes with 95.5, 13.2 and 2.1 million displacement DOF demonstrated efficient scaling up to 8192, 4096 and 1024 compute cores which allowed the simulation of a single heart beat in 235.3, 87.8 and 44.3 minutes, respectively. The efficiency of the method allows fast simulation cycles without compromising in terms of anatomical or biophysical detail, thus enabling modeling studies which elucidate the role of fine scale anatomical features, substrate heterogeneities or mechanistic inquiries into bidirectional feedback loops governing cardiac adaptation processes at a fundamental level. Moreover, meaningful verification of EM models becomes feasible as spatial resolutions can be refined sufficiently to gauge convergence errors.

2. Governing Equations

2.1. Mechanical Model

Cardiac tissue is mechanically characterized as a hyperelastic, nearly incompressible, orthotropic material with a nonlinear stress-strain relationship. The deformation gradient F describes the deformation U of a body from the reference configuration Ω₀(X) to the current configuration Ω_t(x),

$$F_{\mathit{ij}}=\frac{\partial x_i}{\partial X_j},i,j=1,2,3.$$ (1)

By convention, we denote J = det F > 0 and introduce the right Cauchy–Green tensor C = F^⊺F. The nearly incompressible behavior is modeled by a multiplicative decomposition of the deformation gradient, see, e.g., [31], of the form

$$\mathbf{F}=J^{1∕3}\overline{\mathbf{F}},\mathbf{C}=J^{2∕3}\overline{\mathbf{C}},\text{with}\text{det}\overline{\mathbf{F}}=\text{det}\overline{\mathbf{C}}=1,$$ (2)

where J^1/3 and J^2/3 are associated with volumetric deformations, while $\overline{\mathbf{F}}$ and $\overline{\mathbf{C}}$ are associated with isochoric processes. Mechanical deformation is governed by the stationary equilibrium equations given as

$$-\text{Div}\left[\mathbf{FS}\right(\mathbf{U},\mathbf{X}\left)\right]={\mathbf{b}}_0\left(\mathbf{X}\right)\text{for}\mathbf{X}\in {\Omega }_0,$$ (3)

where U(X, t) is the unknown deformation, S(U, X) is the second Piola–Kirchhoff stress tensor, b₀(X) are the body forces and Div denotes the divergence operator in the reference configuration.

On the boundary $\partial {\Omega }_0={\overline{\Gamma }}_{0,\mathrm{D}}\cup {\overline{\Gamma }}_{0,\mathrm{N}}$ Dirichlet displacement boundary conditions are imposed

$$\mathbf{U}\left(\mathbf{X}\right)={\mathbf{U}}_{\mathrm{D}}\left(\mathbf{X}\right)\text{on}{\Gamma }_{0,\mathrm{D}},$$ (4)

with U_D(X) being a prescribed displacement, and Neumann force boundary conditions

$$\mathbf{FS}(\mathbf{U},\mathbf{X}){\mathbf{N}}_0\left(\mathbf{X}\right)={\mathbf{G}}_0\left(\mathbf{X}\right)\text{on}{\Gamma }_{0,\mathrm{N}},$$ (5)

where N₀(X) is the exterior normal vector and G₀(X) denotes a prescribed surface traction.

The deformation of cardiac tissue is governed by imposed external loads such as intracavitary and pericardial pressures, and active stresses intrinsically generated during contraction. The total stress S is additively decomposed according to

$$\mathbf{S}={\mathbf{S}}_{\mathrm{p}}+{\mathbf{S}}_{\mathrm{a}},$$ (6)

where S_p and S_a refer to the passive and active stresses, respectively. Passive stresses are modeled based on the constitutive equation

$${\mathbf{S}}_p=2\frac{\partial \Psi \left(\mathbf{C}\right)}{\partial \mathbf{C}},$$ (7)

where Ψ is an orthotropic, invariant-based strain-energy function [49, 30], which results in a frame-independent stress tensor. The strain-energy function Ψ is additively composed of three functions

$$\Psi \left(\mathbf{C}\right)={\Psi }_{\mathrm{vol}}\left(J\right)+{\overline{\Psi }}_{\mathrm{iso}}\left(\overline{\mathbf{C}}\right)+{\overline{\Psi }}_{\text{aniso}}(\overline{\mathbf{C}},{\mathbf{f}}_0,{\mathbf{s}}_0),$$ (8)

where Ψ_vol(J) is the volumetric contribution to the hyperelastic response while ${\overline{\Psi }}_{\text{iso}}$ and ${\overline{\Psi }}_{\text{aniso}}$ are two volume-preserving parts which relate to the isotropic and an anisotropic behavior, respectively. The prevailing myocyte orientation, referred to as the fiber axis, is denoted as f₀ and the sheet axis, which is perpendicular to f₀ and parallel to the collagen layers which envelop bundles of myocytes, is s₀. Together with the sheet-normal axis n₀, defined to be orthogonal to the sheet and the fiber orientations this forms a right-handed orthonormal set of basis vectors [49]. The volumetric contribution is given as

$${\Psi }_{\mathrm{vol}}\left(J\right)=\frac{\kappa }{2}\ln {\left(J\right)}^2,$$ (9)

with k > 0 being a penalty parameter for enforcing the nearly incompressible behavior of the tissue. The volume-preserving isotropic part is based upon an exponential model [25]

$${\Psi }_{\mathrm{iso}}=\frac{a}{2b}\exp \left[b\right({\overline{I}}_1-3\left)\right],{\overline{I}}_1=\mathrm{tr}\left(\overline{\mathbf{C}}\right),$$ (10)

where a > 0 is a stress-like and b > 0 a dimensionless material parameter, and the anisotropic function is described by ${\overline{\Psi }}_{\text{aniso}}={\overline{\Psi }}_{\mathrm{f}}+{\overline{\Psi }}_{\mathrm{s}}+{\overline{\Psi }}_{\mathrm{fs}}$ and

$${\overline{\Psi }}_{\mathrm{f}}(\overline{\mathbf{C}},{\mathbf{f}}_0)=\frac{a_{\mathrm{f}}}{2b_{\mathrm{f}}}\left\{\exp \left[b_{\mathrm{f}}{({\overline{I}}_{\mathrm{f}}-1)}^2\right]-1\right\},$$ (11)

$${\overline{\Psi }}_{\mathrm{s}}(\overline{\mathbf{C}},{\mathbf{s}}_0)=\frac{a_{\mathrm{s}}}{2b_{\mathrm{s}}}\left\{\exp \left[b_{\mathrm{s}}{({\overline{I}}_{\mathrm{s}}-1)}^2\right]-1\right\},$$ (12)

$${\overline{\Psi }}_{\mathrm{fs}}(\overline{\mathbf{C}},{\mathbf{f}}_0,{\mathbf{s}}_0)=\frac{a_{\mathrm{fs}}}{2b_{\mathrm{fs}}}\left\{\exp \left[b_{\mathrm{fs}}{\overline{I}}_{\mathrm{fs}}^2\right]-1\right\},$$ (13)

with the invariants ${\overline{I}}_{\mathrm{f}}≔{\mathbf{f}}_0\cdot \overline{\mathbf{C}}{\mathbf{f}}_0,{\overline{I}}_{\mathrm{s}}≔{\mathbf{s}}_0\cdot \overline{\mathbf{C}}{\mathbf{s}}_0$ and ${\overline{I}}_{\mathrm{fs}}≔{\mathbf{f}}_0\cdot \overline{\mathbf{C}}{\mathbf{s}}_0$. All parameters, the stress-like a_f, a_s, a_fs as well as the dimensionless b_f, b_s, b_fs are considered to be fs positive. Moreover, the anisotropic responses (11) and (12) only contribute for ${\overline{I}}_{\mathrm{f}}>1$ or ${\overline{I}}_{\mathrm{s}}>1$, respectively. Note that in the anisotropic model (11–13) sheet-normal properties and corresponding shear terms are omitted as they are either expressible in terms of other invariants or were shown to be of minor significance in shear tests, see [49, §5].

Stresses due to active contraction are assumed to act along the fiber orientation f₀ in the reference configuration. Thus, the active Cauchy stress can be defined as

$${\mathbf{S}}_{\mathrm{a}}=S_{\mathrm{a}}{({\mathbf{f}}_0\cdot {\mathbf{Cf}}_0)}^{-1}{\mathbf{f}}_0\otimes {\mathbf{f}}_0,$$ (14)

where S_a is the scalar active stress induced in fiber direction f₀, see Section 2.3.

2.2. Electrophysiology Model

The spread of electrical activation and repolarization is described by the bidomain equations. Cast in the elliptic-parabolic form, these are given in material coordinates by

$$-\text{Div}\left[J{\mathbf{F}}^{-1}\right({\sigma }_{\mathrm{i}}+{\sigma }_{\mathrm{e}}){\mathbf{F}}^{-⊺}\nabla {\varphi }_{\mathrm{e}}]-\text{Div}[J{\mathbf{F}}^{-1}{\sigma }_{\mathrm{i}}{\mathbf{F}}^{-⊺}\nabla V_{\mathrm{m}}]=0,$$ (15)

$$-\text{Div}[J{\mathbf{F}}^{-1}{\sigma }_{\mathrm{i}}{\mathbf{F}}^{-⊺}\nabla V_{\mathrm{m}}]-\text{Div}[J{\mathbf{F}}^{-1}{\sigma }_{\mathrm{i}}{\mathbf{F}}^{-⊺}\nabla {\varphi }_{\mathrm{e}}]=-\beta I_{\mathrm{m}}+\beta I_{\mathrm{i}},$$ (16)

$$\frac{\partial \eta }{\partial t}=\mathit{g}(V_{\mathrm{m}},\eta ,\lambda ),$$ (17)

where σ_i and σ_e are the intracellular and extracellular conductivity tensors, respectively;

$$V_{\mathrm{m}}={\varphi }_{\mathrm{i}}-{\varphi }_{\mathrm{e}}$$ (18)

is the transmembrane voltage, with ϕ_i and ϕ_e being the intracellular and extracellular potentials, respectively; β is the bidomain surface to volume ratio;

$$I_{\mathrm{m}}=C_{\mathrm{m}}\frac{\partial V_{\mathrm{m}}}{\partial t}+I_{\mathrm{ion}}(V_{\mathrm{m}},\eta ,\lambda )$$ (19)

is the transmembrane current density; C_m is the membrane capacitance per unit area; I_ion is the membrane ionic current density, which depends on V_m, a set of state variables η, and the stretch ratio $\lambda ={\mathbf{f}}_0^{⊺}{\mathbf{Cf}}_0$, defined as the ratio between current muscle length and reference length in the unloaded state measured along the fiber direction; I_i is an intracellular current density, specified on a per membrane area basis, which serves as a stimulus to initiate propagation; and ∇ is the gradient operator in the reference configuration.

A computationally less costly monodomain model can be derived from the bidomain model under the assumption that intracellular and interstitial conductivity tensors can be related by σ_i = ασ_e. In this scenario Eq. (15) can be neglected and the intracellular conductivity tensor in Eq. (16) is replaced by the harmonic mean tensor

$${\sigma }_{\mathrm{m}}={\sigma }_{\mathrm{i}}{\sigma }_{\mathrm{e}}{({\sigma }_{\mathrm{i}}+{\sigma }_{\mathrm{e}})}^{-1}.$$

That is, the monodomain equation is given by

$$-\text{Div}[J{\mathbf{F}}^{-1}{\sigma }_{\mathrm{m}}{\mathbf{F}}^{-⊺}\nabla V_{\mathrm{m}}]=-\beta I_{\mathrm{m}}+\beta I_{\mathrm{tr}},$$ (20)

where I_tr is a transmembrane stimulus current.

At tissue boundaries, no-flux boundary conditions are imposed on ϕ_i and ϕ_e, i.e.,

$${\int }_{\partial \Omega }{\sigma }_{\mathrm{i}}\nabla {\varphi }_{\mathrm{i}}\mathrm{d}\Gamma ={\int }_{\partial \Omega }{\sigma }_{\mathrm{e}}\nabla {\varphi }_{\mathrm{e}}\mathrm{d}\Gamma =0$$ (21)

holds. Note that for the numerical examples in Section 6 any effects of deformation upon the distribution of potentials is neglected. Under the conditions considered in here, feedback mechanisms of deformation upon current flow are of minor importance as the period in time where the magnitude of gradients ∥σ_i∇ϕ_i∥ and ∥σ_e∇ϕ_e∥ is large, is temporally disjoint from deformation [90]. Thus, in Eqs. (15–17), F is replaced by the identity matrix I and J = det(F) is replaced by 1.

2.3. Active Stress Model

Most physiological models of active stress generation are based on sets of ordinary differential equations of the form

$$\frac{\partial \zeta }{\partial t}=\mathbf{h}(\zeta ,\eta ,\lambda ,\stackrel{.}{\lambda })=\mathbf{h}(\zeta ,\eta ,\mathbf{U},\stackrel{.}{\mathbf{U}})$$ (22)

$$S_{\mathrm{a}}=f(\zeta ,\eta ,\lambda ,\stackrel{.}{\lambda })=f(\zeta ,\eta ,\mathbf{U},\stackrel{.}{\mathbf{U}}),$$ (23)

where the cellular mechanisms of active stress generation are represented by the state vector ζ, which depends on the electrophysiological state of the cell η, stretch ratio λ and stretch rate $\stackrel{.}{\lambda }$.

The dependency on λ and $\stackrel{.}{\lambda }$ accounts for length and velocity dependence of active stress generation while the dependency on the electrophysiological state η, in particular on the cytosolic calcium concentration [Ca²⁺], models excitation-contraction coupling.

Conversely, deformation exerts influence over cellular EP via mechano-electric feedback (MEF) mechanisms which are mediated through the length dependence of η through Eq. (17), mainly via λ dependent modulation of troponin C binding affinity for calcium, the stretch dependent modulation of ion channel conductivity through Eq. (19), and changes in intracellular and interstitial current flow through Eqs. (15–16). If any of these MEF mechanisms is under study, coupling between EP and deformation is bidirectional. In such scenarios, often referred to as strong coupling, the two physics cannot be solved sequentially since deformation influences EP. Hence, they must be solved in a coupled fashion which can be achieved using a monolithic approach or, as used in this study, by updating the variables involved in the coupling when solving for deformation.

3. Finite Element Method for Cardiac Electromechanics

3.1. Temporal and Spatial Discretization

Temporal dynamics of cardiac EP and mechanical deformation are governed by very different time scales, suggesting that the use of different time steps for the two physics is advantageous to obtain a computationally efficient numerical scheme. Even when considering the EP problem alone, the time scales involved differ significantly between reaction and diffusion terms. Fast transients such as the upstroke of the action potential are governed by time constants in the μs range whereas slower processes occur at the order of tens or hundreds of μs up to ms. Mechanical deformation can be assumed to occur at similar time scales as slower EP processes such as the cytosolic calcium transients. As mass momentum is not considered here, there is no explicit time dependence in the stationary equilibrium equation (3). Efficient numerical schemes aim to exploit these differences in temporal resolution by using appropriate update intervals [93].

In this study, we conform to the following conventions. Time is discretized according to t^k = k · Δt where Δt is considered to be a global synchronization clock. Time steps used in the individual subproblems are multiples n of Δt with $n\in \mathbb{N}$. In our numerical scheme, three different time steps are used: Δt_ode, to discretize the systems of ODEs in Eq. (17) and Eq. (22) representing EP reaction terms and active stress generation; Δt_par, to discretize the parabolic diffusion terms given by the system of PDEs in Eq. (16); and Δt_mech, to update the current configuration given by the PDE in Eq. (3). Unless otherwise noted, throughout this paper we use Δt_ode = Δt_par = Δt and Δt_mech = n · Δt. Time is discretized by t^k = k · Δt for solving the bidomain equations and t^ℓ = ℓ · Δt_mech for referring to the time-varying states of deformation. The instants t^k = k and t^ℓ are equal for k = n · ℓ.

A global time step of Δt = 5 μs is used for time discretization of the bidomain equations while a larger time step of Δt_mech = 1 ms is used to solve the mechanical model, owing to its slower dynamics. Thus, the system of ODEs (17) and (22) as well as elliptic PDE (38) and parabolic PDE (42) are integrated in n time steps of size Δt from $t_{\mathrm{m}}^{\ell -1}$ to $t_{\mathrm{m}}^{\ell }$ using λ and $\stackrel{.}{\lambda }$ derived from U and $\stackrel{.}{\mathbf{U}}$ at time t^ℓ–1.

We employ piecewise continuous linear ansatz functions for both the electric potentials in Eqs. (38–42) as well as the displacements in Eq. (32). To account for the near incompressibility condition (9) the idea of element-by-element static condensation is applied [50, 105]. As we use tetrahedral elements for domain tessellation, this leads to P₁ – P₀ elements for the nonlinear deformation problem. Although this type of element is prone to volumetric and shear locking, as discussed in the limitations section 7.5, it is commonly used for cardiac electromechanics applications [5, 102]. The same spatial discretization is used for both physics (see Fig. 4). Hence, the projection of coupling variables between electrical and mechanical grid is avoided.

Figure 4.

The CM (left), MM (middle) and FM (right) models of a four chamber heart geometry, discretized with average resolutions of 880 μm, 440 μm and 220 μm respectively. Insets illustrate level of detail in geometry resolution and smoothness of organ surfaces.

3.2. Variational Formulation and Discretization of Nonlinear Elasticity Problems

In the numerical examples considered in this study no Neumann boundary conditions were applied (see Section 5.3), and body forces b₀ were neglected. We introduce w = [η; ζ]^⊺ which is the combined electrophysiological (17) and myofilament (22) state vector and q = [g; h]^⊺ which is the respective right-hand side term. Then, the boundary value problem (3–5) combined with the cellular EP model (17) and the myofilament model (22) is formally equivalent to the equations

$$\frac{\mathrm{d}\mathbf{w}}{\mathrm{d}t}=\mathbf{q}(\mathbf{w},V_{\mathrm{m}},\mathbf{U}),$$ (24)

$${\langle {\mathcal{A}}_0\left(\mathbf{U}\right),\mathbf{V}\rangle }_{{\Omega }_0}=0,$$ (25)

which is valid for all smooth enough vector fields V vanishing on the Dirichlet boundary Γ_0,D [20, Theorem 2.6-1]. The nonlinear operator ${\mathcal{A}}_0$ is induced by the stress tensor representation (6) and ⟨·, ·⟩ is the related duality pairing. The left hand side of the variational equation (25) has the physical interpretation of the rate of internal mechanical work and is given by

$${\langle {\mathcal{A}}_0\left(\mathbf{U}\right),\mathbf{V}\rangle }_{{\Omega }_0}≔{\int }_{{\Omega }_0}\mathbf{S}\left(\mathbf{w}\right(\mathbf{U}),\mathbf{U}):\Sigma (\mathbf{U},\mathbf{V})\mathrm{d}\mathbf{X},$$ (26)

with the second Piola–Kirchhoff stress tensor

$$\mathbf{S}\left(\mathbf{w}\right(\mathbf{U}),\mathbf{U})={\mathbf{S}}_{\mathrm{p}}\left(\mathbf{U}\right)+{\mathbf{S}}_{\mathrm{a}}\left(\mathbf{w}\right(\mathbf{U}),\mathbf{U})$$ (27)

and Σ(U, V) is the directional derivative of the Green–Lagrange strain tensor, see [48]. Choosing a time discretization as described in Section 3.1 we compute the stress tensor (27) at time step ℓ using

$$\mathbf{S}\left({\mathbf{w}}^{\ast }\right({\mathbf{U}}^{\ell }),{\mathbf{U}}^{\ell })={\mathbf{S}}_{\mathrm{p}}\left({\mathbf{U}}^{\ell }\right)+{\mathbf{S}}_{\mathrm{a}}\left({\mathbf{w}}^{\ast }\right({\mathbf{U}}^{\ell }),{\mathbf{U}}^{\ell })$$ (28)

where U^ℓ is the displacement at time step ℓ and w* (U^ℓ) is an approximation of w^ℓ which is computed similarly to [42] using a forward Euler predictor step as

$${\mathbf{w}}^{\ast }\left({\mathbf{U}}^{\ell }\right)={\mathbf{w}}^{\alpha }+\mathbf{q}({\mathbf{w}}^{\alpha },t^{\alpha },{\mathbf{U}}^{\ell })\Delta t$$

where t^α is the previous electrical time step, i.e.

$$t^{\alpha }=\ell \Delta t_{\text{mech}}-\Delta t=(\ell \cdot n-1)\Delta t.$$

That means that a prediction is repeatedly computed from the previous electrical time step, which is then corrected in each iteration of the Newton loop to account for the feedback of deformation upon the EP state of the tissue.

To apply the finite element method (FEM) we consider an admissible decomposition of the computational domain $\Omega \subset {\mathbb{R}}^3$ into M tetrahedral elements and introduce a conformal finite element space

$$X_h\subset H^1\left({\Omega }_0\right),N=\text{dim}{X}_h$$

of piecewise polynomial continuous basis functions φ_i. Using Newton’s method and a Galerkin finite element discretization results in solving the nonlinear discretized system to find δU_h ϵ [X_h]³, δU_h = 0 on Γ_0,D such that

$${\langle \delta {\mathbf{U}}_h,{\mathcal{A}}_0^{\prime }\left({\mathbf{U}}_{\nu ,h}^{\ell }\right){\mathbf{V}}_h\rangle }_{{\Omega }_0}=-{\langle {\mathcal{A}}_0\left({\mathbf{U}}_{\nu ,h}^{\ell }\right),{\mathbf{V}}_h\rangle }_{{\Omega }_0},$$ (29)

$${\mathbf{U}}_{\nu +1,h}^{\ell }={\mathbf{U}}_{\nu ,h}^{\ell }+\delta {\mathbf{U}}_h$$ (30)

holds for all V_h ϵ [X_h]³, V_h = 0 on Γ_0,D at time step ℓ and ν is the index of the Newton iteration. The left hand side of (29), omitting (·)_h to increase readability, is computed by

$${\langle \delta \mathbf{U},A_0^{\prime }\left({\mathbf{U}}_{\nu }^{\ell }\right)\mathbf{V}\rangle }_{{\Omega }_0}={\int }_{{\Omega }_0}\nabla \delta \mathbf{U}\mathbf{S}\left({\mathbf{U}}_{\nu }^{\ell }\right):\nabla \mathbf{V}\mathrm{d}\mathbf{X}+{\int }_{{\Omega }_0}{\mathbf{F}}^{⊺}\nabla \delta \mathbf{U}:\mathbb{C}\left({\mathbf{U}}_{\nu }^{\ell }\right):\nabla \mathbf{V}\mathrm{d}\mathbf{X}.$$ (31)

For more details see [2, 3, 48]. The standard finite element method now yields a linear system of equations which is equivalent to the discretized variational formulation (29–30). Finally, we have to solve

$${\mathbf{K}}^{\prime }\left({\underset{¯}{U}}_{\nu }^{\ell }\right)\delta \underset{¯}{U}=-\underset{¯}{K}\left({\underset{¯}{U}}_{\nu }^{\ell }\right),{\underset{¯}{\mathbf{U}}}_{\nu +1}^{\ell }={\underset{¯}{U}}_{\nu }^{\ell }+\delta \underset{¯}{U},$$ (32)

with the solution vector ${\underset{¯}{U}}_{\nu }^{\ell }\in {\mathbb{R}}^{3N}$ at the ν–th Newton step and the increment $\delta \underset{¯}{U}\in {\mathbb{R}}^{3N}$. The tangent stiffness matrix ${\mathbf{K}}^{\prime }\in {\mathbb{R}}^{3N\times 3N}$ is calculated according to

$${\mathbf{K}}^{\prime }\left({\underset{¯}{U}}_{\nu }^{\ell }\right)[j,i]≔{\langle {\phi }_i,{\mathcal{A}}_0^{\prime }\left({\mathbf{U}}_{\nu ,h}^{\ell }\right){\phi }_j\rangle }_{{\Omega }_0},$$ (33)

and the terms of the right hand side are constructed by

$$\underset{¯}{K}\left({\underset{¯}{U}}_{\nu }^{\ell }\right)\left[i\right]≔{\langle {\mathcal{A}}_0\left({\mathbf{U}}_{\nu ,h}^{\ell }\right),{\phi }_i\rangle }_{{\Omega }_0}.$$ (34)

Because of symmetry properties of the stress tensor S and the elasticity tensor $\mathbb{C}$, the operator ${\mathcal{A}}_0^{\prime }\left({\mathbf{U}}_{\nu ,h}^{\ell }\right)$ is self-adjoint and thus the tangent stiffness matrix K′ is symmetric. On the other hand, we can not guarantee that ${\mathcal{A}}_0^{\prime }\left({\mathbf{U}}_{\nu ,h}^{\ell }\right)$ is always an elliptic operator, see, e.g., [75]. Nevertheless, we use the conjugate gradient (CG) method as the preferred Krylov solver, and in the case that ellipticity fails and the CG method is not converging we switch to the more robust GMRES method.

3.3. Variational Formulation and Discretization of the Bidomain Equation

The corresponding weak form to the elliptic-parabolic system (15–17) is to find sufficiently smooth (V_m, ϕ_e, η) such that

$${\int }_{{\Omega }_0}({\sigma }_{\mathrm{i}}+{\sigma }_{\mathrm{e}})\nabla {\varphi }_{\mathrm{e}}\nabla q_1\mathrm{d}\mathbf{X}+{\int }_{{\Omega }_0}{\sigma }_{\mathrm{i}}\nabla V_{\mathrm{m}}\nabla q_1\mathrm{d}\mathbf{X}=0$$ (35)

for all q₁ ϵ H¹(Ω₀) with ∫_Ω0 q1 (X) dX = 0,

$${\int }_{{\Omega }_0}\beta C_{\mathrm{m}}\frac{\partial V_{\mathrm{m}}}{\partial t}{q}_2\mathrm{d}\mathbf{X}+{\int }_{{\Omega }_0}{\sigma }_{\mathrm{i}}(\nabla V_{\mathrm{m}}+\nabla {\varphi }_{\mathrm{e}})\nabla q_2\mathrm{d}\mathbf{X}+{\int }_{{\Omega }_0}\beta I_{\mathrm{ion}}(V_{\mathrm{m}},\eta )q_2\mathrm{d}\mathbf{X}=0$$ (36)

for all q₂ ϵ H¹ (Ω₀) and

$${\int }_{{\Omega }_0}\frac{\partial \eta }{\partial t}{q}_3\mathrm{d}\mathbf{X}+{\int }_{{\Omega }_0}\mathit{g}(V_{\mathrm{m}},\eta ,\lambda )q_3\mathrm{d}\mathbf{X}=0,$$ (37)

for all q₃ ϵ L² (Ω₀).

For more details including regularity assumptions and the proof of existence and uniqueness of a solution of the weak system (35–37), we refer to [65].

To reduce the complexity of the problem, the elliptic (35) and parabolic portions (36) of the bidomain equations are decoupled and solved sequentially [119]. First, the elliptic system (35) is solved, then an operator splitting approach is employed to subsequently solve the parabolic system [107, 97, 109]. This allows us to treat reaction and diffusion terms separately. The reaction term represented by the ODE system (37) is solved using an accelerated Rush–Larsen technique, see [93], and the diffusion problem is solved using a Crank–Nicolson scheme. A Galerkin finite element approach and a time discretization as described in Section 3.1 with t^k = kΔt yields the following linear system of equations to obtain $({\underset{¯}{\varphi }}_{\mathrm{e}}^{k+1},{\underset{¯}{V}}_{\mathrm{m}}^{k+1},{\underset{¯}{\eta }}^{k+1})\in {\mathbb{R}}^N\times {\mathbb{R}}^N\times {\mathbb{R}}^N$ such that

$${\mathbf{K}}_{\mathrm{i}+\mathrm{e}}{\underset{¯}{\varphi }}_{\mathrm{e}}^{k+1}=-{\mathbf{K}}_{\mathrm{i}}{\underset{¯}{V}}_{\mathrm{m}}^k,$$ (38)

$${\underset{¯}{\eta }}_{\mathrm{f}}^{k+1}={\underset{¯}{\eta }}_{\mathrm{f}}^k{e}^{-\frac{\Delta t}{\tau }}+{\underset{¯}{\eta }}_{\mathrm{f}\infty }\left(1-e^{-\frac{\Delta t}{\tau }}\right)$$ (39)

$${\underset{¯}{\eta }}_{\mathrm{s}}^{k+1}={\underset{¯}{\eta }}_{\mathrm{s}}^k+\underset{¯}{g}(V_{\mathrm{m}}^{k^{\ast }},{\underset{¯}{\eta }}_{\mathrm{s}}^k,{\lambda }^{\ell -1})\Delta t$$ (40)

$${\underset{¯}{V}}_{\mathrm{m}}^{k+\frac{1}{2}}={\underset{¯}{V}}_{\mathrm{m}}^k-\frac{\Delta t}{C_{\mathrm{m}}}{I}_{\mathrm{ion}}({\underset{¯}{V}}_{\mathrm{m}}^k,{\underset{¯}{\eta }}^{k+1},{\underset{¯}{\lambda }}^{\ell -1}),$$ (41)

$$({\tilde{\mathbf{M}}}_{\mathrm{i}}+\frac{1}{2}{\mathbf{K}}_{\mathrm{i}}){\underset{¯}{V}}_{\mathrm{m}}^{k+1}=-{\mathbf{K}}_{\mathrm{i}}\left(\frac{1}{2}{\underset{¯}{V}}_{\mathrm{m}}^{k+\frac{1}{2}}+{\underset{¯}{\varphi }}_{\mathrm{e}}^k\right)+{\tilde{\mathbf{M}}}_{\mathrm{i}}{\underset{¯}{V}}_{\mathrm{m}}^{k+\frac{1}{2}},$$ (42)

where

$${\mathbf{K}}_{\mathrm{i}+\mathrm{e}}[j,i]≔{\int }_{{\Omega }_0}({\sigma }_{\mathrm{i}}+{\sigma }_{\mathrm{e}})\nabla {\phi }_i\nabla {\phi }_j\mathrm{d}\mathbf{X},$$ (43)

$${\mathbf{K}}_{\mathrm{i}}[j,i]≔{\int }_{{\Omega }_0}{\sigma }_{\mathrm{i}}\nabla {\phi }_i\nabla {\phi }_j\mathrm{d}\mathbf{X},$$ (44)

$${\tilde{\mathbf{M}}}_{\mathrm{i}}[j,i]≔\frac{\beta C_{\mathrm{m}}}{\Delta t}{\int }_{{\Omega }_0}{\phi }_i{\phi }_j\mathrm{d}\mathbf{X},$$ (45)

η = {η_f, η_s} with η_f being fast acting gating variables solved for by evaluating an analytical solution, τ and η_f∞ are functions of the rate coefficients which govern channel gating kinetics, and η_s are slower acting states, integrated with a forward Euler method [93]. From this construction and ${\sigma }_{\mathrm{i}}={\sigma }_{\mathrm{i}}^{⊺},{\sigma }_{\mathrm{e}}={\sigma }_{\mathrm{e}}^{⊺}$ we conclude that the matrices (43–45) are symmetric and positive definite. The CG method is therefore applicable to solve the linear systems of equations (38) and (42).

4. Linear Solvers

All linear solvers used in this study were implemented in the publicly available numerical package Parallel Toolbox (pt) [72, 44, 73]. The cardiac bidomain equations and the nonlinear deformation equation were solved in parallel using the Cardiac Arrhythmia Research Package (CARP) [118] as an outer framework which was interfaced with the pt library. Technical aspects have been described elsewhere [80]. CARP makes use of the MPI based library PETSc [7] as the basic infrastructure for handling parallel matrices and vectors. Parallel partitioning of unstructured grids relies upon the graph based domain decomposition ParMetis [54, 55].

Details on the methods employed for solving the bidomain equations in parallel have been reported previously [92, 119, 80]. Briefly, the elliptic portion of the bidomain equations is solved using the algebraic multigrid preconditioner ptAMG [72] with an iterative CG solver method (ptAMG-PCG). The parabolic portion of the bidomain equation is solved using an w-Jacobi preconditioner, also in combination with the iterative CG solver (wJ-PCG), using an w of 2/3 and two iterations per CG iteration [80]. The ptAMG-PCG method used for solving the scalar elliptic PDE turned out to be of very limited efficiency for solving the nonlinear deformation problem, thus motivating an adaptation of the AMG preconditioner for nonlinear biomechanical applications. The Krylov solver of choice remained the CG method. In the case that ellipticity fails, we switch to the GMRES method (see section 3.2). An extensive description of the underlying concepts is presented in the following section.

Other AMG implementations such as hypre/BoomerAMG [45] or ML [33] are suitable for the electrical and mechanical problems described in this paper. However, using these packages, we were not able to to match the overall performance of our own AMG implementation. This is not surprising as those packages are designed to accommodate a wide range of applications, whereas our own much simpler implementation is tuned for a small set of applications in cardiac modeling.

4.1. Algebraic Multigrid Design

For a detailed introduction to the AMG method we refer to [115, 108]. The initial design of the ptAMG preconditioner was geared towards solving linear systems arising from the discretization of scalar PDEs such as the elliptic PDE embedded in the bidomain equations. To be suitable for systems of equations where each node in a finite element mesh is associated with more than one DOF, the AMG setup had to be altered.

We assume an equation system Ku = b with K ∈ ${\mathbb{R}}^{3N\times 3N}$ and $\underset{¯}{u},\underset{¯}{b}\in {\mathbb{R}}^{3N}$ as in (32). The total number of unknowns is 3N where N represents the number of finite element grid points and 3 is the number of physical unknowns per grid point. Classical multigrid coarsening methods would compute the coarse grid selection based on the discrete operator K, taking into account the matrix graph alone [9], or additionally also the magnitude of matrix entries [103]. However, in our application, this would lead to different coarse grids for the physical unknowns of one grid point which can translate into bad convergence behavior.

As in [39], this problem is circumvented by computing an auxiliary matrix $\tilde{\mathbf{K}}\in {\mathbb{R}}^{N\times N}$, defined as

$$\tilde{\mathbf{K}}[i,j]≔{\Vert K_{\mathit{ij}}\Vert }_F$$ (46)

where $K_{\mathit{ij}}\in {\mathbb{R}}^{3\times 3}$ denotes the subblock of K which couples the DOF associated with a finite element grid point i with the DOF of the grid point j, and ∥ · ∥_F is the Frobenius matrix norm. A coarsening scheme inspired by the Ruge-Stüben scheme [103] is applied to the auxiliary matrix using

$$\tilde{\mathbf{K}}[i,j]>ϵ\tilde{\mathbf{K}}[i,i]$$ (47)

as strong connection criterion for the edge (i, j). The set of FE grid point indices T := {0, … , N – 1} is split into the disjoint subsets of coarse grid C ⊂ T and fine grid indices F ⊂ T. The prolongation operator P given by

$$\mathbf{P}≔\left(\begin{array}{c}\hfill {\mathbf{I}}_{\mathrm{CC}}\hfill \\ \hfill {\mathbf{P}}_{\mathrm{FC}}\hfill \end{array}\right)$$ (48)

interpolates from C onto T. The coarse grid C ⊂ T can be mapped directly via the identity I_CC while the fine grid F ⊂ T needs to be interpolated from the coarse grid via P_FC. The interpolation weight n_i is computed as the cardinality of the set of all coarse grid points interpolating any i ϵ F :

$$n_i≔\mid \{j\in C:\tilde{\mathbf{K}}[i,j]>ϵ\tilde{\mathbf{K}}[i,i\left]\right\}\mid$$ (49)

Then, the coarse to fine interpolation P_FC is defined as

$${\left({\mathbf{P}}_{\mathrm{FC}}\right)}_{\mathit{ij}}≔\{\begin{array}{cc}(1∕n_i)I_{3\times 3}\hfill & \text{if}\tilde{\mathbf{K}}[i,j]>ϵ\tilde{\mathbf{K}}[i,i]\hfill \\ 0\hfill & \text{otherwise}\hfill \end{array}\phantom{\}}$$ (50)

with i ϵ F and j ϵ C. Finally, the coarse grid operator K_c is defined as the triple matrix product

$${\mathbf{K}}_{\mathrm{c}}≔{\mathbf{P}}^{⊺}\mathbf{KP}.$$ (51)

By default, the multigrid coarsening in ptAMG is continued until no strong connections between the remaining unknowns are found. The inverse of the resulting coarse grid system can therefore be approximated by the inverse 3 × 3 block diagonal. Alternatively, the coarsening process can be configured to finish earlier at a finer level. The resulting coarse grid system needs to be inverted by a direct solver. While using a direct coarse-grid solver can greatly reduce the number of solver iterations, the overall setup and solving time was consistently worse in our numerical benchmarks. One pre- and post-smoothing iteration with a Jacobi-smoother is used in a multigrid V-cycle as a preconditioner step. The inverse 3 × 3 block diagonal of the system matrix is used as the approximate inverse in the Jacobi iteration.

4.2. Strongly Scalable Parallelization

Devising iterative solvers and suitable preconditioners with favorable strong scaling properties, i.e., where execution time reduces when the number of compute cores used increases, is of pivotal importance when attempting to model cardiac electromechanics at high spatial resolutions. Departing from an initial implementation of the ptAMG method described above, strong scaling experiments were performed, in which an entire single heart beat was simulated with a varying number of processors. Parallel performance and scalability was gauged based on the following performance metrics:

1. Real time lag factor ξ_r: Execution time divided by the time span simulated.

2. Speedup S: The reduction in execution time achieved when using N_p parallel processes relative to a reference run using N_pr processes

   $$S=\frac{T_{\mathrm{pr}}}{T_{\mathrm{p}}}.$$ (52)
3. Parallel efficiency E: The algorithm’s ability to reduce execution time as N_p increases, defined as the ratio of the fractional reduction in execution time to the fractional increase in N_p

   $$E=\frac{T_{\mathrm{p}}}{T_{\mathrm{pr}}}.\frac{N_{\mathrm{pr}}}{N_{\mathrm{p}}},$$ (53)

   where T_p refers to the execution time when using N_p processes, and T_pr refers to the same quantity measured in a reference simulation using N_pr processes. An E value of 1 is referred to as linear scaling, because computation time is reduced by the same factor as N_p is increased. E values below 1 are known as sub-linear scaling whereas those above 1 are known as super-linear scaling.

Whenever parallel efficiency degraded, parallel profiling data was generated and analyzed using the Scalasca toolset [34] (Score-P, CUBE4, Scalasca) to identify the scaling bottlenecks. Based on these analyses, the following parallelization techniques were implemented:

1. Data exchange on the interfaces between individual subdomains was reworked to better load balance the cost of accumulation of interface data, as shown previously for solving a scalar elliptic PDE [74].

2. Parallel data layout of the multigrid hierarchy was modified to increase the effective network bandwidth of the data exchange across subdomain interfaces. Parallel profiling has shown that a combination of small average message sizes and dense communication patterns significantly decrease the effective network bandwidth on the intermediate multigrid levels. A strategy to counter this phenomenon is to redistribute the parallel data of the intermediate multigrid levels onto fewer active processes [6], thus reducing communication density and increasing average message size at the same time. The reasoning of this approach is, that the increase in effective network bandwidth outweighs the loss of computational units.

   The approach implemented in ptAMG is to systematically reduce the number of active processes on each multigrid level. A constant group size g > 1 is introduced. On any multigrid level, the range of active processes is split into groups of size g. One process of each group, declared to be the master, will be active on the next coarser multigrid level, while the other processes are declared idle. The idle processes redistribute their current coarse grid system to their group’s master, thus reducing the number of active processes on any multigrid level to 1/g of the previous level.

   The main advantage of this approach is that there is no need for tuning the data redistribution process to individual use cases (e.g. problem size, number of computational units). The optimal choice for g depends only on the rate by which the unknowns are reduced during coarsening. By default g is set to 2.

3. The overall AMG algorithm was split into steps which depend on the matrix sparsity pattern (and thus on the FE graph) and those which depend on the actual matrix entries. When solving Eq. (29), the system matrix is updated in each iteration of the Newton solver, but the sparsity structure remains unchanged as our method is not h-adaptive. Therefore only those setup steps which depend on the matrix values have to be executed.

5. Four Chamber Electromechanical Model

5.1. Generation of Whole Heart Anatomy Model

A processing pipeline consisting of multiple stages was implemented, incorporating segmentation, tissue classification, geometry smoothing and resampling, FE mesh generation and tag transfer, and the assignment of orthotropic eigenaxes. The development of the model generation pipeline aimed at achieving maximum flexibility, allowing the generation of complete four chamber models of the heart including the connected major vessels. Such models are preferable from a modeling point of view as they allow the application of more realistic mechanical boundary conditions [32].

5.1.1. Image Acquisition

The anatomical model was derived from an end diastolic 3D balanced steady state free precession (SSFP) cardiac magnetic resonance imaging acquisition in a sagittal orientation with whole-heart coverage and an isotropic resolution of 1.3 mm. The image was processed using automatic model based segmentation of the left and right ventricle chambers, left and right atrial chambers and the left ventricle myocardium wall [91].

5.1.2. Segmentation and Tissue Classification

As the thin walled atrial walls and the right ventricular wall could not be delineated directly from the image stacks, these walls were introduced by dilating the atrial blood pool by 2 voxels and a 4 voxel right ventricular wall was created by dilating the corresponding chamber [21]. Classification tags were added to further discriminate between different anatomical regions and tissue types relevant to the subsequent model parametrization. As there is no region-specific signature in the acquired imaging data which would allow for an automatic discrimination, a semi-automatic approach was implemented using various tools such as CAIPI (Fraunhofer, MEVIS) and itksnap (University of Pennsylvania, USA).

5.1.3. Anatomy Smoothing

Depending on the resolution of a given medical image stack, the geometric appearance of the heart’s surfaces in a generated mesh can be jagged, which is ideal neither for visual representation nor for computational accuracy. These staircasing artefacts were reduced by smoothing and upsampling the segmented image stack using a variational technique [47].

Briefly, a marching cubes algorithm, as implemented in the VTK library [120], was used to generate a triangulated surface representation of the heart’s anatomy. In an iterative variational optimization procedure, the generated surfaces were smoothed. The method avoided volumetric shrinking and limited the maximum nodal displacements to ±0.5 voxels. The smoothed surface mesh was resampled at an isotropic resolution of 110 μm using the 2D Cairo library [16] and stacked together to generate the smoothed 3D volume. Note that only the anatomical mask was smoothed, while previously assigned classification tags were preserved.

5.1.4. Finite Element Mesh Generation

Segmentation and tagged image stacks were fed into the image based unstructured mesh generation software Tarantula (CAE Software Solutions, Eggenburg, Austria) which builds fully unstructured, boundary fitted, locally refined, hex-dominant hybrid tessellations consisting of hexahedra, prisms, pyramids and tetrahedra [96].

Hybrid meshes were converted into purely tetrahedral element meshes. Classification tags were carried over from the tagged image stacks to the finite element mesh to be used later for the assignment of electrical and mechanical tissue properties during finite element matrix assembly. For the sake of numerical testing, three finite element meshes of the same whole heart geometry were generated at different spatial resolutions in multiples of the image stack resolution of 110 μm. A fine model (FM) was discretized at an average $\overline{h}$ of ≈ 220 μm, a medium model (MM) at $\overline{h}$ ≈ 440 μm, and a coarser model (CM) at $\overline{h}$ ≈ 880 μm.

While the CM with a resolution $\overline{h}$ ≈ 880 μm is, to our knowledge, the highest resolution mesh used for modeling cardiac mechanics, for solving the bidomain equations with acceptable accuracy its resolution is far too coarse. With physiological EP models as used here, predictions of conduction velocity would carry unacceptably large errors. At slower conduction velocities, as they govern conduction in a transmural direction, even an artificial conduction block may occur, thus rendering such a model unsuitable for bidirectionally coupled EM modeling. Nonetheless, this coarse model is deliberately included in this study to report performance and scalability limits since such smaller models are suitable for being used in weakly coupled electromechanical applications where EP and mechanics can be treated in isolation. Further, results obtained with the CM model are informative for simulating smaller animal models with appropriately higher spatial resolution.

Cardiac tissues are orthotropic in both electrical [17] as well as mechanical properties [49]. Since measuring the eigenaxes of the orthotropic tensor in vivo is not yet clinical routine and generates sparse and often noisy data sets, a previously developed Laplace–Dirichlet rule based method (LDRB) is employed [8] to assign the principal eigenaxes along fiber direction f₀, sheet direction s₀ and sheet-normal direction n₀ to each finite element in the ventricles. In the atria no fibers were assigned. The atria were modeled as an unexcitable, non-contracting, isotropic hyperelastic material, that had the principal role of providing more physiological boundary conditions in this model.

5.2. Modeling Electrophysiology and Active Stress Generation

Cellular dynamics were described by the recent Grandi–Pasqualini–Bers (GPB) model of the human ventricular myocyte [38], which was coupled to the Land–Niederer (LN) active stress model [69]. To allow for strong coupling, i.e., to account for length effects on the cytosolic calcium transient, the equation governing the binding of calcium to the low affinity regulatory sites on troponin in the myofilament model (see Eq. (114) in the Appendix of [38])

$$\frac{\mathrm{d}\left[{\mathrm{TnC}}_{\mathrm{L}}\right]}{\mathrm{d}t}=k_{{\text{on}}_{{\mathrm{TnC}}_{\mathrm{L}}}}\left[{\mathrm{Ca}}_{\mathrm{i}}^{2+}\right]\left({\widehat{B}}_{{\mathrm{TnC}}_{\mathrm{L}}}-\left[{\mathrm{TnC}}_{\mathrm{L}}\right]\right)-k_{{\text{off}}_{{\mathrm{TnC}}_{\mathrm{L}}}}\left[{\mathrm{TnC}}_{\mathrm{L}}\right]$$ (54)

is replaced by the analogous Eq. (1) of the LN active stress model [69], given as

$$\frac{\mathrm{d}\mathrm{TRPN}}{\mathrm{d}t}=k_{\mathrm{TRPN}}{\left(\frac{\left[{\mathrm{Ca}}_{\mathrm{i}}^{2+}\right]}{{\left[{\mathrm{Ca}}_{\mathrm{i}}^{2+}\right]}_{\mathrm{T}50}\left(\lambda \right)}\right)}^{n_{\mathrm{TRPN}}}(1-\mathrm{TRPN})-k_{\mathrm{TRPN}}\mathrm{TRPN}.$$ (55)

In the combined GPB+LN model (54) is replaced by (55), scaled by the total buffer concentration ${\widehat{B}}_{{\mathrm{TnC}}_{\mathrm{L}}}$ assumed to be 70 μmol L^–1 in the GPB model. This scaling is necessary to correctly relate fractional occupancy, used by the LN myofilament model, with the concentration of troponin bound calcium, used by the GPB model. That is, the state TnC_L in the GPB model state vector η is replaced by an appropriately scaled state TRPN, taken from the LN state vector, ζ. Therefore, the number of states in the combined state vector w is 58, which is one less than the sum of the number of states of the GPB state vector η (54 states) and the LN state vector ζ (5 states). Parameters of the GPB model were left unaltered whereas parameters of the LN model were adapted to give human myocardium tension transients when coupled to the GPB calcium transient [111] (see Tab. 1). This study sought to find an optimal set of parameters for combining cytosolic calcium transients, as predicted by the GPB model, with active stress transients, as predicted by the LN model.

Table 1.

Summary of electrical and mechanical material parameters. Bidomain conductivities were varied as a function of mean resolution $\overline{h}$ to keep conduction velocities at prescribed values.

Used for	Parameter
Passive stress	k = 1000 kPa,	af = 8:535 kPa,	bf = 5:972 [–],
	a = 1:333 kPa,	as = 2:564 kPa,	bs = 10:446 [–],
	b = 9:242 [–],	afs = 0:417 kPa,	bfs = 11:602 [–],
Active stress	Tref = 117:1 mN mm−2,	[Ca2+]T50 = 0:52 μmol L−1,	TRPN50 = 0:37 [–],
	nTRPN = 1:54 [–],	kTRPN = 0:14 ms−1,	nxb = 3:38 [–],
	kxb = 4:9 × 10−3 ms−1,
Bidomain, 220 μm	β = 1400 cm−1,	σil = 0:234 15 Sm−1,	σel = 0:22 Sm−1,
	Cm = 1:0 μF cm−2,	σit = 0:086 03 Sm−1,	σet = 0:13 Sm−1,
		σin = 0:019 33 Sm−1,	σen = 0:13 Sm−1,
Bidomain, 440 μm	β = 1400 cm−1,	σil = 0:294 31 Sm−1,	σel = 0:22 Sm−1,
	Cm = 1:0 μF cm−2,	σit = 0:140 47 Sm−1,	σet = 0:13 Sm−1,
		σin = 0:035 21 Sm−1,	σen = 0:13 Sm−1,
Bidomain, 880 μm	β = 1400 cm−1,	σil = 1:083 96 Sm−1,	σel = 0:22 Sm−1,
	Cm = 1:0 μF cm−2,	σit = 0:760 53 Sm−1,	σet = 0:13 Sm−1,
		σin = 0:090 40 Sm−1,	σen = 0:13 Sm−1.

In single cell experiments, both the unaltered GPB model and the combined GPB+LN model were paced at a cycle length of 500 ms for a duration of 60 seconds. In the combined GPB+LN model the stretch ratio was kept constant at λ = 1.0 over the entire protocol. In both models, all state variable transients were plotted over the entire pacing experiment to ensure that the system had settled close to a stable limit cycle. The effect of the altered troponin buffering equation in the GPB+LN model relative to the GPB model is illustrated in Fig. 1. The effect of strong coupling in the GPB+LN model under fixed stretch conditions in the range λ ϵ [1.0, 1.2], with the same pacing protocol, is illustrated in Fig. 2. The state vector at the end of the pacing protocol is stored to be used later for initializing the organ model. The same GPB+LN model is used throughout the ventricles, known electrophysiological heterogeneities remained unaccounted for.

Figure 1.

Comparison of V_m, TnC_L and $\left[{\text{Ca}}_{\mathrm{i}}^{2+}\right]$ single cell traces between the GPB model (blue) and the GPB+LN model at steady state with λ = 1 (red).

Figure 2.

Steady state traces of V_m, isometric tension, TnC_L and $\left[{\text{Ca}}_{\mathrm{i}}^{2+}\right]$ in a single cell pacing experiment under varying stretch ratios.

5.3. Mechanical Boundary Conditions

Homogeneous Dirichlet boundary conditions were applied at the termini of the meshed pulmonary veins and superior and inferior vena cava (Fig. 3). In addition, an elastic apical boundary condition was applied by attaching a small fragment of soft material to the apex of the heart, and applying homogeneous Dirichlet boundary conditions to the opposite side of this material.

Figure 3.

Mechanical boundary conditions: Homogeneous Dirichlet boundary conditions were enforced at the termini of the meshed pulmonary veins and venae cavae, and at the bottom of the soft material attached to the apex.

The heart was modeled without attachment to the vascular system, with left and right ventricles remaining unpressurized over the entire cardiac cycle. This setup therefore modeled a Langendorff perfused preparation of a human heart [79].

5.4. Material Parameters

The bidomain equations were parametrized to produce conduction velocities of 0.6 m s⁻¹, 0.4 m s⁻¹ and 0.2 m s⁻¹ along the orthotropic eigenaxes f₀, s₀ and n₀ of the tissue. While simulating action potential propagation in the FM (h = 220 μm) was possible with convergence errors that were well within the bounds of measurement uncertainties (2.8 %, 4.5 % and 19.4 % for v_f, v_s and v_n with the numerical settings used in this study), significant errors incurred at coarser mesh resolutions, particularly at slower conduction velocities. Without appropriate adjustments of conductivities, artificial conduction block would occur in the CM for propagation along the n₀ axis. Therefore, to minimize differences in mechanical response induced by deviations in activation pattern intracellular conductivities were varied as a function of h to arrive at comparable conduction velocities for FM, MM and CM [23]. Using these tuned conductivity values (Tab. 1) led to the same conduction velocities for each mesh resolution under planar wavefront propagation conditions. It is worth noting that the necessary variations were well within the uncertainty of conductivity measurements reported in the literature [22, 100, 99] for both the FM and MM model, but not the CM model as an $\overline{h}$ = 880 μm is insufficient when using a biophysically detailed model of cellular dynamics.

5.5. Benchmark Setup

The combined electromechanical four chamber model was used to benchmark performance and scalability of the AMG solver. At time t = 0 the endocardium of both ventricles was stimulated with a transmembrane current of 200 μA cm⁻² for 2 ms, initiating the transmural activation of the ventricles. Electromechanical activity was simulated for 500 ms to cover all phases of an entire heart beat. The same protocol was applied to all three models (CM, MM and FM), with the only alteration being the use of different conductivities (Tab. 1) to arrive at sufficiently similar electrical activation patterns. Simulations were performed with varying numbers of processes (N_p) for each model, with the range of N_p scaled by overall problem size (32–2048, 128–4096, and 512–8192 processes for CM, MM and FM, respectively). For all runs, time spent on solving the individual components of the elliptic PDE (bidomain equations), parabolic PDE and ODE systems was recorded. For the mechanics solver, time spent on matrix assembly and solver time was measured. Furthermore, the number of iterations was stored for both linear and nonlinear solves. The nonlinear mechanical problem was solved using Newton’s method with an absolute ℓ² norm error reduction of the residual (34) of ε = 10⁻⁶. For all linear subproblems, we used the preconditioned CG method with an absolute error reduction of ε = 10⁻⁸.

6. Results

6.1. Model Generation and Baseline Simulations

Finite element meshes were generated from the segmented image stack at three different average mesh resolutions $\overline{h}$ of 220 μm, 440 μm and 880 μm. The resulting FM, MM and CM geometry models are visualized in Fig. 4 and metrics for these meshes are summarized in Tab. 2.

Table 2.

Finite element meshes of human four chamber heart geometry.

	FM	MM	CM
Resolution	220 μm	440 μm	880 μm
Elements	184 554 709	24 593 765	3 698 356
Nodes	31 971 596	4 413 283	709 878

Prior to numerical benchmarking, baseline simulations were carried out for each model using the same settings except for the bidomain conductivities, which were modified as a function of $\overline{h}$ (Tab. 1). Maps of local activation time (T_act) were computed from V_m to validate that the electrical activation sequence of the ventricles was consistent between models. As shown in Fig. 5, the total activation time of the ventricles was around 45 ms. These activation times which are shorter than anticipated for a physiological activation sequence in a human, are due to the simple endocardial stimulation profile which covered the entire endocardia of both ventricles. This is different from a physiological case where the Purkinje system stimulates the endocardia only over apical and mid segments, but not over the basal segments which are not covered by the Purkinje system. Visual comparison of activation isochrones for later activation times around 35–45 ms suggests that the choice of $\overline{h}$-dependent conductivities led to activation sequences which were indeed very similar between the models (Fig. 5). As expected for the case of closely matching activation sequences, close agreement was also found between models for the electrophysiological quantities V_m, ϕ_e and [Ca]_i. Similarly, the mechanical quantities ∥u∥, S_a, S_p and λ, computed from (25) at a temporal resolution of 1 ms were also compared visually between models. Differences in global mechanical performance of the left ventricle were gauged by tracking the time course of the left ventricular cavity volume V_LV(t) over an entire heartbeat. A representative result is shown for the FM model in Fig. 6.

Figure 5.

Electrical activation patterns for the CM, MM and FM models, shown from anterior and posterior views.

Figure 6.

Electromechanical simulation of a human heartbeat. Top row shows displacement ∥u∥ and bottom row transmembrane voltage V_m at instants 0 ms, 40 ms, 80 ms, 150 ms and 350 ms after the stimulus delivered to the endocardia.

6.2. Solver Configuration

Solver parameters were chosen based on the experience gained during the baseline simulations. The parameter ϵ, which is used during the AMG setup to decide which nodes are strongly connected, was set to 0 for both the electrics and mechanics solvers. With ϵ = 0, the AMG setup is based on the matrix graph alone. While this choice had a negative impact on the linear solver convergence, it significantly improved parallel load balancing and, consequently, overall performance. Compared to ϵ = 0, choosing ϵ > 0 yields a larger parallel variability in size of the linear systems arising at the various levels of the multigrid hierarchy, particularly for uneven parallel distributions of the Dirichlet boundaries. These variations in linear system size led to poorly balanced computational load and thus increased MPI waiting time. The number of pre- and post-smoothing sweeps was set to 1. A larger number of smoothing sweeps would improve convergence but overall solving time would be longer as the reduction in iterations is insufficient to compensate for the increased computational cost of the additional smoothing sweeps. Similarly, the use of direct solvers for inverting the AMG coarse-grid was omitted for both the mechanical problem and the elliptic problem. While direct coarse-grid solvers can improve convergence significantly (depending on the number of multigrid levels used), the baseline benchmarks also showed a reduced overall parallel efficiency of both setup and solving phases. Instead, the coarse-grid solving strategy was to continue coarsening until a 1 × 1 or 3 × 3 system was reached for the elliptic and mechanics problems respectively. The most important factor for improving strong scalability was the grouping of parallel data at the intermediate multigrid levels onto fewer active computing units, as described in Section 4.2. The reduction in communication density and the increase in average message size turned out to be beneficial for N_p > 320. This effect was even greater for N_p > 640, above which scalability stalled with a group size of g = 1, meaning data redistribution was turned off (Fig. 7). In terms of overall solving time, a group size of g = 2 proved to be optimal; a more aggressive reduction of the computing units g = 3, 4 was not beneficial. The performance of the linear solver in terms of convergence and cost of individual solver steps was monitored by recording iteration count, N_it, and time per iteration, T_it. Over the range of processes N_p tested, only a very minor dependency of N_it was observed. Thus only one representative graph for N_p = 512 is shown to illustrate the course of N_it over an entire cardiac cycle (Fig. 8). N_it increased with the overall size of the system to be solved, which shows that ptAMG does not fully succeed in preserving convergence independently of the spatial resolution. Nevertheless, the increase in iteration counts is moderate compared to the increase in spatial resolution and thus linear system size from CM to FM (also compare Table 2). The number of nonlinear Newton iterations N_Nit was monitored as well. Depending on the phase of deformation N_Nit varied in the range between 3 to 7 Newton iterations, largely independent of the problem size (Fig. 8).

Figure 7.

Comparison between different multigrid redistribution strategies used in the ptAMG preconditioner. The time needed to solve the linear systems in each newton step of the MM model during a 20 ms simulation was measured. The variable g denotes the group size of the AMG redistribution as described in Section 4.2. With g = 1 redistribution is turned off, while starting with g = 2, multigrid redistribution is activated.

Figure 8.

Comparison of the average number of linear solver iterations (left panel) per linear solver step and the number of nonlinear Newton iterations (right panel) over a full heart beat for simulation runs using N_p = 512.

6.3. Strong Scaling Benchmarks

The final baseline setup, which simulated EP and mechanics of an entire heartbeat over a basic cycle length of 500 ms, was used for performing a strong scaling benchmark. The goal was to determine the strong scaling performance of the major contributing factors (finite element matrix assembly time and solver time) as well as the overall scaling. The initial number of used processes N_p was chosen sufficiently large to be able to fit the electromechanical model into memory. The required N_p was 32, 128 and 512 for CM, MM and FM respectively. Due to its highly parallel nature, the matrix assembly showed linear scaling over almost all N_p probed. With the chosen ptAMG-PCG solver configuration for the deformation problem, strong scaling with high efficiency was achieved over a wide range of processor counts (Fig. 9, left panel). For the CM model a parallel efficiency above 69 % was achieved as far as 512 processes. From 1024 processes parallel efficiency started to degrade, which can be explained by low computational load on each of the very small local partitions (≈ 2500 DOF per process). The MM model showed an efficiency above 64 % up to 2048 processes. Parallel efficiency saturated at N_p = 4096 (≈ 4000 DOF per partition). The FM model showed strong scaling for all N_p probed. At N_p = 8192 (with ≈ 13 000 DOF per partition) parallel efficiency was still at 58 %. Further increasing N_p > 8192 could not be tested, due to an unresolved fault of the MPI library on the available HPC system. Measured timings and parallel efficiency values are listed in Table 3. Overall, for electromechanical simulations using the monodomain equations to represent EP, computing costs are clearly dominated by solving the deformation equation which is between 3× and 7× as expensive as the matrix assembly costs. In comparison, the costs of solving the parabolic PDE and the system of ODEs which make up the monodomain model are negligible.

Figure 9.

Strong scaling profile of an electromechanical model using a monodomain (left) or bidomain (right) EP model. Red, blue and green traces refer to CM, MM and FM simulation runs respectively, over a time frame of 500 ms. On the left hand side the assembly time for mechanics (33–34) and the solve time (i.e. preconditioner setup and solving) for all linearized systems of equations (32) is given. In addition, we show the total solving time of a monodomain electromechanical simulation. Depicted in the right plot is the computational time for all elliptic systems (38), the computational time for all ODE systems (39–40) and the computational time for all parabolic systems (42). In these three cases the computational time includes assembling, preconditioner setup and solving times. Solid lines correspond to the total solving time of a bidomain electromechanical simulation.

Table 3.

Strong scaling benchmark results for a monodomain electromechanical simulation of a heartbeat with CM, MM and FM model: Table columns are number of cores N_p, and timings for matrix assembly (MA), solving (Solve), monodomain (Mono), total time and parallel efficiency E.

	N p	MA	Solve	Mono	Total	E
CM	32	5746 s	25 088 s	12 852 s	43 687 s	1.00
	64	2928 s	13 054 s	6381 s	22 363 s	0.98
	28	1488 s	6946 s	3290 s	11 724 s	0.93
	256	771 s	4006 s	2009 s	6787 s	0.80
	512	404 s	2372 s	1175 s	3951 s	0.69
	1024	213 s	1633 s	812 s	2658 s	0.51
MM	128	9503 s	50 122 s	20 042 s	79 667 s	1.00
	256	4835 s	25 752 s	10 683 s	41 270 s	0.97
	512	2484 s	13 545 s	5541 s	21 572 s	0.92
	1024	1293 s	8163 s	3030 s	12 486 s	0.80
	2048	716 s	5212 s	1889 s	7817 s	0.64
	4096	461 s	3563 s	1246 s	5270 s	0.47
FM	512	17 572 s	79 413 s	34 583 s	131 568 s	1.00
	1024	8896 s	40 842 s	17 760 s	67 498 s	0.97
	2048	4596 s	23 388 s	9952 s	37 936 s	0.87
	4096	2506 s	14 086 s	5078 s	21 670 s	0.76
	8192	1743 s	9363 s	3014 s	14 120 s	0.58

When considering a bidomain model to represent EP in an electromechanical simulation, the distribution of costs is markedly different. As shown in the right panel of Fig. 9, computational costs of solving the elliptic portion of the bidomain equation alone is comparable to solving an electromechanical problem with a monodomain EP model. In addition, solving of parabolic and elliptic PDEs can be seen to be far less scalable as solving the deformation problem on the same grid. This is due to the smaller number of degrees of freedom per node which leads to a linear system to be solved that is smaller by a factor of 3× and a computational load that is smaller by a factor of 9×. Therefore, the critical surface to volume ratio of a local partition which translates to a communication to compute load ratio, is hit earlier, at ≈ N_p/3. This confirmed by Fig. 9, where scaling efficiency can be seen to drop markedly at N_p = 512, 2048 and 4096 for CM, MM and FM respectively.

6.4. Impact of Length Dependence upon Calcium

The effect of length dependent calcium buffering of troponin C on cytosolic calcium was investigated by comparing two sets of simulations using the same settings except for the reference value [Ca²⁺]_T50(λ) in Eq. (55). In the strong coupling scenario [Ca²⁺]_T50(λ) was used as described in [69], i.e.,

$${\left[{\mathrm{Ca}}^{2+}\right]}_{\mathrm{T}50}\left(\lambda \right)={\left[{\mathrm{Ca}}^{2+}\right]}_{\mathrm{T}50}^{\mathit{ref}}(1+{\beta }_1(\lambda -1\left)\right)$$ (56)

whereas in the weak coupling case [Ca²⁺]_T50(λ) was kept constant at the value corresponding to λ = 1.0, i.e., ${\left[\text{Ca}\right]}_{\mathrm{T}50}\left(\lambda \right)={\left[{\text{Ca}}^{2+}\right]}_{\mathrm{T}50}^{\mathit{ref}}$. Fig. 10 illustrates the effect of length dependent calcium binding affinity on calcium transients, stretch ratio λ and active stress S_a.

Figure 10.

Effects of length dependence calcium binding affinity of troponin C (TnC). Shown are the distribution of intracellular calcium [Ca²⁺]_i and stretch ratio λ at t = 130 ms, and active stress S_a at t = 190 ms in the strong coupling (TnC(λ)) and weak coupling (TnC(λ_ref)) scenario.

7. Discussion

This study describes novel methodology for modeling human whole heart electromechanics at a high spatial resolution. Such high resolutions are beneficial for several reasons: (i) cardiac anatomy can be faithfully represented with high geometric fidelity; (ii) achieving fully, or at least nearly, converged simulation results becomes feasible in strong coupling scenarios where mechanical deformation influences the electrophysiological state of cells via MEF mechanisms; and (iii) both fine scale structural heterogeneities such as vessels, fat deposits, scarred tissue and fibrosis as well as functional heterogeneities due to spatial variation in micro-perfusion, metabolic state or protein expression can be explicitly included in a model. The computational burden imposed by such models is significantly larger when compared to lower dimensional models as they are standard in cardiac mechanics applications [59].

This issue of computational cost has been addressed by focusing on a massively parallel iterative solving method for large deformation problems, allowing us to exploit the full potential of modern HPC hardware. In particular, we developed a novel AMG preconditioner for a CG solver for application to nonlinear mechanics. The primary design objective was to achieve favorable strong scaling characteristics for both setup and solution times up to a sufficiently large core count, such that execution times were reduced and simulations were computationally tractable. Depending on the chosen resolution, benchmark results demonstrated efficient scaling up to 1024, 4096 and 8192 compute cores which allowed the simulation of a single heart beat in 44.3, 87.8 and 235.3 minutes with the CM, MM and FM models respectively. The achieved efficiency makes sufficiently short simulation cycles feasible without compromise of anatomical or biophysical detail.

7.1. Computational costs

The computational burden imposed by high resolution electromechanical models demands the most efficient numerical approaches in order to keep simulations tractable. In the cardiac EP modeling community, two approaches are currently investigated, the use of spatio-temporal adaptivity [10, 26], polynomial adaptivity [1], or, as in this study, the use of strongly scalable solvers which reduce execution times by engaging a larger number of computational units, be it traditional CPUs [82, 98], or acceleration devices such as GPUs [80], or a combination of both parallelization and adaptivity [27]. Solving deformation problems in cardiac or vascular modeling applications relies more on reducing cost by resorting to lower dimensional EP models with slow upstroke velocities combined with simplified active stress models [89, 5] and weak unidirectional coupling to allow for coarse spatial discretization of mechanics [85, 5, 42]. In terms of solver methods used, most studies relied upon direct solvers which tend to become unfeasible for high resolution 3D problems due to linear system size and sparsity pattern [46]. Exceptions exist where strongly scalable solvers were employed, so far only in vascular modeling studies [61, 3] or for a small number of cores [42]. Another approach to reduce costs would be the use of grids of different resolution, a fine mesh for discretizing the bidomain equations to capture the steep wave fronts, and a coarser mesh for discretizing the deformation equation, as mechanical solutions are supposed to be much smoother in space.

Departing from our previously developed bidomain solver infrastructure for which strong scaling properties have been reported in detail [82, 92, 80], we aimed to achieve similar scalability for solving nonlinear deformation problems. As shown in Fig. 9, left panel, scaling efficiencies of 51 %, 47 % and 58 % (Tab. 3) could be achieved when using 1024, 4096 and 8192 cores for executing the CM, MM and FM model, respectively. Remember that the coarse model (CM) was discretized at an average $\overline{h}$ of ≈ 880 μm, the medium model (MM) at $\overline{h}$ ≈ 440 μm, and the fine model (FM) at $\overline{h}$ ≈ 220 μm, see also Tab. 2. Scalability of the two major cost factors, matrix assembly and solving of the linear systems, are shown in Fig. 9. As expected, matrix assembly scales almost linearly as this is an embarrassingly parallel problem. Solver scalability levels off when hitting a critical surface-to-volume ratio between communication, ∝ to the surface of the local partition, and compute cost, ∝ to the volume of the local partition. When compared to the scalability of solving the monodomain equation, the much better scalability of the deformation solver becomes apparent. Noticeable degradation in scalability starts at lower N_p of 256, 1024 and 4096 cores for CM, MM and FM model, respectively. This is due to the less favorable compute load to communication ratio of a scalar PDE with one degree of freedom per node as compared to the three degrees of freedom per node in the deformation equation. Moreover, a significant portion of the mechanics solver work is dedicated to matrix assembly which is computationally quite expensive, but highly scalable as there is no communication cost involved in this step.

While for most electromechanical problems the combination of monodomain and deformation suffices, there are applications of practical relevance where a bidomain formulation may be more appropriate, such as investigating the influence of deformation on the T-wave of the ECG [56]. For this sake scaling data for bidomain simulations were added as well as a reference. As shown in Fig. 9, right panel, scalability of the elliptic and parabolic solve is comparable but solving the elliptic PDE is roughly one order of magnitude more expensive than solving the parabolic PDE. Solving the biophysically detailed GPB+LN model is more costly than the parabolic solve, but since state variables η do not diffuse, the ODE solver step is embarrassingly parallel, which is reflected in a linear scaling graph.

A note of caution is warranted when comparing execution times for EP and mechanics, as a much finer time step of Δt = 5 μs was used for EP than for mechanics with Δt = 1 ms, that is, the EP problem was solved 200× more frequently than the deformation problem. Therefore, a single solver step of the deformation problem is 470–560× and 75–160× more expensive than a single solver step of the monodomain and bidomain equation, respectively.

Preliminary data suggest that further significant improvements in scalability are achievable by exploiting shared memory parallelism in a hybrid MPI-OpenMP approach. The main scaling benefits are a result of the shorter latencies of shared memory data access and the more efficient use of interconnect hardware in terms of latency and bandwidth as less messages are passed, each message being of larger size. Another potentially effective way of enhancing scalability is to employ latency hiding Krylov methods [35, 36].

While both approaches are promising, a detailed exploration of their suitability for cardiac electromechanics applications is beyond the scope of this study.

7.2. Representing Cardiac Anatomy

Cardiac anatomy plays a role of pivotal importance in electrophysiological and mechanical function. Computer models therefore aim to represent anatomy as faithfully as possible, however due to limitations in image acquisition, image segmentation and registration techniques and the difficulty in tessellating anatomically complex 3D structures into discrete finite element models of sufficient mesh quality, most reported studies resorted to using simplified, idealized FE models. In earlier modeling studies, the left ventricle was modeled in isolation as an idealized geometry of cylindrical or ellipsoidal shape [14, 81]. Later studies used more elaborate finite element models which incorporated anatomically realistic biventricular geometries based on histological sections [40, 116, 114, 106]. Due to the cost of generating detailed anatomical models, only one representative heart per species was available in these studies. While such an “one heart fits all” approach proved to be of great utility for studying generic mechanisms, this approach is limited in its capability to make predictions for individual patients, due to the wide anatomical inter-subject variability of the heart. Therefore, driven by major advancements in magnetic resonance (MR) and diffusion tensor (DT) MR imaging, a new trend emerged over the past few years towards models derived from tomographic medical images. Traditionally, the most widely used approach is based on using cubic Hermite finite elements to approximate the biventricular geometry [86, 117, 41]. Techniques were developed to register cubic Hermite template models with segmented medical image stacks [66]. More recently, the cubic Hermite method has also been applied to generating models of the thin walled atria, but significant processing e orts and manual steering is necessary [37]. The main advantage of this cubic Hermite element based method is its capability to approximate cardiac geometry with a limited number of finite elements, keeping compute and memory costs tractable even when using modest computing hardware. However, with regard to creating biophysically detailed and anatomically accurate models coarse mesh approaches pose three challenges. Firstly, they are limited in their ability to capture fine anatomical structures, including papillary muscle, thin walled atria and trabeculation. Secondly, They are limited in their ability to capture regional heterogeneities, including ion channel gradients, peri infarct zones, local fiber dispersion and vasculature. Finally, most approaches rely on the adaptation of template meshes which may not confirm to the breadth of patient anatomies. Moreover, finding an optimal tradeoff between geometric fitting accuracy, mesh quality and stability for coarse cubic hermite based models under large deformations can be challenging [67]. These issues can all be addressed through the use of fine resolution unstructured tetrahedral or hybrid meshes [101, 42], generated directly from image segmentations. The ability to generate models automatically from segmentations improves the robustness of the model generation process. This will be beneficial for future clinical applications and will pave the way towards making models available to non-expert users.

Advanced image acquisition and processing [94, 13] as well as unstructured mesh generation pipelines [96] have been developed for such purposes. These methods permit the automatic construction of anatomical FE models which represent individual geometries at an unprecedented level of geometric fidelity. So far, such detailed models have been used in electrophysiological modeling studies [12], but not yet in any mechanical modeling studies. Gurev and co-workers reported a method which combined an anatomically detailed biventricular tetrahedral FE geometry model for simulating electrophysiology, with a cubic Hermite FE model for mechanics [41]. The method allows wrapping a cubic Hermite template model around the detailed tetrahedral FE model. While efforts were made to minimize differences between the meshes, due to the significantly lower level of detailed of the mechanical model, achieving a perfect overlap was impossible. This increases the complexity, of the model generation and simulation process, particularly in strongly coupled electro-mechanical simulations where projection operators would have to be constructed to transfer data between two FE spaces of imperfect overlap.

An alternative method based on CAD reconstruction of hearts, which uses a four chamber representation plus the attached large vessels similar to that used in this study, has been reported recently [5]. As details on the model generation process have not been published yet, it remains to be seen whether a CAD based approach is viable for generating tomographically reconstructed FE models. This study has similarities with ours, in particular that P₁ – P₀ tetrahedral elements were used, facilitating an accurate approximation of the gross anatomy – and the use of the same FE mesh for both electrophysiology and mechanics. The major difference of this study from ours is that the low resolution used precludes the use of physiologically realistic models of cellular dynamics such as the GPB+LN model used in our study.

The four chamber heart model used in our study represents all details which can be delineated in a standard clinical image stack. The mesh generation methodology draws from previously developed tools [96], which permits, in theory, to represent any discernible structure of the heart down to the microscopic level [94]. All FE meshes in this study were generated fully automatically, without any manual intervention. Any anatomical deviations of the model are within the uncertainty of the segmentation. Generating the CM, MM and FM meshes on a desktop computer took 51, 81 and 205 minutes respectively.

7.3. Bidirectional Coupling

From a physiological point of view, strong or bidirectional coupling is important as this mediates acute and regulatory MEF, i.e., changes in electrophysiological state of a cell as a function of tissue deformation, measured as stretch ratio λ. From a modeling point of view, strong coupling adds technical complexity as electrophysiology and mechanics cannot be treated independently, necessitating the solution of a coupled mono- or bidomain and deformation equation system. Three major mechanisms mediate MEF, by which deformation alters the electrophysiological state of the tissue: (i) deformation alters geometry and thus electrical gradients ∇ϕ_i and ∇ϕ_e respectively; (ii) currents through stretch activated channels I_SAC(V_m, λ) directly alter transmembrane voltage V_m which indirectly influences upon cellular signaling; (iii) stretch alters binding affinity of calcium to troponin C which influences cytosolic calcium transients and diastolic calcium levels. As calcium is involved in many important signaling pathways in a cell this effect may play important roles in both acute and long term regulation of contractility. While our framework would easily accommodate all three MEF pathways, for the sake of testing the scalability of the developed solver methods we restricted ourselves to the feedback through calcium binding affinity of troponin C. In the single heart beat scenario, effects on electrical gradients are very small as electrical gradients are large only prior to contraction, but are very small during contraction when cells are at the plateau of the action potential or slowly repolarize [121]. While the existence of stretch activated channels in cardiac muscle has been postulated [52], in models they are largely treated phenomenologically as a “black box”, as the molecular mechanisms which relate stretch to channel conductance remain poorly understood. Various mathematical structures for I_SAC currents have been published, but parametrized for other species than humans [110] or without parametrization at all [64].

As illustrated in Fig. 10, distribution of stretch ratio λ and active stress S_a during a heartbeat is quite heterogeneous. This is particularly apparent around the apex where the ventricles were elastically anchored, as well as along sites of transition between actively contracting myocardium and passive structures such as the superior vena cava or the interface between ventricles and atria which were modeled as mechanically passive material in this study. Differences between strong coupling case TnC(λ) and weak coupling case TnC(λ_ref) in terms of λ and S_a were rather minor (Fig. 10). Overall, stresses were slightly higher in the weak coupling case. This was due to the length dependent calcium binding affinity being higher compared to the TnC(λ) case where under the given loading conditions and in absence of any cavitary pressures the stretch λ was smaller than 1 during contraction, entailing a comparably lower calcium-bound [TnC] and thus less force generation. As length dependence of active stress in the LN model is modeled as S_a ∝ h(λ)× XB([TnC](λ)) where h(λ) and XB refer to filament overlap and fraction of force generating crossbridges, in the weak coupling case length dependent force generation is mediated only by filament overlap h(λ).

The observed heterogeneity in λ(x) translated into a marked heterogeneity in cytosolic calcium $\left[{\text{Ca}}_{\mathrm{i}}^{2+}\right]$(x) only in the strong coupling case TnC(λ), whereas in the weak coupling case a TnC(λ_ref) heterogeneity at the same instant in time was marginal (compare left and right in $\left[{\text{Ca}}_{\mathrm{i}}^{2+}\right]$ panel of Fig. 10). It is important to note that this effect on calcium heterogeneity can only be captured in a strongly coupled model. In most weak coupling scenarios, EP models such as the GPB model are coupled with stress models such as the LN model in a way where calcium buffered [TnC] is tracked by both EP and stress model using different equations. That is, the GPB model tracks TnC concentration based on Eq. (54) and the LN model based on Eq. (55). This is not only inconsistent from a physical point of view, such approaches may also miss a potentially important feedback mechanism. While this study is far from being an exhaustive evaluation the effects of strong coupling, as this was beyond our scope, these results suggest that such effects may be non-negligible, depending on the particular question under study.

Furthermore, it is assumed that deformation is governed by slower time scales and that solutions are much smoother in space compared to electrophysiology which is characterized by fast transients and steep wave fronts. This notion, if correct, suggests that coarser spatio-temporal discretization might be suitable, thereby reducing computational cost when solving the deformation equations. The exact choice of time and space step for mechanics and the resulting errors are likely to depend on the scenario to be modeled and the level of numerical accuracy one aims to achieve. While exhaustive convergence studies are missing for the complex models as used in this study, numerical work with simplified 1D strand models suggest that spatio-temporal discretization in a strongly coupled scenario also requires fairly fine time and space steps [121]. It therefore remains to be seen whether or not the use of a coarser mesh for mechanics pays off in terms of reduced execution time and overall modeling cost. Additional costs would be incurred due to the need of generating two meshes of different spatial resolution which, ideally, perfectly overlap in space. In practice, this is difficult to achieve when modeling complex structures such as an anatomically realistic four chamber representation of the human heart. Due to the complex geometry, the surfaces of both meshes will be different, requiring extrapolation to transfer coupling variables between the two grids. In a parallel context, a coarser mesh for mechanics will be partitioned differently from the mesh used for EP, causing an increase in the communication cost of the projection operation. Using the same mesh for both EP and mechanics, as implemented in this study or as used by others [5], may be less efficient in weak coupling scenarios where a single static offline projection of data such as activation time or calcium transients can be used as an input for mechanics, but in strong coupling scenarios this may not be the case. Thorough numerical convergence testing studies are necessary in future work to address the issue of spatio-temporal discretization for strongly coupled electromechanical simulations. Such convergence studies will lead to a robust quantification of errors, allowing us to find a balanced tradeoff between computational cost and accuracy.

7.4. Convergence Testing

In an ideal numerical scheme for solving PDEs, the solution depends only on the chosen parameters, but not any other factors such as the spatial discretization step h. While it is feasible to construct such numerical schemes, in practice this is rarely the case. From an applied point of view, the relevant aspect is not to drive discretization errors to zero. Rather, it is more important to approximate the discretization error and contrast it with the typically large uncertainties in parameters and model assumptions. By considering all these factors, we are able to find a balanced tradeoff between computational cost, model parametrization and accuracy.

In the EP modeling communities this issue has been addressed, starting with very early pioneering studies [95], by spatio-temporal convergence testing to choose h and dt appropriately. An extension of this approach brought the introduction of N-version benchmarking to verify whether N different and independent implementations yield comparable results [84]. For solving the monodomain equations it has been demonstrated that all N = 19 independent codes used by the EP modeling community yielded the same result, if h and dt were chosen to be sufficiently small. For electromechanical models, convergence testing or N-version benchmarking has not yet been established. In a recent study an N-version benchmark was carried out for solving a cardiac deformation problem, but mechanical deformation was driven by a time triggered active stress models, i.e. a rather simplified weakly coupled model was considered. Moreover, with N = 2 the number of independent implementations was quite small. In a strong coupling context, as the modeling work in this study, testing for convergence of electromechanical simulations is more challenging as any errors in the driving EP solution will feed into the solution of the deformation problem. With biophysically detailed models of cellular dynamics such as the GPB model, very fine spatial resolutions are key to keep discretization errors within acceptable bounds. To minimize the impact of errors of the electrical activation pattern on mechanical deformation, conductivities were fitted to travel at prescribed velocities for each of the CM, MM and FM models. As shown in Fig. 5, this approach indeed results in very similar activation sequences, though deviations are visible in the CM model, indicating that this approach may not be suitable for even coarser discretizations as done by others [42, 5]. While we did not perform a thorough convergence study, as the complex geometry used did not lend itself well to such an investigation, the effect of mesh resolution on global mechanical performance was evaluated by comparing the computed left ventricular cavity volume over time between models. As shown in Fig. 11, volumes at end-diastole are not discernibly different, but noticeable discrepancies arise during systole, which do not change monotonically with $\overline{h}$.

Figure 11.

Volume changes of the left ventricle, V_LV(t), in milliliters.

In weak coupling cases using activation time triggered mechanics [42] or when simplified phenomenological models of FitzHugh–Nagumo type are used [5], electromechanical convergence is likely to be less of a concern. However, such models are not suitable for studying biophysical mechanisms of excitation-contraction coupling and MEF at the organ scale. The mesh resolutions used in such studies are unsuitable for use with physiologically detailed models of cellular dynamics such as the GPB. At such coarse resolutions artificial conduction block can occur, impeding action potential propagation. This is not the case with simple phenomenological models as the upstroke of the action potential is orders of magnitude slower. As shown in [5], the upstroke of the used EP model lasts around 50 ms whereas the upstroke duration of a biophysically realistic model last around 1 ms. Such slow upstroke velocities translate into less steep wave fronts, which can be resolved on coarser meshes, reducing computational costs significantly.

In the context of convergence testing of strongly coupled electromechanical models the presented numerical approach is of pivotal importance. The demonstrated excellent scaling properties enable the use of ultra-high spatial resolutions which are necessary for generating high fidelity reference simulation results. The complex models used in this study, while well suited for scaling benchmarks, are less suitable for rigorous convergence testing. Simpler geometries, as used in standard EP benchmarks [84], lend themselves more easily to such tests since spatio-temporal resolution can be varied over domains of exact overlap in space.

7.5. Limitations

The focus of this study was to design and implement a strongly scalable iterative solver which is sufficiently efficient to enable execution of high resolution strongly coupled whole heart EM models. Other issues which have not been addressed and may limit the applicability of the proposed method in different modeling scenarios include:

1. A penalty formulation and element-by-element static condensation is used to ensure a nearly incompressible material behavior. Depending on the penalty parameter k this may lead to very ill-conditioned problems. However, with the choice of k = 1000 kPa (Table 1) and the design of the AMG method as described in Section 4, we were able to solve the linearized systems within a reasonable number of iterations (Fig. 8). Nevertheless, as the iteration numbers of the linear solver increase with k, a mixed approach [4, 71] may be preferable.

2. Simple linear tetrahedral elements were employed for both the electrophysiological and the mechanical models. While this is a valid and accepted approach for the bidomain equations, the resulting P₁ – P₀ elements for solid mechanics are known to be prone to volumetric and shear locking. A mixed formulation with quadratic shape functions for displacement and a linear hydrostatic pressure field, i.e. P₂ – P₁ tetrahedral elements, would help to avoid these problems [28, 32, 87]. Such an approach is well suited for large deformation problems and shows the same performance as a mixed formulation using Q₂ – P₁ hexahedral elements [19].

   However, as we wanted to stick to our principle of utilizing the same grid for both physics, the usage of higher order elements for mechanics would lead to a highly unbalanced relation between the size of the mechanical and electrical systems. Hence, a reasonable employment of P₂ – P₁ elements requires different spatial resolutions for mechanics and electrics and an appropriate projection of coupling data between the electrical and the mechanical grid. The construction of such projections and therefore the mixed approach are beyond the scope of this specific paper and left to future work.

3. A Langendorff perfused setup was modeled where the ventricles were not pressurized. This is different from a working heart model where isovolumetric phases are enforced when valves are closed and pressure-volume relations are governed by the cardiovascular system or the atria when valves are open.

4. In vivo, the heart is enveloped by the pericardium, which is elastically attached to surrounding tissue via the mediastinum. This important aspect of cardiac function has not been accounted for [32].

Addressing some of these limitations yields block-structured systems that require special treatment. Such systems can be solved iteratively using specially designed AMG methods [51, 70, 122] or properly chosen block preconditioners, generally based on a Schur complement approach [28, 29, 42]. While the solver approach presented here is not directly applicable to such problems, it still constitutes an important building block for solving block systems. However, conceiving AMG solvers for block system or building Schur complement preconditioners is beyond the scope of this study.