UltraWave: An Open-source Multi-GPU Full-wave Simulator for Acoustic and Elastic Wave Scattering in 3-D Heterogeneous Media

Abstract

Fast and accurate three-dimensional (3-D) full-wave ultrasound simulation is critical for applications such as medical imaging, tissue characterization, and synthetic data generation for machine learning. However, computational demands escalate prohibitively at higher resolutions. To address this challenge, we developed UltraWave, an open-source, multi-GPU-accelerated full-wave simulator for acoustic and elastic wave scattering in 3-D heterogeneous media. UltraWave incorporates advanced capabilities including media inhomogeneity, elastic wave propagation, power-law frequency-dependent absorption, and perfectly matched layers. To rigorously validate simulation accuracy, we also developed a companion code suite that computes the scattered waveforms in both time and frequency domains using analytical solutions from scattering theory for several standard scattering scenarios, enabling quantitative benchmarking of numerical simulators and addressing a critical gap in standardized validation tools. We evaluated UltraWave against the widely used k-Wave simulator, and demonstrated its accuracy, computational efficiency, and scalability. For a 3-D elastic scattering simulation, UltraWave achieved equivalent accuracy to k-Wave while reducing the runtime from 8 hours (k-Wave on a single GPU; multi-GPU capability is currently unavailable in k-Wave) to 20 minutes (UltraWave on 4 GPUs). UltraWave offers a powerful and scalable tool for high-resolution wave propagation and scattering simulations in biomedical applications.

Graphical Abstract

I. Introduction

ACOUSTIC simulations are important in biomedical ultrasound, playing a significant role in imaging, tissue characterization, and synthetic data generation for machine learning applications [1–3]. Over the years, methods such as the spatial impulse response [4], k-space [5], finite difference [6–9], and angular spectrum methods [10–12] have been widely adopted. These tools enable critical applications ranging from transcranial ultrasound field prediction [13] to imaging system design [14].

As ultrasound technologies evolve, four capabilities emerge as essential for simulators: 1) accurate modeling of scattering in heterogeneous media, 2) full three-dimensional (3-D) wave propagation, 3) frequency-dependent absorption, and 4) elastic wave support. First, accurate simulation of scattering is critical because biological tissues are inhomogeneous, and acoustic scattering significantly influences the ultrasonic echo signals acquired from tissue. These echo signals are utilized in essentially all ultrasound imaging technologies. In particular, spectral analyses of the raw radiofrequency (RF) echo signals have been used in quantitative ultrasound (QUS) to derive imaging biomarkers [15–17] that assess tissue health and diagnose diseases such as steatotic liver disease [18], cancerous tumors [19, 20], and other conditions [21, 22]. High-fidelity simulations hold the potential to predict RF signals backscattered from tissues based on tissue histology [23, 24], thus enabling a better understanding of ultrasound wave propagation in biological tissues [25, 26]. Second, 3-D simulation capabilities are necessary because two-dimensional (2-D) simulations cannot fully capture the anisotropic medium properties nor the volumetric wave-tissue interactions. Studies have shown significant differences between brain model recovery using 3-D and 2-D simulation data, as 2-D simulations do not model out-of-plane energy present in 3-D data [27]. Third, accurate modeling of acoustic absorption is important due to the significant impact of tissue absorption characteristics on the amplitude, spectrum, and shape of propagating ultrasound pulses. Absorption is an unavoidable factor in real-life ultrasound experiments, as it restricts the penetration of ultrasound waves and the maximum imaging depth. Absorption contributes to attenuation, which is a useful quantitative ultrasound biomarker [18, 28, 29]. Fourth, accurate simulations of elastic wave propagation, including shear waves, are also important for various applications, such as the simulation of ultrasound wave propagation through bones [30] and the development of shear wave elastography techniques [31].

However, achieving these capabilities introduces a practical challenge: the trade-off between accuracy and efficiency. Higher resolution, which leads to greater accuracy, demands significant computational power and time. This is especially true for 3-D simulations, where the computational needs increase significantly with simulation resolution. Generating large training datasets for deep learning methods also requires high simulation efficiency. Therefore, simulation tools that can utilize graphics processing units (GPUs) are highly desirable due to the enhanced computational efficiency.

While several open-source acoustic simulation tools are available [5, 6, 10, 12, 32–35], only a selected few are optimized to take advantage of the power of GPUs to accelerate simulations, such as k-Wave [5], Stride [34], and j-Wave [35]. Other tools, such as SimSonic [6], which supports acoustic and elastic wave simulations particularly in bone modeling contexts, are central processing unit (CPU)-based and not optimized for GPU acceleration. Although promising and powerful, each tool has its own limitations. For instance, k-Wave offers high accuracy with the k-space method and supports GPU accelerations for simulations; however, the latest publicly released version does not yet support the use of multiple GPUs for simulations. Stride is an open-source ultrasound simulation tool that provides flexibility and scalability with high performance; however, this tool supports limited power law frequency dependent absorption (restricting the power-law exponent to values of 0 and 2 exclusively) for acoustic wave simulations and does not support 3-D elastic wave simulations. Another tool, j-Wave, powered by the JAX library [36], does not yet support elastic wave simulation capabilities. These gaps hinder high-resolution, large-domain simulations in heterogeneous media.

To address these challenges, we developed UltraWave, an open-source, multi-GPU-accelerated simulator for acoustic and elastic wave propagation in heterogeneous media. UltraWave advances the state of the art in three ways. First, it leverages Devito [37], a cutting-edge domain-specific language designed for high-performance finite-difference solutions of partial differential equations (PDEs). Devito stands out for its ability to translate symbolic equations into highly efficient C code and employ OpenMP and OpenACC for runtime execution. By leveraging Devito, UltraWave achieves computational efficiency on CPUs and scales on multi-GPU architectures. Second, UltraWave supports elastic waves, power-law frequency-dependent absorption, and perfectly matched layers for boundary handling. Third, we introduce a validation suite with precise analytical solutions for scattering signals in both the time and frequency domains leveraging existing theories [38–40]. These solutions serve as rigorous accuracy benchmarks for simulated scattering from fluid and elastic targets. Given the scarcity of comprehensive time- and frequency-domain characterizations of the accuracy of simulation tools for simulating ultrasound scattering, we address this gap by providing a standardized validation suite to facilitate such evaluations for future simulation tools.

By open-sourcing UltraWave and its validation framework (https://github.com/zixuant5/UltraWave), we provide a tool for high-fidelity, large-scale simulations while establishing quantitative accuracy benchmarks. Example code and detailed instructions are provided in the GitHub repository. In this study, we present details of UltraWave implementation and scattering validation. We validate UltraWave for several fundamental scattering scenarios, including single and multiple fluid spherical scatterers as basic models for soft tissues, and solid cylindrical and spherical scatterers that incorporate shear wave components. In addition to scattering, we assess the accuracy of UltraWave in modeling acoustic absorption. The accuracy and efficiency of UltraWave are assessed by comparing simulated signals with analytical solutions and benchmarking against the well-established k-Wave toolbox.

II. Methodology

A. Governing Equations

1). Acoustic wave simulation

For acoustic simulations with fluid media, a linear system of coupled first-order PDEs is solved with Devito. The following PDEs represent the conservation of momentum and mass, and the pressure density relation in a heterogeneous medium without attenuation and nonlinearity [41]:

$$\frac{\partial \mathbf{v}}{\partial t}=-\frac{1}{{\rho }_0}\nabla p$$ (1a)

$$\frac{\partial \rho }{\partial t}=-{\rho }_0\nabla \cdot \mathbf{v}-\mathbf{v}\cdot \nabla {\rho }_0+S_M$$ (1b)

$$\frac{\partial p}{\partial t}=c_0^2\left(\frac{\partial \rho }{\partial t}+\mathbf{v}\cdot \nabla {\rho }_0\right)$$ (1c)

where $\mathbf{v}$ is the acoustic particle velocity, $p$ is the acoustic pressure, and $\rho$ is the acoustic density. The acoustic medium is characterized by a spatially varying equilibrium density ${\rho }_0$ and sound speed $c_0$. The term $S_M$ represents a mass source field. In our implementation, $S_M$_’is computed from a user-specified pressure source $S_p$, following the convention in k-Wave, using $S_M=\frac{2S_p}{N_D{c}_r\text{Δ}x}$, where $N_D$ is number of dimensions, $c_r$ is sound speed at the source location, and $\text{Δ}x$ is the spatial resolution [5]. This source term will be omitted in subsequent equations for brevity. The $\mathbf{v}\cdot \nabla {\rho }_0$ terms in Equations (1b) and (1c) are dropped off in our implementation, following k-Wave, because they cancel each other when the two Equations are combined.

This system of PDEs is solved by using the finite-difference time-domain (FDTD) method [42]. In our implementation, we use a fourth-order spatial discretization, which requires a two-point halo in each spatial direction. Staggered grids are used to enhance numerical accuracy. The Courant-Friedrichs-Lewy (CFL) number is kept below 0.3 to ensure numerical stability.

2). Elastic wave simulation

In the context of elastic wave simulation within a heterogeneous medium, where damping effects are neglected, the fundamental equations are governed by the following set of first-order differential equations [43, 44],

$$\frac{\partial }{\partial t}{v}_i=\frac{1}{{\rho }_0}\left(\frac{\partial s_{ii}}{\partial {\xi }_i}+{\sum }_{i\ne j}\frac{\partial s_{ij}}{\partial {\xi }_j}\right)$$ (2a)

$$\frac{\partial }{\partial t}{s}_{ii}=(\lambda +2\mu )\frac{\partial v_i}{\partial {\xi }_i}+\lambda {\sum }_{i\ne j}\frac{\partial v_j}{\partial {\xi }_j}$$ (2b)

$$\frac{\partial }{\partial t}{S}_{ij}=\mu \left(\frac{\partial v_i}{\partial {\xi }_j}+\frac{\partial v_j}{\partial {\xi }_i}\right),i\ne j$$ (2c)

where $\xi =\left({\xi }_x,{\xi }_y,{\xi }_z\right)$ denotes the Cartesian coordinate directions, $i$, $j=x,y,z,\mathbf{v}=\left(v_x,v_y,v_z\right)$ is the velocity vector, and $s_{ij}$ represents the stress tensor components. The equilibrium density of the medium is denoted by ${\rho }_0(x,y,z)$, while $\lambda (x,y,z)$ and $\mu (x,y,z)$ are Lamé coefficients, which are spatially varying properties of the medium. The Lamé coefficients are related to the shear $\left(c_s\right)$ and compressional $\left(c_p\right)$ sound speeds through the relationships:

$$\mu =c_s^2{\rho }_0,\lambda +2\mu =c_p^2{\rho }_0$$ (3)

Implementing these equations within a finite difference framework involves discretizing them using a spatial staggered grid scheme.

3). Acoustic absorption

Biological tissues exhibit frequency-dependent absorption following a power-law relationship:

$$\alpha (\omega )={\alpha }_0{\omega }^y$$ (4)

where $\omega$ is the angular frequency, ${\alpha }_0$ is the absorption coefficient in Np(rad/s)^-ym⁻¹, and $y$ is the power law exponent.

There are two common approaches to modeling arbitrary frequency dependent absorption: deriving fractional wave equations [45] and using relaxation processes [8, 46]. Fractional wave equations introduce power-law absorption through fractional derivatives, offering a direct frequency-domain approach but posing numerical challenges in FDTD simulations. UltraWave uses the relaxation-process approach, modeling attenuation via phenomenological operators.

This method assumes tissue absorption mechanisms behave as relaxation processes with varying, yet unknown, relaxation times in biological materials. Thus, parameters are empirically fitted as phenomenological terms rather than representing specific physical mechanisms. Usually, two relaxation processes are sufficient to model the desired absorption across a wide frequency band [8].

To implement power-law absorption, UltraWave fits the relaxation mechanisms to the user-specified frequency-dependent absorption curve. This is achieved by selecting optimal values of ${\tau }_n$ and ${\eta }_n$ to fit the equation ([8, 46])

$$\alpha (\omega )={\sum }_{n=1}^N\frac{{\eta }_n}{2c_0{\tau }_n}\frac{{\omega }^2{\tau }_n^2}{1+{\omega }^2{\tau }_n^2},$$ (5)

where ${\tau }_n$ is the characteristic relaxation time and ${\eta }_n$ is the relaxation parameter associated with the n-th order process [41]. $N=2$ was used in our implementation.

Acoustic simulations considering absorption are based on the following equations [46]:

$$\frac{\partial \mathbf{v}}{\partial t}=-\frac{1}{{\rho }_0}\nabla p$$ (6a)

$$\frac{\partial p}{\partial t}+p{\sum }_{n=1}^N\frac{{\eta }_n{c}_0^2}{{\tau }_n{c}_{\infty }^2}+{\sum }_{n=1}^N\left(\frac{{\eta }_n{c}_0^2}{{\tau }_n{c}_{\infty }^2}-\frac{1}{{\tau }_n}\right)S_n=-{\rho }_0{c}_{\infty }^2\nabla \cdot \mathbf{v}$$ (6b)

$$\frac{\partial S_n}{\partial t}=-\frac{1}{{\tau }_n}{S}_n+\frac{{\eta }_n{c}_0^2}{{\tau }_n{c}_{\infty }^2}\left(p+{\sum }_{n=1}^N{S}_n\right)$$ (6c)

where the acoustic medium is characterized by a spatially varying density ${\rho }_0$ and sound speed $c_0$, $S_n$ is a defined state variable for the n-th order relaxation process, and $c_{\infty }$ is the sound speed in the high frequency limit defined as $c_{\infty }^2=c_0^2\left(1+{\sum }_{n=1}^N{\eta }_n\right)$.

4). Perfectly matched layer

In FDTD simulations, accurately truncating the finite simulation domain is essential for effectively modeling the infinite space. Without proper boundary conditions, boundaries would act as rigid ends, causing significant reflections on boundaries that adversely affect the simulation outcomes. To address this issue, the perfectly matched layer (PML) technique [47] has been integrated into UltraWave. This technique applies a modified, non-physical set of equations in the PML region to absorb outgoing waves and prevent reflections.

The implementation of PML requires the split-field formulation of the PDEs, i.e., dividing acoustic density into Cartesian components, expressed as $\rho ={\sum }_{\xi }{\rho }_{\xi }$. The first-order equations for the acoustic simulations are [5]

$$\frac{\partial }{\partial t}{v}_{\xi }=-\frac{1}{{\rho }_0}\frac{\partial p}{\partial \xi }-{\alpha }_{\xi }{v}_{\xi }$$ (7a)

$$\frac{\partial }{\partial t}{\rho }_{\xi }=-{\rho }_0\frac{\partial v_{\xi }}{\partial \xi }-{\alpha }_{\xi }{\rho }_{\xi }$$ (7b)

$$p=c_0^2{\sum }_{\xi }{\rho }_{\xi }$$ (7c)

where $\xi$ denotes the dimensions $(x,y,z)$, and $\alpha =\left\{{\alpha }_x,{\alpha }_y,{\alpha }_z\right\}$ is the anisotropic absorption in Nepers per second. These components are zero outside the PML and nonzero only within the PML layer perpendicular to their respective directions. For example, in a PML perpendicular to the x-axis, $\alpha =\left\{{\alpha }_x,0,0\right\}$.

The value of ${\alpha }_{\xi }$ as a function of the corresponding coordinate $r_{\xi }$ is determined by [48]

$${\alpha }_{\xi }={\alpha }_{\max }{\left(\frac{r_{\xi }-r_{\xi }^{start}}{r_{\xi }^{end}-r_{\xi }^{start}}\right)}^4$$ (8)

where $r_{\xi }^{start}$ and $r_{\xi }^{end}$ denote the start (inner) and end (outer) coordinates of the PML, respectively.

Because the fundamental equations of elastic wave simulation already use the split-field formulation, equations of PML implementation are more straightforward:

$$\frac{\partial }{\partial t}{v}_i=\frac{1}{{\rho }_0}\left(\frac{\partial s_{ii}}{\partial {\xi }_i}+{\sum }_{i\ne j}\frac{\partial s_{ij}}{\partial {\xi }_j}\right)-{\alpha }_i{v}_i$$ (9a)

$$\frac{\partial }{\partial t}{s}_{ii}=(\lambda +2\mu )\frac{\partial v_i}{\partial {\xi }_i}+\lambda {\sum }_{i\ne j}\frac{\partial v_j}{\partial {\xi }_j}-{\alpha }_i{s}_{ii}$$ (9b)

$$\frac{\partial }{\partial t}{s}_{ij}=\mu \left(\frac{\partial v_i}{\partial {\xi }_j}+\frac{\partial v_j}{\partial {\xi }_i}\right)-\left({\alpha }_i+{\alpha }_j\right)s_{ij},i\ne j$$ (9c)

B. UltraWave Toolbox

The UltraWave toolbox offers a user-friendly interface for simulations. It uses a modular architecture consisting of four well-defined components.

1. Model contains the simulated medium information and the grid setting. Medium defines the compressional wave speed, shear wave speed, and density of the simulated domain. Grid defines a regular Cartesian grid with the specified spacing and shape.

2. Source describes the source of the wavefield, which specifies the number and position of source points and the time-varying pressure source signals.

3. Receiver defines the position of sensors placed within the medium.

4. Operator acts as the computational engine of UltraWave. It processes the model, source, and receiver configurations, using Devito to solve wave equations for 2-D and 3-D acoustic and elastic simulations. UltraWave offers four pre-defined operators, Acoustic2DOperator, Acoustic3DOperator, Elastic2DOperator, and Elastic3DOperator, catering to different simulation scenarios.

C. Scattering Theory

To rigorously validate simulation accuracy, we developed a companion code suite that numerically implements analytical solutions from scattering theory for several standard scenarios, including scattering from fluid spheres, scattering from an infinitely long solid cylinder, and scattering from a solid sphere. Both the time-domain and frequency-domain expressions of the analytical solutions were implemented based on those established in the literature [38–40]. To compare the simulation results with the analytical solutions, we performed the simulations to yield the time-domain signals, and performed the Fourier transform on the time-domain signals to obtain the frequency-domain results.

For fluid sphere scattering, we consider a fluid sphere with a radius $a$, a density ${\rho }_1$, and a sound speed $c_1$, located at the origin of a spherical coordinate system $(r,\theta ,\phi )$. This spherical scatterer is surrounded by a background fluid with a density ${\rho }_2$ and a sound speed $c_2$. An incident plane acoustic wave travels parallel to the polar axis in the –$z$ direction and impinges upon this sphere, as depicted in Fig. 1a. By choosing the wave direction parallel to the polar axis, the dependence on the $\phi$ coordinate is eliminated, simplifying the analysis to consider only the $r$ and $\theta$ variables. The scattered wave solution is derived as described in [38]. Scattering from multiple fluid spheres was also considered (Fig. 1b).

Fig. 1.

Schematic representations of various scattering scenarios. (a) A single spherical scatterer positioned at the coordinate center with a plane wave approaching from the z-direction. (b) Multiple spherical scatterers randomly distributed within a cubic volume at the coordinate center, with a plane wave approaching from the z-direction. (c) An infinitely long cylindrical scatterer aligned with the y-axis, with a plane wave approaching from the z-direction towards the scatterer. The grey parallelogram represents a two-dimensional cross-section on the x-z plane.

Similarly, for an infinitely long solid cylinder with a radius $a$, a compressional wave speed $c_1$, a shear wave speed $c_2$, and a density ${\rho }_1$, located along the y-axis of a rectangular coordinate system (Fig. 1c), the scattering of a plane wave in a surrounding fluid medium (a compressional wave speed $c_3$ and a density ${\rho }_3$) is analyzed. The analytical solutions for the scattered wave follow the equations in [39].

The scattering by an isotropic solid sphere, located at the origin of a spherical coordinate system, under the incidence of a plane wave propagating along the –$z$ direction, is modeled using a similar coordinate framework (Fig. 1a). The analytical solutions are derived based on [39, 40].

III. Results

A. Fluid Sphere Scattering

1). Single fluid sphere

A single fluid sphere was placed within a fluid medium. The sphere had a sound speed of 1520 m/s, a density of 1050 kg/m³, and a radius of 0.25 mm. It was immersed in a background medium with a sound speed of 1500 m/s and a density of 1000 kg/m³. A planar source emitted plane waves toward the sphere as shown in Fig. 1a, with a central frequency of 2 MHz and a −6 dB bandwidth of 80%. The planar source was sufficiently wide to avoid the influences of waves from the source edge. Point sensors were positioned uniformly 3 mm from the sphere to record signals from various angles. These sensors captured both incident and scattered waves. To isolate the scattered signals, a reference simulation without the fluid spherical scatterer was conducted under identical settings, and the reference signals were subtracted from the simulated signals.

Fig. 2 compares the simulated and theoretical scattered signals at different angles ($\theta =0^{\circ }$, 50°, 90°, and 180°). A spatial resolution of 50 μm and a temporal resolution of 2 ns were used in the simulation. Both UltraWave and k-Wave closely matched the theoretical signals in amplitude and phase across all angles, with minor differences in the backscattered signal $\left(\theta =0^{\circ }\right)$ attributed to the spatial and temporal discretization. For the backscattered signals in the time domain, the mean absolute error (MAE) was 1.8% for the k-Wave simulations and 2.0% for the UltraWave simulations compared with the theoretical signals, normalized by the maximum absolute value of the theoretical signal. The normalized MAE is defined as:

$$\text{nMAE}=\frac{1}{\max (|T|)}\frac{{\sum }_{i=1}^N\left|S_i-T_i\right|}{N}$$ (10)

Fig. 2.

Comparison of k-Wave and UltraWave simulation results with the analytical solution for scattered pressure signals from a single fluid sphere at observation angles of (a) $\theta =0°$, (b) $\theta =50°$, (c) $\theta =90°$, and (d) $\theta =180°$. Simulations were performed using a spatial resolution of 50 μm.

where $S_i$ represents the simulated signal value, $T_i$ represents the theoretical signal value, and $\max (|T|)$ is the maximum absolute value of the theoretical signal.

In the time domain, the maximum values of the normalized cross-correlation between the k-Wave and theoretical signals and between the UltraWave and theoretical signals were both 0.99, occurring at a time lag of 2 ns. The maximum normalized cross-correlation is defined as:

$${\max }_{\tau }\left(\frac{{\sum }_i{S}_i{T}_{i+\tau }}{\sqrt{{\sum }_i{\left(S_i-\overline{S}\right)}^2{\sum }_i{\left(T_i-\overline{T}\right)}^2}}\right)$$ (11)

where $\tau$ is the time lag, $\overline{S}$ is the mean of simulated signal, and $\overline{T}$ is the mean of theoretical signal.

These results indicate that UltraWave and k-Wave have a similar accuracy. Notably, UltraWave demonstrated higher computational efficiency than k-Wave: on the same CPU (AMD EPYC 7302 16-Core Processor), the k-Wave simulation time was 80 minutes without C++ optimization and 30 minutes with C++ optimization, while UltraWave simulation time was 20 minutes; on the same GPU (NVIDIA RTX A6000), the k-Wave simulation time was 60 seconds, while UltraWave simulation time was 35 seconds.

The impact of spatial resolution on simulation accuracy was also investigated using this simulation setup, where two resolutions were evaluated: 25 μm and 50 μm, corresponding to 30.4 and 15.2 grid points per smallest wavelength, respectively. A high temporal resolution of 2 ns was used to ensure its non-influence on accuracy, corresponding to CFL numbers of 0.12 and 0.06, respectively. The backscattering configuration $\left(\theta =0^{\circ }\right)$ was used. The simulations (Fig. 3) showed a closer match to the theoretical signal at a finer spatial resolution (25 μm) for both k-Wave and UltraWave simulations, particularly in the discrepancy observed around the 5 μs trough in the time domain (Figs. 3a and 3c). The normalized MAE decreased to 0.8% for k-Wave and to 1.0% for UltraWave at a spatial resolution of 25 μm. In the frequency domain (Figs. 3b and 3d), the two peaks of simulated signals were closer to the theoretical signals at a spatial resolution of 25 μm. This outcome aligns with expectations: finer spatial resolution yields greater accuracy. However, the simulated results did not fully converge to the theoretical solutions at 25-μm resolution. Therefore, we performed an additional simulation at a spatial resolution of 10 μm. Multiple GPUs were needed for this simulation because of the memory requirement. Therefore, this simulation was performed using UltraWave only as k-Wave does not yet support the use of multiple GPUs. The UltraWave-simulated result showed excellent agreement with the theoretical solution (Figs. 3e-f), with the normalized MAE reduced to 0.1%. This result demonstrated the value of UltraWave for high-fidelity simulations.

Fig. 3.

Backscattered signals from a single fluid sphere compared among analytical solution, k-Wave simulations and UltraWave simulations at a spatial resolution of 50 μm in the (a) time domain and (b) frequency domain, and a spatial resolution of 25 μm in the (c) time domain and (d) frequency domain. The k-Wave signals and UltraWave signals virtually show no difference in (a-d). UltraWave simulations at a 10 μm spatial resolution, compared with the analytical solutions, are shown in the (e) time and (f) frequency domain.

Furthermore, we tested a range of scatterer radii from 0.1 mm to 1 mm with the 25 μm spatial resolution. UltraWave maintained the normalized MAE consistently around 1%, which on average is about 0.2% higher than k-Wave, confirming its accuracy across varying scatterer sizes.

In addition to accuracy, memory usage is an important factor to consider when selecting spatial resolution. The theoretical memory usage for a simulation with a grid size of $N_x\times N_y\times N_z$ can be estimated as:

$$\text{Memory}[\text{MB}]\approx \frac{\left(N_x+SO\right)\left(N_y+SO\right)\left(N_z+SO\right)\times 2A\times 4}{{1024}^2}+M_{I/O},$$ (12)

where the symbol $SO$ represents the spatial order used in the FDTD algorithm, the symbol $A$ represents the number of primary simulation variables (e.g., $A=7$ for 3-D acoustic simulations and $A=9$ for 3-D elastic simulations), the factor of 2 before $A$ accounts for storing variables at two time steps, the factor of 4 accounts for the 4 bytes required to store each value in float32 format, and $M_{I/O}$ represents the memory required for input models (e.g., sound speed, density) and recorded outputs. As shown in Fig. 4, CPU memory usage closely follows this prediction. On GPUs, however, Devito stores intermediate results to optimize performance, which increases memory usage by up to 2× the theoretical estimate for acoustic simulations and up to 4× for elastic simulations. Assuming each GPU has 48 GB of memory and using four GPUs, the maximum feasible domain size is approximately 2³¹ grid points (e.g., 1024×1024×2048) for 3-D acoustic simulations and approximately 2²⁹ grid points (e.g., 1024×1024×512) for 3-D elastic simulations.

Fig. 4.

Memory usage as a function of the total number of grid points in 3-D acoustic simulations, comparing theoretical predictions with CPU and GPU measurements in UltraWave.

The use of multiple GPUs not only enables larger 3-D simulations, but also improves computational efficiency. For example, a 3-D acoustic simulation with 2²⁷ grid points (e.g., 512×512×512) ran in 183 seconds on a single GPU (NVIDIA A40). The runtime was reduced to 89 seconds with two GPUs, 122 seconds with three GPUs, and 46 seconds with four GPUs. The slightly increased runtime with three GPUs relative to two GPUs was likely due to increased communication overhead and suboptimal load balancing, which are minimized in even splits (e.g., between two or four GPUs).

2). Multiple fluid spheres

To evaluate the capability of UltraWave in modeling complex scattering environments, we simulated 10 spheres, each with a radius of 0.25 mm, randomly distributed within a 3 mm × 3 mm × 3 mm cubic volume centered at the coordinate origin (Fig. 1b). A plane wave with a central frequency of 2 MHz and a −6 dB bandwidth of 80% traveled through these spheres, and a point sensor was placed 3 mm above the center of the cube to detect the backscattered signal. The spatial resolution of simulations was 50 μm and temporal resolution was 2 ns. Multiple scattering effect was automatically considered in the simulation, but not modeled in the analytical solution, which allowed us to assess the significance of the multiple scattering effect and could provide insights into its consideration in acoustic model construction.

Three different acoustic impedance values of the spheres were evaluated, while keeping the background medium at the same density (1000 kg/m³) and sound speed (1500 m/s). In the first case, with spheres having an acoustic impedance similar to that of the background medium (density 1005 kg/m³ and sound speed 1502 m/s), it was hypothesized that the multiple scattering effect was negligible. Fig. 5a confirms this, showing the simulated signal closely aligned with the theoretical signal with minor differences. In the second case, where spheres had a slightly higher impedance contrast (density of 1050 kg/m³ and sound speed of 1520 m/s), the discrepancies between the simulated and theoretical signals (Fig. 5b) were also small, suggesting minimal multiple scattering effect. In the third case, involving spheres with a high impedance contrast (density of 1100 kg/m³ and sound speed of 1700 m/s), more noticeable differences between the simulated and theoretical signals were observed (Fig. 5c) due to multiple scattering. This aligns with the expectation that larger impedance contrasts introduce stronger multiple scattering effects. The difference was minimal in the initial portion of the scattered signal (e.g., before 4 μs; Fig. 5c) but increased in the subsequent portion (e.g., after 4 μs), consistent with the expected behavior of multiple scattering. Overall, these results not only demonstrate the accuracy of UltraWave in scenarios with multiple scatterers, but also offer valuable guidance on when the multiple scattering effect should be considered in acoustic modeling.

Fig. 5.

Backscattered signals from 10 fluid spheres for various acoustic impedances: (a) $c_1=1502$ m/s, ${\rho }_1=1005$ kg/m³; (b) $c_1=1520$ m/s, ${\rho }_1=1050$ kg/m3; and (c) $c_1=1700$ m/s, ${\rho }_1=1100$ kg/m³; showing results from the analytical solution and UltraWave simulations.

B. Infinitely Long Solid Cylinder Scattering (2-D Disc Model)

This example concerns scattering from a solid cylinder of infinite length immersed in a fluid medium. Scattering from solid material involves propagation of both compressional and shear waves. To efficiently investigate this scenario, 2-D simulations were performed using a disc scatterer and a line source, which represent a cross-sectional view of the corresponding three-dimensional case of an infinitely long cylinder and a plane source (Fig. 1c). This 2-D configuration is commonly used in the literature [49, 50] because the analytical solution for scattering from a disc under line source incidence is equivalent to the solution of the 3-D scattering from an infinite long cylinder under plane wave incidence.

In our simulations, the scatterer was defined with a density of 1960 kg/m³, a compressional wave speed of 4030 m/s, a shear wave speed of 1645 m/s, and a radius of 2 mm. The surrounding medium had a density of 1000 kg/m³, a compressional wave speed of 1500 m/s, and a shear wave speed of 0 (due to its fluid nature). A plane wave was transmitted toward the scatterer, and the backscattered signal was then captured by a point sensor placed 4 mm above the scatterer. 2-D elastic wave simulations were conducted with both k-Wave and UltraWave. The temporal resolution was kept at 0.6 ns. Figs. 6 and 7 compare the simulated and theoretical signals backscattered from the disc scatterer at central frequencies of 0.75 MHz and 2 MHz, respectively. These figures present the results of simulations in both time and frequency domains.

Fig. 6.

Backscattered signals from a 2-D disc model representing an infinitely long solid cylinder with a compressional wave speed of 4030 m/s, shear wave speed of 1645 m/s, and density of 1960 kg/m³, comparing analytical solution and simulation results from k-Wave and UltraWave. Displayed are signals in the (a) time domain and (b) frequency domain for k-Wave at a spatial resolution of 25 μm; (c) time domain and (d) frequency domain for UltraWave at a spatial resolution of 25 μm; and (e) time domain and (f) frequency domain for UltraWave at a spatial resolution of 5 μm, when the central frequency of the incident wave is 0.75 MHz.

Fig. 7.

Backscattered signals from a 2-D disc model representing an infinitely long solid cylinder with a compressional wave speed of 4030 m/s, shear wave speed of 1645 m/s, and density of 1960 kg/m³, comparing analytical solution and simulation results from k-Wave and UltraWave. Displayed are signals in the (a) time domain and (b) frequency domain for k-Wave at a spatial resolution of 25 μm; (c) time domain and (d) frequency domain for UltraWave at a spatial resolution of 25 μm; and (e) time domain and (f) frequency domain for UltraWave at a spatial resolution of 5 μm, when the central frequency of the incident wave is 2 MHz.

The influence of shear waves on the scattered signals was observed in both theory and simulations (Figs. 6 and 7). In theory, when shear waves are considered, a distinctive “ringing” effect appears in the time-domain signals as small waves trailing the main reflected wave. In the frequency domain, this results in additional deep troughs. This phenomenon is common in scattering from solid scatterers and is attributed to the normal modes of free vibration within the scatterer [39]. These modes depend on the material and geometric properties of the scatterer, such as compressional wave speed, shear wave speed, and its size. The frequency content of the incident wave also plays a key role. Higher-frequency waves excite a broader set of normal modes, leading to more complex spectral patterns.

In simulations with a central frequency of 0.75 MHz and a 25 μm resolution (maximum CFL number, 0.097), the main reflected waves, predominantly generated by compressional waves, were accurately replicated in k-Wave (Fig. 6a) and UltraWave (Fig. 6c), including the observable “ringing” effect in the time domain. UltraWave appeared to outperform k-Wave in better agreeing with the theoretical signal in the time domain, particularly from 10 μs to 20 μs, and in the frequency domain, particularly from 1 to 1.5 MHz. The normalized MAE in the time domain from 0 to 30 μs is 4.7% for k-Wave and 3.6% for UltraWave. The time-domain maximum normalized cross-correlation between the k-Wave signal and the theoretical signal is 0.92, with a corresponding time lag of 19.8 ns. The time-domain maximum normalized cross-correlation between the UltraWave signal and the theoretical signal is 0.95, with a corresponding time lag of 25.2 ns.

In simulations with a central frequency of 2 MHz, both k-Wave and UltraWave signals agreed reasonably well with the theoretical signal in the time domain (Fig. 7a and 7c) and in the frequency domain (Figs. 7b and 7d). In the time domain, the normalized MAE was 3.5% at a 25 μm spatial resolution for both simulators. The time-domain maximum normalized cross-correlation between the k-Wave signal and the theoretical signal was 0.91, with a corresponding time lag of 12.0 ns. The time-domain maximum normalized cross-correlation between the UltraWave signal and the theoretical signal was 0.83, with a corresponding time lag of 10.2 ns. UltraWave had a higher computational efficiency compared with k-Wave: On the same CPU (AMD EPYC 7302 16-Core Processor), UltraWave completed this simulation in 1 hour and 30 minutes, whereas k-Wave required 3 hours and 2 minutes. On the same GPU (NVIDIA RTX A6000), UltraWave took only 1 minute, while k-Wave needed 10 minutes due to the lack of a specific GPU-optimized function for elastic wave simulations.

Unlike the fluid simulation example, the accuracy of this solid simulation example was suboptimal for the spatial resolution of 25 μm, primarily due to two factors. First, the higher compressional and shear wave speeds of the disc scatterer led to shorter wavelengths, thereby reducing the number of points per wavelength (PPW). Second, accurately capturing the normal modes of a solid scatterer requires a higher spatial resolution. Therefore, when the spatial resolution was refined to 5 μm, the MAE of UltraWave simulations decreased to 1.8% for the central frequency of 0.75 MHz, and 2.1% for the central frequency of 2 MHz. The maximum normalized cross-correlation increased to 0.99 for 0.75 MHz, and 0.94 for 2 MHz. The corresponding time lag decreased to 0.6 ns for 0.75 MHz, and 7.2 ns for 2 MHz. At this finer resolution, the simulations more accurately reproduced the “ringing” effect in the time domain and showed greater consistency with the theoretical signals (Figs. 6e-f and 7e-f). However, despite these advancements, some of the more abrupt troughs in the frequency domain are not fully resolved, suggesting that even finer spatial resolutions may be necessary to capture all aspects of the scatterer response.

Fig. 8 shows the nMAEs versus PPW for UltraWave and k-Wave with a central frequency of 0.75 MHz using the same time step. The PPW was calculated based on the shear wave speed because the shear wave contributes to the “ringing” effect. The k-Wave curve in Fig. 8 shows slightly lower MAE than UltraWave at low PPW below 7, but both methods have relatively large errors in this range, and their accuracy becomes comparable as PPW increases beyond 10. Based on Fig. 8, the PPW should be greater than 50 to have a normalized MAE of less than 5%, showing a much higher requirement than acoustic simulations.

Fig. 8.

Normalized mean absolute errors (MAEs) versus points per wavelength (PPW) of the shear wave for UltraWave and k-Wave with a central frequency of 0.75 MHz.

These results demonstrate the capability of UltraWave to model complex elastic wave phenomena, including shear-induced normal modes, while showing the trade-offs between resolution, accuracy, and computational cost. The GPU acceleration and scalability of this tool address these challenges, enabling high-fidelity simulations of elastic scattering in biomedical applications.

C. Solid Sphere Scattering

To further validate the capability of UltraWave in 3-D elastic wave modeling, we simulated a solid sphere in a fluid medium. The sphere had a density of 1960 kg/m³, a compressional wave speed of 4030 m/s, a shear wave speed of 1645 m/s, and a radius of 0.5 mm. The surrounding fluid medium was defined by a density of 1000 kg/m³, a compressional wave speed of 1500 m/s, and zero shear wave speed. A planar source generated plane waves directed at the sphere (Fig. 1a) with a central frequency of 2 MHz. A point sensor, positioned 1.5 mm above the sphere, captured the backscattered signals. The spatial resolution was 10 μm, the temporal resolution was 0.6 ns, and the maximum CFL number was 0.24.

The simulation was performed using both UltraWave and k-Wave, with the results shown in Fig. 9 for both the time and frequency domains. In the time domain, the UltraWave-simulated signal aligns well with the analytical solution, showing a slight peak difference around 4 μs. The k-Wave signal presents some differences in magnitude and exhibits more subtle wave deviations. The nMAE in the time domain is 2.9% for UltraWave and 4.3% for k-Wave. In the frequency domain, the differences are more noticeable, showing some misalignments in magnitude and shifts in peak and trough positions. These variations can be attributed to the simulation’s spatial resolution, which is important for accurately capturing elastic wave scattering phenomena.

Fig. 9.

Backscattered signals from a solid sphere with a compressional wave speed of 4030 m/s, shear wave speed of 1645 m/s, and density of 1960 kg/m³, comparing analytical solution and simulation results from UltraWave and k-Wave at a spatial resolution of 10 μm, when the central frequency of the incident wave is 2 MHz. Displayed are signals in the (a) time domain and (b) frequency domain from UltraWave; (c) time domain and (d) frequency domain from k-Wave.

Despite the considerable memory requirements of this 3-D elastic wave simulation, UltraWave effectively completed the task in 20 minutes using four NVIDIA A40 GPUs, each with 48 GB of memory. k-Wave, limited to a single GPU as multi-GPU capability is not supported, took around 8 hours. This comparison demonstrates the scalability and efficiency of UltraWave, particularly for modeling 3-D elastic scattering in complex media.

D. Acoustic Absorption

To assess the ability of UltraWave to correctly simulate acoustic absorption, we investigated the propagation of a plane wave through a homogeneous absorbing medium. The relaxation parameters were calculated to fit the absorption laws with power-law exponents of 1.5 and 2. Figs. 10a and 10d show the ideal absorption laws and the corresponding fitted relaxation mechanisms, which demonstrate consistency across a wide frequency band from 0 to 12 MHz. To further validate the simulation results, we compared the simulated signals with those predicted using the material impulse response function (MIRF), i.e., the inverse Fourier transform of the material transfer function that describes the combined effects of absorption and dispersion on pulse propagation [51]. In the simulation, the transmitted signal had a central frequency of 2 MHz, and the propagation distances were set at 1.35 mm, 6.75 mm, and 13.5 mm. The spatial and temporal resolutions were 50 μm and 0.6 ns, respectively. The homogeneous medium had a speed of sound of 1500 m/s, a density of 1000 kg/m³, and an absorption coefficient of 0.75 Np(rad/s)^−ym⁻¹. The MIRF-predicted results were obtained by directly convolving the transmitted signal with the MIRF. The UltraWave-simulated signals showed excellent agreement with the MIRF-predicted (Figs. 10b and 10d). UltraWave had a maximum nMAE of 0.63% for $y=1.5$ and 0.57% for $y=2$. Minor discrepancies, likely stemming from residual fitting inaccuracies in the relaxation parameters, did not compromise the overall accuracy.

Fig. 10.

Simulation of a plane wave pulse propagating through an absorbing medium with ${\alpha }_0=0.75$ Np(rad/s)^−ym⁻¹ with $y=1.5$ and 2. (a) For $y=1.5$, the ideal absorption laws are presented alongside the fitted curve derived from relaxation mechanisms in UltraWave. (b) MIRF-predicted and UltraWave-simulated pressure pulses at distances of 1.35 mm, 6.75 mm, and 13.5 mm. (c), (d) Corresponding results for $y=2$.

IV. Discussion

In this study, we introduced UltraWave, an open-source multi-GPU-accelerated simulator for both acoustic and elastic wave simulations in 3-D heterogeneous media. Built on Devito, a domain specific language for automated finite difference code generation, UltraWave achieves scalable performance across multiple GPUs while maintaining high accuracy. Validation against analytical solutions and benchmarking against k-Wave demonstrate the scalability, accuracy and efficiency of UltraWave, achieving a 24-fold speedup over k-Wave in an elastic wave simulation with nMAEs below 3%, by leveraging 4 GPUs for acceleration compared with a single GPU used for k-Wave. While k-Wave is a MATLAB-based simulator that supports elastic wave simulations through the Parallel Computing Toolbox, it does not offer a GPU-optimized implementation for elastic waves. UltraWave, built in Python, is both free to use and designed for efficient multi-GPU performance. These capabilities address a critical gap in open-source tools, enabling large-scale simulations for applications such as transcranial ultrasound and quantitative ultrasound biomarker development.

Another key contribution of this work is the validation suite for scattering from basic geometries, including fluid spheres, infinitely long solid cylinders, and solid spheres. While direct experimental validation is not included, the analytical benchmarks used in this study are based on models that have been experimentally verified in previous work [39, 40, 52]. Our validation framework enables quantitative accuracy assessment in both time and frequency domains, which allows developers of new simulators to rigorously evaluate and refine their tools for applications in scattering simulation. This validation can also guide users in selecting appropriate simulation settings, such as a PPW greater than 50 for elastic waves, to balance accuracy and computational cost. Furthermore, our analysis of multiple scattering effects provides practical insights. As demonstrated in Section III, simulations with scatterers of low impedance contrast aligned closely with single-scattering theory, whereas high impedance contrast showed deviations due to multiple scattering. This distinction is critical for modeling tissues such as calcified plaques or tumors, where scattering complexity varies with material properties.

Elastic wave simulations pose unique challenges. Our results showed that the presence of shear waves significantly changes the scattered signals in both time and frequency domains, suggesting the necessity of accurate elastic wave simulation when shear wave speed is non-negligible. Our results also demonstrated that high resolutions are needed to achieve high accuracy of elastic wave simulations, which highlights the need for a scalable computing framework. The multi-GPU capability of UltraWave addresses this demand, completing 3-D elastic simulations in minutes, compared with hours when using tools that do not support multiple GPUs.

UltraWave currently focuses on acoustic and elastic wave propagation in heterogeneous media without accounting for nonlinearity. Similar to k-Wave, our current study assumes isotropic elasticity, which does not capture direction-dependent stiffness and therefore does not support full elastic anisotropy [53]. Future enhancements will aim to integrate these aspects, offering a more comprehensive simulation tool. Given the open-source nature of our code, the research community is invited to adapt and extend our work to meet their specific research needs.

V. Conclusion

In conclusion, UltraWave, an open-source multi-GPU-accelerated ultrasound simulation tool, offers an accurate, efficient, and scalable solution for acoustic and elastic wave simulations. The accompanying validation suite provides analytical solutions for basic scattering problems and enables rigorous quantitative benchmarking of simulation accuracy. By combining scalable performance with robust physics modeling, UltraWave addresses the growing demand for high-fidelity simulations.

Highlights.

• UltraWave is an open-source, multi-GPU-accelerated full-wave simulator incorporating advanced capabilities including media inhomogeneity, elastic wave propagation, power-law frequency-dependent absorption, and perfectly matched layers.

• UltraWave has been rigorously validated using a companion code suite that computes the scattered waveforms in both time and frequency domains using analytical solutions from scattering theory.

• UltraWave provides an accurate, efficient, and scalable solution for computationally intensive ultrasound simulations.

Biographies

Zixuan Tian (Graduate Student Member, IEEE) received the B.S. degree in electronic information science and technology from Fudan University, Shanghai, China, in 2021. She is currently pursuing the Ph.D. degree with the Department of Electrical and Computer Engineering, University of Illinois Urbana–Champaign, Urbana, IL, USA.

Her current research interests include deep learning in ultrasound, numerical ultrasonic simulation, and ultrasound brain imaging.

Yun Jing, (Senior Member, IEEE) photograph and biography not available at the time of publication.

Aiguo Han (Member, IEEE) received the B.S. degree in Acoustics from Nanjing University, Nanjing, China, in 2008, and the M.S. and Ph.D. degrees in Electrical and Computer Engineering from the University of Illinois Urbana-Champaign, Urbana, IL, USA, in 2011 and 2014, respectively.

He was a Research Assistant Professor with the Department of Electrical and Computer Engineering, University of Illinois Urbana–Champaign, Urbana, IL, USA. He is currently an Assistant Professor with the Department of Biomedical Engineering, Virginia Polytechnic Institute and State University, Blacksburg, VA, USA. His current research interests include biomedical ultrasound imaging, transcranial ultrasound, quantitative ultrasound, and machine learning in ultrasound imaging and diagnostics.

Dr. Han was a recipient of the National Institutes of Health (NIH)/National Institute of Biomedical Imaging and Bioengineering (NIBIB) Trailblazer Award and the American Institute of Ultrasound in Medicine New Investigator Award in Basic Science. He is a member of the Institute of Electrical and Electronics Engineers, a Full Member of the Acoustical Society of America, and a Fellow of the American Institute of Ultrasound in Medicine.