Assessing the Robustness of Frequency-Domain Ultrasound Beamforming Using Deep Neural Networks

Abstract

We study training deep neural network (DNN) frequency-domain beamformers using simulated and phantom anechoic cysts and compare to training with simulated point target responses. Using simulation, physical phantom, and in vivo scans, we find that training DNN beamformers using anechoic cysts provided comparable or improved image quality compared with training DNN beamformers using simulated point targets. The proposed method could also be adapted to generate training data from in vivo scans. Finally, we evaluated the robustness of DNN beamforming to common sources of image degradation, including gross sound speed errors, phase aberration, and reverberation. We found that DNN beamformers maintained their ability to improve image quality even in the presence of the studied sources of image degradation. Overall, the results show the potential of using DNN beamforming to improve ultrasound image quality.

I. Introduction

ULTRASOUND imaging continues to be one of the most commonly used imaging modalities because it is inexpensive, portable, has good soft tissue contrast and is capable of real-time imaging. However, B-mode ultrasound images are frequently corrupted by multiple sources of image degradation. For example, cardiac imaging quality can be impeded by strong off-axis scattering from the ribs or lungs [1], [2]. Abdominal imaging can be degraded by superficial layers of fat that cause reverberation clutter to mask the imaging region of interest [3].

Because improved B-mode ultrasound image quality could have clinical impact on many applications, the development of advanced beamforming methods continues to be an active area of research. Examples of advanced beamforming methods that have been developed for ultrasound imaging include the coherence factor [4], [5] and generalized coherence factor [6], minimum variance (MV) beamforming [7]–[9], phase coherence imaging [10], short-lag spatial coherence imaging (SLSC) [11], [12], aperture domain model image reconstruction (ADMIRE) [13]–[16], compressive sensing methods [17], filtered delay multiply and sum (F-DMAS) [18], and frequency space prediction filtering (FXPF) [19].

Recently, there has been growing interest in using deep neural networks (DNNs) for ultrasound beamforming [20]–[28]. The DNN beamformer that we developed utilizes DNNs to filter aperture domain signals in the frequency domain [20], [23]. We showed that it was possible to train DNN beamformers using simulation-based training data and to increase contrast and contrast-to-noise ratio (CNR) in physical phantom and in vivo scans. We explored several methods for studying DNN beamformer operation and found that the DNNs appeared to adjust the beam based on the input signal [23]. We studied the method for selecting DNN beamformers and found that loss, which is normally used for selecting a model, was a limited predictor of image quality [29]. Instead, we proposed using simulated image quality as a predictor for in vitro and in vivo image quality and as a method for model selection [30]. We also showed that DNN beamformers were robust to noise and conducted an initial study of their robustness to other sources of image degradation, including gross sound speed errors and phase aberration [29], [31].

Previously, we found that image quality improvements plateaued at about 10 000 training examples when training with point targets (see [30, Fig. 12]), which motivates exploring new methods for improving image quality with DNN beamformers. The deep learning community has identified three methods for improving the performance of deep learning models: 1) search for improved model architectures; 2) creating larger training data sets; and 3) scaling computation (i.e., training larger models) [32]. In this work, we focus on creating larger and more diverse training data sets for training DNN beamformers. Toward this end, we study anechoic cyst-based training data using simulated and physical phantom anechoic cysts and compare it to training with simulated point targets [29]. This study was also set up to provide insight into data mismatch issues for training DNN beamformers (e.g., training with simulated data and testing on phantom scans). In addition, we expand on our previous study of DNN beamformer robustness to gross sound speed and phase aberration and also study the effect of reverberation clutter on DNN beamforming [31], [33].

Fig. 12.

DAS and DNN beamforming images for anechoic cysts simulated using different amounts of phase aberration. The FWHM for each aberration profile was 5 mm. Images shown with 60-dB dynamic range.

II. Methods

The DNN beamformer that we study in this article operates in the frequency domain [30]. Channel data were converted from the time to the frequency domain using a short-time Fourier transform (STFT). The gate length for the STFT window was 16 samples (one pulselength).

A set of DNNs constituted a DNN beamformer in this work. A separate DNN was trained for each discrete Fourier transform (DFT) bin, and a frequency-specific DNN was trained for each DFT bin. Fig. 1 shows a diagram of the frequency-domain processing by a DNN beamformer.

Fig. 1.

Diagram showing frequency-domain processing by a DNN beamformer. k is the frequency index for each DNN. The number of elements in the receive subaperture was N. The gated data for the nth channel were s_n(t). The input signals were a single-gated depth of channel data. A DFT transformed each channel signal into the frequency domain. $\Re$ indicates the real component and $\mathfrak{J}$ indicates the imaginary component. Processed frequency-domain data are transformed back into the time domain using an inverse discrete Fourier transform (IDFT) [34].

A. Neural Networks

The DNNs were fully connected feedforward multilayer networks, and the hyperparameter search used during training is in TABLE I. All DNNs within a DNN beamformer (i.e., the DNN for each DFT bin) were trained with the same hyperparameter settings and model architecture.

Table I.

Hyperparameter Search Space

Parameter	Search Values
Batch Size	10–1000
Number of Hidden Layers	3–7
Layer Width	520–1040
Input Gaussian Noise	True or False
Input Dropout Probability	0–0.5
Hidden Node Dropout Probability	0–0.5
Batch Norm	True or False
Loss Function	LI, L2, Smooth LI

Adam (adaptive moment estimation) was the variant of stochastic gradient descent that was used during training with the values suggested by Kingma and Ba [37], including α = 10⁻³ (learning rate), β₁ = 0.9 and β₂ = 0.999 (coefficients used for computing running averages of the gradient and its square), and ∊ = 10⁻⁸ (a term to improve the numerical stability of the gradient update). The rectified linear unit (ReLU) was used for the activation function [38]. Three loss functions were included in the hyperparameter search—mean squared error, mean absolute error, and smooth L1, which uses the mean squared error for small values and mean absolute error otherwise. The weights of the network were initialized using a zero-mean Gaussian random variable with variance given by (2/n)^1/2, where n is the size of the previous layer [39], [40]. A 20-epoch patience was used and inputs to the networks had maximum norm equal to one during training and inference.

Pytorch was used to create and train all of the DNNs in this work [41]. Training was performed on a GPU computing cluster maintained by the Advanced Computing Center for Research and Education at Vanderbilt University.

B. Training Data

In the past, we used the responses from individual point targets or the combined responses of two or three point targets to train neural networks for ultrasound beamforming [23], [30]. For this work, we developed a new training data generation technique using anechoic cysts in simulations and tissue-mimicking phantoms (TMPs). Table III contains a summary of the training data types used in this study.

Table III.

DNN Beamforming Training Data Notes

Beamformer	Training data type
DNNpoint	Simulated points
DNNcyst	Simulated cysts
DNNTMPcyst	Tissue-mimicking phantom (TMP) cysts
DNNpoint + cyst	Simulated points and simulated cysts
DNNpoint + TMPcyst	Simulated points and TMP cysts

1). Simulated Point Targets:

For comparison purposes, we include results using training data generated from point targets. A description for how this style of training data was generated can be found in previous work [23]. The responses from single point targets and the combined responses from two point targets were used as described previously [30]. The parameters for the scanning transducer are in Table II. The training set size included 50 000 examples and the validation included 10 000 examples.

Table II.

Linear Array Scan Parameter Values

Parameter	Value
Transmit Frequency	5.208 MHz
Pitch	298 μm
Simulation Sampling Frequency	520.8 MHz
Simulation Decimation Factor	25
Experimental Sampling Frequency	20.832 MHz
Active Elements	65
Transmit Focus	70 mm
F-number	3.6

2). Simulated Anechoic Cysts:

Cysts were scanned with a simulated ATL L7–4 (38 mm) linear transducer array. The parameters for the scanning transducer are in Table II. The background region contained 25 scatterers per resolution cell and the inside region was completely anechoic. Field II was used to perform the ultrasound simulations [35], [36]. The cysts were at a depth of 7 cm and had a radius of 2.5 mm. A cartoon depiction of a single cyst is in Fig. 2. A circle was inscribed on the cyst interior to create a training data region for the inside of the cyst. This circle is shown as a white dashed line in Fig. 2. An annulus was circumscribed around the cyst exterior to create a training data region for outside the cyst. Training examples were generated differently depending on whether the STFT segment was inside or outside of the cyst.

Fig. 2.

Illustration of training data generation from an anechoic cyst. The white dashed circle represents the training data region for inside the cyst. The white dotted annulus represents the training data region for outside the cyst.

An STFT was taken off the anechoic cyst channel data. The STFT segments on the inside of the anechoic cyst region in Fig. 2 were considered to be examples of off-axis scattering. For these off-axis scattering STFT segments, training examples were formed by using STFT segments as input examples and the corresponding target examples were vectors of zeros. The STFT segments in the background training data region in Fig. 2 were considered to be examples of on-axis scattering. For these on-axis scattering STFT segments, training examples were formed by using STFT segments as input examples and the corresponding target examples were the same as the inputs. The same number of training examples was used from the cyst and background training data regions.

For the simulated cysts that had a radius of 2.5 mm, the training data boundary for the cyst interior was set using the known location and size of the cysts and the measured resolution of the imaging system. Using a simulation, the 6-dB lateral width of a point target at the focus was measured to be 1 mm and the 6-dB axial length of a point target was measured to be 0.2 mm. Therefore, the training data boundary for the cyst interior was an ellipse with a lateral radius of 2 mm and an axial radius of 2.4 mm. The interior boundary for the background annulus region was the same as that used for the cyst region. The exterior boundary for the annulus was set so that an equal number of STFT segments were used from both the cyst region and the background region.

A total of 24 anechoic cysts were simulated. Training data were formed from 21 of these cysts and validation data was formed from three of the cysts. The training set size included 50 000 examples and the validation included 10 000 examples.

3). Tissue-Mimicking Phantom Anechoic Cysts:

An ATL L7–4 (38 mm) linear array transducer was operated using a Verasonics Vantage 128 system (Verasonics, Kirkland, WA, USA) to conduct physical phantom scans. The physical phantom was a multipurpose phantom (Model 040GSE, CIRS, Norfolk, VA, USA) and a cylindrical anechoic cyst with approximately 10 mm diameter at a 7 cm depth was scanned. The parameters for the scanning transducer are in Table II.

Training data were generated using the same method as that used for the simulated anechoic cysts in Section II-B2. For the phantom cysts, the training data regions were placed manually so that the cyst region only included STFT segments that were entirely on the inside of the cyst and the background annulus region only included STFT segments that were entirely on the exterior of the background region. A total of 14 scans were made from a 1-cm-diameter cylindrical cyst at different positions along the cylinder: 12 of them were used for training data and 2 of them were used for validation. The training set size included 50 000 examples and the validation included 10 000 examples.

4). Simulated Point Targets and Cysts:

We also studied mixing different kinds of training data. In this example of data mixing, we mixed the point target data from Section II-B1 and the simulated anechoic cyst data from Section II-B2. The training set size included 50 000 examples and the validation included 10 000 examples and the mixing was 50% from each class of training data.

Previously, we showed that image quality improvements plateaued at about 10 000 training examples when training with point targets [30]. In this article, we used 50 000 training examples because we expect the plateau starting point to increase when training with mixed data types such as the one described in this section. A five-fold increase in training data size was selected to give confidence that the training conditions were indeed above the plateau identified previously but for a mixed training data set. In addition, when training with only one type of training data (e.g., point targets or cysts, but not both), using more training data serves as a form of regularization and is not expected to degrade test performance.

5). Simulated Point Targets and Phantom Anechoic Cysts:

In this example of data mixing, we mixed the point target data from Section II-B1 and the phantom anechoic cyst data from Section II-B3. The training set size included 50 000 examples and the validation included 10 000 examples and the mixing was 50% from each class of training data.

C. Image Quality Metrics

We quantified image quality using speckle signal-to-noise ratio (SNRs)

$$\text{SNRS}=\frac{{\mu }_{\text{background }}}{{\sigma }_{\text{background }}}$$ (1)

contrast ratio (CR)

$$\text{CR}=-20{\text{log}}_{10}\left(\frac{{\mu }_{\text{lesion }}}{{\mu }_{\text{background }}}\right)$$ (2)

and CNR [13], [42]–[45]

$$\text{CNR}=20{\text{log}}_{10}\left(\frac{\left|{\mu }_{\text{background }}-{\mu }_{\text{lesion }}\right|}{\sqrt{{\sigma }_{\text{background }}^2+{\sigma }_{\text{lesion }}^2}}\right)$$ (3)

where μ is the mean and σ is the standard deviation of the uncompressed envelope. CR and CNR require specification of a lesion region and a background region.

Traditionally, CNR has been viewed as one of the better metrics for assessing ultrasound image quality because it can be related to the lesion detection probability for an ideal observer [43]. Recently, Rodriguez-Morales et al. [46] observed that when CNR is measured using high contrast lesions, CNR can be artificially increased using a simple dynamic range transformation. They proposed a new image quality metric designed to be immune to such dynamic range transformation called the generalized contrast-to-noise (GCNR). GCNR is computed from the overlap area of two probability distributions

$$\text{GCNR}=1-\int \text{min}\left\{p_{\text{lesion }}(x),p_{\text{background }}(x)\right\}dx$$ (4)

where p_lesion(x) is the density function from the lesion region and p_background(x) is the density function from the background region. Although we have not seen evidence that DNN beamforming behaves like a simple dynamic range transformation, we include GCNR in this work for completeness.

D. Test Data Comparison Scans

1). Simulation:

Anechoic cysts having 1, 2.5, 5, and 10 mm diameter were simulated at a depth of 7 cm and imaged using a simulated L7–4 (38 mm) linear transducer array. The parameters for the scanning transducer are in Table II. No scatterers were located inside the cysts and 25 scatterers per resolution cell were placed in the background region. The speed of sound was 1540 m/s. A total of 20 anechoic cysts were simulated for each of the specified diameters: 10 were used as a validation set and 10 were used as a test set. The validation set was used to select the DNN beamformer that provided the best image quality (i.e., CNR) and the test set was used to report image quality metrics for the selected DNN beamformer.

2). Physical Phantom Scans:

An ATL L7–4 (38 mm) linear array transducer was operated using a Verasonics Vantage 128 system (Verasonics, Kirkland, WA, USA) to conduct physical phantom scans. The parameters for the scanning transducer are in Table II. The physical phantom was a multipurpose phantom (Model 040GSE, CIRS, Norfolk, VA, USA), and cylindrical anechoic cysts at a 7 cm depth with approximately 5 and 10 mm diameters were scanned. Ten scans were made at different positions along the cylindrical cyst: five were used in a validation set for DNN beamformer selection and five as a test set for image quality performance reporting. Note that although this physical phantom was the same one that was used to generate training data in Section II-B3, a different cylindrical cyst inside the phantom was scanned for the evaluation purposes described in this section.

3). In Vivo Scans:

A linear array transducer (ATL L7–4 38 mm) was operated using a Verasonics Vantage 128 system (Verasonics, Kirkland, WA, USA) to scan the liver of a 36-year-old healthy male. Scanning was conducted to look at liver vasculature. The parameters for the scanning transducer are in Table II. The study was approved by the local Institutional Review Board.

The same experimental setup was used to scan the carotid artery of this healthy individual. It was necessary to use standoff pads in order to place the carotid within the depth of field of the DNNs studied in this work. Scans were conducted to obtain a cross-sectional view of the carotid.

The liver of this healthy individual was also scanned using a curvilinear array transducer (ATL C5–2) and a Verasonics Vantage system. A total of six scans were conducted to image different locations in the liver. DNN beamformers for a C5–2 array were trained from scratch, and the training data were generated using Field II using the scan parameters in Table IV. The style of the training data that was used for the C5–2 array corresponded to DNN_cyst in Table III.

Table IV.

ATL C5–2 Curvilinear Array Scan Parameter Values

Parameter	Value
Transmit Frequency	3.125 MHz
Pitch	424.56 μm
Simulation Sampling Frequency	312.5 MHz
Simulation Decimation Factor	25
Experimental Sampling Frequency	12.5 MHz
Active Elements	33
Transmit Focus	49.3 mm
F-number	3.5

E. Robustness Assessment Scans

1). Gross Sound Speed Errors:

The speed of sound is usually assumed to be 1540 m/s when beamforming ultrasound signals. However, the speed of sound in human tissue can actually vary from this value by 10% or more [47]. Therefore, it is important to study the performance of an advanced beamformer in the presence of gross sound speed errors [16]. Anechoic cysts having 5 mm diameter were simulated at a depth of 7 cm and imaged using a simulated L7–4 (38 mm) linear transducer array. No scatterers were located inside the cysts and 25 scatterers per resolution cell were placed in the background region. A total of 20 cysts were simulated. The cysts were simulated using sound speeds between 1294 and 1786 m/s. Transmit beamforming and receive beamforming were performed assuming 1540 m/s. The same 20 cysts were imaged using each of the studied sound speed settings. To prevent overfitting and inflated performance metrics, a validation set was formed using 10 of the cysts for DNN beamformer selection and a test set was formed using 10 of the cysts for image quality performance reporting. Note that the test set was only used for performance reporting.

2). Phase Aberration:

We studied the effect of phase aberration on DNN beamforming using a near-field phase screen model applied to simulated cysts. Aberration profiles were generated by convolving a Gaussian function with random white noise [15], [48]. The generated aberration profiles were made to be zero-mean by fitting a line to the generated profile and then subtracting this linear trend. This phase screen model was applied on transmit and receive to the mathematical elements of the FIELD II simulation. A single aberration profile was characterized by its autocorrelation full-width at half-maximum (FWHM) and its root mean square (rms). Smaller FWHM values and larger rms values indicate stronger phase aberration. In this study, the aberration profiles had an FWHM of 5 mm and the rms values of 0, 10, 20, 30, and 40 ns. The targets were 5-mm-diameter anechoic cysts simulated at a depth of 7 cm using a simulated L7–4 (38 mm) linear transducer array. A total of 20 cysts were imaged using different levels of phase aberration. A validation set was formed using 10 of the cysts for DNN beamformer selection, and a test set was formed using 10 of the cysts for image quality performance reporting.

3). Reverberation:

Evidence continues to grow that reverberation clutter from near-field sources is a major source of ultrasound image degradation [49]. Therefore, it is important to evaluate an advanced beamformer in the presence of reverberation clutter [15]. To study the effect of reverberation clutter on DNN beamforming, we used a pseudononlinear technique developed previously by members of our group [50]. This method operates by simulating the responses from point targets from a region close to the surface of the transducer and then artificially delaying these signals in time with the goal of mimicking near-field reverberation. We also note that this reverberation clutter generating method may not accurately model all of the different kinds of reverberation effects encountered in vivo [53] However, it has been shown that the pseudononlinear reverberation generation method that we used in this article can create a wide variety of reverberation form factors [50].

We used the signal-to-clutter ratio (SCR)

$$\text{SCR}=10{\text{log}}_{10}\left(\frac{P_{\text{SOI }}}{P_{\text{Clutter }}}\right)$$ (5)

where P_SOI is the power of the uncluttered signal and P_Clutter is the power of the reverberation signal to characterize the strength of reverberation clutter. In this study, the SCR was varied in the range from −20 to 10 dB. A total of 20 cysts were imaged using different levels of SCR. A validation set was formed using 10 of the cysts for DNN beamformer selection and a test set was formed using 10 of the cysts for image quality performance reporting.

III. Results

A. Loss Curves

Fig. 3 show the example loss curves for the different training data types. DNN_point had the lowest loss values, which shows that the cyst-based training data were more difficult to learn than point target training data and suggesting that cyst-based training data have higher variance than point target training data. The difference between training and validation losses was minimal for DNN_point, DNN_TMPcyst, and DNN_{point+TMPcyst}, suggesting that these DNNs were not overfitting. The difference between training and validation losses for DNN_cyst and DNN_point+cyst was on the order of 50%, suggesting that some overfitting may be occurring. Using more regularization, such as increasing dropout or using more training data, could help to reduce the observed difference between the training and validation losses in these cases. Finally, the observed variance in the loss curves across training runs with different starting weights, batch order, and dropout was minimal.

Fig. 3.

Example loss curves for the different training data types. Training data were for the center frequency. To facilitate comparison, the same set of hyperparameters was used for all training data types. The boundary regions surrounding each curve represent the minimum and maximum values over five training runs. The sources of randomness across the training runs included starting weights, batch order, and dropout.

B. Test Data Comparison

Fig. 4(a)–Fig. 4(f) shows the examples of point target responses for DAS and DNN beamforming using the styles of training data described in Table III. Previously, we showed that DNN beamformers trained with point targets generalized to diffuse scattering targets [20]. Fig. 4(c) shows how DNN beamformers trained with diffuse targets (i.e., cysts) generalized to point targets. Fig. 4(g) shows how DNN_point, DNN_cyst, and DNN_point+cyst suppressed lateral sidelobes on the order of 40 dB. In contrast, DNN_TMPcyst and DNN_{point+TMPcyst} suppressed sidelobes on the order of 0–10 dB.

Fig. 4.

Simulated point targets for (a) DAS beamforming and (b)–(f) DNN beamforming with different styles of training data (DNN_point, DNN_cyst, DNN_TMPcyst, DNN_point+cyst, and DNN_{point+TMPcyst}, respectively). (g) Axially integrated lateral and (h) axial profiles. DNN beamformer training data types are in Table III. Images shown with 60-dB dynamic range.

Fig. 4(h) shows how all of the DNN beamformers increased axial range lobes relative to DAS, which is consistent with our previous findings [20]. DNN_point produced the lowest range lobes followed by DNN_cyst and DNN_point+cyst. In general, we note that the DNN beamformers trained with simulated training data sets provided the best performance on simulated point targets.

Fig. 5 shows the example images for simulated anechoic cysts using DAS and DNN beamformers. Fig. 5(i), (o), and (u) show how DNN_cyst produced the highest quality images based on visual inspection and Tables V–VIII show how this beamformer also generally produced the best CNR for all of the studied cyst sizes. For the 10-mm-diameter cysts, DNN_TMPcyst and DNN_point+cyst produced CNR values that were 0.07 dB higher than DNN_cyst; however, DNN_cyst provided the best GCNR. These qualitative and quantitative results show that DNN beamformers trained using one-sized anechoic cyst produced improved image quality for cysts having a different size, which demonstrates that training DNN beamformers with one-sized cyst can generalize to other sized cysts. The fact that DNN_cyst produced the best image quality improvements relative to DAS is not surprising because the training data for this DNN beamformer were most similar to this type of evaluation scan. Training with a different kind of simulated target (i.e., DNN_point) or training with physical phantom data instead of simulated data (i.e., DNN_TMPcyst) reduced CNR by about 0.1–1 dB depending on the cyst size.

Fig. 5.

Simulated anechoic cysts for (a), (g), (m), and (s) DAS, (b), (h), (n), and (t) DNN_point, (c), (i), (o), and (u) DNN_cyst, (d), (j), (p), and (v) DNN_TMPcyst, (e), (k), (q), and (w) DNN_point+cyst, and (f), (l), (r), and (x) DNN_{point+TMPcyst}.

Table V.

Speckle Statistics For Simulated 1 mm Diameter Cysts (N = 10)

	CR (dB)	SNRs	CNR (dB)	GCNR
DNNpoint	9.6 ± 2.3	2.09 ± 0.17	2.32 ± 1.51	0.702 ± 0.090
DNNcyst	11.3 ± 3.1	2.06 ± 0.17	2.89 ± 1.69	0.748 ± 0.112
DNNTMPcyst	8.6 ± 2.4	2.13 ± 0.17	1.79 ± 1.79	0.656 ± 0.110
DNNpoint+cySt	9.6 ± 2.4	2.13 ± 0.17	2.39 ± 1.60	0.704 ± 0.106
DNNpoint+TMPcyst	8.6 ± 2.5	2.12 ± 0.16	1.76 ± 1.91	0.668 ± 0.115
DAS	9.7 ± 2.3	2.06 ± 0.16	2.22 ± 1.53	0.701 ± 0.102

Table VIII.

Speckle Statistics For Simulated 10-mm Diameter Cysts (N = 10)

	CR (dB)	SNRs	CNR (dB)	GCNR
DNNpoint	37.3 ± 0.8	1.93 ± 0.04	5.55 ± 0.19	0.972 ± 0.004
DNNcyst	45.5 ± 1.0	1.92 ± 0.04	5.60 ± 0.19	0.992 ± 0.002
DNNTMPcyst	35.5 ± 0.6	1.96 ± 0.04	5.67 ± 0.20	0.966 ± 0.003
DNNpoint+cyst	40.8 ± 1.0	1.94 ± 0.05	5.67 ± 0.21	0.981 ± 0.003
DNNpoint+TMPcyst	35.9 ± 0.7	1.95 ± 0.04	5.61 ± 0.20	0.967 ± 0.004
DAS	30.8 ± 0.4	1.91 ± 0.04	5.35 ± 0.20	0.973 ± 0.003

Example images for phantom anechoic cysts using DAS and DNN beamformers are in Fig. 6. Speckle statistics for phantom anechoic cysts are in Tables IX and X. For these scans, DNN_TMPcyst provided the largest CNR improvements compared with DAS. We note that the cyst edges in the phantom evaluation scans when using DNN_TMPcyst as shown in Fig. 6(d) and (j) were not as well defined as the cyst edges in the simulation scans when using DNN_cyst as shown in Fig. 5(o) and (u). For the phantom cyst training data used to train DNN_TMPcyst, it was not possible to exactly determine the location of the boundary between cyst and background. When setting up the training data, the gap between the cyst interior region boundary and the inner boundary of the exterior cyst region as shown in Fig. 2 was about 0.4 mm. Reducing this gap region could help to improve boundary performance in the phantom scans.

Fig. 6.

Physical phantom anechoic cysts for (a) and (g) DAS, (b) and (h) DNN_point, (c) and (i) DNN_cyst, (d) and (j) DNN_TMPcyst, (e) and (k) DNN_point+cyst, and (f) and (l) DNN_{point+TMPcyst}.

Table IX.

Speckle Statistics For Physical Phantom 5-mm-Diameter Cysts (N = 5)

	CR (dB)	SNRs	CNR (dB)	GCNR
DNNpoint	30.5 ± 1.9	1.86 ± 0.08	5.06 ± 0.35	0.959 ± 0.016
DNNcyst	30.2 ± 2.0	1.89 ± 0.07	5.20 ± 0.33	0.939 ± 0.021
DNNTMPcyst	34.5 ± 3.0	1.92 ± 0.08	5.43 ± 0.36	0.958 ± 0.018
DNNpoint+cyst	40.6 ± 4.4	1.79 ± 0.08	4.93 ± 0.40	0.956 ± 0.019
DNNpoint+TMPcyst	33.9 ± 2.8	1.87 ± 0.07	5.23 ± 0.34	0.955 ± 0.017
DAS	17.7 ± 0.5	1.84 ± 0.07	3.98 ± 0.34	0.893 ± 0.013

Table X.

Speckle Statistics For Physical Phantom 10-mm-Diameter Cysts (N = 5)

	CR (dB)	SNRs	CNR (dB)	GCNR
DNNpoint	35.3 ± 0.9	1.72 ± 0.03	4.56 ± 0.15	0.984 ± 0.002
DNNcyst	34.3 ± 0.6	1.76 ± 0.01	4.73 ± 0.07	0.969 ± 0.003
DNNTMPcyst	45.6 ± 1.6	1.78 ± 0.03	4.95 ± 0.17	0.989 ± 0.002
DNNpoint+cyst	48.8 ± 0.4	1.66 ± 0.04	4.34 ± 0.19	0.983 ± 0.003
DNNpoint+TMPcyst	44.7 ± 1.4	1.74 ± 0.02	4.74 ± 0.13	0.986 ± 0.002
DAS	19.6 ± 0.4	1.70 ± 0.03	3.58 ± 0.19	0.921 ± 0.004

DNN_TMPcyst produced the best CNR as expected because the training data for this beamformer were most similar to this evaluation scan. Training with a simulated target (i.e., DNN_point or DNN_cyst) reduced CNR by about 0.2–0.6 dB depending on the cyst size.

Fig. 7 shows the example images for DAS and DNN beamforming for in vivo liver scans, and speckle statistics for these scans are in Table XI. DNN_TMPcyst produced the best CNR for the in vivo scan. Compared to training with simulated training data, which is the method we had used previously, DNN_TMPcyst improved CNR by 0.8–2.3 dB. DNN_point produced better CNR than DNN_cyst, which is surprising because one might expect training with a diffuse target to be more similar to liver scans than point targets. Overall, the DNN beamformers trained with phantom anechoic cysts produced the best image quality in terms of CNR on the in vivo evaluation scans.

Fig. 7.

In vivo liver scans for (a), (h), and (o) mask regions, (b), (i), and (p) DAS, (c), (j), and (q) DNN_point, (d), (k), and (r) DNN_cyst, (e), (l), and (s) DNN_TMPcyst, (f), (m), and (t) DNN_point+cyst, and (g), (n), and (u) DNN_{point+TMPcyst}.

Table XI.

Speckle Statistics For In Vivo Liver Scans Using L7–4 (N = 15)

	CR (dB)	SNRs	CNR (dB)	GCNR
DNNpoint	20.5 ± 7.1	1.29 ± 0.36	0.52 ± 2.41	0.786 ± 0.096
DNNcyst	19.5 ± 5.5	1.11 ± 0.34	−0.98 ± 2.66	0.694 ± 0.111
DNNTMPcyst	28.1 ± 10.6	1.33 ± 0.37	1.38 ± 2.50	0.870 ± 0.073
DNNpoint+cyst	23.4 ± 7.1	1.12 ± 0.34	−0.37 ± 2.62	0.768 ± 0.085
DNNpoint+TMPcyst	27.6 ± 10.0	1.31 ± 0.37	1.23 ± 2.52	0.855 ± 0.069
DAS	10.9 ± 4.8	1.47 ± 0.39	−1.41 ± 3.12	0.605 ± 0.180

The arrows in Fig. 7(i)–(n) indicate evidence that DNN beamforming revealed a small blood vessel that was unvisualizable using DAS. Similarly, the arrows in Fig. 7(p)–(u) indicate evidence DNN beamforming revealed a small blood vessel that was somewhat visible using DAS. It is interesting to note that the amount of vessel visualization improvement varied depending on the training data type for DNN beamforming.

The same networks that were evaluated using the liver data above were also evaluated using a scan from an in vivo carotid artery, and this scan is shown in Fig. 8 for DAS and DNN beamforming. For DAS and DNN_TMPcyst, CNR was 4.78 and 6.05, respectively. Qualitatively, DNN_TMPcyst provided the best image quality improvement out of the studied DNN beamforming training data types.

Fig. 8.

In vivo carotid artery scans for (a) DAS, (b) DNN_point, (c) DNN_cyst, (d) DNN_TMPcyst, (e) DNN_point+cyst, and (f) DNN_{point+TMPcyst} using an ATL L7–4 linear array transducer.

Examples of DAS and DNN beamforming for an in vivo liver scan using a curvilinear array are in Fig. 9. For DAS and DNN_cyst, the CR was 16.3 ± 4.1 and 31.0 ± 6.9 dB, respectively. For DAS and DNN_cyst, the CNR was 1.7 ± 0.7 and 3.2 ± 1.1 dB, respectively. These quantitative results complement the qualitative improvements in Fig. 9 when using DNN_cyst compared with DAS. These results show that the frequency-domain DNN beamforming method studied here translates to array geometries beyond linear arrays.

Fig. 9.

In vivo liver scans for (a) DAS and (b) DNN beamforming (DNN_cyst) using a C5–2 curvilinear array transducer. Note that the DNNs for these scans were trained using training data from a simulated C5–2 curvilinear array transducer. Images shown with 60-dB dynamic range.

C. DNN Beamforming Robustness

The results in Section III-B showed that DNN_cyst produced the best image contrast in simulated anechoic cyst scans. Therefore, we studied the robustness of DNN_cyst to gross sound speed error, phase aberration, and reverberation using simulated anechoic cyst scans.

1). Gross Sound Speed Errors:

The DNN beamformer was selected to maximize the average CNR observed across the range of studied gross sound speed error values. A validation set was used to select the best DNN beamformer and the images and quantitative results reported here are from a test set. Fig. 10 shows the examples of DAS and DNN beamforming for different gross sound speed errors. Fig. 11 shows CR, CNR, and SNRs as a function of the speed of sound.

Fig. 10.

DAS and DNN beamforming images for anechoic cysts simulated using different speed of sounds. Transmit and receive beamforming was performed assuming the speed of sound was 1540 m/s. Images shown with 60-dB dynamic range.

Fig. 11.

(a) CR, (b) CNR, and (c) SNRs as a function of actual speed of sound. Error bars indicate one standard deviation.

DNN beamforming always produced better CR compared with DAS for the studied speed of sound range. DNN beamforming produced better CNR than DAS as long as the sound speed was within about 15% of the assumed sound speed. DNN beamforming decreased SNRs noticeably when the sound speed deviated by about 8% from the assumed sound speed. Because CR for the DNN beamformer was always better than that of DAS, these results suggest that the observed degradation in CNR was primarily due to the increase in speckle pattern variance. The DNNs were never exposed to gross sound speed errors during training, so it is encouraging to see in Fig. 12 that even for the largest errors in sound speeds, the cyst was still visible in the DNN beamformed images and the background speckle pattern was still similar to the corresponding DAS speckle pattern.

2). Phase Aberration:

The DNN beamformer was selected to maximize the average CNR observed across the range of studied phase aberration rms values and when the FWHM of the aberration profile was 5.0 mm. Fig. 12 shows the example images for the effect of phase aberration on DNN beamforming compared with DAS beamforming. Fig. 13 shows the CR, CNR, and SNRs as a function of the rms value of phase aberration profiles. DNN beamforming always produced better CR and CNR for the studied rms values. When the rms value for the aberration profile was less than 15 ns, DNN beamforming produced similar SNRs values compared with DAS, but the DNN beamformers increased the speckle pattern variance compared to above this cutoff. Fig. 12 shows that the cyst was visible in the DNN beamforming image for all of the studied phase aberration rms values, while it was barely visible in the DAS image for the largest aberration rms values.

Fig. 13.

(a) CR, (b) CNR, and (c) SNRs as a function of phase aberration. The FWHM for each aberration profile was 5 mm. Error bars indicate one standard deviation.

3). Reverberation:

The DNN beamformer was selected to maximize the average CNR observed across the range of studied reverberation levels. The first two columns of Fig. 14 show the examples of the effect of reverberation clutter on DNN and DAS beamforming. Fig. 15 shows the CR, CNR, and SNRs as a function of SCR. For the studied SCR values, CR and CNR were always better for DNN beamforming than DAS. In general, the amount that DNN beamforming degraded SNRs increased as a function of the amount of reverberation clutter.

Fig. 14.

DAS and DNN beamforming images for anechoic cysts simulated using different levels of reverberation clutter. For the DNNs trained with reverberation, the training data included reverberation with SCR −5 dB.

Fig. 15.

(a) CR, (b) CNR, and (c) SNRs as a function of SCR. Error bars indicate one standard deviation.

The final column of Fig. 14 shows the examples of a DNN trained with reverberation clutter. The input training data examples were anechoic cysts with reverberation at a strength of −5 dB, and the output examples were anechoic cysts without reverberation. The results show how training a DNN beamformer with reverberation improved CR, CNR, and SNRs for test scans with SCR less than −5 dB. For example, Fig. 14 shows how the cyst was visible at SCR −20 dB when training a DNN beamformer with reverberation, but it was not visible when training without reverberation and also for DAS.

Fig. 15 shows that as the reverberation became weaker than that used during training, the DNN trained without reverberation provided equivalent CNR and SNRs and better CR than the DNN trained with reverberation. Training over a wide range of reverberation levels could provide a DNN beamformer with the best performance over a wider SCR range than that shown here.

IV. Discussion

DNN beamforming is flexible in multiple ways, including the method used to generate training data. The results in Fig. 7 and Table XI show that the proposed anechoic cyst-based training data generation method improved in vivo image contrast and CNR compared with the point target training data generation method that we used in the past [20], [23]. This anechoic cyst-based training data generation method has several advantages over the point target method. For example, the number of scans required was reduced by multiple orders of magnitude because many training examples can be produced from a single scan of an anechoic cyst. In this work, we implemented this method on simulated and phantom anechoic cysts. However, it may be feasible to implement this method on in vivo anechoic targets in the human body, such as the bladder.

While Table XI shows how the DNN approach improved CR and CNR in larger structures, Fig. 7 shows how the DNN approach also revealed small structures that were not visualized by DAS. In our example, the smaller structures were blood vessels, but in general, the ability of deep networks to image things unvisualized by DAS is a function of the data used to train the networks. This finding demonstrates the importance of developing improved methods to create training data for DNN beamforming in addition to more traditional methods, such as improving network architectures.

The results in Section III-B show a range of degradation caused by mismatched training data and evaluation scans. For example, training using Field II simulation scans and evaluating physical phantom scans or the other way around reduced CNR by at least 0.1 dB and as much as 1.0 dB. In addition, we found that in vivo CNR was improved by 0.8 dB when training DNN beamformers with physical phantom-based training data instead of simulation scans. These results suggest that finding ways to better match the simulation training data to the experimental scans or finding ways to generate training data from experimental scans will offer at least small improvements in CNR and possibly large improvements. As far as we know, the cyst training data proposed here are one of the first physical scan-based methods for training DNN beamforming that uses fully sampled training data. The method by Gasse et al. [21] used experimental training data, but the goal was to recover the image quality achieved using many plane wave transmit angles from a few plane wave transmit angles, which amounts to recovering fully sampled channel data from channel data that has been subsampled along the transmit event dimension. The method by Yoon et al. [27] also used experimental training data, but the goal was to recover fully sampled channel data from subsampled channel data. For these methods, the best performance is already achievable by using fully sampled channel data.

So far, two categories for creating DNN beamformer training data have been explored. The first category of methods relies on the artificial degradation of ultrasound channel data in some form or fashion (e.g., adding noise or subsampling), which is followed by teaching DNNs to recover image quality as if the channel signals had not been degraded [21], [25], [27]. In these methods, the goal was to reduce the number of necessary transmit events or to reduce the data load needed to create ultrasound images. The second category of methods relies on simulation scans in which it is possible to create input and target example training pairs for the image quality improvement task [23], [26], [28]. The main advantage of this second category is that it is possible to push ultrasound image quality beyond the state of the art achievable when fully sampled channel data are available.

Introducing sources of image degradation, such as gross sound speed errors, phase aberration, and reverberation, can be viewed as a way to push channel data outside the data region in which the DNN beamformers were trained. It would not be surprising if the DNN beamformers had failed to generalize to these new regions of channel data space. Instead, the results showed that the DNN beamformers maintained their performance over a fairly wide range of image degradation space. For example, CNR continued to show improvements compared with DAS for all values of the studied image degradation mechanisms except for the extreme values of sound speed errors.

The results in Section III-C3 show how including sources of image degradation in the training data can further improve the robustness of DNN beamforming. For this work, we studied adding reverberation clutter in the training generation process and found that doing so increased the robustness of the DNN beamformer studied here to stronger reverberation levels.

The anechoic cyst training data generation method proposed in this work has several limitations. For example, this method uses off-axis scattering examples from anechoic regions only, while in general, off-axis scattering is not limited to anechoic regions. It is possible that the beamformer is only learning to improve anechoic regions and may not improve CNR for high contrast targets, such as hypoechoic cysts. In addition, the DNN beamformers trained with anechoic cysts tend to exhibit a dark region artifact for hypoechoic and hyperechoic targets and strong point targets in a speckled background. We are actively developing a training data generation method that relies on hypoechoic cysts instead of anechoic cysts in order to address these limitations [51], [52].

For the physical phantom and the in vivo test scans, the effect of increasing simulated training data diversity (in particular, comparing DNN_point+cyst to DNN_point and DNN_cyst) was to increase CR, decrease speckle SNR, and decrease CNR. These results persist (with a few exceptions) across Tables IX–XI and Figs. 6 and 7. Furthermore, the cyst appears slightly larger and the borders appear sharper for DNN_point+cyst compared with DNN_point and DNN_cyst for the physical phantom scans in Fig. 6. Similar trends were not observed for the simulation data as the image quality of DNN_point+cyst tended to be between those of DNN_point and DNN_cyst.

We think these results suggesting that the effect of increasing training data diversity (i.e., using point targets and anechoic cysts for training) was to expand the null space (i.e., the aperture domain signal space over which the DNN beamformer sets the signal amplitude to zero or close to zero) of the DNN beamformers in both a global and local sense. Globally, we expect the null space for DNN_point+cyst to be expanded relative to DNN_point and DNN_cyst because single point target examples of off-axis scattering exist in a different region of aperture domain signal space compared with anechoic cyst examples of off-axis scattering. However, the results mentioned earlier also suggest that an erosion operation was also applied to the null space of DNN_point+cyst compared with DNN_point and DNN_cyst. The presence of this erosion is conditioned on mismatched training and test data (e.g., train with simulation and test with experiment) and also on the use of a training strategy designed to create a distinct null space for the DNN beamformer.

This finding suggests that developing new ways to generate simulated training data similar to the methods examined here (i.e., developing methods to globally expand the null space of the DNN beamformer) will most likely also include the null space erosion effect described earlier. We speculate that the development of training data generation methods not dominated by the creation of a null space for the DNN beamformer, but instead on the recovery of low amplitude signals of interest (e.g., on-axis scattering) in higher amplitude, structured noise (e.g., off-axis scattering or reverberation clutter) may overcome this limitation [51].

We studied selecting DNN beamformers using GCNR. However, in some cases, GCNR selected DNN beamformers that produced poor image quality upon visual inspection. In these cases, CNR was positively correlated with speckle SNR and negatively correlated with CR and the opposite trend was observed for GCNR—GCNR was negatively correlated with speckle SNR and positively correlated with CR. These findings suggest that while GCNR is useful for verifying that CNR improvements are not due to dynamic range transformation, GCNR may not always be a good indicator for image quality assessment based on visual inspection. We note that the correlations we described earlier did not exist across all DNN training data types listed in Table III and also varied across simulations, phantom, and in vivo scans. In this work, we used CNR to pick the best DNN beamformers from within each class of training data described in Table III and so the behavior described earlier is not apparent from the presented results. Note that this discussion is specific to the DNN beamforming method studied in this work and may not apply to other DNN beamforming methods. We briefly discuss the observed phenomena concerning GCNR in relation to the existing discussion in the literature [54].

As noted by Rodriguez-Morales et al. [54], speckle smoothing leads to reduced variance in the probability density functions for both inside and background regions and to higher GCNR values. In order to understand the unexpected negative correlation between GCNR and speckle SNR that we described earlier, we examined cases near the extremes of the described negative correlation and found the following situations. For high GCNR and low speckle SNR, the region inside the cyst had almost zero variance and was concentrated around a small value (i.e., the probability density function for the inside image pixels resembled a delta function at zero), whereas the background region exhibited speckle that was degraded by dropout (i.e., the variance for the background region was increased). This result suggests that as long as the probability distribution for one region is made to resemble a delta function, the variance for the second probability distribution can be increased, but GCNR will still be high. For low GCNR and high speckle SNR, speckle was smoothed for both regions, but the difference between the means of the probability distributions was decreased, resulting in decreased GCNR. These examples illustrate some strategies that a beamformer might use to arbitrarily manipulate GCNR and some of the DNN beamformers trained in this work appear to have arbitrarily learned these strategies.

We would expect most DNN beamforming methods to exhibit similar robustness to the studied sources of image degradation as that observed for the DNN beamformer studied in this article. The point target and anechoic cyst-based training methods proposed in this work and our previous work would be most applicable to the method proposed by Hyun et al. [28]. It should also be possible to mix the training data generation ideas proposed here with other DNN beamforming methods that rely on recovering fully sampled channel data from subsampled channel data [24], [25], [27]. However, we note that the DNN beamformer that we study here is unique in that many training examples can be generated from a single scan, while most other DNN beamforming methods consider a single scan to be a single training example.

V. Conclusion

In this article, we studied generating DNN beamformer training data using phantom anechoic cysts and found that this method improved in vivo image contrast and CNR compared with training data generation methods that rely on simulation scans. We also investigated the robustness of DNN beamforming using common sources of image degradation, including gross sound speed errors, phase aberration, and reverberation clutter. We found that the DNN beamforming method studied in this article was robust to these sources of image degradation over fairly wide ranges of degradation. These sources of image degradation are common in clinical imaging scans and the simulation study results presented here suggest that DNN beamformers maintained their ability to improve image quality relative to DAS in the presence of common sources of image degradation. Overall, the results of this article demonstrate the strong potential of using DNN for ultrasound beamforming.

Table VI.

Speckle Statistics For Simulated 2.5-mm-Diameter Cysts (N = 10)

	CR (dB)	SNRs	CNR (dB)	GCNR
DNNpoint	21.7 ± 1.4	1.95 ± 0.18	4.94 ± 0.84	0.928 ± 0.020
DNNcyst	31.9 ± 2.3	1.92 ± 0.16	5.40 ± 0.77	0.978 ± 0.018
DNNTMPcyst	19.0 ± 1.4	1.99 ± 0.18	4.78 ± 0.81	0.903 ± 0.023
DNNpoint+cyst	25.4 ± 2.1	1.97 ± 0.16	5.30 ± 0.72	0.948 ± 0.024
DNNpoint+TMPcyst	20.1 ± 1.5	1.98 ± 0.17	4.83 ± 0.73	0.903 ± 0.021
DAS	19.6 ± 1.1	1.93 ± 0.18	4.64 ± 0.86	0.924 ± 0.019

Table VII.

Speckle Statistics For Simulated 5-mm-Diameter Cysts (N = 10)

	CR (dB)	SNRs	CNR (dB)	GCNR
DNNpoint	32.9 ± 0.6	1.93 ± 0.02	5.51 ± 0.10	0.968 ± 0.004
DNNcyst	47.1 ± 1.4	1.93 ± 0.02	5.67 ± 0.10	0.992 ± 0.002
DNNTMPcyst	30.1 ± 0.5	1.96 ± 0.03	5.52 ± 0.13	0.963 ± 0.003
DNNpoint+cyst	37.1 ± 1.1	1.96 ± 0.03	5.69 ± 0.14	0.975 ± 0.004
DNNpoint+TMPcyst	31.2 ± 0.6	1.96 ± 0.03	5.55 ± 0.13	0.957 ± 0.003
DAS	27.2 ± 0.6	1.91 ± 0.02	5.23 ± 0.10	0.974 ± 0.004

Biographies

Adam C. Luchies (Member, IEEE) received the B.S. degree in engineering from John Brown University, Siloam Springs, AR, USA, in 2009, and the M.S. and Ph.D. degrees in electrical and computer engineering from the University of Illinois at Urbana–Champaign, Urbana, IL, USA, in 2011 and 2016, respectively. He completed a postdoctoral fellowship at Vanderbilt University, Nashville, TN, USA.

He currently started a position as an Ultrasound Engineer at Siemens Healthineers, Issaquah, WA, USA. His research interests include ultrasound beamforming, quantitative ultrasound, signal processing, and deep learning.

Brett C. Byram (Member, IEEE) received the B.S. degree in biomedical engineering and math from Vanderbilt University, Nashville, TN, USA, in 2004, and the Ph.D. degree in biomedical engineering from Duke University, Durham, NC, USA, in 2011.

He was a Research Assistant Professor with Duke University. In 2013, he joined the Biomedical Engineering Department, Vanderbilt University, as an Assistant Professor. He has spent time working at the Jørgen Jensen’s Center for Fast Ultrasound, Lyngby, Denmark, and Siemens Healthcare’s Ultrasound Division, Mountain View, CA, USA. He currently runs the Biomedical Elasticity and Acoustic Measurement (BEAM) Laboratory, where he and others in the lab pursue solutions to clinical problems using ultrasound. He is also with the Vanderbilt Institute for Surgery and Engineering (VISE) and the Vanderbilt University Institute of Imaging Science (VUIIS). His research interests include beamforming, motion estimation, and other related signal processing and hardware development tasks.