Improved Nondestructive Ultrasound Molecular Imaging with Lightweight Convolutional Neural Network

Abstract

Ultrasound molecular imaging (USMI) is an imaging approach that utilizes targeted microbubbles (MBs) to highlight biomarkers of disease. While differential targeted enhancement (DTE) is the current state-of-the-art for USMI, its reliance on destructive pulses hinders real-time clinical application. We have developed a neural network-based nondestructive USMI, validated in vivo using a transgenic mouse model of spontaneous breast cancer. To enhance training efficacy despite a limited animal number (N=14), we utilized several augmentation strategies including the use of several targeted MB types for each animal to generate independent image and texture patterns, alternative DTE approaches (sham and injection DTE), and random patch selection, overall resulting in a total of 15,350 patches to train the network. The resulting nondestructive USMI produces an image of the pixelwise MB classification score of the presence of targeted MBs. Our nondestructive USMI achieved a correlation coefficient of 0.954 with DTE, a continuous dice coefficient of 0.863 for a molecular signal coverage of the lesion over 20%, and a higher AUC than DTE (0.954 vs. 0.845) compared to the reference image developed from the contrast enhanced ultrasound (CEUS) image and manual lesion contour. Nondestructive imaging during continuous motion of the transducer under elevation sweeps yielded fewer artifacts and higher AUC than DTE (0.953 vs. 0.892), compared to the reference image. This demonstrates the potential of free-hand and real-time nondestructive imaging. Overall, nondestructive imaging showed comparable performance to DTE under stationary conditions and superior performance to DTE under transducer motion, indicating its clinical imaging potential.

I. Introduction

BREAST cancer has the highest incidence and mortality rate among all cancers in women in the United States with an estimate 316,950 new cases and 42,170 deaths projected in 2025 [1]. While Mammography reduced the mortality rate of breast cancer [2], its diagnostic performance is limited in women with dense breast tissue [3]–[5]. Sole reliance on mammography increases the risk of missing cancer cases, necessitating complementary imaging, such as magnetic resonance imaging (MRI) and ultrasound (US). However, MRI has limitations, such as low specificity and high costs, and dependence on gadolinium-based contrast agents, rising safety concerns [6], [7]. MRI is not easily accessible in many regions globally, making it an inadequate choice for routine screening. In contrast, US can be considered as a more practical alternative, with supplemental screenings showing reduced deaths [8]. Despite improved detection rates with combined US and mammography [9], [10], US alone suffers from lower sensitivity, specificity, and receiver operating characteristic (ROC) area under the curve (AUC) compared to mammography alone or combined mammography and US [10], highlighting the need for advancements in US diagnostic accuracy for women with dense breast tissue.

Ultrasound molecular imaging (USMI) is an emerging approach to improve the diagnostic performance of conventional US for breast cancer evaluation [11], [12]. USMI utilizes contrast microbubbles (MBs), gas-filled agents encapsulated by a lipid shell [13]–[15], that are functionalized to target and bind to specific biomarkers of interest [16]. Suitable biomarkers for breast cancer include vascular endothelial growth factor receptor 2 (VEGFR2), programmed cell death ligand 1 (PD-L1), and B7 homolog 3 protein (B7-H3), which are differentially expressed on the surface of endothelial cells within the tumor vasculature. VEGFR2 is an inflammatory biomarker involved in neovasculature [17], whereas PD-L1 [18], [19] and B7-H3 [20] are immune checkpoint inhibitors.

Under appropriate pressures, US reflections from MBs bound to the targeted MBs will generate strong non-linear echoes due to the compression and expansion of the gas in the MBs while tissue produces linear echoes, thereby providing a means to separate the “molecular signal” (the nonlinear echoes) from the targeted MBs from the linear echoes from tissue. The amplitude of the molecular signal depends on the number of bound microbubbles within the ultrasound beam [21]. Thus, the amplitude of USMI acts as a proxy measurement of the amount of biomarker expressed by the vascular endothelial cells. Imaging techniques used to perform USMI include contrast-enhanced ultrasound (CEUS) and differential targeted enhancement (DTE). CEUS technologies, such as pulse inversion [22], amplitude modulation [23], and contrast pulse sequences [24], are designed to detect the nonlinear echoes from MBs while suppressing echoes from tissue. However, strong echoes from tissue can “leak” into the CEUS images and confound detection of the targeted MBs, while unbound MBs create non-specific background noise. To further improve the diagnostic performance of USMI, CEUS is used in conjunction with DTE [25], [26], which has been demonstrated in preclinical imaging [27]. For DTE, CEUS images are recorded before and after MB destruction with high-pressure pulses (but still within FDA regulations), and the averaged post-burst images are subtracted from the averaged pre-burst images to reconstruct the DTE image. Ideally, the averaging used in this process creates a mean background value from unbound (moving) MB that appears in both the pre- and post-burst images, while the bound (targeted) MB signal is only present in the pre-burst image. The subtraction cancels the unbound MB and tissue-signal leakage leaving only the targeted MB signal in the image. Tissue-signal leakage refers to the signals from tissues that are not fully canceled out by the CEUS pulse sequence. While tissue leakage can be reduced by DTE, misalignment between the pre- and post-burst frames of the DTE process hinders the complete suppression of the tissue-leakage signals, which then appear as false-positive molecular signal.

USMI has been applied to several diseases, including inflammatory bowel disease [28], [29], myocardial ischemia [30], atherosclerosis [31], and cancer imaging [32], [33]. Recently, the clinical feasibility of USMI in human patients has been assessed by performing first-in-human studies in breast and ovarian lesions [32], which showed well-matched US molecular signal to immunohistochemical (IHC) analysis. While DTE is useful in preclinical studies, the first-in-human study employed CEUS as the USMI modality [32] due to several limitations: (1) DTE cannot be implemented in real-time due to the collection of several US images before and after MB destruction, requiring the transducer to remain stationary for prolonged time, (2) only one DTE image can be captured due to the destruction of the MBs, thereby requiring the target location to be known prior to scanning, and (3) bubble destruction using high-pressure pulses could cause mild tissue damage [34] and may introduce other regulatory hurdles. Moreover, misalignment between pre- and post-burst scans, such as physiologic movement from respiration or global motion of the transducer or patient, creates additional tissue-signal leakage and contrast signal artifacts.

Previously, we developed a deep convolutional neural network (CNN) to address the limitations of non-real-time and destructive DTE, which achieved nondestructive USMI by employing the DTE images to generate binarized ground truth. We investigated MB detectability using several forms of input data types including time-delayed in-phase and quadrature (IQ) element signals, beamformed IQ data, and magnitude of the beamformed IQ data [35]. The MB prediction using the time-delayed IQ element data achieved the best DTE-like performance compared to the other data types. This is because data types that undergo less processing (e.g. IQ element data) contain more information than processed data types (e.g. magnitude of the beamformed IQ). This is consistent with network classification performance applied to other diseases, including compressed data and post-processed display images (e.g. DICOM images captured from clinical US scanners), demonstrating lower performance with increasing amounts of processing [35], [36]. Although less-processed data types can achieve higher classification performance, they are less accessible in clinical US scanners and require more time and memory for image reconstruction and inference. The low accessibility of IQ element data in commercial US machines prevents our prior approach from being widely translatable to clinical systems.

We improve upon our prior methodology using deep CNN and utilize conventional post-processed US images to achieve comparable or higher performance than DTE as well as nondestructive MB detection. We introduced this approach in [37] and have further improved the method and expanded its validation in this paper. We refine the traditional DTE approach to augment training and to achieve accurate prediction of stationary MBs while avoiding overfitting. The proposed network is trained and validated using images from an animal model of breast cancer. MB detection performance is evaluated and compared between DTE and the network prediction.

II. Materials and Methods

We propose a nondestructive USMI framework based on a neural network approach trained to detect MBs from a pair of B-mode and CEUS images. The goal of the approach is to achieve frame-by-frame prediction, avoiding video captures before and after MB destruction, and yielding stable MB imaging under motion and in real-time.

An overview of the development of our proposed nondestructive USMI framework is shown in Fig. 1 (a). Briefly, US videos were recorded during in vivo animal experimentation where the animals were given up to 4 types of MB injections and imaged with DTE. Various forms of augmentation were performed to increase the training dataset size. A lightweight CNN was trained to predict the pixelwise MB classification score, which represents a higher score in the presence of targeted MB and a lower score in their absence. The MB classification score ranges from 0 to 1, which is used as USMI. We refer to the pixelwise MB classification score as the intensity of the nondestructive image.

Fig. 1:

(a) Overall framework of the development of the proposed nondestructive USMI approach. (b) Imaging protocol.

A. Microbubble Fabrication for USMI

MBs functionalized with avidin (USphere^™ Labeler LS, Trust Bio-sonics Inc., Taiwan) were purchased and stored at 4 °C. Three antibodies were used in the fabrication of targeted MBs: anti-VEGFR2, anti-PD-L1, and anti-B7-H3. The monoclonal antibodies (mAbs) were biotinylated respective to anti-VEGFR2, anti-PD-L1, and anti-B7-H3 with NHS-PEG₄Biotin (ThermoFisher Scientific, Waltham, MA, USA) using 1:2 molar ratio of antibodies to biotin. NHS-PEG₄-Biotin was dissolved in phosphate-buffered saline (PBS, pH 7.2) for conjugation. NHS-PEG₄-Biotin was mixed with the mAbs (1 mg/mL in UltraPure Water), and incubated at room temperature for 30 minutes. The unconjugated excess biotin was removed by passing through a 7 kDa molecular weight cut-off Zeba spin column (ThermoFisher Scientific, Waltham, MA, USA). VEGFR2, PD-L1 and B7-H3 targeted MBs were prepared by separately mixing 1 × 10⁸/mL/vial of USphere Labeler LS bubbles with 10μg of the corresponding biotinylated mAbs. These mixtures were incubated at room temperature for 1 hour, and unconjugated mAbs were removed by washing the MBs 3 times using 500 μL of PBS by microcentrifugation at 300 g for 3 minutes. After each wash, the upper milky layer of targeted MBs was carefully separated by removing the lower liquid layer using a syringe needle. The targeted MBs were resuspended in fresh PBS for use in the animal experiments. USphere Labeler LS MBs without targeting mAbs were used as control non-targeted MBs.

B. In vivo Animal Data Collection - MB Augmentation

All in vivo animal experiments were approved by the Institutional Animal Care and Use Committee (IACUC) at Stanford University. A transgenic mouse model [FVB/N-TG(MMTV-PyMT^634Mul)] of spontaneous breast cancer was used in the study for training and testing the proposed approach. We monitored tumor progression by measuring the diameter with a caliper. In this transgenic mouse model, the tumors grow rapidly, with a maximum tumor diameter of 8 mm imposed by IACUC guidelines. We performed up to 4 scans prior to the tumors reaching their maximum size (frequently reached by the end of 6 weeks). Before scanning, we manually injected nontargeted MBs and the MBs targeting B7-H3, PD-L1, and VEGFR2 via the tail vein. Two hundred microliters of saline containing 5×10⁷ MBs per mouse were administered slowly over one minute; this corresponds to an injection dose of approximately 2×10⁹ MBs/kg, based on the average weight of the mice used in this study. The three types of targeted MBs served to augment the dataset. Because the different molecular receptors are physically located in different positions, the texture patterns of the molecular signals on US are uncorrelated. Therefore, the same animal is able to be used to obtain uncorrelated images for training, even when the tumor and transducer position is the same. For each mouse, scans with each of the targeted MBs were performed with at least 3-day intervals to ensure all bound targeted MBs were fully cleared after each injection. The order of the type of MB injected was randomized for each animal. The injections were separated by 3 or 4 days with the first day having a “back-to-back” injection in some cases; data were acquired on days 0, 2–3, 4–6, and 7–10. For the ”back-to-back” injection, non-targeted MBs were injected prior to one of the three targeted MB injections, and at least 15 minutes time-intervals were given between the two injections to ensure complete clearance of the non-targeted MBs before the targeted MB injection. Between targeted MB scans, some tumors grew larger than the maximum size; in these cases, the animals were euthanized before completing the 4 MB injections. Overall, a total of 14 mice were imaged with a total of 44 MB injections.

Animals were scanned using a Vevo2100 (FUJIFILM VisualSonics, Toronto, ON, Canada) US scanner equipped with an 18 MHz linear transducer (MS250S). Scanning was conducted utilizing dual B-mode and CEUS with a focal depth of 10 mm and a total imaging depth of 20 mm. The US acquisition operated at a frame rate of 6 to 7 frames per second (FPS), while CEUS employed the amplitude modulation technique [23]. The transmit power used was 4%, with B-mode and CEUS gains set to 22 dB and 30–33 dB, respectively, and the beamwidth was set to “wide”. The dual mode display was recorded as an mp4 video with a raw resolution of 1440×848 pixels, capturing all tumors within a 470×470 pixel area in two separate windows, one for B-mode and the other for CEUS, both centered on the tumor. The mp4 videos were imported into MATLAB using the ”VideoReader” function and were cropped into two 470×470 pixel windows to create the two channel input data for the network. Because the frame rate of the mp4 files was five times higher than that of the Vevo2100 (30 FPS compared to 6 to 7 FPS), we excluded the redundant frames from the mp4 data, specifically those repeated for 5 consecutive frames. Fig. 1(b) shows the US scan protocol, where videos of two types of scans were acquired: 1) videos where the transducer is stationary and observing a tumor’s largest cross-section over time (referred to henceforth as stationary videos) and 2) videos where the transducer is in motion, using a controlled elevational sweep to observe the whole tumor (henceforth referred to as sweep videos). The stationary transducer experiments permit development and testing of the proposed method while the elevational sweeps permit analysis under conditions more typical of clinical usage. The sweep videos simulate free-hand scanning, which typically involves translation of the transducer over a surface while imaging. Sweep videos were acquired twice - before and after the injection of MBs - using a motor stage to generate one-dimensional translational motion, rather than actual free-hand scanning, because DTE requires the same regions of interest between scans for subtraction. The elevational sweep ranged over 10.21 mm with a step size of 0.152 mm. Videos were captured before MB injection, after MB injection, before burst, and after burst. Specifically, a sweep video was captured before injecting MBs (named “pre-injection sweep video”) and then a stationary video was obtained while injecting MBs for approximately 3 minutes (named “injection stationary video”); this stationary video included images before and after MB circulation within the tumors. Twenty seconds after the stationary video, another sweep video was obtained (named “post-injection sweep video”). A second stationary video was obtained for one minute that included MB destruction using a high-mechanical-index burst half-way during the scan (named “burst stationary video”). This stationary video included 30 seconds of pre-burst and another 30 seconds of post-burst video. Detailed timing of the video acquisitions and the imaging setups are shown in Fig. 1(b).

This scanning protocol was performed on multiple days with one of the three targeted MBs used on each scanning day. This enabled data augmentation for the same tumor because (1) the tumor progressed between each MB injection, growing in size and changing shape, and (2) the different MB types produced different USMI image patterns within the same tumor. A total of 83 stationary videos composed of 39 injection videos and 44 burst videos were obtained from these animal experiments.

C. DTE Augmentation

Due to averaging of the pre- and post-burst images, conventional DTE produces only one DTE image (Fig. 2), thereby producing just 44 DTE images from the 44 burst stationary videos; this dataset size is not large enough to train the nondestructive USMI neural network. To enhance training, we generated a total of 3,070 DTE images while recording 44 burst stationary videos by introducing additional non-conventional DTE frameworks, namely “injection DTE” and “sham DTE”, with a sliding window approach.

Fig. 2:

DTE production with sliding window approach from burst DTE videos, producing 30 DTE images per video compared to one DTE image with traditional DTE. ${\mathrm{A}}_i$ and B images represent images with and without MBs, respectively, for both burst and injection DTE.

We applied the sliding window approach to the videos where 30 pre-burst or post-injection frames from stationary videos were used for averaging rather than all frames. The sliding window averages were then subtracted from the averaged post-burst or pre-injection image, respectively, as shown in Fig. 2. The averaged image (${\mathrm{A}}_i$) formed from the sliding window is obtained as:

$$A_i=\sum F(n)=\frac{1}{30}\sum _{k=1}^{30}F(M\cdot (i-1)+k)$$ (1)

where $F(n)$ indicates the $n_{th}$ frame from a video and the videos are spaced by M frames to extract the same 30 frames from each video. In this study, videos have different recording times; for example, pre-burst and post-injection videos were recorded for approximately 30 seconds and 2 minutes, respectively. To avoid overfitting to longer videos, the same number ($N=30$) of averaged images, $\left\{A_1,A_2,\dots ,A_{N=30}\right\}$, was extracted from each video using a regular frame interval ($M$), which varies depending on the length of the videos.

The application of equation 1 to the sweep videos reflects the spatial averaging of the volume instead of a temporal average. Because the transducer is in motion, $n$ represents the frame index in the elevation spatial dimension. For the sweep videos, $M$ was set to 1 to include all spatial frames.

The data utilized for sham/injection/burst DTE are depicted in Fig. 1 (b) and are produced as follows:

Burst DTE:

Conventional DTE, as defined in Lindner et. al. [25], which subtracts averaged post-burst video frames from pre-burst frames. This will be henceforth referred to as “burst DTE” in this paper to avoid ambiguity with the non-conventional DTE images. We recorded 44 burst stationary videos, which were augmented into 1,320 burst DTE images by applying the sliding window approach.

Injection DTE:

Injection DTE uses video before and after injecting MBs and subtracts pre-injection video frames from post-injection frames. We recorded 39 injection stationary videos, which were augmented into 1,170 injection DTE images by applying the sliding window approach.

Sham DTE:

Sham DTE subtracts early pre-injection video frames from later pre-injection frames to generate a negative control without MBs. Among the 39 injection stationary videos, only 30 videos contained 30 or more frames necessary for the sliding window process. Consequently, the 30 injection stationary videos were augmented with the sliding window approach into 580 sham DTE images.

D. Patch Augmentation

We performed an in vivo study in a transgenic mouse model of breast cancer. The lesions in these animals develop in the mammary glands, close to the skin surface, and typically presents as rounded feature with a hypoechoic background. This common pattern can potentially bias the prediction model and reduce generalization because this morphology is only observed in small animal models. To prevent overfitting, we implemented random patch generation as depicted in Fig. 4 (a). This process crops each input frame into five patches of random positions and sizes, ranging from 64×64 to 470×470 pixels. These varying crop sizes serve different purposes in the training process. Patches with sizes closer to the original image dimensions of 470×470 provide valuable information on texture patterns for training while maintaining the same or similar resolutions to the original image. In contrast, patches with sizes closer to 64×64 focus on local tissue texture and are less likely to depict the rounded shapes surrounded by hypoechoic backgrounds. After cropping, we resize the patches to a uniform size of 256×256 pixels, creating network inputs through interpolation or downsampling. This approach generates various resolutions, approximating different frequencies in the network input. Based on our prior work [35], variations in resolution and frequency less than those from the original size down to 64×64 do not significantly impact network predictions. However, alterations beyond those associated with patches smaller that 64×64 tend to degrade network performance and therefore were excluded from our training.

Fig. 4:

(a) Patch augmentation strategy and (b) overview of the nondestructive USMI network architecture.

E. Training and Testing Datasets

The data augmentation strategy involving multiple MB injections, multiple video recordings per MB injection, DTE augmentation, and patch augmentation resulted in 15,350 patches for training and testing. We performed 5-fold cross-validation by randomly dividing the animal into 5 subsets to avoid the use of any same-animal images for both training and testing. Table I provides the number of animals, videos, frames, and patches for each training/testing fold. Note that although patches were used to train the network, testing was performed on the full images.

TABLE I:

Training and Testing Folds

Training/testing index		1	2	3	4	5
Training	Number of animals	11	11	11	11	12
	Animal indices	4 — 14	1 — 3,7 — 14	1 — 6,10 — 14	1 — 9,13 — 14	1 — 12
	Number of videos (S,I,B)a	60 (26,0,34)	59 (24,0,35)	59 (24,0,35)	59 (24,0,35)	59 (23,0,37)
	Number of patches	7685	7720	7470	7455	7670
Testing	Number of animals	3	3	3	3	2
	Animal indices	1 — 3	4 — 6	7 — 9	10 — 12	13 — 14
	Number of videos (S,I,B)a	22 (4,8,10)	22 (6,7,9)	23 (7,8,8)	23 (6,8,9)	23 (7,8,8)
	Number of frames	603	566	656	649	606

^aS, I, and B represent the number of sham, injection, and burst DTE videos, respectively.

In the training dataset, frames containing motion, such as respiratory and global motion, were excluded. All videos contained respiratory motion, while injection videos also contained global motion from the animal due to the tail vein injection and sweep videos included additional global motion due to transducer movement. Consequently, injection DTE images and all DTE images from sweep videos were excluded from training due to the production of false positive signals from motion and misalignment of pre- and post-burst images. Thus, the images in the training dataset comprised only sham and burst DTE images from stationary videos. Sham DTE images were utilized as negative controls, providing input images from lesions with no MB present, while burst DTE images represented lesions containing MBs. The inclusion of sham DTE images was to enable the network to differentiate between the presence and absence of MBs during the training process. To mitigate respiratory motion artifacts in the burst DTE images, pre- and post-burst frames with low frame-to-frame correlation were excluded; only frames with a correlation greater than the average correlation were utilized for training, as shown in Fig. 3. Note that DTE images from stationary and sweep injection videos were included for testing to evaluate the proposed network performance under conditions with motion. The specific number of videos used for training and testing, including sham, injection, burst, and sweep videos, is provided in Table I.

Fig. 3:

Motion shown by the correlation between video frames of a mouse during pre-burst scan. Respiratory motion caused small frame-to-frame decorrelation. Frames were excluded from training if their correlations with the next frame were below the average (red line).

F. Nondestructive USMI Network Architecture

We implemented a simple 4-layer CNN [35] in TensorFlow Keras [38] to predict the pixel-wise MB classification score of targeted MB presence, as illustrated in Fig. 4 (b). This simple architecture was chosen to facilitate rapid inference on a single GPU, aiming for real-time processing. Furthermore, we avoided dimensionality reduction across the layers to preserve the original spatial dimensions of the input data, enabling pixel-wise MB classification. The kernel filter sizes were heuristically optimized to retain information about small MB textures throughout the convolution process across the 4-layers. Given a training set $\{\mathrm{X}=(\text{B-mode},\mathrm{C}\mathrm{E}\mathrm{U}\mathrm{S}),\mathrm{y}\}$, where $\mathrm{X}=(\text{B-mode},\mathrm{C}\mathrm{E}\mathrm{U}\mathrm{S})$ denotes the B-mode and CEUS patches concatenated in the channel dimension into a 256×256×2 tensor and $y$ denotes the corresponding binary ground truth tensor (256×256×1), generated using DTE, that provide the labels of the training data: $y\in \{0,1\}$ where 0 and 1 represent no MB and bound MB, respectively. The tensor, X, is passed through the first three CNN layers, where each layer includes a 2D convolutional layer, a rectified linear unit (ReLU) activation layer, and a batch normalization layer. During the convolutions, the three-dimensional input tensor is convolved with 32 kernels of size $3\times 3\times N_m$, where $N_m$ is the number of input tensors to each layer m as shown in Fig. 4 (b), e.g. N₁ = 2 and N₂ = N₃ = 32. In the last CNN layer, the input tensor is convolved with 2 kernels of size 3×3×32, and the resulting feature maps are processed using a softmax activation function to calculate MB classification scores. We opted for softmax activation instead of the more conventionally used sigmoid activation for binary classification, considering the potential expansion of our network model for multi-class classification (for example, three classes including no MB, bound MB, and free-floating MB). Thus, the four-layer CNN produced a tensor with size 256×256×2, which contains two channels of size 256×256 pixels for no MB and bound MBs. We defined the channel for bound MBs as $\stackrel{\mathbf{ˆ}}{\mathbf{y}}$, which represents the pixel-wise MB classification score of bound MB presence and is utilized as the nondestructive USMI image: thus ($1-\stackrel{\mathbf{ˆ}}{\mathbf{y}}$) represents the channel for no MB. This lightweight and basic CNN architecture was chosen to facilitate the development of a network-based USMI that could be implemented in real-time. We trained the network over 30 epochs using an Intel Core i7–12700K (12th Gen, 8 performance and 4 efficient cores, 3.6GHz base clock) with the Adam optimizer [39], a learning rate of 10⁻³, and a batch size of 32.

G. Ground Truth and Loss Function

To generate the ground truth ($\mathit{y}$) for training, we binarized the continuous scores of the raw DTE images (i.e. sham and burst), of which intensities ranged from 0 to 255, using a threshold intensity of 20. This threshold was optimized to enhance contrast in the binarized ground truth images generated from the continuous values of the raw DTE images, reflecting bound MB responses. Optimization was based on the maximum correlation between the averaged raw DTE intensities, excluding injection DTE images, and the percentages of MB labels in the binarized ground truth within each lesion. A threshold of 20 resulted in the highest Pearson correlation coefficient (PCC) of 0.99. Specifically, a threshold above the optimal value may fail to detect weak signals from MBs (lower intensities of DTE), while thresholds below the optimal value do not provide sufficient contrast for strong signals from MBs (higher intensities of DTE). The ground truth ($\mathit{y}$) and network prediction ($\stackrel{\mathbf{ˆ}}{\mathbf{y}}$) were used to calculate losses while training. The network training used a hybrid loss function (${ℒ}_{\mathit{hybrid}}$) integrating the cross-entropy loss (${ℒ}_{\mathit{XEnt}}$) and soft dice coefficient (SDC) loss (${ℒ}_{\mathit{SDC}}$) functions. We included soft dice loss as a typical loss function used for segmentation tasks, and further added cross-entropy loss to enable sensitive detection of small and fine MB textures. The hybrid loss is a weighted sum of ${ℒ}_{\mathit{XEnt}}$ and ${ℒ}_{\mathit{SDC}}$ formulated by:

$${ℒ}_{\text{hybrid}}(\hat{y},y)=(1-\beta ){ℒ}_{\mathrm{X}\mathrm{E}\mathrm{n}\mathrm{t}}(\hat{y},y)+\beta {ℒ}_{\mathrm{S}\mathrm{D}\mathrm{C}}(\hat{y},y)$$ (2)

where $0<\beta <1$ is a weighting coefficient, and

$${ℒ}_{\mathrm{X}\mathrm{E}\mathrm{n}\mathrm{t}}(\hat{y},y)=-\sum _{i\in \mathrm{R}\mathrm{O}\mathrm{I}}\left[y_i\mathrm{l}\mathrm{o}\mathrm{g}{\hat{y}}_i+\left(1-y_i\right)\mathrm{l}\mathrm{o}\mathrm{g}\left(1-{\hat{y}}_i\right)\right]$$ (3)

$${ℒ}_{\mathrm{S}\mathrm{D}\mathrm{C}}(\hat{y},y)=1-\frac{\sum _{i\in \mathrm{R}\mathrm{O}\mathrm{I}}2{\hat{y}}_i{y}_i+ϵ}{\sum _{i\in \mathrm{R}\mathrm{O}\mathrm{I}}\left({\hat{y}}_i+y_i\right)+ϵ}$$ (4)

where $ϵ$ is a smoothing term. Here, we used $\beta =0.3$ and $ϵ={10}^{-10}$ as optimized in our previous study [35].

H. Comparison Metrics

To facilitate fair comparison between different imaging approaches, all CEUS and DTE images were histogram matched [40] to the network prediction of MB classification score prior to any analysis or process described below, as shown in (Fig. 5 (a)). This histogram matching unified both (1) the scales to a range of 0 to 1 and (2) the distribution as a bimodal distribution with two peaks at 0 and 1. The network prediction follows a bimodal distribution with the two peaks located near 0 and 1, while CEUS images show a Burr distribution [41] before log-compression and DTE images show a right-skewed distribution. However, for the negative controls without MBs, network predictions rarely exhibited false-positive detections at the boundaries, resulting in no peak at 1, which does not follow a bimodal distribution. Conversely, CEUS exhibited high false-positive detection at the boundaries, yielding a peak around 1. Histogram matching CEUS to the nondestructive images distorts the CEUS image by suppressing the tissue signal leakage. To avoid improper histogram matching for negative controls, we used the average of all histograms from nondestructive images to create a reference histogram (with peaks at 0 and 1) for histogram matching with CEUS.

Fig. 5:

Schematic illustration of the process for defining: (a) molecular signal coverage, (b) histogram matching, (c) lesion boundary, and (d) reference for evaluating DTE and nondestructive performance.

To assess equivalence between images, we utilized the continuous dice coefficient (CDC) [42], which is designed for continuous measures with a binary ground truth. The CDC is defined as:

$$\mathrm{C}\mathrm{D}\mathrm{C}(\hat{y},y)=\frac{\sum _{i\in \mathrm{R}\mathrm{O}\mathrm{I}}\left(2{\hat{y}}_i{y}_i\right)+ϵ}{\sum _{i\in \mathrm{R}\mathrm{O}\mathrm{I}}\left(c{\hat{y}}_i+y_i\right)+ϵ}$$ (5)

$$c=\frac{\sum _{i\in \mathrm{R}\mathrm{O}\mathrm{I}}{\hat{y}}_i{y}_i}{\sum _{i\in \mathrm{R}\mathrm{O}\mathrm{I}}{y}_i\cdot \mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\left({\hat{y}}_i\right)}$$ (6)

Due to the stochastic nature of spurious signals (e.g. tissue-signal leakage and noise), as well as a variable distribution of biomarker in the tumor, the percentage of molecular signal coverage of the lesion ${\psi }_L$ was computed to assess general similarity between images as well as to compute CDC as a function of the amount of targeted MBs present. ${\psi }_L$ is computed as the percentage of pixels within the lesion boundary with MB probabilities greater than 0.5:

$${\psi }_L=\frac{\text{Number}\text{of}\text{pixels}\text{within}\text{a}\text{lesion}\text{with}\mathrm{M}\mathrm{B}\text{classification}\text{score}>0.5}{\text{Total}\text{number}\text{of}\text{pixels}\text{within}\text{a}\text{lesion}}\times 100\%$$ (7)

In Fig. 5 (b), the gray histogram shows all pixels within the lesion, and the counts within the black outline are considered bound MBs (MB classification score > 0.5), i.e. the molecular signal.

The lesion contour in equation 7 were manually drawn by two observers (authors JB and AN) using the B-mode images, and the two contours were spatially averaged into a single lesion contour to identify the lesion area. Similarly, to evaluate false positive detection at the boundary of the tumor where there is often tissue-signal leakage due to misalignment with pre- and post-burst images in DTE, we defined a boundary region around the tumor by applying morphological dilation and erosion to the manual contours; the morphological erosion of the contour was subtracted from morphological dilation of the contour, producing the boundary area as illustrated in Fig. 5 (c). The false-positive molecular signal coverage in this boundary region $\left({\psi }_B\right)$ is defined as:

$${\psi }_B=\frac{\text{Number}\text{of}\text{pixels}\text{within}\text{a}\text{lesion}\text{boundary}\text{with}\text{MB}\text{classification}\text{score}>0.5}{\text{Total}\text{number}\text{of}\text{pixels}\text{within}\text{a}\text{lesion}}\times 100\%$$ (8)

Because DTE performance will degrades under motion due to spatial averaging in the sweep experiments, computing the CDC between the DTE and nondestructive approach would result in an apparent degradation in performance of nondestructive imaging. Thus, we employed a comparison of DTE and our nondestructive approach with an independent, binarized, reference image developed from the CEUS image, which has previously been used as a molecular imaging approach [32] and is not sensitive to motion. While spatial averaging could be avoided by using a single pre- and post-burst frame for DTE, this approach amplifies noise from misalignment between pre- and post-burst frames. Thus, to create a reference insensitive to motion, the tumor boundaries were manually contoured in the B-mode image (because regions outside tumors rarely indicate biomarker expression) and this contour was used to mask the CEUS image.(see Fig. 5 (d)). As a molecular imaging approach [32], CEUS amplitude corresponds to the molecular signal. Thus, we set a threshold of 0.2 within the normalized scale of CEUS image intensities to differentiate molecular signal from noise, thereby binarizing each histogram-matched CEUS image within the contour. We computed ROC curves and the AUC using this reference. For ROC analysis, the true positive (TP), true negative (TN), false positive (FP), and false negative (FN) are defined as follows:

• TP: MB correctly predicted ($\hat{y}>T$ with the reference=1)

• TN: no MB correctly predicted ($\hat{y}<T$ with the reference=0)

• FP: no MB predicted as MB ($\hat{y}>T$ with the reference=0)

• FN: MB predicted as no MB ($\hat{y}<T$ with the reference=1)

where $T$ is the threshold ranges from 0 to 1. We extended comparison of nondestructive and DTE imaging to reference CEUS images for the stationary videos to provide baseline comparisons for the sweep videos.

I. Image Evaluation

For the stationary videos, we computed the CDC between the nondestructive and DTE images, the nondestructive and reference images, and the DTE and reference images. The molecular signal coverage (${\psi }_L$) and false-positive rate of molecular signal in the boundary of the lesions (${\psi }_B$) were compared for the nondestructive and DTE images using PCC. ${\psi }_B$ was separately evaluated for burst and injection DTE to observe differences due to motion. For both the stationary and sweep videos, the ROC curves and AUC were computed for nondestructive and DTE imaging using the reference to identify correct and incorrect predictions. We employed the ROC averaging approach [43] where the ROC curve was obtained by averaging all ROC curves across all frames and videos and the AUC is computed as the average of the AUCs of the individual ROC curves. ${\psi }_B$ was evaluated for the sweep videos because of expected tissue-signal leakage in the boundaries of the lesions due to motion.

When comparing DTE and nondestructive USMI based on AUC and CDC, a paired t-test or Wilcoxon signed rank test was used depending on the normality of the data as determined by the Lilliefors test. The statistical criteria for interpreting p-values were as follows: ns (no significance) p > 0.05; *p < 0.05; ** p < 0.01; *** p < 0.001; and **** p < 0.0001. Statistical analysis was performed using MATLAB (The MathWorks, Inc., Natick, MA, USA).

III. Results

A. MB Classification Imaging

Fig. 6 (a) displays representative examples of B-mode, histogram matched DTE, binarized DTE, nondestructive, and overlay of nondestructive on binarized DTE from different in vivo breast lesions with varying molecular signal coverage of the lesion. In this image, molecular images from negative control (no MBs), non-targeted MBs, and all targeted MB types are shown. For the targeted MBs, these particular images were selected for their amount of molecular signal coverage, but it should be noted that all molecular targets produced varying amounts of molecular signal coverage. The nondestructive images were very similar to the DTE images but frequently showed better suppression of tissue-signal leakage (indicated by arrows) compared to DTE. Fig. 6(a) presents a negative control case in the left most column, where the breast lesion does not contain MBs because all images were acquired prior to MB injection. The nondestructive image in this case did not show molecular signal inside the lesion (${\psi }_L=0$) and show suppression of tissue-signal leakage from the lesion boundary area, comparable to the binarized DTE. Furthermore, the supplementary video shows nondestructive USMI; the four types of MBs were injected into the same tumor on days 0, 3, 6, and 10. Due to tumor growth, varying cross sectional imaging, and different MB types, the USMI from the same tumor exhibited diverse molecular signals locations and patterns. The targeted MB cases displayed both stationary and non-stationary signals, while the non-targeted case showed non-stationary signals as there are no bound MBs.

Fig. 6:

Comparison of DTE and nondestructive images with varying molecular signal coverages of the lesion (${\psi }_L$) in the presence of non-targeted MBs, as well as VEGFR2, PD-L1, and B7-H3 trgeted MBs, along with a negative control without injected MBs. (a) B-mode, histogram matched DTE, binarized DTE, nondestructive, and nondestructive image overlaid on the binarized DTE image are displayed, with increasing ${\psi }_L$ from left to right, to visualize similarity between the imaging methods. (b) Nondestructive imaging resulted in a high correlation (PCC=0.954) with DTE for the stationary videos. (c) Similarity between DTE and nondestructive images evaluated by the continuous dice coefficient (CDC) as a function of ${\psi }_L$-threshold.

B. Quantitative Performance Evaluation

To evaluate overall performance of the stationary videos, ${\psi }_L$ was compared between nondestructive and DTE images, including injection and burst DTEs; we excluded sham DTE from this evaluation because, in most cases, the denominator of CDC in equation (5) converges to 0 when both the DTE and nondestructive imaging are accurate, resulting in a summed MB classification score of 0. ${\psi }_L$ for nondestructive USMI and DTE showed strong agreement with PCC of 0.954 (p-value < 0.0001) as shown in Fig. 6 (b).

The overlay images in Fig. 6 (a) visualize the similarity between the binarized DTE and nondestructive images, which was quantitatively evaluated by the CDC in Fig. 6 (c) and showed that their similarity increases with ${\psi }_L$. For low ${\psi }_L$, there is more spurious noise compared to detectable signals, which results in lower CDCs. With increasing molecular signal, there is greater correspondence between DTE and nondestructive images. For example, in cases with ${\psi }_L$ higher than 10 and 20%, an average CDC of 0.752 ± 0.192 (N = 44) and 0.863 ± 0.137 (N = 16) were obtained, respectively.

Using the reference image, AUC, CDC, and ${\psi }_B$ were computed for DTE and nondestructive images from the stationary videos and are presented in Fig. 7 (a-d). In Fig. 7 (a), each data point represents a video where the individual AUCs from each frame were averaged. Nondestructive imaging resulted in a significantly higher AUC than DTE (p-value < 0.0001): averaged AUC of nondestructive = 0.954 and average AUC of DTE = 0.845. Fig. 7 (b) shows the averaged CDCs with one standard deviation (shaded region). Nondestructive imaging exhibits higher CDC than DTE imaging with respect to the reference, except for ${\psi }_L$ over 20%. The false positive detection rate (${\psi }_B$) of burst and injection stationary videos, representing scans with maximum and minimum motion, respectively, are shown in Fig. 7 (c-d). While both stationary video types included respiratory and tissue motion, the injection videos contained additional global motion from animal movement when MBs were injected via the tail vein. Under minimal motion conditions (Fig. 7 (c)), nondestructive and DTE performance showed no significant difference; while under maximum motion (Fig. 7 (d)), nondestructive shows significantly lower false positive rate compared to DTE.

Fig. 7:

Quantitative evaluation of DTE and nondestructive imaging for (a-d) stationary and (e-f) sweep videos. (a) Boxplot of average AUC values of DTE and nondestructive imaging for each stationary scan of a lesion. (b) Averaged CDCs for DTE and nondestructive imaging. Nondestructive imaging showed significantly higher CDC for thresholds of 0, 5, 10, and 15 compared to DTE. (c) Comparison of false positive detection rate (${\psi }_B$) for burst DTE and nondestructive imaging; under minimal motion during the stationary videos. (d) Comparison of injection DTE with nondestructive imaging; under maximum motion during the stationary video scan. (e) Boxplot of average AUC values for DTE and nondestructive imaging for each elevation sweep of a lesion. (f) ${\psi }_B$ for DTE and nondestructive imaging under sweeping motion. Overall, there was no significant difference in performance based on the types of MBs. NTMB: nontargeted MB. TMB: targeted MB.

C. Sweep USMI

Fig. 8 shows representative USMI images from the same view of a lesion from stationary and sweep videos. DTE images were histogram-matched to the nondestructive image and the resulting molecular images were overlaid on the B-mode images. Because burst DTE is designed to use frame averaging in the ensemble (time) dimension to suppress noise and free-floating MBs, temporal frame averaging is not available or applicable to the elevational sweeps. To provide a comparable DTE image, one frame from the pre-injection sweep video was subtracted from one frame of the post-injection sweep video, which labeled as ”Single frame DTE” (Fig. 8 (e)). In addition, Fig. 8 (f) shows DTE from averaging frames while the transducer was sweeping in elevation (spatial averaging). Single frame DTE resulted in a similar appearance to the CEUS image due to the poor alignment between the pre- and post-burst images under the elevation sweep as well as the lack of temporal averaging. Thus, the DTE images incorrectly highlight tissue boundary as molecular signal, similar to CEUS. In contrast, nondestructive imaging demonstrated suppression of noise at the lesion boundary.

Fig. 8:

Comparison of DTE and nondestructive imaging for the same lesion view from stationary and sweep videos. (a) B-mode and CEUS images with VEGFR2-targeted MBs. (b) Averaged ROCs for DTE and nondestructive images from stationary and sweep videos. (c) Burst DTE with temporal averaging over 30 frames from a stationary video. (d) nondestructive image from the stationary video. (e) DTE created from subtracting a single pre-injection frame from a post-injection frame under the elevation sweep. (f) DTE with spatial averaging over 30 frames under the elevation sweep. Histogram matched USMI images are overlaid on the B-mode images below each USMI image.

In Fig. 7 (e), a boxplot of the average AUC values is shown for nondestructive and DTE imaging, with each point representing an average AUC over all frames in a sweep. The averaged ROC curves for nondestructive and DTE are shown in Fig. 8 (b). Nondestructive showed a significantly higher AUC than DTE (0.953 vs. 0.892, p-value < 0.0001). Quantitative evaluation of the false-positive MB detection rate on the tumor boundary area (${\psi }_B$) is shown in Fig. 7 (f), where each point represents the average ${\psi }_B$ over all frames in a sweep video and a box plot of the average ${\psi }_B$ is overlaid for each imaging mode. Burst DTE resulted in significantly higher false-positive signal in the tumor boundaries than nondestructive imaging.

Moreover, Fig. 8 (c) shows the DTE image from temporal averaging in the stationary video, which closely matches the nondestructive image. The ROC curves and AUCs for nondestructive and DTE imaging of all experiments are compared in Fig. 8 (b). The single frame DTE image yielded a higher AUC (0.892) than temporal averaged stationary DTE (AUC = 0.845), despite exhibiting more tissue-signal leakage, higher noise levels, and less highlighted MBs compared to stationary DTE. For the DTE images with spatial averaging for the sweep videos, the relatively significant transducer motion produced degraded molecular images and decreased the AUC to 0.751. Overall, nondestructive imaging resulted in the highest AUC (0.954 and 0.953), effective detection of MBs, best suppression of false positive detection at the lesion boundary.

D. Generalizability: Kidney Cancer Model

Our nondestructive model, trained in this study to detect breast cancer, was applied to another in vivo dataset involving a RenCa mouse model of kidney cancer, as shown in Fig. 9. While the training and inference were conducted using different cancers, they utilized the MBs targeting the same biomarkers: non-targeted MBs and B7-H3 and PD-L1 targeted MBs. The kidney cancer case presented in Fig. 9 (a) (B7-H3 targeted MBs) was scanned using the same US system and gain setting employed in this study, whereas the cases in Fig. 9 (b) and (c) were scanned using a different Vevo 2100 system under different gain settings; the gain was adjusted using the time gain compensation (TGC) sliders on each system. Our nondestructive imaging successfully detected MBs comparable to DTE images for all MB types. However, for the cases acquired under different TGC settings (Fig. 9 (b-c)), our nondestructive imaging detected noise beneath the tumors as MBs. This was likely due to the TGC gain sliders being set to higher values than that used for acquiring the training dataset.

Fig. 9:

USMI of a RenCa mouse model of kidney cancer. B-mode, CEUS, DTE, and nondestructive; CEUS and DTE were histogram matched to nondestructive. The nondestructive imaging model trained in this study for breast cancer was applied to the kidney cancer model. Different MBs were injected at different time points: (a) B7-H3 targeted MBs; (b) PD-L1 targeted MBs; (c) non-targeted MBs. In (b-c), inconsistent gain settings between training and testing caused the false-positive detection of noise beneath the tumors as MBs

IV. Discussion

We developed and validated a neural network-based non-destructive USMI modality trained with B-mode and CEUS videos. The trained network is capable of frame-by-frame MB prediction as an image, and yielded an inference time of 58.82 ms, demonstrating the potential for a real-time nondestructive USMI modality and permitting free-hand scanning. Additionally, our model resulted in 2.5 GFLOPS, making it lighter than 10.8 GFLOPS of ResNet-50 [44], which is recognized as one of the best-performing CNN architectures, albeit with a heavy architecture. Moreover, our model demonstrated comparable computational cost to lightweight neural networks such as SqueezeNet [45] (2.2 GFLOPS) and MobileNet [46] (0.8 GFLOPS), balancing efficiency and computational complexity. The network successfully predicted localized MB positions without destroying MBs and resulted in highly correlated performance with burst DTE imaging, based on qualitative image comparisons in Fig. 6(a) and quantitative metric comparisons in Figs. 6(b-c): PCC = 0.954 (p-value < 0.0001) for ${\psi }_L$; CDC = 0.863 ± 0.137 for ${\psi }_L$ greater than 20%.

Under a stationary transducer, nondestructive USMI yielded higher AUC (0.954) compared to DTE (0.845). The higher AUC of nondestructive imaging in this case can be explained by the sensitivity of DTE to motion. The false positive prediction near tumor boundaries in DTE are due to target misalignment between pre- and post-burst/injection images caused by the animal’s global and respiratory motion, whereas nondestructive imaging is not affected by motion because it is dependent only on a single image for prediction. The misalignment of the averaged frames required to produce DTE due to motion led to a significantly higher false-positive detection rate in the boundary area (${\psi }_B$) in Fig. 7 (d) compared to the minimal motion condition as shown in Fig. 7 (c). Similarly, in the elevational sweep experiments that modeled free-hand molecular imaging scanning, nondestructive imaging yielded a higher AUC (0.953) compared to single-frame burst DTE (0.892). This difference is due to the misalignment of the individual pre- and post-burst frames. Reducing the number of frames in burst DTE to minimize error due to animal motion also results in greater tissue-signal leakage, and also leads to more false positive signals because it is unable to suppress the free-floating MBs. Both of these challenges make the single-frame DTE image in Fig. 8(e) more similar to the CEUS image than the nondestructive image. Thus, as shown in Fig. 8, a sweep DTE with “single-frame” processing (AUC = 0.892) appeared superior to stationary DTE with temporal averaging (AUC = 0.845). Applying spatial averaging to sweep DTE, which reduces the similarity to the ground truth of CEUS, resulted in AUC of 0.751, lower than the AUC (0.845) of stationary DTE. It is also worth noting the consistency of the AUCs for the nondestructive imaging between the two experiments where the transducer is stationary (AUC = 0.954) and in a sweeping motion (AUC = 0.953), highlighting the robustness of the nondestructive to all types of motion. Thus, nondestructive imaging is advantageous for US scanning with free-hand motion as is conventionally performed.

Our previous study [35] developed a similar CNN-based nondestructive USMI modality, trained using ground truth generated by short-lag spatial coherence (SLSC) beamforming [47] to reduce motion artifacts and better differentiate free and bound MBs, and concluded that element IQ data input was superior to other types of input data formats for producing DTE-like quality; however, the accessibility of element IQ data from commercial US machines is very limited and SLSC processing is computationally demanding. Consequently, that approach is not easily translatable to existing commercial scanners. In contrast, we trained the same CNN architecture with US videos, which are easily accessible from commercial systems. Although we used post-processed data, which showed poorer performance than element IQ data [35], we achieved enhanced performance of the CNN-based USMI while being relatively less computationally intensive. Although a direct performance comparison is not feasible due to the different data types acquired in [35] compared to this work (element IQ data vs post-processed data), we compared AUCs for nondestructive imaging, where both utilized the training ground truth for AUC calculations. Our approach yielded an AUC (0.93) on par with the best AUC (0.91) reported in [35]. This was enabled by our data augmentation strategies, incorporating refined DTE techniques and extracting independent features by imaging lesions with distinct MB types, as well as our improved ground truth through the reduction of motion artifacts in DTE images by excluding frames with respiratory motion. Furthermore, we demonstrated that the CNN-based nondestructive imaging expands beyond DTE’s original imaging capabilities by allowing it to operate under motion and free-hand scanning while also avoiding the destruction of the contrast agent.

The evaluation of nondestructive USMI here is limited by the use of DTE and CEUS as references. First, higher intensities were defined as MBs in these ground truth images, but they can also contain false positive signals, including noise and high amplitude tissue-leakage signals. For example, the right-most case in Fig. 6 (a) highlighted background noise on the left side of the image where there are no MBs. These false positive references cause a decrease in the evaluation accuracy. When the MB concentration is relatively low in the CEUS image, the molecular signal is mostly from noise rather than MBs. In Fig. 6 (c), lower molecular signal coverage (${\psi }_L$, left images) resulted in lower CDC measures compared to the higher molecular signal coverage (right images). This is largely due to lower reliability of the evaluation metric at low ${\psi }_L$ and does not necessarily mean nondestructive USMI has lower performance for lower molecular signal coverage. Second, we utilized reference images based on CEUS imaging to compare the AUC for DTE and nondestructive imaging. The frame-by-frame processing of the single-frame DTE and nondestructive imaging accounts for some of the observed higher AUC (0.892 and 0.954, respectively) compared to the conventional burst DTE (0.845) because the same CEUS image is used as the reference while averaged CEUS images are used for conventional DTE. However, the higher AUC for nondestructive imaging compared to single-frame DTE is due to the misalignment of pre- and post- images. Nevertheless, our findings demonstrated that nondestructive imaging is superior to DTE when scanning is conducted under motion. DTE resulted in significantly higher ${\psi }_L$ for injection stationary and sweep videos, which were subject to motion, whereas ${\psi }_L$ was equivalent between DTE and nondestructive imaging when motion was minimized in the burst stationary videos, as shown in Fig. 7 (c), (d), and (f)

A major assumption of DTE imaging is that it suppresses signals from free-floating (unbound) MBs by averaging frames; however, we found that DTE images displayed molecular signal from these unbound MBs as shown in the detected molecular signals in Fig. 6 (a), second column, and the molecular signal coverage in Fig. 6 (b). This means that the averaging process employed by DTE did not fully or always suppress signals from free-floating MBs. Thus, unbound MBs were also labeled as targeted MBs in the ground truth. Similarly, and despite the robustness to animal and transducer motion, the nondestructive method is also susceptible to detecting free-floating MBs as bound MBs. While the free-floating MBs are in motion, the frame-by-frame processing of the nondestructive method can treat the free-floating MBs as a stationary signal. Therefore, further development of the nondestructive approach is necessary to either improve the ground truth in order to label only bound MBs and avoid free-floating MBs or design a network architecture that utilizes temporal information to filter out free-floating MB signal. Preliminary work with other methods has demonstrated potential solutions. Robust principal component analysis (RPCA) effectively distinguishes between bound and free-floating MBs [48], which can be employed to label only bound MBs in ground truth. Additionally, transformer networks, which processes spatio-temporal data, can be employed to reduce the detection of free-floating MBs [49].

This work has advanced the CNN-based nondestructive imaging from our previous studies [35], [37] to facilitate its adaptation for commercial scanners and clinical applications. We have identified several areas of improvement for development of this nondestructive approach. First, the network architecture could be enhanced by adopting state-of-the-art models, such as transformers [50], to improve the detection and visualization of bound MBs through the use of temporal information, as demonstrated by preliminary work [49].

Second, the ground truth could be refined by mitigating motion artifacts and unbound MBs, which would improve the identification of bound MBs in the DTE ground truth. Our previous work [35] conducted image registration between pre- and post-destruction frames using the ”imregister” function in MATLAB; however, this approach resulted in only a small improvement in the DTE images. Therefore, more advanced processing techniques are necessary, such as singular value decomposition [51], [52] and RPCA [48]. Specifically, RPCA has demonstrated effective removal of motion artifacts and unbound MBs in USMI, leading to improved detection of bound MBs. In human subjects, motion artifacts are likely to arise from respiratory motion and free-hand scanning. We expect that the level of these artifacts will fall between those observed in burst DTE and injection DTE of animal scans, due to the lower frequency of respiratory motion in humans compared to small animals. Thus, we anticipate that these motion filtering techniques, including RPCA, can effectively suppress the motion effects in humans.

Third, we recognize the potential of our nondestructive imaging framework to be applied to other disease models, which may necessitate additional training and validation of the model within specific diseases. Our nondestructive model, trained using breast cancer, successfully visualized MBs comparable to DTE images of kidney cancer, as shown in Fig. 9; however, it also revealed limitations of the model due to the low signal-to-noise ratio (SNR) caused by different gain settings. Thus, the model would also benefit from training on a broader range of gain and SNR settings to enhance its robustness. Two strategies can be employed. First, we could acquire additional data from more animals using varying gain and TGC settings. Second, we could artificially augment the dataset after acquiring images under high SNR settings. This would involve characterizing the noise present in the images and incorporating appropriate noise levels into the high SNR dataset, while adjusting gain or TGC artificially by adding or subtracting dB scale pixel values as part of the post-processing. However, our patch augmentation incorporated a broad range of resolutions, meaning that this model was robust to variations in frequency and resolution, although significant changes in frequency, transducers, and scanners would require training on appropriate data. Furthermore, because the model only employs a single frame of B-mode and CEUS image in its prediction, we do not anticipate that varying frame rates will affect the performance of the network. Generalization across all systems and settings is likely not a feasible goal; however, this limited dataset approach could be used to rapidly build a set of nondestructive models that could be applicable to more limited generalizations, such as to transducer form factors, scanner types, and frequency/resolution.

Finally, clinical translation could be made more feasible by aligning the experimental data acquisition with those used in clinical practice. Several factors could potentially affect translation to human imaging, such as the “easy target” produced by the mouse model (shallow depth, low noise/attenuation, etc.), and the variability of the training data. To mitigate these issues, our earlier work [35] employed training data where mouse tumors had porcine tissue placed atop the tumor, thereby simulating conditions that more closely resemble those encountered in human imaging with lower SNR. Additionally, although our training data was acquired within the narrower timeframe compared to that for human practice, it included various levels of marker expression on US as shown in Fig. 9 (a) that range from little or no molecular signal coverage to significant molecular signal coverage, thereby making the dataset representative of a broad range of molecular signal coverage. Furthermore, we could use the current network, trained with mouse model data, as a pre-trained model to fine-tune the network on human scans. We anticipate that these strategies could enhance the model’s generalization when adapted for inference on human subjects.

V. Conclusion

A neural network-based USMI modality was developed to enable nondestructive MB detection for free-hand scanning capabilities that would easily translate to clinical US systems. We trained and validated the approach using US videos of a transgenic mouse model of breast cancer and compared its performance to the current state-of-the-art USMI modality, namely DTE imaging. To allow accurate network training with the limited number of enrolled animals (N = 14), we introduced modified DTE approaches, including sham DTE and injection DTE, with the sliding window technique alongside the MB augmentation approach, creating independent features from the same lesions by injecting four distinct MB types, to increase training samples. Nondestructive USMI outperformed DTE USMI based on CDC and ROC analysis. Moreover, nondestructive imaging avoids destroying MBs during scanning, thereby potentially allowing free-hand and repeated scanning of tumors without requiring knowledge of their location prior to scanning while also overcoming any compounding tissue damage. These findings demonstrated the feasibility of neural-network-based approach to nondestructive USMI that could be more clinically useful by resolving issues that limit DTE imaging to preclinical imaging.