Noise reduction with cross-tracer and cross-protocol deep transfer learning for low-dose PET

Abstract

Previous studies have demonstrated the feasibility of reducing noise with deep learning-based methods for low-dose fluorodeoxyglucose (FDG) positron emission tomography (PET). This work aimed to investigate the feasibility of noise reduction for tracers without sufficient training datasets using a deep transfer learning approach, which can utilize existing networks trained by the widely available FDG datasets. In this study, the deep transfer learning strategy based on a fully 3D patch-based U-Net was investigated on a ¹⁸F-fluoromisonidazole (¹⁸F-FMISO) dataset using single-bed scanning and a ⁶⁸Ga-DOTATATE dataset using whole-body scanning. The datasets of ¹⁸F-FDG by single-bed scanning and whole-body scanning were used to obtain pre-trained U-Nets separately for subsequent cross-tracer and cross-protocol transfer learning. The full-dose PET images were used as the labels while low-dose PET images from 10% counts were used as the inputs. Three types of U-Nets were obtained: a U-Net trained by FDG dataset, a pre-trained FDG U-Net fine-tuned by another less-available tracer (FMISO/DOATATE), and a U-Net completely trained by a large number of less-available tracer datasets (FMISO/DOATATE), used as the reference U-Net. The denoising performance of the three types of U-Nets was evaluated on single-bed ¹⁸F-FMISO and whole-body ⁶⁸Ga-DOTATATE separately and compared using normalized root-mean-square error (NRMSE), signal-to-noise ratio (SNR), and relative bias of region of interest (ROI). For cross-tracer transfer learning, all the U-Nets provided denoised images with similar quality for both tracers. There was no significant difference in terms of NRMSE and SNR when comparing the former two U-Nets with the reference U-Net. The ROI biases for these U-Nets were similar. For cross-tracer and cross-protocol transfer learning, the pre-trained single-bed FDG U-Net fine-tuned by whole-body DOTATATE data provided the most consistent images with the reference U-Net. Fine-tuning significantly reduced the NRMSE and the ROI bias and improved the SNR when comparing the fine-tuned U-Net with the U-Net trained by single-bed FDG only (NRMSE: 96.3% ± 21.1% versus 120.6% ± 18.5%, ROI bias: −10.5% ± 13.0% versus −14.7% ± 6.4%, SNR: 4.2 ± 1.4 versus 3.9 ± 1.6, for fine-tuned U-Net and the U-Net trained by single-bed FDG, respectively, with p < 0.01 in all cases). This work demonstrated that it is feasible to utilize existing networks well-trained by FDG datasets to reduce the noise for other less-available tracers and other scanning protocols by using the fine-tuning strategy.

1. Introduction

Positron emission tomography (PET) has been widely applied for the diagnosis of various diseases in oncology, neurology and cardiology (Schwaiger et al 2005, Fletcher et al 2008, Catana et al 2012). The quality of PET images is highly dependent upon the injection dose of radioactive tracers, causing radiation exposure to the patients (Cherry et al 2012). Reducing the injection dose is always critical under the principle of as low as reasonably achievable (Strauss and Kaste 2006). With the reduced tracer injection, low-dose PET images will have higher noise and lower signal-to-noise ratio (SNR) when compared with full-dose PET images (Chang et al 2011, Yan et al 2016). To address the challenge of higher noise in the low-dose PET images, a variety of hardware and software-based approaches have been developed (Badawi et al 2019, Cherry et al 2018, Riddell et al 2001, Conti 2011; Chan et al 2010). Recently, deep learning-based methods have been established to reduce the image noise and recover the image quality for the low-dose PET images with promising performances (Kang et al 2015, Xiang et al 2017, Xu et al 2017, Gong et al 2018, Lu et al 2019).

To our knowledge, existing works on deep learning-based noise reduction have been focused on using images of the same tracer, typically ¹⁸F-fluorodeoxyglucose (¹⁸F-FDG), in the network training and testing. Despite the diversity of PET tracers, ¹⁸F-FDG remains the most widely used tracer in clinical practice (Lopci et al 2010), providing a large number of datasets for network training (Simonyan and Zisserman 2014). However, for many non-FDG PET tracers, particularly the new tracers that are not widely available and the tracers with very long or very short half-lives, it could be challenging to obtain sufficient datasets for the network training. The tracers with very long half-lives, such as ⁸⁹Zr-labeled tracers (3.3 d half-life), require a low injection dose to reduce radiation to patients, causing high noise even in the full-dose images (Jauw et al 2019). On the other hand, tracers with very short half-lives, such as ¹⁵O-water (122.2 s) and ⁸²Rb (76.4 s), decay very fast during scanning, also causing high noise in the full-dose images. Both tracer categories have the problem of lacking a large quantity of imaging studies, which is a major challenge in the network training for deep learning-based noise reduction for such non-FDG tracers.

To address this challenge, we investigated if U-Net networks trained by ¹⁸F-FDG data can be applied to other non-FDG tracers, that is to say whether the denoising performance of U-Net networks trained by ¹⁸F-FDG data is not worse than that of U-Net networks trained by the non-FDG data while reducing the noise for the non-FDG tracers. We also proposed to apply transfer learning on U-Net networks to improve the performance of noise reduction for the non-FDG tracers (Liu et al 2019). In deep transfer learning, the U-Net networks initially trained by ¹⁸F-FDG data were subsequently fine-tuned with a small number of datasets of another non-FDG tracer. In this study, we used two non-FDG tracers with different acquisition protocols as examples: ¹⁸F-fluoromisonidazole (¹⁸F-FMISO) with single-bed acquisition protocol and ⁶⁸Ga-DOTATATE with whole-body acquisition protocol (Virgolini et al 2010, Hendrickson et al 2011). We investigated cross-tracer transfer learning by applying the U-Net trained with single-bed ¹⁸F-FDG to denoise the low-dose PET images of single-bed ¹⁸F-FMISO and applying the U-Net trained with whole-body ¹⁸F-FDG to denoise the low-dose PET images of whole-body ⁶⁸Ga-DOTATATE. In addition, we investigated cross-tracer and cross-protocol transfer learning by applying the U-Net trained with single-bed ¹⁸F-FDG to denoise the low-dose PET images of whole-body ⁶⁸Ga-DOTATATE. To evaluate the feasibility of deep transfer learning, we obtained the U-Net completely trained by the non-FDG tracer, which was used as the reference. The denoising performance of the proposed transfer U-Net strategy, was compared with the reference U-Net in terms of normalized root-mean-square error (NRMSE), SNR and relative bias of the regions of interest (ROI).

2. Materials and methods

2.1. Study subjects and image generation

In this study, we included patient studies with two different acquisition protocols, single-bed protocol and whole-body protocol with continuous-bed-motion scanning, on a Siemens Biograph mCT PET/CT scanner. For each protocol, we used the data of two different tracers to evaluate the proposed deep transfer learning strategy: ¹⁸F-FDG and ¹⁸F-FMISO for the single-bed protocol, as well as ¹⁸F-FDG and ⁶⁸Ga-DOTATATE for the whole-body protocol. With ¹⁸F-FDG as the widely used tracer, ¹⁸F-FMISO and ⁶⁸Ga-DOTATATE were used as two examples of the less-available non-FDG tracers. As a result, this study included four groups of datasets: single-bed FDG, single-bed FMISO, whole-body FDG and whole-body DOTATATE.

The single-bed FDG group included a total of nine scans from nine patients (age: 58 ± 9 years, BMI: 30.0 ± 11.1 kg m⁻²) with CT detected lung nodules with an intravenous injection of 340 ± 30 MBq ¹⁸F-FDG. Each patient underwent one single-bed scan for 20 min at 60 min post-injection. Similarly, the single-bed FMISO group included a total of 12 scans from six lung cancer patients (age: 62 ± 8 years, BMI: 24.1 ± 6.3 kg m⁻²) with a 181 ± 4 MBq ¹⁸F-FMISO injection. Two patients underwent one single-bed scan, another two patients underwent two single-bed scans for each with a 4 d interval, and the other two patients underwent three single-bed scans for each with a 2 d interval between scans. All FMISO scans were acquired as dynamic data starting immediately after the tracer injection for 120 min. In this study, 50 min list-mode data after 70 min post-injection were used to ensure the noise levels of the single-bed FMISO images were consistent with those of the single-bed FDG images in terms of similar normalized standard deviation in the central field of view.

The whole-body FDG group included a total of 12 scans from 12 patients (age: 67 ± 15 years, BMI: 24.7 ± 3.2 kg m⁻²) with an injection of 374 ± 22 MBq ¹⁸F-FDG. Each patient underwent one continuous-bed-motion scan for 16.7 ± 3.8 min at 68.7 ± 11.1 min post-injection. The whole-body DOTATATE group included a total of 15 scans from 15 patients (age: 65 ± 10 years, BMI: 26.8 ± 5.8 kg m⁻²) with an injection of 130 ± 30 MBq ⁶⁸Ga-DOTATATE. Each patient underwent one continuous-bed-motion scan for 21.6 ± 3.2 min at 68.5 ± 10.6 min post-injection. The speed of continues-bed-motion protocol used for both groups was 1.5 mm s⁻¹. The noise levels of the images in these two groups were consistent in terms of similar normalized standard deviation in the abdominal region.

For each dataset in all the four groups, a full-count sinogram was generated with the acquired full-count list-mode data. Then ten independent replicates of low-count sinograms (each containing 10% counts) were generated by uniform sampling of the full-count list-mode data with Siemens E7 tool. All the images were reconstructed using the ordered subset expectation maximization algorithm (21 subsets and 3 iterations) with corrections for attenuation, scatter, normalization and decay. For the single-bed protocol, the image size was 400 × 400 × 109. For the whole-body protocol, the image size was 400 × 400 in the transverse plane and 509 ± 165 in the axial direction depending on patient height. The voxel size was 2.04 × 2.04 × 2.03 mm³ for both protocols. Finally, we obtained ten pairs of low-dose and full-dose PET images from each scan.

2.2. U-Net architectures and training

In this study, we implemented a fully 3D deep convolutional neural network using a patch-based U-Net architecture for noise reduction (Ronneberger et al 2015, Lu et al 2019). The U-Net architecture, as shown in figure 1, included a contracting path, a bottleneck, an expanding path and skip connections between the contracting path and the expanding path. In our network, the contracting path consisted of three stages, each including two stacked layers: one 3 × 3 × 3 convolution layer with a rectified linear unit and one 2 × 2 × 2 max pooling layer. The bottleneck included two 3 × 3 × 3 convolution layers. The expanding path consisted of three stages, each including two stacked layers: one 3 × 3 × 3 convolution layer with a rectified linear unit and one 2 × 2 × 2 up-convolution layer.

Figure 1.

The U-Net architecture.

For the conventional training, the U-Net was trained from scratch, meaning all the weights of the U-Net were initialized with random values. The pair of low-dose and full-dose PET images in each group (single-bed FDG, single-bed FMISO, whole-body FDG and whole-body DOTATATE) were used as the network input and label to train each U-Net separately. In each epoch, patches with the size of 64 × 64 × 16 voxels were randomly selected from the training images. A total of 480 epochs, each including 100 batches with a batch size of 16 patches, were used to ensure the convergence of the training. The Adam optimizer was applied with an initial learning rate of 0.0001 and exponential decay rate of 0.996. The L2 loss function was used in training.

For the deep transfer learning strategy, the U-Net was initialized with the well-trained FDG U-Net from the conventional training, which was pre-trained using the datasets in the single-bed or whole-body FDG group as the training data. Then a small number of datasets in the non-FDG group (single-bed FMISO or whole-body DOTATATE) were used to fine tune the pre-trained U-Net (Weiss et al 2016). For the fine-tuning step, only the weights of the first layer and the final layer were updated iteratively. A total of 80 epochs, each configured with 100 batches and 16 patches in each batch were used. The Adam optimizer was applied with an initial learning rate of 0.0001 and exponential decay rate of 0.999.

All the experiments were carried out on a Dell workstation with NVIDIA Titan Xp GPU.

2.3. Cross-tracer U-Nets

To investigate the deep transfer learning strategy for denoising the low-dose images of the less-available tracers, we trained three different kinds of U-Nets for single-bed FMISO and whole-body DOTATATE data, respectively, resulting in a total of six networks. The training and testing strategies are detailed in table 1. U-Net A was trained with the data of a non-FDG tracer, i.e. single-bed FMISO or whole-body DOTATATE. U-Net B was trained with single-bed and whole-body FDG data for single-bed FMISO and whole-body DOTATATE, respectively. Using the proposed deep transfer learning strategy, U-Net C was initialized with U-Net B and fine-tuned with three scan data from the non-FDG tracer group. To test all the scans in the non-FDG tracer group without the correlation between training and testing data, a ‘leave-3-out’ cross-validation approach was used for U-Net A, resulting in four replicate networks trained for single-bed FMISO and five replicate networks trained for whole-body DOTATATE. The leave-3-out approach is to leave out three scans to be tested, while each scan includes ten replicates of 10% patient data. Similarly, two replicate networks, each fine-tuned with different sets of three scans, were trained for U-Net C for single-bed and whole-body data. In this study, U-Net A was used as the reference U-Net to calculate the denoising performance for the conventional deep learning approach.

Table 1.

Description of cross-tracer U-Nets for denoising of single-bed FMISO and whole-body DOTATATE data.

	Single-bed data	Whole-body data
U-Net A trained from scratch using:	9 FMISO scans	12 DOTATATE scans
U-Net B trained from scratch using:	9 FDG scans	12 FDG scans
U-Net C fined tuned based on U-Net B using:	3 FMISO scans	3 DOTATATE scans
Testing using:	12 FMISO scans	15 DOTATATE scans

2.4. Cross-tracer and cross-protocol U-Nets

In addition to investigating cross-tracer denoising for the same acquisition protocol, meaning that the training and testing data are both single-bed data or whole-body data, we also investigated denoising with cross-tracer and cross-protocol training data, as detailed below. For this investigation, in addition to U-Net A (the reference U-Net), we obtained another set of two different kinds of U-Nets and subsequently tested them on whole-body DOTATATE data for noise reduction. The training and testing strategies are detailed in table 2. U-Net D was trained with single-bed FDG data. U-Net E was initialized with U-Net D and fine-tuned with three whole-body DOTATATE scans. For testing, five replicate networks were trained for U-Net A with a ‘leave-3-out’ cross-validation approach, and two replicate networks (each with three different whole-body DOTATATE scans) were trained for U-Net E.

Table 2.

Description of cross-tracer and cross-protocol U-Nets for denoising of whole-body DOTATATE data.

	Data
U-Net A trained from scratch using:	12 whole-body DOTATATE scans
U-Net D trained from scratch using:	9 single-bed FDG scans
U-Net E fined tuned based on U-Net D using:	3 whole-body DOTATATE scans
Testing using:	15 whole-body DOTATATE scans

2.5. Evaluation

In this study, the denoising performance of the U-Net network was evaluated using the NRMSE between the full-dose image and the denoised low-dose image, the lesion SNR (Surti and Karp 2008) and the normalized ROI bias, as detailed below.

Using the full-dose PET image as the ground truth, the NRMSE between the full-dose image and the denoised low-dose image was calculated as:

$$NRMSE=\frac{\sqrt{\sum _{i=1}^N\mathrm{ }\mathrm{ }{\left({x^{\prime }}_i-x_i\right)}^2/N}}{\sum _{i=1}^N\mathrm{ }{x^{\prime }}_i/N},$$ (1)

where $x_i$ and $x_i^{\prime }$ are the values of voxel $i$ in the denoised image and the full-dose image, respectively; $N$ is the total number of voxels in the image.

To quantify the SNR and the normalized ROI bias, the ROIs were manually drawn on the aligned CT images. We obtained the ROIs of 17 lung nodules (12 009 ± 28 683 voxels) and 12 lung backgrounds (6449 ± 123 voxels) for the single-bed FMISO group and the ROIs of 20 liver lesions (5108 ± 8018 voxels), 15 normal livers (146 392 ± 43 227 voxels) and 15 lung backgrounds (5003 ± 520 voxels) for the whole-body DOTATATE group. Then the SNR and the normalized ROI bias were calculated as follows:

$$\mathrm{S}\mathrm{N}\mathrm{R}=\frac{{\lambda }_n-{\lambda }_b}{\sqrt{\left({\sigma }_n^2+{\sigma }_b^2\right)/2}},$$ (2)

$$\text{Bias}=\frac{{\lambda }_n-{\lambda }_n^{\prime }}{{\lambda }_n^{\prime }}\times 100\mathrm{\%},$$ (3)

where ${\lambda }_n$ and ${\sigma }_n$ are the mean and the standard deviation of the nodule/lesion/liver ROI in the evaluated PET images, including full-dose PET image, low-dose PET image and denoised PET image; ${\lambda }_b$ and ${\sigma }_b$ are the mean and the standard deviation of the background ROI inside lung in the evaluated PET image; ${\lambda }_n^{\prime }$ is the mean of the nodule/lesion/liver ROI in the full-dose PET image.

In order to investigate if the U-Nets trained by FDG data can be applied to the non-FDG tracers, we implemented statistical analysis on these evaluation indices to compare the denoising performance between the reference U-Net (U-Net A) and the other FDG-trained U-Nets with or without further fine-tuning. For evaluating the cross-tracer transfer learning, two-tailed paired student’s t-test was implemented on both NRMSEs and nodule biases to compare the performance of U-Nets B and C to that of the reference U-Net A. The linear regression analysis was performed on SNRs between the full-dose images and the low-dose images, as well as between the full-dose images and the denoised images with U-Nets A, B and C, separately. The Pearson correlation coefficients obtained from the linear regression plots for U-Nets B and C were compared with those for the reference U-Net A, respectively, using Pearson and Filon’s z test (Diedenhofen and Musch 2015). For evaluating the cross-tracer and cross-protocol transfer learning, one-tailed paired student’s t-test was used to compare the NRMSEs and ROI biases between U-Net E and U-Net D, aiming to investigate whether the denoising performance after fine tuning is better. Two-tailed paired student’s t-test was also used to compare the SNRs obtained from U-Nets D, E to those obtained from the reference U-Net A.

3. Results

3.1. Comparison of cross-tracer U-Nets

As shown in figure 2, all three U-Nets (U-Nets A, B and C) effectively reduced the image noise when comparing with the low-dose PET images for both single-bed FMISO and whole-body DOTATATE. Visually, the denoised low-dose images with U-Nets B and C were similar to those with the reference U-Net A.

Figure 2.

Sample slices of full-dose images, low-dose images and denoised images with U-Nets A, B, C for single-bed FMISO and whole-body DOTATATE, respectively. The white arrows indicate the lung nodule in FMISO image and liver lesion in DOTATATE image.

The NRMSEs between the full-dose PET images and the denoised low-dose PET images with the three U-Nets, as well as the NRMSEs between the full-dose and the low-dose PET images are shown in figure 3 for both single-bed FMISO and whole-body DOTATATE. The NRMSEs were greatly reduced after denoising for both protocols. There was no significant difference when comparing the NRMSE of U-Net B and U-Net C to that of the reference U-Net A (p = ns) for both protocols, except for the case between U-Net B and U-Net A for whole-body DOTATATE but the NRMSE difference was still smaller than 3%.

Figure 3.

The NRMSE of low-dose PET images and denoised images with U-Nets A, B, and C using full dose image as ground truth for single-bed FMISO ((a), n = 12) and whole-body DOTATATE ((b), n = 15).

The results of SNR regression analysis between the full-dose images and the low-dose images, as well as between the full-dose images and the denoised low-dose images with U-Nets A, B and C are shown in figure 4. High correlation coefficients were observed in DOTATATE imaging and reasonable correlation coefficients were observed in FMISO imaging for all three U-Nets, indicating that the SNR improvement benefiting from the U-Net noise reduction was consistent across all the scans included in this study. The improvement in SNR after noise reduction was similar for the three U-Nets used. There was no significant difference for the correlation coefficient when comparing U-Nets B, C to U-Net A (p = ns), indicating the denoising performance in terms of SNR improvement was similar for the three U-Nets.

Figure 4.

The SNR regression plot for low-dose PET images and denoised images with U-Nets A, B, and C when compared to the full-dose PET images for single-bed FMISO (n = 17) and whole-body DOTATATE (n = 35).

Using the full-dose images as the ground truth, the mean and standard deviation values of the ROI biases for the low-dose images and the denoised images with the three U-Nets are shown in figure 5 for both protocols. The ROI biases of the denoised images with the three U-Nets were similar for both protocols while U-Net C provided a slightly smaller ROI bias when compared with U-Net B.

Figure 5.

The ROI bias for low-dose images and denoised images with U-Nets A, B, and C when using the full-dose images as ground truth for single-bed FMISO (a) and whole-body DOTATATE (b).

3.2. Comparison of cross-tracer and cross-protocol U-Nets

The denoised PET images of whole-body DOTATATE using the cross-tracer and cross-protocol U-Nets are shown in figure 6. With U-Net D, noise was still noticed in the denoised image. When comparing with U-Net A (the reference U-Net), U-Net E provided similar denoised images and consistent lesion appearance, indicating that the use of fine-tuning may improve the denoising performance of U-Net trained with the cross-tracer and cross-protocol datasets.

Figure 6.

Samples of full-dose PET images, low-dose PET images and denoised images with U-Nets A, D, and E for whole-body DOTATATE.

Figure 7 shows the quantitative comparison results. As shown in figure 7(a), the NRMSEs were greatly reduced after denoising with all the U-Nets. The NRMSEs of U-Net E were significantly smaller than that of U-Net D (96.3% ± 21.1% versus 120.6% ± 18.5%, p < 0.001). Using U-Net A as the reference U-Net (90.6% ± 18.0%), the NRMSE differences between U-Net E and U-Net A were also significantly smaller than that between U-Net D and U-Net A (5.6% ± 3.5% versus 30.0% ± 11.1%, p < 0.001). As shown in figure 7(b), the ROI biases of U-Net E were significantly smaller than that of U-Net D (−10.5% ± 13.0% versus −14.7% ± 6.4%, p < 0.001), although they were still a little bit higher than that of U-Net A (−5.7% ± 9.2%). For the SNR shown in figures 7(c) and (d), the SNR was improved after noise reduction with all the U-Nets when compared with the low-dose image. The SNRs of U-Net E were significantly higher than that of U-Net D (4.2 ± 1.4 versus 3.9 ± 1.6, p < 0.01). These results indicated that the deep transfer learning strategy based on the single-bed FDG data and fine-tuned with a small number of whole-body DOTATATE datasets can effectively improve the denoising performance of U-Net for denoising the whole-body DOTATATE data.

Figure 7.

The NRMSE (a) and ROI bias (b) of low-dose PET image and denoised images with U-Nets A, D, and E for whole-body DOTATATE, SNR (c) of these images and full-dose PET images, and SNR regression plot (d) between these images and full-dose PET images.

4. Discussion

Using FMISO and DOTATATE as examples, the feasibility of noise reduction for low-dose PET images of those less-available non-FDG tracers potentially without sufficient training data was investigated in this study using the deep transfer learning strategy based on the widely available FDG data. The denoising performance of the network including NRMSE, SNR and ROI bias was compared with that of the conventional deep learning network. The results demonstrated that the network trained by the FDG data can effectively reduce the noise of the low-dose PET images from the less-available non-FDG tracers with the similar count level acquired with the same protocol, as shown in figure 2. The network trained by the single-bed FDG datasets and fine-tuned using a relatively small number of whole-body DOTATATE datasets can effectively reduce the noise for the DOTATATE studies with the similar count level acquired with the whole-body protocol, as shown in figure 6. However, for denoising the DOTATATE images obtained with the whole-body protocol, the denoising performance of U-Net trained by single-bed FDG datasets was not comparable with that of the reference U-Net. Instead, the fine-tuning procedure was recommended to improve the cross-tracer and cross-protocol denoising performance. These results indicate that the proposed fine-tuning strategy might be more useful in the cases of cross-tracer and cross-protocol than the cases of cross-tracer but same protocol, and the tracer distribution might have little impact on the denoising performance of U-Net in the cases of cross-tracer but same protocol. One merit of the deep transfer learning strategy is that we can take the advantage of the existing datasets of a large number of FDG patients to train the U-Net to denoise the low-dose images of the less-available tracers, in case the training data are not sufficient. At this point, we only included nine single-bed FDG scans and 12 whole-body FDG scans to make sure the comparison between different U-Nets was fair in terms of the training dataset size. Similarly, the deep transfer learning strategy can be easily applied to other tracers with very long (such as ⁸⁹Zr-labeled tracers) or very short (such as ¹⁵O-water and ⁸²Rb) half-lives, from which the high-count images could be difficult to obtain for network training. Since deep learning-based denoising performance highly depends on the noise level of the training and testing data, the current network trained with 10% dose images might not be able to directly apply to the testing images on different noise levels. However, based on our work, it is feasible to generate a new network using the FDG training data with the similar noise level as the testing images. Furthermore, we did an initial investigation on the impact of different noise levels on the deep learning-based denoising performance. Figure 8 shows the preliminary results of denoising the 30% dose and 50% dose DOTATATE images. For each noise level (30% dose and 50% dose), three additional U-Nets were trained and compared. U-Net G was trained with the 30% dose (or 50% dose) images from 12 DOTATATE imaging subjects and tested with the 30% dose (or 50% dose) images, meaning that U-Net G was trained and tested using the low-dose images with the same noise level. U-Net H was trained with the 10% dose images from 12 DOTATATE imaging subjects and tested with the 30% dose (or 50% dose) images, meaning that U-Net H was trained and tested using the low-dose images with different noise levels. U-Net I was based on U-Net H and fine-tuned using the low-dose images with the same noise level as the testing images. More specifically, U-Net I was trained with the 10% dose images from 12 DOTATATE imaging subjects and fine-tuned using 30% dose (or 50% dose) images from 3 DOTATATE imaging subjects, and then tested with the 30% dose (or 50% dose) images. The results show that U-Net H over-smoothed the low-dose images for both 30% dose and 50% dose cases. After fine-tuning, the denoised images obtained by U-Net I were more similar to those obtained by U-Net G, which we used as the reference network for denoising performance evaluation. These results might indicate that, the fine-tuning strategy using a small number of datasets with the matched noise level of the testing data could improve the denoising performance of the network trained with the FDG data with a mismatched noise level, though further studies are needed to confirm this finding.

Figure 8.

Full-dose image, low-dose images (top row: 30% dose; bottom row: 50% dose), and corresponding denoised low-dose images using different U-Nets, from a sample DOTATATE imaging subject.

We noticed that the texture of the noise was different and the images were smoothed after denoising using the trained U-Nets in our work, other published works also reported similar findings (Kang et al 2015, Xu et al 2017, Xiang et al 2017, Wolterink et al 2017, Gong et al 2018, Lu et al 2019, Kaplan and Zhu 2019). The reason might be that the full-dose images are still noisy. When minimizing the difference between the label voxel values and the output voxel values during training, the U-Net tended to predict the mean values, resulting in difference noise texture (Wolterink et al 2017). Future investigations are needed to further improve the network structure to address this issue.

In the current study, the ROI bias was low for the low-dose images, which was consistent with the finding reported by Yan et al (2016), and was increased for the denoised images. As previously reported (Lu et al 2019), U-Net could introduce smoothing that affects the quantitative accuracy and cause underestimation, leading to an increased bias. This is also what we observed in figure 5. We observed that the biases of the ROIs were −18.0% ± 14.9% for single-bed FMISO and −5.7% ± 9.2% for whole-body DOTATATE with U-Net A, as shown in figure 5, which is a common concern for deep learning-based image noise reduction (Lu et al 2019). The nodule biases in the single-bed FMISO images being larger than those in whole-body DOTATATE images was attributed to the fact that the nodule sizes of the single-bed FMISO group (average diameter: 3.3 ± 3.4 cm) were much smaller than those of the whole-body DOTATATE group (average diameter: 7.6 ± 5.1 cm). As evidenced in figure 9, the mean nodule bias was smaller than 10% when the nodule diameter was larger than 1.7 ± 0.2 cm for single-bed FMISO and larger than 2.4 ± 0.4 cm for whole-body DOTATATE. We also observed larger standard deviation of the lesion bias after denoising. The impact of the increased bias and standard deviation on the diagnosis in terms of lesion uptake and staging is still unknown. Future studies in collaboration with clinical team are needed to investigate this impact on diagnosis in clinic.

Figure 9.

The biases for nodules with various diameters for single-bed FMISO (a) and whole-body DOTATATE (b) datasets.

We also investigated the impact of count levels on the nodule bias using U-Net denoising. More specifically, a series of low-dose PET images from the single-bed FMISO group were obtained with different percentages (1%, 5%, 10%, 20%, 30%, 40% and 50%) used for down-sampling. For each down-sampling percentage, U-Net A was trained and evaluated in terms of the nodule bias. Figure 10 shows the nodule bias for various combinations of nodule diameter and the down-sampling percentage for single-bed FMISO using U-Net A. As seen, the nodule bias becomes smaller with higher count level and larger nodule diameter. All these results indicate that, the lowest count level for different nodule sizes should be optimized in order to maintain an acceptable nodule quantification bias (e.g. smaller than 10%). In the context of dose reduction for quantification of a known tumor, the bulk of nodule size information is generally available, thus the optimal dose reduction can be personalized based on nodule size information.

Figure 10.

The nodule bias for various combinations of nodule diameter and the listmode down-sampling percentage of the low-dose PET image for single-bed FMISO using U-Net A.

One of the limitations in the present study is that there was no theoretical guidance on how to fine tune the well-trained FDG U-Net. As such, considering the symmetrical architecture of the U-Net, we empirically determined that only the weights in the first and final convolution layers were updated for fine tuning. We compared these results to those from another two fine-tuning strategies, one with only the middle layers being updated and the other with the expanding path being updated in the U-Net as shown in figure 11. The data from the same three patients were used in all the fine-tuning strategies. We noticed that updating the first and final layers provided superior results in terms of lower image noise than updating the middle layers. Updating the expanding path also provided a lower image noise than updating the middles lays, but still could not provide comparable results to updating the first and final layers, especially in the spine region. Other fine-tuning strategies with other layers may also be feasible. In the future, however, more comprehensive experiments to investigate the gain of the fine tuning with different layers may be warranted to further improve the efficiency and accuracy of fine tuning.

Figure 11.

Samples of full-dose PET images, low-dose PET images and denoised images with U-Nets A, D, E, U-Net D fine-tuned with middle layers and U-Net D fine-tuned with expanding paths for whole-body DOTATATE.

Ultimately, we used the deep transfer learning strategy to fine tune the pre-trained network for other tracers and other acquisition protocols. This strategy can also be used in multi-center and multi-scanner datasets. For instance, the network trained by data from one scanner and/or one institution can be fine-tuned with a small number of datasets from another scanner or institution. Thus, the fine-tuned network can potentially provide effective noise reduction for other scanners in other institutions, without the need of transferring a huge training dataset but only transferring the trained network, which can both help protect data privacy and improve training efficiency.

5. Conclusion

It is feasible to transfer the existing network trained by one tracer data to reduce the image noise of other tracers or other scanning protocols when using deep learning methods for PET image denoising. Noise in the low-dose PET images from the less-available tracers (FMISO and DOTATATE used in this work) could be effectively reduced with the network trained by FDG datasets when compared with the reference network trained by the less-available tracers. However, when reducing the noise for the DOTATATE studies acquired with the whole-body protocol, the network trained by the single-bed FDG datasets needs to be fine-tuned. The deep transfer learning strategy can potentially help the network training with multi-center and multi-scanner datasets.