Federated Transfer Learning for Low-dose PET Denoising: A Pilot Study with Simulated Heterogeneous Data

Abstract

Positron emission tomography (PET) with a reduced injection dose, i.e., low-dose PET, is an efficient way to reduce radiation dose. However, low-dose PET reconstruction suffers from a low signal-to-noise ratio (SNR), affecting diagnosis and other PET-related applications. Recently, deep learning-based PET denoising methods have demonstrated superior performance in generating high-quality reconstruction. However, these methods require a large amount of representative data for training, which can be difficult to collect and share due to medical data privacy regulations. Moreover, low-dose PET data at different institutions may use different low-dose protocols, leading to non-identical data distribution. While previous federated learning (FL) algorithms enable multi-institution collaborative training without the need of aggregating local data, it is challenging for previous methods to address the large domain shift caused by different low-dose PET settings, and the application of FL to PET is still under-explored. In this work, we propose a federated transfer learning (FTL) framework for low-dose PET denoising using heterogeneous low-dose data. Our experimental results on simulated multi-institutional data demonstrate that our method can efficiently utilize heterogeneous low-dose data without compromising data privacy for achieving superior low-dose PET denoising performance for different institutions with different low-dose settings, as compared to previous FL methods.

I. Introduction

Positron emission tomography (PET) is a functional imaging modality that measures molecular-level activities inside tissues and has wide applications in oncology, cardiology, neurology, and biomedical research. To obtain high-quality PET images for diagnostic purposes, a given dose of radioactive tracer is injected into patients, which inevitably introduces radiation exposure to both patients and healthcare providers. According to the principle of As Low As Reasonably Achievable (ALARA) [1], reducing the administered injection dose is of great interest to the patients, in particular for oncology applications where serial scans are performed to measure response to therapy. However, compared to full-dose PET, low-dose PET with a reduced injection dose often results in low signal-to-noise ratio (SNR) and potential image artifacts in the reconstructed image, leading to negative impacts on the downstream clinical tasks.

To reduce the injection dose and to maintain the PET image quality, deep learning-based PET denoising methods have been developed [2]–[13] to generate full-dose PET image from low-dose PET image, demonstrating superior reconstruction performance when compared with conventional methods [14]–[16]. However, the superior denoising performance of deep learning-based methods often relies on a large amount of representative paired data, which could be prohibitively expensive to collect. One way to alleviate this issue is through building a centralized dataset with multi-institutional data, but the patient data privacy concern poses a major challenge to such a solution [17], [18].

Recently, federated learning (FL) addresses the data privacy issue by building a client-to-cloud platform for different local clients to learn collaboratively using their local data and computing resources without sharing any private local data. Specifically, the cloud server periodically communicates with local clients to collect local models for aggregating a global model, and re-distributes to local clients for local updates. Instead of directly transferring local data for global training, the FL method only involves the communication of model parameters or gradients, thus solving the data privacy concerns.

In the context of low-dose PET imaging, the different low-dose settings with various injection doses of different institutions present data heterogeneity, leading to domain shifts between different institutions. Examples of PET data with different low-dose settings, simulating multi-institution scenarios, are illustrated in Figure 1. The noise level of reconstruction gradually increases as the dose level decreases across different institution’s settings, leading to potential domain shifts. Unfortunately, the generalization capability of the global model trained with classical FL algorithm, such as FedAvg [19], may be non-ideal under these conditions. Previous works have attempted to address the domain shift issues, but only focuses on image classification task and only consider small domain shift scenarios [20]–[23]. In the application of MR reconstruction using FL, Guo et al. [24] tried to address the domain shift issue by iteratively aligning the latent feature of UNet [25] between target and other client sites. However, their cross-site strategy requires the target client to share both the latent feature and the network parameter with other client sites in each communication rounds, which could result in data privacy concerns [26], [27]. Moreover, the cross-site strategy requires communications between local clients with their local data which may contradict the purpose of FL. The communication process during the training will also be tedious and expensive when the number of clients is large, requiring high-frequency communication of both model parameters and latent features. Similarly, Feng et al. [28] tried to address the domain shift issue by using a UNet [25] with a globally shared encoder for generalized representation learning and a client-specific decoder for domain-specific reconstruction. While achieving promising performance for the MR reconstruction task, the network architecture is limited to UNet or its variants due to the constraint in their FL algorithm designs. Moreover, instead of simple encoder-decoder structures, state-of-the-art deep learning-based image restoration networks often involve advanced designs, such as original resolution restoration [29], recurrent restoration [30], and multi-stage restoration [31]. Therefore, designing an FL algorithm that is not limited to specific network architecture while addressing the domain shift issue is an important task for medical imaging restoration. Furthermore, none of the aforementioned FL methods is for low-dose PET denoising and the application of federated learning for low-dose PET denoising is still highly under-explored.

Fig. 1.

Illustration of PET patient data with three unique low-dose settings, where 20%-count, 40%-count, 60%-count low-dose settings were utilized to simulate the multi-institutional data, respectively. The corresponding full-dose PET is shown on the right of the low-dose PET. The lower the dose level, the higher the noise level in the reconstructed image, thus causing a non-identical data distribution.

To address these issues, we propose a federated transfer learning (FTL) framework for low-dose PET denoising using heterogeneous low-dose data. The general idea is to first obtain a global pre-trained model from collaborative training from multi-institution datasets. With a global model trained with large-scale and diverse data, it is then transferred to local institutions for fine-tuning, such that the domain shift issue can be mitigated with local data only training and better generalized to the local data domain. Thus, our FTL consists of two major steps. First, we employ a standard federated averaging method where the models are independently trained at each institution and iteratively sent over to the central server for aggregation. After convergence, the pre-trained global model is downloaded by each institution for transfer learning on local low-dose data distribution. The proposed FL framework can be flexibly adapted to different denoising network architectures. From our experimental results on simulated multi-institutional data, we found our FTL, a simple yet efficient solution, can generate superior PET denoising results at different institutions with different low-dose settings, as compared to previous baseline methods.

II. Related Works

Low-dose PET Denoising.

Prior works on low-dose PET denoising can be categorized into conventional image post-processing methods [14]–[16] and deep learning-based methods [2]–[13]. Conventional methods, such as Gaussian filtering, are standard post-processing procedures for PET denoising. However, with amplified noise levels under low-dose PET settings, these conventional methods are prone to over-smoothing the image and have challenges to preserve local structures. With recent advances in deep learning for natural image restoration [29]–[31], deep learning-based denoising methods have been developed for low-dose PET and have achieved promising performance. Kaplan et al. [5] used a 2D GAN [32] with UNet [25] as a generator to predict full-dose PET images from low-dose PET images. Wang et al. [3] further advanced the idea and used a 3D conditional GAN [33] to directly translate 3D low-dose PET images to full-dose PET images. Similarly, Gong et al. [7] proposed to use a 3D Wasserstein GAN [34] which also achieved promising low-dose PET denoising performance. Based on previous GAN designs, Ouyang et al. [9] proposed to reinforce the denoising performance by incorporating patient-specific information. In addition to approaches using only the low-dose PET images as network input, low-dose PET denoising facilitated by other imaging modalities has also been investigated. For example, Xiang et al. [2] developed an auto-context CNN using both low-dose PET images and T1 MR images as inputs for full-dose PET generation. Similarly, Chen et al. [10] proposed to input low-dose PET images along with multi-contrast MR images into a UNet [25] for ultra-low-dose PET denoising. While deep learning-based methods have achieved promising denoising performance on low-dose PET, these previous works only investigated low-dose PET under single-institution with single-low-dose-level settings, where different institutions usually do not share the same low-dose protocol. Investigation on how to use multi-institutional low-dose data with different low-dose protocols while considering the data privacy issue to train models that can be generalized to different institutions is an important research topic and is still under-explored.

Federated Learning.

FL with a decentralized learning framework enables multiple local institutions to collaborate in training shared models, while maintaining their local data privacy [35]. Even though a classical FL algorithm, such as FedAvg [19], allows collaborative training without sharing data, it does not address the domain shift issue caused by the data distribution difference between different clients. In the classification task, recent studies mitigate this issue by keeping parts of the network parameters local, with solutions such as local batch normalization [20] and keeping the last output layer local [21], [22]. However, these methods are limited to small domain shift, e.g. covariance shift, and may have sub-optimal performance when domain shift is large. To protect patient data privacy while enabling collaborative training, FL has also been applied to medical imaging tasks and demonstrated promising performance [24], [28], [36]–[39]. For example, Li et al. [38] applied the FedAvg [19] for the brain MR tumor segmentation task. Similarly, Feki el al. [37] investigated the use of FedAvg [19] for the chest X-ray COVID-19 classification task. However, none of these works consider the domain shift caused by the data heterogeneity across different institutions. Recently, for the fMRI classification task, Li et al. [39] investigated using federated adversarial domain alignment [40] to reduce the domain shift for improved classification performance. Guo et al. [24] proposed federated learning with cross-site modeling (FLCM) to address the domain shift issues in the application of MR reconstruction by iteratively aligning the latent feature distribution between the clients. However, their method requiring frequent communication between clients increases the communication cost and the risk of potential privacy leakage. As an alternative, Feng et al. [28] proposed to use a client-specific decoder to reduce the domain shift when reconstructing MR images. However, these methods require specific types of network architecture, i.e. the encoder-decoder structure with a latent feature, and cannot be generalized to applications requiring other network architectures for optimal performance. For example, in the application of medical image reconstruction and restoration, cascaded/recurrent network design with fully-complete backbone network architectures have demonstrated superior performance as compared to encoder-decoder type networks [30], [41]–[44]. Therefore, it is desirable to develop a federated learning framework that can be adapted to networks with flexible architecture while addressing the domain shift issue. Furthermore, federated learning for low-dose PET denoising using heterogeneous low-dose data has not been well studied in previous works.

Algorithm 1:

Personalized Federated Learning with Transfer Learning for PET Denoising.

III. Methods

The general pipeline of our federated transfer learning (FTL) is illustrated in Figure 2. The FTL is a two-stage framework, consisting of collaborative training of a global model using all institution data with federated learning, and then using transfer learning to train individual models with local data only, such that the data privacy is preserved.

Fig. 2.

The framework of the Federated Transfer Learning (FTL). During the first stage, with iterative communication between central server and local models in individual institutions, a global model collaboratively trained by all the institution data is obtained in a data-preserving manner. During the second stage, the global model is downloaded by individual institution and further fine-tuned by the dataset hold by each institution.

A. Federated Transfer Learning Framework

Denoting ${\mathcal{D}}_1,{\mathcal{D}}_2,\dots ,{\mathcal{D}}_N$ as low-dose PET datasets from N different institutions. Each local dataset ${\mathcal{D}}_n$ contains pairs of full-dose and low-dose PET 3D images, but with different low-dose settings. Within each institution, the local denoising model can be trained using the local data by iteratively optimizing:

$${\mathcal{L}}_n=\sum _{(x,y)\in {\mathcal{D}}_n}{\Vert G_n(x)-y\Vert }_2^2,$$ (1)

where G_n is the local denoising model at the nth institution, and parameterized by ${\theta }_{G_n}$. x and y form a low-dose and full-dose PET image pair from${\mathcal{D}}_n$. The L₂ loss is computed between the predicted full-dose image $G_n(x)$ and the ground truth full dose image y. Since we assume the data collected at each institution uses different low-dose settings, non-identical data distribution is expected among${\mathcal{D}}_n$. Therefore, a locally trained denoising model may not generalize well to another institution’s low-dose PET data. One potential way to alleviate this issue is to train the denoising model using a combined dataset from N institutions, i.e.,${\mathcal{D}}_{\text{comb }}=\{{\mathcal{D}}_1\cup {\mathcal{D}}_2\cup \dots \cup {\mathcal{D}}_N\}$. However, it is infeasible to directly combine multi-institution data due to data privacy concerns that might prevent data sharing across institutions.

To address this issue and allow multiple institutions to collaboratively train personalized PET denoising models that perform well on their own data distribution, we use a two-stage framework. In stage 1, we employ a federated averaging method to obtain a global model. Assuming there are Q global training epochs and P local training epochs at each institution, the iterative optimization of local model parameter can be written as:

$${\theta }_{G_n}^{p+1}\leftarrow {\theta }_{G_n}^p-\alpha \nabla {\mathcal{L}}_n$$ (2)

where ${\mathcal{L}}_n$ is the denoising loss function defined in Eq. (1), and α is the learning rate in stage 1. At the end of each global training epoch, the local model parameters ${\theta }_{G_n}$ at each institution are uploaded to a central server for aggregating the local model updates. The central server updates the global model with local model updates by averaging the updated parameters from all local models:

$${\theta }_G^q=\frac{1}{N}\sum _{n=1}^N{\theta }_{G_n}^q$$ (3)

where q denotes the q-th global epoch. After Q rounds of communication between local institutions and the central server, we can obtain a collaboratively-trained global model G_A with parameter ${\theta }_{G_A}^Q$ by utilizing multi-institutional datasets while without directly accessing them.

In stage 2, the trained global model $G_A$ with ${\theta }_{G_A}^Q$ is downloaded by each institution for personalized fine-tuning. Assuming there are T fine-tuning training epochs at each institution, the iterative fine-tuning of local model can be written as:

$${\theta }_{G_n}^{t+1}\leftarrow {\theta }_{G_n}^t-\beta \nabla {\mathcal{L}}_n$$ (4)

where t denotes the t-th fine-tuning epoch, and β is the learning rate in stage 2. After T epochs of training using the local dataset${\mathcal{D}}_n$, we obtain the personalized models G₁,...,G_N with parameters of ${\theta }_{G_1}^T,\dots ,{\theta }_{G_N}^T$. The algorithm is summarized in Algorithm 1.

B. Network Architecture and Implementation Details

In our experiment, we use a dual attention residual dense U-Net (DuAttRDUNet) as our PET denoising network [45]. The architecture details are illustrated in Figure 3. The DuAttRDUNet is a U-shape network with dual attention residual dense blocks (DuAttRDB) embedded at different resolution levels. The DuAttRDB consists of a densely connected module [46], [47], a dual attention module [48], and a residual connection. Here, we set the number of input feature of each DuAttRDB to 64. Since PET is 3D imaging data and its FOV varies in the z direction, we implement a 2.5D DuAttRDUNet that takes five continuous axial slices as inputs and predicts the denoised central slice. We implemented our FTL in Python with Pytorch¹. The Adam solver [49] was used to optimize the denoising network in both stages of the FTL. During the first stage training, we used a constant learning rate of 1e−4 and trained for 90 global epochs. The number of local epoch is set to 2. During the second stage training, we used a reduced constant learning rate of 2e − 5 and trained the local model using local dataset for 10 epochs. The batch size is set to 8 in both stages.

Fig. 3.

The network architecture of Dual Attention Residual Dense U-Net (DuAttRDUNet) used in the FTL framework (Figure 2).

C. Heterogeneous Low-dose Human Data

To simulate a multi-institutional dataset with heterogeneous low-dose settings for our study, we collected 175 subjects in our experiments. The subjects were injected with a ¹⁸F-FDG tracer and the whole-body protocol with continuous-bedmotion scanning was used. All data were acquired using a Siemens Biograph mCT PET/CT system. The 175 subjects were split into 120 subjects for training and 55 subjects for evaluation. The 120 subjects were further split into 3 groups with 40 subjects in each group and with no patient overlapping between groups. Each group was treated as one individual institutional data. To create heterogeneous low-dose data that vary between groups, we used uniform down-sampling of the patient list-mode data with down-sampling ratios of 20%, 40%, and 60% for the three groups, respectively. For the 55 subjects in the evaluation set, we also used uniform down-sampling of the patient list-mode data with down-sampling ratios of 20%, 40%, and 60%, such that each subject has three low-dose images for evaluation. In the evaluation set, 10 subjects with lung lesions were annotated on tumor regions at voxel-level which were used for the evaluation of tumor quantification. For both the low-dose and full-dose images, they were reconstructed using the ordered-subsets expectation maximization (OSEM) algorithm with 2 iterations and 21 subsets, provided by the vendor. A post-reconstruction Gaussian filter with 5mm full width at half maximum (FWHM) was used. The voxel size of the reconstructed image was 4.07 × 4.07 × 3mm³. The image size was 200 × 200 in the transverse plane and varied in the axial direction depending on patient height.

D. Evaluation Metrics and Baselines

For denoising evaluation, we computed the Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Normalized Mean Square Errors (NMSE) between the predicted full-dose image and the ground truth full-dose image. RMSE and PSNR stress the evaluation of intensity profile recovery, while SSIM focuses on the evaluation of structural recovery. To evaluate the denoising performance on important pathological regions, such as lung tumor, we also calculated the lesion region’s bias using:

$$Bias=\frac{{\sum }_{m\in R}{Y}_m^{pred}-{\sum }_{m\in R}{Y}_m^{gt}}{{\sum }_{m\in R}{Y}_m^{gt}}$$ (5)

where R is the lesion region. $Y^{pred}$and $Y^{gt}$are the predicted and ground-truth full dose image, respectively. For comparative study, we compared our method against four previous medical image restoration with FL methods, including FedAvg [19], FLCM [24], FedPer [21], [28], and FedSP [28] without the contrastive regularization. For a fair comparison, we used the DuAttRDUNet as the denoising network in all the compared methods. In FedPer, the last output layer was set as the personalized layer and is local.

IV. Results

The visual comparisons between our FTL and previous FL methods under 20%-count low-dose setting are shown in Figure 4. With the coronal view visualization, we can observe that the 20%-count low-dose image suffers from a high noise level due to insufficient counts in the acquisition. Comparing the full-dose to the low-dose image, we can observe that the noise causes artefactual hot spots across the chest, abdominal, and pelvic regions. While FedAvg can reduce the global noise, the artefactual hot spots remain in the liver and intestine regions (zoom-in regions in Figure 4). Similar observations can be found for the FLCM where the false positive hot spots presented in the original low-dose image still remain in the denoised image. As compared to previous methods, our FTL can generate reconstruction with intensity and structure best matched with the full-dose ground truth image, where the artefactual hot spots are eliminated in the liver, intestine, and cardiac regions by our method. In Figure 5, we show an additional visual comparison of our denoising performance under three different low-dose settings corresponding to three different institutional settings. As we can see, original reconstruction with a lower dose level leads to a higher noise level. The personalized denoising model from the FTL can consistently generate high-quality denoised images across various low-dose levels of different institutional settings.

Fig. 4.

Qualitative comparison of low-dose reconstructions using different federated learning methods under the 20%-count low-dose setting (institution #1).

Fig. 5.

Qualitative comparison of FTL reconstructions on multi-institution dataset with 20%-count, 40%-count, and 60%-count low-dose settings.

The quantitative evaluation of different FL methods under different low-dose settings from different institutions is shown in Table I. In the 20%-count low-dose experiment, compared to the original low-dose reconstruction, our method can improve the image quality with NMSE decreased from 0.0292 to 0.0124 and PSNR increased from 25.86 to 29.24 on average, outperforming the previous baseline methods. Similarly, in the 40%-count low-dose experiment, our method achieves the best performance as compared to previous baseline methods, maintaining the NMSE lower than 0.0065 and PSNR over 32.30. Comparing the experiments from 20%-count to 40%-count low-dose settings, the reconstruction improvement of our method is mainly due to the increase in dose level, where the PSNR already increased from 25.86 to 30.24 in the original reconstruction. Similar observations can be found in the 60%-count low-dose experiment. The evaluation of lesion region’s bias is summarized in Table II. In the 20%-count low-dose experiment, compared to the original low-dose reconstruction, our method can reduce the bias from 0.007 ± 0.023 to 0.004 ± 0.015, better than the previous baseline methods. Similar observations can be found in both the 40%-count and 60%-count low-dose experiments.

TABLE I.

Quantitative comparison of reconstructions using different federated learning methods. We evaluated on three different institution data with three different low-dose settings. Best results are marked in bold.

Evaluation	20%-count data			40%-count data			60%-count data
	SSIM	NMSE	PSNR	SSIM	NMSE	PSNR	SSIM	NMSE	PSNR
LDPET	.964	0.0292	25.86	.982	0.0106	30.24	.991	0.0047	33.77
FedAvg [19]	.977	0.0138	28.83	.985	0.0070	31.76	.986	0.0042	34.03
FLCM [24]	.978	0.0132	29.01	.986	0.0065	32.13	.991	0.0042	34.09
FedPer [21]	.977	0.0131	29.02	.985	0.0072	31.73	.992	0.0038	34.48
FedSP [28]	.978	0.0129	29.09	.986	0.0066	32.09	.991	0.0039	34.37
Ours	.978†∗	0.0124 †‡∗⋆	29.24 †‡∗⋆	.987†‡∗⋆	0.0064 †‡∗⋆	32.38 †‡∗⋆	.993†‡∗⋆	0.0037 †‡∗⋆	34.67 †‡∗⋆

^{”†”, ”‡”, ”*” AND ”⋆”}MEAN THE DIFFERENCE OF FEDAVG[19], FLCM[24], FEDPER[21], AND FEDSP[28] TO OUR METHOD ARE SIGNIFICANT AT p < 0.05, RESPECTIVELY.

TABLE II.

Comparison of lesion region’s bias from different federated learning reconstruction methods. We evaluated on three different institution data with three different low-dose settings. Standard deviations were reported in (). Best results are marked in bold.

	20%-count	40%-count	60%-count
LDPET	0.007(0.023)	0.006(0.019)	0.002(0.010)
FedAvg [19]	0.006(0.021)	0.005(0.017)	0.001(0.009)
FLCM [24]	0.005(0.019)	0.004(0.015)	0.001(0.008)
FedPer [21]	0.005(0.018)	0.004(0.016)	0.001(0.008)
FedSP [28]	0.005(0.019)	0.004(0.014)	0.001(0.007)
Ours	0.004(0.015)	0.003(0.012)	0.001(0.004)

For ablative studies, we first compared the performance of denoising models trained from scratch using only local data v.s trained using our FTL framework. In Figure 6, we demonstrate the visual comparison of low-dose reconstructions from denoising models trained from a single institution 20%count low-dose paired data and trained from our FTL. As we can see, while the denoising model trained with only local data can reduce the global noise level, it also reduces the hypermetabolic hot spot signals presented in the full-dose images. On the other hand, our FTL not only reduces the global noise level but also maintains the hot spot signals in the final reconstructions. The corresponding quantitative evaluation is summarized in Table III. When the denoising model is trained from scratch using only 20%-count low-dose paired data (1st row), it yields PSNR of 28.56 and NMSE of 0.0144 when evaluating on the same institutional data. Due to the domain shift caused by different low-dose settings, when applying the denoising model trained from one institutional data to another institutional data, it yields inferior denoising performance. For example, when applying the model trained with 20%-count data to those 40%-count data (1st row and 2nd column), it only yields PSNR of 30.63 and NMSE of 0.0089 which is significantly lower than those of the model trained with 40%-count data and applied on the 40%-count data (2nd row and 2nd column). While training and applying the same institutional local data mitigates the domain shift issue, our FTL (last row) with diverse multi-institutional data collaborative training still yields the best performance with PSNR boost from 28.56 to 29.24 for the 20%-count experiment for example. The corresponding tumor region’s bias is reported in Figure 7. In the 20%-count experiment, the bias can be reduced by using the denoising models trained from scratch using only local data. Our FTL method provides further reduced mean bias and reduced standard deviation. Similar observations can be seen in both the 40% and 60% experiments.

Fig. 6.

Visual comparison of low-dose reconstruction from denoising model trained from single institution data and from our FTL.

TABLE III.

Quantitative comparison of reconstructions when training only on individual institution dataset. The results with training and testing on same institution data are marked with underline. Best results are marked in bold.

Evaluation	20%-count data			40%-count data			60%-count data
	SSIM	NMSE	PSNR	SSIM	NMSE	PSNR	SSIM	NMSE	PSNR
20%-count data	.977	0.0144	28.56	.982	0.0089	30.63	.985	0.0069	31.65
40%-count data	.975	0.0160	28.22	.986	0.0076	31.54	.988	0.0051	33.11
60%-count data	.974	0.0182	27.71	.985	0.0078	31.33	.991	0.0041	34.20
Ours	.978	0.0124	29.24	.987	0.0064	32.38	.993	0.0037	34.67

Fig. 7.

Comparison of lesion region’s bias from the original low-dose reconstruction (red bar), the reconstruction using model trained only from its own institutional data (green bar), and our FTL reconstruction (blue bar). We evaluated on three different institution data with 20%-count (left), 40%-count (middle), and 60%-count (right) low-dose settings.

We also studied the denoising performance when using different federated learning settings of FTL. A visual comparison example is shown in Figure 8. As we can observe, when training a denoising model using FL with multi-institutional data consisting of 40%-count and 60%-count data, directly applying it to the 20%-count data can reduce the global noise level but with noise-induced hot spots remain in the liver region. When the model was further fine-tuned with 20%count local data, the denoising performance further improves with these hot spots being eliminated. The corresponding quantitative evaluation is summarized in Table IV. A denoising model trained with FL and multi-institutional data consisting of 40%-count and 60%-count data yields PSNR of 28.43 when evaluating on 20%-count data (1st row and 4th column), which is inferior to the performance of the denoising model directly trained on 20%-count from scratch (Table III 1st row and 4th column). However, when the model was further fine-tuned on 20%-count local data (2nd row and 4th column), it boosts the PSNR up to 29.15 which is better than the model trained from scratch. Similar observations can be found for the other FTL setups. The last row reports the performance of our FTL which was first trained using all three institutional local data with FL and then fine-tuned on individual local data, providing the best performance across different setups.

Fig. 8.

Visual comparison of low-dose reconstruction using different federated learning settings of FTL. Results on 20%-count low-dose data are shown here. 40% + 60% means two institution data used during the federated learning stage of FTL. → 20% means transfer learning on target institution data.

TABLE IV.

Quantitative comparison of reconstructions when using different federated learning settings of FTL. Best results are marked in bold. For instance, X + Y → Z means federated learning using X and Y dataset, and then transfer learning to Z dataset.

Evaluation	20%-count data			40%-count data			60%-count data
	SSIM	NMSE	PSNR	SSIM	NMSE	PSNR	SSIM	NMSE	PSNR
40% + 60%	.976	0.0153	28.43	-	-	-	-	-	-
40% + 60% → 20%	.978	0.0128	29.15	-	-	-	-	-	-
20% + 60%	-	-	-	.986	0.0067	31.97	-	-	-
20% + 60% → 40%	-	-	-	.987	0.0065	32.16	-	-	-
20% + 40%	-	-	-	-	-	-	.988	0.0051	33.16
20% + 40% → 60%	-	-	-	-	-	-	.992	0.0038	34.49
Ours	.978	0.0124	29.24	.987	0.0064	32.38	.993	0.0037	34.67

V. Discussion

In this work, we developed a federated learning framework, called FTL, for low-dose PET denoising using simulated heterogeneous multi-institutional low-dose data. Specifically, the proposed FTL consists of two major steps. In the first step, a standard federated averaging is used, such that the denoising models are independently trained at individual institutions and iteratively sent over to the central server for model aggregation. During this process, no local data is shared between institutions. After the convergence of federated training, the global model is re-distributed to each institution for transfer learning on local data only, generating a personalized denoising model for each institution such that the domain shift can be mitigated. Unlike previous works of FLCM [24] and FedPer [21], [28] that are bound to specific network architecture, our FTL framework allows flexible network architecture to be integrated.

From our experimental results, we demonstrated the feasibility of using our FTL for low-dose PET denoising with three different low-dose settings corresponding to three different simulated institutions. First of all, as we can observe from Table I and Figure 4, our method generating three personalized denoising models for each individual institution can consistently outperform previous methods, including FLCM, FedPer, and FedSP that also generate three personalized models, one for each institution, as well as FedAvg that only generates one global model for all institutions. The lesion bias analysis, as shown in Table II, also demonstrated the FTL provides less bias quantification for lung lesions as compared to previous methods. Please note the lesion regions usually have higher SNR as compared to other regions, due to the higher amount of tracer concentration in these regions. Thus, the bias in the lesion region is already very low from the original low-dose reconstruction, e.g. 0.007±0.023 from 20%-count data. Even under this condition, our method further reduced the bias to a lower level with reduced standard deviation as well. Second, one of the goals of our FTL is to address the domain shift issue. As we can observe from Table III, Figure 6, and Figure 7, while training a denoising model from scratch using local data with limited diversity can avoid domain shift issues and provide reasonable denoising performance for each institution, our method yields a superior performance as compared to them since we utilize all institutional data with diverse distribution for the federated pre-training. Similar observations can be found in Table IV where the best performance is given when all the institutional data is used in the first step of FTL, and then fine-tune using local data. Even though our FTL achieves superior performance as compared to models trained from local data only, we think increasing the amount of local data and diversity may further boost our performance and may also potentially close the gap between the model trained from local data only and federated learning-based methods.

The presented work also has limitations with several potential improvements that are the subjects of our future studies. First, our work considers a three-institution scenario with three different low-dose settings, i.e. 20%-count, 40%-count, and 60%-count. However, the number of institutions could be more than three with more diverse low-dose settings. For example, Kaplan et al. [5], Ladefoged et al. [50], and Zhou et al. [51] used a 10%-count low-dose setting. Wang et al. [3] and Whiteley et al. [52] deployed a 25%-count low-dose setting, while Ouyang et al. [9] considered a 1%count ultra-low-dose setting. Even though we only consider three different low-dose settings in our work, our proposed framework can be flexibly adapted to a different number of institutions with different low-dose settings. In fact, we believe that expanding the number of institutions with more diverse low-dose settings could potentially further improve our method’s performance. Second, we created our multi-institutional low-dose PET dataset based on simulation which uses data all acquired from the Siemens mCT PET/CT systems and reconstructed with the OSEM reconstruction algorithm incorporated with point-spread-function (PSF) and time-offlight (TOF) information. In real-world applications, low-dose PET data from different institutions could be acquired with different vendor systems, reconstructed with different algorithms, and administered with different tracer types. Our pilot study mainly focuses on low-dose ¹⁸F-FDG PET with OSEM reconstruction which is the most common protocol in clinic PET and previous works of low-dose PET. In our future work, we will investigate the performance of our framework with data acquired from different tracer types, vendor systems, and reconstruction protocols from different institutions. We believe our method with transfer learning could potentially be adapted to changing elements in the system and their reconstruction algorithms. Thirdly, for a fair comparison, we performed all our experiments using the same denoising network architecture, i.e. DuAttRDUNet (Figure 3). While our framework is technically simple, it can be flexibly adapted with different advanced denoising networks that do not use U-shape design. For instance, the residual dense network based on the residual dense block without any downsampling operation has demonstrated superior performance in general image restoration tasks [46]. Similarly, SwinIR [53] based on residual Swin Transformer blocks [54] without any downsampling operation also showed significant advantages for image restoration tasks. Combining these networks with dual domain learning has also shown to yield superior medical image reconstruction performance [30], [44], [55]–[57]. Deploying these networks in our FTL could potentially further improve our denoising performance and will be an important direction of our future studies. Lastly, in our current experiment, we assumed a similar amount of data from each institution, thus we used a equally weighted parameter averaging strategy in our first stage’s training. In our future work with a more diverse and potential imbalanced amount of real data, we will investigate using weighed parameter averaging strategies [58] to further improve our method’s performance.

There are also additional challenges and considerations that the current work did not consider which would require attention in the future works on using FL for PET applications [59]–[61]. First, even though FL has demonstrated its effectiveness in protecting patients’ privacy by keeping institutional data locally, there are still privacy and security challenges in FL. For example, the parameter in the deep learning model still contains sensitive information that could allow one to reconstitute the patients’ information in the local training set [62]. Adversarial attacks from malicious parties can also degrade the deep learning model’s performance, thus causing severe consequences when a model is deployed in real clinical scenarios [63]. Additional data safety measurements in FL, such as model encryption, differential privacy, and adversarial defense against malicious clients, should be considered to better protect the data privacy [64], [65]. Second, the FL requires standardized data from different institutions to allow stable training of deep learning models. Different institutions may have different infrastructure and imaging data management protocols, resulting in imaging data stored in different formats. This introduces additional data heterogeneity that may affect the federated training’s convergence which could negatively impact the model’s final performance. Thus, appropriate data preprocessing and curation measurements should be performed to standardize the data before performing federated learning [66], [67]. Lastly, an appropriate system architecture is also required to perform FL. For example, local institutions need private or cloud-based computation resources in order to perform local training before federated training. Federated learning also requires internet connections with high-performance bandwidth between different institutions to enable model parameter passing. Many institutions may still lack these resources to enable the proposed FL methods. Therefore, the design of FL algorithms should also consider different system architectures, especially when these resources are limited.

VI. Conclusion

We developed a federated transfer learning framework (FTL) for low-dose PET denoising using heterogeneous data. Our FTL aims to generate a personalized denoising model for each institution by first collaboratively training a global model using all institutional data while without sharing data during this process, and then fine-tune using each institution’s local data to alleviate the domain shift issue. Our experimental results demonstrate that our method can provide high-quality denoising reconstruction, superior to previous baseline image reconstruction with federated learning methods under simulated multi-institution with diverse low-dose settings.