Lung nodule synthesis guided by customized multi-confidence masks

Abstract

The generated lung nodule data plays an indispensable role in the development of intelligent assisted diagnosis of lung cancer. Existing generative models, primarily based on Generative Adversarial Networks (GANs) and Denoising Diffusion Probabilistic Models (DDPM), have demonstrated effectiveness but also come with certain limitations: GANs often produce artifacts and unnatural boundaries, and due to dataset limitations, they struggle with irregular nodules. While DDPMs are capable of generating a diverse range of nodules, their inherent randomness and lack of control limit their applicability in tasks such as segmentation. To synthesize controllable shapes and details of lung nodules, in this study, we propose a unified model that combines GAN and DDPM. Guided by multi-confidence masks, our method can synthesize customized lung nodule images by adding spikes or dents to the input mask, allowing control over shape, size, and other medical image features. The model consists of two parts: (1) a Rough Lung Nodule Generator, based on GAN, which synthesizes rough lung nodules of specified sizes and shapes using a multi-confidence mask, and (2) a Lung Nodule Optimizer, based on DDPM, which refines the rough results from the first part to produce more authentic boundaries. We validate our method using the LIDC-IDRI dataset. Experimental results demonstrate that our unified model achieves the best FID score, and the synthetic lung nodules it generates can serve as a valuable supplement to training datasets for segmentation tasks. Our study presents a unified model that effectively combines GAN and DDPM to generate high-quality and customized lung nodule images. This approach addresses the limitations of existing models by leveraging the strengths of both techniques. Our code is available at https://github.com/UtaUtaUtaha/CMCMGN.

Introduction

Lung nodules are critical indicators in the early detection of lung cancer, which remains a leading cause of cancer-related mortality [1, 2]. Accurate identification and classification of these nodules are essential for timely intervention and improved patient outcomes. With advancements in deep learning and artificial intelligence, generating synthetic lung nodules has become increasingly important [3]. This approach not only enhances training datasets for artificial intelligence models but also aids in the development of more robust diagnostic tools. By improving the accuracy and reliability of nodule detection, we can significantly advance the fight against lung cancer.

Existing generative models for synthetic lung nodules were primarily based on Generative Adversarial Networks (GANs) [4–6] and Denoising Diffusion Probabilistic Models (DDPM) [7, 8]. Both approaches have demonstrated effectiveness in generating realistic nodules. However, they come with certain limitations. GANs often struggle with issues such as holes and artifacts, especially when the training data is insufficient, and the synthesized lung nodules exhibit unnatural boundaries [6], as shown in Fig. 1A. In contrast, DDPMs generate nodules in a purely random manner, making it difficult to control the size and shape of the generated nodules. Furthermore, this random generation often results in extremely small images or even images without any visible lung nodules, as seen in Fig. 1B. These limitations underscore the need for improved generative techniques that can better address the specific requirements of lung nodule synthesis.

Fig. 1.

Disadvantages of GAN and DDPM. The red arrow in part A indicates holes and artifacts in the generated nodules. The red box in part B highlights images with either no lung nodules or those that are difficult to identify

To overcome these challenges, we introduce a mask-guided approach for the realistic generation of lung nodules, leveraging the uncertainty region [2] of the mask to produce more natural boundaries. By transforming the input masks, such as random resizing or adding random spikes and dents of varying sizes, we can generate customized lung nodules [9]. Additionally, we treat holes and artifacts as noise and propose a hybrid model that combines a GAN-based approach with Denoising Diffusion Probabilistic Models (DDPM) to optimize the results. We validate our method using the LIDC-IDRI dataset [10], where our approach achieves the best FID score [11], demonstrating that our generated synthetic data can effectively mitigate the problem of insufficient data in segmentation tasks.

The contributions of this study are as follows: (1) We propose a novel fusion model that combines GAN and DDPM techniques to generate more realistic lung nodules, addressing the issues of holes and artifacts commonly found in GAN-based models. (2) We introduce a multi-confidence mask-guided model capable of synthesizing lung nodules with specified sizes, shapes, and more realistic boundaries, thus enhancing the diversity of lung nodules. (3) Our model achieves the highest FID score, and the synthetic data it generates demonstrates its effectiveness in segmentation tasks.

Related work

Image synthesis

Image synthesis is a traditional task in computer vision. In 2014, Ian Goodfellow introduced Generative Adversarial Networks (GANs) [4], which use an adversarial process where the generator aims to produce data indistinguishable from real data, ultimately generating samples that resemble the true data distribution. In 2016, Phillip proposed an image-to-image translation method [5] based on GANs, where a simple outline image is provided to generate specific target images. In 2020, Jonathan introduced high-quality image synthesis results using Diffusion Probabilistic Models (DDPM) [7], which involve training on a weighted variational bound based on a novel connection between diffusion probabilistic models and denoising score matching with Langevin dynamics. In 2022, Schönfeld proposed a novel GAN model for semantic image synthesis, which uses only adversarial supervision, achieving high-quality, diverse, and multi-modal image generation with improved fidelity, alignment to label maps, and robustness to class imbalance and sparse annotations [12]. In 2023, Tan proposed a new method for diverse semantic image synthesis, achieving multimodal generation at both semantic and instance levels through continuous class modulation and instance-adaptive sampling, enhancing both diversity and quality [13]. In 2024, Liu introduced Residual Denoising Diffusion Models (RDDM) [8], which decouple the traditional denoising diffusion process into residual diffusion and noise diffusion, unifying image generation and restoration tasks.

Our study is closely related to [5] and [7], as we combine the core ideas of these methods to provide a unified approach for the more specific task of lung nodule synthesis.

Lung nodule synthesis

One challenge in applying Artificial Intelligence to medical imaging is the high cost of acquiring medical images and annotations, making it crucial to synthesize a large number of high-quality images [6, 14, 15]. In 2019, Han proposed a 3D Multi-Conditional GAN [16] for data augmentation in 3D object detection, generating realistic and diverse lung CT nodules to improve sensitivity and overcome the limitations of small datasets. In 2021, Wang introduced a Multi-Target Co-Guided Adversarial Mechanism for realistic lung nodule synthesis, utilizing a Mask-Guided Generative Network and a Window-Guided Semantic Learning Network to control nodule shape, background, and semantic features [6]. In 2022, Wang proposed using super-resolution GANs to generate synthetic GGOs in low-dose CT images. The generated nodules were evaluated both visually and quantitatively, showing high similarity to real samples in both aspects [17]. In 2023, Mendes explored the use of generative adversarial networks (Pix2Pix and cCGAN) to synthesize artificial lung images from CT scan annotations, aiming to augment medical datasets [18]. In 2024, Xu proposed a Stable Diffusion-based method (TDASD) to generate high-resolution CT images of lung nodules, addressing the challenge of limited data. By fine-tuning stable diffusion on publicly available lung datasets and incorporating expert medical prompts, the method generates diverse, medically relevant images that enhance tumor diagnosis [19].

Compared with existing methods, our approach is based on [6], which utilizes multi-confidence masks [2] and multiple windows to guide lung nodule synthesis, rather than a single window. Additionally, we combine DDPM to overcome the issues of holes and artifacts in GAN-based models, as well as the randomness inherent in DDPM.

Methods

Network architecture

Our model consists of two main components: (1) a Rough Lung Nodule Generator (RLNG), and (2) a Lung Nodule Optimizer (LNO). The pipeline of our model is illustrated in Fig. 2. Specifically, we use the bone window image as the background and combine it with multi-confidence masks to form the input image (I) for the Rough Lung Nodule Generator (G), which produces a rough output (). Then, we add Gaussian noise to and pass it through a DDPM to obtain , while is passed through a second DDPM to generate . Finally, we combine and and apply linear mapping to obtain the final output (O). The pipeline of our model can be summarized as follows:

1

2

where represents DDPM, and O is the final output of our model.

Fig. 2.

The pipeline of CMCMGN is as follows. First, the input is a tensor that stacks the background and multi-confidence masks into a single tensor to serve. We then employ two identical Denoising Diffusion Probabilistic Models (DDPM) to optimize our results. Specifically, during the optimization phase, we use the generated result as input for one DDPM and add Gaussian noise to this result, which then serves as the input for the second DDPM. This process yields two outputs, referred to as and . Finally, we utilize the grayscale information from to remap , producing the final results

Customized masks

For synthesizing customized lung nodules, we use the bone window image as the background and apply multi-confidence masks [2] to guide the synthesis. High-confidence areas are where multiple doctors’ labels intersect, while low-confidence areas are the union of labels. For each background image, some may contain lung nodule information, so we apply the corresponding mask to eliminate irrelevant regions (Fig. 3A). To enhance robustness and generate diverse nodules, we randomly select backgrounds, apply random zooming to masks, and introduce random transformations like spikes and dents. Specifically, we randomly select a pixel on the edge of the mask, then choose another pixel along the straight line between and the center point. The distance between and is randomly set between 4 and 6 pixels. Finally, we randomly select a third pixel on the edge of the mask, where the distance from is approximately 2 pixels. Using these three points , , and , we define an area. If the area extends outside the mask, we fill it with white; otherwise, we fill it with black, thus creating a spike or dent. Just like the ‘Process’ operation in Fig. 3B.

Fig. 3.

Obtain background and mask. In A, we obtain a bone window image (to the left of the arrow) and use the corresponding mask to remove the nodule information, resulting in a clean background (to the right of the arrow). In B, ‘Process’ means randomly resizing the input mask and introducing random spikes or dents; ‘Add’ refers to combining the mask with a bone window image used as the background; ‘Delete’ means removing the overlapping regions between the mask and the bone

We then randomly select the background and masks as inputs for our model. To prevent them from influencing each other, we delete the overlapping parts between the background and mask to ensure that the selected background and mask do not overlap. To help the model distinguish between the background and nodule regions, we map the background values to the range [0, 1], set the high-confidence region values to 2, and assign a value of 1.5 to the low-confidence region. These processed values are then stacked into a tensor, which serves as the input for our model. The pipeline as show in Fig. 3B.

The rough lung nodule generator

The Rough Lung Nodule Generator (RLNG) in our model is based on the Multi-Target Co-Guided Synthesizing Network (MTCGSN) [6]. The structure of RLNG is illustrated in Fig. 4.

Fig. 4.

The structure of MTCGSN. In this image, G represents the Generator, D represents the Discriminator, and T represents the Transforming Network, which transforms the synthesized HU image into lung window and mediastinal window images. In this structure, the transformed lung window image is used as the input for the subsequent parts. Additionally, C represents the Classification Network. By combining the results from two C networks, we determine the class result indicating whether the nodule is malignant

As in [6], we use a mask combined with a background to guide the model in achieving optimal results. The key difference is that we employ multiple confidence masks [2] as input, with the background extracted from the bone window of the CT images. These elements are then combined to serve as input to the RLNG model. Initially, the input image is processed by a U-Net [20], which features a five-step down-sampling and up-sampling structure to synthesize the original HU image (denoted as G in Fig. 4). Subsequently, we apply two transformations (denoted as T in Fig. 4) to convert the original HU image () into lung window images () and mediastinum window images (). All three synthesized images are used to optimize our model. The details of these Transformation are shown in Table 1. After each convolutional layer, we apply an Instance Normalization Layer, and at the final convolutional layer, a ReLU activation is used.

Table 1.

Architecture of the transforming network

Layer	Kernel	Strides	Channel
conv1		1	32
conv2		1	64
conv3		1	128
conv4		1	64
conv5		1	32
conv6		1	1

Similar to GAN-based models, we design a discriminator (denoted as D in Fig. 4) to distinguish between synthetic and real images. Since both and are derived from the transformed , the discriminator is designed to assess whether the synthesized HU image is real or not. The structure of the discriminator is shown in Table 2. After the first convolutional layer, we apply a Leaky ReLU activation with a slope of 0.2. Except for the first and the last convolutional layer, each subsequent layer is followed by an Instance Normalization layer and a Leaky ReLU activation, with the same slope as the first convolutional layer.

Table 2.

Architecture of the discriminator network

Layer	Kernel	Channel	Strides	Padding
conv1		64	2	1
conv2		128	2	1
conv3		256	2	1
conv4		512	2	1
conv5		1024	2	1
conv6		2048	2	1
conv7		4096	2	1
conv8		1	1	1

Furthermore, for the synthesis of and , we implement a classification step (the C in Fig. 4) to determine whether the label is benign or malignant. Given that the texture and shape of lung nodules can reflect the likelihood of malignancy to some extent [21], we incorporate the classification results to enhance our model’s optimization. The details of our Classifer are shown in Table 3.

Table 3.

Architecture of the classifer

Layer	Kernel	Strides	Channel
conv1		1	64
pooling1		2	64
conv2		1	128
pooling2		2	128
conv1		1	256
pooling3		2	256
FC	–	–	100
FC	–	–	3

The lung nodule optimizer

Considering that DDPMs have excellent denoising and image synthesis capabilities, we use them as optimizers to generate more realistic lung nodules. However, in the results produced by RLNG, noise typically appears at the edges of the nodules, so global denoising is unnecessary. Therefore, we use two DDPMs to optimize the results from RLNG. Specifically, we use one DDPM that accepts only the lung window image as input. Due to the limitations of our dataset, we can only train the DDPM from pure noise to clear images, so we add Gaussian noise to before passing it as input to the second DDPM. The added Gaussian noise does not interfere with the primary information in our synthesized , but helps the DDPM generate more realistic results. In this part, we define our LNO as consisting of two main components: (1) a DDPM that takes the image synthesized by RLNG and adds Gaussian noise (-DDPM), and (2) another DDPM that takes the image synthesized solely by RLNG (-DDPM). Finally, we combine the outputs of both DDPMs and apply a linear remapping to obtain the final result. Experimental results demonstrate the effectiveness of this approach. Specifically, we compute the minimum grayscale value () of the result from -DDPM and use to remap the result from -DDPM to the range .

Loss functions

Since our model consists of two independent parts, the loss function is divided into two components as well. In the RLNG, we synthesize three window images, including the original HU images, lung window images, and mediastinum window images. Therefore, we use three L1 losses: , , and to calculate the distance between the three window images and their corresponding synthesized images.

3

Here, represents the generator, and denotes the overall L1 loss.

The discriminator () is used to assess the realism of the synthesized images. The loss for the discriminator is defined as follows:

4

In this equation, represents the discriminator, represents the generator of the HU image, represents the input mask, and and represent the real and synthesized images, respectively.

For the two classification components (), we use cross-entropy loss [22]:

5

Here, represents the target label for the nodule, indicating whether it is benign or malignant, and is the predicted result by .

The final loss function for RLNG is defined as follows:

6

Here, is a hyperparameter that balances the contributions of the generator and discriminator in our RLNG framework. In our experiments, we set to 100 to achieve optimal performance.

In the LNO part, our loss function is designed similarly to [7], optimizing the denoising process by minimizing the difference between the generated noise and the real noise. The goal of the DDPM model is to learn how to recover real data from noise. The training process achieves this by minimizing the following mean squared error (MSE) loss:

7

In this equation, is the noise predicted by the model, is the true noise, and is the current diffusion timestep. By minimizing this loss, the model learns to effectively denoise at each time step and gradually recover the original data. In this context, the maximum value of t is set as the hyperparameter T. A larger value of T implies a longer total number of diffusion steps for the LNO process, which generally leads to better denoising performance. However, in our experiments, the noise is primarily concentrated at the edges of the synthesized lung nodules. Therefore, we empirically set T to 100 to achieve a balance between denoising effectiveness and computational efficiency.

Experiments

Datasets

We evaluate our network on the publicly available LIDC-IDRI dataset [10], which consists of 1018 study instances and over 2600 nodules. For this study, we select 1835 lung nodules that have at least two annotations from different radiologists. This selection allows us to obtain the corresponding CT images and their annotation sets. For each nodule with multiple masks, we compute their union and intersection. The intersection set is considered the high-confidence region [2], while the difference between the union and intersection sets is treated as the low-confidence region [2]. Based on the official annotations, we extract three layers of images for each nodule, resulting in a total of 4328 images. Each pixel represents the Hounsfield Unit (HU) of the CT images. We clip the intensity values of the CT images to the following ranges: [−200, 800] for the bone window, [−1500, 500] for the lung window, [75, 425] for the mediastinum window, and [−1500, 1500] for the original HU values. Additionally, we normalize all intensity values to the range [0, 1] before training.

Experimental settings

The experiments were conducted using an RTX A6000 GPU with CUDA Version 12.1, utilizing PyTorch and Python 3.8. Our experiments consist of two independent parts.

When training the RLNG model, we resize our images to , as RLNG cannot process images that are too small. We employ two Adam optimizers for the Generator and Discriminator, with an initial learning rate of 0.0002 and betas set to (0.5, 0.999). The batch size is 128, and during the 200 epoch training stage, the learning rate is reduced to 50% of its original value after 100 epochs.

For training the LNO model, we use the AdamW optimizer with an initial learning rate of 0.0001 and a weight decay of 0.0001. The parameters , , and T for LNO are set to 0.0001, 0.02, and 100, respectively. In the 500 epoch training stage, we use a Gradual Warmup Scheduler and apply a Cosine Annealing Schedule to adjust the learning rate for each parameter group [23].

Quantification of synthesized lung nodules

In our experiments, we use the Fréchet Inception Distance (FID) to quantitatively evaluate the synthesized lung nodules. Specifically, we perform five-fold cross-validation to generate as many lung nodules as possible. In total, we generate 4328 lung nodule images, matching the quantity found in the LIDC-IDRI dataset. FID uses an Inception network with the last pooling layer removed to extract n-dimensional features. It assumes that the features from real images follow a particular distribution, and then calculates the Fréchet distance between the feature distributions of real and synthesized images. A lower FID score indicates that the synthesized lung nodules closely resemble real ones in terms of deep feature distribution [6, 24].

For further validation of our model, we choose two approaches. At first, we do not use the transform described in Sect. 3.1.1 for the input mask in the validation set. The results are presented in Table 4.

Table 4.

FID scores of different methods

Methods	FID
Pix2Pix [5] with bone window as background	90.67
MTCGSN [6] with original mask from LIDC-IDRI	87.96
INADE [13] with original mask from LIDC-IDRI	112.54
SaUN [3] with original mask from LIDC-IDRI	84.28
StyleP2P [25] with original mask from LIDC-IDRI	81.47
DDPM [7] with random Gaussian Noise	254.77
RDDM [8] with random Gaussian Noise	261.50
Ours with original mask from LIDC-IDRI	77.03

In Table 4, we can observe that our model achieves the best FID score compared to other methods. Additionally, in our experiments, our result for [5] is 90.67 FID, which is better than the result in [6] (98.97). In [6], they use a real background and a single mask to synthesize lung nodules. In our experiments, we use the bone window as the background, which obviously provides less information than the real background, and we combine multi-confidence masks to guide the lung nodule synthesis, yielding better results than [6]. This demonstrates that our multi-confidence mask-guided method is effective. Furthermore, the MTCGSN in our experiments achieves a better result (87.96) than the one reported in [6], further validating the effectiveness of our method.

To validate that our model can synthesize lung nodules of specific shapes and sizes, while maintaining the training set unchanged, we use the transform described in Sect. 3.1.1. In this experiment, we do not compare with the diffusion-based model because it performs random synthesis. The results are shown in Table 5. We can observe that our model achieves the best FID.

Table 5.

FID scores of different methods with transformed mask

Methods	FID
Pix2Pix [5] with bone window as background	102.98
MTCGSN [6] with transformed mask	92.90
INADE [13] with transformed mask	132.38
SaUN [3] with transformed mask	91.76
StyleP2P [25] with transformed mask	118.72
Ours with transformed mask	87.47

In Fig. 5, we display some synthesized lung nodule results. Except , the others use the same input as described in Sect. 3.1.1. In Fig. 5, we observe that when we add spikes or dents as described in Sect. 3.1.1, they produce excessive and coarse artifacts along the boundaries of the lung nodules. In Fig. 5, to the naked eye, it appears that INADE synthesizes better results than , although it gets a worse FID score. However, a closer inspection reveals that these results resemble grid-like noise, making them easily identifiable as fake images. Figure 5 and show the results of DDPM and RDDM. Here, we can see that the diffusion models struggle to synthesize lung nodules from image-mask datasets like LIDC-IDRI. However, in Fig. 5, the results of our model exhibit more natural boundaries, indicating that our model can generate more customized lung nodules compared to other models. Figure 5 shows the results of DDPM using random noise as input. Although it achieves the worst FID score, it must be acknowledged that its results are more realistic than those of our model, but it can only generate random images.

Fig. 5.

These seven groups correspond to seven methods and their results. Except , which shows the results obtained using Gaussian noise as random input, the others use random bone window images as backgrounds and multi-confidence masks, which are transformed according to Sect. 3.1.1, as the model’s input

Synthesized images enhance segmentation

To further validate the quality of the synthesized lung nodules, we use U-Net [20] and Attention U-Net (Att U-Net) [26] to investigate whether our synthesized images can serve as supplementary training data to improve segmentation performance. In this experiment, we split our data into five folds, using one fold as validation data and the remaining four for training to generate lung nodules. We then test three-fold training data and augment it with one fold of synthesized data to evaluate its impact on segmentation performance. The results are shown in Table 6. The ‘Add Data’ column indicates whether we added the synthesized lung nodules to the training set. As can be seen, both U-Net and Attention U-Net benefit from the inclusion of synthesized data, demonstrating that our synthesis improves segmentation effectiveness.

Table 6.

Segmentation results with augmented data

Models	Add data	Dice [27]
U-Net		83.41
U-Net		84.79
Att U-Net		83.61
Att U-Net		84.48

Ablation experiments

In this section, we evaluate the effectiveness of different components of our model. Specifically, we divide our model into three main parts: (1) the Rough Lung Nodule Generator (RLNG), (2) a DDPM that takes the image synthesized by RLNG and adds Gaussian noise (-DDPM), and (3) another DDPM that takes the image synthesized solely by RLNG (-DDPM).

In Table 7, we observe that each part plays a crucial role. The standalone RLNG produces results similar to those in [6]. When we combine a DDPM as an optimizer with RLNG (-DDPM), the FID score actually worsens compared to RLNG alone, indicating that the DDPM has a negative impact in this case. When we introduce Gaussian noise to the RLNG results and input them into the DDPM (-DDPM), we see a significant improvement in FID, suggesting that our trained DDPM performs better when processing images with some noisy regions. However, we find that -DDPM generates clearer backgrounds than either or -DDPM. Therefore, we combine the results of -DDPM and -DDPM and use a linear remapping to obtain the final result.

Table 7.

Results of albation expriment

RLNG	C-DDPM	N-DDPM	FID
			92.90
			96.22
			90.95
			87.47

Discussions

The choice of and T

In this section, we delve into the exploration of two important hyper parameters, and T, which were introduced in Sect. 3.2. We first independently validate which value of performs better, and based on this, we combine RLNG and LNO to validate the optimal value for T.

In RLNG, we choose three values for : 1, 50, and 100. The results are shown in Fig. 6. We find that yields the worst results, while and produce similar outcomes. The difference between 50 and 100 is observed in the boundaries of the lung nodules. The value of 50 tends to simplify irregular boundaries, while 100 better preserves the details of the boundaries compared to 50.

Fig. 6.

The three lines represent the influence of different values. A value of gives better results

We then combine RLNG and LNO to validate the influence of different T values. In DDPM, the data becomes increasingly noisy with each step of T. During these T steps, noise is added at each time step, so a larger T generally leads to a better denoising effect [7]. However, in our experiments, we found that the "noised image" primarily contains artifacts along the boundaries of the lung nodules. Therefore, we explore different T values to determine whether they can effectively remove these artifacts. To further validate that our method can synthesize customized lung nodules guided by multi-confidence masks, we use the input transformation described in Sect. 3.1.1. The results are shown in Table 8. We can see that achieves the best FID score.

Table 8.

Choice of T

T	10	50	100	500	1000
FID	182.02	105.70	87.47	171.44	365.05

Additionally, except for and , which yielded similar results as shown in Fig. 5, other values of T resulted in worse performance. As shown in Fig. 7, when T is too small, such as , the model lacks denoising ability, and the results are filled with noise. When T equals 500 or 1000, the model’s denoising ability becomes too strong. Since the input to LNO only has some noise around the boundaries of lung nodules, these larger T values tend to generate images that are predominantly filled with large areas of white or black.

Fig. 7.

The influence of different T values in our model

Limitation of FID

According to Table 4, we can see that [7] and [8] achieved the worst FID scores, while [6] produced a relatively good result. However, in Figs. 1B and 5, we observe that the results from [7] are closer to real lung nodules. Even though it sometimes synthesizes empty images, the generated images still resemble those cut from a lung window CT image. On the other hand, the results from [6] show obvious artifacts along the nodule boundaries. These artifacts are easy to identify, even with slight magnification, revealing the images as synthetic. Additionally, in Table 5, all methods produce worse FID scores compared to Table 4. It may partially be due to the model’s inability to synthesize highly irregular lung nodules with the limited training dataset [28]. However, the deeper reason is that we synthesized lung nodules with different feature distributions, such as spicular, textural, and malignant features, by transforming the masks, which led to a higher FID. So we can’t judge the authenticity of synthesized lung nodules by a single FID, but should introduce more Human Evaluation [29, 30].

Limitation of our model

In our approach, we design a unified model that combines a GAN-like architecture and a Denoising Diffusion Probabilistic Model (DDPM) to synthesize lung nodules of designated shapes. However, under certain extreme conditions, when the input mask is made excessively irregular by adding too many spikes or dents (more than ten), the results of our model may not be satisfactory. This is because the Generator tends to synthesize artifacts along the borders of the lung nodules. In some cases, the DDPM preserves these boundary regions, as shown in examples a and b in Fig. 8, resulting in synthesized nodules resembling ‘part-solid nodules’ [31]. Conversely, in some instances, the DDPM ignores these spikes and artifacts at the boundary, generating a clear boundary for the lung nodule, similar to a ‘solid nodule’ [31] as shown in example c of Fig. 8. In particularly challenging situations, even when the Generator does not produce a rough result, the DDPM may still struggle, as demonstrated in example d of Fig. 8.

Fig. 8.

The images on first line are results of RLNG, the second line are optimized by LNO

Future directions

In our experiments, we did not use synthesized lung nodules to evaluate classification accuracy, such as distinguishing between malignant and benign nodules. This is because our objective was solely to customize the synthesis of lung nodules based on their shapes, without introducing semantic information related to pathological features through official annotation labels. Therefore, in our future research, we plan to integrate the characteristics of lung nodules-such as lobulation, spiculation, texture, and malignancy-with segmentation masks. This approach will enable us to generate lung nodules with specific shapes and features.

Conclusions

In this study, we use a multi-confidence mask-guided method for synthesizing customized lung nodules and treat the holes and artifacts in lung nodules synthesized by a GAN-based model as a form of noise. Based on this assumption, we explore the use of a Denoising Diffusion Probabilistic Model (DDPM) for denoising and propose a unified model that combines Generative Adversarial Networks (GAN) and DDPM. This model enables the synthesis of more realistic lung nodules with specified sizes and shapes. In five-fold experiments on the LIDC-IDRI dataset, we synthesized 4384 synthetic lung nodules and achieved the best FID score, outperforming the state of the art. Furthermore, in segmentation experiments, the results demonstrate that our synthesized lung nodules can serve as a supplement to the training set, providing strong evidence for the authenticity of the lung nodules synthesized by our model.

Author contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Huashan Chen, Yongxu Liu and Chen Liu. The first draft of the manuscript was written by Huashan Chen, Qiuli Wang and Rongping Wang. And all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Funding

This work was supported by the Guizhou senior innovative talent project (QKHPTRC-GCC[2022]041-1), Guiyang Science and Technology Project (ZKHT [2022]-4-1-3), Excellent Young Talent Fund of the First Affiliated Hospital of the Army Medical University (2024YQBJ-2) and Joint project of Chongqing Health Commission and Science and Technology Bureau (2025MSXM016).

Declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Ethical approval

The collection and use of the LIDC-IDRI dataset [10] have already been approved by the relevant ethics committees of the National Cancer Institute. Therefore, no additional ethics approval is required for this study to use the publicly available dataset.

Consent to participate

Not applicable in this context.

Consent to publish

Not applicable in this context.