A hierarchical GAN method with ensemble CNN for accurate nodule detection

Abstract

Purpose

Lung cancer can evolve into one of the deadliest diseases whose early detection is one of the major survival factors. However, early detection is a challenging task due to the unclear structure, shape, and the size of the nodule. Hence, radiologists need automated tools to make accurate decisions.

Methods

This paper develops a new approach based on generative adversarial network (GAN) architecture for nodule detection to propose a two-step GAN model containing lung segmentation and nodule localization. The first generator comprises a U-net network, while the second utilizes a mask R-CNN. The task of lung segmentation involves a two-class classification of the pixels in each image, categorizing lung pixels in one class and the rest in the other. The classifier becomes imbalanced due to numerous non-lung pixels, decreasing the model performance. This problem is resolved by using the focal loss function for training the generator. Moreover, a new loss function is developed as the nodule localization generator to enhance the diagnosis quality. Discriminator nets are implemented in GANs as an ensemble of convolutional neural networks (ECNNs), using multiple CNNs and connecting their outputs to make a final decision.

Results

Several experiments are designed to assess the model on the well-known LUNA dataset. The experiments indicate that the proposed model can reduce the error of the state-of-the-art models on the IoU criterion by about 35 and 16% for lung segmentation and nodule localization, respectively.

Conclusion

Unlike recent studies, the proposed method considers two loss functions for generators, further promoting the goal achievements. Moreover, the network of discriminators is regarded as ECNNs, generating rich features for decisions.

Introduction

According to the World Health Organization (WHO), cancer is a leading cause of death worldwide, accounting for nearly 10 million deaths in 2020, or nearly one in six deaths. An average of 2.21 million people are annually diagnosed with lung cancer. The death toll of this cancer reaches 1.8 million [1]. Advances in artificial intelligence technology play an important role in medicine, as in many areas of life. In medicine, intelligent computer systems help doctors interpret, diagnose and treat diseases. This reduces human error and saves time. For example, in medical imaging, artificial intelligence can be used for various applications, such as improving the accuracy of diagnosing sensitive diseases such as cancer, increasing the speed of disease diagnosis, and helping with more effective treatment. One of the new approaches in the field of artificial intelligence is deep learning [2].

So far, a series of lung segmentation techniques have been developed based on handmade features and manual adjustment of experimental parameters. Nevertheless, these methods are mainly helpful for specific applications and datasets and their use for medical datasets, especially CT images, is practically useless as various features were ignored in the feature extraction phase. Furthermore, the feature extraction method is monitored by users for obtaining the manual and interactive set of features. In general, machine learning algorithms have two stages of feature extraction and learning, which finally produce an automatic system. In the first step, the expert should determine the type of feature extraction algorithm or adapt it to the problem. In deep learning techniques, feature extraction and learning are combined and there is no need to determine the feature extraction algorithm [3]. Feature extraction has become automated in many areas with the advent of deep learning [4–8]. Deep learning methods commonly perform better than the handmade ones. In this regard, several methods have been proposed for pixel classification based on deep learning, some of which have been employed in medical imaging. The basic technique uses a bounding box in which a patch, including its neighbors, predicts each pixel class label. Kallenberg et al. [9] proposed a deep learning method with a bounding box for segmenting and scoring breast nodules. For comparison, Shin et al. [10] presented some networks for computer-aided detection. They also offered a transfer learning approach using models trained in the computer vision discipline for the medical imaging issue. Lang et al. [11] proposed a fully convolutional network for semantic segmentation, in which all of the fully connected layers were replaced with convolutional ones. Their model lacked pixel-wise classification with localization.

In general, deep learning algorithms pursue specific goals for nodule detection. A group of algorithms distinguish the area of a nodule from other regions. Some of the studies in this category work pixel by pixel. Wang et al. [12] developed a multi-view CNN technique that examines each pixel to check if it belongs to a nodule. Their proposed method comprises three CNN models from sagittal, coronal and axial views as input. Mukherjee et al. [13] hypothesized that combining model-based and data-driven approaches and constructing a hybrid model may work more properly in nodule segmentation. To this end, a deep learning graph cut was proposed in an energy minimization-based segmentation framework, combining a domain-specific cost function using low-level image features and deeply learned object localization. Zhu et al. [14] developed a method based on expectation–maximization (EM) to make a deep 3D CNN framework to find weakly supervised labels. Xie [15] proposed a 3D U-Net that used ResNet and DenseNet architecture for pulmonary nodule detection.

GAN [16] is a type of neural network developed for semi-supervised and unsupervised learning. In GANs, two networks, called generator and discriminator, are trained concomitantly. The generator is aimed to concentrate on image generation. In contrast, the discriminator should isolate real samples from fake ones [17]. The application of GAN is accompanied by several benefits. It generalizes with little data, makes new scenes from small datasets and produces simulated data that look more natural. More recently, GANs have been utilized in medical imaging in two manners. The first one concentrates on the generative element to analyze and find the basic training data structure and generate new images. This characteristic makes GANs significantly favorable in faring with insufficient data and patient privacy. The second one emphasizes the discriminative features to normalize images, regularizing or detecting when offered with unnatural images. Most medical imaging tasks utilize large-scale 3D images, including thorax CTs and brain MRIs; GANs are currently suitable for these applications [18].

Methods presented for lung segmentation and nodule segmentation suffer from two major drawbacks. Most of them consider the problem as a pixel-wise classification such that pixels of a lung or nodule are in a class and the rest will be in another group [19]. Therefore, the pixels of the majority class overcome the minority class, leading to an imbalanced classification and, hence, decreasing system performance. The second weak point involves the feature extraction from 2D images. Most methods use pre-trained models (PMs), including VGG as a feature extractor [20]. However, PMs are developed for natural images. Considering these two drawbacks, this paper proposes a two-step process for nodule detection. In the first step, CT scan images are segmented using a GAN-based model. The generator structure relies on the U-net network, a pixel-wise classification of the input image with two classes. If a pixel is a member of the lung, it is labeled one; otherwise, it belongs to the zero class. The model tends to become imbalanced due to many zero-class members. This problem was resolved by the focal loss function, which gives more importance to the minority class (class one). In the meantime, the discriminator trains the generator while separating the actual data from the fake one. The discriminator network employs ECNNs whose different CNN architectures extract rich features for decision-making. The next step involves the isolation of the disease site in the segmented images of the previous step by using another GAN. For this purpose, the mask R-CNN network was considered as a generator to extract the bounding box of potential nodule locations with its corresponding class by finding the region of interest (ROI). A new error term was also added to the generator loss function to improve the generated bounding box. The discriminator model uses ECNN like the first GAN.

Methods

The proposed model consists of two GANs, one employed for segmentation and the other for nodule detection. An overview of the proposed model is illustrated in Fig. 1 which will be described separately in the following sections.

Fig. 1.

Overview of the proposed model

Segmentation

The purpose of the segmentation is to make the masks for the lungs as close to the ground truth mask as possible. Two networks are used for this purpose: a generator network (G) and a discriminator network (D), whose architectures are based on CNNs. The generator is trained to predict lung masks based on the grayscale input CT slices. At the same time, the discriminator helps the generator to learn authentic masks by reducing the EM distance between the predicted mask and the ground truth one. During training time, the generator takes a lung CT scan image $I_i$ as a condition and outputs the mask $M_i$, indicating that each pixel may belong to the lung. The discriminator network measures the degree of authenticity of $M_i$. In the following, the generator architecture, the discriminator architecture and the training loss are defined.

Generator architecture

In this case, the generator network is aimed to produce the CT scan image mask to label the pixels belonging to a lung. In other words, segmentation can be considered as a pixel-wise classification problem to determine whether a pixel is related to the lung area or not. Given a CT image $I_i$, the generator will specify the class of each pixel and make the corresponding mask $M_i$ for this classification. The generator consists of two parts: an encoder and a decoder. While the encoder has to extract the multi-scale features from image $I_i$ by many convolution blocks, the decoder makes masks of the multi-scale features extricated by the decoder. The encoder and decoder encompass convolution blocks. The generator network is defined as a binary wise-pixel classifier, in which the pixels belonging to the lung are one and the rest of the pixels are zero. As the number of instances of the zero class is more than that of the other class, the classifier becomes imbalanced. Imbalanced classifiers tend to the whole sample to one class, drastically reducing the model performance. To avoid this problem, a focal loss function was taken into account to train the generator. Typically, it is designed to detect a one-stage object in which there is an imbalance between the foreground and background classes. Focal loss is a modified binary cross-entropy, forcing the model to learn complex samples (minority class data) instead of easy ones, which is defined as Eq. 1:

$$\mathrm{FL}\left(p_{\mathrm{t}}\right)=-{\alpha }_t{\left(1,-,p_t\right)}^{\gamma }\log \left(p_{\mathrm{t}}\right),$$ 1

where $\gamma >0$ is a tunable focusing parameter and $\alpha$ varies in the interval [0,1] that can be considered as an inverse class frequency. $p_t$ ∈ [0,1] is the model estimated probability for the class with label one.

Discriminator model

The purpose of a discriminative network is to separate the actual masks from the masks produced by the generator. For this purpose, the EM distance is utilized to measure the difference between actual and learned distributions as it is a smooth metric. The discriminator approximates the E [·] function such that the EM distance defined as $E_{x\sim pz}[x]-E_{x\sim \mathrm{preal}}[x]$ is estimated as $D_{x\sim p_z}(G(x))-D_{x\sim p_{\mathrm{real}}}(x)$. Unlike a discriminator in the vanilla GAN network that performs the classification task, the proposed discriminator conducts regression work, estimating the E (·) function. The generated and real images are similar in image generation tasks with GAN-based networks. Nonetheless, for lung segmentation, the intensity of the pixels in the generator-generated mask varies in the range of [0,1], while the actual mask is binary, i.e., with values of 0 and 1. This may mislead the discriminator into simply recognizing the difference between a real and a manufactured mask. Hence, the generator output or the actual masks should not be directly fed into the discriminator. Consequently, the discriminator input is the lung slice images that have been mapped from the original CT scan image by the segmented mask and the ground truth mask, respectively. As a supplementary, a segmentation mask is given, the original image is changed so that the nodule areas are kept, but the rest of the parts are set to 0. The discriminator loss function, called a product network, is defined by Eq. 2:

$$E_{x\sim p_z}[D(G\left(x\right)\mathrm{o}{I}_i)]-E_x\sim p_{\mathrm{real}}[D(x)]$$ 2

where ο denotes the operator so that $a\mathrm{o}b$ indicates the pixel-wise multiplication of the matrices $a$ and $b$, respectively.

Detection

As shown in Fig. 1, a mask R-CNN network was taken as the generator network. In this model, the generator takes the lungs extracted from the previous stage instead of receiving a random noise as input (similar to a vanilla GAN). It outputs the class and bounding box prediction. In this case, the generator loss function is considered as Eq. 3:

$$L_{\mathrm{G}}=L_{\mathrm{cls}}+L_{\mathrm{box}}+L_{\mathrm{adv}}^{{\mathrm{G}}_{\mathrm{b}}},$$ 3

where $L_{\mathrm{cls}}$ and $L_{\mathrm{box}}$ are exactly the loss defined in the R-CNN Mask network and stand for loss of class, bounding box, respectively. $L_{\mathrm{adv}}^{{\mathrm{G}}_{\mathrm{b}}}$ shows an adversarial loss term that helps to further optimize the generator with the following expression:

$$L_{\mathrm{adv}}^{{\mathrm{G}}_{\mathrm{b}}}=\frac{1}{N}\sum _{i=1}^N-\log D_b\left(G_b,\left({\mathrm{RoI}}_i\right)\right),$$ 4

where $N$ indicates the mini-batch size and $G_b\left({\mathrm{RoI}}_i\right)$ is the $i$th bounding box prediction.

The goal of the discriminator is to look at the bounding boxes generated by the generator and to evaluate their quality. In practice, just looking at the four coordinates of the box does not offer a correct assessment. The solution is to simultaneously send feature maps of real and fake boxes. The discriminator architecture includes the ensemble of two CNNs with the loss function defined as Eq. 5:

$${\mathrm{L}}_{\mathrm{D}}=\frac{1}{N}\sum _{i=1}^N-[\log (D_b({\mathrm{bb}}_i^{g_t}))+\log (1-D_b(G_b\left({\mathrm{RoI}}_i\right)))],$$ 5

where $G_b\left({\mathrm{RoI}}_i\right)$ is the $i$th bounding box ground truth. Eventually, $D_b$ indicates the probability that it is an actual image. Generally, ${\mathrm{L}}_{\mathrm{adv}}^{{\mathrm{G}}_{\mathrm{b}}}$ motivates $G_b$ to generate a bounding box that can fool $D_b$.

Experimental results

Dataset

The Lung Image Database Consortium and Image Database Resource Initiative (LIDC-IDRI) [21] is the most commonly used Computational Tomography (CT) dataset. It was initiated by the US National Cancer Institute (NCI) in 2000 to “convene a consortium of institutions to develop consensus guidelines for the creation of a CT-based lung nodule reference database,” and later supported by the Foundation for the National Institute of Health (FNIH) to include more CT images from the private companies. After a decade of work, the database was published in 2011, including 1018 thoracic CT scans from 1010 different patients from seven academic institutions and eight private companies. Each CT scan image has a resolution of 512 × 512 × width, whose width ranges from 65 to 764 slices. The total number of whole image slices in the LIDC-IDRI is 244527.

In addition, each CT scan is well annotated where four independent radiologists categorize lesions into three classes (nodule >  = 3 mm, nodule < 3 mm and non-nodule >  = 3 mm), and further diagnose the patient into four classes (nonmalignant disease, primary lung cancer, metastatic disease or unknown). This dataset contains a lot of information about the nodule and has been explored a lot in the past 10 years since its inception. Lung Nodule Analysis 2016 or LUNA16 was set up as a challenge for the computer science community to build good computer-aided diagnostics models for lung nodule detection. The corresponding LUNA16 dataset85 is a high-quality subset of the LIDC-IDRI dataset. Since lung nodules are very small, they are not easily identifiable in thick CT scans. Therefore, the LUNA16 dataset excludes any CT scans in LIDC-IDRI with slice thickness greater than 2.5 mm. Additionally, scans with incomplete or inconsistent slice spacing are also excluded. Only scans with lung nodule annotation where all the radiologists reached a consensus are kept in the dataset, leaving only 888 scans with 777 nodules in LUNA16. An example of dataset images used in the proposed method is shown in Fig. 2.

Fig. 2.

LUNA16 samples

The evaluation criteria utilized in this article are intersection over union (IoU) [22] and Hausdorff distance (HD) [23]. Also, the datasets are divided into two categories using the hold-out method, namely training and test sets. Moreover, 70% of the data are considered for training and 30% for testing.

Segmentation

The proposed method was compared with five state-of-the-art models on the LUNA dataset, including LGAN [19], ED-CNN [24], D-CNN [25], U-net [26], and Tiramisu [27]. Table 1 lists the evaluation results of the proposed approach. As can be seen, the proposed approach reaches the highest score as compared with other methods with IoU = 0.933 and HD = 2.600. Although the model structure is similar to LGAN, the proposed approach has reduced its error by 34.95% in the IoU metric. ED-CNN and D-CNN models are in the ranks after LGAN, which despite the use of robust deep learning networks, are still weak compared to the proposed approach. The proposed approach can reduce ED-CNN and D-CNN by more than 51% and 47%, respectively. The proposed method is much shallower and achieves better performance (about 57% improvement) as compared with the 100-layer Tiramisu [27] method. Moreover, traditional methods such as Morph [28] require a series of steps, including thresholding and component analysis, while the proposed model only supplies a one-step approach.

Table 1.

Performance of the proposed approach

HD	IoU	Method
2.118	0.897	LGAN [19]
1.894	0.862	ED-CNN [24]
1.760	0.873	D-CNN [25]
1.328	0.810	Morph [28]
1.870	0.842	Tiramisu [27]
2.600	0.933	Proposed

Analysis of the number of CNNs for the discriminator used in the segmentation model

Each CNN in the discriminator model extracts several vital features from the image, so the performance of the differentiator is expected to improve by enhancing the number of CNNs. However, adding CNN networks to a certain point enhances the performance of the model, and after that, it backfires. Because additional CNNs have virtually no use, they only increase network parameters and reduce model performance. To test how many CNN networks are adequate for the discriminator, the number of CNNs was raised from one to six. The results are shown in Fig. 3. As can be seen, the model improves by increasing the number of CNNs to three, beyond which, the results for both criteria show a downward trend.

Fig. 3.

Performance of the proposed model for different numbers of CNN networks in the discriminator of the lung segmentation model

Analysis of loss function

The focal loss function was employed for generator training. To investigate the effect of other loss functions on the generator, five functions were selected: weighted cross-entropy (WCE) [29], balanced cross-entropy (BCE) [30], dice loss (DL) [31], Tversky loss (TL) [32] and sensitivity specificity loss (SSL) [33]. WCE is a type of BCE in which positive samples are weighed. DL was generally used to measure the similarity between two images but is later adopted as a loss function. The results obtained by these loss functions are given in Table 2. As can be seen, the WCE function improved the IoU measurement metric by only 14%, despite the allocation of weight to the samples. SSL had the best performance and reduced the error by about 38%. However, SSL showed some weaknesses as compared to FL.

Table 2.

Performance of the proposed model with different loss functions for the generator

Loss	IoU	HD
WCE [29]	0.830	1.39
BCE [30]	0.852	1.76
DL [31]	0.870	1.85
TL [32]	0.882	2.108
SSL [33]	0.906	2.201

Nodule detection

In the proposed algorithm, after dividing the lung, the nodule areas are also identified as a bounding box, which can be compared with the actual bounding box to calculate the effectiveness of an algorithm. This section compares the nodule segmentation algorithms (such as 3D CNN [34], PN-SAMP-M [35], R-CNN [20], DA-Net [36] and MC-CNN [37]) with the proposed model. The results of the segmentation evaluation are listed in Table 3 in terms of IoU and HD criteria for the LUNA dataset. According to the analysis, many studies have tried to improve nodule segmentation and have achieved relatively excellent results. Based on the comparisons, the IoU value of the MC-CNN model was 0.584, which was further improved by the 3D CNN model. The proposed PN-SAMP-M model reached an IoU score of 0.682, exhibiting a 17% improvement as compared with the previous two models. The DA-Net model is more recent and showed an IoU score of 0.739. However, the proposed model achieved an IoU value of 0.781, approximately 16% higher than the top algorithm, i.e., DA-Net.

Table 3.

Comparative analysis of the proposed model with the state-of-the-art works

Algorithm	IoU	HD
3D CNN [34]	0.620	0.978
PN-SAMP-M [35]	0.682	1.089
R-CNN [20]	0.702	1.264
DA-Net [36]	0.739	1.479
MC-CNN [37]	0.584	0.872
Proposed	0.781	1.614

In addition, to compare the quality of the proposed nodule detection system based on true positives and false positives, first the proposed results have been compared with the results of DA-Net method based on symmetric volume difference (SVD), sensitivity and Dice similarity coefficient (DSC). Then, false positive reduction in the proposed method was calculated and compared with that of DA-Net method. It is evident from Table 4 that the results of the proposed method outperformed those of the other one.

Table 4.

Comparative analysis of the proposed model with the state-of-the-art by true positives- and false positives-based criteria

Algorithm	SVD	Sensitivity	DSC	False positive reduction
DA-Net [36]	0.19	87.2	81	0.812
Proposed	0.16	90.1	82	0.896

Analysis of the number of CNNs for the discriminator used in the nodule segmentation model

The nodule segmentation discriminator uses multiple CNN networks the same as the proposed differentiator for lung segmentation. To achieve the optimal number of CNNs, its number was increased from 1 to 5. Note that the first part of the proposed model remained the same for a fair judgment and only the structure distinguished in the nodule segmentation model was changed. The results of this evaluation are shown in Fig. 4. As it is clear, value 2 is the best, beyond which the results are almost stable.

Fig. 4.

Performance of the proposed model for different numbers of CNN networks in the discriminator of the nodule segmentation model

Discussion

In the segmentation model, the generator model was considered as a U-Net network, while the discriminator model was based on the ensemble of convolutional neural networks. According to Table 1, two GAN-based models, LGAN and the proposed method, statistically outperform the others. Four CT images are randomly selected from the output of these models. The segmentation results are depicted in Fig. 5. Selected samples illustrate that the LGAN model detects some extra points which did not classify the lung points as the lung. Moreover, LGAN lost some fragments of the lung. However, the proposed method has a numerical and qualitative advantage in lung segmentation. Furthermore, the proposed model converged to an almost perfectly optimal point. To examine this, the discriminator outputs are presented in Fig. 5. The purpose of the proposed method in this article is not to segment the lung and extract the left and right lungs separately. Rather, the goal is to accurately extract the area inside the lung to reduce the search space for finding nodules and to increase the accuracy. The discriminator convergence is such that its output eventually drops to 0.5 for real and fake data after a certain number of epochs. The generator loss function can be also found in Fig. 6. As can be seen from the figure, the generator loss also approaches zero as much as possible. Furthermore, the designed segmentation model creates an essential step to detect nodule location as this step seems to be a preprocessing step for the subsequent step. The generator model can still achieve more significant improvements with better networks. In the nodule detection model, an R-CNN network was implemented as the generator architecture and the discriminator was executed as an ensemble of convolutional neural networks. Statistical results showed that the proposed model could generally improve the results by 30% as compared with the runner-up model, DA-Net. The qualitative results for two images are depicted in Fig. 7 for all of the models. Models PN-SAMP-M, 3D CNN, and MC-CNN point to non-nodule locations, while DA-Net and R-CNN outperformed these models. However, their extraction areas have little intersection with the actual locations. Also, according to Table 4, the proposed model has shown better performance for nodule detection as compared to the results of DA-Net model in terms of true positives and false positives-based evaluation criteria such as SVD, Sensitivity and DSC. The discriminator output diagram and the generator loss function are also shown in Fig. 8. In this model, the best performance was achieved for the generator and the discriminator.

Fig. 5.

Examples of segmented lungs for the proposed models and LGAN

Fig. 6.

Loss function of the generator and discriminator output in the segmentation model

Fig. 7.

Examples of nodule detection for methods

Fig. 8.

Loss function of the generator and discriminator output in the detection model

Conclusion

This paper employed a two-step method based on a generative adversarial network for nodule detection. In the first step, the CT images taken from the LUNA dataset were entered into a generator network implemented as a pixel-wise classification with a U-net network, in which the left and right lungs were extracted. The focal function was applied to handle imbalanced classification, by paying more attention to the minority class. The extracted lungs were then fed separately into another generator built with a mask R-CNN network. This network outputs the nodule class and locations as a bounding box. A novel loss function was also utilized for the generator to help the bounding box production process, further deceiving the discriminator. An ensemble of convolutional neural networks was considered for the discriminator networks in both steps to hierarchically extract each CT image feature. We designed several experiments to determine the effectiveness of each component. Experiments obtained indicate that the proposed model can be considered a state-of-the-art model.

Future studies may focus on providing an end-to-end method, while improving the proposed approaches, for lung segmentation and nodule detection. Moreover, the proposed approach can be applied to other computer vision applications, including COVID-19 diagnosis.

Declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All studies involving postmortem subjects followed the procedures for informed consent as described in the Declaration of Helsinki.

Informed consent

Informed consent was obtained from all individual participants included in the study.