2D CNN versus 3D CNN for false-positive reduction in lung cancer screening

Abstract.

Purpose: To clarify whether and to what extent three-dimensional (3D) convolutional neural network (CNN) is superior to 2D CNN when applied to reduce false-positive nodule detections in the scenario of low-dose computed tomography (CT) lung cancer screening.

Approach: We established a dataset consisting of 1600 chest CT examinations acquired on different subjects from various sources. There were in total 18,280 candidate nodules in these CT examinations, among which 9185 were nodules and 9095 were not nodules. For each candidate nodule, we extracted a number of cubic subvolumes with a dimension of $72\times 72\times 72{\mathrm{mm}}^3$ by rotating the CT examinations randomly for 25 times prior to the extraction of the axis-aligned subvolumes. These subvolumes were split into three groups in a ratio of $8:1:1$ for training, validation, and independent testing purposes. We developed a multiscale CNN architecture and implemented its 2D and 3D versions to classify pulmonary nodules into two categories, namely true positive and false positive. The performance of the 2D/3D-CNN classification schemes was evaluated using the area under the receiver operating characteristic curves (AUC). The $p$-values and the 95% confidence intervals (CI) were calculated.

Results: The AUC for the optimal 2D-CNN model is 0.9307 (95% CI: 0.9285 to 0.9330) with a sensitivity of 92.70% and a specificity of 76.21%. The 3D-CNN model with the best performance had an AUC of 0.9541 (95% CI: 0.9495 to 0.9583) with a sensitivity of 89.98% and a specificity of 87.30%. The developed multiscale CNN architecture had a better performance than the vanilla architecture did.

Conclusions: The 3D-CNN model has a better performance in false-positive reduction compared with its 2D counterpart; however, the improvement is relatively limited and demands more computational resources for training purposes.

1. Introduction

Lung cancer is the leading cause of death associated with cancer, exceeding the next three cancers combined (i.e., colon, breast, and prostate cancer).¹ The high mortality is partly attributed to the lack of early stage symptoms and limited access to screening.² The landmark National Lung Screening Trial (NLST) reported a 20% reduction in lung cancer mortality in smokers screened with low-dose computed tomography (LDCT) scans compared to chest radiograph.^3–6 The Center for Medicare and Medicaid Services decision to cover annual lung cancer screening with chest LDCT scans for asymptomatic adults with a history of tobacco smoking was in part motivated by the NLST findings. The strength of LDCT scanning is primarily attributed to the high spatial resolution in visualizing small nodules, which may be the manifestation of early lung cancer. In practice, a single chest CT examination contains hundreds of CT images. It is time-consuming for a radiologist to visually interpret this large volume of images, which can lead to missed suspicious nodules. To facilitate an accurate and efficient interpretation of the chest CT examinations, a large number of computer algorithms have been developed to automatically detect nodules depicted on CT scans.^7–12 Unfortunately, few algorithms have demonstrated satisfactory performance in practice, thereby limiting their clinical application. The primary issue is the relatively high rate of false-positive nodule detections in order to maintain a satisfactory sensitivity, which can increase the time to interpret an LDCT scan without any benefit.

A variety of image features were explored to reduce the false-positive rate.^13–22 It is not easy to accurately and consistently identify the critical nodule features associated with true nodules. Despite extensive investigative effort dedicated to false-positive reduction, limited progress has been made in this regard. Deep learning technology, such as the convolutional neural network (CNN), has emerged as a novel solution for many medical image analysis problems and demonstrated remarkable performance, particularly in lung nodule detection and false-positive reduction.^23–30 Both two- and three-dimensional (2D and 3D) deep learning architectures have been developed to reduce false-positive nodule detections by classifying a candidate nodule as nodule and non-nodule.^31–44 It is somewhat intuitive that the 3D CNN could achieve better performance than its 2D counterpart could because 3D images contain significantly more information. However, the performance reported in the literature varied significantly.^22,25 The underlying variations could be caused by the differences in the training/testing datasets and/or the CNN architectures. Training and inference of a 3D CNN require much greater computational resources than those of a 2D CNN do. It remains unclear whether the performance improvement of 3D CNN as compared with 2D CNN outweighs the additional computational cost.³¹ In this study, we established a relatively large dataset consisting of 1600 chest CT examinations acquired on different subjects from various sources, which had 18,280 nodule candidates. Of these candidates, 9185 were nodules (“true positive”) and 9095 were not nodules (“false positive”). Our objective is to verify whether the 3D-CNN architectures perform better than the 2D-CNN architectures do for classifying candidate nodules as nodules or non-nodules.

2. Methods and Materials

2.1. Image Datasets

We established an image dataset consisting of 1600 chest LDCT examinations from different sources, including NLST,³ the Lung Image Database Consortium image collection,⁴⁵ and our own database (Table 1).⁴⁶ These CT scans were acquired at different medical centers using different protocols, which can be found in Refs. ^{3, 45}, and ⁴⁶. The image slice thickness ranged from 0.625 to 2.5 mm. The size of the identified nodules ranged from 3 to 30 mm. We excluded the CT scans with a slice thickness $>3.0\mathrm{mm}$ and the nodules (or masses) $>30\mathrm{mm}$. Under the help of a previously developed computer algorithm,¹² we processed these CT examinations and identified a large number of suspicious candidates. The computer detected candidate nodules were reviewed by $>10$ radiologists separately in the past years and grouped into true positive (true nodule) and false positive (non-nodule) categories. To balance the number of cases in the two categories, we selected 9185 and 9095 nodules, respectively, from the two categories. For each candidate nodule, we extracted a cubic subvolume of $72\times 72\times 72{\mathrm{mm}}^3$ to fully enclose a suspicious nodule. To provide a way for data augmentation and thus increase the diversity, we randomly rotated the CT examinations based on the center of the candidate nodules and then extracted the axis-aligned subvolumes. The random rotations were performed $\sim 25$ times. As a result, we have a total of 227,051 subvolumes in the false-positive group and 229,404 in the true positive group. For each subvolume, we also generated its 2D version as a pseudo-RGB image by stacking the three images along the axes (Fig. 1).

Table 1.

Summary of the datasets.

	NLST ($n=651$)	LIDC-IDRI ($n=518$)	PLuSS ($n=431$)	Total
Nodules	2994	3771	2420	9185
Non-nodules	2755	3962	2378	9095
3D/2D postaugmented nodules	74,717	94,215	60,472	229,404
3D/2D postaugmented non-nodules	68,693	98,997	59,361	227,051

Fig. 1.

(a) Illustration of the 2D image generation procedure based on the 3D subvolume that fully encloses a nodule; (b) three images were extracted along the orthogonal axes and (c) their stacking formed a pseudo-RGB image (red, green, and blue channels).

2.2. CNN Architectures

For comparison purposes, we developed two types of CNN architectures. The first one is a vanilla CNN architecture (Fig. 2), where the 2D pseudo-RGB image or 3D image with the same dimension are passed to a backbone network followed by a flattening layer and the combination of a fully connected (FC) layer and a rectified linear unit (ReLu). The output layer was activated by a Softmax function to obtain the predicted probability of each class. This network does not consider the significant variation of the pulmonary nodule size, which may range from 3 to 50 mm in diameter. For classification purposes, we used a cubic subvolume of $72\times 72\times 72{\mathrm{mm}}^3$ to ensure that a nodule could be fully enclosed. The convolution or the pooling operations may smear out the critical details related to the enclosed nodule while smoothing the images, thereby weakening the discriminative capability of the network toward small nodules. Hence, we developed a multiscale CNN architecture (Fig. 3).

Fig. 2.

A vanilla CNN architecture for the classification of the detected pulmonary nodules.

Fig. 3.

Illustration of the developed multiscale CNN architecture formed by three branches. The volumetric input image is cropped to create two additional branches that are centered on a given nodule with different dimensions, namely $48\times 48\times 48$ and $24\times 24\times 24$. The average-pooling and up-sampling operations are used to resize these subvolumes into a uniform dimension, namely $48\times 48\times 48$.

The multiscale CNN architecture is formed by three branches (Fig. 3). These branches have image inputs with the same center but different dimensions, namely $72\times 72\times 72$, $48\times 48\times 48$, $24\times 24\times 24$. We adjusted all these images uniformly to the same size, namely $48\times 48\times 48$ via pooling and up-sampling operations on the fly. The normalized images are passed separately to a backbone network and later concatenated and flattened, followed by the FC layers and the ReLu. Similarly, the output layer is activated by a Softmax function to obtain the predicted probability of each class.

The backbone network is a critical component of the aforementioned CNN architectures. A variety of backbone networks have been developed.^47,48 In this study, we used a basic backbone network that is formed by a sequence of convolution blocks with different numbers of filters (Fig. 4). For fair comparisons, the 3D convolutional kernels were replaced with the 2D counterparts.

Fig. 4.

Illustration of the basic backbone network formed by a sequence of convolution blocks with different numbers of filters (without padding). The convolutional layer has a filter size of $3\times 3\times 3$ ($3\times 3$ in the 2D version) and different numbers of filters. The batch normalization layer, the ReLU, and a pooling size of 2 were used.

2.3. Training of the 2D- and 3D-CNN Models

On the basis of the aforementioned CNN architectures, we developed and trained four models (Table 2). We split the generated 2D/3D samples into three groups: (1) training, (2) validation, and (3) independent test in a ratio of $8:1:1$, respectively, at the patient level. The samples were first clipped between [$-\mathrm{1024,600}$] Hounsfield and then normalized to [0, 1]. To improve the reliability and prevent overfitting, we performed online data augmentation, such as affine transformation (e.g., flip, scale, translation, and rotation), Gaussian additive noise, and intensity shift, during training. All learnable parameters were initialized by the Xavier initialization⁴⁹ and updated by the Adam stochastic optimization algorithm.⁵⁰ Reducing the learning rate on plateau and early stopping were adopted during training. The parameters for training the 2D- and 3D-CNN models were summarized in Table 2. The development of these models was performed on a PC with a 3.50-GHz Intel^® Xeon^® E5-1620 CPU, 8 GB RAM, and NVIDIA Quadro P5000.

Table 2.

Developed classification models.

	Model-VB-2D/model-MB-2D	Model-VB-3D/model-MB-3D
Batch size	128	8
Initial learning rate	0.001	0.001
Learning rate decay patience	2	1
Learning rate decay factor	0.2	0.1
Early stop patience	15	5

Note. VB: vanilla framework with basic backbone; and MB: multiscale framework with basic backbone.

2.4. Performance Validation

We assessed the performance of the developed classification models by testing on the independent testing datasets using the nonparametric receiver operating characteristic analysis. We note that no image in the testing datasets was involved in the algorithm development and the training procedures. We also summarized the average training/inferring time, memory requirement, and floating-point operations (FLOPs) of the involved 3D- and 2D-CNN models for comparison. The number of FLOPs was used as an index for the computational complexity of the CNN models. Whether there was a significant difference in the performance among the developed CNN models was verified using the DeLong’s test.⁵¹ The 95% confidence interval (CI) and $p$-values were computed in all analyses. We considered a $p\text{-}\text{value}<0.05$ statistically significant. The analyses were performed using IBM SPSS Version 24.

3. Results

Model-MB-3D had the best performance in classification with an area under the receiver operating characteristic curves (AUC) of 0.9541 (CI: 0.9495 to 0.9583), a sensitivity of 89.98%, and a specificity of 87.30% (Table 3). In contrast, the best 2D-CNN model was model-VB-2D, which has an AUC of 0.9307, a sensitivity of 92.70%, and a specificity of 76.21%. Although the DeLong’s test showed that there was a significant difference in classification performance between the 2D- and 3D-CNN models ($p<0.05$), the performance difference between the 2D-CNN models and the 3D-CNN models was very limited. Also the proposed multiscale strategy had a better performance than its counterpart with only a uniform dimension did.

Table 3.

Comparison of the classification performance of the 2D-/3D-CNN models and the involved numbers of the parameters.

Model	Sensitivity (%)	Specificity (%)	Accuracy (%)	AUC (95% CI)
Model-VB-2D	92.70	76.21	84.63	0.9307 (0.9285, 0.9330)
Model-MB-2D	77.48	88.21	82.73	0.9205 (0.9181, 0.9229)
Model-VB-3D	85.12	88.87	86.99	0.9375 (0.9324, 0.9430)
Model-MB-3D	89.98	87.30	88.64	0.9541 (0.9495, 0.9583)

We also summarized the computational cost of the 2D/3D-CNN models based on our desktop (GPU: NVIDIA RTX 2070S, CPU: 3.00 GHz Intel^® Core™ i7-9700) in Table 4. The number of FLOPs was much higher for the 3D models than for the 2D models. The batch size for training the two proposed 2D models can be set at the maximum number of 800 and 1000, respectively, whereas the maximum batch size for the proposed two 3D models can only be at 32 and 24, respectively. It took more than 10 h to train a 3D model, whereas the training time of 2D models was only about 4 to 6 h. Once the trained models were used for prediction, the difference of the computation costs between the 2D and 3D models was narrow, namely 0.02 to 0.03 s for processing a single image.

Table 4.

Summary of the computational cost of the 2D-/3D-CNN models.

Model	Maximum batch size in training	Training time (h)	Prediction time (per image) (s)	#Params (M)	#FLOPs (M)
Model-VB-2D	$\sim 1000$	$\sim 4$	0.0158	0.65	1.31
Model-MB-2D	$\sim 800$	$\sim 6$	0.0200	1.37	2.74
Model-VB-3D	$\sim 32$	$\sim 11$	0.0210	1.69	3.39
Model-MB-3D	$\sim 24$	$\sim 13$	0.0260	3.69	7.40

4. Discussion

In this study, we used a relatively large and diverse dataset ($n=1600$ CT scans), which was acquired using different protocols and from various medical centers, to compare the performance difference between the 3D-CNN models and their 2D counterparts to classify computer detected nodule candidates. To our knowledge, no investigation has ever used such a large dataset as we did in this study. We expect that this relatively large and diverse dataset could support a reliable conclusion. In practice, we always need to decide which type of CNN models (i.e., 2D or 3D) should be used for a specific application. Our results demonstrate that a 3D-CNN model narrowly outperformed a 2D-CNN model to classify computer detected candidate nodules as true or false positive detections. Since it is much easier to train and implement a 2D-CNN model than a 3D-CNN model (Table 4), it is desirable to understand the trade-off between the performance gain, which may or may not be statistically significant, and the increased computational cost, such as memory and training time.

In addition, we developed a multiscale strategy in this study in an attempt to maximize the potential of both 2D and 3D images for classification. As demonstrated by our experiments (Table 3), the multiscale strategy had much better performance for both 2D and 3D classification tasks. Notably, the objective of this study is not to develop a novel CNN model to maximize the classification performance but to clarify whether and to what extent 3D CNN is superior to 2D CNN when applied to reduce false-positive detections in the scenario of LDCT lung cancer screening.

We are aware that different implementations of the CNN architectures and the training methods (e.g., data and parameters) may lead to different results. Hence, to perform a relatively fair performance comparison of the 2D and 3D models, we used the same dataset and the same CNN architecture. In the past years, there have been a large number of investigations performed to classify 2D or 3D images,^{31–35,38,39,44} as summarized in Table 5. Although these studies used different datasets, they did not report a better performance for the 3D CNNs as compared with their 2D counterparts as expected, which is consistent with our findings. Our study used a much larger dataset than other investigations did, but the differences between 2D- and 3D-CNN architectures in classifying lung nodules are limited. Given the much richer information in 3D images, we expect that there should be obvious differences in classification performance between 3D-CNN architecture and 2D-CNN architecture. Unfortunately, our experimental results do not support this. Additional investigative efforts are needed to clarify the underlying reasons and develop novel 3D-CNN architectures that could maximize the potential of the rich information in 3D volumetric images.

Table 5.

Comparison of the classification performance of the previous 2D-/3D-CNN models and the involved number of nodules.

Model	Dim	#Candidates/#nodules/#non-nodules	Sensitivity (%)	Specificity (%)	Accuracy (%)	AUC (%)
MV-ResNet38	2D	831/421/410	91.1	88.6	89.9	94.6
SL-CNN31	2D	1882/—/—	78.6	91.2	86.7	—
NL-CNN31	2D	1882/—/—	88.5	86.0	87.3	—
CNN31	3D	1882/—/—	89.4	85.2	87.4	—
PN-SAMP32	3D	1404/898/506	—	—	97.6	—
MC-CNN33	3D	1375/496/880	77	93	87.1	93
MS-CNN34	3D	1375/496/880	—	—	86.8	—
HS-CNN35	3D	4252/3212/1040	70.5	88.9	84.2	85.6
CNN39	3D	326/163/163	—	—	—	73.2
MO-DenseNet44	3D	686/316/370	90.5	90.3	90.4	—

Note: — denotes not available.

There are limitations with this study. First, in the past decade, we have been actively working on medical image analysis related to lung cancer. More than 10 thoracic radiologists have been involved in reviewing the chest CT images from different sources as presented in this study to establish a dataset with “ground truth.” Considering that the interpretation of chest CT images for the purpose of nodule detection is not very challenging for a thoracic radiologist, we did not ask them to read the images as a group but independently. As a result, there could be some errors with the interpretation results unavoidably; however, the errors and their effect on the results should be very limited and should not change our conclusion. Second, the developed multiscale strategy may not be the optimal solution for the nodule classification problem. As we explained, our emphasis is not on the development of classification models but on the clarification of the performance difference between 3D CNN and 2D CNN in classification. Hence, we did not perform extensive compassion with other available CNN architectures. However, compared with other available methods (Table 5), our developed models demonstrated a very promising classification performance. Third, the parameters involved in the CNN models may not be fully optimized for this particular classification task. The training of a CNN model involves a number of parameters, such as learning rate, batch size, filter numbers, and the choice of optimizer. Tuning these parameters may somehow affect model performance. However, the most important factor that affects the performance of a CNN model is the scale and diversity of the dataset for model developing. Given the relatively large and diverse dataset in this study, we believe that the impact of the training parameter choices on our conclusion should be limited.

5. Conclusions

We developed a multiscale CNN framework to perform binary classification of computer detected lung nodule candidates (i.e., false positive and true positive) depicted on LDCT scans. We implemented the 2D and 3D versions of this framework to clarify whether and to what extent a 3D CNN outperforms its 2D counterpart in classifying lung nodules. Our results on a relatively large and diverse dataset showed that the proposed 2D-CNN and 3D-CNN models had comparable performance to classify candidate lung nodules, albeit the latter one yielded slightly better performance. We believe that the existing CNN architectures may not fully utilize the rich information in 3D images. Additional investigative effort may be needed to develop sophisticated CNN models to maximize the rich information in 3D images.

Biographies

Juezhao Yu is a research assistant in the Department of Radiology at the University of Pittsburgh. His research interests are in machine learning, medical image analysis, and computer-aided diagnosis.

Bohan Yang is a research assistant in the Department of Radiology at the University of Pittsburgh. His research interest is primarily related to the development of machine learning algorithms for quantitative image analysis.

Jing Wang is a research assistant in the Department of Radiology at the University of Pittsburg. Her research interest focuses on quantitative medical image analysis and computer graphics.

Joseph Leader is a research assistant professor of radiology at the University of Pittsburgh. His primary research emphasis is on quantitative lung image analysis.

David Wilson is an associate professor of medicine at the University of Pittsburgh. His primary clinical practice and research interest include lung cancer screening and chemoprevention, diagnosis, staging, and treatment; COPD, especially as it relates to lung cancer.

Jiantao Pu is an associate professor of radiology and bioengineering at the University of Pittsburgh. The primary objective of his research is to develop innovative computational tools for non-invasive, quantitative, and accurate assessment of pathological conditions.

Disclosures

The authors have no relevant financial interests in the manuscript and no other potential conflicts of interest to disclose.