Automatic detection of pulmonary nodules on CT images with YOLOv3: development and evaluation using simulated and patient data

Abstract

Background

To develop a high-efficiency pulmonary nodule computer-aided detection (CAD) method for localization and diameter estimation.

Methods

The developed CAD method centralizes a novel convolutional neural network (CNN) algorithm, You Only Look Once (YOLO) v3, as a deep learning approach. This method is featured by two distinct properties: (I) an automatic multi-scale feature extractor for nodule feature screening, and (II) a feature-based bounding box generator for nodule localization and diameter estimation. Two independent studies were performed to train and evaluate this CAD method. One study comprised of a computer simulation that utilized computer-based ground truth. In this study, 300 CT scans were simulated by Cardiac-torso (XCAT) digital phantom. Spherical nodules of various sizes (i.e., 3–10 mm in diameter) were randomly implanted within the lung region of the simulated images—the second study utilized human-based ground truth in patients. The CAD method was developed by CT scans sourced from the LIDC-IDRI database. CT scans with slice thickness above 2.5 mm were excluded, leaving 888 CT images for analysis. A 10-fold cross-validation procedure was implemented in both studies to evaluate network hyper-parameterization and generalization. The overall accuracy of the CAD method was evaluated by the detection sensitivities, in response to average false positives (FPs) per image. In the patient study, the detection accuracy was further compared against 9 recently published CAD studies using free-receiver response operating characteristic (FROC) curve analysis. Localization and diameter estimation accuracies were quantified by the mean and standard error between the predicted value and ground truth.

Results

The average results among the 10 cross-validation folds in both studies demonstrated the CAD method achieved high detection accuracy. The sensitivity was 99.3% (FPs =1), and improved to 100% (FPs =4) in the simulation study. The corresponding sensitivities were 90.0% and 95.4% in the patient study, displaying superiority over several conventional and CNN-based lung nodule CAD methods in the FROC curve analysis. Nodule localization and diameter estimation errors were less than 1 mm in both studies. The developed CAD method achieved high computational efficiency: it yields nodule-specific quantitative values (i.e., number, existence confidence, central coordinates, and diameter) within 0.1 s for 2D CT slice inputs.

Conclusions

The reported results suggest that the developed lung pulmonary nodule CAD method possesses high accuracies of nodule localization and diameter estimation. The high computational efficiency enables its potential clinical application in the future.

Introduction

Automated pulmonary nodule detection has been a longstanding topic for lung cancer diagnosis. The implementation of computer-aided detection (CAD) systems for nodule detection is a hallmark endeavor used to optimize efficiencies and cost during routine clinical practice. Since the 1980s, conventional CAD studies have investigated multiple methods to detect nodule candidates [Hessian matrix (1-3); Stable 3D Mass-Spring Models (4); thresholding (5); 3D template matching (6)] and minimizing positive detection rates [support vector machine (SVM) (1); neural network (7)]. However, the high false-positive rates and feature annotation costs have remained as hurdles to their clinical implementation.

Recently, deep learning techniques have become popular in academia and industry. Convolutional neural networks (CNN) is the most common deep learning technique in imaging science, with the ability to perform pixel-wise feature extraction (8). Their multiple applications are evidenced by their use in self-driving cars, image segmentation, and facial recognition. Several studies have applied CNN architectures to nodule classification challenges (9-13). However, as they are typically constrained by their binary output (i.e., exist/absence, benign/malignant), these classification systems have not been able to provide detailed nodule information, such as anatomic location and diameters. As such, preliminary two-stage CNN-based CAD systems were developed for pulmonary nodule detection (14,15). In general, these methods consisted of two sub-systems, responsible for (I) the detection of suspicious nodules and (II) the reduction of false-positive rates. Due to the two-stage design, these CAD systems have complex detection procedures and are therefore prone to exceedingly high computational costs. As an attempt to address the issues associated with two-stage CAD systems, a few single-stage CAD systems for lung nodule detection have been reported with reduced computational costs (16,17). However, these single-stage CAD systems were associated with high false-positive rates compared to the two-stage systems. Also, computational costs are high for 3D volumetric detection (18). Furthermore, the ability of these CAD systems to determine nodule anatomic localization and diameter measurement has not been investigated.

In this study, we developed a single-stage pulmonary nodule CAD method based on a novel CNN algorithm—You Only Look Once (YOLO) (19). YOLO has been used in multiple diagnostic modalities, including digital mammograms (20,21), lung X-ray and computed tomography (CT) (22,23), and electroencephalography (EEG) (24). The proposed method customized the latest YOLO v3 algorithm as a CNN implementation (25). The developed method can simultaneously achieve nodule localization and diameter measurements with streamlined computational efficiency using a light-weight architecture. The accuracy of the developed method was first evaluated in a computer simulation study, using a digital phantom. It was subsequently examined in a patient study where data was extracted from a public database, and compared with 9 current lung nodule detection methods.

Methods

CAD method design

The developed CAD method customized a YOLO v3 CNN algorithm for the detection of pulmonary nodules. Figure 1 demonstrates a schematic illustration of the CNN architecture. In general, the network consists of two major components: (I) a feature extractor that screens nodule characteristics among the input data, and (II) a bounding box generator that determines nodule coordinates and diameter. This CAD method takes 2D axial CT slices as input and yields nodule-specific quantitative values (i.e., number, existence confidence, central coordinates, and diameter) as output.

Figure 1.

Convolutional neural networks (CNN) structure illustration of the developed computer-aided detection (CAD) method. (A) Feature extractor; (B) extension version of the feature map; (C) bounding box generator.

As shown in Figure 1A, the feature extractor is, more specifically, a residual network that contains seven residual units (ResUnit) (26). Each ResUnit has two or three convolutional layers with a skip connection design. A series of pooling layers in the ResUnits (ResUnit 5, 6, and 7) allow the feature extractor to screen potential nodules across three spatial scales. Three feature maps are subsequently generated by the feature extractor with 1/4, 1/8, and 1/16 of the input image resolution. The information stored at each site within the correlating feature space is responsible for a confined input range (i.e., it is spatially dependent). In the low-, medium-, and high-resolution feature maps, the voxel range coverage is 4×4, 8×8, and 16×16 pixels of the input image, respectively. Each feature map has 128 feature slices, as demonstrated in Figure 1B.

The bounding box design is then utilized to describe the nodule location and diameter. As demonstrated in Figure 1C, a seven-layer generator network utilized feature maps as inputs to predict the parameters of the bounding box. These encompassed central coordinates ($b_x/b_y$), height and width ($b_h/ b_w$), and nodule score ($b_0$). Bounding boxes were derived from anchor boxes, which were pre-defined bounding boxes at each position in the feature spaces. By associating the feature information at each map position, the bounding box generator predicted the values of coordinate shifts ($t_x/t_y$) and length changes ($t_h/ t_w$) from the anchor box to bounding boxes. Therefore, the central coordinates ($b_x/b_y$) were predicted by,

$$b_x=\sigma \left(t_x\right)+c_x$$ [1]

and

$$b_y=\sigma \left(t_y\right)+c_y$$ [2]

where, and were the coordinates of the anchor boxes in the feature space. The bounding box lengths ($b_h/ b_w$) were determined by,

$$b_w=p_w\cdot e^{t_w}$$ [3]

and

$$b_h=p_h\cdot e^{t_h}$$ [4]

where, $p_w$ and $p_h$ were the lengths of the anchor boxes. The nodule score ($b_0$) was calculated by the multifaction of two variables: the object score ($t_o$) and the nodule class score (c), which indicated the presence of an object in a feature position and the category of the detected object independently. For each feature map, 3 anchor boxes are designed at each position in the feature space. As seen in Table 1, they have different sizes to measure nodules across a wide range of diameters. As multiple anchor boxes at each voxel position may create detection redundancy, a non-maximum suppression algorithm was used when a nodule was detected by several bounding boxes simultaneously (27). This algorithm calculated the overlap among all detected bounding boxes. Bounding boxes that exceeded an overlap threshold were excluded.

Table 1. Size of anchor boxes in high-/medium-/low-resolution feature maps.

Feature map resolution	Anchor boxes	Heights (pixels)	Width (pixels)
High	1	8	8
	2	10	10
	3	12	12
Medium	4	15	15
	5	20	20
	6	25	25
Low	7	30	30
	8	35	35
	9	40	40

The loss function used for network training includes three terms (i.e., object loss ($L_{object}$), class loss ($L_{class}$), and bounding box loss ($L_{box}$), and was mathematically defined as,

$$L=L_{object }+L_{class}+{\lambda }_{box}\cdot L_{box}$$ [5]

where referred to the weighting factor for bounding box loss. Object loss ($L_{object}$) was defined by,

$$L_{object}={\displaystyle \sum }_{i=0}^{S^2}{\displaystyle \sum }_{j=0}^B{1}_{i,j}^{obj}\left[-log\left(t_o\left(i,j\right)\right)\right]+{\displaystyle \sum }_{i=0}^{S^2}{\displaystyle \sum }_{j=0}^B\left(1-1_{i,j}^{obj}\right)\cdot \left[-log\left(1-t_o\left(i,j\right)\right)\right]\cdot \kappa \left(t_o\left(i,j\right),\tau \right)$$ [6]

where S was the feature map resolution, and B represented the number of bounding boxes in each feature map. $1_{i,j}^{obj}$ was a binary value that indicated the existence of an object inside the jth bounding box of ith feature position. $\kappa \left(t_o\left(i,j\right),\tau \right)$ is a binary function and is triggered when $t_o\left(i,j\right)$ was bigger than the threshold (τ). Class loss ($L_{class}$) was defined by,

$$L_{class}={\displaystyle \sum }_{i=0}^{S^2}{\displaystyle \sum }_{j=0}^B{1}_{i,j}^{obj}\left[{\displaystyle \sum }_{m\in classes}^{M}BCE\left(c_m\left(i,j\right),{\widehat{c}}_m\left(i,j\right)\right)\right]$$ [7]

where BCE represented binary cross-entropy function. $c_m\left(i,j\right)$ represented the predicted class score of category m inside the jth bounding box of ith feature position and ${\widehat{c}}_m\left(i,j\right)$ was the corresponding ground truth. Bounding box loss ($L_{box}$) evaluated the difference between the predicted bounding box parameters ($t_x,t_y,t_h,t_w$) and the ground truth (${\widehat{t}}_x,{\widehat{t}}_y,{\widehat{t}}_h,{\widehat{t}}_w$). It was mathematically defined as,

$$L_{box}={\displaystyle \sum }_{i=0}^{S^2}{\displaystyle \sum }_{j=0}^B{1}_{i,j}^{obj}\left\{BCE\left(t_x\left(i,j\right),{\widehat{t}}_x\left(i,j\right)\right)+BCE\left(t_y\left(i,j\right),{\widehat{t}}_y\left(i,j\right)\right)+SE\left(t_h\left(i,j\right),{\widehat{t}}_h\left(i,j\right)\right)+SE\left(t_w\left(i,j\right),{\widehat{t}}_w\left(i,j\right)\right)\right\}$$ [8]

where SE is the square error function.

Experiment design

The proposed CAD method was trained and evaluated via two independent studies: (I) a computer simulation study and (II) a patient study from a public database. These two studies assessed the CAD performance in the computer-based ground truth and human-based ground truth, separately.

In the computer simulation study, 300 3D CT scans containing detailed anatomical information were simulated using the Cardiac-torso (XCAT) digital phantom environment (28). Spherical nodules of various sizes (i.e., 3–10 mm in diameter) were randomly implanted within the lung region of these simulated images. Transverse CT slices that intersect with the center of these spherical nodules were extracted to form the dataset, and a 10-fold cross-validation procedure was implemented to evaluate network hyper-parameterization and generalization.

In the patient study, patient data from the lung image database consortium and image database resource initiative (LIDC–IDRI) (29) were used. The LIDC–IDRI database has 1,018 thoracic CT scans with corresponding nodule information. In this study, CT images with a slice thickness greater than 2.5 mm were excluded, leaving 888 CT images for analysis. Nodule information was marked by four experienced radiologists into one of three groups: (I) no nodule, (II) nodules <3 mm, and (III) nodules ≥3 mm. Among these CT images, 1,186 nodules were considered as positive examples by the criteria of nodule sizes above 3 mm and marked by at least three out of four radiologists. The central transverse CT slices between each nodule’s upper and lower boundary in axial direction were used for 10-fold cross-validation.

The simulated data was generated by digital phantom and the patient data was obtained from a publicly available open-source database, namely LIDC-IDRI database (29). Thus, no ethics approval of an institutional review board or ethics committees was required for this study. The authors acknowledge the National Cancer Institute and the Foundation for the National Institutes of Health, and their critical role in the creation of the free publicly available LIDC/IDRI Database used in this study.

Evaluation method

The average performance among the 10 cross-validation folds was used for evaluation in both the computer simulation study and the patient study. Evaluation metrics included nodule detection accuracy, nodule localization accuracy, and nodule diameter measurement accuracy.

Detection accuracy was assessed based on the sensitivity of nodule identification. Under different nodule score thresholds, nodule detection sensitivities, and the corresponding average false positives (FPs) per image were calculated in the 10 testing folds. Seven sensitivity results (FPs = 1/8, 1/4, 1/2, 1, 2, 4, 8) were reported in two studies. Also, the detection accuracy in the patient study was compared against 9 recently published CAD studies which were developed using the LIDC-IDRI database. The techniques used in these methods included feature-based conventional techniques (30,31), two-dimensional (2D) CNN-based techniques [e.g., regional-based CNN (16), U-NET (32)], and three-dimensional (3D) CNN-based techniques (14,17). Free-receiver response operating characteristic (FROC) curve analysis was used in this comparison study (33). Specifically, detection sensitivities under a wide range of FPs (0<FPs≤8) were acquired by evaluating the proposed method with multiple nodule score thresholds ($b_0\in \left[0.01,0.6\right]$). As for the methods used in the comparison study, 7 sensitivity results covering the same FPs range (FPs = 1/8, 1/4, 1/2, 1, 2, 4, 8) were acquired from their original reports.

Localization accuracy was quantified by central coordinate shifts between the predicted nodule bounding boxes and the ground truth bounding boxes, expressed as the mean value of shifts in x/y direction. Also, 2D histograms were plotted to visualize the spatial deviation in nodule localization.

The standard error was used in evaluating the diameter measurement accuracy,

$$Erro{r}_d=\frac{1}{n}{\displaystyle \sum }_i^n\sqrt{{\left(d_i-{\widehat{d}}_i\right)}^2}$$ [9]

where, $d_i$ was the diameter prediction of i^th nodule, and ${\widehat{d}}_i$ was the corresponding ground truth value. n was the total number of nodules.

Activation maps from the two studies were produced to investigate the performance difference between the simulation database and the patient database. To align with the page space requirements of the manuscript, only activation maps generated by the first 10 convolutional layers were reported.

Results

Table 2 summarizes the nodule detection sensitivity results under different false positives (FPs), per image. In the computer simulation study, the developed CAD method achieved an average sensitivity of 99.5%. In the patient study, the developed CAD method reached high detection sensitivities in high FPs settings (FP =1, 2, 4, 8). Sensitivity results were found to be suboptimal when FP <1.

Table 2. 10-fold cross-validation results of detection sensitivities in the computer simulation study and the patient study.

Study	Sensitivity of false positives (FPs) per image							Averaged sensitivity
	1/8	1/4	1/2	1	2	4	8
Computer simulation study	98.9%	99.3%	99.3%	99.3%	99.5%	100%	100%	99.5%
Patient study	69.0%	79.7%	86.0%	90.0%	93.5%	95.4%	97.7%	87.3%

Comparison results are shown in Figure 2 concerning FROC curve analysis. Here, the FROC curve of the developed CAD method is presented as the blue line. The upper/lower dotted blue curves represent the best/poorest testing performance across the 10-fold cross-validation procedure. In Figure 2A, the developed CAD method demonstrated superiority over 3 conventional CAD methods. Similarly, Figure 2B demonstrates improved performance in the developed method, compared to 2 2D-CNN-based detection methods. Figure 2C reveals comparable results between the developed method and computationally-expensive 3D-CNN-based CAD methods. However, two 3D CNN methods (Pingan and JianPei CAD) have higher sensitivity than the developed method when FP <2.

Figure 2.

FROC curve comparison between the developed and (A) conventional CAD methods, (B) 2D-CNN-based method, (C) 3D-CNN-based method. FROC, free-receiver response operating characteristic; CAD, computer-aided detection; CNN, convolutional neural network.

Table 3 summarizes the central coordinate shifts between the predicted nodule and the ground truth bounding boxes. As shown, the average shifts in the x/y direction were less than 1 mm in both studies. The shift in the x-direction was slightly higher than the shift in the y-direction. Figure 3 illustrates the 2D histogram of central coordinate shifts. No apparent spatial deviation in nodule localization was observed, and most shifts were close to the origin.

Table 3. 10-fold cross-validation results of the coordinate shifts.

Study	Coordinate shifts (mm)
	x	y
Computer simulation study	0.345±0.328	0.267±0.272
Public patient database study	0.736±0.657	0.575±0.608

Figure 3.

2D histograms of the coordinate shifts in (A) the computer simulation study, and (B) the patient study.

Compared to the ground truth, the standard error of diameter measurements was 0.26 mm in the computer-simulated study. The corresponding standard error in the patient study was 0.99 mm. In terms of implementation efficiency, screening a 2D image required 0.07 seconds, when using the developed CAD method.

Figure 4 presents the activation maps in the computer simulation study. Activation slices 1 to 10 represent the results of the first 10 filters in each convolution layer. As indicated by the red arrow (Conv 1 – Activation slice 3), two nodules were implanted in this simulated CT image. Figure 5 demonstrates the activation maps of a patient image. A nodule was indicated by the red arrow (Conv 1 – Activation slice 1).

Figure 4.

Activation maps in the simulation study. Red arrow indicates two nodules were implanted in the simulated CT image.

Figure 5.

Activation maps in the patient study. A nodule was indicated by the red arrow.

Figure 6 presents 2 independent test results from the simulation and the patient study, where ground truth bounding boxes are indicated by the blue squares on the left column. The detected bounding boxes are illustrated in the middle column, and bounding boxes at a magnified scale is indicated by the right column. The nodule scores ($b_0$) are attached to the detected bounding boxes. In the simulation study results presented in Figure 6A, the nodule score threshold was set to 0.2. Three nodules were detected by the developed CAD method using this threshold. Two true positive detections had higher nodule scores (0.9992/0.9768) than false positive detection (0.2074). For the false positive detection, the CAD misinterpreted the descending aorta as nodules. Figure 6B illustrates the patient study results under a threshold value of 0.4, where the developed CAD method is shown to successfully detect a nodule with a high nodule score (0.9666). Although the ground truth of this detection had a speculated perimeter, the developed method defined the boundary of the nodule accurately. In contrast, the false-positive detection was associated with a much lower nodule score (0.4272).

Figure 6.

Computer-aided detection (CAD) method results of two test patients from (A) the computer simulation study, and (B) the patient study.

Discussion

In this study, we developed a CAD method for the detection of pulmonary nodules in diagnostic CT images. The goal of this CAD method was to achieve accurate nodule localization and diameter estimation. Our approach centralized a YOLO v3 CNN design, which, to the best of our knowledge, has not been used for pulmonary nodule detection. One possible reason that has delayed its application is the reported low accuracy in detecting small objects (34). In our study, we approached this problem by reducing the down-sampling scale in the feature extractor to increase the detection accuracy for small nodules. Quantitative evaluation results demonstrated improved performances in nodule localization and diameter measurements. Mainly, the CAD method was able to localize central nodule coordinates in a range of 1-pixel width (<1 mm) without angular dependence for both studies. Also, this CAD method achieved clinically acceptable precision in nodule diameter estimation. The error in diameter measurement was less than 1 mm, which is smaller than the basic dimensional unit (1 mm) in clinical measurement guidelines and inter-/intra- reader variability (1.73/1.32 mm) (35,36).

The reliability of the developed CAD method was evaluated concerning the computer-based ground truth of the computer simulation study. These simulated images could be used as a digital phantom for quality testing and assurance in future nodule detection studies (37). However, the performance of the developed CAD method is slightly different between the simulated database and the patient database (i.e., the computer simulation study achieved a higher detection sensitivity than patient study). The superior performance of the simulation study may result from the simplicity of the images (i.e., noise-free, well-circumscribed simulated nodules, and homogeneous lung tissue).

The performance difference observed between the two studies may be reflected by the activation maps in Figures 4 and 5. As demonstrated in Figure 4, the edges of the patient’s body and multiple internal organs were highlighted in activation slices of the first five columns (conv 1–5). It can be inferred that the filters in the shallow convolutional layers were responsible for the detection of edges in multiple directions. The information inside the lung region was highlighted in the subsequent deep layers. After that, two nodules were detected. Figure 5 indicates that the filters in the patient database study. With the assistance of edge filters in the shadow layers, the information outside the lung region (e.g., heart, bone) is greatly inhibited, while the information within the lung is enhanced. However, the deep filters in the patient study focused on highlighting the bronchus and pulmonary vessels instead of the nodules directly in the simulation study. This difference may be caused by the complexity inside the lung region, which may be associated with complex functional information associated with imaging data (38). Initial extraction of the bronchus and vessel features may be required in the CAD method before excluding their interference for patient nodule detection.

This CAD method is most prominently characterized by its computationally efficient design. The developed CNN structure consists of 19 convolutional layers in the feature extractor, which was primarily reduced, compared with its original algorithm (i.e., 53 convolutional layers) (25). Also, our proposed approach did not require the computationally-intensive false positive reduction procedure compared to other two-stage methods. Computation consumption is reflected by the number of parameters in any CNN model. As such, our method reduced the total number of parameters compared to other CNN approaches (i.e., 3D U-net or 3D Resnet) due to a lower number of layers and 2D convolution operations. A low-performance GPU, such as GTX1060Ti 6GB used in this study, has adequate memory for loading the full set of parameters needed for model training. However, it is noted that a quantitative comparison of parameters was not feasible due to the limited availability of the source code of other CAD methods.

Finally, we note that the improved computational efficiency did not compromise detection sensitivity. As demonstrated in Figure 2, the developed CAD method out-performed all conventional methods and 2D CNN-based methods. It also achieved comparable results at high FPs settings, compared with computationally expensive 3D CNN-based CAD systems. In low FP settings, however, the developed CAD method did not achieve comparable results to the PATECH and JianPei CAD methods. Future works of task-specific developments of the 3D version of the presented method would emphasize a balance between detection accuracy and implementation cost (39). For example, on-board (i.e., when a patient is under treatment), lung nodule detection for lung radiotherapy using a linear accelerator (LINAC) requires rapid implementation from a light-weight CNN design to achieve real-time detection. Clinical practice preference and evaluation would be the guidelines for future development works towards 3D CNN architecture.

While this study presents a novel approach to nodule localization and diameter estimation, it possesses limitations due to the simplicity of phantom images. Simulating more realistic phantom images (i.e., extra noise, morphological variations) will be essential to fully understand the rationale and robustness of the developed CAD method (40). More sophisticated digital phantoms could customize the simulated image database with specific nodule texture, location, size, and density. This way, further investigation of filter preference (i.e., texture, size, density, location) in feature extraction could be conducted by such simulated images.

Conclusions

In this work, a novel deep-learning CAD method was developed for lung nodule detection with improving computational efficiency and reducing false-positive rates. Preliminary results demonstrated that the developed method achieved nodule localization and diameter estimation with sub-millimeter accuracy. With promising nodule detection accuracy and reduced computation power cost, the developed CAD method has an excellent potential for clinical application.

Ethical Statement: The simulated data was generated by digital phantom and the patient data was obtained from a publically available open-source database, namely LIDC-IDRI database (29). Thus, no ethics approval of an institutional review board or ethics committees was required for this study.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.