Predicting Unnecessary Nodule Biopsies from a Small, Unbalanced, and Pathologically Proven Dataset by Transfer Learning

Abstract

This study explores an automatic diagnosis method to predict unnecessary nodule biopsy from a small, unbalanced, and pathologically proven database. The automatic diagnosis method is based on a convolutional neural network (CNN) model. Because of the small and unbalanced samples, the presented method aims to improve the transfer learning capability via the VGG16 architecture and optimize the related transfer learning parameters. For comparison purpose, a traditional machine learning method is implemented, which extracts the texture features and classifies the features by support vector machine (SVM). The database includes 68 biopsied nodules, 16 are pathologically proven benign and the remaining 52 are malignant. To consider the volumetric data by the CNN model, each image slice from each nodule volume is selected randomly until all image slices of each nodule are utilized. The leave-one-out and 10-folder cross validations are applied to train and test the randomly selected 68 image slices (one image slice from one nodule) in each experiment, respectively. The averages over all the experimental outcomes are the final results. The experiments revealed that the features from both the medical and the natural images share the similarity of focusing on simpler and less-abstract objects, leading to the conclusion that not the more the transfer convolutional layers, the better the classification results. Transfer learning from other larger datasets can supply additional information to small and unbalanced datasets to improve the classification performance. The presented method has shown the potential to adapt CNN architecture to improve the prediction of unnecessary nodule biopsy from small, unbalanced, and pathologically proven volumetric dataset.

Introduction

As a multitude of survey data and conclusions published in the whole world, lung cancer has been the most threatening cancer to human health [1–5]. And early detection would increase the survival rates of lung cancer patients significantly. For the last 30 years, advancement in radiology brought a transformative tool to a noninvasive detection of the cancer in early stage. Valuable information can be derived from the radiological volumetric images, so that radiologist experts could detect and even predict the cancer at an early stage. While significant progress has been made, challenge of desiring as early as possible in detecting the cancer remains. This challenge has been motivating great research effort to develop sophisticated computer algorithms to extract as much valuable information as possible from the volumetric images to achieve the goal of detecting the cancer as early as possible. A brief review of the past effort for the goal of detecting early lung cancer is given below.

The detection of early lung cancer includes two steps: In the first step, the task is to detect the pulmonary nodules (PNs), which have been recognized as the precursor of the lung cancer. So far, the developed computer algorithms for PN detection have achieved very high sensitivity. However, among the detected PNs, more than 95% are benign [6]. Identifying the malignant from the benign nodules and characterizing the identified malignant nodules become the task of the second step for the goal of detecting early lung cancer. In the following presentation, we will focus on the second step, also called computer-aided diagnosis (CADx).

In the initial stage of developing computer algorithms to assist experts to perform the task of differentiating malignant from benign nodules, the efforts were devoted to calculate the features, described by the experts, from the images, and then design adequate classifiers on the features for the differentiation task. There are host of representative papers published during this initial stage period. For example, Iwano et al. [7] calculated the shape of PNs for CADx; Saito et al. [8] and El-Baz et al. [9, 10] described both two-dimensional (2D) and three-dimensional (3D) characters of surface morphology of PNs for CADx; Yankelevitz and Kostis et al. [11, 12] classified small nodules according to the growth-rates estimation. Although these efforts considered the experts’ experiences as a priori knowledge, the extracted features only included very few special characters which are limited by human’s recognitions.

Later on, researchers found that the radiological images might contain much more useful information than those that could be described by the experts. Therefore, more quantitative image features were described by mathematics and calculated by computer technologies beyond the prior knowledge, such as the texture features described by the co-occurrence matrices or filters [13], histograms of oriented gradients (HOG) [14], local binary patterns (LBP) [15], and other methods [16, 17]. Then the features and the associated labels were trained by a classification module, such as support vector machine (SVM) [18], decision trees [19], and random forest [20]. This later effort could be seemed as the second stage of developing CADx. However, as multiple features have been extracted and the number of features increased significantly, then how to determine the contributions of the individual features for classification and how to combine them as integrated features are becoming extremely arduous problems. Although these computed features may contain more useful information that could not be described by experts’ prior experience, these features are still artificial features in computer vision, and they are picked up in a more or less manual manner. We cannot guarantee these features are the right/proper features for the CADx task in different images/dataset and different situations.

Recently, with the development and successful applications of machine deep learning techniques, the image features could be analyzed and categorized by the computer techniques automatically for an optimal solution. This could be seemed as the third stage of developing CADx. For early lung cancer diagnosis, one widely cited example of the machine deep learning techniques, called convolutional neural network (CNN) [21], has shown encouraging classification results on PNs as compared to other conventional classification methods, such as the deep belief network (DBN) [22], the stacked denoising auto-encoder (SDAE) [23], and the hand-tuned features [24, 25]. But till now, they all generated good results on the publicly available datasets, such as LIDC-IDRI and LUNA [26, 27]. These observations may be understood as follows. Although these public datasets have great amount of useful information, few of them has the pathological biopsy reports for ground truth on the nodule diagnosis, instead the radiologist experts’ visual assessment on the nodules is treated as the ground truth. Thus, even though the CNN classifications could achieve high accuracies, it can only demonstrate that the deep learning for nodule malignancy classifications could reach the best doctors’ level of visual diagnosis on images, as the conventional classification methods try to achieve. Unfortunately, as we all know, the expert doctor’s visual diagnosis is not the golden standard. Therefore, experiments on the nodules with pathological diagnosis are exceedingly indispensable.

However, it is truly challenging to get enough data with the pathological diagnosis (or biopsy reports) to satisfy the need of large volume of training data for the deep learning networks. Therefore, confronted with this problem, transfer learning strategy emerges as an option to relieve the challenge with encouraging outcome. For example, Shin et al. [28] have demonstrated that transfer learning from pre-trained ImageNet (via fine-tuning) could be useful for the computer-aided detection (CADe) of lung nodules and the CADx of interstitial lung diseases. Hosny et al. [29, 30] have tested transfer learning and AlexNet on small datasets for skin lesions and obtained a significant improvement. To the best of our knowledge, there are few reports (if not all) for the malignant diagnosis of lung nodules published until now. Therefore, in this paper, we aim to explore whether and how deep learning with transfer learning techniques could be useful to the malignant diagnosis of a small number of pulmonary nodules with pathologically proven reports.

As previously mentioned in [21], CNN is most suitable for image recognitions. Driving by different ideas and applications, now there are several architectures built on CNN, such as AlexNet, VGG, GoogleNet, and ResNet, all are widely cited architectures [31–34]. Which one could be selected for our preliminary study in this paper has been considered gravely. Our own dataset with biopsy reports only contains 68 lung nodules (16 benign and 52 malignant proved by pathological diagnosis on biopsied nodule specimens) whose diameters are from 9.1 mm to 130.8 mm (mean size of 31.5 mm). With consideration of the spacing of the 2D CT (computed tomography) images of 1.25 mm, this means that the smallest nodule may occupy only 8×8 pixels or less in the CT images. To study the features of these small nodules, a CNN architecture containing smaller convolution kernels and less convolution layers might be the best choice. The reasons are firstly, smaller receptive fields mean fewer parameters; secondly, more activation functions followed by smaller convolution kernel could bring more local features and increase the ability of discrimination; finally, the nodule images are too small to improve the performance of the CNN architecture by adding more convolution layers. Therefore, we selected the VGG16 architecture in our experiments and considered very small 3×3 receptive fields and performed convolution with stride 1 pixel.

In order to apply the VGG16 architecture on a small dataset, transfer learning strategy was adapted for our datasets with the pathological biopsy ground truth. More concretely, we adapted the parameters trained by the public database containing a large number of natural images – ImageNet as the initial features to initialize the VGG16 architecture. Then we add part of our pulmonary nodule dataset (labeled with the ground truth) into the initial features (as the training dataset) to fine-tune the VGG16 architecture parameters. To evaluate the efficiency of this transfer learning strategy, we compared the classification results from the presented deep transfer learning with a typical conventional method which extracts the gray-level co-occurrence texture features and classifies the texture features by SVM [16]. Both leave-one-out and 10-fold validation experiments were performed to evaluate the robust of the classification results, which makes the conclusion more stable and credible.

The main contributions of this paper could be summarized to three points below. First, we designed a deep transfer learning method on a small and unbalanced CT image dataset of the PNs and obtained higher efficiency than the traditional machine learning method for malignant classifications. Second, we validated that more transfer convolutional layers for the small and unbalanced data would not guarantee better classification results. Third, all the CT images of PNs used in this paper were scanned by automatic exposure low-radiation-dose protocol, all of the subjects were recruited randomly, and all the PNs have the pathological diagnosis as the ground truth labels.

The rest of this paper is organized as follows. Section 2 describes the main steps for deep learning training and transfer learning, and the details of our adaption of machine deep learning for the task of classifying PNs on a small dataset with pathological ground truth. The experimental design and algorithm implementation are then presented in Section 3. The comparisons of the classification results from the presented deep transfer learning method and the traditional machine learning method are reported in Section 4. Finally, the contributions of the current discoveries and the plan of future research work are discussed in Section 5 and concluded in Section 6.

Methods

In this study, we adopt the VGG16 architecture to explore the potential of classifying PN malignancy and compare it to a traditional machine learning method. From the limited amount of clinical data we can gather with pathological ground truth, we further adopt the transfer learning strategy to fine-tune the parameters trained by public ImageNet and investigate the different performances of different layers of the VGG16 architecture. The flowchart of the deep transfer learning method is shown in Fig. 1.

Fig. 1.

The flowchart of deep transfer learning method

VGG16 Architectures Improved by Different Transfer Layers

VGG [32] is the pronoun of very deep convolutional networks, named by Visual Geometry Group who proposed the concept. The main idea is to improve the convolutional network performance through increasing the depth of architecture design with very small (3 × 3) convolution filters in all layers. For illustration purpose, a typical VGG16 architecture was constructed by five different stacks of convolutional layers, with each stack followed by a max-pooling layer. After the last max-pooling layer, three full-connected layers (the first two each has 4096 channels and the third one has 1000 channels) and one soft-max layer were attached. According to the different combinations of convolutional layers, the architecture was divided into six ConvNet configurations, named by A to E in order. As many studies referred that the configuration D (viz, VGG16 as abbreviation in public) showed the best performance on most types of medical data, thus VGG16 was selected as the basic study architecture of the deep learning network in this paper.

Since the purpose of this paper is intended to explore an efficient deep transfer learning method for classification of a small dataset of PNs, thus, the choice of transfer learning architecture types imitates the classical configurations as proposed in reference [32]: A (11 transfer weight layers), B (13 transfer weight layers), and D (16 transfer weight layers) of the original ConvNet configurations, as well as the simplest configuration (8 transfer weight layers) named by S hereafter. All these four transfer learning architectures of VGG16 named by S, A, B, and D are shown in Fig. 2 and will be described in the following sections.

Fig. 2.

The transfer learning architectures of VGG16 named by S, A, B, and D

Transfer Learning

Transfer learning is an ability of a machine learning system, which is similar to the concept that people can solve new problems faster or with better solutions by the knowledge learned previously [35]. The aim of transfer learning strategy is trying to transfer the knowledge from the previous tasks to a target task when the latter has fewer high-quality training data. Obviously, there should exist some explicit or implicit relationships between the features of the previous and target tasks, thus the transfer learning strategy could be used.

By mathematics, suppose χ is a feature space containing {x₁, x₂, …, x_n}, the corresponding marginal probability distribution is P(X), and then a domain D = {χ, P(X)} can be given. A task T should be performed based on that D consists of two components, a label space L and a predictive function f(·) denoted by

$$T=\left\{L,f\left(·\right)\right\}·$$ 1

If defining a source domain D_S and the corresponding learning task T_S, a target domain D_T (D_T ≠ D_S) and learning task T_T (T_T ≠ T_S), transfer learning is used to help improve the learning efficiency of the target predictive function f_T(·) in T_T with the knowledge from D_S and T_S. More details on transfer learning are interpreted in the paper of Pan and Yang [35]. Because it is extremely challenging to obtain large volume medical data with annotations for deep learning techniques to train, and in order to learn more efficient features by CNN architectures, transfer learning was proposed as an option to relieve the challenge. How many layers should be transferred is according to the features from different levels of abstraction. More theoretical studies and experiments will be needed for future works.

Our Training Protocol with Transfer Learning

The hypothesis of this study is that the CNN trained by the well-labeled natural images from ImageNet could be useful for the recognition of medical images. To test the hypothesis, we train a classification model by a small medical image dataset of PNs with the prior knowledge acquired from the labeled natural images of ImageNet. With the limitation of our computing resource (Intel Core i7–6700@3.40GHz quad-core, Dell 0Y7WYT, 8GB Hynix DDR4 213 MHz, and Nvidia GeForce GTX 10502GB), we could not train a database as large as ImageNet. To circumvent this limitation, the public parameters of the convolutional layers of the VGG16 network are downloaded from the online resources (http://download.tensorflow.org /models/vgg_16_2016_08_28.tar.gz). Then a method for normalizing initialization [36] is applied to the downloaded parameters to set the parameters of the full-connected layers of the VGG16 architecture in order to ensure that the output and input have as similar probability distribution as possible. For example, the Gaussian random initialization method with mean 0 and variance 0.01 was used for the parameter setting. We set the data of each dimension on the same scale in order to (1) avoid the parameters trained by the model being biased toward some dimensions, and also to (2) accelerate the gradient descent algorithm converge during the multiple updating steps. In other words, our goal is to prevent the gradient explosion or disappearance of the output from losing gradient of the activation function in the forward propagation of the deep neural network. The reason is that if the gradient is too big or too small, the network would take very long time or need lot computing power to converge. Our software architecture runs in Python 3.5 environment. The deep learning framework is based on TensorFlow. The python libraries in the deep learning framework include TensorFlow 1.0, Numpy, Math, Matplotlib, Pandas, and PlIL.

To optimize the training process, we used the well-known gradient descent algorithm for the CNN model. In addition, the dropout method proposed in [37] is also adopted to avoid “overfitting” by randomly omitting some feature detectors. Although the dropout method has been shown more efficient on large datasets than small ones, based on our comparison experiments, it has better performance than the batch normalization method [38] on our small dataset of PNs. The comparison tests of the above two methods on 68 cases (45 cases for training and 23 cases for testing) for ten classifications are shown in Table 1. Based on the accuracies of these ten classifications, we can conclude that dropout method is superior to the batch normalization for our data. Therefore, we set the dropout functions in the first and second full-connected layers to avoid “overfitting.” At last stage of VGG16 architectures of Fig. 2, the SoftMax program is used as a classifier for the CNN model to divide the nodules into two classes of benign and malignant. Based on the concept of transfer learning, part of the pathologically labeled nodule images is used to fine-tune the parameters of the CNN model so that the CNN model would be more adaptable to the classification of the PNs. Then the optimized CNN model by the fine-tune training is used to test the rest part of the PNs.

Table 1.

Comparison tests of dropout and batch normalization methods on 68 nodule cases

	Dropout	Batch normalization
1	0.782609	0.695652
2	0.826087	0.695652
3	0.869565	0.652174
4	0.869565	0.695652
5	0.869565	0.73913
6	0.826087	0.608696
7	0.913043	0.695652
8	0.826087	0.695652
9	0.869565	0.608696
10	0.826087	0.739130
Average	0.847826	0.682609
Max	0.913043	0.739130
Min	0.782609	0.608696

Although it is shown that the parameters trained by the ImageNet data could be transferred for lung cancer study in [28], the question still remains open which layer features are the efficient ones for PN classification. In other words, whether transferring different layer features could affect the efficiency of PN classification. Inspired by the multiple architectures of VGG network and our experiments, we transferred the parameters of 8, 11, 13, and 16 layers from the initial ones trained by ImageNet by the use of our pathologically labeled PN image samples. In general, transfer learning strategy usually starts on the features from the bottom, middle, or top of the network [39]. As described in the paper [35], the transferability of features from each layer of a neural network could be quantified, but the performance degradation occurs frequently during the transferring at the higher layers. So far, there is no study showing how the performance changes as the repeated convolutional operations vary while retaining the same parameters. Because of these complexities in transferring learning, we follow a logic principle of decreasing the complexities as much as possible for each group of convolutional layers before reaching each max-pooling layer by only transferring the features among different combinations of the convolutional layers. The width of the convolutional layers is started from 64 in the first group of the convolutional layers up to 512 in the last group.

Experiments Design and Implementation

As mentioned above, the purpose of this study is to explore a more efficient method by computer machine learning techniques to distinguish the malignant and benign PNs from a limited volume CT images with pathological reports on the PNs. Thus, the VGG16 architecture plus transfer learning strategy is adapted as a novel computer machine learning technique for the task of CADx of the PNs. To evaluate the efficiency of the proposed technique, the classification experiments are designed and carried out as follows. The flowchart of the main experiments is shown in Fig. 3.

Fig. 3.

The flowchart of the main experiments

Data Preparations

Total of 68 patients, who were scheduled for PN biopsy by medical reasons, were recruited to this study under informed consent after approval by the Institutional Review Board. The average age of the patients is 69.5 which is ranging from 33 to 91 years old, and 52% is male and 48% is female. Each patient underwent a routine clinical CT scanning for needle preparation. The CT images were obtained by a clinical routine protocol of tube voltage of 120kVp and modulated tube current or mAs level with automatic low-exposure dose control. The thickness of the original CT slices in our dataset varies from 1.00 to 1.25 mm, depending on the coverage of axial length. The border of each nodule in the routine patient CT images was drawn by an experienced radiologist. The routing CT image, the drawn nodule border, and the biopsy pathological report of each PN were collected for the CNN-based machine learning classification of PNs. The smallest PN contains 3 slices, the largest one contains 49 slices, and the average is approximately 16 slices. The diameters of the nodules in these 68 datasets are ranged from 9.1 mm to 130.8 mm (mean size of 31.5 mm). These patients have nodule description on their CT reports, where nodule size is given. Therefore, the range of the nodules size occupies from 10 × 10 to 145 × 145 pixels (average of 35 × 35 pixels) according to the mean distance 0.9 mm of the neighbor pixels on x/y direction. The average number of nodule image pixels 35 × 35 was calculated by the summation of all the 68 nodules’ pixels and then division by 68 (which is the total number of nodules). To satisfy the unity size requirement for the input images of the CNN architecture, an image of 48 × 48 pixels containing the entire nodule area as the central part was extracted from the routing CT scan of each patient. The center of the to-be-extracted rectangle area would be the center of the candidate nodule region. Thus, cutting images would ensure that the main area of each nodule is centered in the extracted rectangle area.

In addition, the CNN architecture (i.e., the VGG16 here) adapted by this study is implemented in TensorFlow. In order to utilize the above architecture designed for color images and the parameters transferred from the ImageNet, the extracted nodule CT images should be transformed from gray to colored. After normalization of the whole data set, the color images were generated by the picked colors corresponding to the distribution of the entire range of gray images. The color-transforming operations of three examples are shown in Fig. 4.

Fig. 4.

The color transforming operations of three examples

To keep the overall consistency of the datasets and follow the rule of natural distribution, we performed a preprocessing operation on the color images by scaling the whole image sets to satisfy the normal distribution with the mean value being zero and the standard deviation being a unit norm.

VGGNet Training and Transfer Learning

As stated above, the VGG16 Network was implemented in the TensorFlow framework, and the configurations corresponding to 8, 11, 13, and 16 transfer convolutional layers were tested and compared. All the parameters of the CNN model were initialized with the ones trained by the public large-scale ImageNet datasets of nature images with 1000 classes, which were trained for 200 epochs with the mini-batch size of 16 image instances after initialization. The other hyper-parameters of the CNN model are the learning rate: 0.001 without decreasing; the convergence: 1000 steps; and the omitted probability: 0.5. The parameters of the CNN model were fine-tuned by the training with our small dataset from the 68 PNs on NVidia Quadro P2000 GPUs.

To discover the performance contribution from different model layers, we used a visualization method [40]. Figure 5 shows the feature maps of each convolutional layer extracted from two examples containing one benign and one malignant nodule. In order to make the observations clearer, we zoomed each feature map to the same size.

Fig. 5.

Visualization of different convolutional layer (cov-layer) feature maps of the example nodules (one malignant and one benign)

Classification and Evaluation Experiments

Because the VGG16 implemented in this study is based on 2D images, and each nodule usually contains a different number of slices, thus, it is necessary to consider how to select the image slices of each nodule for training and testing to avoid information bias. In our dataset, the smallest nodule contains 3 slices, the largest one contains 49 slices, and the average slice number of each nodule is 16 slices approximately. Therefore, we make a principle for image selection that only one slice is selected randomly from each nodule for every training and testing experiment. Thus, no matter how big or small of one nodule, the weight of each nodule will be the same. To get robust results, we had hundred random selections to form the training and testing sets. After the slice selection and dataset preparation, leave-one-out and 10-fold cross validation methods were used to measure the efficiency of the VGG16 architecture plus transfer learning strategy separately. Then the total number of training and testing experiments of leave-one-out method is 68 × 100 = 6800. Considering the possible effects brought by the large bias of 16 negative (benign) and 52 positive (malignant) samples of our dataset, we also added the classification experiments on a dataset formed by 16 negative and 8 positive samples (which were selected randomly) for comparison. For the leave-one-out method, we selected one slice from one nodule randomly to do the verification each time and repeated 50 times to get the best hyper-parameters. For the 10-fold cross validation experiments, our training, verification, and testing sets are divided into 6:2:2.

Traditional Classification on Texture Features as Baseline for Comparison

For comparison purpose, we implemented a traditional machine learning method as a baseline reference. This traditional method uses the SVM classifier to train and test statistical texture features of the 68 pathologically proven PNs. The 2D and 3D texture features were calculated from the gray-tone spatial-dependence co-occurrence matrices (GTSDMs) of four, five, and thirteen directions for the center image and volume data [13, 16]. The features were sent to the SVM classifier for training and testing. The training and testing sets were formed randomly by half of all the nodules respectively for each time. And we also randomized the training and testing process for 100 times. This traditional classification method was tested on the large public LIDC-IDRI database, and the outcome was excellent as reported in our prior study [16].

Results

Three receiver operating characteristics (ROC) curves of the traditional SVM classifications on the GTSDM texture features calculated from the 4, 5, and 13 directions of 68 nodules were drawn in Fig. 6. The mean and standard deviation values of the area under the average ROC curve (AUC) measures for 100 training and testing times are shown in Table 2. Each time, half of the dataset was selected randomly as the training set, and the other half was used as the testing set. The average accuracies based on the texture features calculated from the GTSDMs of 4, 5, and 13 directions are all 76.47%.

Fig. 6.

The ROC curves of the three types of texture features from 4, 5, and 13 directions

Table 2.

The AUC information about the performances of the three types of texture features

	4 Directions	5 Directions	13 Directions
Mean	0.5492	0.5462	0.5971
Standard deviation	0.0722	0.0551	0.0369

The outcomes from the experiments of evaluating the presented machine deep transfer learning of 68 nodules are shown by Fig. 7 and reported in Table 3, where Fig. 7 shows the average ROC curves of 100 random selections for leave-one-out training and testing and Table 3 lists the mean and standard deviation (SD) values of the AUC measures. The average accuracies of 8, 11, 13, and 16 transfer learning layers are 85.32%, 85.76%, 71.13%, and 70.69%, respectively.

Fig. 7.

The ROC curves of the three types of deep transfer learning layers

Table 3.

The AUC information about the performances of the four transfer layers

	8 Layers	11 Layers	13 Layers	16 Layers
Mean	0.7712	0.7435	0.7375	0.6762
Standard deviation	0.0591	0.0834	0.0776	0.1216

The confusion matrices for the 68 nodules (16 benign and 52 malignance) classified by VGG16 with different transfer convolutional layers are shown in Table 4. The statistical numbers of prediction nodules are the sum of ten classification experiments of 10-fold cross validation. For each classification experiment, there are totally 7 test samples (5 malignant and 2 benign).

Table 4.

Confusion matrices for 68 nodules (52 malignant and 16 benign) by VGG16 with different transfer layers

Actual	Prediction				Prediction
	8 Layers	Benign	Malignant	Actual	11 Layers	Benign	Malignant
	Benign	16	4		Benign	16	4
	Malignant	14	36		Malignant	9	41
Actual	Prediction				Prediction
	13 Layers	Benign	Malignant	Actual	16 Layers	Benign	Malignant
	Benign	10	10		Benign	12	8
	Malignant	11	39		Malignant	12	38

The classification measures of VGG16 with different transfer convolutional layers on the 68 nodules are shown in Table 5.

Table 5.

The classification measures of VGG16 with different transfer convolutional layers on 68 nodules

	Sensitivity	Specificity	Precision	F1-Score	Accuracy
8 Layers	0.5333	0.9000	0.8000	0.6400	0.7428
11 Layers	0.6400	0.9111	0.8000	0.7111	0.8142
13 Layers	0.4762	0.7959	0.5000	0.4878	0.7000
16 Layers	0.5000	0.8261	0.6000	0.5455	0.7142

To further investigate the performance of the presented classification method on different distributions of benign and malignant nodules in our dataset, a distribution of 16 benign and 8 malignant nodules was selected randomly from the 68 nodules (16 benign and 52 malignant) and classified by the presented transfer learning VGG16 architectures. We did 10-fold cross validation for 5 times, and over-fitting occurred for 2 times. Finally, we counted the average of the results of three classifications when eliminating the two over-fittings. The confusion matrices for the subset 24 nodules classified by VGG16 with different transfer convolutional layers are shown in Table 6. The statistical numbers of prediction nodules are the sum of ten classification experiments of 10-fold cross validation. For each classification experiment, there are totally 3 test samples (1 malignant and 2 benign).

Table 6.

Confusion matrices for 24 nodules (8 malignant and 16 benign) by VGG16 with different transfer layers

Actual	Prediction				Prediction
	8 Layers	Benign	Malignant	Actual	11 Layers	Benign	Malignant
	Benign	18	2		Benign	19	1
	Malignant	3	7		Malignant	2	8
Actual	Prediction				Prediction
	13 Layers	Benign	Malignant	Actual	16 Layers	Benign	Malignant
	Benign	18	2		Benign	19	1
	Malignant	4	6		Malignant	5	5

The classification measures of VGG16 with different transfer convolutional layers on the subset 24 nodules are shown in Table 7.

Table 7.

The classification measures of VGG16 with different transfer convolutional layers on 24 nodules

	Sensitivity	Specificity	Precision	F1-Score	Accuracy
8 Layers	0.8571	0.7778	0.9000	0.8780	0.8333
11 Layers	0.9048	0.8889	0.9500	0.9268	0.9000
13 Layers	0.8182	0.7500	0.9000	0.8571	0.8000
16 Layers	0.7917	0.8333	0.9500	0.8636	0.8000

It is interesting to see the different classification performances from the benign/malignant ratio (16:8) of the subset 24 nodules and the benign/malignant ratio (16:52) of the original 68 nodules. The sensitivity and specificity go different directions when the ratio goes different directions. This interesting observation indicates that both the sample size and ratio may affect the prediction outcome, because both contribute to the parameters fitting in the CNN model. Despite the different performances from different ratios, the presented transfer learning shows the benefit to the CNN performance.

Discussion

The performance of the adapted CNN model with the transfer learning strategy was investigated in this study for differentiating pulmonary nodules from a small sample size (68 nodules) and unbalanced dataset (16 benign and 52 malignant) with pathological reports as the ground truth. Because of the small sample size of PNs, we adopted the parameters from successful machine learning of the large natural image dataset – the ImageNet as the initial parameters, and tested the feasibility of transfer learning from the ImageNet to our small PN medical images based on a popular CNN model – the VGG16 architecture. We adopted the best architecture with 16 layers (as reported in references [31]) and initialized the parameters the presented machine deep learning architecture by the training from the ImageNet database, followed by refining or optimizing the parameters of the presented architecture with transferring learning on our PN dataset. From the experimental outcomes, we found that the VGG16 architecture with the presented transfer learning strategy is efficient for classification of small, unbalanced, and pathological-proven benign and malignant nodule dataset from routine medical CT images in terms of the ROC curves, AUC measures, confusion matrices, and other classification measures, such as accuracy, sensitivity, specificity, precision, and F1-score. The experiment results showed improvement over the traditional machine learning on the GTSDM texture features.

As mentioned in the introduction section above, there are multiple designs of the VGG16 architecture, which can be divided by the width of the convolutional layers, such as 64, 128, 256, and 512. According to these different designs, there are different repeated convolutional layers before each max-pool layer. Each of these multiple designs will encounter the difficulty of determining the corresponding complexity degree of the transfer learning features at different levels. To avoid this difficulty, we took the logic of simplifying the transfer learning features by reducing the initial parameters of repeated convolutional layers. Then we compared the accuracy, sensitivity, specificity, F1-score, precision, the average ROC curves, and the AUC values by evaluating different transfer learning options. By the experiments, we found that not the more the repeated convolutional layers, the better the classification performance. The best performance to which the transfer convolutional layers can achieve may be relying on the dataset, so it would be a learning process to find the adequate transfer convolutional layers for a particular application.

To get more robust and reliable evaluation results, we performed a large number of experiments for high statistical power. All of the CT image slices covering the entire volume of each nodule were included in the random selections, and each image slice of each nodule was selected with equal possibilities for each model training and testing. This selection and combination of training and testing sets were repeated for 100 times. Thus, we obtained 100 combinations of training and testing images of 68 nodules in total. For each combination, we used the leave-one-out validation method to evaluate the performance of the presented VGG16 with transfer learning the parameters of different convolutional layers for the nodule classification. It is expected that these experiments would take very long time. It is worthy to see that the results are very robust. We also did 10-fold cross validation experiments on the original 68 (52 malignant and 16 benign) and a subset of 24 (8 malignant and 16 benign) nodules, respectively, for comparison of different malignant rates of the datasets. The classification results were shown by confusion matrices and five classification measures, which can also give eloquent proof of the good performance of the transfer learning VGG16 scheme proposed in this paper. It is interesting to see the different classification performances from the benign/malignant ratio (16:8) of the subset 24 nodules and the benign/malignant ratio (16:52) of the original 68 nodules. This interesting observation indicates that both the sample size and ratio may affect the prediction outcome, because both contribute to the parameters fitting in the CNN model. Despite the different performances from different ratios, the presented transfer learning shows the benefit to the CNN performance.

Conclusion

In summary, we have performed experiments to prove that the CNN model can benefit from initialization with features learnt from natural images for the task of distinguishing benign and malignant PNs in medical CT images even the dataset is very small and unbalanced. The benefit was quantified by the use of the VGG16 architecture with initial feature learning from the large public nature image database of ImageNet and then transfer learning from a small and unbalanced medical image dataset. The classification performance for predicting the unnecessary biopsies of 16 benign nodules from the small medical image dataset of pathologically proven 68 nodules (including 52 malignant) is satisfactorily with noticeable improvement over the traditional machine learning method. In addition, the satisfactory performance on the benign/malignant ratio (16:52) was also seen for a different benign/malignant ratio (16:8). Therefore, we can have a conclusion that the CNN model with transfer learning has the ability to classify small and unbalanced medical image dataset with the potential to outperform the traditional machine learning method. Furthermore, the experiments revealed that the features between the medical and the natural images share the similarity of focusing on the simpler and less-abstract objects, leading to the experimental outcomes of not the more repeated convolutional layers, the better the classification results. Because of the limitation on data and computing resource, the questions on how the present method would work on larger datasets and how the complexity of the convolutional transfer learning would affect the classification performance remain open and will be our future research topics.

Funding Information

This work was supported by the National Natural Science Foundation of China under Grant No. 81671773, 61672146, and 61802055, the Fundamental Research Funds for the Central Universities of China under grant N151903001, N171902001 and N171903003, and Natural Science Foundation of Liaoning Province of China 20180550182. Zhengrong Liang was supported by NIH Grant #CA206171 of the National Cancer Institute, USA.