Optimize Transfer Learning for Lung Diseases in Bronchoscopy Using a New Concept: Sequential Fine-Tuning

Abstract

Bronchoscopy inspection, as a follow-up procedure next to the radiological imaging, plays a key role in the diagnosis and treatment design for lung disease patients. When performing bronchoscopy, doctors have to make a decision immediately whether to perform a biopsy. Because biopsies may cause uncontrollable and life-threatening bleeding of the lung tissue, thus doctors need to be selective with biopsies. In this paper, to help doctors to be more selective on biopsies and provide a second opinion on diagnosis, we propose a computer-aided diagnosis (CAD) system for lung diseases, including cancers and tuberculosis (TB). Based on transfer learning (TL), we propose a novel TL method on the top of DenseNet: sequential fine-tuning (SFT). Compared with traditional fine-tuning (FT) methods, our method achieves the best performance. In a data set of recruited 81 normal cases, 76 TB cases and 277 lung cancer cases, SFT provided an overall accuracy of 82% while other traditional TL methods achieved an accuracy from 70% to 74%. The detection accuracy of SFT for cancers, TB, and normal cases are 87%, 54%, and 91%, respectively. This indicates that the CAD system has the potential to improve lung disease diagnosis accuracy in bronchoscopy and it may be used to be more selective with biopsies.

This paper presents the development and clinical trial of a novel approach to bronchoscopy inspection following radiological imaging. The computer-aided diagnosis system for lung diseases, including cancers and tuberculosis, is a novel transfer learning (TL) method on top of DenseNet: sequential fine-tuning. The new method demonstrated an overall accuracy of 82%, compared to traditional TL methods that are 70% to 74% accurate.

I. Introduction

Lung cancer is also called bronchogenic carcinoma, because about 95% of primary pulmonary cancers originate from bronchial mucosa. Lung cancer is the most deadly cancer, with a five-year survival rate of 18.1% (based on 2007–2013 SEER database [1]). In 2014, it was estimated that there were 527,228 people living with bronchogenic carcinoma in the United States.¹ In China, lung cancer is the most common cancer and the leading cause of cancer death, especially for men in urban areas [2]. There were 546,259 tracheal, bronchus, and lung (TBL) cancer deaths, about one third of the 1,639,646 deaths on a global scale in 2013 [3]. Another serious health problem in developing countries derived from the lung is tuberculosis (TB). China accounts for more than 10% of the global TB burden. Currently, Chinese government aims to suppress the TB prevalence from 390 per 100,000 people to 163 per 100, 000 people and stabilize it by 2050 [4].

Chest X-ray is a cheap and fast imaging device which are commonly used for the diagnosis of lung diseases including pneumonia, tuberculosis, emphysema and cancer. It is particularly useful for emergency case. With a very small dose of radiation, X-ray generates a 2D projection image of chest and lungs. However, due to its limitation in visualizing lung in 3D, it is gradually replaced by chest CT that offers 3D imaging. The chest CT is widely used for lung nodule detection. The downside of chest CT is its relatively higher radiation. However, in developing countries, chest X-ray is still used as a primary tool for tuberculosis screening or diagnosis. With these radiological imaging tools, radiologists are able to diagnose lung diseases or make a referral in a screening situation.

Once patients are suspected to have lung cancer or TB with X-ray or CT, bronchoscopy can be a followed-up from radiological imaging. Bronchoscopy is used as one invasive tool to directly detect diseases since 1960s [5]. Fig. 1 shows an example of normal tissue, TB and lung cancer in bronchoscopy. We can observe that for invasive TB, the lumen surface suffers from inflammatory change with hyperemia, edema and ulceration. Lung adenocarcinomas grow extraluminally and lead to lumen stenosis without affecting mucosal surface of lumen and therefore, the mucosal surface of lumen is relatively smooth. However, squamous lung cancers always form intruding nodules and are difficult to be differentiated from TB granuloma visually. Computational aid is therefore needed in bronchoscopy, especially for lesion discrimination and detection. Accurately targeting disease areas could significantly reduce the biopsy trauma and increase diagnostic accuracy [6].

FIGURE 1.

A bronchoscopy image of a normal (a), TB (b) and cancer (c) case.

One typical computer-aided diagnosis (CAD) technique for bronchoscopy is so called virtual bronchoscopy (VB) [7], [8]. VB is normally created from CT scans and used for guiding the bronchoscopy to locate lesions [9], [10]. Several techniques, such as segmentation [11], [12], registration [13]–[16] and tracking [17]–[19], has been introduced to the VB to facilitate the guiding process. Despite the guiding, VB also improves diagnostic accuracy for peripheral lesions compared with traditional bronchoscopy [9].

Due to the limitation of traditional bronchoscopy in detecting small lesions (e.g. a few cells thick), autofluorescence bronchoscopy (AFB) [20]–[22] and narrow band imaging (NBI) [23] are introduced. These new imaging techniques improved the sensitivity [24] or specificity [23] for the early stage cancer detection. Although AFB and NBI have their advantages for lung cancer diagnosis, the traditional bronchoscopy is still the most used technique in daily clinical routine practice and bronchoscopic biopsy is a cornerstone for the pathological diagnosis. However, bronchoscopic biopsy may cause life-threatening bleeding during the operation. Therefore, it is very necessary to be selective on the bronchoscopic biopsies. To improve the diagnostic accuracy with less trauma in bronchoscopic biopsies, CADs technique can play a role.

Although CADs were widely studied in other medical imaging area [25]–[31], to the best of our knowledge, CAD system has not properly studied for the traditional bronchoscopy yet. In this study, we are the first group to develop a CAD system to classify cancer, TB and normal tissues in traditional bronchoscopy. To boost classification performance, we applied the latest deep learning techniques in this study. To deal with the limited number of labeled data, we are the first to propose a novel transfer learning concept: sequential fine-tuning.

II. Method

A. Convolutional Neural Networks

Convolutional neural networks (CNNs) [32] is a powerful tool for automatically classifying 2D or 3D image patches (input). It usually contains several pairs of a convolution layer and a pooling layer. The intermediate outputs of these layers are fully connected to a multi-layer perception neural network. Recently, new tricks, including dropout [33], batch normalization [34] and resnet block [35], have been introduced. The purpose of dropout is to solve over-fitting that is caused by co-adaptations during training. Batch normalization helps to accelerate the training of deep networks by normalizing activations. It achieves the same accuracy with 14 times fewer training steps [34], and outperformed the original model. Resnet block is proposed in [35], where the authors find that identity shortcut connections and identity after addition activations are important for smoothing information propagation. They also design 1000-layer deep networks and obtain a better classification accuracy. Their novel deep networks are designed for creating a direct path for propagating information through the entire networks instead of within one residual unit and are trained easily in comparison with the original ResNet architecture [36].

B. Using Pre-Trained Networks / Transfer Learning

A big advantage of CNN is that when the number of training images increases, the performance of the networks improves. Training a deep learning model requires a large amount of labeled data. However, in many medical image classification cases, the number of labeled data is limited for training. Transfer learning has been proposed [37] to effectively tackle this problem. Transfer learning literally means that experience gained from one domain can be transfered to other domains. From CNN perspective, it means that the parameters trained on one type of dataset can be reused for a new type of dataset. Practically, the first layers of a trained CNN model could be copied to the first n layers of a new model. The remaining layers of the new model are initialized randomly and trained according to the new task [38]. Afterwards, to train the networks from the new dataset, either we only allow the parameters from the fully connected layers of the networks to be tuned, or we fine-tune more layers or even the whole network layers. It is also possible to keep the first convolutional layer fixed, as this layer is often used for edge extraction which is common for all kinds of problems. Since not all parameters are retrained or trained from scratch, the transfer learning is beneficial to problems with a small labeled dataset which is common in medical imaging field.

Transfer-learning have been applied on two specific CAD problems: thoraco-abdominal lymph node (LN) detection and interstitial lung disease (ILD) classification [39]. The state-of-the-art performance has been achieved on the mediastinal LN detection. This CNN model analysis can be extended to design high performance CAD systems for other medical imaging tasks. In ultrasound, transfer learning is used in fetal ultrasound [40], [41] breast cancer classification [42], prostate cancer detection [43], thyroid nodules classification [44], [45], liver fibrosis classification [46] and abdominal classification [47]. Christodoulidis et al. [48] pre-train networks on six public texture datasets and further fune-tune the network architecture on the lung tissue data. The resulting conversational features are fused with the original knowledge, which is then compressed back to the network. Compared to the same network without using transfer learning, their proposed method improves the performance by 2%. In [49], experiments are conducted to answer the research question “Can the use of pre-trained deep CNNs with sufficient fine-tuning eliminate the need for training a deep CNN from scratch?”. They conclude that deeply fine-tuned CNNs are useful for analyzing medical images and they perform as well as fully trained CNNs. When the training data is limited, the fine-tuned CNNs even outperformed the fully trained ones.

C. Our System

In this work, we took a pre-trained DenseNet as our pre-trained model. Huang et al. [50] propose DenseNet as a network architecture where each layer is directly connected to every other layer in a feed-forward fashion (within each dense block). For each layer, the feature maps of all preceding layers are treated as separate inputs whereas its own feature maps are passed on as inputs to all subsequent layers. In their work, this connectivity pattern yields state-of-the-art accuracies. On the large scale ILSVRC 2012 (ImageNet) dataset, DenseNet achieves a similar accuracy as ResNet by using less than half the amount of parameters and roughly half the number of FLOPs (floating-point operations per second). Fig. 2 shows the architecture of this work. The number 121 corresponds to the number of layers with trainable weights (excluding batch normalization layer). The additional 5 layers include the initial 77 convolutional layer, 3 transitional layers and a fully connected layer.

FIGURE 2.

Demonstration of Densenet 121 used in our system.

Inspired by previous work on transfer learning, in this paper we proposed a novel way of performing transfer learning. Instead of only fine-tuning the fully connected (FC) layers or fine-tuning the whole networks, we made best use of limited data meanwhile also had the power of all layers of the networks. We sequentially fine-tuned a pre-trained networks from FC layers to layers before. For a following epoch or a following set of epochs, we additionally allowed fine-tuning layers prior to layers which were already fine-tuned in previous epochs. For example, we used epochs for training and our networks consisted of layers. For each sequential step, we performed epochs. In total, to train the whole networks, we needed to perform steps of SFT and for each step, we allowed extra * layers to be trained. , and were parameters to set.

Since the percentage of each class was different during training, to eliminate the effect of the unbalanced data, we used weighted-cross-entropy as the cost function to update the parameters of our networks. To obtain the final label or class of each sample, we just assigned the label or class where the corresponding node in the last layer gave the highest likelihood value. For training, we limited the number of epochs () to be 150 and set to be 5. We resized all input images to the size of 224*224 to fit the pre-trained model.

III. Materials

A total of 434 patients who were suspected to have lung diseases by CT/X-ray images were enrolled in this study. All patients were processed at the Department of Respiration in the First Hospital of Changsha City, China, from January 2016 to November 2017. Inclusion criteria included: 1) image-suspected lung/bronchus diseases; 2) aged between 18 and 70 years old; 3) liver, renal and blood test with neutrophil count > 2.0 g/l, Hb > 9 g/l, platelet count > 100 g/l, AST and ALT > 0.5 ULN, TBIL <1.5 ULN, and Cr < 1.0 ULN. The exclusion criteria included: 1) patients with immune-deficiency or organ-transplantation history; 2) patients with severe heart disease or heart abnormalities, such as cardiac infarction, or severe cardiac arrhythmia. This study was approved by the Ethics Committee of the First Hospital of Changsha City. Informed consent was obtained from each patient before study. Basic demographic and clinical information, including age, sex, radiological images and treatment history were recorded.

Before bronchoscopy was performed, patients were prohibited to eat and drink for at least 12 hours. 5–10 minutes before performing bronchoscopy, patients received 2 percent lidocaine (by high pressure pump) plus localized infiltrating anesthesia. Some received additional conscious sedation or general anesthesia. During the operation, flexible biferoptic bronchoscopy (Olympus BF-260) was inserted to nosal cavaty, glottis and bronchus. Computer workstation was configured to receive bronchoscopy images. Once abnormality of suspect was detected visually, the area was captured by a camera from a high-definition television (HDTV) system and saved as JPG or BMP files (319 by 302 pixels).

Pathological test was the gold standard for the diagnosis of malignant/premalignant airway disease. Therefore, specimens from bronchial biopsy were obtained in all cases of this study. Specimens for pathologic diagnosis were obtained from the following ways: brushing from the lesion or bronchial washings, fine needle aspiration biopsy, and forcep biopsy from visible tumor or TB lesions or suspected regions. Histological diagnosis was made by experienced pathologists. Two independent pathologists firstly made their diagnosis individually. If their diagnosis were inconsistent, another arbitrator pathologist would make the decision. Such histological results were used as ground truth of this study. According to pathological confirmation, among recruited 434 patients, 81 cases were diagnosed as healthy, 76 were diagnosed as TB, and 277 were diagnosed as lung cancer patients. Table 1 summarized patient distribution in terms of age and gender.

TABLE 1. Patient Number Statistics.

Male	Female	Age	Age	Age	Age	Age
272	162	35	52	122	160	65

IV. Experiments and Results

A. Experiments

In this work, considering the limited number of samples, in order to obtain an unbiased evaluation of the classification performance, a 2-fold cross-validation was employed to evaluate the performance of our method. Specifically, the input dataset was randomly divided into 2 equal parts, where one part was left for testing, and the other part was split again for training (70%) and validation (30%) to avoid the bias. The best classifier based on the validation set was used for testing. Such a procedure was repeated 2 times with a different part used for testing. We pooled the results from both two parts and evaluated the performance measurements.

Since we aimed to solve a three-class classification problem, the measurements of accuracy (ACC) and confusion matrix were used for the evaluation purpose. We also investigated two-class (binary) classification problems such as abnormal versus normal cases, TB versus cancer cases and non-cancer cases versus cancer cases. The receiver operating characteristic (ROC) analysis and the area under the ROC curve (AUC) were used for the evaluating the two-class classification performance.

B. Results

The overall three-class classification accuracy of the three methods was 73.7%, 70.2% and 82.0% for from fine-tuning all layer together, fine-tuning the fully connected layers and our proposed method (sequential fine-tuning), respectively. Our proposed method achieved the most accurate result. Fig. 3(a) and Fig. 3(b) showed the confusion matrix and the normalized confusion matrix from our proposed method (sequential fine-tuning).

FIGURE 3.

The confusion matrix and the normalized confusion matrix of our proposed method using sequential fine-tuning. (a) Confusion matrix. (b) Normalized confusion matrix.

Fig. 4 showed ROC for the binary classification between normal cases and abnormal cases (TB+cancer) from fine-tuning all layer together, fine-tuning the fully connected layers and our proposed method. The AUC was 0.98, 0.97 and 0.99, respectively.

FIGURE 4.

ROC curve of binary classification of abnormal (TB+cancer, positives) and normal (negatives) with fine-tuning all layers together (orange), only fine-tuning the fully connected layers (red), our proposed method using sequential fine-tuning (blue), respectively.

Fig. 5 showed the ROC for the binary classification between TB cases and cancer cases from fine-tuning all layers together, fine-tuning the fully connected layers and our proposed method. The AUC was 0.73, 0.68 and 0.77, respectively.

FIGURE 5.

ROC curve of binary situations of TB (negatives) and cancer (positives) with fine-tuning all layers together (orange), only fine-tuning the fully connected layers (red), our proposed method using sequential fine-tuning (blue), respectively.

Fig. 6 showed the ROC for the binary situation between non-cancer cases and cancer cases from fine-tuning all layers together, fine-tuning the fully connected layers and our proposed method. The AUC was 0.85, 0.83 and 0.87, respectively.

FIGURE 6.

ROC curve of binary situations of non-cancer (negatives) and cancer (positives) with fine-tuning all layers together (orange), only fine-tuning the fully connected layers (red), our proposed method using sequential fine-tuning (blue), respectively.

Fig. 7 showed examples of misclassified cases. Fig. 7a indicated a cancer case that had pale mucosa and yellow secretion and it was mis-classified as a normal case by CAD, while Fig. 7b indicated a normal case that had smooth red mucosa and was mis-classified as a cancer case by CAD. Fig. 7c. indicated a TB case that had a round nodule with smooth mucosa and was mis-classified as a cancer case by CAD. Mucosa color, secretions and smoothness were important features for cancer and TB discrimination. In cancer cases, the tumor mucosa is pale, rigid and has dirty secretion. The mis-classification in Fig. 7 might be due to a small dataset for training. Larger training set would extract more minor mucosa features to avoid such mis-classification.

FIGURE 7.

A cancer case was misclassified as a normal case (a), a normal case was misclassified as a cancer case (b), a TB case was misclassified as a cancer case(c).

Table 2 summarized different performance measures from different methods. Our proposed methods outperformed other compared methods regarding all measures.

TABLE 2. Performance Measures Included Three-Class Classification Accuracy (ACC), AUC for Problem 1 (P1) of Abnormal Versus Normal Cases, Problem 2 (P2) of Cancer Versus TB Cases and Problem 3 (P3) of Non-Cancer Versus Cancer Cases From Different Methods.

method	ACC	AUC P1	AUC P2	AUC P3
FT all layers	0.74	0.98	0.73	0.85
FT FC layers	0.70	0.97	0.68	0.83
SFT all layers	0.82	0.99	0.77	0.87

V. Conclusion and Discussion

A computer aided diagnosis system was developed for the classification of normal, tuberculosis and lung cancer cases in bronchoscopy. In the system, a deep learning model based on pre-trained DenseNet was applied. Using the sequential fine tuning, our model in combination with 2-fold-cross-validation, obtained a overall accuracy of 82.0% for a dataset of 81 normal cases, 76 tuberculosis cases and 277 lung cancer cases. The detection accuracy for cancers, TB and normal cases were 87%, 54% and 91% respectively. This indicated that the CAD system had potential to improve diagnosis and it also might be used for more selective biopsies. Furthermore, we showed that the performance of the deep-learning model was improved with our proposed sequential fine-tuning.

To our best knowledge, we were the first group to bring up the concept of sequential fine-tuning in deep learning networks and we showed the benefits of using sequential fine-tuning compared to fine-tuning all layers and fine-tuning only fully connected layers. Our explanation was that since the dataset size was relatively small, it was not reasonable to fine-tune a very large set of parameters of the whole networks at the beginning. Therefore, we chose to sequentially and gradually fine-tune more and more layers from a pre-trained model. The other benefits of doing sequential fine-tuning was that instead of fitting data to two sub-models of the DenseNet (a model with no layers fixed and a model with fully connected layers fixed), we fitted our data to more sub-models as sequentially we fixed different sets of layers. By doing so, we had a better chance of finding a good model for the data.

We also investigated our classification power of different binary classification situations. The area under the ROC curve from the binary classification of abnormal cases and normal cases was very high (0.99). From the ROC curve, we could see that we were able to keep the sensitivity of detecting abnormal cases of our CAD system to be 1 while the specificity was 0.65. It meant that our CAD system could identify 65% normal cases without missing any abnormal cases. It had the potential to reduce the false positive rate of doctors and avoid further with biopsies of these normal patients. The area under the ROC curve from the binary classification of TB cases and cancer cases was 0.77 where there was still space to improve. Although the discrimination power was not very high, we could still triage these abnormal patients and almost 10% of TB patients were correctly identified by our CAD system without missing any cancer patients. Again, these patients would not necessarily go for biopsies. For some cases, the CAD system did not perform well. Fig. 7 showed misclassified cases. The TB case was mistakenly classified as a lung cancer case by our CAD system. This TB nodule looked very like a malignant tumor. However, for doctors, there was still one feature for the discrimination: TB surface was more smooth than cancer surface. With a larger training set, more features would be extracted automatically, and this kind of mistakes would be eliminated or suppressed. In this study, TB cases from the training set was small and thus the trained model was not good enough to capture subtle features of difficult cases.

In this study, we investigated neither the actual diagnostic performance of doctors on bronchoscopy images nor the performance of doctors with the aid of our CAD system. In the future, we would conduct a reader study to evaluate the benefits of using our CAD system. Bronchoscopy as an invasive instrument plays a key role in lung disease diagnosis and determining treatment plans for the patients. With bronchoscopy doctors can directly observe the lung tissue and diagnose the problem to some extent. The doctors needs to make a decision whether to biopsy the patients timely when performing bronchoscopy. Although mortality because of the biopeis is only between 0%–0.04% [51], bleeding during diagnostic bronchoscopy occurs in between 0.26% and 5% of cases. In the case of an massive bleeding or an acute massive iatrogenic haemoptysis after biopsy, a life-threatening situation associated with a high mortality rate can develop. Furthermore, avoiding unnecessary biopsies will also reduce the anxiety of patents. With aid of our computer system, doctors can already correctly eliminate 65%normal patients, 10% of TB patients to avoid unnecessary biopsies/risk for patients which is of great help in clinical operation. To further suppress the number of biopsies, in the future, we will investigate the possibility of boosting this CAD system for identifying specific types of lung cancers. That means more labeled data should be collected in the future. The future work may also extend our CAD system in combining with other imaging techniques(e.g. AFB) to cover broader a range of diseases and meanwhile combining deep learning networks together with human crafted features from domain knowledge.

Funding Statement

This work was supported in part by the Hunan Provincial Natural Science Foundation of China, under Grant 2018sk2124, the Health and Family Planning Commission of Hunan Province Foundation under Grant B20180393, and Grant C20180386 and the Changsha City Technology Program under Grant kq1701008.