A novel transfer-learning based physician-level general and subtype classifier for non-small cell lung cancer

Abstract

Confirming histological patterns of lung carcinoma is important for determining the prognosis and the next steps of treatment for a patient. Confirming the histologic patterns (subtype) of lung adenocarcinoma is important for determining the prognosis and treatment options for a patient. The task is challenging, and often requires the input of experienced pathologists, who by themselves lack interobserver concordance. A computer-aided diagnosis holds the potential to accelerate the time to diagnosis. As many adenocarcinoma tissue samples contain multiple histologic patterns, accurate computer-aided diagnosis requires annotations manually labeled by pathologists. We propose a method that merges weak supervised learning and Integrated Learning using Transfer Learning using two datasets: The Cancer Genome Atlas (TCGA), and the Clinical Proteomic Tumor Analysis Consortium (CPTAC) to reduce the need for manual annotation by a pathologist while maintaining accuracy. Whole-slide images (WSI) are first determined to be either adenocarcinoma or squamous cell carcinoma, then further identify the subtypes by generating weak classifiers for each subtype, then using integrated learning to create a strong classifier.

Our model was evaluated with independent datasets from the CPTAC dataset and a dataset from a private hospital. It can achieve AUC values of 0.86, 0.91, 0.82, 0.77, 0.96, 0.98 in Acinar, LPA, Micropapillary, Papillary, Solid, and Normal, respectively.

Lung adenocarcinoma; Adenocarcinoma subtype classification; Transfer learning; Weak supervised learning; The cancer genome atlas, Squamous cell carcinoma, NSCLC.

1. Introduction

Lung cancer is the most common malignant tumor in the world and is a leading of death cancer patients [1, 2]. Lung adenocarcinoma and squamous cell carcinoma are the primary types of non-small cell carcinoma (NSCLC), of which adenocarcinoma accounts for almost half of all cases [3]. The most serious invasive lung adenocarcinoma typically consists of complex mixtures of multiple patterns [4]. In 2015, the World Health Organization released guidelines with five subtype patterns: lepidic, acinar, papillary, micropapillary, and solid with prognostic differences that may be helpful in identifying candidates for adjunctive therapy [4, 5, 6].

Identification of subtle histopathological patterns in complex tissue images under microscope is a time-consuming and subjective process [7, 8]. Due to the complexity and subjectivity of the classification, concordance between different pathologists is low regardless of the image source: microscopic or WSI. Notably, in a study classifying classical and difficult images of lung adenocarcinoma subtypes, the Cohen's kappa (κ) among 26 lung cancer pathologists were 0.77 ± 0.07 for the classical images and 0.38 ± 0.14 for difficult examples [10].

Recent studies have shown that advanced deep learning algorithms can enhance the ability of pathological image analysis across a multitude of tasks, such as discriminating cancer subtypes [12], identifying tumor regions [13], semantic segmentation [14], detecting tumor metastasis [15], mitotic counting [16], and it has also been used across difference species of cancer [17, 18, 19]. With regards to lung cancer, studies [20, 21] in 2017 have shown morphological features of WSI images can be used to predict the prognosis of lung cancer. We believe that artificial intelligence can be used to assist the pathologist—working alongside, rather than replacing. Wei, et al., in their 2019 study, first attempted to automatically classify histological subtypes of lung cancer on stained sections using emerging deep learning techniques, demonstrating that deep learning models could potentially help pathologists improve the classification of lung adenocarcinoma subtypes by automatically screening and highlighting cancer areas, with objective, physician-level results [22].

This study expands the dataset used to include The Cancer Genome Atlas (TCGA) [23], one of the largest publicly available pathological image datasets, and the Clinical Proteomic Tumor Analysis Consortium (CPTAC) [24]. Images for each cancer type are selected and fused weakly supervised learning and integrated learning with transfer learning to create a model that utilizes a voting strategy to determine the classification.

2. Methods

2.1. Overview

The experimental flow of our lung adenocarcinoma subtype classifier is shown in Figure 1.

Figure 1.

ASC 6-class model experimental process. Data processing: The WSIs are first sliced according to the doctor's label results to obtain the tiles of each subtype. After that, the patients are divided into training patients and test patients, and the tiles are divided into training sets and test sets according to the patients. Then, use a resampling strategy to select training patient of sub-model from huge training patients and the rest patient as validation patient which is using to control training time of sub-model (A) Training: Each sub-model is trained using respective training set and validation set, and the result will be fed into bagging model as input (B) Testing: Put test tiles into the trained sub-model and get the predicted subtypes of the sub-models, then the sub-models vote to get the merged result.

First, images for adenocarcinoma and squamous cell carcinoma from the TCGA dataset are used to create a lung carcinoma classifier.

The adenocarcinoma subtype classifier (ASC) is then created using transfer learning from the lung carcinoma classifier model. The ASC consists of 10 different learners trained using the bagging strategy that use a voting strategy to determine the classification between six different subtypes. The data for the ASC comes from the CPTAC dataset, which is manually labelled by pathologists from the First Affiliated Hospital of Xinjiang Medical University.

2.2. Data collection

Two open datasets, the Clinical Proteomic Tumor Analysis Consortium (CPTAC) and the Cancer Genome Atlas (TCGA) are the data sources for this experiment. To reduce the amount of labelling needed to be done by doctors, TCGA lung cancer data is used to establish a classifier for lung adenocarcinoma and squamous cell carcinoma. The pre-training dataset is composed of 822 Lung Adenocarcinoma (LUAD), 751 Lung Squamous Cell Carcinoma (LUSC), and 591 Normal WSIs. This provides pre-training weights for the ASC.

Four pathologists from the First Affiliated Hospital of Xinjiang Medical University labelled specific areas of the CPTAC WSIs for lung cancer subtypes, which were subsequently used for both the training and testing sets of the models. The CPTAC dataset was employed over the TCGA dataset because it contains multiple subtypes of adenocarcinoma. All adenocarcinoma images were annotated with the following subtypes: acinar, adherent, papillary, micropapillary, and solid. A small number of adenocarcinoma WSIs were added to balance the model data, for a total of 113 WSI from 65 patients. 14 WSIs from 8 patients containing all subtypes are selected as testing patients to assess the ability of the model, and the 99 WSIs from the remaining 56 patients are used for training the model.

2.3. Data pre-processing

For the carcinoma classifier model, TCGA WSIs were sliced into 512 × 512 pixel sized tiles at 5x magnification. Then, each tile was given the same label as the WSI that it came from.

For the ASC, the WSI from the CPTAC dataset were manually labelled by three doctors of thoracic surgery from the First Affiliated Hospital of Xinjiang Medical University. Each WSI was annotated by the doctor into what they found to be appropriate subtypes.

First, the annotated areas were sliced into 512 × 512 pixel sized tiles at 5x magnification. The boundary tiles and tiles that are primarily background are removed. To balance the amount of data in each subtype, tiles were sliced with overlap. Thus, some pixels may appear in multiple slices. The number of tiles for each type was normalized to the number of tiles labelled “Solid” – the most abundant type. Tiles were sliced from outside the labelled area, and these were used for the “Normal” label.

Table 1 contains the number of tiles that are used for training and testing of the ASC. We utilize the Bagging method, which involves creating multiple models. The mean and standard deviation represent the distribution of tiles between the ten different models created under bagging. Bagging is described further below.

Table 1.

Mean and Std. Dev of the number of WSI/tiles in training and validation sets used by all learners in each subtype, and the numbers of WSI/tiles of the test sets for each subtype.

Class	WSI					Tiles
	μtrain	σtrain	μvalid	σvalid	test	μtrain	σtrain	μvalid	σvalid	test
Acinar	22	2	12	2	5	773	180	452	180	624
Lepidic	5	0	3	0	2	528	114	261	114	562
Micropapillary	13	2	9	2	4	501	147	333	147	431
Papillary	28	2	15	2	4	823	161	494	161	447
Solid	25	3	14	3	6	959	247	495	247	591
Background	36	3	18	3	14	960	134	403	133	770
Total	99				14	-				3425

2.4. Classifier

DeepPATH (DP) code adapted from Coudray et al. [12] and Deepslide (DS) code adapted from Wei, et al. [22] are used as the lung carcinoma classifier and the adenocarcinoma subtype classifier (ASC). The DeepPATH model is based on inception v3 architecture with initial 5 convolution nodes combined with 2 max pooling operations and followed by 11 stacks of inception modules. It ends with a fully connected and then a softmax output layer. The Deepslide model is based on an 18-layer ResNet using multi-class cross entropy loss function.

2.5. Bootstrap aggregating (bagging)

This study uses Bootstrap aggregating (Bagging) integrated learning to combine weak learners into strong one. Bagging is a method proposed by Breiman to reduce the variance of learning algorithms [25]. Given a model, bagging extracts samples with replacement from the training population several times, then uses the samples extracted each time to build multiple models (weak learners), and finally uses the mean (regression problem) or majority voting (classification problem) method to aggregate the results of the weak learners to get a strong learner [25, 26]. In this study, we utilize the majority voting method.

2.6. Data partitioning

Although overlapping is used when slicing, the number of tiles that we have is still unbalanced. The number of tiles from different patients in each subtype varies greatly, so to ensure diversity of data, an upper limit is set for the number of tiles that can be provided by any single patient. For different patients in each subtype, a resampling strategy is used to select patients for the training and verification sets, so that the number of patients in each weak learner is similar, but the patients are different. The mean and standard deviation of the numbers of WSI and tiles for the training and verification set used by the learners for each subtype are shown in Table 1.

2.7. Model training using transfer learning

Previous research [12] has shown that the Inceptionv3 [27] architecture can distinguish lung adenocarcinoma and squamous cell carcinoma well. In this experiment, the model parameters obtained from the pre-training model are taken as the initialization weights of the six classification model parameters, except the last layer. For the pre-training model we trained for 230,000 steps, and the ASC model adopts early stopping. In this experiment, the loss function is the cross entropy between the prediction probability and the real class label, and the optimization algorithm utilizes RMSPropOptimizer with an initial learning rate of 0.001, weight attenuation of 0.9, momentum of 0.9, and epsilon of 1.0.

2.8. Test process and visualization of results

A comprehensive assessment of the model is carried out at both WSI and patch levels in this study. Since the model is trained at the patch level, the test is also conducted on the patches cut from WSI. The predictive probability of each subtype for each type is determined and the category corresponding to the maximum prediction probability is obtained.

Because each test patch has a true subtype label from the pathologist, this study uses the confusion matrix to evaluate the model prediction results in all subtypes. We use ROC curves to reflect the generalization of the predicted subtypes of the model. At the WSI level, the prediction results of all patches on the same WSI are counted. In order to consider both quantity and probability, we sum the probability of all patches for their given subtype and the result is taken as the criterion of predicting the main and secondary components, which is then compared with the results provided by doctors.

We visualize the histological patterns of lung adenocarcinoma detected on full slide images. They are displayed by covering the slices of WSI with color blocks representing the predicted categories and then generating an overlay for the original WSI image. This visualization method can directly display the prediction results of the model and provide an easy-to-understand reference for doctors. If the doctor's annotation is provided, the prediction results of the model can be evaluated at the same time as shown in Figure 2.

Figure 2.

Heatmap generated from the lung carcinoma classifier (A.i-iv) and ASC imode (B.i-iv). The latter simultaneously shows doctor's annotated curve to compare with the model results.

3. Results

3.1. Accurate prediction of subtype regions

This experiment demonstrates that the proposed method can directly predict the subtype and location of adenocarcinoma in WSI. First, we can judge whether an WSI contains adenocarcinoma or squamous cell carcinoma via our pre-training model. We can then further predict the subtype categories and location by our ASC subtype model if it is adenocarcinoma. The experimental results show that the ASC subtype model can accurately predict the locale of the subtype and the primary subtype category as shown in Figure 2.

On the WSI level, the quantity of the tiles and the predictive results of tiles on a WSI are used to determine the predominant and minor subtypes. The experimental results show that for predominant subtype accuracy of our model can reach 75% accuracy for primary subtypes and 67% accuracy for secondary subtype classification.

3.2. Transfer learning improves model accuracy

Because the training data of the ASC subtype model requires pathologists to annotate and provide ground truth labels, the process is highly time-consuming and laborious. In this experiment, we use the pre-training model without additional annotated data to provide pre-training weights for the ASC subtype model. This helps improve the accuracy of the model and reduces the amount of detailed annotated data required. Table 2 shows that compared to the average weak learners represented by the mean, the strong learners represented by bagging achieve better results. The DP rows are ones using the DeepPATH code adapted from Coudray et al. [12], while DS represents Deepslide code adapted from Wei, et al. [22]. Furthermore, using the pre-training model parameters as the pre-training weight, the average evaluation index of each model is 5%–20% higher than that of the unused model.

Table 2.

Precision, Recall, F1 score, AUC value of each model for training the ASC subype classifier. DP represents the DeepPATH model, DS represents Deepslide model. Rows with mean are the average of each weak learner, and rows with bagging represent the strong learner. Pretrain represents models using the pre-training weights.

Model	Precision	Recall	F1	AUC
DP_mean	0.22	0.29	0.22	0.68
DP_bagging	0.28	0.34	0.26	0.75
DP_pretrain_mean	0.51	0.51	0.48	0.82
DP_pretrain_bagging	0.59	0.55	0.52	0.88
DS_mean	0.55	0.52	0.49	0.84
DS_bagging	0.63	0.56	0.53	0.89
DS_pretrain_mean	0.57	0.55	0.51	0.86
DS_pretrain_bagging	0.66	0.58	0.54	0.91

Figure 3 shows the comparison of ROC curves with and without pre-training weights of the model under different subtypes. For nearly all subtypes, the result of the model with pre-training weights is better, and particularly improved for Lepidic-primary adenocarcinoma. Testing was also carried out on tiles at 20x magnification and 256∗256px size slices, but results were worse than those at 5x magnification and 512 × 512px size slices.

Figure 3.

ROC curves with and without pre-training weight of our model on test set, and AUC value are show in legend.

3.3. Inconsistent annotations

Wei et al. [22] demonstrate that the consistency of annotations between different doctors only achieves a kappa of 0.4 for a group of 3 doctors. We achieved a kappa score between our model and a professional's annotations of 0.43, matching the kappa found by Wei et al..

In order to examine the effect of different doctors' labeling on the experimental results, we respectively selected one doctor's and two doctors' labeling data to carry out the experiment. Figure 4 shows the confusion matrix between the prediction result of each subtype in the test set and corresponding real label. In Figure 4A, the results of the two doctors are consistent with the result found by Wei et al. Figure 4B shows that one doctor's result is more concentrated than using labeled data from two doctors and average accuracy is higher although testing sets are not the same and the former has less training data.

Figure 4.

The confusion matrix between the prediction result and the real label of each subtype in the test set under two doctors' annotation (A), under one doctor's annotation (B).

4. Discussion

The study demonstrates that transfer learning can be helpful to relieve the pressure of model tagging, and that the bagging strategy can be applied to the deep learning model for the classification of lung cancers and subtype recognition for adenocarcinomas. The results suggest that the strong learner result will be better when the weak classifier result is better. Both the present study and Wei et al. [22] show that the result of micropapillary is the worst in all invasive adenocarcinoma subtypes, while micropapillary and solid adenocarcinoma are the most likely pathological subtypes of recurrence, and adherent growth-oriented adenocarcinoma has a lower risk of recurrence. Acinar and papillary adenocarcinoma are the pathological subtypes with moderate recurrence. Therefore, it is necessary to improve the accuracy of microemulsion prediction. As shown in Figure 4B the accuracy of microemulsion is improved, demonstrating the importance of consistent annotation, as adding more pathologists when measuring concordance harms the metric.

Our study grows the number of classes that we can identify to the five subtypes within invasive adenocarcinomas as identified by the IASLC [3], as well as background material. Previous studies into utilizing machine learning for classification on adenocarcinoma slides have generally identified fewer classes, as shown in Table 3, this study is on par with other studies that are classifying the most number of potential classes in the tissue image. Furthermore, this study, along with Wei, et al. [22], are the only two studies that we could find that classified all adenocarcinoma subtypes, an important factor in determining a patient's prognosis.

Table 3.

Abbreviations: ACC, adenocarcinoma; SCC, squamous cell carcinoma; LP, lepidic; AC, acinar; PA, papillary; MP, micropapillary; SO, solid.

Researchers	Year	Objective	ACC Subtype Identified	Method
Gertych, et al. [28]	2019	5-class ACC subtype classification: AC, MP, SO, Cribriform, Non-tumorous	AC, MP, SO, cribriform	Fine-tuned and de-novo CNN
Nishio, et al. [29]	2021	5-class lung tissue classification: normal, emphysema, atypical adenomatous hyperplasia, lepidic pattern of ACC, and invasive ACC	LP	Homology-based
Yang, et al. [30]	2021	6-class lung tissue classification: ACC, SCC, small-cell lung cancer, pulmonary tuberculosis, organizing pneumonia, normal lung	None	DNN
Wei, et al. [22]	2019	6-class ACC WSI primary subtype classification; LP, AC, PA, MP, SO, benign/imperfect sample	LP, AC, PA, MP, SO	DNN
This Paper	–	6-class ACC WSI primary subtype classification; LP, AC, PA, MP, SO, background	LP, AC, PA, MP, SO	Transfer Learning + DNN

We note that the consistency of different doctors’ labeling results is very low, which can signal that the human labelers may not be able to determine what the common characteristics are for a particular subtype. This phenomenon is because various adenocarcinoma subtypes can be histologically divided into different types, and there are some differences in the morphological characteristics of each type. For one, acinar adenocarcinoma can be divided into simple acinar, complex acinar, glandular fusion, sieve arrangement, and the pathological manifestations of papillary adenocarcinoma can be divided into pseudo-clay, moderate papillary size, and different papillary size, and the prognosis of each type is also different. Therefore, we can try to predict the different categories of subtypes directly. Even with less data of each type, the consistency between the same data and the difference between the different data will be higher.

One challenge facing adenocarcinoma subtype classification is achieving high concordance with professionals, as well as establishing a gold standard. Comparing the accuracy of the trained model with one versus two pathologists changes the kappa significantly. Furthermore, even within doctors, concordance is low. Within the dataset used in this study, we found that variance between tiles marked Background and Acinar were higher than the rest. This led to worse performance for these classes, demonstrating the need for higher concordance between just pathologists, not just pathologists and AI models. Thus, we believe that our study ultimately reinforces the need for a new method to compare classification accuracies.

Declarations

Author contribution statement

All authors listed have significantly contributed to the development and the writing of this article.

Funding statement

Dr. Ilyar Sheyhidin was supported by National Key Research and Development Program of China [2017YFC0909903].

Data availability statement

The authors do not have permission to share data.

Declaration of interest's statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.