Interpretable deep learning model to predict lymph node metastasis in early gastric cancer using whole slide images

Abstract

In early gastric cancer (EGC), the presence of lymph node metastasis (LNM) is a crucial factor for determining the treatment options. Endoscopic resection is used for treatment of EGC with minimal risk of LNM. However, owing to the lack of definitive criteria for identifying patients who require additional surgery, some patients undergo unnecessary additional surgery. Considering that histopathologic patterns are significant factor for predicting lymph node metastasis in gastric cancer, we aimed to develop a machine learning algorithm which can predict LNM status using hematoxylin and eosin (H&E)-stained images. The images were obtained from several institutions. Our pipeline comprised two sequential approaches including a feature extractor and a risk classifier. For the feature extractor, a segmentation network (DeepLabV3+) was trained on 243 WSIs across three datasets to differentiate each histological subtype. The risk classifier was trained with XGBoost using 70 morphological features inferred from the trained feature extractor. The trained segmentation network, the feature extractor, achieved high performance, with pixel accuracies of 0.9348 and 0.8939 for the internal and external datasets in patch level, respectively. The risk classifier achieved an overall AUC of 0.75 in predicting LNM status. Remarkably, one of the datasets also showed a promising result with an AUC of 0.92. This is the first multi-institution study to develop machine learning algorithm for predicting LNM status in patients with EGC using H&E-stained histopathology images. Our findings have the potential to improve the selection of patients who require surgery among those with EGC showing high-risk histological features.

Introduction

Gastric cancer is the fifth most common cancer, and third most common cause of cancer-related mortality worldwide. It has a high incidence rate in Central and East Asia [1,2]. In Korea and Japan, which are some of the hotspot countries for gastric cancer, the development of population-based screening programs has enabled early detection and treatment [3,4]. For early gastric cancers (EGC) with a minimal risk of lymph node metastasis (LNM), the preferred treatment is endoscopic curative resection, such as endoscopic mucosal resection or endoscopic submucosal dissection [5,6]. Endoscopic resection is superior to surgical resection for improving patient quality of life and reduce the risks associated with surgery. However, it should be performed only when LNM is absent. If LNM is suspected, additional gastrectomy with lymphadenectomy is necessary [7,8]. The JGCA (Japanese Gastric Cancer Association) guidelines recommend the eCura scoring system to categorize patients at risk after endoscopic resection (e.g., of patients with eCura C-2 have certain LNM risk) [6]. Currently, there is no definite criteria for identifying patients who may require additional surgery after endoscopic resection. Hence, though only about 15% of patients with EGC have LNM, some of these patients undergo unnecessary additional surgery [9,10]. Despite the use of various imaging modalities to aid in deciding the treatment strategy, this situation continues to persist [11-14]. Considering the frequency of LNM in EGC, additional surgical treatment may not be effective, but the potential risk of LNM cannot be disregarded.

Recently, as artificial intelligence has revolutionized the medical field, several studies have been conducted to predict LNM using computational analyses [8,15-21]. Most of these studies are based on radiologic images [16-18], clinicopathologic features [8,19,21], or both [15,20]. Histological imaging provides a higher density of morphological information for the primary tumor than other clinicopathological parameters or radiological images. Considering that the risk of LNM is determined by the primary tumor, it is necessary to apply advanced computational analysis to identify the intricate and diverse morphological patterns (which cannot be seen by naked human eyes) with hematoxylin and eosin (H&E)-stained slides.

Several studies have predicted LNM using computational pathology [22-27]. However, most previous studies have used histological images of advanced cancers, for an understanding of cancer-related information. There is currently only one study [23] that has reported the prediction of LNM in early-stage cancer with relatively limited cancer-related image information. Moreover, no study has employed deep learning techniques using histological images to predict LNM in gastric cancer. In our previous study, we successfully classified histological subtypes of gastric cancer using deep learning [28]. Considering the histopathological patterns as a significant factor for predicting lymph node metastasis in gastric cancer, we aimed to provide an enhanced treatment guide for EGC patients with LNM. To achieve this goal, we obtained H&E-stained histological images from multiple institutions and extracted morphological features from the slides. Subsequently, machine learning was applied to predict the LNM status.

Materials and methods

Study design

We conducted LNM classification using 1,237 whole slide image (WSI) datasets obtained from five different institutions (HGH, Hanyang University Guri Hospital; KBSMC, Kangbuk Samsung Medical Center; SS, Seoul St. Mary’s Hospital; ISH, International St. Mary’s Hospital; KUMC, Korea University Medical Center). Our approach consisted of three main steps: tumor segmentation, tumor morphology feature extraction, and lymph node metastasis classification. The workflow of the proposed method is illustrated in Figure 1.

Figure 1.

Overview of study design. A. Tumor regions on whole slide image were identified and annotated as regions of interest. B. Semantic segmentation was performed into three types: differentiated tumor, undifferentiated tumor, and normal. C. 70 morphologic features (see Supplementary Materials for details) were computed. D. Morphologic features were processed to predict lymph node metastasis using machine learning. E. Lymph node metastasis prediction performance was visualized by ROC curves and confusion matrix.

Data collection and image preprocessing

Ethical approval was obtained from the ethics committees of the respective institutions (HGH, 2020-09-002; KBSMC, 2021-06-030; SS, KC20RISI0329; ISH, IS21TISI0033; KUMC, 2020AN0385). Representative slide sections of surgical specimen from EGC cases were selected and scanned with Aperio, Mirax, Phillips and Hamamatsu at X40 equivalent magnification (0.25 μm per pixel) and digitalized into svs, mrxs, tiff and ndpi format. ASAP (Automated Slide Analysis Platform) was used for pixel-based annotation of differentiated tumors, undifferentiated tumors, and normal tissues. The annotation was performed by two gastrointestinal pathologists (S.A., S.H.L.) and two general pathologists (Y.-N.S., Y.J.L), and in case of disagreement consensus was reached through discussion.

The WSIs obtained from the five different institutions exhibited color differences owing to batch effects. Therefore, we examined the color distribution of patches generated from the WSIs from each institution, and the results are shown in Supplementary Figure 1. Considering these color differences, we divided the internal datasets for tumor segmentation into HGH, KBSMC, and SS, and the external datasets into ISH and KUMC.

A total of 13,538 patches were generated by the five institutions at a magnification of 10×, with a patch size of 512 × 512 pixels. Details of the patches are summarized in Table 1.

Table 1.

Patch-wise data scheme trained for tumor subtype segmentation

	HGH	KBSMC	SS	ISH	KUMC	All
Training set	2,692	2,361	2,156	0	0	7,209
Validation set	770	675	616	0	0	2,061
Test set	384	337	308	1,017	2,222	4,268
Total	3,846	3,373	3,080	1,017	2,222	13,538

Development of multi-class segmentation network for the subtype of gastric cancer

For tumor segmentation, we used SE-ResNeXt101-32x4d as the encoder model in DeepLabV3Plus. The four classes included normal tissues, differentiated tumors, undifferentiated tumors, and background. Due to class imbalance, we employed an imbalance sampler to achieve a 1:1:1 ratio for normal, differentiated, and undifferentiated tumor patches. During training, the batch size was set to 64, and we used Dice loss with a 1e-4 learning rate and the CosineAnnealingLR scheduler. Augmentation techniques were applied for training, including location augmentation (horizontal flip, vertical flip, shift scale rotate), transform augmentation (elastic transform, grid distortion, optical distortion), and color augmentation (random brightness contrast). Before training, normalization was conducted by dividing the patches by the maximum pixel value of 255.

For all 1,237 WSIs, we used a trained tumor segmentation model to infer the tumor masks at the patch-level, and the images were processed using a sliding window approach to obtain slide-level results. For patch-level inference, we compared each patch with the corresponding tissue mask patch, and only performed inferences on patches where the tissue ratio was greater than 0.3. We did not apply morphology operations (e.g., opening or closing) as a postprocessing method to prevent distortion of tumor morphology.

Extraction of interpretable morphologic features

We extracted over 70 morphological features related to differentiated and undifferentiated tumor regions from slide-level masks of the 1,237 WSIs. These features were quantified and analyzed in the total tumor and main tumor regions. The total tumor region includes all the segmentation results, whereas the main tumor region comprises the largest ones. We extracted eight quantitative and five qualitative features related to the characteristics of the tumor area from the main tumor region because we expected that the main tumor regions would exhibit the most distinct qualitative features. Detailed information regarding these features is summarized in Supplementary Table 1.

Development of machine learning model to predict lymph node metastasis using the interpretable morphologic features

We stratified the 1,237 WSIs acquired for LNM classification based on the institution they were collected from, and then split them into training and test datasets in an 8:2 ratio. This ensured that the occurrence of lymph node metastatic cases in each institution was distributed proportionally in both datasets. As a result, the training dataset consisted of 987 WSIs and the test dataset consisted of 250 WSIs. The distribution of slides and lymph node metastasis cases at each institution is shown in Table 2.

Table 2.

Slide-wise data scheme trained for LNM classifier

	Train set		Test set		Total
	LNM (-)	LNM (+)	LNM (-)	LNM (+)
HGH	202	29	51	7	289
KBSMC	196	16	50	4	266
SS	161	18	41	4	224
ISH	48	4	13	1	66
KUMC	280	33	71	8	392
Total	887	100	226	24	1,237

Results

Robust deep learning network for tumor subtype segmentation on internal and external datasets

The inferences for differentiated/undifferentiated tumors and normal tissues are shown in Figure 2 (slide-wise) and Supplementary Figure 2 (patch-wise). The performance of the segmentation network at the patch and slide levels is presented in Table 3. The mean pixel accuracy of the external cohort, which was not used for training, was as high as 0.8939 and 0.9877 at the patch and slide levels, respectively. The performance of the segmentation network for differentiated and undifferentiated tumors is shown in Supplementary Table 2. Overall, the segmentation performance for differentiated tumors was significantly better than that for undifferentiated tumors, likely because of the higher number of patches available for differentiated tumors. The slide-level inference results for both tumor types were low; however, this was because the areas occupied by differentiated and undifferentiated tumors were very small on the entire slide.

Figure 2.

Results of segmentation network for tumor subtype in slide level. H&E-stained slides (A) alongside corresponding visualizations of tissue-type ground truth (B) and prediction (C) in external dataset. Slide regions are classified into three tissue types: differentiated tumor (dark orange), undifferentiated tumor (brown), or normal (light orange).

Table 3.

Performance of tumor subtype segmentation across path-wise and slide-wise

Datasets	mIoU	F1-Score	mPA	Recall	Precision
Patch - internal (1,029)	0.77	0.8696	0.9348	0.8694	0.8698
Patch - external (3,239)	0.6551	0.7878	0.8939	0.7875	0.7881
Slide - external (39)	0.6152	0.6701	0.9877	0.7803	0.6663

mPA = mean Pixel Accuracy.

Introspection of segmentation network by visualization of the learned representation

To verify that the encoder of the segmentation model was well-trained, the latent vectors of the patch-level test datasets through the encoder were mapped to UMAP (Uniform Manifold approximation and Projection for visualization) [29]. Figure 3A shows a scatter plot of UMAP embedding colored according to the tissue type. Representative patches with similar features, according to the encoder, were clustered closely in distinct regions, proving the discriminant performance of the model. Among the undifferentiated tumors, some appeared to be integrated into the differentiated tumor cluster. Those outside of the distribution patches were reviewed by pathologists and confirmed to be differentiated tumors (Figure 3B).

Figure 3.

Visualization of representation learning and actual segmentation results for differentiated tumor area of UMAP. A. Uniform Manifold Approximation and Projection (UMAP) visualization showing disentangled representation in differentiated/undifferentiated tumor and normal tissue. B. H&E-stained slides alongside corresponding visualizations of tissue-type ground truth and prediction (differentiated tumor, dark orange; undifferentiated tumor, brown, MMP=1).

Interpretable morphologic features can predict lymph node metastasis in early gastric cancer

The basic characteristics according to each cohort are shown in Supplementary Tables 3 and 4. And Table 4 shows the results of lymph node metastasis prediction using XGBoost. The model was trained using 70 morphological features obtained from 987 masks and tested on 250 cases. The feature importance plot used in this model is shown in Supplementary Figure 3. Of the 70 morphologic features included in the model, the top 5 ranked features were ‘filled area of total tumor’, ‘area of total tumor’, ‘equivalent diameter of total tumor’, ‘perimeter of total tumor’, and ‘perimeter of undifferentiated’ in sequential order. The overall AUC approached 0.75, whereas ISH, SS, and HGH achieved AUC above 0.8. Remarkably, despite not being used in segmentation training, the ISH showed a notable AUC of 0.92. However, the performance of KBSMC external dataset was slightly lower at approximately 0.65. The ROC curves and confusion matrix for each institution and the overall institution are shown in Figure 4 and Supplementary Figure 4. The results of the same detailed statistical analyses using SGD and LGB are presented in Supplementary Tables 5 and 6, respectively. The corresponding ROC curves are shown in Supplementary Figures 5 and 6.

Table 4.

Performance of XGBoost for predicting lymph node metastasis

	HGH	KBSMC	SS	ISH	KUMC	Total
AUC (± CI)	0.8011 (± 0.0058)	0.6450 (± 0.0202)	0.8293 (± 0.0064)	0.9231 (± nan)	0.7007 (± 0.0050)	0.7487 (± 0.0021)
Accuracy	0.59	0.87	0.73	0.86	0.62	0.7
Weighted Precision	0.87	0.85	0.9	0.95	0.86	0.87
Weighted Recall	0.59	0.87	0.73	0.86	0.62	0.7
Weighted F1-Score	0.66	0.86	0.79	0.89	0.7	0.76

Figure 4.

The ROC curve for predicting LNM using XGBoost.

Clinicopathologic analysis of false negative results

Nine false-negative slides were obtained from the XGBoost model. Pathologists reviewed the pathology reports and inference masks. In five of the nine false-negative cases, poorly differentiated tumor cells were scattered diffusely as individual cells in the stroma (Supplementary Figure 7). In those cases, although the analysis was performed using a representative section, the segmentation model often predicted suboptimal results. This could be interpreted as a limitation of data labeling in supervised learning because the annotation is not pixel-based. In addition, there were two cases in which the images used for the analysis were not representative. Although the initial pathological diagnosis indicated the depth of SM3 invasion or the presence of lymphovascular invasion, the corresponding area was not included in the image used for analysis (Supplementary Figure 8).

Discussion

Predicting lymph node metastasis status in patients with EGC is very important for determining appropriate treatment. In this study, we developed an LNM prediction model that is morphologically interpretable using whole-slide images of primary tumors. The model yielded satisfactory predictions with an AUC of 0.8 for the internal dataset. Although its performance was reduced to an AUC of 0.65 in one of the external datasets, it was as high as 0.92 in another external dataset. These variations suggest that the model learns by extracting biologically relevant features from HE images rather than random correlations between artifacts and LNM in the training and test cohorts. Various studies have attempted to predict LNM from primary tumor histology using deep learning algorithms in other cancers including the colorectal [23,26], bladder [24], prostate [25], breast [30], and melanoma [22]. However, to the best of our knowledge, this is the first study to demonstrate the potential of deep learning to predict LNM status from routine H&E-stained slides of primary EGC.

Current guidelines for the endoscopic treatment of EGC include histological differentiation, tumor size, depth of invasion, and the presence of an ulcer. This indicates that a patient’s prognosis and the likelihood of lymph node metastasis, can be influenced by the histological phenotype of the cancer. Several studies have shown that an undifferentiated histological pattern is a risk factor for LNM [31-35]. Based on the assumption that the ratio and distribution pattern of tumor differentiation in the slide images might affect LNM, we annotated tumors into differentiated and undifferentiated subtypes. Subsequently, a multiclass segmentation network was developed using the annotated image, and a LNM prediction model was developed by extracting morphological features from each area of the tumor.

This paper proposes a novel approach for developing a prediction model. Unlike previous studies on other organs that used scoring heat maps for LNM prediction [24-26,30], we utilized morphological features extracted from differentiated, undifferentiated, and combined tumor areas. Consequently, it was possible to inversely estimate the crucial features used in the prediction model. Specifically, among the 70 morphological features incorporated into the model, more of the features associated with the undifferentiated tumor area (perimeter, number of regions, and filled area) were included in the top 10 features as opposed to the differentiated tumor area. Furthermore, the fact that features representing the quantity of a tumor have been identified as meaningful indicators has biological plausibility considering previous studies showing that undifferentiated histologic type is a risk factor for LNM.

This study was performed in a large multicenter cohort, and the generalizability of the model was confirmed through external validation. Furthermore, the model demonstrated remarkable prediction accuracy (AUC of 0.8), even in the absence of clinical or radiological information other than the histological features of the primary tumor. This is comparable to the AUC value of 0.836 obtained by the modified eCura system [36]. This is important because our method overcomes the limitations of the classical scoring system, which has limited accuracy and consistency for variable measurement.

However, this study has some limitations. Firstly, as a multimodal study, the AUC for predicting LNM differed between cohort (Figure 4). Specifically, the ISH cohort exhibited a high AUC of 0.92. Besides differences in tumor size and submucosal invasion among the basic characteristics (Supplementary Table 5), the confusion matrix (Supplementary Figure 4) suggests that the proportion effect may have influenced the results due to the smaller number of true LNM cases in the ISH cohort. Comparing the color distribution (Supplementary Figure 1) between the cohort with a relatively larger number of cases and a high AUC (SS: 0.83) and the cohorts with lower AUCs (KUMC: 0.7, KBSMC: 0.64), it is evident that the SS cohort has a much more favorable color distribution. This indicates that data disparity due to color distribution is a crucial factor affecting LNM prediction results, highlighting the need to overcome data disparity through the pre-training strategy for robust feature extraction [37]. Secondly, although a good accuracy for predicting LNM was achieved using the deep learning model, there is a drawback inherent to supervised learning stemming from the labor-intensive nature of the image annotation process. In addition, it has constraints in identifying new image biomarkers because it relies on predefined targets. To overcome the limitations of supervised learning, automatic feature extraction from images should be attempted using a recent medical foundation model trained on unlabeled data [38,39]. Although pathological images containing high density information were used, a comprehensive analysis requires multimodal data that integrates AI with various data types, including histopathological images, tabular, and textual information from pathology reports. This comprehensive approach will enhance the model’s accuracy and robustness [40,41]. To achieve this, cross-attention model, which has been shown to effectively integrate multimodal data for improved performance in predictive tasks is necessary [42-44]. Additionally, model lighting and extensive external validation are required to ensure feasibility in clinical practice.

Conclusion

In conclusion, we developed a prediction model for identifying patients who might be at a higher risk of developing LNM in EGC based on digital H&E slides of primary gastric cancer. The model is not subject to intra- or inter-observer variations, which enables its application in various institutions. Ultimately, this model could aid in reducing the overtreatment of patients with EGC who do not need surgery, while ensuring appropriate treatment for those who need surgery.

Disclosure of conflict of interest

None.