Deep learning quantified mucus-tumor ratio predicting survival of patients with colorectal cancer using whole-slide images

Abstract

Background

In colorectal cancer (CRC), mucinous adenocarcinoma differs from other adenocarcinomas in gene-phenotype, morphology, and prognosis. However, mucinous components are present in a large number of adenocarcinomas, and the prognostic value of mucus proportion has not been investigated. Artificial intelligence provides a way to quantify mucus proportion on whole-slide images (WSIs) accurately. We aimed to quantify mucus proportion by deep learning and further investigate its prognostic value in two CRC patient cohorts.

Methods

Deep learning was used to segment WSIs stained with hematoxylin and eosin. Mucus-tumor ratio (MTR) was defined as the proportion of mucinous component in the tumor area. A training cohort (N = 419) and a validation cohort (N = 315) were used to evaluate the prognostic value of MTR. Survival analysis was performed using the Cox proportional hazard model.

Result

Patients were stratified to mucus-low and mucus-high groups, with 24.1% as the threshold. In the training cohort, patients with mucus-high had unfavorable outcomes (hazard ratio for high vs. low 1.88, 95% confidence interval 1.18–2.99, P = 0.008), with 5-year overall survival rates of 54.8% and 73.7% in mucus-high and mucus-low groups, respectively. The results were confirmed in the validation cohort (2.09, 1.21–3.60, 0.008; 62.8% vs. 79.8%). The prognostic value of MTR was maintained in multivariate analysis for both cohorts.

Conclusion

The deep learning quantified MTR was an independent prognostic factor in CRC. With the advantages of advanced efficiency and high consistency, our method is suitable for clinical application and promotes precision medicine development.

Introduction

In colorectal cancer (CRC), extra-cellular mucinous components are present in a large number of adenocarcinomas.¹ Molecular evidence suggests that over-expression of MUC2 could form a mucous layer that protects against antitumor immune factors, and that absence of MUC5AC expression can promote tumor development.², ³ Previous studies have focused on qualitative analysis of mucinous adenocarcinoma, which is defined as the extra-cellular mucinous component that accounts for more than half of the whole tumor volume.⁴ Several studies reported that mucinous adenocarcinoma has a much shorter overall survival (OS) than non-mucinous adenocarcinoma in CRC.^5–7 However, there is still a lack of quantification of precise mucous proportion and evaluation of its prognostic value in CRC.

The tumor microenvironment (TME) consists of complex components, including tumor epithelium, immune cell, stroma, and mucus.⁸, ⁹ The interaction between TME components, from cellular to tissue levels, plays a crucial role in prognosis of patients with CRC.^9–12 For example, the tumor-stroma ratio (TSR) is associated with survival of patients with CRC.¹³, ¹⁴ Patients with high stroma had worse prognoses than those with low stroma. Immune TME (iTME) biomarkers, including tertiary lymphoid structure (TLS), Immunoscore, and tumor-infiltrating lymphocytes (TILs), have been demonstrated to be essential supplements for tumor node metastasis (TNM) staging.¹¹, ¹⁵, ¹⁶ Other invasive front markers, such as perineural invasion (PN), tumor budding (TB), and poorly differentiated clusters (PDCs), are also well-studied prognostic factors in CRC.¹⁷ We hypothesized that the mucus proportion, or the mucus-tumor ratio (MTR), is associated with CRC prognosis.

Artificial intelligence has a critical role in medical image analysis, which has advanced development of precision medicine.^18–22 We used deep learning technology in a previous study of nine decomposed tissue types on whole-slide images (WSIs) stained with hematoxylin and eosin (HE).¹³ The results showed that the quantified TSR was a strong predictor for OS of CRC. With the advantages of full automation, advanced efficiency, and high consistency, the MTR can be calculated precisely by deep learning. Therefore, in this study, we aimed to quantify the MTR via deep learning and further investigate its prognostic value in two CRC cohorts.

Methods

Patients

Two CRC patient cohorts were respectively enrolled in this study. Patients from Guangdong Provincial People's Hospital served as the training cohort, and patients from Yunnan Cancer Hospital served as the validation cohort. This retrospective study was approved by both institutional review boards, and informed consent was waived. The institutional medical record databases were analyzed to identify patients with histologically confirmed CRC who underwent surgical resection with curative intent. Exclusion criteria were as follows: (1) neoadjuvant therapy; (2) death within 1 month after surgery; (3) clinical information missing; (4) HE stained WSI missing or poor image quality. Only stage II-III patients were included in the TMR analysis, as treatment decisions regarding these two patient groups are of more clinical interest.²³ Clinicopathological information, including age, sex, T-category, N-category, TNM stage, and tumor site, was collected from medical records. OS was the outcome of interest in this study. Follow-up information was last updated in December 2019.

Tissue type segmentation on WSIs

A formalin-fixed, paraffin-embedded (FFPE) tissue section at the most invasive part of the primary tumor was selected for HE staining. The HE-stained slides were scanned as WSIs at 40× magnification. The origin scanned WSIs (40× magnification) were scaled to 20× magnification for segmentation, as digital pathology images with this resolution can provide sufficient information to distinguish different tissue types.⁸, ²⁴ Following our previous definition, eight CRC tissue types and one slide background were determined: tumor epithelium (TUM), stroma (STR), mucus (MUC), debris (DEB), normal mucosa (NOR), smooth muscle (MUS), lymphocytes (LYM), adipose (ADI), and background (BAC).¹³

The fully automatic CRC tissue segmentation was performed using a convolutional neural network (CNN) model released in our previous work (doi: 10.5281/zenodo.4023999). The CNN model was trained using 283k image tiles, and achieved 97% nine-category classification accuracy in an independent test dataset. The HE-stained WSIs were cropped into small overlapped image tiles with a fixed size (224 pixels × 224 pixels) sliding window. These image tiles were then input into the CNN model. Each tile was given a classification probability, and the tissue type with the maximal probability was assigned as the final tissue label. After arranging the classified tissue labels as to where they belong, a rough segmentation map was obtained (Fig. 1A).

Figure 1.

Study workflow. (A) The HE-stained WSI was segmented using a deep learning model to generate a segmentation map containing eight tissue types and one slide background. (B) The mucus-tumor ratio (MTR) calculation process. The mucus and tumor epithelium areas were calculated from the segmentation map. The MTR was defined as the mucus proportion in the sum of the mucus and tumor epithelium areas. ADI, adipose; BAC, background; DEB, debris; HE, hematoxylin and eosin; LYM, lymphocyte aggregates; MUC, mucus; MUS, muscle; NOR, normal mucosa; STR, stroma; TUM, tumor epithelium; WSI, whole-slide image.

MTR calculation process

Mucus and tumor epithelium areas can be easily obtained from the segmented WSI (Fig. 1B, left, middle). Following the definition of TSR, we define the MTR as the mucus proportion in the sum of mucus and tumor epithelium areas (Fig. 1B, right). This definition ensures that MTR values range from 0 to 100%. According to the definition, interferences from other components (such as stroma, necrosis, etc.) were excluded.

Statistical analysis and software

The fully automatic tissue segmentation was performed in MATLAB environment (R2019a, MathWorks, USA). Maximally selected rank statistics from the ‘maxstat’ R package were used to find the threshold binarizing MTR in the training cohort.²⁵ Kaplan–Meier curves of OS were plotted to show the difference in survival rates between patient groups, and P values were calculated via the log-rank test. For both continuous and categorized MTRs, the Cox proportional hazard model was used for univariate and multivariate analysis. Backward step-wise selection was used in multivariate analysis to determine the independent predictors. Harrell's C-index was used to measure the discrimination ability of a predictor or a model. We used R language to perform statistical analyses (www.r-project.org). The reported statistical significance levels were all two-sided, with statistical significance set at 0.05.

Results

Patients

In total, 734 patients were included in the study. Among them, 419 patients (252 males and 167 females; mean age 63.05 ± 12.52 years) from Guangdong Provincial People's Hospital formed the training cohort, and 315 patients (188 males and 127 females; mean age 57.89 ± 12.97 years) from Yunnan Cancer Hospital formed the validation cohort. The median follow-up time (interquartile range, IQR) was 64 (53, 95) months. The clinicopathologic characteristics distribution between the two cohorts are summarized in Table 1. There were no significant differences between the two cohorts, except for age, T-category, and N-category.

Table 1.

Distributions of demographic and clinicopathologic characteristics of patients with colorectal cancer in the two cohorts.

	Training cohort	Validation cohort	P
Age			<0.001
≤ 60 years	157 (37.5%)	180 (57.1%)
> 60 years	262 (62.5%)	135 (42.9%)
Sex			0.960
Male	252 (60.1%)	188 (59.7%)
Female	167 (39.9%)	127 (40.3%)
T-category			<0.001
T1	2 (0.5%)	0 (0%)
T2	17 (4.1%)	0 (0%)
T3	355 (84.7%)	261 (82.9%)
T4	45 (10.7%)	54 (17.1%)
N-category			0.039
N0	190 (45.3%)	141 (44.8%)
N1	141 (33.7%)	117 (37.1%)
N2	88 (21.0%)	57 (18.1%)
Stage			0.951
II	191 (45.6%)	142 (45.1%)
III	228 (54.4%)	173 (54.9%)
Tumor site			0.098
Colon	245 (58.5%)	164 (52.1%)
Rectum	174 (41.5%)	151 (47.9%)

Note: P values were obtained using the χ² test.

Prognostic value of MTR

For continuous MTR, we found that more mucus was associated with a worse OS in the training cohort (hazard ratio [HR] 2.56, 95% confidence interval [CI] 1.15–5.67, P = 0.021). The same trend was observed in the validation cohort (4.34, 1.62–11.6, 0.003).

Using 24.1% as the threshold, which was determined in the training cohort, patients were stratified into mucus-low and mucus-high groups (Fig. 2A and B). In the training cohort, 42 (10.0%) patients were classified as the mucus-high group. In the validation cohort, 43 (13.7%) patients were classified as the mucus-high group. Compared with mucus-low, patients with mucus-high had unfavorable outcomes in the training cohort (HR for high vs. low 1.88, 95% CI 1.18–2.99, P = 0.008; Fig. 2C). Five-year survival rates of mucus-low and mucus-high groups were 73.7% and 54.8% in the training cohort, respectively. The results were confirmed in the validation cohort (2.09, 1.21–3.60, 0.008; 79.8% vs. 62.8%; Fig. 2D).

Figure 2.

HE-stained WSIs and corresponding segmentation maps for mucus-high (A) and mucus-low (B). Kaplan–Meier survival curves for mucus-high vs. mucus-low groups in training (C) and validation (D) cohorts. ADI, adipose; BAC, background; DEB, debris; HE, hematoxylin and eosin; LYM, lymphocyte aggregates; MTR, mucus-tumor ratio; MUC, mucus; MUS, muscle; NOR, normal mucosa; STR, stroma; TUM, tumor epithelium; WSIs, whole-slide images.

Combined analysis of MTR and TSR

As our previous study had shown that the deep learning quantified TSR was an independent predictor for OS,¹³ we speculated that combination analysis of TSR and MTR could provide more accurate prognosis information for patients with CRC. According to the combination of MTR and TSR, patients were stratified into four subgroups: mucus-low & stroma-low (L/L), mucus-low & stroma-high (L/H), mucus-high & stroma-low (H/L), and mucus-high & stroma-high (H/H). Survival curves of the four groups are plotted in Fig.   3A. As only 23 patients were included in the H/L group, we merged the H/L and L/H groups as one group (H/L or L/H group, Fig. 3B). A plot of Kaplan–Meier curves shows that patients with L/L have the most favorable OS, and those with H/H have the most unfavorable OS (log-rank test, P < 0.001; Fig. 3B).

Figure 3.

Combined analysis of MTR and TSR. (A) Kaplan–Meier survival curves for four groups, namely mucus-low & stroma-low (L/L), mucus-low & stroma-high (L/H), mucus-high & stroma-low (H/L), and mucus-high & stroma-high (H/H). (B) Kaplan–Meier survival curves for three groups (H/L and L/H groups were merged as one group: H/L or L/H group). MTR, mucus-tumor ratio; TSR, tumor-stroma ratio.

MTR as an independent prognostic factor

We performed univariate and multivariate analyses on MTR, TSR, and other clinicopathologic factors. We identified age, TNM stage, TSR, and MTR as independent predictors for OS (Table 2, all P < 0.05). In both cohorts, MTR was independent with TNM stage and TSR (training cohort: adjusted HR for high vs. low 1.65, 95% CI 1.01–2.71, P = 0.047; validation cohort: 1.96, 1.12–3.42, 0.018).

Table 2.

Uni- and multivariate analyses of MTR, TSR, and other clinicopathological variables.

	Univariate analysis				Multivariate analysis
	Training cohort		Validation cohort		Training cohort		Validation cohort
	HR (95% CI)	P	HR (95% CI)	P	HR (95% CI)	P	HR (95% CI)	P
Age	1.03 (1.01–1.04)	<0.001	1.01 (0.99–1.03)	0.184	1.02 (1.01–1.04)	<0.001	1.02 (1.00–1.04)	0.049
Sex
Male	1
Female	0.84 (0.60–1.18)	0.317	1.44 (0.90–2.28)	0.125
Stage
II	1		1		1		1
III	2.67 (1.85–3.84)	<0.001	3.10 (1.80–5.35)	<0.001	2.73 (1.90–3.94)	<0.001	3.04 (1.76–5.25)	<0.001
Tumor site
Colon	1		1
Rectum	1.07 (0.77–1.48)	0.704	1.12 (0.71–1.78)	0.625
TSR
Stroma-low	1		1		1		1
Stroma-high	1.72 (1.23–2.41)	0.001	2.17 (1.32–3.57)	0.002	1.62 (1.14–2.32)	0.008	1.99 (1.20–3.28)	0.007
MTR
Mucus-low	1		1		1		1
Mucus-high	1.88 (1.18–2.99)	0.008	2.09 (1.21–3.60)	0.008	1.65 (1.01–2.71)	0.047	1.96 (1.12–3.42)	0.018

Abbreviations: HR, hazard ratio; CI, confidence interval; TSR, tumor-stroma ratio; MTR, mucus-tumor ratio.

To evaluate the added prognostic value of MTR, we developed two Cox models. The model with MTR was established by the independent factors in the multivariate analysis (Table 2); the model without MTR was built with these factors except MTR. C-indexes were calculated for both cohorts and also for two tumor site (colon/rectum) groups. We found that the model with MTR has a higher C-index than the model without MTR in the training cohort, but the performance improvement was not statistically significant (C-index: 0.6824 vs. 0.6819, P = 0.448). In the validation cohort, a similar result was observed (0.6968 vs. 0.6910, 0.226). However, when performing stratified analysis in the rectal cancer group, we found that the model with MTR has better discrimination ability than the model without MTR (0.6723 vs. 0.6600, 0.020).

Discussion

With deep learning technology, we present a fully automatic workflow to assess the MTR on HE-stained WSIs for patients with stage II-III CRC. The tissue sections examined were taken from the most invasive part of the primary tumor. The precise proportion of mucus in the tumor was calculated, saving pathologists' workload and increasing assessment consistency. We found that the MTR was independent of the TNM stage and other clinicopathologic factors. Our automatic workflow is well suited to clinical deployment because of its efficiency, accuracy, and simplicity.

As one of the adenocarcinoma subtypes, mucinous adenocarcinoma is characterized by plentiful extra-cellular mucinous components (> 50%) in the tumor volume.¹, ⁷ However, mucinous components are present in a large number of CRC, and the prognostic value of mucus proportion has not been investigated. To fully automatically evaluate mucus proportion, we used the CNN model to quantify various CRC tissue components on WSIs. We defined the MTR to measure the proportion of mucinous components in the tumor area. The MTR calculation process does not require human participation. Our method significantly improves efficiency and precision compared with the traditional method for evaluating mucinous proportion.¹, ⁴ Instead of merely using 50% as the threshold for qualitative analysis, our method can calculate the exact proportion of mucus components (between 0 and 100%) and exclude the interference of other components (such as necrosis).

We found that the continuous MTR was associated with OS in the two cohorts, and more mucus trended to a worse outcome. Mucus is a complex hydrogel that forms a barrier to infiltration of immune cells and some drugs, explaining our results.³, ²⁶ A mucus-low group and a mucus-high group were stratified using 24.1% as the cutoff, differing from the cutoff of 50% to classify mucinous adenocarcinoma and non-mucinous adenocarcinoma. We evaluate the mucus proportion in the most invasive part of the primary tumor section,²³ while mucinous adenocarcinoma was examined in the whole tumor volume.⁴ As other TME-related invasive front markers (such as TSR, TLS, PDCs, etc.) have been shown to be associated with the prognosis of CRC, the prognostic value of MTR on the invasive frontier of the tumor is worth exploring. Our results suggest that the mucus proportion quantified in the invasive boundary can be used for risk stratification in patients with CRC. The results indicate that mucus-high group patients have a much lower 5-year survival rate than the mucus-low group (58.8% vs. 76.3%), which is consistent with previous studies of mucinous adenocarcinoma.⁷, ²⁷ Multivariate analysis confirmed MTR as a prognostic factor independent of the TNM stage. Together with the TNM stage, TMR can more accurately stratify patients with CRC into different risk groups.

Other CRC prognostic factors, such as TSR, could also be evaluated simultaneously on the same HE-stained WSI. In multivariate analysis for both cohorts, MTR was independent from TSR. We reasoned that combining MTR and TSR could provide more subtle risk stratification for patients with stage II-III CRC. The results show that patients with stroma-high and mucus-high (H/H group) have the worst outcome, and L/L group patients have the most favorable survival. For the one high one low (H/L or L/H) group of patients, the survival curve was between those of the H/H and L/L groups. The result is consistent with previous research reports³, ¹³ and this study's findings: more mucus and more stroma mean a poorer prognosis for patients with CRC.

In the stratified analysis for the colon and rectum subgroup, the MTR only remains statistically significant in the rectum group, where marginal significance was observed in the colon group (P = 0.060). The result is consistent with mucinous adenocarcinoma and non-mucinous adenocarcinoma study, which shows that mucinous adenocarcinoma only has different survival from non-mucinous adenocarcinoma in rectal cancer.²⁸ Our results also indicated that the discrimination performance between the model with MTR and the model without MTR was not statistically significant in patients, except in the rectal cancer group (P = 0.020).

One limitation of the study is that we performed patch-level segmentation on WSIs instead of pixel-level segmentation because of the lack of pixel-level annotations. Also, the quantification of the exact proportion of mucus in the whole tumor and its prognostic value is still worthy of further investigation. In addition, it is also beneficial to combine MTR with other biomarkers of tumor invasion front, such as TB, PDCs, TLS, etc. Furthermore, the presented CRC tissue segmentation model and the prognostic value of MTR require prospective validations in the future.

In short, we present a workflow to quantify mucus proportion in the most invasive part of the primary tumor for CRC. We defined the MTR and evaluated its prognostic value in two patient cohorts. The MTR was an independent predictor for OS, especially in rectal cancer. With advantages of high efficiency and consistency, our method is suitable for clinical application, promoting precision medicine development.

Author contributions

KZ, YLH, and YQH designed the study. KZ, HHW, YTL, CHL, MNZ, and XC wrote the draft report. ZYL, ZHL, and YL initiated and coordinated the study. LW, SY, and ZYX labeled images. KZ, YX, and JMP analyzed images. KZ and HL did the statistical analysis. All authors discussed the early version of the report and provided comments and suggestions for change. All authors have approved the final report for publication.

Conflict of interest

As an Editorial Board Member of Precision Clinical Medicine, the corresponding author Zaiyi Liu was blinded from reviewing or making decisions on this manuscript.