Computerized tumor-infiltrating lymphocytes density score predicts survival of patients with resectable lung adenocarcinoma

Summary

A high abundance of tumor-infiltrating lymphocytes (TILs) has a positive impact on the prognosis of patients with lung adenocarcinoma (LUAD). We aimed to develop and validate an artificial intelligence-driven pathological scoring system for assessing TILs on H&E-stained whole-slide images of LUAD. Deep learning-based methods were applied to calculate the densities of lymphocytes in cancer epithelium (DLCE) and cancer stroma (DLCS), and a risk score (WELL score) was built through linear weighting of DLCE and DLCS. Association between WELL score and patient outcome was explored in 793 patients with stage I-III LUAD in four cohorts. WELL score was an independent prognostic factor for overall survival and disease-free survival in the discovery cohort and validation cohorts. The prognostic prediction model-integrated WELL score demonstrated better discrimination performance than the clinicopathologic model in the four cohorts. This artificial intelligence-based workflow and scoring system could promote risk stratification for patients with resectable LUAD.

Graphical abstract

Highlights

• •TILs assessment system was developed on H&E-stained WSIs of lung adenocarcinoma
• •WELL score was an independent prognostic factor in terms of OS and DFS
• •Prediction model integrated with WELL score demonstrated better performance

Health sciences; Immunology; Cancer; Artificial intelligence

Introduction

Lung cancer is one of the most common cancers and the leading cause of cancer-related death worldwide.¹ Among all patients with lung cancer, lung adenocarcinoma (LUAD) accounts for the most significant proportion, and its incidence rate continues to rise.² The American Joint Commission on Cancer (AJCC) tumor-node-metastasis (TNM) staging system has played a critical role in routine prediction and patient risk stratification.³ Although the TNM stage, integrated with clinical risk factors, provides valuable prognostic information and improves the risk stratification ability of patients to a certain extent,^4,5 the effect is not satisfiable at present and there is an urgent need for new biomarkers. Recently, evaluation of the tumor immune microenvironment (TIME) in solid tumors has been demonstrated as a complementary prognostic biomarker for TNM stage.⁶

The development, metastasis, and prognosis of solid tumors are closely associated with the status of TIME,^7,8,9 which is influenced by the type, abundance, and location of tumor-infiltrating lymphocytes (TILs). Several studies have shown that TILs are associated with the prognosis of many tumor diseases, such as breast cancer,¹⁰ colorectal cancer,¹¹ and oropharyngeal cancer.¹² In the context of lung cancer, high TILs and appropriate spatial architecture appeared to have a protective effect and had favorable outcomes for patients with LUAD.^13,14 Immunohistochemical (IHC) and multiplex immunofluorescence (mIF) are the popular methods to depict the landscape of TILs subpopulation in patients with LUAD.^4,15,16 Our group has recently proposed an immune cell biomarker named I-score, which integrated the density of CD3⁺ and CD8⁺ T cells in the tumor region of the IHC pathological images in patients with non-small cell lung cancer (NSCLC).⁴ Experiments demonstrated that the high I-score was associated with superior disease-free survival (DFS) compared with the low I-score. However, Federico et al. demonstrated that a high number of CD3⁺ T cells in the tumor was not associated with longer recurrence-free survival that was evaluated by mIF images.¹⁶ This is mainly due to the inconsistency or opposite results assessment via IHC images which produced with uncontrollable factors (e.g., antibody selection, experimental operation process, and different automated immunohistochemical staining instruments) across different laboratories. Moreover, mIF imaging is challenging to be embedded into the current clinical practice because it is not a conventional examination compared with Hematoxylin and Eosin (H&E)-stained pathology examination, and the cost is high.¹⁷

H&E-stained pathological sections, which are routinely used by pathologists for diagnosis with low cost and are easy to acquire, contain rich TIME information at the tissue and cell levels. It might be an alternative solution to assess TIME from H&E-stained images when IHC or mIF examinations are unavailable. Brambilla et al.¹³ claimed that intense lymphocytic infiltration was an independent prognostic factor in patients with LUAD through manual evaluation of the H&E-stained section. However, manual TILs quantification of hotspots or regions of interest (ROI) under the microscope is time consuming and laborious, with poor consistency.¹⁸ Restricted by computer hardware, the early automatic quantitative analysis of digital pathology images mainly focused on tissue microarrays (TMAs), which was proved to be a cost-effective method for exploring TILs-based image biomarkers in pathological practice.^19,20 Nevertheless, the TMAs-based method is challenging to obtain a complete picture of a biological sample, often accompanied by selection bias. With the development of high-speed scanner technology, pathological sections could be quickly imaged into whole-slide images (WSIs) with high spatial resolution. This will provide sufficient fuel for computational pathology and make it possible to automatically quantify the TILs in WSIs. At present, there are few artificial intelligence-based workflows²¹ to automatically quantify and locate TILs in H&E-stained WSIs, and the exploration of the prognostic value of patients with LUAD is still insufficient.

In this study, a deep learning-driven digital image biomarker (named WELL score), based on the density and location of TILs, was proposed and the prognostic value of this biomarker was explored in patients with LUAD in four independent cohorts. We hypothesized that the WELL score was an independent prognostic factor and could improve the accuracy in predicting overall survival (OS) and DFS, along with clinicopathologic characteristics, in resectable LUAD.

Results

Baseline characteristics of cohorts

The details of the clinicopathologic characteristics of the discovery and validation cohorts are shown in Table 1. There were statistically significant differences in all of the clinicopathologic characteristics (age at surgery, sex, smoking status, pT stage, pN stage, TNM stage, tumor site, and adjuvant chemotherapy) across the four cohorts (p < 0.05).

Table 1.

Summary of clinicopathologic characteristics of all four cohorts

	D1	V1	V2	V3	p value
Age at surgery (years, median [IQR])	62.0 (55.0, 69.0)	62.0 (57.0, 66.5)	54.0 (48.0, 62.5)	65.0 (58.0, 72.0)	<0.001a
<65	181 (65.6%)	97 (69.8%)	98 (85.2%)	136 (51.7%)	<0.001b
≥65	95 (34.4%)	42 (30.2%)	17 (14.8%)	127 (48.3%)
Sex					0.009b
Male	141 (51.1%)	86 (61.9%)	58 (50.4%)	116 (44.1%)
Female	135 (48.9%)	53 (38.1%)	57 (49.6%)	147 (55.9%)
Smoking status					<0.001b
Never	208 (75.4%)	65 (46.8%)	74 (64.3%)	37 (14.1%)
Former/current	68 (24.6%)	74 (53.2%)	41 (35.7%)	226 (85.9%)
pT stage					<0.001b
T1	124 (44.9%)	42 (30.2%)	81 (70.4%)	108 (41.1%)
T2	130 (47.1%)	60 (43.2%)	21 (18.3%)	132 (50.2%)
T3	19 (6.9%)	24 (17.3%)	6 (5.2%)	20 (7.6%)
T4	3 (1.1%)	13 (9.3%)	7 (6.1%)	3 (1.1%)
pN stage					<0.001b
N0	218 (79.0%)	80 (57.6%)	83 (72.2%)	182 (69.2%)
N1	18 (6.5%)	19 (13.7%)	13 (11.3%)	52 (19.8%)
N2	40 (14.5%)	40 (28.7%)	19 (16.5%)	29 (11.0%)
TNM stage					<0.001b
I	198 (71.7%)	50 (36.0%)	70 (60.9%)	157 (59.7%)
II	32 (11.6%)	33 (23.7%)	17 (14.8%)	72 (27.4%)
III	46 (16.7%)	56 (40.3%)	28 (24.3%)	34 (12.9%)
Tumor site					0.033b
Upper/middle lobe	183 (66.3%)	76 (54.7%)	65 (56.5%)	170 (64.6%)
Lower lobe	93 (33.7%)	63 (45.3%)	50 (43.5%)	90 (34.2%)
Unknown	0 (0%)	0 (0%)	0 (0%)	3 (1.1%)
Adjuvant chemotherapy					<0.001b
No	211 (76.4%)	63 (45.3%)	56 (48.7%)	164 (62.4%)
Yes	65 (23.6%)	76 (54.7%)	59 (51.3%)	99 (37.6%)
Follow-up (month, median [95% CI])c	85.0 (82.4–88.6)	36.9 (36.4–43.0)	60.8 (57.8–65.1)	29.6 (25.8–35.3)
No. of OS events	62 (22.5%)	46 (33.1%)	30 (26.1%)	90 (34.2%)
No. of DFS events	116 (42.0%)	67 (48.2%)	48 (41.7%)	113 (43.0%)

IQR, interquartile range; CI, confidence interval. Data in parentheses are IQR, percentages, or 95% confidence intervals.

^ap values were determined by Kruskal-Wallis rank-sum test.

^bp values were determined by the Pearson’s Chi-square test or Fisher's exact test, where appropriate.

^cmedian follow-up time was estimated by the reverse Kaplan-Meier method.

Prognostic ability of the WELL score

Kaplan-Meier (K-M) survival curves showed that OS was superior for the low-risk group compared with the high-risk group (D1, hazard ratio [HR], 2.68; 95% confidence interval [CI], 1.56–4.62; p < 0.001; V1, HR, 3.05; 95% CI, 1.36–6.83; p = 0.004; V2, HR, 2.39; 95% CI, 0.98–5.84; p = 0.049; V3, HR, 1.99; 95% CI, 0.96–4.13; p = 0.058; Figure 2). In addition, we built a WELL score in terms of DFS and K-M survival curves, which showed that the low-risk group survived better than the high-risk group (D1, HR, 2.07; 95% CI, 1.42–3.01; p < 0.001; V1, HR, 1.54; 95% CI, 0.90–2.62; p = 0.11; V2, HR, 3.80; 95% CI, 1.61–8.96; p = 0.001; Figure 3). In subgroup analyses, the patients were pooled together to increase the discovery power. We found that the WELL score was associated with significantly better OS when stratified by age at surgery, sex, smoking status, and TNM stage (p < 0.05), and this trend (p = 0.210) was demonstrated in patients that received adjuvant chemotherapy (Figure S6).

Figure 2.

Kaplan-Meier curves of patients in four cohorts in terms of OS

Kaplan-Meier curves of patients stratified by WELL score in the discovery cohort D1 (A) and three validation cohorts V1, V2, and V3 (B, C, and D) in terms of OS.

Figure 3.

Kaplan-Meier curves of patients in three cohorts in terms of DFS

Kaplan-Meier curves of patients stratified by WELL score in the discovery cohort D1 (A) and two validation cohorts V1 and V2 (B and C) in terms of DFS.

The univariable and multivariable Cox regression analysis results for OS in different cohorts are shown in Table 2. The risk factors (WELL score and clinicopathologic characteristics) that demonstrated statistical significance at p < 0.05 in the univariable analysis (WELL score, age at surgery, sex, smoking status, and TNM stage) were included in the multivariable analysis. After adjusting for relevant clinical parameters, the multivariable Cox regression analysis revealed that the WELL score was an independent prognostic factor for OS in D1 and V1 (D1, HR, 1.95; 95% CI, 1.12–3.41; p = 0.019; V1, HR, 3.65; 95% CI, 1.61–8.27; p = 0.002). This trend could be found in other two validation cohorts ([V2, HR, 2.06; 95% CI, 0.83–5.07; p = 0.118] and [V3, HR, 1.91; 95% CI, 0.91–3.99; p = 0.085]) (Table 2). Similarly, the WELL score was also an independent prognostic factor for DFS in D1 and V2 (D1, HR, 1.68; 95% CI, 1.14–2.47; p = 0.008; V2, HR, 3.66; 95% CI, 1.53–8.76; p = 0.004). This trend could be found in another validation cohort (V1, HR, 1.58; 95% CI, 0.92–2.71; p = 0.097) (Table S1).

Table 2.

Univariable and multivariable Cox regression analyses for OS in four cohorts

	Discovery cohort D1a		Validation cohort V1		Validation cohort V2		Validation cohort V3
	HR (95% CI)	p value	HR (95% CI)	p value	HR (95% CI)	p value	HR (95% CI)	p value
Univariable analysis
Age at surgery (years)
≥65 vs. ＜65	2.04 [1.24, 3.36]	0.005	1.08 [0.57, 2.02]	0.814	1.39 [0.57, 3.41]	0.468	1.37 [0.91, 2.08]	0.133
Sex
Female vs. male	0.49 [0.29, 0.83]	0.007	0.43 [0.22, 0.85]	0.015	0.95 [0.46, 1.95]	0.894	0.83 [0.55, 1.25]	0.371
Smoking status
Former/current vs. never	2.00 [1.19, 3.34]	0.008	2.04 [1.11, 3.75]	0.021	0.69 [0.32, 1.51]	0.351	1.35 [0.70, 2.61]	0.372
TNM stage
Stage III vs. stage I/II	5.54 [3.30, 9.30]	<0.001	2.02 [1.13, 3.61]	0.018	3.48 [1.69, 7.14]	<0.001	2.27 [1.35, 3.81]	0.002
Tumor site
Upper/middle vs. lower	1.39 [0.80, 2.44]	0.243	1.38 [0.76, 2.51]	0.296	1.01 [0.49, 2.07]	0.986	0.82 [0.54, 1.26]	0.367
WELL scoreb
High vs. low	2.68 [1.56–4.62]	<0.001	3.05 [1.36–6.83]	0.004	2.39 [0.98–5.84]	0.049	1.99 [0.96–4.13]	0.058
Multivariable analysis
Age at surgery (years)
≥65 vs. ＜65	2.29 [1.38, 3. 81]	0.001	1.29 [0.67, 2.47]	0.445	1.26 [0.50, 3.22]	0.625	1.30 [0.85, 1.98]	0.232
Sex
Female vs. male	0.60 [0.33, 1.09]	0.129	0.58 [0.22, 1.57]	0.285	0.81 [0.31, 2.11]	0.662	0.78 [0.51, 1.18]	0.237
Smoking status
Former/current vs. never	1.30 [0.71, 2.36]	0.366	1.59 [0.64, 3.93]	0.316	0.57 [0.20, 1.60]	0.286	1.29 [0.66, 2.52]	0.458
TNM stage
Stage III vs. stage I/II	5.39 [3.14, 9.24]	<0.001	1.92 [1.06, 3.45]	0.030	3.20 [1.53, 6.68]	0.002	1.95 [1.14, 3.34]	0.015
WELL scoreb
High vs. low	1.95 [1.12, 3.41]	0.019	3.65 [1.61, 8.27]	0.002	2.06 [0.83, 5.07]	0.118	1.91 [0.91, 3.99]	0.085

^aD1 is the discovery cohort.

^bContinuous variables DLCE score and DLCS score were employed for Cox modeling, and the coefficients of the two variables in terms of OS were −0.4181 and −0.4400, respectively. We combined the two variables into a score through linear weighting, which was named WELL score, WELL score = −0.4181^∗LDCE score −0.4400^∗LDCS score.

Development and validation of the prognostic model

Because age at surgery, TNM stage, and WELL score were identified as independent factors for OS in multivariable Cox regression analysis in the discovery cohort, we developed a predictive model (full model) integrating the above independent factors. We further compared the performance of the full model with four other models that included the TNM stage model, the WELL score model, the TNM stage & WELL score combined (TNM_WELL) model, and the TNM stage & age at surgery combined (clinicopathologic) model. The baseline hazard, coding, and regression coefficients of each model are summarized in Table S2.

The model performance is presented in Table 3. The full model showed better discrimination performance (evaluated by C-index and iAUC) and better calibration ability (evaluated by AIC) than the clinicopathologic model in the four cohorts. The TNM_WELL model showed better discrimination and calibration performance than the TNM stage model in the four cohorts. Integrating the WELL score into the TNM stage model improved the prediction for OS (likelihood ratio test, χ² = 8.269, p = 0.004); also, integrating the WELL score into the clinicopathologic model improved the prediction for OS (likelihood ratio test, χ² = 6.967, p = 0.008). The receiver operating characteristic curve (ROC, t = 60 months) and resulting time-dependent AUC plotted over time are presented in Figure S7. The full model showed higher AUC across most time points compared to the reference model.

Table 3.

Performance metrics of the full model (integrated WELL score and clinicopathologic characteristics) and reference models in the discovery and validation cohorts

Cohort	Model	C-index	iAUC	AIC
Discovery cohort (D1)	TNM stage	0.651 (0.592–0.710)	0.634	629.2
	WELL score	0.622 (0.563–0.681)	0.618	650.2
	TNM_WELL	0.709 (0.646–0.772)	0.700	622.9
	Clinicopathologic	0.723 (0.658–0.788)	0.715	618.9
	Full	0.751 (0.686–0.815)	0.743	614.0
Validation cohort 1 (V1)	TNM stage	0.586 (0.511–0.661)	0.590	417.4
	WELL score	0.589 (0.523–0.655)	0.609	413.7
	TNM_WELL	0.627 (0.548–0.707)	0.659	411.4
	Clinicopathologic	0.596 (0.512–0.680)	0.593	417.9
	Full	0.622 (0.535–0.709)	0.664	413.0
Validation cohort 2 (V2)	TNM stage	0.632 (0.547–0.717)	0.634	267.2
	WELL score	0.587 (0.509–0.664)	0.587	273.5
	TNM_WELL	0.679 (0.592–0.767)	0.676	264.3
	Clinicopathologic	0.650 (0.556–0.744)	0.648	267.0
	Full	0.683 (0.593–0.773)	0.684	264.3
Validation cohort 3 (V3)	TNM stage	0.565 (0.516–0.613)	0.552	853.4
	WELL score	0.527 (0.488–0.565)	0.543	857.3
	TNM_WELL	0.585 (0.531–0.639)	0.580	850.7
	Clinicopathologic	0.586 (0.521–0.651)	0.575	852.5
	Full	0.596 (0.529–0.663)	0.597	850.3

Visualization of TILs and assessment of the immune cell phenotypes

Visual examples of high-risk and low-risk groups identified by the WELL score were shown in Figure 4. We could observe that densities of TILs in the cancer epithelium and cancer stroma regions of low-risk patients in the discovery and validation cohorts were much higher than those of high-risk patients. CIBERSORT algorithm^22,23 was employed to calculate the relative level of 22 immune cell phenotypes using RNA transcripts for the patients in V3 (n = 263). 25 samples were excluded because of p > 0.05 or lack of immune infiltration. Among the 22 immune cells, the difference in the abundance of 7 immune cells between the high-risk and low-risk groups was statistically significant (p < 0.05, Figure 5). The fractions of plasma cells, CD8⁺ T cells, and activated CD4⁺ memory T cells in the low-risk group were higher than those of the high-risk group (p < 0.05). The fractions of CD4⁺ memory resting T cells, macrophages M2, and mast cells resting in the low-risk group were lower than those of the high-risk group (p < 0.05).

Figure 4.

heatmaps of TILs

DLCE and DLCS maps of high-risk and low-risk groups from different cohorts (discovery cohort D1 and three validation cohorts V1, V2, and V3). The first and second columns were DLCE maps of the high-risk and low-risk groups identified by the WELL score. The third and fourth columns were DLCS maps of the high-risk and low-risk groups identified by the WELL score. The color bar (on the right of the figure) indicates the TILs density measurement (blue represents the low density, while red represents the high density).

Figure 5.

Immune cell phenotypes evaluated by CIBERSORT algorithm

Among the 22 immune cells, the distribution of 7 immune cells between the high-risk and low-risk groups was statistically significantly different (p < 0.05). The blue violin represents the low-risk group and the red represents the high-risk group. The fractions of plasma cells, CD8 T cells, and CD4 memory-activated T cells in the low-risk group are higher than those of the high-risk group. The fractions of CD4 memory resting T cells, macrophages M2, and mast cells resting in the low-risk group are lower than those of the high-risk group.

Discussion

In this study, we developed a digital biomarker named WELL score, which integrated the densities of TILs in cancer epithelium and cancer stroma region of H&E-stained WSIs. The prognostic analysis found that the WELL score was an independent prognostic factor in patients with resectable LUAD. Compared with a high WELL score, a low WELL score was associated with significantly superior OS and DFS in patients with LUAD. To our knowledge, this is the first image biomarker based on the artificial intelligence-driven automatic quantification of H&E-stained WSIs that focus on the prognosis analysis of resectable LUAD.

The analysis of TIME largely depends on the accurate quantification of tissue- and nuclei-level information. However, the heterogeneity of lung cancer pathological images has brought a great challenge in building the analysis manner.^24,25 This work proposed a deep learning-based pipeline for the TILs quantification. In the tumor region segmentation process, a transfer learning framework was employed to pre-train the deep learning model by using pathological images of other tissues (breast cancer). We then used a small amount of annotated pathological images of LUAD for fine-tuning to accurately segment of tumor region while making full use of the labeled data. For the overstained vascular tissues or atelectasis that belong to normal tissue, it is easy to confuse with tumor-associated stroma (which belongs to the tumor region), so pathologists with relevant experience are essential to identify these issues. In the process of tissue segmentation, our proposed weakly supervised method was applied to segment the tumor region into cancer epithelium, cancer stroma, lymphoid aggregates, and necrosis.²⁶ Compared with the full supervision method which requires a lot of pixel-level annotations,²¹ the weakly supervised semantic segmentation method only needs patch-level labels, which will greatly reduce the burden of data annotation for pathologists. The lymphoid aggregates and necrosis region were excluded according to the practical review.²⁷ When calculating the density of TILs, the numerator of the calculation formula was the number of TILs, and the denominator represented the area of cancer epithelium or cancer stroma. Owing to the unique structure of the cavity in lung cancer tissue and the fact that the area and ratio of the blank region varied significantly in different pathological grades,²⁸ we removed this area, which would help to calculate the density of TILs accurately and was conducive to downstream analysis.

TILs scoring process in breast cancer is relatively mature,²⁹ but the uniform workflow is still unavailable for LUAD. Furthermore, there is no unified statement about which regions/compartments’ TILs scores are calculated. Most of the TILs-related research focused on the density of TILs in the stromal region. For example, Donnem et al.³⁰ evaluated the density of stomal CD8⁺ TILs, which was demonstrated as the independent prognostic factor in NSCLC. Two pathologists evaluated stromal TILs in H&E-stained WSIs in a semiautomatic manner, and comparative experimental results showed that high-density TILs in the stroma were positively associated with better survival for the patients that received immunotherapy.³¹ Recently, some research work advocated considering the importance of TILs in cancer epithelium region.^6,21 Park et al.²¹ firstly classified the tiles into three different immune phenotypes (IPs) according to the TILs density of the cancer epithelium region and cancer stroma region. Then the ratio of different IPs patches determined the IP of WSIs, which formed an image biomarker. It needed to set twice cutoff values on the processing pipeline in the discovery cohort, which may weaken the generalization ability of the model, especially when encountering a gap in data distribution between the discovery cohort and validation cohorts. In this study, we directly quantified the TILs density in cancer epithelium and cancer stroma of the tumor region in the WSIs. Furthermore, we developed and validated a new biomarker by integrating DLCE score and DLCS score and conducted the prognostic analysis.

Despite the differences in clinicopathologic characteristics among the four cohorts (Table 1), the WELL score also had good prognostic performance across different cohorts using a cutoff threshold learned from the discovery cohort. Image metrics related to brightness and contrast that were computed via HistoQC³² were embedded using the t-SNE algorithm³³ and demonstrated in Figure S8A, which showed that each cohort was aggregated separately, indicating a strong batch effect. However, the image features (DLCE score and DLCS score) extracted by us from different cohorts were interlaced or overlapped with each other, indicating that two features of TILs were resilient to batch effects and reproducible across multiple cohorts (Figure S8B). Both in cancer epithelium and cancer stroma regions, the TILs densities of low-risk groups were higher than those of high-risk groups (Figure 4). This observation may suggest that more infiltration of lymphocytes was conducive to stronger anti-tumor immune reaction and thus conducive to the better survival of patients with LUAD. This was consistent with the experimental results or conclusions of¹⁸ previous studies mainly focused on the associations between DLCS and prognosis.^15,30 This study comprehensively considered the impact of DLCE and DLCS on the prognosis of LUAD patients. It suggested that the risk stratification and generalization ability of the WELL score were better than those of DLCE or DLCS alone (Figures 2, S2, and S3).

Although the WELL score could enable risk stratification of LUAD by the density of TILs assessed via H&E-stained sections, the association of the WELL score with the immune infiltration landscape remained unclear. Hence, we applied the CIBERSORT,²² a tool for inferring the composition of immune cells by RNA transcripts, to clarify the lymphocyte infiltration landscape between the high-risk and low-risk groups. We found that the patients in the low-risk group had a higher proportion of CD8⁺ T cells, activated CD4⁺ memory T cells, and plasma cells versus those in the high-risk group, and these lymphocyte subsets participated in many aspects of antigen presentation and anti-tumor immune reaction.³⁴ It was widely reported that the high abundance of CD8⁺ T cells,^30,35 activated (but not resting) CD4⁺ memory T cells,³⁶ and plasma cells³⁷ contributed to favorable survival outcomes in LUAD, which was consistent with our findings. In addition, these lymphocyte subsets were involved in the formation of tertiary lymphoid structures, which played an important role in controlling tumor invasion and metastasis.³⁸ We speculated that the differences in survival between the high-risk and low-risk groups might be due to the dual effect on both the TILs density and the abundance of these lymphocyte populations that were associated with better prognosis. However, the spatial correlation of these lymphocyte populations and the density of TILs remained to be further explored by using multiplex immunohistochemistry analysis.

In conclusion, we developed and validated a TILs-based digital image biomarker (named WELL score) by using H&E-stained WSIs. The WELL score was an independent prognostic factor and would improve the risk stratification ability when integrated with clinicopathologic characteristics for resectable LUAD patients. A future prospective study of the clinical application of the WELL score in the prognostic analysis of LUAD is warranted.

Limitations of the study

Our research still had some limitations. First, this work was retrospective with significant differences in clinicopathologic characteristics of different cohorts, which brought great challenges to the model development and validation. In addition, the protocol for the production of pathological sections and the parameters of WSI scanners in each cohort were different, which resulted in great differences in pathological images and was more prone to batch effects. Thanks to the two indexes (DLCE and DLCS), they alleviated batch effects to a certain degree that benefited multi-institute prognostic analysis. Second, we quantified DLCE and DLCS via routine H&E-stained slides but did not distinguish between lymphocyte (immune cell) subtypes. Automatic identification of subtypes of TILs (eg., CD3, CD4, CD8, CD20, etc.) could refine and enrich our studies. Third, this study was based on the analysis of a single WSI without considering the case of multiple slides. Although multiple slides would significantly increase the computational overhead, it may alleviate the bias in slide selection. In addition, the auxiliary results of lymph node metastases sections may further improve the discriminant ability of the WELL score-based model.

STAR★Methods

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Software and algorithms
R code for statistical analysis	This paper	https://github.com/panxipeng/HE-WSI-lungTILs
Hover-Net	Graham et al.39	https://github.com/vqdang/hover_net
Tissue segmentation toolbox	Han et al.26	https://github.com/ChuHan89/WSSS-Tissue
ResNet-50	He et al.40	https://arxiv.org/abs/1512.03385
HistoQC	Janowczyk et al.32	https://github.com/choosehappy/HistoQC
CIBERSORT	Newman et al.22	https://cibersort.stanford.edu/
Python	Open resource	https://www.python.org/
RStudio	Open resource	https://www.rstudio.com/
R software	R CRAN	https://cran.r-project.org/
survival R package (v3.2-7)	R CRAN	https://cran.r-project.org/web/packages/survival/index.html
survminer R package (v0.4.8)	R CRAN	https://cran.r-project.org/web/packages/survminer/index.html
Rmisc R package (v1.5.1)	R CRAN	https://cran.r-project.org/web/packages/Rmisc/index.html
timeROC R package (v0.4)	R CRAN	https://cran.r-project.org/web/packages/timeROC/index.html
lmtest R package (v0.9-40)	R CRAN	https://cran.r-project.org/web/packages/lmtest/index.html
lpSolve R package (v5.6.15)	R CRAN	https://cran.r-project.org/web/packages/lpSolve/index.html
irr R package (v0.84.1)	R CRAN	https://cran.r-project.org/web/packages/irr/index.html
ggplot2 R package (v3.3.6)	R CRAN	https://cran.r-project.org/web/packages/ggplot2/index.html
vioplot R package (v0.3.7)	R CRAN	https://cran.r-project.org/web/packages/vioplot/index.html
openxlsx R package (v4.2.2)	R CRAN	https://cran.r-project.org/web/packages/openxlsx/index.html
gbm R package (v2.1.8)	R CRAN	https://cran.r-project.org/web/packages/gbm/index.html
Other
Camelyon Challenge dataset	Litjens et al.41	https://camelyon17.grand-challenge.org
Lung adenocarcinoma whole-slide images dataset	The Cancer Genome Atlas	https://portal.gdc.cancer.gov/
Lung adenocarcinoma RNA transcripts dataset	The Cancer Genome Atlas	https://portal.gdc.cancer.gov/

Resource availability

Lead contact

Further information and requests for resources and reagents should be directed to and will be fulfilled by the lead contact, Zaiyi Liu (liuzaiyi@gdph.org.cn).

Materials availability

This study did not generate new unique reagents.

Experimental model and subject details

Patients cohorts

Patients with LUAD who underwent curative-intent resection were retrospectively recruited from four institutions: Guangdong Provincial People’s Hospital (discovery cohort, D1), Shanxi Cancer Hospital (validation cohort 1, V1), Yunnan Cancer Hospital (validation cohort 2, V2), and The Cancer Genome Atlas (validation cohort 3, V3). The study was approved by the Research Ethics Committee of Guangdong Provincial People’s Hospital, the Ethics Committee of Shanxi Provincial Cancer Hospital, and the Institutional Review Board of Yunnan Cancer Hospital (approval number: KY-Z-2021-030-02, 202106, and KY2020139). As this was a retrospective image analysis study, the informed consent of patients was waived. The clinical factors, including age at surgery, sex, smoking status, pT stage, pN stage, TNM stage, tumor site, and adjuvant chemotherapy were collected from the hospital information system. The patients were excluded because of no intact clinicopathologic characteristics, receiving neoadjuvant therapy, incomplete resection, history of other tumors, and obvious artifacts on pathology image quality evaluation. The detailed inclusion criteria and exclusion criteria are shown in Figure S1. The endpoints were overall survival (OS) and disease-free survival (DFS).

H&E-stained slides digitalization and quality control

In discovery cohort (D1) and validation cohort 2 (V2), the glass sections were scanned as whole-slide images (WSIs) at 40× magnification (0.25μm/pixel) using a scanner (Leica, Aperio-AT2, USA), and the scanning equipment brand was Hamamatsu (Japan, 40× magnification, 0.22μm/pixel) in validation cohort 1(V1) cohort. WSIs of TCGA came from multiple institutes with large variances in terms of color and contrast. H&E-stained WSIs were reviewed by two pathologists (BBL and LXY) to select the most representative tumor slide. In addition, other low-quality (e.g., out-of-focus, blurred, pen markings, noticeable artifacts) WSIs were removed.

Method details

Semi-automated tumor region segmentation

The image processing pipeline was shown in Figures 1B–1D. Tumor region segmentation was a prerequisite for tissue and nuclei segmentation and classification. Firstly, the Camelyon Challenge dataset⁴¹ was cut into tumor patches and non-tumor patches, and fed into the ResNet-50 ⁴⁰ model for pre-training; Then, 67 LUAD WSIs from D1 were cut into tumor patches and non-tumor patches for fine-tuning the pre-trained model. Finally, the fine-tuning model was adopted in the tumor region segmentation for the remaining WSIs from D1 and all WSIs from the other three institutes. The segmentation masks of the tumor region were checked by two pathologists (BBL and LXY), and the imperfect masks were corrected for the following workflow.

Figure 1.

The overall workflow of the study

(A) The HE-stained glass sections were scanned into whole-slide images.

(B) Semiautomated tumor region segmentation. A transfer learning framework was deployed for tumor region segmentation and segmentation masks of the tumor region were checked by two pathologists.

(C) Fully automated tissue and nuclei segmentation. To accurately obtain the boundary of the tissue region, our proposed semantic segmentation method was proposed to divide the tumor region into cancer epithelium, cancer stroma, lymphoid aggregate, and necrosis regions on WSIs under 10× magnification. Hover-Net was deployed to segment and classify the nuclei into tumor nuclei, stromal nuclei, lymphocytes, and other nuclei in the tumor region of WSI at the maximum magnification.

(D) TILs in cancer epithelium and cancer stroma. The densities of lymphocytes in cancer epithelium (DLCE) and cancer stroma (DLCS) were calculated.

(E) Risk score and model development. A risk score (named WELL score) was built through linear weighting of DLCE score and DLCS score. A discovery cohort was used to optimize WELL scores and the median value of WELL scores in the discovery cohort was selected as the cutoff to stratify patients into high- and low-risk groups. Univariable and multivariable Cox regression analyses were performed to evaluate the association of WELL score and clinical risk factors with OS and DFS.

(F) Model validation. The other two cohorts and TCGA were used to validate the performance of the model.

Tissue segmentation of tumor region

To accurately obtain the boundary of the tissue region, we introduced the semantic segmentation (pixel-level classification) method to divide the tumor region into cancer epithelium, cancer stroma, lymphoid aggregate, and necrosis regions. The tissue semantic segmentation was performed on WSIs under 10× magnification to save the computation cost. To save the workload of manual annotation, we applied our proposed weakly supervised segmentation method based on patch-level classification labels. The method was divided into two stages: image classification and image segmentation. In the image classification stage, we used a class activation map (CAM)-based model to generate pseudo segmentation; in the image segmentation stage, we used our proposed Multi-Layer Pseudo-Supervision strategy²⁶ to improve the accuracy of image segmentation. The schematic diagram is shown in Figure 1C.

Nuclei segmentation and classification

Hover-Net,³⁹ consisting of one encoding branch and three decoding branches, was one of the state-of-the-art deep learning models for nuclei segmentation and classification. We deployed Hover-Net to segment and classify the nuclei into tumor nuclei, stromal nuclei, lymphocytes, and other nuclei in the tumor region of WSI at the maximum magnification. The schematic diagram is shown in Figure 1C.

TILs quantification and WELL score establishment

We quantified the TILs density using the following equations.

$$\mathrm{D}\mathrm{L}\mathrm{C}\mathrm{E}=\frac{\#\mathrm{o}\mathrm{f}\mathrm{l}\mathrm{y}\mathrm{m}\mathrm{p}\mathrm{h}\mathrm{o}\mathrm{c}\mathrm{y}\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{p}\mathrm{i}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{l}\mathrm{i}\mathrm{u}\mathrm{m}}{\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{a}\mathrm{o}\mathrm{f}\mathrm{c}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{p}\mathrm{i}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{l}\mathrm{i}\mathrm{u}\mathrm{m}}$$ (Equation 1)

$$\mathrm{D}\mathrm{L}\mathrm{C}\mathrm{S}=\frac{\#\mathrm{o}\mathrm{f}\mathrm{l}\mathrm{y}\mathrm{m}\mathrm{p}\mathrm{h}\mathrm{o}\mathrm{c}\mathrm{y}\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{r}\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{o}\mathrm{m}\mathrm{a}}{\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{a}\mathrm{o}\mathrm{f}\mathrm{c}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{r}\mathrm{s}\mathrm{t}\mathrm{r}\mathrm{o}\mathrm{m}\mathrm{a}}$$ (Equation 2)

$$\text{DLCE score }=\text{ log}(\text{DLCE}+1)$$ (Equation 3)

$$\text{DLCS score }=\text{ log}(\text{DLCS}+1)$$ (Equation 4)

$$\text{WELL score }=(\text{w}1\times \text{ DLCE score})+(\text{w}2\times \text{ DLCS score})$$ (Equation 5)

where DLCE indicated the density of TILs in cancer epithelium (Equation 1); DLCS indicated the density of TILs in cancer stroma (Equation 2). We used a logarithmic operation⁴² to compress the dynamic range of DLCE and DLCS for the sake of better visualization and analysis (Equations 3 and 4). A risk score, named WELL score (Equation 5), was constructed through linear weighting of DLCE score and DLCS score. A Cox regression model was built in the discovery cohort to estimate the weights w₁ and w₂. The median value of the WELL score in the discovery cohort was set as a cutoff. That is, a patient with a WELL score higher than the median value belonged to the high-risk group and vice versa.

Quantitative assessment of immune cells via CIBERSORT algorithm

A well-known bioinformatics tool (named CIBERSORT algorithm^22,23) was used to evaluate the proportion of the relative level of 22 immune cell phenotypes according to the expression of immune-related and other genes in the complex expression mixture. These 22 kinds of immune cells are mainly composed of T cells, B cells, natural killer cells, monocytes, macrophages, dendritic cells, mast cells, eosinophils, neutrophils, etc. We set the threshold p<0.05 and excluded the sample without immune infiltration.

Quantification and statistical analysis

Continuous clinicopathologic characteristics (age at surgery) were reported as median (IQR) and compared via Kruskal-Wallis rank-sum test. Categorical clinicopathologic characteristics (sex, TNM stage, etc.) were reported as count (percentage) and compared via Pearson’s Chi-square test or Fisher’s exact test, where appropriate. The follow-up time across the institutions was estimated using the reverse Kaplan-Meier (K-M) method. K-M curves and Cox proportional hazards regression models were employed for survival analyses. The log-rank test was used to compare the high-risk group and the low-risk group in terms of OS and DFS. The univariable Cox regression analysis was performed to evaluate the risk factors that are associated with OS and DFS, and the factors that demonstrated statistical significance (p < 0.05) would be served as candidates in multivariable Cox regression analysis. The final model was determined by stepwise regression, and Akaike information criterion (AIC) was used to balance the model complexity and the goodness of the fit. The discrimination performance of the prediction models was evaluated using Harrell’s C-statistics (C-index) and the integrated area under the curve (iAUC). The time-dependent area under the curve (tAUC) was computed and plotted over time.

All statistical analyses were performed using R (version 4.0.2; https://www.r-project.org/) with packages survival, survminer, Rmisc, timeROC, lmtest, lpSolve, and irr. All statistical tests were two-sided, and p-values < 0.05 were considered statistically significant. Since this study was retrospective and the patients with end-point events (i.e., 62/278 in our case) in the discovery cohort had been identified, our risk variables could not exceed 6 to meet the 10 EPP rule.⁴³

Published: December 22, 2022

Data and code availability

• •Image and genetic data of The Cancer Genome Atlas are publicly available as of the date of publication. DOIs are listed in the key resources table. Image and clinicopathological information from other institutions are available from the lead contact (Prof. Zaiyi Liu via liuzaiyi@gdph.org.cn) on reasonable requests.
• •The code is available in the github repository (https://github.com/panxipeng/HE-WSI-lungTILs).
• •Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.