Skip to main content
Scientific Reports logoLink to Scientific Reports
. 2026 Mar 19;16:14144. doi: 10.1038/s41598-026-44633-y

Estimating major pathological response in non-small cell lung cancer patients with post-neoadjuvant therapy using MMT-net

Chenghan Yang 1,2, Liqin Liu 3, Weijin Xiao 1, Feng Wu 3, Lei Guo 4,✉, Chao Li 1,3,4,5,✉
PMCID: PMC13136302  PMID: 41857185

Abstract

Neoadjuvant therapy (NAT) has demonstrated considerable effectiveness in treating locally advanced non-small cell lung cancer (NSCLC). Major pathological response (MPR), defined by the proportion of viable tumor cells in the tumor bed assessed through whole slide images (WSIs), is regarded as a crucial marker for predicting patient prognosis. While artificial intelligence (AI)-assisted WSI analysis has proven to be more effective than manual inspection, the intricate nature of analyzing post-NAT NSCLC WSIs still poses substantial clinical and computational challenges. In this study, we introduce a self-supervised learning model named the Multi-Magnifier Transformer Network (MMT-Net), designed for estimating MPR in NSCLC patients with post-NAT. The MMT-Net framework comprises three main components: a CNN backbone pretrained on unannotated pathological images for feature extraction, a multi-attention mechanism for fusing multi-scale features and a transformer layer for final classification. Validation results on the curated NSCLC post-NAT dataset have demonstrated that MMT-Net is more effective than commonly used methods for estimating MPR. Additionally, attention map visualizations show strong agreement with pathologist annotations, and the model outputs significantly correlate with immunohistochemistry markers such as CK and Ki67, confirming the reliability of MMT-Net. The proposed MMT-Net holds promise for improving the efficiency of MPR estimation, further advancing the clinical application of NAT in NSCLC.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-026-44633-y.

Keywords: Whole slide image analysis, Neoadjuvant therapy, Major pathological response estimation, Deep learning

Subject terms: Cancer, Computational biology and bioinformatics, Oncology

Introduction

Neoadjuvant therapy (NAT), encompassing chemotherapeutic and immunotherapeutic modalities, is an established intervention for individuals diagnosed with non-small cell lung carcinoma (NSCLC) at clinical stage IIIA or less1. The major pathological response (MPR) is defined as the proportion of surviving tumor cells within the tumor bed. It serves as a commonly used surrogate marker for long-term survival in patients with post-NAT2,3. The International Association for the Study of Lung Cancer (IASLC) has proposed an assessment scheme for MPR based on extensive data collected from both existing and ongoing clinical trials4. According to the IASLC recommendations, the tumor bed is subdivided into viable tumor (residual tumor cells with preserved morphology), stroma (therapy-induced fibrotic, inflammatory, or desmoplastic regions), and necrosis & keratinization (areas of coagulative necrosis or keratin debris). It is widely accepted that NSCLC patients who show a favorable response to NAT typically have 10% or fewer surviving tumor cells within the tumor bed.

The whole slide imaging (WSI) offers improved durability, easier retrieval and remote diagnosis support, is the most commonly used technique by pathologists to estimate the MPR in NSCLC patients with post-NAT. Recently, a pathological image generated by the advanced WSI instruments contains billions of pixels, making manual inspection exceedingly time-consuming, laborious, and susceptible to human error. Benefiting from the remarkable success of deep learning in the field of computer vision, artificial intelligence (AI)-assisted WSI analysis has emerged as a more effective strategy than manual inspection, thereby improving the reliability of MPR estimation.

Numerous AI-assisted WSI analysis methods have been developed to aid in the examination of pathological images, and have demonstrated their competitive performance relative to experienced pathologists across various cancer types. For example, Cheng et al. develop a deep neural network model, called HnAIM, to facilitate early detection and risk stratification of liver cancer, and emphasize its superior diagnostic performance compared to pathologists in identifying hepatocellular nodular lesions5. Kather et al. design a convolutional neural network (CNN)-based model to analyze colorectal cancer tissue slides, and they present a prognostic biomarker called “deep stroma score” to demonstrate significant predictive value for patient survival outcomes6. Coudray et al. employ CNN to differentiate histological subtypes of lung cancer and predict mutations in lung adenocarcinoma, highlighting the promising role of AI-assisted WSI analysis in enhancing oncological diagnostics7. Mohi et al. conduct a comprehensive review of the current deep learning algorithms, and also discuss future prospects for the application of artificial intelligence in WSIs analysis8.

Nevertheless, the existing AI-assisted WSI analysis methods still exhibit limited performance in complex medical scenarios, particularly for MPR estimation in NSCLC patients with post-NAT. On the one hand, the existing deep learning-based models are typically fine-tuned from pretrained models that are initially trained on massive natural image datasets, e.g. the ImageNet dataset9. These models often struggle to capture domain-specific features from pathological images that are characterized by discontinuous spatial priors, ultra-high resolution, and complex textural patterns compared with natural images10,11. On the other hand, there is a broad consensus that low magnifications are necessary for evaluating tumor architecture, spatial distribution and therapy-induced necrosis, while high magnifications are needed for identifying nuclear atypia, mitotic figures and residual tumor cells12. Most existing methods focus on WSI analysis using the single-magnification approach, potentially missing crucial texture patterns related to MPR that are present at other magnification scales. These limitations hinder the application of AI-assisted WSI analysis for estimating MPR in NSCLC patients with post-NAT.

To address these issues, we propose an effective AI-assisted WSI analysis method, called MultiMagnifier-Transformer Net (MMT-Net), to improve the MPR estimation of NSCLC patients with post-NAT. By leveraging self-supervised learning13 using SimCLR framework, MMT-Net extracts more effective pathological image features than other models pretrained on ImageNet, showcasing its capability in handling high-resolution, complex pathological images. To capture features related to MPR that are present at multi-magnification scales, an effective muti-head attention mechanism14 and a Transformer layer15 are introduced here to fuse multi-magnification features to achieve better performance compared to single-magnification training. Numerous results have demonstrated that MMT-Net achieves the highest accuracy among current AI-assisted WSI analysis methods, exhibiting a remarkable ability to focus on regions akin to those identified by pathologists during MPR estimation. Additionally, the model reliability is validated by immunohistochemistry, with results showing a high degree of correlation between model predictions and the markers such as CK and Ki67 staining. The proposed MMT-Net model is expected to be a powerful tool for MPR estimation.

Materials and methods

Patient cohort and whole-slide imaging

In this study, we retrospectively analyze 23 consecutive cases of non-small cell lung cancer (NSCLC) resected between 2022 and 2023 at Fujian Cancer Hospital. The majority of histological subtypes in this cohort are determined based on preoperative biopsy specimens. All patients receive neoadjuvant chemoradiotherapy. Due to the retrospective nature of this study and the analysis of pathological specimens with limited clinical information, the requirement for informed consent is waived. This study received approval from the Institutional Ethics Committee of Fujian Cancer Hospital (SQ2021-070-01). Pathological hematoxylin and eosin (H&E) sections from each patient are scanned using the EasyScan WSI scanner (MOTIC CHINA GROUP CO., LTD.), resulting in a total of 68 WSIs. The ground truth is annotated by two experienced thoracic oncology pathologists that underwent rigorous, standardized training specifically focused on the IASLC criteria. The pathologists independently review and annotate half of the WSIs using Motic Digital Slide Assistant Lite software. Here, in accordance with the IASLC standards, the tumor bed is defined as histological changes comprising viable tumor, stroma and necrosis & keratinization. To ensure accuracy and consistency in delineating the tumor bed components, all discrepancies in the annotations are resolved through consensus discussions.

Data preprocessing and dataset construction

To ensure the effectiveness of our training model, we randomly divide the entire patient cohort into a training set, which includes 11 cases with 39 H&E-stained slides, and a testing set, which comprises 12 cases with 29 H&E-stained slides. The training set is further split into a training subset and a validation subset at a ratio of 4:1. The training subset is used to train the model and perform inference, while the validation subset is used to select optimal model parameters during the training process. The experimental configuration is detailed in the Supplementary Table 1.

The data preprocessing for WSI data is conducted as follows: Firstly, the binary OTSU algorithm16 is employed to remove background regions, pen marks, and artifacts from the WSIs, ensuring that only tissue areas are retained for subsequent analysis. Then, patches measuring 256 × 256 pixels are extracted from the tissue regions at 40× magnification without overlap, and the corresponding 256 × 256 pixels patches are then extracted from the same regions at 20× and 10× magnifications, centered on the 40× magnification images, as illustrated in Fig. 1. More Details can be found in Supplementary Text 1. It is important to note that any patch containing less than 80% of a specific tissue category is excluded from the dataset.

Fig. 1.

Fig. 1

Collection of WSI patches. (A) Detailed extraction of patches from H&E-stained images; (B) Visualization of image patches at various magnification levels.

Finally, the resulting dataset comprises 98,037 patches at 40× magnification, with an equivalent number of patches extracted at 20× and 10× magnifications. Taking the tissue types at 40× magnification as an example, Table 1 summarizes the number of image patches per tissue category, including viable tumor, stroma, and necrosis & keratinization, across the training, validation, and testing sets.

Table 1.

Dataset statistics for different tissue types at 40× magnification.

Dataset Viable tumor Stroma Necrosis &
Keratinization
Total Patches
Training set 21,325 38,713 3991 64,029
Validation set 5410 9,621 977 16,008
Testing set 6000 6000 6000 18,000

Model construction

By introducing the self-supervised learning strategy, the proposed Transformer-based model, MMT-Net, can effectively extract pathology-specific features from multi-magnification tissue samples to enhance the classification performance of viable tumor, stroma, and necrosis & keratinization, and thereby improving MPR estimation. The workflow of MMT-Net comprises three stages: feature extraction, feature fusion, and classification, as illustrated in Fig. 2.

Fig. 2.

Fig. 2

The workflow of MMT-Net for classifying viable tumor, stroma, and necrosis & keratinization involves three stages. In Stage 1, the SimCLR-based CNN backbone is trained to extract high-quality features from pathological images. Stage 2 implements a multi-attention feature fusion mechanism to integrate features from 10× and 40× magnifications. In Stage 3, the fused features are fed into a Transformer layer for classification. The performance of MMT-Net model is optimized throughout the workflow using the focal loss function during gradient backpropagation. Each stage builds upon the previous one to ensure robust and accurate predictions.

Stage 1: feature extraction

In this stage, we train the CNN backbone with SimCLR in a self-supervised way to extract high-quality features from pathological patches without using labels, enabling it to serve as a general feature extractor for downstream tasks. For each input image Inline graphic, we apply two random data augmentations Inline graphic and Inline graphic to obtain two correlated views Inline graphic and Inline graphic. These are fed into a shared encoder network Inline graphic, such as ResNet18, to generate intermediate representations Inline graphic and Inline graphic. A projection head Inline graphic then maps these representations into a contrastive embedding space, resulting in Inline graphic and Inline graphic. We randomly sample a minibatch of Inline graphic original images, yielding Inline graphic augmented samples. Each positive pair Inline graphic shares the same original image, while the remaining Inline graphic samples are treated as negative examples. The cosine similarity between vectors Inline graphic and Inline graphic is defined as Inline graphic, and the contrastive loss for a positive pair Inline graphic is computed as:

graphic file with name d33e532.gif 1

where Inline graphic is the temperature parameter, and Inline graphic is an indicator function to exclude the sample itself. The final loss is averaged across all positive pairs Inline graphic and Inline graphic in the minibatch:

graphic file with name d33e554.gif 2

This stage enables the pre-defined CNN backbone to learn the effective feature representation from unannotated pathological images, which is then used for downstream pathological image analysis tasks. Detailed descriptions of the training process are provided in Supplementary Text 2.

Stage 2: feature fusion

In this stage, we present a multi-attention feature fusion mechanism to fusion the multi-magnification features extracted from CNN backbone. Detailed of the multi-head attention mechanism is designed as follows:

Firstly, the 40× features are used as the input matrix Inline graphic and Inline graphic, while the 10× features serve as the input matrix Inline graphic. This asymmetric design emphasizes the high-resolution 40× features to guide attention calculation, capturing fine structural details, while the 10× features provide complementary semantic information. Such a setup effectively balances detailed structure and global context for improved feature fusion. For each attention Inline graphicits output is computed as:

graphic file with name d33e582.gif 3

where Inline graphic, Inline graphic and Inline graphic are the query, key, and value matrices, and Inline graphic, Inline graphic, and Inline graphic are the learnable weight matrices for the Inline graphic-th head. The output of the multi-head attention is calculated as follows:

graphic file with name d33e617.gif 4

where Inline graphicis a learnable weight matrix for the concatenated output, and Inline graphicdenotes the total number of attention heads. This mechanism allows the model to focus on multiple aspects of the input representations simultaneously.

Stage 3: classification

In this stage, the fused features are input into the transformer layer for classification. The transformer layer consists of multiple transformer blocks and a non-linear multilayer perception (MLP) connection layer. Each transformer block includes a multi-head attention layer, a feed-forward network consisting of two fully connected layers with a ReLU activation function, and two residual connections. To mitigate the impact of dataset imbalance on model training, the focal loss function is utilized, defined as follows:

graphic file with name d33e635.gif 5

where Inline graphic is the predicted probability for the target class, Inline graphic is a weighting factor that balances the importance of positive and negative samples, and Inline graphic is a focusing parameter that reduces the loss contribution from well-classified examples. Details about training process are provided in the Supplementary Text 3.

The output from the transformer layer is designated as the final output of proposed MMT-Net, which classifies the tissue into viable tumor, stroma, and necrosis & keratinization categories, to help pathologist estimate MPR.

Model evaluation

To evaluate the performance of the proposed model in classifying viable tumor, stroma, and necrosis & keratinization, metrics such as accuracy, F1-score are adopted. We also plot the receiver operating characteristic (ROC) curve, the macro-average area under the ROC curve (macro-AUC) is calculated to provide an overall measure of the model performance, as detailed in the Supplementary Text 4. Additionally, to gain insight into the model training, the gradient-weighted class activation mapping (Grad-CAM) technique17 is employed to generate classification heatmaps, highlighting the regions that most significantly influenced the model predictions.

Results

Highest accuracies in classification of tumor sub-regions by MMT-Net model

To evaluate the performance of MMT-Net on the classification of viable tumor, stroma, and necrosis & keratinization, we conduct the comparative experiments between MMT-Net and four commonly used models, including Ensemble learning-based method18, MSDNet19, Res2Net20, and MSBP-Net21, as presented in Table 2 and supplementary Table2. Detailed descriptions of these model constructions and implementation are provided in the Supplementary Text 5 and Supplementary Text 6. Here, two training strategies are conducted in each model: the first involves training directly with initialized weights, referred to as “without pre-training” and the second involves fine-tuning models initially pre-trained on ImageNet, referred to as “with pre-training”. Especially, the proposed MMT-Net pre-trained using self-supervised learning framework on pathological images is also conducted for comparison. SimCLR was selected for MMT-Net based on its superior performance compared to alternative self-supervised approaches, including SimSiam and MoCo-v3, as demonstrated in supplementary Table 3. On the one hand, when adopting the “without pre-training” strategy, the proposed MMT-Net achieves the highest accuracy and F1-score among the five models, with values of 97.15% and 97.17%, respectively. This performance surpasses existing methods, showcasing the advantages of the MMT-Net model architecture, particularly its multi-magnification feature fusion design and exceptional adaptability to data distribution. On the other hand, when employing the “with pre-training” strategy, all methods exhibit improved accuracy and F1-scores compared with “without pre-training”, underscoring the critical importance of pre-training in classification tasks. Specifically, MMT-Net fine-tuned with pathological images, as opposed to ImageNet, achieves an accuracy increase from 98.99% to 99.21% and an F1-score increase from 98.99% to 99.21%. Similarly, the second-best model, Ensemble learning-based method, shows an accuracy increase from 98.43% to 99.04% and an F1-score rise from 98.44% to 99.04%, as detailed in the Supplementary Table 4. The ResNet50 model demonstrates the lowest performance among the evaluated models both employing “without pre-training” and “with pre-training”, suggesting pretrained models on natural images may not be directly suitable for capturing the complex features present in histopathology data. These results collectively underscore the superior effectiveness of the proposed pre-training strategy on pathological images in capturing more pertinent domain-specific features compared to natural images.

Table 2.

Performance comparison of MMT-net and other multi-scale models.

Method Without Pre-training With Pre-training
Accuracy (%) F1-score (%) Accuracy (%) F1-score (%)
ResNet50 92.90 93.01 98.06 98.07
MSDNet 94.59 94.66 97.47 97.49
MSBP-Net 92.68 92.78 96.53 96.56
Res2Net 91.78 91.92 97.29 97.31
Ensemble learning-based method 96.29 96.32 98.43 98.44
MMT-Net 97.15 97.17 98.99 98.99
MMT-Net (SimCLR) — — 99.21(11) 99.21(20)

* The values in parentheses (11) and (20) represent the last two digits after the decimal point of the accuracy and F1-score values (e.g., 99.2111% and 99.2120%).

Furthermore, ROC curves for each model are plotted using “without pre-training” and “with pre-training” strategy, as shown in the Supplementary Fig. 1 and Supplementary Fig. 2. The proposed MMT-Net consistently achieves the highest macro-AUC value among the five models. These results indicate that the proposed MMT-Net model, integrated with the pre-training strategy, achieves the best performance in classifying viable tumor, stroma, and necrosis & keratinization.

MMT-Net provides comparable MPR estimation to pathologists for NSCLC patients with post-NAT

To demonstrate the performance of MMT-Net in practical MPR estimation, we apply it to evaluate the response of NSCLC patients with post-NAT, comparing the results with those of experienced pathologists. Four representative cases, featuring MPR values ranging from high to low, are selected for evaluation. The first two cases exhibit MPR values greater than 10%, while the latter two cases have MPR values below 10%, as displayed in Fig. 3A. Fig.3B gives the panoramic classification of four representative cases using the proposed MMT-Net, with red indicating viable tumor, green indicating stroma, and yellow indicating necrosis & keratinization, respectively. These classification results are converted into mask images and annotated using an external contour method, as demonstrated in the Supplementary Fig. 3. It is observed that these annotations are compatible with the Motic Digital Slide Assistant Lite software, suggesting the proposed MMT-Net can aid pathologists in diagnosing and quantitatively assessing the NAT response in NSCLC cases.

Fig. 3.

Fig. 3

WSI-level panoramic classification map. (A) Original WSIs; (B) Classification maps are constructed based on the predictions of MMT-Net for the corresponding patches; (C) Pie charts quantitatively display the percentage of different categories within each WSI.

In addition, pie charts are utilized to represent the proportions of each category (Fig. 3C). These visualizations offer an intuitive understanding of the distribution of viable tumor, stroma, and necrosis & keratinization across the selected cases, emphasizing the variability in tissue composition and aiding in the comprehensive evaluation of tumor response post-NAT. Additional cases are available in the Supplementary Fig. 4. These results demonstrate the reliability of MMT-Net model in classifying distinct sub-regional tissues and capturing intra-tumoral heterogeneity, thereby providing critical support for accurate MPR estimation in clinical applications.

MMT-Net focuses on similar regions as pathologists in MPR estimation

Deep learning models are frequently criticized for their lack of interpretability22. To address this issue, we employ the Grad-CAM technique to analysis the features extracted by our model. This method can highlight the focal points of the proposed MMT-Net when processing pathological image patches. Fig. 4 displays the regions of interest at 40× and 10× magnifications for predictions of viable tumor, stroma, and necrosis & keratinization. In the Grad-CAM heatmaps, darker regions indicate higher model attention. Remarkably, the areas emphasized by the model closely correspond to the regions that pathologists examine during diagnosis. The significant overlap underscores the notable alignment between the prediction of model and the evaluations of expert pathologists, providing supportive evidence for the consistency and reliability of MMT-Net outputs.

Fig. 4.

Fig. 4

Visualization of image patches by the MMT-Net model. (A) Viable tumor area; (B) Stroma area; (C) Necrosis & keratinization area.

Strong agreement between MMT-Net predictions and immunohistochemistry results

Cytokeratins (CK) are intermediate filament proteins present in epithelial cells, extensively utilized in pathology to identify epithelial-origin tumors via immunohistochemical staining23. Different types of CK, such as CK7, CK20, and CK5/6, exhibit distinct expression patterns across various cells and tissues, making them crucial for tumor origin identification and classification. Ki67, a nuclear protein expressed during cell proliferation (in G1, S, G2, and M phases), is a widely recognized marker for assessing cell proliferation activity. Its expression levels are commonly used in tumor pathology to evaluate the growth rate and prognosis of cancers24. In this study, we selected WSIs from the testing set of NAT-treated NSCLC patients and performed staining using CK and Ki67. The immunohistochemical results are subsequently compared with the prediction of MMT-Net model. As illustrated in Fig. 5, both global and local views showed a high degree of concordance between the model predictions and the actual staining results.

Fig. 5.

Fig. 5

Results of Ki67 and CK staining. (A) Original WSI. (B)The MMT-Net model predicted the classification map. (C, D) Ki67 and CK stained WSI; (E, F, G) Details of the original image and stained images at the local level.

To further quantify this alignment, we calculate the Pearson correlation coefficients between the model predictions and CK, Ki67 stained results. The coefficients are 0.8837 and 0.8754 with a p-valueInline graphic0.001, respectively, indicating a statistically significant correlation between the model prediction and immunohistochemistry results, as detailed in the Supplementary Text 7 and Supplementary Fig. 5. This strong alignment further illustrate the proposed MMT-Net model in assessing NAT-treated NSCLC, highlighting its potential for effective clinical use in diagnostics.

Discussion

Although numerous studies have reported on AI-assisted WSI analysis methods for identifying tumor areas25,26, cancer subtypes27,28, microenvironment analysis29,30, and lymph node metastasis detection31,32, there is a relative paucity of research focused on MPR estimation in NSCLC patients with post-NAT. The WSI analysis in this field presents unique challenges due to extensive therapy-induced pathological changes such as necrosis, fibrosis, and inflammation, which can obscure residual tumor cells. Additionally, NSCLC demonstrates a highly heterogeneous response to NAT, with viable residual tumor regions often interspersed with therapy-affected areas, complicating precise localization. The morphological characteristics of residual tumor cells are often subtle and difficult to distinguish from stromal cells or treatment artifacts, posing significant challenges to existing AI-assisted WSI analysis methods in accurately estimating MPR of NSCLC patients with post-NAT.

In this study, we propose an effective AI-assisted WSI analysis model, called MMT-Net to estimate the MPR of NSCLC patients with post-NAT. Many results show its better performance on the classification of active tumor areas, stroma, and necrotic & keratinized areas than the existing methods. The designed network architecture and training strategy support the better performance of MMT-Net. On the one hand, a self-supervised training strategy using the SimCLR framework is introduced to provide effective initialization parameters for MMT-Net, facilitating downstream pathological image analysis tasks. As shown in the Supplementary Fig. 6, Supplementary Fig. 7 and Supplementary Table 3, the feature extractor trained with SimCLR outperformed other transfer learning methods in both linear evaluation and fine-tuning tasks, demonstrating the effectiveness of self-supervised strategy we adopted in enhancing predictive performance for domain-specific datasets, such as post-NAT NSCLC. On the other hand, the Transformer-based multi-magnification network architecture is designed to extract pathology-specific features at various magnifications, capturing global pathological changes at low magnification and fine cellular details at high magnification. By leveraging a multi-attention mechanism, the model effectively combines both micro and macro information, enabling precise identification of residual tumor cells and their relationships with surrounding tissues, thereby enhancing MPR estimation. Supplementary Table 5 presents the ablation studies evaluating magnification settings and fusion strategies. These results confirm that both strategies enhance the performance of the MMT-Net model, establishing it as an effective analytical tool capable of providing treatment recommendations for NSCLC patients with post-NAT.

Our study also demonstrates a strong correlation between MMT-Net predictions and immunohistochemistry results, underscoring the reliability of MMT-Net in practical clinical research. Additionally, we investigate the capability of MMT-Net to highlight high-attention regions within pathological images. Notably, the predictions of MMT-Net closely align with the diagnostic judgments of experienced pathologists, suggesting that MMT-Net can effectively emulate and learn the decision-making processes of pathologists. By mapping the model prediction results for WSI back to the original images, pathologists can achieve results comparable to manual annotations within the corresponding software. Moreover, a comparison of these prediction results with the staining results for CK and Ki67 revealed a high degree of consistency. This finding further validates the accuracy and efficacy of the MMT-Net model, aligning with the immunohistochemistry results.

There are, however, the dataset constructed is still relatively limited in size because of strict inclusion criteria that lead to rare clinical occurrences. Although self-supervised learning strategy is applied to unannotated data have partially mitigated the lack of dataset, further validation across larger, multicenter, multi-institutional datasets and external patient cohorts is warranted to further enhance the generalizability and clinical applicability of proposed MMN-Net.

In summary, we present an effective AI-assisted WSI analysis method, MMT-Net, a Transformer-based model utilizing self-supervised learning strategy to improve the estimation of MPR in NSCLC patients after receiving NAT. Crucially, the model is architecture-agnostic, requiring no task-specific assumptions. Its core components are readily transferable to related computational pathology applications, such as tumor-stroma analysis, necrosis quantification, or multi-tissue classification across diverse cancer types. Our study offers valuable insights and tools for the diagnosis and treatment of NSCLC, potentially advancing the application of NAT and improving the accuracy of pathological evaluations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary Material 1 (1.3MB, docx)
Supplementary Material 2 (1.7MB, docx)

Author contributions

C.Y., L.G., and C.L. designed the study, analyzed the clinicopathological data, and performed the histopathological evaluation of the therapeutic effect. L.L., and F.W. annotated the WSIs, which were confirmed by C.L., W.X., and C.Y. constructed the DL model, and helped with the data analysis. All authors reviewed the paper and approved the final draft of the manuscript.

Funding

This work was supported by the Natural Science Foundation of Fujian Province (2021J01432), the Joint Funds for the Innovation of Science and Technology, Fujian Province (2021Y9212), and the National Natural Science Foundation of China (22404024). We thank Fujian Provincial Cancer Hospital for providing the data, and the Interdisciplinary Institute for Medical Engineering at Fuzhou University for their technical support.

Data availability

The data presented in this study are available from the corresponding author on reasonable request.

Declarations

Competing interests

The authors declare no competing interests.

Ethics approval

This study received approval from the Institutional Ethics Committee of Fujian Cancer Hospital (SQ2021-070-01). All experiments were performed in accordance with relevant guidelines and regulations (e.g., Declaration of Helsinki). Informed consent was obtained from all participants, either in written form or through verbal notification, and all identifying information was removed from the data.

Footnotes

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Contributor Information

Lei Guo, Email: gankLei@fzu.edu.cn.

Chao Li, Email: lichao3501@fjmu.edu.cn.

References

  • 1.Hellmann, M. D. et al. Pathological response after neoadjuvant chemotherapy in resectable non-small-cell lung cancers: Proposal for the use of major pathological response as a surrogate endpoint. Lancet Oncol.15, e42-50 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.William, W. N. et al. Computed tomography RECIST assessment of histopathologic response and prediction of survival in patients with resectable non–small-cell lung cancer after neoadjuvant chemotherapy. J. Thorac. Oncol.8, 222–8 (2013). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Pataer, A. et al. Histopathologic response criteria predict survival of patients with resected lung cancer after neoadjuvant chemotherapy. J. Thorac. Oncol.7, 825–32 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Travis, W. D. et al. IASLC Multidisciplinary Recommendations for Pathologic Assessment of Lung Cancer Resection Specimens After Neoadjuvant Therapy. J. Thorac. Oncol.15, 709–740 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Cheng, N. et al. Deep learning-based classification of hepatocellular nodular lesions on whole-slide histopathologic images. Gastroenterology162, 1948-1961.e7 (2022). [DOI] [PubMed] [Google Scholar]
  • 6.Kather, J. N. et al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS Med.16, e1002730 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Coudray, N. et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med.24, 1559–67 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Din, N. M. U., Dar, R. A., Rasool, M. & Assad, A. Breast cancer detection using deep learning: Datasets, methods, and challenges ahead. Comput. Biol. Med.149, 106073 (2022). [DOI] [PubMed] [Google Scholar]
  • 9.Deng, J. et al. ImageNet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. pp. 248–55. (2009).
  • 10.Wu, W., Gao, C., DiPalma, J., Vosoughi, S. & Hassanpour, S. Improving Representation Learning for Histopathologic Images with Cluster Constraints. In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 21347–57. (2023). [DOI] [PMC free article] [PubMed]
  • 11.Chen, R. J. & Krishnan, R. G. Self-Supervised Vision Transformers Learn Visual Concepts in Histopathology. (2022).
  • 12.Chen, R. J. et al. Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning. (2022).
  • 13.Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. A Simple Framework for Contrastive Learning of Visual Representations. (2020).
  • 14.Vaswani, A. et al. Attention Is All You Need. (2023).
  • 15.Dosovitskiy, A. et al. An Image is Worth 16x16 Words (Transformers for Image Recognition at Scale, 2021).
  • 16.Otsu, N. A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern.9, 62–66 (1979). [Google Scholar]
  • 17.Selvaraju, R. R. et al. Grad-CAM: Visual explanations from deep networks via gradient-based localization. Int. J. Comput. Vis.128, 336–59 (2020). [Google Scholar]
  • 18.Terada, K. et al. Deep Learning for Predicting Effect of Neoadjuvant Therapies in Non-small Cell Lung Carcinomas With Histologic Images. Mod. Pathol.36, 100302 (2023). [DOI] [PubMed] [Google Scholar]
  • 19.Huang, G., Chen, D., Li, T., Wu, F. & van der Maaten, L. Weinberger KQ. Multi-Scale Dense Networks for Resource Efficient Image Classification. (2018).
  • 20.Gao, S.-H. et al. Res2Net: A new multi-scale backbone architecture. IEEE Trans. Pattern Anal. Mach. Intell.43, 652–62 (2021). [DOI] [PubMed] [Google Scholar]
  • 21.Vuong, T. T. L., Song, B., Kim, K., Cho, Y. M. & Kwak, J. T. Multi-scale binary pattern encoding network for cancer classification in pathology images. IEEE J. Biomed. Health Inform.26, 1152–63 (2022). [DOI] [PubMed] [Google Scholar]
  • 22.Li, X. et al. Interpretable deep learning: Interpretation, interpretability, trustworthiness, and beyond. Knowl. Inf. Syst.64, 3197–234 (2022). [Google Scholar]
  • 23.Moll, R., Divo, M. & Langbein, L. The human keratins: Biology and pathology. Histochem. Cell Biol.129, 705–733 (2008). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Menon, S. S., Guruvayoorappan, C., Sakthivel, K. M. & Rasmi, R. R. Ki-67 protein as a tumour proliferation marker. Clin. Chim. Acta.491, 39–45 (2019). [DOI] [PubMed] [Google Scholar]
  • 25.Wang, S. et al. Comprehensive analysis of lung cancer pathology images to discover tumor shape and boundary features that predict survival outcome. Sci. Rep.8, 10393 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Wang, S. et al. ConvPath: A software tool for lung adenocarcinoma digital pathological image analysis aided by a convolutional neural network. EBioMedicine50, 103–10 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Wang, X. et al. Weakly supervised deep learning for whole slide lung cancer image analysis. IEEE Trans. Cybern.50, 3950–62 (2020). [DOI] [PubMed] [Google Scholar]
  • 28.Cao, L. et al. E2EFP-MIL: End-to-end and high-generalizability weakly supervised deep convolutional network for lung cancer classification from whole slide image. Med. Image Anal.88, 102837 (2023). [DOI] [PubMed] [Google Scholar]
  • 29.Albusayli, R. et al. Artificial intelligence-based digital scores of stromal tumour-infiltrating lymphocytes and tumour-associated stroma predict disease-specific survival in triple-negative breast cancer. J. Pathol.260, 32–42 (2023). [DOI] [PubMed] [Google Scholar]
  • 30.Shaban, M. et al. A digital score of tumour-associated stroma infiltrating lymphocytes predicts survival in head and neck squamous cell carcinoma. J. Pathol.256, 174–85 (2022). [DOI] [PubMed] [Google Scholar]
  • 31.Ehteshami Bejnordi, B. et al. Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer. JAMA318, 2199–2210 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Chuang, W.-Y. et al. Identification of nodal micrometastasis in colorectal cancer using deep learning on annotation-free whole-slide images. Mod. Pathol.34, 1901–11 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary Material 1 (1.3MB, docx)
Supplementary Material 2 (1.7MB, docx)

Data Availability Statement

The data presented in this study are available from the corresponding author on reasonable request.


Articles from Scientific Reports are provided here courtesy of Nature Publishing Group

RESOURCES