Advances in the use of Radiomics and Pathomics for predicting the efficacy of neoadjuvant therapy in tumors

Highlights

• •Currently, biomarkers for early assessment of neoadjuvant response are lacking.
• •Radiomics and pathomics offer new ways to refine neoadjuvant therapy strategies.
• •The integration of multi-omics represents an emerging area of research.
• •Challenges such as lack of interpretability and poor generalizability remain.

Abstract

Neoadjuvant therapy is widely used for treating malignant tumors, but its efficacy varies among patients. Currently, tools or biomarkers for early and accurate evaluation of the efficacy of neoadjuvant therapy are lacking. The advent of radiomics and pathomics offers new avenues for refining neoadjuvant therapy strategies and could provide high-performance predictive tools. The integration of multi-omics represents an emerging area of research. The introduction of radiopathomics offers innovative approaches to studying the efficacy of neoadjuvant therapy. This article reviews the current developments in multi-omics integration, the advances in the use of radiopathomics to predict the efficacy of neoadjuvant therapy, and the challenges faced by ongoing research.

Graphical abstract

Introduction

Neoadjuvant therapy is a focal point in the comprehensive treatment of malignant tumors. It can shrink tumors, reduce their stage, and improve the survival rate of patients. It is increasingly used for various tumors, including those of the breast, gastric, colorectal, lung, and head and neck [[1], [2], [3], [4], [5]]. Neoadjuvant therapy followed by radical surgery has become the standard treatment for many tumors. Pathologic complete response (pCR) serves as the primary endpoint of neoadjuvant therapy trials and a surrogate marker for disease-free survival (DFS) and overall survival (OS) [6]. Patients achieving pCR often choose more conservative surgical plans to preserve organ function and improve quality of life [3]. However, due to tumor heterogeneity and complexity, not all patients benefit. Those who do not respond well may suffer side effects without any therapeutic benefit, potentially experience disease progression during treatment, miss optimal surgical timings, and face a higher risk of recurrence and lower survival rates [6].

Currently, histopathologic examination of surgical specimens remains the gold standard for efficacy assessment, but results are only available after the completion of all preoperative neoadjuvant therapies and surgery, and do not guide clinical adjustment of treatment strategies. Therefore, the search for early predictive biomarkers is critical [7]. Presently, imaging examinations such as computed tomography (CT) and magnetic resonance imaging (MRI) are commonly used to predict responses to neoadjuvant therapy. Dynamic monitoring of changes in tumor size is possible, but imaging after therapy may show unchanged or increased tumor sizes due to tissue fibrosis or immune cell infiltration, which are insufficient for accurate response evaluation [4]. Increasingly, research is focusing on identifying biomarkers for neoadjuvant therapy, including tumor markers such as carcinoembryonic antigen and tumor microenvironment markers such as epidermal growth factor receptor (EGFR) and vascular endothelial growth factor (VEGF) [8]. With the rapid advancement of neoadjuvant immunotherapy, biomarkers such as programmed cell death ligand 1 (PD-L1), tumor mutational burden, tumor-infiltrating lymphocytes (TILs), and inflammatory cytokines have also been shown to correlate with responses to neoadjuvant immunotherapy [4]. However, the methods for analyzing these markers are time-consuming and costly, highlighting the need for the development of new predictive tools [9].

Advances in radiomics and pathomics are opening new avenues for the development of neoadjuvant therapeutic strategies, which could emerge as high-performance predictive tools. Radiomics involves extracting quantitative features from medical images to characterize tumors at a macroscopic level [10]. Pathomics focuses on deriving features from digitized pathology slides, providing microscopic details of the tumor [11]. However, relying solely on single-modal data limits the analysis to one perspective; therefore, integrating multimodal data in multi-omics studies is a growing area of research [12]. Radiopathomics, which combines macroscopic and microscopic features, offers new insights for predicting the efficacy of neoadjuvant therapy.

This review aims to discuss the research progress and controversies surrounding radiomics and pathomics in predicting neoadjuvant therapy efficacy, explore the current state of multi-omics integration, examine the application of radiomics and pathomics in predicting neoadjuvant therapy outcomes, and address the challenges and future directions of this research field.

Radiomics

Overview of Radiomics

Radiomics refers to the high-throughput extraction of high-dimensional quantitative data from medical images, using techniques such as machine learning to identify the most clinically relevant features. These features can capture intra-tumor heterogeneity and provide insights into the tumor microenvironment [10]. Numerous studies have developed radiomics-based predictive models targeting various clinical endpoints including cancer diagnosis and staging, survival, recurrence, lymph node metastasis, distant metastasis, and complications [[13], [14], [15], [16], [17]]. These models have demonstrated strong performance, offering the potential for accurate and efficient non-invasive tools that facilitate personalized treatment and precision medicine.

The radiomics workflow generally includes the following steps: image acquisition and preprocessing; image segmentation, focusing on the region of interest (ROI) or volume of interest (VOI); feature extraction; feature selection; model construction; and performance evaluation and model validation, using methods such as decision curve analysis (DCA), and the Hosmer–Lemeshow test, and metrics such as the area under the curve (AUC), sensitivity, and specificity [[18], [19], [20]].

Radiomics for predicting the efficacy of neoadjuvant therapy

Research status

With the rapid advancement of radiomics, numerous studies have been conducted to predict the efficacy of neoadjuvant therapy using radiomics-based models. Most of these studies have focused on tumors such as esophageal cancer [21]. rectal cancer [3], breast cancer [1], gastric cancer [2], and lung cancer [4].

There are two primary strategies for developing these models. The first approach is to construct a model that directly predicts prognosis using radiomics. Sun et al. developed an MRI-based pre-treatment radiomics model to predict the response to neoadjuvant chemotherapy in patients with locally advanced cervical cancer [22].

The second strategy involves using radiomics as an alternative to biomarkers by predicting biomarker expression through radiomic features. This approach not only enables non-invasive biomarker assessment but also provides a visual correlation between imaging data and molecular markers. Liu et al. demonstrated that the tumor-to-stromal ratio (TSR) correlates with the response of bladder cancer to neoadjuvant chemotherapy. Based on this finding, they developed a CT-based radiomics model to predict TSR; it showed strong predictive performance, offering a non-invasive method for TSR evaluation [23]. Li et al. developed a deep learning radiomics nomograms to predict isocitrate dehydrogenase genotypes in brain glioma [24]. Zhu et al. pioneered a method using radiomics to predict the expression of CTLA4, a gene closely related to targeted therapy in head and neck squamous cell carcinoma [25]. (Fig. 1)

Fig. 1.

Methods for prognosis assessment and strategies for constructing radiomics or pathomics models: (A) Imaging examinations are commonly used to predict responses to neoadjuvant therapy and can dynamically monitor changes in tumor size, but the evaluation is inaccurate. Histopathologic examination is the gold standard for efficacy assessment, but the results are only available after the completion of all preoperative neoadjuvant therapies and surgery. Biomarkers have also been proven to be related to the prognosis of tumors. (B) Radiomics features or pathomics features can be analyzed based on medical images or whole-slide imagings to construct radiomics models or pathomics models. (C) Models for predicting prognosis can be directly constructed through radiomics or pathomics. (D) Models can be constructed through radiomics or pathomics to predict the expression of biomarkers that have been proven to be related to prognosis.

Despite the large body of research in this area, there remains no clear consensus on several important issues. In this section, we will focus our discussion on three key aspects: the imaging modality, the timing of imaging examinations, and the selection of ROIs.

Imaging modalities

The imaging modalities used in radiomics include CT, PET, MRI, and US, with each modality offering distinct advantages.

CT is a widely accessible, time-efficient, and cost-effective option. Its radiomics features are known for being highly reproducible and robust [19]. PET provides insight into the functional and biochemical changes that often precede anatomical changes [26]. However, PET radiomics features can be influenced by variations in reconstruction parameters, which can affect reliability [19]. US, a real-time imaging technique, is commonly used for characterizing breast lesions [27]. However, the high operator variability of US leads to reduced reliability and reproducibility [19].

MRI, a radiation-free modality, offers high soft-tissue contrast resolution [28]. It is also capable of functional imaging at high resolution, including via techniques such as diffusion-weighted imaging (DWI) and dynamic contrast-enhanced MRI (DCE-MRI) [19]. Different MRI sequences capture various functional states of tumors, making multiparametric MRI (mpMRI)-based radiomics more capable of reflecting tumor heterogeneity [29]. Nie et al. developed both single- and multiparameter models for predicting pCR after neoadjuvant radiotherapy for rectal cancer, using preoperative T1/T2 anatomical imaging, DWI, and DCE. Their results showed that the multiparameter model outperformed the single-parameter model [3].

In the era of precision medicine, relying on a single imaging modality is insufficient to meet the demands of personalized treatment. Comprehensive assessments of tumor biology require information from multiple imaging techniques, which is why multimodal models hold great promise. Qi et al. developed a radiomics model based on ¹⁸F-FDG PET, enhanced CT, and clinical features, achieving superior performance in predicting pCR in esophageal cancer patients undergoing neoadjuvant radiochemotherapy. The fusion of multimodal radiomics features significantly enhanced prediction performance compared to single-modality models [30].

Timing of imaging examination

Since pretreatment images can capture the heterogeneity of the primary tumor, most radiomics studies have focused on pretreatment images for research. While combining multisequence and multimodal data has improved the predictive performance of radiomics models, pretreatment images alone are limited to reflecting the characteristics of the primary tumor. However, tumor heterogeneity evolves dynamically throughout treatment, and post-treatment images can more directly reflect the pathological remission status. Therefore, combining both pre- and post-treatment imaging data, which captures the entire diagnostic and therapeutic process, may further enhance predictive accuracy [7]. Wei et al. constructed a model to predict lymph node metastasis based on mpMRI images taken before and after neoadjuvant therapy in rectal cancer patients. Their combined sequence model, modelpre_T2_DWI_post, outperformed both modelpre_T2_DWI and all single-sequence models [15].

Delta radiomics presents a new approach by constructing models based on changes in radiomics features during treatment [31]. Lu et al. created MRI-based pre-treatment, post-treatment, and delta radiomics models, with the delta model achieving the highest performance in predicting the pathological response in esophageal cancer; the pre-treatment model performed the worst [32]. The advantages of delta radiomics have also been shown in studies on osteosarcoma and gastric cancer [31,33].

Pre-treatment, post-treatment, and delta radiomics features each represent different aspects of tumor biology, including primary tumor characteristics, pathological remission status, and treatment sensitivity. Peng et al. have attempted to integrate features from all these stages. The results indicated that combining information from multiple time points enables a more thorough evaluation of tumor treatment response [8].

Region of interests

Most studies focus on the primary tumor, making the intratumoral region the most common choice. However, as tumor biology is explored more deeply, increasing attention is being given to the peritumoral region. Liu et al. extracted radiomics features from both the intratumoral and peritumoral regions of gastric cancer and developed separate models: an intratumor model, a peritumor model, and a combined model. Their findings revealed that peritumoral models provide added value in predicting pathological responses to neoadjuvant chemotherapy in gastric cancer. The tumor immune microenvironment plays a crucial role in tumor progression, metastasis, and treatment efficacy, and peritumoral lymphocytic infiltration in gastric cancer has been significantly associated with the prognosis and response to chemotherapy [2]. Sun et al. reported similar findings in their study of cervical cancer [22].

Additionally, some studies have expanded the scope of ROIs to include regions such as rectal mesenteric fat [34], lymph nodes [35], and hemodynamic features [36]. Including these additional regions has helped to improve the predictive efficacy of radiomics models by providing a more comprehensive analysis of tumor and surrounding tissue characteristics.

Pathomics

Overview of pathomics

In clinical practice, histological examination of tumor specimens is regarded as the definitive standard for tumor diagnosis. Evaluation of hematoxylin and eosin (H&E)-stained sections by a pathologist is essential for determining the TNM stage and histological classification of tumors. Additionally, tumor pathology reflects the heterogeneous characteristics of the tumor microenvironment and has been shown to be predictive of prognosis [37]. However, traditional histopathology relies on manual quantification and assessment, which can be inefficient and heavily dependent on the expertise of the pathologist [38]. With advancements in digital slide scanning technology and the decreasing cost of digital storage, whole-slide imaging (WSI) has rapidly developed, bringing increased attention to the concept of “pathomics.” [39]

Pathomics applies artificial intelligence to extract large-scale, quantifiable data from digital pathology images and analyze them to obtain valuable insights [11]. The workflow of pathomics is similar to that of radiomics. Previous studies have demonstrated that pathomics, in combination with artificial intelligence, can be used to predict lymph node metastasis [40], tumor staging [41], biomarkers including BRAF mutations [42], microsatellite instability (MSI) [43], and the malignancy of diseases such as oral leukoplakia [44]. Pathomics is also a powerful tool for predicting the prognosis of malignant tumors, with potential for developing clinical treatment strategies and advancing personalized medicine. Studies have attempted to develop pathomics models to predict survival [45], treatment response [46], and recurrence [47] in malignant tumors, with promising results.(Fig. 1)

Pathomics for predicting the efficacy of neoadjuvant therapy

Pathomics has made notable progress in predicting treatment efficacy. Jiang et al. conducted a retrospective analysis of stage III colon cancer patients who received chemotherapy and developed a pathomics model. The results showed that the pathomics signature was independently associated with DFS and OS [45]. Similarly, Han et al. created pathomics models to predict the response to immunotherapy in gastric cancer patients. The model effectively stratified responses to immune checkpoint inhibitors in the training cohort and was validated in one internal and two external cohorts [48]. Pathomics models have also been developed to predict the efficacy of targeted therapy in ovarian cancer [46], radiotherapy in small cell lung cancer (SCLC) [38], and immunotherapy in esophageal squamous cell carcinoma [37], and they all show great promise.

Studies specifically focusing on pathomics for predicting the efficacy of neoadjuvant therapy are limited. Saednia et al. used quantitative digital histopathology and machine learning to predict the pCR to neoadjuvant chemotherapy in breast cancer patients. They applied a pre-trained weighted U-Net model to segment nuclei within tumor regions and extracted five subsets of pathomics features from the segmented samples. Models were built for each feature subset, with AUC values ranging from 0.67 to 0.87 [49]. Fisher et al. developed a machine-learning-based two-step pipeline to differentiate various histological components in WSI of breast cancer tissue biopsies. They identified histological features that predict the response to neoadjuvant chemotherapy, with tumor and TIL being strongly associated with pCR, and microvessel density (MVD) and polyploid giant cancer cells (PGCCs) being linked to residual disease [50]. These studies suggest the potential of pathomics features for predicting neoadjuvant therapy outcomes.

In addition to directly predicting treatment efficacy, pathomics can also serve as a tool for biomarker prediction [48], including MSI [43], CDKN2A expression [51], and TNFRSF4 expression [52]. Studies have shown that MSI plays an important role in predicting the efficacy of immunotherapy in advanced solid tumors. Cao et al. developed a deep learning model based on pathomics that efficiently predicted MSI from histopathological images and was transferable to new patient populations [43]. While biomarkers such as PD-L1 expression and tumor lymphocyte infiltration(4) have been extensively studied in the context of neoadjuvant therapy, the methods for detecting these biomarkers are often costly and complex. Pathomics, as a minimally invasive and efficient predictive method for biomarkers, is a promising direction for future research (Fig. 1).

Multi-omics integration

Research status

It has been shown that integrating multimodal data can complement tumor heterogeneity at multiple scales, significantly enhancing the predictive power of models compared to using unimodal data alone [12]. Currently, multi-omics research, which spans genomics, proteomics, transcriptomics, epigenomics, metabolomics, pathomics, and radiomics, is a major focus. Each of these “omics” disciplines contributes to a more comprehensive understanding of biological systems and diseases, revealing multiple levels of anatomical, molecular, and cellular interactions. This integrated approach lays the foundation for precision medicine and personalized healthcare strategies.

Radiomics reflects tumor characteristics at the macroscopic level but does not capture the biological nature of the tumor. Genomics, by contrast, reveals tumor heterogeneity more precisely at the molecular level, although it requires invasive methods for tissue sample collection. Combining radiomics and genomics ("radiogenomics") has the potential to advance precision medicine. Radiogenomics links radiomic features to genetic profiles, enabling more accurate tumor diagnosis and prognosis [36].

One application of radiogenomics is predicting gene mutation status through radiomics. Zhu et al. developed a radiomics model based on enhanced CT to predict CTLA4 expression in head and neck cancer [25]. By integrating radiomics and genomics, the radiogenomics model can serve as a valuable biomarker, improving predictive efficacy. Zhou et al. demonstrated this by integrating radiomics, genomics, and pathology features into a fusion model, which achieved an AUC of 0.93 in the validation cohort, significantly higher than that of the radiomics model alone [36]. Kazerooni et al. integrated MRI and RNA sequencing data to predict prognosis for low-grade gliomas in children, and they believe this study provides directions for identifying patients who might benefit from targeted therapies [53].

Additionally, radiogenomics can help decode the underlying biological mechanisms of radiomics models. Gene set enrichment analysis (GSEA) is commonly used for this purpose. Fan et al. developed radiomics features to predict the response to neoadjuvant chemotherapy in breast cancer, finding correlations with patient survival outcomes. Through gene expression and GSEA, they discovered that the IL-17 and estrogen signaling pathways were associated with treatment response, a finding consistent with known evidence [54].

The integration of pathomics with genomics or transcriptomics can also help us analyze the underlying biological mechanisms of cancer. As previously mentioned, Cao et al. developed a deep learning-based pathomics model to predict MSI and conducted genomic and transcriptomic association analyses. Their results demonstrated that pathomics features were strongly correlated with the expression of the IFN-γ-JAK-STAT1 signaling pathway, which plays a well-established role in immune activation and responses to immunotherapy. Additionally, transcriptomic profiling revealed a strong association between pathomics features and antitumor activity [43].

The convergence of radiomics and pathomics has also shown promise in predicting the efficacy of neoadjuvant therapy, which will be elaborated on in the next subsection.

The interconnections between radiomics, pathomics, and genomics have significantly deepened our understanding of cancer biology. Multi-omics fusion, which integrates multi-dimensional data, is the next step in advancing precision medicine, though research in this area remains limited. Vanguri et al. developed a multi-omics model to predict the immunotherapy response in non-SCLC (NSCLC) by integrating radiomics, pathomics, and genomics features. Their model achieved an AUC of 0.80, outperforming models based on any single variable [55]. The integration of imaging features, pathological features, and genetic data represents the future of cancer research, enabling a comprehensive understanding of cancer at both the spatial and molecular levels, uncovering the biological mechanisms behind imaging features, and identifying novel biomarkers. While some researchers have begun constructing multi-omics heterogeneous networks that encompass multimodal data, significant challenges remain, particularly in addressing the heterogeneity of multimodal data [56].

Radiopathomics

The strengths and weaknesses of radiomics and pathomics

Histological examination is generally considered the gold standard for the diagnosis of most solid tumors. Pathomics extracts quantitative features from pathological images, capturing microstructural information within the tumor region such as cellular and subcellular features, as well as microenvironmental characteristics. These features can directly reflect tumor malignancy and treatment sensitivity [57,58]. However, since pathomics typically relies on H&E-stained tissue sections from biopsy samples, there are limitations due to the inherent heterogeneity of tumors and the potential lack of representativeness in biopsy specimens [48].

In contrast, radiomics provides spatial macrostructural information about the tumor and surrounding tissues, capturing features such as shape, size, intensity distribution, density variations, and spatial relationships [12,57]. The strength of radiomics also lies in the fact that by constructing a multi-parameter model, it can combine the advantages of multiple imaging modes and capture richer information. However, radiomics features lack cytological detail [59]. Additionally, as previously mentioned, incorporating post-treatment images can improve the predictive efficacy of radiomics models. However, these models are based on imaging data collected after at least one round of neoadjuvant therapy, limiting their ability to guide early clinical decision-making and reducing their overall clinical practicality.

Radiopathomics combines radiomics and pathomics features to provide a more comprehensive view of both the macroscopic and microstructural characteristics of tumors. This combination enhances the predictive ability of the model and offers a deeper understanding of the disease.

Research status

Recently, several studies have constructed promising predictive models by fusing radiomics and pathomics.

In terms of prognostic prediction, Hu et al [60] and Zhou et al [61]created prediction models for lung adenocarcinoma and prostate cancer, respectively. Lymph node metastasis and distant metastasis represent significant prognostic factors affecting clinical outcomes. Specifically, Zhang et al [62] designed a model to predict bone metastasis in prostate cancer, while Xiao et al [63] established a model to predict large-number cervical lymph node metastasis in papillary thyroid carcinoma. Similarly, Zhao et al. constructed a prediction model for distant metastasis in locally advanced rectal cancer [64]. Regarding recurrence prediction, Xie et al. conducted a study on hepatocellular carcinoma [57]. In addition, Tan et al. developed radiopathomics nomograms to predict the pathological staging of gastric cancer [65].

Additionally, several studies have focused on predicting biomarker expression through radiopathomics models. For instance, Mao et al. developed a deep learning-based radiopathomics model to predict carcinogenesis promotor cyclooxygenase-2 expression in common bile duct in children with pancreaticobiliary maljunction [66].

Radiopathomics has also advanced in terms of predicting the efficacy of neoadjuvant therapy (Table 1).

Table 1.

Literature compilation of radiopathomics for predicting the efficacy of neoadjuvant therapy.

Author	Year	Disease	Study type	Treatment	Study population	Method	Result(AUC)	Validation
Feng [58]	2022	rectal cancer	retrospective study(include a prospective validation study)	Neoadjuvant chemoradiotherapy	retrospective training cohort 303 retrospective external validationl cohort 1 480 retrospective external validationl cohort 2 150 multicenter, prospective validation cohort 100	machine learning	retrospective training cohort: RAPIDS 0.868 (95% CI 0.825 - 0.912) retrospective external validationl cohort 1: RAPIDS 0.860 (95%CI 0.828 - 0.892) retrospective external validationl cohort 2: RAPIDS 0.872 (95%CI 0.810 - 0.934) multicenter, prospective validation cohort: RAPIDS 0.812 (95%CI 0.717-0.907); pathomics microenvironment model 0.630 [0.507–0.754], p < 0.0001; radiomics MRI model 0.716 [0.580-0.852], p<0.0001; pathomics nucleus model 0.733 [0.620-0.845], p<0.0001	external
Jiang [59]	2024	breast cancer	retrospective study	Neoadjuvant chemotherapy	traning cohort 150 test cohort 65 external validation cohort 96	machine learning deep learning	traning cohort: multiomics signature 0.900; TNM staging 0.539, p<0.0001; RADL signature 0.816, p=0.0019 test cohort: multiomics signature 0.814; TNM staging 0.505, p=0.0055; RADL signature 0.791, p=0.5735 external validation cohort: multiomics 0.792; TNM staging 0.432, p=0.0110; RADL signature 0.660, p=0.0004	Internal external
Zhang [81]	2023	breast cancer	retrospective study	Neoadjuvant chemotherapy	training cohort 155 validation cohort 56	deep learning	training cohort: DLRPM 0.933 (95% CI 0.895–0.971) validation cohort: DLRPM 0.927 (95% CI 0.858–0.996); radiomics signature 0.821[0.700–0.942], p<0.05; pathomics signature 0.766[0.629–0.903], p<0.05	internal
Xu [29]	2024	breast cancer	retrospective study	neoadjuvant chemotherapy	training cohort 124 validation cohort 31	machine learning	training cohort: radiopathomics signature model 0.83 validation cohort: radiopathomics signature model 0.91; radiomics signature 0.83, p>0.05; pathomics signature 0.60, p>0.05	internal
Wan [82]	2022	rectal cancer	retrospective study	neoadjuvant chemoradiotherapy	training cohort 107 validation cohort 46	machine learning deep learning	training cohort: multiscale model 0.93 (95% CI 0.88–0.98); traditional clinicoradiological model 0.69 [0.55-0.82] validation cohort: multiscale model 0.90 (95% CI 0.78–1.00); traditional clinicoradiological model 0.68 [0.46-0.91]	internal

Currently, the application of radiopathomics primarily focuses on breast and rectal cancers, with most studies being retrospective in nature. Feng et al. developed and validated a radiopathomics model to predict a pathological complete response to neoadjuvant radiochemotherapy in locally advanced rectal cancer. The model used pre-treatment mpMRI and H&E-stained tissue images. Separate models were constructed based on radiomics features from MRI, pathomics features from nuclei, and microenvironment features, which were then integrated into the RadioPathomics Integrated Prediction System (RAPIDS). RAPIDS demonstrated strong performance in two retrospective external validation cohorts and one prospective external validation cohort. Notably, RAPIDS significantly outperformed unimodal models, with the study concluding that the improved performance was due to the integration of complementary features rather than redundancy in the input data [58]. Jiang et al. developed a multi-omics model combining DLG3, radiomics, and pathomics using machine learning and deep learning techniques. Their multi-omics model accurately predicted pathological complete response and was validated in both internal and external cohorts [59].

Multimodal fusion in radiopathomics typically occurs in three stages: early fusion, intermediate fusion, and late fusion. Early fusion, also known as data-level fusion, combines information from multiple modalities before features are input into a classifier. Intermediate fusion, or inter-layer fusion, integrates different modalities during the modeling process between the input and output layers [18]. Late fusion, also known as decision-level fusion, trains separate models for each modality before fusing them into a final composite model [61].

Most radiopathomics studies, including those by Feng et al. and Jiang et al., have used late fusion, where separate radiomics and pathomics models are built and subsequently combined. Late fusion is favored due to its simpler design, especially when the data from different modalities are not highly complementary [61]. However, other fusion strategies have also been explored. Tan et al. applied early fusion in a model using CT scans and WSIs to predict gastric cancer staging. They combined 1,834 radiomics features with 311 pathomics features, screening 17 features from among the total of 2,145 to develop a radiopathomics model [67].

Despite these advancements, there is insufficient evidence to determine which fusion method is most effective in radiopathomics applications. Further exploration is required to compare the efficacy of these fusion techniques.

The strengths of radiopathomics

Radiopathomics combines the strengths of radiomics and pathomics, comprehensively captures the spatial heterogeneity both macroscopic and microscopic levels. The predictive ability of the model has been significantly improved, and encouraging results have been achieved in several studies.

Imaging and histological examinations can achieve longitudinal monitoring of tumors. For example, the radiomics pipeline designed by Gu et al. mentioned earlier has achieved multi-stage screening of patients [68]. Zhao et al. analyzed the changes of tumor-infiltrating immune cells before and after neoadjuvant chemotherapy and designed an immune microenvironment score to predict the therapeutic effect [69].

Additionally, for multi-omics models to be successfully applied in clinical settings, both their utility and affordability are crucial. Models that rely on expensive or technically complex data are challenging to implement on a large scale in routine practice [70]. Radiomics and pathomics respectively rely on commonly available clinical imaging and pathological images without additional invasive sampling, and are very promising in the construction of predictive models.

Radiopathomics can also indirectly reflect changes at the molecular level such as gene expression. In the aspect of pathomics, Chen et al. applied a pathomics model to predict the stemness index of lung adenocarcinoma, an indicator based on mRNA expression [71]. Cao et al. established a microsatellite instability prediction model for colorectal cancer [43]. In the aspect of radiomics, Fan et al. constructed a model for predicting γδ T-cell abundance [72]. Zhu et al. created a model to predict the expression of CTLA4 [25].

The weaknesses of radiopathomics

Radiomics and pathomics analyze images through machine learning or deep learning. The features obtained lack biological interpretability, which leads to limited trust in the clinical applications. It is necessary to combine it with other methods such as genomics and immunohistochemistry to explain the possible biological significance [18].

Furthermore, the workflow of radiomics and pathomics involves many factors that may affect the performance of the model, such as equipment parameters, staining techniques, image preprocessing, ROI segmentation methods, etc., which leads to poor generalizability of the model and limits its wide application [18].

Challenges and prospects

With the rapid advancement of artificial intelligence, research in radiomics and pathomics is flourishing. While initial progress has been made in multi-omics integration, such as radiopathomics, significant challenges still hinder its further development (Table 2).

Table 2.

Challenges and solutions of radiopathomics.

Challenge	Solution
Lack biological interpretability ("black box" issue)	Identifying biological significance (eg. radiogenomics, pathogenomics, IHC)
	XAI techniques (eg.SHAP)
Single center retrospective study	Multicenter prospective study
Internal validation	External validation
Poor generalizability	Standardized workflow
	Multicenter, large, high-quality datasets
	Few-shot learning and manifold learning
	Enhancing the repeatability and reproducibility of features (eg. ComBat harmonization)
	Reducing the risk of overfitting (feature selection, increase sample size)
	Improving data imbalance (oversampling and undersampling techniques)
Single-modality data	Multi-omics integration

Interpretability

Radiomics and pathomics rely on data-driven feature extraction, which often lacks biological interpretability. Additionally, as deep learning gains prominence in this field, the “black box” issue becomes more pronounced compared to traditional machine learning approaches [18]. This lack of model transparency limits trust and hinders the broader application of radiomics and pathomics in clinical practice.

One approach to improving feature interpretability is identifying their biological significance. Radiogenomics and pathogenomics offer effective methods for linking extracted features to biological processes. Immunohistochemistry (IHC) is another viable technique. Cai et al. used IHC staining for Ki67, p53, and PD-L1 to assess correlations between molecular expression and pathomics features [44].

Explainable artificial intelligence (XAI) techniques are also being developed to improve model transparency [73]. SHapley Additive exPlanations (SHAP), uses Shapley values to interpret machine learning results. Wang et al. applied SHAP analysis to confirm the correlation between tumor heterogeneity and neoadjuvant treatment outcomes in rectal cancer, highlighting the differential contributions of various radiomics features to assessment of the treatment response [74].

Despite these advances, many studies do not include explanations of their models and predictions, which is an area that requires increased attention in future research.

External validation and multicenter prospective evidence

One of the obstacles to the clinical implementation of radiomics and pathomics models is the lack of external validation and prospective evidence. Much of the current research is based on retrospective studies, which are prone to selection bias and residual confounding [75]. Additionally, most studies are conducted using data from a single center, with only internal validation and no external validation. As a result, the models fail to account for the heterogeneity of patient populations treated at different medical centers [76].

Generalizability

Generalizability refers to the ability of a model to accurately predict outcomes for new, unseen data. A key reason for poor generalizability is the difference between the distribution of training data and unknown data [18]. There are five main ways to improve the generalizability of a model.

The first is the establishment of a standardized workflow. Currently, no standard workflow exists, and any changes to the process can introduce discrepancies that negatively affect model performance [18].

The second is to increase the number and diversity of samples. Clinical models need large, high-quality datasets from multiple medical centers to prevent algorithmic bias caused by differences in data structure, staining techniques, and patient populations [70]. Constructing digital biobanks to facilitate the sharing of standardized imaging, clinical, pathological, and molecular data is crucial [77].

When large datasets are unavailable, generalization can be improved through techniques such as few-shot learning and manifold learning. Few-shot learning combines data augmentation and transfer learning [18]. Cao et al. trained a deep learning model using The Cancer Genome Atlas Colon Adenocarcinoma Collection (TCGA-COAD) dataset and found that it performed poorly in an Asian cohort. They then applied transfer learning to fine-tune the model with local data from the Asian cohort, significantly improving its efficacy [43].

The third method is enhancing the repeatability and reproducibility of features. Reproducibility reflects the stability of a model. Feature values are influenced by various stages of the workflow, including image acquisition, preprocessing, ROI segmentation, and feature selection. ComBat harmonization is a widely used standardization technique in radiomics because radiomics features are sensitive to variations in acquisition equipment and reconstruction parameters. Image preprocessing steps, such as z-score normalization and Gaussian filtering, are necessary before segmenting ROIs. In the ROI segmentation stage, reproducibility can be improved by using automatic segmentation or performing manual segmentation with input from multiple experts. During the feature selection stage, features with high reproducibility can be filtered using metrics such as the intraclass correlation coefficient (ICC) to quantify reproducibility [18].

The fourth method is reducing the risk of overfitting. Using too many features relative to a small sample size can lead to overfitting, which reduces the ability of the model to generalize. One solution is to reduce the number of features by performing feature selection, to retain only the most relevant features and remove redundant ones. Another solution is to increase the sample size [78]. Additionally, integrating multi-omics data can reduce the reliance of the model on a single feature, thereby reducing the risk of overfitting [67].

The fifth method is improving data imbalance. In an imbalanced dataset, there is a significant disparity in the sizes of different data classes, which leads the model to prioritize learning from the majority class and biases its predictions. This can be addressed using oversampling and undersampling techniques. Oversampling involves replicating data from the minority class, while undersampling discards some data from the majority class [18]. Huang et al. improved the performance of their model by oversampling a relatively small percentage of non-luminal cases [12].

Multi-omics integration

The insights gained from single-modality data are limited, making multimodal data fusion an inevitable direction for future research. Multi-omics research has become a growing focus, but current studies are often constrained to small, single-institution datasets. Furthermore, there is a lack of in-depth research on fusion methods and applicable algorithms. Moving forward, the development of large-scale multimodal datasets and more sophisticated fusion strategies will be necessary to reveal tumor characteristics in a more comprehensive and detailed manner, thereby advancing personalized medicine [79]. An excellent example is the work of Migliozzi et al., who integrated radiomics, proteomics, phosphor-proteomics, metabolomics, lipidomics, and acetylomics to build a probabilistic classification tools [80].

Conclusion

Radiomics and pathomics offer valuable tools for understanding tumor heterogeneity at the macroscopic and microscopic levels, respectively, and provide promising avenues for predicting the efficacy of neoadjuvant therapy. As multi-omics integration continues to evolve, more comprehensive and multi-dimensional analyses of tumors are expected to emerge. Although challenges such as lack of interpretability and poor model generalizability persist, the standardization of workflows and the establishment of large-scale databases will likely pave the way for further developments in multi-omics studies such as radiopathomics, ultimately guiding personalized treatment approaches.

Funding sources

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

CRediT authorship contribution statement

Jiayi Wang: Writing – original draft, Conceptualization. Jiahui Dai: Investigation, Data curation. Yangxi Cheng: Validation, Data curation. Xirui Wang: Investigation, Data curation. Rui Deng: Supervision, Investigation. Huiyong Zhu: Writing – review & editing, Conceptualization.

Declaration of competing interest

I have nothing to declare.