Automating classification of treatment responses to combined targeted therapy and immunotherapy in HCC

Abstract

Tyrosine kinase inhibitor (TKI) combined with immunotherapy regimens are now widely used for treating advanced hepatocellular carcinoma (HCC), but their clinical efficacy is limited to a subset of patients. Considering that the vast majority of advanced HCC patients lose the opportunity for liver resection and thus cannot provide tumor tissue samples, we leveraged the clinical and image data to construct a multimodal convolutional neural network (CNN)–Transformer model for predicting and analyzing tumor response to TKI–immunotherapy. We employed an automatic liver tumor segmentation system, based on a two-stage 3D U-Net framework, to delineate lesions by first segmenting the liver parenchyma and then precisely localizing the tumor. This approach effectively addresses the variability in clinical data and significantly reduces bias introduced by manual intervention. Next, we developed a clinical model using only pretreatment clinical information, a CNN model using only pretreatment magnetic resonance imaging data, and an advanced multimodal CNN–Transformer model fusing both imaging and clinical parameters from a training cohort (n = 181) and then compared their predictive performances in an independent cohort (n = 30). In the validation cohort, the area under the curve (95% confidence interval) values were 0.720 (0.710–0.731), 0.695 (0.683–0.707), and 0.785 (0.760–0.810), respectively, indicating that the multimodal model significantly outperformed the single-modality baseline models across validations. Finally, single-nucleus sequencing with the surgical tumor specimens reveals tumor ecosystem diversity associated with treatment response, providing a preliminary biological validation for the prediction model. In summary, this multimodal model effectively integrates imaging and clinical features of HCC patients, has a superior performance in predicting tumor response to TKI–immunotherapy, and provides a reliable tool for optimizing personalized treatment strategies.

Introduction

Hepatocellular carcinoma (HCC) is the second leading cause of cancer-related mortality and the fifth most common malignancy in China (Llovet et al., 2021). The median survival of HCC patients is 6–20 months, and the 5-year survival rate is only 10% (Konyn et al., 2021). HCC is characterized by its hidden onset and rapid progression, often resulting in a high proportion of patients presenting with metastasis at the time of diagnosis. Consequently, many patients miss the optimal window for surgical treatment. Despite undergoing systemic treatments such as surgery, transarterial chemoembolization (TACE), and radiotherapy, a significant proportion of HCC patients remain at high risk for metastasis and recurrence. Additionally, many patients develop substantial resistance to various chemotherapeutic agents (Abou-Alfa et al., 2018; Zhu et al., 2019), leading to a generally poor prognosis following treatment.

Currently, the commonly used anti-angiogenic, molecularly targeted drugs for HCC are sorafenib, lenvatinib, and apatinib, all of which are tyrosine kinase inhibitors (TKIs). These agents could obviously increase the overall survival of cancer patients (Qin et al., 2019), offering a ‘dawn of hope’ for perioperative conversion therapy in HCC (Gordan et al., 2020). Meanwhile, several immune checkpoint blockades (ICBs) have been approved for clinical use in advanced HCC. They include programmed death receptor-1 (PD-1) and CTLA-4 inhibitors (Sangro et al., 2013; El-Khoueiry et al., 2017). Although TKIs act quickly and are highly specific, tumor cells can easily develop resistance to them (Finn et al., 2020a). While ICBs have shown clinical efficacy in various cancers, the response rate to ICBs in HCC is only ~20% (Sangro et al., 2013). Current research indicates that the differential sensitivity to targeted therapies and ICBs is significantly associated with the tumor microenvironment (TME) and the heterogeneity of tumors (Aldea et al., 2021). It is particularly noteworthy that the TME of HCC exhibits significant heterogeneity in clonal origins during its early formation. Subsequent oncogenic mutations and immune selection pressures drive further heterogeneous evolution of the TME, contributing to its complexity and adaptability. This evolutionary process may also be driven by mechanisms such as chromosomal instability, neoantigen depletion, and immune-editing events, including T-cell receptor rearrangements and the upregulation of immunosuppressive ligands. These factors contribute to the variability in sensitivity to targeted therapies and ICB among HCC patients (Craig et al., 2020). In 2020, FDA approval of the ICB combination therapy atezolizumab/bevacizumab for HCC marked the dawn of the ‘targeted–immune 2.0 era’ in liver cancer treatment (Finn et al., 2020b). Current clinical evidence indicates that patients who exhibit limited responses to targeted therapy or ICB monotherapy may achieve better outcomes with combination therapies. Consequently, identifying the optimal targeted–immune therapeutic strategy to prevent overtreatment and minimize associated side effects, while establishing a comprehensive multimodal treatment approach, is crucial for enhancing the survival rates of HCC patients.

The progress in radiomics has enabled the extraction of high-throughput features from medical imaging modalities such as magnetic resonance imaging (MRI), X-rays, and computed tomography. This capability allows for quantitative analysis, demonstrating robust performance in diagnosis, therapeutic response prediction, and prognosis assessment (Zhang et al., 2023). This trend marks the emergence of a new era in TME analysis, emphasizing the integration of 3D spatial information and tissue morphology to achieve more comprehensive insights (Wu et al., 2022).

A prominent area of research in artificial intelligence (AI)-driven healthcare is machine learning, which aims to empower computer systems to autonomously learn from data and improve performance without explicit programming. By analyzing and identifying patterns within datasets, machine learning algorithms can extract valuable insights, enabling applications such as prediction, classification, clustering, and decision-making. A variety of machine learning tools, including artificial neural networks (ANNs), Bayesian networks (BNs), decision trees (DTs), and support vector machines (SVMs), have been developed for a wide range of oncological applications. These encompass diagnosis, prognosis prediction, tumor classification, detection of pathological features, and identification of genetic alterations (Kourou et al., 2015; Jiang et al., 2020).

Deep learning, a specialized branch of machine learning, eliminates the need for manual feature selection by autonomously and adaptively identifying and extracting intricate, high-level lesion information from image samples based on predefined learning objectives. As a result, its application to image-based medical data, including histopathology and 3D imaging, has increasingly become a prominent area of research focus (Bi et al., 2019; Lotter et al., 2021). For instance, Sun et al. (2021) created a biomedical interpretable AI model designed for precision diagnosis and treatment by integrating tumor imaging, genomic data, and other multiomics information. In another study, Park et al. (2018) developed an elastic net survival model using preoperative MRI results for breast cancer prognosis analysis. They constructed a nomogram model that combined radiomics and clinicopathological features, which demonstrated superior performance compared to models relying solely on individual feature sets (Park et al., 2018). By integrating multilevel data and utilizing ensemble learning models, we can more accurately evaluate the biological characteristics of tumors, enabling personalized treatment strategies for HCC patients.

In this study, we aim to develop an ensemble learning classification model. This model will integrate patient clinical information and multidimensional 3D image data to predict variations in sensitivity to combined targeted therapy and immunotherapy among advanced HCC patients. By doing so, we hope to advance the concept of ‘smart healthcare’ and offer novel perspectives for research in precise HCC stratification, personalized treatment strategies, and efficacy monitoring.

Results

Participant characteristics

The study enrolled a total of 211 patients including a training cohort of 181 patients from October 2018 to October 2022 and an independent validation cohort of 30 patients from January 2023 to December 2023. The flowchart of our study is depicted in Figure 1. The clinicopathologic characteristics of the participants in the two cohorts were similar and are detailed in Table 1 and Supplementary Tables S1 and S2, respectively. The training cohort, comprised of 181 patients (162 men and 19 women) with the age of 55 ± 10 years, was utilized to construct the convolutional neural network (CNN) prediction models. Among these patients, the majority were classified as the non-progressive disease (non-PD) group (n = 115, 63.5%), comprising cases of complete response (CR; n = 4, 2.2%), partial response (PR; n = 32, 17.7%), and stable disease (SD; n = 79, 43.6%). In contrast, 66 patients (36.5%) were classified as the progressive disease (PD) group. The baseline characteristics of the PD and non-PD groups were largely comparable, without significant differences except for tumor number (P = 0.003), AST level (P = 0.016), CD4% (P = 0.050), and CD8% (P = 0.037) (Table 1). Univariate logistic regression analyses revealed that tumor number (multiple vs. single) was significantly associated with predicting for PD (P < 0.05) (Supplementary Table S3).

Figure 1.

Framework and flowchart of our study. (A) A framework for the development and validation of prediction models. (B) A flowchart for this study.

Table 1.

Baseline clinical characteristics of the participants in the training cohort.

	Number (proportion, %) or median (IQR)
Variables	non-PD (n = 115)	PD (n = 66)	P-value
Age, ≥50/<50 (years)	85/30 (73.9/26.1)	47/19 (71.2/28.8)	0.694
Gender, male/female	103/12 (89.6/10.4)	59/7 (89.4/10.6)	0.971
Lymph node metastasis, positive/negative	29/86 (25.2/74.8)	17/49 (25.8/74.2)	0.860
Extrahepatic metastasis, positive/negative	29/86 (25.2/74.8)	21/45 (31.8/68.2)	0.339
Ascites, positive/negative	23/92 (20.0/80.0)	15/51 (22.7/77.3)	0.665
Tumor number, multiple/single	78/37 (67.8/32.2)	58/8 (87.9/12.1)	0.003*
Maximum tumor size, cm	6.8 (3.2–10.3)	5.1 (2.1–9.3)	0.055
Tumor thrombus, positive/negative	57/58 (49.6/50.4)	30/36 (45.5/54.5)	0.594
PLT, ×109/L	140 (107–193)	139 (96–184)	0.613
Neutrophil, ×109/L	3.3 (2.4–4.4)	3.2 (2.4–4.7)	0.533
Lymphocyte, ×109/L	1.2 (0.9–1.6)	1.1 (0.8–1.5)	0.385
Monocyte, ×109/L	0.47 (0.38–0.61)	0.45 (0.33–0.57)	0.131
NLR	2.88 (1.88–4.13)	2.88 (1.83–4.90)	0.332
PLR	113.1 (84.3–172.7)	129.6 (85.4–198.4)	0.158
Total bilirubin, μmol/L	15.3 (11.0–21.1)	15.8 (12.3–20.8)	0.982
Conjugated bilirubin, μmol/L	5.5 (4.1–7.8)	5.9 (4.1–8.8)	0.065
Albumin level, g/L	40 (37–43)	40 (36–42)	0.282
ALT, U/L	31 (21–49)	37 (23–62)	0.072
AST, U/L	42 (27–63)	45 (30–82)	0.016*
AKP, U/L	119 (82–153)	109 (79–151)	0.635
γ-GT, U/L	122 (54–202)	123 (46–246)	0.808
Creatinine	71 (63–83)	70 (62–79)	0.531
Prothrombin time, sec	12.5 (11.8–82.8)	12.7 (11.8–13.4)	0.177
AFP, ng/ml	112 (7–3173)	249 (11–7734)	0.227
PIVKA-II, mAU/ml	1184 (109–10316)	1284 (123–21100)	0.129
CEA, ng/ml	2.7 (1.8–4.2)	2.4 (1.6–4.3)	0.429
CA19-9, U/ml	20.2 (12.3–61.7)	22.8 (10.9–35.9)	0.258
Laminin, ng/ml	112.2 (87.9–175.6)	114.4 (72.8–187.8)	0.166
IV-Col, ng/ml	118.1 (81.9–151.8)	5.9 (4.1–8.8)	0.625
CD4, %	40.3 (35.8–46.0)	38.2 (32.8–45.1)	0.050*
CD8, %	26.2 (19.3–31.9)	25.9 (18.9–30.3)	0.037*

IQR, interquartile range; PLT, platelet; NLR, neutrophil to lymphocyte ratio; PLR, platelet-to-lymphocyte ratio; ALT, alanine aminotransferase; AST, aspartate aminotransferase; AKP, alkaline phosphatase; γ-GT, gamma-glutamyl transpeptidase; AFP, alpha fetoprotein; PIVKA-II, protein induced by vitamin K absence or antagonist-II; CEA, carcinoembryonic antigen; CA19-9, carbohydrate antigen 19-9; IV-Col, type IV collagen. *P ≤ 0.05.

Two-stage automatic liver tumor segmentation using a 3D U-Net network

Liver tumor distribution is highly complex, encompassing single and multiple tumors with widely varying sizes. Additionally, the contours of some tumors are poorly defined, further complicating the segmentation process. In this study, we employed a U-Net-based network and developed a two-stage approach to automatically and precisely localize tumor regions of interest (ROIs), thereby minimizing the influence of irrelevant regions on segmentation accuracy. The total architecture of the model is illustrated in Figure 2. In the first stage, MRI images are resized to 256 × 256 × 48 and fed into a 3D U-Net network to create a pixel-level liver segmentation model, accurately identifying the liver area. In the second stage, the MRI sequences of the segmented liver region are resized to the same dimensions (256 × 256 × 48) and processed through a second 3D U-Net network to generate pixel-wise tumor predictions, ensuring accurate tumor localization.

Figure 2.

Network architecture of the 3D U-Net-based liver tumor segmentation. In Stage 1, the full MRI volume is used as the input to obtain liver segmentation. The liver region is then extracted from the original MRI and serves as the input for Stage 2. The tumor segmentation is derived from the output of Stage 2. Both stages utilize the same 3D U-Net architecture.

Two cases from the PD group and two from the non-PD group were selected to evaluate the performance of our proposed two-stage U-Net model. The automatic liver tumor segmentation results are illustrated in Figure 3, indicating that our method achieves accurate and reliable tumor segmentation. Notably, the model demonstrates consistent performance across different slices of the same patient’s MRI sequence, underscoring its robustness and precision.

Figure 3.

Liver tumor segmentation results based on the proposed two-stage U-Net model. (A and B) Cases from the PD cases. (C and D) Cases from the non-PD cases. Each row shows different slice segmentation results of one MRI sequence with the contour labeled in green.

Comparative analysis of predictive performance between clinical model, CNN model, and integrated multimodal CNN model

To differentiate between the PD and non-PD groups, we developed three network models: an SVM model based solely on clinical information, a 3D CNN model based solely on MRI images, and a multimodal network model integrating both MRI images and clinical data. The multimodal CNN model was designed to extract quantitative features from both sources, enhancing classification accuracy and effectively categorizing each patient as PD or non-PD. The overall architecture of the CNN models for tumor response prediction is shown in Figure 4.

Figure 4.

The architectural construction of the CNN models for tumor response prediction in this study. (A) The architecture of 3D CNN model utilizing only preoperative MRI data for tumor response prediction. Images provided to the 3D CNN are processed initially in a convolutional layer and followed by BN, ReLU, and one maximum pooling layer. Then, four residual blocks are followed in a sequential order. The data are fed to average pooling and finally processed in fully connected layers followed by a softmax activation function. (B) The total architecture of the multimodal CNN model for tumor response prediction. Both the tumor mass image data and clinical data are taken as the input for the corresponding data process. The CNN module shares the same architecture as above, and Transformer module, consisting of two encoder layers, is used for the feature extraction for clinical data. Both the CNN and Transformer module outputs are concatenated and fed into two fully connected layers in a sequential order. The data are finally fed to a softmax activation for the binary classification task.

The predictive performances of the clinical model, the 3D CNN model, and the multimodal CNN model are summarized in Table 2. Using a 10-fold cross-validation approach, the mean values of standard assessment metrics, including the area under the receiver operating characteristic (ROC) curve (AUC), accuracy, sensitivity, and specificity, were determined, along with the 95% confidence intervals (CIs). For the clinical model, the performance metrics in the validation cohort were as follows: AUC 0.720 (0.710–0.731), accuracy 0.658 (0.627–0.690), sensitivity 0.650 (0.555–0.745), and specificity 0.662 (0.573–0.752). For the 3D CNN model, the performance metrics in the validation cohort were AUC 0.695 (0.683–0.707), accuracy 0.679 (0.654–0.714), sensitivity 0.594 (0.514–0.674), and specificity 0.850 (0.779–0.921). In contrast, the multimodal model achieved superior results, with AUC 0.785 (0.760–0.810), accuracy 0.721 (0.686–0.755), sensitivity 0.606 (0.548–0.664), and specificity 0.950 (0.910–0.990) in the validation cohort. These results indicate that the multimodal model outperformed the 3D CNN model and was more balanced in terms of specificity and sensitivity. The ROC curves of these models are shown in Figure 5A and B. To illustrate the variability of the models, boxplots of all performance metrics across each fold are shown for the clinical model, the 3D CNN model, and the multimodal model, respectively (Supplementary Figure S1).

Table 2.

Comparison of the performances achieved by the clinical model, the 3D CNN model, and the multimodal model integrating MRI images and clinical variables.

	AUC	Accuracy	Sensitivity	Specificity
Clinical model
Training cohort	0.662 (0.626–0.697)	0.648 (0.619–0.676)	0.572 (0.504–0.639)	0.690 (0.631–0.750)
Validation cohort	0.720 (0.710–0.731)	0.658 (0.627–0.690)	0.650 (0.555–0.745)	0.662 (0.573–0.752)
3D CNN model
Training cohort	0.689 (0.663–0.714)	0.680 (0.654–0.705)	0.652 (0.594–0.709)	0.742 (0.686–0.799)
Validation cohort	0.695 (0.683–0.707)	0.679 (0.645–0.714)	0. 594 (0.514–0.674)	0.850 (0.779–0.921)
Multimodal model
Training cohort	0.798 (0.782–0.815)	0.741 (0.721–0.761)	0.667 (0.633–0.701)	0.882 (0.855–0.910)
Validation cohort	0.785 (0.760–0.810)	0.721 (0.686–0.755)	0.606 (0.548–0.664)	0.950 (0.910–0.990)

The results are evaluated in terms of mean percentage AUC, accuracy, sensitivity, and specificity values by implementing a 10-fold cross-validation scheme. The related 95% CI is highlighted in parentheses.

Figure 5.

The predictive performance and visual explanation of the CNN models. (A and B) ROC curves for tumor response prediction models in the training and validation cohorts, respectively. The ROC curve for the clinical model utilizes clinical data as the input. The ROC curve for the 3D CNN model utilizes MRI image sequences as the input. The ROC curve for the multimodal model incorporates both MRI image sequences and clinical data. The curves represent the average values from the 10-fold cross-validation scheme, with the corresponding 95% CIs indicated by shaded areas. (C) 3D Grad-CAM for visualizing both PD and non-PD samples. The left panels present tumor mass MRI sequences as the input for the 3D CNN, while the right panels present the 3D Grad-CAM heatmaps overlaid on the MRI sequences. Each row shows the slices of one MRI sequence from an individual patient. The saliency regions are highlighted using a jet colormap, with the gradient from blue to red indicating areas of increasing importance within the tumor MRI sequence, corresponding to the decision of interest.

Visual explanation of the combined CNN model

Gradient-weighted class activation mapping (Grad-CAM) is a non-intrusive method that does not require any modifications to the network architecture. It can enhance the transparency of the model’s decision-making process and also aid in validating the clinical relevance of the identified regions. Figure 5C provides a visual representation of the decision-making process of the 3D CNN model. The top panel presents cases from the non-PD group, while the bottom panel illustrates PD cases. Each row shows the slices of one MRI sequence alongside their corresponding heatmaps, highlighting regions identified by the model. The results demonstrate that, regardless of the tumor’s appearance as a single mass or multiple instances, the model predominantly focuses on the tumor’s internal regions. While some variations in saliency zones are observed across different slices, most of these highlighted areas remain contiguous.

Single-nucleus transcriptomic analysis reveals tumor ecosystem in PD and non-PD patients

Considering the significant variability in the response to TKI combined with immunotherapy among advanced HCC patients, we collected surgical tumor specimens from four eligible patients and performed single-nucleus RNA sequencing analysis, aiming to preliminarily explore differences in the TME between PD and non-PD patients.

Using established marker genes, we identified and categorized 20 distinct cell subsets into 10 major cell types, including epithelial cell, cancer cell, CD4+ T cell, macrophage, endothelial cell, liver bud hepatic cell, progenitor cell, fibroblast, NK/NKT cell, and B cell (Figure 6A). Next, we employed bubble plots to visualize the distribution of top differentially expressed genes (DEGs) across different single-cell subclusters (Figure 6B). Compared to non-PD group (sensitive), PD group resistant to TKI–immunotherapy exhibited a significantly lower proportion of immune cells, particularly CD4+ T cells (11.00% in sensitive vs. 1.12% in resistant) and macrophages (8.60% in sensitive vs. 0.02% in resistant) (Figure 6C). These results imply that the mechanisms underlying responses to combined TKI–immunotherapy could be strongly linked to the abundance and activity of immune cells. We next analyzed the DEGs in tumor cells between the PD and non-PD groups. Volcano plots revealed that AC106771.1, AC097520.2, AL157378.1, LINC01194, and LINC01446 were significantly upregulated in the resistant group, whereas PIGR, PTH2R, AF233439.1, MIF, and RHEX were downregulated (Figure 6D). Furthermore, the resistant group exhibited higher expression of genes involved in cytoskeletal processes in muscle cells, vascular smooth muscle contraction, and ECM–receptor interactions, while the sensitive group displayed significant enrichment for ferroptosis-related pathways (Figure 6E).

Figure 6.

Single-nucleus transcriptomic analysis deciphers tumor ecosystem in PD and non-PD patients. (A) t-SNE plots for distinct cell subsets in four HCC cases, i.e. two resistant (PD) and two sensitive (non-PD) to TKI–immunotherapy. (B) Bubble plot of the top DEGs in each single-cell subcluster. (C) Proportional distribution of each cell cluster in patients resistant and sensitive to TKI–immunotherapy. (D) Volcano plot of DEGs between the tumor cells from sensitive and resistant groups. (E) KEGG enrichment analysis of DEGs between the tumor cells from sensitive and resistant groups.

Given the differences in immune cell composition between the PD group and non-PD group, we statistically analyzed the proportions of CD4+ and CD8+ T cells in peripheral blood from the patients. Although the non-PD group exhibited a higher proportion of CD4+ and CD8+ T cells in peripheral blood (Table 1), univariate analysis revealed that neither CD4+ nor CD8+ T cell proportion showed statistical significance in predicting tumor response (Supplementary Table S3). Furthermore, in a subset of HCC patients, we performed multiplex immunofluorescence staining for CD86 and CD206 in tumor tissues but did not observe significant differences between the non-PD and PD groups (Supplementary Figure S2). These results suggest that future studies may need to incorporate other immune cell markers for more accurate prediction of tumor response.

Discussion

In this study, we developed an intelligent network for automated liver tumor identification and segmentation using MRI images. Building on this, we constructed two CNN models to predict tumor response to combined TKI–immunotherapy in patients with unresectable HCC. While previous CNN-based studies in HCC predominantly focused on tumor identification and classification (Cheng et al., 2021), there is limited research leveraging CNNs to evaluate the therapeutic efficacy. Recent investigations have mainly concentrated on tumor response assessment in unresectable HCC patients receiving either immunotherapy alone or lenvatinib monotherapy combined with TACE, where prediction models were developed using traditional methods such as the least absolute shrinkage and selection operator (LASSO) and standard neural networks (Yuan et al., 2020; Luo et al., 2022; Xu et al., 2023). Our 3D CNN model surpasses traditional methods by automatically extracting spatial features from 3D medical image data, enabling end-to-end learning, and capturing complex volumetric patterns. This makes it significantly more effective for tasks such as tumor response prediction and personalized treatment optimization. In addition, to ensure analytical rigor in evaluating the efficacy of TKI–immunotherapy combination therapy, we applied stringent exclusion criteria, excluding patients with TACE or other confounding treatments that could potentially interfere with the accurate assessment of therapeutic outcomes. To the best of our knowledge, this study represents the first comprehensive effort to develop a computer-aided evaluation framework for predicting tumor response in HCC patients receiving TKI–immunotherapy combination treatment, integrating both MRI-derived features and clinical parameters.

Many patients are diagnosed with HCC at an advanced stage, where systematic therapy remains the primary treatment option that can potentially improve survival outcomes (Reig et al., 2022). For over a decade, single-agent targeted therapies, such as sorafenib and lenvatinib, have been the standard treatment options, demonstrating efficacy in patients with advanced HCC (Llovet et al., 2008; Kudo et al., 2018). However, the therapeutic outcomes of these agents remain far from satisfactory. The emergence of immunotherapy has transformed this landscape, offering new hope and significantly improving treatment prospects (Llovet et al., 2008; Kudo et al., 2018). A significant turning point in the treatment of HCC has been achieved with the evolution of systemic therapies, transitioning from single-agent targeted therapy to the combination of immunotherapy and targeted therapy, particularly involving TKIs (Finn et al., 2020a; Cheng et al., 2022). Given the increasingly recognized role of the TME in tumor progression, developing a deeper understanding of the relationship between the TME and therapeutic outcomes is becoming ever more critical as immunotherapy advances. Previous studies have demonstrated that tumor heterogeneity is closely linked to tumor response (Sun et al., 2018; Dercle et al., 2020), which is consistent with our results of single-nucleus RNA sequencing. Specifically, we observed a quantitatively lower presence of CD4+ T cells and macrophages within the TME of patients who exhibited resistance to TKI–immunotherapy combination treatment. Therefore, there is an urgent need to develop a method capable of predicting treatment outcomes for advanced HCC patients undergoing TKI–immunotherapy combination therapy.

Given the proven robustness and effectiveness of AI-assisted systems in disease diagnosis and treatment outcome prediction (Comes et al., 2024), we developed a deep learning model to predict therapeutic outcomes for advanced HCC patients undergoing TKI–immunotherapy combination treatment. In our study, we implemented a three-step approach: first, liver tumors were automatically segmented as ROI using the 3D U-Net algorithm for feature extraction; second, we developed a CNN model based solely on preoperative MRI and a multimodal CNN model integrating preoperative MRI with clinical variables; and third, we evaluated and compared the performances of the two CNN models along with the clinical model and utilized 3D Grad-CAM to interpret the results. Our findings demonstrate that the multimodal CNN model serves as an effective tool to support both clinicians and patients in treatment decision-making, outperforming the conventional CNN model in predictive performance (Table 2). Our visualization results further revealed that the features learned by the model are predominantly localized within the liver tumor region. While the interpretation of learned features in deep models can be inherently complex, these heatmaps provide valuable insights by emphasizing the most relevant regions within each MRI slice, effectively illustrating the attention weights and importance assigned by the model during prediction.

To overcome the limitations associated with the relatively small sample size in our study, we adopted widely recognized methods to ensure robust outcomes and prevent overfitting. Notably, data augmentation and dropout techniques were systematically incorporated into the design of our model. Furthermore, to assess the robustness of the predictive performance, we conducted multiple rounds of cross-validation and calculated the CIs for the evaluation metrics.

Our study advances the field through three key innovations. (i) We developed a fully automated two-stage 3D U-Net segmentation pipeline that sequentially processes liver and tumor regions. This approach effectively addresses the variability in clinical data and significantly reduces manual intervention bias. (ii) The multimodal model significantly outperformed unimodal benchmarks across validations, with ROC/boxplot analyses confirming the statistical value of clinical data integration. (iii) Through 3D Grad-CAM visualization, we demonstrated that the model consistently focuses on relevant tumor regions, confirming its ability to learn genuine pathological features rather than artifacts. Single-nucleus RNA sequencing revealed that the treatment-resistant patients exhibit distinct TME characteristics, including reduced CD4+ T cell infiltration and suppressed ferroptosis-related pathways. These findings provide biological substantiation for the model’s predictions. The non-invasive prediction model using imaging and clinical data not only represents a technical achievement but also provides imaging-based complementary value to conventional tumor biomarker-dependent clinical predictions.

Our current study has several limitations that should be acknowledged. First, the patient cohort is relatively homogeneous and moderately sized, sourced from a single center over a short period (between 2018 and 2022). In fact, the advanced HCC patient population is highly heterogeneous, encompassing a wide variety of tumors with diverse clinicopathological characteristics. For instance, some patients present with diffuse-type HCC, while others exhibit massive HCC. Second, as most participants in this study were diagnosed with HBV-associated HCC, the applicability of the proposed network to HCC cases stemming from other etiological factors may be limited. Third, we lacked clinical data on side effects associated with TKI plus immunotherapy during and after treatment. Fourth, owing to the unavailability of tumor tissue specimens, we could not perform comparative analyses between our model and the established biomarkers, such as tumor mutational burden or PD-L1 expression levels. However, our model may serve as an adjunct diagnostic tool for tissue-based biomarkers. Finally, the retrospective nature of our study introduces inherent selection bias, potentially influencing the interpretation of the results. Therefore, it is essential to incorporate data from diverse devices, clinical centers, and countries for external validation in the future.

In conclusion, our proposed deep learning model integrates multimodal input from MRI data and clinical information using 3D U-Net and CNN architectures, enabling fully automated liver tumor recognition and treatment response evaluation for HCC patients undergoing TKI plus immunotherapy. This tool assists clinicians in making informed therapeutic decisions and selecting appropriate surveillance strategies in clinical practice, demonstrating the significant potential of AI-assisted systems in oncology.

Materials and methods

Study cohort

This study was approved by the Ethics Committee of Zhongshan Hospital in accordance with the ethical guidelines of the Declaration of Helsinki (No. B2021-248). Informed consent was waived due to the retrospective design of the study. A total of 211 patients receiving the combination therapy with TKIs and PD-1 inhibitors were included in this study. Among them, MRI scans and clinical parameters from 181 patients enrolled between October 2018 and October 2022 were utilized to construct the CNN prediction models, with the 30 patients enrolled between January 2023 and December 2023 serving as an independent validation cohort. Both cohorts exhibited similar backgrounds in terms of clinicopathological characteristics (Supplementary Table S1). Additionally, we randomly selected 172 HCC patients between January 2023 and December 2023 to obtain their MRI images for the establishment of an automatic liver tumor segmentation model. The diagnosis of HCC was confirmed either by liver biopsy or according to the diagnostic criteria established by the American Association for the Study of Liver Diseases. The exclusion criteria for this study were as follows: (i) patients younger than 18 years; (ii) cases with metastatic HCC; (iii) patients with other malignant tumors; (iv) individuals who underwent palliative-intent primary resection; (v) patients who received additional treatments following TKI–immunotherapy combination therapy; (vi) cases with incomplete follow-up data; and (vii) patients lacking pretreatment MRI or those with MRI scans obtained >1 month prior to treatment initiation.

Data collection and MRI acquisition

The clinical characteristics of 211 patients were collected, including best tumor response, age, gender, lymph node metastasis, extrahepatic metastasis, ascites, tumor number, maximum tumor diameter, tumor thrombus, circulating blood cell laboratory values, liver function, hepatitis panel, serum liver fibrosis indicators (laminin and type IV collagen), carbohydrate antigen 19-9, and carcinoembryonic antigen.

All MRI examinations for each patient were performed using a 3-Tesla MR scanner following the standard liver MRI protocol (Quan et al., 2024). Patients were advised to fast for ~4 h before the examination to minimize gastrointestinal fluid secretion. For this study, enhanced T1-weighted sequences (arterial phase) were employed to extract radiomic features.

Follow-up and response evaluation

In this study, all patients received standardized care and regular follow-up assessments. Every three treatment cycles, or approximately every 2 months, two physicians evaluated the tumor response by using enhanced T1-weighted sequences to determine the objective response and assess potential tumor progression. Tumor responses were categorized into four groups: PD, SD, PR, and CR. The definitions of these criteria were guided by the Response Evaluation Criteria in Solid Tumors version 1.1, which served as the standard for assessing tumor response in this study (Eisenhauer et al., 2009). For subsequent analysis, patients were divided into two groups: the PD group and the non-PD group (including CR, PR, and SD).

Single-nucleus RNA sequencing

We chose four HCC samples from our study cohort, of which two were PD and two were non-PD based on tumor response evaluation (Supplementary Table S4). The frozen tissues were cut into small pieces measuring 1–4 mm³ over dry ice and subsequently transferred into glass tubes. Before filtering debris using a 40-μm Flowmi Cell Strainer, the tissues were submerged in lysis solution (40 U/ml RNase inhibitor, 21 mM MgCl₂, 146 mM NaCl, 1 mM CaCl₂, 0.1% NP-40, and 10 mM Tris–HCl) and placed on ice for 7 min. The nuclei were isolated by spinning the homogenate at 500× g for 5 min at 4°C, followed by washing with 5 ml of resuspension buffer containing 1× phosphate-buffered saline (PBS) and 1.0% bovine serum albumin (BSA). The nuclei were suspended with 100 μl PBS containing 1.0% BSA after another washing, labeled with trypan blue, and counted using a fluorescent microscope. Based on the guidelines provided by the manufacturer, nuclei underwent prompt processing by using the 10× Chromium platform immediately. Libraries were built using the 10× Genomics Chromium Next GEM Single Cell 3ʹ Reagent Kits v3.1 (1000268), and the Nova 6000 platform was used for sequencing.

Construction of clinical model for tumor response prediction

According to univariate logistic regression analyses of clinical features in the training cohort, we chose a statistically significant variable (tumor number) and clinically meaningful variables (including ALT, AST, maximum tumor size, AFP, and PIVKA-II) to establish a clinical model. All the data were normalized by the Z-score method. We used the SVM model with the standard RBF kernel, enabling regularization (C = 1.0) and probability estimates. The average values of the standard evaluation metrics were determined over a 10-fold cross-validation scheme.

Data preprocessing and two-stage automatic liver tumor segmentation

To extract meaningful features from 3D MRI images of tumor masses, it is crucial to first accurately identify the ROIs within liver tumors. Since the MRI sequences had varying spacing values, we resampled the spacing to a uniform value of [0.83, 0.83, 3.60] to avoid anisotropic effects on the final results and preserve the spatial information of the tumor at a consistent scale. This value was derived by calculating the mean spacing across the entire dataset.

In the initial preprocessing stage, the MRI images were resampled to a 3D volume of 256 × 256 × 48 voxels using the Spline Interpolated Zoom method. Corresponding segmentation labels were resized to match the same dimensions using the Nearest Neighbor interpolation technique. The resized images were then fed into a U-Net network to generate voxel-wise semantic predictions for liver segmentation. Subsequently, the ROI was extracted from the MRI sequence based on the predictions obtained from this first step.

In the second stage, the extracted liver ROI sequence was resampled to a consistent 3D volume of 256 × 256 × 48 voxels to maintain uniformity with the preprocessing of the initial MRI images. This resized liver sequence was then passed through a second U-Net network, which generated a voxel-wise semantic segmentation map specifically for tumor regions. Based on this tumor segmentation, the tumor mass sequence was subsequently extracted and served as the input for the subsequent phase, where treatment response prediction was performed.

The U-Net architecture used here is a widely recognized framework in deep learning and consists of two main components: an encoder and a decoder. In our proposed network, the encoder is constructed using four sequential basic blocks. Each block consists of two convolutional layers, followed by an instance normalization layer and a Leaky ReLU activation layer. The number of feature channels in these basic blocks progressively increases across the encoder, with channel dimensions of 30, 60, 120, and 240, respectively, to capture hierarchical features at different scales. The decoder component of the network consists of three transposed convolutional layers, each with a kernel size of 2 × 2 × 2 and a stride of 2 × 2 × 2, followed by a basic block. Each transposed convolutional layer upsamples the feature map by a factor of 2, which is then concatenated with the corresponding feature map from the encoder through skip connections before being processed by the subsequent basic block. The basic blocks in the decoder progressively reduce the number of feature channels, with dimensions of 120, 60, and 30, respectively. Finally, the last basic block is followed by a 1 × 1 × 1 convolutional layer with a stride of 1 × 1 × 1 to produce the final output. The total architecture of the model is illustrated in Figure 2.

The model was optimized using the Adam optimizer with a learning rate of 0.001. Due to graphics processing unit (GPU) memory limitations, a batch size of 2 was used throughout the training process, which ran for 100 epochs. The loss function was a combination of cross-entropy loss and dice loss, designed to address both class imbalance and segmentation accuracy.

Construction of 3D CNNs for tumor response prediction

Before being input into the 3D CNN model, the extracted liver tumor sequences underwent a series of preprocessing and augmentation steps to enhance their suitability for training. First, all ROIs were resized to a standardized 3D volume of 128 × 128 × 48 voxels to ensure uniformity in input dimensions. Following this, the intensity levels of the sequences were normalized by computing their mean and standard deviation. Finally, to increase the diversity and robustness of the training data, various data augmentation techniques were applied, including random rotation, scaling, flipping, and elastic transformations. These steps collectively improve the model’s ability to generalize and perform well on unseen data.

The 3D CNN model proposed in this study is depicted in Figure 4A. The network architecture consists of four 3D basic residual blocks, each employing a residual structure with skip connections that combine the input x with the output of an alternate branch. Specifically, the first residual block employs 64 filters, the second utilizes 128 filters, the third incorporates 256 filters, and the fourth integrates 512 filters. Following these residual blocks, a global average pooling layer is applied to summarize the spatial information, and its output is passed to a fully connected layer. To mitigate overfitting, a dropout layer with a probability of 0.5 is introduced before the fully connected layer. Finally, a softmax layer is used to perform the binary classification task, producing the final output for distinguishing between PD and non-PD cases.

To train and validate the network, a 10-fold cross-validation strategy was implemented, repeated over five iterations, with each fold being randomly stratified to ensure balanced data distribution. The binary cross-entropy loss function was used to guide the training process. The network weights were optimized using the Adam optimizer with a learning rate of 0.0001. Training was conducted for 100 epochs, with a fixed batch size of 8. The model’s performance in distinguishing between PD and non-PD cases was evaluated using the AUC. Additionally, standard evaluation metrics—sensitivity, specificity, and accuracy—were computed to assess classification effectiveness. Mean values and CIs were derived from the 10-fold cross-validation results to provide robust statistical insights.

Development of a multimodal model for tumor response prediction

To leverage clinical data effectively, we extracted clinical features and integrated them with image features to enhance predictive performance. We developed a multimodal architecture designed to simultaneously learn from both imaging and clinical data for accurate prediction of PD and non-PD cases.

In the multimodal model, we integrated the previously introduced CNN module with a Transformer module. The overall architecture of the model is illustrated in Figure 4B. The Transformer block comprised two encoder layers, each with a hidden size of 256 and 8 self-attention heads. A dropout probability of 0.2 was applied within the encoder layers to prevent overfitting. To preserve positional information, a position embedding was added to the patch embeddings before feeding them into the encoder layers. We employed a standard learnable 1D position embedding with a maximum length of 60 and a dimension of 64. Both the CNN and Transformer modules independently generated output vectors with the size of 512. These two vectors were then concatenated into a single 1024-dimensional vector, which was subsequently passed through a dense layer to reduce its dimensionality to 256. Finally, for the binary classification task, a dense layer followed by a softmax activation was applied to produce the final output. This architecture effectively combines the strengths of both CNN and Transformer modules, enabling robust feature extraction and classification.

A 10-fold cross-validation procedure, repeated across five independent rounds, was employed to train and evaluate the network. Each fold was randomly stratified to ensure balanced representation of classes. Throughout the training process, key hyperparameters—learning rate, optimizer, number of epochs, batch size, and loss function—were maintained consistently with those used in the CNN model training. The implementation was carried out using the Python programming language and the PyTorch 2.3.0 framework. For computational resources, both the CNN and multimodal trainings were executed on a hardware setup comprising a GeForce GTX 3090 GPU, an Intel^® Xeon^® Bronze 3204 CPU @ 1.90 GHz processor, and 256 GB of random-access memory.

Grad-CAM

In our study, we employed 3D Grad-CAM to visually interpret the decision-making process of our 3D CNN model in classifying subjects as either PD or non-PD. It generates a class activation heatmap for each slice of the 3D MRI sequence, highlighting the regions that contribute most significantly to the model’s prediction for each class (PD or non-PD).

For this analysis, the heatmaps were derived from the final residual block of the network, which captures both low-level and high-level semantic features. To demonstrate the effectiveness of this approach, we selected four representative cases from both the PD and non-PD groups for detailed visualization and interpretation.

Statistical analyses

Categorical variables were given as counts with percentages (%), while continuous variables were expressed as medians with interquartile ranges. Fisher’s exact test or the χ² test was used to compare categorical variables. The Student’s t-test was used to compare parametric continuous variables, while nonparametric variables were compared using the Kruskal–Wallis H test or the Mann–Whitney U test. Multivariate logistic regression analysis was employed to determine independent predictive factors. Statistical tests were two-tailed, and P ≤ 0.05 was used to classify variables as statistically significant. R software (version 3.5.1) and Python (version 3.13) were used for all statistical analyses and calculations.

Funding

This study was supported by the National Science Foundation for Young Scientists of China (81301821).

Conflict of interest: none declared.

Author contributions: B.Q. constructed the CNN models, analyzed data, and drafted the manuscript. M.D. collected data, constructed liver tumor segmentation and the CNN models, analyzed data, and drafted the manuscript. P.Z. collected data, conducted single-nucleus RNA sequencing analysis, and drafted the manuscript. S.C. analyzed data and co-drafted the manuscript. J.C. established a liver tumor segmentation model and co-drafted the manuscript. P.X. conceived the study and supervised the project. P.L. conceived the study and supervised the project. L.Y. conceived the study, provided funding, and supervised the project. All authors have read and approved the final manuscript.