Lightweight hybrid foundation model for lung cancer prognosis based on low-dose chest X-ray images

Abstract

Background

Lung cancer (LC) is the leading cause of cancer mortality worldwide. Accurate prognosis in vulnerable populations, particularly in low-resource settings remains challenging with high radiation dose of standard imaging modalities. While chest X-rays (CXRs) are safer and more accessible, their low spatial resolution limits prognostic accuracy. Existing multimodal models primarily rely on computationally intensive architectures, such as computed tomography (CT) or positron emission tomography (PET) inputs, which reduce their clinical utility. This study aimed to develop and validate a lightweight hybrid foundation model (LHFM) that integrates CXR imaging with clinical data to enable accurate LC prognosis in low-resource settings.

Methods

We developed a LHFM that integrates visual features from CXR images extracted by a segment anything model (SAM)-Med2D encoder with semantically enriched prompts generated by BioGPT and clinical metadata. These multimodal features are fused via a dual-branch transformer architecture for survival prediction. The model was trained and validated on the JSRT and PadChest datasets, with external validation on multicenter datasets including the NIH CXR. Performance was evaluated using the concordance index (C-index), area under the receiver operating characteristic curve (AUROC), and Kaplan-Meier (KM) survival analysis.

Results

The proposed LHFM achieved superior prognostic performance, with a C-index of 0.910 [95% confidence interval (CI): 0.898–0.922, standard deviation (SD) =0.006] and AUROC of 0.935 (95% CI: 0.927–0.943, SD =0.004), outperforming existing multimodal benchmarks (P<0.001). KM curves demonstrated significant separation between the high-risk and low-risk groups. Domain-shift robustness testing across heterogeneous external datasets demonstrated representation stability under distribution shift.

Conclusions

LHFM establishes a new paradigm for prognostic precision by exhibiting significant performance from low-dose CXR. This hybrid approach directly addresses the implementation gap in clinical artificial intelligence (AI), offering a scalable, equitable, and immediately applicable solution for personalized cancer care in resource-limited and radiography first workflows, with potential applicability across other cancer types.

Highlight box.

Key findings

• This study proposes a lightweight hybrid foundation model (LHFM) that integrates chest X-ray (CXR) image features extracted by segment anything model (SAM)-Med2D with semantic prompts generated by BioGPT and clinical metadata for lung cancer prognosis.

• LHFM achieved a concordance index of 0.910 and an area under the receiver operating characteristic curve (AUROC) of 0.935, outperforming existing multimodal benchmarks.

• The model demonstrated strong external generalizability and robustness under domain shift across three independent datasets [concordance index (C-index): 0.835–0.879], confirming its robustness and translational potential.

What is known and what is new?

• Lung cancer prognostic models often rely on computed tomography (CT) or positron emission tomography (PET), which are limited by high radiation exposure, cost, and accessibility in low-resource settings. Although CXR is more accessible and involves lower radiation, its reduced spatial resolution hinders the performance of traditional deep learning models.

• In this study, we introduce a LHFM that integrates CXR features with clinical data through a multimodal framework, offering an accurate prognosis. LHFM provides a scalable, low-dose solution suitable for resource-limited environments.

What is the implication, and what should change now?

• LHFM provides a cost-effective and scalable solution for integrating artificial intelligence (AI)-aided prognosis into clinical workflows using standard radiography equipment.

• The framework can support early triage, treatment prioritization, and equitable access to advanced prognostic tools in low-resource and vulnerable populations environments.

• Future clinical adoption should focus on prospective validation and integration into radiology information systems to enhance precision oncology and workflow efficiency.

Introduction

Lung cancer (LC) remains the leading cause of cancer-related mortality worldwide, accounting for over 1.8 million deaths annually (1,2). Accurate prognostication is crucial for personalized treatment, yet existing models rely on computed tomography (CT) or positron emission tomography (PET) imaging modalities (3-6) which are limited by cost, radiation exposure, and access barriers in vulnerable populations and low-resource settings (7,8). By contrast, chest X-ray (CXR) is inexpensive, widely accessible, and involves substantially lower radiation dose (9-12), but its two-dimensional, lower-resolution nature hinders reliable extraction of subtle prognostic signals.

Recent deep learning (DL) studies have demonstrated that CXR images encode prognostic information (13,14); however, most frameworks remain unimodal, image-only, underutilizing complementary clinical context available in radiology reports or electronic health record (EHR) data (15,16). Consequently, these models lack interpretability and are impractical for real-world deployment. Existing multimodal architectures, including Lite-ProSENet (17) and FGCN (18), achieve significant improvements concordance index (C-index) ≈0.76–0.78, but they require high-resolution CT and computationally expensive transformers, which restricts their clinical scalability (19). In contrast, other radiomics and omics-based approaches report more modest performance (C-index ≈0.65–0.78) and often lack external validation or interpretability (20-23). Yet, these methods depend on high-resolution, high-radiation imaging and computationally intensive architectures, limiting scalability beyond specialized centers. Collectively, current evidence underscores a critical gap: there is no lightweight, interpretable, CXR-based multimodal framework that leverages recent advances in medical foundation encoders while remaining deployable in routine practice and resource-limited clinical environments.

Emerging medical vision-language models, such as segment anything model (SAM)-Med2D for segmentation (24) and BioGPT for biomedical language understanding (16), enable rich visual-text representations but are typically embedded in large, resource-intensive pipelines (25). Harnessing their representational strength within a compact architecture optimized for CXR-based risk stratification offers an opportunity to deliver accurate, interpretable prognosis without relying on CT/PET or high-end computing resources.

To address these gaps, we propose a lightweight hybrid foundation model (LHFM) designed to integrate CXR-derived embeddings from SAM-Med2D with BioGPT-generated clinical prompts and structured metadata through a dual-branch transformer. We hypothesize that integrating SAM-Med2D visual embeddings with BioGPT-generated semantic prompts and clinical data will achieve significant survival prediction accuracy from low-dose CXRs compared with unimodal and heavy multimodal baselines. The main contributions of this study are as follows.

• ❖ We propose a LHFM that integrates SAM-Med2D CXR features, BioGPT-generated semantic prompts, and structured clinical metadata through a unified dual-branch transformer architecture for LC prognosis.

• ❖ LHFM bridges the gap between high-capacity foundation models and deployable clinical aids by combining visual embeddings from SAM-Med2D with textual semantics from BioGPT, achieving high accuracy while preserving interpretability and scalability.

• ❖ Across multiple CXR-based datasets, LHFM consistently outperformed existing unimodal and multimodal baselines, with semantic prompt–guided fusion enhancing both interpretability and prognostic performance.

• ❖ The framework demonstrates robust cross-domain generalization and enables real-time inference on standard hardware, underscoring its translational potential for resource-limited and clinical settings.

We present this article in accordance with the TRIPOD reporting checklist (available at https://tlcr.amegroups.com/article/view/10.21037/tlcr-2025-aw-1299/rc).

Methods

Datasets

This study utilized multiple publicly available chest CXR and CT datasets for model development and validation, and to assess domain adaptability. The model was developed and validated primarily on the JSRT and PadChest CXR datasets, which provide high-quality annotated images alongside demographic and clinical metadata, enabling robust prognostic model development across diverse patient groups and imaging protocols (26,27). Baseline demographic and clinical characteristics for the primary development cohort (JSRT) are summarized in Table S1. For experimental validation, these datasets were split into training (80%) and testing (20%). To minimize unintended correlations and prevent information leakage, this study used advanced splitting such that samples from the same subject did not appear in both training and testing partitions. Model stability was assessed using 5-fold cross-validation on the development datasets, and the held-out test set remained strictly isolated until final evaluation, and sample images are presented in Figure 1. The model’s generalizability and translational potential was performed on heterogeneous datasets, including the large-scale NIH ChestX-ray and the NSCLC-Radiomics-Interobserver1 datasets, which represent varied demographic distributions and clinical contexts. In addition to internal testing, we evaluated LHFM under marked distribution shifts using heterogeneous external imaging cohorts. Because some datasets (e.g., Shenzhen/Montgomery) (28,29) are not LC cohorts, these analyses are reported as domain-shift robustness and feature transferability tests, rather than clinical external validation of LC prognosis. This evaluation strategy probes whether multimodal representations learned from low-dose radiographs remain stable under substantial differences in acquisition, pathology prevalence, and cohort composition. Furthermore, the NSCLC-Radiomics-Interobserver1 dataset was incorporated to assess the model’s performance (30).

Figure 1.

Sample lung CXR images. The figure depicts three distinct clinical classifications: (A) a malignant nodule, (B) a benign nodule, and (C) a non-nodule appearance for comparative analysis. CXR, chest X-ray.

Accordingly, the primary endpoint used for training and evaluation was an algorithmically computed risk label (high-risk vs. low-risk) derived from available demographic and clinical metadata under controlled experimental assumptions. This design enables systematic assessment of whether hybrid fusion of imaging features and semantic prompt representations improves predictive discrimination in a low-dose CXR setting, while avoiding unsupported claims of actual time-to-event clinical outcome prediction.

The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. All datasets used in this study were publicly available (as presented in Appendix 1) and fully anonymized, and institutional review board approval was waived. Missing data in demographic or clinical metadata were handled by imputation with median values for continuous variables and mode for categorical variables, confirming balanced input across patient groups.

Clinical prompts enhance LC prognosis by integrating key tumor features such as size, density, margins, and location into a lightweight predictive model. This method integrates imaging and clinical data to provide interpretable, efficient, and highly accurate predictions, making it suitable for diverse healthcare settings. In this study, clinical prompts were used as auxiliary inputs during model training.

Proposed methods

This section describes the proposed method for determining the prognosis of patients with LC using low-dose CXR. The main workflow includes four steps: data preprocessing, feature extraction, hybrid model training, and prognostic prediction, as presented in Figure 2. We begin by outlining data preprocessing techniques for robust and generalizable inputs, followed by model architecture, training, and prognostic prediction.

Figure 2.

Block diagram of the proposed LHFM framework. The pipeline begins with multimodal data input, including chest X-ray images and clinical metadata prompts, followed by preprocessing tasks such as normalization and augmentations. Feature extraction is then performed via SAM-Med2D for visual representations and BioGPT for semantic text embeddings. These complementary features are combined within a hybrid fusion model, which subsequently enables robust survival prediction and prognostic classification for accurate prognostic prediction. LHFM, lightweight hybrid foundation model.

Data preprocessing

The preprocessing pipeline standardized the CXR images from the adult and other cohorts by resizing them to 224×224 pixels and normalizing the pixel intensities. Data augmentation, including rotation, scaling, and cropping, was applied to improve model robustness. Clinical metadata were processed to handle missing values, ensuring consistent and balanced input for subsequent multimodal feature extraction and model training. Pixel intensity values were normalized via z-score standardization as depicted in Eq. [1]. Clinical metadata were processed via a parallel preprocessing pipeline.

$$I^{\prime }=\frac{I-\mu }{\sigma }$$ [1]

where $I$ is the raw pixel intensity, $\mu$ is the dataset-wide mean, and $\sigma$ is the standard deviation (SD), which is computed per dataset to account for acquisition-specific illumination and contrast variations.

Statistical analysis

Statistical analysis was performed to evaluate the model’s prognostic performance and robustness. The C-index assessed survival ranking, while the area under the receiver operating characteristic curve (AUROC) evaluated binary classification. Kaplan-Meier (KM) analysis with the log-rank test compared survival between risk groups. Performance is reported as mean ± SD with 95% CI from five-fold cross-validation. Sensitivity analyses included modality ablation and external validation. All analyses were conducted using Python (version 3.10) as shown in Table S2. P values smaller than 0.05 were considered statistically significant.

Constructing a LHFM

Figure 3 demonstrates the conceptual transition from traditional and DL models to the proposed model (LHFM), highlighting its multimodal efficiency and interpretability. The framework and implementation of the proposed model, are shown in Figure 4, which consists of a dual-branch feature extractor, a fusion model, and a prognostic prediction module. The details are explained below.

Figure 3.

Comparative schematic of lung cancer prognosis architectures. (A) Shows that traditional statistical models rely solely on clinical data. (B) shows DCNN-based approaches that use single-modality features. (C) presents multimodal models such as Lite-ProSENet, which integrate clinical data and CT scans but require high-radiation imaging. (D) illustrates the proposed LHFM, which integrates SAM-Med2D CXR embeddings with BioGPT-generated prompts and clinical metadata through a hybrid fusion model. CT, computed tomography; CXR, chest X-ray; DCNN, deep convolutional neural network; LHFM, lightweight hybrid foundation model.

Figure 4.

Overall framework of the lightweight hybrid foundation model. Preprocessed CXR images and clinical text are encoded by SAM-Med2 and BioGPT to yield 512-D image and 768-D text embeddings with clinical metadata. A fusion module, comprising a fusion layer, an attention-based block, and fully connected layers, outputs a sigmoid-activated risk probability dichotomized into low- versus high-risk classes. CXR, chest X-ray.

Dual hybrid feature extraction

In this stage, visual features are extracted from the preprocessed CXR images via the SAM-Med2D (24) visual encoder. It generates high-fidelity embeddings that capture spatial context and critical visual features from CXR, facilitating the subsequent integration of visual information into the model. In parallel, BioGPT (16), a biomedical language model, prompts are image-conditioned semantic representations, generated from CXR-derived embeddings using a lightweight learned prompt generator that bridges the visual latent space to BioGPT’s token embedding space. The generated prompt functions as an auxiliary semantic representation that supports multimodal fusion and interpretability. Importantly, the prompt branch is designed as a representation-learning mechanism, rather than a standalone clinical text prognostic model. Qualitative examples of generated prompts and additional prompt-sensitivity experiments are provided in the Table S3. For example, a generated prompt might be: “Presence of a speculated mass with lesion margins in the right upper lobe, approximately 3.2 cm in diameter.” These prompts offer interpretable descriptions of clinical conditions, effectively bridging pixel-level features to high-level semantic descriptions.

Dual hybrid feature extraction combines visual and textual pathways to create clinically informative CXR representations. Let $X={\left\{x_i\right\}}_{i=1}^N$ denote the set of preprocessed CXR images, where each $x_i\in {ℝ}^{H\times W\times 3}$ starts by projecting through a vision transformer (ViT) backbone whose patch-token output is linearly mapped to a compact visual embedding $v_i=W_{V}ViT\left(x_i\right)+b_v$. A lightweight prompt generator $\phi p(\cdot )$ then converts $v_i$ into a clinically interpretable prompt $p_i$. This prompt is tokenized and passed through BioGPT, whose [CLS] output $h_o\left(p_i\right)$ is similarly linearly projected to yield the text embedding $t_i=W_T{h}_o\left(p_i\right)+b_T$. By aligning $v_i$ and $v_i$ in a shared latent space (${ℝ}^{d_v}$ and ${ℝ}^{d_t}$), we fuse pixel-level detail with semantic descriptors, enabling the model to ground high-level clinical concepts directly in image features.

Let $X={\left\{x_i\right\}}_{i=1}^N$ denote the set of preprocessed CXR images, where each $x_i\in {ℝ}^{H\times W\times 3}$. We first extract a visual embedding $v_i\in {ℝ}^{d_v}$ for each image via the SAM-Med2D encoder as shown in Eqs. [2-4].

$$v_i=f_{SAM}\left(x_i\right)$$ [2]

where $f_{SAM}:{ℝ}^{H\times W\times 3}\to {ℝ}^{d_v}$ is implemented as a ViT with a projection head. Specifically, if $z_i=ViT\left(x_i\right)\in {ℝ}^D$ are the raw patch-token outputs, then visual extraction employs the SAM-Med2D (24).

$$v_i=W_v{z}_i+b_v,W_v\in {ℝ}^{d_v\times D},b_v\in {ℝ}^{d_v}$$ [3]

encoder, $v=f_v\left(X\right)$, which processes input image $X$ into spatial embeddings $v\in {ℝ}^d$ using a transformer architecture fine-tuned for medical imaging. Concurrently, the clinical textual prompt $p_i$ from $v_i$, via prompt-generator $g$ is embedded with BioGPT (16):

$$p_i=g\left(v_i\right),t_i=f_{BioGPT}\left(p_i\right)$$ [4]

where $t_i\in {ℝ}^{d_t}$ represents the number of clinical text embeddings. Here $g$ is a small feed-forward network and $t_i=W_t{h}_o\left(p_i\right)+b_t$, with $h_o\left(p_i\right)\in {ℝ}^{H_o}$ the token output of BioGPT, and $W_t\in {ℝ}^{d_t\times H_o}$, $b_t\in {ℝ}^{d_t}$. $t=f_t\left(v\right)$, where cross-modal attention dynamically synthesizes clinical prompts conditioned on $v$ (visual embeddings). This cross-attention mechanism, $Attn\left(Q=v{W}_Q,K=t{W}_K\right)$, explicitly aligns pixel-level features with semantic descriptors, ensuring that prompts are anatomically grounded and clinically interpretable.

Hybrid multimodal fusion model

The main model training stage involves integrating visual embeddings and semantic prompts via a lightweight dual-branch transformer model. This multimodal fusion architecture outputs scalar risk scores indicative of prognosis. The fusion model $f_i$ integrates visual ($V$), textual ($T$), and clinical ($\text{ C}\in {ℝ}^{d_c}$) features via a dual-branch transformer. Let $X\in {ℝ}^{H\times W\times 3}$ denote an input CXR. The vision encoder $f_v:{ℝ}^{H\times W\times 3}\to {ℝ}^{d_v}$ employs a patch-based transformer. The inputs are visual features $V\in {ℝ}^{d_v}$, textual features $T\in {ℝ}^{d_t}$, and $V\in {ℝ}^{d_c}$. The modality-specific projections are fused as depicted in Eq. [5].

$$\begin{array}{l}{h}_v=ReLU\left(W_{v}V+b_v\right),\\ h_t=ReLU\left(W_{t}T+b_t\right),\\ h_c=ReLU\left(W_{c}C+b_c\right)\end{array}$$ [5]

where $W_{\text{*}}\in {ℝ}^{3d_h}$ are learnable weights. Tokenization and iterative fusion are employed as shown in Eqs. [5,6].

$$M_i^{\left(0\right)}=LN\left(W_m\left[v_i\left|\left|t_i\right|\right|c_i\right]+b_m\right)\in {ℝ}^d$$ [6]

where $LN$ represents layer normalization, || represents the concatenation operator, $W_m\in {ℝ}^{d\times \left(d+d+d\right)}$, and $b_m\in {ℝ}^d$ represents the fusion weights/bias.

Prognostic prediction

Our prognosis-prediction module embeds CXR features, BioGPT-derived semantic prompts, and tabular covariates into a shared latent vector $H_i$. A dual-head architecture then produces (I) a calibrated continuous hazard score for time-event analysis and (II) a SoftMax posterior that dichotomizes patients into low versus high-risk strata. The transformer backbone first aligns modality-specific embeddings through stacked multihead fusion-attention layers, a design that has already outperformed conventional pipelines (31). The fused representation is routed to a survival head, $S_i=\sigma \left(W_r^T{H}_i+b_r\right)$, mirroring the continuous-risk formulation and to a classification head, $\widehat{y_{\mathfrak{i}}}=\left(W_c{H}_i+b_c\right)$, which supports binary triage consistent with recent vision-language prognostic frameworks for CXR analysis. The survival risk and discrete outcome are depicted in Eqs. [7,8], respectively. A sigmoid-activated risk score $S_i$ quantifies individual survival probability, whereas a parallel SoftMax head $\widehat{y_{\mathfrak{i}}}$ yields class-level prognosis. Both heads share the fused latent $H_i$, ensuring that predictions are informed simultaneously by pixel-level patterns, semantic prompts, and structured patient data (32).

$$r_i=W_r^T{H}_i+b_r,S_i=\sigma \left(r_i\right)\in \left(0,1\right)$$ [7]

$$\widehat{y_{\mathfrak{i}}}=softmax\left(W_c{H}_i+b_c\right)\in {\Delta }^{K-1}$$ [8]

Loss function of LHFM

The model was optimized via weighted binary cross-entropy loss (Eq. [9]) to address class imbalance, with a positive-class weight $\alpha =\raisebox{1ex}{$N_{neg}$}\!\left/ \!\raisebox{-1ex}{$N_{pos}$}\right.$ applied to increase the recall of high-risk cases. Training employed the Adam optimizer (β₁=0.9, β₂=0.999, weight decay =1×10^–4, learning rate =1×10^–4) with early stopping, yielding stable convergence and well-calibrated risk predictions suitable for clinical decision support. The simplified gradient $\sigma \left(z_i\right)-y_i$ ensures numerically stable updates under mixed-precision training.

$$\mathcal{L}BCE=\frac{1}{N}{\displaystyle \sum _{i=1}^N\left[\text{max}\left(0,z_i\right)-y_i{z}_i+nl\left(1+e^{-\left|z_i\right|}\right)\right]}$$ [9]

Reproducibility and implementation details

All experiments were implemented in Python using PyTorch and trained on an NVIDIA GeForce RTX 2080Ti GPU. To enhance reproducibility and minimize run-to-run variability, random initialization was controlled using a deterministic seed (applied across Python/NumPy/PyTorch, including CUDA where applicable). CXR images were preprocessed using a standardized pipeline, including resizing all inputs to 224×224 pixels and applying consistent intensity normalization to stabilize optimization and improve convergence. Model training was performed using the Adam optimizer with a batch size of 16, and an early stopping strategy was applied based on validation performance to reduce overfitting and retain the best-performing checkpoint. To improve generalization, radiography-appropriate data augmentation was incorporated during training. Model evaluation was conducted using 5-fold cross-validation, and final performance is reported as mean ± SD across folds along with 95% confidence intervals (CIs) to quantify uncertainty and ensure reliable comparison across experiments. Specific details were presented in Table S2.

Evaluation metrics

The experimental results were evaluated via the concordance index (C-index) to assess prognostic accuracy (33) as depicted in Eq. [10], alongside the area under the AUROC to compute classification performance. Model performance was reported as mean ± SD across folds, with 95% CI, enabling robust estimation of variability and ensuring reproducibility under repeated sampling. The statistical significance of improvements over baselines was validated by CIs and interpretability analysis quantified the contributions of imaging and textual features to the predictions (34).

$$\text{C-index}=\frac{{\displaystyle {\sum }_{i,j}I\left({\widehat{y}}_i<{\widehat{y}}_j|e_i=1\text{,\hspace{0.17em}}{y}_i<y_j\right)}}{{\displaystyle {\sum }_{i,j}I\left({\widehat{y}}_i<{\widehat{y}}_j|e_i=1\right)}}$$ [10]

where ${\widehat{y}}_i<{\widehat{y}}_j$ are the predicted risk scores for individuals $i$ and $j$, and where $y_i$ and $y_j$ are the actual observed survival times. $e_i$ is an event indicator for individual $i$ (1 if the event occurred, 0 if censored), and $I(\cdot )$ is an indicator function that returns a value of 1 if the condition inside is true and 0 otherwise.

Moreover, KM survival estimation is utilized in LC prognosis, which robustly handles values, allowing accurate survival function estimation even when patients are lost to follow-up or when the event has not occurred by the end of the study (35), as depicted in Eq. [11].

$$\widehat{S}\left(t\right)={\displaystyle \prod _{t_{\left(j\right)}\le t}\left(1-\frac{d_j}{n_j}\right)}$$ [11]

where $\widehat{S}\left(t\right)$ represents the probability that a patient survives longer than $t$ times after diagnosis or treatment starts, $t_{\left(j\right)}=j^{th}$ represents the ordered event time, $d_j$ indicates the number of events at time $t_{\left(j\right)}$, $n_j=$ represents the number of individuals at risk just before $t_{\left(j\right)}$, and the product is taken over all distinct event times up to $t$.

Results

Performance comparisons with related works

The progressive improvement from unimodal to multimodal fusion models highlights the intrinsic value of heterogeneous data integration in survival prediction tasks. Specifically, the baseline clinical model demonstrated limited prognostic discrimination (C-index: 0.696±0.008) as presented in Table 1, underscoring the inherent limitations of traditional approaches that rely on structured tabular data. In contrast, the FGCN model’s modest gains (C-index: 0.769±0.007) suggest that model inter-feature dependencies can partially address these limitations.

Table 1. Performance comparisons with related works on the primary test set (JSRT and PadChest).

Model	Modality	AUROC			C-index
		Mean ± SD	95% CI		Mean ± SD	95% CI
Baseline	Unimodal	0.750±0.007	0.736–0.764		0.696±0.008	0.680–0.712
Lite-ProSENet (17)	Multimodal	0.919±0.004	0.911–0.927		0.893±0.005	0.883–0.903
FGCN (18)	Multimodal	0.834±0.005	0.824–0.844		0.769±0.007	0.755–0.783
Schulz et al. (21)	Multimodal	0.852±0.006	0.840–0.864		0.779±0.007	0.765–0.793
Deng et al. (22)	Multimodal	0.732±0.006	0.720–0.744		0.659±0.008	0.643–0.675
LHFM (ours)	Multimodal	0.935±0.004	0.927–0.943		0.910±0.006	0.898–0.922

AUROC, area under the receiver operating characteristic curve; C-index, concordance index; CI, confidence interval; LHFM, lightweight hybrid foundation model; SD, standard deviation.

These trends are consistent with the recent studies where multimodal or habitat-aware CT models achieved strong but CT-dependent survival or response prediction in NSCLC patients (19,36-38). The C-index of 0.910±0.006 (95% CI: 0.898–0.922) confirms that LHFM effectively integrates image and text information, outperforming multimodal baselines, thereby overcoming the gap that often hinders unimodal DL approaches.

The most significant improvement is observed over Deng et al. (22), with AUROC and C-index increases of 0.203 and 0.251, respectively, highlighting a substantial increase in both classification discrimination and survival ranking capabilities. Similarly, compared with the baseline unimodal model, LHFM achieved improvements of 0.185 in the AUROC and 0.214 in the C-index, underscoring the benefit of integrating multimodal data. Notably, while Lite-ProSENet (17) exhibited strong performance, LHFM still achieved significant improvements (AUROC: +0.016, C-index: +0.017), reflecting that its advantages persist even against high-performing multimodal architectures.

These improvements validate LHFM’s multimodal hybrid fusion approach, effectively leveraging complementary information from different data types. The consistent superiority across all comparisons suggests that LHFM generalizes well to different baseline architectures and is not overly tuned to any single reference model. This makes LHFM a clinically promising framework for LC prognosis, with potential applicability across diverse datasets and clinical scenarios.

KM survival curve

KM survival analysis confirmed significant stratification between the high-and low-risk groups identified by LHFM (log-rank P ≈6.42×10⁻¹³), as shown in Figure 5. The high-risk group exhibited a survival decline, with the probability of survival decreasing to less than 0.4 within 12 months and approaching zero by 60 months, whereas the low-risk group maintained substantially greater survival throughout follow-up. The minimal overlap of the 95% CI beyond the initial time points reinforced the robustness of this separation. LHFM risk stratification aligns with clinically significant outcome differences, demonstrating strong prognostic utility (C-index: 0.910; AUROC: 0.935) as validated in Table 2.

Figure 5.

KM survival curves for LHFM risk stratification. The LHFM model strongly differentiates between high and low-risk patients, with a marked gap in survival curves. KM, Kaplan-Meier; LHFM, lightweight hybrid foundation model.

Table 2. LHFM performance on external multimodal datasets.

Dataset	C-index (mean ± SD, 95% CI)
NIH ChestX-ray dataset (28)	0.851±0.006 (0.839–0.863)
NSCLC-Radiomics-Interobserver1 dataset (30)	0.879±0.005 (0.869–0.889)
The Shenzhen + Montgomery datasets (29)	0.835±0.007 (0.821–0.849)

CI, confidence interval; C-index, concordance index; LHFM, lightweight hybrid foundation model; SD, standard deviation.

Generalizability

Table 2 shows that LHFM preserves exceptional prognostic discrimination across a spectrum of external cohorts. To evaluate external validity and generalizability, the model was trained on the JSRT and PadChest datasets and directly tested on independent external datasets. For the expansive NIH CXR dataset (28), the model achieves a C-index of 0.851, attesting to its robustness within adult radiography pipelines. The performance further increased to 0.879 on the volumetric, expert-annotated NSCLC Radiomics Interobserver1 (30), indicating that the integration of rich 3D morphological information can meaningfully enhance survival stratification.

Notably, even on the comparatively limited and tuberculosis-oriented Shenzhen and Montgomery CXR datasets (29), LHFM achieved a C-index of 0.835, reflecting its performance reduction amidst domain and pathology shifts. All values exceeded 0.80, confirming the clinical relevance and robustness under distribution shifts, collectively underscoring LHFM’s capacity to generalize across diverse imaging modalities, spatial resolutions, cohort scales, and disease spectra with minimal adaptation. All external validation values are reported with 95% CIs.

External validation across three independent datasets (NIH CXR, NSCLC-Radiomics-Interobserver1, and Shenzhen-Montgomery) demonstrated strong generalizability (C-index range: 0.835–0.879), confirming that LHFM maintains prognostic accuracy under significant demographic and modality variation.

Ablation study

An ablation study conducted on a stratified hold-out test set (20% of the dataset) confirmed that visual and textual components are essential for LHFM’s prognostic performance. The removal of either modality resulted in significant performance degradation, as depicted in Table 3. Ablation study performance with the text-only variant exhibited particularly substantial declines, underscoring the indispensability of imaging data for capturing significant anatomical details. Multimodal fusion via attention-based integration outperformed naive fusion strategies, with the full model (LHFM) achieving optimal performance (AUROC: 0.935±0.004; C-index: 0.910±0.006). These results confirm the synergistic interaction between imaging and clinical text modalities, whereas visualization analyses, as depicted in Figure 6, further validated the model’s interpretability and class-agnostic robustness.

Table 3. Ablation study performance.

Model variant	Modification	AUROC			C-index
		Mean ± SD	95% CI		Mean ± SD	95% CI
Image-only	Text encoder removed	0.873±0.007	0.859–0.887		0.842±0.009	0.824–0.860
Prompt-only	Image encoder removed	0.824±0.006	0.812–0.836		0.794±0.008	0.778–0.810
Full LHFM (ours)	None (image encoder + text encoder)	0.935±0.004	0.927–0.943		0.910±0.006	0.898–0.922

AUROC, area under the receiver operating characteristic curve; CI, confidence interval; C-index, concordance index; LHFM, lightweight hybrid foundation model; SD, standard deviation.

Figure 6.

Performance visualization of the proposed model. (A) AUROC for the full LHFM model compared against image-only and prompt-only variants on the hold-out test set. (B) Confusion matrix for binary risk classification at the optimal threshold determined by Youden’s index. (C) Sample visualization from the test set: (left) original CXR image with ground-glass opacity, (middle) model-generated heatmap localizing the region of high prognostic saliency, (right) attention map from the fusion module highlighting features correlated with the generated clinical prompt. AUC, area under the curve; AUROC, area under the receiver operating characteristic curve; CXR, chest X-ray; LHFM, lightweight hybrid foundation model; ROC, receiver operating characteristic.

Sensitivity analyses and robustness evaluation

To address concerns regarding potential prompt learning and unusually high discrimination metrics, we conducted sensitivity analyses across benchmark comparisons, modality ablations, and external robustness evaluation. The full multimodal LHFM achieved the strongest overall performance (AUROC =0.935±0.004; C-index =0.910±0.006). Removing individual branches resulted in consistent degradation, including the image-only model (AUROC =0.873±0.007; C-index =0.842±0.009) and the prompt-only model (AUROC =0.824±0.006; C-index =0.794±0.008), supporting that performance gains are not attributable to a single dominant modality. These experiments confirm that LHFM benefits from complementary multimodal integration rather than trivial shortcuts (Table 4).

Table 4. Sensitivity analyses and model robustness evaluation (95% CI).

Model	Analysis type	AUROC			C-index		Fusion description
		Mean ± SD	95% CI		Mean ± SD	95% CI
LHFM (ours)	Reference (primary)	0.935±0.004	0.927–0.943		0.910±0.006	0.898–0.922	Full hybrid fusion model (image + prompt- integration)
Image-only	Ablation	0.873±0.007	0.859–0.887		0.842±0.009	0.824–0.860	Prompt encoder removed (imaging backbone only)
Prompt-only	Ablation	0.824±0.006	0.812–0.836		0.794±0.008	0.778–0.810	Image encoder removed (prompt-only)
Baseline	Benchmark	0.750±0.007	0.736–0.764		0.696±0.008	0.680–0.712	Unimodal baseline
Lite-ProSENet (17)	Benchmark	0.919±0.004	0.911–0.927		0.893±0.005	0.883–0.903	Multimodal comparator
FGCN (18)	Benchmark	0.834±0.005	0.824–0.844		0.769±0.007	0.755–0.783	Multimodal comparator
Schulz et al. (21)	Benchmark	0.852±0.006	0.840–0.864		0.779±0.007	0.765–0.793	Multimodal comparator
Deng et al. (22)	Benchmark	0.732±0.006	0.720–0.744		0.659±0.008	0.643–0.675	Multimodal comparator

AUROC, area under the receiver operating characteristic curve; CI, confidence interval; C-index, concordance index; LHFM, lightweight hybrid foundation model; SD, standard deviation.

Sensitivity analyses demonstrated that LHFM remained robust across ablation experiments, benchmark comparisons, and external evaluations (Table 4). The full multimodal LHFM achieved the strongest overall performance (AUROC =0.935±0.004; C-index =0.910±0.006). Removing individual branches resulted in consistent performance degradation, including the image-only variant (AUROC =0.873±0.007; C-index =0.842±0.009) and the prompt-only variant (AUROC =0.824±0.006; C-index =0.794±0.008), supporting the complementary value of hybrid fusion. External robustness was maintained across independent datasets with C-index values ranging from 0.835 to 0.879, confirming generalizable prognostic discrimination under dataset shift (Table S3).

Discussion

This study introduces LHFM, which integrates visual features extracted from CXR via a fine-tuned SAM-Med2D encoder with semantically enriched clinical text prompts generated by BioGPT for LC prognosis. The LHFM demonstrated consistent superiority over state-of-the-art unimodal and multimodal benchmarks across diverse cohorts. It achieved a C-index of 0.910 (95% CI: 0.898–0.922; P<0.001), representing a significant improvement over existing model such as Lite-ProSENet (17). Ablation studies validated that integrating clinical text embeddings enhanced prognostic accuracy by approximately 7% compared with image-only baselines and substantially improved model interpretability through attention-guided risk stratification. Unlike existing ViT models focused primarily on report generation (39,40), the LHFM is specifically designed for survival analysis, explicitly modeling time-to-event endpoints with a computationally efficient architecture. This directly addresses the unmet need for lightweight, interpretable prognostic models in vulnerable populations and resource-limited contexts.

The interpretability and clinical relevance of the LHFM were enhanced through BioGPT-guided saliency maps that consistently co-localized with clinically relevant anatomical regions, thereby reinforcing clinician trust and adhering to emerging standards for AI transparency and accountability (39). This aligns with the findings of a recent study that emphasized the importance of structured AI reporting and its integration into radiology workflows (41). This alignment is particularly impactful in vulnerable populations oncology, where limited datasets and the imperative to minimize radiation exposure make interpretable, low-dose prognostic tools essential for both scientific and ethical precision medicine (41). The model’s prognostic utility was further confirmed through KM survival analysis, which provides visualization and complements statistical metrics with risk stratification (42). By leveraging the accessibility and safety of CXR (9), our approach provides a clinically relevant and evidence-driven framework for advancing precision oncology in diverse healthcare settings.

Furthermore, LHFM employs a lightweight architectural design that uses substantially fewer parameters than conventional transformer-based survival models do (43), ensuring computational efficiency without compromising predictive performance. This framework is particularly crucial for clinical deployment in resource-constrained environments. The model demonstrated robust generalizability across diverse patient demographics and imaging protocols, achieving consistent prognostic performance with C-index values ranging from 0.835 to 0.879 on external validation datasets, exceeding the clinically meaningful threshold of 0.80 (44,45). The clinical implication is that this framework could be integrated into bedside radiology systems to support real-time triage and informed decision-making (3,46).

The strong discrimination reported in this study should be interpreted within the context of the study design. Because the primary endpoint is an algorithmically computed risk label derived from available demographic and clinical data, the reported AUROC and C-index reflect methodological feasibility and multimodal integration capacity, rather than direct clinical survival prediction performance. Therefore, these results should not be directly compared with CT-based survival radiomics studies using verified time-to-event endpoints. Instead, LHFM demonstrates that lightweight hybrid fusion of foundation-level visual embeddings and semantically enriched prompt representations can deliver robust risk stratification performance on low-dose CXR data under controlled assumptions.

In addition to its methodological contributions, the LHFM has significant implications for medical system integration (47,48). By enabling accurate LC prognosis from low-dose CXR, the framework directly supports clinical decision-support systems deployable at the point of care. Its lightweight architecture reduces computational requirements, facilitating adoption in health facilities where advanced imaging modalities are not available. The scalability of the framework enables integration with radiology workflows and EHR platforms, supporting risk stratification, follow-up scheduling, and treatment prioritization. By leveraging CXR, the most widely available imaging modality, our model enhances cost-effectiveness, reduces reliance on high-radiation techniques, and promotes equitable access to advanced prognostic tools. In this way, the model contributes not only to technical innovation but also to healthcare efficiency and system-wide optimization. Moreover, this work aligns with the translational focus of Translational Lung Cancer Research by signifying how foundation models can be adapted into lightweight, clinically deployable prognostic tools.

We performed dedicated model generated prompt-sensitivity experiments to verify that the prompt pathway does not artificially inflate performance through pairing artifacts. Compared with the reference LHFM, replacing prompts with a null token reduced performance, while prompt shuffling across subjects produced a substantial drop, and random prompt injection further degraded discrimination. These findings confirm that the prompt branch contributes meaningful value only when prompts remain semantically relevant and correctly paired with the corresponding image, supporting the interpretation that LHFM learns clinically grounded multimodal representations rather than prompt-driven outcome.

However, this study is limited by its retrospective design and potential biases arising from multicenter heterogeneity. Although robustness and prompt-sensitivity analyses were performed, prospective validation on survival-linked LC cohorts is needed. Furthermore, the reliance on CXR, while a strength in terms of accessibility, inherently limits the morphological detail available compared to CT, which may cap the ultimate prognostic performance. Future studies will prospectively evaluate LHFM within integrated genomic and EHR to assess workflow impact, trust calibration, and prognostic depth.

Conclusions

This study presents LHFM, which integrates SAM-Med2D visual features with BioGPT-generated prompts and clinical metadata for the prognosis of LC. Our model demonstrated significant prognostic accuracy (C-index =0.910 and AUROC =0.935) and interpretability compared to existing frameworks, while maintaining computational efficiency, making it suitable for vulnerable populations and resource-limited healthcare environments. By leveraging CXRs, the most widely accessible imaging modality, LHFM reduces reliance on costly and high-radiation imaging, facilitating seamless integration into EHR and radiology workflows. In conclusion, the proposed framework provides a scalable, cost-effective decision support tool that integrates seamlessly into healthcare delivery systems, improving workflow efficiency and access to accurate prognoses in diverse clinical settings. Future work will focus on prospective validation and the incorporation of genomic data to enhance clinical utility across oncological domains.

Supplementary

The article’s supplementary files as

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments.