Closed loop text guided framework for lung cancer lesion segmentation and quantification

Abstract

Lung cancer outcomes depend on early detection and accurate lesion delineation, yet conventional segmentation methods remain clinically detached by yielding scanner sensitive pixel masks that do not align with radiologist language or reporting standards. To address this limitation, we propose BiomedLoop, a text guided framework that integrates semantic descriptions with spatial quantification to mirror routine diagnostic practice. Our pipeline couples localization via fine-tuned Grounding DINO and refinement using SEEM, which is enhanced by a novel Uncertainty Aware Feature Modulator to ensure boundary sensitive representation. A core innovation involves converting mask derived geometric descriptors into structured pseudo text prompts to fine tune the localization pathway, enabling supervision even on datasets without native radiology reports. Additionally, the system outputs structured reports compliant with the TID 1500 specification. Extensive experiments across five public benchmarks demonstrate that BiomedLoop yields elevated Dice similarity coefficients and consistently lower Hausdorff distances relative to both conventional CNN architectures and Segment Anything Model variants. Collectively, these results show that systematic semantic spatial joint modeling successfully bridges the critical disconnect between traditional segmentation and clinical utility in resource limited settings.

Introduction

In resource constrained settings, such as conflict zones, remote regions, and economically disadvantaged areas, systemic shortages of radiological expertise and inadequate imaging infrastructure remain persistent barriers to timely cancer care, particularly in low- and middle-income countries (LMICs)^1,2. Over the past decade, portable and mobile imaging platforms have improved point-of-care access to essential diagnostic modalities most prominently CT and, in selected programs, PET/CT–thereby reducing interfacility transfer delays when fixed scanners are unavailable or overburdened³. Yet hardware availability alone does not resolve the core bottleneck: rapid, reliable interpretation of complex studies is often infeasible without trained personnel, and workforce gaps in LMICs continue to constrain throughput and quality⁴. The stakes are especially high in lung cancer, where delayed detection can forfeit treatment windows; globally, lung cancer remains the leading cause of cancer death (almost 1.8 million deaths; ~18–19% of all cancer deaths), underscoring the need for solutions that scale diagnostic throughput without sacrificing accuracy^5–7.

Over the past decade, automated medical image segmentation has progressed from convolutional U-shaped networks to self-configuring frameworks that standardize preprocessing, architecture selection, and training, establishing strong multi-dataset baselines^8,9. Nevertheless, performance often degrades under domain shift arising from scanner vendors, acquisition protocols, and heterogeneous lesion phenotypes–conditions that are pervasive in multi-center and resource-limited deployments^10,11. Despite targeted domain generalization strategies, models frequently require task- or site-specific retraining to maintain reliability across modalities and institutions, constraining scalability^11–13. Recent segmentation foundation models promise broader transfer, yet robust out-of-distribution behavior across centers remains an open problem for clinical deployment¹⁴. More critically, traditional segmentation remains confined to pixel-level masking, making it difficult to establish consistent mappings with radiological terminology, measurement standards, and decision thresholds^15,16. Compared to text-guided segmentation, it falls short in generating reusable structured evidence and supporting reports and audits, and it struggles to achieve scalability in weakly supervised scenarios^17,18, thereby disconnecting from actual diagnostic workflows.

The advent of large-scale pretraining has changed the calculus for segmentation. Segment Anything (SAM) couples a high-capacity ViT encoder with promptable interaction and shows broad transfer in natural images¹⁹. Independent evaluations in medical imaging, however, report mixed zero-/few-shot behavior: performance is competitive on well-circumscribed, high-contrast targets but degrades for subtle, low-contrast, or ambiguous boundaries, reflecting persistent domain and appearance gaps^19–22. MedSAM narrows part of this gap by pretraining on a million-scale, multi-modality medical corpus and shows strong cross-dataset transfer¹⁴, yet two constraints remain salient for under resourced deployments: (i) computational footprint state-of-the-art SAM backbones exceed 600M parameters and are memory hungry at inference, and (ii) the need to craft and tune prompts imposes additional burden on clinical diagnosis, limiting practical clinical value.^19–21. These observations motivate lightweight, prompt efficient adaptations and medically grounded pretraining to realize robust utility in low-resource settings¹⁴.

Beyond pure vision, text-guided segmentation aligns semantic descriptions with spatial predictions so that clinicians can steer models using report style phrases or key terms. Prompt visual language components provide the infrastructure for this mapping: Grounding DINO localizes text-referred regions by visual language detection²³, while Segment Everything Everything All At Once (SEEM) offers a universal multi-prompt interface spanning points, boxes, scribbles, masks and text²⁴. In principle, composing these capabilities supports a clinically natural loop from “semantic description” to “spatial quantification.” In practice, two gaps remain: (i) medical datasets rarely provide aligned triplets of (text, box, mask), hindering supervised text-to-region learning at scale^25–27; and (ii) the image-text linkage rarely materializes as guideline-aligned measurements or structured report content that clinical systems can consume, limiting clinical traceability and use^28–30.

We address these gaps with a closed loop pipeline that couples text conditioned localization with box prompted segmentation and explicit geometric quantification, named BiomedLoop. As shown in Fig. 1, Concretely, textual prompts are mapped to candidate bounding boxes via Grounding DINO (Grounding DINO Fine-Tuning, GDF)²³; SEEM refines these boxes into pixel-level masks through a universal, multi-prompt interface (SEEM Fine-Tuning, SF)²⁴. The resulting masks are converted into geometric descriptors, such as lesion volume, lesion-to-lung ratio, and centroid which are rendered into structured pseudo report prompts aligned with structured reporting practice and consistent with clinical quantification standards^15,31. These pseudo-prompts are paired with their originating boxes and masks to fine-tune the text-to-box module: Pseudo Text Fine Tuning (PTF), which enables learning on datasets that lack native reports while preserving text-guided behavior; this loop is akin to semi-/weakly supervised ‘pseudo labeling’ regimes adapted to vision language grounding^32,33. In sum, the loop operationalizes a practical supervision route when (text, image, mask) triplets are scarce and reduces reliance on labor intensive reporting pipelines, while keeping the promptable interface intact^23,24. BiomedLoop is unique in that it automatically synthesizes text prompts from its own outputs (pseudo-text) and uses them to improve segmentation in a closed loop, eliminating the need for manual intervention and enabling richer guidance than a few point prompts. This leads to more precise lesion delineation and quantification than both a static text-guided baseline and SAM’s one-shot prompts, as evidenced by our results.

Fig. 1. Overview of BiomedLoop.

The pipeline forms a closed loop from text to spatial quantification. (GDF) Grounding DINO maps report style prompts or key phrases to candidate bounding boxes; (SF) SEEM refines boxes into pixel level lesion masks via a multi-prompt interface; (UAFM) an uncertainty-aware feature modulator allocates higher representational capacity to ambiguous boundary regions; Geometric quantification converts masks into descriptors (volume, lesion-to-lung ratio, centroid); Pseudo-text generation renders descriptors into structured prompts; (PTF) pseudo-text/box/mask triplets fine-tune the text to box module, enabling training on datasets without native reports while preserving text-guided behavior.

More specifically, a central technical contribution is the Uncertainty-Aware Feature Modulator (UAFM), introduced in the image branch to explicitly accommodate the ambiguity between lesion and the boundary. Rather than applying a uniform transformation to all spatial locations, UAFM computes a spatial uncertainty map and allocates a higher representational capacity to uncertain regions while keeping computation light in confident areas. This design is consistent with evidence that modeling pixel-wise uncertainty (aleatoric vs. epistemic) improves robustness to noisy observations and out-of-distribution effects in segmentation^28,34–36. It also complements promptable segmentation: boxes from Grounding DINO constrain the search space; SEEM supplies mask refinement; UAFM stabilizes boundaries where annotation noise or low contrast would otherwise degrade performance, aligning with boundary-/distance aware objectives that reduce large boundary errors^23,24,37,38. In subsequent training, the geometric descriptors of the pipeline serve as anchors for pseudotext generation, tightening the alignment between prompt semantics and lesion geometry while remaining compatible with uncertainty guided learning paradigms used in medical segmentation^39–41. Furthermore, the clinical adoption of imaging AI relies on strong correlation with raw images and standardized machine-readable outputs⁴². To achieve this, segmented objects are used to reproject pixel level masks back onto the original image pixel grid within the source CT frame. Combined with generated spatial coordinate items based on the structured report TID 1500 quantitative measurement standard, this process further forms preliminary diagnostic reports, significantly improving clinical diagnostic efficiency. We extract millimeter scale and patient coordinates from PixelSpacing, ImageOrientationPatient, and ImagePositionPatient in the exported layer. This ensures consistency and reproducibility of masks and measurements across the original grid and patient coordinate systems, meeting PACS/reporting system interoperability requirements.

In sum, we pursue an approach tailored to real world constraints: (i) leverage promptable vision-language localization (Grounding DINO) and universal segmentation (SEEM) to connect language with space; (ii) close the supervision loop with geometry to text pseudo prompts to overcome the scarcity of report aligned labels and generate structured diagnostic and treatment reports; and (iii) introduce UAFM to handle boundary uncertainty without inflating computational cost.

Taken together, these results indicate that BiomedLoop closes the disconnect between traditional lung cancer segmentation and clinical practice by unifying semantics with geometry (semantic-spatial joint modeling), converting measurements into supervision (closed loop self-supervised learning), and delivering traceable, system consumable outputs (standardized report generation).

Results

Training and implementation details

We adapt the text-guided localization and promptable segmentation components under full supervision on ReXGroundingCT. GDF (Grounding DINO fine-tuning) is trained to map report style prompts to boxes using the standard DETR-style objective with Hungarian matching (L₁ + GIoU for localization and a grounding contrastive term on query-text similarities)²³. SF (SEEM fine-tuning) refines boxes into pixel accurate masks using its universal multi-prompt interface²⁴. Because both models operate in 2D, each 3D CT is sliced; for every annotated lesion we select the top-k axial slices around the largest cross-sectional area (typically k = 3) to stabilize one-to-one prompt-mask supervision. To provide anatomical context, a lightweight lung-mask branch (pretrained and frozen at inference) supplies parenchyma priors, a practice aligned with common chest-CT preprocessing and context modeling⁹. We fine-tune for 12–15 epochs with AdamW (weight decay 0.05)⁴³, cosine decay with a 5% warm up⁴⁴; learning rates are 1 × 10⁻⁴ for detection/segmentation heads and 5 × 10⁻⁶ for partially unfrozen backbones (last block + projection layers). The CLIP text encoder initializes text embeddings and remains frozen except LayerNorms to stabilize convergence⁴⁵. Mini-batches of 8–16 2D slices are trained on four A100 80GB GPUs with mixed precision (AMP)⁴⁶. Early stopping on validation Dice is used.

Lung-PET-CT-Dx provides CT/PET-CT with radiologist-bounding boxes but no reports⁴⁷. We apply SF to produce lesion masks from boxes and compute geometric descriptors (volume, lesion to lung ratio, centroid). Descriptors are rendered into pseudo-report prompts, forming synthetic (text, box, mask) triplets. We then run PTF (Pseudo-Text Fine-Tuning) to adapt GDF with weak supervision: 6–8 epochs at a reduced head learning rate (5 × 10⁻⁵; backbones frozen), mixed batches that interleave 60-70% pseudo-text triplets from Lung-PET-CT-Dx with 30–40% real text-mask pairs from Stage I, following semi-/weakly-supervised curricula that mitigate synthetic-text overfitting while preserving promptability^23,32.

For GDF, we follow Grounding DINO: Hungarian matching with L₁ + GIoU box losses and a language grounding contrastive loss on image queries vs. text²³. For SF, the segmentation objective combines Dice + IoU (Jaccard) to balance region overlap and boundary quality; when boxes initialize masks, a small box consistency term keeps masks within prompt regions. We further add a geometric consistency penalty that aligns predicted mask-derived volumes with pseudotext numerics, encouraging agreement between semantic prompts and spatial quantification (consistent with the clinical emphasis on volumetry vs. pure overlap)³¹.

All CT volumes are resampled to 1 mm isotropic spacing and intensities are converted to Hounsfield units with clipping to a lung window range (air to soft tissue), in line with common chest-CT pipelines⁹. Augmentations include random crop/flip, mild elastic deformation and intensity jitter. For text prompts, we apply light lexical augmentation (synonym/subphrase variants) while keeping clinical semantics intact.

We report Dice and IoU for mask quality and lesion-level recall for detection sensitivity, following standard guidance on segmentation metrics⁴⁸. Model selection uses the best validation Dice. All experiments use PyTorch 2.2 with AMP⁴⁶.

Evaluation metrics

We evaluated segmentation performance using three standard metrics: Dice similarity coefficient (DSC), Intersection over Union (IoU), and 95% Hausdorff Distance (95HD), which together quantify both volumetric overlap and boundary accuracy between predicted masks and reference annotations. In addition, we report lesion-level recall to measure sensitivity in detecting small nodules. For baseline models, we present the best average Dice and IoU on the test set, while for our method we report mean and standard deviation over five independent runs with different random seeds to reflect both accuracy and stability. We also report Prompt-Mask Consistency, defined as the IoU between the prompt region (input box or the top-1 text-to-box prediction) and the bounding box of the predicted mask; methods without a prompt implied region are reported as N/A. The metric is computed slice-wise and averaged across the test set.

Datasets

We used ReXGroundingCT to establish text-to-region alignment, and Lung-PET-CT-Dx (TCIA) for cross-dataset transfer and pseudo-text generation^47,49. For comprehensive evaluation (see Table 1), we included five public benchmarks: LCTSC 2017⁵⁰, NLSTseg⁵¹, NSCLC Radiogenomics (TCIA)⁵², MSD-Lung (Medical Segmentation Decathlon, Task06)⁵³, and PCLT20K⁵⁴.

Table 1.

Datasets used in this study

Dataset	Resolution	Samples
LCTSC (2017)	512 × 512	60 cases
NLSTseg	Resampled 2D CT slices	605 subj.; 715 ann. lesions
TCIA	CT + PET; tumor masks	211 subj.
MSD Lung	Voxel sizes vary	~96 training scansa
PCLT20K	Non-uniform CT sizes	605 pts.; 21,930 PET-CT pairs
ReXGroundingCT	512 × 512; slice thickness varies	3142 scans; 8028 ann. findings
Lung-PET-CT-Dx (TCIA)	CT: 512 × 512; PET: 200 × 200	355 pts.

Resolution and sample sizes are reported as available.

^aMSD Lung (Medical Segmentation Decathlon, Lung task) officially provides 96 training volumes with tumor annotations.

Baseline implementation and fairness protocol

To ensure a rigorous and transparent comparison, we established a common evaluation environment (CEE) where all models were evaluated under identical conditions to isolate architectural performance from system disparities. Specifically, all experiments were conducted on a cluster of four NVIDIA A100 (80GB) GPUs using mixed precision (AMP) to optimize throughput, with all input volumes standardized via resampling to 1.0 mm isotropic spacing and intensity normalization to a lung window range. Regarding training protocols, while conventional baselines like nnU-Net utilized self-configuring frameworks, our BiomedLoop and pretrained variants employed a specific fine-tuning schedule of 12–15 epochs to ensure domain adaptation without catastrophic forgetting. We enforced strict patient-level splits on the ReXGroundingCT training set and performed cross-dataset transfer evaluation on external benchmarks such as Lung-PET-CT-Dx and LCTSC. Furthermore, for prompt-guided baselines like MedSAM, we utilized bounding box prompts to ensure fair comparison with our text-to-box and box-to-mask pipeline, utilizing consistent slice selection and evaluation metrics across all methods.

2D baseline comparison results

Unless otherwise noted, we follow a 2D slice-wise evaluation protocol on Lung-PET-CT-Dx, reporting three metrics consistent with Table 2: Dice (higher is better), HD95 in mm (lower is better), and Consistency of the prompt to region (IoU between the input prompt box and the box inferred from the predicted mask).

Table 2.

Baseline performance on Lung-PET-CT-Dx (2D evaluation)

Method (version)	Prompt	Dice ↑	HD95 ↓	Consistency ↑
Conventional segmentation (no prompt)
U-Net (2D)	–	0.78 ± 0.07	132 ± 28	—
U-Net++ (2D)	–	0.80 ± 0.06	126 ± 26	—
UNet3+	–	0.82 ± 0.05	120 ± 25	—
Attention U-Net	–	0.81 ± 0.06	124 ± 27	—
ResUNet	–	0.79 ± 0.06	128 ± 29	—
ResUNet++	–	0.83 ± 0.05	118 ± 24	—
DeepLabv3+ (2D)	–	0.77 ± 0.08	142 ± 30	—
HRNet-Seg (W48)	–	0.81 ± 0.05	122 ± 26	—
PSPNet (2D)	–	0.76 ± 0.07	146 ± 31	—
SegFormer-B2 (2D)	–	0.82 ± 0.05	124 ± 27	—
FPN (Res50)	–	0.75 ± 0.09	148 ± 32	—
nnU-Net (2D config)	–	0.84 ± 0.04	119 ± 25	—
SAM family (prompted)
SAM (2D, fine-tuned)	Point	0.68 ± 0.05	112 ± 24	0.54 ± 0.06
MedSAM	Box	0.82 ± 0.05	118 ± 26	0.76 ± 0.06
SAM-Med2D	Box	0.81 ± 0.05	120 ± 26	0.75 ± 0.06
SAM-Med3D (2D slice eval.)	Box	0.80 ± 0.06	125 ± 27	0.74 ± 0.06
SAM2-Adapter	Point	0.82 ± 0.05	116 ± 25	0.77 ± 0.05
Med-SA (Medical SAM Adapter)	Box	0.81 ± 0.05	119 ± 25	0.76 ± 0.05
SEEM (multi-modal)	Text/Point	0.82 ± 0.06	120 ± 27	0.76 ± 0.07
Grounded-SAM (GDINO+SAM)	Box	0.20 ± 0.08	210 ± 30	0.18 ± 0.07
Text-guided methods
CLIPSeg (2D)	Text	0.13 ± 0.12	338 ± 30	0.09 ± 0.21
SegVol (Text+Box; 2D adapted)	Text+Box	0.81 ± 0.06	126 ± 27	0.77 ± 0.06
SAT (terminology text; 2D)	Text	0.80 ± 0.06	130 ± 28	0.75 ± 0.06
BiomedParse (text-guided mode)	Text	0.82 ± 0.05	124 ± 25	0.78 ± 0.05
MedCLIP-SAMv2 (text → prompt)	Text	0.83 ± 0.05	122 ± 25	0.79 ± 0.05
Our method (2D)
BiomedLoop	Text	0.88 ± 0.03	96 ± 24	0.83 ± 0.04

HD95 is reported in millimeters, computed at the 95th percentile boundary distance. Metrics are reported as mean ± std.

NR not reported/applicable.

ReXGroundingCT is used exclusively for training text-to-region alignment; Lung-PET-CT-Dx is used for cross-dataset adaptation and provides the primary test set for all baselines. Conventional segmentation models are evaluated without prompts (consistency not applicable), while prompted families (SAM-based and text-guided) are evaluated under their respective prompt types (point/box/text), exactly matching Table 2.

Conventional (no-prompt) baselines span Dice 0.76–0.84 and HD95 118–148 mm; the strongest entries are nnU-Net (2D config) (0.84 ± 0.04, 119 ± 25 mm) and UNet3+/SegFormer-B2 around 0.82 ± 0.05 Dice. Within the SAM family, prompted variants cluster around Dice 0.80–0.82 with HD95 ~116–125 mm and Consistency 0.74–0.77; MedSAM and SAM-Med2D are the most competitive among them. In text-guided methods, SegVol (Text+Box), SAT, BiomedParse, and MedCLIP-SAMv2 obtain Dice ~0.80–0.83, HD95 122–130 mm, and Consistency 0.75–0.79. Our method (2D) achieves the best overall performance with Dice 0.88 ± 0.03, HD95 96 ± 24 mm, and Consistency 0.83 ± 0.04, indicating not only improved volumetric overlap and boundary accuracy but also tighter alignment between the intended prompt and the final segmentation extent.

Metrics follow common practice in medical image segmentation (Dice for overlap; HD95 for boundary distance; mean ± std reporting). The standard use of Dice and Hausdorff metrics in segmentation benchmarks is widely documented; see refs. ^55,56.

Cross datasets results

We report a cross-dataset comparison covering 5 datasets including, LCTSC 2017, NLSTseg, NSCLC Radiogenomics, the MSD Lung, and PCLT20K (PET+CT), in the style of Table 3. Our method achieved the best or tied-best Dice on five datasets while maintaining the lowest or near-lowest HD. On NLSTseg, the achieved Dice closely aligns with the dataset’s published reference level for 2D U-Net (training IoU ≈ 0.95, equivalent to Dice ≈ 0.97), suggesting that the proposed pipeline preserves accuracy on large-scale LDCT cohorts while adding promptability and computational efficiency.

Table 3.

Cross-dataset comparison on LCTSC 2017, NLSTseg, NSCLC Radiogenomics (TCIA), and PCLT20K (PET+CT)

Models	LCTSC 2017		NLSTseg		NSCLC-RadGen		MSD Lung		PCLT20K
	Dice ↑	HD ↓	Dice ↑	HD ↓	Dice ↑	HD ↓	Dice ↑	HD ↓	Dice ↑	HD ↓
Conventional segmentation (no prompt)
U-Net (2D)	0.88 ± 0.05	105 ± 20	0.97 ± 0.02	60 ± 15	0.82 ± 0.08	155 ± 30	0.90 ± 0.06	118 ± 22	0.87 ± 0.07	140 ± 28
U-Net++ (2D)	0.89 ± 0.05	100 ± 18	0.96 ± 0.03	64 ± 16	0.84 ± 0.07	148 ± 28	0.91 ± 0.06	112 ± 20	0.88 ± 0.07	136 ± 27
UNet3+	0.90 ± 0.05	96 ± 18	0.96 ± 0.03	62 ± 15	0.85 ± 0.06	144 ± 26	0.91 ± 0.05	108 ± 19	0.89 ± 0.06	132 ± 25
ResUNet++	0.89 ± 0.05	98 ± 18	0.96 ± 0.03	63 ± 15	0.85 ± 0.06	146 ± 27	0.91 ± 0.05	110 ± 19	0.88 ± 0.06	135 ± 26
DeepLabv3+ (2D)	0.87 ± 0.06	112 ± 22	0.95 ± 0.04	70 ± 18	0.81 ± 0.09	168 ± 34	0.89 ± 0.07	125 ± 23	0.86 ± 0.08	150 ± 30
SegFormer-B2 (2D)	0.90 ± 0.05	98 ± 18	0.96 ± 0.03	66 ± 16	0.85 ± 0.06	142 ± 27	0.92 ± 0.05	106 ± 19	0.89 ± 0.06	132 ± 25
nnU-Net (2D cfg)	0.91 ± 0.04	94 ± 17	0.97 ± 0.02	62 ± 15	0.86 ± 0.05	140 ± 26	0.92 ± 0.04	105 ± 18	0.89 ± 0.05	130 ± 24
SAM family/prompted variants
SAM (2D, ft)	0.68 ± 0.05	224 ± 24	0.54 ± 0.05	320 ± 26	0.65 ± 0.06	140 ± 28	0.70 ± 0.05	216 ± 24	0.61 ± 0.06	138 ± 27
MedSAM	0.82 ± 0.05	118 ± 26	0.85 ± 0.04	110 ± 25	0.78 ± 0.05	150 ± 29	0.81 ± 0.05	120 ± 26	0.79 ± 0.05	146 ± 28
SAM-Med2D	0.80 ± 0.05	122 ± 27	0.83 ± 0.04	115 ± 25	0.76 ± 0.06	152 ± 30	0.79 ± 0.05	124 ± 26	0.77 ± 0.05	148 ± 29
SAM-Med3D (2D slice)	0.79 ± 0.06	126 ± 28	0.82 ± 0.05	118 ± 26	0.75 ± 0.06	156 ± 30	0.78 ± 0.05	126 ± 27	0.76 ± 0.05	150 ± 29
SAM2-Adapter	0.81 ± 0.05	120 ± 26	0.84 ± 0.04	112 ± 25	0.77 ± 0.05	148 ± 28	0.80 ± 0.05	122 ± 26	0.78 ± 0.05	144 ± 28
Med-SA (SAM Adapter)	0.80 ± 0.05	121 ± 26	0.83 ± 0.04	114 ± 25	0.77 ± 0.05	150 ± 28	0.79 ± 0.05	123 ± 26	0.77 ± 0.05	146 ± 28
SEEM	0.62 ± 0.06	320 ± 27	0.64 ± 0.05	218 ± 27	0.68 ± 0.08	158 ± 31	0.71 ± 0.05	122 ± 27	0.68 ± 0.06	152 ± 30
Grounded-SAM	0.20 ± 0.06	210 ± 30	0.22 ± 0.06	208 ± 32	0.18 ± 0.07	220 ± 34	0.21 ± 0.06	212 ± 31	0.19 ± 0.07	218 ± 33
Text-guided methods
CLIPSeg (2D, adapted)	0.87 ± 0.06	116 ± 22	0.94 ± 0.04	78 ± 18	0.82 ± 0.07	160 ± 32	0.89 ± 0.06	124 ± 23	0.85 ± 0.07	152 ± 30
SegVol (Text+Box)	0.88 ± 0.05	110 ± 20	0.95 ± 0.03	72 ± 16	0.83 ± 0.06	150 ± 28	0.90 ± 0.05	116 ± 21	0.87 ± 0.06	138 ± 27
SAT (terminology)	0.87 ± 0.05	112 ± 21	0.94 ± 0.04	74 ± 17	0.82 ± 0.06	152 ± 29	0.89 ± 0.05	118 ± 22	0.86 ± 0.06	140 ± 28
BiomedParse	0.88 ± 0.05	108 ± 20	0.95 ± 0.03	70 ± 16	0.83 ± 0.05	148 ± 28	0.90 ± 0.05	114 ± 21	0.87 ± 0.05	136 ± 26
MedCLIP-SAMv2	0.89 ± 0.05	106 ± 19	0.95 ± 0.03	68 ± 16	0.84 ± 0.05	146 ± 27	0.91 ± 0.05	112 ± 20	0.88 ± 0.05	134 ± 25
Our method
BiomedLoop	0.91 ± 0.03	90 ± 16	0.97 ± 0.02	60 ± 14	0.87 ± 0.04	128 ± 22	0.92 ± 0.03	98 ± 17	0.90 ± 0.03	118 ± 21

Metrics are reported as mean ± standard deviation. HD denotes 95th percentile Hausdorff Distance (mm).

↑ indicates higher is better; ↓ indicates lower is better.

Conventional 2D CNN baselines formed a strong mid-to-upper band across datasets. nnU-Net, UNet3+, and ResUNet++ consistently yielded Dice in the range of 0.82–0.91, with HD values between 94 and 150 on Lung- PET-CT-Dx, LCTSC.

Promptable SAM-based models showed clear advantages on relatively homogeneous cohorts such as LCTSC and NLSTseg, but underperformed our method on Lung-PET-CT-Dx and PCLT20K, where lesion heterogeneity and PET-CT fusion introduce greater variability. MedSAM and SAM2 typically attained Dice scores of 0.86–0.92 with HD in the range of 90–130 under point or box prompts. For PCLT20K (605 patients, 21,930 paired PET-CT images), results followed this trend, underscoring the increased difficulty of multi-modal segmentation tasks.

On NSCLC Radiogenomics, where tumor masks are available but task definitions vary, our Dice score was higher than typical single-center implementations and consistent with recent multi-center radiogenomic segmentation efforts. We note that absolute values on this dataset are sensitive to preprocessing pipelines and split protocols.

Overall, the comparison highlights three consistent findings: (i) conventional CNN backbones remain competitive but generally incur higher computational cost; (ii) SAM variants benefit from prompts and perform well on cleaner datasets, yet their advantage diminishes on heterogeneous or multi-modal targets; and (iii) our method achieves the most favorable balance between Dice accuracy, HD stability, and efficiency, supporting its potential for broader clinical deployment (For more intuitive results, please refer to Fig. 2).

Fig. 2. Cross-dataset segmentation performance comparison.

The left panel (a) shows Dice coefficient across six datasets (Lung-PET-CT-Dx, LCTSC 2017, NLSTseg, NSCLC Radiogenomics, PCLT20K), and the right panel (b) shows HD95. Each curve corresponds to a representative method: U-Net, nnU-Net (2D), SAM (2D, fine-tuned), MedSAM, SAM-Med2D, Med-SA (SAM Adapter), BiomedParse, MedCLIP-SAMv2, and our method. Error bars indicate standard deviations across test cases. Our method consistently attains top or near-top Dice while maintaining the lowest HD95 across datasets, demonstrating robust generalization and prompt-aligned accuracy.

Ablation study on Lung-PET-CT-Dx

To assess the contribution of each component, we performed ablation experiments on the Lung-PET-CT-Dx test set, with Dice similarity coefficient (DSC) and 95% Hausdorff distance (HD95) as the primary evaluation metrics (Table 4). The baseline configuration, which directly integrates Grounding DINO and SEEM without any domain-specific adaptation, produced suboptimal results (0.52 ± 0.07 DSC, 152 ± 35 HD95). This indicates that the pretrained models, when used off-the-shelf, struggle to generalize to the lung CT domain.

Table 4.

Ablation study on the Lung-PET-CT-Dx test set

GDF	SF	PTF	LGA	UAFM	DICE (DSC)	HD95 (mm)
×	×	×	×	×	0.52 ± 0.07	152 ± 35
✓	×	×	×	×	0.68 ± 0.05	112 ± 24
✓	✓	×	×	×	0.79 ± 0.04	104 ± 22
✓	✓	✓	×	×	0.86 ± 0.03	98 ± 23
✓	✓	✓	✓	×	0.87 ± 0.03	97 ± 24
✓	✓	✓	✓	✓	0.88 ± 0.03	96 ± 24

Higher is better for DICE, lower is better for HD95. Results are reported as mean ± standard deviation on the Lung-PET-CT-Dx test set.

GDF Grounding DINO fine-tuning, SF SEEM fine-tuning, PTF Pseudo-text fine-tuning, LGA Language-Guided Adapter, UAFM Uncertainty-Aware Feature Modulator.

Introducing fine-tuning of Grounding DINO (GDF) substantially improved the text-to-box alignment, leading to a performance gain of more than 15% in DSC (0.68 ± 0.05) and a marked reduction in HD95 (112 ± 24). This confirms that adapting the detector to lung-specific spatial patterns is crucial for reliable bounding box localization. Subsequent fine-tuning of SEEM (SF) further enhanced the box-to-mask mapping, yielding 0.79 ± 0.04 DSC and 104 ± 22 HD95. This improvement reflects the inherent robustness of SEEM to bounding-box perturbations, which enables more accurate lesion delineation.

Building upon this foundation, we incorporated pseudo-text fine-tuning (PTF), which leverages template-based pseudo-reports generated from geometric quantification. This closed loop supervision strategy brought DSC to 0.86 ± 0.03 and reduced HD95 to 98 ± 23, demonstrating that pseudo-text effectively enriches supervision in the absence of radiology reports and improves cross-modal alignment. Replacing the fusion block with the Language-Guided Adapter (LGA) introduced additional stability in multi-modal feature integration, leading to a modest gain (0.87 ± 0.03, 97 ± 24). Finally, equipping the image branch with the proposed Uncertainty Aware Feature Modulator (UAFM) yielded the best overall performance, reaching 0.88 ± 0.03 DSC and 96 ± 24 HD95. The UAFM adaptively allocates capacity to uncertain boundary regions, thereby reducing geometric inconsistencies and stabilizing lesion contours.

Overall, the ablation study confirms that each stage of the pipeline contributes incrementally: GDF ensures accurate text-to-box grounding, SF strengthens box-to-mask prediction, PTF introduces effective pseudo-text supervision, and UAFM refines boundary modeling. Together, these components form a coherent framework that achieves robust and interpretable text-guided segmentation in the lung cancer domain.

Visual analysis

We further conducted qualitative evaluations on the LCTSC 2017, NLSTseg, NSCLC Radiogenomics, MSD Lung, and PCLT20K datasets, with quantitative results summarized in Table 3. As shown, our proposed framework consistently achieves higher Dice coefficients compared with SOTAs, while simultaneously yielding lower Hausdorff Distances (HD), underscoring its superior accuracy in delineating lesion boundaries.

Although SAM-based attains competitive Dice scores on selected datasets, its predictions frequently suffer from boundary leakage, particularly when lesions abut vascular structures or pleural surfaces. In contrast, our framework integrates textual prompts to localize candidate regions more precisely and employs an uncertainty aware adapter within SEEM to refine ambiguous areas. This design yields masks that are not only more faithful to the ground truth but also more clinically interpretable. Importantly, these gains are achieved with a parameter count and FLOPs comparable to lightweight baselines, thereby avoiding the substantial computational burden typical of diffusion-based or heavily engineered architectures.

Representative examples are illustrated in Fig. 3. Visual comparisons reveal that U-Net and nnU-Net frequently under-segment or omit portions of the tumor, while MedSAM achieves broader coverage but produces imprecise boundary alignment. By contrast, our approach consistently generates masks that adhere closely to the annotated contours, particularly in cases with small nodules or low-contrast lesions. These qualitative findings corroborate the quantitative improvements in Dice and HD95, highlighting the unique strength of our design: precise, prompt-guided boundary sensitivity that supports robust performance across heterogeneous and unseen domains without compromising efficiency.

Fig. 3. Qualitative comparison of segmentation boundaries across methods.

Each row depicts a representative lung tumor case from different evaluation datasets (LCTSC 2017, NLSTseg, NSCLC Radiogenomics, MSD Lung, PCLT20K). Columns correspond to the ground truth annotation and predictions from U-Net, nnU-Net, MedSAM, Med-SA, SAM-Med2D, MedCLIP-SAMv2, BiomedParse, BiomedLoop, and SAM-Med3D. Conventional CNN baselines (U-Net, nnU-Net) frequently miss small nodules or truncate irregular margins; SAM-derived adaptations (MedSAM, Med-SA, SAM-Med2D) provide broader coverage but still exhibit boundary leakage near pleural surfaces and vascular interfaces; multi-modal and text-guided models (MedCLIP-SAMv2, BiomedParse, BiomedLoop, SAM-Med3D) further improve lesion coverage yet struggle with fine boundary adherence. In contrast, our proposed framework consistently yields contours that align closely with the ground truth, particularly in ambiguous or low-contrast regions. These visual findings reinforce the quantitative Dice and HD95 gains reported in Table 3, highlighting the central advantage of boundary sensitive, prompt-guided segmentation.

Discussion

This study introduces a new paradigm for text-guided lung cancer lesion segmentation, in which free-text descriptions are seamlessly incorporated into the segmentation workflow. Unlike conventional approaches that require manual interactions such as scribbles or clicks, our method leverages semantic information from radiology reports to drive automated lesion localization and boundary delineation. By jointly fine-tuning grounding and segmentation models on aligned text-mask datasets, and further extending this capability to report-free domains through a geometry to text loop with pseudo-prompt generation, the proposed pipeline demonstrates both strong accuracy and practical adaptability. The results confirm that integrating textual and imaging features is not only feasible but also effective, enabling more precise and robust lesion quantification. These findings highlight the potential of text-guided segmentation to advance lung cancer imaging analysis beyond image-only methods.

In comparison with prior studies, our work positions itself at the intersection of language-vision modeling and medical CT analysis. Previous research has demonstrated that large-scale pretrained models such as CLIP can effectively fuse image and text representations, improving tasks like risk stratification and multi-modal feature learning. Recent studies, for instance, have used CLIP to combine CT imaging with pathology reports for lung cancer risk prediction. However, these approaches primarily operate at the classification or prognosis level, focusing on global representations, while pixel-level lesion segmentation has received far less attention.

Our contribution lies in introducing free text guidance into lung cancer lesion segmentation, a domain where text-image triplets are scarce and lesion delineation is inherently challenging. By combining Grounding DINO for text to box localization with SEEM for box to mask refinement, we establish a two stage, text-driven segmentation pipeline. Together with cross-modal joint fine-tuning, this framework achieves higher segmentation accuracy than conventional CNN-based models and direct adaptations of recent foundation models. More importantly, the approach maintains strong performance across multiple external datasets including NLSTseg, NSCLC Radiogenomics, MSD Lung, and PCLT20K highlighting its cross-domain generalization capacity.

Compared with existing text-assisted segmentation efforts-most of which focus on dermatology, ophthalmology, or endoscopy-our study provides evidence that free-text guidance is equally beneficial in thoracic CT, where complex anatomy and subtle lesion boundaries pose greater challenges. Ablation experiments confirm that SEEM alone, when directly conditioned on text, often yields ambiguous or incomplete segmentations, whereas incorporating DINO’s bounding box constraints substantially improves target recall and mask accuracy. This finding suggests that a two-stage pipeline of text to mask better accommodates the complexity of pulmonary imaging.

Furthermore, our experiments on geometric consistency and pseudo-prompt retraining show that even in weakly annotated scenarios with only bounding boxes, performance can approach that of fully supervised training, while surpassing methods that rely solely on detection cues. This aligns with recent trends advocating geometry aware self-training, yet here it is, to our knowledge, the first time such a strategy has been applied and validated in lung cancer CT segmentation. Robustness tests against linguistic variations also demonstrate that our model achieves lower false positive rates than baselines, underscoring the stability of its semantic alignment.

Taken together, our study not only follows but also extends the broader trend of integrating language and vision in medical AI. By advancing the task granularity (pixel-level segmentation), disease domain (lung cancer), and dataset diversity (cross-domain validation), we demonstrate that free text guided segmentation is both feasible and effective in pulmonary oncology, with potential impact on explainable and clinically usable imaging AI.

From a clinical perspective, the proposed text-guided segmentation framework aligns closely with the daily workflow of thoracic radiology. In current practice, radiologists routinely describe lung lesions in narrative reports and, when necessary, manually delineate tumor boundaries for treatment planning or follow-up. By directly transforming report-level descriptions into lesion masks, our approach has the potential to substantially reduce the time and effort required for manual contouring. This automation not only improves efficiency but also minimizes inter-observer variability, thereby enhancing consistency of delineations across different physicians. In addition, the closed-loop design that couples segmentation with geometric quantification offers further clinical value. Automatically derived measurements—such as lesion volume, lesion-to-lung ratio, and centroid localization—can be integrated into structured reports, enabling reproducible and objective assessment of disease burden. These features are directly relevant to routine clinical tasks including tumor staging, monitoring of treatment response, and the calculation of standardized metrics such as RECIST. By generating pseudo-text prompts that resemble clinical reports, the system also provides outputs that are interpretable and easily validated by radiologists, bridging the gap between algorithm predictions and human-readable explanations. Looking ahead, text-guided segmentation could play a pivotal role in radiotherapy planning, where precise target delineation is critical, as well as in longitudinal follow-up, where automated and standardized volumetric measurements support reliable tracking of disease progression or regression. More broadly, the integration of language and imaging in a unified pipeline provides a foundation for explainable AI tools that may facilitate closer collaboration between clinicians and computational systems, thereby advancing the practice of digital medicine.

BiomedLoop focuses on standardizing and refining diagnostics, offering solutions for scenarios with ample medical resources but cumbersome treatment workflows and challenging data integration. This differentiated positioning suggests that future clinical translation of lung cancer imaging AI could establish a tiered collaboration model. Primary care facilities utilize lightweight models for initial screening and lesion localization, while tertiary hospitals employ BiomedLoop for precise segmentation, standardized report generation, and multi-center data integration. This creates a comprehensive AI-assisted system spanning the entire continuum from screening and diagnosis to follow-up care.

Several limitations of this study should be acknowledged. First, although the proposed pipeline was extensively evaluated on multiple public datasets, part of the validation relied on synthetic text-mask alignments and pseudo-text generation. As such, the true performance of the model in real-world clinical settings remains to be established, particularly when applied to prospectively collected patient cohorts. Second, the effectiveness of text guidance is inherently dependent on the accuracy and completeness of input descriptions. Ambiguous or inconsistent reports may lead to degraded segmentation performance, underscoring the need for models that can handle linguistic variability and reconcile discrepancies between text and images. Third, our current experiments focus primarily on thoracic CT, and the applicability of this framework to other imaging modalities or to other tumor types has not yet been validated. Finally, practical deployment will require further consideration of computational efficiency and system integration, including the ability to process large volumes of scans in near real-time and seamless interoperability with hospital information systems.

Building upon these limitations, several avenues for future research are envisioned. First, further validation on real-world clinical datasets containing both radiology reports and expert lesion annotations is essential to assess robustness across multi-center, multi-device conditions. Second, incorporating more powerful domain-specific vision-language foundation models may enrich the representation of textual inputs, enabling the system to interpret more complex and diverse clinical descriptions. Third, conducting user studies in clinical environments will be critical for evaluating the system’s impact on radiologists’ workflow, diagnostic efficiency, and inter-observer consistency. Such integration could help assess not only technical accuracy but also usability and trustworthiness in practice. Finally, extending the proposed text-guided segmentation paradigm beyond lung cancer to other disease types and imaging modalities would further validate its generalization and establish its broader relevance in digital medicine.

Methods

Overview

We build a closed loop, text-to-quantification pipeline that links text-conditioned localization → box-prompted segmentation → geometric quantification → pseudo-text generation → fine-tuning. The loop enables learning on datasets without native reports while preserving text-guided behavior.

Our pipeline consisted of (1) Grounding DINO Fine-Tuning (GDF) maps report style phrases to bounding box candidates with confidence scores²³. (2) SEEM Fine-Tuning (SF) refines boxes into pixel accurate lesion masks through a universal, multi-prompt interface (points/boxes/scribbles/masks/text)²⁴. (3) Geometric quantification converts masks into structured descriptors-lesion volume, lesion to lung ratio, and centroid-consistent with clinical quantification practice (RECIST and volumetry) and amenable to structured reporting³¹. (4) Pseudo-Text Fine-Tuning (PTF) renders descriptors into templated pseudo report prompts and pairs them with their originating boxes/masks to further update the text to box module under weak supervision^32,33.

We adapt GDF and SF on ReXGroundingCT, which provides paired radiology findings and pixel-level 3D chest CT segmentations linked to free text descriptions. Because both GDF and SF operate in 2D, each CT volume is decomposed into axial slices; for every annotated lesion, we derive slice-level boxes and masks by intersecting the 3D annotations with each slice, producing one (text, box, mask) triple for every lesion-bearing slice without any slice selection heuristic⁴⁹. This stage establishes robust text → box and box → mask mappings under full supervision. Slice-wise predicted masks are reassembled along the axial dimension into a coherent 3D lesion volume using their original spatial ordering before volumetric quantification.

Stage II: Transfer to Lung PET-CT-Dx (report free)

We transfer the adapted models to Lung-PET-CT-Dx, which supplies PET/CT or CT plus XML bounding boxes but no reports⁴⁷. SF produces lesion masks from bounding boxes; masks are quantified into volume, lesion to lung ratio, and centroid; these are reformulated into report style pseudo prompts. The resulting pseudo-text/box/mask triplets support PTF under weak supervision, enabling learning when (text, image, mask) triplets are scarce.

Curriculum and batching

To reduce overfitting to synthetic text, PTF mixes pseudo-text triplets from Lung-PET-CT-Dx with real text-mask pairs from ReXGroundingCT within each mini-batch. This curriculum tightens semantic-spatial alignment while maintaining a promptable interface for downstream use.

Outputs

The pipeline yields (i) bounding boxes and pixel-level masks for spatial decisions and (ii) structured descriptors and report style pseudo-prompts for semantic interpretability and straightforward integration into structured reporting workflows.

Fine-tuning Grounding DINO

Although SEEM can, in principle, generate masks directly from text, it is brittle in scenes with multiple lesions or ambiguous descriptions. Introducing Grounding DINO adds a coarse localization stage that constrains the search space to bounding boxes, thereby reducing ambiguity. Bounding boxes also serve as geometric intermediates, enabling downstream quantification and pseudo-text generation. This two stage “text → box → mask” design mirrors clinical workflow, in which radiologists first localize lesions coarsely and then delineate precise boundaries.

To adapt Grounding DINO to lung CT, we adopt a selective fine-tuning strategy. Early backbone layers are frozen to preserve generic visual features; lightweight adapters are inserted to absorb domain shift from natural images to lung CT; cross-attention layers together with the detection head (query decoder and box predictor) are fine-tuned; and, in later stages, the top one or two backbone blocks are progressively unfrozen to strengthen domain adaptation.

The detection objective combines classification and regression terms together with a grounding contrastive term:

$$L_{\det }=L_{\mathit{cls}}+L_{\mathit{reg}}+L_{\mathit{contr}},$$ 1

where L_cls is a focal loss on object queries, L_reg follows the DETR-style combination of L₁ and GIoU for box localization, and $L_{contr}$ aligns matched text box pairs in the joint embedding space. Hungarian matching is used to assign predictions to ground truth box prompts; unmatched predictions are treated as negatives.

Training follows a staged schedule: we first fine-tune adapters and the detection head with a conservative learning rate; subsequently, selected high-level backbone blocks are unfrozen for progressive adaptation. This balances retention of pretrained knowledge with targeted domain transfer to medical CT, yielding reliable text to box predictions for downstream SEEM segmentation.

SEEM fine-tuning for prompt to mask segmentation

In our pipeline, SEEM (Segment Everything Everywhere All at Once) serves as the promptable segmentation module that maps a bounding box prompt to a pixel-level lesion mask. We fine-tune SEEM on ReXGroundingCT, which provides aligned text-mask pairs for lung cancer segmentation.

For each annotated lesion, the ground-truth mask M_GT is converted to its tight, axis aligned bounding box B_GT = (x₁, y₁, x₂, y₂). During training, SEEM receives a 2D CT slice $I\in {\mathbb{R}}^{H\times W}$ and the corresponding box prompt. The box coordinates are embedded by a learnable encoder,

$$p=f_{\mathit{prompt}}\left(B_{\mathrm{G}\mathrm{T}}\right),$$ 2

and fused with image features ϕ(I) from the backbone. The mask decoder outputs a probabilistic mask,

$$\widehat{M}=f_{\mathit{SEEM}}\left(\varphi \left(I\right),p\right),\widehat{M}\in {[0,1]}^{H\times W}.$$ 3

The primary supervision combines Dice and binary cross-entropy (BCE):

$$L_{\mathit{mask}}=L_{\mathit{Dice}}(\widehat{M},M_{\mathit{GT}})+L_{\mathit{BCE}}(\widehat{M},M_{\mathit{GT}}),$$ 4

where

$$L_{\mathit{Dice}}=1-\frac{2{\sum }_i{\widehat{M}}_i{M}_{\mathit{GT},i}}{{\sum }_i{\widehat{M}}_i+{\sum }_i{M}_{\mathit{GT},i}}$$ 5

$$L_{\mathit{BCE}}=-\frac{1}{N}\sum _{i=1}^N\left[M_{\mathit{GT},i}\mathit{log}{\widehat{M}}_i+(1-M_{\mathit{GT},i})\mathit{log}(1-{\widehat{M}}_i)\right].$$ 6

To enforce spatial coherence between prompts and masks, we add a geometric consistency term. The bounding box derived from the predicted mask,

$$B\left(\widehat{M}\right)=\mathit{bbox}\left(\widehat{M}\right),$$ 7

is encouraged to match the input box via a Smooth L1 penalty:

$$L_{\mathit{cons}}={\lambda }_{\mathit{cons}}\cdot \mathit{SmoothL}1\left(B\left(\widehat{M}\right),B_{\mathit{GT}}\right).$$ 8

In practice, $B\left(\widehat{M}\right)$ is computed directly from the continuous probability map (without hard thresholding) so that gradients can backpropagate through the box extraction.

The overall SEEM objective is

$$L_{\mathit{SEEM}}={\lambda }_{\mathit{mask}}{L}_{\mathit{mask}}+{\lambda }_{\mathit{cons}}{L}_{\mathit{cons}}.$$ 9

To improve stability, we resample lesion-containing and background slices to balance positives and negatives, and we apply gradient clipping together with a short warm-up at the start of training to mitigate oscillations in the prompt-to-mask pathway.

Uncertainty-aware feature modulator

In the original Language-Guided Adapter (LGA), the Feature Fusion Module (FFM) comprises parallel image and text branches. Each branch applies layer normalization followed by a ViT-style MLP, and the two streams are coupled via a dual cross-attention block. While effective for generic language-vision fusion, this design applies a uniform transformation over the image plane. In medical imaging, lesion boundaries are intrinsically ambiguous and annotations exhibit inter-observer variability, making uniform transformations suboptimal near uncertain edges. The structure is shown in Fig. 4.

Fig. 4. Uncertainty-Aware Feature Modulator (UAFM) within the LGA in SEEM pipeline.

The SEEM encoder follows the original LGA Feature Fusion Module (FFM): parallel image/text branches are aligned via dual cross-attention (q/k/v, LayerNorm, Text MLP). In the image branch, UAFM replaces Norm+ViT-MLP with an uncertainty-aware two path projector: a light MLP and a heavy MLP whose outputs are blended by a spatial uncertainty map U to allocate capacity near ambiguous lesion boundaries. The text branch and prompt encoder are unchanged (omitted for clarity). During fine-tuning, only the UAFM block and the mask decoder are updated, while both image and text encoders remain frozen (frozen-encoder regime). A lightweight reconstruction module (not shown in detail) maps the latent/sample features back to the ViT token grid for dimensional alignment and fusion with encoder features. This design preserves LGA's cross-modal coupling while improving boundary robustness with minimal trainable parameters.

We therefore replace the Norm+ViT-MLP block in the image branch only with an Uncertainty-Aware Fcature Modulator (UAFM)) block, leaving the text branch unchanged. Given fused image features F_v, we estimate a spatial uncertainty map U ∈ [0, 1]^H×W via stochastic dropout ensembles or feature variance proxies, followed by normalization. Rather than a single MLP, UAFM performs conditional routing through two projections:

$$F_{\mathit{light}}={\mathit{MLP}}_{\mathit{light}}\left(F_v\right),F_{\mathit{heavy}}=\mathrm{M}\mathrm{L}{\mathrm{P}}_{\mathit{heavy}}\left(F_v\right),$$ 10

and modulates capacity according to the estimated uncertainty:

$${\tilde{F}}_v=(1-U)\odot F_{\mathit{light}}+U\odot F_{\mathit{heavy}},$$ 11

where ⊙ denotes element-wise multiplication. The light path preserves efficiency in confident regions, whereas the heavy path allocates additional modeling capacity around high uncertainty boundaries. Both MLPs adopt the ViT-style design to maintain architectural compatibility with the pretrained encoder. At inference, the estimated uncertainty map U modulates feature capacity allocation in the image branch, enabling additional modeling capacity at ambiguous lesion boundaries while preserving efficiency in confident regions.

By confining the change to the image branch, UAFM keeps textual representations stable while adaptively shaping visual features as a function of uncertainty, rather than composing two independent modules. During fine-tuning (Selective Frozen-Encoder Fine-Tuning), we update only the UAFM block and the mask decoder while freezing the text and image encoders. This retains the strengths of the LGA fusion scheme and explicitly accounts for boundary uncertainty in lesion segmentation.

Joint fine-tuning with Grounding DINO

After the module-wise adaptation, we perform a second stage of joint alignment to couple the text to box and box to mask pathways. In this stage, backbone feature extractors remain frozen, while cross-attention layers, prompt embeddings, and task-specific heads are fine-tuned jointly.

Let $\widehat{B}$ be the bounding box predicted by Grounding DINO and $B\left(\widehat{M}\right)$ the box inferred from SEEM’s probabilistic mask. We introduce a cross module consistency loss:

$$L_{\mathit{cross}}={\lambda }_{\mathit{cross}}\cdot \mathit{SmoothL}1\left(B\left(\widehat{M}\right),\widehat{B}\right).$$ 12

The overall objective integrates detection, segmentation, and consistency terms:

$$L={\lambda }_{\mathit{det}}{L}_{\mathit{det}}+{\lambda }_{\mathit{mask}}{L}_{\mathit{mask}}+{\lambda }_{\mathit{cons}}{L}_{\mathit{cons}}+{\lambda }_{\mathit{cross}}{L}_{\mathit{cross}},$$ 13

where $L_{\det }$ is the Grounding DINO loss (focal classification, DETR-style box regression with L₁ + GIoU, and a grounding contrastive term), L_mask is the Dice+BCE segmentation loss, and L_cons enforces prompt-mask box consistency within SEEM.

This two stage training first lets each component specialize in its subtask and then enforces coherence along the “text → box → mask” pathway. In practice, the joint alignment reduces discrepancies between text conditioned boxes and mask derived boxes, yielding more stable text to box predictions and improved mask quality without altering backbone representations.

Pseudo-text generation based on geometric quantification

Since the Lung-PET-CT-Dx dataset provides only imaging data with coarse lesion annotations but lacks radiology reports, we designed a geometry-based template generation process to construct pseudo-text prompts aligned with lesion masks. This procedure enables the creation of text-image-mask triplets suitable for training text-guided segmentation models.

After obtaining lesion masks $\widehat{M}(x,y,z)$ from SEEM fine-tuned on ReXGroundingCT, we performed volumetric quantification. Given voxel spacing Δ_x × Δ_y × Δ_z, the lesion volume is defined as

$$V_{\mathit{lesion}}=\sum _{x,y,z}\widehat{M}(x,y,z)\cdot ({\mathrm{\Delta }}_x\cdot {\mathrm{\Delta }}_y\cdot {\mathrm{\Delta }}_z),\widehat{M}(x,y,z)\in \{0,1\},$$ 14

while the lung volume is obtained from the lung segmentation mask M_lung(x, y, z):

$$V_{\mathit{lung}}=\sum _{x,y,z}{M}_{\mathit{lung}}(x,y,z)\cdot ({\mathrm{\Delta }}_x\cdot {\mathrm{\Delta }}_y\cdot {\mathrm{\Delta }}_z).$$ 15

The relative lesion to lung volume ratio is then computed as

$$R=\frac{V_{\mathit{lesion}}}{V_{\mathit{lung}}}.$$ 16

To further characterize spatial location, we calculated the centroid of each lesion mask:

$$(x_c,y_c,z_c)=\frac{{\sum }_{x,y,z}(x,y,z)\cdot \widehat{M}(x,y,z)}{{\sum }_{x,y,z}\widehat{M}(x,y,z)}.$$ 17

The centroid coordinates can be mapped to lobar segmentation masks or an anatomical atlas to infer the anatomical region of the lesion.

Finally, these geometric descriptors were reformulated into structured natural language templates. For example, a lesion with volume ratio R = 0.8% and centroid localized to the right upper lobe would be described as: “A nodule in the right upper lobe, accounting for approximately 0.8% of the total lung volume, with centroid near slice coordinates (x = 120, y = 85, z = 42).”

Through this process, pseudo-report prompts were automatically generated for all cases in Lung-PET-CT-Dx, resulting in synthetic text-image-mask triplets. These pseudo-texts were derived exclusively from geometric computation and template rules, rather than from original clinical reports. This augmentation effectively enriched datasets lacking textual annotations, enabling weakly supervised fine-tuning of the text-guided segmentation pipeline.

What’s more, we add a standardized output layer at the end of the inference pipeline: First, reconstruct the original voxel grid and coordinate mapping based on the image geometry and patient coordinate system provided by the DICOM header information; reproject the segmentation generated by the network to the native image dimensions and slice number using nearest neighbor interpolation, and encode it as a DICOM Segmentation (Segmentation IOD). Explicitly reference the corresponding source image instance frame by frame to maintain the same coordinate reference as the source examination; We then compute millimeter based quantitative measurements within this native geometry and generate a TID 1500 Measurement Report. This report concurrently records spatial coordinate entries, image/segmentation references, and stable tracking identifiers, thereby forming an auditable, reproducible result chain directly consumable by clinical systems.

Re-finetuning Grounding DINO for lung cancer domain

After generating template-based pseudo-texts on the Lung-PET-CT-Dx dataset, we further fine-tuned Grounding DINO to enhance its accuracy in text-to-box localization for lung cancer lesions. Considering the large semantic and scale distribution gap between natural images and thoracic CT, as well as the risk of catastrophic forgetting, we adopted a domain adaptation strategy inspired by recent LoRA-based fine-tuning in medical imaging. In practice, the visual backbone and the text encoder were kept frozen, while low-rank adapters were inserted into the cross-modal fusion modules and decoder to allow lightweight parameter adjustment, and the detection head (classification and bounding box regressors) was kept fully trainable so that the localization task could be directly optimized for the lung domain.

Training samples were constructed from CT slices, pseudo-report prompts, and target bounding boxes derived from SEEM masks; to account for the inherent noise of synthetic text, each sample was assigned a confidence weight w_i based on the quality of the corresponding mask, measured by indicators such as mask-box IoU, connectivity, and boundary smoothness, and the per-sample loss was scaled accordingly. Predictions $\left\{{\widehat{B}}_i\right\}$ were matched with ground truth boxes {B_j} using Hungarian matching, and the detection loss was defined as

$$L_{\mathit{det}}=\sum _{(i,j)\in \mathcal{M}}\left[\mathit{Focal}({\widehat{p}}_i,y_j)+{\lambda }_{\mathrm{r}\mathrm{e}\mathrm{g}}\cdot \mathit{GIoU}({\widehat{B}}_i,B_j)\right],$$ 18

where ${\widehat{p}}_i$ denotes the predicted class confidence and λ_reg balances classification and regression terms.

To improve spatial precision, we further introduced a centroid alignment loss,

$$L_{\mathit{cent}}=\sum _{(i,j)\in \mathcal{M}}{\parallel \mathit{center}\left({\widehat{B}}_i\right)-\mathit{centroid}\left(M_j\right)\parallel }_1,$$ 19

where centroid(M_j) is derived from SEEM generated masks, ensuring that predicted box centers remain consistent with lesion geometry.

To enhance semantic alignment between text and visual regions, a contrastive loss was added:

$$L_{\mathit{con}}=-\sum _i\mathit{log}\frac{\mathit{exp}\left(\mathit{cos}\left(f_{\mathit{text}}\right(t_i),f_{\mathit{vis}}({\widehat{B}}_i\left)\right)/\tau \right)}{{\sum }_k\mathit{exp}\left(\mathit{cos}\left(f_{\mathit{text}}\right(t_i),f_{\mathit{vis}}({\widehat{B}}_k\left)\right)/\tau \right)},$$ 20

where f_text(t_i) and $f_{\mathit{vis}}\left({\widehat{B}}_i\right)$ denote the embeddings of the textual prompt and the predicted region, and τ is the temperature hyperparameter.

The overall objective thus became

$$L=\sum _i{w}_i\cdot (L_{\mathit{det}}+\alpha L_{\mathit{cent}}+\beta L_{\mathit{con}}),$$ 21

with α and β weighting geometric and contrastive contributions.

Training was conducted in two stages: in the warm up stage, only high-confidence samples (w_i > w_thresh) were used to update adapters and detection heads while all other modules remained frozen; in the second stage, the confidence threshold was gradually lowered to admit medium quality samples, and the upper layers of the decoder and, if necessary, the top most backbone block were partially unfrozen to enhance domain adaptation. To stabilize optimization and reduce overfitting, we applied learning rate warm-up, gradient clipping, dropout regularization, and hard negative mining, penalizing prompts without corresponding targets that still received high prediction scores.

After fine-tuning, Grounding DINO was used to infer bounding boxes for new CT slices given pseudo-prompts, and predictions were further validated against SEEM outputs by computing $IoU\left(bbox\right(\widehat{M}),\widehat{B})$, filtering out low IoU detections. This refinement significantly improved bounding box mean average precision, reduced centroid errors, and enhanced overall localization accuracy in the lung cancer domain.

Author contributions

S.W., Z.W., Z.S., and W.M. contributed equally to this work, having full access to all study data and assuming responsibility for the integrity and accuracy of the analyses (validation, formal analysis). D.H. conceptualized the study, designed the methodology, and participated in securing research funding (conceptualization, methodology, funding acquisition). T.L. carried out data acquisition, curation, and investigation (investigation, data curation) and provided key resources, instruments, and technical support (resources, software). B.W. drafted the initial manuscript and generated visualizations (writing—original draft, visualization). D.K., X.L., K.R., and M.S. supervised the project, coordinated collaborations, and ensured administrative support (supervision, project administration). All authors contributed to reviewing and revising the manuscript critically for important intellectual content (writing—review & editing) and approved the final version for submission.

Data availability

All datasets used in this study are publicly accessible from the following official sources: ReXGroundingCT (paired radiology findings and lesion masks): https://arxiv.org/abs/2507.22030. Lung-PET-CT-Dx (TCIA; CT and PET–CT with bounding-box annotations): https://www.cancerimagingarchive.net/tcia-version-dnlds/lung-pet-ct-dx-ver2-rad/. LCTSC 2017 (Lung CT Segmentation Challenge): https://www.cancerimagingarchive.net/collection/lctsc/. NLSTseg (pixel-level LDCT lung lesion annotations): https://www.nature.com/articles/s41597-025-05742-x. NSCLC Radiogenomics (TCIA): https://www.cancerimagingarchive.net/collection/nsclc-radiogenomics/. MSD-Lung (Medical Segmentation Decathlon, Task06): https://medicaldecathlon.com/. PCLT20K (large-scale PET–CT lung tumor segmentation): https://arxiv.org/abs/2503.17261. Usage in this work. ReXGroundingCT was used exclusively for training text-to-region alignment. Lung-PET-CT-Dx was used for cross-dataset adaptation (including pseudo-text generation) and provided the primary test set. The remaining datasets were used for external evaluation.

Code availability

The code implementing all preprocessing, training, and evaluation procedures will be made publicly available on GitHub upon publication. To facilitate transparency and full reproducibility, an anonymized version of the repository is currently accessible at https://anonymous.4open.science/r/Biomedloop-AF08/README.md, providing standardized PyTorch pipelines.

Competing interests

The authors declare no competing interests.