OpenVocabCT: Towards Universal Text-driven CT Image Segmentation

Abstract

Computed tomography (CT) is extensively used for accurate visualization and segmentation of organs and lesions. While deep learning models such as convolutional neural networks (CNNs) and vision transformers (ViTs) have significantly improved CT image analysis, their performance often declines when applied to diverse, real-world clinical data. Although foundation models offer a broader and more adaptable solution, their potential is limited due to the challenge of obtaining large-scale, voxel-level annotations for medical images. In response to these challenges, prompting-based models using visual or text prompts have emerged. Visual-prompting methods, such as the Segment Anything Model (SAM), still require significant manual input and can introduce ambiguity when applied to clinical scenarios. Instead, foundation models that use text prompts offer a more versatile and clinically relevant approach. Notably, current text-prompt models, such as the CLIP-Driven Universal Model, are limited to text prompts already encountered during training and struggle to process the complex and diverse scenarios of real-world clinical applications. Instead of fine-tuning models trained from natural imaging, we propose OpenVocabCT, a vision-language model pretrained on large-scale 3D CT images for universal text-driven segmentation. Using the large-scale CT-RATE dataset, we decompose the diagnostic reports into fine-grained, organ-level descriptions using large language models for multi-granular contrastive learning. We evaluate our OpenVocabCT on downstream segmentation tasks across 14 public datasets and 1 institutional dataset for organ and tumor segmentation, demonstrating the superior performance of our model compared to existing methods. All code, datasets, and models will be publicly released at https://github.com/ricklisz/OpenVocabCT.

I. Introduction

In clinical practice, computed tomography (CT) is widely used for detailed anatomical visualization and precise segmentation of organs-at-risk (OAR) and tumor lesions. With the advent of deep learning models such as convolutional neural networks (CNNs) [1]–[3] and vision transformers (ViTs) [4], [5], medical image segmentation has shown significant promise in CT image analysis. While many purpose-built models achieve remarkable precision for specific tasks [6]–[9], their effectiveness is often limited when dealing with the generalizability observed in diverse multimodal clinical data or handling various imaging tasks.

Foundation models, on the other hand, are designed to provide a broad base of capabilities that can be adapted to many tasks without extensive retraining [11]. With the substantial increase in the availability of annotated imaging datasets, numerous studies have been conducted on building foundation segmentation models for 3D CT images [12]–[14]. However, obtaining abundant voxel-level annotations for medical images remains time-consuming and expensive. As a result, prompting-based foundation models are being explored as a more data-efficient solution [15]–[17]. These models leverage either visual prompts (points or bounding boxes) or text prompts (natural language) to guide the segmentation process. While visual-prompting methods such as the Segment Anything Model (SAM) [18] are widely used for various tasks, they still face challenges when applied to medical settings due to: 1) SAM requires significant image-level annotations for training and considerable manual prompting for inference, which is not a scalable solution for clinical settings; 2) visual prompts lack the necessary clinical details to guide accurate segmentation. These limitations reduce the clinical utility of visual-prompting methods and underscore the importance of vision-language models, as both visual and textual features are essential for effective segmentation to facilitate the diagnosis, treatment planning, and treatment response assessment.

Text-prompting models have emerged as a promising solution for building foundation segmentation models in CT [16], [19]. Recent developments in learning visual representations from text supervision have shown tremendous success in computer vision [20] and medical imaging [21]. However, prior works primarily focus on 2D medical images, such as chest X-rays [22], [23], which limits their application to the more widely used volumetric CT imaging for individual segments of the whole-body. Additionally, text-driven segmentation methods for 3D CT scans often rely on text embeddings from pretrained text encoders, with limited or insufficient use of image-text alignment strategies [16], [19], [24]. This insufficient alignment can restrict the model’s ability to generalize effectively to diverse unseen clinical text prompts or varying medical vocabularies, hindering its utility in real-world clinical settings.

In real-world practice, healthcare professionals use varied prompts to describe or annotate specific organs, making generalization particularly important for achieving universal organ segmentation. Furthermore, radiology reports frequently contain detailed but lengthy diagnostic information, qualitatively deduced from the images, some of which is unrelated to organs or lesions, which poses challenges for efficient image-to-text alignment. To address these challenges, we developed a dedicated image-text supervision framework for 3D CT that generalizes to diverse and unseen text prompts, and that can be applied to text-prompt segmentation tasks. Our main contributions are summarized as follows:

1. We curate organ-level captions from paired CT radiology reports in CT-RATE (n=50188) [10] by prompting large language models and filtering with RadLex. The pipeline yields concise organ phrases aligned to each volume and is supported by a clinical audit (high factuality) and distributional analyses.

2. We introduce a multi-granularity, multi-positive contrastive objective that aligns vision encoder with both organ-level captions and report-level descriptions. This transfers the text encoder’s lexical invariance into a joint image–text space, improving generalization to training-unseen prompts (e.g., merged organs and synonyms).

3. We evaluate our method OpenVocabCT on 14 public datasets for organ and tumor segmentation, demonstrating strong finetuning performance and generalization capabilities. Compared to vision-only segmentation models, our approach achieves comparable performance while enhancing usability by allowing clinicians to interact with the model using natural language. Moreover, compared to text-driven models, it delivers superior performance in generalization to diverse, previously unseen prompts. Finally, we demonstrate clinical transfer by adapting the model to an institutional gynecologic oncology cohort (GYN), where it still achieves competitive segmentation performance.

II. Related Work

A. Medical image segmentation

Deep learning-based methods [1], [2] have been widely applied for organ segmentation and tumor segmentation and detection, yielding promising results. However, these methods are often task-specific or organ-specific, such as organ segmentation [25] or tumor detection [26], [27]. Recently, there has been a growing interest in building foundation segmentation models for various organs and tumors [13], [19], [28], and for this reason adapting SAM for ‘universal’ segmentation [15], [29]; however, SAM’s interactive nature still requires considerable manual input, limiting its applicability in clinical settings [30]. Additionally, since SAM was developed using natural images, it may lack the medical semantic understanding to differentiate between healthy organs and tumors. In contrast, our method integrates medical professional’s knowledge from radiology reports into a text model, which is more practical than visual prompting for clinical usage.

B. Vision-language model for medical imaging

Vision-language models such as CLIP [20] have demonstrated the ability to learn transferable visual features through language supervision, without manual image annotations. In medical imaging, where diagnostic radiology-founded reports complement imaging data, CLIP-based models have shown promise across various tasks, including organ segmentation [19], disease classification [22], and image-text retrieval [31]. However, adapting CLIP to medical imaging presents unique challenges. Unlike natural images, which can often be summarized in a few sentences, medical images like CT scans contain complex diagnostic information, resulting in detailed radiology reports. This complexity demands improved local-level alignment between image and text. Fine-grained vision language modeling aims to explicitly construct region–text pairs for grounded alignment. CT-GLIP constructs organ-level region–text pairs for 3D CT and performs grounded contrastive pretraining to recognize organs and abnormalities in zero-shot/fine-tuning settings [32]. fVLM aligns anatomy-level CT regions with report sentences and calibrates contrastive pairs to mitigate false negatives, reporting broad diagnosis results across 15 anatomies [33]. These methods leverage explicit region supervision (grounded pairs) and primarily target recognition, whereas our approach does not rely on segmentation masks and is built for text-driven 3D segmentation, aligning global reports and multiple organ captions via MGCL while freezing a segmentation-pretrained encoder to preserve dense features. Another significant challenge is data scarcity. While CLIP is ideal for large-scale datasets, medical image-text datasets are relatively limited, which requires a more label-efficient approach. Our method addresses this by curating a radiology-specific corpus of image-text pairs, designed to supplement the radiology-specific knowledge that conventional CLIP training lacks.

C. Text-driven segmentation model

Prompt engineering has demonstrated strong performance improvements in both natural language processing and computer vision tasks. By utilizing pre-trained vision-language models [20], text prompting methods results in open-vocabulary segmentation [34], [35] and referring segmentation [36] tasks. Previous studies demonstrate that pretrained CLIP models are effective for 2D medical image segmentation tasks [17], [37]. Li et al. propose a text-augmented segmentation model that utilizes medical text annotations and pseudo labels in a semi-supervised framework [24]. However, extending these approaches to 3D vision-language models for CT segmentation remains an active research area due to the limited availability of paired CT image-text datasets. Recent efforts have centered on developing text-prompted universal models for segmenting various organs and tumors in 3D volumes. Liu et al. proposed to leverage CLIP’s text embeddings to guide the segmentation model for partially-labeled datasets [19]. Zhao et al. introduced SAT, a large-scale segmentation model with a knowledge-enhanced text encoder for multimodal organ and tumor segmentation [16]. However, no existing approach fully leverages a vision-language-aligned model like CLIP for 3D medical image segmentation tasks. Given CLIP’s strong grounding capabilities, we argue that vision-language alignment is essential for building robust text-driven segmentation models. To address this, we propose a pre-trained vision-language model trained on a large-scale CT image-report dataset, incorporating diverse captions for each organ and region.

III. Method

A. Preliminary

In recent years, language supervision methods such as CLIP [20] and SigLIP [38] have been shown to be effective in learning transferable representations. Concretely, given a batch of images $\left\{I_1,I_2,\cdots ,I_N\right\}$ and paired text descriptions $\left\{T_1,T_2,\cdots ,T_N\right\}$, CLIP leverages an image encoder $E_i$ to extract the image embeddings $\left\{v_1,v_2,\cdots ,v_N\right\}$, and a text encoder $E_t$ to extract the text embeddings $\left\{t_1,t_2,\cdots ,t_N\right\}$:

$$v_N=E_i\left(I_N\right),t_N=E_t\left(T_N\right).$$ (1)

The extracted image embedding and text embedding are used to compute InfoNCE loss, in which paired image-text samples are considered as positives and the unpaired ones as negatives. The image-to-text loss $L_{i2t}$ can be formulated as:

$$L_{\mathrm{C}\mathrm{L}\mathrm{I}\mathrm{P}}^{i2t}=-\sum _{i=1}^N\mathrm{l}\mathrm{o}\mathrm{g}\left(\frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(\mathrm{c}\mathrm{o}\mathrm{s}\left(v_i,t_i\right)/\tau \right)}{\sum _{j=1}^N\mathrm{e}\mathrm{x}\mathrm{p}\left(\mathrm{c}\mathrm{o}\mathrm{s}\left(v_i,t_j\right)/\tau \right)}\right).$$ (2)

where $\mathrm{c}\mathrm{o}\mathrm{s}⟨\cdot ,\cdot ⟩$ denotes the cosine similarity and $\tau$ is a learnable temperature parameter. The final bidirectional total loss can be formulated as ${ℒ}_{\text{CLIP}}=\frac{1}{2}\left({ℒ}_{\text{CLIP}}^{i2t}+{ℒ}_{\text{CLIP}}^{t2i}\right)$.

B. Improving image-text alignment with language models

While CLIP has demonstrated promising results on natural images, it still faces considerable challenges in medical imaging. First, paired medical image and text data are very limited, necessitating the use of image-level and text-level data augmentations. Second, medical text typically comes in the form of diagnostic reports, which are longer and more complex than the natural image captions. Third, reports typically have broad, patient-level observations without granular, organ-specific details for text-driven segmentation.

To address these limitations, we propose to leverage existing LLMs to automatically extract concise, granular descriptions from reports. We employ in-context learning, which is known to enhance alignment between LLM outputs and human-aligned standards, to guide this process. First, we craft some few-shot examples of concise organ-level captions from the radiology impressions (Figure 1). Subsequently, we prompt GPT-4 and Llama-3 APIs using a structured query: ”You are a radiology expert. Given the following radiology report, generate concise captions describing each organ or pathology.”. This ensures the model to maintain consistency in its response and preserves the semantic details in the original caption. On average, the LLM produces 7 raw organ-level captions per report.

Fig. 1.

We use CT-RATE [10], a large-scale paired CT and radiology report dataset, to generate pre-training data. Some key information about this dataset is present: (a) Example of paired CT scans and detailed captions for each organ; (b) Top 100 captions containing detailed organ-level information; (c) Distribution of the top 20 organs/tissues from these captions. We leverage pre-trained large language models to break down radiology reports into fine-grained captions and filtered out low-quality captions using substring matching for open-vocabulary segmentation training.

Finally, observing that the generated captions may contain redundant information, we followed MetaCLIP’s [39] approach to filter out low-quality captions via substring matching. We utilize a radiology-specific text corpus RadLex [40], which provides an agreed set of named entities for radiology procedures, as our text metadata. We then apply sub-string matching on the granular captions with the metadata entries, which identifies high-quality captions that contain any of the metadata entries, filtering the various types of noise that the LLM may introduce. This process produces 3 captions per report. By generating concise and granular organ descriptions, we enhance the quality of textual supervision, leading to segmentation performance. We have performed a manual check by randomly sampling a subset (N=100) and asked a clinician to verify quality of the curated captions.

C. Multi-granularity contrastive learning

To improve image-text alignment, we introduce a multi-granularity contrastive learning framework that leverages both LLM-generated granular captions and the original radiology reports. Given the granular captions $C^l=\left[c_1,\dots ,c_M\right]$, we implement a simple random sampling strategy to gather short captions:

$$S_{i,j}\sim \mathrm{U}\mathrm{n}\mathrm{i}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}\left(\left[c_1,\dots ,c_M\right]\right),$$ (3)

where $S_{i,j}$ refers to the $j$-th caption of $i$-th sample from the short caption set. The multi-granular contrastive text-to-image loss becomes:

$${ℒ}_{\mathrm{M}\mathrm{G}\mathrm{C}\mathrm{L}}^{t2v}=-\sum _{i=1}^N\sum _{j=1}^K\mathrm{l}\mathrm{o}\mathrm{g}\frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(\mathrm{c}\mathrm{o}\mathrm{s}\left(v_i,t_{i,j}\right)/\tau \right)}{\sum _{n=1}^N\mathrm{e}\mathrm{x}\mathrm{p}\left(\mathrm{c}\mathrm{o}\mathrm{s}\left(v_n,t_{i,j}\right)/\tau \right)},$$ (4)

where $t_{i,j}$ refers to the text embedding of $S_{i,j}$, and $K$ denotes the number of randomly sampled short captions ($\mathrm{K}=3$ in our implementation). We formulate the bi-directional multi-granular contrastive loss: ${ℒ}_{\text{MGCL}}=\left({ℒ}_{\text{MGCL}}^{t2v}+{ℒ}_{\text{MGCL}}^{v2t}\right)/2$. By introducing short captions as text augmentations, we increase the diversity of the training data, resulting in a more complete and aligned representation of both images and text.

Empirically, we find that image-text pretraining may not result in optimal finetuning performance for dense prediction tasks like segmentation. To enhance the model’s dense prediction capability, we first pretrained the image encoder on TotalSegmentator for segmentation for 500 epochs, providing a good initialization for the image encoder. We then initialized the image encoder with the pretrained model’s weights before image-text alignment. During image-text contrastive learning, we locked the image encoder and only tuned the text encoder to align the text embeddings with the vision embeddings.

For the final pretraining objective, we combine the original CLIP loss ${ℒ}_{\text{CLIP}}$ using the full radiology report with the proposed ${ℒ}_{\text{MGCL}}$ loss using the generated short captions:

$${ℒ}_{\text{Final}}=L_{\mathrm{C}\mathrm{L}\mathrm{I}\mathrm{P}}+L_{\mathrm{M}\mathrm{G}\mathrm{C}\mathrm{L}},$$ (5)

This approach helps the model to capture detailed connections between images and text while keeping the broader context provided in the radiology reports, improving its ability to generalize to a variety of image-text pairs.

IV. Experiments

A. Datasets and evaluation metrics

1). Vision-language pretraining dataset:

We leverage a large-scale chest CT dataset CT-RATE [10], that have 21,304 patients and 50,188 image-radiology report pairs. The CT images were obtained using a range of reconstruction techniques. The radiology reports have four sections: 1. clinical information inlcuding symptoms and history, 2. imaging technique and acquisition protocol, 3. imaging findings (anatomical/pathological observations), and 4. impression/diagnosis.

2). Segmentation datasets:

For finetuning on segmentation datasets, we connect our pretrained encoder to STUNet decoder and finetuned on a variety of datasets: TotalSegmentator [41], MSD Lung, MSD Pancreas, MSD Hepatic Vessel, MSD Colon, MSD Liver [42], KiTS23 [43], BTCV, BTCV Cervix, AMOS22 [44], SegRap2023 [45], COVID-19 [46]. For TotalSegmentator, we use only the first fold as test set and the remaining four folds as the training set. For the remaining dataset (i.e. MSD Lung, MSD Pancreas, MSD Hepatic Vessel, MSD Colon, MSD Liver, KiTS23, BTCV, BTCV Cervix, AMOS22, SegRap2023, COVID-19), we perform 5-fold cross validation for each method on each dataset. We also evaluate the segmentation model’s generalization capabilities on the FLARE22 [47] and SegTHOR [48] datasets. Dice Similarity Coefficient (DSC) and Normalized Surface Distance (NSD) are utilized to quantitatively evaluate organ/tumor segmentation performance. We calculate NSD at the original volume spacing with 1.0 mm tolerance. A summary of the datasets used in this paper is provided in Table I.

TABLE I.

Summary of datasets used for pretraining, finetuning, and evaluation.

Dataset	Usage	Target Type	Anatomical Region	Sample size
CT-RATE	Pretraining (vision-language encoders)	Report and organ texts	Chest/abdomen	50,188 volumes (21,304 patients)
TotalSegmentator	Pretraining (vision encoder) + Finetuning	Organ	Whole-body	1204 volumes
MSD Lung	Finetuning	Tumor	Chest	63 volumes
MSD Pancreas	Finetuning	Organ + Tumor	Abdomen	281 volumes
MSD Hepatic Vessel	Finetuning	Organ + Tumor	Abdomen	303 volumes
MSD Colon	Finetuning	Tumor	Abdomen	126 volumes
MSD Liver	Finetuning	Organ + Tumor	Abdomen	131 volumes
KiTS23	Finetuning	Organ + Tumor	Abdomen	489 volumes
BTCV	Finetuning	Organ	Abdomen	30 volumes
BTCV-Cervix	Finetuning	Organ	Pelvis	30 volumes
AMOS22	Finetuning	Organ	Abdomen	360 volumes
FLARE22	Generalization	Organ	Abdomen	50 volumes
SegTHOR	Generalization	Organ	Thorax	40 volumes
SegRap2023	Finetuning	Organ	Head and neck	120 volumes
COVID-19 Segmentation	Finetuning	Lesion	Chest	199 volumes
Institutional GYN	Finetuning	Organ	Pelvis	15 volumes (15 female patients)

B. Implementation details

1). Pretraining:

Our OpenVocabCT is composed of an image encoder, a text encoder and a segmentation decoder. For pretraining, we use the STUNet-Large [13] as the backbone for our image encoder due to its excellent performance in various benchmarks. The image encoder is pretrained on TotalSegmentator for 500 epochs using the 4 training folds mentioned previously. We use BIOLORD [49] as our text encoder, which was pretrained on both clinical sentences and biomedical concepts. The text encoder’s latent feature is projected to the image encoder’s latent feature dimension using a simple MLP. We preprocessed CT images by resampling them to an isotropic spacing of 1.5 mm × 1.5 mm × 1.5 mm and padding them to a size of 220 × 220 × 220. For image captions, we randomly sample three granular captions from the filtered dataset, along with the findings section from the original report. Pretraining is conducted on four NVIDIA A100 GPUs, with a batch size of 32 per GPU.

2). Finetuning on segmentation datasets:

We directly use the aligned image encoder and text encoder from pretraining. We then connect the pretrained image encoder to the decoder of STUNet-Large. To avoid catastrophic forgetting, we freeze the weights of both image encoder and text encoder and only tune the segmentation decoder. For text-driven segmentation, we generate organ prompts such as ‘Liver’ or ‘Left kidney’ and feed the tokenized prompts to the text encoder similar to [16], [19]. The text features are further processed through a text-guidance connector to generate query embeddings, which are then multiplied with the image features to produce segmentation maps. In our ablation study, we explored various strategies for designing the text-guidance connector.

C. Finetuning performance

We compare our method with both vision-only segmentation models and text-driven segmentation frameworks. For vision-only methods, we compare with the supervised method nnUNetv2 [2] and self-supervised methods UniMiSS [28], S3D [50], and Voco [51]. We implement self-supervised pretraining on CT-RATE dataset to maintain a fair comparison. For text-driven approaches, we compare with CLIP-Driven Universal Model [19], SAT-Pro [16], and CT-CLIP [10]. As shown in Table II, our method is able to achieve better performance than both the previous SOTA image-only methods and the text-driven SOTA methods on TotalSegmentator dataset. Compared to the best image-only method nnUNet, our method outperforms by 3.8% DSC and 4.1% NSD. Compared to the best text-driven method (SAT-Pro), our method outperforms by 3.1% DSC and 5.0% NSD. This shows that our method can effectively segment the majority of organs with superior performance. For tumor segmentation tasks, as shown in Table III, our method achieves comparable performance as the best text-driven method Universal model and outperforms the best vision-only method nnUNet by 2.6% in DSC and 2.4% in NSD on average. In Table IV, OpenVocabCT achieves the highest average Dice (81.6%) and NSD (76.7%) across BTCV, BTCVCervix, and AMOS22, outperforming all vision-only and text-driven baselines. In Table V, OpenVocabCT achieves the highest DSC and NSD on both SegRap2023 and COVID-19 datasets, outperforming strong baselines such as nnUNet and SAT Pro. Paired t-test show improvements over nnUNet are statistically significant (p < 0.05) for SegRap NSD (p = 0.0008), COVID-19 DSC (p = 0.04), and COVID-19 NSD (p = 0.0446). SegRap DSC shows a smaller but consistent gain (+0.43%) that does not reach statistical significance (p=0.46). This supports that our method offers consistent, statistically significant improvements on most metrics.

TABLE II.

Finetuning performances on TotalSegmentator. Results reported in DSC (%) and NSD (%). BOLD indicates best result, Underline second best.

Method	Vertebrae		Cardiac		Muscles		Organs		Ribs		Avg
	DSC	NSD	DSC	NSD	DSC	NSD	DSC	NSD	DSC	NSD	DSC	NSD
CLIP-Driven	81.1	77.4	84.5	66.8	88.8	74.9	86.4	76.1	82.1	77.2	84.6	74.5
SAT Pro	85.4	80.5	89.2	70.6	88.0	72.5	87.7	76.7	83.7	78.1	87.6	75.7
CT-CLIP	76.6	71.2	78.1	64.3	81.2	67.7	79.1	69.4	74.2	70.9	77.8	68.7
nnUNet	87.0	82.2	88.7	67.7	85.1	72.4	87.5	76.0	86.1	84.8	86.9	76.6
UniMiSS	85.4	81.0	86.0	67.8	86.6	74.1	88.6	75.4	86.5	85.6	86.5	76.8
S3D	85.4	83.6	85.5	68.2	85.2	74.2	88.2	76.5	85.2	86.2	85.8	77.7
Voco	84.1	83.2	85.8	68.2	84.3	73.8	88.1	76.5	85.0	86.1	85.3	77.6
OpenVocabCT	90.4	87.8	90.3	72.0	90.0	77.2	91.3	78.3	91.6	88.1	90.7	80.7

TABLE III.

Finetuning performance on tumor Segmentation. Results are reported in DSC (%) and NSD (%) for different datasets (MSD Lung, Pancreas, Hepatic Vessel, Colon, Liver, and KiTS23) across various segmentation tasks. BOLD indicates best result. Underline second best result.

Method	MSD Lung		MSD Pancreas						MSD Hepatic Vessel						MSD Colon
	Lung Tumor		Pancreas		Pancreas Tumor		Avg		Hepatic Vessel		Hepatic Vessel Tumor		Avg		Colon Tumor
	DSC	NSD	DSC	NSD	DSC	NSD	DSC	NSD	DSC	NSD	DSC	NSD	DSC	NSD	DSC	NSD
CLIP-Driven	67.1	59.4	82.7	80.2	60.8	58.1	71.7	69.2	62.6	73.7	69.4	63.0	66.0	68.4	62.1	62.9
SAT Pro	61.8	59.1	76.2	72.9	41.6	43.4	58.9	58.2	65.2	77.6	61.8	62.3	63.5	70.0	32.4	29.7
CT-CLIP	52.9	51.3	68.3	64.6	33.2	31.8	50.8	48.2	52.9	64.0	55.0	51.8	53.9	57.9	28.1	23.2
nnUNet	68.2	65.6	81.6	79.2	53.1	53.6	67.4	66.4	67.7	81.1	72.1	69.8	69.9	75.5	49.2	56.7
UniMiSS	66.6	58.5	79.0	75.4	45.7	45.1	62.4	60.3	67.0	80.8	71.8	68.4	69.4	74.6	45.3	48.1
S3D	66.9	60.2	79.4	76.5	45.5	44.9	62.5	60.7	67.4	80.9	72.2	68.3	69.8	74.6	36.1	35.8
Voco	69.6	61.9	79.8	76.2	43.9	44.1	61.8	60.2	67.2	80.7	71.1	67.2	69.2	73.9	35.6	36.4
OpenVocabCT	70.3	65.3	81.9	80.4	60.0	58.3	71.0	69.9	67.3	81.7	70.2	69.0	68.8	75.4	62.8	64.1
Method	MSD Liver						KiTS23								Overall Avg
	Liver		Liver Tumor		Avg		Kidneys		Kidney Cysts		Kidney Tumor		Avg		Avg
	DSC	NSD	DSC	NSD	DSC	NSD	DSC	NSD	DSC	NSD	DSC	NSD	DSC	NSD	DSC	NSD
CLIP-Driven	96.5	82.0	71.9	72.7	84.2	77.4	95.2	90.5	76.3	61.8	34.9	60.8	68.5	71.0	70.9	69.6
SAT Pro	92.7	76.9	59.7	57.2	76.2	67.1	93.2	91.0	80.6	79.8	45.7	77.5	73.2	82.7	64.6	66.1
CT-CLIP	86.7	68.0	51.8	50.6	63.5	59.3	85.4	81.7	74.3	73.6	31.2	58.2	63.6	71.2	56.4	56.2
nnUNet	93.8	80.4	66.0	65.7	79.9	73.1	96.1	94.1	84.4	84.3	47.9	81.3	76.1	85.6	70.9	73.8
UniMiSS	94.0	80.3	64.9	65.0	79.5	72.7	96.3	94.4	83.2	81.6	47.0	78.7	75.5	84.9	69.2	70.6
S3D	93.4	78.6	60.9	61.3	77.2	70.0	96.0	93.7	85.0	82.1	46.4	79.4	75.8	85.1	68.1	69.3
Voco	92.6	78.0	63.2	67.5	77.9	72.8	96.1	94.0	84.9	82.6	45.8	80.4	75.6	85.7	68.2	69.9
OpenVocabCT	96.6	82.5	68.5	71.2	82.6	76.9	96.8	94.9	86.3	86.9	48.1	83.8	77.6	88.5	73.5	76.2

TABLE IV.

Finetuning performances on BTCV, BTCVCervix and AMOS22. Results reported in DSC (%) and NSD (%). BOLD indicates best result; Underline second best.

Method	BTCV		BTCVCervix		AMOS22		Avg
	DSC	NSD	DSC	NSD	DSC	NSD	DSC	NSD
CLIP-Driven	78.8	80.0	68.2	47.9	84.6	86.3	77.2	71.4
SAT Pro	80.1	81.5	72.4	54.0	86.3	87.8	79.6	74.4
CT-CLIP	63.0	64.4	58.5	40.1	70.3	72.6	63.9	59.0
nnUNet	78.2	81.1	68.6	49.6	88.1	90.0	78.3	73.6
UniMiSS	78.5	78.6	65.6	47.6	88.5	90.3	77.5	72.2
S3D	76.5	77.7	66.1	47.4	87.5	88.9	76.7	71.3
Voco	76.6	77.8	66.5	48.4	88.1	89.8	77.1	72.0
OpenVocabCT	82.2	84.2	73.6	55.2	89.0	90.7	81.6	76.7

TABLE V.

Finetuning performances on SegRap2023 and COVID-19. Results reported in DSC (%) and NSD (%). BOLD indicates best result; Underline second best.

Method	SegRap2023		COVID-19
	DSC	NSD	DSC	NSD
UniMISS	84.19	81.83	72.65	57.76
SAT Pro	85.44	81.53	73.11	57.97
nnUNet	86.69	83.29	72.39	56.95
OpenVocabCT	87.12	86.01	73.99	58.44

D. Generalization to training-unseen text prompts

Compared to vision-only models, text-driven segmentation models are more flexible by parsing a wide range of clinical descriptions to guide the segmentation process. This allows text-driven models to generalize to partially labeled data that are incomplete or inconsistent as compared to training data. In real-world scenarios where healthcare professionals use varied prompts to describe or annotate specific organs, this generalization capability becomes particularly valuable for achieving universal organ segmentation. To assess these generalization capabilities, we evaluate how well the text-driven segmentation model handles two categories of training invisible prompts: 1) prompts obtained by merging multiple organs and 2) prompts that are synonymous terms for the target organ. Specifically, for category 1, we obtain these prompts by merging various suborgans used in training (i.e. left lung is obtained by merging the lung upper lobe left, lung lower lobe left classes in TotalSegmentator). For category 2, we take the training visible prompt and substitute it for a synonym. For example, renal organs is synonym for kidney and hepatic system is synonym for liver.

Generalization results to merged suborgans is shown in Table VI. Compared to the CLIP-Driven Universal Model, SAT Pro, and CT-CLIP, our method consistently achieves superior performance on merging simple left and right organs (e.g. left and right lungs, left and right kidneys). These results highlight the model’s ability to interpret novel combinations of suborgans effectively. Generalization results to synonyms is shown in Table VII. Notably, our method also achieves significantly higher performance in challenging cases (e.g., cervical vertebrae, lumbar vertebrae, thoracic vertebrae, veins, arteries) without explicitly being trained on such prompts. Our method consistently outperforms the existing text-driven methods, achieving the highest average performance (73.2% DSC) across all categories.

TABLE VI.

Generalizability to training invisible text prompts: merging suborgans. L.: Left. R.: Right. AG: Adrenal gland. V.: Vertebrae. BOLD indicates best result. Underline second best result.

Method	L. Lung	R. Lung	L&R Lung	L.Heart	R. Heart	L. and R. Kidney	L. and R. AG	Heart	L. Ribs	R. Ribs
CLIP-Driven	80.8	50.1	21.0	42.3	64.0	45.7	40.4	63.1	0	10.3
SAT Pro	63.2	62.4	6.0	0	52.9	28.7	34.2	8.4	63.2	62.4
CT-CLIP	48.5	47	26.5	47.0	54.2	64.9	0	25.7	3.1	0
OpenVocabCT	95.4	92.8	85.6	51.3	68.2	89.3	77.3	54.2	68.1	66.7
Method	Trachea and Esophgaus		L. Gluteus	R.Gluteus	Cervical V.	Lumbar V.	Thoracic V.	Veins	Arteries	Avg
CLIP-Driven	69.0		50.0	19.8	20.3	5	21.1	36.1	63.1	39.0
SAT Pro	1.1		49.6	49.8	29	4.9	13.8	10.4	5	23.3
CT-CLIP	47.1		1.2	5.6	0	22.8	0	38.9	52.8	26.9
OpenVocabCT	67.1		78.1	77.5	66.2	31.6	43.0	62.8	63.3	68.8

TABLE VII.

Generalizability to training invisible text prompts: synonyms. Parentheses represent the ground truth class. BOLD indicates best result. Underline second best result.

Method	Renal organs (kidney)	Hepatic system (liver)	Heart muscle (myocardium)	Aortic vessel (aorta)	Cerebrum (brain)	Small intestine (small bowel)	Avg
CLIP-Driven	54.5	0	37.4	42.0	78.5	24.3	39.4
SAT Pro	0	74.7	76.5	83.6	0	76.2	51.8
CT-CLIP	23.6	45.7	65.1	57.9	72.5	19.3	47.4
OpenVocabCT	77.1	84.5	69.0	71.1	79.8	79.9	76.9

Our results also reveal that using descriptive prompts merging suborgans leads to better performance than using synonym prompts. For example, both prompt ‘left & right kidney’ and ‘renal organs’ refer to the same anatomy. However, OpenVocabCT attains 86.1% DSC on former prompt and 78.4% DSC on the latter one. Similarly, it reaches 95.2% DSC for liver yet 88.7% DSC for hepatic system. We attribute this performance gap primarily to tokenization differences: “Kidney” is a in-vocabulary token with high occurrence during both pretraining and fine-tuning. In contrast, “renal organs” is an open-vocabulary prompt and never appears in the fine-tuning captions, so its embedding is less precise and semantically diffuse. Nonetheless, even on these more challenging generic prompts, OpenVocabCT still outperforms SAT Pro by 6.9% DSC on kidney and 7.4% DSC on liver, confirming that our approach is robust across both specific and synonymous clinical terms.

We visualize the segmentation results of generalization study (Figure 4). As shown in Figure 4 rows 1, 3, and 4, our model performs reasonably well when merging left and right organs in both the chest region (lung) and the abdominal region (kidney, autochthon). In row 2, the model demonstrates its flexibility by accurately segmenting organs described using synonyms, such as ”hepatic system” for the liver and ”renal organs” for the kidneys. Additionally, for bones and vertebrae, our method effectively segments merged categories like left ribs, right ribs, lumbar vertebrae, and thoracic vertebrae. This demonstrates the superior ability of our model to handle diverse and unseen clinical terminology as text prompts, suitable for real-world deployment in diverse clinical environments.

Fig. 4.

Segmenting organs with training-unseen prompts (axial view). Each segmentation model is evaluated under training unseen prompts, as depicted in the corresponding color-coded legend.

E. Ablation Study

1). Ablation study on pretraining:

We conduct an ablation study on the effectiveness of our proposed pretraining strategy, shown in Table VIII. Compared with a baseline model using random initialization, CLIP pretraining does not improve the finetuning performance, which corroborates our hypothesis that image-text alignment may not benefit a dense segmentation task. Incorporating our proposed MGCL on average improves the finetuning performance by 1.2% DSC and the generazaibility performance by 6.8% DSC and 12.4% DSC. We also investigate adding fine-grained vision alignment into our MGCL pipeline. Specifically, we obtain 37 paired segmentation masks and captions from RadGenome [52] dataset. For the image encoder, we extract the latent embedding from the last convolution layer, and then use the downsampled segmentation mask to locate the foreground position and perform global average pooling. This fine-grained visual feature is used to align with organ captions. Surprisingly, we found that this reduces segmentation performance across all metrics, possibly due to lack of global visual features. We find that the optimal performance emerges when initializing the image encoder with pretrained weights and locking the image encoder, considerably improving finetuning score by 8.5% DSC and generalizability scores by 7.7% DSC for merging suborgans.

TABLE VIII.

Ablation Study on Pretraining Strategies and text-branch connector. Results are reported in DSC (%).

Pretraining Strategy	Segmentation text-branch	Finetuning	Generalizability
			Merging	Synonyms
Random Init	MLP	83.4	-	-
CLIP Loss	MLP	81.0	54.3	60.7
Multi-Granularity Contrastive Loss	MLP	82.2	61.1	73.1
Fine-grained vision alignment + Multi-Granularity Contrastive Loss	MLP	79.7	51.5	55.8
Multi-Granularity Contrastive Loss	Cross Attention	85.6	43.4	49.4
Pretrained Image Encoder + Multi-Granularity Contrastive Loss	MLP	90.7	68.8	73.2

2). Ablation study on text-branch connector:

We also conduct an ablation study on 2 architectures of the text-branch connector: Multi-Layer Perceptron (MLP) vs Cross-Attention mechanism in Table VIII. Intriguingly, we find that the MLP connector achieves strong generalizability (61.1% DSC for merging and 73.1% DSC for synonyms) by efficiently aligning image and text without overfitting. In contrast, while cross-attention can result in a higher finetuning score of 85.6% DSC, it results in significantly lower generalizability, likely due to its reliance on specific text features. To mitigate this trade-off, our approach leveraging pretrained image encoder can provide a good initialization weight for our image-text alignment, achieving the best average performance, balancing the finetuing and generalizabitlity to diverse text prompts.

F. Generalization to external datasets

We further study the generalization capabilities to external datasets. Table IX summarizes the performance on FLARE22 dataset. Compared to the other methods, our method consistently achieves superior generalization performances in 10 of 13 organs in the abdominal region. The second best performing model is the vision-only model nnUNet, which our method slightly outperforms by 0.4% DSC. Generalization performance for SegTHOR dataset is shown in Table X. Our method also achieves superior performance in esophagus, heart and aorta segmentation. For inferring the training unseen heart category, we prompt text-driven models with the prompt heart. Our method outperforms best existing text-driven models in heart segmentation by 9.4% DSC and 6% DSC on average. To further explore the generalization capability of nnUNet on the unseen heart category, we combined predictions for its sub-organ components (i.e., heart myocardium, heart ventricle, heart atrium, and pulmonary artery). Interestingly, nnUNet demonstrates comparable performance on this cardaic organ category, achieving results superior to most existing text-driven models (except ours). We hypothesize that this is because text-driven models may suffer from insufficient image-text alignment during training, limiting their ability to generalize effectively to unseen categories (heart in this case).

TABLE IX.

External generalization study on the Flare22 dataset. All models are trained only on TotalSegmentator for fair comparison. Results are reported in DSC (%). BOLD indicates best result. Underline second best result.

Method	Liver	RK	LK	LAG	RAG	Spleen	Pancreas	Gallbladder	Esophagus	Stomach	Duodenum	IVC	Aorta	Avg
CLIP-Driven	95.0	87.9	88.3	83.1	83.7	88.1	79.3	87.8	84.5	93.5	66.7	91.4	90.6	86.2
SAT-Pro	97.9	91.1	90.9	84.4	81.6	95.1	81.4	90.1	85.1	95.3	73.8	93.7	93.5	88.8
CT-CLIP	86.7	76.2	75.7	71.0	71.5	83.3	74.7	78.7	75.0	89.8	67.3	82.3	81.2	78.0
nnUNet	97.5	92.5	93.1	85.0	84.8	98.0	83.6	90.0	82.3	95.9	76.6	94.1	95.1	89.9
OpenVocabCT	97.6	94.3	93.3	86.0	84.9	98.1	83.9	90.1	84.2	96.1	75.4	94.1	95.7	90.3

TABLE X.

External generalization study for the SegTHOR dataset. All models are trained only on TotalSegmentator for fair comparison. Heart* is training unseen organ. Results are reported in DSC (%). BOLD indicates best result. Underline second best result.

Method	Esophagus	Heart*	Trachea	Aorta	Avg
CLIP-Driven	72.5	70.0	79.5	69.4	72.9
SAT-Pro	76.8	73.4	87.9	78.3	79.1
CT-CLIP	65.9	68.3	65.1	52.1	62.9
nnUNet	83.8	78.6	91.3	82.4	84.0
OpenVocabCT	85.8	82.8	88.9	82.9	85.1

G. Transferring to clinical gynecological segmentation

We further evaluate the clinical applicability of OpenVocabCT on real-clinical data. We perform finetuning evaluation on an institutional GYN dataset with 15 gynecologic brachytherapy patients. All CT scans were acquired using a SOMATOM go.Open Pro scanner (Siemens Healthineers). The images were reconstructed with a slice thickness of 1 mm, and the axial resolution was standardized to 1 mm × 1 mm pixel spacing. Three OARs (rectum, sigmoid and uterus) were manually contoured by an experienced radiation oncologist and subsequently reviewed by a senior radiation oncologist for consistency. We conduct finetuning by using models finetuned on Totalsegmentator, using 5-fold cross validation. As shown in Table XI, our OpenVocabCT still demonstrates strong finetuning performance on GYN segmentation, outperforming nnUNet. Paired t-test confirm that the improvements over nnUNet are statistically significant (p < 0.05) for average DSC and average NSD.

TABLE XI.

Finetuning evaluation on GYN dataset. Results reported in DSC (%) and NSD (%). BOLD indicates best result, Underline second best.

Method	Rectum		Sigmoid		Uterus		Avg
	DSC	NSD	DSC	NSD	DSC	NSD	DSC	NSD
UniMiSS	93.8	91.3	58.5	37.0	64.8	31.1	72.4	53.1
SAT Pro	95.0	91.9	65.3	44.4	68.8	32.7	76.3	56.3
nnUNet	96.0	92.7	69.5	48.4	76.7	46.0	80.7	62.4
OpenVocabCT	96.4	93.4	72.5	56.3	85.2	58.4	84.7	69.3

V. Conclusion

In this paper, we propose a novel framework for pretraining and adapting vision-language models for universal text-driven CT image segmentation. Our approach introduces a multi-granular contrastive learning loss that effectively captures organ- and disease-specific information extracted from radiology reports. To ensure high-quality caption selection, we leverage a radiology corpus for generating informative and relevant text descriptions. We first show that our method achieves superior results on organ and lesion segmentation compared to both vision and vision-language models. We also show that our method can successfully generalize to training-unseen text prompts for universal organ segmentation, outperforming other methods.

While our evaluation includes 14 public benchmarks and one institutional dataset spanning thoracic, abdominal, pelvic, and head-and-neck CT, we acknowledge that this still does not cover the full diversity of clinical imaging scenarios. Certain domains, such as cardiovascular, musculoskeletal CTs remain underrepresented. In addition, our pretraining dataset only includes non-contrast chest CT for image-text alignment and TotalSegmentator for segmentation understanding. Our future work will expand the pretraining on larger-scale image-text datasets and include diverse segmentation targets (e.g. tumor) for pretraining. Our clinical evaluation can also be expanded to multi-center datasets and include underrepresented anatomical regions to further validate robustness and fairness. For LLM-caption validation, a clinical expert independently reviewed a random subset of 100 patient-level captions, scoring anatomical correctness, clinical relevance, and linguistic clarity on a 1–5 scale. Overall, 92% of captions received a score ≥ 4, indicating that the generated organ-level descriptions were accurate and clinically meaningful. As demonstrated in our ablation studies, incorporating these curated captions improved segmentation performance by 1.2% on fine-tuning tasks and enhanced generalization by 6.8% on merged prompts and 12.4% on synonym-based prompts. These findings show that the LLM-curated, RadLex-filtered captions provide diverse, clinically valid supervision that strengthens both representation quality and prompt robustness.

In our future work, we aim to extend this framework for other imaging modalities (such as PET and MRI) and also explore pretraining with more diverse CT image sites (such as abdominal, head and neck regions). We also plan to transfer our framework to real-world clinical data and demonstrate its effectiveness in other tasks such as tumor detection and image synthesis.

Fig. 2.

Overall workflow for OpenVocabCT. (a) We first curate granular organ-level captions from CT-RATE’s radiology report using LLMs with few-shot examples. The LLMs break down long radiology findings into organ-level captions, which are further filtered via substring matching to our metadata. (b) We pretrain our vision language model using a multi-granularity contrastive loss. Each CT image is paired with multiple granular captions and the original report to enhance text representation learning. (c) We finetune the vision language model on CT segmentation datasets with text prompts for each organ.

Fig. 3.

Visualization of tumor segmentation on MSD Pancreas, MSD Colon and KiTS. Each segmentation model is evaluated under training seen prompts.