Generalist foundation models from a multimodal dataset for 3D computed tomography

Abstract

Advancements in medical imaging AI, particularly in 3D imaging, have been limited due to the scarcity of comprehensive datasets. We introduce CT-RATE, a public dataset that pairs 3D medical images with corresponding textual reports. CT-RATE comprises 25,692 non-contrast 3D chest CT scans from 21,304 unique patients. Each scan is accompanied by its corresponding radiology report. Leveraging CT-RATE, we develop CT-CLIP, a CT-focused contrastive language–image pretraining framework designed for broad applications without the need for task-specific training. We demonstrate how CT-CLIP can be used in multi-abnormality detection and case retrieval, and outperforms state-of-the-art fully supervised models across all key metrics. By combining CT-CLIP’s vision encoder with a pretrained large language model, we create CT-CHAT, a vision–language foundational chat model for 3D chest CT volumes. Fine-tuned on over 2.7 million question–answer pairs derived from the CT-RATE dataset, CT-CHAT underscores the necessity for specialized methods in 3D medical imaging. Collectively, the open-source release of CT-RATE, CT-CLIP and CT-CHAT not only addresses critical challenges in 3D medical imaging but also lays the groundwork for future innovations in medical AI and improved patient care.

The CT-RATE dataset pairs 3D chest computed tomography volumes with radiology reports to train a foundation model for medical imaging.

Main

Three-dimensional (3D) medical imaging, such as computed tomography (CT) and magnetic resonance imaging (MRI), provides a more detailed and comprehensive view of a patient’s health status compared with two-dimensional (2D) imaging¹. Among these medical imaging modalities, CT stands out due to its speed and widespread availability, making it the preferred modality for diagnosing various abnormalities². As global demand for CT scans continues to grow, the need for rapid and accurate diagnostic results becomes more pressing. However, the manual generation of reports from these complex images is time consuming and prone to errors, underscoring the urgent need for automation³. Artificial intelligence (AI) has the potential to address this challenge by efficiently identifying pathologies, reducing the workload on radiologists and potentially saving lives in situations where immediate radiological expertise is not available.

AI has already begun to revolutionize the interpretation of medical images^4–8. Recent advances in automated diagnosis, classification and segmentation have been driven by numerous publicly available datasets^9–13. Beyond supervised methods, general AI systems have been developed using open-source image–text pairs in self-supervised learning, particularly in 2D imaging¹⁴. These versatile AI foundation models integrate comprehensive natural language annotations into the learning process without requiring task-specific training, thereby eliminating the need for explicit manual image annotations. Unconstrained by predefined classes or tasks during training, they enhance adaptability and have been successfully applied to tasks such as disease classification¹⁵, case retrieval¹⁶ and interactive diagnostics via chat interfaces¹⁴. Despite these advancements in 2D imaging, the application of advanced AI techniques to 3D imaging remains underexplored. A major challenge is the lack of open-source 3D medical imaging datasets paired with textual reports, which require extensive computational resources, specialized knowledge and technical expertise to create^17–19. Currently, the largest 3D medical imaging datasets available contain only a few thousand cases and do not include paired textual data^20–22, limiting the development of advanced AI models. Moreover, 3D models face higher computational demands and have fewer available pretrained open-source models compared with their 2D counterparts^23,24. Previous studies have only begun to address these challenges; for example, one involving chest CT is limited by its supervised approach, restricting detection to predefined abnormalities and offering only a small portion of its dataset publicly without paired textual reports²⁰.

To address these challenges, we introduce CT-RATE, a dataset that pairs 50,188 reconstructed 3D chest CT volumes (over 14.3 million 2D slices) from 25,692 scans of 21,304 patients with corresponding radiology reports. Each volume is annotated with 18 abnormalities, supporting extensive validation and fine-tuning. CT-RATE is fully open source. Leveraging CT-RATE, we develop CT-CLIP, a chest CT-focused contrastive language–image pretraining framework. CT-CLIP integrates extensive natural language annotations into the learning process, enabling the model to comprehend image-based semantic knowledge and perform various tasks without specific training. Our evaluation shows CT-CLIP’s ability to detect multiple abnormalities in a zero-shot setting²⁵, outperforming state-of-the-art fully supervised methods on three test sets from two different countries. We extend CT-CLIP’s capabilities by combining its vision encoder with a pretrained large language model to create CT-CHAT, a vision–language chat model for 3D chest CT volumes. CT-CHAT is fine-tuned with over 2.7 million question–answer pairs derived from the CT-RATE dataset. CT-CHAT outperforms open-source conversational vision–language AI assistants, such as LLaVA 1.6 (ref. ²⁶), LLaVA-Med²⁷ and CXR-LLaVA²⁸, underscoring the need for specialized tools in 3D medical imaging. We also benchmark CT-CHAT against state-of-the-art 3D medical report-generation methods, including RadFM²⁹ and CT2Rep³⁰, using clinically meaningful accuracy metrics. On two independent test sets, CT-CHAT outperforms these baselines by a significant margin, demonstrating both its clinical relevance and technical efficacy in report generation. In addition to quantitative benchmarks, we conduct expert evaluations with three board-certified radiologists, who consistently rated CT-CHAT’s reports as superior to those from RadFM and CT2Rep. While these results are encouraging, the experts emphasized that substantial progress is still needed for clinical deployment in 3D medical imaging. Our work marks a first step toward that goal, but important challenges remain.

Results

Pairing CT volumes with radiology text reports

We curate and open source CT-RATE, a dataset comprising 25,692 non-contrast 3D chest CT volumes and corresponding radiology text reports from 21,304 unique patients (Fig. 1a). These 3D volumes, reconstructed using various methods to suit different window settings³¹, total over 14.3 million 2D slices. Each CT exam is accompanied by a report that includes an ‘impression’ and a ‘findings’ section, authored by a radiologist, along with additional sections detailing patient information and scan technique. CT-RATE is divided into training and internal validation sets for experiments (see ‘Creating the CT-RATE dataset’ in Methods). While CT-RATE is a single-centre dataset, it captures substantial heterogeneity in CT scanners, acquisition protocols and reconstruction techniques.

Fig. 1. Overview of our dataset and CT-CLIP.

a, Data acquisition protocol for the CT-RATE dataset, comprising non-contrast 3D chest CT volumes and corresponding radiology text reports written by radiologists. b, Extraction of multi-abnormality labels from radiology reports using a fine-tuned text encoder. c, Training of our CT-CLIP framework, which pairs 3D chest CT volumes with radiology reports from the CT-RATE dataset through a contrastive learning approach. d, Illustration of the contrastive learning procedure, employed in the correlation matrix during training to ensure precise match recognition and mismatch differentiation. V_i shows volume embeddings and R_i shows report embeddings. e, Inference for zero-shot multi-abnormality detection uses normalized probabilities derived from softmax-applied contrastive loss for each positive and negative prompt.

To evaluate the generalization of our models (developed on CT-RATE) under distribution shifts, we incorporate two external validation sets from independent studies conducted in different countries^20,32 (see ‘Adding external validation datasets to CT-RATE’ in Methods). First, we use the public RAD-ChestCT dataset, which includes multi-abnormality labels, to assess model generalizability. Second, we perform real-world validation by inviting an independent clinical research team (uninvolved in model development) to download our open-source codebase, pretrained models and user interface from our public repository and apply them to their own local, private dataset without any adaptation. The performance scores they obtained are presented in the following sections. In particular, none of the external validation data was used during development; all model development was conducted solely on CT-RATE.

Conventional supervised machine learning approaches for medical image classification necessitate manual annotation of large datasets, a process that requires significant time and expertise¹⁹. Although our self-supervised learning approach does not require abnormality labels, the baseline supervised model and fine-tuning methods do. To meet this need, we enrich CT-RATE with global abnormality labels extracted from the radiology reports using an automated text classifier (Fig. 1b). This process involves fine-tuning a pretrained language model, RadBERT³³, similar to recent work³⁴. Within CT-RATE, we manually annotate 1,000 reports for 18 distinct abnormalities for this fine-tuning procedure. The fine-tuned model’s precision, recall and F1 scores, detailed in Supplementary Table 2, show robust classification across abnormalities, ensuring the CT-RATE dataset is enriched with high-quality labels.

Figure 2 summarizes CT-RATE alongside the open-source external validation set, RAD-ChestCT. CT-RATE spans a wide age range (18–102 years) (Fig. 2a) and includes diverse pathologies primarily related to the heart and lung (Fig. 2b). The reports vary in length and are authored by numerous radiologists, reflecting a variety of writing styles (Fig. 2c). CT studies in CT-RATE also vary in slice numbers (Fig. 2d), slice thickness and axial pixel (XY) spacing (Fig. 2e), and include different manufacturers and a balanced sex distribution (Fig. 2f). This heterogeneity is crucial for developing robust models capable of handling out-of-distribution cases. Supplementary Table 1 provides detailed statistics on abnormalities, manufacturers and sex distributions in the training and internal validation sets. In conclusion, CT-RATE does not represent a curated study dataset but rather reflects real-world scenarios characterized by heterogeneity. This diversity ensures that the models trained on CT-RATE are well suited for deployment in clinical settings, offering reliable performance across a broad range of conditions.

Fig. 2. Comprehensive analysis of the novel CT-RATE dataset.

a, CT-RATE comprises data from 21,304 unique patients, with ages ranging from 18 to 102 years. b, For validation and fine-tuning purposes, multi-abnormality labels for 18 distinct abnormalities are extracted from the corresponding radiology reports for each CT volume. c, The training process of the CT-CLIP model uses the ‘impression’ and ‘findings’ sections of radiology reports, which vary in length, along with the chest CT volumes. d, The number of slices in the chest CT volumes within CT-RATE ranges from 100 to 600. e, Spacings in the x, y and z axes vary across volumes; all chest CT volumes are preprocessed to ensure consistent spacing before training each model. f, Despite originating from a single institution, CT-RATE includes volumes acquired using scanners from three different manufacturers, ensuring variability.

Source data

CT-CLIP: developing a visual–language foundation model with CT-RATE

Unlike other methods using supervised machine learning for multi-abnormality classification of CT volumes, our study employs radiology reports, which provide a rich source of semantic knowledge. This significantly enhances the understanding of CT volumes and supports a wide range of potential zero-shot applications. Accordingly, we develop CT-CLIP, a foundational model for chest CT-focused contrastive language–image pretraining, adapting a technique from a previous study³⁵. We use both the impression and findings sections of radiology reports combined for CT-CLIP due to their superior performance compared with using either one alone, as demonstrated by the abnormality classification scores in an ablation study (Supplementary Fig. 1a). While the best performance is achieved by training on both impression and findings together, using only the findings section performs better than using only the impression section, indicating that the findings section contains more clinically informative content.

CT-CLIP comprises a vision transformer and a text transformer that generate two distinct embedding vectors: one for chest CT volumes and another for reports, respectively (see ‘Developing the CT-CLIP model’ in Methods). The vision transformer is adapted from CT-ViT²³, incorporating a two-stage spatial and causal transformer, enabling 3D attention across volumes. CT-CLIP is trained to maximize the similarity between embedding vectors for paired volume–text data samples, while minimizing similarity for unpaired data. Our training approach leverages contrastive learning³⁶ (Fig. 1c,d), which enables the training of a foundation model suitable for various applications without task-specific training, including multi-abnormality detection, volume-to-volume retrieval and report-to-volume retrieval. Subsequent sections demonstrate how CT-CLIP can be applied to these downstream tasks without additional training. To further enhance CT-CLIP’s capabilities, we explore fine-tuning methods for detection to achieve greater accuracy and adapt CT-CLIP to create a vision–language AI assistant.

We analyse the latent spaces of reports and chest CT volumes generated by CT-CLIP, by computing t-distributed stochastic neighbour embedding (t-SNE) projections of these spaces. Figure 4d shows the distribution according to the number of abnormalities present in a case. Figure 4e depicts the age distribution on the t-SNE projections. The well-clustered t-SNE projections confirm CT-CLIP’s ability to accurately map latent spaces with closely aligned visual and linguistic information. For instance, in both image and text latents, the alignment between age and abnormality distributions indicates that older patients are more prone to abnormalities, which the model effectively identifies. Extended Data Figs. 1 and 2 provide additional t-SNE projections for each abnormality using volume and report embeddings, respectively.

Extended Data Fig. 1. Abnormality-based t-SNE projections for chest CT volumes and radiology text reports.

Two-dimensional t-SNE visualizations showing the joint embedding space of chest CT volumes and their associated radiology text reports, color-coded by abnormality categories.

Extended Data Fig. 2. Abnormality-based t-SNE projections for chest CT volumes and radiology text reports (continued).

Additional t-SNE projections illustrating the joint embedding space of chest CT volumes and radiology text reports across further abnormality categories.

Surpassing supervised levels with CT-CLIP’s zero-shot multi-abnormality detection

CT-CLIP adopts a zero-shot classification approach for multi-abnormality detection without explicit training on specific abnormality labels. Zero-shot classification refers to the ability of a model to categorize previously unseen classes without being explicitly trained on those specific categories³⁵. Instead of relying on predefined labels, CT-CLIP leverages the semantic information embedded in paired radiology reports to infer multiple abnormalities. For instance, a model trained to recognize features such as ‘reticulation’, ‘consolidation’ or ‘nodules’ in common lung diseases could potentially identify a rare lung condition it was not explicitly trained on by recognizing combinations of these features.

Zero-shot classification with foundation models is typically applied as a multiclass classification task, assigning a single label to each image³⁵. However, a medical image may present multiple abnormalities, necessitating a unique approach. Inspired by recent work on 2D chest radiographs¹⁵, we generate positive and negative prompts for each abnormality. The embeddings for the input 3D chest CT volume and the two prompts are projected into a shared latent space, and the contrastive loss between these projections is calculated on the basis of the distance between the image and language spaces (see ‘CT-CLIP for zero-shot multi-abnormality detection’ in Methods for more details). This process extracts and identifies the relevant features in both modalities. The model then outputs logits for each prompt, which are interpreted as the likelihood of abnormality presence, as shown in Fig. 1e. We conduct a systematic evaluation of CT-CLIP’s zero-shot detection capabilities on our internal validation set from CT-RATE and two external validation sets from RAD-ChestCT and UPMC (Fig. 4a).

The careful engineering of prompts is instrumental in zero-shot classification tasks with vision–language models, as it significantly impacts performance³⁷. Therefore, we evaluate seven distinct prompts commonly used in our reports, with detailed information provided in the Methods section. Figure 4b presents the mean AUROC, F1 score, accuracy and precision for these various prompts. On the basis of its consistently superior performance, we use prompt 7 (‘{Abnormality} is {$\varnothing$/not} present.’) in further experiments (see ‘CT-CLIP for zero-shot multi-abnormality detection’ in Methods).

To evaluate CT-CLIP’s zero-shot multi-abnormality detection, we benchmark it against CT-Net²⁰, a state-of-the-art fully supervised model for 3D chest CT volumes. CT-Net requires manual annotation for each predefined abnormality, which demands significant time and expertise¹⁹. We choose CT-Net as a baseline to demonstrate that CT-CLIP can surpass traditional fully supervised performance without relying on manual annotations or being restricted to predefined classes. Since the pretrained CT-Net model is not available and only a small portion of its dataset is open source, we train CT-Net on CT-RATE using our abnormality labels for a fair comparison (see ‘Benchmarking the CT-CLIP models for multi-abnormality detection’ in Methods). Despite using a different dataset for training, the retrained CT-Net’s classification scores are comparable to those reported in the original study²⁰. Figure 4a shows the retrained CT-Net’s performance metrics, while Extended Data Figs. 3–5 and Supplementary Figs. 2 and 6 detail classification scores and confusion matrices.

Extended Data Fig. 3. Comparison of abnormality-based performance metrics in the internal validation set.

This figure provides a detailed analysis of performance metrics, including AUROC, accuracy, precision, and F1 scores, for detecting various abnormalities with our models in the internal validation set, compared to the fully supervised baseline model. The proposed CT-CLIP based models demonstrate improvement over the baseline in almost all comparisons.

Source data

Extended Data Fig. 5. Comparison of abnormality-based performance metrics in the UPMC validation set.

This figure provides a detailed analysis of performance metrics, including AUROC, accuracy, precision, and F1 scores, for detecting various abnormalities with our models in the UPMC validation set, compared to the fully supervised baseline model. It highlights the models’ remarkable adaptability and superior effectiveness with distribution shifts, setting a new standard in performance compared to a fully supervised baseline model.

Source data

We compare our zero-shot approach to this supervised baseline and observe significantly higher performance metrics for our approach. On the internal validation set from CT-RATE, the zero-shot CT-CLIP model achieves a mean AUROC that is 0.102 higher and a mean F1 score that is 0.050 higher than the supervised baseline. On the first external validation set, RAD-ChestCT, CT-CLIP outperforms the baseline with a mean AUROC gain of 0.085 and an F1 score improvement of 0.073. Similarly, on the second external validation set from UPMC, CT-CLIP maintains strong performance with a 0.087 increase in mean AUROC and a 0.037 boost in F1 score. Results on the external validation sets demonstrate that the zero-shot CT-CLIP model performs well not only on similar data distributions but also under distribution shifts, indicating its strong generalization to external datasets. A two-sided unpaired permutation test using 10,000 permutations highlights a statistically significant improvement (P < 0.05) in AUROC as a zero-shot classifier over the supervised (baseline) model (detailed in Supplementary Table 5).

Extended Data Figs. 3–5 provide scores for all abnormalities on the CT-RATE internal, RAD-ChestCT external and UPMC external validation sets, respectively. Supplementary Figs. 3 and 7 show confusion matrices on the CT-RATE internal and RAD-ChestCT external validation sets, respectively. Notably, on the internal validation set, all abnormalities (18 out of 18) exhibit higher scores in our zero-shot approach compared with the supervised baseline in mean AUROC. Surprisingly, CT-CLIP excels by just leveraging existing radiology text reports in electronic health records, thus avoiding the need for expensive manual annotations. Moreover, CT-CLIP’s zero-shot capability offers flexible predictions beyond predefined classes and adapts efficiently to changing needs post training.

Since the availability of sufficiently large datasets is essential for machine learning, especially for transformers, which require more data than convolutional neural networks^38,39, we explore the impact of dataset size on classification performance. We train CT-CLIP on different portions of the CT-RATE dataset, specifically 9.8% (equivalent to the open-source portion of the RAD-ChestCT dataset²⁰, currently the largest publicly available chest CT dataset), 20%, 40%, 60%, 80% and 100%. Figure 4c reports the classification performance across different dataset sizes. Notably, a consistent trend emerges: larger dataset sizes correlate with improved scores. This finding highlights the role of dataset size in enhancing CT-CLIP’s zero-shot classification performance and underscores the value of our extensive CT-RATE dataset for computational research. It also offers a clear avenue for future improvements, suggesting that further expanding the dataset or incorporating more diverse data could lead to even better performance.

Enhancing CT-CLIP’s detection performance with fine-tuning

To enhance CT-CLIP’s detection performance, we explore a task-specific fine-tuning method, ClassFine, inspired by conventional linear probing, by adding a new classification layer to the pretrained vision transformer (Fig. 3a). Supplementary Fig. 1b indicates that training only the final layer may be sufficient for fine-tuning (see ‘Fine-tuning CT-CLIP for multi-abnormality detection’ in Methods). While the ClassFine approach improves detection performance (Fig. 4a), it restricts the model to predefined labels, similar to any supervised classification method. Extended Data Figs. 3–5 present abnormality scores, with confusion matrices in Supplementary Figs. 5 and 9 on the internal and external validation sets, respectively. To address the limitations of ClassFine, we develop the VocabFine approach, a novel open-vocabulary fine-tuning method inspired by WISE-FT⁴⁰ (see ‘Fine-tuning CT-CLIP for multi-abnormality detection’ in Methods). VocabFine, similar to CT-CLIP’s zero-shot inference, is fine-tuned to improve label prediction accuracy (Fig. 3c). Specifically, we use positive and negative prompts for abnormalities, calculate the logits with pretrained CT-CLIP and align these logits with the ground-truth labels. While the fine-tuned model with VocabFine retains the open-vocabulary inference capability of CT-CLIP (Fig. 3d), it also consistently outperforms ClassFine (Fig. 4a), further emphasizing the value of our open-vocabulary fine-tuning. Abnormality-specific improvements are detailed in Extended Data Figs. 3–5, while Supplementary Fig. 1b underscores the importance of training all weights in fine-tuning. Supplementary Figs. 4 and 8 provide confusion matrices on the internal and external validation sets, respectively. Notably, both ClassFine and VocabFine are trained solely on CT-RATE; thus, their generalization to external sets (RAD-ChestCT and UPMC) demonstrates the robustness of CT-CLIP’s pretrained representations under distribution shifts. This performance advantage across multiple test sets supports the effectiveness of our fine-tuning strategies. However, despite these promising results, the abnormality detection performance still falls short of clinical deployment thresholds.

Fig. 3. Fine-tuning CT-CLIP and the validation strategy.

a, Illustration of linear probing fine-tuning method (ClassFine) for CT-CLIP, where a linear layer is incorporated into the vision encoder. b, ClassFine enables multi-abnormality classification but is limited to the classes predefined during fine-tuning. c, Illustration of CT-CLIP’s open-vocabulary fine-tuning method (VocabFine) for each abnormality. d, VocabFine allows for open-vocabulary abnormality classification even after fine-tuning, although it is constrained to the prompts provided during fine-tuning. ${\mathcal{L}}_{\mathrm{BCE}}$, binary cross-entropy loss. e, Validation: models are trained on the CT-RATE dataset from Turkey and then tested on an internal CT-RATE validation set and two external datasets from the United States; a detailed analysis of the CT-RATE dataset is presented in Fig. 2. f, Comparison: a comprehensive evaluation is performed in multi-abnormality detection across three different cohorts, evaluating CT-CLIP, the two fine-tuned models and a fully supervised method.

Fig. 4. Detailed evaluation of CT-CLIP.

a, Multi-abnormality classification performance is evaluated on an internal and two external validation sets using four key metrics. Our CT-CLIP outperforms the fully supervised baseline across all metrics. Performance improves with both fine-tuning techniques, demonstrating the robustness of the CT-CLIP’s encoder in feature extraction from CT volumes. For F1, a weighted score is computed per abnormality and then averaged across all findings to ensure balanced evaluation. The scores are colour highlighted to emphasize the performance increase compared to random benchmarks. b, Different prompts are used for zero-shot multi-abnormality detection with CT-CLIP. The best-performing prompt, highlighting the importance of prompt engineering, is selected for further experiments. Data are presented as mean ± s.d. of the bootstrapped values. c, To explore how the performance scales with dataset size, we train CT-CLIP using different portions of CT-RATE (9.8% of CT-RATE matches the size of the only open-source CT dataset with multi-abnormality labels, as indicated by single asterisk). Contour lines denote ±1 s.d. calculated via bootstrapping, with the centre line indicating the mean. d, t-SNE projections map the distribution of the number of abnormalities present in cases, using the latent values from individual reports and CT volumes. e, t-SNE projections illustrate the age distribution of patients. The clustering shows that CT volumes and reports from older patients tend to correspond with a higher number of abnormalities, indicating a correlation between age and disease prevalence.

Source data

Beyond its superior performance, VocabFine retains the open-vocabulary capabilities of the original CT-CLIP model, enabling inference on abnormalities that were not explicitly included during fine-tuning. This makes the model inherently more flexible and clinically useful in real-world scenarios where emerging or rare findings may need to be recognized without retraining. In addition, unlike ClassFine, which only adds a classification head on top of the vision encoder and is therefore restricted to vision-only downstream tasks, VocabFine preserves and fine-tunes both the vision and text encoders. This dual-modality design allows it to generalize to a wider range of tasks, such as text-to-image retrieval and multimodal question answering, which rely on aligned visual and linguistic representations. In summary, our VocabFine approach improves multi-abnormality detection accuracy and also maintains the foundational model’s versatility with adaptability across diverse downstream applications.

Retrieving relevant cases with CT-CLIP

In clinical practice, radiologists frequently refer to past cases with similar findings to guide diagnosis and treatment decisions. This process, known as case retrieval, enhances diagnostic accuracy and builds confidence in decision making. However, manually searching for comparable cases is time consuming and inefficient. An automated, robust solution can streamline case retrieval, saving both time and effort.

Beyond zero-shot detection, CT-CLIP serves as a powerful tool for image-to-image retrieval⁴¹, identifying the relevant volumes from a query volume by comparing cosine similarities between the embeddings (Fig. 5a). We test on both CT-RATE internal and RAD-ChestCT external validation sets using the vision transformer of CT-CLIP, fine-tuned models and the supervised baseline. We use the mean average precision at K (MAP@K) metric, where K represents the number of returned 3D chest CT images (see ‘Benchmarking the CT-CLIP models for CT volume retrieval’ in Methods). As K increases, MAP@K scores typically decrease due to less accurate matches. Results, as in Fig. 5c, show comparable performance among methods, with all significantly outperforming random retrieval. Although the external validation set from RAD-ChestCT has higher MAP@K scores, the fold increase over random retrieval is lower (three times on the external vs six times on the internal set), due to differing abnormality distributions. This suggests that examining fold change offers a more accurate comparison. Moreover, fine-tuning models does not significantly alter the vision transformer’s weights, as CT-CLIP and the fine-tuned models perform similarly across MAP@K metrics.

Fig. 5. Using CT-CLIP for retrieval of 3D chest CT volumes.

a, CT-CLIP can be used for volume-to-volume retrieval using the vision encoder within CT-CLIP. b, CT-CLIP can be used for report-to-volume retrieval using both the vision and text transformers within CT-CLIP. c, Comprehensive evaluation of volume-to-volume retrieval performance on internal and external validation sets, with scores colour highlighted to emphasize improvements over random scores, shows similar retrieval accuracy even for the fine-tuned models. d, The evaluation of report-to-volume retrieval is limited to the internal validation set due to the absence of paired radiology reports in the external validation set. Notably, fine-tuning CT-CLIP for multi-abnormality classification diminishes its retrieval performance. This reduction in performance is primarily observed in report-to-volume retrieval, indicating that VocabFine fine-tuning predominantly impacts the text transformer component of CT-CLIP.

CT-CLIP also functions as a text-to-image retrieval tool⁴², evaluating cosine similarities between target volume embeddings and query report embeddings (Fig. 5b). Given the limitations of the supervised model and the ClassFine approach in encoding text, we employ CT-CLIP and the VocabFine approach for our experiments. Due to the lack of reports in the RAD-ChestCT dataset, only the internal validation set is used for report-to-volume retrieval. We evaluate performance using Recall@K, where K represents the number of returned images, reflecting the percentage of correctly matched images within the top K returns (see ‘Benchmarking the CT-CLIP models for CT volume retrieval’ in Methods). While Recall@K scores increase with more returned volumes, considering fold change relative to random performance is more meaningful. Figure 5d shows that CT-CLIP performs significantly higher than the random baseline. However, the Recall@K scores for VocabFine are substantially lower than those of CT-CLIP, consistent with the expectation that fine-tuning a foundation model for a specific task might diminish its efficacy for other tasks. The reduced performance with VocabFine in report-to-volume retrieval further indicates significant modifications to the text transformer weights due to fine-tuning.

Developing a multimodal AI assistant with CT-RATE and CT-CLIP

Observing CT-CLIP’s remarkable capabilities for open-vocabulary inference, we build on this foundation to enable an interactive exchange between textual queries and diagnostic image information. To achieve this, we develop CT-CHAT, a multimodal generalist AI assistant specifically designed for 3D chest CT. CT-CHAT integrates both visual and language processing to perform various tasks, making it a versatile tool for clinical use. To train CT-CHAT, we create a visual question-answering (VQA) dataset, comprising over 2.7 million question–answer pairs derived from CT-RATE using Llama 3.1 7B⁴³ (see ‘Creating the dataset for CT-CHAT’ in Methods). This dataset consists of several components, including long-answer questions, short-answer questions, multiple-choice questions and report-generation tasks (Extended Data Fig. 8), reflecting the diverse types of inquiry encountered in clinical practice.

Extended Data Fig. 8. Representative examples from VQA.

This dataset includes various question types, such as long-answer (which encompass conversational, descriptive, free-response, and text-only questions), multiple-choice, short-answer, and report-generation questions.

CT-CHAT’s architecture leverages the strengths of CT-CLIP by using its pretrained vision transformer to encode 3D chest CT volumes. These encodings are then passed to a large language model (LLM) through a multimodal projector, as shown in Fig. 6a. We perform an ablation study evaluating CT-CHAT’s performance with 4 different state-of-the-art LLMs, as detailed in Extended Data Fig. 6. We select 70B Llama 3.1 (ref. ⁴³), outperforming others in key metrics, for the optimal configuration of CT-CHAT.

Fig. 6. Overview and detailed evaluation of CT-CHAT.

a, Architecture of CT-CHAT, a vision–language AI assistant for 3D chest CT volumes, trained by fine-tuning LoRA weights on a state-of-the-art large language model using CT tokens encoded by the 3D vision encoder of CT-CLIP. b, Illustration showing an example CT-CHAT output, highlighting its superior performance as a vision–language assistant specifically designed for 3D chest CT volumes compared with its 2D counterparts, with DRRs provided. c, The question-answering performance of various vision–language assistants is evaluated across five key metrics, with radar charts providing a detailed breakdown of evaluations for each of the four different visual question-answering tasks. d, Report-generation performance, evaluated by three board-certified radiologists and a violin plot of scores, confirms CT-CHAT’s clear superiority. e, Report-generation performance is evaluated using clinical-efficacy metrics, calculated by adapting CT-RATE’s automatic report labeller to assess clinical correctness across multiple findings on two test sets.

Source data

Extended Data Fig. 6. Comparative metrics for CT-CHAT and baselines.

This figure illustrates the performance of CT-CHAT and baselines across various tasks, demonstrating that CT-CHAT consistently outperforms the baselines, achieving the highest accuracy in every metric for each task.

We create CT-CHAT by fine-tuning the multimodal projector and low-rank adaptation (LoRA) weights⁴⁴ for the LLM (see ‘Developing the CT-CHAT model’ in Methods). Before this, we perform a medical concept alignment step, where the LLM is frozen, and the multimodal projector is trained solely to generate radiology reports. This step slightly enhances performance by aligning the multimodal projector with biomedical concepts (Supplementary Table 10b). The ablation study for the volume provision proves that CT-CHAT extracts relevant information from the given 3D volumes (Supplementary Table 10a). Although CT-CHAT shows the ability to retrieve information from the given image, it is also expected to hallucinate some pathologies when the input volume is not provided. Thus, we implement a guardrail after the ablation study, with the system prompt (Supplementary Table 9). Supplementary Fig. 11 provides an example of the guardrail in case the volume is not provided.

Finally, CT-CHAT generates contextually relevant and clinically accurate responses across a variety of tasks, from simple question-answering to the generation of detailed radiology reports as shown in Extended Data Fig. 9. The model is also capable of dynamic task switching, facilitated by special tokens that guide the LLM in generating the appropriate type of response based on the specific task. This flexibility makes CT-CHAT a powerful tool for enhancing radiologists’ workflows, reducing the time and effort required for interpretation, and improving diagnostic accuracy in 3D medical imaging.

Extended Data Fig. 9. Example outputs of CT-CHAT across different question types.

Representative CT-CHAT responses for (a) long-answer, (b) short-answer, (c) multiple-choice, and (d) report-generation questions, illustrating the model’s ability to handle diverse clinical query formats.

Surpassing other multimodal AI assistants with CT-CHAT

Given the absence of generalist models for 3D imaging, we evaluate CT-CHAT against several state-of-the-art open-source vision–language AI models, including LLaVA 1.6 (Mistral 7B and Vicuna 13B)²⁶, LLaVA-Med²⁷ and CXR-LLaVA²⁸. To ensure a fair comparison, these models, designed for 2D imaging tasks, are tested using digitally reconstructed radiographs (DRRs) of 3D chest CT volumes, a method that ensures accurate representation in a 2D context, as suggested by ref. ⁴⁵, rather than using random or central 2D slices of a 3D volume (see ‘Benchmarking the CT-CHAT model’ in Methods). In contrast, CT-CHAT directly processes 3D chest CT, giving it a significant advantage in understanding and interpreting the complex anatomical and pathological information within these scans.

CT-CHAT generates clinically accurate answers to questions about the given CT, as in Fig. 6b. To evaluate results, we employ multiple metrics, including BLEU, METEOR, ROUGE-L and CIDEr, which assess the quality of the generated text. Also, inspired by previous work²⁷, we use an LLM-based metric, Llama score, to evaluate the clinical accuracy of responses, with the models’ outputs compared against ground truths (see ‘Benchmarking the CT-CHAT model’ in Methods). The results indicate that CT-CHAT, being capable of evaluating the full information of 3D volumes, consistently outperforms the other open-source vision–language AI assistants across all tasks. It shows superior ability in generating long and short answers, handling multiple-choice questions and producing detailed radiology reports that align closely with clinical expectations, as shown in Fig. 6c. Extended Data Fig. 6 provides a detailed breakdown, demonstrating that CT-CHAT significantly outperforms others in all tasks. There are two key reasons why CT-CHAT significantly outperforms 2D-based models: (1) Image: CT-CHAT leverages the 3D spatial information processed by CT-CLIP’s pretrained vision transformer, which captures much richer and more detailed anatomical structures than 2D models—a critical advantage for CT interpretation. (2) Text: CT-CHAT is trained on 3D CT volumes paired with radiology-specific linguistic data from CT-RATE, enabling it to generate highly precise and clinically relevant responses, far surpassing those produced by models trained on general-purpose texts. This powerful combination of CT-CLIP’s capacity to handle complex 3D imaging data and the domain-specific paired image–text data from CT-RATE allows CT-CHAT to push the boundaries of multimodal tasks in 3D medical imaging. In summary, CT-CHAT not only surpasses existing 2D tools but also provides a scalable solution for enhancing chest CT interpretation, underscoring the need for generalist models in 3D medical imaging.

In the absence of existing vision–language baselines for 3D medical imaging, we extend our evaluation of CT-CHAT beyond comparisons with 2D multimodal AI assistants by benchmarking its report-generation capabilities against state-of-the-art 3D radiology report-generation models, specifically RadFM²⁹ and CT2Rep³⁰. This task is particularly important as it closely reflects the daily workflow of radiologists and provides a more clinically relevant benchmark than VQA. To robustly assess report-generation accuracy, we conducted expert evaluations with three board-certified radiologists who independently rated 150 reports for each model (CT-CHAT, RadFM and CT2Rep). To ensure a fair and comprehensive assessment, we did not select cases randomly; instead, we used the Llama score on CT-CHAT outputs to stratify cases into low (0–3), medium (4–-6) and high (7–10) clinical accuracy ranges, selecting 50 cases from each range. Each radiologist then scored the reports on a scale from 0 to 10 based on predefined clinical criteria. The mean expert scores demonstrate CT-CHAT’s clear superiority: Radiologist 1 rated CT-CHAT at 3.44 (vs RadFM 0.55, CT2Rep 2.66); Radiologist 2 at 4.10 (vs RadFM 0.91, CT2Rep 2.52); and Radiologist 3 at 4.52 (vs RadFM 0.86, CT2Rep 2.74). These results are presented in Fig. 6d.

To further strengthen report generation, we enhanced CT-CHAT with structured 3D segmentation inputs by providing detailed nodule information, such as lobar location, centroid, diameters and volumes, extracted using a state-of-the-art 3D nodule-segmentation algorithm on the CT-RATE training set. For implementation details on how we integrate nodule-segmentation information, see ‘Developing the CT-CHAT model’ in Methods. This additional structured input significantly improved report accuracy, as shown in Fig. 6e, demonstrating that leveraging high-quality 3D segmentation models meaningfully boosts clinical performance. Currently, we incorporate only nodule information; incorporating structured details for other abnormalities would probably enhance performance further, but robust 3D segmentation models for these additional findings are not yet available.

For quantitative evaluation, we exclude language generation metrics because they are limited in capturing clinical fidelity. Instead, we use clinical-efficacy metrics (precision, recall, F1 and CRG score) that better reflect diagnostic value. Labels are extracted with our automated report labeller, the same model used to annotate CT-RATE, and CT-CHAT is benchmarked using its 8B Meta Llama 3.1 configuration. Although the RAD-ChestCT dataset lacks text reports, its binary abnormality labels allow us to apply the text labeller and compute metrics consistently. CT-CHAT outperforms RadFM and CT2Rep on the CT-RATE internal and RAD-ChestCT external validation sets across all metrics, as shown in Fig. 6e. CT-CHAT’s superior performance over CT2Rep is particularly notable, given that both models are trained on CT-RATE. Unlike CT2Rep, which does not leverage pretraining, CT-CHAT benefits from CT-CLIP’s pretrained vision encoder, demonstrating the strength of our foundational approach. Our results show that integrating outputs from other advanced 3D algorithms, such as robust segmentation tools, can further improve tasks such as report generation. These findings highlight that continued development and integration of high-quality 3D models are essential for clinically deployable AI systems. By providing first-of-their-kind resources and benchmarks, especially through CT-RATE, our work takes an important step toward this goal, although significant challenges remain for routine clinical use.

Discussion

Radiology reports provide comprehensive descriptions of medical images without adding extra labelling burden, as they are already part of the daily routine for radiologists⁴⁶. Recent studies show that foundational models, trained on paired radiology reports and 2D medical images, perform comparably to fully supervised methods in various tasks¹⁵. However, these studies have not extended to 3D medical images, primarily because a publicly available 3D medical imaging dataset paired with text reports is absent. In this study, we introduce CT-RATE, a comprehensive dataset, which we first use to develop CT-CLIP, a foundation model for 3D medical imaging using radiology text reports (Fig. 1c).

CT-CLIP provides a versatile solution for various tasks without requiring task-specific training. Unlike supervised methods, it eliminates the need for explicit labels and enables predictions beyond predefined classes (Fig. 1e), making it particularly valuable when learning objectives evolve post training. CT-CLIP’s zero-shot multi-abnormality detection outperforms the state-of-the-art supervised method across all metrics (Fig. 4a). Importantly, this experiment suggests that explicit labels are not necessary for high performance in 3D medical image interpretation when corresponding radiology reports are available for training. Furthermore, we externally validate CT-CLIP on two independent datasets, RAD-ChestCT and UPMC, from a different country (Fig. 3e). The model’s ability to generalize to different distributions, a primary challenge in deploying medical AI⁴⁷, is confirmed as it outperforms the fully supervised method in all metrics in the external validation sets as well (Fig. 4a). To establish a fully supervised baseline and fine-tune CT-CLIP, we extract multi-abnormality labels from reports, making them publicly available. Following recent examples, we develop an automatic labeller by fine-tuning a language model (Fig. 1b)⁴⁸. However, developing such a system requires time, expertise and domain knowledge, as evidenced by our effort in manually annotating a small portion of the reports. CT-CLIP, as a zero-shot classifier, overcomes this bottleneck by detecting abnormalities at a supervised level without the need for an automatic labeller. Fine-tuning experiments further confirm CT-CLIP’s representational learning capabilities, demonstrating superior performance (Fig. 4a). Despite being trained without explicit supervision, CT-CLIP’s vision encoder captures clinically meaningful features that are highly transferable to downstream tasks. These fine-tuning strategies, one tailored for classification with predefined labels (ClassFine), and the other enabling open-vocabulary inference (VocabFine), build upon the same pretrained encoder and consistently yield superior results compared with a fully supervised baseline. This indicates that the encoder does not rely on memorizing specific abnormality classes but instead learns a more general and semantically aligned representation of the 3D chest CT volumes. Such flexibility underscores the foundational nature of CT-CLIP and its potential for broad applicability in medical imaging tasks, even beyond those explicitly considered during training. Finally, and indicating a direction for future developments, the effective clustering observed in t-SNE projections (Fig. 4d,e) unveils another aspect of CT-CLIP’s clinical potential. The successful clustering of abnormalities (Extended Data Figs. 1 and 2) suggests that CT-CLIP could extend its use beyond multi-abnormality detection to assist in prognosis, disease grading and the calculation of lung-related parameters in future research.

Analysing CT-CLIP’s open-vocabulary inference capabilities, we observe potential for developing multimodal algorithms that address broader and more diverse clinical applications. To expand these capabilities, we fine-tune four state-of-the-art LLMs through visual instruction tuning with the CT tokens created by CT-CLIP, resulting in the creation of CT-CHAT. For fine-tuning, we create and open source a visual question-answering dataset derived from CT-RATE (Extended Data Fig. 8). As a result, CT-CHAT outperforms existing 2D models, underscoring the urgent need for generalist models specialized in 3D medical imaging (Fig. 6c). We find that the choice of LLM (Supplementary Table 10a) and the use of concept alignment pretraining (Supplementary Table 10b) have minimal impact on CT-CHAT’s accuracy. In contrast, including the volume as input is crucial, particularly in report generation (Supplementary Table 10b). This highlights the value of the CT-RATE dataset and CT-CLIP in developing a generalist AI assistant for 3D medical imaging. Also, we demonstrate that engineering the system prompt of CT-CHAT (Supplementary Table 9) helps mitigate pathology hallucinations when the volume is not provided (Supplementary Fig. 11). Our model, with new special tokens, can respond in various modes, including long and short answers, multiple choice and report generation (Extended Data Fig. 9).

To further evaluate CT-CHAT’s clinical use, we benchmark its report-generation performance against state-of-the-art 3D medical models RadFM²⁹ and CT2Rep³⁰. Since radiology report generation closely mirrors the daily workflow of radiologists, it serves as a more clinically relevant benchmark than VQA. Instead of relying solely on conventional text similarity metrics, which are limited in assessing the clinical accuracy of generated content, we employ clinical-efficacy metrics, using our in-house automated report labeller, which was also used to annotate CT-RATE (see ‘Creating the CT-RATE dataset’ in Methods for more details). CT-CHAT consistently outperforms both RadFM and CT2Rep on the CT-RATE internal and RAD-ChestCT external validation sets across all clinical metrics (Fig. 6e). Moreover, we demonstrate that integrating structured outputs from other advanced 3D algorithms can further improve report generation. By incorporating detailed nodule-segmentation information, such as lobar location and size, extracted using a state-of-the-art 3D segmentation model, we substantially enhanced CT-CHAT’s report accuracy (Fig. 6e). This experiment emphasizes that leveraging robust 3D models for different abnormalities could further boost performance, yet such high-quality models are still scarce. Continued development and open-source release of more domain-specific 3D segmentation algorithms are essential for future progress. Our expert evaluation adds a robust layer of validation, consistently confirming CT-CHAT’s superior performance (Fig. 6d). This direct clinical assessment highlights the real-world value of our approach. Notably, CT-CHAT’s advantage over CT2Rep, despite both models being trained on the CT-RATE dataset, demonstrates the effectiveness of pretraining with CT-CLIP. These results reinforce the strength of our foundational model, CT-CLIP, in learning clinically meaningful representations and enabling high-quality performance in downstream tasks. However, it remains far from ready for clinical deployment, as emphasized by our radiologists. As with all AI systems, challenges such as robustness, interpretability, regulatory compliance and prospective validation must be addressed before integration into routine radiology workflows. To help broaden accessibility, we introduce a user-friendly graphical interface (via our GitHub repository) that allows non-experts to interact with the CT-CHAT model. This interface not only makes CT-CHAT a valuable tool for educational and clinical research settings but also invites a wider audience to engage with and contribute to the evolution of interactive AI-driven inference in clinical practice.

Despite these advancements, our CT-RATE dataset and the CT-CLIP and CT-CHAT models have certain limitations. First, biases inherent to our single-institution dataset may influence our self-supervised training. Although we curate a large and diverse set of volume–report pairs, the data originate from a single geographic and clinical setting, which may limit the generalizability of our models to other populations. We conduct an ablation study to assess the impact of training set size on CT-CLIP’s performance, revealing that performance increases with larger training sets (Fig. 4c). This suggests that incorporating data from multiple institutions and more diverse patient populations could further enhance model robustness. Second, CT-CLIP depends on the specific terminology used in reports and the choice of prompts during zero-shot detection. Variability in how diseases are described across radiology reports can affect detection accuracy. To assess the importance of prompt engineering, we evaluate zero-shot multi-abnormality detection using seven different prompts (Fig. 4b). While we select the best-performing prompt from commonly used structures in reports, continued optimization may further improve performance. Third, the evaluation of CT-CHAT faces limitations due to the lack of pretrained 3D medical multimodal AI assistants for benchmarking. As a result, we compare CT-CHAT against 2D models, which may not fully capture the 3D spatial information inherent in CT volumes. To address this, we use DRRs generated from our dataset⁴⁵. This approach is suggested by previous work showing that models trained with DRRs can yield comparable results to those trained with 3D chest CT volumes⁴⁵. However, the absence of true 3D benchmarks highlights the need for further development and validation of multimodal AI assistants in 3D medical imaging. Due to the absence of 3D VQA baselines, we also benchmark CT-CHAT’s report-generation function against state-of-the-art 3D radiology report-generation models. Despite CT-CHAT being trained for VQA, CT-CHAT outperforms these models; however, the quality of its generated reports remains insufficient for clinical use, underscoring the need for further improvements in performance and clinical validation. Finally, our current models are limited to chest CT volumes. To broaden their clinical applicability, future work should focus on extending CT-RATE, CT-CLIP and CT-CHAT to include other 3D imaging modalities such as MRI and PET scans, as well as other anatomical regions. Expanding to different modalities and body parts would enhance the use of our models across a wider range of medical imaging tasks and clinical scenarios.

In conclusion, the development of CT-CLIP and CT-CHAT marks a major step forward in 3D medical imaging. Enabled by CT-RATE, an open-source 3D dataset paired with radiology reports, these models set a benchmark for multimodal AI in this domain. Their open availability provides a strong foundation for future research and, ultimately, for the development of clinically deployable systems.

Methods

Ethics statement

Our study was granted ethics approval by the Clinical Research Ethics Committee at Istanbul Medipol University (E-10840098-772.02-6841, 27/10/2023) for open sourcing our CT-RATE dataset and associated trained models. To uphold patient privacy, we anonymized all chest CT volumes, associated metadata and radiology text reports before conducting any computational analysis.

Creating the CT-RATE dataset

We curated a novel dataset, CT-RATE, consisting of chest CT volumes and corresponding radiology text reports from Istanbul Medipol University Mega Hospital. The CT-RATE dataset included non-contrast chest CT volumes acquired between May 2015 and January 2023, totalling 50,188 reconstructed CT volumes from 25,692 distinct CT experiments conducted on 21,304 unique patients. We divided the cohort into two groups: 20,000 patients were allocated to training and 1,304 for validation. Figure 1a illustrates the dataset accumulation process, and Fig. 3e shows the training and validation split.

In clinical practice, CT volumes are reconstructed with various techniques for specific diagnostic requirements. Sharper kernels, for instance, improve resolution for lung abnormalities, whereas smoother kernels are favoured for assessing mediastinal issues³¹. For this reason, we incorporated every reconstructed volume from the hospital database into our dataset. This strategy facilitated a thorough evaluation of various abnormalities. The patients ranged in age from 18 to 102 years, with a mean of 48.8 years and a mode of 40 years. The sex distribution is 41.6% female and 58.4% male. Of the scans, 61.5% were taken using Philips CT scanners, 30.1% with Siemens scanners and 8.4% with PNMS (Philips-Neusoft Medical Systems) scanners. The CT volumes were provided in various resolutions: 65.4% in 512 × 512 px, 4.2% in 768 × 768 px and 30.4% in 1,024 × 1,024 px. The pixel spacing in the axial (XY) plane ranged from 0.227 to 1.416 mm, with a mean of 0.605 mm, while slice thickness ranged from 0.035 to 6 mm, with a mean of 1.231 mm. The number of slices per volume ranged from 100 to 600, averaging 304.7 with a mode of 255 slices. The radiology reports were structured into four sections: clinical information, technique, findings and impression. The clinical information section outlines the patient’s symptoms and medical history. The technique section specifies the acquisition protocol, encompassing details such as the field of view. The findings section covers anatomical and pathological observations, whereas the impression section offers a diagnosis based on these findings. Figure 2 summarizes the CT-RATE dataset alongside the first external validation set (RAD-ChestCT). Unfortunately, detailed metadata for the second external validation set (UPMC) are not publicly available, as it is a private dataset and cannot be shared; thus, it was not included.

To anonymize the CT-RATE dataset, we first anonymized reports by removing all personal information pertaining to patients and medical professionals. This process involved targeted searches using regular expressions to identify and eliminate date-related information and patient–doctor identifiers. Subsequently, these anonymized reports were translated from Turkish to English using the Google Translate API, and bilingual final-year medical students reviewed and corrected all translations. They also thoroughly examined each report to remove any residual personal information and ensure coherence. Only the English versions of these reports were included in our CT-RATE dataset. For the CT volumes, we implemented several quality control measures. We excluded volumes that involve contrast material or body parts other than the chest. Only the remaining data were used for developing our models and open sourced as the CT-RATE dataset. Each chest CT volume’s orientation was standardized by examining its metadata to ensure consistency. In addition, we removed all identifying information about patients and medical professionals from the metadata. The anonymized metadata were open sourced as part of the CT-RATE dataset. Extended Data Table 1 provides a detailed explanation of these metadata attributes.

Extended Data Fig. 4. Comparison of abnormality-based performance metrics in the Rad-ChestCT validation set.

This figure provides a detailed analysis of performance metrics, including AUROC, accuracy, precision, and F1 scores, for detecting various abnormalities with our models in the Rad-ChestCT validation set, compared to the fully supervised baseline model. It highlights the models’ remarkable adaptability and superior effectiveness with distribution shifts, setting a new standard in performance compared to a fully supervised baseline model.

Source data

To develop an automated label extractor model, we annotated 1,000 reports by merging findings and impression sections, as some abnormalities were not explicitly mentioned in a single section. This created a multilabel dataset for each report, identifying 18 abnormalities. Conditions that were not referenced in either section were considered absent. Similar conditions were grouped: left and right mucoid impactions under ‘mucoid impaction’, and various lung opacities (lung opacity, density increase and ground-glass opacities) under ‘lung opacity’. Lung and fissural nodules were categorized as ‘lung nodules’. The annotated reports were divided into training (80%, 800 reports) and validation (20%, 200 reports) sets. Leveraging this dataset, we fine-tuned the RadBERT-RoBERTa-4m model³³, which was originally trained on over 4 million radiology reports. Then, we applied it to classify the remaining 24,692 reports. These classified reports served three purposes: training the supervised baseline model, fine-tuning CT-CLIP and internal evaluation of the models. Figure 1b illustrates the automated classification process for radiology reports. Supplementary Table 2 shows the precision, recall and F1 scores of the report classifier.

Our preprocessing approach, inspired by previous work²⁰, involved several key steps to standardize all chest CT volumes in the dataset. First, each volume was resized to achieve uniform spacing of 0.75 mm in the x axis and y axis, and 1.5 mm in the z axis. The CT volumes were then either centre cropped or padded to reach a consistent resolution of 480 × 480 × 240, ensuring comprehensive inclusion of the patient’s relevant area for the interpretation. Subsequently, we converted each CT volume to Hounsfield Unit (HU) values using intercept and slope values from the metadata. HU values, a measure of radiodensity, were clipped to a range of −1,000 to 1,000. This range represents the practical lower and upper limits of the HU scale⁴⁹. During training, we normalized these values to a range of −1 to 1. The CT-RATE dataset contains the original raw data, and all preprocessing scripts are provided in our GitHub repository.

In addition to radiology reports and metadata, we planned to extend CT-RATE with multi-abnormality segmentation and localization labels for 3D chest CT volumes. To our knowledge, no publicly available dataset currently provides such comprehensive annotations for 3D medical imaging, making this a highly valuable contribution. Given the substantial effort and expert input required, this will be pursued as part of future work. As an initial step, we included organ-level anatomical-segmentation labels generated using a state-of-the-art segmentation model⁵⁰ (see Supplementary Table 11). These labels are openly available through the CT-RATE repository on Hugging Face and hopefully support tasks such as visual grounding and anatomical localization, and serve as a foundation for future fine-grained abnormality segmentation.

Adding external validation datasets to CT-RATE

For the first external validation set, we used the RAD-ChestCT dataset, which includes a cohort from another country²⁰. The dataset comprised 3,630 non-contrast chest CT volumes, uniformly reconstructed using a single technique. Half were conducted with Siemens CT and the other half with GE Medical scanners. The number of slices ranged from 46 to 1,277, with an average of 511.6 and a mode of 472. All CT volumes had a resolution of 512 × 512 pixels. The pixel spacing in the axial (XY) plane ranged from 0.189 to 0.977 mm, with an average of 0.692 mm, while the slice thickness varied from 0.125 to 5 mm, with an average of 0.706 mm. To ensure consistent evaluation, we applied identical preprocessing methods to this external dataset as used with our CT-RATE cohort. Since the RAD-ChestCT dataset lacked a ‘mosaic attenuation’ label, we excluded it from our external evaluation. Furthermore, while the RAD-ChestCT dataset included a ‘Calcification’ label, our dataset distinguished between ‘Arterial wall calcification’ and ‘Coronary artery wall calcification’. In assessing the model accuracies on the external evaluation set, we used the higher classification probabilities between these two labels for the ‘Calcification’ prediction.

For the second external validation set, we used the validation split from the MedSyn paper³², which contains 1,024 chest CT scans and paired radiology reports, each from a different participant. These scans originated from the University of Pittsburgh Medical Center (UPMC) and had been fully de-identified. We employed the same automated report labeller used for CT-RATE to extract 18 abnormality labels from the text reports for evaluation. To ensure consistent and fair comparison, we applied the same preprocessing pipeline to chest CT volumes as used with our CT-RATE cohort.

Developing the CT-CLIP model

CT-CLIP is a novel 3D adaptation of the contrastive language–image pretraining (CLIP) which is originally designed for natural image–text pairs³⁵ and has shown excellent performance in 2D medical data^15,51. It learns to identify similarities between text and images via contrastive learning (Fig. 1d), enabling efficient zero-shot classification. We extended CLIP to work with 3D CT paired with reports. It integrates a 3D encoder, previously used for generating CT²³, as the image encoder, and CXR-Bert⁴⁸ as the text encoder. Similar to the 2D CLIP, our method extracted latent embeddings from input volumes and radiological reports. We then computed the contrastive loss, which measures the similarity between these embeddings. Training the model to align the 512-dimensional projection layers from text and CT volume embeddings with the identity matrix (as shown in Fig. 1c) allows identifying meaningful correlations between textual information and 3D medical images. This process enables zero-shot classification and can be further enhanced through task-specific fine-tuning for supervised classification.

As a 3D vision transformer, we used a component of CT-ViT²³, a network designed to generate low-dimensional tokens from CT volumes and reconstruct CT volumes from these tokens. CT-ViT divides CT volumes into patches measuring 20 × 20 × 10 voxels in the coronal, sagittal and axial planes, ensuring uniform spacing. These patches are processed by spatial and causal transformers to produce low-dimensional CT tokens, which are then decoded to reconstruct the CT volumes²³. For CT-CLIP, we used only the encoder part of this network to generate CT tokens. During the training phase of CT-CLIP, the encoded CT tokens were averaged across the axial plane, then flattened and transformed into a 512-dimensional layer. This created latent representations for the CT volumes, as shown in Fig. 1c. This approach reduces GPU memory requirements significantly through the linear transformation layer for latent representations, while still capturing information from every slice due to the axial attention mechanism in the CT-ViT. The improved memory efficiency enables the use of a larger 3D CT encoder.

For latent representations of radiology reports, we employed the CXR-Bert text encoder⁴⁸. This encoder can process up to 512 tokens, which allows the inclusion of both impression and findings sections from reports. For reports with content of fewer than 512 tokens, we used padding with the <PAD> token, where padded tokens are masked during the attention mechanism. Each token is represented in 768 dimensions in the CXR-Bert text encoder. These tokens are then summed across the 512-token span and linearly transformed into a compact 512-dimensional layer, encapsulating the essential textual information from the reports. Figure 1c visually represents this encoding process. In addition, to assess the impact of including the impression and findings sections, we conducted an ablation study on impression only and findings only training in Supplementary Fig. 1a. To highlight the impact of scaling the dataset size on model performance, we conducted an ablation study. This experiment involved training the CT-CLIP model with varying sizes of the CT-RATE dataset: 9.8% (matching the size of the RAD-ChestCT dataset, the only public chest CT dataset with multi-abnormality labels²⁰), 20%, 40%, 60%, 80% and 100%. By systematically altering the sizes of the dataset, we explored how the accuracy of the model scales with the amount of available training data. Figure 4c illustrates the outcomes of this ablation study.

CT-CLIP for zero-shot multi-abnormality detection

In our CT-CLIP model, we implemented a similar approach to CheXzero¹⁵ for zero-shot multi-abnormality classification. This method adapts CLIP’s zero-shot classification technique to handle multiple classes instead of multiple labels for the same class, which is particularly important in medical imaging where images often contain several abnormalities simultaneously. We started by extracting the model’s preliminary output scores, known as logits, using both positive (for example, ‘Consolidation is present.’) and negative (for example, ‘Consolidation is not present.’) prompts. The prompting method for inference was chosen through prompt engineering experiments. We then applied softmax to these logits to convert them into a likelihood of each abnormality’s presence or absence in a CT volume. Figure 1e illustrates CT-CLIP’s inference strategy for the zero-shot multi-abnormality detection task.

To optimize CT-CLIP’s performance for zero-shot multi-abnormality detection, we experimented with various prompts, derived from common sentence structures in radiology reports. The positive–negative formulations for prompts 1 to 7 are as follows: ‘{$\varnothing$/No} {abnormality}.’, ‘Findings are {$\varnothing$/not} compatible with {abnormality}.’, ‘There is {$\varnothing$/no} {abnormality}.’, ‘{Abnormality} is {$\varnothing$/not} seen.’, ‘{$\varnothing$/Not} {abnormality}.’, ‘{Abnormality} is {$\varnothing$/not} observed.’ and ‘{Abnormality} is {$\varnothing$/not} present.’. We evaluated these prompts on the basis of their classification scores (Fig. 4b) and applied the most effective one to fine-tune the CT-CLIP model using the VocabFine approach and during inference.

Fine-tuning CT-CLIP for multi-abnormality detection

To enhance the performance of the CT-CLIP model in classifying multiple abnormalities, we implemented two fine-tuning approaches. The first approach, called VocabFine, involves fine-tuning the model while retaining its open-vocabulary capabilities. The second approach, adapted from linear probing, is called ClassFine. This method entails training the model with an additional classification layer. We also conducted ablations to investigate the impact of freezing pretrained layers within CT-CLIP during fine-tuning (Supplementary Fig. 1b). Specifically, in addition to training every layer, we fine-tuned only the projection layers of the image and text encoders in the case of VocabFine. For ClassFine, alongside training every layer, we trained the model only for the newly added classification layer.

The first approach, VocabFine, is a modification of WISE-FT, a technique for fine-tuning foundation models⁴⁰. In WISE-FT, the zero-shot model generates logits using classification labels as input text prompts. The model is then fine-tuned using binary cross-entropy (BCE)⁵² to compare these logits with the actual ground-truth labels. This process allows the model to learn specific classification labels while maintaining its ability to function with an open vocabulary⁴⁰. Our VocabFine approach adapted this method for the multilabel classification. We generated both positive and negative prompts for each abnormality and then concatenated their respective logits into a unified array of 36 elements (two logits for each of the 18 abnormalities). The ground-truth labels were represented as binary arrays (1s and 0s), where 1 indicates a positive logit if the ground truth is ‘True’ and vice versa for ‘False’. For example, if a volume’s label is true for a specific abnormality, it corresponds to [1, 0] in the ground-truth array; false would be represented as [0, 1]. The training aligned these ground-truth values with the computed logits. To optimize computational efficiency, we divided the 36-element array into smaller segments of 12 elements for each gradient run. Figure 3c depicts the fine-tuning process in VocabFine. The inference strategies of both CT-CLIP and the fine-tuned model with Vocabfine are the same (Fig. 3d).

Our second fine-tuning strategy, ClassFine, was achieved by integrating a linear layer into the visual encoder network, a technique inspired by ref. ⁵³. The architecture of the ClassFine approach is demonstrated in Fig. 3a. This modified model undergoes training using ground-truth labels. The model fine-tuned with ClassFine is then evaluated using both internal and external evaluation sets. While this approach offers the potential for improved classification accuracy, it also makes the foundation model more specific for the trained classification task. Since ClassFine is based on fully supervised fine-tuning, its capabilities are primarily limited to the classes defined during this fine-tuning phase. Figure 3b visualizes the inference process of ClassFine, highlighting its specific application to the predetermined classes.

Benchmarking the CT-CLIP models for multi-abnormality detection

In assessing our CT-CLIP and fine-tuned models, we employed the CT-Net model as a fully supervised baseline²⁰. Renowned for its performance in a similar CT dataset²⁰, CT-Net provides a robust comparison point. We trained the CT-Net model on our CT-RATE dataset, adapting the chest CT volumes to fit the model’s input requirements by resizing them to 420 × 420 × 402. Consistent with our models, we focused on the abnormalities present in the CT-RATE dataset (Fig. 3d), instead of the 98 and 9 abnormalities explored in the original CT-Net study. For each abnormality, we determined classification thresholds by identifying the left-upper point on the receiver operating characteristic (ROC) curve, using the 10% of each validation set. This analysis was performed across all three evaluation sets: CT-RATE (internal), RAD-ChestCT (external) and UPMC (external), and for all multi-abnormality detection models: zero-shot, fine-tuned and fully supervised. In addition, we computed the area under the receiver operating characteristic (AUROC) values for each abnormality.

To provide a comprehensive evaluation, we also calculated accuracy, precision and weighted F1 scores for each identified abnormality. For the weighted F1, each of the 18 abnormalities was treated as a binary task (present vs absent); we computed a weighted F1 score that balances positive and negative cases within each abnormality and then averaged these scores to produce a single overall metric. This ‘macro of weighted F1 scores’ ensures that (1) each abnormality contributes equally to the final score and (2) class imbalance within each abnormality is addressed fairly, resulting in a clinically meaningful measure of performance. To further validate the latent spaces generated by CT-CLIP for radiology reports and 3D chest CT volumes, we also conducted a t-SNE analysis⁵⁴. This allowed us to visually map the distribution of abnormalities and other relevant attributes within the t-SNE embeddings (Fig. 4d,e, and Extended Data Figs. 1 and 2).

For the statistical analysis of our model evaluations, we used the non-parametric bootstrap method to examine the distribution of predictions on internal and external validation sets. This involved 1,000 iterations of random sampling of the volumes with replacement, each using sample sets of size n, matching the number of evaluated volumes. We calculated the average metric values for each sample and quantified dispersion by computing the standard deviation of these bootstrap values. For statistical significance, we employed a two-sided paired permutation test with 10,000 permutations to assess differences in performance between the two models across AUROC, weighted F1 score, accuracy and precision metrics, following previous work⁵⁵. Detailed comparisons are provided in Supplementary Table 5.

Benchmarking the CT-CLIP models for CT volume retrieval

Image retrieval involved identifying the most relevant image from a pool of candidates on the basis of either textual⁵⁶ or visual⁵⁷ input. With CT-CLIP, trained to associate CT volumes with text reports, volume retrieval becomes possible. This was achieved by computing the cosine similarity among all pairs of volumes and reports, as well as among volume pairs, within a unified embedding space. This embedding space was generated by projecting CT tokens derived from the volumes through the projection layer (Fig. 1c). To conduct volume-to-volume retrieval, we calculated the cosine similarity between the latent embeddings of a specific chest CT volume in the validation set and all other abnormal chest CT volumes within the same set. We then ranked the volumes on the basis of their cosine similarity scores to identify the most relevant matches for a given volume. This volume-to-volume retrieval process is shown in Fig. 5a.

To evaluate this retrieval task’s performance, we adapted the mean average precision at K (MAP@K) metric, typically used in image-to-image retrieval tasks⁴¹. In this metric, an image is considered relevant if it matches the classes of the target input. Precision is then calculated as the proportion of relevant images within the top K retrieved images. Relevance is determined using the intersection over the union of the abnormality labels marked as ‘True’ in the ground truth with those in the retrieved chest CT volume. The average precision at K (AP@K) is defined as follows:

$$\text{AP@K}=\frac{1}{K}\sum _{i=1}^K{P}_i\times R_i$$ 1

Here, P_i is the precision at the ith position, and R_i is the relevance at the ith position. The final class retrieval accuracy is given by:

$$\mathrm{MAP@K}=\frac{1}{n}\mathrm{AP@K}$$ 2

where n is the total number of samples. It is important to note that as K increases, MAP@K may decrease due to the inclusion of less relevant images. We used the vision encoders of the supervised baseline, CT-CLIP and the fine-tuned models for volume-to-volume retrieval and assessed their performance across both internal and external validation sets using MAP@1, MAP@5, MAP@10 and MAP@50 metrics. The Recall@K metric was not applied in image-to-image retrieval due to the inherent limitation that the cosine similarity for identical images would inevitably be the highest, resulting in a consistent Recall@K value of 1. However, this issue does not arise in text-to-image retrieval since the query is based on text embeddings rather than image embeddings.

For report-to-volume retrieval, we used the pretrained text transformer and 3D vision transformer to generate vector embeddings for both radiology reports and CT volumes, which were transformed from the token space to the embedding space with the CT-CLIP’s trained projection layers (Fig. 1c). Our goal was to find the CT volume that best matched each query report. To achieve this, we computed the cosine similarity between the latent representations of each report and all the CT volumes. Subsequently, we ranked the CT volumes on the basis of their cosine similarity scores to identify and return the most relevant matches. The process of report-to-volume retrieval is depicted in Fig. 5b.

For the evaluation of this task, we employed the Recall@K metric, where K represents the number of returned images. This method serves as a standard evaluation tool in text-to-image retrieval tasks⁴². We assessed four Recall@K metrics: Recall@5, Recall@10, Recall@50 and Recall@100. These metrics measure the frequency of the target image appearing among the top 5, 10, 50 and 100 retrieved images, respectively. It is worth noting that the external validation set lacks text reports. Therefore, we restricted the report-to-volume retrieval evaluation to the internal validation set. We assessed both CT-CLIP and the fine-tuned model with VocabFine for this task. However, since the supervised model and ClassFine do not generate latent embeddings for reports, they were not used in the report-to-volume retrieval.

Creating the dataset for CT-CHAT

To train CT-CHAT, we curated a comprehensive VQA dataset derived from free-form reports and chest CT volumes, using the same train–validation split as CT-RATE. The dataset had four main components, with the following details for the training set.

The first component is the long-answer questions, which were further divided into four subcategories: conversation-based questions, description questions, free-response questions and text-only questions. The conversation, description and free-response questions were generated from radiology reports, while the text-only questions were created using 48,343 articles related to chest CT, retrieved from the PubMed Central–OA (www.ncbi.nlm.nih.gov/pmc/tools/openftlist). In total, the long-answer component of the training set included 281,689 conversations, with a total of 518,053 conversation questions, 138,104 description questions, 138,061 free-response questions and 250,760 text-only questions. All questions were generated using the Llama 3.1 8B large language model⁴³, following the methodology described in ref. ¹⁴. The scripts for generating the question–answer pairs are available in the GitHub repository, and the prompts are provided in Supplementary Table 7. The second component consists of multiple-choice questions, also generated by Llama 3.1 8B⁴³, following the same approach described in ref. ¹⁴. This component of the training set included a total of 46,911 multiple-choice conversations, comprising 138,608 questions in total. The third component included short-answer questions derived from radiology reports in the CT-RATE dataset using a rule-based approach, as described in ref. ⁵⁸. These questions covered categories such as location, presence, abnormality and disorder. A total of 1,417,828 short-answer questions were sampled to create 318,710 short-answer conversations in the training set. The fourth component involved radiology report-generation questions, totalling 47,149 conversations in the training set, where each conversation contained a single report-generation question along with the corresponding generated report. The report-generation questions were randomly selected from a predefined set of questions (Supplementary Table 6).

The validation set was curated similarly to the training set, using the free-form reports from the CT-RATE validation set. The validation dataset included 33,326 conversation questions, 8,886 description questions, 8,888 free-response questions, 12,013 short-answer questions, 8,915 multiple-choice questions and 3,039 report-generation questions. Extended Data Fig. 7 shows example entries from the described dataset. Since the VQA dataset was generated by a large language model, we carefully selected only accurately generated and properly formatted data, that is, data that could be successfully parsed as JavaScript Object Notation for training our models. In addition, we limited the initial number of generated questions to maintain a reasonable training duration. Our VQA dataset has been made publicly available alongside CT-RATE.

Extended Data Fig. 7. Comparative metrics for CT-CHAT and baselines in report generation.

This figure presents the report generation performance of CT-CHAT compared to baseline models across multiple abnormalities, evaluated on two independent validation sets. CT-CHAT consistently outperforms the baselines, demonstrating superior clinical accuracy.

Source data

Developing the CT-CHAT model

Adapted from the LLaVa framework⁵⁹, CT-CHAT integrates the frozen CT-CLIP vision encoder, a multimodal projector module and a large language model to enhance multimodal processing capabilities. The visual encoder compresses 3D chest CT volumes into a low-dimensional feature space. The multimodal projector module then transforms these visual features into the LLM’s embedding space, aligning it with text tokens. The LLM processes these visual–language tokens as input to generate contextual responses. Figure 6a summarizes the architecture of the CT-CHAT model.

In our implementation, we evaluated and compared four different LLMs. For the primary configuration of CT-CHAT, we used the 70B Meta Llama (3.1)⁴³, one of the state-of-the-art open-source LLMs. This model consists of 80 transformer layers, 64 attention heads, an embedding dimension of 8,192 and a feed-forward network (FFN) dimension of 28,672, and supports a context size of up to 128,000 tokens. For a smaller model, we used the 8B Meta Llama (3.1)⁴³, which includes 32 transformer layers, 32 attention heads, an embedding dimension of 4,096, an FFN dimension of 14,336 and a context size of up to 128,000 tokens. In addition, for comparison, we implemented two LLMs used in the official LLaVA (1.6)²⁶: Vicuna 13B and Mistral 7B. Figure 6d shows the scores for each of these LLMs. To dynamically adjust the model’s output mode on the basis of the specific task, we added four special tokens to the tokenizers of each model: <long_answer>, <short_answer>, <report_generation> and <multiple_choice>. These tokens were selected for different tasks during training and inference. The multimodal projector, based on the approach described in ref. ¹⁴, consists of an attention-pooling layer, followed by a 2-layer multilayer perceptron (MLP). The attention-pooling mechanism uses 256 learned latent queries in conjunction with multiheaded cross-attention to compress the final layer’s features from the encoder backbone into a fixed-length sequence of image tokens. This approach optimizes both training and inference, ensuring that the sequence length remains within the LLM’s context window and making the model focus more on the important features with the attention mechanism. The MLP that follows was adapted from LLava (1.6)²⁶ and includes a single hidden layer activated by GeLU, transforming the image tokens to match the LLM’s embedding dimension.

Training CT-CHAT involved two steps. First, in the medical concept alignment step, the LLM was frozen, and the multimodal projector was trained with the input ‘Please provide the radiology report for the following chest CT volume <image>’ and the output report. This step improves subsequent fine-tuning performance and helps the multimodal projector to be aligned in biomedical concepts²⁷. We conducted an ablation study of this step to show its effect (Supplementary Table 10b). In the second step, medical instruction tuning, the model was trained with the CT-VQA dataset that includes long-answer, short-answer, multiple-choice and report-generation question–answer pairs. During this step, the multimodal projector, along with LORA weights⁴⁴ for the LLM with rank 128 and alpha 256, was fine-tuned. <image> tokens were added to indicate the embedded image input to the model, except for text-only questions and answers. The four special tokens were added to the respective question–answer pairs in the dataset, enabling the model to learn different output modes. These tokens were later used during inference to select different modes of the model. The hyperparameters for each model for pretraining and fine-tuning are provided in Supplementary Table 4a,b. To boost report-generation accuracy, we fine-tuned CT-CHAT with structured metadata extracted exclusively from pulmonary-nodule masks. Each CT-RATE volume was processed with TotalSegmentator⁶⁰, from which we recorded, per nodule, the lobar location, centroid, orthogonal diameters and volume (mm³). These fields were embedded in a fixed prompt:

<volume_tokens>

Generate a detailed radiology report for the given chest CT volume.

<BEGIN_NODULE_INFO>

Nodule_Count: n

Nodule_ID: 1 7D2Lobe: left lower

Centroid_mm: (x,y,z)

Diameter_mm: (x: d_x, y: d_y, z: d_z)

Volume_mm3: v

… (repeat for each nodule)

<END_NODULE_INFO>

Each sample included (1) CT volume features from the frozen CT-CLIP encoder, (2) the structured nodule template and (3) the reference report. Fine-tuning was performed on the 8B Llama 3.1 configuration of CT-CHAT using LoRA (rank 128, α = 256); only the LoRA weights and projector were updated. Supplying these segmentation-derived cues markedly improved performance (Fig. 6e). The same workflow could incorporate additional abnormalities as more 3D segmentation models become available.

Benchmarking the CT-CHAT model

Given the lack of pretrained VQA models for 3D imaging, we conducted a comparative analysis of CT-CHAT against four 2D state-of-the-art open-source VQA models: LLaVA 1.6 (Mistral 7B)²⁶, LLaVA 1.6 (Vicuna 13B)²⁶, LLaVA-Med²⁷ and CXR-LLaVA²⁸, using the benchmarking framework established in ref. ¹⁴. To address the challenge of representing 3D information using their 2D encoders and to ensure a fair comparison, we used DRRs derived from CT-RATE, as suggested by ref. ⁴⁵. This approach was chosen over the selection of random or central CT slices, which could not adequately capture all relevant pathologies. The use of DRRs was justified, as ref. ⁴⁵ demonstrates that models trained with DRRs show comparable results to those trained with 3D chest CTs.

To evaluate the baseline models and CT-CHAT models with different LLMs, we employed four NLP metrics: BLEU-1 (ref. ⁶¹), METEOR⁶², ROUGE-L⁶³ and CIDEr⁶⁴, along with the Llama score, which measures the clinical accuracy of the generated responses compared to the ground truth, using the Llama 3.1 70B model⁴³. A similar LLM-based clinical accuracy measurement approach was used in Llava-Med with the GPT-4 API²⁷. Supplementary Table 8 presents the system prompt for the Llama 3.1 model, which was used to score the generated answers. We calculated scores specifically for each task: long answer, short answer, multiple choice and report generation. Figure 6c illustrates mean and task-specific scores for each model. To visualize the clinical accuracy of the models, we categorized the Llama scores for every question in each task into three ranges: 0–3 for low clinical accuracy, 4–6 for average clinical accuracy and 7–10 for high clinical accuracy, and created Sankey plots within these ranges (Supplementary Fig. 10).

In addition to benchmarking CT-CHAT on VQA tasks, we further evaluated its performance in radiology report generation, a task more reflective of routine radiological workflows, by comparing it with two state-of-the-art 3D report-generation models: RadFM²⁹ and CT2Rep³⁰. While VQA metrics such as BLEU, ROUGE and the Llama score assess linguistic and clinical quality, they do not fully capture diagnostic use. Therefore, for this comparison we employed clinical-efficacy metrics: precision, recall, F1 and the CRG score⁶⁵. CRG weights findings by prevalence and severity, giving a distribution-aware score aligned with radiologists’ priorities. Metric computation used our automated report labeller, which extracts structured abnormality labels from the generated reports. Evaluation was conducted on both the internal CT-RATE validation set and the external RAD-ChestCT dataset; for RAD-ChestCT, which lacks text reports, we leveraged its binary abnormality annotations to enable indirect evaluation through our labeller. Notably, we used the original pretrained weights for both RadFM and CT2Rep without retraining.

To complement automatic metrics, we performed a blinded, three-reader study of report quality. Two board-certified thoracic radiologists from Zurich (R1, R2) and one from Istanbul (R3) independently rated reports generated by CT-CHAT, RadFM and CT2Rep. Because free-text scoring is time-intensive, we limited the pool to 150 CT-RATE validation cases per model. Cases were stratified rather than randomly sampled: after computing Llama scores for CT-CHAT outputs, we selected 50 volumes from each accuracy band (low 0–3, medium 4–6, high 7–10) to ensure balanced difficulty. For every selected scan, the three anonymized model outputs were presented, and readers scored each report on a 0–10 scale against predefined criteria covering finding description, localization, terminology and overall clinical adequacy. Figure 6d shows the results: CT-CHAT achieved mean scores of 3.44 (R1), 4.10 (R2) and 4.52 (R3), outperforming RadFM (0.55, 0.91, 0.86) and CT2Rep (2.66, 2.52, 2.74). Violin plots display a right-shifted distribution for CT-CHAT, confirming superior perceived clinical fidelity. The same protocol can be repeated for larger or external cohorts as new models emerge. It is important to note CT-CHAT’s superiority over CT2Rep, even though both models were trained on the same CT-RATE dataset. Moreover, CT2Rep is optimized solely for report generation, while CT-CHAT is tuned primarily for VQA tasks, classification and question answering, and only secondarily for report generation. Despite this, CT-CHAT consistently outperformed CT2Rep across all evaluations. The performance gap underscores the value of contrastive pretraining with CT-CLIP in capturing clinically meaningful 3D CT representations that transfer effectively to downstream tasks, even when the model is not explicitly trained for them.

To evaluate CT-CHAT’s ability to retrieve information from chest CT, we conducted an ablation study for the image provision, as shown in Supplementary Table 10a. Since the model is expected to hallucinate pathologies when the input volume is not provided, we implemented a guardrail for prevention with the system prompt in Supplementary Table 9. Supplementary Fig. 11 provides an example of the guardrail.

Computational hardware and software

Experiments for CT-CLIP were conducted using Python 3.11.5 with PyTorch 2.0.1, CUDA 11.7 and other libraries such as SciPy 1.11.1, Torchvision 0.15.2, Scikit-learn 1.3.0, Pandas 2.0.3, NumPy 1.24.4, Transformers 4.30.1 and Accelerate 0.21.1. For CT-CHAT, Python 3.12.4 was used with PyTorch v.2.4.0, CUDA v.12.4, SciPy v.14.0, Torchvision v.0.19.0, Scikit-learn v.1.2.2, Pandas v.2.2.2 and NumPy v.1.26.4. We used four 80 GB A100 GPUs, 500 GB RAM and 24 CPUs, mainly for CT-CLIP training with distributed data parallel and fully sharded data parallelism. Fine-tuning of CT-CLIP and other experiments was performed on a single A100 GPU. CT-CHAT with the Llama 3.1 70B model was trained on four A100 GPUs using DeepSpeed ZeRO-3, while other CT-CHAT models used two GPUs. Hyperparameters are given in Supplementary Tables 3 and 4. All software details are available in our respective GitHub repositories.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Extended data

Extended Data Table 1.

Metadata attributes

Detailed breakdown of the metadata attributes accessible within the CT-RATE dataset.

Author contributions

I.E.H., S.E. and B.M. designed the study. I.E.H. and S.E. handled data collection, data analysis, model construction and validation, figure preparation and writing of the paper. F.A., A.G.S., S.N.E. and I.D. managed data anonymization and annotation. M.F.D. contributed to model construction. C.W., W.D. and K.B. carried out external evaluation. O.F.D. developed the graphical user interface and assisted with writing of the paper. M.X. and T.A. generated anatomical-segmentation labels. S.S. maintained the model repository. B.H. assisted in preparing NIfTI files with metadata and integrating nodule-segmentation masks into the report-generation pipeline. M.P. assessed report-generation performance. B.K. provided technical expertise and contributed to writing of the paper. H.R., B.W., E.S., M.S., E.B.E., A.A., A.S., A.K., Z.L., B.L., C.B. and M.K.O. provided domain-knowledge support. B.M. supervised the entire study. All authors reviewed and approved the final manuscript.

Peer review

Peer review information

Nature Biomedical Engineering thanks Imon Banerjee, Chang Min Park and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.

Funding

Open access funding provided by University of Zurich.

Data availability

Our CT-RATE dataset, comprising paired chest CT volumes and radiology text reports, is publicly accessible via Hugging Face at https://huggingface.co/datasets/ibrahimhamamci/CT-RATE (ref. ⁶⁶). The external validation set, RAD-ChestCT, is also openly accessible through Zenodo at https://doi.org/10.5281/zenodo.6406114 (ref. ⁶⁷). Source data are provided with this paper.

Code availability

Our trained models and codebase are publicly available in GitHub at https://github.com/ibrahimethemhamamci/CT-CLIP (ref. ⁶⁸) and https://github.com/ibrahimethemhamamci/CT-CHAT (ref. ⁶⁹) for further research.

Competing interests

The authors declare no competing interests.

Extended data

is available for this paper at https://doi.org/10.1038/s41551-025-01599-y.

Supplementary information

The online version contains supplementary material available at https://doi.org/10.1038/s41551-025-01599-y.