All-in-one medical image-to-image translation

Summary

The growing availability of public multi-domain medical image datasets enables training omnipotent image-to-image (I2I) translation models. However, integrating diverse protocols poses challenges in domain encoding and scalability. Therefore, we propose the “every domain all at once” I2I (EVA-I2I) translation model using DICOM-tag-informed contrastive language-image pre-training (DCLIP). DCLIP maps natural language scan descriptions into a common latent space, offering richer representations than traditional one-hot encoding. We develop the model using seven public datasets with 27,950 scans (3D volumes) for the brain, breast, abdomen, and pelvis. Experimental results show that our EVA-I2I can synthesize every seen domain at once with a single training session and achieve excellent image quality on different I2I translation tasks. Results for downstream applications (e.g., registration, classification, and segmentation) demonstrate that EVA-I2I can be directly applied to domain adaptation on external datasets without fine-tuning and that it also enables the potential for zero-shot domain adaptation for never-before-seen domains.

Graphical abstract

Highlights

• •EVA-I2I unifies multi-domain medical I2I translation in a single model
• •DCLIP prompt encoding enables semantic domain control and scalability
• •Outperforms baselines in I2I translation quality and zero-shot domain adaptation
• •Enables direct deployment in clinical tasks without fine-tuning

Motivation

The rise of large-scale multi-domain medical imaging datasets has motivated the development of generalizable image-to-image (I2I) translation models. A core challenge lies in effectively encoding diverse imaging protocols while maintaining semantic consistency and adaptability across domains. Traditional multi-domain I2I approaches often rely on task-specific architectures or discrete domain labels, limiting their scalability and generalization. Inspired by advances in contrastive language-image pre-training and prompt-driven generation, we introduce a unified and extensible framework that leverages DICOM-informed semantic representations to perform I2I translation across multiple medical imaging domains simultaneously.

Han et al. propose EVA-I2I, a scalable image-to-image translation framework that synthesizes medical images across diverse domains using DICOM-tag-informed language-image pre-training. This method enables generalization to unseen domains and direct clinical application, advancing robust AI integration in medical imaging.

Introduction

The application of deep learning in medical image analysis has gained popularity, but addressing the issue of model generalization remains crucial to its practical use in clinical settings.¹ One of the reasons is the domain shift between datasets,² i.e., the training set (source domain) and the test set (target domain) are in different distributions,³ often due to data being collected from different centers with varying scanning parameters or even different modalities.⁴ Many studies^5,6,7 have focused on combating the domain shift in medical image analysis by implementing domain adaptation (DA) or data harmonization due to the lack of a large number of annotated training data that can cover the test set distribution.

Image-to-image (I2I) translation can be used for DA that directly aligns images by converting them from the source domain to the target domain.⁴ This technique has been widely used in the DA of medical images. One of its benefits is that it decouples DA from downstream models, making it applicable to various downstream tasks, such as registration,⁸ classification,⁹ and segmentation.¹⁰ It is worth noting that some I2I works have shown promising results, but few have conducted external validation studies¹¹ due to the cynical performance dropping when migrating I2I. Additionally, in clinical settings, small sample sizes and missing sequences can complicate matters and result in poor results when directly training I2I on the target domains.⁴

Because the number of multi-domain public datasets increases, it now becomes possible to develop an omnipotent multi-domain I2I translation model. The large number of learned domains can cover more target domains, increase model generalization capabilities, and facilitate the acquisition of shared representation of anatomical structures across various domains.⁴ However, as shown in Figure 1A, due to the diverse scanning devices and subtle differences in parameter settings used, integrating datasets and constructing multi-domain I2I translation models can be challenging in encoding different domains while preserving semantic relationships and ensuring scalability. Existing studies partition data into a set of domains (e.g., based on modality or sequence), and it is typical to use one-hot encoding to differentiate between the various domains. Han et al.¹² organize BraTS2021¹³ by placing the T1 (T1-weighted), T1Gd (contrast-enhanced T1), T2 (T2-weighted), and Flair (fluid-attenuated inversion recovery) sequences in a specific order and use one-hot encoding to represent the output sequences, such as “0100” for outputting T1Gd. Some works^14,15 consider enriching the representation by additionally merging the encoding of the input sequence. One-hot encoding has some quite apparent limitations. Expanding the encoded sequence length to accommodate domain expansion can be quite challenging. Moreover, one-hot encoding cannot distinguish semantic differences across multiple domains, such as capturing the semantic relationship among A3, B2, and N2 in Figure 1A. Although all three are T2 images, the semantic distance between A3 (removed skull) and others (with skull) should be larger than B2 and N2.

Figure 1.

Overview of the proposed EVA-I2I model

(A) Integrating domains from different datasets allows training an I2I translation model with strong generalization.

(B) Datasets used in this work, where datasets for external validation are in red.

(C) The proposed EVA-I2I model transforms the input image to different domains, given prompts. Note that the reference image is an example of the given prompt and is not used for inference.

Recent one-to-one methods^16,17 have made progress in medical I2I but are difficult to apply to multi-domain scenarios due to the geometrical increase in the number of required models and training time as the number of domains grows. For natural images, ComboGAN¹⁸ sets up separate encoders, decoders, and discriminators for each domain, which achieves multi-domain I2I in one training process, but the number of models still grows linearly with the increasing number of domains. MUNIT¹⁹ disentangles images into separate style and texture components and then couples textures with different styles through adaptive instance normalization (AdaIN) to achieve multi-domain I2I. StarGAN²⁰ introduces a multi-branch discriminator and a mapping net to enhance style encoding clustering. Disentangle-based methods can bring diverse synthesis results¹⁰ but have certain limitations, as they require the extraction of style encoding from the target domain during inference. For medical images, Sharma et al.²¹ propose a multi-channel input and output generator that produces various domains using distinct channels. This architecture avoids increasing the number of generators, but it may present challenges when expanding to new domains. Some studies^12,14,22 use one-hot encoding to define the output domain. Chen et al.¹⁴ implement AdaIN to control the generator for synthesizing images of the desired domain, while Han et al.¹² choose dynamic convolutions to achieve this. Referring to disentangle-based models, Jiang et al.²² combine one-hot coding with additional style encoding extracted from the images and use variational auto-encoder (VAE) to constrain the style encoding to match a Gaussian universal prior. Conditional models ensure flexible inference and scalability in modeling, but using one-hot encoding can be challenging in presenting subtle semantic relationships between domains. Therefore, we propose using prompts to define distinct domains while controlling the image synthesis process with contrastive language-image pre-training (CLIP) encoding. Large-scale vision-language models, e.g., CLIP, present widespread success in semantic understanding^23,24 and have been applied to medical image analysis. Several studies^25,26 have explored the correlation between chest X-ray (CXR) and its corresponding medical report, attaining zero-shot classification capabilities with image-report contrastive learning. Qin et al.²⁷ create medical prompts to guide the vision-language model for object recognition and detection, showing success in zero/few-shot detection in the medical imaging field. Liu et al.²⁸ propose using CLIP to help the model understand the relationship between different anatomical structures, resulting in the successful segmentation of desired targets from computed tomography (CT) images. Zhao et al.²⁹ propose a joint-learning-based foundation model that determines different tasks through a textual description. For multi-domain I2I, CLIP has been incorporated with various vision generative models^30,31,32 but rarely in the medical domain.³³

In this work, we propose a virtual sequence with an “every domain all at once” I2I (EVA-I2I) translation model, which utilizes natural language to describe and identify different domains and uses DICOM-tag-informed CLIP (DCLIP) encoding to drive I2I synthesis. Specifically, we generate a corresponding prompt (text description) for a scan, which includes organ, modality, sequence, etc. We propose combining different datasets into one by pairing a scan and its prompt as the smallest unit, which can cluster domains and differentiate similar ones based on semantics. Encoding these prompts can guide the model to generate corresponding scans. Due to the characteristics of natural language, DCLIP encoding retains dense semantic relationships and is extensible. Compared to prior methods, our model offers three key advantages: (1) prompt-based representations allow flexible, scalable domain specification beyond fixed one-hot codes; (2) DCLIP enables aligned image-text embedding for semantically driven synthesis; and (3) the model supports translation across heterogeneous datasets without explicit domain labels. Additionally, as shown in Figure 1C, our method can learn and present all seen domains in the dataset simultaneously. Some of these domains have similar appearances, and subtle differences can be effectively distinguished through prompts and reflected in the generated images.

Results

Performance of in-domain and out-domain MRI I2I translation

As illustrated in Figure S1, our EVA-I2I consists of three main components: a text encoder, a structure encoder, and a conditional decoder. The text encoder embeds the medical prompts, which are composed of organ type, modality, sequence, view, etc., using DCLIP to produce a semantic representation aligned with the target domain. The structure encoder extracts anatomical features from the input image, which are finally rendered to the target image by the decoder according to the given domain representation. More details of the model components and training procedure are provided in the STAR Methods.

We compare the image translation performance of our EVA-I2I with that of other multi-domain I2I methods, such as MUNIT,¹⁹ StarGAN v.2,²⁰ Chen et al.,¹⁴ and Seq2Seq.¹² Table 1 illustrates the peak signal-noise rate (PSNR), structural similarity index measure (SSIM), and learned perceptual image patch similarity (LPIPS) of translated images generated by each method on the in- and out-domain datasets. All the comparison methods are trained on the training cohort of the in-domain datasets. Except for providing an input image when inferring, MUNIT and StarGAN v.2 need a reference image to extract style encoding from the target domain, and Chen et al., Seq2Seq, and the proposed EVA-I2I require an encoding (one hot or DCLIP) of the target domain to control the synthesis. Figure 2 shows the axial visualization of synthesized images generated by comparison methods. Results show that our EVA-I2I achieves significantly ($p<0.05$ for PNSR and LPIPS) better synthesis performance and more closely resembles the target style than other comparisons on both in- and out-domain datasets. To assess the computational efficiency of each method, we report the number of giga floating-point operations per second (GFLOPs) and the average inference time per slice in Table 1. Our method achieves moderate GFLOPs and inference times compared to other image translation models. Notably, the inference times of all compared methods are within a similar range and are negligible relative to the typical duration of multi-sequence MRI scanning (30–60 min). This suggests that our approach, along with the others, can be seamlessly integrated into clinical workflows without introducing much latency. More quantitative and visualization results are provided in Table S3 and Figures S2–S5. Furthermore, we evaluated the semantic consistency between the input and translated images using the MedSAM segmentation model.³⁴ Specifically, MedSAM was applied independently to both the source and generated images, and the overlap between the resulting segmentation masks was measured. As shown in Figure S7, the images generated by our method yield the highest segmentation consistency with the input images, indicating that our approach preserves structural information and achieves strong semantic alignment across domains.

Table 1.

The quantitative results of image translation for different comparisons on in- and out-domain datasets

Method	In-domain datasets (testing cohort)			Out-domain datasets			GFLOPS	Infer time (s)
	PSNR (dB) ↑	SSIM ↑	LPIPS ↓	PSNR (dB) ↑	SSIM ↑	LPIPS ↓
MUNIT19	22.0 ± 2.9	0.805 ± 0.102	28.1 ± 16.5	20.4 ± 2.8	0.880 ± 0.014	17.2 ± 2.2	152.1	0.212
StarGAN v.220	22.5 ± 2.9	0.793 ± 0.125	32.0 ± 16.6	21.9 ± 2.2	0.887 ± 0.014	19.7 ± 1.8	90.7	0.075
Chen et al.14	20.8 ± 3.4	0.799 ± 0.096	29.8 ± 17.1	23.3 ± 1.5	0.930 ± 0.017	10.7 ± 1.7	266.5	0.062
Seq2Seq12	21.8 ± 4.2	0.806 ± 0.136	26.2 ± 19.1	22.7 ± 2.4	0.899 ± 0.039	10.7 ± 1.9	296.4	0.078
EVA-I2I (ours)	23.6 ± 3.6	0.807 ± 0.151	21.1 ± 15.7	24.3 ± 1.7	0.931 ± 0.017	9.54 ± 1.71	214.6	0.109

The best results are in bold.

Figure 2.

Axial visualization of translated images generated by comparison methods

Cases 1–5 from the in-domain datasets show the transformation from T1 to T2, susceptibility weighted imaging (SWI) to T1, CT to T1, non-fat saturated to fat saturated, and out of phase to in phase. Case 6 and case 7 from the out-domain datasets show the transformation from T2 to T1 and post-contrast dynamic contrast-enhanced (DCE) to pre-contrast DCE. Our EVA-I2I attains better quality and a style that resembles the target image.

Table 2 illustrates the quantitative results of the ablation study on domain encoding, model backbone, and training loss function. We compare three domain encodings to equip our EVA-I2I, including one hot,¹² BioBERT,³⁵ and DCLIP (proposed). BioBERT encoding can extract semantic information from medical prompts, leading to better synthesis results than one-hot encoding, and our proposed DCLIP encoding aligns text and images, making its synthesis performance significantly ($p<0.05$ for PNSR and LPIPS) better than BioBERT encoding, which cannot align text and images. We also compare three model backbones of the structure encoder ${\mathrm{E}}_{\mathrm{\Phi }}$ in EVA-I2I, including ResNet,³⁶ Swin-Transformer,³⁷ and ConvNeXt (proposed). The ConvNeXt backbone slightly outperforms other methods. Moreover, we also evaluate the effectiveness of cross-dataset collaborative learning. Results show that the model trained with ${\mathcal{L}}_{\text{fine}}$ significantly ($p<0.05$ for PNSR and LPIPS) outperforms the model only trained with ${\mathcal{L}}_{\text{init}}$.

Table 2.

The quantitative results of ablation study for EVA-I2I on domain encoding, model backbone, and training loss function

Ablation	Method	In-domain datasets (testing cohort)			Out-domain datasets
		PSNR (dB) ↑	SSIM ↑	LPIPS ↓	PSNR (dB) ↑	SSIM ↑	LPIPS ↓
Encoding	one hot	21.9 ± 4.0	0.806 ± 0.139	26.2 ± 18.9	22.8 ± 2.3	0.901 ± 0.028	10.7 ± 1.9
	BioBERT	22.2 ± 2.9	0.806 ± 0.121	24.7 ± 17.5	23.5 ± 1.8	0.920 ± 0.019	10.4 ± 1.7
	DCLIP (ours)	23.6 ± 3.6	0.807 ± 0.151	21.1 ± 15.7	24.3 ± 1.7	0.931 ± 0.017	9.54 ± 1.71
Backbone	ResNet	23.4 ± 3.6	0.807 ± 0.152	21.3 ± 15.9	24.1 ± 1.7	0.930 ± 0.017	9.86 ± 1.73
	Swin-Transformer	23.5 ± 3.6	0.806 ± 0.151	21.3 ± 15.9	24.3 ± 1.7	0.929 ± 0.017	9.80 ± 1.73
	ConvNeXt (ours)	23.6 ± 3.6	0.807 ± 0.151	21.1 ± 15.7	24.3 ± 1.7	0.931 ± 0.017	9.54 ± 1.71
Loss	EVA-I2I-${\mathcal{L}}_{\text{init}}$	23.2 ± 3.7	0.805 ± 0.153	22.8 ± 16.4	23.8 ± 1.8	0.926 ± 0.016	10.2 ± 1.7
	EVA-I2I-${\mathcal{L}}_{\text{fine}}$	23.6 ± 3.6	0.807 ± 0.151	21.1 ± 15.7	24.3 ± 1.7	0.931 ± 0.017	9.54 ± 1.71

The best results are in bold.

Training with multiple datasets increases model generalization

We compare the generalization performance of models trained on a single dataset and models trained on multiple datasets. For the single-dataset training experiments, we train Pix2Pix³⁸ for T1-T2 brain MRI translation tasks on BraTS2021, IXI, and OASIS-3, respectively, and then directly test the model on LEMON and HCP Retest datasets. For our proposed EVA-I2I and other multi-domain I2I translation models, we utilize multiple in-domain datasets for training. Tables S4 and S5 show that Pix2Pix trained with BraTS2021 achieves better results than those trained with IXI and OASIS-3 because BraTS2021 has brain MRI with skull removal, which is the same in the LEMON and HCP Retest datasets. The compared multi-domain I2I translation methods are inferior to Pix2Pix because they fail to manage the domain semantics between multiple datasets. In contrast, our method can capture the subtle differences between the semantics of different domains, resulting in better generalization performance than Pix2Pix.

Fine-grained control of synthesis images given by the prompt

Text offers better control over details than one-hot encoding. As shown in Figure 3, modifying the modality or MRI sequence in the prompt can bring the most intuitive changes to the image appearance, followed by some imaging techniques, such as fat saturation, that will bring subtle changes in the image appearance. Other information, such as devices and scanning parameters, has little impact on the appearance. This may be because the latter causes smaller appearance changes than the former, and relevant information is often missing from the training data. Furthermore, the description of the time points can control the intensity enhancement produced by the contrast agent in DCE-MRI. If time-point descriptions are missing, the model tends to generate average intensities across multiple time points. Our EVA-I2I can control subtle changes in the generated image by altering the input prompt, which is not achievable by other methods unless using well-designed one-hot encoding to encode each keyword.

Figure 3.

Axial visualization of the impact of changing prompts on generated images

In case 1, altering the sequence (prompt 1) results in a noticeable change in the appearance of the image. Including additional scan information in prompt 2 has a minimal impact on the appearance. Fat saturation (prompt 3) further suppresses the intensity of fat tissue under the skin, thus rendering the generated image more consistent with the reference image. In case 2, when the specific time point of DCE-MRI is not specified (prompt 1), the intensity enhancement of the cardiac and tumor (red circle) is between the time points before contrast (prompt 2) and the first post-contrast (prompt 3). Note that the proposed model requires only the input image and the target prompt. The reference image is not used as an input but is provided solely for comparison. It corresponds to the given prompt but is from a different subject because the input case lacks the corresponding image of the reference prompt.

Translate images from never-before-seen domains

We further explore the potential of zero-shot I2I translation using our EVA-I2I, i.e., translating images from never-before-seen domains to a seen domain. Figure S6 shows the translation from wash-in (subtraction between post- and pre-contrast DCE) to T1 images of the in-house dataset and from T1Gd (with fat saturation and remaining skulls) to T2 images of the crossMoDA22 dataset. The wash-in and T1Gd images are never-before-seen domains in the in-domain datasets. StarGAN v.2 outperforms other comparison methods because it acquires style information from the target domain. Our EVA-I2I surpasses StarGAN v.2 in terms of structure details, and it does not require style encoding, only medical prompts.

Visualization of prompt embedding and prompt-based zero-shot classification

We show the t-distributed stochastic neighbor embedding (t-SNE) visualization of prompt embedding space on different brain MRI sequences for BioBERT³⁵ encoding and DCLIP encoding in Figure 4. DCLIP encoding highlights a more cohesive feature cluster, effectively conveying domain semantics. Specifically, for DCLIP encoding, T1 and T1Gd (T1 after contrast injection) are closer in the prompt embedding space and far from other sequences. In contrast, BioBERT does not maintain a clear separation between different sequences in the embedding space. It is shown that DCLIP-based encoding enables the model to capture complex domain relationships and learn sophisticated and structured feature embedding.

Figure 4.

Visualization of prompt embedding

(A) The t-SNE visualization of prompt embedding space of BioBERT encoding.

(B) The t-SNE visualization of prompt embedding space of DCLIP encoding.

(C) Bar chart of zero-shot classification on medical imaging characters.

DCLIP-based encoding can effectively characterize prompt embeddings and also help establish a relationship between prompt and image. To support this, we experiment to evaluate the zero-shot classification ability of DCLIP encoding on the in-domain datasets. Specifically, we use images to match their most relevant prompts, i.e., the item with the largest cosine similarity, to classify some image characteristics, e.g., organ, the plane of view, modality, sequence, manufacturer, etc. As a comparison, we train a ResNet50³⁶ model using these given characteristics as labels in a supervised pattern. Figure 4 shows the image classification results. Without additional annotations, the classification performance of DCLIP encoding has exceeded the supervised learning-based method.

Downstream clinical tasks: Brain MRI registration

As shown in Figure 5A and Table S6, we utilized out-domain datasets to build cross-modality brain MRI registration experiments. The Dice coefficient similarity (DSC) for VoxelMorph (VM) is improved from 0.574 on Rigid to 0.837 on Learn2Reg-OASIS but drops a lot more on HCP Retest and LEMON datasets than that of Learn2Reg-OASIS (0.837) due to never-before-seen domains during training. Transforming T2 to T1 by DA methods can resolve the issue, but this is challenging without training on the source and target domain datasets. We compare different DA methods with the proposed EVA-I2I. Except for image modality translation (IMT)⁸ (with fine-tuning), all the DA methods are trained on the in-domain datasets and directly apply T2-to-T1 transformation on the HCP Retest and LEMON datasets. Table S6 shows that EVA-I2I achieves a better DSC than other DA methods.

Figure 5.

Performance of external domain adaptation

(A) The bar chart of brain MRI registration on the HCP Retest dataset.

(B) The bar chart of vestibular schwannoma segmentation on the crossMoDA22 dataset.

(C) The receiver operating characteristic (ROC) curve of the neoadjuvant therapy (NAT) response prediction on the in-house dataset. Data are represented as mean ± SEM.

Downstream clinical tasks: Vestibular schwannoma segmentation

Figure 5B and Table S7 illustrate the external DA performance of our method on vestibular schwannoma segmentation. Directly applying the segmentation model to another domain leads to impairment of the segmentation performance. The results show that the proposed EVA-I2I extremely enhances the segmentation performance of nnU-Net, whose improvement outperforms other DA methods, showing the potential of zero-shot DA. The segmentation performance of the DA methods for small vestibular schwannoma (VS) is hard to compare with MSF-Net because DA methods are trained with an image spacing of $1\times 1$ $\times$ 1 mm, while MSF-Net is trained with an image spacing of $0.412\times 0.412$ $\times$ 1 mm, showing better VS structure. This is a limitation of EVA-I2I: it is trained with fixed image spacing and cannot adapt to tasks needing smaller image spacing.

Downstream clinical tasks: Breast cancer pCR early prediction

Figure 5C shows the external DA performance of our method on breast cancer pathological complete response (pCR) early prediction. The performance of ResNet18 (DCE + wash-in) is severely degraded compared to ResNet18 (wash-in), which makes sense because non-fat-saturated (FS) DCEs have more noticeable differences in appearance than FS DCEs. After employing different DA methods, the predicted area under the ROC curve (AUC) has improved. The proposed EVA-I2I achieves the most apparent improvement with AUC and exceeds the performance of ResNet18 (wash-in) by leveraging breast tissue information in DCEs.

Discussion

In this work, we introduce medical prompts to define and identify medical images in distinct domains and further propose EVA-I2I to achieve multi-domain I2I translation via the guide of DCLIP encoding of text descriptions of target domains. Medical prompts have advantages over one-hot encoding: (1) they can integrate multiple datasets and avoid conflicts between different domains and (2) they contain rich semantic information, which can effectively differentiate between inner-sequence and inter-sequence imaging differences. Table S4 illustrates that training models with multiple datasets may not always enhance synthesis performance. One-hot encoding fails to address domain conflicts that may arise when combining diverse datasets, while using medical prompts has been proven to address this issue and result in improved synthesis performance compared to using a single dataset. As shown in Figure 3, adding the keyword “fat saturation” and a description of the time point in the prompt can distinguish Turbo Spin Echo (TSE)-T2 with different appearances and DCE-MRI with different intensity enhancement of tumors, respectively. Achieving equivalent capabilities with one-hot encoding requires a more intricate design and is hard to broaden. Moreover, we propose using DCLIP for image-text alignment rather than encoding text with an unaligned text encoder such as BioBERT. Table 2 demonstrates that our EVA-I2I, guided by DCLIP encoding, produces better synthesis results than the one guided by BioBERT encoding. This is because DCLIP encoding can more effectively capture the semantic relationship between domains after aligning text and image. As shown in Figure 4, this semantic relationship is related to the difference in image appearance and cannot be represented by BioBERT encoding. Furthermore, the proposed cross-dataset collaborative learning approach can establish a connection between distinct datasets and achieve I2I translation across datasets. This effectively broadens the range of target domains that our model can generate, as shown in Figure 1, and improves the synthesis performance, as presented in Table 2.

Our EVA-I2I demonstrates strong generalization performance with the help of multi-dataset training and the guidance of DCLIP encoding for medical prompts. Table 1 and Figure 2 illustrate that the proposed model can achieve better synthesis performance on external validation than other comparisons. Although it may seem counterintuitive that performance on some out-domain datasets surpasses that on in-domain datasets, this can be explained by differences in organ type and synthesis difficulty. For example, HCP (out-domain) contains skull-stripped brain MRIs that resemble BraTS2021 (in-domain), leading to comparable results. In contrast, CHAOS (in-domain) involves abdominal MRIs with greater anatomical complexity, resulting in lower synthesis quality. As a result, the average in-domain metrics may appear lower, but method comparisons within each dataset remain consistent and valid. Figure S6 further demonstrates the zero-shot I2I translation capability of the proposed model, which can translate never-before-seen domains in the training datasets into seen domains. It dramatically expands the scope of application for our EVA-I2I to include tasks that cannot be accomplished by the compared models. Moreover, the generalization of our EVA-I2I is also reflected in downstream applications, which is also the purpose of applying DA in clinical practice. Figure 5 and Tables S6 and S7 show the DA performance of our model for cross-modality registration, classification, and segmentation on external datasets, respectively. Many studies on DA have shown that in order to achieve good results, the model needs to be fine-tuned on the target domain. However, our approach does not require fine-tuning and can directly yield good results on external validation datasets. Even in the case of zero-shot DA, our method can achieve performance that is comparable to the fine-tuned approach.

In this paper, we propose a DCLIP-driven model for medical I2I translation. We integrate DCLIP encoding into a conditional synthesis model to establish semantic relationships between the description and appearance of multi-domain images, resulting in a flexible and powerful model. The model effectively learns from diverse public datasets and identifies shared information between medical images, achieving synthesis accuracy surpassing other DA methods on in- and out-domain datasets. Moreover, downstream applications demonstrate that the EVA-I2I can be utilized for DA on external datasets without fine-tuning, highlighting its potential to enhance the generalization of existing deep learning models.

Limitations of the study

This study also has some limitations. First, the proposed model has a fixed image resolution for training, limiting its ability to perform well in higher-resolution situations when performing downstream applications. Moreover, our model is only capable of processing 2D slices. Although we have incorporated the plane of view in the medical prompts to enable our model to make inferences from different views, it still struggles to comprehend complex 3D structures efficiently, which may lead to inter-slice discontinuities, particularly during zero-shot I2I translation tasks. Furthermore, the out-domain datasets do not cover the same organs or imaging domains as the in-domain datasets, which may lead to inconsistencies and hinder direct comparability of performance metrics between the in-domain and out-domain datasets.

Resource availability

Lead contact

Requests for further information and resources should be directed to and will be fulfilled by the lead contact, Tao Tan (taotan@mpu.edu.mo).

Materials availability

This study did not generate new unique materials or reagents.

Data and code availability

• •This paper analyzes existing, publicly available data. Accession numbers or web links for the public available datasets are listed in the key resources table.
• •All source codes are publicly available at https://github.com/fiy2W/eva_i2i. An archival DOI is listed in the key resources table.
• •Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.

Acknowledgments

L.H. was funded by a Chinese Scholarship Council (CSC) scholarship (no. 202006240065). This work is supported by Science and Technology Development Fund of Macao (0105/2022/A) and a Macao Polytechnic University grant (RP/FCA-08/2024). The authors would like to acknowledge the Research High Performance Computing (RHPC) facility of the Netherlands Cancer Institute (NKI).

Author contributions

Conceptualization, L.H., T.T., and R.M.; methodology, L.H., Y.H., H.D., and T.T.; data curation, H.L., X.W., Y.G., and T.Z.; formal analysis, L.H., X.W., Y.G., T.Z., H.D., Y.H., C.L., X.L., and R.M.; software, J.T.; project administration, R.M. and T.T.; resources, R.M. and T.T.; writing – original draft, L.H.; writing – review & editing, T.T., Y.S., and S.K.Z.; supervision, T.T. and R.M.

Declaration of interests

The authors declare no competing interests.

STAR★Methods

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Deposited data
AMOS22 dataset	Multi-Modality Abdominal Multi-Organ Segmentation Challenge 2022	https://amos22.grand-challenge.org/
BraTS2021 dataset	Brain Tumor Segmentation Challenge 2021	http://www.braintumorsegmentation.org/
CHAOS dataset	Combined (CT-MR) Healthy Abdominal Organ Segmentation Challenge	https://chaos.grand-challenge.org/
Duke Breast Cancer MRI dataset	The Cancer Imaging Archive	https://www.cancerimagingarchive.net/collection/duke-breast-cancer-mri/
IXI dataset	Human Connectome Project	https://brain-development.org/ixi-dataset/
SynthRAD2023 dataset	Synthesizing computed tomography for radiotherapy challenge	https://synthrad2023.grand-challenge.org
OASIS-3 dataset	Open Access Series of Imaging Studies	https://www.oasis-brains.org/
crossMoDA22 dataset	Cross-Modality Domain Adaptation Challenge	https://crossmoda2022.grand-challenge.org/
Learn2Reg-OASIS dataset	Learn2Reg Challenge	https://learn2reg.grand-challenge.org
LEMON dataset	Max Planck Institut Leipzig Mind-Brain-Body Dataset	https://fcon_1000.projects.nitrc.org/indi/retro/MPI_LEMON.html
HCP Retest dataset	Human Connectome Project	https://db.humanconnectome.org/data/projects/HCP_Retest
Software and algorithms
EVA-I2I	This paper	https://github.com/fiy2W/eva_i2i and https://doi.org/10.5281/zenodo.15815446
pytorch 2.0.0	PyTorch	https://pytorch.org
torchvision 0.16.2	PyTorch	https://pytorch.org
huggingface-hub 0.27.0	Hugging Face	https://huggingface.co
tokenizers 0.15.2	Hugging Face	https://huggingface.co/docs/tokenizers/index
transformers 4.39.3	Hugging Face	https://huggingface.co/docs/transformers/index
numpy 1.26.3	NumPy	https://numpy.org/
scipy 1.11.4	SciPy	https://scipy.org/
scikit-learn 1.4.1	scikit-learn	https://scikit-learn.org/stable/
scikit-image 0.22.0	scikit-image	https://scikit-image.org/
SimpleITK 2.3.1	SimpleITK	https://simpleitk.org/
timm 0.9.12	timm	https://github.com/huggingface/pytorch-image-models
lpips 0.1.4	LPIPS	https://github.com/richzhang/PerceptualSimilarity

Experimental model and study participant details

Data collection

Table S1 illustrates the detailed information of datasets used in this work, divided into in- and out-domain datasets. In-domain datasets (D01-D07) comprise 7 public datasets with 27,950 brain, breast, abdomen, or pelvis scans (3D volumes). They are randomly split into training, validation, and independent testing cohorts with a ratio of 3:1:1 based on patients. Out-domain datasets include 4 public datasets (D08-D11) and an in-house dataset (D12) aiming for extra external validation and downstream applications. Within each sub-dataset, multi-domain cases are pre-aligned. In contrast, alignment across different datasets is not required. The in-house dataset includes 2,101 breast cancer patients treated with neoadjuvant therapy (NAT) between 2000 and 2020 at the Netherlands Cancer Institute (NKI) in Amsterdam, the Netherlands. This retrospective study was approved by the Institutional Review Board of the NKI, with registration number IRBd21-059. All data collection and usage were conducted in compliance with institutional ethical guidelines. The in-house cohort consisted exclusively of female patients, ranging in age from 19 to 82 years, with a median age of 50. The overall pathological complete response (pCR) rate in this cohort was 31%. Each patient has two to four multi-sequence MRI visits to acquire T1, T2, DCE, and diffusion-weighted imaging (DWI). Because of the long inclusion period, the MRI protocol for breast examination has changed, with variations in the scanning techniques used. Before 2015, scanning was acquired using 2D sequences in the coronal plane, while after 2015, it shifted toward 3D scanning in the transverse plane. Furthermore, in recent years, T1 and DCE have been scanned using fat-saturation techniques. DCE is acquired with 6 timepoints within 7.5 min after administering contrast. Additionally, DWI is performed using four different b values of 0, 150, 800, and 1,500. The MRI scans are obtained using Philips Ingenia 3.0-T scanners. We also generate the corresponding skull-removed images for brain scans of the in-domain datasets to increase the number of domains. The downstream tasks delineated in this paper are exclusively related to MRI, thus, CT datasets are not considered a part of the out-domain datasets. Images for the brain and breast are resampled to $1\times 1$ $\times$ 1 mm, and images of the abdomen and pelvis are resampled to $2.5\times 1$ $\times$ 1 mm.

Method details

Medical prompt for domain description

With available information from scans, we design a template to build medical prompts (e.g., “A T2-weighted magnetic resonance imaging of the brain for the subject …”), which can describe and identify the corresponding scan. Table S2 illustrates the tags and keywords we extract from DICOM files. For datasets in which DICOM files are not provided, we manually collect relevant information from the summary Excel files provided by the corresponding public dataset. Besides these keywords from DICOM, we additionally define [Organ] token for organ (e.g., brain, breast, abdomen, and pelvis), [Modality] token for modality (e.g., magnetic resonance imaging, computed tomography, and cone beam computed tomography), [Sequence] token for MRI sequence (e.g., T1-weighted, T2-weighted, T2 star-weighted, and T2-weighted-fluid-attenuated inversion recovery), [SequenceFrameNumber] token for the numbers of the multi-timepoint sequences (e.g., first, second, and third), [Contrast] token for contrast agent (e.g., Gadolinium), [Preprocess] token for image preprocess (e.g., skull removal), and [Plane] token for the plane of view (e.g., axial, coronal, and sagittal) with the help of radiologists. We generate the medical prompt automatically based on the above keywords to describe the corresponding scan. Referred from,²⁸ we define the medical prompt template for MRI as follows,

“A [Sequence] [Modality] of [Organ] for the subject. The subject undergoes the contrast-enhanced scan with the [Contrast] agent. The scan is acquired on the [Manufacter] [ManufacturerModelName] [MagneticFieldStrength] Tesla scanner. The scanning sequences consist of [ScanningSequence]. The sequence variants consist of [SequenceVariant]. The scan options consist of [ScanOptions]. The image is acquired using a [FlipAngle] degree flip angle, an echo time of [EchoTime] ms, a repetition time of [RepetitionTime] ms, an inversion time of [InversionTime] ms. The image preprocessing consists of [Preprocess]. The image is visualized in the [Plane] plane.”

The medical prompt template will be slightly adjusted when some keywords are unavailable. Prompt augmentation can enhance generalization by reducing reliance on fixed templates, and strengthen contextual understanding and language flexibility. We randomly delete some sentences (but preserve essential sentences extremely influencing the image style, e.g., involving information on organ, modality, and sequence) during the training process to achieve data augmentation. We initialize a prompt as a list of sentence dictionaries, which includes the content of the sentence and the mask of the sentence. If the sentence is essential, the mask is set to 1. During augmentation, the essential sentences are preserved, and others are randomly deleted. Moreover, a random shuffle is applied to the selected sentences in each training step.

DCLIP encoding with text and image encoders

To build the domain representation ${\mathrm{\Theta }}_s$ with prompt, we apply DCLIP to encode the text because it can learn a well-aligned space between images and texts. As shown in Figure S1A, we directly adapt text encoder ${\mathbf{E}}_T$ from CLIP²³ and improve the image encoder ${\mathbf{E}}_I$ with ConvNeXt³⁹ backbone for better representation. We improve the pre-training referring to supervised contrastive learning⁴⁰ because some prompts can correspond to multiple images, while the loss in²³ cannot maximize the similarity between these samples.

The text encoder ${\mathbf{E}}_T$ is a 12-layer Transformer with a width of 768 and 12 attention heads, adapted from CLIP.²³ This transformer operates on a lower-cased byte pair encoding (BPE) representation of the text with a vocabulary size of 49,152. The maximum content length is limited to 256 to handle the long medical prompt. Furthermore, the text sequence is enveloped with [SOS] and [EOS] tokens, and the feature representation of the text is obtained by treating the activations of the highest layer of the transformer at the [EOS] token. The feature representation is then layer-normalized and linearly projected into the domain encoding space as the domain representation ${\mathrm{\Theta }}_s$.

The image encoder ${\mathbf{E}}_I$ is improved from the ResNet-based architecture of CLIP,²³ whose backbone is replaced with ConvNeXt-T³⁹, making it more efficient, scalable, and accurate. ${\mathbf{E}}_I$ commences with a convolutional layer featuring a kernel size of $7\times 7$, a stride of $1\times 1$, and an output channel of 96. It is then followed by ConvNeXt-T, which comprises four stages exhibiting channel numbers of 96, 192, 384, and 768, block numbers of 3, 3, 9, and 3, and strides of 4, 2, 2, and 2, respectively. The global average pooling layer is substituted with an attention pooling mechanism, a single Transformer-style multi-head QKV attention layer with the image global average-pooled conditioned query.

Since different subjects may have performed the same scan, i.e., one prompt corresponds to multiple images, we improve the DCLIP loss ${\mathcal{L}}_{\text{CLIP}}$ referring the supervised contrast loss⁴⁰ so that it can reduce the distance between these samples. For image $X$ and prompt $P$ pairs from a mini-batch of the dataset $\mathcal{D}$, we define ${\mathcal{L}}_{\text{DCLIP}}$ as follows,

$${\mathcal{L}}_{\text{DCLIP}}=-\sum _{i\in \mathcal{D}}\left(\frac{1}{\left|\mathcal{M}\left(i,·\right)\right|}\sum _{m\in \mathcal{M}\left(i,·\right)}\log \frac{\exp \left(z_T^{\left(i\right)}·z_I^{\left(m\right)}/\tau \right)}{{\sum }_{j\in \mathcal{D}}\exp \left(z_T^{\left(i\right)}·z_I^{\left(j\right)}/\tau \right)}+\frac{1}{\left|\mathcal{M}\left(·,i\right)\right|}\sum _{m\in \mathcal{M}\left(·,i\right)}\log \frac{\exp \left(z_I^{\left(i\right)}·z_T^{\left(m\right)}/\tau \right)}{{\sum }_{j\in \mathcal{D}}\exp \left(z_I^{\left(i\right)}·z_T^{\left(j\right)}/\tau \right)}\right)$$ (Equation 1)

where $z_I$ and $z_T$ are projections of the image representation ${\mathbf{E}}_I\left(X\right)$ and the text representation ${\mathbf{E}}_T\left(P\right)$, $\mathcal{M}$ indicates the label of the confusion matrix whose value is 1 when the prompt matches the image, and 0 for the rest. Note that, one prompt could match multiple images. $\mathcal{M}\left(i,·\right)$ presents items with the label of 1 from the $i$ th row of $\mathcal{M}$, $\mathcal{M}\left(·,i\right)$ presents items with the label of 1 from the $i$ th column of $\mathcal{M}$, $\tau =0.07$ refers to the scalar temperature parameter. To construct $\mathcal{M}$, we compare each image’s original prompt with each candidate text prompt: a match is established if all sentences in the candidate prompt appear in the original prompt, and all necessary components of the original prompt are present in the candidate. This ensures robust matching even under text augmentation and partial paraphrasing.

DCLIP-driven I2I translation model

As shown in Figure S1B, the architecture of the proposed EVA-I2I consists of the text encoder ${\mathbf{E}}_T$, a structure encoder ${\mathbf{E}}_{\mathrm{\Phi }}$, and a conditional decoder $\mathbf{G}$. First, we use the weight-frozen ${\mathbf{E}}_T$ to convert the prompt into a DCLIP encoding $T$, which is an estimation of ${\mathrm{\Theta }}_s$. Then, using ${\mathbf{E}}_{\mathrm{\Phi }}$, we extract the approximate structure representation $\mathrm{\Phi }$ of the input image. Finally, with $\mathbf{G}$, we can render the structure representation of the image into the corresponding scan using the input DCLIP encoding as a control. ${\mathbf{E}}_{\mathrm{\Phi }}$ and $\mathbf{G}$ have ConvNeXt³⁹ backbone equipped with self-attention blocks, and $\mathbf{G}$ is implemented with dynamic convolutions¹² to embed $T$. Moreover, we develop a DCLIP-driven discriminator to simplify the training process of adversarial learning.

The structure encoder ${\mathbf{E}}_{\mathrm{\Phi }}$ starts with a convolutional layer with a kernel size of $7\times 7$, a stride of $1\times 1$, and an output channel of 96. Then it follows with a two-stage ConvNeXt backbone, exhibiting channel numbers of 192 and 384, block numbers of 1 and 3, and strides of 2 and 2, respectively. In addition, a self-attention block is inserted between each two ResNeXt blocks. The structure of the conditional decoder $\mathbf{G}$ is symmetrical to that of ${\mathbf{E}}_{\mathrm{\Phi }}$, except that the convolutional layers in the ResNeXt blocks are replaced by the HyperConv layers¹² to receive the control of DCLIP encoding. The convolutional layer with a stride of 2 is replaced with a two-times upsampling layer followed by a convolutional layer with a stride of 1. The output of $\mathbf{G}$ is activated with a LeakyReLU layer.

Instead of defining a discriminator for each domain, we use a DCLIP-driven discriminator $\mathbf{D}$, inspired by our generator, to handle all domains effectively. We employ ResNeXt blocks equipped with HyperConv layers¹² in $\mathbf{D}$. Specifically, $\mathbf{D}$ consists of three convolutional layers with a kernel size of 4, a stride of 2, and output channels of 96, 192, and 384, four conditional ResNeXt blocks inserted with self-attention blocks, and one convolutional layer with a kernel size of 1 and output channel of 1.

Inner-subject supervision

Figure S1C shows the training procedure of the proposed EVA-I2I. For paired samples from the same dataset, we can use supervised learning to constrain the synthesized images at the pixel and perceptual levels. The reconstruction loss for each paired source image $X_{\text{src}}$ and target image $X_{\text{tgt}}$ is as follows,

$${\mathcal{L}}_{\text{rec}}={\lambda }_{L1}·\left({\Vert X_{\text{src}}^{\prime }-X_{\text{src}}\Vert }_1+{\Vert X_{\text{tgt}}^{\prime }-X_{\text{tgt}}\Vert }_1\right)+{\lambda }_p·\left({\mathcal{L}}_p\left(X_{\text{src}}^{\prime },X_{\text{src}}\right)+{\mathcal{L}}_p\left(X_{\text{tgt}}^{\prime },X_{\text{tgt}}\right)\right)\text{,}$$ (Equation 2)

where generated target image $X_{\text{tgt}}^{\prime }=\mathbf{G}\left({\mathbf{E}}_{\mathrm{\Phi }}\left(X_{\text{src}}\right)\Vert T_{\text{tgt}}\right)$, generated source image $X_{\text{src}}^{\prime }=\mathbf{G}\left({\mathbf{E}}_{\mathrm{\Phi }}\left(X_{\text{tgt}}\right)\Vert T_{\text{src}}\right)$, $T_{\text{src}}$ and $T_{\text{tgt}}$ are DCLIP encoding for prompts of $X_{\text{src}}$ and $X_{\text{tgt}}$, ${\Vert ·\Vert }_1$ refers to an ${\mathcal{L}}_1$ loss, and ${\mathcal{L}}_p$ indicates the perceptual loss based on pre-trained VGG19. ${\lambda }_{L1}$ and ${\lambda }_p$ are weight terms and experimentally set to be 10 and 0.1.

For paired samples, we also constrain their structure representation to be consistent, and the structure loss is,

$${\mathcal{L}}_{\text{con}}={\Vert {\mathbf{E}}_{\mathrm{\Phi }}\left(X_{\text{src}}\right)-{\mathbf{E}}_{\mathrm{\Phi }}\left(X_{\text{tgt}}\right)\Vert }_2^2+{\mathcal{L}}_{\text{PCL}}\left({\mathbf{E}}_{\mathrm{\Phi }}\left(X_{\text{src}}\right),{\mathbf{E}}_{\mathrm{\Phi }}\left(X_{\text{tgt}}\right)\right)\text{,}$$ (Equation 3)

where $\Vert ·{\Vert }_2^2$ refers to a mean square error (MSE) loss, and ${\mathcal{L}}_{\text{PCL}}$ indicates the contrastive loss at the pixel level. We propose using additional ${\mathcal{L}}_{\text{PCL}}$ to restrict the consistency of image content, as MSE can lead to over-compression on the value range of features. For image $X_{\text{src}},X_{\text{tgt}}\in {\mathbb{R}}^{1\times W\times H}$, ${\mathcal{L}}_{\text{PCL}}$ is defined on the feature pixels located by common foreground mask $M_0=\left(X_{\text{src}}>0\right)\cap \left(X_{\text{tgt}}>0\right)$,

$${\mathcal{L}}_{\text{PCL}}=-\sum _{i\in M}\log \frac{\exp \left(p_i·q_i/\tau \right)}{{\sum }_{j\in M}\exp \left(p_i·q_j/\tau \right)}·\frac{\exp \left(q_i·p_i/\tau \right)}{{\sum }_{j\in M}\exp \left(q_i·p_j/\tau \right)}\text{,}$$ (Equation 4)

where $M\in {\mathbb{Z}}^{1\times \frac{W}{4}\times \frac{H}{4}}$ is four times downsampled from $M_0\in {\mathbb{Z}}^{1\times W\times H}$, $p,q\in {\mathbb{R}}^{C\times \frac{W}{4}\times \frac{H}{4}}$ are normalization of the image content representations ${\mathbf{E}}_{\mathrm{\Phi }}\left(X_{\text{src}}\right)$ and ${\mathbf{E}}_{\mathrm{\Phi }}\left(X_{\text{tgt}}\right)$, $i$ and $j$ are pixel indices, $p_i,q_i,p_j,q_j\in {\mathbb{R}}^C$ are features corresponding to the pixels selected in $M$, and $\tau =0.07$ refers to the scalar temperature parameter.

Cross-dataset collaborative learning

To maximize the utilization of multiple datasets, we propose constraints on unpaired samples between different datasets. Besides paired images $X_{\text{src}}$ and $X_{\text{tgt}}$, we randomly introduce an intermediate image $X_{\mathrm{int}}$ and its corresponding DCLIP encoding $T_{\mathrm{int}}$ from another subject in the mini-batch, and build a triangular consistency loss to establish mappings between domains across datasets. Note that, $X_{\mathrm{int}}$ is traversed from other datasets and for the same organ with $X_{\text{src}}$ and $X_{\text{tgt}}$.

$${\mathcal{L}}_{\text{tri}}={\lambda }_{L1}·{\Vert X_{\mathrm{int}\to \text{tgt}}^{\prime }-X_{\text{tgt}}\Vert }_1+{\lambda }_p·{\mathcal{L}}_p\left(X_{\mathrm{int}\to \text{tgt}}^{\prime },X_{\text{tgt}}\right)\text{,}$$ (Equation 5)

where $X_{\mathrm{int}\to \text{tgt}}^{\prime }=\mathbf{G}\left({\mathbf{E}}_{\mathrm{\Phi }}\left(X_{\mathrm{int}}^{\prime }\right)\Vert T_{\text{tgt}}\right)$, and the generated intermediate image is $X_{\mathrm{int}}^{\prime }=\mathbf{G}\left({\mathbf{E}}_{\mathrm{\Phi }}\left(X_{\text{src}}\right)\Vert T_{\mathrm{int}}\right)$.

To make $\mathbf{G}$ to generate an image with an appearance similar to the intermediate image, we introduce adversarial loss and cosine similarity loss to constrain the image style and semantics, respectively.

$${\min }_{\mathbf{D}}{\max }_{{\mathbf{E}}_{\mathrm{\Phi }},\mathbf{G}}{\mathcal{L}}_{\text{adv}}={\Vert \mathbf{D}\left(X_{\mathrm{int}}\Vert T_{\mathrm{int}}\right)-1\Vert }_2^2+{\Vert \mathbf{D}\left(X_{\mathrm{int}}^{\prime }\Vert T_{\mathrm{int}}\right)\Vert }_2^2\text{,}$$ (Equation 6)

$${\mathcal{L}}_{\cos }=-\frac{⟨{\mathbf{E}}_I\left(X_{\mathrm{int}}^{\prime }\right),T_{\mathrm{int}}⟩}{\Vert {\mathbf{E}}_I\left(X_{\mathrm{int}}^{\prime }\right)\Vert ·\Vert T_{\mathrm{int}}\Vert }\text{,}$$ (Equation 7)

where $\mathbf{D}$ is the DCLIP-driven discriminator, $⟨,⟩$ refers to the inner product, and ${\mathbf{E}}_I$ is frozen during training.

Total loss

We train the generator using only inner-subject supervision in the initial stage and additionally add cross-dataset collaborative learning in the fine-tuning stage. The total loss function in the initial stage and fine-tuning stage of the generative model can be summarized as,

$$\begin{array}{cc}{\mathcal{L}}_{\text{init}}=& {\mathcal{L}}_{\text{rec}}+{\lambda }_{\text{con}}{\mathcal{L}}_{\text{con}}\\ {\mathcal{L}}_{\text{fine}}=& {\mathcal{L}}_{\text{rec}}+{\lambda }_{\text{con}}{\mathcal{L}}_{\text{con}}+{\mathcal{L}}_{\text{tri}}+{\lambda }_{\text{adv}}{\mathcal{L}}_{\text{adv}}+{\lambda }_{\cos }{\mathcal{L}}_{\cos }\text{,}\end{array}$$ (Equation 8)

where we set ${\lambda }_{\text{con}}=1$, ${\lambda }_{\text{adv}}=5$, and ${\lambda }_{\cos }=0.1$ with experimental experience.

Training details

All the models are developed with PyTorch, and experiments are conducted on the NVIDIA Quadro RTX A6000 GPU. The EVA-I2I is trained using the AdamW optimizer with an initial learning rate of ${10}^{-4}$ and an input image size of $256\times 256\text{.}$ We select the best model by evaluating the best PSNR in validation cohorts of in-domain datasets. During training, images are first normalized with $\mathcal{N}\left(X\right)=\left(\max \left(X,R_{\min }\right)-R_{\min }\right)/R_X$, where $R_{\min }$ is set to 0 for MRI and −1024 for CT and CBCT. $R_X$ is 99.5 percentile of $X$ for MRI, 3000 for CT, and 1500 for CBCT, respectively. And then, the normalized images are padded or randomly cropped to $256\times 256\times 256$. Finally, 2D slices are randomly selected in axial, coronal, or sagittal planes from the image volume. DCLIP is adopted with ${\mathcal{L}}_{\text{DCLIP}}$ and a batch size of 60 for 50 epochs on the in-domain datasets. We initialize the proposed EVA-I2I by training 5 epochs on paired scans with ${\mathcal{L}}_{\text{init}}$ and a batch size of 6. Then, we train the model with ${\mathcal{L}}_{\text{fine}}$ and a batch size of 6 for 45 epochs on both paired and unpaired samples.

Downstream clinical tasks: Brain MRI registration

Cross-modality registration is widely used to align anatomical structures between different domains of medical images. Three public datasets from the out-domain datasets are utilized to build the brain image registration experiments. Learn2Reg-OASIS⁴¹ (T1-to-T1 mono-modality registration) is split into three cohorts, with 350 subjects for training the learning-based registration model (VoxelMorph⁴²), 14 for validation, and 50 for testing. HCP Retest and LEMON (T2-to-T1 cross-modality registration) are utilized for external validation. All the deformation registration is based on rigid registration results. VM is only trained on the Learn2Reg-OASIS dataset and is directly tested on HCP Retest and LEMON datasets.

Downstream clinical tasks: Vestibular schwannoma segmentation

Cross-modality segmentation is a common clinical task that migrates knowledge to other domains, relieving the pressure of data annotation. We conduct the experiment on the crossMoDA22^43,44 dataset for segmenting vestibular schwannoma (VS). Specifically, we train the segmentation model based on 227 annotated T1Gd (never-before-seen domain) images and test the model on another 96 unpaired T2 images. MSF-Net⁴⁵ trains DA between the source and target domains as an upper bound for comparisons. We train the nnU-Net⁴⁶ as a baseline model with annotated T1Gd images. We use different DA methods to transform T1Gd images into fake T2 images to re-train the nnU-Net. All the DA methods are trained with the in-domain datasets, which makes this task a zero-shot DA due to the never-before-seen T1Gd for VS in the training sets.

Downstream clinical tasks: Breast cancer pCR early prediction

Classification models have generalization problems when dealing with cross-domain or multi-center data. In the in-house dataset, the DCE protocol for breast examination has changed due to the long inclusion period. The early DCEs do not apply fat saturation (FS) techniques and are acquired using 2D sequences in the coronal plane. In contrast, the recent DCEs apply FS and shift toward 3D scanning in the transverse plane. This can lead to significant differences in image appearance. The wash-in image is a subtraction between post-contrast and pre-contrast DCE, which can highlight the tumor region and is not affected by FS but loses the context of breast tissue.¹² Here, we conduct an experiment on the in-house dataset that predicts the response to neoadjuvant therapy (NAT) for breast cancer patients based on pre-NAT DCE-MRI. Specifically, we train a ResNet18⁴⁷ model on the combination of FS DCEs and wash-in images from the in-house dataset and test the classification performance of the model on the combination of non-FS DCEs and corresponding wash-in images. We also train ResNet18 only on wash-in images as a benchmark for the prediction performance.

Quantification and statistical analysis

Statistical analysis

Statistical analysis was done using Python 3.11.3. All statistical tests were conducted with a two-sided T-test, and $p<0.05$ indicated statistical significance. The assessment of translation performance is quantified through peak signal noise rate (PSNR), structural similarity index measure (SSIM), and learned perceptual image patch similarity (LPIPS⁴⁸). Higher PSNR indicates better image fidelity, higher SSIM reflects better contrast and structural information, and lower LPIPS implies better human visual perception. Downstream applications involve registration, classification, and segmentation tasks. Dice coefficient similarity (DSC) is utilized to evaluate image registration quality, which measures spatial overlap between anatomical regions. The area under the ROC curve (AUC) is used to evaluate the classification performance. DSC and average symmetric surface distance (ASSD) are used to assess the segmentation performance on area and boundary overlapping, respectively. All calculation methods are as follows:

$$\text{PSNR}=10\times {\log }_{10}\frac{R^2}{{\Vert X^{\prime }-X\Vert }_2^2}$$ (Equation 9)

where $R=1$ presents the image intensity range. $X^{\prime }$ is the synthesized image of comparison methods, and $X$ is the ground truth.

$$\text{SSIM}\left(x,y\right)=\frac{\left(2{\mu }_x{\mu }_y+c_1\right)\left(2{\sigma }_{xy}+c_2\right)}{\left({\mu }_x^2+{\mu }_y^2+c_1\right)\left({\sigma }_x^2+{\sigma }_y^2+c_2\right)}$$ (Equation 10)

where $x$ and $y$ are windows of $X^{\prime }$ and $X$, ${\mu }_x$ and ${\mu }_y$ are the pixel sample mean of $x$ and $y$, ${\sigma }_x^2$ and ${\sigma }_y^2$ are the variance of $x$ and $y$, ${\sigma }_{xy}$ is the covariance of $x$ and $y$, $c_1=0.0001$ and $c_2=0.0009$ are set due to the image intensity range of 1.

$$\text{LPIPS}=\sum _l{\Vert {\mathcal{F}}_l\left(X^{\prime }\right)-{\mathcal{F}}_l\left(X\right)\Vert }_2^2$$ (Equation 11)

where $\mathcal{F}$ is the pre-trained AlexNet,⁴⁹ ${\mathcal{F}}_l(·)$ indicates the features of the $l$ th layer of $\mathcal{F}$.

$$\text{DSC}=\frac{2\left|M^{\prime }\cap M\right|}{\left|M^{\prime }\right|+\left|M\right|}$$ (Equation 12)

where $|·|$ is the cardinality of the set.

$$\text{ASSD}=\frac{{\sum }_{x\in \partial M}\mathbf{d}\left(x,\partial M^{\prime }\right)+{\sum }_{y\in \partial M^{\prime }}\mathbf{d}\left(y,\partial M\right)}{\left|\partial M^{\prime }\right|+\left|\partial M\right|}$$ (Equation 13)

where $\partial M^{\prime }$ and $\partial M$ are the contour sets of $M^{\prime }$ and $M$, $\mathbf{d}\left(·,·\right)$ indicates the minimum distance between a point and a contour.

$$\text{AUC}=\frac{{\sum }_{in{s}_i\in pos}\mathit{ran}{k}_{in{s}_i}-\frac{P\times \left(P+1\right)}{2}}{P\times N}$$ (Equation 14)

where $P$ and $N$ are the number of positive and negative samples. $\mathit{ran}{k}_{in{s}_i}$ refers to the serial number of sample $i$. $\sum _{in{s}_i\in pos}\mathit{ran}{k}_{in{s}_i}$ indicates the sum of serial numbers of the positive samples.

Published: August 11, 2025