On the Utility of Foundation Models for Fast MRI: Vision-Language-Guided Image Reconstruction

Abstract

Purpose:

To investigate whether a vision-language foundation model can enhance undersampled MRI reconstruction by providing high-level contextual information beyond conventional priors.

Theory and Methods:

We proposed a semantic distribution-guided reconstruction framework that uses a pre-trained vision-language foundation model to encode both the reconstructed image and auxiliary information into high-level semantic features. A contrastive objective aligns the reconstructed representation with the target semantic distribution, ensuring consistency with high-level perceptual cues. The proposed objective works with various deep learning-based reconstruction methods and can flexibly incorporate semantic priors from multimodal sources. To test the effectiveness of these semantic priors, we evaluated reconstruction results guided by priors derived from either image-only or image-language auxiliary information.

Results:

Experiments on knee and brain datasets demonstrate that semantic priors from images preserve fine anatomical structures and achieve superior perceptual quality, as reflected in lower LPIPS values, higher Tenengrad scores, and improved scores in the reader study, compared with conventional regularization. The image-language information further expands the semantic distribution and enables high-level control over reconstruction attributes. Across all evaluations, the contrastive objective consistently guided the reconstructed features toward the desired semantic distributions while maintaining data fidelity, demonstrating the effectiveness of the proposed optimization framework.

Conclusion:

The study highlights that vision-language foundation models can improve undersampled MRI reconstruction through semantic-space optimization.

1. INTRODUCTION

Foundation models have emerged as a transformative paradigm in artificial intelligence.¹ These models are pre-trained on large-scale datasets using self-supervised objectives^2–5 to learn semantic and contextual representations with strong transferability across downstream tasks.⁶ Recently, vision-language foundation models have extended this paradigm to jointly model images and text by learning a shared semantic embedding space that enables cross-modal understanding and generation.⁷ Beyond natural images, foundation models are increasingly explored in medical imaging, showing promise in tasks such as segmentation,^8,9 disease classification,¹⁰ and automated diagnosis.^11,12

Despite these advances, applications of foundation models in fast MRI remain relatively underdeveloped. Unlike conventional predictive tasks, fast MRI involves reconstructing high-fidelity images from undersampled k-space data under physical constraints. Current reconstruction methods primarily rely on physics-based and image-level priors. For example, parallel imaging exploits the information redundancy of multiple receiver coils.^13,14 Compressed sensing relies on non-uniform k-space undersampling to generate incoherent artifacts, which can be effectively suppressed by imposing priors such as total variation and transform-based sparsity,^15–17 dictionary learning,^18,19 low-rank^20,21 and structured low-rank models.^22,23 Deep learning approaches further introduce powerful data-driven priors that capture statistical image regularities from large-scale datasets,²⁴ through either supervised^25–32 or self-supervised training.^33,34 More recently, generative diffusion models have emerged to provide priors by learning the distribution of fully sampled images and incorporating physics constraints during reconstruction.^35–39 Additionally, approaches such as deep image prior^40,41 and implicit neural representation^42,43 utilize the inherent inductive biases of neural networks as implicit regularizers.

While existing reconstruction methods have substantially advanced the field, the priors they employ are generally derived from image-level representations. These priors effectively capture low- to mid-level image features, such as edges, textures, and spatial smoothness, but provide limited access to higher-level semantic information that aligns with human understanding of MRI anatomy and pathology.⁴⁴ Moreover, their effectiveness is shaped by the limited scope of the training data, which may reduce the generalizability to out-of-distribution cases.⁴⁵ Additionally, prior information for fast MRI could in principle come from multiple sources beyond image data. For example, textual information that includes brief notes on image quality and artifacts, descriptions of anatomical structures, scanning parameters from headers, or radiology reports, represents a potentially valuable source of prior knowledge that has not yet been fully explored. Such information may offer several advantages. First, it is often readily available, abundant, and can be obtained at low additional cost. Second, it can provide semantic representations about tissue characteristics, which may complement image-level priors when image information is limited or degraded. This highlights the potential of integrating textual information to expand the prior knowledge available for MRI reconstruction.

In this study, we integrate a pre-trained vision-language foundation model to enhance MRI reconstruction. Unlike recent foundation model studies in medical imaging that pretrain or fine-tune large models on medical datasets to serve as backbones,^46–52 our approach leverages a well-trained and frozen vision-language foundation model through its semantic space. This design provides access to high-level features of image content and, at the same time, broadens the available prior knowledge by incorporating information from natural language descriptions. To exploit these properties, we propose a general optimization framework formulated in the semantic space, which aligns the reconstructed embedding with a target distribution while preserving data fidelity. By fusing these complementary sources of information, our method enriches the contextual guidance for undersampled MRI reconstruction and introduces a preliminary mechanism for incorporating natural language into the reconstruction process. The main contributions of this work are summarized as follows:

1. Semantic prior integration: We demonstrate how a large-scale vision-language foundation model, pre-trained on extensive image-text datasets, can enhance MRI reconstruction by incorporating semantic priors into the reconstruction process.

2. Semantic-space optimization framework: We propose a general optimization objective formulated in the semantic embedding space, which can be integrated with a variety of deep learning-based reconstruction approaches.

3. Language-guided reconstruction: We explore the feasibility of natural language as auxiliary guidance to extend the semantic representations for MRI reconstruction. Compared with purely image-based priors, textual descriptions offer greater accessibility and flexibility, thereby enriching the prior information available for undersampled MRI reconstruction.

2. THEORY

2.1. Overview of Vision-Language Foundation Models

Figure 1(A) illustrates a general overview of vision-language foundation models for image understanding. The input consists of an image paired with a language instruction. The image is processed by an image encoder, often implemented with a Vision Transformer architecture (highlighted in yellow),⁵³ while the language instruction is processed by a text encoder that maps tokenized text into a continuous embedding space (highlighted in blue). The resulting visual and textual representations are subsequently aligned or fused within a shared semantic space through mechanisms such as multimodal transformers, cross-attention modules, or contrastive objectives, depending on the specific model design. This shared embedding space enables semantically related images and texts to be mapped to nearby representations. When generating a textual response, the model produces an output token sequence that is subsequently decoded into natural language using a standard text decoding procedure.

Figure 1.

Overview of the proposed semantic distribution guided MRI reconstruction framework. (A) Architecture of vision-language foundation models used to extract semantic embeddings. (B) Deep learning-based MRI reconstruction networks with data consistency constraints. (C) Semantic space optimization using a contrastive objective. Two types of semantic embedding are investigated, including those derived from image-only exemplars and those incorporating image-language auxiliary information. Both the reconstructed output and the auxiliary information are encoded into the same semantic space, where the contrastive objective aligns the reconstructed representation with the target semantic distribution.

In addition to these architectural components, many vision-language foundation models employ contrastive learning objectives during pretraining to encourage the learning of meaningful and well-organized representations from large-scale multimodal data. In contrastive learning, semantically related samples are treated as positive pairs, whereas unrelated samples serve as negative pairs. A commonly used contrastive objective is the InfoNCE loss,⁵⁴ which encourages embeddings of positive pairs to be close while pushing apart those of negative pairs. Given $N$ samples in a batch, the InfoNCE contrastive loss is defined as:

$${ℒ}_{con}=-\frac{1}{N}\sum _{i=1}^N\text{log}\frac{\text{exp}(\text{sim}(e_i,e_i^{+})/\tau )}{{\sum }_{j=1,j\ne i}^N\text{exp}(\text{sim}(e_i,e_j^{-})/\tau )},$$ (1)

where $e_i$ represents the embedding of the $i$th sample, $e_i^{+}$ and $e_j^{-}$ denote positive and negative embeddings relative to $e_i.\tau$ is a temperature parameter controlling the sharpness of the similarity distribution, with smaller values yielding more peaked distinctions between positive and negative pairs. The similarity term $\text{sim}(·)$ typically refers to the cosine similarity, defined as:

$$\text{sim}(a,b)=\frac{a·b}{‖a‖‖b‖},$$ (2)

which measures the angular closeness between two vectors independent of their magnitudes. While the original InfoNCE formulation assumes a single positive counterpart per sample, later variants generalize it to multi-positive settings,^55,56 in which several semantically similar samples are simultaneously treated as positives:

$${ℒ}_{con}(e_i,𝒫,𝒩)=-\frac{1}{N}\sum _{i=1}^N\text{log}\frac{{\sum }_{p\in 𝒫}\text{exp}(\text{sim}(e_i,p)/\tau )}{{\sum }_{p\in 𝒫}\text{exp}(\text{sim}(e_i,p)/\tau )+{\sum }_{n\in 𝒩}\text{exp}(\text{sim}(e_i,n)/\tau )},$$ (3)

where $𝒫$ represents the set of positive embeddings, $𝒩$ is the set of negative embeddings. Through minimizing this loss, the model organizes the multimodal embedding space such that samples with similar semantic content lie closer together.

To learn the structured semantic space, vision-language foundation models are pretrained on large-scale multimodal datasets containing diverse images and textual descriptions. A representative example is the Janus model⁵⁷ (https://github.com/deepseek-ai/Janus), a recent open-source vision-language foundation model. Janus uses a multimodal transformer-based large language model to jointly process visual and textual embeddings and is pretrained on about 1.25 million image-text pairs, enabling both perceptual and contextual semantic understanding. In this study, Janus is adopted as the underlying foundation model.

2.2. Leveraging Vision-Language Foundation Models for MRI Reconstruction

The pre-trained vision-language foundation model provides unified semantic representations that capture high-level visual structures and linguistic context, which could offer complementary prior knowledge for undersampled MRI reconstruction. To incorporate such priors, we formulate the problem using the conventional fast MRI forward model. Let:

$$S_j=\mathbf{M}\mathbf{F}{\mathbf{C}}_{j}I+{\delta }_j,$$ (4)

where $S_j$ is the undersampled k-space data of the $j$th coil. $I$ is the vectorized MR image to be reconstructed and ${\delta }_j$ represents the noise. The matrix ${\mathbf{C}}_j$ corresponds to the diagonalized coil sensitivity map for the $j$th coil, $\mathbf{F}$ denotes the Fourier transform, and $\mathbf{M}$ is the diagonalized sampling mask. Recovering $I$ from $S_j$ can be formulated as the following optimization problem:

$$\text{arg}\underset{I}{\text{min}}\frac{1}{2}\sum _{j=1}^c{‖S_j-\mathbf{M}\mathbf{F}{\mathbf{C}}_{j}I‖}_2^2+\lambda ℛ(I),$$ (5)

where $c$ is the number of receiver coils. The first term enforces data consistency with the acquired measurements, while $ℛ(\bullet )$ denotes a regularization term incorporating prior knowledge about the reconstructed image $I$, and $\lambda$ controls the contributions between these two components. In this study, we introduce a semantic consistency prior by formulating $ℛ(\bullet )$ with a contrastive learning objective. The idea is to ensure the semantic embedding of the reconstructed image stays close to semantically relevant examples and far from irrelevant ones within the foundation model’s embedding space. This results in the following regularized reconstruction problem:

$$\text{arg}\underset{I}{\text{min}}\frac{1}{2}\sum _{j=1}^c{‖S_j-\mathbf{M}\mathbf{F}{\mathbf{C}}_{j}I‖}_2^2+\lambda {ℒ}_{con}(e_I,𝒫,𝒩),$$ (6)

where $e_I$ is the semantic embedding extracted from the current reconstruction $I$. $𝒫$ and $𝒩$ denote the sets of positive and negative embeddings, respectively.

Equation (6) establishes a general goal for optimizing semantic space. By leveraging the high-level perceptual and contextual representations provided by the foundation model, this goal guides the reconstruction toward images that better reflect realistic MRI content. Furthermore, the formulation can be incorporated into a range of deep learning architectures for MRI reconstruction. For a given network $f_{\theta }$ with parameters $\theta$ and input $d$, the overall loss function is formulated as:

$$ℒ\left(\theta \right)=\frac{1}{2}\sum _{j=1}^c{‖S_j-\mathbf{M}\mathbf{F}{\mathbf{C}}_j{f}_{\theta }\left(d\right)‖}_2^2+{\lambda ℒ}_{con}\left[e_{f_{\theta }\left(d\right)},𝒫,𝒩\right],$$ (7)

The network parameters $\theta$ are optimized via gradient backpropagation to minimize $ℒ(\theta )$, ensuring that the reconstructed images satisfy both the MR data consistency and the high-level semantic consistency imposed by the foundation model.

As shown in Figure 1(B), we investigated three deep learning architectures in this study:

1. End-to-end mapping²⁵ with a U-Net: This approach learns a direct transformation from undersampled inputs to reconstructed images.

2. Unrolled iterative network:^29,33 This method unrolls a variable-splitting iterative reconstruction algorithm into a sequence of learnable stages. Each stage alternates between a neural network update block (e.g., U-Net) and an explicit data-consistency step solved using conjugate gradient descent.

3. Implicit neural representation (INR) network:⁴² This method represents the MR image as a continuous function of spatial coordinates. The function is parameterized by an MLP, which takes spatial coordinates as input and predicts the corresponding image intensities.

All three reconstruction backbones are optimized in a self-supervised and scan-specific manner using the loss function defined in Eq. (7).

2.3. Semantic Embedding Extraction from the Foundation Model

Figure 1(C) illustrates the process of extracting semantic embeddings from the Janus foundation model for semantic space optimization. Specifically, we consider two scenarios that incorporate different forms of auxiliary information:

1. Image-only prior: A set of available fully sampled and undersampled MR images is used as auxiliary information to construct the positive and negative prior embeddings. These embeddings are obtained through the image-encoder ${𝚽}_{img}$ of the Janus model, i.e.:

   $$𝒫={\left\{{𝛟}_{img}\left(I_i^{\mathit{\text{fully}}}\right)\right\}}_{i=1}^M,𝒩={\left\{{𝛟}_{img}\left(I_i^{\mathit{\text{under}}}\right)\right\}}_{i=1}^M,$$ (8)

   where $M$ is the total number of auxiliary image examples. The reconstructed image $I$ is encoded using the same procedure so that its semantic embedding lies in the same space as the prior embeddings:

   $$e_I={𝛟}_{img}\left(I\right)$$ (9)
2. Image-language prior: A small set of auxiliary positive and negative MR images (e.g., fully sampled and undersampled) is paired with a language instruction that prompts the model to perform an image-quality assessment, such as “Determine whether this image is high-quality or low-quality.” As illustrated in Figure 1(C), each auxiliary image is encoded by the image encoder ${𝚽}_{img}$, while the accompanying language instruction is encoded through the text encoder ${𝚽}_{txt}$. The visual and textual embeddings are then jointly processed by the large language model ${𝚽}_{llm}$, which produces a language semantic embedding representing the model’s quality-assessment response for the given image (e.g., “high-quality” or “low-quality”). In this scenario, the number of auxiliary MR images is intentionally reduced, potentially leading to an incomplete representation of the underlying semantic distribution. To mitigate this limitation, we expand the semantic space by introducing Gaussian perturbations to the input language embeddings. The standard deviation of the perturbation is carefully tuned to broaden semantic coverage while preserving the separability between semantically relevant and irrelevant embeddings. The process can be formulated as:

   $$\begin{gathered} 𝒫=\left\{{𝛟}_{llm}\left({𝛟}_{img}\left(I_i^{pos}\right),{𝛟}_{txt}\left(T\right)+{ϵ}_k\right)|i=1,2,\dots ,M,k=1,2,\dots K\right\}, \\ 𝒩=\left\{{𝛟}_{llm}\left({𝛟}_{img}\left(I_i^{neg}\right),{𝛟}_{txt}\left(T\right)+{ϵ}_k\right)|i=1,2,\dots ,M,k=1,2,\dots K\right\} \end{gathered}$$ (10)

   where $I_i^{pos}$ and $I_i^{neg}$ denote positive and negative image examples, respectively. $T$ represents the language instruction. $K$ is the number of perturbations, and ${ϵ}_k\in \text{Gaussian}(0,\sigma )$ denotes the $k$th perturbation. The reconstructed image $I$ is passed through the same image-language encoding pipeline, ensuring that it is mapped into the same language semantic space defined by the model’s quality-assessment responses:

   $$e_I={𝛟}_{llm}\left({𝛟}_{img}\left(I\right),{𝛟}_{txt}\left(T\right)\right)$$ (11)

By extracting high-level semantic priors from the auxiliary information and optimizing the reconstructed embeddings accordingly, the reconstruction process is guided toward the semantic distribution defined by the positive examples. In the image-only setting, this corresponds to aligning the reconstruction with the visual features and anatomical details represented in the fully sampled reference images. In the image-language setting, the reconstruction is steered toward the semantic attributes conveyed by the positive textual descriptions, thereby enabling the model to recover image features consistent with the intended linguistic cues.

3. METHODS

3.1. Implementation Details

In Janus, the vision transformer within the image encoder divides each input image into non-overlapping 24×24 patches, which are then linearly projected and processed through 24 transformer layers. Each transformer layer maps the input into 1024-dimensional visual embeddings. The large language model also consists of 24 transformer layers and outputs 2048-dimensional embeddings. These embeddings constitute the semantic space within which the reconstruction is optimized. The weights of the Janus model are fixed in this study.

For deep learning backbones used in MRI reconstruction, the U-Net architecture⁵⁸ in the end-to-end mapping and the unrolled model follows a standard encoder-decoder design. Each U-Net contains five resolution levels with two 3×3 convolutional layers per level, followed by batch normalization and ReLU activations. The encoder employs 64, 128, 256, 512, and 1024 feature channels across the five levels, with symmetric decoding pathways and skip connections. The U-Net operates on complex-valued images represented as two input channels corresponding to the real and imaginary components. The output is also produced in two channels representing the reconstructed real and imaginary parts. No activation function is applied to the final output layer. The unrolled iterative model consists of four unrolled stages, and the network parameters are not shared across stages. The MLP in INR consists of eight fully connected layers, each containing 256 hidden units with sine activation functions⁵⁹ to improve the network’s ability to learn high-frequency information. The final output layer does not apply an activation function. The MLP maps each spatial coordinate to a two-channel output representing the real and imaginary components of the reconstructed image. All inputs are normalized to the range of $[-1,1]$ in coordinate space to facilitate stable training.

For the image-only prior, three hierarchical embedding levels extracted from the 1^st, 13^th, and 24^th transformer layers of the image encoder were used to capture different semantic information. The corresponding weights for these three layers were set to 0.005, 0.5, and 1, respectively. For the image-language prior, the large language model’s final output was used as the semantic representation, as it provides the most comprehensive and semantically complete information for highly abstract language descriptions. To augment the language semantic distributions, 100 Gaussian noise perturbations $(K=100)$ with a standard deviation of $\sigma =0.03$ were added to the textual embeddings. The temperature parameter in the contrastive loss was fixed at 0.07 and kept consistent across all experiments.

All methods were implemented in Python 3.9 using PyTorch 2.8.0 on a workstation equipped with an INTEL(R) XEON(R) PLATINUM 8562Y+ CPU and an NVIDIA H100 GPU with 96 GB of memory. The code is available at: https://github.com/I3Tlab/Foundation-Model-MRI-Reconstruction.

3.2. Experimental Settings

3.2.1. Evaluation of Image-only Prior

This experiment evaluates the performance of the image-only prior. We used knee (proton density weighted 2D-FSE with fat suppression) and brain (T2 FLAIR) datasets from the FastMRI repository.^60,61 All FastMRI data used in this study are publicly available, de-identified, and were originally acquired under institutional review board approval with informed consent. To construct the image embedding distributions for $𝒫$ and $𝒩$, a subset of the training data, including 20 subjects for knee and 50 subjects for brain, was used as auxiliary images, while reconstruction was conducted on the corresponding validation datasets (99 subjects for knee data, 69 subjects for brain data) by retrospectively undersampling k-space data. For both the end-to-end and the unrolled backbones, the networks were first warm-started on auxiliary images to learn general structural representations and subsequently fine-tuned on the specific undersampled data by minimizing the loss function in Eq. (7), with a low-rank adaptation scheme for fast network convergence.⁶² The performance of the proposed semantic priors was compared with the conventional method using total variation (TV) regularization under the same initialization and optimization settings.

Reconstruction results were assessed using peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM),⁶³ Tenengrad,⁶⁴ and learned perceptual image patch similarity (LPIPS).⁶⁵ Tenengrad measures image sharpness by computing the gradient magnitude, while LPIPS evaluates perceptual similarity based on deep neural network features. To further assess image quality beyond these quantitative metrics, a reader study was conducted in which two experienced musculoskeletal radiologists independently and blindly evaluated the reconstructions from all three methods under two regularization conditions on 20 subjects. Images were randomly presented, and each was scored on a 5-point Likert scale⁶⁶ (0=very poor, 5=excellent; higher scores indicate better quality) across four criteria: presence of artifacts, sharpness of anatomical structures, diagnostic confidence, and overall image quality. Artifacts arising from imperfect fat suppression were not considered during scoring, as they originate from the acquisition rather than the reconstruction methods. Embedding distributions in the semantic space were visualized using the uniform manifold approximation and projection (UMAP) method.⁶⁷

3.2.2. Robustness and Generalization of Image-only Prior

We further evaluated the generalization and robustness of the image-only prior under substantial abnormalities and contrast variations that may not be fully represented in the auxiliary images. Four representative abnormalities were selected, including knee joint replacement, edema, ventricular enlargement, and white matter hyperintensities with tumors. Cross-contrast robustness was assessed by reconstructing fat-suppressed or non-fat-suppressed FastMRI knee data using semantic embeddings extracted from the opposite contrast.

3.2.3. Effectiveness of Image-language Prior

In this experiment, we investigated the role of language in expanding the distribution of semantic embeddings and its impact on image reconstruction. We used only 3 subjects from the FastMRI knee dataset to construct two types of language-augmented semantic priors:

1. “High-quality” vs. “Low-quality” semantic space: Embeddings were obtained from fully sampled and undersampled images paired with the instruction “Determine whether this image is high-quality or low-quality.”

2. “Blurred” vs. “Aliased” semantic space: Embeddings were derived from TV-regularized compressed sensing reconstruction results (causing blurring) and undersampled images paired with the instruction “Determine whether this image is blurred or aliased.”

We monitored the reconstruction outputs across iterations, along with the corresponding quality assessment responses generated by the large language model, to illustrate the influence of language embeddings on the reconstruction trajectory. To evaluate the flexibility of the language instructions, we generated 20 semantically equivalent prompts with varied phrasings, as summarized in Table S1. Variations in the resulting reconstructions were analyzed to assess the consistency of language-conditioned behavior.

4. RESULTS

4.1. Reconstruction Performance with Image-only Prior

Figure 2 presents the reconstruction results on knee data using the conventional TV regularization and the proposed semantic prior. At an acceleration rate of R=4 (Figure 2(A)), both approaches effectively suppress aliasing artifacts. The zoomed-in views show that the proposed prior better preserves realistic textures in the bone region, producing results that visually resemble the fully sampled reference, whereas TV regularization tends to over-smooth the structures. Figure 2(B) presents reconstruction results at R=6. The displayed image corresponds to a representative knee slice containing a meniscus tear, indicated by the yellow arrow. This pathological feature is depicted with greater clarity in the reconstructions guided by semantic priors. Moreover, the layered cartilage (blue arrow) and the surrounding tissue (green arrow) exhibit improved sharpness and structural definition across all three deep learning methods when incorporating semantic priors. Figure 3 presents reconstruction results on brain data. At R=4 (Figure 3(A)), incorporating semantic priors yields higher visual fidelity and better preservation of fine anatomical details across all three deep learning methods, as shown by the boundaries of the globus pallidus (red arrow). Figure 3(B) highlights a lesion region, where the surrounding anatomical structures are delineated more distinctly in the results of end-to-end and unrolled methods guided by semantic priors, as indicated by the yellow arrow.

Figure 2.

Reconstruction results on knee data using conventional TV regularization and the proposed semantic prior. (A) Results at an acceleration rate of R = 4. Both approaches effectively suppress aliasing artifacts, while the semantic prior preserves more realistic bone textures and produces images that are visually closer to the fully sampled reference. (B) Results at R = 6. The meniscus tear (yellow arrow) is more clearly depicted when using semantic priors. The layered cartilage (blue arrow) and surrounding tissue (green arrow) also show improved sharpness and structural definition across all three deep learning methods.

Figure 3.

Reconstruction results on brain data using conventional TV regularization and the proposed semantic prior. (A) Results at an acceleration rate of R=4. The semantic prior improves visual fidelity and enhances fine anatomical details, such as the boundaries of the globus pallidus (red arrow). (B) Results at R=5. In the lesion region, the surrounding structures are more distinctly delineated in the end-to-end and unrolled reconstruction guided by semantic priors (yellow arrow).

Table 1 compares quantitative results for both the knee and brain datasets using the conventional and proposed priors. For the knee data at R=4 and R=6, the end-to-end and unrolled methods with proposed semantic priors achieve better LPIPS and Tenengrad scores than the TV-regularized reconstruction, although they yield lower PSNR and SSIM. The INR model combined with semantic priors demonstrates improved performance across all four metrics. Similar trends are observed in the brain data. Integrating semantic priors consistently improves LPIPS and Tenengrad, while producing lower PSNR for all three architectures. For SSIM, the semantic prior produces slightly lower or comparable values in the end-to-end and unrolled methods, and higher values in the INR network. Figure 4 shows the reader study results comparing the proposed semantic-prior reconstruction with the conventional TV-regularized baseline across four evaluation criteria. The proposed semantic prior yields higher mean scores across all criteria for each network architecture. Statistical testing using the Wilcoxon signed-rank test indicates significant improvements (p<0.05) for several comparisons.

Table 1.

Comparison of PSNR, SSIM, LPIPS, and Tenengrad metrics on knee and brain datasets at different acceleration rates. Values are reported as mean±standard deviation, with the best performance in each setting highlighted in bold. Semantic priors achieve better performance in terms of LPIPS and Tenengrad across all architectures.

Knee (R=4)
Methods	End-to-end		Unroll		INR
	TV	Semantic	TV	Semantic	TV	Semantic
PSNR (↑)	30.31±2.33	29.47±2.05	31.10±2.37	30.52±2.26	31.27±2.31	31.76±2.36
SSIM (↑)	0.760±0.064	0.733±0.063	0.757±0.069	0.735±0.070	0.750±0.074	0.768±0.071
LPIPS (↓)	0.178±0.047	0.173±0.040	0.155±0.055	0.124±0.042	0.111±0.038	0.096±0.036
Tenengrad (↑)	0.038±0.014	0.043±0.015	0.043±0.015	0.052±0.018	0.049±0.017	0.051±0.018
Knee (R=6)
Methods	End-to-end		Unroll		INR
	TV	Semantic	TV	Semantic	TV	Semantic
PSNR (↑)	29.39±2.18	28.47±2.05	30.25±2.21	29.42±2.11	30.43±2.13	31.44±2.30
SSIM (↑)	0.712±0.069	0.679±0.069	0.711±0.071	0.678±0.072	0.726±0.073	0.756±0.071
LPIPS (↓)	0.245±0.048	0.224±0.041	0.200±0.049	0.165±0.042	0.155±0.043	0.125±0.040
Tenengrad (↑)	0.030±0.011	0.045±0.017	0.033±0.011	0.050±0.017	0.043±0.015	0.048±0.017
Brain (R=4)
Methods	End-to-end		Unroll		INR
	TV	Semantic	TV	Semantic	TV	Semantic
PSNR (↑)	33.76±2.61	32.83±2.32	34.34±2.79	33.97±2.56	34.25±2.70	34.09±2.56
SSIM (↑)	0.885±0.037	0.861±0.040	0.890±0.044	0.890±0.044	0.884±0.041	0.897±0.040
LPIPS (↓)	0.110±0.044	0.109±0.040	0.096±0.043	0.092±0.041	0.109±0.043	0.089±0.040
Tenengrad (↑)	0.067±0.030	0.071±0.032	0.078±0.035	0.081±0.037	0.074±0.033	0.077±0.034
Brain (R=5)
Methods	End-to-end		Unroll		INR
	TV	Semantic	TV	Semantic	TV	Semantic
PSNR (↑)	32.74±2.48	31.78±2.27	32.99±2.73	32.38±2.54	33.32±2.59	33.13±2.47
SSIM (↑)	0.868±0.040	0.841±0.045	0.870±0.047	0.864±0.046	0.869±0.045	0.881±0.042
LPIPS (↓)	0.128±0.045	0.122±0.042	0.114±0.044	0.110±0.042	0.122±0.045	0.100±0.041
Tenengrad (↑)	0.064±0.030	0.068±0.031	0.071±0.032	0.075±0.034	0.072±0.033	0.075±0.034

Figure 4.

Reader study results comparing reconstructions with the conventional and the proposed semantic priors across three deep learning architectures. The semantic priors consistently achieve higher subjective scores in Artifacts, Sharpness of Anatomical Structures, Diagnostic Confidence, and Overall Quality, demonstrating improved perceptual quality. Asterisks indicate statistically significant differences (p < 0.05).

Figure 5 visualizes the embedding distributions of undersampled and fully sampled auxiliary images, as well as the reconstructed images across three hierarchical levels in the semantic space. Figure 5(A) shows the clustering of data points. The reconstructed embeddings (red) overlap with the fully sampled cluster (orange) across all three levels, while remaining distinct from the undersampled embeddings (blue). Figure 5(B) shows the optimization trajectory of a representative example over the course of reconstruction. The trajectories are illustrated using a color gradient, with the initial point shown in blue and the final point marked by a red star. The zoomed-in panels highlight the local optimization paths. These trajectories reveal a consistent convergence from the undersampled distributions toward the fully sampled cluster across all three levels, indicating that the contrastive loss successfully drives the reconstructed representations toward the positive distribution while repelling them from the negative distribution, thereby enforcing semantic consistency with the fully sampled data

Figure 5.

Visualization of embedding distributions and optimization trajectories in the image semantic space. Each point corresponds to the embedding of a single image slice. (A) Clustering of undersampled, fully sampled, and reconstructed data across three hierarchical semantic levels. Reconstructed embeddings (red) lie closer to the fully sampled cluster (orange), indicating successful semantic alignment. (B) Optimization trajectories of a representative example, shown with color gradients from blue (initial) to red (final). The reconstructed embeddings converge toward the fully sampled cluster, demonstrating effective semantic-space optimization.

4.2. Generalization and Robustness of Image-only Prior

Figure S1 presents the reconstruction results of the three methods guided by semantic priors, for cases with substantial abnormalities at a 4× acceleration. Although the auxiliary images may not include these atypical structures, all three methods are able to recover the major pathological features. In contrast, the zero-filled images exhibit pronounced aliasing and substantial loss of anatomical detail. Figure 6 shows the reconstruction results for knee data acquired with 2D-FSE without fat suppression, using either conventional regularization or the semantic prior extracted from auxiliary 2D-FSE images with fat suppression. In the zoomed-in region containing a meniscus tear (yellow arrow), the pathological structure and adjacent tissues (red arrow) are more distinctly visualized in the reconstructions guided by the proposed prior. In contrast, conventional regularization tends to produce smoother appearances in these areas, reducing the visibility of fine structural details.

Figure 6.

Reconstruction results on knee data without fat suppression. Reconstruction guided by the conventional regularization and the proposed prior derived from auxiliary fat-suppressed images is compared. The meniscus tear (yellow arrow) and surrounding tissue (red arrow) are more clearly delineated when using the proposed semantic prior.

4.3. Effectiveness of Image-language Prior

Figure 7 illustrates the feasibility and effectiveness of the proposed image-language prior in guiding MRI reconstruction at R=4. Figure 7(A) visualizes the embeddings in the language semantic space. The red and black points correspond to high-quality and low-quality embeddings obtained from representative image examples paired with the instruction “Determine whether this image is high-quality or low-quality.” The orange and blue points denote the augmented embeddings generated by adding Gaussian perturbations to the original language embeddings of the instruction. The color-gradient trajectory depicts the optimization path of a representative reconstruction, showing a gradual transition of its embedding from the low-quality region toward the high-quality region. Figure 7(B) shows the corresponding evolution of the reconstructed images across iterations. The visual quality improves progressively from low-quality intermediate results to a high-quality final reconstruction. The large language model’s responses follow the same trend, shifting from “Low-quality” at early iterations to “High-quality” in later stages.

Figure 7.

High-quality versus low-quality scenario under the proposed image-language prior for semantic distribution-guided MRI reconstruction at R=4. (A) Language semantic embedding space showing separable high-quality and low-quality clusters, with Gaussian-perturbed embeddings extending the semantic distribution. A representative reconstruction trajectory transitions from the low-quality to the high-quality region. (B) Display of the reconstruction evolution at different iterations. The reconstructed images progressively improve, accompanied by a shift in the language model’s assessments from “Low-quality” to “High-quality.”

Figure 8 illustrates the effectiveness of the proposed image-language prior in a second case, using the “blurred” versus “aliased” semantic setting. Similarly, the prior distributions are generated from representative image examples paired with the instruction “Determine whether this image is blurred or aliased,” and are further augmented using Gaussian perturbations. Notably, under this “blurred” versus “aliased” semantic prior, the reconstruction exhibits a different evolution compared with the case presented in Figure 7. As shown in Figure 8(B), the reconstruction begins with an aliased appearance due to undersampling and progressively reduces aliasing artifacts throughout the iterations. At later iterations, the reconstruction becomes increasingly smoothed, eventually resembling the TV-regularized compressed sensing result that was used to generate the auxiliary blurred examples. The large language model’s responses follow the same progression, transitioning from “Aliased” in the early iterations to “Blurred” as the reconstruction converges.

Figure 8.

Further validation using the blurred versus aliased case under the proposed image-language prior for MRI reconstruction at R=4. (A) Language semantic embedding space showing separable blurred and aliased clusters, with Gaussian-perturbed embeddings extending the semantic distribution. A representative reconstruction trajectory transitions from the aliased to the blurred region. (B) Display of the reconstruction evolution at different iterations. The reconstructed image progressively reduces aliasing artifacts as it becomes increasingly smoothed, and the language model’s assessments shift accordingly from “Aliased” to “Blurred.”

Figure S2 presents reconstruction results obtained under four representative language instructions that are semantically equivalent but phrased differently. The reconstructed images exhibit minimal visual variation across instructions. Quantitatively, the pixel-wise standard deviation across all 20 reconstructions is shown in the first row, with the average remaining below 0.02, corresponding to subtle fluctuations in normalized MRI intensities in the 0–1 range.

5. DISCUSSION

This study explores the potential of vision-language foundation models for MRI reconstruction. Using the pre-trained Janus model, we introduce a contrastive semantic objective that guides reconstruction toward target semantic prior distributions. Both image-only and image-language auxiliary information were investigated. The image-only prior improves preservation of fine anatomical structures and perceptual quality with good generalizability and robustness, while the image-language setting further extends the distributions under limited image examples and enables explicit language-guided control of reconstruction attributes. Overall, these results demonstrate the promise of vision-language semantic priors for incorporating perceptual and linguistic knowledge as a complement to conventional physics-based constraints in MRI reconstruction.

5.1. Results Analysis

The experimental results offer insights into how the proposed semantic priors influence MRI reconstruction. In the image-only setting, the semantic prior improves visual fidelity (Figures 2 and 3) and yields superior Tenengrad and LPIPS values (Table 1). Although PSNR and SSIM are reduced, this does not necessarily indicate the introduction of hallucinated structures. First, PSNR and SSIM primarily emphasize pixel-wise intensity agreement and may not reliably reflect anatomical coherence.⁶⁸ Second, the semantic prior is incorporated within a physics-constrained reconstruction framework, where strict data-consistency enforcement limits the reconstruction to structures supported by the acquired k-space measurements. The reader study further confirms that semantic priors derived from vision-language foundation models encode perceptually meaningful features that complement conventional regularizations. Nevertheless, in high-acceleration settings with reduced data consistency, these perceptual improvements should be interpreted with caution, and comprehensive clinical evaluation is warranted prior to deployment. Furthermore, as an additional validation, the proposed semantic prior was evaluated within a SwinUNETR architecture (see Figure S3).⁶⁹ Similar improvements were observed, suggesting that the effectiveness of the semantic prior extends to transformer-based reconstruction backbones. Beyond matched acquisition settings, the proposed priors exhibit robustness to abnormalities and contrast variations (Figures S1 and 6). Further analyses and evaluations with “cross-domain” priors (see Figure S4) suggest that cross-protocol generalization within the same anatomy is relatively robust, whereas cross-anatomy generalization is more challenging, highlighting the potential need for anatomy-specific priors or adaptation strategies. Finally, sensitivity analysis of the image embedding weights from different layers reveals multiple near-optimal configurations (Figure S5), indicating reduced dependence on precise parameter tuning. This behavior likely reflects representational redundancy and complementary information across embeddings from different layers, motivating future exploration of more efficient and adaptive embedding selection and weighting strategies.

In the image-language setting, the embedding distributions in Figures 7(A) and 8(A) show that incorporating language instructions with Gaussian perturbations expands the embedding distribution under limited auxiliary image samples (three subjects). This effect likely arises because Gaussian perturbations in the language embedding space approximate alternative phrasings and act as a data-augmentation mechanism that broadens semantic coverage. Similar feature-space perturbation strategies for images and texts have been used in prior studies to enrich semantic variability without additional annotated data.^70–72 The “Instruction Following” experiment in Figure S6 further demonstrates that Gaussian perturbations applied to the input language embeddings produce varied responses that remain semantically consistent and comply with the given instruction, indicating preservation of semantic intent. The strength of this augmentation is governed by the Gaussian standard deviation $\sigma$. Based on the analysis in Figure S7, σ = 0.03 is selected as a trade-off between preserving semantic separability and introducing variability. Additionally, the reconstruction consistency across differently worded but semantically equivalent instructions (Figure S2) highlights the flexibility of language-based priors. These results suggest that linguistic information may further extend the semantic distribution and enrich the prior knowledge available for MRI reconstruction.

Taken together, image-only and image-language priors offer complementary benefits. Table 2 summarizes their relative accessibility and reconstruction accuracy. As shown, accessibility increases from same-contrast image priors to cross-contrast priors, and further to image-language priors; however, this increased accessibility is accompanied by a gradual reduction of precision in guiding MRI reconstruction. This trade-off suggests that the optimal prior choice may depend on the availability of auxiliary information and the required reconstruction accuracy for a given task.

Table 2.

Summary of the semantic priors evaluated in this study, illustrating the trade-off between accessibility and precision.

Prior type	Accessibility	Accuracy
Same-contrast images	Low: Requires multiple high-quality MRI images with the same contrast as the reconstructed image.	High: Provides the most accurate guidance, as the prior matches both anatomy and contrast.
Different-contrast images	Medium: More accessible, though it still requires multiple images.	Medium: Moderately accurate, as anatomical consistency is preserved, but different contrasts introduce variability.
Images with language	High: Requires far fewer image samples, with language descriptions providing an efficient mechanism for expanding the semantic embeddings.	Low: Less accurate, as guidance relies on the semantic alignment of language, which is highly abstract.

5.2. Relation to Foundation Model Approaches for MRI

Recent studies have explored the use of foundation models in MRI through large-scale pretraining on diverse MRI datasets.^{46–52,73,74} These models learn generic and transferable MRI representations that can be adapted to downstream tasks with limited supervision. Representative examples include MRI-CORE,⁷³ which is trained on millions of MRI slices and evaluated on classification and segmentation, as well as other MRI foundation models such as Triad⁵⁰ for segmentation, classification, and registration, and BrainSegFounder⁴⁶ for brain tumor and brain lesion segmentation. In parallel, several studies have investigated MRI reconstruction frameworks that benefit from foundation models. For instance, FedGAT⁷⁵ constructs a generative MRI prior by fine-tuning a natural-image foundation model through federated learning across institutions, which is then used to augment local reconstruction models and improve generalization. MambaRoll⁷⁶ adapts a state space model architecture to physics-driven MRI reconstruction. Similarly, our work leverages the representational power of existing foundation models to regularize MRI reconstruction. A key distinction is that we do not modify or fine-tune the foundation model itself; instead, we directly exploit its semantic embedding space as a prior. Building on insights from these related studies, future work may explore several directions. First, MRI-specific foundation models could serve as priors, providing richer representations that capture multi-contrast and multi-anatomical structures to support high-fidelity reconstruction. Second, future efforts could focus on efficiently adapting natural-image-trained foundation models to the MRI domain and integrating them effectively into reconstruction frameworks. Third, the structured representations learned by foundation models could inspire the design of reconstruction architectures that better capture multi-scale and contextual dependencies, potentially enhancing both reconstruction accuracy and interpretability.

5.3. Technique Considerations

The proposed auxiliary information-guided reconstruction shares similarities with previous studies that incorporate side information, such as multi-contrast^77–79 or longitudinal MRI scans⁴³ from the same subject, to enhance fast MRI reconstruction. These approaches provide anatomically aligned and subject-specific priors, but such side information is often unavailable. In contrast, semantic priors from foundation models enable the use of more flexible auxiliary information, including images from different subjects, different contrasts, and even language descriptions. Additionally, our method shares a similar motivation with generative adversarial network (GAN)- or diffusion-based approaches in leveraging data distributions, but differs fundamentally in how these data distribution priors are obtained. GAN and diffusion models learn distributions from MRI datasets through adversarial or generative training. By contrast, our method does not learn a new distribution from MRI data. Instead, it constructs a distribution by mapping MRI data into a fixed semantic embedding space provided by a pre-trained foundation model. This enables the extraction of a data distribution without requiring large fully sampled datasets. In this study, only a small set of fully sampled auxiliary data is used to estimate the target image semantic distribution (20 subjects for knee and 50 subjects for brain), because the foundation model has been pre-trained on large-scale multimodal data and yields a structured and semantically meaningful embedding space. Our method therefore operates by identifying the relevant subspace within this pre-trained semantic manifold, rather than learning a new distribution from scratch. This setting is fundamentally different from GAN or diffusion training paradigms. Figure S8 illustrates the advantages of our approach under limited fully sampled data, showing that adversarial training struggles in this regime due to insufficient data to reliably train a discriminator, whereas our method can leverage the pre-organized semantic space to guide reconstruction.

While our method benefits from a well-trained foundation model as a semantic prior extractor, it may be affected by the domain gap between natural images and MRI data. As illustrated in Figure S9, MRI knee and brain embeddings are more dispersed across different anatomical slices than natural images, suggesting that the model does not fully capture the shared consistency across slices of the same anatomy. Such suboptimal semantic representations may degrade reconstruction performance when auxiliary MRI examples are limited, or require larger and more diverse auxiliary datasets to adequately characterize anatomy-specific semantic distributions. Addressing these limitations will require a deeper understanding of the semantic representations learned by foundation models, including strategies for domain adaptation, semantic manifold augmentation, and alignment with human perceptual and conceptual structures. Recent advances in diffusion models highlight that refining latent representations can substantially improve generative performance and controllability.^80–83 Inspired by these developments, exploring similar semantic-space manipulation strategies in vision-language foundation models may benefit medical imaging applications and contribute to the development of more robust, interpretable, and intelligent foundation models.

5.4. Limitations

Our method has several limitations beyond those discussed above. First, the language used in this work is limited to simple semantic directions and is incorporated indirectly through contrastive embedding alignment. A promising future direction is to explore image-generation foundation models, where language instructions directly shape the generated image and could therefore serve as explicit constraints within the MRI reconstruction optimization process. Second, the current framework incurs substantially increased parameter counts and GPU memory consumption compared with baseline models (Table S2). Although the foundation model remains frozen during reconstruction, its large-scale architecture still introduces considerable computational overhead, potentially limiting scalability and clinical deployment. Techniques such as model compression,^84,85 pruning,^86,87 or distilling the semantic guidance into lightweight networks⁸⁸ may help improve efficiency while retaining performance. Third, the current implementation adopts a scan-specific reconstruction setting to more clearly evaluate the effect of the proposed semantic prior on the reconstruction process. This setting results in relatively long reconstruction times for the baseline methods, with further runtime increases when incorporating the semantic prior (Table S2). Since the proposed semantic prior is formulated as a plug-and-play semantic contrastive optimization scheme, future work may integrate it with more efficient reconstruction backbones or alternative learning strategies to enhance computational efficiency.

6. CONCLUSION

In summary, this work provides a proof-of-concept for incorporating visual-language foundation model priors into MRI reconstruction. The proposed framework demonstrates that semantic embeddings from both images and language can guide the reconstruction toward a perceptually faithful result. By extending the prior space beyond traditional handcrafted features, this approach opens new avenues for instruction-driven, concept-aware reconstruction, laying the groundwork for future research on multimodal guidance in medical imaging.

Supplementary Material

SM

Table S1. Twenty linguistically varied but semantically equivalent language instructions used to construct the embedding distribution. Each instruction prompts a binary judgment of whether an image is high-quality or low-quality, providing diverse textual formulations that express the same underlying semantic intent for the contrastive objective.

Table S2. Comparison of computational resources and reconstruction times for baseline models and the proposed framework on the FastMRI knee dataset. Total model parameter counts, GPU memory consumption, and reconstruction runtimes are reported.

Figure S1. Reconstruction results at a 4× acceleration for cases with substantial abnormalities. Across all three deep-learning methods, semantic priors enable recovery of major pathological features, whereas the zero-filled images show severe aliasing and loss of anatomical detail.

Figure S2. Reconstruction results under four representative language instructions that are semantically equivalent but phrased differently. The reconstructed images show minimal visual variation across instructions. The pixel-wise standard deviation is calculated across all 20 reconstructions, indicating subtle fluctuations across these instructions.

Figure S3. Reconstruction results on the FastMRI knee dataset using the same SwinUNETR backbone with different regularization strategies. Compared with conventional total variation regularization, the proposed semantic prior preserves fine structural details more effectively, while avoiding excessive smoothing, demonstrating its compatibility with transformer-based reconstruction frameworks.

Figure S4. Analysis of cross-protocol and cross-anatomy generalization of the proposed semantic prior. (A) UMAP visualization of embeddings from three types of auxiliary images: knee MRI with fat suppression (FS), knee MRI without fat suppression (w/o FS), and brain MRI. Knee (FS) and knee (w/o FS) embeddings form nearby clusters, while brain embeddings are more distant. (B) Reconstruction results on knee MRI with fat suppression using the three types of priors.

Figure S5. Grid-search-based sensitivity analysis of reconstruction performance with respect to the low-, middle-, and high-layer embedding weights of the image semantic prior, evaluated using (A) SSIM and (B) LPIPS.

Figure S6. Illustration of the “Instruction Following” experiment with Gaussian perturbations. (A) The input knee MRI image and language instruction requesting a description of image quality. (B) Original response and responses generated after applying Gaussian perturbations to the input language embeddings. Text in bold highlights the varied wording introduced by the perturbations, while the overall meaning remains consistent.

Figure S7. UMAP visualization of the embedding distributions under different Gaussian perturbation intensities. Real (unperturbed) and augmented (perturbed) embeddings for “High-quality” and “Low-quality” samples are shown. $\sigma$ = 0.03 provides a balance between semantic separability and distribution expansion.

Figure S8. Comparison of MRI reconstruction results between the proposed method and a GAN-based learning paradigm under limited fully sampled data. Four representative knee slices are shown at an acceleration factor of 4. The proposed method more effectively suppresses aliasing artifacts and achieves more stable improvements.

Figure S9. Visualization of embedding distributions of natural images and MRI images within the foundation model semantic space.