RadGazeGen: radiomics and gaze-guided chest X-ray generation using diffusion models

Abstract.

Purpose

We present RadGazeGen, a framework for integrating experts’ eye gaze patterns and radiomic feature maps as controls within text-to-image diffusion models to enable high-fidelity medical image generation. Although recent text-to-image diffusion models have achieved impressive success, textual descriptions alone often fail to capture disease-specific details necessary for generating clinically accurate and anatomically faithful images.

Approach

To address these limitations, RadGazeGen leverages radiologists’ eye gaze trajectories and radiomics feature descriptors as spatial and semantic controls in the diffusion process. Eye gaze patterns encode visuo-cognitive attention and spatial localization of subtle disease cues, whereas radiomics features capture subvisual phenotype characteristics such as texture, intensity, and shape. By combining these multimodal cues, the proposed framework guides the generative model toward anatomically consistent and disease-aware image synthesis. The quality of the generated images were evaluated using a board-certified radiologist.

Results

RadGazeGen was evaluated on the REFLACX dataset for image generation quality and diversity. Furthermore, to assess its downstream clinical utility, the generated images were used for disease classification tasks on the CheXpert test set ($n=500$) and for long-tailed learning evaluation on the MIMIC-CXR-LT test set ($n=\mathrm{23,550}$), demonstrating high fidelity and diagnostic relevance of the synthesized images.

Conclusions

By jointly conditioning on gaze and radiomic representations, RadGazeGen bridges the gap between human visual cognition and machine perception, improving both realism and clinical validity in medical image generation. This framework underscores the importance of incorporating anatomically grounded and disease-aware controls in diffusion-based medical image synthesis.

1. Introduction

Current clinical applications of AI encounter significant hurdles such as data scarcity, missing modalities, and dataset imbalances, highlighting the importance for enhancing the generation of high-quality medical images to bolster automated patient diagnosis, prognosis, and to expedite clinical research. In the last few years, diffusion models have gained prominence in image generation.^1,2 More recently, these models have made remarkable progress in conditional image generation, mainly in text-to-image (T2I) generation.³

These models take text descriptions or captions as input and attempt to synthesize images that have maximum resemblance with the provided description.

Recent T2I models trained on huge amounts of data have achieved state-of-the-art performance.⁴ However, this type of textual conditioning has several limitations. In the case of complex textual structuring, i.e., when the text has a description of complex objects or ill-defined contents, the T2I diffusion models show poor image generation capabilities.⁵ In addition, vice versa, to generate a complex image, the text description design is often ill-defined. Even in the case of a well-defined text description, the generated images from T2I models occasionally have missing objects, incorrect shapes, color, position, etc.⁶ This problem stems from the fact that there is an inherent ambiguity of obviousness between text description and real-life images. A simple example is “an elephant and a bird flying” intuitively means that only the bird is flying. However, the ambiguity can be that in an unrealistic scenario both the elephant and the bird are flying. Hence, instead of generating an image where there is an elephant on the ground and a bird in the sky, the generated image shows both the elephant and bird flying.⁵ This particular problem is in fact more common in the case of medical images. For example, if the radiologist text indicates that there is a presence of a certain abnormality, “enlarged cardiac mediastinum,” but it is not very specific about the location and severity, the generated CXR can have enlargement is any direction and may have less severe enlargement than anticipated. Hence, location and severity are critical to clinically accurate image generation. Even after several iterations of prompt design, the exact desired image is extremely difficult to generate. Another important problem is that radiology reports contain common disease related diagnostic information and often exclude any global anatomical or patient demographic information. Moreover, description of local changes in image patterns, and their relationship to the disease, are also not present in such reports. This limitation is a real challenge to medical image generation. This motivates us to explore other clinically relevant conditioning approaches.

Generating images with the desired structure, texture, and attribute location may require several trials to design an unambiguous and clearly instructed prompt. To address this, additional controls are often fed to diffusion models to generate the desired structure.⁷ These controls are fine-grained information that contain required structural, textural, and positional features to generate specific desired images. Images of edge maps, depth maps, pose skeletons, and segmentation maps can all be used as controls. Recently, remarkable progress has been made in this domain where more fine-grained additional controls such as spatial maps have demonstrated superior image generation capabilities.⁸ However, in the case of medical image generation, these controls are not well-defined, and the field is unexplored.

Medical image generation is a complex task because apart from generating high-quality images, the quality of generated anatomical structure and, more importantly, the precise representation of disease patterns are extremely critical. Current text-conditioned diffusion models fall short in effectively addressing these critical aspects of medical image generation. In medical imaging, radiomics features have long been used for various tasks involving disease diagnosis, prognosis, and treatment response assessment. Radiomics features are image transformations that capture a variety of properties of medical images pertaining to local and global changes in intensity distributions—in this process, the textural patterns of disease manifestation are quantified. In this work, we hypothesize that these different clinically-relevant radiomic feature maps can be used as specialized controls for accurate medical image generation.

In clinical domains such as radiology and pathology, experts’ visual patterns have been shown to have an impact on medical image interpretation.^9–11 These eye gaze patterns harbor contextually rich visuo-cognitive data related to the occurrence and patterns of diseases.¹² Recently, several methods have shown that integrating gaze patterns into deep learning frameworks helps improve the model performance for several diagnostic tasks.^9,10 In summary, eye gaze patterns may play a crucial role as controls for diffusion models, complementing text-based conditioning to generate disease pathologies in the relevant regions. Our next hypothesis is that the human visual attention (HVA) maps computed from these eye gaze patterns can be used as relevant controls to translate the disease occurrence and visual patterns into a diffusion model. In this work, we propose to integrate these eye gaze patterns into diffusion models for medical image generation. Previous works have explored medical image generation from text conditions where disease patterns generated can be unrealistic and not located in the desired regions. By integrating these gaze patterns, we aim to generate medical images with not just realistic disease patterns but ones where such patterns are in the regions where an expert is focusing during diagnostic interpretation.

1.1. Summary of Motivation

Owing to the ambiguity between the radiology text description and images, diffusion models face difficulty in generating realistic images with accurate disease patterns. In recent works in the nonmedical domain, the addition of spatial features as controls improves T2I generation capabilities. In medical image generation, by adding such controls, anatomical structure of the CXRs can be accurately generated; however, translating the disease patterns still remains a challenging problem.¹³ Our work is based on the premise that, to translate the disease patterns and locations, the radiologists’ eye gaze fixations and radiomic feature maps can be used as clinically relevant controls.

In this work, the radiomics filter maps are combined with HVA maps and are provided as controls to diffusion models. However, composing these filters as a single control is difficult; hence, we propose separate frameworks for radiomics filters maps and HVA maps. For training diffusion models with the radiomic filters maps, we design Radiomics ControlNet (Rad-CN) and for training diffusion models with the HVA maps, we design and Human Visual Attention ControlNet (or HVA-CN). The combined Rad-CN and HVA-CN make up our RadGazeGen architecture. We demonstrate the efficacy of our approach for several tasks. RadGazeGen outperforms other diffusion baselines, highlighting the importance of radiomic filters and HVA maps as controls to improve the clinical diagnostic quality of generated images. Furthermore, we use a pretrained disease classification model for inference on the generated images. We show that our generated images contains the disease patterns as in the training dataset distribution and achieve comparable AUCs on different disease pathologies; this is further qualitatively corroborated by our collaborating radiologists. Finally, we also show results on a long-tailed disease classification task where we generate additional CXRs of the medium and tail classes and train the classifier on the re-balanced data distribution. We observe that injection of our generated images improve the balanced accuracy and other metrics when compared with the injection of baseline generated images to the data distribution.

1.2. Contributions

The primary contributions of this work are as follows:

• •We propose a gaze-guided diffusion model RadGazeGen to generate disease pattern-preserving CXR images. This model comprises of two modules: Rad-CN, which takes radiomic feature maps as inputs to generate anatomically and texturally-aware CXR images. and HVA-CN, which takes radiologists’ eye gaze patterns and feeds the same to the Rad-CN to generate disease pattern and location-aware CXR images.
• •We demonstrate the performance of RadGazeGen for the classification of thoracic diseases and long-tailed classification tasks.

In Fig. 1, we show our RadGazeGen pipeline and compare it with existing methods. RadGazeGen takes both radiomics filter maps and HVA maps, which are computed from eye gaze patterns of radiologists, as control inputs and generates a CXR image.

Fig. 1.

(a) Baseline methods add generic controls (for example text) to the diffusion models for medical image generation. (b) Our method, RadGazeGen, combines different radiomics filter maps and lung segmentation mask with radiologist’s eye gaze patterns, and uses these controls for clinically accurate medical image generation.

2. Related Works

2.1. Eye-Gaze in Medical Image Analysis

The eye gaze of experts has demonstrated immense potential in several medical applications such as disease classification ^9,10 and segmentation,¹⁴ for various modalities, including, radiology,¹⁵ pathology,¹⁶ retinopathy,¹⁷ and ECG.^18,19 After successfully leveraging eye gaze patterns in general computer vision tasks such as object detection,^20,21 image segmentation,^22–24 action localization,^25,26 and activity recognition,^27,28 recent applications have ventured into utilizing them in the clinical domain. Understanding gaze patterns of medical professionals, such as radiologists/pathologists, and providing this information as an auxiliary signal to ML models can potentially reduce misdiagnosis.²⁹ This claim was validated by recent works,^9,10 which employed teacher–student knowledge distillation models to improve disease diagnosis. In this work, we propose a new framework of diffusion models that generate gaze-guided clinically relevant medical images.

2.2. Text-to-Image Diffusion

Since their introduction,³⁰ diffusion models have been applied extensively to image generation tasks.^1,2 Latent diffusion models (LDMs) deal with latent spaces of the images to perform the diffusion steps.³ Text-to-image diffusion models integrate text embeddings with image latent space embeddings using CLIP,³¹ achieving state-of-the-art performance in image generation tasks. Recent commercial applications of text-to-image generations are DALL-E2^32,33 and Midjourney.³⁴ Recently, LDMs have been used for different medical imaging modalities for a wide range of applications such as image-to-image translation,^35,36 reconstruction,^37–44 denoising,^45,46 registration,^47,48 segmentation,^49–54 and classification.^55–57 Despite their overall success, the ambiguity between text descriptions and images in text-to-image diffusion models can severely impact the quality of generated images and often lead to suboptimal generations.⁵

2.3. Controllable Diffusion Models

Recently, diffusion models in which one or more controls are used are gaining traction. These controls help in generating custom and task-specific images. The controls such as color variation⁵⁸ and inpainting^33,59 can be directly provided by the image diffusion process. Controllable diffusion models can be broadly classified into two categories: models trained from scratch⁶⁰ and finetuned lightweight adapters that are finetuned by freezing the pretrained T2I diffusion models.^7,61 Composer⁶⁰ uses single or multiple conditions to generate images by training large diffusion models. T2I-Adapter,⁶¹ GLIGEN,⁶² and ControlNet⁷ finetune an adapter from frozen large pretrained diffusion models. More recently, several methods such as UniControl⁶³ and UniControlNet⁶⁴ have been proposed where multiple controls are unified to generate more specific images with different styles and contents. Att-Adapter⁶⁵ and Multi-ControlNet⁷ are variations of T2I-Adapter and ControlNet, respectively, where models trained with different conditions are fused to generate images from composite conditions. However, for medical image generation, the use of additional controls is still in its infancy. Given the diversity of modalities and downstream tasks in medical images, the design of controls in this domain depends largely on the specific modality and task. Medical image translation¹³ is an important field of research that has immense clinical importance with applications ranging from generating multimodality images to data augmentations. However, this domain of image translation using diffusion models with additional controls is relatively unexplored.

2.4. Radiomics

Radiomics encompasses the computerized extraction of quantifiable data from radiological images, typically subtle radiographical features. These features, which capture morphological and textural attributes, have been shown to be effective in several clinical applications involving radiographs.^66,67 The role of such clinically relevant descriptors in generative models is unexplored.

In summary, there is no existing work that studies clinically relevant controls for radiology image generation. To address this limitation, we propose RadGazeGen, which uses radiomics filter maps and radiologists’ eye gaze patterns as controls to generate clinically accurate medical images.

3. Method

An overview of the proposed method is shown in Fig. 2. In this work, we first train individual ControlNet (CN) models with different radiomics maps and segmentation masks as a control [shown in Fig. 2(a)]. Then, we perform zero-shot inference by combining two or more of these controls [shown in Fig. 2(b)]. The proposed method RadGazeGen comprises two main modules, namely, Rad-CN and HVA-CN. For Rad-CN training, the radiomics maps and the lung segmentation mask are combined to form unified controls. HVA maps are computed from the eye gaze patterns of radiologists. For HVA-CN training, these HVA maps in a specified manner are fed as controls to the trained Rad-CN.

Fig. 2.

Overview of RadGazeGen pipeline. (a) Generated CXRs using radiomics filter maps (Canny, Gaussian-Laplacian, Sobel) and lung segmentation masks as conditioning controls. (b) Multicontrol fusion demonstrating combined Canny and Sobel filters with lung segmentation masks respectively for enhanced CXR generation. (c) Pipeline components: (1) hypothesis generation framework for control selection, (2) Rad-CN architecture employing radiomics filters for initial RadGazeGen module training, then (3) HVA-CN module applying hypotheses from (1) through hierarchical visual attention mechanisms to finalize model training.

In this section, we start by introducing the preliminary concepts that are used in developing the methodology of the paper (Sec. 3.1). In Sec. 3.2, we discuss the different components of the proposed methodology RadGazeGen. The primary components of RadGazeGen, Rad-CN is discussed in Sec. 3.2.1, and HVA-CN is discussed in Sec. 3.2.2. Finally, we discuss the inference of RadGazeGen in Sec. 3.2.3.

3.1. Preliminary

Diffusion probabilistic models, often referred to as diffusion models, are generative models that are parameterized forms of Markov chains trained using variational inference.¹ Let us consider an image input $x$, the forward process is defined as adding Gaussian noise $ϵ\sim \mathcal{N}(0,I)$ to $x$ according to a variance schedule ${\beta }_t\in \{{\beta }_0,{\beta }_1,\dots ,{\beta }_T\}$, where $T$ is the number of total time steps.

$$\begin{gathered} q(x_{1:T}|x_0)≔\prod _{t=1}^{T}q(x_t|x_{t-1}),q(x_t|x_{t-1}) \\ ≔\mathcal{N}(x_t;\sqrt{1-{\beta }_t}{x}_{t-1},\beta I), \end{gathered}$$ (1)

and the reverse process is defined as

$$p_{\theta }(x_{0:T})≔p(x_T)\prod _{t=1}^T(x_{t-1}|x_t).$$ (2)

The training is done using variational lower bound (ELBO). In our case, we train the stable diffusion (SD) model with chest X-rays (CXRs) as image $x$, and the radiologist’s reports as text conditioning, $c$, in a T2I diffusion manner. Radiologists eye gaze patterns, represented as $\mathbb{G}$, are collected when they scan a CXR during diagnosis. The radiologists also provide text transcripts after diagnosis. We compute HVA maps from $\mathbb{G}$, similar to the method described by Bhattacharya et al.⁹ In short, the HVA maps are heatmaps computed by adding Gaussian blur to $\mathbb{G}$.

3.2. RadGazeGen

An overall pipeline of RadGazeGen is shown in Fig. 2. We first train an SD model with CXRs and radiology reports as text conditions, $c_{\mathrm{sd}}$. By doing this, the SD model is generalized for CXR images (represented as SD-CXR) and can be used for further finetuning to serve different medical image generation tasks. Details of Rad-CN and HVA-CN architecture are shown in Fig. 3. Next, we discuss the two major components of RadGazeGen as follows.

Fig. 3.

Overview of the Rad-CN and HVA-CN architecture. (a) Rad-CN architecture: adapts the frozen pre-trained SD model (initially trained on CXR images and paired radiology reports) from the MIMIC-CXR dataset. A trainable copy of the SD model is fine-tuned in a ControlNet framework, enabling text-to-image synthesis with radiomics filters as conditioning. (b) HVA-CN architecture: modifies the trained CN model with the hypotheses as conditioning.

Algorithm 1.

RadGazeGen pipeline.

1: Input: CXR images $I_R$, radiology reports $c$, radiomics filters $R_i=\{R_1,\dots ,R_n\}$, gaze data $G$, disease image bag $B_K$

2: Output: Generated CXR image $x$

3: Train stable diffusion using $(I_R,c)$ to obtain SD-CXR

4: for each image $I_R$ do

5:  Compute radiomics maps $R_i$

6:  Sample timestep $t$ and noise $ϵ$

7:  Add noise to latent $z_t^R$

8:  $\widehat{ϵ}\leftarrow {ϵ}_{{\theta }_R}(z_t^R,t,c_R,R_i)$

9:  Compute loss $L_{R-CN}=\Vert ϵ-\widehat{ϵ}{\Vert }^2$

10:  Update ${\theta }_R$

11: end for

12: Compute HVA maps $V_m$ from gaze data $G$

13: for each disease type $m$ do

14:  Threshold $V_m$ to obtain mask ${\widehat{V}}_m$

15:  for each image $B_j^k\in B_K$ do

16:   Generate hypothesis $H_j^k=B_j^k\wedge {\widehat{V}}_m$

17:  end for

18:  Compute similarity $D(f_B,f_H)$

19:  Select best hypothesis ${\beta }_k=\arg \max D(f_B,f_H)$

20: end for

21: for each selected hypothesis ${\beta }_k$ do

22:  Sample timestep $t$ and noise $ϵ$

23:  Add noise to latent $z_t^V$

24:  $\widehat{ϵ}\leftarrow {ϵ}_{{\theta }_V}(z_t^V,t,c_V,V)$

25:  Compute loss $L_{\mathrm{HVA}-\mathrm{CN}}=\Vert ϵ-\widehat{ϵ}{\Vert }^2$

26:  Update ${\theta }_V$

27: end for

28: Freeze ${\theta }_R,{\theta }_V$

29: Input controls $I(R_i,V)=(x_{R_i},x_V)$

30: Fuse models $F=F_{{\theta }_R}\oplus F_{{\theta }_V}$

31: for $t=t^{*}\dots 1$ do

32:  $x_t\leftarrow F(I(R_i,V))$

33: end for

34: return generated CXR image $x$

3.2.1. Radiomics, ControlNet

The Rad-CN module is used to finetune the SD-CXR with $n$ different radiomics feature maps, ${\mathcal{R}}_i$, to generate radiomics-guided CXR images with a text condition $c_{\mathcal{R}}$. Here, ${\mathcal{R}}_i\in \{{\mathcal{R}}_1,\dots ,{\mathcal{R}}_n\}$ are computed by applying different radiomic filters to the CXR image ${\mathcal{I}}^{\mathcal{R}}$. The objective of Rad-CN is to translate from ${\bigcup }_i{\mathcal{R}}_i$ to real images $x$ based on $c_{\mathcal{R}}$. In Fig. 2(c.2), we show the different ${\mathcal{R}}_i$ is fed to ${\mathcal{F}}_{\theta }^{\mathcal{R}}$, which generates ${\widehat{\mathcal{I}}}^{\mathcal{R}}$. The training loss is defined as

$${\mathcal{L}}_{\mathrm{R}\text{-}\mathrm{CN}}({\theta }_{\mathcal{R}})={\mathbb{E}}_{z^{\mathcal{R}},ϵ(x),t,c_{\mathcal{R}},{\mathcal{R}}_i}[\Vert ϵ-{ϵ}_{{\theta }_{\mathcal{R}}}(z_t^{\mathcal{R}},t,c_{\mathcal{R}},{\mathcal{R}}_i){\Vert }_2^2],$$ (3)

where $t$ is time step, $z_t^{\mathcal{R}}$ is the noise at each time step $t$, and ${\theta }_{\mathcal{R}}$ is the trainable parameter of Rad-CN.

3.2.2. Human visual attention, ControlNet

Here, we discuss how eye gaze patterns are used as a control for CXR generation. HVA-CN [shown in Fig. 2(c.3)] has two major components: gaze-guided hypotheses generation and training based on the best hypotheses generated. First, we compute HVA maps, ${\mathcal{V}}^m$ from $\mathbb{G}$, shown in Fig. 4. Here, $m$ is the different disease types. These HVA maps are then used to generate multiple CXR hypotheses images. We then select the best hypothesis and use that as the gaze-guided control for HVA-CN training. A detailed discussion of these components is provided next.

Fig. 4.

Hypothesis generation. HVA maps are thresholded to create masks, which are then mapped to various disease pathologies to generate hypotheses.

Gaze-guided hypothesis generation

These hypotheses are multiple generated CXRs based on a bag of images ${\mathbb{B}}^K$. This bag contains CXRs of $K$ different thoracic diseases. To generate these hypotheses, first, we apply a threshold to ${\mathcal{V}}^m$, such that ${\widehat{\mathcal{V}}}^m(\mathbf{x},\mathbf{y})=\{\begin{array}{cc}255& \text{if}\mathcal{V}(\mathbf{x},\mathbf{y})\hspace{0.17em}\\ 0& \text{otherwise}\end{array}$, where $\mathbf{x},\mathbf{y}$ are the $x$- and $y$-coordinates of the image. Here, $\widehat{\mathcal{V}}$ is the image obtained after the threshold, as shown in Fig. 4. Then, we apply the bitwise AND operation to a set of images ${\mathbb{B}}_j^k$ and ${\widehat{\mathcal{V}}}^m$ as a binary mask (Fig. 4), such that $k=m$. Finally, we obtain the gaze-guided hypotheses, represented as ${\mathcal{H}}_j^k$. From these multiple hypotheses, the best hypothesis is selected by comparing it with ${\mathbb{B}}^k$. The hypothesis with the maximum similarity score is finally selected, shown as ${\beta }^k$. This score is calculated by computing the distance between the features ofsla ${\mathbb{B}}^k$, $f_{\mathbb{B}}$ and ${\mathcal{H}}_k^k$, $f_{\mathcal{H}}$, shown as $\mathcal{D}(f_{\mathbb{B}},f_{\mathcal{H}})$, shown in Fig. 2(c.1).

Training

The training of HVA-CN, represented as ${\mathcal{F}}_{\theta }^{\mathcal{V}}$, as shown in Fig. 2(c.3)

$${\mathcal{L}}_{\mathrm{HVA}\text{-}\mathrm{CN}}({\theta }_{\mathcal{V}})={\mathbb{E}}_{z^{\mathcal{V}},ϵ(x),t,c_{\mathcal{V}},\mathcal{V}}[\Vert ϵ-{ϵ}_{{\theta }_{\mathcal{V}}}(z_t^{\mathcal{V}},t,c_{\mathcal{V}},\mathcal{V}){\Vert }_2^2],$$ (4)

where $t$ is time step, $z_t^{\mathcal{V}}$ is the noise at each time step $t$, and ${\theta }_{\mathcal{V}}$ is the trainable parameter of HVA-CN.

3.2.3. Inference

During inference, the input set ${\mathcal{I}}_{({\mathcal{R}}_i,\mathcal{V})}≔({\mathbf{x}}_{{\mathcal{R}}_i},{\mathbf{x}}_{\mathcal{V}})$ is fed to fused ${\mathcal{F}}_{\theta }^{\mathcal{R}}$ and ${\mathcal{F}}_{\theta }^{\mathcal{V}}$, represented as $({\mathcal{F}}_{\theta }^{\mathcal{R}}\oplus {\mathcal{F}}_{\theta }^{\mathcal{V}})$. This is our proposed RadGazeGen architecture. The RadGazeGen module, $\widehat{\mathcal{F}}(.)$ is expressed as

$$\widehat{\mathcal{F}}({\mathcal{I}}_{({\mathcal{R}}_i,\mathcal{V})})≔\prod _{t^{*}}{\mathcal{F}}_{\widehat{\theta }}^{\mathcal{R}}\oplus {\mathcal{F}}_{\widehat{\theta }}^{\mathcal{V}}({\mathcal{I}}_{({\mathcal{R}}_i,\mathcal{V})}),$$ (5)

where $\widehat{\theta }$ are the frozen parameters of Rad-CN and HVA-CN and $t^{*}$ are the inference steps of the model. Now, we show that our model also facilitates inference using a subset of controls from the set $({\mathcal{I}}_{({\mathcal{R}}_i,\mathcal{V})})$ using the indicator function $1(.)$, and Eq. (5) can be rewritten as $\prod _{t^{*}}\sum _i^{{\mathcal{R}}_n}1(i=={\mathcal{R}}_i){\mathcal{F}}_{\widehat{\theta }}^{\mathcal{R}}\oplus {\mathcal{F}}_{\widehat{\theta }}^{\mathcal{V}}({\mathcal{I}}_{({\mathcal{R}}_i,\mathcal{V})})$.

4. Experiments and Results

We use CXR images and free-text radiology reports from the MIMIC-CXR dataset for SD training. We use the eye-gaze data from publicly available Eye Gaze Data for Chest X-rays⁶⁸ ($n=1083$) and REFLACX dataset⁶⁹ ($n=2507$). For evaluation, we use $n=110$ CXRs from the REFLACX dataset. For classification experiments, we use the CheXpert test dataset ($n=500$), and for long-tailed classification experiments, we use the MIMIC-CXR-LT dataset.⁷⁰ This dataset is divided into the training ($n=68,058$), validation ($n=300$), balanced test ($n=600$), and test ($n=23,550$) set.

4.1. Implementation

4.1.1. SD training

We use CXR images of size $515\times 512$ from the MIMIC-CXR dataset to finetune an SD v1.5 model. We train for 50,000 steps with a learning rate of $1e-05$ and batch size of 4. We use gradient checkpointing and mixed precision of “fp16.” The gradient accumulation steps are set to 4, and max_grad_norm is set to 1. Training is done on a single Quadro RTX 8000 (48 GB) GPU.

4.1.2. RadGazeGen training

To train the different components of RadGazeGen, the two eye gaze datasets are used. For training, we use $n=3580$ CXRs by combining the entire first dataset with $n=1083$ CXRs and the samples for which only single radiologist eye gaze are collected from REFLACX ($n=2507$). We remove the overlapping samples from the first dataset. For testing, we use the remaining $n=110$ from REFLACX dataset. First, we extract different radiomics filter maps using OpenCV and Sklearn libraries and lung segmentation from.⁷¹ We then train Rad-CN with these radiomics filters and lung masks. The HVA maps are computed, as discussed in Sec. 3.2.2. We train the HVA-CN with these HVA maps. For training, we use $512\times 512$ image size as input and train for 10 epochs with a batch size set to 4 and a learning rate of $1e-5$. We use mixed precision of “fp16,” and the gradient accumulation steps are set to 4.

4.1.3. Classification with generated images as test set

We generate the captions for the test set of CheXpert using RGRG.⁷² Radiomics maps are computed as described above, followed by the computation of HVA maps. The captions, radiomics filter maps, and HVA maps computed from the test set of CheXpert are used to generate CXRs using RadGazeGen. The pretrained DenseNet-121 model from TorchXRayVision⁷³ is used to predict the classes for the generated images.

4.1.4. Long-tailed (LT) classification

For long-tailed classification, we use the MIMIC-CXR-LT dataset. We identify the medium and tail classes and randomly select text reports for these medium and tail classes from the MIMIC-CXR dataset. Note that there is no potential concern about data overlap. Then, we randomly select sets of radiomic filter maps and HVA maps and combine them with the texts. For each disease pathology in the medium and tail classes, we generate CXR images using this set as controls such that for each pathology the CXR count is $n_i=1100$, where $i$ is the different medium and tail classes. In this way, all the classes in the MIMIC-CXR-LT dataset have $\ge 1000$ CXR images. A ResNet-50⁷⁴ model is then trained for LT classification.

4.2. Quantitative Results

In Sec. 4.2.1, we evaluate the quality of generated images compared with original images. In Sec. 4.2.2, we discuss the classification results on the CheXpert dataset. In Sec. 4.2.3, we discuss the long-tailed classification results on the MIMIC-CXR-LT dataset. Finally, in Sec. 4.2.4, we discuss the ablation analysis results.

4.2.1. Quality of image generation

First, we discuss the quality and diversity of the generated images. We compare our proposed method with different SD, ControlNet, and MultiControlNet baselines. Here, MultiControlNet is a variant of ControlNet, which inputs multiple controls as inputs. In Table 1, we report the Fréchet inception distance (FID) scores for the generated images and compare with T2I-Adapter, ControlNet, and Multi-ControlNet for different radiomic maps and HVA controls. RadGazeGen outperforms the baselines on four out of five controls. Though T2I-Adapter achieves the best performance for canny edge filter maps as control, RadGazeGen outperforms ControlNet and MultiControlNet on all the controls (Table 2). In Table 3, we use different quantitative metrics to assess the controllability. Structural similarity index measure (SSIM) is used for Canny, Sobel, and GL filter maps; mean intersection over union (mIoU) is used for segmentation maps overlap; and CLIP score is used to assess the generated content. RadGazeGen outperforms SD and ControlNet baselines on three out of five controls. From Tables 1 and 3, we observe that ControlNet performs better than MultiControlNet on image quality and controllability, whereas RadGazeGen performs better than MultiControlNet when both the models are using multiple controls. These results suggest the strength of the composability of Rad-CN and HVA-CN modules for better-quality image generation. It is important to note that Sobel and Gaussian Laplacian maps primarily capture edge-based information rather than the full image structure; therefore, SSIM may yield relatively lower similarity scores when applied to such edge representations.

Table 1.

FID (↓) metric is shown for different radiomics filter maps, segmentation masks, and HVA controls. We compare with different baselines. Best results in bold.

	Canny	Sobel	GL	Segmentation	HVA
T2I-Adapter	1.94	2.64	3.43	3.10	3.53
ControlNet	2.9338	2.9855	3.7689	2.6059	3.1240
MultiControl	3.1674	3.1707	2.9685	4.6870	3.0573
RadGazeGen	2.8697	2.9065	2.9111	2.5820	3.0021

Table 2.

Comparison of the proposed method with existing foundation models. We report FID and SSIM metrics. Best results in bold.

	FID (↓)	SSIM (↑)
MedFusion	2.6187	0.0082
CXR-IRGen	3.3686	0.0085
Ours	2.8697	0.70

Table 3.

Assessment of controllability using SSIM (↑), mIoU (↑), and CLIP-score (↑) for different controls. Best results in bold.

	Canny (SSIM)	Sobel (SSIM)	GL (SSIM)	Segment. (mIoU)	Style/content (CLIP score)
SD	0.3091	0.0080	0.0090	0.3111	27.6936
RoentGen	0.1942	0.0079	0.0087	0.3689	27.4786
T2I-Adapter	0.53	0.0079	0.0087	0.06	27.62
ControlNet	0.4526	0.0078	0.0095	0.3599	27.7032
MultiControl	0.5816	0.0077	0.0090	0.1056	22.2958
RadGazeGen	0.7045	0.0076	0.0101	0.3555	27.7757

4.2.2. Classification performance

The classification results on the generated images using text and other controls from the CheXpert test set are shown in Table 4. Disease-wise AUC for the RadGazeGen-generated images are compared with the text-to-image diffusion model-generated images. For context, we also provide the AUCs of the model evaluated on the real images (first row) from the CheXpert test dataset. RadGazeGen outperforms the text-to-image baseline on seven out of 12 baselines, further validating the quality of the generated CXRs using the radiomics feature and HVA maps as controls.

Table 4.

Classification performance on generated images using text and other controls from the CheXpert test dataset. AUC(↑) is reported for different disease pathologies. Best results in bold.

	Ate.	Cmg.	Cns.	Edm.	Les.	Opa.	Eff.	PTh	Pne.	PTX	Fra.	Enl.	Avg.
Originala	81.95	86.45	89.35	86.85	78.76	90.31	89.77	85.04	90.09	76.49	58.26	78.11	82.62
SD	72.57	73.64	73.77	75.28	64.24	83.32	74.59	68.07	59.24	58.36	71.70	78.67	71.12
T2I-Adapter	47.8	47.2	64.2	62.3	36.3	63.0	48.0	58.6	60.64	66.2	57.2	53.76	55.43
RadGazeGen	80.71	70.34	77.67	73.47	66.57	88.20	78.31	53.29	77.46	62.74	52.32	74.90	71.33

^aThe first row reports the model’s performance on the real images from the CheXpert test dataset.

4.2.3. Long-tailed classification performance

LT classification results on the MIMIC-CXR-LT dataset are shown in Table 5. Our model is compared with ControlNet and MultiControlnet baselines. RadGazeGen outperforms both ControlNet and MultiControlNet on head and medium and reports comparable performance on the tail class samples. Overall, RadGazeGen improves the long-tailed classification performance on both the balanced and imbalanced test sets.

Table 5.

Long-tailed classification on the MIMIC-CXR-LT balanced test ($n=570$) and the test set ($n=23550$). Accuracy is reported for the individual head, medium, and tail classes and average of all the classes (Avg) is reported for the balanced set. For the imbalanced set, macro-F1 (mF1) and balanced accuracy (bAcc) are reported.

	Balanced test			Test
	Head	Medium	Tail	Avg	mF1	bAcc
ControlNet (Canny)	0.497	0.016	0.011	0.175	0.174	0.169
MultiControlNet (Canny + Seg)	0.504	0.011	0.022	0.179	0.181	0.171
RadGazeGen	0.506	0.027	0.022	0.185	0.182	0.178

4.2.4. Ablation analysis

Here, we discuss the image quality metrics for different components of RadGazeGen, shown in Table 6. We show the performance of Rad-CN, HVA-CN, and also individual radiomics maps with HVA-CN. For Rad-CN, we combined all the radiomics maps and do not use the HVA-CN component. For ${\mathcal{R}}_i+\mathrm{HVA}\text{-}\mathrm{CN}$, we use the different ${\mathcal{R}}_i\mathrm{s}$, Canny, Sobel, GL, and segmentation masks individually with HVA-CN. We observe that RadGazeGen outperforms all the individual components. Given that we use a similar experimental setup, it is interesting to note that Rad-CN performs slightly better than HVA-CN. Adding different ${\mathcal{R}}_i\mathrm{s}$ to HVA-CN improves the performance. This validates our initial hypothesis that adding radiomics filter maps to experts’ gaze patterns will improve the quality of the generated medical images as they characterize complementary attributes (not captured by radiology reports).

Table 6.

Ablation results. FID (↓), SSIM (↑), and PSNR (↑) for different components of the RadGazeGen architecture.

	FID (↓)	SSIM (↑)	PSNR (↑)
Rad-CN	3.0679	0.6373	10.6014
HVA-CN	3.0511	0.6316	10.1067
Canny + HVA-CN	3.0554	0.6373	10.2443
Sobel + HVA-CN	3.0765	0.6366	10.2286
GL + HVA-CN	3.0552	0.6368	10.2068
Segmentation + HVA-CN	3.0485	0.6362	10.2484
RadGazeGen	2.9912	0.6510	10.6014

4.3. Qualitative Results

Figure 5 shows the generated images for different disease pathologies such as consolidation, effusion, and pneumothorax. We compare the images generated from RadGazeGen with those from ControlNet and Multi-ControlNet and original images shown as a reference. We show the disease pattern annotations by a radiologist (7 year experience). We observe that RadGazeGen generates not only high-quality images but also clinically accurate disease patterns, whereas ControlNet and MultiControlNet generate visually poor-quality images as confirmed by a radiologist. The ground-truth text is shown in the first column with the corresponding disease highlighted in red (details in the Supplementary Material). Figure 7 shows the generated images along with the controls. We observe that RadGazeGen shows skillful composability of different controls. RadGazeGen generates CXRs taking into consideration the textural patterns from radiomics feature maps with adherence to the segmentation masks and translates disease patterns and location from the HVA maps. Figure 6 shows the RadGazeGen-generated images for different disease pathologies using text and other controls from the CheXpert test and MIMIC-CXR-LT datasets. We show the bounding box annotations by the radiologist in red. We observe that RadGazeGen generates clinically accurate disease patterns and hence improves disease classification performance.

Fig. 5.

Qualitative comparison. The generated CXRs for different baselines are shown along with the corresponding radiologist’s text and the original image. We also show the radiologist’s annotations of the disease occurrence (red bounding box).

Fig. 7.

(a) A single control is used to generate CXRs. (b) Multiple controls are used to generate CXRs. (c) ${\mathcal{R}}_i$ and $\mathcal{V}$ combined to generate CXRs.

Fig. 6.

(a) CXRs generated for different pathologies from the CheXpert test dataset. We show the radiologist’s annotation in red bounding boxes showing the location of the disease patterns. Additional annotations of other diseases occurrence shown in yellow bounding boxes. (b) CXRs generated for the tail classes from the MIMIC-CXR-LT dataset.

4.4. Radiologist Evaluation

To assess the clinical relevance and visual fidelity of the generated images, a radiology fellow with 4 years of experience and an attending radiologist with 8 years of experience independently scored the outputs from our proposed method and three baseline models—stable diffusion, ControlNet, and MultiControlNet. Each generated image was evaluated based on three criteria: anatomical plausibility (0 to 4), pathological realism (0 to 4), and overall image quality (0 to 2). As summarized in Table 7, our method achieved the highest mean scores across all three evaluation criteria: 3.81 for anatomy, 3.44 for pathology, and 2.00 for image quality. Compared with stable diffusion, our method showed improvements of 10.8% in anatomy, 10.3% in pathology, and 14.3% in image quality. Relative to ControlNet, the gains were even more substantial: 45.4% (anatomy), 37.6% (pathology), and 44.9% (quality). Compared with MultiControlNet, our method improved by 64.9% in anatomy, 31.3% in pathology, and 88.7% in quality. These results underscore the clinical utility of our approach in generating diagnostically meaningful synthetic images.

Table 7.

Mean radiologist scores across 16 cases for anatomy, pathology, and image quality (max scores: 4 for anatomy and pathology, 2 for quality).

Method	Anatomy (0 to 4)	Pathology (0 to 4)	Quality (0 to 2)
SD	3.44	3.12	1.75
CN	2.62	2.50	1.38
MCN	2.31	2.62	1.06
Ours	3.81	3.44	2.00

5. Discussion

The findings presented in Sec. 4 showcase the potential of RadGazeGen in generating high-quality, clinically relevant CXR images from eye gaze patterns of radiologists and other control signals such as radiomics features. This discussion will delve deeper into the implications of these results, highlight the strengths and limitations of our approach, and suggest avenues for future research.

5.1. Key Findings

5.1.1. Quality and controllability

The quantitative results, as detailed in Tables 1 and 3, unequivocally demonstrate that RadGazeGen outperforms state-of-the-art models such as T2I-Adapter, ControlNet, and Multi-ControlNet in generating images with superior FID scores, higher SSIM, and mIoU. Eye gaze patterns of radiologists add clinically relevant information about disease pathology and fine-grained location to the generative models. Furthermore, the integration of radiomics filter maps with eye gaze enables the diffusion model to generate medical images with enhanced clinical fidelity, surpassing the capabilities of SDs. This multicontrol composability is crucial for generating images that are not only visually realistic but also rich in clinically relevant details. Although the composability of multiple controls increases computational cost, the generated images demonstrate significant improvements in precision and reliability. Note that although we experimented with three radiomics filters due to computational restrictions, it can be extended to multiple other filters. In clinical scenarios, where these qualities are critical, our method offers a substantial benefit.

5.1.2. Pulmonary disease classification performance

The classification results on the CheXpert dataset, summarized in Table 4, demonstrate that the images generated by RadGazeGen are highly effective for disease classification tasks. The significant improvement in AUC scores for seven out of 12 disease categories showcases the diagnostic utility of the synthetic images. This suggests that RadGazeGen captures and preserves critical diagnostic features in the generated images, making them suitable for training and validating machine learning models.

5.1.3. Long-tailed pulmonary disease classification performance

The results on the MIMIC-CXR-LT dataset, shown in Table 5, highlight RadGazeGen’s potential in addressing class imbalance. By generating additional images for underrepresented medium and tail classes, our model enhances the classification accuracy across these categories. This approach effectively mitigates the class imbalance problem, a common challenge in medical imaging datasets. The improvement in balanced and imbalanced test set performance demonstrates RadGazeGen’s capability to generate diverse and representative synthetic data that can bolster the robustness of classification models.

5.1.4. Clinical relevance

The qualitative comparisons, depicted in Figs. 5–7, further validate the clinical relevance of the images generated by RadGazeGen. The generated images accurately reflect disease patterns and are consistent with the ground-truth annotations provided by experienced radiologists. This level of clinical fidelity is essential for applications in medical education, diagnostic support, and research. The model’s ability to produce images that are both diagnostically and anatomically indistinguishable from real images represents a significant advancement in medical image synthesis.

5.2. Limitations

Despite the promising results, our approach has certain limitations. The reliance on specific control signals, such as HVA maps, may restrict the generalizability of the model to other imaging modalities or clinical tasks where such data are not readily available. In addition, the computational complexity associated with generating and training using multiple control signals requires substantial resources, which may limit the accessibility of our approach in resource-constrained environments. Although our method is readily applicable to 3D imaging modalities, the computational cost of generating HVA maps increases significantly in 3D, thereby adding to the overall complexity of the proposed approach.

5.3. Future Directions

Incorporating eye gaze from radiologists into text-to-image diffusion models offers promising avenues for enhancing medical image generation. By aggregating gaze data from multiple experts, we can integrate diverse expertise into the generation process, potentially improving the accuracy and relevance of the produced images. Moreover, utilizing gaze patterns to generate images at different time points could provide insights into disease progression and treatment efficacy, offering a dynamic tool for personalized medicine and longitudinal studies. These advancements could significantly enhance the utility of generative models in medical imaging, contributing to better treatment decisions and patient outcomes.

Furthermore, although RadGazeGen demonstrates strong performance in thoracic disease classification and long-tailed learning tasks, its effectiveness has not yet been validated across other medical imaging domains,^75–77 which remains an important direction for future work.

Finally, incorporating additional control signals, such as patient demographics, clinical history, or genetic information, could further refine the specificity and accuracy of the generated images. This integration would allow for the creation of more contextually relevant medical images, enhancing their utility in precision medicine and patient-specific diagnostics.

6. Conclusion

In this work, we present RadGazeGen, a visual attention and radiomics-guided diffusion model for clinically accurate medical image generation. Unlike traditional text-conditioned image generation approaches, which have suboptimal information on the structure, anatomy, and disease-relevant patterns in the medical images, our approach uses radiomics filter maps and eye gaze to generate better structured, anatomically accurate, and clinically relevant disease patterns. The efficacy of our model is demonstrated in the quality of image generation and downstream disease classification problems.

Biographies

Moinak Bhattacharya is a PhD student in biomedical informatics at Stony Brook University, working under the mentorship of Prof. Prateek Prasanna. His research focuses on developing AI and deep learning methods for medical imaging, with applications in neuro-oncology and cancer treatment response prediction. He specializes in foundation models and generative modeling to advance precision medicine.

Gagandeep Singh is an assistant professor in the Department of Radiology at Columbia University Irving Medical Center, specializing in brain tumor imaging. His research uses radiomics and deep learning to identify imaging markers linked to clinical and pathological outcomes. He has expertise in MR perfusion, spectroscopy, and functional MRI. Dr. Singh completed his residency at Rutgers-Newark Beth Israel Medical Center and fellowship at NYP-Columbia, where he also contributes actively to medical education and training.

Shubham Jain is an associate professor in computer science at Stony Brook University and leads the PiCASSo Lab. Her research focuses on smart environments and cyber-physical systems, emphasizing scalable analytics for pervasive sensing devices. She previously served as an assistant professor at Stony Brook and Old Dominion University. She earned her PhD in electrical and computer engineering from Rutgers University in 2017, specializing in systems that enable intelligent, data-driven environments.

Prateek Prasanna is an associate professor in the Department of Biomedical Informatics at Stony Brook University, where he directs the Imaging Informatics for Precision Medicine (IMAGINE) Lab. His research spans radiomics, pathomics, deep learning, explainable AI, and generative AI, with a focus on developing robust and clinically meaningful AI systems for precision medicine applications across cancer, ophthalmic, and dental imaging. His work has been published in leading venues including MICCAI, CVPR, ECCV, NeurIPS, Radiology, and IEEE Transactions on Medical Imaging, resulting in over 200 publications, 8 issued patents, and multiple innovation awards, including the NSF CAREER Award. He also serves as the founding co-director of the Radiology Informatics Microcredential Program at Stony Brook University and leads efforts in multimodal AI that integrate imaging, pathology, and genomics to advance AI-powered diagnostics and treatment response assessment.

Funding Statement

This research was partially supported by the National Institutes of Health (NIH) and National Cancer Institute (NCI) (Grant Nos. 1R21CA258493-01A1, 1R01CA297843-01, 3R21CA258493-02S1, and 1R03DE033489-01A1) and the National Science Foundation (NSF) (Grant No. 2442053). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Disclosures

The authors have no relevant financial interests in the paper and no other potential conflicts of interest to disclose.

Code and Data Availability

The datasets used in this work are publicly available. Eye gaze data for chest X-rays is available at https://physionet.org/content/egd-cxr/1.0.0/, the REFLACX dataset is available at https://physionet.org/content/reflacx-xray-localization/1.0.0/, the CheXpert dataset is available at https://stanfordmlgroup.github.io/competitions/chexpert/, and the MIMIC-CXR-LT dataset is available at https://physionet.org/content/cxr-lt-iccv-workshop-cvamd/2.0.0/. Code is available at https://github.com/lordmoinak1/radgazegen.