Cross-modality synthesis of ultra-widefield fluorescein angiography from ultra-widefield color fundus photography for diabetic retinopathy via UWFDR-GAN

Abstract

Background

While ultra-widefield fluorescein angiography (UWF-FA) is essential for evaluating retinal vascular pathology in diabetic retinopathy (DR), its invasive nature limits its clinical application. This study aimed to develop and evaluate UWFDR-GAN, a generative adversarial network (GAN) framework for translating ultra-widefield color fundus photography (UWF-CFP) into UWF-FA specifically for DR patients.

Methods

A total of 270 paired UWF-CFP and UWF-FA images were collected from patients with DR, comprising 73 pairs of mild non-proliferative diabetic retinopathy (NPDR), 47 pairs of moderate NPDR, 82 pairs of severe NPDR, and 68 pairs of proliferative diabetic retinopathy (PDR). We first employed a self-supervised keypoint detection framework for precise cross-modal image registration. The generation network incorporated discrete wavelet transform/inverse transform (DWT/IDWT) to preserve high-frequency details and a Swin Transformer-based multi-scale discriminator to enhance structural realism. We quantitatively compared the performance of our model against several state-of-the-art methods, including pix2pix, pix2pixHD, and UWAFA-GAN, using objective evaluation metrics: the Multi-Scale Structural Similarity Index Measure (MS-SSIM), Peak Signal-to-Noise Ratio (PSNR), Fréchet Inception Distance (FID), and Inception Score (IS).

Results

UWFDR-GAN achieved the best quantitative performance (MS-SSIM: 0.7214; PSNR: 20.00; FID: 77.48; IS: 1.0123), outperforming all comparison models. Qualitatively, it preserved global vascular architecture and demonstrated superior reconstruction of DR-specific lesions, particularly neovascularization and non-perfusion areas.

Conclusions

UWFDR-GAN provided a non-invasive ultra-widefield vascular assessment solution for clinical DR management, demonstrating potential to reduce reliance on invasive fluorescein imaging.

Introduction

Diabetic retinopathy (DR) is a severe retinal microvascular complication of diabetes and a leading cause of vision loss and blindness worldwide, with a prevalence of 22.27% among the diabetic population [1]. It is characterized by microvascular changes, inflammation and retinal neurodegeneration, which lead to pathologies such as capillary occlusion, ischemia and vascular leakage [2]. Early detection and timely intervention are crucial for preventing severe visual impairment.

Ultra-widefield (UWF) imaging has fundamentally transformed the clinical approach to DR. By capturing up to a 200° view of the retina in a single image, UWF color fundus photography (UWF-CFP) has revealed that a significant proportion of DR lesions occur exclusively in the retinal periphery outside the standard Early Treatment Diabetic Retinopathy Study (ETDRS) seven-field [3]. According to the DRCR Retina Network Protocol AA study, predominantly peripheral lesions identified on UWF fluorescein angiography (UWF-FA) were significantly associated with an increased risk of disease worsening [4]. Furthermore, the study established UWF-FA as an essential tool in modern DR management, providing prognostic information that allows for more accurate risk assessment and clinical decision-making than UWF-CFP alone. However, the procedure of UWF-FA is invasive, requiring the intravenous injection of a fluorescent dye, and carries a risk of adverse reactions [5, 6]. Moreover, its implementation in primary care settings is largely impractical due to prohibitive costs of the equipment.

To bridge the gap between clinical necessity and practical inaccessibility of UWF-FA, cross-modality image generation has emerged as a promising solution. Research into generating FA from CFP gained traction around 2018 [7–9]. Generative adversarial networks (GANs) and their variants have achieved significant breakthrough in recent years. Early models like conditional GANs and the Fundus2Angio established the feasibility of this approach, inspiring numerous other GAN variants [10–17]. More recently, the focus has recently shifted to UWF domain, where models like Ultra-Wide-Angle Fluorescein Angiography GAN (UWAFA-GAN) represent significant progress [18–20].

However, a key limitation of most existing models is their reliance on training datasets composed of mixed ocular diseases, rather than focusing on a specific disease. To address these shortcomings, we developed UWFDR-GAN, a GAN-based model aiming at converting UWF-CFP images to UWF-FA images specifically for DR and demonstrated its superior performance for this critical application.

Methods

Data collection and preprocessing

Patients diagnosed with DR according to ETDRS in Guangdong Provincial People’s Hospital from January 2023 to December 2024 were enrolled [21]. The UWF-CFP and arterial-venous phase (30–90 s after fluorescein injection) UWF-FA images of these patients were collected. The exclusion criteria were as follows: (1) previous retinal laser treatment; (2) severe refractive media opacities; (3) presence of other fundus vascular diseases; (4) image pairs that were difficult to register; (5) low-quality image pairs. All images were captured by Optos California.

For patients with bilateral DR, images were collected from the eye presenting with the more severe DR grade. A total of 270 image pairs were ultimately included in the analysis. According to the International Clinical Diabetic Retinopathy severity scale, these image pairs were categorized into four groups: mild non-proliferative diabetic retinopathy (mild NPDR, 73 pairs), moderate NPDR (47 pairs), severe NPDR (82 pairs), and proliferative diabetic retinopathy (PDR, 68 pairs). The dataset was randomly partitioned into training and test sets with an 8:2 ratio.

After data collection, we first preprocessed the input UWF-CFP and UWF-FA images by standardizing their dimensions to 3 × 768 × 768 and applying Z-score normalization to each sample. To optimize image pair clarity, we applied the Contrast Limited Adaptive Histogram Equalization (CLAHE) algorithm, with a contrast limiting threshold of 2.0 and a tiling grid size of 8 × 8.

Vessel segmentation

Vascular segmentation was performed using two groups of nnU-Nets with identical architectures [22]. Each network comprised six upsampling and downsampling layers, with channel counts of 32, 64, 128, 256, 320, and 320, respectively. To mitigate the dependence on large fully annotated datasets, an iterative high-confidence self-training strategy was applied. Initially, samples from the RECOVERY-FA19 and PRIME-FP20 were used to pretrain five nnU-Nets [23]. Subsequently, each sample was segmented by all five nnU-Nets, and those with a mean overlap exceeding 90% were considered high-confidence samples. These high-confidence samples, together with public dataset samples, were used to retrain the nnU-Nets. The procedure was repeated three times to yield the final nnU-Nets. The final segmentation results were obtained by averaging the outputs of the trained models.

Cross-modality registration

UWF-CFP images were registrated to UWF-FA images by learning anatomically meaningful vascular keypoints and estimating a single spatial transformation between the two modalities (Fig. 1). For keypoints detection, a U-Net-based keypoint detector was trained with limited manual annotations of vascular bifurcations and crossings, and regularized through two priors: (1) geometric consistency, enforcing correspondence under known spatial transforms; and (2) anatomical consistency, constraining keypoints to vascular topology features [24]. Operating on vessel segmentation results, this framework facilitated cross-modal normalization and ensured robust alignment across modalities. The priors were further extended to unannotated data to enhance the model’s generalization capability. Matched keypoints between UWF-CFP and UWF-FA images were used to compute a robust planar transformation via Scale-Invariant Feature Transform (SIFT) descriptors and Random Sample Consensus (RANSAC) [25].

Fig. 1.

Self-supervised keypoint detection framework based on U-Net

For visual evaluation of registration accuracy, vascular masks from aligned UWF-CFP and UWF-FA images were pseudo-color fused using green and red channels respectively. To systematically assess tissue continuity, the registered image pairs were also partitioned into 100 × 100-pixel alternating display blocks enabling cross-modal structural consistency verification at subregional levels.

Generative model architecture

In the generation phase, registered UWF-CFP and UWF-FA image pairs were used as input data to perform image translation via an improved Pix2PixHD model. Specifically, two targeted improvements were introduced to better capture vascular and lesion structures: (1) a wavelet-integrated generator to preserve high-frequency vascular detail and lesion boundaries, and (2) a multi-scale Swin Transformer discriminator to enhance synthesis realism [26].

In the wavelet-integrated generator, convolutions and deconvolutions were replaced with discrete wavelet transform (DWT) and inverse DWT (IDWT) to minimize the loss of microvascular detail caused by stride-based down/up-sampling. As illustrated in Fig. 2, DWT decomposed features into low-frequency content and high-frequency sub-bands emphasizing vessel edges, bifurcations, and leakage fronts. These components were processed in parallel and then reconstructed via IDWT, thereby improving the fidelity of thin vessels and lesion borders while reducing artifacts. For the discriminator, the original multi-scale convolutional discriminator was upgraded to a Swin Transformer-based design operating across multiple image scales. The windowed self-attention mechanism captured long-range dependencies that shape the retinal vascular tree (e.g., arcades, branching patterns) while maintaining sensitivity to local lesions. The structure-aware adversarial feedback further improved the realism of the synthesized UWF-FA image.

Fig. 2.

Complete architecture of UWFDR-GAN

Implementation details

To ensure effective training for all models, we employed task-specific loss functions and data augmentation strategies. With respect to loss functions, we used a hybrid loss function that equally combines cross-entropy and Dice losses (ratio 1:1) for segmentation. For registration, training primarily targeted the keypoint-detection module, utilizing a Dice-based loss function. For generation, we followed the Pix2PixHD objective, which consists of adversarial and feature-matching losses (ratio 1:1). Regarding data augmentation strategies, we employ the default data augmentation protocols of nnUNet for segmentation, such as affine transformations, Gaussian noise, contrast adjustments, and similar techniques. For registration, the augmentation strategy encompassed Gaussian blur, contrast alterations, and illumination modifications. For generation, we utilized solely affine transformation-based augmentation. Additionally, other hyperparameters and training strategies, including the optimizer, learning rate, batch size, and number of epochs, are detailed in Table 1.

Table 1.

Training hyperparameters for each task

Task	Optimizer	(Initial) learning rate	Batch size	Number of epochs
Segmentation	Stochastic Gradient Descent	0.01	13	1000
Registration	Adam	0.0001	1	300
Generation	Adam	0.0002	2	70

Quantitative evaluation

We presented global and local lesion visual comparison of various benchmarked models such as pix2pix, pix2pixHD and UWAFA-GAN [18, 27, 28]. For quantitative evaluation, we employed four metrics. The Multi-Scale Structural Similarity Index Measure (MS-SSIM) evaluates the generation of fine vessels and peripheral regions by assessing image luminance, contrast, and structure at various scales, while the Peak Signal-to-Noise Ratio (PSNR) quantifies pixel-level differences [29, 30]. Furthermore, we assessed overall realism and diversity using the Fréchet Inception Distance (FID), which compares the feature distributions of generated and real UWF-FA images, and the Inception Score (IS), which measures image quality and diversity via an Inception-based classifier [31–33].

Clinical validation

A comprehensive clinical evaluation was designed and conducted, involving two ophthalmologists (A and B), each with over five years of experience in DR. The 54 image pairs in the test set were randomly divided into two non-overlapping subsets of 27 pairs each. The first subset of 27 pairs was used for a classic Turing test to evaluate the perceptual realism of the generated images. The 27 real UWF-FA images and their 27 corresponding generated counterparts were pooled, resulting in a set of 54 images. These images were presented to the readers in a randomized and fully blinded sequence. For each image, ophthalmologist A and B were asked to perform a single task: classify the image as either “Real” or “Generated”. Diagnostic performance was evaluated using accuracy and confusion matrices, and inter-rater agreement was assessed with Cohen’s Kappa coefficient.

The second subset of 27 pairs was used to evaluate diagnostic agreement for critical lesions. Ophthalmologist A and B were provided with UWF-CFP and generated UWF-FA images and were asked to identify the presence or absence of neovascularization (NV) and non-perfusion areas (NPA). Sensitivity, specificity, and accuracy were calculated using real UWF-FA images as the gold standard. Inter-rater agreement was quantified with Cohen’s Kappa coefficient.

External validation

To assess the model’s generalizability, we performed external validation on PRIME-FP20 dataset, which contains paired UWF-CFP and UWF-FA images. All DR image pairs from the dataset were selected. The image generation procedure followed the same protocol as that used for the test set. Performance was quantified using the same metrics (MS-SSIM, PSNR, FID, and IS) by comparing the generated images to their ground truth counterparts.

Results

Summary of the study

Our study achieved accurate cross-modal registration between UWF-CFP and UWF-FA images, and developed a GAN framework trained on the DR dataset to enable the generation of UWF-FA from non-invasive UWF-CFP. The framework was validated through quantitative metrics.

Registration performance

Figure 3 demonstrated vascular mask overlay effects, where yellow overlapping zones covering major vessels and branches indicate successful registration between UWF-CFP and UWF-FA masks. Figure 4 employed checkerboard fusion to visually verify seamless transition consistency, revealing that while minor spatial shifts occurred in peripheral branches (particularly temporal quadrant), global vascular architecture remained coherent across modalities.

Fig. 3.

Overlay of UWF-CFP/FA vascular mask. Green lines correspond to UWF-CFP vessel segmentation masks, red lines correspond to UWF-FA vessel segmentation masks, and yellow lines correspond to overlapping areas of red and green

Fig. 4.

Checkerboard fusion of registered UWF-CFP/FA images

Quantitative evaluation of image generation

Quantitative evaluation results were demonstrated in Table 2. Our model consistently outperformed other frameworks across the key metrics. Notably, it achieved significant improvement over the next-best model, UWAFA-GAN, with a superior MS-SSIM score (0.7214 vs. 0.6644), reflecting higher sensitivity in identifying true lesions and greater fidelity in reconstructing fine vascular details such as the boundaries of NPA. The improvement in PSNR (20.00) indicated reduced global pixel distortion, resulting in cleaner images with fewer distracting artifacts. Most importantly, the substantial reduction in FID suggested that the generated images were perceptually much closer to ground truth, confirming the high realism and reliability for clinical review.

Table 2.

Objective evaluation for model generation effectiveness

	MS-SSIM (95% CI,↑)	PSNR (95% CI, ↑)	FID (95% CI, ↓)	IS (mean ± SD,↑)
pix2pix	0.6354 [0.6160, 0.6509]	19.17 [18.86, 19.58]	111.39 [101.83, 122.08]	1.0110 ± 0.0010
pix2pixHD	0.6544 [0.6353, 0.6731]	19.18 [18.79. 19.54]	90.75 [81.08, 100.15]	1.0121 ± 0.0012
UWAFA-GAN	0.6644 [0.6508, 0.6860]	19.06 [18.67, 19.42]	96.24 [87.71, 105.81]	1.0104 ± 0.0010
UWFDR-GAN (Ours) *	0.7214 [0.7044, 0.7441]	20.00 [19.62, 20.35]	77.48 [69.23, 88.06]	1.0123 ± 0.0014

CI, confidence interval; SD, standard deviation; MS-SSIM, multi-scale structural similarity index measure; PSNR, peak signal-to-noise ratio; FID, Fréchet inception distance; IS, inception score; ↑, a higher value is better; ↓, a lower value is better; *, p < 0.05 by pairwise t-test between models

Clinical validation

The two ophthalmologists correctly identified over 88% of the real UWF-FA images. Ophthalmologist A misidentified 59.3% (16 out of 27) of the generated images as real, while ophthalmologist B misidentified 66.7% (18 out of 27). As detailed in Table 3, both examiners demonstrated high sensitivity and specificity in detecting NV and NPA with generated UWF-FA images, with inter-rater agreement reaching a substantial level.

Table 3.

Clinical validation of generated images

	Examiner	Sensitivity	Specificity	Accuraty	Cohen’s Kappa
NV	A	95.0%	71.4%	88.9%	78.6%
	B	100%	85.7%	96.3%
NPA	A	54.5%	93.8%	77.8%	74.6%
	B	81.8%	93.8%	88.9%

NV, neovascularization; NPA, non-perfusion areas

Global synthesis performance

Figure 5 visually illustrated the global synthesis performance of each model on the test set. In comparison with the ground truth, images generated by pix2pix and pix2pixHD exhibited abnormal textures and vascular discontinuities. The results from UWAFA-GAN were superior to the former two, while our UWFDR-GAN achieved the best overall performance, with its synthesized images most closely resembling the real UWF-FA images.

Fig. 5.

Comparison of UWF-FA global effects generated by different models

Local lesion synthesis performance

The performance disparities among the generative models in detailing local lesion features were further amplified through comparison in Fig. 6. Notably, UWFDR-GAN demonstrated superior pathological characterization, with the generated lesion regions exhibiting a high degree of agreement with the real angiographic images in terms of vascular morphology. In contrast, UWAFA-GAN and pix2pixHD were only capable of reflecting macroscopic differences in local perfusion, and presented anomalous anastomoses between retinal and neovascular vessels. The pix2pix model revealed fundamental limitations, manifesting discontinuous vascular generation alongside extraneous textures unrelated to vascular trajectories.

Fig. 6.

Comparison of local lesions in UWF-FA images generated by different models. (A, B) NV (circled in yellow); (C) NV (circled in yellow) and NPA (yellow arrows); (D) pre-retinal hemorrhage (red arrows) and NPA (yellow arrows); (E) cotton-wool spots (circled in yellow)

The lesion-specific comparison revealed that the performance of UWFDR-GAN varied across different types of pathologies. NV achieved optimal synthesis fidelity, approximating localized hyperfluorescence patterns of real angiograms despite imperfect lesion boundary (A, B, C). The generation of NPA (C, D) was also prominent, manifesting as relatively hypofluorescent regions with localized vascular texture absence, although reduced contrast was observed in a limited cases. UWFDR-GAN successfully reconstructed fluorescence blocking caused by pre-retinal hemorrhage (D), contrasting with diminished or absent hemorrhagic regions in other models. Additionally, the hypofluorescent features of cotton-wool spots (E) were also captured in the generated images.

External validation

In external validation using the public PRIME-FP20 dataset, our model demonstrated strong structural synthesis, achieving MS-SSIM values of 0.6510 and PSNR values of 19.58. The FID increased to 150.40 and IS to 1.1688, which was attributable to the small sample size of the external dataset and residual misalignments between image pairs.

Discussion

In this study, we successfully developed UWFDR-GAN, a novel GAN designed to translate non-invasive UWF-CFP into UWF-FA images for patients with DR. By providing a safe, rapid, and accessible alternative to conventional invasive angiography, our approach addresses a critical unmet need in ophthalmic diagnostics and embodies a true bench-to-bedside translation of deep learning.

A foundational challenge before model training is the precise alignment of UWF-CFP and UWF-FA images. Peripheral retina area saw more distortions while posterior pole exhibits superior alignment due to its stable anatomical landmarks [34, 35]. Previously, Zhang et al. proposed a two-step registration method based on deep convolutional networks and a distortion correction method and Martı ´nez-Rı ´o et al. present a weakly supervised deep learning methodology for robust deformable registration of multimodal retinal images [36, 37]. However, their efficacy significantly diminishes when applied to UWF images. To address this, we developed self-supervised keypoint detection framework, which automatically expanded manually annotated keypoints and improved global registration point density. In the model training phase, our focus on DR-specific training dataset enabled UWFDR-GAN to learn the specific features of DR lesions. Architecturally, the integration of DWT and IDWT preserved high-frequency details, and the use of a Swin Transformer-based multi-scale discriminator enhanced the model’s ability to enforce structural realism.

In large-scale screening settings, reliance on CFP alone can underestimate risk, as critical pathologies are always invisible. By generating UWF-FA images, UWFDR-GAN enables the precise identification of high-risk individuals who might otherwise be overlooked, facilitating timely referral. In resource-limited regions, technicians can capture fundus images, and our model can generate FA-like images which are then transmitted to remote specialists for expert interpretation. This approach bridges geographical and resource gap, ensuring that retinal vascular assessments are feasible in primary care hospitals. Furthermore, UWFDR-GAN provides a non-invasive alternative for assessing retinal vasculature in patients with renal disease and pregnant women, who are contraindicated for FA due to the nephrotoxicity and teratogenicity risks of fluorescein.

As the results showed, UWFDR-GAN excels in the high-fidelity reconstruction of NV and NPA. Its capacity to synthesize the characteristic hyperfluorescent leakage from NV serves as a non-invasive confirmation of high-risk proliferative disease, facilitating more accurate therapeutic intervention, while clear visualization of NPA provides ophthalmologists with a detailed ischemic map, offering a direct guide for precise retinal photocoagulation.

Effective management of DR requires regular follow-up to adjust treatment plans according to the disease’s dynamic progression. While the invasive nature prevents the use of UWF-FA for frequent monitoring, the application of UWFDR-GAN overcomes this critical limitation by enabling non-invasive, longitudinal tracking of DR. It empowers ophthalmologists to objectively monitor ischemic progression by quantifying key prognostic markers, such as the number and activity of NV and the expansion rate of NPA, allowing for more proactive interventions.

Furthermore, the clinical utility of UWFDR-GAN extends to patient communication and education. For individuals who are not suitable for UWF-FA, retinal ischemia remains an abstract concept, leading to poor treatment adherence. By generating a retinal vascular map in real-time during a consultation, clinicians can visually demonstrate the presence of retinal lesions, emphasize the urgency for intervention and promote patient engagement in self-management.

Despite the promising results, our study has relevant limitations that frame the context for future work. First, our model was trained on a single-center dataset with a limited number of cases. The imbalanced proportion of NPDR versus PDR cases (74.8% vs. 25.2%) may have resulted in insufficient weight allocation for learning the complex features of advanced PDR. Second, a key challenge in UWF-CFP to UWF-FA translation is the inherent anatomical interference from the choroid. UWF-CFP captures both the retinal and choroidal vascular networks, and in DR patients, pathologically attenuated choroidal thickness can enhance the visibility of choroidal vessels through the retina, creating structural ambiguity and artifacts that can interfere with the training process [38].

It is crucial to emphasize that UWFDR-GAN is intended as a clinical decision-support tool to augment rather than conventional UWF-FA. Clinicians should remain aware of its inherent risks, including the potential to miss subtle pathologies or generate plausible but incorrect artifacts. Consequently, any clinical decisions should be based on a comprehensive evaluation of all available clinical information.

Future research will focus on addressing these limitations. Our primary goal is to enhance model generalizability and robustness by training and validating on a large-scale, multi-center, multi-device dataset. Another key priority is enhancing clinical safety, for instance by developing uncertainty maps that highlight regions of low confidence, thereby serving as an important safeguard against potential misinterpretation.

In conclusion, this study introduces UWFDR-GAN, a novel generative framework that accurately translates UWF-CFP into UWF-FA for DR. The model demonstrates superior performance in generating both global vascular architecture and local lesions, providing a safer and more accessible approach to DR management.

Author contributions

Research design and supervision: DC, QF and LZ; Data collection and processing: ZX, HL, AL, ZX, TL and DH; Model design and training: TW and GL; Writing: ZX and TW; Review and editing: DC and DY. All authors read and approved the final manuscript.

Funding

This study was supported by grants from National Natural Science Foundation of China (No. 82371063 and 62471214) and the National Key Research and Development Program of China (No. 2024YFA1012002).

Data and code availability

The datasets are not publicly available due to privacy considerations. The code is available at https://github.com/SPWtZzt/UWFDR-GAN.

Declarations

Ethics approval and consent to participate

This study was performed in accordance to the tenets of the Declaration of Helsinki and approved by the Medical Research Ethics Committee of the Guangdong Provincial People’s Hospital (No. KY-H-2022-017-02).

Consent for publication

All authors agree to publish this manuscript.

Competing interests

The authors declare that they have no competing interests for this work.