Weakly-Supervised Vessel Detection in Ultra-Widefield Fundus Photography Via Iterative Multi-Modal Registration and Learning

Abstract

We propose a deep-learning based annotation-efficient framework for vessel detection in ultra-widefield (UWF) fundus photography (FP) that does not require de novo labeled UWF FP vessel maps. Our approach utilizes concurrently captured UWF fluorescein angiography (FA) images, for which effective deep learning approaches have recently become available, and iterates between a multi-modal registration step and a weakly-supervised learning step. In the registration step, the UWF FA vessel maps detected with a pre-trained deep neural network (DNN) are registered with the UWF FP via parametric chamfer alignment. The warped vessel maps can be used as the tentative training data but inevitably contain incorrect (noisy) labels due to the differences between FA and FP modalities and the errors in the registration. In the learning step, a robust learning method is proposed to train DNNs with noisy labels. The detected FP vessel maps are used for the registration in the following iteration. The registration and the vessel detection benefit from each other and are progressively improved. Once trained, the UWF FP vessel detection DNN from the proposed approach allows FP vessel detection without requiring concurrently captured UWF FA images. We validate the proposed framework on a new UWF FP dataset, PRIME-FP20, and on existing narrow-field FP datasets. Experimental evaluation, using both pixel-wise metrics and the CAL metrics designed to provide better agreement with human assessment, shows that the proposed approach provides accurate vessel detection, without requiring manually labeled UWF FP training data.

I. Introduction

Ophthalmologists recognize features of retinal vasculature as important biomarkers associated with multiple diseases. For example, diabetic retinopathy and retinal vein occlusion are characterized by increase in retinal vasculature tortuosity, vessel caliber expansion, and retinal non-perfusion [1]. Therefore, detecting vessels is a fundamental problem in retinal image analysis that has been extensively researched. Existing approaches can be classified into two main categories, supervised and unsupervised, depending on whether they do or do not use labeled training data [2]. Traditionally, the focus was on unsupervised methods that addressed the problem from a variety of perspectives, incuding hand-crafted match filtering [3], [4], morphological processing [5]–[7], multi-scale approaches [8], [9], and matting-based techniques [9]. Recently, supervised learning approaches, specifically deep neural networks (DNNs), have led to significant improvements in retinal vessel segmentation. A variety of DNN architectures have been proposed for retinal vessel segmentation, including per-pixel classifier [10], fully convolutional network [11], [12], U-Net [13]–[16], graph neural network [17], context encoder network [18], and generative adversarial networks [19]. Additionally, several works exploit novel loss functions [20]–[22] and training strategies [23]. These DNN based methods have primarily focused on narrow field (NF) fundus photography (FP), both because NF FP is the predominant format and modality of capture in the clinical setting and because recent efforts have created reasonable sized labeled ground truth datasets for DNN training [4], [24]–[29].

Due to the additional diagnostic information they can offer, vessel detection is also of interest in formats and modalities other than NF FP [30]. Specifically, in this paper, we focus on ultra-wide field (UWF) FP [31] leveraging concurrently captured UWF fluorescein angiography (FA) images. Like NF FP, UWF FP is noninvasive and only involves capture of the retinal images under low-power illumination; even pupil dilation is not required [31]. As shown in Fig. 1(a), UWF FP images provide a wide 200° field-of-view (FOV) in a single high-resolution image, as opposed to the much narrower 30°–50° FOV for NF FP. Manual examination of the UWF images in diagnosis achieves reliable performance comparable to direct clinical examination using an opthalmoscope with pupil dilation. At the same time, UWF FP also reveals additional peripheral retinal vasculature structure that is of diagnostic importance when compared to NF FP [32], [33]. UWF FA, which is shown in Fig. 1(b), represents an alternative modality that also offers a wide FOV and additional diagnostic utility, but has the limitation that it is more invasive, requiring intravenous injection of fluorescein sodium dye.

Fig. 1:

Concurrently captured ultra-widefield (UWF) fundus photography (FP) and UWF fluorescein angiography (FA) image pair. Cyan circles depict the approximate field-of-view for narrow-field FP.

While DNNs trained on NF FP can be applied to UWF FP, the performance is relatively poor in the peripheral region (as demonstrated in Section III-F). The development of DNNs specifically for detecting vessels in UWF FP has been stymied by the paucity of labeled ground truth data. Manually annotating the binary vessel maps for UWF FP is particularly time-consuming and requires clinical-expertise. High-resolution UWF FP exhibits non-uniform illumination and contrast between vessels and background, which makes it challenging and time-intensive to accurately annotate both major and minor vessels across the large FOV; estimates indicate that approximately 18 hours are required for de novo manual annotation for one UWF FP image [34]. Prior work on UWF FP vessel detection [34] therefore proposed the use of pixel-wise hand-crafted features with a shallow, two-layer, multi-layer perceptron that were trained on a limited number of small labeled patches. The approach, however, does not take full advantage of deep learning advances that employ end-to-end training and also learn features in a data-driven fashion.

In this paper, we focus on innovative methodologies that train DNNs for UWF FP vessel detection in an annotation-efficient fashion and eliminate the requirement of manually labeled datasets for supervised learning. To this end, we make the following contributions:

• We present a novel iterative framework for vessel detection in UWF FP using DNNs that does not require de novo labeled UWF FP vessel maps. Instead, we rely on datasets that also include concurrently captured UWF FA images, for which effective deep learning approaches for vessel detection have recently become available allowing for accurate vessel detection. The proposed framework then jointly addresses precise registration between the vessel images for the modalities and vessel segmentation in UWF FP, where the two tasks synergistically benefit each other as iterations progress despite the differences in geometry and modality.

• We construct a new ground truth labeled dataset, PRIME-FP20, to evaluate retinal vessel detection in UWF FP and to facilitate further work on this problem.

• The proposed framework provides a method for accurate vessel detection in UWF FP imagery, a modality that has received limited attention in prior works. The proposed approach significantly outperforms existing methods on the PRIME-FP20 dataset and, on NF FP datasets, achieves performance comparable with state-of-the-art methods designed specifically for NF FP.

We note that an alternative framework for joint vessel segmentation and registration on paired NF FP and NF FA images has also been proposed in [35]. This approach formulates vessel segmentation as a style transfer task (from retinal images to binary vessel maps) and uses one vessel map from the existing dataset as the style target. As we discuss in Section IV and demonstrate in the Supplementary Material, the proposed framework is more effective and achieves better performance than the method in [35].

The rest of the paper is organized as follows. Section II describes the proposed iterative registration and learning framework. In Section III, we perform the detailed analysis of the proposed framework and present the experimental results of vessel detection. A discussion of prior work in [35] that addresses a similar problem and the broader utility of the proposed framework is included in Section IV. Section V concludes the paper.

II. Iterative Registration and Learning approach

As already mentioned, instead of labeled data, training in the proposed approach is accomplished by using a set of concurrently captured UWF FP and UWF FA images, which we denote as ${\left\{\left(X_c^i,X_a^i\right)\right\}}_{i=1}^M$, where $\left(X_c^i,X_a^i\right)$ denotes a simultaneously captured UWF FP and UWF FA image pair (in that order) and M is the number of image pairs. Importantly, we note that while the image pairs for the two modalities are captured during the same clinical visit, they are not aligned and have significant differences in geometry in addition to fundamental differences in the information they contain arising from the differences in the modalities. In the ensuing discussion, we illustrate and describe the processing for one pair $\left(X_c^i,X_a^i\right)$, the i^th pair, for situations where the same processing flow applies to all pairs.

For each UWF FA image $X_a^i$, a corresponding vessel map $Y_a^i$ is obtained using a pre-trained DNN for this modality (shown in green Fig. 2). Our implementation uses [36] though alternative approaches could also be utilized for this purpose. The training of the desired DNN for FP vessel detection is then accomplished as shown in Fig. 2 by iterating between two steps comprising (a) multi-modal registration between the estimated UWF FA vessel map and a current estimate for the UWF FP vessel map and (b) weakly-supervised learning from noisy labels. Specifically, at iteration t, using parametric chamfer alignment, the detected UWF FA vessel map $Y_a^i$ is registered with the current estimate $Y_c^{i,t}$ of the UWF FP vessel map. The UWF FA vessel map $Y_a^i$ is warped using the estimated registration transformation to obtain tentative/noisy training labels $Y_{a\to c}^{i,t}$ for pixels in the corresponding UWF FP image $X_c^i$. The collective set of such pairs of images for the concurrent UWF FP and FA captured images form the (noisy-labeled) training data ${\left\{\left(X_c^i,Y_{a\to c}^{i,t}\right)\right\}}_{i=1}^M$. The fundamental differences between FA and FP imaging modalities and invariable errors in the registration contribute to the noise in the labeling. In particular, FA imaging captures fine vessels that are not visible in FP [31]. Consequently, the warped vessel maps $Y_{a\to c}^{i,t}$ contain a large amount of “false positive” labels that are actually background in $X_c^i$.

Fig. 2:

Proposed iterative registration and learning approach for retinal vessel detection in UWF FP without requiring labeled FP vessel data. For clarity, the figure illustrates the processing flow for only the i^th pair $\left(X_c^i,X_a^i\right)$ of UWF FP and UWF FA images, from the complete set of M pairs ${\left\{\left(X_c^i,Y_{a\to c}^{i,t}\right)\right\}}_{i=1}^M$ used for the training.

In the learning step, we propose a robust weakly-supervised learning approach to train DNN that identifies and corrects the noisy labels in the generated dataset. The detected UWF FP vessel map $Y_c^{i,t+1}$, estimated by the trained DNN, is used for the registration step in the (t + 1)^th iteration.

The proposed framework iteratively addresses precise registration and vessel detection, where two tasks synergistically benefit each other as iterations progress. Precise alignment is important to obtain high-quality training labels. Even a small misalignment between the FA and FP images can significantly deteriorate the training data quality by assigning incorrect labels to the image pixels. On the other hand, accurate UWF FP vessel detection, estimated using the weakly-supervised learning approach, helps estimate the registration parameters because chamfer alignment uses detected UWF FP vessel maps for anchoring. Using concurrently captured UWF FP and FA images, the proposed framework accomplishes the training of a DNN for FP vessel detection without requiring labeled UWF FP data. Note that vessel detection in UWF FP images can be performed using the trained DNN without requiring concurrently captured UWF FA images.

Next we provide details for the registration and learning steps that constitute the two major steps in the proposed iterative framework.

A. Vessel Registration via Chamfer Alignment

Binary UWF FA vessel maps $Y_a^i$ are transferred to the corresponding UWF FP images $X_c^i$ by using a geometric transform that is estimated using the chamfer alignment technique from [36]. To make the presentation self-contained, we include a brief overview here that conveys the key intuition.

We denote the locations of estimated vessel pixels in the UWF FA vessel map $Y_a^i$ by ${\mathcal{Q}}_a={\left\{q_j^a\right\}}_{j=1}^{N_a}$, where $q_j^a$ are the 2D coordinates of vessel pixel j and N_a is the number of vessel pixels in $Y_a^i$. Similarly, the locations of the N_c vessel pixels in the estimated UWF FP vessel map $Y_c^{i,t}$ at iteration t are represented as ${\mathcal{Q}}_c^t={\left\{q_k^c\right\}}_{k=1}^{N_c}$. Chamfer alignment [37] estimates a parametric geometric transformation ${\mathcal{T}}_{\beta }$ to register the points in ${\mathcal{Q}}_a$ to those in ${\mathcal{Q}}_c^t$ by minimizing the average squared Euclidean distance between the transformed locations ${\mathcal{T}}_{\beta }\left(q_j^a\right)$ and the closest point in ${\mathcal{Q}}_c$, where β denotes the vector of parameters for the geometric transform. Specifically, define the objective function

$$L(\beta )=\frac{1}{N_a}\sum _{j=1}^{N_a}{D}_j\left(q_k^c,q_j^a\right),$$ (1)

with $D_j\left(q_k^c,q_j^a\right)={\text{min}}_k{\Vert q_k^c-{\mathcal{T}}_{\beta }\left(q_j^a\right)\Vert }^2$. Then the estimated registration transform is obtained as ${\mathcal{T}}_{\beta *}$ where β* minimizes L(β). We use a second order polynomial transformation for ${\mathcal{T}}_{\beta }$, which is parameterized by a 12-dimensional parameter vector β and has been shown to be suitable for retinal vessel registration in prior work [36], [38].

In practice, we use a refinement of the basic chamfer alignment approach outlined above that uses a latent-variable based probablistic formulation along with the expectation maximization (EM) algorithm [39] to provide robustness against outlier points that exist in ${\mathcal{Q}}_a$ but do not have correspondences in ${\mathcal{Q}}_c$. The robustness against such outliers is particularly crucial in this application setting because, as noted earlier, some fine vessels appear only in ${\mathcal{Q}}_a$ because the FA modality detects these much better than FP. We refer readers to [36] for detailed derivations of the parameter estimation with the EM approach. Here we only note that the key intuition can be understood from the fact that, in the EM approach, the arithmetic average in (1) is replaced by a weighted average where the weight for the squared error $D_j\left(q_k^c,q_j^a\right)$ corresponding to the j^th point in ${\mathcal{Q}}_a$ corresponds to the estimated posterior probability that it is not an outlier (and has a corresponding point in ${\mathcal{Q}}_c$). When these posterior probabilities are accurately estimated, the errors for the outlier points effectively drop out from the weighted average, as desired.

For the t^th iteration, once the registration transform parameters have been estimated, by applying the corresponding transformation ${\mathcal{T}}_{\beta *}$ to the UWF FA vessel maps $Y_a^i$ we obtain the warped version $Y_{a\to c}^{i,t}$ as the current estimate of the FA vessel map aligned with the FP imagery, which serves as “noisy labels” for the learning step.

B. Weakly-Supervised Learning with Noisy Labels

While the multi-modal registration provides tentative dataset ${\left\{\left(X_c^i,Y_{a\to c}^{i,t}\right)\right\}}_{i=1}^M$ to train a DNN for detecting vessels in UWF FP, the labels in $Y_{a\to c}^{i,t}$ inevitably contain noise (incorrect labels) due to the fundamental differences in FA and FP modalities. In this sub-section, we analyze the characteristic of the label noise and propose a weakly-supervised learning method to train DNN against label noise.

FA imaging is able to capture the fine retinal vessels better than FP [31]. Consequently, the warped FA vessel maps $Y_{a\to c}^{i,t}$ contain a large number of vessel branches, especially fine vessels, that are not visible in FP modality. Figures 3(a) and Figures 3(b) show a sample UWF FP patch selected from the peripheral region and the corresponding warped UWF FA vessel map, respectively. From these two figures, one can appreciate that the majority of fine vessels are not captured in UWF FP image. In Fig. 3(c), we compare and visualize the differences between the warped vessel map and ground truth labels that are manually annotated from scratch by a human annotator. The red pixels in Fig. 3(c) depict a large proportion of vessel labels in the warped UWF FA vessel map $Y_{a\to c}^{i,t}$ that are actually background in the UWF FP image. On the other hand, the FP vessel pixels that are not in the warped UWF FA vessel map $Y_{a\to c}^{i,t}$, shown in blue in Fig. 3(c), are a rather small fraction of the FP vessel pixels. Thus, treated as an estimate of the FP vessel map, the warped FA vessel map $Y_{a\to c}^{i,t}$ has low precision but high recall. Therefore, the label noise in $Y_{a\to c}^{i,t}$ is asymmetric: the background labels are largely accurate and the vessel labels potentially have errors.

Fig. 3:

(a) Sample UWF FP patch and (b) the corresponding labels from warped UWF FA vessel map. Red and blue pixels in (c) indicate incorrect vessel labels (“false positive”) and background labels (“false negative”), respectively, in the warped UWF FA vessel map.

We exploit the asymmetry of the label noise and propose a weakly-supervised learning approach to train a DNN using $Y_{a\to c}^{i,t}$ as noisy labels. Formally, we divide pixels in $Y_{a\to c}^{i,t}$ into two sets, ${\mathcal{Y}}_v^{i,t}$ and ${\mathcal{Y}}_b^{i,t}$, where pixels in ${\mathcal{Y}}_v^{i,t}$ are labeled as vessels (white pixels in Fig. 3(b)) and those in ${\mathcal{Y}}_b^{i,t}$ are labeled as background (black pixels in Fig. 3(b)). We further denote ${\mathcal{Y}}_v^t={\mathcal{Y}}_v^{1,t}\cup {\mathcal{Y}}_v^{2,t}\cup \cdots {\mathcal{Y}}_v^{M,t}$ and ${\mathcal{Y}}_b^t={\mathcal{Y}}_b^{1,t}\cup {\mathcal{Y}}_b^{2,t}\cup \cdots {\mathcal{Y}}_b^{M,t}$. Our goal is to train a DNN, modeled as a function f with learnable weights W, that outputs a probabilistic vessel map Y_c = f(X_c; W) in response to an input FP image X_c. In the t^th iteration, weight parameters W^t for the DNN are estimated by minimizing the binary cross-entropy loss, viz.,

$${\mathcal{L}}^t=\frac{1}{\left|{\mathcal{Y}}_v^t\right|+\left|{\mathcal{Y}}_b^t\right|}\left(\sum _{v\in {\mathcal{Y}}_v^t}{l}_v^t+\sum _{b\in {\mathcal{Y}}_b^t}{l}_b^t\right),$$ (2)

where $l_v^t=-\text{log}\left(y_{c,v}^t\right)$ and $l_b^t=-\text{log}\left(1-y_{c,b}^t\right)$ are the binary cross-entropy loss computed from the predicted vessel probability $y_{c,v}^t$ and $y_{c,b}^t$ in $y_v^t$ and $y_b^t$, respectively, and | · | represents the cardinality. Our motivation is that while DNNs can be over-fitted on noisy labels with sufficient training epochs, in the early training epochs [40], DNNs tend to first learn on the correct labels. Thus the correct and the incorrect labels can be distinguished based on the loss values [41].

In Fig. 4 (left), we plot the training loss values $l_v^t$ computed after each training epoch for both correct (green) and incorrect (red) labels in $y_v^t$. At the early stage of the training, pixels with incorrect labels have larger loss values than the correctly labeled pixels, allowing one to identify the noisy labels from the loss values. In Fig. 4 (right), we show the loss distribution after 20 training epochs for both correctly and incorrectly labeled pixels. We see that the distribution is bimodal and can be modeled as a two-component mixture model.

Fig. 4:

Left: The binary cross-entropy loss for correct (green) and incorrect (red) vessel labels in the course of training. The curve shows the median loss value and the shaded region represents the range between the 15th and the 85th percentile of the loss values. Right: histogram of the training loss after 20 training epochs (indicated by the dash line in the left plot).

To estimate the distribution of $l_v^t$, we use the latent variable $Z_v^t\in \left\{0,1\right\}$ to indicate if the pixel v in $y_v^t$ is mislabeled. Given that the label is correct $\left(Z_v^t=1\right)$, the conditional probability of $l_v^t$ is modeled as an exponential distribution $\lambda \text{exp}\left(-\lambda l_v^t\right)$ with parameter λ. See the green distribution in Fig. 4. And, given $Z_v^t=0$, the conditional probability of $l_v^t$ is modeled as a Gaussian distribution $\mathcal{N}(\mu ,\sigma )$ with mean μ and standard deviation σ. See the red distribution in Fig. 4. The distribution of the mixture model for $l_v^t$ takes the form of

$$p\left(l_v^t\right)=\pi \lambda e^{-\lambda l_v^t}+(1-\pi )\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{{\left(t_v^t-\mu \right)}^2}{2{\sigma }^2}},$$ (3)

where $\pi =p\left(Z_v^t=1\right)$ is the mixing weight that represents the prior probability of latent variable $Z_v^t$. We adopt the EM algorithm [39] to fit the proposed mixture model. EM algorithm alternates between the E-step and the M-step. In the E-step, we compute the posterior probability $p_v^t=p\left(Z_v^t=1\mid l_v^t\right)$, which can be obtained using Bayes’ rule:

$$p_v^t=\frac{\pi \lambda e^{-\lambda l_v^t}}{\pi \lambda e^{-\lambda l_v^t}+(1-\pi )\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{{\left(l_v^t-\mu \right)}^2}{2{\sigma }^2}.}}$$ (4)

In the M-step, we update the parameters of the mixture model. Using the estimated posterior probability, we obtain

$$\begin{array}{l}{\pi }^t=\frac{{\sum }_{v\in {\mathcal{Y}}_v^t}{p}_v^t}{\left|{\mathcal{Y}}_v^t\right|},{\mu }^t=\frac{{\sum }_{v\in {\mathcal{Y}}_v^t}\left(1-p_v^t\right)l_v^t}{{\sum }_{v\in {\mathcal{Y}}_v^t}\left(1-p_v^t\right)},\\ {\lambda }^t=\frac{{\sum }_{v\in {\mathcal{Y}}_v^t}{p}_v^t}{{\sum }_{v\in {\mathcal{Y}}_v^t}{p}_v^t{l}_v^t},{\sigma }^t=\sqrt{\frac{{\sum }_{v\in {\mathcal{Y}}_v^t}\left(1-p_v^t\right){\left(l_v^t-u\right)}^2}{{\sum }_{v\in {\mathcal{Y}}_v^t}\left(1-p_v^t\right)}}.\end{array}$$ (5)

The process is repeated until parameters converge. The fitted mixture model provides a tool for analyzing the label noise in the warped vessel maps. The prior probability $p\left(Z_v^t=1\right)$ is an estimate of the amount of correct labels in ${\mathcal{Y}}_v$. More importantly, the posterior probability $p\left(Z_v^t=1\mid l_v^t\right)$ indicates the probability of pixel being correctly labeled, which allows us to update labels in $y_v^t$. Specifically, the label in updated ground truth ${\mathcal{Y}}_u^{i,t}$ is computed as

$$y_{u,v}^t=\{\begin{array}{lll}{p}_v^t{y}_{a\to c,v}^t+\left(1-p_v^t\right)y_{c,v}^t,\hfill & \text{if }v\in {\mathcal{Y}}_v^i,\hfill & \left(6\text{a}\right)\hfill \\ 0,\hfill & \text{if }v\in {\mathcal{Y}}_b^i.\hfill & \left(6\text{b}\right)\hfill \end{array}$$

In (6a), the updated label for pixel in ${\mathcal{Y}}_v^t$ is a linear combination of the label in the warped vessel map $y_{a\to c,v}^t$ and the predicted probability vessel map $y_{c,v}^t$ where the coefficients are determined by the posterior probability $p_v^t$. Intuitively, if the posterior probability $p_v^t$ is close to 1, we trust the label $y_{a\to c,v}^t$ in the warped vessel map because the corresponding pixel is correctly labeled. Otherwise, we reduce the weights of label $y_{a\to c,v}^t$ and rely more on the network-predicted probability vessel maps $y_{c,v}^t$. In (6b), we do not update background labels in $y_b^t$ because these labels are considered accurate.

The training process is divided into two stages. First, we train the DNN on the tentative noisy dataset ${\left\{\left(X_c^i,Y_{a\to c}^{i,t}\right)\right\}}_{i=1}^M$ for E₀ epochs. Then we fit the proposed mixture model on the loss values $l_v^t$ and obtain the updated labels $Y_u^t$. In the second stage, we continue to train the DNN on ${\left\{\left(X_c^i,Y_u^{i,t}\right)\right\}}_{i=1}^M$ for another E₁ epochs. The overall algorithm for the proposed framework is summarized in Algorithm 1.

Note that both the vessel registration and the robust learning steps utilize the EM framework to estimate the posterior probabilities of a pixel being outlier/mislabeled. However, the objectives in these two steps are different and we can not use the posterior probabilities estimated in one step for the other. In the registration step, the EM framework mitigates the effects of outlier vessel points. The outliers are defined as the vessel points in $Y_a^i$ that do not have correspondences in the current estimated vessel map $Y_c^{i,t}$. As we show in Section III-D, some vessels are not properly detected in $Y_c^{i,t}$ in the first few iterations. As a result, the outlier pixels in the registration step are not necessarily the same as the mislabeled pixels that need to be identified in the training step.

III. Experiments

In this section, we first introduce a new dataset, PRIME-FP20, that is used for implementing the proposed iterative framework and for evaluating the vessel detection performance. Next, we summarize the evaluation metrics in Section III-B, and describe the implementation details and alternative methods used as baselines in Section III-C. The experimental results are structured as follows. We provide detailed analysis to demonstrate the effectiveness of the proposed iterative framework and the weakly-supervised learning method in Section III-D and Section III-D, respectively. We then compare the proposed framework with alternative methods on the PRIME-FP20 dataset in Section III-F. Finally, we show the boarder utility of the proposed framework for detecting vessels in NF FP in Section III-G.

A. PRIME-FP20 Dataset

We construct a new dataset, PRIME-FP20 [42], for evaluating the performance of vessel detection in UWF FP. The PRIME-FP20 dataset consists of 15 pairs of concurrently captured UWF FP and UWF FA images that are selected from baseline images of patients enrolled in the PRIME study¹. The images are captured using Optos California and 200Tx cameras (Optos plc, Dunfermline, United Kingdom) [43]. The system uses a scanning ophthamoscope with a low power laser to capture dual red and green channel UWF FP images and a single channel FA image. All images have the same resolution of 4000 × 4000 pixels and are stored as 8-bit TIFF format with lossless LZW compression. The green channel UWF FP image is used as the input X_c for our vessel detection because it captures information for layers with the retinal vasculature, whereas the red channel captures information from other layers (from the retinal pigment epithileum to the choroid) [43]. For evaluation, ground truth vessel maps for the UWF FP modality are manually labeled by a human annotator using the ImageJ software [44] with the segmentation editor plugins. The available selection tools in ImageJ, such as brush tool and free-hand selection tool were used to mark the vessel pixels in the UWF FP. The annotator repeatedly adjusted image brightness and contrast to precisely label both major and minor vessel branches in different regions. For each UWF FP, we also provide a binary mask for the FOV of the image. To obtain the mask, we simply binarize the green channel of the UWF FP because the pixels intensities out of FOV are close to zero.

B. Evaluation Metrics

For quantitative evaluation, we report the area under the Precision-Recall curve (AUC PR)², the Dice coefficient (DC), and the CAL metric [45]. The computation of these metrics is summarized in Section S.III of the Supplementary Material.

The AUC PR and the Dice coefficient, although widely used in prior literature, are based on the pixel-wise comparison of the ground truth and the estimated vessel map. However, the pixel-wise comparison does not consider the structure of retinal vasculature and is sensitive to the label ambiguities, particularly for peripheral pixels that only partially belong to vessels. The CAL metric [45] is designed to be less sensitive to label uncertainties and provides better agreement with human assessment of higher level structure. CAL evaluates the consistency between the binary ground truth and the binary predicted vessel map by calculating three individual factors that quantify the consistency with respect to the connectivity (C), the area (A), and the corresponding length of skeletons (L). Each factor ranges between 0 and 1 where 1 indicates perfect consistency to the ground truth. The product of three factors is defined as the overall CAL metrics. The computation of the CAL metrics requires a binary vessel map, to obtain which, we binarize the predicted probabilistic vessel map Y_p with a threshold τ = 0.5.

For the experiments on the PRIME-FP20 dataset, we perform the K-fold cross-validation [46] to evaluate the performance of vessel detection, where K is set to 5, and report the statistics of the five evaluation metrics. We only consider pixels within the FOV mask when computing the metrics.

C. Implementation Details and Alternative Methods

To detect UWF FA vessels X_a, we train the U-Net [13] model on the RECOVERY-FA19 dataset [36] that provides eight high-resolution (3900 × 3072 pixels) UWF FA images and the ground truth vessel maps. We use the U-Net model because of its superior performance in medical image segmentation [47]. Detailed training protocol is included in the Supplementary Material (Section S.II-B). We apply the trained model to the UWF FA images $X_a^i$ and binarize the estimated vessel map $Y_a^i$ with a threshold τ = 0.5.

For the proposed iterative framework, we use the pairs of UWF FP and UWF FA images in the PRIME-FP20 dataset. Note that the proposed framework does not require the ground truth vessel maps for UWF FP in the PRIME-FP20 dataset. These manually labeled ground truth are only used for evaluation in our experiments. We implement the chamfer alignment and the weakly-supervised learning using MATLAB^™ and PyTorch [48], respectively. We perform three iterations between registration and learning (T = 3) and provide an empirical evaluation of different number of iterations in Section III-D. In the first iteration, we use a preliminary UWF FP vessel map $Y_c^{i,0}$ for chamfer alignment, which is obtained from a DNN pre-trained on existing NF FP dataset. For the weakly-supervised learning step, we use the U-Net [13] model. We set the training epochs E₀ = 25 and E₁ = 30. Detailed network architectures and training protocol are included in Section S.II of the Supplementary Material.

We consider existing learning-based vessel detection methods for as baselines for comparison. These methods include HED [49], U-Net [13], DRIU [11], CRF [50], NestUNet [14], M2U-Net [51], CE-Net [18], CS-Net [52], RU-Net [15], and IterNet [16]. For each method, we obtained two versions of the network by training on different datasets, and evaluate each version on the PRIME-FP20 dataset. The first dataset for training consisted of UWF FP images and the warped FA labels obtained from the first iteration using the proposed framework, i.e., ${\left\{\left(X_c^i,Y_{a\to c}^{i,t}\right)\right\}}_{i=1}^M$, where t = 1. The second dataset used for training was the IOSTAR [26] dataset that provides 30 images and binary vessel maps. While the images in the IOSTAR dataset are narrow-field, they are captured with the scanning laser ophthalmoscopy (SLO) technique that is also used in the PRIME-FP20 dataset³.

D. Iterative Registration and Learning Framework

We demonstrate the effectiveness of the proposed iterative framework by showing that both registration and learning benefit from each other and improve progressively.

To quantify registration accuracy, we compute the chamfer distance as the average Euclidean distance between each point in the ground truth binary vessel maps and its closest point in the transformed vessel maps detected in UWF FA. The average chamfer distance under the second-order transformation can be treated as a proxy for the registration error. The blue line with circle markers in Fig. 5(a) shows the average chamfer distance over 4 iterations. In the first iteration, the chamfer distance is on average 1.66 pixels. While the misalignment is slight, it can significantly deteriorate the quality of the training data. The generated tentative ground truth in the first iteration only has a recall of 0.63, which means that 37.0% of true vessels are labeled as background (false negative labels). The third column in Fig. 5(b) shows sample results of the generated ground truth in the first iteration, where the blue pixels highlight the false negative labels. In the third iteration, the chamfer distance drops to 0.77 pixels, yielding accurate training data with a recall increased to 0.83. The fifth column in Fig. 5(b) shows training data obtained from the third iteration.

Fig. 5:

Registration and vessel detection performance as a function of iteration count. (a) the residual chamfer distance, which serves as a good proxy for the registration error, is labeled on the left axis. The axis on the right-hand side, with labels shown in different colors, corresponds to the metrics used for evaluating vessel detection performance. (b) Sample vessel maps obtained in the first and the third iterations. Red and blue pixels indicate incorrect vessel labels (“false positive”) and background labels (“false negative”), respectively, in the warped UWF FA vessel map.

The improved ground truth dataset in turn benefits network training for vessels detection in UWF FP. The fourth and the last columns in Fig. 5(b) show the predicted vessel maps in the first and the third iteration, respectively. The yellow arrows highlight the improved vessel detections that are not correctly identified in the first iteration. We quantify and visualize the performance of vessel detection obtained over 4 iterations in Fig. 5(a). The axes on the right side correspond to the three metrics used for evaluation. It is clear that, as the registration and training proceed, the performance of vessel detection is improved progressively. Additionally, we see that the DNN performance becomes stable after three iterations and going to the fourth iteration offers limited improvement. Thus, we set the total number of iterations T to 3 in our experiments.

E. Robust Learning with Noisy Labels

We conduct detailed analysis for a better understanding of the proposed method for robust learning from noisy labels. Because we focus on the robust learning method, all experimental results reported in this section are performed on the noisy training data that is generated from the last iteration in the proposed framework.

To justify the effectiveness of the proposed robust learning method, we compare the performance of vessel detection with the following alternative training strategies: (1) the standard training approach that directly trains a DNN without any techniques particularly attuned to noisy labels, (2) the re-labeling method that dynamically updates the labels in the training dataset [53], and (3) a re-weighting method that reduces the weight for the noisy labels in the loss function. The re-labeling method seeks to obtain a clean dataset by dynamically updating the training labels using the probabilistic vessel maps predicted from the DNN. The training process is formulated as a joint framework that alternatively optimizes the DNN parameters and the training labels. For the re-weighting method, the idea is to adaptively assign small weights to the potential noisy pixels and to emphasize the clean pixels in the loss function. Specifically, we assign the posterior probability p_v as the weighting factor to each pixel in ${\mathcal{Y}}_v$ and set the weights to 1 for all pixels in ${\mathcal{Y}}_b$.

The quantitative results obtained from different training methods are listed in Table I. Directly training on the incorrect labels adversely impacts the performance of vessel detection, even though we apply early stopping to prevent the DNN from over-fitting the noisy labels. In addition, it is difficult to determine the stopping criterion because no validation dataset is available in this settings. The re-weighting and the proposed approaches, both of which utilize the posterior probabilities p_v to train DNNs, show significant improvement over the direct training and the re-labeling methods. This also demonstrate the effectiveness of the proposed mixture-model-based noisy label identification. Unlike the re-weighting method, which uses p_v to reduce the effects of incorrect labels, the proposed robust training approach updates the noisy labels and therefore explicitly forces DNN to learn on the correct prediction.

TABLE I:

Accuracy metrics for vessel detection results obtained with alternative training strategies. All DNNs are trained on the dataset obtained from the third iteration in the proposed framework. The best result is shown in bold.

Methods	AUC PR	Max DC	CAL (C, A, L)
Direct Training	0.802	0.745	0.628 (0.998, 0.777, 0.809)
Re-labeling [53]	0.837	0.769	0.586 (0.999, 0.729, 0.805)
Re-weighting	0.842	0.768	0.713 (0.999, 0.833, 0.856)
Proposed	0.842	0.772	0.730 (0.999, 0.849, 0.860)

Next, we assess the effects of different mixture models on fitting the loss distribution and estimating the posterior probabilities p_v. Specifically, we compare the proposed mixture model with a two-component Gaussian mixture model (GMM) and a two-component beta mixture model (BMM) [41]. A proper mixture model, which provide a good approximation to the loss distribution, should lead to an accurate estimation of the posterior probability p_v and an accurate update on the training labels Y_u. Thus, we compare the quality of the updated labels with respect to the manually labeled ground truth⁴. To do so, we fit the mixture models on the same loss distribution and update the labels using (6a) and (6b). Figure 6 plots the AUC PR obtained after each training epoch for different mixture models. We have several observations from this figure. First, the GMM is not a good approximation for the loss distribution and the accuracy of noisy label correction decreases as the training proceeds and is significantly worse than other two mixture models. Second, compared to the BMM, the proposed mixture model provides the more accurate results and the performance is largely stable in the first 70 training epochs. In Fig. 7, we show the sample results of updated labels, the corresponding noisy labels from the warped vessel maps, and the manually labeled ground truth. The “false positive” labels are removed from the warped vessel maps, highlighted by the yellow arrows in Fig. 7, yielding to updated labels that is similar to the ground truth labels. In Section S.IV of the Supplementary Material, we present intermediate results in a visual format, specifically, the predicted vessel maps and the posterior probabilities estimated using the proposed mixture model, that provide additional insight into the working of the EM-based noisy label correction in the proposed approach.

Fig. 6:

AUC PR obtained with alternative mixture models for modeling the loss distribution as a function of training epochs. The curves show the average AUC PR values over 5-fold cross-validation, and the shaded region represents the one standard deviation from the mean AUC PR values.

Fig. 7:

Sample results of noisy label correction in the proposed framework. Additional results, including the predicted vessel maps and the posterior probabilities, are provided in Section S.IV of the Supplementary Material.

F. Evaluation on the PRIME-FP20 Dataset

As mentioned in Section III-B, we perform 5-fold cross-validation to assess the results of vessel detection on the PRIME-FP20 dataset. Table II lists the quantitative results obtained from the proposed iterative framework and the existing methods. The proposed iterative framework performs remarkably well and significantly outperforms other methods with respect to all evaluation metrics, achieving an AUC PR of 0.845, the maximum Dice coefficient of 0.776, and an overall CAL of 0.730. Notably, the performance metrics for the proposed framework are quite close to the annotation-intensive approach, where a U-Net model is trained manually labeled clean dataset with the same 5-fold cross-validation (The row labeled U-Net* in Table II). We show sample results of the detected vessel maps obtained from different methods in Fig. 8 and provide more visual results in the Section S.V of the Supplementary Material. In Fig. 8, we see that the existing DNNs trained on NF fundus images perform poorly in the peripheral region. We attribute this poor performance to the fact that the peripheral region contains artifacts that are not visible in the NF dataset. Such artifacts normally have dark and curvilinear structures that can be misinterpreted as vessels in the image. For example, the yellow arrows in the enlarged view of region III highlight the “false positive” detection region that is not a vessel but an eyelash shadow appearing in the periphery. Compared to the DNNs trained on NF images, the proposed iterative framework accurately detects vessel maps from different regions in UWF FP. See the enlarged view of regions III and IV for the result patches selected from the periphery and the central retina, respectively.

TABLE II:

Quantitative metrics assessing vessel detection accuracy for different methods on the PRIME-FP20 dataset. Existing methods are trained on two different datasets: the registration-transfer-labeled dataset ${\left\{\left(X_c^i,Y_{a\to c}^{i,t}\right)\right\}}_{i=1}^M$ obtained from the first iteration of the proposed framework and the IOSTAR [26] dataset. The row U-Net* lists the results from a U-Net trained on manually labeled PRIME-FP20 dataset that does not contain noisy labels. The baseline CRF method does not converge when trained on transfer-labeled dataset. The best result is shown in bold.

Method	Year	Trained on noisy PRIME-FP20 ${\left\{\left(X_c^i,Y_{a\to c}^{i,t}\right)\right\}}_{i=1}^M$			Trained on IOSTAR
		AUC PR	Max DC	CAL (C, A, L)	AUC PR	Max DC	CAL (C, A, L)
U-Net* [13]	-	0.869	0.796	0.755 (0.999, 0.869, 0.870)	-	-	-
HED [49]	2015	0.752	0.690	0.584 (0.999, 0.765, 0.761)	0.723	0.683	0.451 (0.997, 0.640, 0.700)
U-Net [13]	2015	0.773	0.713	0.547 (0.998, 0.723, 0.751)	0.746	0.704	0.547 (0.998, 0.727, 0.752)
DRIU [11]	2016	0.752	0.700	0.558 (0.999, 0.747, 0.739)	0.728	0.691	0.495 (0.999, 0.698, 0.705)
CRF [50]	2017	-	-	-	0.563	0.550	0.341 (0.994, 0.577, 0.590)
NestUNet [14]	2018	0.749	0.698	0.596 (0.999, 0.776, 0.765)	0.754	0.716	0.567 (0.999, 0.747, 0.757)
M2U-Net [51]	2019	0.730	0.680	0.531 (0.998, 0.731, 0.719)	0.727	0.694	0.534 (0.998, 0.720, 0.742)
CE-Net [18]	2019	0.753	0.702	0.510 (0.998, 0.689, 0.731)	0.757	0.718	0.574 (0.999, 0.752, 0.762)
CS-Net [52]	2019	0.771	0.712	0.559 (0.998, 0.747, 0.742)	0.772	0.721	0.565 (0.998, 0.746, 0.755)
RU-Net [15]	2019	0.758	0.702	0.579 (0.999, 0.767, 0.753)	0.757	0.707	0.559 (0.999, 0.737, 0.758)
IterNet [16]	2020	0.798	0.736	0.607 (0.999, 0.765, 0.785)	0.746	0.717	0.553 (0.999, 0.732, 0.753)
Proposed	2020	0.841	0.771	0.730 (0.999, 0.849, 0.860)	-	-	-

Fig. 8:

Sample images and detected vessel maps for the proposed approach and alternatives from the PRIME-FP20 dataset. The contrast-enhanced enlarged views I-IV, marked by the cyan rectangles in the full image, are included. Additional visual results are provided in Section S.V of the Supplementary Material.

Comparing the results in Table II across different training datasets for each of the alternative approaches, we observe that most networks perform better when trained on the noisy PRIME-FP20 dataset obtained in the first iteration of the proposed framework compared with training on the IOSTAR dataset, even without using any learning approaches that are attuned to noisy labels. These results further reinforce the benefits of the proposed registration based transfer approach for generating training data for UWF FP modality.

G. Evaluation on Narrow-Field Fundus Photography

Fundus photography shares common characteristic between the ultra-widefield and the narrow-field modalities. In this section, we demonstrate that the DNN trained only on ultra-widefield images using the proposed framework is capable of detecting vessels in NF FP. To this end, we test the performance of the trained DNN on two public datasets, DRIVE [24] and STARE [4], and compare with the existing learning-based methods for vessel detection. Note that we train the DNN on ultra-widefield images using the proposed weakly-supervised learning approach and evaluate the performance on the NF images. We refer to this experiment as the cross-training evaluation [12], [54] where the training and the test data come from two independent sources. For existing learning-based methods, the models are trained on the DRIVE [24] and evaluated on the STARE [4], and vice versa. These two datasets provide two independent ground truth vessel maps manually labeled by two human annotators. We choose the vessel maps from the first annotator as the ground truth also report the human performance by evaluating the vessel maps made by the second annotator, which is commonly accepted approach in the literature.

Complete results are listed in Table S.III in the Supplementary Material. On the DRIVE dataset, the proposed framework achieves the best performance with the AUC PR of 0.886, the maximum DC of 0.803, and the overall CAL metric of 0.827. Note that the CAL metric is significantly better than those obtained from prior alternatives by large margins and is close to human performance (0.839). The second-best performing method, HED [49], achieves an overall CAL of 0.743. The performance on the STARE dataset, while slightly worse than the best performing method, is comparable to other methods. Specifically, the results obtained from the proposed framework has the AUC PR of 0.884, the maximum DC of 0.795, and the overall CAL metric of 0.756. The results on both datasets reinforce the robustness and the accuracy of the proposed iterative framework. We provide visual results of detected vessel maps in Section S.VII of the Supplementary Material.

IV. Discussion

For narrow field FP and FA images, the problem analogous to the one we address for UWF FP and FA images, has been previously addressed in an alternative approach called SegReg [35]. This approach utilizes paired image-pairs from FP and FA modalities and jointly addresses registration and the training of DNNs for vessel segmentation for both modalities. Trained networks can then be used for vessel detection in each modality individually. We evaluated SegReg on the PRIME-FP20 dataset. Implementation details and visual results are included in Section S.VI of the Supplementary. SegReg performs rather poorly on the PRIME-FP20 dataset, achieving an AUC PR of 0.535 and the maximum Dice coefficient of 0.560, which is not competitive with the other prior methods used for the comparisons in Table S.III. The reasons for the poor performance are twofold. First, SegReg assumes that the registered vessel maps in FA and FP modalities are identical. The assumption, however, does not hold for UWF FP and UWF FA pairs. Second, SegReg estimates the displacement field for registration, which only handles small deformation and therefore requires a good initialization. In addition to its poor detection performance, SegReg also consumes more computational resources for training than the proposed framework – approximately 11.5 hours as opposed to about 3.25 hours for the proposed network on an Nvidia V100 GPU.

Although, in this paper, we considered a very specific problem setting in retinal image analysis and presented our work in that context, the proposed methodology could also be adapted and applied in other situations. Alternative imaging modalities that vary in their ability to resolve different anatomical features (or physiology) are quite common and it is also a common situation that for some patients, data is acquired with a subset of the modalities. Techniques motivated by and/or developed using appropriate adaptations of the proposed method could be useful in these alternative settings, for both medical imaging and other image processing applications. Within the domain of image-based retinal diagnosis, an additional potential application of the proposed framework would be for vessel detection in optical coherence tomography angiography (OCT-A), which is also being increasingly used for examining retinal vasculature. Outside of medical imaging, an analogous problem also arises in understanding the evolution of road networks from a progression of satellite/aerial images captured over an extended duration of time. Later times typically contain more roads than earlier ones and additionally are likely to have precise labeled data compared with older imagery. Although beyond the scope of the current paper, techniques motivated by and/or developed using appropriate adaptations of the proposed method can be expected to useful in these alternative settings. Promisingly for such broader applications, our preliminary experiments indicate that the methodology for learning from noisy labels used in the proposed approach is not heavily reliant on the assumption that the supervising (FA) modality has greater detail than the supervised (FP) modality.

V. Conclusion

The iterative registration and deep-learning framework proposed in this paper provides an effective and annotation-efficient approach for detecting retinal blood vessels in UWF FP imagery without requiring manually labeled UWF FP vessel maps. Experimental evaluations demonstrate that the proposed approach significantly outperforms the existing methods on a new UWF FP dataset, PRIME-FP20, and achieves comparable performance with the state-of-the-arts on existing NF FP datasets. The PRIME-FP20 is made publicly available [42] to facilitate further work on retinal image analysis.