Neovascularization detection in diabetic retinopathy from fluorescein angiograms

Abstract.

Although a lot of work has been done on optical coherence tomography and color images in order to detect and quantify diseases such as diabetic retinopathy, exudates, or neovascularizations, none of them is able to evaluate the diffusion of the neovascularizations in retinas. Our work has been to develop a tool that is able to quantify a neovascularization and the fluorescein leakage during an angiography. The proposed method has been developed following a clinical trial protocol; images are taken by a Spectralis (Heidelberg Engineering). Detections are done using a supervised classification using specific features. Images and their detected neovascularizations are then spatially matched by an image registration. We compute the expansion speed of the liquid that we call diffusion index. This last one specifies the state of the disease, permits indication of the activity of neovascularizations, and allows a follow-up of patients. The method proposed in this paper has been built to be robust, even with laser impacts, to compute a diffusion index.

1. Introduction

The detection and follow-up of diabetic retinopathy, an increasingly important cause of blindness, is a public health issue. Indeed, loss of vision can be prevented by early detection of diabetic retinopathy and increased monitoring by regular examination. There are now many algorithms for the automatic detection of common anomalies of the retina (microaneurysms, hemorrhages, exudates, tasks, etc.). However, very few research studies have been done on the detection of a major pathology, which is neovascularization (NV), corresponding to the growth of new blood vessels due to a large lack of oxygen in the retinal capillaries.

Our work has not been to substitute manual detections of experts but to help them do it by suggesting what areas of the retina could or could not be considered as having NVs and by providing quantitative and qualitative proliferative diabetic retinopathy (PDR) such as the area of NV, the location to the optical nerve, and the activity of NV (diffusion index). The main goal has been to provide a diffusion index of the injected fluorescent liquid, which indicates the severity of the pathology, to follow the patient over the years.

Diabetic retinopathy is one of the first causes of visual impairment worldwide, due to the increasing incidence of diabetes. PDR is defined by the outgrowth of preretinal vessels leading to retinal complication, i.e., intravitreous hemorrhages and retinal detachments. Today, the laser photocoagulation is the standard of care treatment of PDR, leading to a decrease of growth factors secretion in photoagulated areas of the retina.

Vascular endothelial growth factor (VEGF) is responsible for the growth of healthy vessels as well as the NVs due to diabetes. Research is active on finding a specific type of anti-VEGF that could stop the growth of the NVs specifically. A clinical trial (ClinicalTrials.gov Identifier: NCT02151695), called “Safety and Efficacy of Aflibercept in Proliferative Diabetic Retinopathy” is in progress at the CHU of Poitiers, testing the effects of aflibercept, a specific anti-VEGF. This drug has been approved by the European Medicines Agency and the United States Food and Drug Administration for treatment of exudative age-related macular degeneration, another retinal disease characterized by choroidal new vessels. The aim of this pilot study is to evaluate the efficacy and the safety of aflibercept intravitreal injections compared to panretinal photocoagulation for PDR.

In ophthalmology, the majority of the works on the retinal diseases is about the detection of the exudates,^1–5 the healthy vessels segmentation,^6–8 the detection of the neural disk,^9,10 but none of them is about the detection of the PDR within angiograms.

Some works have also been done on the image registration for retinal images. Can¹¹ et al. have proposed the registration of a pair of retinal images using branching points and crossover points in the vasculature. Zheng et al.¹² have developed a registration algorithm using salient feature region description. Legg et al.¹³ have illustrated the efficiency of mutual information for registration of fundus color photographs and color scanning laser ophthalmoscope images.

Few steps are needed to compute diffusion index. It is a growth in time, which means that we have to detect and quantify the pathology on both injection times and compare their area. As it is nearly impossible to have exactly the same conditions during the acquisition (eye movements, focus of the camera, and angle), we need an image registration to have an estimation of deformations and to correctly spatially correlate NVs. For the segmentation, we used a supervised classification by random forests¹⁴ using intensity, textural, and contextual features, and a database of trained images from the clinical trials. These steps are shown in Fig. 1.

Fig. 1.

Example of a full detection by our method.

The paper is organized as follows. In Sec. 2, we present the microscope and the acquisition protocol. The image registration on both injection times is proposed in Sec. 3. We then propose an NV detection method in Sec. 4 and an automatic diffusion index computation in Sec. 5.

2. Materials

Our database is made of images taken by the Spectralis Heidelberg Engineering^© microscope with an ultrawidefield lens covering a 102 deg field of view. It delivers undistorted images of a great part of the retina, making the detection easier and to monitor abnormal peripheral changes.

Images are in gray levels with a resolution equal to $1500\times 1500\text{pixels}$. The areas that are bright are mainly (1) those leaking during fluorescein angiography due to NV, (2) normal retinal vessels, or (3) laser impacts. Some images we have taken from patients who have been treated by laser photocoagulation visible in Fig. 2. These impacts are annoying because they are also very bright for some parts. The blood still spread through some impacts and can be big enough to be wrongly assimilated to an NV.

Fig. 2.

Image of an angiogram taken with the Heidelberg Spectralis. Laser impacts are present all over the image, and some examples are highlighted in red.

To qualify the PDR by the index leakage, different times of acquisition during fluorescein angiography were used. The protocol presented below is the clinical trial’s protocol, which is composed of two image acquisitions: (1) fluorescein injection into the patient’s arm $t_0$, (2) acquisition of the early time injection $t_1$, and (3) acquisition of the late time injection $t_2$.

The image at time $t_1$ has to be acquired 10 s after the fluorescein injection at time $t_0$. Three minutes between acquisition times $t_1$ and $t_2$ allow the visualization of the fluorescein leakage defined as a progressive increase of the NV area with blurring edges of the NV. No leakage is observed on normal retinal vessels. In Fig. 3 we can see pictures of the same eye with acquisitions at times $t_1$ and $t_2$, where we can see the fluorescein spreading first into arteries and then bleeds in NVs.

Fig. 3.

Acquisitions of a retina: (a) the image and (b) a zoom of size $333\times 337$ on an NV at time $t_1$, (c) the image, and (d) a zoom of size $484\times 489$ on the same NV at time $t_2$.

As images are taken by 3 min, some spatial differences occur and we need to spatially correlate both images with an image registration, which is presented in the next part.

Fifteen diabetic patients were included in the analysis and ophthalmologists have identified 60 different NVs from fluorescein angiographies on wide field imaging.

3. Image Registration

The image registration does not aim to be perfect but to allow spatial comparison between NV taken in both images to compute quantitative data. The best registration model should be a local method, but for that reason just explained, a global method is widely enough for the comparison. Some of them are very popular and have been tested by many experts such as scale invariant feature transform (SIFT),¹⁵ maximally stable extremal regions,¹⁶ or speeded up robust features.¹⁷ We found that SIFT was robust and fast enough for the deformations we have on images.

3.1. Constraints

Images are taken with a manually movable camera with no special landmarks to help. Moreover, the eye of the patient slightly moves between each capture, even when focusing a specific direction, which means that the images for the two injection times can be geometrically different, with translations ($x$ and $y$), scaling ($z$), and some small rotations ($\theta$).

Furthermore, tissues on the retina can slightly be different over the time, depending on several biological factors, such as the heat, the light, or the blood flood. We then have global and local geometrical deformations. The brightness of the images mainly depends on the diffusion of the fluorescent liquid injected in the patient. Some tissues will appear more or less bright between both images and sometimes will simply be or not be present onto them. For example, healthy arteries will appear darker on the late time injection because the liquid first flood into them ($t_1$) and then spread into different tissues such as neovessels ($t_2$). That is why NVs appear brighter and are easier to detect on $t_2$. We finally have global colorimetry changes, which impact on the general contrast of the image, and very local changes.

3.2. Deformation Computation

First steps are the extraction and the description of keypoints on the image. These keypoints need to be invariant to image scaling and rotation and mostly invariant to change in illumination; they also need to be highly distinctive by their description to be matched further.

A scale space extrema detection using a difference-of-Gaussian algorithm is done to identify primary interest points. For each of these candidates, a local extrema detection is done to eliminate most sample points. An orientation is then assigned to each keypoint based on their local image gradient directions. These gradients are finally transformed into a representation, based on the region around and the selected scale.

To match the extracted keypoints, we use the brute force method because time computation is not our main constraint. Then, for each keypoint we take the two nearest points in terms of Euclidean distance, and we only keep those who the first nearest point is inferior to 0.8 times the second nearest neighbor (as proposed in Ref. 15), which mean that the first nearest neighbor has to be much closer than the second one.

The deformation matrix is finally computed by the RANSAC algorithm¹⁸ (random sample consensus), which is a nondeterministic iterative algorithm that tests random patterns from a data set containing correct and wrong values. This method estimates the model parameters that seem to best match the input data with a probability that increases with the number of iterations. RANSAC algorithm is robust even when some outliers are present in the dataset but can be poor when only a small portion of data is inliers, in this case the model is not able to find a good pattern to fit with. Therefore, we have to reduce our data by eliminating matchings that are obviously noise or that are not enough to represent an entire pattern.

3.3. Results and Discussion

The deformations we have between both the images are relatively small. Even with small movements from the eye or the camera, the lens used takes wide images enough to avoid big deformations, so we removed matching points that obviously are too far from each other. Hospital practitioners center the imaging system, and we have a displacement between both the acquisitions. Within the accordance of the ophthalmologists, we denote $r$ the maximum displacement, which is lower than the half of the image.

We can see in Fig. 4 that the registration process works well. It is still a global image registration that could be more precise with a specific local nonrigid algorithm, but the aim is to pair NVs and be spatially correct when comparing the leakage areas, so we do not need to have a perfect registration.

Fig. 4.

Registration of NVs: (a) detection of NVs at time $t_2$, (b) registration of these detected NVs on the image acquired at time $t_1$, (c) and (d) examples of detected NVs inside boxes, yellow boxes and NVs are computed at time $t_2$, green boxes and NVs are their registrations on the image acquired at time $t_1$.

According to head position or moving eye, the pictures are different between two fluorescein angiographies for the same patient. Due to this registration model, the ophthalmologists could compare easier several exams.

4. Neovascularizations Detection

4.1. Principle

The aim of a supervised classification is to set the rules that will let the algorithm classify objects into classes from features describing these objects. The algorithm is first trained with a portion of the available data to learn the classification rules (see Fig. 5).

Fig. 5.

Supervised classification process.

As supervised classification tends to give better results when it is possible to have a good trained database, we choose to use the random forests of decision trees¹⁴ (RF), which is a supervised classification algorithm that gives good results even with a small database.

The noise is very high in most images, notably the laser impacts that some patients can have (see Fig. 2). They have some close properties such as the brightness that is very high and sometimes the same shape and size. Some noise is also due to the acquisition itself: eyelashes can blur part of the images and the automated settings of the camera can lead to more or less blur, just as examples.

All the possible precomputations that could be done are instead represented as features in the classification. For example, we applied a healthy vessels segmentation that is included into the features (see Sec. 4.2), instead of a binary data preprocessing (healthy vessel or not) resulting in a probability, represented as a feature included in the classification.

4.2. Classification Algorithm

4.2.1. Features

Supervised classification can be used with as many features as we want but can be poor if too many bad features are used. To prune the features, it is possible to use multiple features and try to see which are the best by some tests. Once you have found the most important features, you can decide if you want to get rid of the other features or not, depending on your needs in terms of accuracy and time computation. Note that images being only in gray level.

We choose to take enough features to prune our selection because our database is not big enough to take on many features and still be a good predicate. NVs being very bright, we choose to have several features based on the intensity, we also add textural and one contextual feature as listed below.

Intensity: Because leakages are bright, we put a lot of weight onto the features based on the intensity: mean, maximum, and minimum intensity in the neighborhood. We also take into account the single value of the classified pixel. Mean, maximum, and minimum values are computed in a $5\times 5$ and a $9\times 9$ neighborhood, which leads to six features.

Texture: It can be a discriminator between laser impacts and NVs because laser impacts are more likely heterogeneous than the second one. For that, we calculate the variance on a $5\times 5$ and a $9\times 9$ neighborhood. We compute an isotropic gradient with a $3\times 3$ and $5\times 5$ Sobel operator. We add some Haralick’s texture features: angular second moment and contrast.¹⁹

Contextual features: They are very important because the intensity is often not enough and is very sensitive to the noise. We add a vessel segmentation in our process, which we translate into a feature. Healthy vessels could sometimes be classified as NVs if we only take into account the intensity features, because they are very similar. We base our method on the method proposed in Ref. 8. It is a morphological segmentation based on the width and the homogeneity of the vessels and weighted by the luminance. The vessels being very homogeneous, the more the value of the square and the value of the closest line (in terms of mean value) are close, the more is the chance that the pixel belongs to a vessel. See Fig. 6.

Fig. 6.

Detection of the healthy vessels: (a) noncomputed image and (b) image of the segmented vessels.

4.2.2. Classification by random decision forests

Random decision forests (RF) classification is a supervised algorithm based on an ensemble of binary decision trees. Each tree corresponds to a classifier having a weight, using only random features to classify the data.²⁰

Each tree gives a classification and “votes” for the class and the forest then selects the class having the most votes over all the trees used in the forest, given the result of the RF classification.

We limit the maximum tree depth to 500 and a node is splitted from four samples. Different appropriated thresholds are tested and discussed in Sec. 4.4.

With our dataset, the importance of the proposed features is listed in Fig. 7 (values have been rounded for visibility). The most important features are the minimum intensity and the mean intensity in the $9\times 9$ neighborhood. As expected, the intensity of the classified pixel is poor because several noise is also bright (e.g., laser impacts and healthy vessels). Haralick’s features cannot characterize NVs. Indeed, NVs are very homogeneous areas without texture.

Fig. 7.

List of the feature importance.

Figure 8 is an example of a classification with the RF algorithm. Compared to the ground truth, the true positives are in green, false positives in red, false negatives in blue, and true negatives in white.

Fig. 8.

Results of the RF classification on image (a). In (b) green regions represent the true positives, red regions the false positives, and blue the false negatives.

4.3. Postprocessing

Because it is a pixelwise classification, it is not sufficient enough by itself to have compact and fulfilled regions, so we added a few posttreatments.

As the leakage is almost isotropic in the vitreous, it is correct to compare the leakage with a cloud that is more or less dense but mostly filled (i.e., without holes). Classification sometimes gives regions with little holes that can easily be filled with a closing operation by mathematical morphology. Moreover some thin line detections can happen onto laser impact edges or healthy vessels for example, we can remove them with an opening operation.

Thus, after the classification, morphological operators are directly applied to remove thin false detections and they fill the holes of the detected NVs.

4.4. Results and Discussion

RF algorithm gives a probability for each pixel to belong to the class “NV” or to the class “other.” Because the results may vary due to the probabilities, we tried the algorithm with different thresholds $\lambda$ of probability. Results are obtained using a cross-validation process on our database. A query image is randomly excluded from the image database, and the other images constitute the training set. NVs are detected on this query image. In this way, the data of the image are not taken into account for the training and the statistical model is not wrongly influenced.

As results, we compare expert manual and automated segmentation to classify the resulting pixels into four classes: true positive (TP), false positive (FP), true negative (TN), and false negative (FN).

Given these classes, we can calculate the sensibility (S), the specificity (Sp), and the pixel prediction value (PPV) as following:

$$\mathrm{S}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}};\mathrm{Sp}=\frac{\mathrm{TN}}{\mathrm{FP}+\mathrm{TN}};\mathrm{PPV}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}.$$

However, NVs are mainly small compared to the size of the image and results in a big disparity between the number of positive and negative pixels. The specificity is then always very close to 1 because the pixel number belonging to the background is compared too much to the positives, so we neglect this feature from our results.

Results of the detection for the images at time $t_2$ are given in Fig. 9. We can see that the pixel prediction value is greatly influenced by probability threshold $\lambda$ compared to the sensibility, which is less influenced. A $\lambda$ under 0.8 gives a good detection of the NVs (high $S$, but very poor PPV). When the $\lambda$ is above 0.8, the $S$ decreases a bit but stays very high, whereas the PPV becomes more reliable around a $\lambda$ of 0.8 and becomes $>90\%$ for a $\lambda$ superior to 0.9.

Fig. 9.

Sensibility (blue dots) and pixel prediction value (red squares) results depending on probability threshold $\lambda$: (a) at time $t_2$ and (b) at time $t_1$.

Results of the detection for the images at time $t_1$ are given in Fig. 9. They are not as high as for the images at time $t_2$, as expected, because it is not easy to distinguish them from healthy vessels before the big part of the spread. As for the images at time $t_2$, PPV is very poor below a $\lambda$ of 0.8 and becomes very high above. The problem is that above this threshold, the $S$ decreases more than expected, until 60% for a $0.99\lambda$.

5. Diffusion Index

5.1. Methodology and Results

The diffusion index has to give an indication about the severity of the diabetic retinopathy, which means that it has to compare two liquid spread volumes. As we only work with two dimensional images, we can only guess that the spreading is isotropic and that an index computed only with the surface is enough to tell the strength of the leakage.

Figure 10 recalls the processing: we detect the NV surfaces at time $t_2$ and inside these surfaces, we detect NV surfaces at time $t_1$. The diffusion index is then computed by the ratio of the NV areas at $t_2$ and at $t_1$.

Fig. 10.

Methodology of diffusion index computation.

5.2. Results and Discussion

The detection of NVs into the images at times $t_1$ and $t_2$ is quite complex and really depends on many parameters. The parameters are linked to the fact that the eyes of the patient moves between each capture and the images between two injections can be geometrically different.

In our experience, for NV of diabetic retinopathy, the algorithm shows a sensibility and pixel predictive values are effective to describe lesions. We compute the mean square error (MSE) between the diffusion indices of the ground truth and the computed diffusion indices.

The detection of NVs onto the images at time $t_1$ is quite complex and really depends on many parameters. As shown in Fig. 11, we obtain a low MSE for probability $\lambda$ equal to 0.8.

Fig. 11.

MSEs for each computed diffusion index according to probability threshold $\lambda$ used for the classification.

A statistical mean of the diffusion indices are also computed. Table 1 shows the results with probability threshold $\lambda =0.8$. We observe that computed diffusion indices are close to the ground truth, indeed the error is only 0.01. We note that the surfaces of NVs globally double as observed in practice.

Table 1.

Diffusion index results with $\lambda =0.8$.

Ground truth		Automated	Difference
Mean	2.09	2.10	0.01
$\sigma$	0.33	0.54	0.52

Moreover, the retina can slightly be different in the time, depending biological factors, and the healthy arteries appear darker according to the time of the capture.

The diffusion index would help the ophthalmologists to follow-up the patients with PDR. The ophthalmologists could use this index in order to evaluate efficacy of treatment by panretinal photocoagulation or anti-VEGF intravitreal injection.

6. Conclusion

We propose to compute diffusion indices after detecting NVs in noisy angiogram images at times $t_1$ and $t_2$. First, we extract the NV areas at time $t_2$ and we use the area to detect the NV areas at time $t_2$. We also need to register images between the two acquisitions, and we choose to detect interest points using SIFT and we estimate the geometrical transformation for each NV.

To detect NVs, we learn features that characterize the NV. Therefore, we choose a random tree forest and this approach gives good detection results and the computed diffusion index is close to the ground truth.

A clinical study about this algorithm and manual method is now necessary to compare them, to permit the evaluation of clinical effectiveness, and to propose a software solution for the ophthalmologists.

Biographies

Benjamin Béouche-Hélias received his master’s degree in biology and computer sciences in 2014. He was an engineer at the Poitiers Hospital and has been working on automatic retina detection algorithms. Currently, he is a software development engineer at Tata Consultancy Services.

David Helbert received his PhD in signal and image processing in 2005 at the University of Poitiers, France. Currently, he is an associate professor at the University of Poitiers and is the head of the image processing team (ICONES) at Xlim Research Institute, UMR CNRS 7252. His main research areas include signal and image processing, medical imaging, multiresolution analysis, and mathematical modeling on hyperspectral images.

Cynthia de Malézieu is resident of ophthalmology at the Hospital of Poitiers, France. She received European Board of Ophthalmology in May 2017.

Nicolas Leveziel is a full professor at the University of Poitiers since 2013, hospital practitioner at the hospital of Poitiers. Currently, he is the head of the Ophthalmology Department. His research activities are focused on age-related macular degeneration and diabetic retinopathy.

Christine Fernandez-Maloigne received her PhD in image processing at the University of Technology of Compiegne in 1989. Currently, she is a full professor at Xlim research institute, University of Poitiers, Vice-Rector of Poitiers University, in charge of International Relations, and director of a CNRS research federation (MIRES). Her research activities are focused on color imaging, including fundamental researches about introduction of human visual system models in multiscale color image processes.

Disclosures

For this study, an informed consent was obtained from all human subjects. The authors state no conflict of interest, financial or otherwise and have nothing to disclose.