Nested U-Net for Segmentation of Red Lesions in Retinal Fundus Images and Sub-image Classification for Removal of False Positives

Abstract

Diabetic retinopathy is a pathological change of the retina that occurs for long-term diabetes. The patients become symptomatic in advanced stages of diabetic retinopathy resulting in severe non-proliferative diabetic retinopathy or proliferative diabetic retinopathy stages. There is a need of an automated screening tool for the early detection and treatment of patients with diabetic retinopathy. This paper focuses on the segmentation of red lesions using nested U-Net Zhou et al. (Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Springer, 2018) followed by removal of false positives based on the sub-image classification method. Different sizes of sub-images were studied for the reduction in false positives in the sub-image classification method. The network could capture semantic features and fine details due to dense convolutional blocks connected via skip connections in between down sampling and up sampling paths. False-negative candidates were very few and the sub-image classification network effectively reduced the falsely detected candidates. The proposed framework achieves a sensitivity of $88.79\%$, precision of $71.50\%$, and F1-Score of $79.21\%$ for the DIARETDB1 data set Kalviainen and Uusutalo (Medical Image Understanding and Analysis, Citeseer, 2007). It outperforms the state-of-the-art networks such as U-Net Ronneberger et al. (International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2015) and attention U-Net Oktay et al. (Attention u-net: Learning where to look for the pancreas, 2018).

Introduction

Diabetic retinopathy (DR) has become one of the most vulnerable eye diseases that cause blindness [5]. It is observed that one out of every three diabetic patients suffers from DR [6], and one out of every ten suffers from the severity of the disease. In the USA, blindness within the age of 20 to 70 is caused mostly due to DR [6]. DR shows no early symptoms of vision difficulty and blindness can occur even before the age of 50 [5]. Blindness due to DR can be avoided if detected in an early stage [6]. The significant risk factors for the onset and progression of DR are the duration of diabetes and degree of diabetic control. The pathological findings in DR are red lesions and white lesions. The red lesions include micro-aneurysms and haemorrhages and white lesions include soft and hard exudates [7]. Micro-aneurysms indicate the primary stage of DR which resemble tiny red circular spots as shown in Fig. 1. Haemorrhages are different sized red patches that determine the severity of the disease [8].

Fig. 1.

Blue box represents haemorrhage, yellow box represents vessel crossover, black box represents micro-aneurysm, and green box depicts excudates

Detection of micro-aneurysms and haemorrhages are usually done manually by the ophthalmologists, who measure the haemorrhage size and determine the severity of DR. Differences in image contrast, variation in image quality, variation in size, colour, and shapes of red lesions are the challenges for automatic detection or segmentation of red lesions. Ambiguity often arises between small red lesions and the vessel junctions. The characteristic of small red lesions that hinders the most in its accurate detection or segmentation is the variation of shape and size (25 to 100 $\mu m$). The variation in size and shapes considerably increases the possibility of false detection of pixel candidates and chances of missing out on small lesions. Manual segmentation is time-consuming, prone to error, and challenging for clinical practice. The number of ophthalmologists in India is limited compared to the increasing rate of DR cases. An automatic screening tool for DR would reduce the burden on the clinicians and help in accurate and effective diagnosis.

Micro-aneurysms are only of few pixels. The reported works mainly focussed on detection of red lesions rather than segmentation. The classical approaches primarily include mathematical morphological methods and template matching techniques to automate the classification, detection, and segmentation of red lesions. The classical approaches follow common steps like preprocessing, initial candidate detection, feature extraction, and classification. The preprocessing is essential for enhancing the red lesions from the background [9]. The shade correction method and normalization of green channel [10] were performed as preprocessing techniques. Watershed transform was also applied to distinguish micro-aneurysms from other dot-like structures [9]. Red lesions were detected using diameter closing, automatic threshold followed by classification. The classification of candidates was accomplished using kernel density estimation with variable bandwidth [10]. Template matching was used [7] for extraction of different shaped haemorrhages. The Radon Cliff operator  [11] was applied for the successful detection of circular structures irrespective of their size. Wavelet transform [12] and support vector machine (SVM) [13] were also used for detection of red lesions. Lazar et al. constructed a score map on micro-aneurysms and extracted the micro-aneurysms by thresholding operation [14]. Rocha et al. avoided the need for preprocessing or postprocessing [15] and built a visual word dictionary for detecting the points of interest within the lesions. Directional cross-section profiles were estimated on the local maxima pixels of the images, followed by peak detection with the calculation of size, shape, and height of the peaks to detect micro-aneurysms [16]. Circular Hough transform with feature extraction method was used for successful detection of microaneurysms  [17]. Kar et al. implemented a curvelet-based enhancement method to separate the red lesion from the background. The information of matched filter along with Laplacian and Gaussian filters were maximized jointly for detection of red lesion [18]. Zhou [19] used an unsupervised classification method for micro-aneurysms detection. Wang  [20] localized the candidates using a dark object filtering process, with singular spectrum analysis to process their cross-sections.

The classical methods for detection and segmentation of red lesions mostly rely on hand crafted features based on low-level information. The morphological methods also suffer from the dependency of several parameters which are determined empirically. Fully convolutional networks (FCN) were introduced for pixel-level segmentation [21] and were applied for medical image segmentation [6, 22, 23]. The U-Net [3] is very popular for medical image segmentation. The proposed network is based on U-Net++ [1] network with dense convolution layers connected through skip connections in between the up sampling and down sampling paths. The decoder provides the high-level feature maps, whereas the encoder provides the low-level feature maps. It shows that with the dense convolution layers, the feature maps captured from the background and foreground improved compared to that of U-Net [3] and attention U-Net [4]. The segmentation results are further refined by sub-image classification with ResNet-18 [24]. The falsely detected candidates for the proposed architecture are less than results obtained using U-Net [3] and attention U-Net [4]. The novelty of this work lies in exploring an improved version of U-Net++ [1] with a postprocessing method for further reduction in false positives based on a sub-image classification approach. An ablation study of U-Net++ [1] was performed to determine its depth. The proposed framework is robust and effective for fast segmentation of red lesions with an improved performance.

Materials and Methods

Materials

Publicly available MESSIDOR data set [25] was used for initial training of the network from scratch. MESSIDOR data set [25] consists of 1200 colour fundus images, captured using a non-mydriatic camera with a 45 degree field of view. Images were of $1440\times 960$, $2240\times 1488$ or $2304\times 1536$ pixels. Out of 1200 images of the MESSIDOR data set [25], 742 images belonging to grade-1, grade-2, and grade-3 categories were selected. Five hundred and seventy-two were for training, and one hundred and seventy were for validation. The ground truths for MESSIDOR data set [25] are not publicly available. The annotations of red lesions in MESSIDOR [25] images were done under the supervision of ophthalmologists.

DIARETDB1 data set [2] consists of 89 colour fundus images, where 84 images contain at least mild or non-proliferative signs and five are normal. These images were captured using a 50 degree field of view. The course ground truths of DIARETDB1 [2] images are publicly available which can be used only for detection. Four medical experts had done the annotations. We redefined the boundaries of red lesions under the supervision of ophthalmologists.

Methods

The proposed method has two stages: initial segmentation using Nested U-Net [1] of red lesions and subsequent reduction in false-positives candidates based on sub-image classification. The block diagram of the proposed method is shown in Fig. 2, and the methodology is explained in subsequent sections.

Fig. 2.

Block diagram of the proposed methodology

Preprocessing

Retinal fundus images often suffer from uneven illumination and poor contrast. It is observed that illumination is maximum near the optic disc and gradually decreases towards the periphery. The red lesions have the maximum contrast in the green channel. To cope with GPU memory, the large-sized retinal fundus images were divided into small pieces of $256\times 256$ pixels so that semantic information is not lost and high resolution is maintained. Contrast limited adaptive histogram equalization (CLAHE) [5, 26, 27] was performed to improve the visibility of red lesions as shown in Fig. 3.

Fig. 3.

(a) Normal fundus image and (b) preprocessed fundus image

Network Architecture for Red Lesions Segmentation

The state-of-the-art networks for the segmentation of medical images are composed of encoders and decoders. The decoder provides the semantic, high level, and course feature maps, and the encoder provides the low level and fine feature maps. The proposed network has an encoder and decoder paths connected through several dense convolutional blocks. An encoder–decoder model based on U-Net++ [1] is proposed for segmentation of red lesions. The proposed network is a fully convolutional neural network and its purpose is to use the high-level and shallow-level information of an object for better segmentation. The network consists of a main up-sampling and down-sampling paths as shown in Fig. 4. The encoder and decoder are connected via skip connections with a series of convolution blocks. There are twenty-one convolutional blocks, and each block has two convolutional layers. The encoder path has six convolutional blocks with kernel size of $3\times 3$ and padding of 1. Maxpooling operation with a filter size of $2\times 2$ and stride 2 is performed after each encoder block except the last block. The size of feature map increases from 64 to 2048 along the down sampling path. Along the encoder path, image size reduces from $256\times 256$ to $8\times 8$. The decoder path has five convolutional blocks following a sequence of up-sampling and concatenation after each block. Feature map size increases from 2048 to 64 along the decoder path and image size increases from $8\times 8$ to $256\times 256$. There is padding with dense convolutional blocks along the horizontal line between encoder and decoder. Four dense convolutional blocks are present between the topmost encoder and decoder blocks, and these blocks decrease by unity as we go downwards along with the network. Along the horizontal paths of the network, every block is connected to the other through skip connections. The convolutional blocks parallel to the main decoder path, undergo up-sampling. Thus, each convolutional block of the dense layer is preceded by a concatenation operation that adds the output from the previous dense convolutional blocks with the up sampled output from the lower dense convolutional block. Essentially, the dense blocks bring together the feature maps of the encoder and decoder such that both the feature maps are semantically similar. This network aims to segment the red lesions with minimum false-negative candidates.

Fig. 4.

Architecture of the network for segmentation of red lesions

Here, $W^{\mathit{ij}}$ denotes each convolutional block and $w^{\mathit{ij}}$ is its output, where i denotes convolutional blocks downwards along the encoder and j depicts the dense convolutional blocks with the skip connections. All blocks with $j=0$ receive the output from its preceding block of the encoder following max pooling. All block with $j=1$ receives two inputs; one from the block preceding it along the skip connection path and the other from the block below along the encoder path following up-sampling. For $j>1$ , the convolutional blocks receive a j number of inputs from the preceding dense convolutional blocks along the skip path and one from the dense layer below after up-sampling. Thus each block receives the feature maps from all its previous blocks. The input image data propagates along all possible paths of the network and produces the feature maps at the end of the network.

$$\begin{array}{c}\hfill w^{\mathit{ij}}=\mathrm{\Gamma }(\{w^{i-1,j}\})\text{for}j=0,i>0\end{array}$$ 1

$$\begin{array}{c}\hfill w^{\mathit{ij}}=\mathrm{\Gamma }([w^{i,k}{]}_{k=0}^{j-1},\mu (w^{i+1,j-1})])\text{for}i,j>0\end{array}$$ 2

where $\mathrm{\Gamma }(.)$ denotes convolution, $\{.\}$ denotes max pooling, [.] denotes concatenation and $\mu (.)$ denotes upsampling.

Sub-image Classification for False-Positive Reduction

ResNet is one of the most common convolutional neural networks used for classification. The ResNet-18 [24] is used to classify the sub-images with and without red lesions. The optimal sub-image size is $80\times 80$. The ResNet-18 [24] has 18 convolutional layers each of kernel size is $3\times 3$. The feature map size increases from 64 to 512 down the network. The last layer is a fully connected one. Leaving the first convolutional layer, skip connections are there at an interval of two convolutional layers; this structure is called the residual block [24]. The residual block is shown in Fig. 5, where the curved arrow denotes the skip connection. The equations that define the residual block are as follows:

$$\begin{array}{c}\hfill H(x)=F(x)+x\end{array}$$ 3

$$\begin{array}{c}\hfill F(x)=H(x)-x\end{array}$$ 4

where x and H(x) are the input and output of the residual block. Each residual block has two weighted layers, as shown in Fig. 5. The skip connections help to retain the information before the previous layer. The residual blocks decrease the network’s distortions caused by the increased number of convolutional layers. It is known that shallow layers learn low-level features like object edges, object structure, or texture features, whereas deep layers learn high-level features like spatial and semantic features. Thus if the networks are made deeper, low-level features are often lost. Residual blocks in ResNet-18 [24] restrain the loss of low-level features and it works better than other networks.

Fig. 5.

Residual block

Training

The proposed network was trained with images of MESSIDOR data set [25]. Each image of the MESSIDOR data [25] set was divided into eight pieces and re-sized to $256\times 256$ pixels. The network was trained up to 75 epochs. ADAM [28] optimizer was used with a learning rate of 0.00001 for 1 to 50 epochs. The learning rate was decreased to 0.000001 for 51 to 75 epochs. Patches of size $80\times 80$ were extracted to update the weights of the pre-trained network. The total number of epochs for training was 60. Stochastic gradient descent (SGD) [29] was used with a learning rate of 0.01 and momentum 0.6. The learning rate decreased by 0.0001 after every 15 epochs.

Results and Discussions

Metric

The performance of the proposed method and competing methods were evaluated using conventional metrics like sensitivity, specificity, precision, accuracy, and F1-score as mentioned below.

$$\begin{array}{c}\hfill Sensitivity=\frac{\mathit{TP}}{TP+FN}\end{array}$$ 5

$$\begin{array}{c}\hfill Specificity=\frac{\mathit{TN}}{TN+FP}\end{array}$$ 6

$$\begin{array}{c}\hfill Accuracy=\frac{TP+TN}{TP+TN+FP+FN}\end{array}$$ 7

$$\begin{array}{c}\hfill Precision=\frac{\mathit{TP}}{TP+FP}\end{array}$$ 8

$$\begin{array}{c}\hfill F1-Score=\frac{2TP}{2TP+FP+FN}\end{array}$$ 9

Connected component level validation was done inspired from the following method [22, 30]. In Fig. 6, the red-coloured contours represent the ground truth lesions and the green-coloured contours represent the predicted lesions. Instead of counting the truly classified pixels, only the true-positive pixels of connected components, overlapping with the ground truth by a minimal ratio of 0.2, are considered. Suppose the predicted lesion does not overlap the ground truth by a minimal ratio of 0.2. In that case, the non-overlapping portion of the predicted lesion is considered false-positive pixels. The non-overlapping part of the ground truth is regarded as the false-negative pixels. The rest of the pixels are true negatives. The boundaries of red lesions are not clearly defined and therefore, connected component level validation is used for calculating the performance metrics.

Fig. 6.

Overlapping of ground truth and predicted lesions

Segmentation Results

The performance of the proposed and competing methods were evaluated on DIARETDB1 data set [2]. The 89 images of DIARETDB1 data set [2] were divided into two parts. Twenty-eight images were used for fine-tuning the pre-trained model, and results were reported on the rest of the 61 images. The state-of-the-art networks (U-Net [3], attention U-Net [4]) were also trained on 28 images, and the results were reported on 61 images of DIARETDB1 data set [2]. It is observed that there is an improvement in sensitivity and precision (Table 1) with the proposed architecture. Improvement in sensitivity is $8.87\%$ over U-Net [3], and precision has increased by $8.83\%$ in comparison with the results obtained using attention U-Net [4].

Table 1.

Performance evaluation for red lesion segmentation

Segmentation	Sensitivity	Specificity	Accuracy	Precision	F1-Score
Architecture
Used
U-Net [3]	79.92	99.65	99.48	67.88	73.41
Attention U-Net [4]	85.10	99.60	99.49	62.67	72.18
Multi-task Architecture [22]	85.18	99.89	99.83	78.96	81.95
Proposed method	88.79	99.64	99.53	71.50	79.21

In Fig. 7, the first column represents the ground truths for the red lesions, the second column represents the initial segmentation results of the proposed method, and the third column shows the results after false-positive reduction. The proposed method could segment red lesions of poor contrast and small size. It is robust enough to segment tiny as well as large sized red lesions. In Fig. 8, the first column, second column, and third column represent the segmentation of red lesions using U-Net [3], attention U-Net [4], and proposed method, respectively. False-positive false-negative candidates were sufficiently low in the proposed method compared to the other competing techniques.

Fig. 7.

(a), (d) and (g) are the ground truths, (b), (e) and (h) are the segmentation of red lesions using nested U-Net, (c), (f) and (i) are the final results after false-positive removal. Blue represents the true-positive candidates, green represents the false-positive candidates, and red represents the false-negative candidates

Fig. 8.

(a), (d) and (g) are the results of U-Net, (b), (e) and (h) are the results of attention U-Net, and (c), (f) and (i) are the results of proposed method. Blue represents the true-positive candidates, green represents the false-positive candidates, and red represents false-negative candidates

Conclusion

An architecture based on convolutional neural network is developed for the segmentation of red lesions in fundus images. The architecture uses minimum and informative training images and has the potential to assist ophthalmologists in diagnosing diabetic retinopathy with high accuracy. The proposed architecture could segment red lesions with very few false-negative candidates. False-positive candidates were also reduced to a large extent. More work is still needed to minimize false-negative and false-positive candidates further. There is a scope of improvement in the preprocessing technique for better visualization of small pathologies in fundus images that are difficult to segment.

This article purely focused on the segmentation of red lesions (micro-aneurysms and haemorrhages) for non-proliferative diabetic retinopathy diagnosis. The limitation of this article is that segmentation and detection of neovascularization were not addressed. The method used in this article can successfully segment micro-aneurysms with a minimum number of false positives. The method would be helpful in the early screening of DR and can prevent vision loss as well as the cost of treatment. The proposed method can be useful for computer-aided diagnosis of non-proliferative diabetic retinopathy and assist ophthalmologists while taking clinical decisions.

Author Contributions

The authors contributed to an improved version of U-Net++ with a postprocessing method for further reduction of false positives by sub-image classification approach. Ablation study of nested U-Net was performed to determine its depth. A robust, fast, improved, and effective segmentation of red lesion could be achieved.

Funding

The study was not funded by anyone.

Data Availability

The study was done using two publicly available datasets MESSIDOR and DIARETDB1.For this type of study, formal consent is not required.

Code Availability

Codes for U-Net++ and ResNet-18are available publicly. For this work, code of UNet ++ was modified.

Declarations

Conflicts of Interest

The authors declare that they have no conflict of interest.