Automated segmentation of hyperreflective foci in spectral domain optical coherence tomography with diabetic retinopathy

Abstract.

We propose an automated segmentation method to detect, segment, and quantify hyperreflective foci (HFs) in three-dimensional (3-D) spectral domain optical coherence tomography (SD-OCT). The algorithm is divided into three stages: preprocessing, layer segmentation, and HF segmentation. In this paper, a supervised classifier (random forest) was used to produce the set of boundary probabilities in which an optimal graph search method was then applied to identify and produce the layer segmentation using the Sobel edge algorithm. An automated grow-cut algorithm was applied to segment the HFs. The proposed algorithm was tested on 20 3-D SD-OCT volumes from 20 patients diagnosed with proliferative diabetic retinopathy (PDR) and diabetic macular edema (DME). The average dice similarity coefficient and correlation coefficient ($r$) are 62.30%, 96.90% for PDR, and 63.80%, 97.50% for DME, respectively. The proposed algorithm can provide clinicians with accurate quantitative information, such as the size and volume of the HFs. This can assist in clinical diagnosis, treatment, disease monitoring, and progression.

1. Introduction

Diabetic retinopathy (DR) is the leading cause of blindness in patient’s age 16- to 64-years-old in developed countries.^1,2 Based on the presence of its clinical features, DR is classified into five types, namely mild nonproliferative diabetic retinopathy (NPDR), moderate NPDR, severe NPDR, PDR, and diabetic macular edema (DME).³ In eyes with PDR [as shown in Fig. 1(a)], fibro-vascular proliferation results from ischemia and release of vasoproliferative factors. Proliferation extends beyond the internal limiting membrane (ILM) and results in vitreous hemorrhage and vitreoretinal traction, which can lead to neovascular glaucoma and severe vision loss.^4,5 NPDR is swelling of the retina in the area that serves the central vision. The small blood vessels in the retina become damaged and can leak fluid into the retinal tissue, resulting in blurry vision and loss of portions of the field of vision. DME as shown in Fig. 1(d) is one of the most common causes of visual loss in both proliferative and nonproliferative diabetic retinopathies. It occurs when fluid and protein deposits collect on or under the macula of the eye, causing it to thicken and swell (edema). The swelling may distort a person’s central vision. Diabetic patients without NDR are at risk since the longer a person has diabetes, the higher their risk of developing some ocular problem.⁶ DR often goes unnoticed until vision loss occurs; early detection, timely treatment, and appropriate follow-up care of diabetic eye disease can protect against vision loss.

Fig. 1.

B-scan of (a) PDR and (d) DME, showing HFs (in yellow arrows) at different retinal layers. (b) and (e) are the 2-D false color images of (a) and (d), (c) and (f) are the 3-D intensity value distributions.

Several algorithms were designed for the segmentation of retinal anatomy in DR, but early research in the automated detection of pathology generally investigated fluorescein angiography images, rather than the present advanced frameworks. More recently, researchers are exploring different frameworks, such as OCT and fundus images, using different techniques in which some relied on global image processing, whereas others investigated the localization and segmentation of DR. In this literature, we highlight different methods that have been widely used in the localization and segmentation of DR. In one of the studies,⁷ the authors filtered the images using fixed scales and orientations of a Gabor filter bank of several filters. Then, the stages of NPDR were obtained by analyzing the numbers of maxima in the energy versus orientation plot. Finally, various DR diseases, such as PDR, NPDR, and DME, were differentiated using thresholding and edge detection techniques. They obtained a good classification rate in macula-related diseases using 24 eye images. In another study, Sopharak et al.⁸ proposed an exudate detection technique based on mathematical morphology on retinal images of nondilated pupils that are of low-quality images. Their work was based on this technique because it is very fast and requires lower computational time. The method in Ref. 9 presented an extension of the gradient method for the detection of red lesions on fundus images. The images were corrected for shading and noise, and then the expanding gradient was calculated for each of the pixels. A map of the image was obtained, and thresholding was performed to obtain the target object. From the 687 true red lesions included in 47 images of eye fundus, 584 red lesions were found using this method and 106 false red spots were additionally found. Another study¹⁰ developed an automated system to analyze the retinal images for important features of DR using image-processing techniques and an image classifier based on an artificial neural network, which classifies the images according to the disease conditions. Vascular network, optic disk, and lesions such as exudates were identified by this work.

Another work based on computer-aided detection was developed in Ref. 11. It detected the fovea, blood vessel network, optic disk, and bright and dark lesions associated with DR. The lesion detection was accomplished through the process of eliminating the normal retinal components: blood vessels, fovea, and optic disk. The image was partitioned into two regions: fovea and nonfovea since each has a different background. Statistical adaptive thresholding and filtering were then applied. In another study,¹² the proposed algorithm for hard exudates detection is composed of four main stages: image preprocessing and enhancement, feature extraction, classification, and postprocessing. The image was enhanced to normalize the intensity and contrast, whereas the algorithm extracts the dynamical training sets from each image. Then, the algorithm classified the pixels using a Fisher’s linear discriminant, after which a postprocessing technique was applied to differentiate between the hard exudates and the cotton wool spots and other artifacts. The work discussed so far focused mainly on the automatic identification, detection, and segmentation of DR in fundus images. Thus, these techniques are not applicable to the segmentation and localization of DR in SD-OCT images. Therefore, in this paper, we present an automated segmentation of hyperreflective foci (HFs) method in SD-OCT with DR.

HFs are commonly found in diabetic patients.^13–15 Coscas et al.^13,14 first reported the presence of HFs in spectral domain optical coherence tomography (SD-OCT), as small in size and scattered throughout all retina layers but mainly located in the outer retina layers around fluid accumulation in the intraretinal cystoid spaces. They suggested that the presence of HFs could affect the prognosis and treatment decisions, particularly in patients with age-related macular degeneration (AMD). Subsequently, HFs have also been reported in retinal vein occlusion and DME.^16,17 Bolz et al.¹⁵ reported and described the HFs as punctiform hyperreflective elements distributed throughout all retinal layers in eyes with different types of DME. Uji et al.¹⁸ also reported the presence of HFs in the outer and inner retinas in eyes with DME. Reduction in HFs has been reported in patients with diabetic diseases after treatment and has correlated positively to visual acuity outcomes.^19,20 Additionally, other reports suggest that the number and potentially the location of HFs may be a predictor of final treatment outcome in diabetic disease.^17–19 As such, we expect that the volume occupied by HFs in the retinal may be a useful diagnostic metric. Quantitative tools for assessing HFs may lead to better metrics for choosing treatment protocols. Automated methods for segmentation of the HFs are necessary to efficiently assess an entire 3-D OCT image stack and to estimate the total HFs volume.

Segmentation of HFs is tedious and complicated because HFs are difficult to identify, segment, and quantify. The major challenges in this segmentation process are highlighted below. Figure 2(a) shows the presence of HFs in all retinal layers, most especially between the ILM and inner segment–outer segment (IS–OS) layers. In this case, the layers are difficult to segment. Ruptured retinal layers, as shown in Fig. 2(b), are another characteristic of the HFs. This tends to make the HFs difficult to locate since the layers cannot be identified for segmentation. Layer segmentation, which is the most important step toward segmenting the HFs, is difficult to perform for both cases. To date, there is no layer segmentation algorithm for this problem. Figure 2(c) shows another challenge of HFs segmentation based on its similarity with the background: indifference between the HFs and the background pixels; the foreground (HFs) and the background exhibit extremely close similarity. Identification of the foreground and differentiating between the foreground and the background are a challenge in HFs segmentation. This problem is shown in Figs. 1(c) and 1(f), where the intensity distributions of the HFs and the background are difficult to separate, and in Figs. 1(b) and 1(e), where it is virtually impossible to identify which pixel belongs to the HFs or the background. The weak boundary of HFs is shown in Figs. 2(d) and 2(e). For successful segmentation, the target object must possess a strong boundary for identification, recognition, and classification. Since this feature is rarely available in HFs segmentation, this formulates into another major challenge in this segmentation process. Another setback is the presence of noise in the OCT images. The primary noise is the speckle noise, which has a significant effect on HFs segmentation since speckles are similar to HFs in reflectivity (brightness), size, and shape (both have no definite shape). The application of a denoising algorithm tends to eliminate smaller HFs.

Fig. 2.

Identification of HFs in retinal OCT images. (a) HFs in different layers, (b) HFs in ruptured layers, (c) similarity of HFs with background, and (d), (e) HFs weak boundary.

We note that currently there is no known software available to manually or automatically obtain the HFs volume from OCT machines. Using image-editing software, which is available under a contractual agreement with such vendors to manually mark out the HFs region by an expert grader, is possible. With massive HFs segmentation in OCT, analyzing and segmentation of HFs in large data acquisition is time consuming and tiring. Furthermore, this manual segmentation through an expert grader depends on an individual’s visual strength, which is less reliable. This technique lacks efficiency and accuracy; hence, a better technique is proposed in this research. To address this need, this paper presents a fully automated HFs segmentation technique for 3-D SD-OCT images to evaluate the potential role of HFs as an independent prognostic of the visual outcome in treated patients, as well as disease monitoring and progression. To our knowledge, this is the first study to segment the HFs in SD-OCT in patients with DR.

This paper’s main contributions include (1) an automatic seed generation proposed, making the traditional grow-cut algorithm an automated process, (2) amendment to the core grow-cut algorithm, making its performance better than the state-of-the-art, (3) improvement in the computational time of the grow-cut algorithm, and (4) experimental results with 20 eyes in 10 PDR and 10 DME patients demonstrate that our method can achieve high HFs segmentation accuracy and effectively measure the HFs volume.

2. Materials and Methods

2.1. Patients and Data Acquisition

OCT images of patients diagnosed with varied levels of retinopathy severity of DME and PDR were obtained using a Cirrus SD-OCT device (Carl Zeiss Meditec, Inc., Dublin, California). The scan covered a $0.51\mathrm{mm}\times 1.02\mathrm{mm}\times 0.13\mathrm{mm}$ area centered on the fovea. The volume dimension was $512\times 1024\times 128$ with a voxel size of $1\mu \mathrm{m}\times 1\mu \mathrm{m}\times 1\mu \mathrm{m}$. Out of the 20 subjects, 10 were diagnosed with PDR and the other 10 were diagnosed with DME. These patients, with various forms and levels of DR, ranged in age from 35- to 65-years-old with an average age of 42. The 20 3-D SD-OCT cubes from the 20 patients were carefully reviewed for the presence of HFs, noticeably as round or oval shapes in different retinal layers. Informed consent was obtained, and the study was approved by the Institutional Review Board of the First Affiliated Hospital of Nanjing Medical University.

2.2. Algorithm Overview

Figure 3 summarizes the algorithm. A preprocessing step, which includes denoising and intensity normalization, was first performed to remove artifacts and to establish intensity consistency. Layer segmentation using an optimal graph search method, coupled with a supervised classifier (random forest), followed. Finally, an automated grow-cut method was applied to segment the HFs. The full process is described in the sections below.

Fig. 3.

Overview of the proposed automated HFs segmentation algorithm. B-filter, bilateral filtering; ILM, inner limiting membrane; and IS–OS, inner segment–outer segment.

We have chosen the random forest classifier based on our need in this research as follows: (a) easy, simple, and fewer parameters are needed, (b) its ability to learn complex nonlinear relationship accurately, (c) it performs better than the state-of-the-art classifiers, (d) its ability to handle multilabel problems, (e) it is computationally efficient, and (f) it generates a probability for each of the labels, which we needed in soft classification of a one pixel wide boundary in this research since hard classification of such pixel suffers dramatically from both false positives and false negatives. These probability maps are then input to a boundary identification algorithm (3-D graph search), which finds contours that separate the retinal layers in each OCT image.

2.3. Preprocessing

2.3.1. Denoising

A bilateral filter is quite effective in removing modest levels of noise, and its performance has received renewed attention in the image-processing community.^21,22 In image denoising applications, edge preservation is a priority, and this important feature of a bilateral filter^23–25 is required in this study. The OCT images are corrupted with noise, most especially speckle and additive white Gaussian noise,²⁶ and the corrupted image is defined as

$$f(x)=f_0(x)+\sigma ·w_x,$$ (1)

where $x(x\in X)$ is finite rectangular domain of ${\mathbb{Z}}^2$, $f_0(x)$ is the clean image, $\sigma$ is the level of noise, and $w_x$ is an independent variable distributed as $\mathbb{N}(\mathrm{0,1})$. The denoised image is obtained as

$${\hat{f}}_x(x)=\frac{{\sum }_{j\in \mathrm{\Omega }}{g}_{{\sigma }_s}(j)g_{{\sigma }_r}[f(x-j)-f(x)]f(x-j)}{{\sum }_{j\in \mathrm{\Omega }}{g}_{{\sigma }_s}(j)g_{{\sigma }_r}[f(x-j)-f(x)]},$$ (2)

where $g_{{\sigma }_s}(x)=\exp (-\frac{{\Vert x\Vert }^2}{2{\sigma }_s^2})$, $g_{{\sigma }_r}(t)=\exp (-\frac{t^2}{2{\sigma }_r^2})$ are the spatial and range kernels, respectively, and $\mathrm{\Omega }=[-W,W]\times [-W,W]$, where $W=3{\sigma }_s$.²³ The domain $\mathrm{\Omega }$ is a restricted square neighborhood.^23,27 The result is shown in Fig. 4.

Fig. 4.

Denoised image. (a) Input image and (b) denoised output image using bilateral filter.

2.3.2. Intensity normalization

The denoised OCT images suffer from intensity variation. This problem made it difficult to extract features successfully from the OCT images. To overcome this problem, we applied the intensity normalization to balance these variations. This is always an important step in quantitative OCT image analysis, especially when extracting features based on intensity. To normalize these intensity variations, we rescaled the intensity values on each of the OCT B-scan by contrast rescaling in the range $[0,I_{\max }]$. Contrast rescaling was applied on each B-scan in which the intensity in the range $[0,I_{\max }]$ is rescaled to [0,1], where $I_{\max }$ denotes the maximum intensity value, which is obtained by median filtering (kernel size 12 pixels) each of the individual A-scans within the same B-scan. Intensity values larger than $I_{\max }$ are rescaled to 1. The major problem with this is that some hyperintense spots were found at the surface of the image. The effect of these spots on the performance of the algorithm cannot be overlooked. Therefore, we applied a more robust approach by setting the $I_{\max }$ value to a value that is 5% more than the maximum of the whole median filtered image. The 5% of $I_{\max }$ was chosen after several trial-and-error experiments to ascertain which $I_{\max }$ gives the best result with the least effect. At 5% of $I_{\max }$, the normalization of intensity inconsistency by constrast rescaling removes hyperintense reflections that are found at the surface of the retinal while retaining the overall intensity values in the B-scan.

2.4. Layer Segmentation

Layer segmentation consists of several subprocesses. The processes include boundary identification, feature extraction, boundary classification, and layers boundary tracing using the graph search method. These subprocesses are discussed in detail below.

2.4.1. Boundary identification

After the preprocessing steps, an anisotropic Gaussian smoothing was applied at different orientations and scales.²⁸ The Gaussian smoothing of scale ${\sigma }_{(x,y)}=(7,1)$ and ${\sigma }_{(x,y)}=(12,4)$ with an orientation 0 deg was then rotated at $-10$ and $+10\mathrm{deg}$ from the horizontal (on each B-scan) direction to obtain the top $(-10)$, center (0), and the bottom $(+10)$. Then, an effective feature $3\times 3$ neighborhood²⁹ was applied, with an additional filter bank in the A-scan direction and a second derivative was taken. The two boundaries with the highest positive gradient profile (the ILM and the IS–OS layer, in Fig. 5) were once again smoothed with Gaussian filter ${\sigma }_{(x,y)}=(22,4)$ and ${\sigma }_{(x,y)}=(97,210)$ for ILM and IS–OS for proper boundary points detection since HFs are mainly present between the ILM and IS–OS in this study.¹⁸

Fig. 5.

Layers detection in SD-OCT, (a) vertical layer detection and (b) horizontal detection. The arrows show the respective layers (orange arrow—ILM and the blue arrow—IS–OS).

2.4.2. Features extraction

Since the layers were identified, adequate features were needed to properly outline the layers. Features extraction is an important step for segmenting any structures in medical images.^30,31 To avoid feature redundancy and repetitions, we applied a wrapper feature selection method.³² Wrapper methods utilize the classifier as a black box to score the subsets of features based on their predictive power. Wrapper methods based on support vector machine (SVM) have been widely used in the machine learning field.^33,34 We used the SVM recursive feature elimination (SVM-RFE)^33–37 to select useful and important features. This method uses a backward feature elimination scheme to recursively remove insignificant features from subsets of features. In each recursive step, it ranks the features based on the amount of reduction in the objective function. It then eliminates the bottom ranked feature from the results. Out of the 70 features inputted, 30 features were ranked as significantly important. These features are spatial, local, and context awareness. The spatial features help to localize retinal pixels within a generic retina using a coordinate system that is defined by the geometry of the subject-specific retina. The context-aware features allow the classifier to learn local relationships between neighboring points without explicitly calculating any new features. The local features provide information about the current pixel. These extracted 30 features based on the boundary detected for each layer were retained for later usage. Features 1 to 9 are the nine intensity values of the $3\times 3$ neighborhood applied on each pixel. Features 10 to 21 were extracted per pixel from the signed value of the first and magnitude of the second derivatives on 128 B-scan images corresponding to two scales and three orientations. Features 22 to 24 are the relative distances of each pixel along the A-scan among the layer boundaries obtained in the boundary identification section. Features 25 to 30 are the average vertical gradients in the $12\times 12$ neighborhood under the current pixel. These features were used in training the random forest classifier. The random forest classifier uses ground truth labels, created by an expert, to learn. This classifier is then used to obtain the boundary probability at each pixel by computing these features for a new set of data inputted.³⁸

2.4.3. Boundary classification based on features

To classify these identified layers, we used a supervised classifier (random forest). Random forest³⁸ was chosen because of its probability generation for each label, which tends to provide a soft classification with each tree providing a separate classification for an input feature vector since the training is randomized between the trees. Random forests are a combination of tree predictors such that each tree depends on the values of a random vector sampled independently and with the same distribution for all trees in the forest.³⁸ Each decision tree is constructed using a random subset of the training data. Given an ensemble of classifiers $\{h_1(x),h_2(x),\dots h_{N_a}\}$ of $N_a$ decision tree, a random vector ${\theta }_k$ is generated, independent of the past random vectors but with the same distribution, and a tree is grown using the training set and ${\theta }_k$, resulting in a classifier $h(x,{\theta }_k)$, where $x$ is an input feature vector. The dimension of $x$ is 30 in this paper. For vector data, we provide an estimated label $\ddot{y}=h_i(x)\in P$, where $P$ is the set of all labels and $i=1,2,\dots N_a$. The posterior probability for any predicted label can be obtained as $p(y=b|x)=\frac{1}{N_a}\sum _{i=1}^{N_a}{I}_b({\ddot{y}}_i)$, where $I_b(·)$ is an indicator function for class $b$. Then a final label estimate for the classification can be obtained as $\ddot{y}={\mathrm{argmax}}_{p}p(y|x)$.

This supervised classifier is trained to find the two boundaries between layers. The full training data set comprises 3-D OCT volumes manually delineated from 18 subjects, which is not part of the test image. Each volume contains 128 B-scans. Thus, for a given B-scan, we use all of the segmented boundary points for training (i.e., 1024 points) for each of the boundaries. In as much as each B-scan contains 1024 A-scans. This approach focuses on identifying the one pixel wide boundary between layers. It is difficult in classifying the pixel found in between the boundaries due to weaker feature response. These boundaries are converted to layer segmentations by assigning each boundary pixel to the layer above it.³⁹ The random classifier outputs a set of boundary probabilities that are well defined visually, and then an optimal graph search algorithm is applied to generate the boundary curve from the boundary probabilities to compute the final boundaries.

2.4.4. Optimal graph search

After the classifier classified the boundaries, we applied the Graph search algorithm in which we adhered to the method used in Ref. 40, using the basic algorithm. Graph search optimization has been widely used in OCT.⁴⁰ The approach in Ref. 40 defines a cost based on the boundary probability estimated in all B-scans and finds the collection of boundary surfaces with their respective orders coupled with the minimum cost over the entire 3-D volume. Then, multiple constraints are used to bound the intra- and intersurface distances between adjacent pixels in the $x$- and $z$-directions and the min and max distances between the surfaces. The cost is calculated as 1 minus the boundary probability since it is a minimum nonnegative cost solution. The max-flow/min-cut algorithm was then used to obtain the surface.⁴¹ Since the layers were now successfully segmented into ILM and IS–OS layers, this restricted region becomes the region-of-interest (ROI).

2.5. Hyperreflective Foci Segmentation

After obtaining the ROI, the HFs are now restricted to a region. But HFs segmentation is tedious and complicated because of the following: (1) pixel intensity similarity between the HFs and the background, (2) difficulty in layer extraction and segmentation, (3) HFs have no fixed location, (4) speckles are similar to small HFs, and (5) HFs boundaries are weak. These problems possess a challenge, which calls for a suitable solution to address it. Hence, we propose a framework for HFs segmentation based on the grow-cut algorithm,⁴² which models the image using cellular automata. The grow-cut method uses continuous state cellular automata to interactively label (segment) the image using user manually supplied seeds.⁴² Many authors have used the grow-cut algorithm in medical image segmentation.^43–45 Although cellular automata are spatially and temporally discrete, abstract computational systems have proven to be useful tools both as general models of complexity and as specific representations of nonlinear dynamics in various scientific fields. Cellular automaton is a model of a system of cell objects with some specific characteristics.⁴⁶ These characteristics include (1) the cells live on a grid, (2) each cell has a state, and (3) each cell has a neighborhood. It is represented by the four tuple $(Z,S,N,f)$, where $Z$ is the finite or infinite lattice, $S$ is a finite set of cell states, $N$ is the finite neighborhood, and $f$ is the local transition function.^46–57 In the grow-cut method, the cells represent the image pixels and the feature vector is the gray-scale intensities (as in our case). The state of each cell is given by $S(\theta ,l,C)$, for each image pixel consists of a strength $\theta$ value in an interval [0,1], an image feature vector $C$, and a label $l$.

The traditional grow-cut algorithm requires manually selecting the seed, which is tiring and time consuming when the quantity of the images is large. In addition, each HF in the image exhibits a different range of pixel values, although close enough, yet it is difficult for the ordinary eye to differentiate between the HFs and the speckle noise or other reflective spots when selecting the initial seed. In this paper, we proposed an automated seed supply method. The automata are initialized by assigning corresponding labels seeds, using the regional and local maximum concept.

• Step 1:

  Obtain the regional maxima from the constrained boundary of ILM and IS–OS within 26-connected neighborhoods. The regional maxima are connected components of pixels with a fixed intensity value, $i$, whose external boundary pixels have a lesser pixel value than $i$. Identified regional maxima pixels are set to “1;” all others pixels are set to “0.”

• Step 2:

  From step 1, the location of the regional maxima is known. Extract the pixel intensity value of all the regional maxima. Compute the values in a $n\times 1$ matrix size, where $n$ denotes number of regional maxima.

• Step 3:

  Removing false (bright) pixel—speckles. Let $B$ represent the computed matrix in step 2; then arrange the pixel values in $B$ in ascending order. Get the successive minimum adjacent difference of $B$, represented as $\mathrm{\Delta }{B}_i$, where $i=\mathrm{1,2},3,\dots ,n-1$. Use $\overrightarrow{B}=B+\text{random}(n)\times \min (\mathrm{\Delta }{B}_i)$.

• Step 4:

  Apply dilation operation with an appropriate spherical structuring element whose radius is seven pixels. The dilation operator used is to expand unconnected edges while erosion smoothing the image by cleaning up small bright spots in the image. All pixels are considered; thus, new pixel regional maxima are obtained and compute in a matrix $M$, by replacing each point with these values.

• Step 5:

  At the point where $\overrightarrow{B}=M$, we obtain the local maxima location in $B$.

• Step 6:

  Compute the average of local maxima, which is the initial seed point (label).

The pixel with initial labels is assigned the cell strength of 1, whereas the unlabeled cells are set to 0. A pseudocode for the automated grow-cut algorithm is given below in Algorithm 1.

After the initialization of the labels, an iteration rule in Eqs. (3) and (4, see Algorithm 1) is applied

$$l_i^{t+1}=l_{i^{*}}^t.$$ (3)

$${\theta }_i^{t+1}=g(i,i^{*}){\theta }_{i^{*}}^t\text{where}{i}^{*}=q^{*},$$ (4)

where $g$ is the pixel similarity function at different nodes $i$ and $j$, which range from [0,1], as⁵⁸

$$g(i,j)=e^{-\Vert C_i-C_j\Vert }.$$ (5)

This function is equivalent to the weight function in which $c$ is the absolute intensity difference between neighboring nodes of $i$ and $j$. HFs intensities are used as the image features, with a 26-cell cubic neighborhood since our images are in 3-D. Additionally, HFs voxels are brighter than the background, and a prior awareness of HFs voxels can be used in modifying the transition function $g(i,j)$. The significant contribution of our method to the grow-cut algorithm includes (a) an automatic seed generation proposed, (b) amendment to the core grow-cut algorithm making its performance better than the state-of-the-art, and (c) improvement in the computational time of the grow-cut algorithm.

Algorithm 1.

Automated grow-cut.

//For each cell

For $\forall m\in \{HFs,Background\}$

Start $l_p$ and ${\theta }_p$

//label obtained from steps 1-6

for ${\forall }_p\in P$

if seed $p$ is of class $m$

${\theta }_{m,p}^0=1$ else ${\theta }_{m,p}^0=0$

end for

Do (until converge)

For ${\forall }_p\in P$

For ${\forall }_q\in N(p)$

//pixel neighbors attack current cell

where $q=g({\Vert c_q-c_p\Vert }_2)·{\theta }_{m,q}^t$

//$c_p$ and $c_q$ are cell features (intensity)

$q^{*}=argma{x}_{q\in N(p)}q$

//update state at p

${\theta }_{m,p}^{t+1}=g({\Vert c_q-c_p\Vert }_2)·{\theta }_{m,q^{*}}^t$

end for

// copy previous state

${\theta }_{m,p}^t={\theta }_{m.p}^{t+1}$

end for

end do

end for

3. Results and Analysis

3.1. Evaluation Parameters

3.1.1. Dice similarity coefficient

Dice is defined as the intersection between two similar labeled regions $L_{\text{region}}$ in ground truth segmentation $(G)$ and automatic segmentation $(A)$ over the average volume of these two regions. Its values range between 0 (no overlap) and 1 (perfect agreement). In this study, the Dice values are expressed as percentages and obtained using

$$\text{Dice}=\frac{2|A_{\text{region}}\cap G_{\text{region}}|}{|A_{\text{region}}|+|G_{\text{region}}|}\times 100.$$ (6)

3.1.2. Correlation coefficient–$r$

The correlation coefficient $r$ measures the strength and direction of a linear relationship between two variables. In this paper, the two variables are the ground truth $(G)$ and automatic segmentation $(A)$. The closer the absolute value of $r$ is to one, the closer the data are described by a linear equation. A correlation ($r=-1$) indicates a perfect negative correlation, whereas a correlation ($r=1$) indicates a perfect positive correlation. Datasets with values of $r$ close to zero show little to no straight-line relationship

$$\mathrm{Corr}(r_{G,A})=\frac{\mathrm{Cov}(G,A)}{{\sigma }_G{\sigma }_A},$$ (7)

where $G$ is the ground truth segmentation, $A$ is the automatic segmentation, $\mathrm{Cov}(G,A)$ is the covariance between ground truth and automatic, ${\sigma }_G$ is the standard deviation of ground truth, ${\sigma }_A$ is the standard deviation of automatic, and probability value is the $p$-value.

Probability value is the probability for a given statistical model that, when the null hypothesis ($H_0$) is true, the statistical model summary would be the same as or more extreme than the actual observed results. It simply tests a null hypothesis against an alternative hypothesis using a dataset.

3.2. Segmentation Results

3.2.1. Comparative analysis of automatic and ground truth segmentations

Data from 20 patients with PDR and DME were analyzed. The results from the proposed algorithm were compared with the results from the ground truth. Tables 1 and 2 show the mean volume of both the PDR and DME of the HFs, and Table 1 shows the results of applying the metrics of Sec. 3.1, to the segmentation results of each retinal cube using the expert labels as ground truth. The results are averages for each PDR and DME retinal cubes. The proposed algorithm and the expert grading (ground truth) were quantitatively and qualitatively compared. The performance of the algorithm for HFs in PDR and DME in Tables 1 and 2 could be observed for the total of 20 experiments, 18 of which were undersegmented, with only 2 being over-segmented. The under-segmentation effects were largely due to the inconsistent intensity of the HFs, where the expert grader considered some areas to be HFs and the proposed algorithm rejected such areas due to high significant property difference as shown in Fig. 6. Another causative agent is the weak boundary of the HFs, as shown in Fig. 7. In such cases, the expert grader marked and included the faded trailing boundary pixels of the HFs, whereas the proposed algorithm considered such pixels as background pixels since their properties were more similar to the background pixels than the HFs pixels. Appropriate layers segmentation could have reduced the under-segmentation of the proposed algorithm. As shown in Fig. 8(d), such layers segmentation would increase the segmentation accuracy of the algorithm. Therefore, our layers segmentation algorithm reduced the under-segmentation caused by the collapsed IS-OS layers by allowing segmentation of HFs IS–OS layers.

Table 1.

Mean volume of HFs in PDR using the ground truth (pixel).

Experiments	Ground trutha	Automatica	Differencea	Vol. seg. (%)
1	0.049	0.090	0.041	183.673
2	0.229	0.189	0.040	82.533
3	1.993	1.617	0.376	81.134
4	1.296	0.657	0.639	50.694
5	0.255	0.194	0.061	76.078
6	0.047	0.045	0.002	95.745
7	0.635	0.414	0.221	65.197
8	0.994	0.687	0.307	69.115
9	0.633	0.365	0.268	57.662
10	0.097	0.053	0.044	54.639

^aMultiply each value with $1.0\times {10}^{04}$.

Table 2.

Mean volume of HFs in DME using the ground truth (pixel).

Experiments	Ground trutha	Automatica	Differencea	Vol. seg. (%)
1	0.046	0.042	0.004	91.304
2	0.003	0.018	0.015	600.000
3	0.299	0.284	0.015	94.983
4	0.831	0.791	0.040	95.187
5	0.252	0.174	0.078	69.048
6	3.133	1.735	1.398	55.378
7	0.793	0.498	0.295	62.800
8	0.157	0.079	0.078	50.318
9	0.829	0.627	0.202	75.633
10	0.305	0.065	0.240	21.311

^aMultiply each value with $1.0\times {10}^{04}$.

Fig. 6.

Unsegmentated region result compared with ground truth. (a) original image, (b) the ground truth, (c) automatic result, (d) ground truth binarized image, and (e) automatic binarised image. Yellow arrows identify the ambigious region classified as HFs region by the expert human grader.

Fig. 7.

Segmentation results compared with ground truth. (a) original image, (b) the ground truth, (c) the automatic result, (d) ground truth binarized image, and (e) automatic binarized image. Yellow arrows identify the false negative segmentation results corrected by an expert human grader.

Fig. 8.

Effect of layers segmentation. (a) original image, (b) the ground truth, (c) the automatic result, (d) layers segmentation result, (e) ground truth binarized image, and (f) automatic binarized image. Yellow arrows identify the false negative segmentation results corrected by an expert human grader.

To date, there is no layers segmentation algorithm suitable for these cases. The over-segmentations were due to the false pixels exhibiting the same properties, such as with HFs. These pixels are scattered and are present in every layer of the retinal image, making it difficult for ordinary eyes and even the proposed algorithm to differentiate from HFs since both have the same properties. This fluctuation in the intensity, and the uneven intensity range of the background, resulted in false negative segmentation for the proposed algorithm as shown in Fig. 9. The assumption was that any bounded area $<50\text{pixels}$ should be discarded and removed since it is regarded as a speckle noise in this research work. Figure 10 shows one of the cases in which, amid the challenges of scattered varying HFs-like object, both the expert grader and the proposed algorithm rejected such pixel regions and classified such regions correctly.

Fig. 9.

Segmentation results compared with ground truth. (a) original image, (b) the ground truth, (c) the automatic result, (d) ground truth binarized image, and (e) automatic binarized image. Yellow arrows identify the false positive segmentation results corrected by an expert human grader.

Fig. 10.

Automatic segmentation results. (a) original image, (b) the ground truth, and (c) the automatic result. Yellow arrows indicate the pixels both rejected by the expert grader and the proposed automatic algorithm.

As shown in Table 3, the proposed algorithm agreed well with the ground truth. The dice similarity coefficient (DSC) for both the PDR and DME is 62.30% and 63.80%, respectively, with the correlation coefficient of 96.90% and 97.50%, respectively. The results show a good correlation between the proposed algorithm and the ground truth. The indication of this measurement revealed that the proposed algorithm performed better in DME than in PDR. This is due to the fact that, in most DME images, the HFs are bigger (size) than in PDR, so the proposed algorithm could easily lose smaller bounded area pixels due to many aforementioned challenges. The high $p$-values of 0.555 and 0.473 and a high estimate of correlation coefficient could only occur with a very small sample size.⁵⁹ The alternative hypothesis might in fact still be true, but, owing to a small sample size, the study did not have enough power to detect that the $H_0$ was likely to be false. Due to the cost of groundtruth and the time consumption, this research work could not afford to add more groundtruth data to verify that the $H_0$ is false.

Table 3.

Mean of HFs using the ground truth (%).

Metrics	PDR	DME
dice	0.623	0.638
correlation coefficient ($r$)	0.969	0.975
$p$-value	0.555	0.473

Figure 10 shows the automatic segmentation result of the SD-OCT images slice with boundaries clearly segmented. The first row shows the original image, the second column shows the groundtruth, and the third shows the automatic segmentation results. It shows a case in which the HFs boundaries were segmented accurately amid the aforementioned challenges. Both the HFs and the background pixels were classified accurately. Figures 11(a)–11(e) show a 3-D volumetric rendering HFs for proper visualization. This is a result of a cube (128 images) with high segmentation accuracy. Figure 11(a) is the frontal view of the segmented HFs in white, Fig. 11(b) is the imposed automated segmentation and the ground truth (white), and Figs. 11(c)–11(e) are the rear and orthogonal views in different angles, respectively.

Fig. 11.

3-D visualization of segmented HFs. (a–e) are quantification of volumetric spaces of HFs in different 3-D projections. (a) the frontal of view of the ground truth (white), (b) the automatic segmented HFs (red) imposed on the ground truth, (c) rear view of the imposed automatic on the ground truth, and (d), (e) are the orthogonal views in different angles.

Figure 12 shows the linear regression analysis results, and Fig. 13 shows the Bland–Altman plots for the automatic segmentation results versus the ground truth for PDR and DME. It is observed from Figs. 12 and 13 that the automatic segmentation results have strong correlation with the PDR ground truth ($R^2=0.9406$) and the DME ground truth ($R^2=0.9499$). In this research work, 20 retinal 3-D OCT images were segmented, with bias and limits of agreement calculated separately for each of the PDR and DME. An analysis based on these 20 subjects obtained the agreement plot as shown in Fig. 13. The 95% limit of agreement $(-3.93,0.11)$ of PDR and $(-6.34,1.68)$ of DME contain 95% (8/10) PDR and (9/10) DME of the difference scores. The mean difference (bias) of the measurements between the ground truth and the proposed automatic segmentation algorithm is $-1.91$ for PDR and $-2.33$ for DME. The standard deviations of the difference are 3.82 for PDR and 4.66 for DME, and the width of the 95% limits of agreements are 4.04 for PDR and 8.02 for DME. Hence, the automatic segmentation results from the proposed algorithm can replace the expert (ground truth). Figure 14 shows the dissimilarity between the ground truth and the proposed automatic algorithm. The volumes obtained in both the PDR and DME have a good correlation with the ground truth, with only an abrupt difference in volume between the ground truth and the proposed automatic algorithm in experiment 10 of the DME. This is associated with difficulty in segmenting the layers as shown in Fig. 15.

Fig. 12.

Correlation (a) PDR and (b) DME of the volume of HFs measured by automatic segmentation result and ground truth.

Fig. 13.

Agreement (a) PDR and (b) DME of the volume of HFs measured by automatic segmentation result and ground truth.

Fig. 14.

Discrepancy (a) PDR and (b) DME of the volume of HFs measured by automatic segmentation result and ground truth.

Fig. 15.

Comparison between the proposed method and other state-of–the-art medical image segmentation algorithms. (a1–a3) Experimental images used in the comparison, (b1–b3) the ground truths, (c1–c3) the results of the proposed method, (d1–d3) the results of the level set method, (e1–e3) the results of the active contour method, and (f1–f3) the results of the traditional grow-cut method.

4. Discussion and Conclusion

In this paper, we developed an automated HFs segmentation algorithm to segment HFs in 3-D retinal SD-OCT using cellular automata and the grow-cut algorithm, which involves three main steps: (1) the preprocessing, (2) the layer segmentation, and (3) the HFs segmentation. The proposed method was compared with other segmentation algorithms in a clinical application. Active contour,⁶⁰ level set,⁶¹ and traditional grow-cut algorithm possess high segmentation accuracy and performance; therefore, these algorithms are widely used in medical image segmentation. As shown in Fig. 15, the results of the proposed method and the three state-of-the-art methods are presented. It is obvious that all three methods are incomparable in terms of accuracy in segmenting the HFs to the proposed method. Additionally, active contour performs better than the level set method. The active contour method uses the initialization point (manually selected HFs pixel) to segment the HFs from an initial contour and automatically continues to trace the connected neighboring pixels in the image. The level set method does not depend solely on the initialization point (manually selected HFs pixel) and then splits automatically to detect more connected HFs component in the image. However, the traditional grow-cut uses cellular automaton as an image model in which the automata evolution models segmentation process by manually selected HFs pixel as the foreground. Each cell of the automata has some label, and during the evolution some cells captures their neighbors and replace their labels. All these methods are widely used in medical image segmentation, and their performances vary in different frameworks. In this case, the active contour method might be more suitable than the level set and the traditional grow-cut method. Although these methods (level set, active contour, and traditional grow-cut) perform poorly in this study, their performance in some other framework is highly commendable.

We experienced three major challenges, namely layer segmentation, HFs weak boundary, and the speckle noise. The effect of these challenges is quite visible in the overall results of the algorithm, yet we were able to minimize the effects. The layer segmentation part of the algorithm could segment the layers to their respective layers as required by the overall algorithm. This implies that any HFs out of the designated layer boundaries are not considered for segmentation, whereas an expert human grader would.

While most of the HFs in this research were found within the designated layer boundaries, the algorithm performed poorly in some special case (the layers were totally invisible) as shown in Fig. 16, due to total layers rupture of the retinal. These phenomena are often in HFs segmentation, yet its effect is substantial in the overall performance of the algorithm.

Fig. 16.

Examples of total layers rupture. (a) Some layers are still visible, (b) most layers in the left part of the image are invisible, and (c) all layers are unidentifiable except (ILM).

Since HFs suffer from seriously weak boundaries, this poses a serious challenge for the algorithm to properly classify the boundary pixels into their respective classes. The boundary pixels and the background pixel intensity values are so similar that there is a high tendency for misclassifying those pixels. This effect is shown in Fig. 7, in which an expert human grader corrected the segmentation result. The images used in this research contained speckle noise, which is one of the problems related to OCT images. These speckles are also found where HFs are situated, with a similar appearance to HFs. To reduce these speckles, we denoised the images with caution. The aim is to avoid eliminating some smaller HFs while removing speckles. The remaining speckles in the image are the major issues in the segmentation process since both HFs and the speckles are similar. Speckles could easily be mistaken for HFs by the algorithm. To reduce this tendency, size was used as a key criterion for the HFs segmentation. Thus, this minimized the error due to speckle noise.

The algorithm was tested on 20 datasets of 3-D OCT images of patients diagnosed with PDR and DME diseases. These are the various levels of the DR in the dataset: (1) PDR [(a) early PDR, (b) high risk PDR, and (c) advance PDR] and (2) DME [(a) nonclinically significant DME and (b) clinically significant DME].⁶² The PDR dataset consists of one early PDR, three high-risk PDR, and six advance PDR. The DME dataset consists of three nonclinically significant DME and seven clinically significant DME. Our experimental results achieve reasonable consistency with the ground truth. The performance evaluation of segmentation results demonstrates that our HFs segmentation method is efficient and accurate when compared with the ground truth. The mean of DSC, correlation coefficient ($r$), and probability value ($p$-value) for the proposed algorithm are 62.30%, 96.90%, and 0.55, respectively, for PDR and 63.80, 97.50, and 0.47, respectively, for DME. The linear regression analysis also shows close correlation between the automatic segmented. The proposed algorithm shows a close correlation with the ground truth ($R^2=0.9406$) and ($R^2=0.9499$) for both PDR and DME. This indicates that the segmented HFs are in agreement with that of the ground truth. To the best of our knowledge, this is the first work on HFs segmentation. Hence, our algorithm could only be compared with the ground truth done by experts. The traditional grow-cut algorithm has been improved significantly based on time and segmentation efficiency. One of the disadvantages of the proposed algorithm is the time consumption in operation since it works on each pixel of the 3-D images. The proposed algorithm was implemented in MATLAB R2013a and tested on a PC with Intel^® core (TM) i5-4200U CPU@ 1.60 GHz and 8 GB of RAM. The mean running time is $20(\pm 3)\min$ per cube (128 images). The computational time for each of the state-of-the-art segmentation algorithms and the proposed algorithm is presented in Table 4. The proposed algorithm has better computational time as compared with the other segmentation algorithms, followed by the level set algorithm, which is faster than the active contour algorithm in this case. The traditional grow-cut algorithm has the highest computational time as reported in Table 4.

Table 4.

Mean computational time of various segmentation methods as compared with the proposed method (min).

Segmentation methods	Computational time (minutes) per image cubes (128 images)
Proposed method	20 $(\pm 3)$
Traditional grow-cut	50 $(\pm 2)$
Level set	25 $(\pm 1)$
Active contour	27 $(\pm 1)$

To the best of our knowledge, several works have only described or rather reported the presence of HFs in DR and other related diseases. Since HFs are only present in retinal diseases, this formation leads to the conclusion that HFs are not regarded as a disease, but their presence could lead ophthalmologists to make accurate conclusions about the diseases, and make efficient treatment processes for patients with such diseases. Hence, segmentation of HFs could serve as one of the diagnosis tools in analyzing such diseases. The unsegmented ambiguous area by the proposed algorithm will have less clinical impact since the HFs are not the main disease but rather one of the several diagnosis tools used in analyzing such diseases. In addition, no evidence has shown that the volume of HFs is proportional to any of such diseases, but rather the segmented volume would support the ophthalmologists in their decision-making and diagnosis. We present an algorithm that can detect, segment, and quantify HFs in SD-OCT, which has never been available by either commercial software, open source, or individuals. This work presents an eye-opener for researchers by breaking the barrier that HFs could only be segmented manually rather than by a computer algorithm. Although this algorithm serves as a groundbreaking research area for other researchers or commercial softwares to explore and propose better methods, our future work will include improving the algorithm for better performance. In conclusion, the segmentation results of our algorithm can be used in clinical diagnosis, treatment planning, and disease monitoring and progression of diabetic patients.

Biographies

Idowu Paul Okuwobi received his BS degree in aeronautical engineering and his MS degree in mechanical and electrical engineering from Nanjing University of Aeronautics and Astronautics, China, in 2012 and 2015, respectively. He is a PhD student at the Nanjing University of Science and Technology, China. His current research interests include medical image segmentation and analysis, digital image processing, image segmentation, machine learning, and pattern recognition.

Wen Fan received her MD degree from Wuhan University in 2012. She is a lecturer at the Medical College of the Nanjing Medical University and an attending doctor in the ophthalmology department of the First Affiliated Hospital of Nanjing Medical University. She is mainly engaged in fundamental and clinical research of ophthalmology imaging.

Chenchen Yu received his bachelor’s degree from Nanjing University of Science and Technology in 2015. Currently, he is a master’s degree student in the same university. His research is about medical image processing and analysis.

Songtao Yuan is a chief physician of ophthalmology and an associate professor. Currently, he is a member of the Chinese Medical Association as well as the Association for Research in Vision and Ophthalmology of the United States (ARVO). His fields of expertise are diseases of retina and vitreous, retinal degeneration diseases, and research on fundamental and transformation of ophthalmic regeneration. He is an expert in vitreous and retinal surgery, especially in complex and joint surgeries.

Qinghuai Liu is a vitreoretinal specialist, professor of ophthalmology, and the head of the Department of Ophthalmology, the First Affiliated Hospital of Nanjing Medical University (NMU). He is a committee member of Chinese Ophthalmological Society (COS) and the chairman of Jiangsu Ophthalmological Society. He has mainly focused his research on the prevalence, gene susceptibility, pathological mechanism, biomarker identification, and translational medicine of retinal diseases, such as AMD and diabetic retinopathy (DR).

Yuhan Zhang earned his master’s degree from Nanjing University of Science and Technology in April 2017. Currently, he is a doctorate student at the Nanjing University of Science and Technology. His doctoral research is about SD-OCT image processing.

Bekalo Loza received her BS degree in computer science from Hawassa University, Ethiopia, in 2009 and her MS degree in geoinformatics from the University of Twentee, Netherlands, in 2014. She worked as a lecturer at the Hawassa University, Ethiopia from 2009 to 2012. Currently, she is a doctorate student at Nanjing University of Science and Technology. Her doctorate research is medical image analysis.

Qiang Chen received his BSc degree in computer science and his PhD in pattern recognition and intelligence systems from Nanjing University of Science and Technology, China, in 2002 and 2007, respectively. Currently, he is a professor with the School of Computer Science and Engineering at the Nanjing University of Science and Technology. His main research topics are image processing and analysis.

Disclosures

No conflicts of interest, financial or otherwise, are declared by the authors.