Multimodal Retinal Vessel Segmentation from Spectral-Domain Optical Coherence Tomography and Fundus Photography

Abstract

Segmenting retinal vessels in optic nerve head (ONH) centered spectral-domain optical coherence tomography (SD-OCT) volumes is particularly challenging due to the projected neural canal opening (NCO) and relatively low visibility in the ONH center. Color fundus photographs provide a relatively high vessel contrast in the region inside the NCO, but have not been previously used to aid the SD-OCT vessel segmentation process. Thus, in this paper, we present two approaches for the segmentation of retinal vessels in SD-OCT volumes that each take advantage of complimentary information from fundus photographs. In the first approach (referred to as the registered-fundus vessel segmentation approach), vessels are first segmented on the fundus photograph directly (using a k-NN pixel classifier) and this vessel segmentation result is mapped to the SD-OCT volume through the registration of the fundus photograph to the SD-OCT volume. In the second approach (referred to as the multimodal vessel segmentation approach), after fundus-to-SD-OCT registration, vessels are simultaneously segmented with a k-NN classifier using features from both modalities. Three-dimensional structural information from the intraretinal layers and neural canal opening obtained through graph-theoretic segmentation approaches of the SD-OCT volume are used in combination with Gaussian filter banks and Gabor wavelets to generate the features. The approach is trained on 15 and tested on 19 randomly chosen independent image pairs of SD-OCT volumes and fundus images from 34 subjects with glaucoma. Based on a receiver operating characteristic (ROC) curve analysis, the present registered-fundus and multimodal vessel segmentation approaches [area under the curve (AUC) of 0.85 and 0.89, respectively] both perform significantly better than the two previous OCT-based approaches (AUC of 0.78 and 0.83, p < 0.05). The multimodal approach overall performs significantly better than the other three approaches (p < 0.05).

I. Introduction

With the relatively recent development and commercial availability of three-dimensional spectral-domain optical coherence tomography (SD-OCT) [1], [2] and its increasing use for the diagnosis and management of blinding diseases, such as glaucoma, age-related macular degeneration, and diabetic retinopathy, a pressing need exists for the automated segmentation of important ophthalmic structures. Most prior work in this area (including methodology developed primarily for time-domain OCT) has focused on the segmentation of the intraretinal layers [3]–[11], but the segmentation of other structures, such as the optic disc/neural canal opening [12]–[15], fluid-filled regions [16], [17], and retinal vessels [18], [19], is also important. In this work, we focus on the problem of segmenting the retinal vessels in SD-OCT volumes centered at the optic nerve head (ONH), which is not only important for the assessment of diseases that affect the vessels directly, but is perhaps even more important as a component of other image analysis tasks such as atlas generation, image registration [20], [21], the removal of the vessel influence from thickness measurements, and, through the use of the vessels as orienting landmarks, the segmentation of other ophthalmic structures.

However, vessel segmentation in SD-OCT volumes of the ONH is not a trivial problem. As the retinal vessels themselves are generally not readily visible in the current generation of commercially available SD-OCT volumes, their silhouettes (within the deeper layers) are used for the segmentation of their projected locations, requiring a sufficient contrast between the vessel silhouettes and the surrounding tissue. Outside the of the ONH, such a contrast exists primarily due to the presence of bright outer layers near the retinal pigment epithelium (RPE). However, within the ONH proper, the tissue composition changes (e.g., the retinal pigment epithelium doesn't exist) and the contrast between the vessel silhouettes and surrounding tissues is decreased. The lack of contrast is especially noticeable near the projected location of the neural canal opening (NCO) as the bright tissue of the retinal pigment epithelium ends (Fig. 1.a and b), making the local appearance of the projected neural canal opening similar to that of retinal vessels.

Fig. 1.

Illustration of NCO location and NCO false positives from Niemeijer's OCT-based vessel segmentation approach [18]. (a-b) Central slice of raw SD-OCT volume and vessel-oriented OCT projection image (Section II-B1) obtained from the SD-OCT volume with the highlight of NCO location (yellow arrows) and retinal vessels (purple arrows). A green outline is used to emphasize that the SD-OCT image is three dimensional. (c) Vessel segmentation from Niemeijer's OCT-based vessel segmentation [18]. Note that the typical false positives near the NCO (red arrows).

These challenges are apparent in the few reported approaches for the segmentation of retinal vessels within ONH-centered SD-OCT volumes [18], [19]. For example, the approach of Niemeijer et al. [18] focused entirely on the region outside the NCO (Fig. 1.a and b) due to difficulties associated with the relatively low visibility in the ONH center and the similar high contrast projective appearance of the NCO contour to the vessels, causing false positives indicated by the red arrow in Fig. 1.c.

Because of the obvious difficulty in segmenting the vessels near the neural canal opening, we have previously presented a unimodal approach for segmenting retinal vessels in ONH-centered SD-OCT volumes by incorporating the pre-segmented NCO positional information in the vessel classification process to help suppress the false positive tendency near the NCO boundary [22]. The performance of the algorithm demonstrated an obvious improvement over the approach of Niemeijer et al. [18]. However, the relatively low visibility in the ONH center remains a problem for vessel identification in some scans.

While only a few approaches have been reported for the segmentation of vessels in SD-OCT volumes [18], [19], [22], many vessel segmentation approaches performed on fundus photographs have been reported [23], [24]. The blood vessels on fundus photographs present a relatively high contrast in the region near and inside the NCO over SD-OCT images (Fig. 2). As fundus photographs are often taken alongside SD-OCT volumes, it can be expected that use of this information can help to improve existing SD-OCT vessel segmentation approaches. Thus, in this work, we present two approaches that utilize information from fundus photographs to segment the retinal vessels in ONH-centered SD-OCT volumes.

Fig. 2.

Illustration of the high visibility of the NCO in SD-OCT image and that of vessels on fundus image. Left column: vessel-oriented OCT projection image (bottom) (Section II-B1) with the zoomed ONH center (upper). Note that the high NCO contrast as indicated by the yellow arrows. Right column: fundus photograph (bottom) with the zoomed ONH center (upper). Note that the high vessel contrast as indicated by the purple arrows.

In our first approach (the simpler of the two approaches), referred to as the registered-fundus vessel segmentation approach, vessels are first segmented on fundus photographs directly using a k-nearest-neighbor (k-NN) pixel classifier [23] and this vessel segmentation result is mapped to the SD-OCT volume through registration of the fundus photograph to the SD-OCT volume. However, such an approach would not fully take advantage of the multimodal information available as SD-OCT information would only be used in the registration process. Thus, in the second approach, referred to as the multimodal vessel segmentation approach, after automated fundus-to-SD-OCT registration, vessels are simultaneously segmented with a k-NN classifier using features from both modalities. Such an approach is expected to better be able to take advantage of the complimentary information from both modalities, including the use of segmented three-dimensional structures from the SD-OCT volume, such as the intraretinal layers [10] and the neural canal opening [15]. Note that a preliminary version of this second approach (but using manual registration) was reported in [25].

II. METHODS

An overview of our registered-fundus and multimodal vessel segmentation approaches is presented in Fig. 3. In both approaches, we first segment the retinal vessels on original fundus photographs using a pixel-classification-based approach (Section II-A). Once we have the vessel profiles on the original fundus photographs, we register the fundus photographs to the corresponding SD-OCT volumes using a feature-point-based registration approach [26]. More specifically, in our first approach, we register the segmented fundus vessel image to the OCT modality and use the registered fundus vessels as the OCT vessels (Section II-B2). In our second approach, we register the original fundus images to the OCT modality using the same transformation and apply a multimodal vessel segmentation (Section II-C) by incorporating the complementary information from SD-OCT volumes (e.g. the NCO positional information) and fundus photographs (e.g. better vessel contrast in the region inside the NCO).

Fig. 3.

Overview of registered-fundus and multimodal vessel segmentation, where the dashed-blue-line and light-gray-background blocks indicate the registered-fundus vessel segmentation and the dashed-red-line and light-gray-background blocks indicate the multimodal vessel segmentation.

A. Vessel segmentation in original fundus photographs

A supervised pixel-classification-based segmentation method is used to segment the vessels in the original fundus photographs [23]. More specifically, features are first extracted using Gaussian derivative filters. Each image L consists of Gaussian filter derivatives up to and including order 2 (i.e. L, L_x, L_y, L_xx, L_xy, L_yy) at scales σ equal to 1, 2, 4, 8, and 16 pixels. A previously trained k-NN classifier (k=31) is then applied and each pixel is assigned a soft label. This results in a vesselness image, with the image intensity of each pixel representing the likelihood of belonging to a vessel. An example of an original fundus photograph and its vessel segmentation is shown in Fig. 4. Note that because the purpose of this step is to obtain a vessel segmentation from fundus photographs alone (without the use of SD-OCT image information), we are able to use our existing vessel segmentation approach that has been previously trained from an independent set of fundus photographs [23]. However, the k-NN classifiers used in the other steps involving SD-OCT data are trained using the training data described further in Section III. While the training process requires expert delineations, after training, such classifiers enable a fully automated segmentation on previously unseen data.

Fig. 4.

Example fundus photograph and its vessel segmentation. (a) Original color fundus photograph. (b) Segmented vesselness map of the original fundus photograph.

B. Registered-fundus vessel segmentation

1) Vessel-oriented OCT projection image creation

The fundus-to-OCT registration needs a reference image in the SD-OCT modality. Thus we create an OCT projection image at the level of the NCO in the SD-OCT volume. More specifically, four intraretinal 3-D surfaces are simultaneously identified in the 3-D raw SD-OCT volumes using an optimal graph-theoretic multilayer segmentation algorithm [10], [13]. Based on a segmented surface, we then flatten the raw SD-OCT volumes (Fig. 5.b) [10], [13]. The four segmented surfaces are also flattened by applying the same transformation.

Fig. 5.

Illustration of vessel-oriented OCT projection image creation. (a) Central slice of the raw SD-OCT volume. (b) Central slice of the flattened SD-OCT volume with four segmented retinal surfaces indicated. (c) 3-D visualization of segmented surfaces. (d) Vessel-oriented OCT projection image obtained from the layer indicated by the yellow arrows in (b).

After the flattening of the original 3-D SD-OCT volume, the projection image is obtained by computing the mean intensity values from a small number of slices surrounding the NCO plane, i.e. the thin layer between the segmented surface 2 (orange) and 4 (yellow) in the flattened OCT image, as indicated by the yellow arrows in Fig. 5.b and is referred to as the “vessel-oriented” OCT projection image as it is also used for the OCT vessel feature extraction. Fig. 5.d is an example of the resulted vessel-oriented OCT projection image.

2) Registered-fundus vessel segmentation

To register the segmented original fundus vessels to the vessel-oriented OCT projection image, we also segment the vessels on the vessel-oriented OCT projection image using the OCT vessel segmentation approach of Niemeijer et al. [18]. Note that although this preliminary approach presents some false positives around the NCO boundary in some images as described previously, it is good enough to provide feature points in the region outside the NCO for the registration.

The Random Sample Consensus (RANSAC) algorithm [27] is then applied to remove the outliers of the initial matching points found by the Scale Invariant Feature Transformation (SIFT) detector [28]. An exhaustive search is utilized to find the best set of matching points. The original fundus vesselness map is registered to the vessel-oriented OCT projection image using an affine transformation and cropped to the same size as the vessel-oriented OCT projection image. Fig. 6 is an illustration of the registered-fundus vessel segmentation.

Fig. 6.

Illustration of registered-fundus vessel segmentation. (a) Original color fundus image. (b) Segmented vesselness map of the original fundus image. (c) Vessel-oriented OCT projection image. (d) Preliminary OCT vessel segmentation [18]. (e) Cropped fundus-to-OCT registered vessel image.

C. Multimodal retinal vessel segmentation

1) Overview

The multimodal approach utilizes features from both modalities of SD-OCT volumes and color fundus photographs for pixel classification to segment the vessels. In the SD-OCT modality, to suppress the false positives near the NCO boundary, the algorithm first pre-segments the NCO using our previous graph-based NCO segmentation approach presented in [14], [15]. Oriented Gabor wavelets rotated around the center of the NCO along with the corresponding oriented NCO-based templates are applied to extract OCT features (Section II-C3) [22]. To extract the fundus features, the fundus images are first registered to the vessel-oriented OCT projection images (using vessel information) in the same manner as described in Section II-B2 and cropped to the same size as the vessel-oriented OCT projection images. Gabor wavelets with oriented NCO-based templates and an additional Gaussian filter family with different Gaussian derivatives and scales are applied on the registered fundus images to extract the fundus features. A k-NN classifier is utilized to detect the retinal vessels by combining the complementary feature spaces of the two modalities as described in Section II-C5 [25]. The major steps of the multimodal vessel segmentation can be found in Fig. 3 and further details are provided in the paragraphs below.

2) NCO pre-segmentation in SD-OCT volumes

To incorporate the NCO information into the vessel classification process, the NCO is pre-segmented from a NCO-aimed OCT projection image using our previous graph-based approach [14], [15]. Fig. 7 summarizes the NCO segmentation process. A graph-theoretic approach [10], [13], [29] is used to segment the internal limiting membrane and three surfaces near the retinal pigment epithelial layer and these surfaces are radially interpolated inside the expected neural canal opening region (Fig. 7.b). A projection image (referred to as the NCO-aimed OCT projection image) is obtained by averaging the intensity values between these layers (Fig. 7.c). In the NCO-aimed OCT projection images, the outer boundary corresponds to the projected location of the NCO and the inner boundary corresponds to the cup at the level of the NCO reference plane. A two-dimensional graph search is then applied to simultaneously segment the optimal NCO and cup boundaries [15]. Having the NCO segmentation enables the projected position of this 3-D structure to be utilized in the computation of features for the classification.

Fig. 7.

Illustration of NCO segmentation. (a) Central slice of the raw SD-OCT volume. (b) Central slice of the flattened SD-OCT volume with three radially interpolated surfaces. (c) NCO-aimed OCT projection image obtained from the layer indicated by the yellow arrows in b. (d) NCO (outer boundary) and optic cup (inner boundary) segmentation overlapping with the NCO-aimed projection image.

3) Feature extraction in SD-OCT volumes

The pixel features in the SD-OCT image modality are generated from the vessel-oriented OCT projection images (Fig. 8.b) using a similar Gabor wavelet filter family with NCO-based templates as used in our unimodal OCT vessel segmentation approach [22] as described below and summarized in Table I. Note that the blood vessels generally radially distribute around the center of the NCO. Gabor wavelets present the desirable characteristics of the spatial frequency, spatial locality, and orientation selectivity [30], [31]. Thus, oriented Gabor wavelets around the center of NCO are well suitable for the feature generation for the blood vessel detection. The idea behind using NCO-based templates is to help suppress the false positive tendency from the NCO, while simultaneously not affecting the true positives of the blood vessels (based on the assumption that the blood vessels are not parallel with the NCO).

Fig. 8.

Illustration of multimodal retinal vessel segmentation. (a) Registered fundus to OCT image. (b) Vessel-oriented OCT projection image. (c) A schematic illustration of the Gabor wavelet responses and the NCO-based templates oriented at 20 degrees. Blue arrow = NCO contour. Purple arrows = template pair centered on the NCO boundary. (d) Vessel segmentation from the multimodal approach.

TABLE I.

Pixel features from the SD-OCT and fundus images

		OCT features	Fundus features
Oriented Gabor wavelets with NCO-templates	Frequency	ν ∈ {1, ..., 6}	ν ∈ {1, ..., 5}
	Gaussian scale	σ ∈ {1, 2}	σ = 2
	Orientation	μ ∈ {0, ..., 8}	μ ∈ {0, ..., 8}
Gaussian derivative filters	Gaussian scale	None	σ ∈ {1, 2, 4, 8, 16}
	Order	None	N ∈ {1, 2}
Original intensity		Included	Included

In particular, a Gabor wavelet ψ_μ,ν(z) [30] can be defined as:

$${\psi }_{\mu ,\nu }\left(z\right)=\frac{{\Vert k_{\mu ,\nu }\Vert }^2}{{\sigma }^2}{e}^{-\frac{{\Vert k_{\mu ,\nu }\Vert }^2{\Vert z\Vert }^2}{2{\sigma }^2}}\left[e^{i{k}_{\mu ,\nu }z}-e^{-\frac{{\sigma }^2}{2}}\right],$$ (1)

where z = (x, y), ||·|| is the norm operator, μ and ν define the orientation and spatial frequency (scale) of the Gabor kernel, and σ is related to the standard derivation of the Gaussian window in the kernel and determines the ratio of the Gaussian window width to the wavelength. The wave vector k_μ,ν is defined as

$$k_{\mu ,\nu }=k_{\nu }{e}^{i{\varphi }_{\mu }},$$ (2)

where $k_{\nu }=\frac{k_{max}}{f^{\nu }}$ in which k_max is the maximum frequency and f^ν is the spatial frequency between kernels in the frequency domain. ${\varphi }_{\mu }=\frac{\pi \mu }{n}$ in which n is the total number of orientations.

Based on the vessel profiles, the Gabor wavelet parameters are chosen as follows: the Gaussian scale σ ∈ {1, ...,2}, spatial frequency (scale) ν ∈ {1, ..., 6}, and orientation μ ∈ {0, 8}. Together, a Gabor wavelet family with two Gaussian scales, six spatial frequencies (scales), and nine orientations is generated.

To suppress the NCO false positive tendency, a pair of NCO templates as shown in Fig. 8.c is created. The center of the template pair is that of the pre-defined NCO and the center of each of them lies on the NCO boundary. The NCO-based templates rotate in the same orientations with the Gabor wavelets [22]. The shapes of the templates are defined as:

$$\frac{{\{x_1-x_c-r_1\}}^2}{W^2}+\frac{{\{y_1-y_c\}}^2}{H^2}=1,$$ (3)

and

$$\frac{{\{x_2-x_c+r_2\}}^2}{W^2}+\frac{{\{y_2-y_c\}}^2}{H^2}=1,$$ (4)

where (x_c, y_c) is the NCO center, r₁ and r₂ are the distances of the center of each template to the center of NCO, and W and H are the maximum width and height of the templates which are defined based on prior knowledge of NCO profiles.

The NCO-based templates rotate in the same orientations as the Gabor wavelets. Wherever they rotate, the average pixel value of the Gabor wavelet response in that orientation is assigned to those regions. The intensity value from the vessel-oriented OCT projection image is also included in the SD-OCT modality feature space (Table I).

4) Feature extraction on fundus photographs

All of the pixel features in the color fundus image modality are generated from the green channel of the registered fundus image as shown in Fig. 8.a because the green channel provides a relatively high vessel contrast. Oriented Gabor wavelets with NCO-based templates and additional Gaussian filter banks in the x- and y-direction are applied for the feature extraction.

More specifically, oriented Gabor wavelets with NCO-templates at Gaussian scale of σ = 2, spatial frequencies (scales) ν ∈ {1, ..., 5}, and orientation μ ∈ {0, ..., 8} are created. Together, they come to a Gabor wavelet family with one Gaussian scale, five spatial frequencies (scales), and nine orientations [22]. This number of Gabor-wavelet-based features (with NCO templates) is smaller than that used in the OCT images because of our decreased expectation of NCO-based false positives. In addition, a Gaussian filter family with the Gaussian derivatives with the orders of N ∈ {1, 2} and five Gaussian scales at σ ∈ {1, 2, 4, 8, 16} is also generated (based on the prior success of using such features with fundus photographs [23]). The Gaussian derivatives are applied only in the x- and y-direction. The intensity value from the green channel of the transformed color fundus photograph is also included in the fundus modality feature space (Table I).

5) k-NN classification

After separately extracting the pixel features from the vessel-oriented OCT projection and registered fundus images, the feature spaces of the two modalities are combined. In other words, for each sample in the multimodal image space, the multimodal feature vectors include the features from both modalities and each feature vector is normalized to zero mean and unit variance. In a training set (as described in Section III), each pixel is labeled “vessel” or “non-vessel” for use within a k-NN classifier.

In the testing phase, each pixel pair of the vessel-oriented OCT projection images and corresponding registered fundus images in the test image set is treated as a single query sample and is classified using a k-NN classifier with k = 31. This value of k was chosen based on preliminary experiments from independent data sets and for consistency with our prior work [18], [23]. In such experiments, small changes in k did not have a major influence on the segmentation results. To save the running time, in this work, the searching of the nearest neighbor training samples for each query sample is implemented using an approximate-nearest-neighbor approach with a tolerance of a small amount of error [32]. Based on the obtained k nearest neighbor-training samples, each query sample (pixel pair) in the test image is assigned to a soft label, i.e. a posterior probability defined as p_vessel = n/k, where n is the number of the training samples labeled as “vessel” among the k nearest neighbor training samples. Fig. 8 is an example illustration of the multimodal retinal vessel segmentation.

III. EXPERIMENTAL METHODS

The data is based on the dataset used in our previous NCO segmentation work [15], which includes 34 independent deidentified ONH-centered SD-OCT scans (Cirrus™ HD-OCT) and corresponding stereo color fundus photographs (Nidek 3Dx) image pairs from 34 subjects with glaucoma. Fifteen image pairs are randomly chosen as the training set and the remaining 19 pairs are used as the test set.

Each SD-OCT volume consists of 200 × 200 × 1024 voxels, corresponding to 6 × 6 × 2 mm³. Each stereo color fundus photograph has 768 × 1019 pixels. The NCO-aimed OCT projection images, the vessel-oriented OCT projection images, and the cropped color fundus registered images have a size of 200 × 200 pixels. The pixel depth of the color fundus photograph is 3 × 8-bits in red, green, and blue channels.

Expert-defined manual tracings with each pixel labeled as “vessel” or “non-vessel” are obtained based on the vessel-oriented OCT projection and cropped fundus registered images (using a viewer that enables the registered images to be examined on top of one another simultaneously), resulting in 200 × 200 × 15 = 600000 training samples for the training set of 15 images and 200 × 200 = 40000 test samples for each test image. In cases where small misalignments between the two images exist, experts are instructed to favor the vessel information from the vessel-oriented OCT projection images in the region outside the NCO. In the region inside the NCO, the manual vessel tracings are to reflect the union of the vessel information from both images. In this work, one expert traced the vessels, with two additional experts providing corrections, which effectively resulted in one tracing that was the result of the consensus of three experts through a discussion.

Four vessel segmentation approaches, i.e. the previously reported Niemeijer et al. [18] OCT-only approach, the previously reported Hu et al. [22] OCT-only approach, the registered-fundus approach, and the multimodal approach, are evaluated using the data in the test set. In particular, the performance of each vessel segmentation approach is evaluated based on the AUC of the ROC curves. The AUC between the ROC curves between each pair of approaches are compared using the non-parametric approach proposed by DeLong et al. [33], which is based on the theory of generalized U-statistics. The pROC package [34] for R is used to perform this test and p-values less than 0.05 are considered significant.

IV. RESULTS

Table II and III demonstrate the quantitative results by comparing our two current approaches, i.e. the registered-fundus and multimodal approaches, with two closest previous approaches, i.e. Niemeijer's previous OCT [18] and our previous unimodal OCT [22] approaches. Note that in Table II, three of the 19 test eyes are excluded to the statistical analysis because of the relative large automated registration errors due to the low vessel visibility and/or “non-optimal” OCT projection images resulting from motion artifacts. However, Table III provides the AUC analysis results for all the 19 test eyes. In the three cases that the automated registration presents relative large errors, a manual registration is applied. The ROC curves are presented in Fig. 9.

TABLE II.

AUC comparison among the Niemeijer et al. [18] OCT-only approach, the Hu et al. [22] OCT-only approach, the registered-fundus approach, and the multimodal approach for the 16 automated-registration-based test eyes

Region	Inside NCO
Modality	Unimodal (OCT)		Multimodal (OCT + fundus)
Algorithm	Niemeijer et al. [18]	Hu et al. [22]	Registered-fundus	Multimodal
AUC	0.65	0.68	0.87	0.87
(Pairwise AUC test)*	(p < 0.05)	(p < 0.05)	(p > 0.05)
(Pairwise AUC test)**	(p < 0.05)	(p < 0.05)

Region	Outside NCO
Modality	Unimodal (OCT)		Multimodal (OCT + fundus)
Algorithm	Niemeijer et al. [18]	Hu et al. [22]	Registered-fundus	Multimodal
AUC	0.85	0.87	0.92	0.94
(Pairwise AUC test)*	(p < 0.05)	(p < 0.05)	(p < 0.05)
(Pairwise AUC test)**	(p < 0.05)	(p < 0.05)

Region	Entire region
Modality	Unimodal (OCT)		Multimodal (OCT + fundus)
Algorithm	Niemeijer et al. [18]	Hu et al. [22]	Registered-fundus	Multimodal
AUC	0.79	0.83	0.87	0.90
(Pairwise AUC test)*	(p < 0.05)	(p < 0.05)	(p < 0.05)
(Pairwise AUC test)**	(p < 0.05)	(p < 0.05)

^*pairwise AUC test [33] performed between the final multimodal approach and each of the other three indicated vessel segmentation approaches.

^**pairwise AUC test [33] performed between the registered-fundus and each of the two indicated OCT-based vessel segmentation approaches.

TABLE III.

AUC comparison among the Niemeijer et al. [18] OCT-only approach, the Hu et al. [22] OCT-only approach, the registered-fundus approach, and the multimodal approach for all 19 test eyes

Region	Inside NCO
Modality	Unimodal (OCT)		Multimodal (OCT + fundus)
Algorithm	Niemeijer et al. [18]	Hu et al. [22]	Registered-fundus	Multimodal
AUC	0.64	0.67	0.85	0.86
(Pairwise AUC test)*	(p < 0.05)	(p < 0.05)	(p > 0.05)
(Pairwise AUC test)**	(p < 0.05)	(p < 0.05)

Region	Outside NCO
Modality	Unimodal (OCT)		Multimodal (OCT + fundus)
Algorithm	Niemeijer et al. [18]	Hu et al. [22]	Registered-fundus	Multimodal
AUC	0.84	0.87	0.90	0.92
(Pairwise AUC test)*	(p < 0.05)	(p < 0.05)	(p < 0.05)
(Pairwise AUC test)**	(p < 0.05)	(p < 0.05)

Region	Entire region
Modality	Unimodal (OCT)		Multimodal (OCT + fundus)
Algorithm	Niemeijer et al. [18]	Hu et al. [22]	Registered-fundus	Multimodal
AUC	0.78	0.83	0.85	0.89
(Pairwise AUC test)*	(p < 0.05)	(p < 0.05)	(p < 0.05)
(Pairwise AUC test)**	(p < 0.05)	(p < 0.05)

^*pairwise AUC test [33] performed between the final multimodal approach and each of the other three indicated vessel segmentation approaches.

^**pairwise AUC test [33] performed between the registered-fundus and each of the two indicated OCT-based vessel segmentation approaches.

Fig. 9.

ROC curves of four different vessel segmentation approaches for all the 19 test eyes. ROC curves in the region (a) inside the NCO, (b) outside the NCO, and (c) entire region.

From Table III, for the region inside the NCO, the AUC of the OCT-only approach of Niemeijer et al. [18], our previous OCT-only approach [22], our current registered-fundus and multimodal approaches are 0.64, 0.67, 0.85, and 0.86 respectively; for the region outside the NCO, are 0.84, 0.87, 0.90, and 0.92 respectively; for the entire region are 0.78, 0.83, 0.85, and 0.89 respectively. Based on the p-values of the AUC comparison, the two present vessel segmentation approaches (registered-fundus and multimodal) both perform significantly better than the two previous OCT-based unimodal approaches in both the region inside and outside the NCO. In the region outside the NCO, the multimodal approach performs significantly better than the registered-fundus approach and in the region inside the NCO, it presents a similar performance to the registered-fundus approach. Overall, the multimodal approach performs significantly better than the registered-fundus approach.

Two example visual comparisons of the four different vessel segmentation approaches are provided in Fig. 10 and 11 respectively. As can be seen, both the quantitative and qualitative results of the two present fundus-related approaches demonstrate a great improvement over the two previous unimodal OCT-based approaches.

Fig. 10.

Example comparison of the four different vessel segmentation algorithms. (a) Cropped fundus registered image. (b) Vessel-oriented OCT projection image. (c-f) Vessel segmentation from Niemeijer's previous OCT [18], our previous unimodal OCT [22], registered-fundus, and multimodal approach respectively. The red arrows indicate the false positives or vessel breaks due to the presence of the optic disc/NCO boundary.

Fig. 11.

Example comparison of the four different vessel segmentation algorithms. (a) Cropped fundus registered image. (b) Vessel-oriented OCT projection image. (c-f) Vessel segmentation from Niemeijer's previous OCT [18], our previous unimodal OCT [22], registered-fundus, and multimodal approach respectively. The red arrows indicate the false positives or vessel breaks due to the presence of the optic disc/NCO boundary. The green arrow indicates the false positive from the choroidal vessels.

In order to also evaluate the performance of the algorithms on the data originally assigned to the training set, we also repeated the experiments by flipping the roles of the training data and test data (i.e., we trained the algorithms on the 19 images originally assigned to the test set and tested the algorithms on the 15 images originally assigned to the training set). Similar results were obtained. In particular, for the region inside the NCO, the AUC of the OCT-only approach of Niemeijer et al. [18], our previous OCT-only approach [22], our current registered-fundus and multimodal approaches are 0.65, 0.69, 0.87, and 0.87 respectively; for the region outside the NCO, are 0.85, 0.88, 0.91, and 0.92 respectively; for the entire region are 0.79, 0.82, 0.88, and 0.89 respectively.

V. DISCUSSION AND CONCLUSIONS

We present two novel retinal vessel segmentation approaches, i.e. the registered-fundus and the NCO false-positive-suppression-based multimodal vessel segmentation approach by utilizing the vessel information from fundus photographs with the aim to obtain a better vessel identification in the SD-OCT volumes. The registered-fundus vessel segmentation segments the vessels from the original fundus photographs and then registers the segmented vessel images to the SD-OCT modality. The multimodal approach simultaneously segments retinal vessels using the complementary information from both SD-OCT volumes and color fundus images. The two fundus-related vessel segmentation approaches perform significantly better than two previous OCT-based unimodal approaches in the regions both inside and outside the NCO.

The registered-fundus approach in general provides an accurate vessel segmentation due to the high vessel contrast on fundus photographs. However, it has certain typical limitations, for instance, the typical vessel breaks near the optic disc/NCO as indicated by the red arrows in Fig. 10.e and similar optic disc/NCO false positives to Niemeijer's OCT vessel segmentation approach as indicated by the red arrow in Fig. 11.e. In addition, it also presents typical false positives from choroidal vessels as indicated by the green arrow in Fig. 11.e. Such limitations are overcome, at least in part, by the multimodal approach for the following reasons. First, the incorporation of the pre-segmented NCO positional information to both the OCT and fundus pixel features allows the suppression of the false positives near the NCO/optic disc. Second, as the fundus photographs are the projection of the whole structures in the ONH, the choroidal vessels may present on fundus photographs, although the visibility is relatively low compared to the retinal vessels on fundus photographs. However, the vessel-oriented OCT projection images are taken from the layer around the RPE/BM plane in SD-OCT volumes, where the choroidal vessels are not visible. The incorporation of the OCT features helps to suppress the responses from the choroidal vessels. In addition, the choroidal vessels do not have the same radial-distribution tendency around the center of the NCO as the retinal vessels. The oriented Gabor wavelets thus likely also help to suppress their responses.

The presented multimodal algorithm overall performs better than the other three algorithms. However, this approach still has limitations. First, as can be seen from a qualitative analysis of the results, some segmented vessels present breaks and some small vessels are missing. As the smaller retinal vessels are, in general, more visible on the full-resolution fundus photographs, enhanced multiscale approaches to better utilize the original resolution of the fundus photographs may help in addressing this issue. (Nevertheless, in many applications, having the larger vessels appropriately segmented is most important.) Second, the registration errors of the fundus-to-OCT could misguide the classification. For instance, when large mis-alignment of the two modality images occurs, the classifier may treat a non-overlapped vessel as two vessels.

Within our dataset, we primarily noticed registration issues in the few cases with a visible motion artifact in the SD-OCT projection images (e.g., see Fig. 12). Thus, in future work, incorporating methods to first correct such artifacts [35] is expected to be useful. Furthermore, exploring alternate registration approaches (including use of nonlinear transformations) may be beneficial. Third, the expert may not have always traced very small vessels they deemed “incomplete”, whereas the algorithm would have found portions of these vessels, thus (incorrectly) causing the identification of these regions as false positives.

Fig. 12.

Example “worst-case” performance of multimodal approach. (a) Original color fundus image. (b) Registered fundus image. (c) Vessel-oriented OCT projection image. (d) Multimodal vessel segmentation result. Note that the motion artifacts within the SD-OCT volume as illustrated with yellow arrows made registration difficult and correspondingly resulted in a relatively poor segmentation result compared to the other results in the dataset. However, such a segmentation result would still be expected to be usable in many applications.

While we have compared the algorithms using a ROC analysis and presented the qualitative results using the continuous soft-label vesselness maps, many applications will require the choice of a single threshold for differentiating vessels from background. Such a threshold can be selected by choosing an operating point on the ROC curve with the desired tradeoff between sensitivity and specificity for the given application. For example, for the multimodal approach, the operating point on the ROC curve on a 45-degree line closest to the upper-left corner (0,1) of the plot would provide a sensitivity of 0.88 and a specificity of 0.89. This corresponds to using a threshold of 70 out of 255.

Our implementation (using non-optimized and non-parallelized C++ code) of the complete multimodal approach requires approximately 11 minutes to segment the vessels within a single eye (using a Windows 64-bit PC with a 3.06 GHz Intel Xeon W3550 processor and 24 GB RAM) with the majority of the time devoted to the SD-OCT layer+NCO segmentation (approximately 3 minutes), the segmentation of vessels on the fundus photograph (approximately 0.5 minutes), the preliminary segmentation of vessels on the SD-OCT projection image (approximately 1 minute), the fundus-to-OCT registration (approximately 0.5 minutes) and the final feature extraction and k-NN pixel classification steps (approximately 6 minutes). Because the registered-fundus approach does not require the final feature extraction and pixel classification steps, it requires approximately 5 minutes of total computing time. Reflecting a subset of the above processing steps, the Niemeijer et al. [18] OCT-only approach requires approximately 4 minutes of computing time (3 minutes for layer segmentation and 1 minute for feature extraction/classification). The OCT-only approach of Hu et al. [22] requires approximately 5.5 minutes of total computing time (3 minutes for layer+NCO segmentation and 2.5 minutes for feature extraction/classification).

Because of the dependence of all approaches on the time required for layer segmentation (3 minutes) and pixel classification (ranging from 0.5 minutes for the registered-fundus approach to 0.5 + 1 + 6 = 7.5 minutes for the multimodal approach), we are actively exploring use of parallel implementations of the graph-based segmentation approach and the k-NN classifier. In addition, because of the dependency of the running time of a k-NN classifier on the number of features, reducing the number of features via a feature-selection approach (e.g., the sequential forward-floating search [36]) may help to further reduce the running time.

While this work has focused on the evaluation of the multimodal approach for purposes of obtaining a vessel segmentation for further use in SD-OCT volumes (e.g., for further SD-OCT-based image analysis tasks), it is also important to consider its potential applicability for obtaining a vessel segmentation for further use in fundus photographs. Admittedly, because of the increased visibility of vessels within fundus photographs and often sufficient performance of existing approaches, using a multimodal approach for purposes of obtaining a slightly better vessel segmentation within fundus photographs is not as critically needed as is obtaining improved SD-OCT vessel segmentations. Nevertheless, it is our expectation that use of multimodal information would enable improved vessel segmentations, primarily due to a reduction in false positives near the NCO and choroidal vessels as discussed previously. Use of SD-OCT information may also help in other ways, such as more readily enabling the definition of important landmarks such as the fovea. Note that because of the increased resolution of fundus photographs compared with the x-y projected resolution of SD-OCT volumes, the typically larger field of view of fundus photographs, and the possibility of registration errors, it will be most fair (at least at first) to develop and evaluate such an approach through a multimodal training set designed specifically for the segmentation of vessels within fundus photographs.

In conclusion, we present a novel registered-fundus and a novel multimodal vessel segmentation approach to help obtain better vessel profiles in the SD-OCT volumes. Overall, the two present fundus-related approaches perform better than two closest previous OCT-based vessel segmentation approaches. The multimodal approach performs better than all the three unimodal vessel segmentation approaches of the registered-fundus and the two OCT-based approaches quantitatively and qualitatively.