RAC-CNN: multimodal deep learning based automatic detection and classification of rod and cone photoreceptors in adaptive optics scanning light ophthalmoscope images

Abstract

Quantification of the human rod and cone photoreceptor mosaic in adaptive optics scanning light ophthalmoscope (AOSLO) images is useful for the study of various retinal pathologies. Subjective and time-consuming manual grading has remained the gold standard for evaluating these images, with no well validated automatic methods for detecting individual rods having been developed. We present a novel deep learning based automatic method, called the rod and cone CNN (RAC-CNN), for detecting and classifying rods and cones in multimodal AOSLO images. We test our method on images from healthy subjects as well as subjects with achromatopsia over a range of retinal eccentricities. We show that our method is on par with human grading for detecting rods and cones.

1. Introduction

Analysis of rod and cone photoreceptors is valuable for the study, diagnosis, and prognosis of various retinal diseases. The high resolution of adaptive optics (AO) ophthalmoscopes enables the visualization of photoreceptors in the living human retina [1,2], and these AO ophthalmoscopes have been used to study the properties of cones [3–8] and rods [6,8–10] in both healthy and pathological eyes. AO ophthalmic imaging was first demonstrated with a flood illumination camera [11]. Currently however, the most widely used AO ophthalmic technology is the AO scanning light ophthalmoscope (AOSLO), due to the superior image contrast provided by its axial sectioning, and its potential higher transverse resolution [1,12–16]. In addition to reflectance imaging, the point scanning in AOSLO enables multiple-scattering imaging, which reveals transparent retinal neuronal and vascular structures [17–19], as well as single- and two-photon fluorescence [20–23], and fluorescence lifetime imaging [24]. Finally, AO optical coherence tomography allows for acquisition of 3-dimential images with comparable transverse resolution and an order of magnitude superior axial resolution, only limited by the light source spectrum [25–30]. Confocal AOSLO is able to visualize rods and foveal cones, the smallest photoreceptors in the retina [9]. In the past decade, the benefits presented by non-confocal AOSLO modalities have been explored, with split detector AOSLO [17] providing a number of advantages. Split detector AOSLO has a reduced ability to visualize rods in comparison to confocal AOSLO, but due to the different source of contrast, split detector AOSLO is often able to reduce ambiguity in identifying cones, especially in diseased eyes [31].

To utilize the captured images for clinical or research purposes, quantitative metrics such as rod and cone density, spacing, size, and luminance are often measured [32]. Generally, before these quantitative metrics can be calculated, each individual rod or cone in a region of interest (ROI) must be localized. The current gold standard method of manually marking these photoreceptors is highly subjective [33] and time consuming, which acts as a bottleneck limiting the clinical utilization of AOSLO systems. To combat this problem, several automated algorithms have been developed to detect cones in ophthalmic AO images taken from healthy [34–50] and pathological [31,48,50] subjects. To date, no well validated automatic method for detecting individual rods in AOSLO images or method for classifying between cones and rods has been published.

In recent years, deep learning has achieved state-of-the-art results for a variety of image processing tasks. Of particular note are convolutional neural networks (CNNs), which apply a sequence of transforming layers to an image with weights learned directly from training data [51]. CNNs have been utilized for a variety of tasks in ophthalmic image processing including classification [52–56], segmentation [57–62], and image enhancement [63]. CNNs have been used to achieve state-of-the-art performances for cone localization in ophthalmic AO images in healthy [46,49] and pathologic [31,50] eyes. Our recent work [31] showed that a CNN using multimodal confocal and split detector AOSLO information could improve the performance of detecting cones in subjects with achromatopsia (ACHM) by utilizing the complimentary information captured in both modalities.

In this work, we present the first validated method for automatically detecting and classifying rod and cone photoreceptors in AOSLO images. We develop a novel CNN semantic segmentation architecture that combines information from both the confocal and split detector AOSLO modalities. We show that our method is able to accurately localize and classify rods and cones in AOSLO images from both healthy subjects and from those with ACHM. We validate the results of our method against the current gold standard of manual marking.

2. Review

2.1. AOSLO photoreceptor imaging

AOSLO imaging is able to visualize rod and cone photoreceptors within the human retina [9]. Although the size and density of cones and rods vary as a function of retinal eccentricity [64], rods are generally smaller and more numerous than cones in healthy subjects. Rods are often more difficult to visualize, which may be due to their small size [9]. Figures 1(a) and 1(b) provide an example of co-registered confocal and split detector AOSLO images taken from a healthy subject. It can be seen that the confocal image is able to visualize both the cones and relatively small rods. The split detector image is not able to visualize rods, but it has high contrast in visualizing cones and can be helpful for reducing ambiguities in identifying cone photoreceptors [31,45].

Fig. 1.

Rod and cone photoreceptor visualization on AOSLO. (a) Confocal AOSLO image at 7° from the fovea in a normal subject. (b) Co-registered non-confocal split detector AOSLO image from the same location as (a). (c) Confocal AOSLO image at 3° from the fovea in a subject with ACHM. (d) Simultaneously captured split detector AOSLO image from the same location as (c). Cone photoreceptor examples are shown with magenta arrows, and rod photoreceptor examples are shown with yellow arrows. Scale bars: 10 μm.

Figures 1(c) and 1(d) show an example of simultaneously captured confocal and split detector AOSLO images from a subject with ACHM. ACHM is a genetic retinal disorder characterized by a lack of cone function resulting in color blindness, photophobia, nystagmus, and severely reduced visual acuity. In confocal AOSLO images of subjects with ACHM, cones appear as dark spots instead of the bright spots seen when imaging healthy subjects [17,65]. Because rods can still be visualized, it is sometimes possible to infer the positions of cones by finding areas surrounded by rods. Split detector AOSLO imaging is able to visualize remnant cone structures in ACHM, and visualized cones have similar features as compared to cones imaged in healthy subjects. Even though split detector AOSLO can visualize most cones in ACHM, there is still ambiguity in determining some cone locations due to poor contrast seen in pathologic subjects [33], and using multimodal information has been shown to be beneficial in identifying cones [31]. Additionally, it has been seen that rods may sometimes be visualized with split detector AOSLO imaging in ACHM subjects [17].

2.2. Convolutional neural networks

CNNs apply a series of transforming layers to complete a specific task. These networks learn filters to extract relevant features directly from training data. Of note for medical imaging processing is U-net [66], a semantic segmentation network that takes a full-sized image and outputs a classification for each pixel in the original image. U-net and its variants have been used for a variety of semantic segmentation [61,66–68] and detection [68] tasks across multiple imaging modalities.

The performance and function of a CNN depends on the layers that compose it. Specific layer types used in this work include convolutional, batch normalization, rectified linear units (ReLU), max pooling, concatenation (i.e. fusion), unpooling, and soft-max layers. Convolutional layers convolve an input of size W×H×D (before padding) with N kernels of size F×G×D with a stride of 1 which produces an output of size W×H×N, where the output can be considered a stack of N feature maps. For each of these N feature maps, a potentially different bias value is added. The weight values for the kernel and the bias values are learned automatically through training. Batch normalization layers [69] normalize their inputs based on mean and variance statistics, which can reduce internal covariate shift to decrease overfitting during training. ReLU layers [70] transform their inputs by setting all negative values to 0, which speeds up the training process and improves the performance of the network by adding a source of non-linearity [71]. Max pooling layers apply a max operation over the first two dimensions of their input in a P×Q window with a stride of 2, which effectively down-samples the input by a factor of 2 in the first two dimensions. Concatenation layers combines two inputs of size A×B×Y and A×B×Z into a single output of size A×B×(Y + Z). Unpooling layers are used in a decoder portion of a network to up-sample their input while preserving the spatial information from the max pooling layer in the corresponding encoder portion of the network [72]. Finally, a soft-max [73] layer takes an input of size I×J×C, where C is the number of classes, and applies the soft-max function across the third dimension to get the probability of each point in the first two dimension belonging to each class. For semantic segmentation, I×J is the size of the original image.

3. Methods

The steps for training and testing our proposed rod and cone detection and classification algorithm are outlined in Fig. 2. In the training phase, we used manual rod and cone markings on paired confocal and split detector images to create label and weight maps. These maps and the original images were then used to train a semantic segmentation CNN to create probability maps of rod and cone locations, on which the parameters for rod and cone localization were learned. In the testing phase, we then used the trained CNN and learned detection parameters to detect and classify rods and cones in paired confocal and split detector images that had not been previously seen.

Fig. 2.

Outline of the CNN AOSLO rod and cone detection algorithm.

3.1. Data sets

The images for the data sets used in this work were acquired at the Medical College of Wisconsin using a previously described AOSLO system [13,17], which simultaneously captures both confocal and split detector images. The images were obtained from the Advanced Ocular Imaging Program image bank (MCW, Milwaukee, Wisconsin). Images were acquired from healthy subjects and subjects with ACHM. For each subject, a series of image sequences was captured along the temporal meridian over a range of eccentricities using a 1.0° field of view for each sequence. For the healthy subjects, image sequences were repeated at each location over a 0.05 D focus range at 0.005 intervals. Each image sequence was strip-registered and averaged using between 19 and 70 frames, as previously described [17,74]. For healthy subjects, processed images from the multiple focus positions were manually aligned and overlapping regions were then cropped and used to create a new image sequence that was strip-registered and averaged once more, resulting in a further enhanced image. We extracted ROIs from the averaged images to form our data sets. Lateral scale/sampling for each subject was calculated using axial length measurements from an IOL Master (Carl Zeiss Meditec Inc., Dublin, California, USA).

We used separate data sets for the healthy and ACHM cases. Our healthy data set consisted of 40 confocal and split detector image pairs from 8 subjects. For each subject, there were 5 image pairs that were approximately evenly spread between 3° and 7° from the fovea. The average image size for the healthy data set was 115×115 µm². Our ACHM data set consisted of 49 confocal and split detector image pairs from 7 subjects. For each subject, there were 7 image pairs that were approximately evenly spread between 1° and 7° from the fovea. The average image size for the ACHM data set was 116×116 µm². All image pairs from both data sets had their cones and rods marked by two manual graders independently. Both modalities were used when creating the markings, resulting in a single set of markings for each confocal and split detector image pair for each grader. The first manual grader marked a total of 7847 cones and 32111 rods across the healthy data set, and a total of 3204 cones and 32641 rods across the ACHM data set. The second manual grader marked a total of 7853 cones and 31664 rods across the healthy data set, and a total of 3071 cones and 28418 rods across the ACHM data set.

3.2. Labeling and weighting

Training our semantic segmentation CNN to classify and detect rod and cone locations requires label and weight maps for the training data. For each confocal and split detector image pair in the training data set, both of size I×J, we generated a single label map of size I×J. Each pixel in the label map corresponds to the same location in the confocal and split detector images and classifies those pixels as one of three possible classes: rod, cone, or background. Because our manual markings (Figs.3(c) and 3(d)) are only photoreceptor locations and not segmentations, we developed a method similar to the one presented for cell detection in Falk et al. [68] to use the manual markings to create label maps that could be used for photoreceptor detection and classification. We rounded the coordinates of the cone markings from the first grader to the nearest pixel location and labeled all pixels within a 3.5 pixel radius of the coordinates as the cone class. Next, we labeled all pixels in a 2×2 pixel block around the manual rod coordinates from the first grader as the rod class. If there would be any overlap between the cone and rod labeled pixels, we labeled them as rods. We labeled all remaining pixels as background to produce the final label map (Fig. 3(e)). The sizes of the cone and rod masks were chosen empirically to be smaller than the sizes of rods and cones observed in a separate data set.

Fig. 3.

Creating label and weight maps from AOSLO image pairs. (a) Confocal AOSLO image. (b) Co-registered non-confocal split detector AOSLO image from the same location. (c-d) Manually marked rod positions shown in yellow and cone positions shown in magenta on the confocal image shown in (a) and on the split detector image shown in (b). (e) Label map generated from the markings in (c-d). (f) Weight map corresponding to the label map in (e).

For each label map, we also generated a weight map of the same size that determined how much weight to place on correctly classifying each pixel when training the CNN. Because there was a large difference in the number of labels for each class, we adjusted the weights to balance for this difference in class representation. The weight was determined by:

$$w(x)=\{\begin{array}{c}{L}_{\mathit{\text{Background}}}/L_{\mathit{\text{Rod}}}\\ L_{\mathit{\text{Background}}}/L_{\mathit{\text{Cone}}}\\ 1\end{array}\begin{array}{c}l(x):\mathit{\text{Rod}}\\ l(x):\mathit{\text{Cone}}\\ l(x):\mathit{\text{Background}}\end{array},$$ (1)

where l(x) is the label for pixel x, and w(x) is the associated weight for pixel x. L_Rod, L_Cone, and L_Background are the number of pixels throughout the entire training data set labeled as rod, cone, or background, respectively. An example weight map is shown in  Fig. 3(f).

3.3. Convolutional neural network

We built a novel dual-mode semantic segmentation network which we call the rod and cone CNN (RAC-CNN). The architecture of the network is shown in Fig. 4, and was based on the encoder-decoder architectures with skip connections seen in U-net [66] and ReLayNet [61], along with the dual-mode nature and filter sizes used in the DM-LF-CNN [31] used previously for AOSLO images. Our RAC-CNN is composed of two structurally identical paths, with one path taking a confocal image as the input and the other taking the matching split detector image. Each path consists of three contracting encoder blocks, followed by a transition block, and then three expanding decoder blocks. Each encoder block consists of a convolutional layer, a batch normalization layer, a ReLU layer, and a max pooling layer in that order. The transition block has a convolutional layer, a batch normalization layer, and a ReLU layer. Each decoder block consists of an unpooling layer that uses the max pooling indices from the corresponding encoder block, a concatenation layer that combines the input with the output from the ReLU layer from the corresponding encoder block (known as a skip connection [75]), a convolutional layer, a batch normalization layer, and a ReLU layer. All convolutional layers in the encoder, transition, and decoder blocks used 64 kernels of size 5×5×D (where D is the size of the third dimension of the input into the convolutional layer), and all max pooling layers had windows of size 2×2. The output of both paths is concatenated together and put through a final convolutional layer with 3 kernels of size 1×1×128 and a soft-max layer, which outputs the probability of each pixel pair in the original images belonging to each class.

Fig. 4.

The rod and cone CNN (RAC-CNN) architecture, which consists of the following layers: convolutional (Conv(F,G,N) where F and G are the kernel sizes in the first two dimensions and N is the number of kernels), batch normalization (BatchNorm), ReLU, max pooling (MaxPool(P,Q) where P and Q are the window dimensions), unpooling, concatenation, and soft-max. The same structure is used in the split detector AOSLO and confocal AOSLO paths.

The weight and bias parameters must first be learned before a network can be used for inference. We trained separate networks for the healthy and ACHM cases. First, we initialized the weights of the convolutional layers by drawing from a Gaussian distribution with a standard deviation of $\sqrt{2/K}$, where K is the number of nodes in one kernel, and initialized the biases of the convolutional layers to 0 [66]. The network parameters were then learned using stochastic gradient descent to minimize the weighted cross-entropy loss:

$$L=-\sum _{x\in \mathrm{\Omega }}w(x)\text{log}(p_{l(x)}(x)),$$ (2)

where x is a pixel in the image domain Ω, w(x) is the associated weight for the pixel, and p_l(x)(x) is the probability output from the soft-max layer for the pixel x associated with the true class label l(x). As in [66], we used a mini-batch size of 1 so that each step of the stochastic gradient descent was over a single image. Note that when a mini-batch size of 1 is used, batch normalization layers act as instance normalization layers [76]. During inference, we used the mini-batch mean and variance for the batch normalization layers as opposed to the moving average mean and variance which can be inaccurate when small batch sizes are used. We performed the training over 100 epochs, where in each epoch all the training images are seen once. The learning rate was set initially to 0.1 and was gradually lowered to 0.0001 by the final epoch, with logarithmically equally spaced values between epochs. Weight decay was set to 0.0005, and momentum was set to 0.9, which are the default values in MatConvNet [77]. We also used data augmentation in the form of vertical flipping and translations. Every time an image pair is seen during training, there was a 50% chance for both images to be flipped vertically. To train for translational invariance, a square 200×200 pixel region was randomly chosen and used for training each time an image pair was seen. The same region was used for the paired confocal and split detector images.

3.4. Photoreceptor localization

Finally, we used the probability maps generated by the trained RAC-CNN to detect cone and rod locations for split detector and confocal image pairs. Each image pair is fed into the trained network which outputs a 3-dimensional matrix of size I×J×3, where I×J is the size of either of the input images. This matrix can be thought of as a stack of three probability maps, where each probability map corresponds to one of the classes: cones, rods, or background. From this matrix, we pulled out the probability maps for rods (Fig. 5(c)) and cones (Fig. 5(d)) and processed them separately using the method presented in Cunefare et al.[31]. In brief, we smoothed each map by convolving it with a Gaussian filter with standard deviation σ to filter out spurious maxima. Next, we applied the extended-maxima transform using MATLAB’s imextendedmax function [78], which finds connected maximal regions where the probability difference in the region is less than or equal to a set value H and outputs these regions in a binary map (Figs. 5(e) and 5(f)). We found all connected clusters in each binary map to use as potential candidates for rod or cone positions, depending on the probability map used, and eliminated weak candidates by removing any cluster whose maximum value in the filtered probability map was less than a threshold T. Finally, we found the center of mass for all the remaining clusters which were considered to be the rod or cone positions (Figs. 5(g) and 5(h)). We set the values of σ, H, and T automatically by maximizing the average Dice’s coefficient (explained in 3.5) across the same training images and manual markings used to train the RAC-CNN over a set of potential parameter combinations. The values of σ, H, and T differed and were found separately for the cone and rod cases, and for the healthy and ACHM data sets.

Fig. 5.

Detection of rods and cones in confocal and split detector AOSLO image pairs. (a) Confocal AOSLO image. (b) Co-registered non-confocal split detector AOSLO image from the same location. (c) Rod probability map and (d) cone probability map generated from (a) and (b) using the trained RAC-CNN. (e) Extended maxima of (c). (f) Extended maxima of (d). (g-h) Detected rods marked in yellow and cones marked in magenta on the confocal image shown in (a) and on the split detector image shown in (b).

3.5. Validation

We validated the RAC-CNN detection method against the current gold standard of manual grading. We used leave-one-subject-out cross validation to evaluate our method, where the images from one subject were held back, and the images from the remaining subjects were used for training the network and photoreceptor localization parameters. The images from the held back subject were then evaluated using the trained network and parameters. This was repeated for all subjects in the data set, so that all subjects could be used for validation without overlap between subjects used for training and testing of the algorithm. We performed the validation separately for the healthy and ACHM data sets. The first set of manual markings was used for validation of our method. In order to compare to state-of-the-art cone detection methods, we also evaluated the performance of the LF-DM-CNN cone detection method [31]. The LF-DM-CNN was retrained and evaluated using leave-one-subject-out cross validation performed separately for the healthy and ACHM data sets. There are no other published methods for rod detection to compare to.

To quantify the performance of our RAC-CNN for detecting rods and cones, we first matched the automatically detected photoreceptors to the photoreceptors marked by the first grader one-to-one for each image pair in a similar fashion as presented in Cunefare et al.[31]. We performed the matching separately for rods and cones. An automatically detected photoreceptor was considered a true positive if it was located within a distance d of a manually marked photoreceptor of the same type. The value d was set to the smaller between 0.75 of the median spacing between manually marked photoreceptors of the specific type in the image and an upper limit based on the type of photoreceptor. The upper limit for cones was set to 16 pixels based on the value used in Cunefare et al. [31], and the upper limit for rods was set to 5 pixels empirically based on rod size observed on a separate data set. Automatically detected photoreceptors that were not matched to a manually marked photoreceptor of the same type were considered false positives, and manually marked photoreceptors that did not have a matching automatically detected photoreceptor of the same type were considered false negatives. If a manually marked photoreceptor matched to more than one automatically detected photoreceptor of the same type, only the automatically marked photoreceptor with the smallest distance to the manually marked photoreceptor was considered a true positive, and the remaining were considered false positives. Finally, automatically detected and manually marked photoreceptors within 7 pixels of the edges of the image were removed to avoid border artefacts. After matching for one photoreceptor type in an image pair, the number of automatically marked photoreceptors (N_Automatic) and manually marked photoreceptors (N_Manual) of that type can then be expressed as:

$${\mathrm{N}}_{\text{Automatic}}={\mathrm{N}}_{\text{TP}}+{\mathrm{N}}_{\text{FP}},$$ (3)

$${\mathrm{N}}_{\text{Manual}}={\mathrm{N}}_{\text{TP}}+{\mathrm{N}}_{\text{FN}},$$ (4)

where N_TP is the number of true positives, N_FP is the number of false positives, and N_FN is the number of false negatives. For each image pair, we then calculated the true positive rate, false discovery rate, and Dice’s coefficient [79,80] for rods and cones separately as:

$$\text{True positive rate}={\mathrm{N}}_{\text{TP}}/{\mathrm{N}}_{\text{Manual}},$$ (5)

$$\text{False discovery rate}={\mathrm{N}}_{\text{FP}}/{\mathrm{N}}_{\text{Automatic}},$$ (6)

$$\text{Dice′s coefficient}=\text{2}{\mathrm{N}}_{\text{TP}}/\text{(}{\mathrm{N}}_{\text{Manual}}+{\mathrm{N}}_{\text{Automatic}}\text{)}\text{.}$$ (7)

To assess inter-observer variability, the second set of manual markings was compared to the first set of manual markings following the same procedure. Additionally, the LF-DM-CNN results were computed in the same way for only cones.

4. Results

We implemented and ran all methods in MATLAB 2017b (The MathWorks, Natick, MA) and used MatConvNet [77] 1.0-beta23 for training and running the RAC-CNN. We ran all experiments on a desktop PC with an i7-5930 K CPU at 3.5 GHz, 64 GB of RAM, and a GeForce GTX TITAN X GPU. Using the trained RAC-CNNs, the average run time for detecting and classifying rods and cones in new images was 0.2 seconds across the healthy data set (average image size of 236×237 pixels) and 0.2 seconds across the ACHM data set (average image size of 231×231 pixels). Using the trained LF-DM-CN, the average run time for detecting only cones was 30.2 seconds across the healthy data set and 29.0 seconds across the ACHM data set. The mean values and standard deviations of the automatically chosen photoreceptor detection parameters σ, H, and T across the validation groups are given in Table 1. The average time to train our RAC-CNN and to learn the detection parameters was under 2 hours for both the healthy and ACHM data sets.

Table 1. Average detection parameters across the healthy and ACHM validation groups (standard deviations shown in parenthesis).

	σ	H	T
Healthy-Rod	0.7 (0.2)	0.01 (0.02)	0.66 (0.12)
Healthy-Cone	2.5 (0.4)	0 (0)	0.48 (0.09)
ACHM-Rod	0.9 (0.2)	0.03 (0.03)	0.63 (0.08)
ACHM-Cone	2.2 (0.7)	0.01 (0.02)	0.66 (0.15)

The performance of the automatic methods with respect to the first set of manual markings over the 40 images in the healthy data set is summarized in Table 2, along with a comparison between the two manual graders. Table 3 shows the same performance comparisons across the 49 images in the ACHM data set. The median Dice’s coefficient values across the healthy data set was 0.92 for rods and 0.99 for cones for the RAC-CNN, and 0.90 for rods and 0.98 for cones for the comparison between graders. The median Dice’s coefficient values across the ACHM data set was 0.91 for rods and 0.90 for cones for the RAC-CNN, and 0.84 for rods and 0.89 for cones for the comparison between graders.

Table 2. Average performance of the automatic methods and second grader with respect to the first set of manual markings across the healthy data set (standard deviations shown in parenthesis).

	True positive rate	False discovery rate	Dice’s coefficient
RAC-CNN (Rods)	0.92 (0.06)	0.10 (0.06)	0.91 (0.04)
Grader #2 (Rods)	0.88 (0.06)	0.10 (0.06)	0.89 (0.05)
RAC-CNN (Cones)	0.97 (0.04)	0.03 (0.03)	0.97 (0.03)
LF-DM-CNN [31] (Cones)	0.98 (0.04)	0.03 (0.04)	0.97 (0.03)
Grader #2 (Cones)	0.97 (0.03)	0.02 (0.03)	0.97 (0.03)

Table 3. Average performance of the automatic methods and second grader with respect to the first set of manual markings across the ACHM data set (standard deviations shown in parenthesis).

	True positive rate	False discovery rate	Dice’s coefficient
RAC-CNN (Rods)	0.91 (0.07)	0.11 (0.10)	0.89 (0.05)
Grader #2 (Rods)	0.78 (0.07)	0.08 (0.08)	0.84 (0.05)
RAC-CNN (Cones)	0.84 (0.14)	0.08 (0.09)	0.87 (0.10)
LF-DM-CNN [31] (Cones)	0.86 (0.12)	0.10 (0.10)	0.87 (0.09)
Grader #2 (Cones)	0.88 (0.11)	0.12 (0.09)	0.87 (0.07)

Figure 6 shows examples of the performance of our RAC-CNN method with respect to markings by the first manual grader on images from the healthy data set, and Fig. 7 provides examples from the ACHM data set. In the marked images, automatically detected photoreceptors that were matched to a manually marked photoreceptor of the same type (true positives) are shown in green, photoreceptors missed by the RAC-CNN method (false negatives) are shown in blue, and automatically detected photoreceptors with no corresponding manually marked photoreceptor (false positive) are shown in gold. For the sake of visual clarity, the rods and cones are shown on separate images. Figure 8 provides comparisons of the performances of the automatic methods for cone detection.

Fig. 6.

Performance of the RAC-CNN method on healthy images. Confocal AOSLO images from different subjects are shown on the top row, and the co-registered split detector AOSLO images are shown in the row second from the top. Rod detection results for the RAC-CNN method with respect to the first set of manual markings are shown on the second row from the bottom, and cone detection results are shown on the bottom row. Green points denote true positives, blue denotes false negatives, and gold denotes false positives. Dice’s coefficients for the rods and cones are 0.98 and 1 in (a), 0.94 and 0.99 in (b), and 0.91 and 0.95 in (c), respectively.

Fig. 7.

Performance of the RAC-CNN method on ACHM images. Confocal AOSLO images from different subjects are shown on the top row, and the simultaneously captured split detector AOSLO images are shown in the row second from the top. Rod detection results for the RAC-CNN method with respect to the first set of manual markings are shown on the second row from the bottom, and cone detection results are shown on the bottom row. Green points denote true positives, blue denotes false negatives, and gold denotes false positives. Dice’s coefficients for the rods and cones are 0.93 and 0.98 in (a), 0.94 and 0.93 in (b), and 0.89 and 0.88 in (c), respectively.

Fig. 8.

Performance of the automated algorithms for cone detection in a healthy (top) and ACHM (bottom) image pair. Simultaneously captured confocal and split detector images are shown in the two left columns. Performance with respect to manual cone markings for the RAC-CNN and our previous LF-DM-CNN [31] methods are shown in the right two columns and displayed on the split detector images. Only cones are included in this figure as LF-DM-CNN cannot detect rods. Green points denote true positives, blue denotes false negatives, and gold denotes false positives. Dice’s coefficients are 0.99 for both methods for the healthy image pair, and 0.92 for both methods for the ACHM image pair.

5. Discussion

We developed the RAC-CNN, an automatic deep learning based method for detecting and classifying rod and cone photoreceptors in multimodal AOSLO images. Our semantic segmentation based RAC-CNN worked very fast, able to detect and classify rods and cones in 0.2 seconds on image sizes that would take our previous patch based CNNs [31,46] over 10 seconds to only detect cones. We validated our method on images taken over a wide range of retinal eccentricities from healthy subjects and from subjects with ACHM. We showed that our method had good agreement with the current gold standard of manual grading. Finally, we showed that the performance of our method was similar to that between two different manual graders.

In Table 2, we show the performance of the RAC-CNN, LF-DM-CNN, and manual methods over the healthy data set. There was a lower performance for detecting rods than cones. This is likely due to the increased difficulty in visualizing rods which are smaller than cones in non-foveal regions of the retina. Table 3 shows the performance of the RAC-CNN, LF-DM-CNN, and manual methods across the ACHM data set, which as expected was worse than those reported for the healthy data set. There is often more uncertainty in identifying photoreceptors, and the images are generally nosier and blurrier in comparison to images taken from healthy subjects. Additionally, the healthy data set had extra averaging done to further enhance the quality of the confocal images which was not done for the ACHM set. Cone detection had a larger drop in performance in comparison to rod detection, likely because ACHM is a cone, rather than rod, dysfunction. On both healthy and ACHM images, performance of our RAC-CNN method was comparable to those reported by the state-of-the-art LF-DM-CNN cone-only detection method [31]. This is encouraging because we had previously shown that LF-DM-CNN cone detection performance is on a par with human grading. Yet, RAC-CNN was roughly 150 times faster and for the first time adds rod segmentation capability with human level accuracy.

As AOSLO is currently commonly utilized for assessment of known ophthalmic diseases, we utilized an automatic method to learn optimal parameters for the healthy and ACHM groups. For the sake of completeness, we tested an alternative scenario in which RAC-CNN and detection parameters are trained with leave-one-subject-out cross validation across both the healthy and ACHM data sets combined (with no indication given whether an image is from a healthy or pathological subject). The performance of the algorithm was identical at the reported precision of the Dice’s coefficient value for cone detection in healthy and ACHM groups, and for rod detection in the ACHM group. Only, the Dice’s coefficient value for rod detection in the healthy set was reduced from 0.91 to 0.90.

From Table 3 it can be seen that there is noticeable inter-observer variability for detecting photoreceptors in the ACHM data set, especially for rods. This is consistent with previous studies showing the difficulty in marking pathological images [33]. From the table it can be seen that the first manual grader had a tendency to mark more rods than the second grader in the ACHM data set. A CNN trained using the second set of manual markings would be expected to detect less rods than one trained on the first. Yet, if the goal is to measure the longitudinal change in photoreceptor density, it is expected that such biases to largely cancel out when comparing photoreceptor counts by a consistent automatic algorithm in the images of the same subject at different timepoints. Assessment and validation of the automated photoreceptor counting consistency in longitudinal studies is part of our future work.

To assess how each path contributed to the classification of the AOSLO images, we inspected the weights in final convolutional layer, which combines the features from the confocal and split detector paths. For the rod class in healthy subject networks, on average 67% and 33% of the weight magnitude was associated with the confocal and split detector information, respectively; while in ACHM subject networks on average 60% and 40% of the weight magnitude was associated with the confocal and split detector information, respectively. For both the healthy and diseased networks, more weight was given to the confocal modality for detecting rods, as rods are generally less visible in split detector AOSLO. However, the split detector information is still useful for resolving ambiguities sometimes seen in differentiating cones from rods in confocal AOSLO [31,45]. For the cone class in healthy subject networks, on average 59% and 41% of the weight magnitude was associated with the confocal and split detector information, respectively; while in ACHM subject networks on average 41% and 59% of the weight magnitude was associated with the confocal and split detector information, respectively. For detecting cones, more weight was given to the confocal information in the networks trained on healthy data, but more weight was given to the split detector information for the ACHM networks. This is consistent with the loss of ability to directly visualize cones in confocal AOSLO images of ACHM subjects. Finally, for the sake of completeness to further show the importance of using the information of both channels for detecting rods, we retrained the networks to use only the confocal information path for rod detection. We found that rod detection based only on the confocal channel reduced the accuracy as reported by Dice’s coefficient from 0.91 to 0.90 in the healthy data set, and from 0.89 to 0.86 in the ACHM data set.

There are a few possible avenues to improve our RAC-CNN method. First, accurate manual segmentations of the rod and cone boundaries could be useful for training the RAC-CNN. As our manual markings only contained the rod and cone positions, the label maps for the data did not reflect the variations seen between individual photoreceptors. This could be especially important for cones, whose sizes can vary significantly at different retinal eccentricities. Accurate segmentations could allow the network to be better trained or allow it to be trained for segmentation as well as classification and localization. Additionally, our method could be improved by increasing the amount of training data. Studies on rod photoreceptors have been limited in part due to the subjective and time-consuming nature of manually marking rods in AOSLO. We hope that by providing an automated method for detecting rod and cone photoreceptors we will enable researchers to perform more studies on the rod mosaic, and in turn generate more data which may then be used to further refine this and other automated methods. Finally, rather than simply averaging the registered AOSLO frames together, a modified network that uses temporal information [81] is expected to further enhance rod and cone localization.

Funding

Foundation Fighting Blindness (FFB)10.13039/100001116 (BR-CL-0616-0703-Duke); Research to Prevent Blindness (RPB)10.13039/100001818; National Institutes of Health (NIH)10.13039/100000002 (F30EY027706, P30EY001931, P30EY005722, P30EY026877, R01EY017607, R21EY027086, T32EB001040, U01EY025477, R21EY029804); Google10.13039/100006785 Faculty Research Award.

Disclosures

The authors declare that there are no conflicts of interest related to this article.