Repeatability of Adaptive Optics Automated Cone Measurements in Subjects With Retinitis Pigmentosa and Novel Metrics for Assessment of Image Quality

Abstract

Purpose

We determine the intersession repeatability of cone measurements via flood-illuminated adaptive optics (AO) imaging in patients with retinitis pigmentosa (RP), to better differentiate variation due to imaging inaccuracies versus pathology-driven change.

Methods

A total of 25 4° × 4° AO images were acquired three times on the same day in 10 subjects with RP, registered in i2K Retina, and cones were identified using a custom-built MATLAB algorithm. Nine equally spaced regions of interest were selected for each imaging set. A subset of subjectively “poor” and “good” quality images was selected by three independent graders, analyzed using cone density, cone location similarity (CLS) and cone spacing, and compared to age-matched normals.

Results

The coefficient of variation (CoV), repeatability, and percent repeatability of automated cone density were slightly higher in patients with RP compared to age-matched normals, but showed no statistically significant difference. The standard deviation of CLS and cone spacing of nearest-neighbor distance demonstrated a statistically significant difference between good- and poor-quality images.

Conclusions

Repeatability of automated cone density measurements in patients with RP is comparable to normals. Misidentification of cones due to image quality variability is a major limitation of automated cone counting algorithms in patients with RP. Our study suggests that CLS and cone spacing metrics could be used to help define image quality and, thus, increase confidence in automated cone counts in patients with RP.

Translational Relevance

The novel AO image quality assessment metrics described in our study could help to improve patient image interpretation, prognosis, and longitudinal care.

Introduction

Flood-illuminated adaptive optics (AO) is a high-resolution retinal imaging technique that uses a flash infrared imaging light source and finely tuned deformable mirrors to continuously sample imaging waveform distortions to reduce the inherent optical aberrations of the human eye. Flood-illuminated AO has been used to study the cone mosaic in numerous retinal conditions, including acquired and inherited retinal disorders, and color deficiencies.1–7 However, the majority of these studies assess a single imaging session for each subject and, thus, do not provide data about the intersession repeatability of cone identification via AO imaging. Intersession repeatability of cone density via flood-illuminated AO has been shown to be reliable in healthy subjects8 and, while this provides an important reference database, it does not describe the repeatability of AO imaging for individuals with retinitis pigmentosa (RP). Qualitative patterns and findings on flood-illuminated AO imaging have been described in subjects with RP,9 but there have not been any intersession quantitative studies to date. Recently, there have been studies investigating the repeatability of cone photoreceptor imaging via adaptive optics scanning laser ophthalmoscopes (AO-SLO) in subjects with no pathology as well as various retinal genetic diseases.10–15 AO-SLO images are created by scanning a small point source of light in a raster fashion; when combined with a pinhole filter, this technique leads to improved axial and lateral resolution compared to flood-illuminated AO. Unfortunately, these devices remain relatively expensive compared to flood-illuminated AO cameras and require substantial infrastructure and expertise to operate. Given that repeatability studies have yet to be performed via flood-illuminated AO in subjects with RP, establishing the repeatability in this population may help to improve clinical management through more accurate prognosis, disease monitoring and assessment of future therapeutic interventions. As AO technology improves, it may provide the ability to track not only global trends in retinal degeneration, but also monitor the health of individual cones in a longitudinal manner. Therefore, the initial goal of our study was to evaluate the repeatability of flood-illuminated AO images obtained in patients with RP via intersession cone analysis. However when reviewing the data, it became apparent that the heterogeneous image quality in our study population made automated cone identification unreliable. This problem is not new with cone identification algorithms, but one that tends to be exacerbated in subjects with abnormal retinal architecture secondary to a pathologic process. Through studying repeatability in these patients, we developed two new metrics that might allow for more objective image qualification, and, thus, increased confidence in the validity of automated cone identification for a given image.

Methods

Patients

This study adhered to the tenets of the Declaration of Helsinki and was approved by the Oregon Health & Science University IRB. Before enrollment, all patients signed an informed consent after the nature and possible consequences of the study were explained. Ten patients with a clinical diagnosis of RP (age range, 22–57 year; mean, 36.9 11.7) were recruited for this study (Table 1, Supplementary File S1) and were compared to 11 normal patients from a previous study8 with similar characteristics. Exclusion criteria included patients with significant opacification of the ocular media, subjects with uncontrolled nystagmus, head movement that prevented target fixation, visual acuity of <20/50, history of cataract surgery, visual field of <30° (Octopus Perimetry; Haag-Streit Diagnostics, Koniz, Switzerland) or cystoid macular edema detected via OCT imaging (Spectralis OCT, Heidelberg Engineering, Heidelberg, Germany). In summary, we selected for patients with good fixation, media clarity, visual acuity, and diverse genetic mutations.

Table 1.

RP Subject Characteristics

Age (years)	Sex	Axial Length OD (mm)	BCVA OD	BCVA OS	Genetic Testing
22	F	25.41	20/30	20/40	Unsolved
31	F	22.88	20/30	20/30	None
56	M	23.12	20/20	20/20	SEMA4A + CRB1
40	F	25.09	20/20	20/20	PRPH2
24	F	23.39	20/20	20/20	Unsolved
57	F	25.01	20/20	20/30	RHO
35	F	22.23	20/20	20/20	Unsolved
34	F	25.36	20/20	20/30	Unsolved
38	M	23.84	20/20	20/20	USH2A
32	F	24.23	20/15	20/15	RHO

Image Acquisition

Both eyes were dilated with 1% phenylephrine and 2.5% tropicamide before each imaging session. For each session, a series of 25 overlapping 4° × 4° images was acquired using the rtx1 flood-illuminated AO camera (Imagine Eyes, Orsay, France) covering a 12° × 12° field of the central macula in both eyes (Fig. 1A). Three imaging sessions were performed on each patient on the same clinic visit day between 9:00 AM and 3:00 PM. Retinal eccentricities were determined based on distance from the fovea (Fig. 1B). After image acquisition, each set of three images at each corresponding location were registered with an affine transformation in i2K Retina (DualAlign LLC, Clifton Park, NY) to create a stack (Fig. 1C). Images were subsequently prefiltered, and cone photoreceptors were identified using a custom-built MATLAB cone counting algorithm (MathWorks, Natick, MA) as described previously.8 In short, prefiltering was accomplished by background subtraction using a moving 11 × 11 moving average filter, and subsequent cone detection through local maxima intensity thresholding. Binary cone detection maps were generated for each image and used to create Voronoi diagrams representing cone density maps. The RP subjects were compared against age-matched normal subjects obtained from a prior AO repeatability study.8

Figure 1.

Image location, registration, and ROI selection. (A) Fundus photo OD with overlaid AO imaging region. (B) Macular AO imaging area with representative imaging locations (green-outlined boxes), the five different foveal eccentricities (white arrows), and ROIs at each imaging location (green-shaded squares). (C) Representation of full overlay across each imaging session capturing all ROIs (green boxes). (D) Representative incomplete overlay between images, causing exclusion of certain ROIs (red boxes).

Regions of interest (ROIs) were used to determine cone density variation in images across the three sessions for each fixation point. A grid of nine 100 × 100 μm ROIs was selected for each imaging set, with the central ROI positioned at the center of the padded stack overlay and the ROIs equally spaced 0.5° from each other (Fig. 1C). ROIs that did not completely lie within the boundary of the three registered images were excluded from analysis (Fig. 1D). There were 4500 total ROIs (10 subjects, 25 image stacks per eye, nine ROIs per location), of which 33 ROIs were excluded, all from the same subject. After registration using the Voronoi density plots, the preferred retinal locus, which for the purposes of our study we equated to the foveal center based on a prior study,16 was manually selected at the center of fixation when the fixation target was located at [0°, 0°]. For data analysis, ROIs were grouped based on the distance from the fovea of the central ROI at each imaging location. This created five different foveal eccentricity groups, as shown in Figure 1B. In a prior AO study of healthy subjects,8 images within 1.5° of the fovea were excluded from analysis due to the inability to reliably resolve cones secondary to condensed foveal photoreceptor packing. In patients with RP, it usually is possible to visualize cones throughout the fovea due to photoreceptor loss and increased cone spacing.2,9 Therefore, we decided to include all retinal eccentricities for analysis in this study to avoid introducing subjective assessment of each subject imaged.

A subset of ROIs ranging from 2° to 4° retinal eccentricity were evaluated for further subgroup analysis based on subjective image quality, which was determined by the ability to clearly visualize a hexagonal cone mosaic. These ROIs avoided regions with major blood vessels, but there were no other exclusion criteria. Three independent graders reviewed a set of training images to establish “good,” “intermediate,” and “poor” image quality. Following training, each grader independently reviewed the images and assigned one of the three grades. Images for which there was consensus of “good” and “poor” were included for analysis. When there was disagreement on an image grade, the graders collectively reviewed the image and a group decision was reached. Due to a low number of images graded as “intermediate” and a wide variation in image appearance resulting in reduced intergrader agreement, this group was not included in the final analysis.

Cone Density Statistical Methods

From each ROI that was included in analysis, average cone density, coefficient of variation (CoV), coefficient of repeatability, and repeatability were calculated. Cone densities for each ROI across all three sessions were found from the corresponding Voronoi maps and used to calculate average cone density. Coefficient of repeatability was found using an adaptation of the method described by Garrioch et al.15 Within-subject standard deviation (SD) of cone density across all sessions for a given ROI was found and converted to coefficient of repeatability using , where S_D = SD, n = number of subjects, and m = number of observations for each subject. Using the previous measurements, repeatability was computed as . Repeatability explains the variability in cone density attributable to the methods involved in image acquisition, which is independent of axial length as this is a metric of intersession measurements from individual subjects. Here, it is reported in terms of its raw value as well as a percentage of the mean cone density in the given region. In addition, 95% confidence intervals (CIs) are reported for average cone density, CoV, coefficient of repeatability, repeatability, and percent repeatability. For all statistical analyses we considered P < 0.01 to be statistically significant. One-way analysis of variance (ANOVA) statistical analysis was used to compare mean SD of cone spacing between subgroups with F, F_crit and P values reported. The F value represents the ratio of the variance calculated between the means to the variance within the samples of each group, while the F_crit is the minimum ratio required to demonstrate a statistically significant difference.

Cone Location Similarity

To our knowledge, cone location similarity (CLS) is a novel metric that attempts to determine “shared” cones over repeated imaging session at the same location using the spatial information as pixel coordinates. With our imaging system, average cone diameter varies from 4 to 6 pixels (3.13–4.69 μm); therefore, if any three identified cone locations were within 5 pixels (3.91 μm) of each other across all three aligned imaging sessions, we considered this to be a shared cone location. A visual representation of this process is shown in Figure 2. In the case that multiple cones exist within five pixels from a repeat session, the cone that minimizes the mean distance between the three identified cones is selected. CLS is reported as a percentage , where N = number of imaging sessions. We considered adjusting for axial length, as this can affect the imaged cone density, and theoretically cone size. However, given the inherent heterogeneity in cone luminance and reflectivity, which can cause notable variation in imaged cone size, we found that axial length adjustments did not impact our CLS measurements. Furthermore, as the algorithm seeks to detect shared cones across repeated sessions of one individual, axial length can be assumed constant in the individual and, therefore, not required for this task. In the final analysis, absolute number of cones and percent shared cones were used to evaluate image reliability between the “poor” and “good” quality image sets.

Figure 2.

Images from set (A) display analysis from one region of an adaptive optics image with good quality. Images from set (B) are from a region of lesser quality. For both sets (A) and (B), subset images 1 to 3 display a cropped view (100 μm²) from each of the three sessions. Subsets 4 to 6 show the presumptive cones detected by automated cone counting from each session, with the absolute number in the upper right corner. Subset 7 shows an overlay of identified cones from all three sessions, with the larger gray circles representing shared cones detected in all three sessions with nearest Euclidean distances ≤5 pixels. Subset 8 displays the final shared cones across all three sessions, with the absolute number in the upper right corner.

Cone Spacing

Cone spacing metrics were performed using the binary cone identification maps from the “poor” and “good” image sets. Cone spacing was analyzed using nearest-neighbor distance (NND) based on center-to-center spacing of adjacent cones, which is one of the techniques that has been explored and validated by prior studies.4,10,17–18 We also explored the SD and CoV of NND across these “poor” and “good” 100 × 100 μm ROIs as a method to evaluate and compare cone mosaic uniformity. Individual t-tests with a Bonferroni correction were used to evaluate statistical differences between sub-analysis groups.

Results

The intersession repeatability of adaptive optics images in patients with RP were evaluated using three main metrics: cone density, CLS, and cone spacing.

Cone Density

The average cone density, average CoV of cone density, repeatability, and percent repeatability are reported in Table 2. Average CoV between imaging sessions in subjects with RP range from 3.93% to 4.25%, which is slightly larger than findings in normal subjects (1.19%–4.14%),8 but shows no substantial difference. Repeatability ranges from 2464 to 3593 cones/mm² and percent repeatability ranges from 14.74% to 16.38% at varying eccentricities, which is at the higher end of the range observed in normal subjects (6.5%–15.0%),8 but is not substantially different.

Table 2.

Cone Density of RP Subjects

Foveal Distance, deg	Average Cone Density (CI), cones/mm2	Average CoV (CI), cones/mm2	Repeatability (CI), cones/mm2	Percent Repeatability in RP Subjects (CI)	Percent Repeatability in Normalsa
0	16,784 (± 533)	566 (±50)	3593 (±533)	15.42% (±3.58)	–
2	15,987 (±256)	597 (±33)	3514 (±256)	16.38% (±2.26)	10.50%
2.8	14,713 (±230)	530 (±29)	3154 (±230)	15.91% (±4.60)	8.25%
4	13,402 (±201)	522 (±25)	2764 (±201)	15.71% (±5.59)	6.50%
4.47	13,278 (±134)	526 (±18)	2573 (±134)	14.74% (±4.28)	11.75%
5.66	12,895 (±181)	547 (±25)	2464 (±181)	15.66% (±3.61)	15.0%

RP patient average cone density, average CoV, repeatability, and percent repeatability with corresponding CIs, as a function of foveal distance. N for each foveal distance: 0 = 174, 2 = 722, 2.8 = 720, 4 = 720, 4.47 = 1415, 5.66 = 708.

^aNormals data in column 6 obtained from a prior study by Feng et al.8

Cone Location Similarity

The CLS (CLS) between imaging sessions varied greatly across subanalysis groups (Table 3). Average CLS was 80.54% in normal “good” subjects and 40.47% in normal “poor” subjects, while it was 76.85% in RP “good” subjects and 39.57% in RP “poor” subjects. There was a statistically significant difference between the average CLS percentage of the “good”-quality versus the “poor”-quality groups (P < 0.01).

Table 3.

Summary of Cone Spacing and CLS Statistics

Subject Group	Average Cone Spacing, mm	Average SD of Cone Spacing, mm	CoV of Cone Spacing	Average CLS % (SD)
Normals good	5.73	1.02	11.73%	79.49 (7.21)
Normals poor	5.67	1.45	16.55%	41.07 (3.97)
RP good	5.27	0.87	10.53%	75.15 (9.98)
RP poor	5.64	1.52	17.89%	39.32 (11.18)

Average SD of cone spacing, CoV of cone spacing, and cone similarity percentage, across all four subanalysis groups. SD and CoV of cone spacing values were statistically significant between subgroups (Table 4). Average cone similarity percentages showed a statistically significant difference between the “good” and “poor” quality groups (P < 0.01).

Cone Spacing

The average SD and CoV of cone spacing values for each subanalysis group were 1.02 μm and 11.73% for normal “good” images, 1.45 μm and 16.55% for normal “poor” images, 0.87 μm and 10.53% for RP “good” and 1.52 μm and 17.89% for RP “poor” (Table 3, Fig. 3). When comparing subanalysis groups via 1-way ANOVA, there was a statistically significant difference between the means (F = 31.04, Fcrit = 2.79, P < 0.001). When analyzing subgroups via individual t-tests with a Bonferroni correction, all groups had a statistically significant difference except for normal “poor” versus RP “poor” and normal “good” versus RP “good” (Table 4).

Figure 3.

Boxplots showing the SD of cone spacing across all four subanalysis groups. Individual t-tests with a Bonferroni correction were run for every group combination, showing a statistically significant difference between all group pairings (P < 0.05), except for normals poor versus RP poor (P = 0.52) and Normals good versys RP good (P = 0.06, see Table 4).

Table 4.

t-Test Comparisons Across Subanalysis Groups

t-Test	P Value
Normals good versus normals poor	<0.01
Normals good versus RP good	0.06
Normals good versus RP poor	<0.01
Normals poor versus RP good	<0.01
Normals poor versus RP poor	0.52
RP good versus RP poor	<0.01

Individual t-tests with a Bonferroni correction were run between each subanalysis group for SD of cone spacing. All comparisons demonstrated a statistically significant difference, except for NP versus RPP.

Discussion

Accurate quantitative analysis of cone changes in patients with RP may prove to be a valuable tool by helping to augment the qualitative interpretation of disease progression in clinical patients and therapeutic trials. However, this is a difficult task, as the repeatability of flood-illuminated AO imaging in this patient population is limited by a variety of confounding factors. Our study attempted to quantify the changes in cone density across multiple imaging sessions to better delineate between random variations and true pathology-driven loss of cone photoreceptors. Additionally, while studying repeatability, we developed cone spacing and CLS image analysis techniques to use as objective metrics for image quality assessment.

A major challenge in AO imaging is determining objective image quality. Good-quality images with a clearly visible, uniform hexagonal cone mosaic are fairly easy to identify. However, when an image shows blurred or hazy photoreceptors, it often is very difficult to determine whether impaired cone visualization is due to retinal disease affecting reflectance, camera imaging artifact, or some combination thereof. An experienced AO image grader usually can provide accurate and repeatable interpretations of image quality, but this requires a significant amount of subjective interpretation and contextual awareness. Automated cone detection algorithms have been shown to perform very well in high quality images, but they can often misidentify retinal debris and noise as photoreceptors.19 Because automated cone detection is necessary for practical applications of flood-illuminated AO imaging systems, it is crucial to be able to differentiate between a good-quality and a poor-quality image to achieve confidence in the validity of identified cones. While there have been recent advances in cone detection algorithms and improvements on traditional techniques,20–22 this remains an area that requires significant attention.

Cone Density

As expected, average cone density was reduced in a topographic fashion in patients with RP when compared to healthy subjects.8 Our automated cone density profiles and values are similar to those reported in prior studies evaluating scanning laser ophthalmoscope (SLO) and flood-illuminated adaptive optics images in subjects with RP.23–25 The automated cone density repeatability metrics, such as CoV, repeatability, and percent repeatability were slightly higher in patients with RP compared to normal subjects,8 but still within a similar range. These findings make intuitive sense given the pathophysiologic disruption of the outer retina caused by RP. However, these findings must be interpreted with caution. The similar repeatability metrics in RP patients compared to normal subjects, especially in areas of disease with poor image quality, likely results from inherent inaccuracies in the automated cone-identification algorithm. Automated algorithms, at times, identify cellular debris and noise as photoreceptors in an inaccurate manner with regard to location, but a consistent manner with regard to average density over a given retinal area. This can create a false sense of confidence in cone density measurements, and, thus, repeatability measurements, especially over larger imaging areas in which disease is present. Due to the nonspecificity of cone density detection algorithms in differentiating between true and false cones, we decided to evaluate two additional photoreceptor imaging metrics in an attempt to improve the validity of cone identification.

Cone Location Similarity

In the highly organized, wave-guiding structure of a healthy cone photoreceptor mosaic, the location of identified cones should not change from one imaging session to the next. Many prior studies have demonstrated that cones may vary in degree of reflectivity signal due to a variety of factors. Physiologic diurnal variations and outer-segment renewal have been shown to affect cone reflectivity.26–29 The Stiles-Crawford effect and cone alignment changes can alter incident light angles and create heterogeneity in cone imaging.30–33 Although the cone reflectivity profiles are influenced by these factors, the retinal location and spatial orientation has been shown to remain constant.34–35 However, there often is more reflectivity variability and artifact in the diseased or dying retina.19 This can result in similar intersession cone density values via misidentification of noise as cones, but the specific cone locations themselves are inconsistent.

While there are some promising novel methods for cone identification,21–22 these techniques are not able to take image quality into consideration when performing cone counting, which generates uncertainty in the validity of the identified cones. We subjectively judged a subset of images as “good” or “poor,” and then decided to create and investigate a metric of spatial organization repeatability, called CLS, to assess how this spatial repeatability correlates with image quality. Figure 2 illustrates that intersession CLS is notably higher in “good” quality compared to “poor” quality images. Given these findings, CLS may be a very specific quantitative metric for grading image quality. It would help to differentiate between true photoreceptor signals and imaging noise when using an automated cone-identification algorithm, while also making image quality assessment entirely grader independent.

Cone Spacing

Another potential AO image analysis metric is the SD of cone spacing via NND. In a healthy cone mosaic, the photoreceptors are arranged in a hexagonal matrix for optimal packing efficiency. This creates theoretically uniform cone spacing at a given foveal eccentricity, with increasing inter-photoreceptor distances as foveal eccentricity increases. Therefore, in a healthy photoreceptor mosaic, the SD and CoV of cone spacing values should approach zero at a given eccentricity. Additionally, prior studies have shown that AO image cone spacing metrics can be used to differentiate healthy retina from retinal pathology at a given foveal eccentricity, based on an increase in the distance between neighboring cones due to photoreceptor loss as a result of the disease process.4,10 However, these studies only evaluated relatively high-quality images with a clearly visible cone mosaic. In a poor-quality image with artifact and noise, the absolute number of cones identified may remain constant across imaging sessions, but the cone location distribution, and, thus, inter-cone spacing, often is nonuniform. As shown in Table 3, even though the average distance between cones may be similar between good and poor-quality images, the SD and CoV of cone spacing values can help to further determine image quality. Good-quality images with a clearly visualized cone mosaic have a significantly lower SD and CoV of cone spacing when compared to poor-quality images. We believe that SD and CoV of cone spacing are essentially surrogate measurements for uniform hexagonality, which likely correlate with higher image quality. Therefore, this may be another useful quantitative metric for determining image quality. To our knowledge, this is the first time that subjective image quality has been correlated to SD and CoV of cone spacing values.

Limitations/Future Work

Primary limitations to this study are the manner in which we selected patients and the relatively small number of patients imaged. Selective criteria were used to screen for patients with mild-to-moderate RP, so no cases of severe or advanced RP were included. However, there was no post-imaging exclusion of patients with lower quality images, so our findings portray an accurate representation of the recruited patient population. Future studies should investigate the feasibility and repeatability of flood-illuminated AO imaging in patients with advanced RP and other retinal diseases. Another methodologic limitation was that we used the preferred retinal locus as a surrogate for foveal center based on a prior study.16 However, an anatomic correlation with OCT would have been a more accurate method for determining the true foveal center. Additionally, further investigation into SD of cone spacing and CLS as metrics for determining image quality and differentiating healthy retina from areas of pathology is warranted. After adjustment for normal changes in cone density and spacing at various foveal eccentricities, it may be possible to discriminate between organized, healthy cone mosaics and the altered reflectivity patterns and spacing seen in diseased retina. This could also help to provide more information about regional changes throughout the retina and better delineate areas of pathologic progression. Furthermore, it may lead to the development of an objective algorithm for assessing image quality and automated cone identification reliability without the need for subjective assessment by an experienced grader. However, the CLS analysis technique is limited in that it requires repeated imaging at the same retinal location, which often is not feasible in the clinical setting.

Conclusions

We found that automated cone density repeatability metrics are similar for subjects with RP compared to normal subjects. However, our confidence in the validity of these cone counts depends on image quality, which is highly variable when imaging the diseased retina via flood-illuminated adaptive optics. While human graders can easily determine image quality via subjective assessment, there is not currently a reliable automated computer algorithm for judging AO image quality. We investigated two novel quantitative metrics as a way to objectively determine image quality: intersession CLS and CoV of nearest neighbor distance. We first selected subsets of subjectively good- and poor-quality images, then identified patterns of CLS and cone spacing that were associated with each image type. Future studies could investigate the ability of these metrics to perform image quality grading, and evaluate if these grades correlate with subjective human assessment.