Reliability and repeatability of cone density measurements in patients with congenital achromatopsia

Abstract

Adaptive optics scanning light ophthalmoscopy (AOSLO) allows non-invasive assessment of the cone photoreceptor mosaic. Confocal AOSLO imaging of patients with achromatopsia (ACHM) reveals an altered reflectivity of the remaining cone structure, making identification of the cells more challenging than in normal retinas. Recently, a “split-detector” AOSLO imaging method was shown to enable direct visualization of cone inner segments in patients with ACHM. Several studies have demonstrated gene replacement therapy effective in restoring cone function in animal models of ACHM and human trials are on the horizon, making the ability to reliably assess cone structure increasingly important. Here we sought to examine whether absolute estimates of cone density obtained from split-detector and confocal AOSLO images differed from one another and whether the inter- and intra-observer reliability is significantly different between these modes. These findings provide an important foundation for evaluating the role of these images as tools to assess the efficacy of future gene therapy trials.

XX.1 Introduction

AOSLO enables visualization of the cone photoreceptor mosaic in the living human eye (Dubra et al. 2011; Rossi et al. 2011). Quantitative measurements from such images include cone density (Chui et al. 2008), cone spacing (Duncan et al. 2007; Rossi and Roorda 2010; Cooper et al. 2013) and Voronoi geometry (Baraas et al. 2007). These methods typically rely on identification of individual cones in the image and thus whether this is done manually or via an automated (or semi-automated) process, there is a need to assess the inherent reliability and repeatability of each approach.

Previously we assessed the repeatability of cone density measurements in a population of young healthy individuals using a semi-automated method and found that if repeated images of the same retinal location were precisely aligned, the repeatability was 2.7% (Garrioch et al. 2012). Chiu et al. ( 2013) demonstrated similar repeatability using the same data set and a fully automatic algorithm based on graph theory and dynamic programming. Most recently, we examined the inter-observer and inter-instrument reliability of cone density measurements and found that the inter-observer study’s largest contribution to variability was the subject (95.72%) while the observer’s contribution was only 1.03% (Liu et al. 2014). For the inter-instrument study, we reported an average cone density ICC of between 0.931 and 0.975 (Liu et al. 2014).

These studies are only relevant for individuals with intact cone mosaics and do not apply in conditions such as ACHM, where cone appearance can be greatly altered (Genead et al. 2011; Merino et al. 2011). This makes it difficult to disambiguate cones from other reflective material in the outer retina. Scoles et al. (2014) developed a split-detector AOSLO method to directly visualize cone inner segments in a manner independent of the cone’s waveguide properties (from which the reflective confocal AOSLO signal arises) allowing for easier and more complete visualization of residual cone structure in patients with ACHM. In patients with ACHM, we sought to (1) assess whether estimates of cone density obtained with split-detector AOSLO images differed from those obtained from confocal AOSLO images and (2) determine whether the inter- and intra-observer reliability is significantly different between the two imaging modes. The findings presented here serve as a foundation for subsequent studies aimed at monitoring residual cone structure over time in patients with ACHM.

XX.2 Materials and Methods

XX.2.1 Subjects

All research followed the tenets of the Declaration of Helsinki and study protocols were approved by IRBs at the Medical College of Wisconsin and Moorfields Eye Hospital. Subjects provided written informed consent after the nature and possible consequences of the study were explained. Images from 7 subjects with molecularly confirmed ACHM (5 with CNGB3 mutations, 2 with CNGA3 mutations) were used in this study (5 males and 2 females, aged 11–64 years). Axial length measurements were obtained from all of the subjects using an IOL Master (Carl Zeiss Meditec, Dublin, CA) in order to calculate the lateral scale of each retinal image.

XX.2.2 AOSLO Imaging of the Photoreceptor Mosaic

Each patient’s head was stabilized using a dental impression on a bite bar and both eyes were dilated and cyclopleged using a combination of phenylephrine hydrochloride 2.5% and tropicamide 1%. Images of the photoreceptor mosaic were obtained using 790-nm light with two previously described AOSLOs that allow simultaneous acquisition of confocal and split-detector images as in Figure XX.1 (Scoles et al. 2014). Image sequences (100–200 frames) subtending either 1×1° or 1.5×1.5° were collected between the foveal center and 20° temporal to fixation. Each confocal image sequence was registered to produce a single image with improved signal-to-noise ratio (Dubra and Harvey 2010), with the same transforms applied to the corresponding split-detector image sequence, yielding a second image of the exact same retinal location. From these images, a total of 80 100×100μm areas were cropped for analysis.

Fig. XX.1.

Confocal (a,b) and split-detector (c,d) AOSLO images from two subjects with ACHM – JC_10069 (a,c) and MM_0005 (b,d).

XX.2.3 Analyzing the Cone Mosaic

The data set consisted of 960 images (80 images, 2 modalities, 3 observers, 2 trials/observer). Three observers with varying familiarity in analyzing AOSLO images reviewed each image and manually identified cones after adjusting the brightness and contrast of the image to assist in determining cone presence. Images were displayed in random order, with the identity and retinal location of the images masked (ensuring any effect of fatigue is captured by the observer’s variance component). The number of cones in the cropped 100×100μm region was divided by its area to derive an estimate of the cone density for that image.

XX.2.4 Statistical Methods

The sample size and other characteristics of this study were chosen using a Monte Carlo simulation with preliminary estimates of unknown quantities estimated on a pilot data set. The objective was to secure the half-width of the 90% CI for the relative contribution to the total variance, such that it is bounded by 1% for observer, trial and image and the half-width of the 90% CI for subjects’ relative contribution to the total variance is not higher than 2.5%. In this simulation study, 1000 repetitions were performed for a variance components model to assess the contribution of subject, mode, observer and trial to overall variability.

XX.3 Results

Highly significant biases prevented further analysis of the variance components model. Table XX.2 reports the fixed effects (regression coefficients) of the parsimonious linear mixed regression model, including both random and fixed effects for predicting cone density values on a natural logarithmic (LN) scale. As each image was assessed 12 times (2 modes, 3 observers, 2 trials) we needed to account for possible correlation between measurements. To do so, our model used three random effects: mode, observer and trial. In addition to the three random effects accounting for within image correlation, we investigated three fixed effects of mode, observer and trial, as well as the two-way interactions and the three-way interaction. Statistical significance was declared at 5%. We found that the three-way interaction between mode, observer and trial was not significant (P=0.194). The two-way interaction between mode and trial was also not significant (P=0.479). The interactions between observer and trial and between observer and mode were highly significant (P<0.0001).

Table XX.2.

Cone Density Measurements for Each Observer

-	Observer 1		Observer 2		Observer 3
Mode (Trial)	Estimate	Std. Err.	Estimate	Std. Err.	Estimate	Std. Err.
Confocal (1)	8.34	0.062	8.32	0.058	8.32	0.058
Confocal (2)	8.43	0.057	8.31	0.058	8.31	0.058
Split (1)	8.28	0.071	8.16	0.068	8.16	0.068
Split (2)	8.31	0.067	8.16	0.068	8.16	0.068

The linear mixed model used to build Table XX.2 absorbed information from all 960 observations. The presence of significant interactions with the observer prevents easily explaining the content of Table XX.1. To simplify the explanation of the regression modeling we fitted separate models for each observer, allowing us to interpret findings separately for each observer. Table XX.2 reports the estimates of mean LN(cone density) separately for each observer. Observer 1 had a significantly different interaction between trial and mode (P=0.006), precluding investigation of further interactions for this observer. The interactions between mode and trial (P=0.951) and the main effect of trial (P=0.447) were not significant for observer 2. Only the effect of mode was significant for observer 2 (P<0.0001). The interaction between mode and trial was not significant for observer 3 (P=0.632), nor was the main effect of mode (P=0.160). Surprisingly, we observed a strong effect of trial (P<0.0001).

Table XX.1.

Fixed Effects for All Observers

Variable	Estimate	Std.Err.	t
(Intercept)	8.35	0.060	138.53*
Mode=Split	−0.083	0.035	−2.34*
Trial=2	0.061	0.016	3.78*
Obs=2	−0.030	0.020	−1.47
Obs=3	−0.119	0.029	−4.04*
Trial=2:Obs=2	−0.068	0.022	−3.08*
Trial=2:Obs=3	0.035	0.022	1.59
Mode=Split:Obs=2	−0.074	0.022	−3.34*
Mode=Split:Obs=3	0.025	0.022	1.12

The model intercept corresponds to the expected LN(cone density) for the confocal mode, observer=1 and trial=1.

^*Statistically significant

This result indicates that the observers’ counts differ for different trials and modes, such that one observer may have different responses between modes and another observer may show no difference. Likewise, one observer may have different responses between trials with another observer showing no difference.

XX.4 Discussion

The results of the linear mixed regression model analysis demonstrated a strong effect of observer in cone counting in images from patients with ACHM using two different imaging modalities. This strong observer effect prevents further analysis of the reliability and repeatability of cone measurements in these retinas. Upon further analysis two of three observers showed a strong effect of trial (independent effect for one and interacting with mode for another), indicating that they were not able to consistently identify the same number of cells in the image set between two trials, with observer 1 showing an effect in the interaction between trial and mode. Observer 2, however, showed no effect of trial and the effect of mode indicates a difference between the confocal and split-detector measurements for this observer. Varying experience working with ACHM images (observer 2 had the most and observer 3 the least) may partially explain these results – thus analysis of diseased retinas may require a more experienced observer than analysis of cone structure in normal retinas. This result demonstrates the need for more experienced observers to analyze images of diseased retinas and development of automated methods for split-detector analysis.