Intraobserver Repeatability and Interobserver Reproducibility of Ellipsoid Zone Measurements in Retinitis Pigmentosa

Abstract

Purpose

To examine repeatability and reproducibility of ellipsoid zone (EZ) width measurements in patients with retinitis pigmentosa (RP) using a longitudinal reflectivity profile (LRP) analysis.

Methods

We examined Bioptigen optical coherence tomography (OCT) scans from 48 subjects with RP or Usher syndrome. Nominal scan lengths were 6, 7, or 10 mm, and the lateral scale of each scan was calculated using axial length measurements. LRPs were generated from OCT line scans, and the peak corresponding to EZ was manually identified using ImageJ. The locations at which the EZ peak disappeared were used to calculate EZ width. Each scan was analyzed twice by each of two observers, who were masked to their previous measurements and those of the other observer.

Results

On average, horizontal width (HW) was significantly greater than vertical width (VW), and there was high interocular symmetry for both HW and VW. We observed excellent intraobserver repeatability with intraclass correlation coefficients (ICCs) ranging from 0.996 to 0.998 for HW and VW measurements. Interobserver reproducibility was also excellent for both HW (ICC = 0.989; 95% confidence interval [CI] = 0.983–0.995) and VW (ICC = 0.991; 95% CI = 0.985–0.996), with no significant bias observed between observers.

Conclusions

EZ width can be measured using LRPs with excellent repeatability and reproducibility. Our observation of greater HW than VW is consistent with previous observations in RP, though the reason for this anisotropy remains unclear.

Translational Relevance

We describe repeatability and reproducibility of a method for measuring EZ width in patients with RP or Usher syndrome. This approach could facilitate measurement of retinal band thickness and/or intensity.

Introduction

Retinitis pigmentosa (RP) and its syndromic forms (e.g., Usher syndrome) are inherited retinal degenerations characterized by progressive loss of rod and cone photoreceptors. Peripheral vision is lost first, and damage approaches the fovea centripetally as the disease progresses, eventually impacting central vision.¹ Patients often experience night blindness and impaired dark adaptation in adolescence, followed by visual field constriction in young adulthood, commonly resulting in legal blindness by age 40.² RP is a genetically heterogenous condition, and clinical presentation and severity of the disease varies across different modes of inheritance. Autosomal recessive RP (arRP) is the most common form² and typically has earlier onset and more rapid progression than the autosomal dominant form (adRP).³ X-linked RP (xlRP) is the most severe form of the disease and starts earlier and progresses faster than arRP or adRP.⁴

Objective measurement of photoreceptor damage is critical to monitoring disease progression and outcomes in clinical trials.^2,5 Optical coherence tomography (OCT) enables direct visualization of retinal layers and assessment of retinal health. One of these layers, the ellipsoid zone (EZ) (also referred to as the IS/OS junction),^6,7 is a hyperreflective band in the outer retina that is used to assess the structural integrity of photoreceptors.⁸ In fact, it has been shown that as cone photoreceptors degenerate the EZ band diminishes in intensity.⁹ The width of the EZ band has been used to monitor the progression of RP¹⁰ as changes in EZ width over time reflect disease advancement and have been used as an anatomical correlate of the visual field.^11–13 Examination of the EZ has also proven useful in studying the natural history of other retinal disorders including Stargardt disease,^14,15 achromatopsia,¹⁶ branch retinal vein occlusion,¹⁷ retinopathy of prematurity,¹⁸ macular telangiectasia type 2,¹⁹ choroideremia,^14,20 blue-cone monochromacy,²¹ cone-rod dystrophy,²¹ age-related macular degeneration,²² drug toxicity,²² and postoperative changes following macular surgery.²²

Several methods for measuring EZ width have been described, such as manual identification of the EZ band boundary,^4,23,24 delineation of the EZ area from an en face projection of the layer,²⁵ and segmentation of the EZ band layer with the EZ boundary defined as the location where outer segment thickness decreases to zero.^{4,12,13,24,26} The accuracy of segmentation algorithms can be affected by irregularities in layer contour and/or band intensity. Indeed, segmentation errors are significantly increased in pathologic eyes when compared to normal eyes.²⁷ Additionally, segmentation-based approaches can imply differences in layer thickness when hyperreflective bands attenuate as part of the disease process.²⁸ Here, we used longitudinal reflectivity profile (LRP) based analysis for quantification of EZ width. Here we sought to assess the repeatability and reproducibility of this method for measuring the width of retained EZ band in patients with RP.

Methods

Subjects

All research methods followed the tenets of the Declaration of Helsinki. The study protocol was approved by the Institutional Review Board at the Medical College of Wisconsin (PRO17439 and PRO30741). Subjects provided informed written consent after the nature and possible consequences of the study were explained. Subjects with clinically diagnosed RP or Usher syndrome were eligible for inclusion. We retrospectively examined horizontal and vertical Bioptigen SD-OCT line scans acquired through the foveal center (Bioptigen, Research Triangle Park, NC) from a total of 73 subjects. Scans with inferior image quality (due to corneal defects, media opacity, high refractive errors [greater than ±10 diopters], and/or significant macular edema) were excluded. Likewise, subjects were excluded for whom their disease was advanced to a point at which no EZ was clearly discernable in their OCT images. In addition, horizontal scans in which the EZ band did not terminate prior to the nasal and/or temporal scan boundary and vertical scans in which the EZ band did not terminate prior to the inferior and/or superior scan boundary were excluded due to inability to accurately assess EZ band width. Scans from 48 subjects (22 males, 26 females; mean ± SD age = 41.6 ± 18.6 years) remained for analysis, with 23 of the 48 subjects having scans available from both eyes (total of 71 eyes). Thirty-eight subjects (55 eyes) had RP, and 10 subjects (16 eyes) had Usher syndrome. The diagnosis of RP or Usher syndrome was based on inheritance pattern, clinical symptoms, and/or genotype. A summary of subject demographics is provided in Supplementary Table S1.

SD-OCT Imaging

Scans were acquired between March 2011 and November 2015 at the Medical College of Wisconsin. Nominal scan lengths were 6, 7, or 10 mm, and each line scan comprised of 1000 A-scans/B-scan and between 80 to 120 repeated B-scans. Horizontal and vertical line scans were acquired for each imaging session. OCT line scans of both eyes were available for 23 of 48 subjects, resulting in a total of 144 line scans. The B-scans for a given line scan were registered and averaged to remove speckle noise as previously described,²⁹ resulting in a .tif image for each line scan. Depending on eye motion and image quality, each .tif scan was an average of between 6 and 86 individual B-scans. The lateral scale of each .tif image was calculated by correcting the nominal scan length for the ratio between the assumed axial length of the OCT system (24 mm) and the actual axial length measurement for that eye (Zeiss IOL Master; Carl Zeiss Meditec, Dublin, CA).

EZ Width Measurement

Each averaged .tif image was analyzed as follows. LRPs were generated from OCT line scans as previously described,³⁰ and peaks corresponding to the external limiting membrane (ELM), EZ, interdigitation zone (IZ), and retinal pigment epithelium (RPE) were manually identified in ImageJ.³¹ Horizontal width (HW) and vertical width (VW) were calculated using the boundaries of the EZ band, defined as the locations at which the EZ peak disappeared nasally/temporally for horizontal scans and superiorly/inferiorly for vertical scans (Fig. 1; see Supplementary Video S1). For the 23 subjects for whom scans of both eyes were available, we had a single observer (M.R.S.) measure the 46 registered .tif images (23 horizontal, 23 vertical) a single time to confirm interocular symmetry. For repeatability and reproducibility analyses, one eye was chosen at random for subjects for who had images from both eyes, while images from whichever eye was available were used for the remaining 25 subjects. In these repeatability and reproducibility analyses, the EZ width was measured twice per image (96 total images; 48 horizontal, 48 vertical) by each of two observers (M.R.S. and A.L.H.), with each observer masked to their previous measurements as well as those of the other observer. The repeated measurements within each observer were separated by 1 week. The observers had different levels of experience in working with OCT images, one being a relative novice (M.R.S.) and the other being more experienced (A.L.H.).

Figure 1.

Horizontal SD-OCT line scan with superimposed LRPs from the right eye of a 50-year-old female (IE_0508) with ADRP (RHO; p.Pro23His). Numbered retinal layers correspond to the (1) ELM, (2) EZ, (3) IZ, and (4) RPE. Outer LRPs correspond to nasal (right) and temporal (left) boundaries of the EZ band. Scale bars = 100 μm.

Statistics

Unless otherwise noted, all statistical tests (including the Bland-Altman analyses) were performed using Prism version 7.0c (GraphPad, LaJolla, CA). The bias, limits of agreement (LOA), and 95% confidence intervals (CIs) for the bias and LOA were calculated following the methods of Bland and Altman.^32–34 For all data sets, normality was assessed using the Shapiro-Wilk normality test. Where normality could not be confirmed, nonparametric tests were used. The specific tests used are included alongside each result, as appropriate. Intraclass correlation coefficients (ICCs) were calculated for log-transformed HW and VW measurements using R statistical software (Foundation for Statistical Computing, Vienna, Austria). Variance components models fitted separately for HW and VW data were used to evaluate the contributions of subject, observer and reading within observer (trial) to the total variance of the measurements (SAS version 9.4; SAS, Cary, NC).

Results

Interocular Symmetry

OCT line scans of both eyes were available for 23 of 48 subjects. Right and left eye EZ width measurements were highly correlated for both HW (Spearman's rank correlation coefficient = 0.915 [95% CI = 0.803–0.965]) and VW (Spearman's rank correlation coefficient = 0.887 [95% CI = 0.743–0.953]) measurements (Fig. 2). Using a Wilcoxon matched-pairs signed rank test, we found no difference between mean OD and OS EZ width measurements for both HW (P = 0.2479) and VW (P = 0.6010). These data imply a high degree of interocular symmetry of the retained EZ in patients with RP and Usher syndrome.

Figure 2.

Interocular comparison of EZ width illustrating symmetry between EZ width measurements of left (OS) and right eyes (OD). HW measurements are represented by gray open circles, while VW measurements are represented by crosses. The trend line for all EZ width measurements has a slope of 0.907 (solid line), and the dashed lines show the 95% CIs for the linear regression. No significant difference was observed between OD and OS measurements (see text).

Intraobserver Repeatability

For each observer, the HW measurements were significantly greater than VW measurements (observer 1 = P < 0.0001, observer 2 = P < 0.0001; Wilcoxon matched-pairs signed rank test). The median and interquartile ranges are provided in Table 1. An example of this anisotropy is shown in Figure 3. HW and VW measurements were highly correlated (Spearman's rank coefficient observer 1 = 0.961 [95% CI = 0.929–0.9978], Spearman's rank coefficient observer 2 = 0.981 [95% CI = 0.967–0.990]). Given the difference in magnitude and variability of HW and VW, we assessed the HW and VW data separately.

Table 1.

Results and Intraobserver Repeatability of EZ Width Measurements

Figure 3.

Anisotropy in retained EZ area. Shown are horizontal (A) and vertical (B) SD-OCT line scans from the left eye of a 44-year-old female (TC_1176) with ADRP (RP1; p.R677X). Lines representing the location of each scan are superimposed on the corresponding autofluorescence image (C), which shows a horizontally elongated elliptical ring of hyperautofluorescence. The HW measurement was 1543 μm or 39% (observer 1) and 1479 μm or 37% (observer 2) greater than the VW measurement. Arrows represent the EZ boundaries identified by observer 1 (black arrows) and observer 2 (white arrows). OCT scale bars = 100 μm.

The test–retest difference was calculated from the absolute value of the differences between measurements for each observer, for both horizontal and vertical measurements (Fig. 4). The median test–retest difference was between 11 and 42 μm, though there was significant variability. Shown in Figure 5 are exemplar images illustrating variable intraobserver repeatability. As can be seen in these images, areas of discordance often occurred in proximity to blood vessel shadows projecting through the outer hyperreflective bands. As each of the eight sets of measurements (two scans, two observers, two measurements per observer) failed our normality test, the data were log transformed for the following repeatability analyses. Excellent intraobserver repeatability was observed for both HW and VW measurements, as seen by the ICC values provided in Table 1 and the Bland-Altman plots in Figure 6. Back-transforming the results of the Bland-Altman analysis provides values that relate to the ratio of the measurements from the two trials for that given observer and scan set (horizontal or vertical).³³ This ratio ranged from 0.9957 to 1.0002, meaning that the bias between measurements ranged from −0.43% to 0.02%. As shown in Figure 6, no proportional bias was observed in any of the plots, and the scatter of differences was homoscedastic as a function of the mean. Individual results, along with LOA and CIs (expressed as percentages) are provided in Table 1.

Figure 4.

Test–retest differences in EZ width measurements. Shown are the median (dashed horizontal lines) for each observers HW and VW measurements. The 25th and 75th quartiles are represented by the rectangles, while the error bars extend to the minimum and maximum values for each data set. Note that the y-axis extends below 0 for clarity, as a minimum value of 0 was observed in all four data sets.

Figure 5.

Variable repeatability of EZ width measurements. Shown are four SD-OCT line scans (A, D: horizontal; B, C: vertical) representing the range of differences observed (0–491 μm). The EZ boundaries from measurement 1 are represented by white arrows, while those from measurement 2 are represented by black arrows; in cases where there was no difference between the EZ boundaries, a single white arrow with a black border is shown. Panel (A) is from a 28-year-old female (KS_10084) with Usher type II (USH2A; p.C3281F, p.C1195F, p.T5006M), panels (B, D) are from a 19-year-old female (KS_10243) with ADRP (RP1; p.R677X), and panel (C) is from a 55-year-old female (JC_1088) with Usher type III (GPR98; p.C5921R, p.L5999P). The absolute difference in EZ width is provided for each scan. Scale bars = 200 μm.

Figure 6.

Bland-Altman plots illustrating the intraobserver repeatability of EZ width measurements for (A, B) HW and (C, D) VW. HW values are represented by circles, and VW values are represented by crosses. The mean log EZ width difference (bias) is represented by the solid black line, while the dashed lines represent the 95% LOA for the bias. Shaded regions represent the confidence limits on the bias and LOA (see Methods). To assess whether repeatability was dependent on the magnitude of EZ width, we calculated the Pearson correlation coefficient (r) for each data set. (A) r = 0.117, 95% CI = −0.173 to 0.388, P = 0.43; (B) r = 0.003, 95% CI = −0.281 to 0.287, P = 0.98; (C) r = −0.032, 95% CI = −0.314 to 0.254, P = 0.83; (D) r = 0.100, 95% CI = −0.189 to 0.374, P = 0.49. These values indicate that there is no significant proportional bias. All four data sets passed the test for homoscedasticity, with P > 0.05.

Interobserver Reproducibility

To assess interobserver reproducibility, we averaged the two trials within each observer. Again each of the four sets of measurements (two scans, two observers, one averaged measurement per observer) failed our normality test, so the data were log transformed for this analysis. There was a high interobserver agreement between observers 1 and 2 for the HW (ICC = 0.989; 95% CI = 0.983−0.995) and VW (ICC = 0.991; 95% CI = 0.985−0.996) measurements. As with the intraobserver data, back-transforming the results of the interobserver Bland-Altman analysis shown in Figure 7 provides values that relate to the ratio of the measurements from the two observers for that given scan set (horizontal or vertical).³³ For HW, this ratio was 1.01, meaning that for most measurements, observer 1 exceeded observer 2 by 1.007%. However, the upper and lower LOA put the agreement between −7.07% and 9.80%, suggesting no consistent bias between the observers. For VW, this ratio was 1.002, meaning that for most measurements, observer 1 exceeded observer 2 by 0.19%. Here, the upper and lower LOA put the agreement between −7.357% and 8.35%, again suggesting no consistent bias between the observers. As with the intraobserver data no proportional bias was observed in these plots, and the scatter of differences was homoscedastic as a function of the mean (Fig. 7).

Figure 7.

Bland-Altman plots illustrating the interobserver reproducibility of EZ width measurements for (A) HW and (B) VW. HW values are represented by circles, and VW values are represented by crosses. The mean log EZ width difference (bias) is represented by the solid black line, while the dashed lines represent the 95% LOA for the bias. Shaded regions represent the confidence limits on the bias and LOA (see Methods). To assess whether repeatability was dependent on the magnitude of EZ width, we calculated the Pearson correlation coefficient (r) for each data set. (A) r = 0.156, 95% CI = −0.134 to 0.422, P = 0.29; (B) r = 0.178, 95% CI = −0.112 to 0.440, P = 0.23. These values indicate that there is no significant proportional bias. Both data sets passed the test for homoscedasticity, with P > 0.05.

A final way to assess the variability in our EZ width data is through the analysis of variance components, which might be thought of as a generalization of ICC analysis. Table 2 provides a summary of the magnitude of variances as well as their percent values of the total variance. These data show that the variance associated with observers and readings within observer (trials) are extremely small compared to the variance associated with subjects. This suggests this method can be used to reliably measure EZ width in subjects with RP or Usher syndrome.

Table 2.

Variance Components Modeling Results

Discussion

In this study, we determined the repeatability of measurements of EZ width obtained using an LRP-based analysis in patients with RP and Usher syndrome. We observed excellent intraobserver repeatability and interobserver reproducibility. Our test–retest differences were generally comparable to those in previous studies, which have reported mean test–retest differences ranging from 10 to 110 μm when comparing first and second measurements of EZ width in RP.^{4,12,24–26,35} Across both observer, the average test–retest difference was 52.6 μm, with 22% of measurements having a test–retest difference of 0 μm. It should be noted, however, that our study relied on comparison of different measurements taken of the same scan by the same observer, whereas prior studies compared measurements from two different scans obtained at closely spaced visits. As such, we might expect slightly better agreement in our measurements. We observed the worst repeatability in images with low signal-to-noise ratio, extensive vessel shadowing in the parafoveal region, and blurring between outer retinal bands. Importantly, while both observers had excellent repeatability (as demonstrated by the ICC analysis), observer 1 (relative novice) had a number of test–retest differences above 200 μm, whereas observer 2 (more experienced) had no such disparate measures. This observation suggests that (1) observer training is critical with these methods and (2) development of automated and/or objective measures of EZ integrity will likely be needed.

As is seen with other OCT-based measures of retinal structure (e.g., foveal pit morphology, retinal thickness), we observed high interocular symmetry for the EZ width measurements. This is consistent with previously reported functional symmetry in patients with RP.³⁶ Strong structural interocular symmetry was also seen in a group of 32 subjects with RPGR-associated RP, though some subjects showed interocular differences in EZ width as great as 51%.³⁷ For the 23 subjects for whom we had right and left eye data, the largest difference in EZ width we observed was 27%, though only two of these subjects had RPGR-associated RP. Recently, Sujirakul et al.³⁵ reported asymmetrical structural progression between right and left eyes in 19% of their patients. It would be interesting to monitor the progression rate in the few subjects in our cohort who displayed moderate interocular asymmetry.

Average HW was greater than VW, which is consistent with previous findings.^4,35 The reason for this anisotropy is unclear; however, previous studies have observed higher rod and cone packing density along the horizontal meridian.^38,39 Whether this bias impacts disease progression remains to be seen; however, previous studies have not reported any significant differences in the rate of progression along the vertical and horizontal meridians.^4,24,35 As has been seen in measurement of the foveal avascular zone (FAZ) using OCT angiography (OCTA),⁴⁰ estimates of EZ area may be inaccurate due to the anisotropy of the region. Hariri et al.²⁵ have developed a method for measuring EZ area by segmenting its en face projection using SD-OCT, but even that may be insensitive to small changes in progression that might occur preferentially along a specific meridian. That said, Ho et al. examined EZ area in RP and reported a correlation between EZ area and the hill of vision (Ho A, et al. IOVS. 2014;55:ARVO E-Abstract 3380). Thus the importance of EZ area as a possible biomarker seems clear. Exploration of metrics currently being used to assess the FAZ in OCTA images such as acircularity and axis ratio⁴¹ may be helpful in tracking the EZ area with high sensitivity.

Limitations of our study include the fact that we utilized OCT scans from a single device. It remains to be seen whether the automated averaging techniques employed on many clinical devices permits the same examination of LRP features utilized here. In addition, we were limited to a retrospective data set, which introduced a couple of limitations. First, the scan size was variable, and this contributed to a number of scans being unanalyzable. In an extreme case, use of a scan size of 3 mm would obviate measurement of EZ width in nearly all of our subjects. While 6 mm was our smallest scan size, had we used a 10 or 12 mm scan in all subjects we likely could have included more subjects for analysis, though the lateral resolution of the resultant images would have been lower (possibly compromising the LRP analysis). A second limitation introduced by the retrospective analysis is that we could not control for disease stage (either due to different ages or different mutation subtypes), thus subanalysis of any relationship between EZ width and age or genotype was not possible. Finally, we restricted this analysis to images of sufficient quality to examine the EZ. In a clinical environment, image quality may be more variable; thus, our estimates of repeatability are likely a best-case scenario. In the future, it would be worth examining the utility of LRP-based analyses in more “realistic” data sets.

In conclusion, measuring EZ width in patients with RP and Usher syndrome using LRPs derived from freely available software is a method with excellent intraobserver repeatability and interobserver reproducibility. One advantage of this method over manual marking of the EZ band boundaries is that an LRP can also be used to measure the thickness^42–44 and intensity^45,46 of the EZ and other hyperreflective bands. Further improvements could be made by automating the process of LRP generation or objective peak identification using OCT Reflectivity Analytics software.⁴⁷ Our group has taken an interest in elucidating the repeatability of image analysis tools in other ocular imaging modalities,^48,49 and the ophthalmic imaging community has also deemed it important to accurately interpret and compare results from different studies. It is important to note that these findings should not be extrapolated to other retinal diseases that may show qualitatively similar transition zones on OCT such as choroideremia²⁰ or Stargardt disease,¹⁴ and separate repeatability and reliability studies are likely needed for those specific patient populations.