Automated segmentation errors when using optical coherence tomography to measure retinal nerve fiber layer thickness in glaucoma

Abstract

Purpose

To characterize the error of OCT measurements of retinal nerve fiber layer (RNFL) thickness when using automated retinal layer segmentation algorithms without manual refinement.

Design

cross-sectional study

Methods

Setting

glaucoma clinical practice.

Study Population

3490 scans from 412 eyes of 213 individuals with a diagnosis of glaucoma or glaucoma suspect.

Observational Procedures

We used spectral domain OCT (Spectralis, Heidelberg Engineering, Heidelberg, Germany) to measure RNFL thickness in a 6-degree peripapillary circle, and exported the native ‘automated segmentation only’ results. In addition, we exported the results after ‘manual refinement’ to correct errors in the automated segmentation of the anterior (internal limiting membrane) and the posterior boundary of the RNFL.

Main Outcome Measures

Differences in RNFL thickness and glaucoma classification (normal, borderline, or outside normal limits) between scans with ‘automated segmentation only’ to scans using ‘manual refinement’.

Results

‘Automated segmentation only’ resulted in a thinner global RNFL thickness (1.6 μm thinner, P<0.001) when compared to ‘manual refinement’. When adjusted by operator, a multivariate model showed increased differences with decreasing RNFL thickness (P<0.001), decreasing scan quality (P<0.001), and increasing age (P<0.03). Manual refinement changed 298/3486 (8.5%) of scans to a different global glaucoma classification wherein 146/617 (23.7%) of borderline classifications became normal. Superior and inferior temporal clock hours had the largest differences.

Conclusions

Automated segmentation without manual refinement resulted in reduced global RNFL thickness and overestimated the classification of glaucoma. Differences increased in eyes with a thinner RNFL thickness, older age, and decreased scan quality. Operators should inspect and manually refine OCT retinal layer segmentation when assessing RNFL thickness in the management of patients with glaucoma.

Introduction

Clinicians and researchers use optical coherence tomography (OCT) to measure peripapillary retinal nerve fiber layer (RNFL) thickness and use this information to discriminate between healthy patients and those with glaucomatous damage, as well as to monitor for structural progression related to glaucoma.^1,2 OCT software determines the RNFL thickness by automatically segmenting the acquired images into distinct retinal layers.^3,4

However, automated image segmentation algorithms can fail to accurately delineate the layers of the retina and 19.9% to 46.3% of scans contain at least one segmentation artifact.^5–9 These artifacts have been associated with several findings, such as image decentration, epiretinal membranes, long axial length, poor visual acuity, cataract, and advanced glaucoma. While artifacts are common, many OCT operators do not routinely inspect the acquired scans or perform manual refinement of layer segmentations, which may lead to errors in the diagnosis of glaucoma.¹⁰

We were interested in the magnitude, associations, and locations of errors associated with automated segmentation of the RNFL. We studied this by comparing RNFL thickness resulting from automated segmentation to RNFL thickness when operators manually refined the RNFL boundaries to correct errors in their location. Clinicians and researchers will use this information to reduce the likelihood of artifact due to errors of automated segmentation algorithms and understand the implications of manually refining retinal layer segmentations in circumpapillary scans for glaucoma management.

Methods

We included data from participants enrolled in the ongoing Portland Progression Project^11–14 at Legacy Devers Eye Institute, Portland, OR. The Legacy Health Institutional Review Board approved this study. All participants gave their consent after they were informed about the risks and benefits of their participation. The study adhered to the tenets of the Declaration of Helsinki.

The Portland Progression Project includes participants with a diagnosis of glaucoma suspect or open angle glaucoma. All participants had a full ophthalmic examination including measurement of visual acuity, slit lamp evaluation of the anterior segment, manifest refraction, gonioscopy, pachymetry, and evaluation of the fundus and optic disc. At enrollment, participants had optic discs suspicious for glaucoma (large cup to disc ratio, cup-to-disc asymmetry ≥ 0.2 between eyes, history of disc hemorrhage, rim notching, nerve fiber layer thinning or defect); ocular hypertension (untreated IOP ≥ 22 mmHg on at least two occasions); and at least one additional risk factor for glaucoma (e.g. first-degree family history of POAG). We excluded participants with visual acuity worse than 20/40 and/or other ocular diseases that could cause changes in visual acuity or visual fields.^11–14

Trained operators used a spectral domain OCT (Spectralis, 870-nm center wavelength, diode laser, Heidelberg Engineering, Heidelberg, Germany) and the circle scan imaging pattern to measure peripapillary RNFL thickness in both eyes of each individual. The final circle scan image was the average of 9 sweeps collected with active image stabilization and manually centered around the optic disc. Each circle scan included 1536 A-scans at a radius of 6-degree from the optic disc center. The operator focused images to optimize the clarity of blood vessels within the RNFL. We exported the native retinal layer automated segmentation results and the OCT global glaucoma classification of each scan [normal, borderline (P<0.05 of normative limits), or outside normal limits (P<0.01 of normative limits)] for the automated segmentation.

Subsequently, the operators manually refined the instrument’s automated segmentation results using the Heidelberg Eye Explorer Software tool (Software version 5.6.1.0, Heidelberg Engineering, Heidelberg, Germany) to achieve an accurate delineation of the anterior RNFL boundary (between the vitreous and the internal limiting membrane) and the posterior boundary of the RNFL¹⁵ (between the RNFL and the ganglion cell layer) when there were obvious errors in the automated results.^12,16 These manually refined layer segmentation results were also exported. The OCT glaucoma classification for each scan (normal, borderline, or abnormal) was also exported for all scans after manual refinement. Figures 1 and 2 show two examples of automated segmentation algorithm failure, and the images after manual refinement for the anterior and posterior boundaries of the RNFL.

Figure 1.

Top panel is a section of raw images from the optical coherence tomographic circle scan. Middle panel shows machine delineation of the retina layers that overestimates (white arrow) and underestimates (yellow arrow) the retinal nerve fiber layer (RNFL) thickness. Bottom panel shows the RNFL thickness after manual refinement of the automated segmentation. Green line: Inner Limiting Membrane. Red line: Posterior RNFL boundary.

Figure 2.

Panel A shows the circle scan intercepting a retinal vessel for approximately one clock hour creating a shadow and segmentation artifact shown in Panel B. Panel C shows the manual delineation of the nerve fiber layer thickness.

Data Analysis

We excluded scans with an OCT scan quality score less than 15 as suggested by the manufacturer¹⁵, or when vitreous traction or epiretinal membranes distorted the RNFL to such an extent that defining an accurate RNFL thickness was impossible. We used a custom program developed in R (Available at: www.R-project.org, version 3.2.1) to generate from the 1536 thickness samples of each circumpapillary B-scan, a global average thickness value and an average for each of 12 (30-degree) clock hour sectors. We converted left eye sectors to the corresponding right eye configuration.

We used the difference between RNFL thickness using ‘manual refinement’ to scans using ‘automated segmentation only’ as an estimate of error (i.e. as when a clinic does not use manual refinement). We also compared the OCT glaucoma classification (normal, borderline, or outside normal limits) between the results of each scan based on ‘automated segmentation only’ to the results after ‘manual refinement’.

This cross-sectional analysis included all OCT scans of both eyes of participants who were being tested longitudinally. We used a linear, mixed effects model^17,18 with eye nested within subject as a random effect to account for correlations between the two eyes of an individual and multiple tests over time. This model used the difference in RNFL thickness (automated minus manually refined data) as the dependent variable and compared the univariate and multivariate associations of average RNFL thickness, OCT scan quality score, age, mean deviation during visual field testing on the same day as OCT testing (Humphrey SITA standard algorithm, Size III stimulus, 24-2 test pattern, Carl Zeiss Meditec, Dublin, CA), operator, and central corneal thickness. We performed all analyses using the R statistical program and used the ‘Ime’ function within the nIme R library (version 3.1–122) for linear mixed effects models.

Results

This study included 3490 scans from 412 eyes (209 right and 203 left eyes) of 213 individuals. Four scans (0.1%) did not include complete measurements in clock hour 9 (temporal sector) and were not included in the evaluation of this sector or in the changes in glaucoma classification. Table 1 describes the participants. They were mostly white, 58% female, early glaucoma by mean deviation of their visual field, and a mean (+/− SD) age of 66.7 +/− 10.9 years.

Table 1.

Demographics at last exam. Data are presented in mean (standard deviation) unless otherwise specified.

	Overall [n=213 participants (n=209 right eyes, 203 left eyes)]
Age, years	64.5 (11.2)
Gender, % Female	58.2
Ethnicitya, %
White	93.0
AI/AN	1.9
African American	2.8
Hispanic/Latino	0.9
Asian	1.4
Mean Deviation right eyes, dB	−0.86 (2.8)
Mean Deviation left eyes, dB	−1.51 (3.5)
Central corneal thickness right eyes, microns	560.1 (38.5)
Central corneal thickness, left eyes, microns	559.6 (37.3)

^aEthnicity by self-report

Global differences in retinal nerve fiber layer thickness analysis

Automated segmentation resulted in a thinner global RNFL thickness, on average (1.6 μm thinner, 1^st /3^rd quartile 0.2/3.1 microns, P <0.001) when compared to manual refinement. Univariate linear mixed effects models showed increased differences (error) with decreasing RNFL thickness (microns, beta=.11, P <.001), decreasing OCT scan quality score (beta=.11, P <.001), decreasing visual field mean deviation during visual field testing (dB, beta=.18, P <.001), and increasing age (years, beta=−.03, P =.002). Operator was associated with differences in retinal nerve fiber layer thickness (p<.001, mean difference in μm, operator 1: 2.47; operator 2: 2.34; operator 3: 0.78; and operator 4: 1.76). Central corneal thickness (microns, beta=.01, P =.19) was not associated with differences (error).

A multivariate model adjusted by operator showed increased differences with decreasing RNFL thickness (P <.001), decreasing scan quality (P <.001), and increasing age (P =.03), but visual field mean deviation was no longer significant (P =.15), most likely because it was correlated (r = −.369, P <.001) with RNFL thickness. Overall, this suggests that OCT automated segmentation resulted in a thinner RNFL measurement when compared to manual refinement, and errors with OCT increased with decreasing RNFL thickness, decreasing scan quality, and increasing age.

30-degree sectoral (clock hour) analysis

We examined each clock hour sector using a similar univariate and multivariate analysis as above and found similar results (results not shown). Figures 3 and 4 are notched box plots depicting the mean difference between automated and manually refined RNFL thickness in each 30-degree sector (clock hours) using a TSNIT (temporal to superior to nasal to inferior to temporal) orientation for a right eye. Figure 3 shows that the largest differences between automated and manual segmentation occurred in the superior temporal (11 and 12 o’clock sectors) and inferior temporal (6 and 7 o’clock) sectors. Figure 4 shows the magnitude of error that may be as high as 150 microns and errors occur in all sectors.

Figure 3.

This is a notched box plot with global difference (manual refinement minus automated segmentation) and twelve 30-degree sectors (clock hours) in a right eye configuration, oriented by TSNIT (temporal to superior to nasal to inferior to temporal) location. If the notches of two plots do not overlap then the medians are significantly different at the 5 percent level. It excludes outliers to demonstrate the difference in errors between sectors.

Figure 4.

This is a notched box plot with global difference (manual refinement minus automated segmentation) and twelve 30-degree sectors (clock hours) in a right eye configuration, oriented by TSNIT (temporal to superior to nasal to inferior to temporal) location. It is similar to Figure 3 but displays outliers to demonstrate the magnitude of errors within sectors.

A multivariate mixed effects model with backward elimination by t-value showed that two temporal sectors (8 and 9 o’clock, P>.05 for each) had the least association with differences (error). Overall, this suggests that clock hour sectors located superiorly and inferiorly may require manual refinement more often while temporal sectors may have decreased need for manual refinement.

Glaucoma Classification Analysis

Table 2 shows that manual refinement changed 298/3486 (8.5%) of scans to a different global glaucoma classification; 268/3486 (7.7%) had a less severe classification of glaucoma (borderline to normal, abnormal to borderline, or abnormal to normal) after manual refinement when compared to automated segmentation. Only 30/3486 (0.8%) of scans had a more severe Spectralis glaucoma classification after manual refinement. However, 146/617 (23.7%) of borderline classifications with automated segmentation became normal after manual refinement. Overall, this demonstrates that automated segmentation without manual refinement generally overestimated glaucoma severity.

Table 2.

Contingency table of glaucoma classification based on spectral domain OCT (Spectralis, 870-nm, software version 5.6.1.0, Heidelberg Engineering, Heidelberg, Germany), n=3486 scans. Data are presented in number (% of total).

Manually Refined	Outside Normal Limits	Borderline	Within Normal Limits
Automated Segmentation
Outside Normal Limits	879 (25.2%)	115 (3.3%)	7 (0.2%)
Borderline	14 (0.4%)	457 (13.1%)	146 (4.2%)
Within Normal Limits	0 (0%)	16 (0.5%)	1852 (53.1%)

Discussion

OCT provides objective and repeatable measurements of optic nerve head structure and RNFL thickness.^19–22 However, variability in RNFL thickness measurements and failure of RNFL segmentation algorithms can lead to over- or underestimations of RNFL thickness.¹⁰ We were interested in the magnitude, associations, and locations of errors associated with automated segmentation of the RNFL when operators do not use manual refinement. In this study, we used manual refinement of the automated RNFL segmentation to correct errors in automated RNFL segmentation to better reflect the anatomy represented in each scan. OCT automated segmentation resulted in thinner RNFL measurements when compared to manual refinement and differences (error) increased with decreasing RNFL thickness, decreasing scan quality, and increasing age. We also found that the superior and inferior temporal quadrants had larger differences while the temporal sectors had the least differences. Finally, automated segmentation overestimated global classification of glaucoma and glaucoma suspect. Clinicians and researchers should use this information to be aware of circumstances under which, and locations where, automated segmentation errors are most likely to occur as well as to understand the importance of manually refining retinal layer segmentations.

The effect of manual refinement of RNFL segmentation was important in eyes with a thinner RNFL. Hwang and colleagues show that accuracy of the automated OCT segmentation algorithm of the macular ganglion cell inner plexiform layer is most accurate in eyes with less structural distortion of this layer.²³ Glaucomatous pathology can thin the RNFL and reduce reflectivity (attenuation) of RNFL axons, which can distort and decrease contrast of the RNFL boundaries that automated segmentation algorithms (and human operators) rely upon.^24–26 Our results are consistent with the suggestion that glaucomatous pathology increases the likelihood of automated segmentation errors and an increased need for inspection and manual refinement. Correct assessment of the RNFL in glaucoma patients may improve our ability to monitor for progressive RNFL thinning.

Poorer OCT scan quality score was associated with larger errors. This observation is supported by other studies that show that eyes with lower scan quality have higher rates of segmentation errors.^{20,21,27–29} However, poor scan quality does not always correlate with poor image quality^5,28; and an operator may still be able to adequately visualize and manually refine the RNFL boundaries. We also recognize that our study may underestimate the effect of poor scan quality since we excluded scans with very poor scan quality (<15) and utilized sweep averaging to improve signal-to-noise ratio (scan quality) by a factor of approximately 3x by reducing speckle noise in each final B-scan.

Our study found older age to be associated with a greater difference between manual and automated segmentation results. This may be related to RNFL thickness decreasing with age,^30,31 and segmentation errors may occur with decrease RNFL thickness as explained in the above paragraph. Aging is also associated with media opacities (i.e. cataracts), epiretinal membranes, peripapillary atrophy, and other common abnormalities. Chong and colleagues¹⁰ demonstrated that these abnormalities may lead to misclassification of eyes as glaucomatous i.e. ‘red disease’ in eyes that do not have glaucoma as defined by other structural measurements or visual field testing. Our study did not determine the physiologic etiology (e.g. cataract, peripapillary atrophy, etc) of the segmentation errors, which would be an important investigation in the future.

We found that OCT automated segmentation resulted in a thinner RNFL thickness when compared to manual refinement. Global glaucoma classifications changed in 7.7% of eyes, and interestingly, 23.7% of borderline classifications with automated segmentation became normal after manual refinement. Overall, this suggests that the failure to inspect and manually refine automated segmentations may overestimate the proportion of eyes with a glaucoma classification. Overall, researchers and clinicians may overestimate glaucoma and clinicians may over treat patients if they do not have access to the retinal layer segmentation and do not inspect and manually refine these segmentations for errors.

The superior and inferior temporal sectors may have exhibited the largest errors because these sectors are most commonly where the major retinal vessels cross the circumpapillary scan location. These vessels create shadows due to signal attenuation, thus making it more challenging to delineate the boundaries of the RNFL. Figure 2 demonstrates this shadowing and resultant segmentation artifact. Moreover, the major retinal vessels can become more prominent and appear to ‘emerge’ from the RNFL as axons are lost, further complicating segmentation in these vessel-dense sectors.²⁶ Manual segmentation of the RNFL can decrease errors associated with vessel artifact although, it is possible that they may persist to a lesser degree. In addition, the superior and inferior areas of the retinal nerve fiber layer may also be areas of lower image resolution. Overall, clinicians and researchers should manually inspect and refine the autosegmentation results if possible to achieve the most accurate RNFL thickness measurements.

Our study provides new information about the magnitude, associations, and locations of errors with automated segmentation of the RNFL when operators do not use manual refinement. However, our study has limitations. For example, our study used data from a cohort of individuals with diagnoses ranging from glaucoma suspect to moderate glaucoma and our findings may not be applicable to subjects with more severe glaucoma. Our operators have up to 15 years of experience with ocular imaging in various forms. They had inter-individual differences in the error (differences) in manual refinement of automated imaging. Overall, it is likely that imaging artifacts would be more common for less experienced operators and be different between operators. Finally, our results may not be applicable to some OCT manufacturers that do not allow manual refinement of the automated segmentation. However, it is likely that these errors occur in other OCT devices, but that given each device may use a different segmentation algorithm, the nature and severity of the errors may vary.

In conclusion, OCT automated segmentation without manual refinement resulted in thinner RNFL measurements and overestimation of the glaucoma classification. Segmentation errors occur more commonly in eyes with decreased RNFL thickness, eyes with decreased scan quality, participants with increased age, and the superior and inferior temporal quadrants. Manual refinement of automated RNFL segmentation can help correct these errors to decrease misclassification of glaucoma. Future studies should compare automated and manual refinement for their ability to detect decreases in RNFL thickness longitudinally.