Effectiveness of a Qualitative Approach Toward Evaluating OCT Imaging for Detecting Glaucomatous Damage

Abstract

Purpose

The purpose of this study was to determine whether a qualitative approach toward evaluating optical coherence tomography (OCT) imaging improves the ability to detect glaucomatous damage compared to a conventional metric of global circumpapillary retinal nerve fiber layer (cpRNFL) thickness.

Methods

A total of 394 healthy eyes and 272 glaucoma eyes were evaluated. Glaucoma eyes were categorized as perimetric (156 eyes) based on a history of three or more consecutive abnormal 24-2 visual field tests or suspected glaucoma if they did not (116 eyes). Customized one-page reports derived using OCT volume scans of the optic disc and macula from these eyes were qualitatively graded for the probability of optic neuropathy affecting the eye.

Results

The sensitivity of detecting perimetric glaucoma eyes with the global circumpapillary RNFL thickness metric and qualitative evaluation of the OCT imaging results were 86.5% and 95.5% at a specificity of 95%, being significantly higher for the latter (P < 0.001). There were seven eyes with perimetric glaucoma missed by the qualitative evaluation. Based upon examination of all available visual fields, at least four of these seven eyes had visual fields that either improved or had abnormalities that were inconsistent over time or with patterns of glaucomatous damage.

Conclusions

Qualitative evaluation of OCT imaging results allows glaucoma eyes with repeatable visual field abnormalities to be detected with a high level of accuracy, performing better than a conventional summary metric of global cpRNFL thickness.

Translational Relevance

Clinical detection of glaucomatous damage with OCT imaging can be optimized through a qualitative evaluation of its results.

Introduction

Glaucoma is an optic neuropathy characterized by the progressive loss of retinal ganglion cells (RGCs) that is clinically observable as characteristic changes to the appearance of the optic nerve head. Optical coherence tomography (OCT) is a high-resolution imaging modality becoming widely used in clinical settings that allows three-dimensional visualization and quantification of the neuroretinal tissue affected in glaucoma, thus showing great promise for improving the detection of glaucomatous damage.

Nonetheless, some studies have reported that circumpapillary retinal nerve fiber layer (cpRNFL) thickness measurements from OCT imaging did not perform any better than a careful qualitative evaluation of the optic nerve head appearance on stereophotographs for discriminating between glaucoma eyes with repeatable visual field abnormalities and healthy eyes.^1–5 However, these studies relied primarily on using conventional metrics such as the global thickness rather than making full use of the wealth of information available from the OCT scans interpreted alongside a knowledge of the nature of glaucomatous damage.⁶ Indeed, a previous study showed that the sensitivity of detecting glaucomatous damage was improved by scoring RNFL defects on the RNFL thickness deviation plots.⁷

We therefore hypothesized that careful qualitative evaluation of OCT imaging results alone could allow glaucoma eyes with evident visual field loss to be detected with a high degree of accuracy. This hypothesis is based on the findings of our two previous studies, where a qualitative evaluation of OCT imaging results provided a high accuracy for detecting eyes considered to have glaucomatous damage (having a sensitivity of 100% and specificity of 95% in our first study⁸ and a sensitivity and specificity of 98% in our second⁹). However, a key limitation of these two studies was the inclusion of OCT imaging results in the formation of the reference standard. This can overestimate the actual diagnostic performance of the qualitative evaluation of OCT imaging since the technique being evaluated was used for establishing the reference standard itself.

Instead, a diagnostic test should ideally be compared against an independent measure that closely reflects a relevant clinical endpoint or “[a] characteristic or variable that reflects how the patient feels, functions, or survives.”¹⁰ In patients with glaucoma, visual field results have been accepted as such a relevant surrogate functional metric,^11,12 given its close association with functional disability.^13,14 Therefore, it is important to determine the diagnostic performance of the qualitative evaluation of OCT imaging results for detecting glaucoma eyes with established visual field abnormalities in order to establish the effectiveness and clinical relevance of this technique. This study was performed to undertake this evaluation.

Methods

Participants

This study included participants who were part of a prospective study to understand the role of OCT imaging in glaucoma and was approved by the Institutional Review Boards of Columbia University and the New York Eye and Ear Infirmary of Mount Sinai. It adhered to the tenets of the Declaration of Helsinki and the Health Insurance Portability and Accountability Act, and written informed consent was obtained from all participants.

Participants were considered to have either suspected glaucoma or glaucoma based on a comprehensive clinical examination by the referring glaucoma specialist (RR), and all eyes were required to have a best-corrected visual acuity of 20/40 or better. Participants were excluded if they had any ocular or systemic conditions that could affect visual field or OCT imaging results (e.g., retinal vein occlusion, demyelinating disease).

All participants were also required to have performed at least three reliable visual field tests prior to the date of OCT imaging using the Swedish Interactive Threshold Algorithm standard 24-2 testing strategy on a visual field analyzer (Humphrey Field Analyzer II-I; Carl Zeiss Meditec, Inc., Dublin, CA). An unreliable visual field test was defined as a test with >33% fixation losses or false-negative errors (except for the latter when mean deviation [MD] was less than −12 dB) or with >15% false-positive errors. Eyes were categorized as having perimetric glaucoma if they had a history of three or more consecutive abnormal visual field test results prior to the date of OCT imaging, with an abnormal result defined as having a pattern standard deviation (PSD) at P < 0.05 or a Glaucoma Hemifield Test (GHT) result outside normal limits.¹⁵ This criterion was met on an average ± standard deviation of 4.6 ± 3.4 years (range, 0–12 years) prior to the OCT imaging date. Eyes that did not meet this criterion of having a history of at least three consecutive abnormal visual field tests were categorized as having suspected glaucoma.

Healthy participants consisted of those included in a study that determined normal reference limits by the OCT device manufacturer (data provided by Topcon, Inc., Tokyo, Japan). One eye, randomly selected, from each participant was included in this study; the eligibility criteria for these participants are described further in Supplementary Material S1.

OCT Imaging and Customized Report

Volume scans consisting of 512 × 128 A-scans over a 6 × 6-mm region centered on the optic disc, and macula were obtained using a spectral-domain OCT device (3D OCT-2000; Topcon, Inc.). Scans contaminated by significant blink or eye movement artifacts were excluded from this study. These macular and disc scans were used to generate a customized one-page report similar to one described in our previous study,^8,16 and two examples of this report are shown in Figure 1. This report includes a circumpapillary B-scan derived from the optic disc volume scan averaged over an annulus of 100 μm in width, and its corresponding RNFL thickness profile (upper left panels), both of which were presented with the temporal region of the disc at the center to provide greater ease for evaluating the topographical relationship between regions of the cpRNFL thickness profile and retinal and visual field locations.⁸ It also includes the macular and optic disc RNFL and macular RGC plus inner plexiform layer (RGC+) thickness map (lower left), with the latter included because glaucomatous damage often affects this region¹⁷ and can be more easily visualized with this map. Corresponding thickness deviation probability plots for each of these thicknesses were also included because subtle damage may be easier to see with these plots than when using thickness maps alone. The optic disc and macular RNFL thickness maps were coregistered using retinal features (such as blood vessels) to provide greater ease in evaluating the spatial extent of abnormalities in this retinal layer and were presented in field view (right panels). The 24-2 and 10-2 visual field locations were overlaid on these RNFL and RGC+ thickness deviation probability plots to allow a topographical comparison of the spatial extent of their abnormalities (right panels). Finally, a pie chart consisting of the average cpRNFL thickness of each quadrant was also included.

Figure 1.

Examples of the customized one-page reports used in this study, consisting of a cpRNFL thickness profile and its corresponding B-scan (top left), macular and optic disc RNFL thickness plots and a macular retinal ganglion cell plus inner plexiform layer (RGC+) thickness plot (bottom left), an optic disc and macular RNFL thickness probability plot in field view with 24-2 visual field locations overlaid (top right), a macular RGC+ thickness probability plot in field view with 10-2 visual field locations overlaid (bottom right), and a circumpapillary RNFL quadrant thickness pie chart (bottom right). These two examples illustrate eyes with perimetric glaucoma that were correctly identified with a qualitative approach (due to the local thinning of the cpRNFL in the inferior-temporal region) but were missed with the global circumpapillary cpRNFL parameter.

Qualitative Evaluation of OCT Imaging

One report specialist (DCH) graded all the one-page reports included in this study and was masked to the characteristics of the cohort. Each report was graded for the probability of optic neuropathy affecting the eye along a continuous scale (between 0% and 100%), with higher values denoting a higher likelihood of optic neuropathy. Note that the term optic neuropathy was used because the report specialist was not tasked to distinguish between evidence of glaucomatous and nonglaucomatous optic neuropathy on OCT imaging. However, because this study excluded eyes with nonglaucomatous causes of optic neuropathy, this term will also be referred to as glaucomatous damage. The reports in this study were presented in a randomized manner, and the time required to grade each report was automatically recorded using a custom-written program.

Statistical Analysis

The primary outcome measure of this study was the ability to discriminate between eyes with perimetric glaucoma and healthy eyes. Eyes with suspected glaucoma were included to ensure that the full spectrum of the disease was presented during the grading process to discourage the examiner from considering all eyes as either clearly healthy or glaucomatous. However, the discriminatory ability of the qualitative evaluation and the age-adjusted global cpRNFL thickness parameter (both methods as continuous predictors) was evaluated only in healthy and perimetric glaucoma eyes and was visualized with the receiver operating characteristic (ROC) curve of each method. It was compared by performing a Wald test of the difference in sensitivity for detecting the perimetric glaucoma eyes at a 95% specificity using a bootstrap resampling procedure (n = 1000 resamples).¹⁸ The eyes with suspected glaucoma were not included in this analysis because their true disease state remains unknown based on the independent visual field criterion (i.e., three consecutive abnormal visual field tests defined using the GHT or PSD results).

Results

Participant Characteristics

A total of 272 glaucoma eyes from 169 participants met the eligibility criterion of this study, and 156 eyes from 110 participants had a history of three consecutive abnormal visual field results (defined using the GHT and PSD values), while 116 eyes from 87 participants did not. These eyes are referred to below as having perimetric and suspected glaucoma, respectively. The mean ± standard deviation of their age was 65 ± 12 (range, 29–89) and 59 ± 16 (range, 20–84) years old, respectively. The median (interquartile range [IQR]) MD and PSD of the perimetric glaucoma eyes were −5.83 dB (IQR = −10.57 to −3.4 dB) and 6.36 dB (IQR = 3.45–10.45 dB), respectively, and were −0.74 dB (IQR = −2.32 to 0.18 dB) and 1.63 dB (IQR = 1.43–1.96 dB), respectively, for the suspected glaucoma eyes. A total of 394 healthy eyes were also included in this study, and they were 47 ± 16 (range, 18–89) years old. The characteristics of these participants and the qualitative grading results (described below) are summarized in Table 1.

Table 1.

Summary of the Characteristics of the Participants Included and Qualitative Grading Results

Qualitative Grading

The median (IQR) qualitatively graded probability of optic neuropathy (noting that higher values indicate a graded higher likelihood of optic neuropathy) for the healthy, suspected glaucoma, and perimetric glaucoma eyes were 1% (IQR = 1%–1%), 40% (IQR = 1%–98%), and 99% (IQR = 99%–99%) respectively. Note that 21% of healthy eyes and 10% of perimetric glaucoma eyes had a probability grading >1% and <99%, respectively. Thus, the IQR was relatively narrow, indicating how most eyes were graded with relative certainty. The median time required to grade the OCT imaging reports for these eyes was 8 seconds (IQR = 6–18 seconds), 29 seconds (9–56 seconds), and 17 seconds (9–32 seconds), respectively; the median time to grade all the OCT imaging reports in this study was 11 seconds (IQR = 6–31 seconds). These findings are also summarized in Table 1.

Diagnostic Performance of the Qualitative Approach

The ROC curves in Figure 2 illustrate the diagnostic performance of the qualitative OCT evaluation using the report to discriminate between the healthy and perimetric glaucoma eyes.

Figure 2.

ROC curve of the diagnostic performance of the qualitative evaluation of the OCT imaging results and the age-adjusted global circumpapillary retinal nerve fiber layer (cpRNFL) thickness parameter for discriminating between healthy eyes and eyes with perimetric glaucoma.

At a specificity of 95%, the sensitivity of detecting glaucoma with the qualitative approach was significantly higher than using the global cpRNFL thickness parameter when evaluating all eyes with perimetric glaucoma (95.5% vs. 86.5%; P < 0.001) and when including only perimetric glaucoma eyes with a visual field MD better than −6 dB (92.5% vs. 82.5%; P = 0.007); these findings are summarized in Table 2. Note that including the global RGC+ thickness along with cpRNFL thickness in a multivariable logistic regression model only improved the sensitivity of detecting the perimetric eyes from 86.5% to 88.5% (still significantly lower than the qualitative approach, P = 0.010) and made no difference when evaluating only eyes with an MD > −6 dB. In addition, the qualitative approach also had a significantly higher sensitivity for detecting perimetric glaucoma eyes compared to individual cpRNFL thickness sectors (including the superior-nasal, superior-temporal, temporal, inferior-temporal, inferior-nasal, and nasal; all P ≤ 0.011).

Table 2.

Sensitivity of Detecting Perimetric Glaucoma Eyes at a Specificity of 95%

Therefore, overall at a specificity of 95%, 7 out of 156 eyes in this study were missed with the qualitative approach, as opposed to 21 eyes that were missed when using the global cpRNFL thickness parameter. Two examples of eyes missed by the global cpRNFL thickness parameter are shown in Figure 1.

Perimetric Glaucoma Eyes Missed by the Qualitative Approach

Of the seven eyes missed with the qualitative approach, six of the eyes had a visual field MD > −6 dB at the time of OCT imaging. To better understand why the qualitative approach may have missed seven glaucoma eyes, all the available visual field results from the first of the three consecutive abnormal test results were reviewed alongside the OCT imaging results. (Recall that our definition of perimetric glaucoma depended on having an abnormal GHT and/or PSD result on at least three consecutive 24-2 visual field tests.) Two cases are presented below, and the remaining five cases are described in Supplementary Material S2. Broadly, in two eyes (cases 1 and 3), the more recent visual fields improved and were no longer abnormal (based on the GHT or PSD results). In two other eyes (cases 2 and 4), the pattern of abnormal points on the visual field were not consistent and did not resemble patterns typical of glaucomatous damage.

For example, case 1 involved the left eye of a 55-year-old man suspected of having normal tension glaucoma. The OCT imaging results were graded as having a 1% probability of glaucomatous damage being present, but the visual field results revealed inferior hemifield defects for the three consecutive abnormal tests (Fig. 3). However, the subsequent three visual field results demonstrated a gradual improvement of these defects, where the GHT and PSD results were no longer abnormal, highlighting how this eye may not actually have true visual field losses. The results for case 3 (found in Supplementary Material S2) were similar to the findings of this case.

Figure 3.

Case 1: An eye missed by a qualitative evaluation of the one-page report of OCT scans (top) but with a history of three consecutive abnormal visual field test results (with time from the OCT scan shown at the bottom) based on the GHT result and PSD values, as indicated by the asterisks (bottom). WNL, within normal limits; ONL, outside normal limits.

Case 2 involves the left eye of an 80-year-old male participant with pigment dispersion syndrome and pseudoexfoliation syndrome, currently under medical therapy. The OCT imaging results were graded as having a 5% probability of glaucomatous damage. The three consecutive abnormal visual field tests and the one available subsequent visual field test had an abnormal PSD or GHT. However, the locations of the abnormal points were not consistent and looked more like a rim artifact in last test (Fig. 4). The results for case 4 were similar to the findings of this case (see Supplementary Material S2).

Figure 4.

Case 2: An eye missed by a qualitative evaluation of the one-page report of OCT scans (top) but with a history of three consecutive abnormal visual field test results (with time from the OCT scan shown at the bottom) based on the GHT result and PSD values, as indicated by the asterisks (bottom).

Discussion

This study demonstrated that a qualitative evaluation of OCT imaging results allows eyes with perimetric glaucoma to be detected with a high degree of accuracy, performing better than a conventional metric of global cpRNFL thickness. Furthermore, at least four of the seven eyes with perimetric glaucoma that were missed by the qualitative evaluation on OCT imaging showed visual field results that could be considered inconclusive for glaucomatous damage. This suggests that the actual accuracy might be even higher than we reported. In any case, these findings underscore the potential of OCT imaging in the clinical management of patients with glaucoma.

Previous studies have consistently suggested that OCT imaging does not perform any better than does a careful qualitative evaluation of optic disc stereophotographs for detecting glaucoma eyes with repeatable visual field defects when relying on conventional summary metrics like global circumpapillary RNFL thickness.^1–5 Instead, this study demonstrates that applying a similar careful qualitative approach toward evaluating OCT imaging results substantially improves the detection of glaucomatous damage compared to using a summary metric. This should not be surprising. In the same way that the presence, extent, nature, and details of glaucomatous damage visible on a clinical examination of the optic disc are insufficiently represented by a summary metric such as a cup-to-disc ratio, characteristics of glaucomatous damage visible on OCT imaging are likewise insufficiently represented by a metric of global cpRNFL thickness.

Indeed, these findings are in agreement with a previous study⁷ that demonstrated how the ability to detect glaucomatous damage was improved by scoring RNFL defects based on the RNFL thickness deviation plots of OCT optic disc scans. However, our study is distinguished by also considering patterns of loss on the RNFL thickness plots in addition to the deviation plots and we evaluated macular OCT scans as well. This process may improve the ability to detect subtle glaucomatous defects, especially those that may not fall outside normative limits (and thus be present on the deviation plots) due to healthy interindividual variability. It may also improve the ability to correctly discriminate between normal variations in neuroretinal thickness (which may result in arcuate-like abnormalities on the optic disc RNFL thickness plots, for instance) and true glaucomatous defects.⁶

Our findings also agree with our previous studies,^8,9 although there are some key differences in the methodology and thus its interpretation. For instance, the judgment of whether an eye was abnormal or not (used for defining the reference standard) in our first study⁸ was performed by three glaucoma specialists who were provided with a commercial OCT report, optic disc stereophotographs, and a 24-2 visual field test. In our second study,⁹ the reference standard was determined by two glaucoma specialists who were given a widefield OCT report and its interpretation by the grader (similar to a neurosurgeon being provided with results by a radiologist), along with the patient chart information, optic disc stereophotographs, and 24-2 and 10-2 visual field results. An important limitation with evaluating the qualitative approach with a reference standard that depends on using the OCT imaging results as well is the risk of overestimating its diagnostic performance, since both the technique studied and the reference standard evaluate the same source of information (although the latter includes other information as well).

Another important methodological difference was the participants examined; our previous studies^8,9 included only eyes with abnormal or suspicious-appearing optic discs and with a 24-2 visual field MD > −6 dB. Consequently, the eyes judged as being normal were used as the control group instead of the use of healthy eyes, as in this study. The eyes included in our previous studies are more likely to represent the challenging cases faced daily by glaucoma specialists in clinical practice, whereas the eyes included in this study (including a larger proportion of healthy participants) might be more akin to those seen in a general ophthalmology setting. In either case, the inclusion of a large sample of healthy participants in this study is relevant given the relatively low prevalence of glaucoma.¹⁹ In addition, ensuring that a diagnostic technique has a high level of specificity is paramount because the initiation of lifelong treatment, let alone the diagnosis of glaucoma, can adversely affect an individual.^20,21

On the other hand, it is also crucial that a diagnostic technique like the qualitative approach of evaluating OCT imaging results not miss eyes with severe visual field loss. Indeed, we observed in this study that only 7 out of 156 eyes (4.5%) with perimetric glaucoma were missed by the qualitative evaluation of OCT imaging, although only one of these seven eyes had a 24-2 visual field MD worse than −6 dB. It is essential to note that while the use of a visual field endpoint for defining perimetric glaucoma is clinically relevant, it is by no means perfect because this endpoint can be reached simply as a result of measurement variability or factors unrelated to glaucomatous damage. In fact, a previous study demonstrated that 12% of the visual field tests in eyes with ocular hypertension (requiring three consecutive abnormal visual field tests based on the GHT and PSD results) after the visual endpoint was reached showed normal results.¹⁵ For this reason, we presented the visual field results of the seven eyes missed by the qualitative evaluation of the OCT imaging results to the clinical and scientific community so that they may determine the significance of those visual field abnormalities for themselves. Our observations were that there was no compelling evidence of glaucomatous visual field damage in at least four of these eyes and that further investigations into the visual field abnormalities observed in the remaining three eyes are warranted. We thus believe that the diagnostic performance of the qualitative evaluation of the OCT results might be higher than we actually report.

Whereas the presence of glaucomatous damage in the majority of the seven eyes missed by the qualitative approach was somewhat inconclusive, the same was not true for many of the 21 eyes (13.5%) missed when using the global cpRNFL thickness parameter but detected by the qualitative OCT evaluation. This can be clearly seen in the two examples shown in Figure 1, where the inferior-temporal RNFL defect associated with macular RGC+ abnormalities in both eyes were missed by the global parameter. Missing cases with obvious disease—especially disease that often affects the macula,^6,17 which is a part of the retina that is crucial for vision-related quality of life²² and daily functioning¹³—is problematic, and hence we caution against a reliance on a global thickness metric alone.

Nonetheless, the key implication of the findings of this study is clear: the full potential of OCT imaging can be realized through a careful qualitative evaluation of the wealth of information it provides during the clinical management of patients with glaucoma. This careful evaluation took a median of 11 seconds to perform in this study and often took no longer than 1 minute even in challenging cases (such as eyes with suspected glaucoma), highlighting its feasibility in real-world clinical practice. Furthermore, the results highlight the potential utility of using the qualitative evaluation of OCT imaging results for defining the presence of glaucomatous damage in clinical research, although further studies are required to validate such an approach. In particular, it would be important to establish whether this method can accurately predict clinically relevant outcomes longitudinally in eyes without repeatable visual field loss (such as the those with suspected glaucoma in our study).^12,23 The continuous scale used for grading the probability of glaucomatous damage in this study is also useful for clinical research since it allows sensitivities to be compared at matched specificities, accounting for differences between graders. In addition, continuous measures tend to be more powerful than categorized measures, as useful information can be lost through the categorization process.²⁴

An important limitation should be recognized when interpreting the results of this study, namely that an observer with extensive experience in the evaluation of the OCT imaging results (DCH)⁶ performed all the grading in this study, although previous work indicated others, including those without a medical degree, could be trained to perform as well.^6,8 In any case, it remains to be determined whether a similar diagnostic performance with the qualitative approach can be achieved currently in clinical practice or whether a knowledge gap exists. However, one of the authors in this study (ZW) also achieved the same diagnostic performance when undertaking the same evaluation, but the results were not included because the author was not masked to the proportion of healthy and glaucoma eyes included in the study (but otherwise performed all the grading in the same manner). Of interest, there was a near-perfect agreement between the grading by this author (ZW) and the single observer in this study (DCH; κ = 0.93 ± 0.04; P < 0.001) for the healthy and perimetric glaucoma eyes, but studies are needed to further establish the interobserver agreement for this method and are indeed underway in our lab. Nonetheless, this study demonstrates a proof in principle that OCT imaging can be a powerful tool in the detection of glaucomatous damage, and the use of other novel measures (e.g., RNFL volume deviation²⁵) or artificial intelligence (e.g., deep learning methods²⁶) could make better use of its information beyond the current conventional summary metric of global cpRNFL thickness. Future studies are also needed to determine whether a qualitative evaluation also performs better than such summary metrics when evaluating eyes that are difficult to judge, such as those with high myopia or macrodiscs and eyes with preperimetric glaucoma.

In conclusion, this study demonstrated that a qualitative evaluation of the OCT imaging results allowed a detection of glaucoma eyes with repeatable visual field abnormalities with a high level of accuracy, superior to that achieved by a global thickness parameter. These findings show the potential of OCT imaging in clinical management of patients with glaucoma when making full use of the information that it provides.