Evidence based guidelines for the number of peripapillary OCT scans needed to detect glaucoma worsening

Abstract

Objective:

Estimate the number of OCT scans necessary to detect moderate and rapid rates of retinal nerve fiber layer (RNFL) thickness worsening at different levels of accuracy using a large sample of glaucoma and glaucoma-suspect eyes.

Design:

Descriptive and simulation study

Participants:

12,150 eyes from 7,392 adult patients with glaucoma or glaucoma-suspect status followed at the Wilmer Eye Institute from 2013–2021. All eyes had at least 5 measurements of RNFL thickness on the Zeiss Cirrus OCT with signal strength of 6 or greater.

Methods:

Rates of RNFL worsening for average RNFL thickness and for the four quadrants were measured using linear regression. Simulations were used to estimate the accuracy of detecting worsening — defined as the percentage of cases where the true rate of RNFL worsening was at or below different criterion rates of worsening when the OCT measured rate was also at or below these criterion rates — for different testing frequencies and strategies. Two different measurement strategies were simulated: “evenly spaced” (equal time intervals between measurements) and “clustered” (approximately half the measurements at each of the endpoints of the time period).

Main Outcome Measures:

75th percentile (moderate) and 90th percentile (rapid) rates of RNFL worsening for average RNFL thickness, and the accuracy of diagnosing worsening at these moderate and rapid rates.

Results:

The 75th and 90th percentile rates of worsening for average RNFL thickness were −1.09 μm/yr and −2.35 μm/yr, respectively. Simulations showed that for the average measurement frequency in our sample of approximately 3 OCT scans over a 2-year period, moderate and rapid RNFL worsening are accurately diagnosed only 47% and 40% of the time, respectively. Estimates for the number of OCT scans needed to achieve a range of accuracy levels are provided. For example, 60% accuracy requires 7 measurements to detect both moderate and rapid worsening within a 2-year period if the more efficient “clustered” measurement strategy is used.

Conclusions:

In order to more accurately diagnose RNFL worsening, the number of OCT scans must be increased compared to current clinical practice. A clustered measurement strategy reduces the number of scans required compared to evenly spacing measurements.

The accuracy of diagnosing moderate and rapid rates of retinal nerve fiber layer thickness worsening using optical coherence tomography is estimated from over 12,000 glaucomatous eyes. More frequent scans are needed to improve accuracy.

Introduction

The rate of peripapillary retinal nerve fiber layer (RNFL) thickness loss measured by optical coherence tomography (OCT) is a widely used metric in the assessment of glaucoma worsening^1,2. It is therefore important to estimate the number of OCT scans needed to detect glaucoma worsening at desired levels of accuracy. One measure of accuracy is how often the true rate of OCT RNFL worsening — the rate without OCT measurement error — is at or below a criterion rate (e.g., 75th and 90th percentile rates in the population) when the rate measured by OCT is also at or below this criterion rate. Previous studies have estimated the frequency and severity of errors in OCT retinal layer segmentation algorithms^3,4 and measured OCT test-retest reliability^5,6. However, none have estimated the accuracy (as just defined) of OCT RNFL trend analysis at identifying different criterion rates worsening and assessed how accuracy varies with different testing frequencies and strategies. This is an important question. If eyes that are labeled as worsening using values measured on OCT are often not worsening, or vice versa, then providers may over or under call progression which may have significant implications for patient care (e.g., over/under treatment, inappropriate follow-up etc.).

In this study, rates of worsening are measured for average RNFL thickness and for superior, inferior, temporal and nasal quadrants in a sample of over 12,000 glaucoma and glaucoma-suspect eyes under care at a tertiary care glaucoma center. As current OCT devices do not provide a normative database of rates of RNFL worsening, we define “moderate” and “rapid” worsening to be the 75th and 90th percentile rates in our sample. We estimate the accuracy of diagnosing worsening at these criterion rates using simulations. Different testing strategies are also simulated (evenly spaced vs. clustered testing), since it is well known that clustering multiple measurements at fewer time points leads to more accurate estimates of the slope of a linear regression line than evenly spacing measurements^7–9, and linear regression is the standard method used to estimate the rate of RNFL worsening with OCT¹⁰. We conclude with evidenced based guidelines for the number of OCT scans required to achieve desired levels of accuracy for detecting moderate and rapid rates of RNFL worsening.

Methods

Measurements of RNFL thickness using OCT (Cirrus, Carl Zeiss Meditec, Dublin, CA, USA) were obtained from glaucoma and glaucoma suspect patients who were 18 years or older under care at the Wilmer Eye Institute from 2013 to 2021. Institutional Review Board (IRB)/Ethics Committee approval was obtained for this study. Eyes were included only if they had at least 5 measurements of RNFL thickness from different time points with a minimum superior/inferior quadrant thickness of 50 μm and a minimum signal strength (SS) of 6. Measurements below these thresholds were either below the RNFL thickness floor or likely to be unreliable due to artifact or poor image quality^11,12.

The mean deviation (MD) and glaucoma hemifield test (GHT) from the Humphrey visual field (24–2 SITA Standard) closest to baseline OCT were used to classify eyes as either glaucoma suspect, mild, moderate or severe using established guidelines^13,14. Eyes with GHT “within normal limits” and MD > −6 dB were classified as glaucoma suspect. If GHT was not within normal limits eyes were classified as mild if MD > −6 dB, moderate if −12 dB ≤ MD ≤ −6 dB, and severe if MD < −12 dB.

Figure 1 shows how simulated data were generated to estimate the accuracy of detecting RNFL worsening at or below different criterion rates for varying numbers of OCT scans and different testing strategies. The ith simulated data point for the kth eye had the form y_k,i = β_k + ε_k,i where β_k represents the true rate of worsening and ε_k,i is the ith residual for the kth eye. Though the distribution of true rates β_k is unknown, it is best approximated by the distribution of measured rates in a large sample of eyes. Thus, we set the true rate β_k in our simulation to the measured rate of worsening for the kth eye in our sample.

Figure 1.

Schematic of how simulated data (red dots) were generated for one eye, in this case for an eye whose true rate of worsening was β_k = −1 μm/yr (black line). Residuals are randomly chosen from the combined distribution of residuals (figure at top left) for rates of worsening within 0.25 μm/yr of β_k. The simulated measured rate (blue line) is the result of linear regression on the simulated data points. This simulation is repeated 100 times for each eye. Accuracy was measured as the percentage of cases where the true rate of worsening was at or below a criterion rate of worsening when the simulated OCT measured rate was also at or below this criterion. Accuracy was computed for different numbers of measurements and two different measurement strategies: evenly spaced (center) and clustered (right).

For each β_k the ith residual was randomly chosen from the combined distribution of residuals for all eyes whose measured rates of worsening were within 0.25 μm/yr of β_k. This assumes that measurement noise depends on β_k and is independent of time, which is a simplification that enables us to simulate different time intervals between measurements as well as generate more simulated data points than were measured for each eye.

For each eye, 100 sets of simulated RNFL thickness data were generated. Each simulated dataset consisted of M data points within a time period T, and simulated datasets were generated for different M and T. For each M and T we tested two conditions: “evenly spaced” and “clustered”. In the evenly spaced condition, consecutive measurements were separated by a constant time interval. In the clustered condition, all simulated measurements were either at the beginning of the time period or at the end of the time period, with approximately half the measurements at either endpoint. If M was even, precisely half the data points were at each endpoint, while if M was odd, either (M + 1)/2 data points were at the beginning and (M − 1)/2 at the end, or vice versa (randomly determined for each simulated dataset). Figure 1 shows a schematic of both evenly spaced (center) and clustered (right) measurement conditions. The same set of residuals are used in both conditions.

We chose to test only clustered and evenly spaced measurement strategies because there are theoretical reasons for why these two measurement strategies should lie close to the upper and lower bounds for accurately estimating the slope of a linear regression line⁷. Hybrid measurement strategies⁹ such as clustering at multiple time points, or combining evenly spaced and clustering measurement strategies, are likely to fall between these bounds. Accuracy of diagnosis was defined as the percentage of cases where the true rate (i.e., β_k) was at or below a criterion rate when the measured rate (i.e., slope fit to the simulated data) was also at or below this criterion. Accuracy was calculated for different criterion rates as well as for different values of M and T. All analysis was done using R (https://www.R-project.org/).

Results

A total of 12,150 eyes from 7,392 patients satisfied our inclusion criteria. The mean (SD) age of patients was 67.6 (14.8), 61% were female, and 55% were white. Glaucoma severity was defined for 71% of patients; 29% did not have a measured visual field. Table 1 provides further details on the demographics of our sample, distribution of glaucoma severity and baseline RNFL thickness.

Table 1.

Demographics and OCT data

Sample size
Patients	7,392
Eyes	12,150
Age
Mean (SD)	67.6 (14.8)
Median	70.2
Range	18.9, 102.3
Gender, n (%)
Male	2,901 (39.25%)
Female	4,488 (60.71%)
Other	3 (0.04%)
Race, n (%)
White	4,100 (55.46%)
Black	2,077 (28.10%)
Asian	444 (6.01%)
Other	543 (7.35%)
N/A	228 (3.08%)
Severity, n (%)
Suspect	3,677 (30.26%)
Mild	3,602 (29.65%)
Moderate	875 (7.20%)
Severe	479 (3.94%)
N/A	3517 (28.95%)

Baseline RNFL thickness (μm), Median (IQR)
All	83.6	74.7, 92.7
Suspect	85.8	78.1, 93.6
Mild	80.6	72.0, 89.6
Moderate	74.7	65.8, 86.2
Severe	68.7	61.2, 81.6

Figure 2A shows the distribution of rates of RNFL worsening (i.e., slopes from linear regression) for average peripapillary RNFL thickness over the range [−10, 10] μm/yr. The median rate of worsening is −0.39 μm/yr and the 75th and 90th percentiles are at −1.09 μm/yr and −2.35 μm/yr respectively. Table 2 (top half) lists the mean, median, 75th percentile and 90th percentile rates of worsening for the distributions of average RNFL thickness as a function of different levels of severity, and the distributions of RNFL thickness for superior, inferior, temporal and nasal quadrants. Median rates of worsening were more negative for superior (−0.61 μm/yr) and inferior (−0.74 μm/yr) quadrants compared to average RNFL thickness, and less negative for temporal (−0.18 μm/yr) and nasal (−0.09 μm/yr) quadrants. Table 2 (bottom half) also lists the percentage of cases where the measured rate of RNFL worsening was at or below different criterion levels.

Figure 2.

Distribution of the rate of RNFL worsening (A), signal strength (B), baseline RNFL thickness (C) and average time between measurements (D) for average RNFL thickness.

Table 2.

Rates of RNFL thickness worsening

	Distribution of rates of RNFL worsening (μm/yr)
	Mean	Median	75th Percentile	90th Percentile
Average	−3.08	−0.39	−1.09	−2.35
Suspect	−1.57	−0.37	−1.01	−1.96
Mild	−1.72	−0.39	−1.05	−2.07
Moderate	−6.21	−0.43	−1.13	−2.75
Severe	−8.49	−0.39	−1.37	−5.16
Superior	−4.37	−0.61	−1.77	−3.72
Inferior	−4.34	−0.74	−1.94	−4.02
Temporal	−1.30	−0.18	−0.75	−1.73
Nasal	−2.33	−0.09	−0.88	−2.21

	Percentage of eyes with a measured rate of RNFL worsening at or below:
	0	−1	−2	−3	−4	−5	−6	−7	−8
Average	68.9%	27.0%	12.1%	7.6%	5.9%	4.9%	4.2%	3.8%	3.5%
Suspect	69.2%	25.2%	9.6%	5.6%	4.1%	3.2%	2.5%	2.2%	2.0%
Mild	69.6%	25.8%	10.5%	6.2%	4.8%	3.8%	3.0%	2.9%	2.7%
Moderate	71.2%	28.5%	13.0%	8.9%	7.2%	6.7%	6.0%	5.4%	5.1%
Severe	65.1%	31.8%	20.0%	13.5%	11.5%	10.4%	9.6%	8.7%	8.2%
Superior	67.7%	40.0%	21.6%	13.0%	9.2%	7.0%	5.8%	5.1%	4.5%
Inferior	71.7%	42.7%	24.1%	14.6%	10.1%	7.7%	6.2%	5.3%	4.7%
Temporal	60.2%	18.7%	8.7%	5.6%	4.0%	3.2%	2.7%	2.4%	2.2%
Nasal	53.1%	22.6%	11.3%	7.2%	5.5%	4.6%	4.0%	3.6%	3.3%

Figure 2B shows the distribution of signal strength (SS) for SS ≥ 6. The distribution is highly symmetric with a mean of 8.0, a standard deviation of 1.0, and a ceiling effect at SS = 10 with 3.3% of all measurements. Figure 2C shows the distribution of baseline average RNFL thickness in the range [50, 150] μm. RNFL thickness values below 50 μm were filtered out, while RNFL thickness values above 150 μm constituted 1.03% of our sample. The mean (SD) baseline average RNFL thickness is 87.50 (32.08) μm, the median is 83.62 μm, and the skewness is 6.87 μm for the right-skewed distribution (i.e., most data falls to the right of the peak). Figure 2D shows the distribution of mean time intervals between measurements per eye with a mean (SD) of 390 (186) days. The average number of samples per eye was 6.3, although we note that this was after filtering out eyes with 4 or fewer measurements of RNFL thickness.

Figure 3 shows the percentage of simulated eyes (y-axes) whose true rate of average RNFL thickness worsening was at or below different criterion rates of worsening (x-axes) when the simulated OCT measured rate was also at or below that criterion. Results are shown as a function of the number of simulated OCT measurements (different colored lines). Figure 3A shows results for a 2-year period in the evenly spaced condition. Figure 3B shows the comparable result when simulated measurements are clustered at the beginning and end of the 2-year period. For example, the black curve in Figure 3B shows that a rate of worsening of −3 μm/yr or lower can be detected with 75% accuracy if a total of 13 measurements are made, assuming half the measurements (either 6 or 7 measurements) are clustered at the beginning of the 2-year period and the rest are clustered at the end. The comparable accuracy when measurements are evenly spaced is 58% (black curve in Figure 2A at −3 μm/yr) — “evenly spaced” for 13 measurements over a 2-year period corresponds to one measurement every 2 months. Figures 3C and 3D are the analogs of Figures 3A and 3B for a 1-year period. Color coding across for all 4 graphs in Figure 3 are by total number of measurements.

Figure 3.

Accuracy (percent correct) for detecting average RNFL thickness worsening at or below different criterion rates of worsening (x-axis) in evenly spaced (A, C) and clustered (B, D) conditions. Colored lines represent different numbers of measurements over a 2-year (A, B) or 1-year (C, D) period. The legends in the evenly spaced conditions specify the frequency of measurement.

The precision of the estimates in Figure 3 was estimated by re-running all simulations. The mean absolute error (MAE) between simulations was 0.15 percent correct across all conditions (number of measurements and measurement strategy) for the 2-year period and 0.09% across all conditions for the 1-year period. Thus, 100 simulations per slope is sufficiently large for the results in Figure 3 to be reliable to within less than 1% error.

Figure 4 plots percent correct as a function of the total number of measurements over a 2-year period for moderate (Figure 4A) and rapid (Figure 4B) rates of worsening using average RNFL thickness. Data points (black triangles and squares) represent conditions where percent correct was actually estimated. The equations for determining the accuracy of detecting worsening for the clustered (blue) and evenly spaced (red) curves based on the total number measurements are provided in Supplemental Table S1 (available at www.aaojournal.org).

Figure 4.

Accuracy (percent correct) plotted as a function of the total number of measurements for 75th percentile (A) and 90th percentile (B) rates of average RNFL worsening in clustered (blue) and evenly spaced (red) conditions over a 2-year period. The x-axis begins at 2 measurements. Data points (black triangles and squares) represent conditions where percent correct was actually estimated. The equations for the blue and red curves fitted to the data points are provided in Supplemental Table S1.

Figure 5 plots percent correct as a function of the total number of measurements over a 2-year period for average RNFL thickness (black curves: the same curves as in Figure 4) and the 4 quadrants: superior (yellow), inferior (green), temporal (magenta) and nasal (blue). Definitions of “moderate” and “rapid” worsening in Figure 5 are different for each quadrant (see Table 2). Across all conditions — moderate (5A and 5B) and rapid (5C and 5D) rates of worsening as well as evenly spaced (5A and 5C) and clustered (5B and 5D) measurement strategies — accuracy is highest with inferior and the lowest with nasal. The average difference between inferior and nasal is approximately 10 percent correct across all conditions.

Figure 5.

Accuracy (percent correct) plotted as a function of total number of measurements for 75th percentile (A, B) and 90th percentile (C, D) rates of RNFL worsening for average (black), superior (yellow), inferior (green), temporal (magenta) and nasal (blue) RNFL thickness in evenly spaced (A, C) and clustered (B, D) measurement conditions over a 2-year period.

Figure 6 is analogous to Figure 5 except that the colored curves stratify by severity instead of by quadrant: glaucoma suspect (blue), mild (green), moderate (yellow), severe (magenta) and all eyes (black: same curves as in Figure 4). Unlike Figure 5, all curves in Figure 6 use the same definitions of “moderate” and “rapid” glaucoma worsening, namely for average RNFL thickness (i.e., −1.09 μm/yr for moderate and −2.35 μm/yr for severe). In general the more severe the glaucoma the better the accuracy.

Figure 6.

Accuracy (percent correct) plotted as a function of total number of measurements for 75th percentile (A, B) and 90th percentile (C, D) rates of average RNFL worsening for all eyes in our sample (black), glaucoma suspects (yellow), and mild (green), moderate (magenta) and severe (blue) glaucoma in evenly spaced (A, C) and clustered (B, D) measurement conditions over a 2-year period.

The results in Figures 5 and 6 are mostly explained by: 1) the percentage of eyes with measured rates of worsening well below the criterion, and 2) the mean absolute residual after linear regression. For measured rates of worsening well below the criterion, the true rate of worsening is nearly guaranteed to be below the criterion, increasing accuracy. This helps explain Figure 6: as severity increases there is a higher percentage of measured rates of worsening well below the criterion (see bottom half of Table 2), increasing accuracy. Counteracting this effect is accuracy decreasing due to a larger mean absolute residual. The larger the mean absolute residual, the more likely a measured rate of worsening at or just below the criterion comes from a true rate above it. Mean absolute residuals increase with severity: 2.47 μm for suspect, 2.73 μm for mild, 3.90 μm for moderate and 4.74 μm for severe. However, Figure 6 shows that in the case of severity, the effect of having a larger mean absolute residual is insufficient to overcome the effect of having a larger percentage of eyes with measured rates of worsening well below the criterion.

The situation is somewhat different for Figure 5. Table 2 (bottom half) explains why inferior has the highest accuracy, but on its own Table 2 predicts that temporal, not nasal, should have the lowest accuracy. Here, the larger mean absolute residual for nasal (3.86 μm) versus temporal (2.41 μm) makes a difference. Mean absolute residuals for inferior (4.52 μm) and superior (4.67) were nearly identical and did not affect the relative accuracies of inferior and superior predicted by Table 2.

There are other factors that influence accuracy, such as the ratio of the percentage of eyes right above the criterion versus right below it. The larger the ratio the lower the accuracy because it is more likely that a measured rate of worsening at or just below the criterion comes from a true rate above it. In other words, the shape of the distribution in the neighborhood of the criterion matters. However, the results in Figures 5 and 6 can be mostly explained by the two factors described above.

Discussion

This study shows that the number of OCT scans needed to accurately diagnose glaucoma worsening over a 2-year period is far greater than is typically obtained in current clinical practice. OCT scans in our sample were taken on average once every 390 days, which is consistent with current insurance coverage¹⁵. At this rate, the estimated accuracy is 47% for detecting moderate RNFL worsening and 40% for detecting rapid worsening. The current frequency of obtaining OCT scans may be sufficient to diagnose the presence of glaucoma but this frequency is not sufficient to accurately diagnose glaucoma worsening.

To use OCT to diagnose glaucoma worsening accuracy should be increased. To obtain an accuracy of at least 60%, a total of 7 OCT scans over 2 years are needed to detect both moderate and rapid worsening using the more efficient clustered measurement strategy (i.e., 3–4 OCT scans at the first and last visit). For 70% accuracy, 14 OCT scans to detect moderate worsening and 16 OCT scans for rapid worsening. OCT scans take little time and are patient friendly, and 60% or even 70% accuracy may be practical to achieve by taking many OCT scans at each visit.

From a practical standpoint, it is unclear whether multiple scans can be obtained serially (with the patient not leaving the OCT machine) or whether the patient should be told to leave the OCT machine for a short period and return and retake the test. The latter strategy may induce more variance in the RNFL measurements in a same-visit cluster of scans and thus result in a mean measure of RNFL thickness in the cluster that is closer to a ground truth value. Future studies will have to analyze the best approach for obtaining clustered OCT exams. Current OCT progression software (i.e., Zeiss FORUM) does not, to our knowledge, allow for clustered measurements to be included in linear regression of RNFL worsening. This will need to be modified to make employing clustered OCT scans practical.

This study also provides measured rates of RNFL worsening for average RNFL thickness for different levels of glaucoma severity and measured rates of RNFL worsening for the 4 quadrants from a large sample of 12,150 eyes with glaucoma or glaucoma-suspect status. The median, 75th and 90th percentile rates of worsening for average RNFL thickness are: −0.39 μm/yr, −1.09 μm/yr and −2.35 μm/yr, respectively. A separate study with a large sample of 6,138 glaucoma or glaucoma-suspect eyes from 3,669 patients found a similar value of −1.03 μm/yr for the 75th percentile but a median rate of −0.67 μm/yr¹⁶. Our 75th percentile and median rates compare favorably to results from a recent study based on 541 eyes from 357 clinically followed glaucoma suspects which found that eyes developing visual field defects had on average a −1.13 μm/yr rate of average RNFL thickness worsening as opposed to a −0.27 μm/yr rate for those that did not¹⁷. As the Cirrus device does not include a normative database of rates of RNFL worsening, the 75th and 90th percentile rates may be used as guidelines for whether a patient with glaucoma is progressing at a moderate or fast rate.

A major strength of this study is the large dataset of glaucoma and glaucoma-suspect patients followed in a clinical population. However, there are several limitations. First, simplifying assumptions were made in our simulations that may not hold. For example, residuals for clustered measurements may not be similar to residuals for evenly spaced measurements since clustering is conceptually closer to test-retest variability^5,6 than trend-based analysis. Second, our estimates of percent correct at the 75th and 90th percentiles only apply to diagnosis of worsening after a 2-year period. However, 2 years is a reasonable timeframe to diagnose glaucoma worsening given that current recommendations are to perform 6 visual field tests over this period to assess disease trajectory¹⁸. Finally, our data were collected from the Zeiss Cirrus OCT and may not be generalizable to other OCT devices.

In summary, we recommend obtaining a much larger number of OCT scans than is currently the norm to accurately detect glaucoma worsening. When practical, a clustered measurement strategy should be used with multiple scans per visit. As OCT is a relatively quick and patient friendly test, we believe providers can easily increase the number of tests per visit to more accurately identify glaucoma worsening.