Comparing the “Rule of 5” to Trend-based Analysis for Detecting Glaucoma Progression on Optical Coherence Tomography

Abstract

Objective:

The “rule of 5” is a simple rule for detecting retinal nerve fiber layer (RNFL) change on spectral-domain optical coherence tomography (SDOCT), in which a loss of 5μm of global RNFL on a follow-up test is considered evidence of significant change when compared to the baseline. The rule is based on short-term test-retest variability of SDOCT and is often used in clinical practice. The purpose of this study was to compare the rule of 5 with trend-based analysis of global RNFL thickness over time for detecting glaucomatous progression.

Design:

Prospective cohort.

Subjects:

300 eyes of 210 glaucoma subjects followed for an average of 5.4 ± 1.5 years with a median of 11 (interquartile range: 7–14) visits.

Methods:

Trend-based analysis was performed by ordinary least-squares (OLS) linear regression of global RNFL thickness over time. For estimation of specificity, false positives were obtained by assessing for progression on series of randomly permutated follow-up visits for each eye, which removes any systematic trend over time. The specificity of trend-based analysis was matched to that of the rule of 5 to allow meaningful comparison of the “hit rate”, or the proportion of glaucoma eyes categorized as progressing at each time point, using the original sequence of visits.

Main Outcome Measures:

Comparison between hit rates of trend-analysis versus rule of 5 at matched specificity.

Results:

After 5 years, the simple rule of 5 identified 37.5% of eyes as progressing at a specificity of 81.1%. At the same specificity, the hit rate for trend-based analysis was significantly greater than that of the rule of 5 (62.9% vs. 37.5%; P<0.001). If the rule of 5 was required to be repeatable on a consecutive test, specificity improved to 93.4%, but hit rate decreased to 21.0%. At this higher specificity, trend-based analysis still had a significantly greater hit rate than the rule of 5 (47.4% vs. 21.0%, respectively; p<0.001).

Conclusion:

Trend-based analysis was superior to the simple rule of 5 for identifying progression in glaucoma eyes, and should be preferred as a method for longitudinal assessment of global SDOCT RNFL change over time.

Precis

The “rule of 5” detects a much lower proportion of glaucoma eyes exhibiting progressing on optical coherence tomography compared to trend-based analysis when matched for specificity.

Glaucoma is an optic neuropathy characterized by progressive, gradual loss of retinal ganglion cells that results in typical thinning of the retinal nerve fiber layer (RNFL) and neuroretinal rim.¹ In many patients, early structural changes may occur in the absence of detectable visual field loss on standard perimetry.^{2, 3} These early structural changes are important to detect as they may enable timely intervention to reduce intraocular pressure (IOP) with the goal of preventing vision loss from glaucoma.^{4, 5}

Over the past decade, spectral-domain optical coherence tomography (SDOCT) has emerged as a valid and reliable technique for assessing structural changes due to glaucoma.^{2, 4, 6} Although SDOCT can image several different topographical areas affected by the disease, circumpapillary measurements of RNFL thickness remain the most commonly used parameter in clinical practice. These measurements have been shown to be useful for detection of glaucomatous changes over time. Nevertheless, there is no consensus as to the best way to detect glaucomatous progression on SDOCT.

Prior studies have commonly defined glaucomatous RNFL progression on SDOCT by a statistically significant negative slope, a method also known as trend-based analysis.⁷ With trend-based analysis, the sequence of all tests available during follow-up is analyzed to detect whether there is a trend of worsening over time that is statistically significant and, therefore, exceeds variability. In clinical practice, however, many clinicians apply a simple “rule of 5”, in which a loss of 5 μm of global RNFL is considered evidence of glaucomatous progression when comparing a follow-up test with the baseline scan.^{8, 9} This general rule is a loose application derived from published data on the short-term variability of SDOCT technology. Leung et al. conducted repeated testing with Cirrus HD-OCT (Carl-Zeiss Meditec, Inc., Dublin, CA) within a few weeks and estimated the 95% confidence interval for reproducibility of average RNFL thickness to be 4.86μm in healthy subjects without glaucoma or other ocular pathology.¹⁰ In another study, the threshold limit of intervisit test-retest variability for SDOCT was 4.95 μm for Spectralis (Heidelberg Engineering, GmbH, Dossenheim, Germany) and 4.89 μm for Cirrus HD-OCT.¹¹

Thresholds of variability have led some to conclude that a loss of at least 5 μm should be considered diagnostic of progression.⁹ However, short-term variability thresholds may not accurately reflect longer term variability.¹² In addition, derivations of this rule do not take into account its repeated application for detecting change over time in the same patient, which can potentially lead to an increased number of false positives due to inflation of the type 1 error rate. In addition, by ignoring subtle changes that may only be detected by analyzing the whole sequence of tests available during follow-up, the rule of 5 may be less sensitive for glaucoma progression when compared to a trend-based analysis.

Delivery of an inappropriate diagnosis of glaucoma or glaucoma progression can have negative ramifications, not only because it may lead to intensification of unnecessary, expensive and invasive treatments, but also because such diagnoses can adversely impact patient quality of life.^{13, 14} On the other hand, failure to diagnose glaucoma progression can lead to delays in the timely and appropriate delivery of care to control IOP and decrease the rate of disease progression. The purpose of the present study was to investigate and compare the performance of the simple rule of 5 to a trend-based definition of progression.

METHODS

The data used in this study were drawn from participants in a prospective longitudinal cohort study designed to evaluate optic nerve structure and visual function in glaucoma. The study was approved by the Institutional Review Board and was conducted in accordance with the tenets of the Declaration of Helsinki and the Health Insurance Portability and Accountability Act.

All study subjects had to have open angles on gonioscopy and meet the following inclusion criteria. Subjects were initially diagnosed with glaucoma at the baseline visit on the basis of repeatable visual field defects on standard automated perimetry (SAP) using the 24–2 Swedish interactive threshold algorithm (SITA) Standard of the Humphrey Field Analyzer II, model 750 (Carl Zeiss Meditec, Inc., Dublin, CA, USA) and/or glaucomatous optic neuropathy on optic disc stereoscopic photographs. Repeatable visual field defects on SAP were defined as at least three consecutive abnormal SAP results in which the pattern standard deviation (PSD) p-value was <0.05, and/or the glaucoma hemifield test results were outside the range of normal limits. SAP results also had to meet reliability parameters including ≤33% fixation losses, and ≤15% false-positive errors or false-negative errors. Glaucomatous optic neuropathy was evaluated by masked assessment of stereophotographs and it was defined based on the presence of neuroretinal rim notching, excavation, or thinning, and/or characteristic RNFL defects.¹⁵

Subjects were excluded if they had any ocular or systemic disease that could affect the optic nerve or visual field, visual acuity worse than 20/40, spherical equivalent outside ±5.0 diopters, or cylinder correction outside ±3.0 diopters. Included subjects completed a total of at least 5 SDOCT tests, which were acquired approximately every 6 months over a minimum of 2 years.

Spectral Domain-Optical Coherence Tomography

SDOCT tests were acquired with the Spectralis SDOCT (software version 5.4.7.0, Heidelberg Engineering, GmbH, Germany) at approximately semi-annual intervals during follow-up. All subjects completed at least 5 reliable SDOCT tests over 2 years. All images were reviewed to ensure that the signal strength was sufficient (i.e. >15 dB), that the scan was centered, and that there were no artifacts or RNFL segmentation algorithm errors. Scans that were inverted, clipped or those that had coexistent retinal pathological abnormalities were excluded. The SDOCT parameter used in the study was the global RNFL thickness corresponding to the average of all RNFL thickness measurements acquired from a 3.45-mm circle centered on the optic disc.

Data analysis

In the present study, we compared the performance of the rule of 5 to that of trend-based analysis by ordinary least squares regression for detection of glaucomatous progression over time. Eyes were considered to have progressed by trend-based analysis if the slope was negative and statistically significantly different (p<0.05) from zero.¹⁶ This is the conventional definition for trend-based analysis of RNFL that has been applied in numerous studies⁷ and that is generally available in commercially available SDOCT software. Progression by the rule of 5 was considered to have occurred when the global RNFL thickness at follow-up was at least 5 μm thinner than the baseline RNFL thickness. We also evaluated whether the specificity of the rule of 5 could be improved by requiring that the difference from baseline be repeatable, i.e., two consecutive tests had to have a difference of at least 5 μm compared to the baseline.

For each method, the first two visits were averaged and considered as baseline.^{6, 17, 18} For each eye, we evaluated the presence of progression at each successive visit, starting with a minimum of 3 visits. For a meaningful comparison of the ability of trend-based analysis and rule of 5 to detect glaucomatous progression, the specificities of these criteria needed to be matched at each follow-up time point. For this purpose, we obtained estimates of false-positives (FPs) by assessing the presence of progression on series of randomly permutated follow-up visits for each glaucoma eye. The random permutation removes any systematic trend over time, providing a reasonable estimate of false positives that arise due to test-retest variability. For each eye, we obtained 5 series of random permutations of the follow-up visits.^{17, 19, 20}

As there is no ‘gold standard’ test for glaucoma progression to provide a ground truth, relative measures were used. Instead of sensitivity, the “hit rate”, or proportion of eyes identified as progressing at each time point, was calculated for each method (i.e, trend-based analysis vs. rule of 5). This approach has been adopted by previous studies in the literature.^21–24 Generalized estimating equations (GEE) were used to obtain 95% confidence intervals for the hit rate, and the total proportions were compared using a Wald test with robust standard errors from the GEE to account for repeated observations at the subject level. The overall agreement between the rule of 5 and trend-based analysis over time was evaluated by a Kappa statistic, with values < 0 as indicating no agreement, 0–0.20 slight, >0.20–0.40 fair, >0.40–0.60 moderate, >0.60–0.80 substantial, and >0.80 almost perfect agreement. All statistical analyses were performed with commercially available software (STATA, version 15.1, StataCorp, College Station, TX and R version 3.4.2).

RESULTS

A total of 300 eyes of 210 glaucoma subjects were included in the study. Mean follow-up time was 5.4 ± 1.5 years and a median number of 11 (interquartile range: 7 – 14) tests were available during follow-up. Table 1 summarizes the baseline demographic and clinical characteristics of study subjects.

Table 1.

Demographic and clinical characteristics of subjects included in the study.

	N = 300 eyes of 210 subjects
Baseline Age, years
Mean ± SD	68.2 ± 10.2
Median (IQR)	69.6 (60.8 – 75.0)
Sex, N (%)
Female	107/210 (51.0%)
Male	103/210 (49.0%)
Race, N (%)
African American	59/210 (28.1%)
Non-African American	151/210 (71.9%)
Baseline mean deviation, dB
Mean ± SD	−5.21 ± 5.80
Median (IQR)	−3.47 (−6.37 – −1.65)
Baseline RNFL thickness, μm
Mean ± SD	74.4 ± 17.0
Median (IQR)	73.5 (62.5 – 85.5)
Number of SDOCT scans,
Mean ± SD	11.2 ± 4.6
Median (IQR)	11 (7 – 14)
Follow-up time, years
Mean ± SD	5.4 ± 1.5
Median (IQR)	5.9 (4.5 – 6.6)

IQR = interquartile range; RNFL = retinal nerve fiber layer; SD = standard deviation; SDOCT = Spectral domain optical coherence tomography; dB = decibels; μm= microns

Table 2 provides hit rates for detection of glaucoma progression at matched specificities for the rule of 5 and trend-based analysis at each year during follow-up. At each year, hit rates obtained for the trend-based analysis were significantly greater than those for the rule of 5. As an example, after 5 years of semi-annual testing, the rule of 5 detected 37.5% (95% CI: 32.0% to 43.3%) of eyes as progressing at a specificity of 81.1%. When matched to this same specificity, the proportion of eyes progressing by trend-based analysis was 62.9% (95% CI: 56.8% to 68.6%), which was significantly greater than the hit rate of the rule of 5 (p<0.001). Figure 1 illustrates hit rates for each method during follow-up.

Table 2.

Percentage of glaucoma eyes exhibiting progression (hit rates) by trend-based analysis versus the rule of 5

	Matched Specificity (%)	Hit Rate (95% CI)	Hit Rate (95% CI)
Year		Rule of 5	Trend-based Analysis
1	84.4	14.1 (8.1, 23.4)	32.6 (23.9, 42.8)
2	80.8	25.1 (20.3, 30.7)	44.1 (38.0, 50.5)
3	80.4	31.3 (26.2, 36.9)	55.7 (49.5, 61.8)
4	80.8	29.9 (24.8, 35.6)	56.1 (49.9, 62.0)
5	81.1	37.5 (32.0, 43.3)	62.9 (56.8, 68.6)

Figure 1.

Percentage of glaucoma eyes exhibiting progression (hit rates) by trend-based analysis versus (A) the rule of 5 and (B) repeatable rule of 5 after matching on specificity [1- False Positives (FP)].

When we required the rule of 5 to be repeatable on two consecutive tests, the specificity improved to 93.4% at 5 years. The hit rate, however, decreased to 21.0% (95% CI: 16.6% to 26.1%) (Table 3). When trend-based analysis was matched to the same specificity, the hit rate was 47.4% (95% CI: 41.5% to 53.4%), which was significantly greater than that of the repeatable rule of 5 (p<0.001).

Table 3.

Percentage of glaucoma eyes exhibiting progression (hit rates) by trend-based analysis versus the repeatable rule of 5

	Matched Specificity (%)	Hit Rate (95% CI)	Hit Rate (95% CI)
Year		Repeatable Rule of 5	Trend-based Analysis
1	95.2	0.0	15.3 (9.0, 24.8)
2	93.7	11.2 (7.2, 17.2)	23.5 (18.9, 29.0)
3	93.1	21.4 (17.2, 26.4)	38.3 (32.7, 44.3)
4	93.2	20.2 (15.9, 25.3)	41.9 (36.2, 47.9)
5	93.4	21.0 (16.6, 26.1)	47.4 (41.5, 53.4)

Finally, we evaluated the agreement in eyes flagged as progressing by each definition of the rule of 5 and trend-based analysis over time (Supplemental Figure 2, available at http://www.ophthalmologyglaucoma.org). Overall the two methods had only fair to moderate agreement over time.

DISCUSSION

Misclassification of progression is a critical issue in the management of glaucoma. Receiving a glaucoma diagnosis can negatively impact a patient’s quality of life.^{13, 14} Moreover, inappropriate initiation or escalation of treatment may prove expensive and harmful to patients. On the other hand, failure to appropriately diagnose glaucoma progression can lead to a delay in care and result in ongoing vision loss. Despite the important role of SDOCT in monitoring patients with glaucoma, there is no agreed upon gold standard criterion for determining how much of a loss in RNFL constitutes true glaucomatous progression. Our study found that the commonly used rule of 5 tends to significantly underperform for the detection of glaucomatous progression if compared to trend-based analysis when the methods are matched for specificity.

Our results showed that over a 5-year period, trend-based analysis of global RNFL thickness over time detected a significantly larger number of glaucoma eyes as progressing compared to the rule of 5, when the methods were matched for the same specificity. By comparing hit rates for trend-based analysis and the rule of 5 as shown in Table 2, one can see that trend-based analysis detected, in absolute terms, from 18.5% to 26.2% more eyes as progressing at each year. In relative terms, hit rates for trend-based analysis were from 1.7 to 2.3 times greater than those of the rule of 5. One challenge in the analysis of longitudinal data is that of repeated testing, which can inflate the type 1 error rate. In clinical practice, as more tests are obtained over time, the likelihood of erroneously flagging an eye as progressing due to chance will increase. We found that the rule of 5 resulted in a false positive rate that ranged from 15.6% to 19.6% over a 5-year period (i.e. specificity of 80.4% to 84.4%); if applied in clinical practice, this could lead to a high cumulative proportion of eyes erroneously diagnosed as progressing over time. We then evaluated what happened when the rule of 5 was required to be repeatable on a consecutive test before it was flagged as progression. As expected, this approach resulted in a smaller proportion of false positives, improving the specificity of the rule of 5 from 81.1% to 93.4%. However, this improvement in specificity led to a trade-off in the hit rate, resulting in a lower proportion of eyes detected as progressing. Importantly, even at this increased specificity, trend-based analysis still had higher hit rates compared to the repeatable rule of 5, as showed in Table 3.

The general rule of 5 was developed based on published short-term test-retest variability thresholds of SDOCT global RNFL thickness.^10,11 Although conceptually appealing, we have previously demonstrated that the rule of 5 may lead to a high number of false positives if used to flag change between consecutive SDOCT visits.²⁵ Since the short-term variability thresholds of SDOCT RNFL measurements were determined using healthy subjects, they do not necessarily reflect either the short- or long-term variability in patients with glaucoma. The variability of RNFL measurements in glaucoma eyes tends to be higher than in healthy eyes. In addition, other factors, such as environmental and ocular conditions (e.g. media opacities due to dry eyes or cataract, pupil size) may result in increased long-term variability compared to the short-term one. In fact, a recent study by Urata et al.¹² showed that, in glaucomatous eyes, the long-term variability of RNFL thickness measurements was over two times greater than the short-term variability. Thus, event-based algorithms based on short-term variability data, of which the rule of 5 is a very simple version, may tend to overestimate progression over time.

The design of any study on glaucoma progression is limited by the lack of a perfect reference standard for detecting change in the disease. Given the lack of a standard, we used the hit rate, or proportion of eyes detected as progressing, as the metric to compare performances of the different approaches after they were matched on specificity. The specificity at each timepoint was determined by estimating the false positive proportion in randomly permuted visits of glaucoma eyes. Random permutations of visits are supposed to remove systematic trends in the measurements over time. Therefore, if progression is detected in randomly permutated visits, it should not generally be representative of “true change, therefore serving as a surrogate for false-positives. However, it should be noted that our comparisons of hit rates had to rely on the imperfect, albeit reasonable, assumption that whichever method “flags” more progression is likely to be the better one, given that the two methods were matched for specificity. Previous studies have also used this approach in the literature.^{21, 26, 27} One could argue that visual field progression could be used as a reference standard for progression. However, SDOCT and SAP have been shown to have a poor agreement for detecting progression, at least when global metrics are used.^{21, 28–30} Despite this fact, SDOCT changes have been shown to be clinically relevant, being predictive of future visual losses and also of decline in quality of life in glaucoma. Subsequent studies should attempt to investigate whether progression detected by the rule of 5 or trend-based analysis is associated with future changes in visual function.

Another drawback of the rule of 5 is that it does not account for declines in RNFL that may occur due to normal aging.²⁵ Such age-related loss can confound the detection of glaucoma progression on SDOCT.^{6, 16, 31–34} Wu et al.¹⁶ have proposed different adjustments that could be made to trend-base analysis in order to account for normal age-related loss in RNFL. For example, in one definition progression was defined by a significantly negative slope (p<0.05) with the magnitude of the estimated slope being more negative than the 5% lower limit of the controls.¹⁶ In this analysis, we did not adjust for age-related loss in trend-based analysis in order to be fair in the comparison with the rule of 5. Although in theory one could attempt to correct the rule of 5 for age by requiring the change in RNFL thickness to exceed 5 microns plus the expected change in normal variability using the lower 5% cut-off from healthy controls, this is not typically done in routine clinical care. Moreover, age-adjustment to the rule of 5 would largely defeat the practicality of the rule.

Our study had limitations. We used the global average RNFL to assess progression rather than sectorial measurements, which may be more sensitive for localized glaucomatous damage.³⁵ However, sectorial measurements may be more prone to variability without necessarily being better suited to the detection of longitudinal progression. In clinical practice, it may be best to incorporate highly specific statistical definitions for progression using the global RNFL with one’s clinical assessment of SDOCT for patterns of RNFL loss that are characteristic of glaucoma, so that one can improve the sensitivity of SDOCT for detecting glaucomatous progression while maintaining high specificity.³⁵ It should be noted that most eyes in our sample had mild or moderate glaucoma. This should be kept in mind in terms of the generalizability of the findings. Also, in our analyses, we graphed the total proportion of eyes that were progressing at each time point. However, over a lifetime, individual eyes with glaucoma may experience periods where they are actively progressing, followed by times of relative stability. Since all of the definitions in this study assessed progression with respect to the original baseline, they might not be able to distinguish successive periods of stability alternating with progression. Thus, after each period of progression has passed, the ‘baseline’ might need to be reset to reflect the most recent losses in RNFL.

CONCLUSIONS

In summary, this study showed that trend-based analysis was superior to the simple rule of 5 for identifying progression in glaucoma eyes, and should be preferred as a method for longitudinal assessment of global SDOCT RNFL change over time. It is important to emphasize, however, that given the lack of a perfect reference standard for progression in glaucoma, any statistical trend-based definitions or simple algorithms such as the rule of 5 should be applied with caution and interpreted in the setting of other clinical criteria before care is escalated.

Supplementary Material

1

Supplemental Figure 2: Plot of the Kappa score for the agreement in progression of the (A) rule of 5 or the (B) repeatable rule of 5 and trend-based analysis versus time. The agreement improves over time, but the Kappa score remains below 0.8 (where >0.8 represents good agreement).

Abbreviations:

FP

false positives

IOP

intraocular pressure

OLS

ordinary least squares

PSD

pattern standard deviation

RNFL

retinal nerve fiber layer

SDOCT

spectral domain-optical coherence tomography

SAP

standard automated perimetry

TP

true positives