Combined Probability Test for Sectoral Progression of the Circumpapillary Retinal Nerve Fiber Layer Thickness

Abstract

Purpose

Optical coherence tomography has become a widely used tool to assess structural changes at the optic nerve head and the peripapillary retina. Often, global analyses are supplemented with sectoral analyses, but it is unclear how to control specificity as trend analyses are conducted on a larger number of sectors. We introduce a random permutation analysis for a combined probability test of progression in circumpapillary retinal nerve fiber layer (cpRNFL) thickness applied to different number of sectors.

Methods

A series of seven cpRNFL scans were extracted for 428 eyes of 255 patients with glaucoma from the DIGS/ADAGES dataset. The combined probability test was run for 2^k sectors, where k = 0, ⋯, 8 in addition to the maximum possible number of pixels, 768. Positive rates were derived for specificity ranging from 100% to 85%.

Results

At 95% specificity, the positive rate for 768 pixels was 41% [37%, 46%]. The positive rates for global thickness, and for 12 sectors, were statistically significantly smaller (28% and 35%, respectively). Positive rates remained at the observed maximum until the number of sectors fell below 128.

Conclusions

The permutation of cpRNFL thickness profiles makes it possible to detect highly localized change in cpRNFL profiles from optical coherence tomography.

Translational Relevance

Glaucoma-related changes in the optic nerve fiber layer are often localized rather than global. Permutation analysis provides a framework to detect such changes without sacrificing specificity.

Introduction

If left untreated, glaucoma can lead to progressive optic nerve damage associated with irreversible loss of vision.^1–3 Although static automated perimetry remains the clinical standard for monitoring glaucoma progression,⁴ recent technological advancements have increased traction toward the use of optical coherence tomography (OCT), which provides objective quantification of structural changes in glaucoma.^5–7 Global circumpapillary retinal nerve fiber layer (cpRNFL) thickness is often used to quantify overall thickness and its rate of change, but analyses are also reported for sectors. Current commercially available OCT devices return average thicknesses for four, six, and 12 sectors along with the thickness profile along the temporal, superior, nasal, and inferior (TSNIT) retinal areas.

The use of sectors for diagnosis based on cpRNFL thickness can lead to a statistical artifact referred to as red disease in the clinical research literature.⁸ This artefact increases with the number of sectors and is a consequence of statistical inference with multiple comparisons which reduces specificity. A similar statistical artefact occurs for progression analysis.

Progression in cpRNFL thickness over time is assessed by the one-tailed t-test with the alternative hypothesis that linear regression slope is lower than 0 microns per year. For global cpRNFL thickness and a nominal significance level of 0.05, the specificity of the significance test is expected to be 95%. But, as the number of sectors increases, the specificity decreases until it reaches 0% at about 128 sectors as illustrated in Figure 1.

Figure 1.

Theoretical prediction for the effects of “multiple testing” on specificity as a function of the number of sectors. The expected specificity is (1 − 0.05)ⁿ, where n is the number of sectors and 0.05 is the significance level. Specificity decreases rapidly as the number of sectors increases. It falls far below tolerable limits with as few as four sectors.

One way to control the specificity is to apply a Bonferroni correction. For example, to achieve a specificity of 95%, we would divide the significance of 0.05 by the number of sectors. A more principled approach is to combine evidence of local progression into a statistic, such as the χ² test used in Fisher's combined probability test (Section 21.1, page 103 in Statistical Methods for Research Workers⁹). Both approaches are flawed, however, because they assume that the significant tests are mutually independent. This can be circumvented by using a nonparametric version of Fisher's approach. One such version based on random permutation analysis was previously developed to determine the statistical significance of deterioration within a series of visual fields.¹⁰ This statistical approach (Permutation of Pointwise Linear Regression, PoPLR) has been shown to improve sensitivity to progression over global indices in visual field progression analysis, without compromising specificity.^10,11 In this study, we introduce a random permutation analysis applied to cpRNFL scans and compare the results to a global progression analysis.

Methods

Datasets

The data included in this study were selected from the Diagnostic Innovations in Glaucoma Study and the African Descent and Glaucoma Evaluation Study (DIGS/ADAGES).^12,13 We selected eyes from patients with POAG who had series of seven or more circumpapillary OCT scans. For those eyes with more scans, we selected the first seven only. The scans were obtained with the Spectralis spectral-domain OCT (Heidelberg Engineering, Heidelberg, Germany) and had signal strength ≥15. When multiple reliable OCT scans were taken on the same day, their average was used. In total, 428 eyes from 255 patients met the inclusion criteria. At baseline these patients had a mean age (standard deviation) of 65.4 (9.9) years and an average cpRNFL thickness of 80.5 (16.4) µm. The mean follow-up time was 3.9 (0.8) years, corresponding to an average of 6.7 (0.4) months between visits. The DIGS/ADAGES studies adhered to the tenets of the Declaration of Helsinki for research involving human subjects and conformed to the Health Insurance Portability and Accountability Act. Informed consent was obtained from all participants.

OCT Data Processing

The OCT circumpapillary scans in the DIGS/ADAGES dataset consists of RNFL thickness at 768 pixels in a circle (diameter, 12°) centered on the optic nerve head. By convention, OCT circumpapillary scans begin at the horizontal line and travel counterclockwise along the temporal quadrant (315° to 45°) toward the superior quadrant (45° to 135°), then the nasal quadrant (135° to 225°), and then the inferior quadrant (225° to 315°) before returning to the temporal quadrant. Following this convention, the 768 pixels were averaged into different sectors. For example, for n = 1, the cpRNFL thicknesses were averaged over all 768 pixels into global cpRNFL thickness. For n = 4, the cpRNFL thicknesses were averaged in four sectors at every 192 pixels.

Sectoral Progression

Sectoral progression was identified by performing linear regression at each sector and obtaining the p-value for the one-tailed t-test for the slope with the alternative hypothesis that the slope is less than 0. The n p-values corresponding to the n sector linear regression significance tests were combined into Fisher's combined statistic $S=-2{\sum }_{i=1}^{n}log{p}_i$. Then random permutation analysis was used as a nonparametric version of the combined probability test. In short, the temporal order of each series of cpRNFL scans was randomly rearranged multiple times, and Fisher's S for each random permutation was recorded. From the set {s_p} of S values, the null distribution (no systematic change) was constructed and the aggregated p-value computed. More precisely, the p-value for the combined probability test for progression is the proportion of values in the set {s_p} obtained from the permuted series that are lower than the value obtained for the non-permuted series.

Statistical Analysis

First, we computed the positive rates (i.e., false-positive + true-positive rates) for the combined probability test for progression for 768 pixels at specificity levels from 100% to 85%. We then calculated the area under the partial receiver operating characteristic, subtracted the area under the curve for random chance from it and expressed this as a ratio of the maximum possible area. This relative area under the curve (rAUC) served as the principal outcome measure for the subsequent analysis. The rAUC for the maximum 768 pixels was compared against that for 360 sectors and for 2^k sectors, with k varying from 0 to 8.

An eye-specific report that includes the combined probability test was generated for sectoral progression analysis based on the statistical analysis described. All computations were performed in R.¹⁴

Results

Figure 2 shows the positive rates for sectoral progression of series of seven circumpapillary OCT scans at a significance level of 0.05 as a function of number of sectors using DIGS/ADAGES data and the specificity calculated from a dataset of 30 non-progressing series.¹⁵ The positive rate increased with number of sectors, but at the expense of a substantial decline in specificity.

Figure 2.

Positive rates versus specificity in sectoral progression for series of seven cpRNFL scans as a function of number of sectors. The upper panel shows the positive rates for progression at a significance level of 0.05 for different numbers of cpRNFL sectors, with 1 being statistically significant reduction in the overall thickness (one sector), 4 being significant reduction in thickness at any of four sectors, and so on up to 128 sectors. The positive rates in the upper panel were obtained for series in the DIGS/ADAGES dataset. The lower panel shows the specificity for the same cpRNFL sectors. The dashed lines represent the 95% specificity. The specificities in the lower panel were obtained for pseudo-series of non-progressing series. In short, 30 non-progressing series¹⁵ were used to generate 3000 series by randomly reordering the scans 100 times for each non-progressing series.

Figure 3 shows the positive rates for the combined probability test with random permutations for the minimum and maximum number of sectors, as well as for four, 12, and 128 sectors. The positive rate (and 95% confidence interval) at a specificity of 95% for the 768 pixels was 41% (37%, 46%). For global cpRNFL thickness (n = 1) it was 28%, and for four and 12 sectors it was 34% and 35%, respectively. These rates were statistically and clinically significantly smaller than the rate of 41% obtained with 768 pixels. For 128 sectors, the positive rate was 41%, and not statistically significantly different than for 768 pixels. Figure 4 shows the rAUC as a function of sectors used for the combined probability test of the cpRNFL thickness, from 1 sector to 768 pixels.

Figure 3.

Positive rates of the combined probability test for global, four, 12, and 128 sectors. The positive rates are compared against the positive rate for all 768 pixels. The region shadowed in light blue represents the 95% confidence bands for the positive rates for the combined probability test. The gray line represents the expected positive rates for a model that assigns progression by random chance.

Figure 4.

The rAUC as a function of the number of sectors. The region shadowed in gray represents the 95% confidence bands for the rAUC.

The maximum rAUC was 0.40 (0.35, 0.44). For 128 sectors, the rAUC was the same as for 768 pixels. For 64 sectors, the rAUC was about 5% smaller at 0.38.

To validate the combined probability test with 128 sectors, we computed the false-positive rates as a function of significance using the permuted retest dataset with 3000 non-progressing series used for Figure 2. The positive-rates curve coincided with the chance straight line except for sampling variability (see Supplementary Fig. S1).

Figure 5 shows an example of the results of sectoral RNFL progression and the combined probability test for a single eye. Supplementary Figures S2 and S3 show two additional examples with comments. Results for all eyes included in this study can be found in the Supplementary file octspa.pdf.

Figure 5.

Sectoral progression analysis for one eye. The top left panel shows a summary of the results, including the age at baseline, follow-up time, number and proportion of progressing sectors (at a significance level of 0.05), the probability score based on the probability levels 0.05, 0.02, and 0.01 and as defined as in Åsman et al.,¹⁶ and the number of arcs (two or more adjacent sectors with probability level 0.05 or below) along with the average number of sectors per arc. The top middle panel shows the distribution from the combined probability test. The red circle and vertical line represent the results for the observed series of seven OCT scans. In this example, the P value of progression was calculated at 0.002. The right top panel shows the sectors progressing at significance levels of 0.05 (orange), 0.02 (red), and 0.01 (dark red). The second row shows the seven temporal, superior, nasal, and inferior (TSNIT) profiles. The third and fourth rows show the result of sectoral linear regression, namely, baseline and rates of change. The sectors and color coding for progression in the fourth row correspond to those in the top right panel. The horizontal dashed line in the fourth row represents the average slope.

Discussion

Glaucoma-related changes in the retinal nerve fiber layer are often distinctly localized rather than global. This means that global change in cpRNFL thickness, and its statistical significance, may not be the most sensitive index of disease progression.^16–19 This is similar to visual field progression where global indices such as mean deviation provide a useful summary measure, but point-by-point analyses are ultimately more responsive to localized change.

Analysis of cpRNFL thickness in discrete sectors leads to the classic statistical issues of multiple comparisons and low specificity that have previously been identified in the literature. A combined probability test addresses this issue and makes it possible to investigate change at finer spatial scales at a constant false-positive rate (i.e. at a fixed specificity). A key finding of our study was that the typically small number of sectors (four, six, 12) used in current analyses does not achieve high sensitivity at detecting change. In the dataset that we investigated in this study, at least 128 RNFL sectors were necessary to achieve near-maximal sensitivity. The difference in positive rate from 28.3% (for a global analysis) to 40.7% (for 128 sectors) means an increase of 45% (see Fig. 3). This is a large difference that is likely to be clinically meaningful. Because our study only included one dataset for the analysis of progression, it will be important to replicate the analyses with datasets from other longitudinal studies.

The current study is consistent with and expands the findings of previous studies with sector-wise progression assessment^20,21 Specifically, we believe that our approach, based on permutation analysis, provides a simple way to control specificity that relies on few assumptions. One of these assumptions is, of course, that the data series are not affected by systematic changes in magnification that could translate into an artefactual progression signal over time.²² Artefactual change as distinct from disease-related progression as well as from random variability must always be considered as a potential confounder; this is a universal issue and not unique to our approach.

Strictly, our findings apply to the particular problem of estimating the statistical significance of change in cpRNFL thickness profiles. However, the method described in this paper can potentially be adapted to other OCT devices, with different scan patterns and resolutions. More importantly, one must also bear in mind that there are several other aspects—clinical, statistical, and technical—that are important to analyses of retinal changes in patients with glaucoma. Ultimately, such analyses must be evaluated not just in terms of specificity and sensitivity to change, but also for how effectively they support visualization and localization of change, and clinical interpretability and decision-making.

We chose a random permutation approach to compute the p-value. An alternative is to consider data correlations to estimate the degree of interdependence among significance tests and correct for multiple comparisons by adjusting the degrees of freedom for the χ² in Fisher's test. We tested different calculations (Nyholt, Li & Ji, Gao, Galway included in the R package poolr²³) with OCT and visual field data and found no method that universally succeeded at controlling specificity as well as random permutations. It is likely, however, that there are several potential modifications that could be made to further optimize our method. For example, one could model the relationships between neighboring sectors, or sectors in some proximity to each other. Similarly, one could also adjust for age-related slopes to account for RNFL thinning associated with natural aging, for issues arising from imaging artifacts, or for changes in the structure of the eye other than thinning due to disease progression, e.g., vessel displacements. Such “tweaks” could further enhance the performance of the approach we described and arguably that of any progression method. But the analysis reported in this article was intended as a proof-of-principle, and therefore we have not attempted to optimize the discriminatory performance of the technique. In a similar vein, future work is needed to assess whether our approach yields stronger correlations with changes in visual field and intraocular pressure over time. From a clinical standpoint, our approach has the potential to assist in determining which patients are truly progressing and would benefit from clinical or surgical interventions or from modifications in their medical treatment regimen.

Like other analyses of change, including that of global cpRNFL thickness, the performance of the combined probability test can be affected by inter-visit differences in eye position, magnification, rotation, and segmentation errors.^17,24 But, unlike with other analyses, these error sources will affect sensitivity only, and not specificity (as long as these errors remain random rather than systematic, as discussed above). The “guarantee” of fixed specificity, not just at the population level but at the level of individual patients, is a principal advantage of permutation analyses over other approaches of detecting change.²⁵ This makes permutation approaches highly attractive where tight control over specificity is paramount, particularly for monitoring patients with glaucoma who need to be examined continually for many years.