Abstract
Objective:
Assess the impact of OCT signal strength (SS) and artifact on retinal nerve fiber layer (RNFL) measurement reliability and understand whether glaucoma severity modifies this relationship.
Design:
Retrospective longitudinal cohort study.
Subjects:
2,992 OCTs from 474 eyes of 241 patients with glaucoma or glaucoma suspect status.
Methods:
We extracted mean RNFL thickness and SS and manually graded scans for artifact. To analyze the effect of SS and artifact on OCT reliability, we: 1) created a multilevel linear model using measured RNFL thickness values and demographic/clinical data to estimate the true (predicted) RNFL thickness 2) calculated model residuals (ΔRNFL) as our reliability measure 3) created a second multilevel linear model with splines and interaction terms which modeled overall and quadrant specific reliability (ΔRNFL) as the outcome, using SS and artifact as predictors.
Main Outcome:
Impact of SS and artifact on ΔRNFL.
Results:
For SS between 10 and 3, the impact of decreases in SS on OCT reliability is modest (−0.67 to −1.25 ΔRNFL per one-point decrease in SS, p<0.05). But below 3, changes in SS have a large impact on reliability (−15.70 to −16.34 ΔRNFL per one-point decrease in SS, p<0.05). At SS between 10 and 3, decreases in SS tend to have a larger impact on reliability in eyes with severe glaucoma (−1.25 per one-point decrease in SS, p<0.05) compared to eyes with mild or moderate glaucoma (−.67 to −0.75 per one-point decrease in SS, p<0.05)
The presence of artifact has a significant impact on OCT reliability independent of the effects of SS (−4.76 ΔRNFL, p<0.05). Artifact affects reliability solely in the quadrant in which it occurs, with artifact in one quadrant showing no impact on ΔRNFL in the opposite quadrant (p>0.05).
Conclusions:
SS decreases down to 3 have relatively mild impacts on OCT reliability. Below 3, the impact of further decreases in SS on reliability are substantial. The effect of SS on reliability is greater in severe glaucoma. Artifacts result in a decrease in reliability independent of the effect of SS. We propose evidence-based guidelines to guide physicians on whether to trust the results of an OCT scan.
Precis:
Decreases in signal strength and artifact independently decrease reliability of peripapillary OCT scans. The impact signal strength decrements varies based on the absolute value of signal strength as well as severity of disease.
Introduction:
Peripapillary measurement of the retinal nerve fiber layer (RNFL) by optical coherence tomography (OCT) has widespread clinical use including the diagnosis and monitoring of glaucomatous and non-glaucomatous optic neuropathies. Many clinical decisions, especially related to initiation or escalation of therapy for glaucoma, hinge on whether the optic nerve OCT shows abnormality in, or worsening of, RNFL thickness. In order for the clinician to trust that the data used to make such a decision are accurate, he or she must trust that the OCT image from which those data were derived is reliable.
OCT reliability can be judged by multiple criteria and two have been studied extensively: signal strength (SS) and the presence of artifact (i.e., image segmentation or acquisition error). Multiple studies have shown that decreases in SS result in decreases in measured RNFL thickness.1-5 Other studies have demonstrated that artifact occurs more often in scans with lower SS.5,6 Although this work has confirmed the importance of these two measures, there are many questions that remain unanswered. For instance, we do not know whether the impact of decreases in SS is similar across the range of values of SS (i.e. is the impact of a decrease in SS from 10 to 7 the same as that of a decrease in SS from 5 to 2?). Additionally, we do not understand whether the stage of glaucoma (mild, moderate or severe) modifies the relationship between SS and reliability of a OCT. Furthermore, we do not know whether the presence of artifact and the level of SS have an independent impact on OCT reliability or if the impact of artifact on reliability is local (i.e. only affects quadrant that it is in) or global (affects the whole scan).
Recently, we refined and expanded a statistical method for assessing reliability of visual field testing7 initially introduced by Junoy-Montolio et al.8 In this study, we apply this method to assess the factors influencing OCT reliability and address many of the unanswered questions above. This approach predicts the values of the variable of interest (in our case RNFL thickness) based on a large array of clinical and imaging measures over time. Reliability (ΔRNFL) is calculated as the difference between the predicted RNFL thickness (determined through advanced modeling) and the RNFL thickness measured by the OCT on a given date. The factors that influence ΔRNFL can then be identified using statistical modeling techniques. The advantage to this approach is that it allows for robust statistical analysis of the factors (SS, artifact) that affect OCT reliability across their range of values and how the relationship between these values is modified by stages of glaucoma. Such an analysis allows us to provide clinicians with evidence-based guidelines on how to judge the utility of an OCT scan to determine presence or worsening of disease.
Methods:
The study protocol was approved by the Johns Hopkins University School of Medicine institutional review board and adhered to the tenets of the Declaration of Helsinki.
Study Population and Data Collection
We included all glaucoma patients aged 57 and older involved in a longitudinal study assessing the functional impact of glaucoma.9 Informed consent was obtained from all participants. All patients included in this study had multiple OCT scans done on the Cirrus HDOCT 4000 (Carl Zeiss Meditec Inc., Dublin, CA, USA) between 5/2011 and 6/2018. All imaged eyes were included in analyses. As the current study was designed to evaluate the effect SS and artifact on reliability, no OCTs were excluded because of poor SS or presence of artifact. A chart review was performed to determine patient age, sex, race and the additional variables presented in Supplementary Table 1. Glaucoma stage was defined based on mean deviation (MD) of the visual field taken at or closest to the baseline OCT scan. MD>−6 was defined as mild /no disease, −12<MD≤−6 as moderate disease, and ≤−12 as severe disease.
Analysis of OCT Images
SS as well as all numeric RNFL parameters were automatically exported from the Cirrus OCT machine. All OCT scans were graded by 2 trained graders for artifact. Artifact was defined by black areas (0-micron RNFL thickness) within the RNFL thickness map circle and/or any irregular contours or dropouts of the red and purple segmentation lines on the tomograms (Figure 1). Graders also noted which quadrant(s) (superior, temporal, inferior, nasal) the artifact(s) occurred in. The 2 graders completed a training session with a senior glaucoma specialist involved in the study (PR) and were required to grade 50 training OCTs before moving onto grading OCT scans required for the study. For scans in the training set, percentage agreement between the senior glaucoma specialist and the trained graders was >95% and 90% of all artifacts in the training set were correctly identified by the graders.
Figure 1.
Example peripapillary OCT scan demonstrating artifact (red arrows). C/D = cup-to-disc; INF = inferior; NAS = nasal; OD = right eye; ONH = optic nerve head; OS = left eye; RNFL = retinal nerve fiber layer; SUP = superior; TEMP = temporal.
Modeling of Reliability
Reliability of the RNFL thickness measurement was defined as the difference between observed and predicted mean RNFL values (RNFLobserved – RNFLpredicted, referred to as ΔRNFL). The effects of disease and OCT parameters on ΔRNFL was assessed using the three-step process summarized in Figure 2.
Figure 2.
Demonstration of how reliability of retinal nerve fiber layer (RNFL) thickness measurement (ΔRNFL) is calculated. Each red circle represents the mean RNFL thickness measured by optic nerve head OCT in a specific eye of a specific patient at a specific time. For each of these OCT scans, we estimated what the RNFL thickness should be (open blue circles) with mixed-effects models. The ΔRNFL is the difference between the measured and the predicted RNFL thickness.
First, predicted RNFL thickness was calculated for eligible OCTs in the database with linear mixed effects regression models. The dependent variable in this model was the mean RNFL thickness for each OCT test in the database, while the independent variables included the features described in Supplementary Table 1. A linear mixed effects regression model approach was employed to account for clustering between eyes within the same patient and OCTs done on the same eye. The model employed random intercepts and an unstructured variance-covariance matrix.
Second, ΔRNFL (ie. model residual) was calculated as a continuous, directional measure of reliability for each OCT test included in the study by subtracting the predicted mean RNFL thickness obtained from the mixed effects model above from the actual observed RNFL thickness for that OCT (RNFLobserved – RNFLpredicted).
Third, predictors of reliability (ΔRNFL) were identified with a mixed effects linear model employing random intercepts. In this final multivariate model, the dependent variable was ΔRNFL (representing reliability) and predictors that were used to explain ΔRNFL included SS and presence of artifact. Interaction terms between the severity of glaucoma at baseline and SS and AR were used to account for the fact that the effects of SS and artifact on ΔRNFL vary by the severity of glaucoma. Additionally, during the exploratory phase of the analysis (Figures 4), we noted an inflection point at a SS of 3, below which ΔRNFL decreases at a more rapid rate. Therefore, a spline term was added to account for this effect across the range of values of SS.
Figure 4.
Locally weighted scatterplot smoothing (LOWESS) curves demonstrating changes in reliability of retinal nerve fiber layer thickness measurement (ΔRNFL) across the range of signal strengths grouped by severity of glaucoma. Grey outlines represent 95% confidence intervals.
The above 3-step modeling process was repeated for quadrant-specific data and used to test the hypothesis that artifact affects ΔRNFL solely in the quadrant in which it occurs. The estimated regression coefficients in the results represent the effect of each factor on ΔRNFL assuming all other factors are held constant.
Derived regression coefficients were used to define the decrease in SS required to produce various degrees of unreliability at different stages of disease severity. Acceptable levels of ΔRNFL were defined as: (1) an absolute level (i.e. >5 microns or <−5 microns), or (2) a level defined relative to the typical range of ΔRNFL values found at that stage of disease. To accomplish the latter, the distribution of within-eye standard deviations (SDs) was first derived for ΔRNFL (SDΔRNFL) at each disease stage and stage-specific median SDΔRNFL values were defined. Standards for acceptable ΔRNFL related to the stage-specific SDΔRNFL values were then derived, i.e., requiring ΔRNFL to be less than the median SDΔRNFL at that level of disease severity.
Poor reliability may be the result of patient/ocular features which cannot be modified even with technician coaching. As such, we wanted to identify variables that predicted poor reliability, suggesting that repeat scanning may not be helpful. We chose factors that are readily available to most technicians at the time of the scan (age, race and refraction/axial length[AL]) to model the likelihood of poor SS and artifact. Linear and logistic regression models were used with SS and artifact, respectively, as the outcomes and age, race and AL as the predictors.
All statistical analyses were performed in R version 3.4.2 (R Foundation for Statistical Computing, Vienna, Austria).
Results:
Demographic, ocular and OCT characteristics
In total, we analyzed 2,992 OCTs from 474 eyes of 241 patients (Table 1). The mean age of our patient population was 70.5 years (SD: 7.6). Each eye had a mean of 6.3 scans performed between 5/2011 and 6/2018. The majority of eyes included in the study had mild disease (70.8%) and a diagnosis of primary open angle glaucoma (55.5%). The mean AL of the eyes included in this study was 24.7 (SD 1.7). Mean RNFL thickness across all scans of all eyes was 71.8 microns. Over 66.8% of scans had a signal > 7. SS < 3 was uncommon, with only 2.3% of scans demonstrating such values. Artifact was also rare, occurring in 4.1 % of all scans.
Table 1.
Demographics, Clinical and OCT Characteristics Clinical Characteristics of Study Subjects
| Subjects, n | 241 |
| Age, mean (SD) | 70.5 (7.6) |
| Females, n (%) | 122 (50.6) |
| African Americans, n (%) | 67 (27.8) |
| Eyes, n | 474 |
| Glaucoma Stage at Baseline | |
| Mild, n (%) | 320 (70.8) |
| Moderate, n (%) | 64 (14.2) |
| Severe, n (%) | 68 (15.0) |
| Glaucoma Type | |
| Glaucoma Suspect, (%) | 134 (28.2) |
| Primary Open Angle Glaucoma, (%) | 263 (55.5) |
| Angle Closure Glaucoma, (%) | 36 (7.6) |
| Other, (%) | 41 (8.6) |
| IOP at Baseline, mean (SD) | 14.6 (5.0) |
| Number of Medications at Baseline, mean (SD) | 1.1 (1.7) |
| Axial Length in mm, mean (SD) | 24.7 (1.7) |
| OCTs, n | 2,992 |
| RNFL Thickness, mean (SD, IQR) | 71.8 (15.5, 61-82) |
| Mild Glaucoma, mean (SD, IQR) | 74.9 (14.7, 65-85) |
| Moderate Glaucoma, mean (SD, IQR) | 64.0 (12.8, 56-70) |
| Severe Glaucoma, mean (SD, IQR) | 59.7 (12.9, 53-65) |
| Signal Strength | |
| 0-3, n (%) | 69 (2.3) |
| 4-6, n (%) | 923 (30.8) |
| 7-10, n (%) | 2,000 (66.8) |
| Segmentation Error | |
| Present, n (%) | 124 (4.1) |
| Absent, n (%) | 2,868 (95.9) |
OCT: Ocular Coherence Tomography, SD: Standard Deviation, IOP: Intraocular Pressure, IQR: Interquartile Range, RNFL: Retinal Nerve Fiber Layer
Patients with severe glaucoma had, on average, 0.42 lower SS than patients with mild glaucoma (p<0.05). Scans with artifact had an average of 0.40 lower SS values than scans without artifact (p<0.05). However, the distribution of SS was broad in scans with artifact and some of these scans had high SS values (Figure 3).
Figure 3.
Density plots showing distribution of signal strength values based on severity of glaucoma (colors) and the (bottom) presence or (top) absence of artifact. The vertical lines demonstrate the mean signal strength for each density plot. Note that the means in the moderate and severe distributions overlap, so only 2 vertical lines appear in each panel.
Factors predicting reliability:
In both adjusted and unadjusted analyses, measured RNFL thickness is lower than expected RNFL thickness for lower SS values (Table 2; Figures 4 and 5). The impact of SS on ΔRNFL varies across the range of SS values. In the final adjusted model, decreases in SS between 10 and 3 tend to have modest impacts (0.67 to 1.25 microns decrease in ΔRNFL per one-point decrease in SS). However, changes in SS below 3 tend to have dramatic impacts (15.70 to 16.34 microns decrease in DRNFL per one-point decrease in SS). (Table 2; Figures 4 and 5)
Table 2.
Effects of OCT Parameters on ΔRNFL (Observed – Predicted) Stratified by Severity of Glaucoma.
| Change in ΔRNFL Thickness in Microns (Observed – Predicted) for 1 Point Change in SS |
||
|---|---|---|
| Severity of Glaucoma |
SS 10 to 3 (95% CI) | SS 3 to 0 (95% CI) |
| Mild | −0.67 (−0.92 to −0.41) * | −16.34 (−18.47 to −14.20) * |
| Moderate | −0.75 (−1.37 to −0.14) * | −15.70 (−17.96 to −13.44) * |
| Severe | −1.25 (−1.97 to −0.52) * | −16.03 (−18.34 to −13.71) * |
| Change in ΔRNFL Thickness in Microns (Observed – Predicted) if Artifact is Present (95% CI) |
||
| All Severities | −4.76 (−6.35 to −3.17) * | |
| Location of Artifact |
Change in Quadrant ΔRNFL Thickness in Microns (Observed – Predicted) when Artifact Present in Opposite Quadrant |
|
| Superior | −3.64, (−7.41 to 0.115) in inferior ΔRNFL | |
| Inferior | 4.04 (−0.61 to 8.6) in superior ΔRNFL | |
OCT: Optical Coherence Tomography, RNFL: Retinal Nerve Fiber Layer, SS: signal strength, CI: confidence interval,
- significant finding.
Figure 5.
Locally weighted scatterplot smoothing (LOWESS) curves demonstrating changes in reliability of retinal nerve fiber layer thickness measurement (ΔRNFL) across the range of signal strengths grouped by the presence or absence of artifact. Grey outlines represent 95% confidence intervals.
The relationship between SS and DRNFL for SS > 3 varies based on stage of glaucoma. Patients with severe glaucoma demonstrated a higher ΔRNFL decrement per one-point decrease in SS compared to patients with mild or moderate disease (1.25 microns lower ΔRNFL per one-point decrement in SS compared to 0.67 to 0.75 micron lower ΔRNFL per one-point decrement in SS respectively). However, once SS falls below 3, all three disease stages tend to have similar rates of ΔRNFL loss per unit of SS loss (Table 2 and Figure 4).
In multivariable models (Table 2), the presence of artifact also resulted in lower ΔRNFL (i.e., lower measured than expected thickness) independent of the value of SS. Figure 5 demonstrates that scans with artifact and high SS (>7) almost always have a decreased ΔRNFL, whereas scans without artifact and SS>7 tend to have a ΔRNFL close to zero. Additionally, the effects of isolated artifacts appear to be confined in the quadrant in which the artifact occurred. Specifically, Table 2 demonstrates that artifact occurring in the superior quadrant does not affect ΔRNFL in the inferior quadrant and vice versa.
Amounts of SS decrease required to produce significant changes in ΔRNFL
Significant changes in ΔRNFL were defined as either: 1) a 5-micron change in ΔRNFL 2) a median within eye SD of ΔRNFL change. The amount of SS decrease needed to produce these changes in ΔRNFL varies by disease stage and the presence of artifact (Table 3). In order to produce a 5 micron decrease in measured RNFL thickness when artifact is not present, SS must decrease by 7.02 points in mild disease, 5.68 points in moderate disease and 4.00 points in severe disease. However, when artifact is present, less than a one micron change in SS is needed to produce similar changes across all disease severities. We found the median within eye SD of ΔRNFL to be 3.21 microns and therefore correspondingly lower amounts of changes in SS were required to produce a within eye SD change in the absence of artifact. However, when artifact was present a less than 1 micron change in SS is required to produce a within eye SD change in ΔRNFL.
Table 3.
Amount of SS decrease needed to produce a 5 micron or one median within eye SD (3.21 microns) decrease in ΔRNFL (Observed – Predicted) stratified by severity of glaucoma and the presence or absence of artifact.
| Artifact | Severity of Glaucoma | Amount of Signal Strength Decrease Needed to Produce a 5 Micron Change in ΔRNFL |
Amount of Signal Strength Decrease Needed to Produce the Median Within-Eye Standard Deviation of ΔRNFL |
|---|---|---|---|
| Absent | Mild | 7.02 | 4.79 |
| Moderate | 5.68 | 4.28 | |
| Severe | 4.00 | 2.5 | |
| Present | Mild | <1 | <1 |
| Moderate | <1 | <1 | |
| Severe | <1 | <1 |
SS: Signal Strength, RNFL: Retinal Nerve Fiber Layer
Factors influencing level of SS and presence of artifact:
Increasing age was associated with a decrease in SS (β=−0.43 in SS per 10-year increment in age [95% CI: −0.61 to −0.26]). Older age was also associated with a higher odds of artifact, though this association was not statistically significant (β=1.37 increase in odds of artifact per 10 year increment in age [95% CI: 0.91 to 2.10]) (Table 4). Eyes with longer AL demonstrated lower SS (β=−0.17 point change in SS per one mm increase in AL [95% CI: −0.23 to −0.11]) and a higher odds of artifact (1.40 higher odds of artifact per one mm increase in AL [95% CI: 1.19 to 1.64]) (Table 4). African American race demonstrated lower SS (β=−0.28 point change in SS for African American race compared to others [95% CI: −0.28 to −0.0.02]) and a higher odds of artifact (2.10 higher odds of artifact for African American race compared to others [95% CI: 1.13 to 2.10]) (Table 4).
Table 4.
Factors Affecting Signal Strength and Presence of Artifact.
| Change in Signal Strength (95% CI) |
Odds of Artifact (95% CI) |
|
|---|---|---|
| Increase in Age by 10 Years | −0.43 (−0.61 to −0.26) * | 1.37 (0.91 to 2.10) |
| Increase in Axial Length by 1 mm | −0.17 (−0.24 to −0.11) * | 1.40 (1.19 to 1.64) * |
| African American Race | −0.28 (−0.28 to −0.01) * | 2.10 (1.13 to 3.94) * |
- significant finding.
Discussion:
In this study, we used a robust modeling approach to assess the impact of SS and artifact on the reliability of peripapillary OCT. Our results demonstrate that decreases in SS are associated with decreased measured RNFL thickness compared to predicted RNFL thickness (i.e., decreased ΔRNFL). Of note, the strength of this association varied substantially across the range of SS values, with decreases in SS between 10 and 3 having modest impacts on ΔRNFL while decreases below 3 having significant impacts on ΔRNFL. Additionally, decreases in SS tend to have a larger impact in eyes with severe glaucoma as compared to eyes with mild or moderate glaucoma. Thus, clinicians should be wary of labeling scans as having worsened if the SS has fallen, especially if the eye has severe glaucoma as the true value of RNFL thickness is likely higher the value measured by the OCT. Not surprisingly, we found that the presence of artifact is associated with a large decrease in ΔRNFL and this effect is independent of the effect of SS. Of note, the impact of artifact was confined in the quadrant in which the artifact occurs and did not appear to reduce reliability in other areas of the OCT. Thus, clinicians should be careful to check all quadrants for artifact even if a scan has good SS and should only discount the RNFL measurements in the quadrant in which an artifact occurs.
Our finding that modest decreases in measured RNFL thickness occur when the SS decreases but remains above 3 is mirrored in several other studies. Russel et al. measured multiple RNFL thickness values on the same patient after placing different strengths of diffusion filters in front of subjects’ eyes to mimic degradation in SS values to a lower limit of 3 or 4.3 They report a beta coefficient in their regression model of RNFL thickness to SS of 1.03 which is in line with the regression coefficients reported in Table 2. Vizzeri et al. have reported slightly stronger association between measured RNFL thickness and SS (Beta coefficient 1.98-2.88).2 However, in their study, only scans without artifact and with SS>7 were included which may explains such variations. These results emphasize that changes in SS that occur above a value of 3 have modest impact on RNFL measurements. Our work expands on these studies by also demonstrating that the impact of such SS changes varies by severity.
Prior studies support our finding that falling below a SS of 3 may result in more significant impact on the reliability of the measurements. Russel et al. found that the coefficient of variation for scans was high below a value of 3 or 43 and Zhang et al. also report that the variability of scans becomes greater below on SSI of 37 (on a scale of 0-100) on the RTVue OCT machine.10 However, although both papers assesses variability, neither of these studies modeled what happens to RNFL thickness measurements at SS below 3. Our data show that while scans with SS below 3 are rare in clinical practice (2.3% of scans in our data set), the impact of SS on measured RNFL thickness when falling below this value is large with each one-point change in SS resulting in a 15 to 16 micron drop in ΔRNFL. This is likely because there are much larger variations in RNFL measurements below 3, these variations tend to skew in the negative direction rather than being distributed equally with positive and negative values. This negative bias with increased variability of measurement forces very high drops in measured ΔRNFL values per change in SS below 3.
Eyes with more severe glaucoma at baseline are more affected by decreases in SS than eyes with mild or moderate glaucoma. Wu et al.1 reported that the relationship between SS and measured RNFL thickness was not modified when RNFL thickness was below or above 90 microns. We defined disease severity based on visual field criteria which is supported by published literature.11,12 In our dataset, mean RNFL thickness was 71.8 microns with mild, moderate and severe glaucoma patients having mean RNFL values of 74.9, 64.0 and 59.7 microns respectively. Using our approach, we found that RNFL measurements in eyes with severe glaucoma tend to be most affected by decreases in SS. This may be due to an increase in the variability of RNFL measurements in severe disease. Although Table 1 shows that, SD of RNFL values in eyes with severe disease is similar to eyes with moderate disease, the increased variability may be masked by the fact that there is floor RNFL value which on the Cirrus machine tends to be around 50 microns.13-17 Indeed, if we divide the SD of RNFL values by the inter-quartile range (an estimate of the distribution of a variable), we find that eyes with severe disease tend to have the highest variability across their range (data not shown). This argument is also supported by prior work that shows increased variability of RNFL measurements in eyes with severe glaucoma on visual field testing.15
Our data show that artifact results in a decrease in ΔRNFL independent of the effect of SS and this effect is a local one (i.e., only the quadrant of the scan with the artifact is affected). Previous groups have shown that artifacts tend to become more common with lower SS values4,6 but did not study the impact of artifact on RNFL measurements. These results emphasize that clinicians should diligently assess all scans for the presence of artifact, even those with good SS. As artifacts reduce ΔRNFL by 4.76 microns, the presence of artifact is much more impactful on RNFL measurements than a one-point decrease in SS (when SS is above 3). Indeed, when artifact is present a less than one-point decrease in SS can result in a 5 micron decrease in ΔRNFL compared to a 4 to 7-point drop in SS required to produce the same decrease when artifact is absent. Five micron decreases are beyond the normal test to test variation in RNFL measurements in OCT 18 and may mistakenly suggest progression or presence of disease.
Eyes of older patients, African American patients and eyes with long AL are associated with lower SS values. Additionally, the odds of artifact are increased in eyes of African Americans and eyes with longer AL. Prior work has shown that older age19,20 and increasing AL19 are associated with decreased SS. Prior work also found that cataract is associated with lower SS19-21. To our knowledge this is the first study that has identified African American race as a predictor of worse SS and higher odds of artifact. In this study, we chose to focus on age, race and AL alone as a technician performing OCT on a patient has ready access to these pieces of information and can decide not to repeat scans with low SS or artifact in older, African American patients with higher AL.
Our study has some limitations. First, our measure of reliability (ΔRNFL) is based on a predicted RNFL value as there is no way to know the “true” RNFL value. However, our approach of using predicted values to create a measure of reliability has been validated in multiple prior studies. 7,8 Another approach to measuring reliability would be to perform repeated measures on the same patients as done in some prior work,3,10 but this would be an impractical way to generate the large and relatively generalizable dataset needed to perform the robust statistical analysis we have completed in this study. Second, recent work22 has demonstrated that the two step modeling approach may result in biased covariate estimates that may be minimized with a one-step approach. However, in our work we are interested in the marginal impact of OCT reliability metrics on residuals (ΔRNFL) rather than the conditional impact of OCT reliability on RNFL thickness, therefore we feel that the modeling approach used in this study is the optimal approach to answer the main questions posed in this manuscript. Nonetheless, a sensitivity analysis using a one step and modified two step approach did not change the major inferences of our statistical analysis. Additionally, we found adding that SS and artifact as covariates to the first stage of our model to predict RNFL thickness did not significantly improve the root mean square error of the prediction (results not shown). Third, the generalizability of our work may be limited to adult patients in a glaucoma clinic as prior work has demonstrated less robust relationships in children whose mean RNFL values are closer to 100 microns23. Further work will need to be done to assess the effects of artifact and SS on reliability in a variety of patient populations and across various OCT machines.
In summary, our work provides clinicians with the following evidence-based guidelines to assess the reliability of OCT scans:
Always check for decreased SS values and the presence of artifact, as presence of either may independently decrease measured RNFL values.
SS is associated with lower measured RNFL thickness values, particularly when SS falls below 3. Above a value of 3, variation in SS has only a modest impact.
Eyes with severe glaucoma (MD worse than −12) are most affected by lower SS.
The presence of artifact results in large decreases in measured RNFL thickness but only has a local effect (i.e., one should not disregard the measurements in an entire scan if artifact only affects one quadrant).
We encourage all clinicians using OCT scans to utilize the above criteria to judge the reliability of the test. By doing so, we anticipate better assessment of the information contained in an OCT and ultimately better clinical decision making.
Supplementary Material
Acknowledgments
Financial Support: NIH Grant 1R01EY022976-01, AGS MAPS Award
Footnotes
No conflicting relationship exists for any author
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
References:
- 1.Wu Z, Huang J, Dustin L, Sadda SR. Signal strength is an important determinant of accuracy of nerve fiber layer thickness measurement by optical coherence tomography. J Glaucoma 2009;18:213–216. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Vizzeri G, Bowd C, Medeiros FA, et al. Effect of signal strength and improper alignment on the variability of stratus optical coherence tomography retinal nerve fiber layer thickness measurements. Am J Ophthalmol 2009;148:249–255.e1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Russell DJ, Fallah S, Loer CJ, Riffenburgh RH. A comprehensive model for correcting RNFL readings of varying signal strengths in cirrus optical coherence tomography. Invest Ophthalmol Vis Sci 2014;55:7297–7302. [DOI] [PubMed] [Google Scholar]
- 4.Cheung CYL, Leung CKS, Lin D, et al. Relationship between retinal nerve fiber layer measurement and signal strength in optical coherence tomography. Ophthalmology 2008;115:1347–1351, 1351.e1–2. [DOI] [PubMed] [Google Scholar]
- 5.Cheung CY, Chan N, Leung CK. Retinal Nerve Fiber Layer Imaging with Spectral-Domain Optical Coherence Tomography: Impact of Signal Strength on Analysis of the RNFL Map. Asia-Pac J Ophthalmol Phila Pa 2012;1:19–23. [DOI] [PubMed] [Google Scholar]
- 6.Asrani S, Edghill B, Gupta Y, Meerhoff G. Optical coherence tomography errors in glaucoma. J Glaucoma 2010;19:237–242. [DOI] [PubMed] [Google Scholar]
- 7.Yohannan J, Wang J, Brown J, et al. Evidence-based Criteria for Assessment of Visual Field Reliability. Ophthalmology 2017;124:1612–1620. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Junoy Montolio FG, Wesselink C, Gordijn M, Jansonius NM. Factors that influence standard automated perimetry test results in glaucoma: test reliability, technician experience, time of day, and season. Invest Ophthalmol Vis Sci 2012;53:7010–7017. [DOI] [PubMed] [Google Scholar]
- 9.Ramulu PY, Mihailovic A, West SK, et al. Predictors of Falls per Step and Falls per Year At and Away From Home in Glaucoma. Am J Ophthalmol 2019;200:169–178. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Zhang X, Iverson SM, Tan O, Huang D. Effect of Signal Intensity on Measurement of Ganglion Cell Complex and Retinal Nerve Fiber Layer Scans in Fourier-Domain Optical Coherence Tomography. Transl Vis Sci Technol 2015;4:7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.MD EH, MD RKPI, MD DRA. Clinical Decisions In Glaucoma. 1 edition St. Louis, Mo: Mosby; 1993. [Google Scholar]
- 12.Chakravarti T Assessing Precision of Hodapp-Parrish-Anderson Criteria for Staging Early Glaucomatous Damage in an Ocular Hypertension Cohort: A Retrospective Study. Asia-Pac J Ophthalmol Phila Pa 2017;6:21–27. [DOI] [PubMed] [Google Scholar]
- 13.Hood DC, Kardon RH. A framework for comparing structural and functional measures of glaucomatous damage. Prog Retin Eye Res 2007;26:688–710. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Hood DC, Anderson SC, Wall M, Kardon RH. Structure versus function in glaucoma: an application of a linear model. Invest Ophthalmol Vis Sci 2007;48:3662–3668. [DOI] [PubMed] [Google Scholar]
- 15.Mwanza J-C, Budenz DL, Warren JL, et al. Retinal nerve fibre layer thickness floor and corresponding functional loss in glaucoma. Br J Ophthalmol 2015;99:732–737. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Mwanza J-C, Kim HY, Budenz DL, et al. Residual and Dynamic Range of Retinal Nerve Fiber Layer Thickness in Glaucoma: Comparison of Three OCT Platforms. Invest Ophthalmol Vis Sci 2015;56:6344–6351. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Banister K, Boachie C, Bourne R, et al. Can Automated Imaging for Optic Disc and Retinal Nerve Fiber Layer Analysis Aid Glaucoma Detection? Ophthalmology 2016;123:930–938. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Wadhwani M, Bali SJ, Satyapal R, et al. Test-retest variability of retinal nerve fiber layer thickness and macular ganglion cell-inner plexiform layer thickness measurements using spectral-domain optical coherence tomography. J Glaucoma 2015;24:e109–115. [DOI] [PubMed] [Google Scholar]
- 19.Lee M, Zulauf M, Caprioli J. The influence of patient reliability on visual field outcome. Am J Ophthalmol 1994;117:756–761. [DOI] [PubMed] [Google Scholar]
- 20.Na JH, Sung KR, Lee Y. Factors associated with the signal strengths obtained by spectral domain optical coherence tomography. Korean J Ophthalmol KJO 2012;26:169–173. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.van Velthoven MEJ, van der Linden MH, de Smet MD, et al. Influence of cataract on optical coherence tomography image quality and retinal thickness. Br J Ophthalmol 2006;90:1259–1262. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Chen W, Hribar P, Melessa S. Incorrect Inferences When Using Residuals as Dependent Variables. J Account Res 2018;56:751–796. [Google Scholar]
- 23.Samarawickrama C, Pai A, Huynh SC, et al. Influence of OCT signal strength on macular, optic nerve head, and retinal nerve fiber layer parameters. Invest Ophthalmol Vis Sci 2010;51:4471–4475. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.