Predicting glaucoma development with longitudinal deep learning predictions from fundus photographs

Abstract

Purpose:

To assess whether longitudinal changes in a deep learning algorithm’s predictions of retinal nerve fiber layer (RNFL) thickness based on fundus photographs can predict future development of glaucomatous visual field defects.

Design:

Retrospective cohort study

Methods:

This study included 1072 eyes of 827 glaucoma-suspect patients with an average follow-up of 5.9 ± 3.8 years. All eyes had normal standard automated perimetry (SAP) at baseline. Additional SAP and fundus photographs were acquired throughout follow-up. Conversion to glaucoma was defined as repeatable glaucomatous defects on SAP. An OCT-trained deep learning algorithm (Machine to Machine, M2M) was used to predict RNFL thicknesses from fundus photographs. Joint longitudinal survival models were used to assess whether baseline and longitudinal change in M2M’s RNFL thickness estimates could predict development of visual field defects.

Results:

196 eyes (18%) converted to glaucoma during follow-up. The mean rate of change in M2M’s predicted RNFL thickness was −1.02 μm/y for conversors and −0.67 μm/y for non-conversors (P < 0.001). Baseline and rate of change of predicted RNFL thickness were significantly predictive of conversion to glaucoma, with hazard ratios in the multivariable model of 1.56 per 10 μm lower at baseline (95% CI, 1.33 – 1.82; P < 0.001) and 1.99 per 1 μm/y faster loss in thickness during follow-up (95% CI, 1.36 – 2.93; P < 0.001).

Conclusions:

Longitudinal changes in a deep learning algorithm’s predictions of RNFL thickness measurements based on fundus photographs can be used to predict risk of glaucoma conversion in eyes suspected of having the disease.

INTRODUCTION

Glaucoma is the leading cause of irreversible blindness in the world.^1–3 It is marked by the degeneration of retinal ganglion cells that leads to changes in the optic nerve head appearance⁴ along with an insidious progression of characteristic visual field defects.⁵ However, as glaucoma tends to initially affect the peripheral vision and may remain asymptomatic until late stages, most people are unaware of their disease^6,7 and present for care only after they have lost a substantial number of ganglion cells. From that point on, progression to functional disability may happen quickly in many cases. Thus, early diagnosis and treatment are paramount to preserve quality of life.

Optical coherence tomography (OCT) is an imaging technology capable of acquiring objective and quantitative measurements of neural loss in glaucoma.⁸ Previous studies found that retinal nerve fiber layer (RNFL) thickness measurements from OCT can be used to detect glaucomatous change prior to the development of irreversible visual field defects in glaucoma suspects.^9–13 However, despite its increasing availability, OCT devices are still rare in many clinical settings. In addition, the use of OCT for screening purposes or for monitoring glaucoma suspects outside of specialized centers is difficult, due to the prohibitive costs and the requirement for well-trained technicians.

One potential way to address this problem has been shown in deep learning research using fundus photographs, which are cheap and easy to acquire. A previous study has shown the feasibility of training a deep learning algorithm to obtain predictions of RNFL thickness measurements from simple fundus photos. The model used OCT-acquired RNFL thickness measurements as the ground-truth, in an approach named Machine to Machine (M2M).¹⁴ The M2M predictions from fundus photographs showed excellent correlation with actual OCT measurements.

Although the M2M model predictions showed good correspondence to OCT measurements, with predictions generally more accurate than human gradings,¹⁵ their value in predicting glaucoma development in suspected eyes has not yet been shown. Thus, in the present study, we apply the M2M model to fundus photographs of glaucoma suspect eyes followed over time until they developed the first evidence of glaucomatous visual field defects. In doing so we investigate whether M2M-predicted RNFL thicknesses can be used to detect glaucomatous damage before the appearance of visual field defects. In addition, we estimate how well baseline deep learning predictions of RNFL thickness as well as longitudinal changes predict risk of conversion to glaucoma.

METHODS

This study included a retrospective cohort extracted from the Duke Glaucoma Registry (DGR), a database of electronic medical records developed by the Vision, Imaging and Performance (VIP) Laboratory. The Duke University Institutional Review Board approved this study with a waiver of informed consent due to the retrospective nature of this work. The study was conducted in alignment with the tenets of the Declaration of Helsinki.

The database included data regarding patient demographics as well as patient diagnostic and procedure codes, ocular and medical history, best-corrected visual acuity, slit-lamp biomicroscopy, IOP measurement using the Goldmann applanation tonometry (GAT; Haag-Streit, Konig, Switzerland), central corneal thickness (CCT), and gonioscopy, It also contained data from standard automated perimetry (SAP; Humphrey Field Analyzer II; Carl Zeiss Meditec, Inc., Dublin, CA), stereoscopic optic disc photographs (Nidek 3DX; Nidek, Gamagori, Japan and Visupac FF-450, Carl-Zeiss Meditec, Dublin, CA). SAP was acquired using 24–2 Swedish interactive threshold algorithm (Carl Zeiss Meditec, Inc.). Patients who were deemed glaucoma suspects were those with a history of elevated intraocular pressure, suspicious appearance of the optic disc on slit-lamp fundus examination, or other risk factors for the disease. Patients were excluded if they had a history of ocular or systemic diseases that could affect the visual field or optic nerve. Therefore, tests performed after any diagnosis of retinal detachment, retinal or malignant choroidal tumors, non-glaucomatous disorders of the optical nerve and visual pathways, exudative, atrophic and late-stage dry age-related macular degeneration, amblyopia, uveitis, and venous or arterial retinal occlusion according to International Classification of Diseases (ICD) codes were excluded. In addition, tests performed after treatment with panretinal photocoagulation, according to Current Procedural Terminology (CPT) codes, were also excluded, as described previously.¹⁶

All eyes were required to have a normal SAP test within 3 months of their first fundus photograph. A “normal” visual field was defined based on mean deviation (MD) and pattern standard deviation (PSD) within 95% confidence limits and glaucoma hemifield test (GHT) results within normal limits. Visual fields were excluded for the following reasons: fixation losses of more than 33%, false-positive errors of more than 15%, and GHT classification of “Abnormally High Sensitivity.” The date of the first fundus photograph marked the study start point for each eye. From that point on, the SAP results for each visit were analyzed to assess for development of abnormal visual fields. Development of an abnormal visual field was defined as PSD with P < 0.05 or GHT results outside normal limits. Eyes that had at least 2 consecutive abnormal visual fields during follow-up were classified as “conversors,” which indicated glaucomatous conversion. The other eyes were classified as “non-conversors.” Each of the abnormal visual fields, as well as the repeatability of the defects, was confirmed on manual assessment by 2 masked reviewers with adjudication to resolve discrepancies.

For the conversors, we analyzed only the fundus photographs taken before the date of confirmation of conversion. For non-conversors, all available photographs were used. The study endpoint was the date of the second abnormal visual field (event) for the conversors and the date of last available visual field (censor) in follow up for the non-conversors.

Deep Learning M2M Predictions

The M2M deep learning model was used for this study. Specific details of the M2M model are listed elsewhere,¹⁴ but in brief, the M2M model is a convolutional neural network trained to predict global RNFL thickness measurements using color fundus photographs. The model was trained using a dataset of 26,509 pairs of fundus photographs and spectral domain (SD) OCT scans of 2,521 eyes of 1,391 patients. Importantly, none of the patients included in the development of the M2M model were part of the present study.

To train the model, fundus photographs were downsampled to 256 × 256 pixels, with pixel values scaled to range from 0 to 1. Data augmentation, including random rotations, flips, and lighting, was utilized to increase heterogeneity of input values. ResNet34, a residual deep neural network architecture pre-trained on the ImageNet dataset,¹⁷ was further tuned with our training set. Initially, the final 2 layers of ResNet34 were unfrozen. Then, all layers were unfrozen and fine-tuned with differential learning rates, with lower rates for the earlier layers and gradually higher rates for the later layers. Adam optimizer¹⁸ was used to train the model with minibatches of size 64. The cyclical learning method with stochastic gradient descent with restarts¹⁹ resulted in the best learning rate. A more extensive description of the development of M2M is detailed elsewhere.¹⁴ The algorithm, developed by the VIP Laboratory at Duke, has been previously demonstrated to estimate RNFL thickness from fundus photos with a mean absolute error of only 7.4 μm, and M2M’s RNFL predictions had a correlation of 0.832 with observed RNFL thicknesses from paired OCT scans.¹⁴ In a follow-up study,²⁰ the repeatability of the M2M predictions of RNFL thickness was investigated by measurements obtained from multiple fundus photographs acquired during the same day in the test sample. The intraclass correlation coefficient was 0.946, and the coefficient of variation was 3.2%, indicating high repeatability.²⁰

After being trained and validated, the M2M model was used to obtain RNFL thickness predictions for each fundus photo from our study cohort.

Statistical Analysis

The primary purpose of our study was to assess whether M2M’s predictions of baseline and longitudinal RNFL thicknesses from fundus photos could predict risk of conversion in glaucoma suspects. We used joint longitudinal survival models²¹ to investigate the relationship between baseline and longitudinal predictions of RNFL thicknesses and the risk of developing visual field loss in glaucoma suspects. Details of joint longitudinal survival models applied in this context have been described elsewhere.¹³ In brief, this analysis involves joint evaluation of a longitudinal submodel and a survival submodel, which are tied together by sharing random effects. The longitudinal submodel was composed of a linear mixed effects model, which accounts for measurement error by including a random error term and assumes random slopes and random intercepts, allowing different rates of change for each eye.

The survival submodel was used to quantify the strength of association between longitudinal change in predicted RNFL thickness and the risk for the event (developing visual field loss). It included a baseline hazard specified by the Weibull distribution and confounding factors including baseline age, sex, race, CCT, baseline PSD, and mean IOP obtained during follow-up. This model was estimated jointly with the longitudinal submodel and allowed an evaluation of the relationship between the M2M-predicted RNFL thicknesses and the risk for event. We were mainly interested in whether the slopes of change in M2M-predicted RNFL thicknesses were associated with risk of conversion, though the association of baseline M2M-predicted RNFL thicknesses and risk of conversion was evaluated as well.

To assess the proportion of variance in the outcome (survival time) explained by our predictors—in other words, the strength of the relationship between the predictors and outcome in the survival model—we used a modified R² index,²² which has been proposed as the best way to assess prognostic information of survival models.²³ P-values and confidence intervals for the modified R² indices were acquired by bootstrapping with 1000 replications.

All statistical analyses were performed with commercially available software (STATA, version 16; StataCorp LP, College Station, TX). The alpha level (type I error) was set at 0.05.

RESULTS

The study included 1072 eyes of 827 glaucoma suspect patients. Fundus photos acquired over a 30-year period were used, from 1986 to 2017; a mean number of 4.2 ± 2.7 fundus photos per eye were available during follow-up. A mean number of 9.8 ± 6.1 and 9.6 ± 5.4 visual fields were tested per eye during follow-up among the conversors and non-conversors, respectively. A total of 196 (18%) eyes converted to glaucoma during follow-up. The mean time of follow-up for the conversors and non-conversors was 4.4 ± 3.8 and 6.3 ± 3.7 years, respectively. The mean baseline predicted RNFL thickness from fundus photos was 88.7 ± 9.4 μm for conversors and 92.1 ± 7.2 μm for non-conversors (P < 0.001). Additional demographic data and clinical characteristics are detailed in Table 1.

Table 1.

Baseline and follow-up demographic and clinical characteristics of glaucoma suspect eyes that developed visual field defects over time (conversors) and those that did not (non-conversors).

Variables	Conversors, n = 196	Non-Conversors, n = 876	P
At baseline
Age, years	61.1 ± 12.9	54.4 ± 13.5	< 0.001a
Sex, % female	62%	63%	0.826b
Race
Caucasian %	56%	61%	0.243b
African American %	40%	36%	0.253b
SAP MD, dB	−1.35 ± 1.65	−0.16 ± 1.27	< 0.001c
Median [IQR]	−1.19 [−2.06, −0.26]	−0.04 [−1.00, 0.75]
SAP PSD, dB	2.00 ± 0.79	1.56 ± 0.41	< 0.001c
Median, [IQR]	1.80 [1.56, 2.14]	1.51 [1.32, 1.71]
CCT, μm	548 ± 41	549 ± 40	0.718a
M2M predicted average RNFL thickness, μm	88.7 ± 9.4	92.1 ± 7.2	< 0.001a
During follow-up
Average IOP, mmHg	16.8 ± 4.1	16.7 ± 3.7	0.755a
Average number of SAP tests	9.8 ± 6.1	9.6 ± 5.4	0.690a
Average number of fundus photos	3.7 ± 2.8	4.4 ± 2.7	< 0.001a
Average time of follow-up, years	4.4 ± 3.8	6.3 ± 3.7	< 0.001a

CCT = central corneal thickness; IOP: intraocular pressure, IQR: interquartile range, M2M: Machine to Machine, MD: mean deviation, PSD: pattern standard deviation, RNFL: retinal nerve fiber layer, SAP: standard automated perimetry

^aCompared using t-test

^bCompared using chi-square test

^cCompared using Mann-Whitney rank test

Figure 1 shows the individual trajectories of M2M predictions of RNFL thickness from fundus photographs for each conversor and non-conversor eye included in the study. Figure 2 shows the distribution of rates of change of the M2M predictions of RNFL thickness for conversors and non-conversors. Mean rate of change was significantly faster for conversors (−1.02 μm/year; 95% CI: [−1.05, −1.00]) than for non-conversors (−0.67 μm/year; 95% CI: [−0.68, −0.66]; P < 0.001). In univariable analysis, each 10 μm thinner predicted baseline RNFL thickness was associated with a 71% increase in the hazard of developing visual field defects during follow-up (HR: 1.71; 95% CI: 1.48 to 1.98; P < 0.001). Each 1 μm/year faster decrease in predicted RNFL thickness was associated with approximately 2-fold increase in the hazard of developing visual field loss (HR: 2.10; 95% CI: 1.44 to 3.06; P < 0.001). The univariable model had a modified R² index of 34% (95% CI: 25% to 44%). After adjustments for baseline age, sex, race, CCT, baseline PSD, and mean IOP, each 10 μm thinner predicted baseline RNFL thickness was associated with a 56% increase in hazard (HR: 1.56; 95% CI: 1.33 to 1.82; P < 0.001), and each 1 μm/year faster decrease in predicted RNFL thickness was associated with a 99% increase in hazard (HR: 1.99; 95% CI: 1.36 to 2.93; P < 0.001). The multivariable model had a modified R² index of 51% (95% CI: 41% to 62%). The hazard ratios for the main effects and covariates of the multivariable joint longitudinal survival models are shown in Table 2.

Figure 1.

Trajectory of Machine to Machine (M2M) predicted retinal nerve fiber layer (RNFL) thicknesses from fundus photographs over time (longitudinal response) for non-conversors and conversors.

Figure 2.

Distribution of rates of change in Machine to Machine (M2M) predicted retinal nerve fiber layer (RNFL) thickness from fundus photographs in conversors and non-conversors.

Table 2.

Results of the multivariable joint longitudinal survival model assessing the effect of longitudinal changes in M2M predicted RNFL thicknesses in predicting the risk of development of visual field loss, adjusting for confounding factors.

Parameter	Hazard Ratio (95% CI)	P
Predicted M2M average RNFL thickness
Intercept, per 10 μm lower	1.56 (1.33 to 1.82)	< 0.001
Slope, per 1 μm/y faster decrease	1.99 (1.36 to 2.93)	< 0.001
Age, per decade older	1.54 (1.34 to 1.77)	< 0.001
Sex, female	0.82 (0.61 to 1.12)	0.216
Race, black or African American	1.14 (0.83 to 1.56)	0.413
Mean IOP, per 1 mmHg higher	1.02 (0.97 to 1.06)	0.501
CCT, per 40 μm thinner	1.02 (0.86 to 1.19)	0.851
PSD, per 1 dB greater	2.08 (1.70 to 2.53)	< 0.001

CCT: central corneal thickness, IOP: intraocular pressure, M2M: Machine to machine; PSD: pattern standard deviation, RNFL: retinal nerve fiber layer

Figure 3 shows representative examples of eyes in the study, illustrating fundus photos and M2M predictions of RNFL thickness, as well as visual field results at different time points during follow-up. Figure 3A shows an eye that had significant decline in predicted RNFL thickness over time, from 83.2 μm to 69.5 μm. It can be seen that this translated into a very high risk of conversion (i.e., low survival probability) and, in fact, this eye ended up developing a visual field defect over time. Figure 3B shows an eye with stable M2M predictions of RNFL thickness over time which did not develop visual field loss.

Figure 3.

Examples in which the trend in predicted RNFL thickness was concordant with whether the eye eventually converted to glaucoma. For each eye, five fundus photos and three visual fields (grayscale and pattern deviation plot) are shown at several time points during follow-up, along with the survival probability plot at the respective study endpoint. (A) depicts an eye with a fast rate of change in M2M-predicted RNFL thickness that eventually converted to glaucoma on visual field. (B) depicts an eye with relatively stable RNFL predictions that did not convert to glaucoma.

Joint longitudinal survival modeling also allowed us to model how the risk of conversion to glaucoma would change for a particular eye as more fundus photographs were acquired over time. In this approach, the addition of new data modifies the probabilities of conversion. Two examples of this continuously updated function are shown in Figure 4. Both eyes (top and bottom rows) have similar predicted RNFL thicknesses at baseline. However, the eye depicted in the top row had a much faster rate of decline during follow-up, and its probability of survival, i.e., not converting to glaucoma, diminished correspondingly as more data were acquired over time.

Figure 4.

Predicted individual survival probability plots for two eyes of different subjects. One eye (top) showed a relatively faster rate of change in M2M-predicted RNFL thickness during follow-up, while the other eye (bottom) also showed a decline but at a much slower rate. Survival probabilities are shown for three different time points: baseline visit (left), after six years (middle), and at their respective study endpoints, around ten years (right). A comparison of the predicted survival probabilities shows that the eye with faster rate of decline (top) had a much lower probability of survival, that is, retaining a normal visual field. This eye did in fact develop repeatable glaucomatous visual field defects while the other eye (bottom) did not.

DISCUSSION

In our study we found that deep learning predictions of RNFL thickness based only on fundus photographs were able to predict future development of visual field defects in eyes of glaucoma suspects. Eyes that had lower M2M predictions of RNFL thickness at baseline and faster decline in thickness over time were significantly more likely to convert to glaucoma. While the baseline models alone were predictive of future glaucomatous conversion, incorporating longitudinal changes significantly strengthened this predictive ability. Even after controlling for potential confounding variables such as age, sex, race, CCT, baseline PSD, and mean IOP, baseline and longitudinal changes in M2M’s predictions of RNFL thickness were still predictive of future conversion to glaucoma. These results provide evidence for the potential of deep learning approaches applied to fundus photography to be used for longitudinal monitoring of eyes suspected of glaucoma.

A previous study showed that joint longitudinal survival modeling of actual RNFL thickness measured by OCT was predictive of future glaucomatous conversion on SAP.¹³ That study found that each 10 μm thinner global RNFL thickness at baseline was associated with a hazard ratio of 1.59, and each 1 μm/year faster decrease in RNFL thickness during follow up was associated with a hazard ratio of 1.97.¹³ Our multivariable model, which used M2M predictions of RNFL thickness from fundus photos, had comparable values: hazard ratio of 1.56 per 10 μm thinner predicted RNFL thickness at baseline and hazard ratio of 1.99 per 1 μm/year faster decrease in predicted RNFL thickness. In addition, the modified R² of the multivariable model using OCT average RNFL thicknesses in that study (51%)¹³ was the same as the one in our study (51%). This is a striking result given that our predicted RNFL thickness measurements were obtained solely from fundus photographs, using a previously trained deep learning model. These results agree with previous reports showing a strong correlation (r > 0.8) between predicted RNFL thickness measurements from the M2M model and actual OCT measurements obtained in eyes with varying levels of glaucoma damage.¹⁴ In a more recent study, Jammal and colleagues also showed that M2M predictions generally outperformed human graders in detecting glaucoma on fundus photographs.¹⁵

The use of deep learning based on fundus photographs for monitoring eyes suspected of glaucoma is appealing given its low cost, potential for widespread availability, and objectivity. While effective at diagnosing and staging glaucoma,²⁴ OCTs are expensive and require technical expertise in acquiring and interpreting images. As such, their availability is limited beyond specialized eye centers and hospitals.²⁵ And despite the ongoing developments of portable, lower-cost OCT systems, the cost is still much greater than fundus photography,²⁶ and these innovations do not resolve the hurdle of needing specialized expertise in interpreting OCT outputs. On the other hand, fundus photos are relatively easy to acquire, cheap, and much more accessible, having been used as screening tools for ocular pathologies in non-ophthalmological primary care settings.^27,28 Developments of portable fundus cameras have expanded the role of fundus photographs in ophthalmological screening through telemedicine^29,30 and more recently, fundus photography has even been adapted to simple smartphone cameras.³¹ These innovations could further enable imaging in underserved regions, which have disproportionately higher rates of glaucoma.^1–3 Thus, our method based on fundus photos has potential for widespread availability.

Although other deep learning models have been trained to replicate subjective human gradings on fundus photographs,³² such an approach suffers from the inaccuracies of the subjective gradings used as ground truth for their training. Many previous studies have shown that subjective gradings have poor interrater reliability and result in under- or over-estimation of the amount of glaucomatous damage.^33–35 In addition, because human gradings are generally qualitative, this would make it difficult to use such deep learning models to track change over time. The application of our M2M deep learning algorithm on fundus photos overcomes each of these issues. Because it was trained using RNFL thickness measurements from OCT as the reference standard,¹⁴ M2M’s deep learning predictions based on fundus photos is fully objective and quantitative, and once trained, the deep learning model can be applied to new photographs in real-time, not requiring interpretation by an ophthalmologist.

An attractive feature of the model shown in our study is the ability to obtain updated predictions of risk of glaucoma as more data are acquired over time for an individual patient. This is illustrated in Figure 4, where although both eyes had the same baseline measurements, they eventually developed very different slopes of change in predicted RNFL thickness, and the eye with the faster slope eventually developed a visual field defect. In practice, this model could be applied to reassess the risk of glaucoma in eyes suspected of disease, as new photographs are acquired over time. In fact, previous studies have shown that baseline measurements are insufficient to provide enough predictive ability to estimate risk of future development of glaucoma in suspected eyes. In the confocal scanning laser ophthalmoscopy ancillary study to the Ocular Hypertension Treatment Study, baseline optic disc topographic measurements as analyzed by a Moorfield regression analysis showed a positive predictive value of only 14.1% in predicting future primary open-angle glaucoma.³⁶ Another study showed that baseline gradings of optic disc structural changes were weak in their ability to predict future development of visual field defects.³⁷ Even OCT measurements have not been as effective in predicting risk of future glaucoma when using only baseline measurements.^10,12 Thus, the ability to continuously update predictions and use longitudinal changes in the acquired data makes our model a useful clinical tool in monitoring for glaucomatous change and improve risk prediction.

Limitations

Our study had some limitations. Due to the retrospective nature of the study, the treatment regimen of our cohort was neither specified nor controlled. Eyes that were deemed at higher risk at baseline because of the appearance of their optic nerves may have received more aggressive treatment, decreasing the subsequent risk of conversion. This may have biased our results but in fact would have tended to underestimate the hazard ratios. Despite this, as previously discussed, the results of our analysis were on par with models using actual OCT measurements. Of note, different cameras were used to acquire photos both at baseline as well as during follow-up. This may raise concerns about the influence of camera type on M2M’s RNFL thickness predictions. However, as the training sample also had photos taken from both cameras, the impact of using different cameras on the predictive ability of the M2M model was likely minimal. In any case, changes in cameras would most likely have decreased rather than increase the performance of the model. Another potential concern is the variation in quality of the photos used in the study. Given the enormous number of photographs used for development, validation and testing of the M2M model, it was impractical to manually review all of them. Thus, there is variability in the quality of our photos, and the quality of some of them may have been less than ideal. However, the retention of photographs of lower quality may improve the generalizability of our model. Future studies should evaluate the impact of different camera types and photo quality on predictions that could be obtained by this type of model. Finally, our study used a population of glaucoma-suspect patients presenting to a tertiary care center. As such, the results of this study may not be completely generalizable to use in other settings, namely in the broader population at large as a screening tool or in non-academic community settings. Future studies should be conducted to externally validate and translate the novel approach presented in this study to these other settings.

Conclusion

Our study demonstrated that deep learning predictions of RNFL thickness measurements based on fundus photographs can be used to predict risk of future glaucomatous conversion. This method can be potentially used to develop a cheap and accessible glaucoma screening method as well as to monitor risk of glaucomatous conversion in eyes suspected of the disease.