Highly Accurate and Precise Automated Cup-to-Disc Ratio Quantification for Glaucoma Screening

Abstract

Objective

An enlarged cup-to-disc ratio (CDR) is a hallmark of glaucomatous optic neuropathy. Manual assessment of the CDR may be less accurate and more time-consuming than automated methods. Here, we sought to develop and validate a deep learning–based algorithm to automatically determine the CDR from fundus images.

Design

Algorithm development for estimating CDR using fundus data from a population-based observational study.

Participants

A total of 181 768 fundus images from the United Kingdom Biobank (UKBB), Drishti_GS, and EyePACS.

Methods

FastAI and PyTorch libraries were used to train a convolutional neural network–based model on fundus images from the UKBB. Models were constructed to determine image gradability (classification analysis) as well as to estimate CDR (regression analysis). The best-performing model was then validated for use in glaucoma screening using a multiethnic dataset from EyePACS and Drishti_GS.

Main Outcome Measures

The area under the receiver operating characteristic curve and coefficient of determination.

Results

Our gradability model vgg19_batch normalization (bn) achieved an accuracy of 97.13% on a validation set of 16 045 images, with 99.26% precision and area under the receiver operating characteristic curve of 96.56%. Using regression analysis, our best-performing model (trained on the vgg19_bn architecture) attained a coefficient of determination of 0.8514 (95% confidence interval [CI]: 0.8459–0.8568), while the mean squared error was 0.0050 (95% CI: 0.0048–0.0051) and mean absolute error was 0.0551 (95% CI: 0.0543–0.0559) on a validation set of 12 183 images for determining CDR. The regression point was converted into classification metrics using a tolerance of 0.2 for 20 classes; the classification metrics achieved an accuracy of 99.20%. The EyePACS dataset (98 172 healthy, 3270 glaucoma) was then used to externally validate the model for glaucoma classification, with an accuracy, sensitivity, and specificity of 82.49%, 72.02%, and 82.83%, respectively.

Conclusions

Our models were precise in determining image gradability and estimating CDR. Although our artificial intelligence–derived CDR estimates achieve high accuracy, the CDR threshold for glaucoma screening will vary depending on other clinical parameters.

Financial Disclosure(s)

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Primary open-angle glaucoma is one of the most common glaucoma subtypes, with a global prevalence of 2.4%.¹ Accurate detection of glaucoma is essential to prevent irreversible damage to the optic nerve head (ONH), and this often involves clinically assessing the cup-to-disc ratio (CDR). The CDR is a morphological characteristic of the ONH that can estimate the risk of developing glaucoma.² A larger CDR or interocular asymmetry  >0.2 is one of the key risk factors for the development and progression of glaucoma.3, 4, 5 In patients with advanced glaucoma, even small alterations in CDR may lead to considerable loss of retinal ganglion cells.⁶ Furthermore, many genetic variants are associated with CDR, with a subset of these variants also associated with primary open-angle glaucoma risk.7, 8, 9, 10 Therefore, accurate assessment of CDR is important for glaucoma screening and tracking progression in a clinical setting.

Fundus photography is a noninvasive imaging technique frequently used in glaucoma practice for documenting the condition of the optic nerve and retina, including CDR, neuroretinal rim, disc hemorrhages, and longitudinal monitoring of the optic nerve and surrounding retinal structures. Manual assessment of the CDR is also possible but may be challenging and time-consuming for clinicians. Even among glaucoma specialists, CDR measurement is subject to interobserver and intraobserver variability.11, 12, 13 Advanced imaging technologies, such as OCT and confocal scanning laser ophthalmoscopy, have been observed to provide different CDR values for an individual due to distinct imaging technologies and analysis methodologies.^14,15 To overcome these difficulties, CDR estimation can be automated through leveraging advances in artificial intelligence (AI) and computer vision.¹⁶

Several studies have precisely segmented the optic disc and cup with deep learning–based techniques for calculating CDR using shape-based methods (circular or elliptic ONH) or appearance-based approaches (texture, color, and intensity of the optic disc).17, 18, 19 Accurately and precisely segmenting the optic cup and disc is challenging because of the variability in ONH morphology, which can be influenced by the patient’s ethnicity, age, disease conditions, and image quality.20, 21, 22 Additionally, segmentation techniques that require masking for both optic disc and cup can be more time-consuming and costly.²³ In contrast, nonsegmentation techniques automatically learn the features (disc and cup) from the images. A limited number of studies have used convolutional neural network (CNN)–based regression analysis to estimate CDR using fundus image data.24, 25, 26 Therefore, we developed and validated a CNN-based regression model on a large cohort of retinal images to compute CDR without applying the segmentation techniques for diagnosing glaucoma in clinical and community settings. This selection was primarily driven by the novel application of CNNs for regression analysis in CDR determination, diverging from their common use in image classification due to the considerable amount of resources needed to accurately grade a large dataset in terms of time and costs.

Methods

Study Design

A comprehensive overview of this study is shown in Figure 1. An extensive dataset from the United Kingdom Biobank (UKBB) was used, with each retinal image in the study independently assessed and graded by 2 ophthalmologists (A.W.H. and J.E.C.). We utilized CNN-based models for classification, and regression analyses to determine the gradability and estimate the CDR from 80 225 and 68 695 colored fundus images, respectively.

Figure 1.

Overview of our study design used for model development. A model for gradability assessment was developed (model 1), and then a model for cup-to-disc ratio quantification was generated based on gradable images (model 2). These 2 models were then combined to assess the utility of diagnosing glaucoma in 2 separate cohorts. CDR = cup-to-disc ratio; QC = quality control; UK = United Kingdom.

The UKBB study received ethical approval from the local research ethics boards, and all participants gave informed written consent. This study's procedures were performed in accordance with the ethical guidelines for medical research as described by the World Medical Association Declaration of Helsinki.

Participants and Imaging Modality

Training Dataset

All the participants' data were from the UKBB dataset, which is a large population-based observational study in the United Kingdom that began in 2006 and recruited >500 000 participants aged between 40 to 69 years at the time of recruitment; 67,040 participants had ≥1 gradable retinal image.^27,28 The images were available from both left and right eyes from baseline and repeat assessment visits. The retinal images were obtained with a Topcon 3D OCT 1000 Mark II (Topcon Corp).²⁹ This study included both eyes of the participants during 2 visits, irrespective of ancestry.

As described previously, a total of 80 225 fundus images from the UKBB fundus dataset were evaluated and had CDR assessed.²⁸ Of these images, 68 695 (82.73%) were gradable, and 2812 (3.38%) duplicate images were removed. Two independent graders exhibited a strong positive correlation (Pearson correlation coefficient [R] = 0.70) and high consistency (intraclass correlation coefficient, 3k = 0.82) on a shared set of 2812 images (Fig S1).

External Validation

The models were externally validated on the EyePACS dataset—an excellent and reliable resource for creating AI-powered tools to diagnose glaucoma.³⁰ This collection of 101 442 fundus images represents a diverse population of 60 357 individuals who visited various centers across the EyePACS network in the United States.³¹ This dataset is of exceptional quality and accuracy, given a team of 20 expert graders thoroughly graded the complete dataset, with ≥2 graders reviewing each image. These graders have demonstrated excellent proficiency in detecting glaucoma from fundus photographs, with a minimum sensitivity of 80% and a specificity of 95%. Our models underwent additional validation using the Drishti dataset from India,³² allowing us to assess the consistency and effectiveness of our models across different geographical populations. It includes 70 images from people diagnosed with glaucoma and 31 normal images, all with a resolution of 2047 × 1760 pixels in portable network graphics format.

Image Grading and Preprocessing

In our previous study, 2 fellowship-trained ophthalmologists independently viewed and graded the retinal images.²⁸ All the fundus images were cropped to a pixel ratio of 1080 × 800 before the training or validation of the gradability assessment. However, for regression analysis, we further cropped and removed the noninformative areas, background, or margin around the main content of the images based on their pixel intensity using the OpenCV library.³³ The final downsized images were 512 × 512 pixels. As undertaken previously, retinal photographs with low-quality images (ungradable and artifacts) were removed.^34,35

Model Selection

Twelve pretrained models from the FastAI (TorchVision)^36,37 were utilized and tested on grader “A” data (larger dataset) for CDR estimation using regression analysis. The potential model (vgg19) with batch normalization (bn) layers was selected based on the model’s performance (Fig S2) for both classification and regression analyses. The vgg19_bn pretrained model was initially trained on ImageNet.³⁸ This pretrained model automatically learns high-level features from a large variety of images; thus, it offers a robust feature extractor for classification and regression tasks. We trained and fine-tuned the CNN regression model on a combined dataset of 68 695 images to estimate CDR. The models for detecting glaucoma, using the overall cut-off CDR (≥0.60),³⁹ were extensively validated on 2 publicly available datasets including different ethnicities.

Deep Learning Algorithm and Training

All the models were trained via the FastAI Framework—a deep learning library built on PyTorch that provides a high-level application programming interface and allows deep learning architectures to be trained quickly to achieve state-of-the-art results.⁴⁰ We used 2 separate CNN-based models from the FastAI library for the gradability assessment of CDR and regression analysis for estimating CDR using fundus images. The binary cross entropy was utilized for the loss function for the gradability task, and the mean squared error loss function was used for regression analysis.

Image gradability was defined as binary classification (gradable or ungradable) based on the CDR that clinicians can obtain. Two experienced fellowship-trained ophthalmologists (A.W.H. and J.E.C.) independently assessed the gradability of the images. Images were randomly (80:20) separated into 2 groups, to train and validate the models for classification and regression analyses. The classification model was trained on a dataset of 64 180 fundus images, while the regression model was trained on a dataset of 48 734. The validation sets for the classification and regression model comprised 16 045 and 12 183 images, respectively.

The class weights were balanced for the gradability task, and in-built data augmentation techniques were implemented for both classification (softmax probability >0.5 was designated as gradable) and regression analysis to improve the accuracy for generalizing the models' utility.⁴¹ The augmentation parameters are illustrated in Table S1. Model training was completed in 2 steps: in the first step, we used fine-tuning with frozen layers of the pretrained weights (transfer learning) for 5 to 10 epochs using the Fastai library “valley” function to find the optimal learning rate, validation loss was monitored, and the task was executed until it reached the point where validation loss stopped decreasing. After this, the model unfroze all their layers and used a 1-cycle policy for 10 epochs with the “slice” function to implement a strategy known as discriminative learning rates during the model training.^40,42 The callback function was employed to monitor the validation loss during the training and stop training if the loss failed to improve by ≥0.1 (minimum delta) at patience 2 for a maximum of 10 epochs. The regularization technique (weight decay = 1e-3) was also operated to prevent overfitting. This was repeated 10 times with data shuffling, whereby the images were randomly separated into training and validation datasets, and prior to each iteration the top 1% of images found to be in the “top losses” category were removed.

The experiment was conducted on virtual Ubuntu (22.04) desktop with NVIDIA A100 with 40GB of graphics processing unit RAM at Nectar Research Cloud⁴³ using Python (3.10.6) programming language with PyTorch (2.0.0+cu117), FastAI (2.7.12), TorchVision (0.15.1+cu117), Matplotlib (3.5.1), and Scikit-learn (1.2.2) libraries.44, 45, 46

Model Evaluation

Models were evaluated on their area under the receiver operating characteristic curve (AUROC), sensitivity, specificity, precision, recall, and F1-score for gradability assessment. Mean absolute error, mean squared error, R, and coefficient of determination (R²) were used for regression analysis with a 95% confidence interval (CI) for each outcome.⁴⁷ These metrics assessed model performance by quantifying the difference between actual (graded by clinicians) and predicted CDR by the models. Furthermore, the classification metrics were measured by setting a tolerance bin around each regression point and seeing if the truth value fell within tolerance.

Results

Gradability Assessment

Our classification model obtained a high accuracy of 97.13% on the internal validation set of 20% (16 045) of the total data (80 225) for gradability assessment: gradable and ungradable. The CI with 95% for each metric was estimated through Bootstrap resampling⁴⁸ with 4000 iterations to ensure the reliability of the results (Table 1). The model underwent testing on random images from the validation set, and the outcome from the classification model is visualized in Fig S3.

Table 1.

Performance of Our Classification Model (Model 1) for Both Gradable and Ungradable Fundus Images in the Validation Dataset

Classification Metrics	Gradable (n = 13 717) Mean [95% CI]	Ungradable (n = 2328) Mean [95% CI]
AUROC	0.9656 [0.9614, 0.9697]	0.9656 [0.9613, 0.9699]
Accuracy	0.9713 [0.9688, 0.9738]	0.9712 [0.9687, 0.9738]
Sensitivity (recall)	0.9736 [0.9710, 0.9763]	0.9575 [0.9499, 0.9652]
Specificity	0.9574 [0.9491, 0.9657]	0.9736 [0.9709, 0.9763]
Precision	0.9926 [0.9912, 0.9941]	0.8605 [0.8471, 0.8738]
F1-score	0.9830 [0.9815, 0.9846]	0.9063 [0.8981, 0.9145]

AUROC = area under the receiver operating characteristic curve; CI = confidence interval; n = number of fundus images.

Regression Analysis

Our regressing model achieved an R² of 0.71 on the validation set. To further enhance its performance, we excluded 7780 images from the “top losses” category as predicted by the model using the ImageClassifierCleaner from the FastAI library.⁴⁹ We then retrained the regression model on the final dataset of 60 917 images, with 80% used for training and 20% for validation, resulting in a 19.92% improvement in the R². The Bland-Altman plot displays the accuracy of the CDR estimation on a validation dataset. The plot includes the corresponding CDR ground truth, which ranges from 0 to 0.95 with 0.05 intervals. Figure 2A exhibits the mean offsets, agreement limits with a 95% CI.

Figure 2.

Comparative analysis and final dataset distribution. The Bland-Altman plot displays the cup-to-disc ratio (CDR) estimation on the validation dataset with the corresponding CDR ground truth shown in (A). The mean difference between true and predicted CDR was 0.022 (95% confidence interval: 0.009–0.034). The red dashed line represents the mean difference, while the yellow dashed lines represent the 95% confidence interval for the mean difference for upper and lower limits. The distribution of the CDR in the final analysis data is graphically presented in (B).

The mean CDR values from graders A and B were 0.36 and 0.31, respectively, but the final mean and standard deviation for the analyzed data were 0.33 and 0.18. Figure 2B visualizes the distribution of CDR from the final analyzed dataset, while Fig S4 shows the random images taken from the validation set to estimate CDR by the regression model.

The classification metrics were also assessed using a tolerance-based approach, wherein a predefined tolerance bin was established around each regression point. Subsequently, the true value was evaluated to determine if it fell within the established tolerance bin. Thus, we examined the model’s classification metrics using 4 tolerance thresholds (0.05, 0.10, 0.15, and 0.20) around the ground truth values (Table S2). The regression point was converted into classification metrics with a tolerance of 0.2 for 20 classes, and the classification metrics achieved an accuracy of 99.20%. This approach facilitated the rigorous evaluation of the model's accuracy in correctly assigning class labels; these 20 classes are displayed in the confusion matrix in Figure 3A.

Figure 3.

The confusion matrix for cup-to-disc ratio classification is illustrated in (A). In contrast, (B) presents the conversion of regression points into classification metrics utilizing a particular threshold. Publicly available datasets were used for external validation of the glaucoma screening at the globally accepted cut-off threshold, as shown in (C).

Glaucoma Screening

Our models were externally validated on EyePACS and Drishti datasets with promising results for screening glaucoma. On EyePACS, our models achieved 82.49% accuracy, 72.02% sensitivity, and 82.83% specificity. On Drishti, the models achieved 79.21% accuracy, 78.87% sensitivity, and 80.65% specificity. These models predicted <1% and 0% ungradable images on the EyePACS and Drishti datasets, respectively, as shown in Table S3. Our AI system completed gradability, CDR estimation, and classification within 0.3 seconds, which is approximately 10 times faster with more accurate and consistent results than experienced human graders. We also found our regression model could accurately estimate the CDR even from poor-quality retinal images (Fig S5).

After analyzing the EyePACS and Drishti datasets, the optimal CDR threshold for detecting glaucoma was 0.56 (Figs S6 and S7) as based on receiver operating characteristic curve analysis.⁵⁰ At this threshold, AUROC was 0.87, suggesting this model can appropriately distinguish fundus images with and without glaucomatous optic discs.

Discussion

In this study, we conducted a regression analysis on fundus images utilizing deep learning architectures to estimate the CDR without employing segmentation techniques from image gradability to glaucoma screening in a single study. Our classification model was first trained for the gradability task and successfully predicted gradable fundus images based on CDR with remarkable accuracy, precision, and recall.

Deep learning-based models to directly estimate CDR are a more efficient and reliable method than clinical assessment. Our results are comparable to Yuen et al's fundus image quality assessment study, which reported an accuracy of 92.5%, a sensitivity of 92.1%, and a specificity of 98.3% on internal validation with 11.6 times fewer validation data than ours.⁵¹ However, both studies demonstrate that the deep learning–based model has the potential to grade fundus images accurately.

Accurate estimation of the CDR is useful in diagnosing and monitoring glaucoma. Conventionally, AI-based CDR estimation relies on segmentation techniques.^22,52, 53, 54, 55 This technique can be computationally expensive, particularly when dealing with a large volume of medical images. While advancements in graphics processing units have improved computational efficiency, training deep learning models on a significant number of images can still be resource-intensive and costly. Aljazaeri et al reported that a CNN-based regression model performs better than the standard segmentation technique for calculating CDR.²⁵ Our study utilized a deep learning-based model to directly estimate CDR from the fundus images using feature extraction by pretrained weights (vgg19_bn) without applying segmentation techniques. This approach provides a more efficient and reliable method for estimating CDR.

Interestingly, Hemelings et al also used CNN regression analysis to estimate CDR from fundus images, and their findings were notable. However, our results surpassed theirs with an R² value of 86% compared with their 77% [95% CI 0.77–0.79] on the test set of 4765 for CDR estimation. We observed that their mean of CDR was 0.67, approximately twice our dataset's CDR mean.²⁴ Their different regression model exhibited a strong Pearson's R of 0.88 when predicting the CDR against expert grader.²⁶ Another study reported that the model achieved an R of 0.89 on a dataset of 2115 fundus images from the UKBB cohort.⁵⁶ In contrast, our model achieved a higher R of 0.93, indicating a strong correlation between the predicted and expert-graded CDR.

It is important to note that our present model is not trained to detect glaucoma but to estimate the CDR, which is just one of several features that can be used to predict the presence of glaucoma. Further to this, screening of glaucoma on a global scale poses challenges due to the variability of optic disc sizes among different ethnic groups.^57,58 This creates obstacles in standardizing screening techniques using computer vision based on CDR. Hemelings et al reported an impressive AUROC for external validation but altered in both sensitivity and specificity values across the different datasets when using a fixed threshold (0.7) for glaucoma.²⁶ The highest specificity was observed in the Retinal Fundus Glaucoma Challenge 1 dataset at 0.99 [0.98–0.99], whereas the PAPILA dataset exhibited the lowest specificity at 0.70 [0.63–0.76]. Regarding sensitivity, the Gutenberg Health Study, PAPILA, and Artificial Intelligence for Robust Glaucoma Screening datasets achieved the highest value of 0.94, with the Online Retinal Fundus Image Database for Glaucoma Analysis and Research dataset presenting the lowest sensitivity at 0.68 [0.61–0.75]. They also validated the model externally on the EyePACS dataset; sensitivity was achieved 0.68 and 0.89, with thresholds set at 0.82 and 0.72, respectively. Bhuiyan et al noted 80.11% of sensitivity and 84.96% of specificity on the Online Retinal Fundus Image Database for Glaucoma Analysis and Research dataset for screening glaucoma suspects using retinal images, with a cut-off point of CDR >0.5.⁵⁹ Consequently, depending exclusively on CDR for glaucoma screening may overlook the complexity of glaucoma (Table S3). A robust and comprehensive screening model for glaucoma at a population level requires integrating glaucomatous phenotypic and genotypic information with advanced AI algorithms.

This research has some important limitations. Firstly, the dataset used in the training of the models was mainly from the healthy White population—the mean of CDR was 0.37—potentially biased for glaucomatous discs; the distribution of CDR can be seen in Figure 2B. The sample we used to train our models may not represent other age groups or populations from different ethnic or geographical backgrounds. Second, we downsized the input images (224 × 224 × 3) to train the regression model, which could lose some meaningful information for accurately estimating the CDR. Finally, dropping poor-performing images (i.e., the top 1% categorized as “top losses” during each of 10 rounds) during model construction could have led to imbalance classes. Nevertheless, these exclusions were made when our initial model's predictions of CDR values exhibited high errors, indicating substantial deviations from the ground truth values.

In summary, we developed a fully automated end-to-end computer vision model for estimating CDR from the fundus images using CNN regression analysis for glaucoma care. The models were externally validated on 2 publicly available databases for glaucoma screening based on CDR. Deep learning–based models to directly estimate CDR are a more efficient and reliable method than clinical assessment.⁶⁰ However, a generalized model for screening glaucoma based on CDR is challenging because of uncertainty in what the optimal CDR cut-off should be for diagnosing glaucoma.

Manuscript no. XOPS-D-23-00210.