Efficacy of deep learning-based artificial intelligence models in screening and referring patients with diabetic retinopathy and glaucoma

Abstract

Purpose:

To analyze the efficacy of a deep learning (DL)-based artificial intelligence (AI)-based algorithm in detecting the presence of diabetic retinopathy (DR) and glaucoma suspect as compared to the diagnosis by specialists secondarily to explore whether the use of this algorithm can reduce the cross-referral in three clinical settings: a diabetologist clinic, retina clinic, and glaucoma clinic.

Methods:

This is a prospective observational study. Patients between 35 and 65 years of age were recruited from glaucoma and retina clinics at a tertiary eye care hospital and a physician’s clinic. Non-mydriatic fundus photography was performed according to the disease-specific protocols. These images were graded by the AI system and specialist graders and comparatively analyzed.

Results:

Out of 1085 patients, 362 were seen at glaucoma clinics, 341 were seen at retina clinics, and 382 were seen at physician clinics. The kappa agreement between AI and the glaucoma grader was 85% [95% confidence interval (CI): 77.55–92.45%], and retina grading had 91.90% (95% CI: 87.78–96.02%). The retina grader from the glaucoma clinic had 85% agreement, and the glaucoma grader from the retina clinic had 73% agreement. The sensitivity and specificity of AI glaucoma grading were 79.37% (95% CI: 67.30–88.53%) and 99.45 (95% CI: 98.03–99.93), respectively; DR grading had 83.33% (95 CI: 51.59–97.91) and 98.86 (95% CI: 97.35–99.63). The cross-referral accuracy of DR and glaucoma was 89.57% and 95.43%, respectively.

Conclusion:

DL-based AI systems showed high sensitivity and specificity in both patients with DR and glaucoma; also, there was a good agreement between the specialist graders and the AI system.

The referral process among different sub-specialties forms an essential component of the medical system. Because of advancing age and associated co-morbid conditions, patients with ophthalmic diseases can become non-compliant with cross-referral between ophthalmic sub-specialties, which may lead to a poor visual outcome.[1] Likewise, there is often non-compliance in the referrals between the specialities, such as a general physician, diabetologist, and ophthalmologist.[1] It is evident from several studies that deep learning (DL) algorithms can produce excellent results for detecting diabetic retinopathy (DR), glaucoma suspects, and other retinal diseases.[2]

These algorithms have shown performance similar to that of a retina/glaucoma specialist in research settings.[3] Recently, the American Diabetic Association has included artificial intelligence (AI) for DR screening in the published guidelines. However, these algorithms’ real-time implementation and utility may differ in various health care settings.[4] Its utility in reducing cross-referral in ophthalmic practice has not been explored. The aim of this study was to analyze the real-time performance of the AI-based algorithm in detecting the presence of DR and glaucoma suspects as compared to the diagnosis by the retina and glaucoma specialists. We also tried to explore whether using these algorithms can reduce the cross-referral in three clinical settings, a diabetologist’s or physician’s clinic, a retina clinic, and a glaucoma clinic.

Methods

This was a prospective observational study performed between September 2019 and January 2020. The patients from glaucoma and retina clinics at a tertiary eye care hospital and a physician clinic who fulfilled the inclusion criteria were recruited. All patients aged between 35 and 65 years and patients with a prior diagnosis of glaucoma or suspected of glaucoma who either visited the glaucoma clinic directly or were referred from other sub-specialties were included in the study; likewise, patients with DR referred to a retina clinic and patients with DR visiting the physician clinic were included. Patients with media opacities precluding the acquisition of retinal images and uncooperative patients were excluded from the study. The study was approved by the Institutional Review Board, and written informed consent was obtained from all the participants. The study adhered to the tenets of the Declaration of Helsinki.

Zeiss VISUCAM 224 fundus camera

At the retina and glaucoma clinic, all the patients who were recruited in the study underwent a comprehensive eye examination, including a detailed history and slit lamp evaluation, and all patients underwent fundus photographs using a Zeiss non-mydriatic fundus camera (VISUCAM 224, Germany, 2015),[5] with the macula as the center and a 45-degree color mode. In each eye, five fields were captured, namely, macula center, temporal, nasal, upper, and lower side quadrants of the retina. Patients recruited at the glaucoma clinic also underwent an additional photograph centered on the disc. Patients with DR were graded according to the guideline of the international clinical DR disease severity scale[6], and patients with glaucoma were graded according to the internal council of ophthalmology guidelines for glaucoma.[7]

AI system

All images were graded using the Dr Noon AI system (Medi Whale Inc., Seoul, South Korea, 2018). Dr Noon is a medical AI system (cloud based on a DL algorithm) that uses an integrated database of clinically classified eye diseases to screen and evaluate retinal fundus images (retinal diseases, glaucoma, macular degeneration, and cataract).[8] Our product is a cloud-based web application known as SaMD. The system can be divided into the user side and the server side. On the user side, users can access a web client through their browsers and upload images and patient information using the client program, which automatically uploads the images. Third-party users can also upload and retrieve results by requesting the API server. The client program can send our product’s diagnostic results as DICOM files to the PACS system. The server side consists of a proxy server, client server, API server, AI model server, and data storage. The proxy server transmits user requests to either the client server or the API server. The client server serves our web application to users through their browsers and requests the API from the API server. The API server receives requests, processes them, and requests AI results from the AI model server. It also stores files and queries data in the data storage. The AI model server generates the AI model’s results using images. The data storage stores image files and data in RDS. Our system has similar security threats to those of common web application systems. In this section, we will outline these threats and the methods we use to mitigate them.

The API server and AI model server are developed using the Flask (v. 1.0.2) Python web framework, whereas the client server is developed using the NextJS (v. 10.2.3) web front framework. Data storage is hosted through AWS S3 and RDS services. Fig. 1 elaborates the AI system.

Figure 1.

The AI system

Dr Noon machine learning system consists of three parts: image pre-processing, convolutional neural network’s inference, and processing result. In image pre-processing, the retinal area was cropped using circle detection. The cropped fundus image is resized to a specific size of (ex: (300,300,3). Finally, contrast enhancement methods are applied to enhance contrast of vessels in the fundus image and reduce variability caused by different lightness conditions. Fig. 2 shows the example of a pre-processing fundus image.

Figure 2.

Example of the pre-processing fundus image

The convolutional neural network model consists of two parts: a convolutional feature encoder and a fully connected layer. A fundus image is fed into the convolutional feature encoder and results in a feature map of size (8,8,512). To remove the spatial dimension, global average pooling is performed to get a 512-dimensional feature vector. The last fully connected layer outputs class probability or regression predictions. Mean squared error or sigmoid cross-entropy was used as a loss function to train the model. Every convolutional layer is followed by batch normalization, except the last fully connected layer. Fig. 3 shows the example of a convolutional feature encoder.

Figure 3.

Example of the convolutional feature encoder

Results of the machine learning model, probability of class or regression prediction, are saved in the Dr Noon database system. The saliency map is generated to help understand how our proposed model predicts the result. This saliency map shows which pixel in the image highly affects results of the model. Fig. 4 shows the example of a saliency map.

Figure 4.

Example of a saliency map

Rationale and selection process of the particular ML model

The input size of the ML model was decided (300,300,3) to maintain the accuracy of the model, and the ML system can give response to clients in reasonable time. A convolutional neural network model architecture that achieved a very high accuracy with an ImageNet dataset like VGG was selected and slightly modified. In the current version of Dr Noon, we choose the modified VGG architecture model. It does not have less layers compared to other architectures like Resnet, and it achieved similar or better performance. The VGG architecture could reduce the training and developing time and the inference time in Dr Noon. Some regularization methods are applied to prevent the model’s overfitting. We set the learning rate constant and save the model with early stopping to get the best performance.

The input data or parameters used to generate the corresponding output

Input data should be in a specific image file format: jpg, jpeg, png, gif, tif, or dicom files containing image arrays and some patients’ information. The client should input some patients’ information, such as patient ID, exam date, birth date, and gender, to get the corresponding output.

Grading of images

All the images acquired at the glaucoma clinic were graded independently by three glaucoma specialists, and those from the retina clinic were graded by three retina specialists. The images from the physician clinic were graded by all six ophthalmologists (three glaucoma and three retina specialists). The reasons for referrals from glaucoma, retina, and physician clinics because of retinal problems, glaucoma, and ungradable images were noted. The flow of the patient recruitment process is shown in Figs. 5-8 give the AI output for the retinal disease.

Figure 5.

Flow chart explaining the consecutive process of patient recruitment

Figure 8.

AI output for Proliferative Diabetic Retinopathy

Figure 6.

AI output for Diabetic retinopathy

Figure 7.

AI output for Vessel occlusion

Statistical analysis

Statistical analyses were performed using Microsoft Excel and statistical software (SPSS for Windows V.14.0; SPSS Science, Chicago, Illinois, USA). The sensitivity and specificity of the DL algorithm for detecting DR, glaucoma suspects, and opacity/artifact (ungradable) were calculated by generating 2 × 2 tables taking the ophthalmologist grading as the gold standard. The degree of agreement between automated analysis and manual (ophthalmologist) grading was quantified and assessed using Cohen’s kappa (k) statistics. For all statistical tests, a P value <0.05 was considered statistically significant (retina specialists). The reasons for referrals from glaucoma, retina, and physician clinics because of retinal problems, glaucoma, and ungradable images were noted. The flow of the patient recruitment process is shown in Fig. 1.

Results

Out of 1085 patients, 362 were seen at glaucoma clinics, 341 were seen at retina clinics, and 382 were seen at physician clinics; the demographic and systemic characteristics are listed in Table 1, which presents the demographic and systemic characteristics of patients in three different clinics: glaucoma clinics, retina clinics, and physician clinics. The purpose of this table is to compare and demonstrate the differences between these groups in terms of their demographic information and any underlying systemic conditions that they may have. The table provides information on various factors, such as age, sex, presence of DM, duration of DM, blood pressure, and HbA1c, which can impact the diagnosis and treatment of patients in these clinics.

Table 1.

Baseline characteristics of the study subjects

Parameters n=1085	Glaucoma Clinic	Retina Clinic	Physician Clinic	P
Age, Mean (SD)	50.70±6.16	51.69±6.54	57.57±11.85	<0.001
Gender
Male (%)	207 (57.2)	230 (67.4)	189 (49.5)	<0.001
Female (%)	155 (42.8)	111 (32.6)	193 (50.5)
Presence of DM	46 (27.9)	21 (47.7)	382 (100.0)	<0.001
SBP, Mean (SD), mm of Hg	127.72±18.93	130.55±16.74	126.32±19.03	0.014
DBP, Mean (SD), mm of Hg	76.14±11.41	79.67±7.34	79.71±14.17	<0.001
Duration of DM	1.13±0.34	1.48±0.51	1.51±0.50	<0.001
HbA1c, Mean (SD)	-	-	7.87±1.59	-

DM: Diabetes Mellitus, SD: Standard Deviation, SBP: Systolic Blood Pressure, DBP: Diastolic Blood Pressure

Table 2 demonstrates a level of agreement among specialist graders from the glaucoma clinic (n = 3), retina clinic (n = 3), and AI, as evidenced by Cohen’s kappa values exceeding 80%. Table 3 shows agreement between ophthalmologists’ grading and AI; the retina grader from the glaucoma clinic had 85% agreement, the glaucoma grader from the retina clinic had 73% agreement, and least agreement was found in the glaucoma grader from the physician clinic, with about 54%. Among the referred patients from the glaucoma clinic to retina clinic, the disease prevalence was 19.13%; among the referrals from the retina clinic to glaucoma clinic, the disease prevalence was 6.39%. The accuracies were good in both the groups, Table 4. The accuracy of cross-referral from the retina clinic and glaucoma clinic was 89.75% (95% CI: 85.85–92.58%) and 95.43% (95% CI: 91.76–97.79%), respectively.

Table 2.

Agreement between graders by speciality in grading retinal images

Speciality	No. of Graders	K%	SE	95% CI
Glaucoma	3	85.00	0.038	77.55-92.45
Retina	3	91.90	0.021	87.78-96.02

K% - Cohen’s Kappa agreement; SE: Standard error

Table 3.

Sensitivity, specificity, and AUC of the algorithm with reference to the ophthalmologist’s grading

Clinic	Grading used to compare AI	Sensitivity% (95% CI)	Specificity% (95% CI)	AUC (95% CI)	Kappa (%)
Glaucoma clinic	Retina grader	79.37 (67.30-88.53)	99.45 (98.03-99.93)	96.35 (93.14-99.55)	85
Retina clinic	Glaucoma grader	83.33 (51.59-97.91)	98.86 (97.35-99.63)	83.04 (68.4197.68)	73
Physician clinic	Retina grader	68.63 (54.11-80.89)	97.57 (95.58-98.83)	87.22 (80.07-94.36)	70
	Glaucoma grader	83.33 (51.59-97.91)	96.9 (94.86-98.30)	70.60 (57.30-83.92)	54

AI: Artificial Intelligence, DR: diabetic retinopathy, CI: Confidence interval, AUC: Area Under Curve

Table 4.

Accuracy of AI algorithms to predict the cross-referral in glaucoma, retina, and physician clinics

Clinic	Value (%)	95% CI
Glaucoma clinic to retina
Sensitivity	60.61	47.81-72.42
Specificity	96.42	93.51-98.27
Disease prevalence	19.13	15.12-23.68
PPV	80	67.86-88.34
NPV	91.19	88.46-93.32
Accuracy	89.57	85.85-92.58
Retina clinic to glaucoma
Sensitivity	35.71	12.76-64.86
Specificity	99.51	97.31-99.99
Disease prevalence	6.39	3.54-10.49
PPV	83.33	38.50-97.56
NPV	95.77	93.88-97.10
Accuracy	95.43	91.76-97.79

CI: Confidence Interval, PPV: Positive Predictive Value, NPV: Negative Predictive Level; all the patients with DR and glaucoma from retina, glaucoma, and physician clinics were combined to perform cross-referral analysis

Discussion

In this current study, we aimed to compare the diagnostic ability of AI algorithms and human specialist graders for screening DR and glaucoma and also the efficacy of this AI system to cross-refer to the respective sub-specialties. These days, fundus photography is regularly performed in many centers. They aid in the identification of many ocular disorders. With the rising frequency of ocular disorders, such as DR and glaucoma, there is a need to build an algorithm to reach out to different social backgrounds that lack clinical services.[9]

India has led the way in the development and validation of numerous AI-based algorithms for diagnosing DR and glaucoma, but DR will represent a significant challenge to India’s vision in 2020. Unlike glaucoma, which may be managed solely by ophthalmologists, controlling DR necessitates a multi-modal strategy involving coordination with diabetologists and physicians.[10] Fundus photo images were used in many studies to screen glaucoma using AI models, but the diagnosis of glaucoma is more challenging. The diagnosis of glaucoma is based on a number of characteristics, including the stage of the disease and the eye’s refractive error.[11]

DR is diagnosed based on abnormal retinal features such as hemorrhage, micro-aneurysms, and exudates, whereas glaucoma is diagnosed by interpreting elusive structural changes of the optic disc.[12] Hence, an ultra-high-resolution three-dimensional fundus photo camera was used in this current study. It is difficult to compare the current DL algorithm’s diagnostic performance to that of recent investigations.[11] In the current study, we found the agreement between the human grader and AI to be 85% in identifying glaucoma, whereas the DR grader and AI had 91% of agreement and 70.30% of agreement in detecting DR and glaucoma at the physician clinic, respectively. In a recent study by Jammal AA et al.,[13] a DL algorithm out-performed human graders in detecting glaucomatous optic disc changes from fundus photographs (>90% of sensitivity and specificity).

In this study, the AI model was shown to be clinically useful in diagnosing referable patients with DR and glaucoma. We observed a high specificity of about > 90% and a positive predictive value (PPV) of 80% in cross-referring patients from the retina clinic to the glaucoma clinic and 83.3% of PPV in cross-referring patients from the glaucoma clinic to the retina clinic. Also, the accuracy of the referrals was high at > 85% in both patients with DR and glaucoma, where 19% of the patients were referred from the glaucoma clinic to retina clinic and 6.39% of patients were referred from the retina clinic to glaucoma clinic, but the sensitivity was low in referring patients with glaucoma as identifying glaucoma suspects is influenced by many other factors such as increased intra-ocular pressure, retinal nerve fibre layer thickness changes, and visual field changes.

Although most deep learning algorithms for DR and glaucoma screening have simplified the criteria for referable and non-referral conditions, there exist many discrepancies in real-life scenarios. For example, referrals for pan-retinal photo-coagulation or intravitreal injections may be more urgent for patients with proliferative diabetic retinopathy (PDR), whereas patients with moderate non-proliferative diabetic retinopathy (NPDR) without diabetic macular edema (DME) on the other hand may not require therapy but do require close monitoring by retina specialists on a regular basis, even if they are categorized as referable.[14]

Training AI models based on emergency clinical situations will save community resources by precisely identifying the referable group of people in need of treatment. In our study, AI had a high sensitivity when compared to the retina grader at the glaucoma clinic (DR detecting sensitivity 79.37%), and the glaucoma grader at the retina clinic (glaucoma detecting sensitivity 83.33%) and the AI at the physician clinic also had good sensitivity (DR detecting sensitivity 68.63%, glaucoma detecting sensitivity 83.33%). AI algorithms applied to fundus photographs for screening purposes may yield good results, yet more sophisticated approaches, including data from optical coherence tomography (OCT) and perimetry, should be employed for patients who are suspected glaucoma.[15] Gulshan et al.[16] evaluated two retinal photography-based AI algorithms (EyePACS and Messidor), where both the models had high sensitivity and sensitivity in detecting patients with DR; they recommended that future studies should incorporate large datasets and multiple graders to develop clinically reliable algorithms. In our study, we compared the performance of AI for cross-referring patients between retina, glaucoma, and physician clinics; similarly, Ting DSW et al.[17] performed a multi-center study using DL to identify DR, glaucoma, and age-related macular degeneration (ARMD), where their model showed >90% of sensitivity and specificity. Similarly, utilizing two other public databases (Messidor-2 and E-Ophtha), Gargeya and Leng showed a good DL diagnostic performance in detecting any DR.[18] To aid translation, the DL should be developed and tested in clinical backgrounds employing a wide range of retinal images of varied quality from various camera types as well as in typical DR screening populations.[19]

The limitations of the current study are that DR grading might be possible with the fundus photographs, yet definite identification of DME might not be possible as it needs the assistance of OCT in patients with glaucoma retinal nerve fibre layer thickness and perimetry results play a major role to pick the early glaucomatous changes; hence, referring patients solely based on fundus photographs prevails as a clinical challenge. Future studies can concentrate on incorporating OCT and perimetry changes along with fundus photographs to provide more precise and sophisticated diagnoses; also, these validations should be carried out in large population samples across different ethnicities to ensure reproducibility. These future integrations might facilitate the clinical practice of AI modalities globally.

In conclusion, we found that this DL-based AI system showed high sensitivity and specificity in identifying both patients with DR and glaucoma; there was a good agreement between the specialist graders and the AI system; also, AI had a high accuracy in cross-referring the patients to the sub-specialties.

Retina and Glaucoma Study Group

Shri Bhagwan Mahavir Vitreoretinal services:

Dr Chetan Rao

Dr Jaya Prakash V

Smt Jadhav bai Nathmal Singhvee Glaucoma Services:

Dr Trupti Sudhir Patil

Dr Rathini Lillan David

Dr Vineet Ratra

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.