Diabetes mellitus presents a significant global health care burden with the number of affected persons estimated to be 463 million in 2019. 1 In 2010, 3.7 million persons with diabetic retinopathy (DR) were visually impaired, and 800 000 were blind. 2
The COVID-19 pandemic exposed a need for wider implementation of both tele-medicine and alternative methods of screening patients, such as artificial intelligence (AI). Many patients with diabetes, a high-risk group for COVID-related complications, refused to follow-up when treatment facilities became available due to fear of being exposed to the virus.3,4
Artificial intelligence provides a cost-effective method to enhance the effectiveness of tele-ophthalmology (Tele) to provide necessary screening in a population in which access to in-person clinical examination by an expert is difficult or impossible.5,6 This study assessed the concordance of DR detection via Tele, AI, and the gold standard of an in-person examination by a retinal specialist.
Patients who had undergone both an in-person and Tele evaluation of retinal images between April 2021 and June 2021 were identified. Data collected included noninvasive nonmydriatic fundus photos taken by a nonmydriatic Canon CR-2 Plus AF Retinal Imaging camera (Tokyo, Japan), grading results of the fundus photos by a retina specialist via Tele based on the International Clinical Diabetic Retinopathy (ICDR) classification scale, and diagnosis of DR based on ICDR classification scale during an in-person clinic visit in which a dilated fundus exam was performed. The fundus photos were sent for grading by the AI program, EyeArt (EyeNuk, CA), which is designed to grade fundus photos for DR severity (none, mild, or more than mild).
The concordance between each grading method was compared using Cohen’s Kappa (κ) coefficient. When comparing in-person and Tele to AI, any grade higher than mild (moderate, severe, proliferative, or regressed) was considered equal to the AI grade of “more than mild.”
Forty patients (80 eyes) were studied. Grading results are listed in Table 1. Cohen’s Kappa (κ) ± SE for in-person versus Tele was .859 ± .058, in-person versus AI was .751 ± .082, and Tele versus AI was .883 ± .063 (Table 1). A κ value of .61 to .80 indicates substantial strength of agreement and .81 to 1.00 indicates almost perfect agreement.
Table 1.
Diabetic Retinopathy Diagnosis, Grading, and Rate of Agreement.
| In-person diagnosis | Tele-ophthalmology grade | AI grade | |
|---|---|---|---|
| None | 33 (41.3%) | 24 (30.0%) | 19 (23.7%) |
| Mild nonproliferative diabetic retinopathy | 5 (6.3%) | 2 (2.5%) | 0 (0.0%) |
| Moderate nonproliferative diabetic retinopathy | 9 (11.3%) | 13 (16.2%) | 35 (43.8%) a |
| Severe nonproliferative diabetic retinopathy | 3 (3.8%) | 2 (2.5%) | |
| Proliferative diabetic retinopathy | 7 (8.8%) | 9 (11.3%) | |
| Regressed proliferative diabetic retinopathy | 23 (28.8%) | 19 (23.7%) | |
| Ungradable | 11 (13.8%) | 26 (32.5%) | |
| Comparison | Value (κ) | Asymptotic standard error b | No. of DR grades compared c |
| In-person clinical vs. tele-ophthalmology | .859 | .058 | 69 |
| In-person clinical vs artificial intelligence | .751 | .082 | 54 |
| Tele-ophthalmology vs artificial intelligence | .883 | .063 | 53 |
Abbreviations: AI, artificial intelligence; DR, diabetic retinopathy; NPDR, nonproliferative diabetic retinopathy; PDR, proliferative diabetic retinopathy.
An AI reading of “More than mild diabetic retinopathy” was considered comparable to moderate NPDR, severe NPDR, PDR, and Regressed PDR.
Not assuming the null hypothesis.
In order to compare the grading accuracy, each eye needed a grade. If an eye was ungradable for one of the grading modalities being compared, then that eye was not included.
Artificial intelligence and Tele displayed strong agreement with each other and with the gold standard in-person dilated fundus examination by a retina specialist for detecting DR. Tele-ophthalmology and AI have the potential to serve as effective tools to improve the outreach of diabetic screening across a large population irrespective of geographic location of an ophthalmologist. Artificial intelligence will expand the capabilities of Tele and provide an invaluable resource in triaging patients with urgent needs, especially in medically stressful times such as in a pandemic. Additional studies should assess ways to reduce the number of ungradable images via Tele and AI, create a trend analysis for multiple visits for a given patient, and compare the AI used in this study to multiple Tele readers as well as different AI software. Continued advancement and refinement of these technologies will improve images, reduce artifacts that cause ungradable images, and help integrate AI into routine screening of DR.
Footnotes
Abbreviations: AI, artificial intelligence; DR, diabetic retinopathy; ICDR, International Clinical Diabetic Retinopathy; κ, kappa; NPDR, nonproliferative diabetic retinopathy; PDR, proliferative diabetic retinopathy; Tele, tele-ophthalmology.
Marco A. Zarbin is a consultant for: Genentech/Roche, Novartis Pharma AG, Life Biosciences, Illuminare, Tenpoint, Tamarix, EdiGene, and an equity holder in NVasc. All other authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding: The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Lions Eye Research Foundation of New Jersey and NJ Health Institute. The sponsor or funding organization had no role in the design or conduct of this research.
ORCID iDs: Eric J. Kuklinski 
https://orcid.org/0000-0003-0264-2763
Marco A. Zarbin
https://orcid.org/0000-0002-7811-7132
References
- 1. Saeedi P, Petersohn I, Salpea P, et al. Global and regional diabetes prevalence estimates for 2019 and projections for 2030 and 2045: results from the International Diabetes Federation Diabetes Atlas, 9(th) edition. Diabetes Res Clin Pract. 2019;157:107843. doi: 10.1016/j.diabres.2019.107843. [DOI] [PubMed] [Google Scholar]
- 2. Leasher JL, Bourne RR, Flaxman SR, et al. Global estimates on the number of people blind or visually impaired by diabetic retinopathy: a meta-analysis from 1990 to 2010. Diabetes Care. 2016;39(9):1643-1649. [DOI] [PubMed] [Google Scholar]
- 3. dell’Omo R, Filippelli M, Virgili G, et al. Effect of COVID-19-related lockdown on ophthalmic practice in Italy: A report from 39 institutional centers. Eur J Ophthalmol. 2022;32(1):695-703. doi: 10.1177/11206721211002442. [DOI] [PubMed] [Google Scholar]
- 4. Lindeke-Myers A, Zhao PYC, Meyer BI, et al. Patient Perceptions of SARS-CoV-2 exposure risk and association with continuity of ophthalmic care. JAMA Ophthalmol. 2021;139(5):508-515. doi: 10.1001/jamaophthalmol.2021.0114. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Huang X-M, Yang B-F, Zheng W-L, et al. Cost-effectiveness of artificial intelligence screening for diabetic retinopathy in rural China. BMC Health Serv Res. 2022;22(1):260. doi: 10.1186/s12913-022-07655-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Morrison SL, Dukhovny D, Chan RVP, Chiang MF, Campbell JP. Cost-effectiveness of artificial intelligence–based retinopathy of prematurity screening. JAMA Ophthalmol. 2022;140(4):401-409. doi: 10.1001/jamaophthalmol.2022.0223. [DOI] [PMC free article] [PubMed] [Google Scholar]