Abstract
Background:
To compare the performance of Medios (offline) and EyeArt (online) artificial intelligence (AI) algorithms for detecting diabetic retinopathy (DR) on images captured using fundus-on-smartphone photography in a remote outreach field setting.
Methods:
In June, 2019 in the Yucatan Peninsula, 248 patients, many of whom had chronic visual impairment, were screened for DR using two portable Remidio fundus-on-phone cameras, and 2130 images obtained were analyzed, retrospectively, by Medios and EyeArt. Screening performance metrics also were determined retrospectively using masked image analysis combined with clinical examination results as the reference standard.
Results:
A total of 129 patients were determined to have some level of DR; 119 patients had no DR. Medios was capable of evaluating every patient with a sensitivity (95% confidence intervals [CIs]) of 94% (88%-97%) and specificity of 94% (88%-98%). Owing primarily to photographer error, EyeArt evaluated 156 patients with a sensitivity of 94% (86%-98%) and specificity of 86% (77%-93%). In a head-to-head comparison of 110 patients, the sensitivities of Medios and EyeArt were 99% (93%-100%) and 95% (87%-99%). The specificities for both were 88% (73%-97%).
Conclusions:
Medios and EyeArt AI algorithms demonstrated high levels of sensitivity and specificity for detecting DR when applied in this real-world field setting. Both programs should be considered in remote, large-scale DR screening campaigns where immediate results are desirable, and in the case of EyeArt, online access is possible.
Keywords: artificial intelligence, diabetic retinopathy, fundus-on-phone camera, Mexico, rural health care, screening
Introduction
Approximately 463 million people live with diabetes worldwide, and about 700 million people are projected to have diabetes by 2045. 1 Individuals with diabetes are 25 times more likely to become blind than are those in the general population. 2 Thus, diabetic retinopathy (DR) is one of the leading causes of blindness worldwide, with sight-threatening DR affecting 28.5 million people. 3 In Mexico, increasing rates of obesity and a genetic predisposition for type 2 diabetes have led to the increasing prevalence (15.2%) of diabetes in adults. 4 In 2016, Mexico declared that diabetes had reached epidemic proportions and should be considered a major public health problem. 5 Furthermore, endocrine, nutritional, and metabolic diseases were the second leading causes of death in 2014. 6 Approximately one-third of Mexicans with diabetes have DR, with an alarming incidence of 38.9% in Chiapas. 7
Early detection of DR can help prevent blindness in those with diabetes. However, early detection of DR is challenging because Mexico does not have a national DR screening program. Therefore, Retinacare International, a 501c3 US-based nonprofit organization, partnered with RetimediQ in 2005, a private ophthalmological clinic in Merida, Yucatan, to conduct annual and bi-annual DR screening campaigns throughout the state. To date, we have screened >6000 patients in seven cities and towns using dilated, indirect ophthalmoscopic examinations, often combined with high-magnification mydriatic funduscopic examinations.
Smartphone-based retinal imaging has emerged as an efficient, sensitive, specific, and cost-effective method for DR screening.8-10 However, ophthalmologists or trained graders have been required to grade acquired images for the presence and severity of DR.11,12 This grading requirement is not practical for screening in an outreach field setting. Thus, an automated image-grading system for detecting DR is needed to facilitate large-scale DR detection screening efforts and reduce health care provider burden.
Diabetic retinopathy detection can now be performed by computer-based analysis of fundus images using machine learning and artificial intelligence (AI).13,14 Studies have demonstrated that deep-learning algorithms can accurately detect and grade DR in digital fundus images.15-19 Others have investigated the feasibility of DR detection using smartphone-based fundus photography (Remidio fundus-on-phone [FOP] camera 18 combined with an offline (Remidio Medios) 20 or online version of AI software [Eyenuk EyeArt]). 21 These studies determined that both versions of AI software show high sensitivity and specificity for detecting DR in images acquired by smartphone-based fundus photography in tertiary care centers.20,21
This retrospective, noninterventional AI validation analysis compared the diagnostic accuracy of the offline Medios AI software and online EyeArt AI software for detecting DR on a single set of patient images acquired using the ultra-portable Remidio-FOP camera in an outreach field setting.
Methods
Patients
A total of 248 consecutive patients with a known history of diabetes were invited to participate in a DR screening campaign in the cities of Valladolid and Merida, Yucatan, Mexico, in June 2019. Each patient was assigned a unique identification number, and all data, including fundus images, were de-identified to ensure patient confidentiality. The study protocol (EXT-22-01) was approved by the Research Ethics Committee of the Association to Avoid Blindness in Mexico I.A.P (CONBIOETICA-09-CEI-006-20170306) and the Committee of the Association to Avoid Blindness in Mexico I.A.P. (COFEPRIS: 17 CI 09 003 142). Written informed consent was obtained from all patients.
Screening
Three graduate students (AP, SW, GES) who did not have professional experience in fundus photography acquired the images of dilated eyes using two portable Remidio-FOP cameras (Remidio Innovative Solutions Pvt Ltd, Karnataka, India). A minimum of three fundus fields (ie, posterior pole [disc and macula], nasal, temporal) were attempted to be captured for each eye using the portable devices mounted on a table stand.
The offline AI algorithm on the Remidio smartphone flagged images rated as poor quality and prompted the operator to take additional pictures of the same or near retinal view until the images were deemed acceptable by the AI system. A retinologist (ES-B, MI, JJW) performed indirect ophthalmoscopic examinations on all patients, which were often combined with a high-magnification funduscopic examination using the slit lamp. The presence of DR was determined based on the clinical information collected. A patient was positive for DR if any degree of DR was identified in at least one eye.
Image Analysis
After the DR screening campaign, image sets were fully analyzed by the Medios offline automated application integrated into the smartphone-based retinal imaging system. This application has two components: (1) an algorithm that checks the quality of the images and (2) a mechanism that generates an image-level diagnosis (or not) of DR. If an image is of marginal-to-poor quality, the image quality notification function flagged the image. However, this function could be overridden manually, allowing all patient image sets to be assessed. If at least one image in one eye was positive for any level of DR using the Medios AI software, the patient had a positive DR result.
All images captured by the two smartphones were evaluated further at a later date in masked fashion by two experienced retinologists (GES, JJW) for the presence or absence of DR. The DR status of the patients was determined using the images alone. When the two retinologists did not agree, a third, adjudicating retinologist (JB) evaluated the images to determine the presence or absence of DR.
The “ground truth” presence or absence of DR for each patient was determined using the photographic image determination combined with their clinical examination result. The “true” presence of DR was defined as any DR observed on clinical examination or DR missed on clinical examination but identified on one or more photographs by the two masked readers. The “true” absence of DR was defined as a negative clinical examination (ie, no degree of DR observed) combined with both masked readers not detecting any degree of DR in any photographs from either eye of the patient.
For the online analysis (after the campaign and masked review of images), all images (JPEGs) were uploaded to the Remidio confidential website and then sent to Eyenuk for analysis using an automated process with machine learning-enabled software, EyeArt v2.1. The architecture, data composition, and clinical validation studies for the Medios18,20 and EyeArt algorithms13,21 have been described. Each gradable patient was given a diagnosis (or not) of DR.
The Medios and EyeArt results were compared individually with the “true” presence or absence of DR (per the above criteria). A head-to-head comparison was performed in the limited number of patients for whom every image was deemed to be of good quality by Medios and whose image set was deemed to be gradable by EyeArt. This allowed for an “apples-to-apples” comparison, where all of the patient-level image criteria were met by both AI systems.
Statistical Analysis
The efficacy of the two software programs was evaluated using the following metrics: sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio (LR+), negative likelihood ratio (LR−), and Youden’s J statistic (combines sensitivity and specificity into a single measure of each of the algorithm’s performance). The closer J is to 1, the closer the software is to having no false positives and no false negatives. Each of these metrics provides unique insight into the performance of the diagnostic test. 22
Apparent prevalence of DR was calculated as the number of patients who tested positive divided by the total number of patients tested. The Rogan-Gladen estimator 23 was used to approximate true prevalence. An exact 95% confidence interval (CI) was calculated using the Clopper-Pearson approach for the point estimates of apparent prevalence, true prevalence, sensitivity, specificity, PPV, NPV, and Youden’s J. 24 For LR+ and LR−, 95% CIs were calculated using methods described by Simel et al. 25
Results
Of the 248 patients, 212 were female (mean age 56.4 years [range: 5-80]) and 36 were male (mean age 55.9 years [range: 12-73]). Mean duration of diabetes was 14.7 years (range: 1-39) among female patients and 12.9 years (range: 5-21) among male patients.
A total of 2130 images were acquired. The “ground truth” results for the presence or absence of DR were: 129 patients were diagnosed as having some level of DR and 119 patients did not have any degree of DR. The two masked readers agreed on the presence or absence of DR for 92% (228/248) of patients; images for 20 patients required the third masked reader (JB) to adjudicate and determine the presence or absence of any DR.
Of the 248 patients, Medios recognized 82 patients who had at least one poor-quality image. However, Medios was able to analyze every image, thus yielding a DR determination on all 248 patients. EyeArt was able to evaluate 46 of these patients. Of the 129 patients with “true” DR, the Medios software identified 121 patients as having DR and 8 as not having DR. Table 1 summarizes the performance metrics of the Medios software based on these results.
Table 1.
Medios Analysis (N = 248).
| Metric | Equation | Result | Point estimate | (95% CI) |
|---|---|---|---|---|
| Apparent prevalence (AP) | TP + FP / N | 121 + 7 / 248 | 0.52 | (0.45-0.58) |
| True prevalence | AP + (SP − 1) /SP + (SE − 1) | 0.52 + (0.94 − 1) / 0.94 + (0.94 − 1) | 0.52 | (0.46-0.58) |
| Sensitivity (SE) | TP / TP + FN | 121 / 121 + 8 | 0.94 | (0.88-0.97) |
| Specificity (SP) | TN / TN + FP | 112 / 112 + 7 | 0.94 | (0.88-0.98) |
| PPV | TP / TP + FP | 121 /121 + 7 | 0.95 | (0.89-0.98) |
| NPV | TN / TN + FN | 112 /112 + 8 | 0.93 | (0.87-0.97) |
| LR+ | SE / (1 − SP) | 0.94 / 1 − 0.94 | 15.95 | (7.76-32.76) |
| LR− | (1 − SE) / SP | (1 − 0.94) / 0.94 | 0.07 | (0.03-0.13) |
| J statistic | SE + SP − 1 | 0.94 +0.94 − 1 | 0.879 | (0.76-0.95) |
Abbreviations: CI, confidence interval; TP, true positive; FP, false positive; SP, specificity (estimated probability that a patient without DR tests as not having DR); DR, diabetic retinopathy; SE, sensitivity (estimated probability that a patient with “true” DR tests as having DR); FN, false negative; TN, true negative; PPV, positive predictive value (% of positive tests that are TPs); NPV, negative predictive value (% of negative tests that are TNs); LR−, negative likelihood ratio (estimate of the probability that a patient who has DR is predicted as not having DR divided by the probability that a patient who does not have DR is predicted as not having DR); LR+, positive likelihood ratio (estimate of the probability that a patient with DR is predicted as having DR divided by the probability that a patient who does not have DR is predicted as having DR).
EyeArt was able to evaluate and thus yield a DR determination in 156 of the 248 patients. The “true” results for the presence or absence of DR in this subset of patients determined that 82 patients had some level of DR; 74 patients did not have DR. EyeArt identified 87 patients as having DR and 69 patients as not having DR. Table 2 summarizes the performance metrics of the EyeArt software based on these results.
Table 2.
EyeArt Analysis (N = 156).
| Metric | Equation | Result | Point estimate | (95% CI) |
|---|---|---|---|---|
| Apparent prevalence (AP) | TP + FP / N | 77 + 10 / 156 | 0.56 | (0.48-0.64) |
| True prevalence | AP + (SP − 1) / SP + (SE − 1) | 0.56 + (0.86 − 1) / 0.86 + (0.94 − 1) | 0.53 | (0.44-0.61) |
| Sensitivity (SE) | TP / TP + FN | 77 / 77 + 5 | 0.94 | (0.86-0.98) |
| Specificity (SP) | TN / TN + FP | 64 / 64 + 10 | 0.86 | (0.77-0.93) |
| PPV | TP / TP + FP | 77 / 77 + 10 | 0.89 | (0.80-0.94) |
| NPV | TN / TN + FN | 64 / 64 + 5 | 0.93 | (0.84-0.98) |
| LR+ | SE / (1 − SP) | 0.94 / (1 − 0.86) | 6.95 | (3.89-12.40) |
| LR− | (1 − SE) / SP | (1 − 0.94) / 0.86 | 0.07 | (0.03-0.17) |
| J statistic | SE + SP − 1 | 0.94 + 0.86 − 1 | 0.804 | (0.63-0.91) |
Abbreviations: CI, confidence interval; TP, true positive; FP, false positive; SP, specificity (estimated probability that a patient without DR tests as not having DR); DR, diabetic retinopathy; SE, sensitivity (estimated probability that a patient with “true” DR tests as having DR); FN, false negative; TN, true negative; PPV, positive predictive value (% of positive tests that are TPs); NPV, negative predictive value (% of negative tests that are TNs); LR−, negative likelihood ratio (estimate of the probability that a patient who has DR is predicted as not having DR divided by the probability that a patient who does not have DR is predicted as not having DR); LR+, positive likelihood ratio (estimate of the probability that a patient with DR is predicted as having DR divided by the probability that a patient who does not have DR is predicted as having DR).
A total of 110 patients had image sets that were deemed completely to be of good quality by Medios and gradable by EyeArt. In this head-to-head comparison, the “true” results for the presence or absence of DR were: 76 patients had some level of DR and 34 patients did not have DR. Table 3 summarizes the performance metrics of the head-to-head comparison of the Medios results and the EyeArt results.
Table 3.
Head-to-Head (Medios/EyeArt) Performance (N = 110).
| Metric | Equation | Result | Point estimate | (95% CI) |
|---|---|---|---|---|
| Apparent prevalence (AP) | TP + FP / N | 75 + 4 / 110 74 + 4 / 110 |
0.72 0.69 |
(0.62-0.80) (0.60-0.78) |
| True prevalence | AP + (SP − 1) / SP + (SE − 1) | 0.72 + (0.88 − 1) / 0.88 + (0.99 − 1) 0.69 + (0.88 − 1) / 0.88 + (0.95 − 1) |
0.69 0.69 |
(0.60-0.78) (0.60-0.78) |
| Sensitivity (SE) | TP / TP + FN | 75 / 75 + 1 74 / 74 + 4 |
0.99 0.95 |
(0.93-1.00) (0.87-0.99) |
| Specificity (SP) | TN / TN + FP | 30 / 30 + 4 30 / 30 + 4 |
0.88 0.88 |
(0.73-0.97) (0.73-0.97) |
| PPV | TP / TP + FP | 75 / 75 + 4 75 / 75 + 4 |
0.95 0.95 |
(0.88-0.99) (0.87-0.99) |
| NPV | TN / TN + FN | 30 / 30 + 1 30 / 30 + 4 |
0.97 0.88 |
(0.83-1.00) (0.73-0.97) |
| LR+ | SE / (1 − SP) | 0.99 / (1 − 0.88) 0.95 / (1 − 0.88) |
8.39 8.05 |
(3.34-21.07) (3.20-20.25) |
| LR− | (1 − SE) / SP | (1 − 0.99) / 0.88 (1 − 0.95) / 0.88 |
0.01 0.06 |
(0.00-0.10) (0.02-0.16) |
| J statistic | SE + SP − 1 | 0.99 + 0.88 − 1 0.95 + 0.88 − 1 |
0.869 0.830 |
(0.65-0.97) (0.60-0.95) |
Abbreviations: CI, confidence interval; TP, true positive; FP, false positive; SP, specificity (estimated probability that a patient without DR tests as not having DR); DR, diabetic retinopathy; SE, sensitivity (estimated probability that a patient with “true” DR tests as having DR); FN, false negative; TN, true negative; PPV, positive predictive value (% of positive tests that are TPs); NPV, negative predictive value (% of negative tests that are TNs); LR−, negative likelihood ratio (estimate of the probability that a patient who has DR is predicted as not having DR divided by the probability that a patient who does not have DR is predicted as not having DR); LR+, positive likelihood ratio (estimate of the probability that a patient with DR is predicted as having DR divided by the probability that a patient who does not have DR is predicted as having DR).
Discussion
This is the first analysis to compare the head-to-head performance of two AI algorithms for detecting the presence of any degree of DR in a field setting using a portable fundus camera. Although a real-world, head-to-head analysis comparing multiple AI DR screening algorithms has been published, the analysis was performed on images obtained with a nonportable fundus camera in two strictly controlled Veterans’ Affairs primary care clinical environments and did not include the Medios AI algorithm. 26 To our knowledge, our analysis is the first to use the Medios and EyeArt AI algorithms to analyze images from patients of Spanish-Mexican and Mayan descent in the Yucatan Peninsula, each having a distinct genetic phenotype.
The offline smartphone-based Medios AI algorithm was highly sensitive and specific for detecting the presence or absence of DR in binary fashion. These results are consistent with those from previous studies.12,18,20,27 For the Medios AI analysis, the sensitivities and specificities of 0.94 and 0.94 for all 248 patients and 0.99 and 0.88 for the 110 patients in the head-to-head comparison were comparable with those detecting any DR in patients of Indian descent.12,18
For the cloud-based EyeArt AI algorithm analysis, the sensitivities and specificities of 0.94 and 0.86 for 156 patients with all gradable images and 0.95 and 0.88 in the head-to-head comparison of 110 patients were comparable with the results of previous reports for the presence of any DR and sight-threatening DR for English,28-30 Indian, 21 and American patients.31,32 Interestingly, both AIs had similar sensitivities and specificities despite being trained by different methodologies.13,22,23 This is made more remarkable because DR phenotypes differ by region and ethnicity, 3 and the two AI algorithms used in this analysis were trained on different international DR data sets of patients of Indian descent14,20,27 for Medios and of American and Northern Mexican descent13,33 for EyeArt.
Because three inexperienced graduate students were tasked as camera operators, we confirmed that once they became familiar with the technology, the Remidio-FOP camera with accompanying Medios AI was easy to use, as previously described.12,20 Interestingly, all 248 patients had images that Medios was able to analyze. This can be explained by its proprietary, two-step process: (1) an upstream image quality module with a per image notification function and (2) a downstream image automated analysis application module trained to recognize any sign of DR. Sensitivity increases at the potential expense of specificity. Although any image set could be deemed of marginal quality based on a single image of poor quality, the images could still be subjected to the evaluation module at the camera operator’s discretion. Thus, Medios was able to evaluate all 248 patients, even though 82 patients had at least one poor-quality image. Only 67% of patients had a complete set of good-quality images. While there may be potential safety issues (eg, increased number of false negatives) with a strategy that allows for the analysis of images of marginal-to-poor quality, the overall benefit in an outreach field setting—where media opacities, patient compliance, and image acquisition speed have a greater impact—cannot be marginalized. Many of the 82 patients with marginal-quality images had obvious DR stigmata identified by the two masked readers. However, portions of the image often were partially obscured by vitreous or preretinal hemorrhage, asteroid hyalosis, or cataract; thus, the images were deemed to be of marginal quality. The greater acquisition of fundus images, including those of marginal quality, can potentially improve the quality of the screening process by identifying more patients with sight-threatening DR and perhaps early DR.
EyeArt evaluated 63% (156/248) of the patients, primarily owing to camera operator error. During the screening campaign, the three camera operators were untrained and not familiar with EyeArt’s mandatory image-specific and patient-specific capture criteria. 21 EyeArt processes are designed to produce patient-level results, not individual image-level results. Therefore, if any image was considered ungradable, the entire patient encounter was deemed ungradable because EyeArt does not skip ungradable images. Similar to Medios, EyeArt is trained to reduce the incidence of false negatives. However, EyeArt has a one-step AI algorithm that combines image quality/gradeability and image analysis for the presence of disease. As such, variations in image alignment, resolution, and exposure; not having a macula-centered image per eye; monocular status; or having >14 images per patient resulted in an ungradable encounter and a recommendation for referral to an eye specialist.
The high rate of ungradable images by EyeArt is in contrast to the results of a study conducted in primary care, general ophthalmology, and retina specialty centers with trained photographers using a nonportable tabletop Canon camera. 33 This study, which excluded participants with persistent visual impairment, demonstrated a 97.4% dilated eye gradable rate. 32 Another possible reason for EyeArt’s high ungradable rate was the high incidence of media opacities observed. Incidence rates for these conditions tend to be higher in underserved outreach field settings. In addition, these imaging factors underscore the need for dilation in an outreach setting, as the incidence of ungradability has been shown to be substantially higher for nonmydriatic vs mydriatic images.32,34
In the head-to-head comparison—where all images for every patient were deemed of sufficient quality for grading by both AI algorithms—the sensitivity, specificity, and PPVs/NPVs of Medios and EyeArt were comparable and highly accurate. This very high degree of DR detection accuracy (ie, sensitivity) may be a function, at least in part, of the high DR prevalence (69%) in this cohort. These results are not typical in Western populations and are more representative of underserved populations. The high degree of sensitivity also could be linked to the screening of patients of Mexican-Spanish and Mayan descent who have a greater degree of background contrast from choroidal melanin and fundus hyperpigmentation, which is often not seen in the blonde fundi of Western populations. 19 This also is in contrast to a study showing lower specificities when a significant portion of the screened population had mild nonproliferative retinopathy or no retinopathy. 29 Thus, provided that online access is available, either AI algorithm should be adequate at providing a relatively immediate and robust determination of the presence or absence of DR in a large-scale outreach campaign.
The primary limitation of this analysis is that the definition of “ground truth” DR was unconventionally determined by clinical examination findings combined with image analysis performed by two masked, fellowship-trained, vitreoretinal specialists, each with >30 years’ experience. In previous reports evaluating both AI algorithms, three 45° and four wide-field tabletop fundus photographs evaluated by masked or expert readers were used to define “ground truth” DR.12,32 However, these conventional methods were not feasible in a large-scale, real-world, five-day DR screening campaign in an outreach setting. Furthermore, by adding the clinical examination (including a 20-diopter indirect ophthalmoscopic evaluation) into the definition of “ground truth” DR, eyes with DR primarily limited to the pre-equatorial fundus were properly characterized. Although both AI algorithms were trained using only images of the posterior pole, the pattern recognition and deep-learning ability of each algorithm could potentially allow each to grade an image as having some degree of DR in the absence of any classic stigmata of DR visible to the naked eye on a photographic image. Pathology, including microvascular ischemia (in the absence of microaneurysm, intraretinal hemorrhage, or cotton wool spot), choriocapillary ischemic thinning, and primary neuronal cell loss may lead to retinal thinning, which is only recognizable by deep learning. As such, our definition of “ground truth,” which accounts for these possibilities, potentially increased the accuracy of the results.
Other limitations included using untrained camera operators pressed for time and screening a smaller number of patients who had images deemed acceptable by both AI algorithms. This analysis was limited to detecting any degree of DR in a binary fashion and did not include grading the level of retinopathy if present. This was necessary, as the clinical examination reports did not segregate referable DR from mild nonproliferative DR. In addition, Medios is only trained to recognize the presence of moderate nonproliferative diabetic retinopathy or worse or no DR as a binary outcome. Nonetheless, detecting any level of DR in this underserved and likely poorly controlled population with diabetes takes on an added level of importance where rapid diabetic retinal disease progression and blindness from cataract formation and glaucoma are more probable. 35
Conclusions
Our results demonstrate that both Medios and EyeArt AI-enabled algorithms can be effective in achieving high accuracy in an outreach field setting where portable fundus cameras are used and where medical professionals and other resources are limited. These robust findings suggest this same methodology of DR screening could be readily implemented in any office setting or location in the United States. The benefit of immediately determining, in an offline manner with Medios, the presence or absence of DR cannot be overstated. Both AIs should be considered equally viable options in large-scale DR screening campaigns where rapid results are needed and, in the case of EyeArt, online access is possible.
Acknowledgments
The authors acknowledge Medios Technologies, Singapore, for providing an oral description of the technical design of their software and assistance with image transfers. The authors thank Eyenuk, Los Angeles, CA, for providing a description of EyeArt’s functionality and technical support in the submission of acquired images to EyeArt. Linda Goldstein, PhD, CMPP, provided medical writing assistance funded by Retinacare International.
Footnotes
Abbreviations: AI, artificial intelligence; AP, apparent prevalence; CI, confidence interval; DR, diabetic retinopathy; FN, false negative; FOP, fundus on phone; FP, false positive; LR−, negative likelihood ratio; LR+, positive likelihood ratio; NPV, negative predictive value; PPV, positive predictive value; SE, sensitivity; SP, specificity; TN, true negative; TP, true positive.
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. None of the authors have any conflicts of interest to disclose. No author has a relationship of any kind with Remidio, Medios Technologies, or Eyenuk. The two Remidio-FOP cameras and smartphones were purchased by and are the property of Retina Care International. Retina Care International did not receive any funding of any kind for this study. Medios Technologies and Eyenuk provided the AI for use in the study and had no role in the study design, funding, execution, data collection and analysis, or publication.
Funding: The authors received no financial support for the research, authorship, and/or publication of this article.
ORCID iDs: John J. Wroblewski 
https://orcid.org/0000-0002-0600-8666
George E. Sanborn
https://orcid.org/0000-0002-0444-6988
References
- 1. International Diabetes Federation. IDF diabetes atlas. https://diabetesatlas.org/en/. Accessed August 11, 2021.
- 2. Kahn HA, Hiller R. Blindness caused by diabetic retinopathy. Am J Ophthalmol. 1974;78(1):58-67. [DOI] [PubMed] [Google Scholar]
- 3. Teo ZL, Tham YC, Yu M, et al. Global prevalence of diabetic retinopathy and projection of burden through 2045: systematic review and meta-analysis. Ophthalmology. 2021;128(11):1580-1591. [DOI] [PubMed] [Google Scholar]
- 4. Villalpando S, de la Cruz V, Rojas R, et al. Prevalence and distribution of type 2 diabetes mellitus in Mexican adult population: a probabilistic survey. Salud Publica Mex. 2010;52(suppl 1):S19-S26. [DOI] [PubMed] [Google Scholar]
- 5. Mendoza-Herrera K, Quezada AD, Pedroza-Tobias A, Hernandez-Alcaraz C, Fromow-Guerra J, Barquera S. A diabetic retinopathy screening tool for low-income adults in Mexico. Prev Chronic Dis. 2017;14:E95. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Pan American Health Organization. Health in the Americas+, 2017 Edition. Summary: Regional Outlook and Country Profiles. World Health Organization; 2017. https://iris.paho.org/handle/10665.2/34321. [Google Scholar]
- 7. Polack S, Yorston D, López-Ramos A, et al. Rapid assessment of avoidable blindness and diabetic retinopathy in Chiapas, Mexico. Ophthalmology. 2012;119(5):1033-1040. [DOI] [PubMed] [Google Scholar]
- 8. Rajalakshmi R, Arulmalar S, Usha M, et al. Validation of smartphone based retinal photography for diabetic retinopathy screening. PLoS ONE. 2015;10(9):e0138285. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Russo A, Morescalchi F, Costagliola C, Delcassi L, Semeraro F. Comparison of smartphone ophthalmoscopy with slit-lamp biomicroscopy for grading diabetic retinopathy. Am J Ophthalmol. 2015;159(2):360-364.e1. [DOI] [PubMed] [Google Scholar]
- 10. Ryan ME, Rajalakshmi R, Prathiba V, et al. Comparison Among Methods of Retinopathy Assessment (CAMRA) study: smartphone, nonmydriatic, and mydriatic photography. Ophthalmology. 2015;122(10):2038-2043. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Malerbi FK, Andrade RE, Morales PH, et al. Diabetic retinopathy screening using artificial intelligence and handheld smartphone-based retinal camera. J Diabetes Sci Technol. 2022;16(3):716-723. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Sengupta S, Sindal MD, Baskaran P, Pan U, Venkatesh R. Sensitivity and specificity of smartphone-based retinal imaging for diabetic retinopathy: a comparative study. Ophthalmol Retina. 2019;3(2):146-153. [DOI] [PubMed] [Google Scholar]
- 13. Bhaskaranand M, Ramachandra C, Bhat S, et al. Automated diabetic retinopathy screening and monitoring using retinal fundus image analysis. J Diabetes Sci Technol. 2016;10(2):254-261. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Gulshan V, Peng L, Coram M, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016;316(22):2402-2410. [DOI] [PubMed] [Google Scholar]
- 15. Abràmoff MD, Lavin PT, Birch M, Shah N, Folk JC. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digit Med. 2018;1:39. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Gulshan V, Rajan RP, Widner K, et al. Performance of a deep-learning algorithm vs manual grading for detecting diabetic retinopathy in India. JAMA Ophthalmol. 2019;137(9):987-993. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Lim JI, Regillo CD, Sadda SR, et al. Artificial intelligence detection of diabetic retinopathy: subgroup comparison of the EyeArt system with ophthalmologists’ dilated examinations. Ophthalmol Sci. 2023;3(1):100228. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Natarajan S, Jain A, Krishnan R, Rogye A, Sivaprasad S. Diagnostic accuracy of community-based diabetic retinopathy screening with an offline artificial intelligence system on a smartphone. JAMA Ophthalmol. 2019;137(10):1182-1188. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Raman R, Srinivasan S, Virmani S, Sivaprasad S, Rao C, Rajalakshmi R. Fundus photograph-based deep learning algorithms in detecting diabetic retinopathy. Eye. 2019;33(1):97-109. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Sosale B, Sosale AR, Murthy H, Sengupta S, Naveenam M. Medios—an offline, smartphone-based artificial intelligence algorithm for the diagnosis of diabetic retinopathy. Indian J Ophthalmol. 2020;68(2):391-395. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Rajalakshmi R, Subashini R, Anjana RM, Mohan V. Automated diabetic retinopathy detection in smartphone-based fundus photography using artificial intelligence. Eye. 2018;32(6):1138-1144. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Youden WJ. Index for rating diagnostic tests. Cancer. 1950;3(1):32-35. [DOI] [PubMed] [Google Scholar]
- 23. Rogan WJ, Gladen B. Estimating prevalence from the results of a screening test. Am J Epidemiol. 1978;107(1):71-76. [DOI] [PubMed] [Google Scholar]
- 24. Clopper CJ, Pearson ES. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika. 1934;26:404-413. [Google Scholar]
- 25. Simel DL, Samsa GP, Matchar DB. Likelihood ratios with confidence: sample size estimation for diagnostic test studies. J Clin Epidemiol. 1991;44(8):763-770. [DOI] [PubMed] [Google Scholar]
- 26. Lee AY, Yanagihara RT, Lee CS, et al. Multicenter, head-to-head, real-world validation study of seven automated artificial intelligence diabetic retinopathy screening systems. Diabetes Care. 2021;44(5):1168-1175. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. Sosale B, Aravind SR, Murthy H, et al. Simple, Mobile-based Artificial Intelligence Algorithm in the detection of Diabetic Retinopathy (SMART) study. BMJ Open Diabetes Res Care. 2020;8(1):e000892. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. Bhaskaranand M, Ramachandra C, Bhat S, et al. The value of automated diabetic retinopathy screening with the EyeArt system: a study of more than 100,000 consecutive encounters from people with diabetes. Diabetes Technol Ther. 2019;21(11):635-643. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Heydon P, Egan C, Bolter L, et al. Prospective evaluation of an artificial intelligence-enabled algorithm for automated diabetic retinopathy screening of 30 000 patients. Br J Ophthalmol. 2021;105(5):723-728. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Olvera-Barrios A, Heeren TF, Balaskas K, et al. Diagnostic accuracy of diabetic retinopathy grading by an artificial intelligence-enabled algorithm compared with a human standard for wide-field true-colour confocal scanning and standard digital retinal images. Br J Ophthalmol. 2021;105(2):265-270. [DOI] [PubMed] [Google Scholar]
- 31. Mokhashi N, Grachevskaya J, Cheng L, et al. A comparison of artificial intelligence and human diabetic retinal image interpretation in an urban health system. J Diabetes Sci Technol. 2022;16(4):1003-1007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32. Ipp E, Liljenquist D, Bode B, et al. Pivotal evaluation of an artificial intelligence system for autonomous detection of referrable and vision-threatening diabetic retinopathy. JAMA Netw Open. 2021;4(11):e2134254. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. Cuadros J, Bresnick G. EyePACS: an adaptable telemedicine system for diabetic retinopathy screening. J Diabetes Sci Technol. 2009;3(3):509-516. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34. Piyasena M, Yip JLY, MacLeod D, Kim M, Gudlavalleti VSM. Diagnostic test accuracy of diabetic retinopathy screening by physician graders using a hand-held non-mydriatic retinal camera at a tertiary level medical clinic. BMC Ophthalmol. 2019;19(1):89. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35. Shah P, Mishra DK, Shanmugam MP, Doshi B, Jayaraj H, Ramanjulu R. Validation of Deep Convolutional Neural Network-based algorithm for detection of diabetic retinopathy—artificial intelligence versus clinician for screening. Indian J Ophthalmol. 2020;68(2):398-405. [DOI] [PMC free article] [PubMed] [Google Scholar]