Diabetic Retinopathy Screening Using Artificial Intelligence and Handheld Smartphone-Based Retinal Camera

Abstract

Background:

Portable retinal cameras and deep learning (DL) algorithms are novel tools adopted by diabetic retinopathy (DR) screening programs. Our objective is to evaluate the diagnostic accuracy of a DL algorithm and the performance of portable handheld retinal cameras in the detection of DR in a large and heterogenous type 2 diabetes population in a real-world, high burden setting.

Method:

Participants underwent fundus photographs of both eyes with a portable retinal camera (Phelcom Eyer). Classification of DR was performed by human reading and a DL algorithm (PhelcomNet), consisting of a convolutional neural network trained on a dataset of fundus images captured exclusively with the portable device; both methods were compared. We calculated the area under the curve (AUC), sensitivity, and specificity for more than mild DR.

Results:

A total of 824 individuals with type 2 diabetes were enrolled at Itabuna Diabetes Campaign, a subset of 679 (82.4%) of whom could be fully assessed. The algorithm sensitivity/specificity was 97.8 % (95% CI 96.7-98.9)/61.4 % (95% CI 57.7-65.1); AUC was 0·89. All false negative cases were classified as moderate non-proliferative diabetic retinopathy (NPDR) by human grading.

Conclusions:

The DL algorithm reached a good diagnostic accuracy for more than mild DR in a real-world, high burden setting. The performance of the handheld portable retinal camera was adequate, with over 80% of individuals presenting with images of sufficient quality. Portable devices and artificial intelligence tools may increase coverage of DR screening programs.

Introduction

Diabetic retinopathy (DR) screening is considered one of the most cost-effective initiatives on diabetes care, ¹ fitting into the World Health Organization criteria for the screening of chronic disease; its importance in preventing blindness is well established. ² However, implementation of broad public health DR screening programs remains a substantial challenge where there are insufficient health care structure and resources in terms of funding, trained health care human resources, and facilities. ³

Novel portable smartphone-based retinal cameras are readily available, low-cost, easy-to-use devices with validated sensitivity and specificity for DR screening; their implementation is feasible and realistic. ⁴

Artificial intelligence (AI) systems for DR screening/grading have the potential to increase further the accessibility of DR screening for people with diabetes and increase diagnostic accuracy, efficiency, productivity, reproducibility, and outcomes; ⁵ a robust economic rationale has been found for the use of deep learning (DL) systems as assistive tools for DR screening. ⁶ Such characteristics confer to AI systems a great relevance especially in low-income, underserved areas with insufficient access to ocular exams, as is the case of Itabuna, Northeast Brazil, where patients rely on the Itabuna Diabetes Campaign, a once-a-year initiative that offers DR screening, treatment, and counseling. Involving mass gatherings of patients and retina specialists, for most patients it was the sole opportunity for the annual eye exam. During the 2019 event, a strategy based on handheld portable cameras with an embedded AI algorithm compatible with telemedicine was evaluated; if successful, such a strategy could allow for a more dispersed, continuous screening intervention that would not require mass gatherings.

The study aims to evaluate the diagnostic accuracy of a semiautomated strategy of DR screening with mobile handheld retinal cameras and an AI algorithm, using a manual grading as the independent reference standard, in a real-life, high burden setting in an area with scarce healthcare resources in Northeast Brazil.

Methods

Study Design, Population, and Setting

This retrospective study enrolled a convenience sample of individuals aged over 18 years old with a previous type 2 diabetes mellitus (T2DM) diagnosis who attended the Itabuna Diabetes Campaign, an event that took place on November 23, 2019 at the city of Itabuna, Bahia State, Northeast Brazil (Latitude 14^o 47’08’’ longitude 39º 16’49’’W). This annual event mobilizes a significant amount of the city’s inhabitants and involves diabetes awareness, counseling, screening, and treatment of diabetes complications. The study protocol was approved by the ethics Committee of Federal University of São Paulo (# 1260/2015) and was conducted in compliance with the Declaration of Helsinki, following the institutional ethics committees. After signing informed consent, participants answered a questionnaire for demographic and self-reported clinical characteristics and underwent ocular imaging.

Imaging

Smartphone-based hand-held devices (Eyer, Phelcom Technologies, São Carlos, Brazil) were used for fundus photography and image acquisition: two images of the posterior segment—one centered on macula and another disc centered (45° field of view)—were captured for each eye, after mydriasis induced by 1% tropicamide eye-drops. ⁷ Image acquisition was performed by a team of nine examiners, including medical students, with variable degrees of previous experience in this kind of procedure.

Image Grading

Remote image reading was performed at EyerCloud platform (Phelcom Technologies, São Carlos, Brazil) by a single retinal specialist (FKM) after anonymization. The photographs were evaluated for quality, regarding transparence of the media, focus, and image boundaries, and classified as gradable or ungradable images. ⁷ The lack of images with representative fields was considered as protocol failure. Subsequently, DR and maculopathy classification was manually determined from fundus photographs for those individuals with gradable images. Classification of DR was given per individual, considering the most affected eye, according to a strict standardized protocol (Table 1) ⁸ ; apparently present macular oedema was defined as apparent retinal thickening or hard exudates in posterior pole. ⁹ For analysis, classification grades were combined into no or mild non-proliferative DR (NPDR) versus moderate NPDR, severe NPDR, proliferative DR, or apparently present macular oedema (more than mild DR, mtmDR) in at least one eye. Vision-threatening DR (vtDR) was considered as the presence of severe NPDR, proliferative DR, or apparently present macular oedema in at least one eye. ⁹ Individuals with images of insufficient quality were excluded from analysis, except cases when one gradable eye was classified as vTDR. No information other than ocular images was available for the reader.

Table 1.

Diabetic Retinopathy Severity Levels and Distribution Among Patients.

Diabetic retinopathy severity level	Distribution (%)	Lesions
Absent	62.7	No alterations
Mild NPDR	11.0	At least one hemorrhage or microaneurysm
Moderate NPDR	10.0	Four or more hemorrhages in only one hemi-field a
Severe NPDR	5.1	Any of the following:
		- Four or more hemorrhages in the superior and inferior hemi-fields
		- Venous beading
		- Intraretinal microvascular abnormalities (IRMA)
Proliferative diabetic retinopathy	10.6	Any of the following:
		- Active neovessels
		- Vitreous hemorrhage

Abbreviation: NPDR, non-proliferative diabetic retinopathy.

^aSuperior and inferior hemi-fields separated by the line passing through the center of the macula and the optic disc.

Automated Detection of DR

Images from 824 individuals, totalizing 3,255 color images, were graded by a DL–enhanced method (PhelcomNet), a modified version of the convolutional neural network (CNN) Xception, ¹⁰ with different input and output parameters but the same intermediate convolutional layers. The input was changed to receive images of size 699 × 699 × 3 RGB, and two fully connected layers of 2100 neurons were added at the top. Finally, two neurons with softmax activation classified the network input according to class. Softmax normalized the respective neuron input values, creating a probabilistic distribution in which the sum will be 1. Therefore, the neuron with the highest value identified the class to which the evaluated image belongs.

The PhelcomNet prediction score was defined as x, corresponding to the interval between 0 and 1, which indicates the likelihood of DR. In contrast to the reference standard, the device puts both no DR and mild DR into one grade; all the other levels are considered more than mild DR. Since there are up to four images per individual, each of the images has a score indicating a probability of retinal alteration. In order to obtain a single score for each individual considering all outputs, we used a linear combination y of each prediction for image x₁, x₂, x₃, x₄ordered in descending order—x₁ being the highest and x₄ being the lowest score. In this case, four linear combinations were employed, depending on the number of images per individual and defined as:

$$\begin{array}{l}{y}_4=0.5x_1+0.25x_2+0.125x_3+0.125x_4\hfill \\ y_3=0.5x_1+0.25x_2+0.25x_3\hfill \\ y_2=0.5x_1+0.5x_2\hfill \\ y_1=1.0x_1\hfill \end{array}$$

The inspiration for these linear combinations is that a larger weight for predictions with a higher probability makes it more significant than the others, since fundus images with higher DR evidence tend to be more important on human assessment.

The PhelcomNet went through a training process that evaluated each image individually, progressively adjusting its internal parameters values to obtain an output from the softmax layer closest to the class to which the image belongs. For the algorithm training, a dataset of 10,569 fundus images captured between 2019 and 2020, exclusively using the Phelcom Eyer device (resolution 1600 × 1600 × 3 RGB) was used. For validation, 20% of these images were used to periodically evaluate the performance of the network. Images were separated into two classes: those containing only images from normal eyes (Class 0), and those containing images with retinal alterations (Class 1).

To add more diversity to the data, transformations were applied to images before CNN evaluation, a technique known as data augmentation. Rotation, width and height shift, zoom, and brightness values were randomly applied.

GradCam Heatmap

In order to visualize the location of the most important regions obtained by CNN, as a means to discriminate between classes, the Gradient-Based Class Activation Map (GradCam) was used ¹¹ ; it generates a heat map with the values obtained on the last convolutional layer (Figure 1). The training set was split between Classes 0 and 1, rendering the lesions as the most important discriminatory region for classes to be differentiated from each other; this is notably highlighted on the heat map. The equation to obtain the weights of importance of a neuron is described elsewhere. ¹¹

Figure 1.

Example of heatmap visualization using gradient method. (a) Color fundus photograph depicting hard exudates and microaneurysms in the macular region, suggesting the possibility of diabetic macular oedema. (b) Overlay with the GradCam heatmap visualization can aid in making a diagnosis as the modifications are flagged in a color scale, from blue (low importance) to red (high importance).

Statistical Analysis

Data were collected in MS Excel 2010 files (Microsoft Corporation, Redmond, WA, USA). Statistical analyses were performed using SPSS 19.0 for Windows (SPSS Inc., Chicago, IL, USA). Individual’s characteristics and quantitative variables are presented in terms of mean and standard deviation (SD). A paired two-tailed Student t test was used to compare continuous clinical variables between the two groups. Fisher’s exact or chi-square tests were used for unpaired variables. The 5% level of significance was used. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV), and their 95% confidence intervals (CIs), were calculated for the device outputs with no or mild DR and mtmDR, compared with the corresponding reference standard classifications ⁹ ; comparison was made against human reading as the ground-truth. The 0.75 threshold was chosen as the operating point (see Supplementary Material). Diagnostic accuracy is reported according to the Standards for Reporting of Diagnostic Accuracy Studies (STARD). ¹²

Results

Nine hundred and forty individuals aged over 18 years old with a previous T2DM diagnosis were assessed. Average age was 60.8 + 11.4 years, and diabetes duration was 10.4 + 8.7 years; the majority of participants were women (64.9%). Use of insulin was reported in 25.8% of individuals, systemic blood hypertension was present in 68.4%, and smoking habit was present in 48.4%. The vast majority of participants relied on the public health system (94.1%); most individuals were illiterate or had incomplete primary education (54.4%).

Digital fundus photography images were obtained for both eyes of 824 individuals, 145 of whom (17.6%) could not be graded by the specialist reader due to insufficient quality; age (P < .001) and diabetes duration (P = .001) were inversely associated with gradeability. The remaining 679 individuals had DR classified as follows: absent 426 (62.7%), mild NPDR 75 (11.0%), moderate NPDR 71 (10.0%), severe NPDR 35 (5.1%), and proliferative DR 72 (10.6%) (Table 1). Diabetic macular oedema was detected in 25.4% of individuals. Among individuals with ungradable images as per the specialist reader, the algorithm gave an output of insufficient quality for one participant. Hence, a total of 679 individuals had their images with sufficient quality for classification by both the specialist reader and the AI system (Figure 2).

Figure 2.

Waterfall diagram. Standards for Reporting of Diagnostic Accuracy Studies (STARD) diagram for the algorithm mtmDR output. mtmDR, more than mild diabetic retinopathy.

The sensitivity/specificity, per the human grading standard, for the device to detect mtmDR was 97.8% (95% CI 96.7-98.9)/61.4% (95% CI 57.7-65.1). PPV and NPV for mtmDR were 48.5% (95% CI 44.75-52.27) and 98.7% (95% CI 97.85-99.55), respectively (Table 2). Diagnostic accuracy was 71.3% (95% CI 67.91-74.69). Area under the receiver operating characteristic (ROC) curve was 0.89 (Figure 3).

Table 2.

Confusion Matrix for Reference Standard According to Human Grading and Device Output; Sensitivity, Specificity, and Predictive Values of the Algorithm.

Device output
Human reference standard	mtmDR	Not mtmDR	Total	Sensitivity, specificity
mtmDR	180	4	184	97.8% sensitivity
Not mtmDR	191	304	495	61.4% specificity
All	371	308	679
Predictive value	48.5% PPV	98.7% NPV

Abbreviations: mtmDR, more than mild diabetic retinopathy; NPV, negative predictive value; PPV, positive predictive value.

Figure 3.

Receiver operating characteristic (ROC) curve of the artificial intelligence device for detection of more than mild diabetic retinopathy (mtmDR). Area under the curve = 0.89.

All cases of false negative or positive exams were individually reviewed. There were four false negative exams for the algorithm output according to the human grading reference standard. All of them were classified as moderate NPDR in the most affected eye, with two individuals having apparently present macular oedema.

Discussion

The results show that a portable smartphone-based retinal camera with an AI algorithm achieved a good diagnostic accuracy for detecting mtmDR, compared to an independent reference standard, in a high burden setting. A review of false negatives did not reveal any case of severe NPDR or proliferative DR. We believe that high sensitivity and the setting of mtmDR as the cutoff point allows safe use in a screening strategy (Figure 3); the sensitivity reached by the current algorithm compares well with previous reports.^4,13,14

Implementation and maintenance of DR screening programs worldwide are challenged by financial and workforce constraints; ⁶ under-resourced health systems, particularly in low- to middle-income countries that face severe worker and infrastructure shortages,^15,16 need rational and cost-effective strategies to deal with the increasing demand brought by growing global prevalence of diabetes mellitus and to overcome social and economic barriers; telemedicine and AI have been established as successful and cost-effective strategies for DR screening,^6,17 helping increase program coverage and assisting in the detection of vtDR. ¹⁵

The high sensitivity found in the present study also allows for a semiautomated strategy, with algorithmic assistance increasing efficiency of non-specialist clinicians to diagnose at scale and reducing the workload of the specialist in a high burden setting, as it directs attention to concerning features consistently across large data sets, for example, through heatmaps (Figure 1). ¹⁸ Interestingly, the current algorithm comes embedded in the handheld device and works offline, which allows for a potential point-of-care DR detection.

The algorithm specificity was somewhat lower than previously reported, ^4,13,14 and a review of false positives revealed mainly fundus pigment changes or image artifacts flagged as pathological changes by the algorithm (Figure 4); such an outcome brings insights between the differences of a deep neuronal network and biological intelligence, the former resulting from a feed-forward approach and the latter being characterized by context-sensitive checking. ¹⁹

Figure 4.

Example of a false positive case. (a) Color fundus photograph depicting inferior pigmentary change. (b) Overlay with the GradCam heatmap; the pigmentary alteration is flagged by the algorithm, leading to a false positive output.

Adequate image quality is a major factor upon which the success of the screening strategy is dependent, ²⁰ and the quality of images obtained with portable devices relies heavily on the training of the operator, ²¹ among other variables. Handheld portable retinal cameras are becoming increasingly popular because of its lower cost in comparison with traditional tabletop retinal cameras, and have been shown to help increase access to DR screening.^22-24 In the present study, the team responsible for image acquisition was heterogenous and composed of fully trained members as well as volunteers with no previous experience; there was a high flow of individuals due to the dynamics of the Itabuna Diabetes Campaign, a high-burden setting with more than 900 individuals in a period of six hours. Nevertheless, in the present study, the overall ungradability rate of 17.6% for the specialist reading compares favorably with a recent experience in rural India of 34%. ²⁵ Variables related to the individual profile also have influence on image quality, namely age, diabetes duration, poor collaboration, poor mydriasis and media opacities, especially cataracts: ²⁶ gradeability was inversely associated with age or diabetes duration.

Individuals enrolled in the present study adequately represent the population with T2DM treated in primary health care, characterized by the predominance of the elderly, mostly women and diabetes duration over 10 years. Arterial hypertension (68.4%) was the most prevalent comorbidity associated with T2DM, but smoking habit (48.4%) was reported at a higher rate than in previous reports. ²⁷ The relatively high rate of apparently present diabetic macular oedema may reflect the low level of healthcare access in the region: Brazil contains the fifth largest population of patients with diabetes worldwide, ²⁸ and data from a Brazilian multicenter study indicate that less than half of T2DM patients followed at reference centers reached the glycemic goal. ²⁷ In its Northeast region, access is low to both public healthcare and education,^27,29 and the region presented the higher mortality rate due to diabetes in the country, possible due to lack of access to healthcare, according to a study on the burden of T2DM in Brazil. ³⁰ Data on other diabetes complications are scarce in the region, but the reported prevalence of diabetic foot disease, foot ulcers, and amputation in a previous edition of the Itabuna Diabetes Campaign was 20.6%, 5.8%, and 1.0%, respectively; ³¹ a study performed in a Northeastern Brazilian capital revealed that the most frequent cause of end-stage renal disease was diabetes. ³² Most individuals in our sample relied exclusively on the public health system (94.1%), were illiterate, or had incomplete primary education (54.4%). Due to the lack of a standardized DR screening protocol in the Brazilian public health system, as an effort to raise access to DR screening and mitigate diabetic blindness, some initiatives based on volunteer work have emerged, and the Itabuna Diabetes Campaign is one successful example; besides diabetes complications treatment and prevention, it helps raise diabetes awareness. Its model, which relies on the retina status as a biomarker of other diabetes complications, was replicated in all five regions of Brazil: during November 2019, Diabetes Campaigns occurred in 24 cities distributed throughout 15 Brazilian states, in commemoration of the month of diabetes awareness and prevention, as established by the International Diabetes Federation ³³ ; however, with the COVID-19 pandemic, an alternative to massive gatherings is needed. ³⁴ The current strategy presented herein offers an alternative suitable for this new circumstance, with telemedicine allowing a more dispersed, continuous screening action that does not rely on the presence of specialists, which are scarce in underserved areas; additionally, portable cameras may be operated outside the clinic, even outdoors, rendering this strategy adequate during times of social distancing. A model based on semiautomated screening with mobile units, portable devices, and AI in a primary care setting could contribute to address this novel challenge.

This study has important strengths, as it brings the results of a real-life situation of high burden DR screening in an underserved region of Brazil and a system that involved portable retinal cameras and algorithmic analysis; to the best of our knowledge, this is the first report of such a strategy that could increase access of DR screening in a relevant and cost-effective manner in the country that hosts the fifth most significant population of patients with diabetes on the planet. Of note, such a strategy has already been successfully used in 2020, allowing the first major DR screening initiative after the pandemic.

Our study has several limitations, the most notable of which is that human grading was performed by only one specialist, a potential source of bias. Nevertheless, we believe that the goal of evaluating the potential of AI as an assistive tool for DR screening in a high-burden setting was accomplished; our conclusions point to the relevant role of technology in increasing access to quality healthcare. Additionally, cataracts were not documented or classified systematically, raising concerns on the extent of their influence on gradeability of retinal images. Furthermore, diabetic maculopathy was not evaluated with gold standard methods; instead, its presence was inferred in non-stereoscopic images. The relatively high rate of apparently present diabetic macular oedema found in our sample may be related to the methodology; it may also have been the result of overcall. Finally, the lack of comprehensive clinical and laboratory data is also a limitation of the current study.

Conclusion

In conclusion, the present study shows the feasibility and presents favorable results of a strategy based on retinal images acquired with low-cost, portable devices and automated algorithmic analysis for DR screening in a high burden setting. Such features, combined with telemedicine, may constitute a cost-effective model for middle- to low-income countries where there is insufficient access to ocular healthcare, in a sustainable, continuous manner, as opposed to episodic screening events. The high sensitivity achieved with the algorithm offers the possibility of point-of-care triage. Other challenges remain, such as establishing and complying with the legal and regulatory framework related to algorithm analysis in each healthcare environment. Additionally, DR screening should be envisioned as a part of promoting diabetes awareness, education, and optimizing clinical control; finally, as postulated decades ago, screening should only be offered as long as the health system can provide proper treatment to all detected cases.