Abstract
PURPOSE:
The objective of this study is to evaluate the sensitivity, specificity, and accuracy of artificial intelligence (AI) for diabetic retinopathy (DR) screening in community in Binh Dinh Province in Vietnam.
MATERIALS AND METHODS:
This retrospective, descriptive, cross-sectional study assessed the DR screening efficacy of EyeArt system v2.0 by analyzing 2332 nonmydriatic digital fundus pictures of 583 diabetic patients from hospitals and health centers in Binh Dinh province. First, we selected thirty patients with 120 digital fundus pictures to perform the Kappa index by two eye doctors who would be responsible for the DR clinical feature evaluation and DR severity scale classification. Second, all digital fundus pictures were coded and then sent to the two above-mentioned eye doctors for the evaluation and classifications according to the International Committee of Ophthalmology’s guidelines. Finally, DR severity scales with EyeArt were compared with those by eye doctors as a reference standard for EyeArt’s effectiveness. All the data were analyzed using the SPSS software version 20.0. Values (with confidence interval 95%) of sensitivity, specificity, positive predictive value, negative predictive value, and accuracy were calculated according to DR state, referable or not and vision-threatening DR state or not. P < 0.05 was considered statistically significant.
RESULTS:
The sensitivity and specificity of EyeArt for DR screening were 94.1% and 87.2%. The sensitivity and specificity for referable DR and vision-threatening DR were 96.6%, 90.1%, and 100.0%, 92.2%. Accuracy for DR screening, referable DR, and vision-threatening DR were 88.9%, 91.4%, and 93.0%, respectively.
CONCLUSION:
EyeArt AI was effective for DR screening in community.
Keywords: Accuracy, artificial intelligence, diabetic retinopathy, sensitivity, specificity
Introduction
Diabetic retinopathy (DR) is one of the leading causes of blindness and vision impairment in adults aged over 50 years old in the world,[1] with a prevalence increasing from 14.9% in 1990 to 18.5% in 2020. According to the statistics of the International Committee of Ophthalmology (ICO), there was one in three people with diabetes has a DR and 10.2% of people with diabetes having a vision-threatening DR.[2]
The gold standard of DR diagnosis was the 7-field color stereoscopic fundus imaging or fluorescein angiography (FA) in accordance with the guidelines of Early Treatment Diabetic Retinopathy Study.[3] However, these techniques were not practical for DR screening in community, especially in the regions where specialized equipment and human resources for eye care were not available.[4,5] In recent years, digital fundus cameras have been largely used as an alternative to 7-field color stereoscopic fundus cameras or FA in community-based DR screening. Furthermore, artificial intelligence (AI) has been established to make DR screening results faster, simpler, and more efficient.[6] The sensitivity and specificity of AI were 90.79% and 91.18%, respectively, according to He et al.[7]
Binh Dinh is a poor central coastal province with one city and 10 districts with a population of 1.6 million in Vietnam. The provincial eye-care network has been set up for a long time, but at the district level, there is no eye doctor capable of DR screening. In other words, the whole province does not have enough staff and equipment to diagnose and treat DR. Since 2019, nonmydriatic digital fundus imaging cameras have been used for DR screening in community in Binh Dinh. Since March 2021, an AI software (EyeArt) has been deployed throughout the province with many encouraging results. It was very necessary to evaluate the sensitivity, specificity, and accuracy of EyeArt AI software for DR screening in the community. The result will be a basis for developing a provincial plan to prevent DR in the coming years.
Materials and Methods
Study population and design
This retrospective, descriptive, cross-sectional study was performed on all patients with diabetes at hospitals and health centers in Binh Dinh province in Vietnam from March 2021 to October 2022. With the sensitivity[8] was 90%, the DR prevalence in community[2] in Binh Dinh was 30% and 10% of nonmydriatic digital fundus images were expected to be excluded for any reason, a sample size of 516 patients with 2064 digital fundus images were selected to carry out the study.
This study protocol adhered to the principles of the Declaration of Helsinki and was approved by the Institutional Review Board of the University of Medicine and Pharmacy at Ho Chi Minh City in Vietnam (IRB-VN01002/IORG0008603/FWA00023448), coded 22608-DHYD under the Reduced Procedure. Written informed consent was obtained from the patients in the study.
Inclusion and exclusion criteria
The inclusion criteria were all patients with diabetes, regardless of age, gender, and area of residence. All patients came to hospitals and health centers in the province to have their eyes examined and their fundi photographed. All patient’s fundi were imaged with a nonmydriatic digital camera, Volk Optical (Pictor Plus™ Fundus Camera, image resolution of 2560 × 1920 pixels) or CR-2 AF (Canon CR2 Camera, image resolution of 32.5 megapixels). One patient had two digital fundus pictures for each eye, one centered on the macula and another centered on the optic disc. All retinal fundus pictures were automatically classified with EyeArt system v2.0 (Eyenuk, Inc., Los Angeles, CA, USA).
Exclusion criteria
(1) Patients who had experienced treatment with anti-VEGF or laser photo-coagulation, (2) those who with a second follow-up, (3) those who had one eye classified as non-DR and another ungradable due to any reasons, and (4) those who had not enough four digital fundus pictures or lacked the necessary information for the statistical analysis.
Description of parameters
The study indicators included (1) demographic variable (age, groups of age, and sex). Patient’s age was categorized into <40 years old, 40–50 years old, 50–60 years old, 60–70 years old, and ≥70 years old, (2) DR clinical variable according to the ICO’s guidelines,[2] (3) DR severity scale variable according to the ICO’s guidelines (non-DR, mild nonproliferative diabetic retinopathy [NPDR], moderate NPDR, severe NPDR, and proliferative diabetic retinopathy [PDR]),[2] (4) referable DR variable[2] (if at least one eye has any of the following features: moderate NPDR, severe NPDR, PDR, noncentral-involved diabetic macular edema (DME), and central-involved DME), and (5) vision-threatening DR variable[2] (if at least one eye has any of the following features: severe NPDR, PDR, noncentral-involved DME, and central-involved DME).
Study protocol
The study protocol was as follows: First, we selected thirty patients with 120 digital fundus pictures to perform the Kappa index to obtain the consensus of two eye doctors who would be responsible for DR clinical feature evaluation and DR severity scale classification. A Kappa value of 0.7 and over would indicate a good reliability for the study deployment. Second, all digital fundus pictures were coded and then sent to the two above-mentioned eye doctors to be evaluated and classified according to the ICO’s guidelines. Noted that these two eye doctors did not know any information about patients and DR severity scales classified with EyeArt. In the end, DR severity scales classified with EyeArt were compared with those by eye doctors as a reference standard for EyeArt’s effectiveness.
Statistical analysis
All digital retinal fundus images that responded to the criteria were included in the analysis. If the four digital fundus photos were classified into different DR severity in one patient, the DR severity scale of the given patient was classified according to the image with the most severe damage. If the patient has at least one eye classified DR/referable DR/vision-threatening DR, the four digital fundus images of this patient are included for the analysis. In contrast, the patient was excluded if she/he had two eyes classified as ungradable due to image quality or one eye classified as non-DR and another ungradable. If there was a disagreement between two eye doctors, these photos were sent to a DR expert at the Department of Ophthalmology, University of Medicine, and Pharmacy at Ho Chi Minh City for reading. Her DR severity scale was the last result.
All the data were analyzed using the SPSS software version 20.0 (IBM SPSS software, United States). The true positive, false positive, true negative, and false negative were determined on 2 × 2 table based on DR severity scales classified with EyeArt versus by eye doctor. The values (with confidence interval [CI] 95%) of sensitivity, specificity, positive predictive value, negative predictive value, and accuracy were calculated according to DR state, referable DR state or not, and vision-threatening DR state or not. The continuous data were expressed as means with standard deviation. Ordinal and binary data were expressed as a percentage. Pearson or Fisher exact Chi-square test was used to determine the association between the qualitative variables. McNemar was used to determine the differences in a dichotomous dependent variable between two related groups. The DR severity is an ordinal variable, so the Chi-square test (Pearson or Fisher) is suitable for it. In meanwhile, the DR prevalence is a binary variable with two values of zero and one (0,1). In the study, each picture was evaluated as having a DR or not with EyeArt, then by Eye Doctor, so the McNemar test was suitable for this related binary variable (paired binary variable). The referral DR prevalence and the vision-threatening DR prevalence were similar. P < 0.05 was considered statistically significant.
Results
Kappa index of two eye doctors for consensus
A Kappa of ten clinical signs of DR per eye was evaluated for the consensus of two eye doctors. The Kappa of macular edema in the right eye and in the left eye was 0.839 and 0.800 with P < 0.001. Similarly, the Kappa of hard exudate in the right eye, in the left eye, and the Kappa of venous abnormality in the left eye were 0.918, 0.915, and 0.843 with P < 0.001, respectively. The Kappa of other clinical signs was 1.0 with P < 0.001. In addition, the Kappa of DR screening, referable DR, and vision-threatening DR was also 1.0 with a P < 0.001. This result showed a high consensus of two eye doctors.
Demographic characteristics
There were 684 coded and inputted patients, but only 583 patients (85.23%) with 2332 digital fundus pictures used for the statistical analysis [Figure 1]. The mean age was 61.8 ± 10.5 years old (median 62, max 90, min 16). There were 346 females, accounting for 59.3%. The female-to-male ratio was 1.46. The association between the groups of age, and sex was statistically significant with P < 0.001 (Pearson Chi-square) [Table 1 and Supplementary materials (1.7MB, tif) ].
Figure 1.
Flowchart showing the number of cases included and excluded
Table 1.
Demographic characteristics of the study sample
| Groups of age/sex (years old) | Female, n (%) | Male, n (%) | Total, n (%) |
|---|---|---|---|
| <40 | 8 (1.4) | 12 (2.1) | 20 (3.5) |
| 40–50 | 20 (3.4) | 35 (6.0) | 55 (9.4) |
| 50–60 | 85 (14.6) | 70 (12.0) | 155 (26.6) |
| 60–70 | 143 (24.5) | 76 (13.0) | 219 (37.5) |
| ≥70 | 90 (15.4) | 44 (7.6) | 134 (23.0) |
| Total | 346 (59.3) | 237 (40.7) | 583 (100.0) |
| P | <0.001 | ||
Diabetic retinopathy severity scales classified with artificial intelligence
According to the classification of EyeArt AI, the prevalence of moderate NPDR was the highest (14.4%), followed by severe NPDR (8.7%). The prevalence of mild NPDR was similar to that of PDR (4.6% vs. 4.5%) [Table 2]. The association between DR severity and age, groups of age, and sex were not statistically significant with P = 0.095, 0.505, and 0.571, respectively (Pearson Chi-square).
Table 2.
Diabetic retinopathy severity and prevalence classified with the eyeart system
| DR severity and prevalence (n=583) | Non-DME | DME | Total, n (%) |
|---|---|---|---|
| No DR | 395 | 0 | 395 (67.8) |
| Mild NPDR | 23 | 4 | 27 (4.6) |
| Moderate NPDR | 66 | 18 | 84 (14.4) |
| Severe NPDR | 31 | 20 | 51 (8.7) |
| PDR | 20 | 6 | 26 (4.5) |
| Any DR | 188 (32.2) | ||
| Referable DR prevalence | 165 (28.3) | ||
| VTDR prevalence | 99 (17.0) |
DR=Diabetic retinopathy, NPDR=Nonproliferative diabetic retinopathy, PDR=Proliferative diabetic retinopathy, DME=Diabetic macula edema, VTDR=Vision-threatening diabetic retinopathy
Figure 2a shows the apparent difference in DR severity scales classified with EyeArt AI and by eye doctors. This difference was statistically significant with a P < 0.001 (Fisher’s exact).
Figure 2.
(a) Diabetic retinopathy (DR) severity classified with EyeArt versus eye doctor. (b) DR, referral, vision-threatening DR prevalence classified with EyeArt versus eye doctor
Diabetic retinopathy prevalence, referable diabetic retinopathy prevalence, and vision-threatening diabetic retinopathy prevalence
DR prevalence, referable DR prevalence, and vision-threatening DR prevalence classified with EyeArt were 32.2%, 28.3%, and 17.0%, respectively [Table 2].
The association between DR prevalence and age, groups of age, and sex was not statistically significant with a P = 0.440, 0.827, and 0.409, respectively (Pearson Chi-square). The association between referable DR prevalence and age, groups of age, and sex was not statistically significant with a P = 0.475, 0.773, and 0.796, respectively (Pearson Chi-square). The association between vision-threatening DR prevalence and age, groups of age, and sex was not statistically significant with a P = 0.379, 0.858, and 0.956, respectively (Pearson Chi-square).
Figure 2b shows the clear difference in DR prevalence, referable DR prevalence, and vision-threatening DR prevalence classified with EyeArt and by eye doctor. This difference was statistically significant with a P < 0.001 (McNemar).
Effectiveness of eyeart artificial intelligence software in diabetic retinopathy screening
The sensitivity, specificity, and accuracy of EyeArt AI software for DR screening were 94.1% (CI 95%: 90.4%–97.8%), 87.2% (CI 95%: 83.9–90.2%), and 88.9% (CI 95%: 86.1%–91.3%), respectively [Table 3].
Table 3.
Effectiveness of EyeArt artificial intelligence software
| Indicators (n=583) | DR screening, % (CI 95%) | Referral DR screening, % (CI 95%) | Vision-threatening DR screening% (CI 95%) |
|---|---|---|---|
| TP, FP | 128, 57 | 115, 46 | 58, 41 |
| FN, TN | 8, 390 | 4, 418 | 0, 484 |
| Sensitivity | 94.1 (90.4–97.8) | 96.6 (93.3–99.2) | 100 (100–100) |
| Specificity | 87.2 (83.9–90.2) | 90.1 (87.3–92.9) | 92.2 (89.9–94.3) |
| PPV | 69.2 (62.7–75.7) | 71.4 (64.6–78.3) | 58.6 (49.5–68.7) |
| NPV | 98.0 (96.5–99.2) | 99.1 (98.1–99.8) | 100 (100–100) |
| Accuracy | 88.9 (86.1–91.3) | 91.4 (89.0–93.7) | 93.0 (90.7–95.2) |
TP=True positive, FP=False positive, FN=False negative, TN=True negative, PPV=Positive predictive value, NPV=Negative predictive value, CI=Confidence interval
The sensitivity, specificity, and accuracy of EyeArt for referable screening were 96.6% (CI 95%: 93.3%–99.2%), 90.1% (CI 95%: 87.3%–92.9%), and 91.4% (CI 95%: 89.0%–93.7%), respectively [Table 3].
The sensitivity, specificity, and accuracy of EyeArt for vision-threatening DR screening were 100.0% (CI 95%: 100.0%–100.0%), 92.2% (CI 95%: 89.9%–94.3%), and 93.0% (CI 95%: 90.7%–95.2%), respectively [Table 3].
Figure 3a-e shows some digital fundus pictures in the study sample.
Figure 3.
(a) Non-Diabetic retinopathy (Left Eye). Patient: K H Dang Th – 61 years old. Code: EYEM97016. (b) Moderate nonproliferative diabetic retinopathy (NPDR) with hemorrhages and micro-aneurysms (Right Eye). Patient: B Do – 66 years old. Code: EYEM97024. (c) Severe NPDR with hemorrhages, macular edema, and laser spots (Left Eye). Patient: S Huynh – 58 years old. Code: EYEM97047. (d) Proliferative diabetic retinopathy (PDR) with pre-retinal hemorrhages with neovascular vessels (Right Eye). Patient: D H Dao Th – 27 years old. Code: EYEM97021. (e) PDR with preretinal hemorrhages, fibrous proliferation membrane with neovascular vessels (Right Eye). Patient: X M Mai Th – 52 years old. Code: EYEM97080
Discussion
Demographic characteristics
The mean age in our study was 62 years old, which is the age of retirement in Vietnam, but those at this age can still make a significant contribution to their family and society.
The mean age in our study was similar to that in Singapore,[9] in Thailand,[10] and in the United States.[11] Meanwhile, the mean age of the study in Australia was among young workers (44 years old).[12] In contrast, the mean age in the study in China was high, in the elderly or elderly population (68 years old) [Table 4 and Supplementary materials (1.7MB, tif) ].[7]
Table 4.
Demographic characteristics of the study sample compared to other studies
| Author | Year | Location of study | Number of patient | Mean age | Female: Male ratio |
|---|---|---|---|---|---|
| He et al.[7] | 2019 | China | 889 | 68 | 1.13 |
| Ting et al.[9] | 2017 | Singapore | 14,880 | 60.2 | 0.83 |
| Raumviboonsuk et al.[10] | 2019 | Thailand | 7517 | 61.13 | 2.08 |
| Abràmoff et al.[11] | 2018 | America | 819 | 59 | 1.11 |
| Keel et al.[12] | 2018 | Australia | 96 | 44 | 0.75 |
| Thanh nguyen van | 2022 | Vietnam | 583 | 62 | 1.46 |
Regarding gender, the participation rate of females in our study was almost 1.5 times higher than that of males, but the participation rate of females was twice as high as that of males in the study in Thailand (female/male ratio was 2).[10] Meanwhile, the participation rate of females was equal to that of males in studies in the United States[11] and in China.[7] On the contrary, the participation rate of females was lower than that of males in the studies in Singapore[9] and in Australia.[12]
The differences in mean age and sex among studies were probably due to different customs and habits, different DR patterns, and different DR screening programs.
Diabetic retinopathy severity classified with artificial intelligence in different studies
Table 5 and Supplementary materials shows that the prevalence of moderate NPDR in all studies was the highest compared with other severity scales. It was probably a model of DR in the world. The prevalence of moderate NPDR in our study was equal to that of Malavika study using the available digital fundus picture set of EyePACS[8] but was higher than that in the study in China[7] and lower than that in the study in India.[13] The prevalence of mild NPDR in our study was similar to the study in China and in India.[7,13] Note that the prevalence of severe NPDR and PDR in the study of Rajalakshmi[13] in India was too high. It was probably because the study was conducted on people who had a higher prevalence of diabetes and did not have the opportunity to have access to specialized eye-care services. The study of Hsieh et al.[14] in Taiwan showed that the prevalence of mild NPDR was the highest and it was much higher than other levels. In other words, in the study, there was a high prevalence of patients coming to the eye doctors at an early stage with only micro-aneurysms. It was possible that the DR screening and referral network was very excellent in Taiwan. It was possible that the author conducted the study on a group of low-risk patients at a particular time or the data collection method was not standardized, leading to the skewed data and causing an increased prevalence of mild NPDR.
Table 5.
Diabetic retinopathy severity was classified with artificial intelligence in different studies
| Author (location, year) | Number of patient | Non-DR prevalence (%) | Mild NPDR (%) | Moderate NPDR (%) | Severe NPDR (%) | PDR (%) |
|---|---|---|---|---|---|---|
| He et al.[7] (China, 2019) | 889 | 16.3 | 4.72 | 9.0 | 2.59 | 0.0 |
| Bhaskaranand et al.[8] (EyePACS, 2019) | 107,001 pictures | 67.5 | 8.2 | 14.2 | 2.5 | 2.6 |
| Raumviboonsuk et al.[10] (Thailand, 2019) | 7517 | 87.83 | 9.8 | 0.81 | 1.57 | |
| Rajalakshmi et al.[13] (India, 2018) | 296 | 68.6 | 4.7 | 35.5 | 10.8 | 17.6 |
| Hsieh et al.[14] (Taiwan, 2021) | 1875 pictures | 65.12 | 22.99 | 8.91 | 1.23 | 1.76 |
| Thanh Nguyen Van (Vietnam, 2022) | 583 | 67.8 | 4.7 | 14.4 | 8.7 | 4.5 |
NPDR=Nonproliferative diabetic retinopathy, PDR=Proliferative diabetic retinopathy
Diabetic retinopathy prevalence, referable prevalence, and vision-threatening diabetic retinopathy prevalence in the study sample compared to other studies
The DR prevalence and referable DR prevalence in our study (32.2% and 28.3%) were similar to those in the study of Malavika[8] using 107,001 digital fundus pictures from EyePACS (32.5% and 24.8%). However, the vision-threatening DR prevalence in our study (17.0%) was three times higher than that of Malavika’s study (5.1%).[8] Note that the two studies were performed with the same AI system (EyeArt).
The DR prevalence in our study (32.2%) was similar to those in the study of Hsieh et al.[14] in Taiwan (34.9%), but the referable DR prevalence in our study (28.3%) was 2.3 times higher than that of Hsieh et al.[14] in Taiwan (11.9%). The clear difference was due to the reasons we discussed above.
Compared with other studies, we found that DR prevalence, referable DR prevalence, and vision-threatening DR prevalence in our study were higher than that in the study in Singapore,[9] in Thailand,[10] and in China[15] but ½ times lower than that in the study in India,[13] although the study in India also used the same system as ours (EyeArt) for DR screening [Table 6 and Supplementary materials (1.7MB, tif) ].
Table 6.
Diabetic retinopathy prevalence, referral diabetic retinopathy prevalence, and vision-threatening diabetic retinopathy prevalence in the study sample compared to other studies
| Author | Year | Location of study | Number of patient | DR prevalence (%) | Referal prevalence (%) | VTDR prevalence(%) |
|---|---|---|---|---|---|---|
| Bhaskaranand et al.[8] | 2019 | EyePACS | 107,001 pictures | 32.5 | 24.8 | 5.1 |
| Ting et al.[9] | 2017 | Singapore | 14880 | - | 3.0 | 0.6 |
| Raumviboonsuk et al.[10] | 2019 | Thailand | 7517 | - | 18.41 | 8.61 |
| Rajalakshmi et al.[13] | 2018 | India | 296 | 68.6 | 63.9 | 28.4 |
| Hsieh et al.[14] | 2021 | Taiwan | 1875 pictures | 34.88 | 11.89 | - |
| Zhang et al.[15] | 2020 | China | 47,269 | 28.8 | 24.4 | 10.8 |
| Thanh Nguyen Van | 2022 | Vietnam | 583 | 32.2 | 28.3 | 17.0 |
DR=Diabetic retinopathy, VTDR=Vision-threatening DR
The referable DR prevalence and vision-threatening DR prevalence in our study were quite high. This is a burden for the provincial diabetic eye care network, requiring more investments in the future.
Effectiveness of diabetic retinopathy screening compared among studies performed with the same artificial intelligence system
The sample size in our study (583 patients with 2332 digital fundus pictures) was higher than that in India[13] (296 patients), but lower than that of studies[16,17] using available picture sets of Mesidor-2 and EyePACS. The accuracy in our study was similar to that of Bhaskaranand,[17] but lower than that of Solanki.[16] The sensitivity in our study was higher than that in the study of Solanki[16] and Bhaskaranand,[17] but lower than that in the study of Rajalakshmi.[13] Meanwhile, the specificity in our study was higher than those of Rajalakshmi[13] in India and those of Solanki[16] and Bhaskaranand[17] using available picture sets of Mesidor-2, EyePACS [Table 7].
Table 7.
Sensitivity and specificity in studies implemented with the same artificial intelligence system
| Author | Year | Location | Number of patient | Sensitivity | Specificity | AUC |
|---|---|---|---|---|---|---|
| Rajalakshmi et al.[13] | 2018 | India | 296 | 99.30 | 68.80 | - |
| Solanki et al.[16] | 2015 | Messidor-2 | 874 | 93.80 | 72.20 | 0.94 |
| Bhaskaranand et al.[17] | 2016 | EyePACS | 5084 | 90.00 | 63.20 | 0.88 |
| Thanh Nguyen Van | 2022 | Vietnam | 583 | 94.1 | 87.2 | 0.889a |
aAccuracy. AUC=Area under the curve
The higher sensitivity in our study compared with other studies[16,17] was probably due to our stricter inclusion and exclusion criteria (each patient had enough four digital fundus pictures, excluded all patients with the second DR screening visit, excluded those who experienced treatment with laser photocoagulation or antiVEGF, excluded patients with insufficient information) that excluded digital fundus pictures classified as a positive by eye doctor, but classified as a negative with AI. Of the 101 patients excluded from the study, 83 patients with 332 pictures were identified by eye doctor or EyeArt as unreadable, eight patients with the second visit, three patients with insufficient information, and seven cases with other reasons. Of the eight patients excluded for the second visit, three were classified by eye doctors as having mild NPDR, but classified with EyeArt as not having DR. In other words, this directly reduced the false-negative rate. Therefore, it contributed to the higher sensitivity in our study compared with other studies.
A prospective study by Ip et al.[18] showed that fundus image and extent of DR damage can improve following anti-VEGF therapy without significant retinal reperfusion. In addition, many studies found that the classification of DR digital fundus imaging was misleading in patients who had undergone laser photo-coagulation or antiVEGF.[19,20,21] That was the reason that we excluded patients who had already been treated. Similarly, we only selected patients who came to have their eyes examined at the first visit for the analysis because, on the second visit, we could not determine if these patients had eyes treated or not due to available data. According to us, all these exclusions would not affect the clinical applications of AI-assisted screening on DR in the real world. Furthermore, these exclusions would help more accurately evaluate the sensitivity and specificity of AI in DR screening in community.
Effectiveness of different artificial intelligence systems for diabetic retinopathy screening
The accuracy of DR screening in our study was similar to that in the study of Oliveira[22] and Malerbi,[23] but lower than that of He,[7] Keel,[12] and Hsieh et al.[14] The sensitivity and specificity of DR screening in our study (EyeArt, in Vietnam) were similar to those in the study of Hsieh et al.[14] (VeriSee™, in Taiwan). The sensitivity of DR screening in our study was lower than that of Oliveira[22] in Portugal and of Malerbi in Brazil,[23] but the specificity was higher than those of two studies. In contrast, the sensitivity in our study was higher, but the specificity was lower than in the other remaining studies [Table 8].
Table 8.
Sensitivity, and specificity in the different studies with different artificial intelligence systems
| Author | Year | Location | Number of patient | Sensitivity | Specificity | AUC |
|---|---|---|---|---|---|---|
| For diabetic retinopathy screening | ||||||
| He et al.[7] | 2019 | China | 889 | 91.80 | 98.79 | 0.946 |
| Abràmoff et al.[11] | 2018 | The United States | 819 | 87.20 | 90.70 | - |
| Keel et al.[12] | 2018 | Australia | 96 | 92.30 | 93.70 | 0.937–0.989 |
| Hsieh et al.[14] | 2021 | Taiwan | 1875 pictures | 92.2 | 89.5 | 0.955 |
| Oliveira et al.[22] | 2011 | Portugal | 5386 | 95.80 | 63.20 | 0.849 |
| Malerbi et al.[23] | 2021 | Brazil | 824 | 97.80 | 61.40 | 0.890 |
| Thanh Nguyen Van | 2022 | Vietnam | 583 | 94.1 | 87.2 | 0.889a |
| For referable diabetic retinopathy | ||||||
| Ting et al.[9] | 2017 | Singapore | 14,880 | 90.5 | 91.6 | 0.936 |
| Raumviboonsuk et al.[10] | 2019 | Thailand | 7517 | 97.0 | 96.0 | - |
| Hsieh et al.[14] | 2021 | Taiwan | 1875 pictures | 89.2 | 90.1 | 0.955 |
| Thanh Nguyen Van | 2022 | Vietnam | 583 | 96.6 | 90.1 | 0.914a |
| For vision-threatening diabetic retinopathy | ||||||
| Ting et al.[9] | 2017 | Singapore | 14.880 | 100.0 | 91.1 | 0.958 |
| Thanh Nguyen Van | 2022 | Vietnam | 583 | 100.0 | 92.2 | 0.930a |
aAccuracy. AI=Artificial intelligence, AUC=Area under the curve
The accuracy of referable DR screening in our study was lower than that in the study of Hsieh et al. in Taiwan[14] and Ting in Singapore.[9] The sensitivity of referable DR screening in our study was similar to that of Raumviboonsuk[10] in Thailand, but higher than that of Hsieh et al.[14] in Taiwan and of Ting[9] in Singapore. The specificity of referable DR screening in our study was similar to that of Hsieh et al.[14] in Taiwan and of Ting,[9] but lower than that of Raumviboonsuk[10] [Table 8].
The accuracy for vision-threatening DR screening in our study was lower than that in the study of Ting in Singapore.[9] Meanwhile, the sensitivity and specificity of vision-threatening DR screening in our study were similar to those of Ting[9] in Singapore [Table 8].
The study by Van der Heijden et al.[24] showed that the validation results were less precise when using real-world data sets compared to those using open-access data sets with the same algorithm. Hsieh et al.[14] in Taiwan found that the VeriSee™ system had a better sensitivity than the eye doctor in referable DR screening and data set validation could reduce the false positive rate, which resulted in a higher accuracy in detecting referable DR.
It is more important to use an AI system with high sensitivity for DR screening, but a system with low specificity will risk resulting in a high false-positive rate in real-world practice, which will increase the cost of unnecessary treatment. The sensitivity and specificity of DR screening, referable DR screening, and vision-threatening DR screening in our study were good for screening for DR in community.
Strengths and limitations
In this study, the eye doctors who participated in ICO-guided digital fundus image classification had to undergo training on DR.[2] The high-resolution digital fundus cameras were used by trained technicians. One patient had two digital fundus pictures for each eye, one centered on the macula and another centered on the optic disc. Thus, this study satisfied the demand of Bayer Educational Initiative in 2020 for an effective DR screening program in community.[25]
Another clear strength of the study was that in the condition of not having enough eye doctors, endocrinologists and a lack of specialized equipment in a poor province in Vietnam, the application of AI is effective in DR screening in the community. It decreased training costs, equipment expenditure, and patient transport expenses and helped diabetes have access to hi-tech eye-care services.
The small sample size and retrospective method were the limitations in this study. Although the study conducted consecutive sampling to make it representative of the population, geographical, and visual characteristics of the sample, the study sample was difficult to represent the population due to available data. Under favorable conditions, we wish to conduct a cohort study to evaluate the effectiveness of DR intervention in community in Binh Dinh province in Vietnam.
Conclusion
AI is an effective, easy-to-use tool for screening DR in the community, especially in conditions where there are not enough eye doctors, endocrinologists, and lack of specialized equipment in a poor province like Binh Dinh in Vietnam. Besides, AI will be a useful alternative for eye doctors in diabetic eye care in future. The results of this study will open great opportunities for improving the diabetic eye care network in community and contribute to persuading provincial leaders and policymakers to invest more in the blindness prevention program in the coming years.
Data availability statement
All data generated or analyzed during this study are included in this published article.
Financial support and sponsorship
Nil.
Conflicts of interest
The authors declare that there are no conflicts of interest in this paper.
Supplementary Materials
Acknowledgments
This study was supported by the Department of Ophthalmology at the University of Medicine and Pharmacy at Ho Chi Minh City, Binh Dinh Eye Hospital, Binh Dinh Provincial General Hospital, Quy Nhon City Health Center, Phu Cat District Health Center.
References
- 1.GBD 2019 Blindness and Vision Impairment Collaborators, Vision Loss Expert Group of the Global Burden of Disease Study. Causes of blindness and vision impairment in 2020 and trends over 30 years, and prevalence of avoidable blindness in relation to VISION 2020: The right to sight: An analysis for the Global Burden of Disease Study. Lancet Glob Health. 2021;9:e144–60. doi: 10.1016/S2214-109X(20)30489-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Wong TY, Sun J, Kawasaki R, Ruamviboonsuk P, Gupta N, Lansingh VC, et al. Guidelines on diabetic eye care: The international council of ophthalmology recommendations for screening, follow-up, referral, and treatment based on resource settings. Ophthalmology. 2018;125:1608–22. doi: 10.1016/j.ophtha.2018.04.007. [DOI] [PubMed] [Google Scholar]
- 3.Ghasemi Falavarjani K, Wang K, Khadamy J, Sadda SR. Ultra-wide-field imaging in diabetic retinopathy; an overview. J Curr Ophthalmol. 2016;28:57–60. doi: 10.1016/j.joco.2016.04.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Panozzo G, Cicinelli MV, Augustin AJ, Battaglia Parodi M, Cunha-Vaz J, Guarnaccia G, et al. An optical coherence tomography-based grading of diabetic maculopathy proposed by an international expert panel: The European School for advanced studies in ophthalmology classification. Eur J Ophthalmol. 2020;30:8–18. doi: 10.1177/1120672119880394. [DOI] [PubMed] [Google Scholar]
- 5.Lira RP, Oliveira CL, Marques MV, Silva AR, Pessoa Cde C. Adverse reactions of fluorescein angiography: A prospective study. Arq Bras Oftalmol. 2007;70:615–8. doi: 10.1590/s0004-27492007000400011. [DOI] [PubMed] [Google Scholar]
- 6.Peng J, Zou H, Wang W, Fu J, Shen B, Bai X, et al. Implementation and first-year screening results of an ocular telehealth system for diabetic retinopathy in China. BMC Health Serv Res. 2011;11:250. doi: 10.1186/1472-6963-11-250. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.He J, Cao T, Xu F, Wang S, Tao H, Wu T, et al. Artificial intelligence-based screening for diabetic retinopathy at community hospital. Eye (Lond) 2020;34:572–6. doi: 10.1038/s41433-019-0562-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Bhaskaranand M, Ramachandra C, Bhat S, Cuadros J, Nittala MG, Sadda SR, et al. The value of automated diabetic retinopathy screening with the EyeArt system: A study of more than 100,000 consecutive encounters from people with diabetes. Diabetes Technol Ther. 2019;21:635–43. doi: 10.1089/dia.2019.0164. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Ting DS, Cheung CY, Lim G, Tan GS, Quang ND, Gan A, et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA. 2017;318:2211–23. doi: 10.1001/jama.2017.18152. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Raumviboonsuk P, Krause J, Chotcomwongse P, Sayres R, Raman R, Widner K, et al. Deep learning versus human graders for classifying diabetic retinopathy severity in a nationwide screening program. NPJ Digit Med. 2019;2:25. doi: 10.1038/s41746-019-0099-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Abràmoff MD, Lavin PT, Birch M, Shah N, Folk JC. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digit Med. 2018;1:39. doi: 10.1038/s41746-018-0040-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Keel S, Lee PY, Scheetz J, Li Z, Kotowicz MA, MacIsaac RJ, et al. Feasibility and patient acceptability of a novel artificial intelligence-based screening model for diabetic retinopathy at endocrinology outpatient services: A pilot study. Sci Rep. 2018;8:4330. doi: 10.1038/s41598-018-22612-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Rajalakshmi R, Subashini R, Anjana RM, Mohan V. Automated diabetic retinopathy detection in smartphone-based fundus photography using artificial intelligence. Eye (Lond) 2018;32:1138–44. doi: 10.1038/s41433-018-0064-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Hsieh YT, Chuang LM, Jiang YD, Chang TJ, Yang CM, Yang CH, et al. Application of deep learning image assessment software VeriSee™for diabetic retinopathy screening. J Formos Med Assoc. 2021;120:165–71. doi: 10.1016/j.jfma.2020.03.024. [DOI] [PubMed] [Google Scholar]
- 15.Zhang Y, Shi J, Peng Y, Zhao Z, Zheng Q, Wang Z, et al. Artificial intelligence-enabled screening for diabetic retinopathy: A real-world, multicenter and prospective study. BMJ Open Diabetes Res Care. 2020;8:e001596. doi: 10.1136/bmjdrc-2020-001596. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Solanki K, Ramachandra C, Bhat S, Bhaskaranand M, Nittala MG, Sadda SR. EyeArt: Automated, high-throughput, image analysis for diabetic retinopathy screening. Invest Ophthalmol Vis Sci. 2015;56:1429. [Google Scholar]
- 17.Bhaskaranand M, Ramachandra C, Bhat S, Cuadros J, Nittala MG, Sadda S, et al. Automated diabetic retinopathy screening and monitoring using retinal fundus image analysis. J Diabetes Sci Technol. 2016;10:254–61. doi: 10.1177/1932296816628546. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Ip MS, Domalpally A, Hopkins JJ, Wong P, Ehrlich JS. Long-term effects of ranibizumab on diabetic retinopathy severity and progression. Arch Ophthalmol. 2012;130:1145–52. doi: 10.1001/archophthalmol.2012.1043. [DOI] [PubMed] [Google Scholar]
- 19.Wykoff CC, Nittala MG, Zhou B, Fan W, Velaga SB, Lampen SI, et al. Intravitreal aflibercept for retinal nonperfusion in proliferative diabetic retinopathy: Outcomes from the randomized RECOVERY trial. Ophthalmol Retina. 2019;3:1076–86. doi: 10.1016/j.oret.2019.07.011. [DOI] [PubMed] [Google Scholar]
- 20.Goldberg R, Hill L, Abolian A, Stoilov I. San Francisco, CA: AAO; 2019. PA041 What Happens to Diabetic Retinopathy Severity Scores With Less Aggressive Treatment? A Post Hoc Analysis of the RISE/RIDE Open-Label Extension Study. [Google Scholar]
- 21.Tadayoni R. Time to call into question the fundus-based evaluation of diabetic retinopathy after intravitreal injections. J Ophthalmic Vis Res. 2020;15:4–6. doi: 10.18502/jovr.v15i1.5971. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Oliveira CM, Cristóvão LM, Ribeiro ML, Abreu JR. Improved automated screening of diabetic retinopathy. Ophthalmologica. 2011;226:191–7. doi: 10.1159/000330285. [DOI] [PubMed] [Google Scholar]
- 23.Malerbi FK, Andrade RE, Morales PH, Stuchi JA, Lencione D, de Paulo JV, et al. Diabetic retinopathy screening using artificial intelligence and handheld smartphone-based retinal camera. J Diabetes Sci Technol. 2022;16:716–23. doi: 10.1177/1932296820985567. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.van der Heijden AA, Abramoff MD, Verbraak F, van Hecke MV, Liem A, Nijpels G. Validation of automated screening for referable diabetic retinopathy with the IDx-DR device in the Hoorn diabetes care system. Acta Ophthalmol. 2018;96:63–8. doi: 10.1111/aos.13613. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Lanzetta P, Sarao V, Scanlon PH, Barratt J, Porta M, Bandello F, et al. Fundamental principles of an effective diabetic retinopathy screening program. Acta Diabetol. 2020;57:785–98. doi: 10.1007/s00592-020-01506-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
All data generated or analyzed during this study are included in this published article.