Table 2.
Characteristics of included studies in systematic review.
| First author, reference | Factor addressed of training /testing dataset | Data points | Training dataset | Number of Images (training dataset) | Testing dataset | Number of images (testing dataset) | Outcome measures | Results | Implications | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gulshan9 | Dataset size (% of total training dataset of 103,698) (Training) | 0.2% | EyePACS | 207 | EyePACS | 24,360 | SP (at pre-set 97% SN) |
SP 38% |
60,000 Images may be the minimum training dataset size needed for maximum performance | |||||
| 2% | 2073 | 61% | ||||||||||||
| 10% | 10,369 | 77% | ||||||||||||
| 20% | 20,739 | 86% | ||||||||||||
| 30% | 31,109 | 91% | ||||||||||||
| 40% | 41,479 | 98% | ||||||||||||
| 50% | 51,849 | 100% | ||||||||||||
| 60% | 62,218 | 96% | ||||||||||||
| 70% | 72,588 | 97% | ||||||||||||
| 80% | 82,958 | 100% | ||||||||||||
| 90% | 93,328 | 99% | ||||||||||||
| 100% | 103,698 | 100% | ||||||||||||
| Mydriasis (testing) | Mydriatic | EyePACS | 128,175 | EyePACS-1 | 4236 | SN SP |
SN 89.6% |
SP 97.9% | Mydriasis may not be required for optimal performance | |||||
| Non-Mydriatic | 4534 | 90.9% | 98.5% | |||||||||||
| Both | 8770 | 90.1% | 98.2% | |||||||||||
| Ting15 | Retinal cameras (testing) | Canon | SiDRP | 76,370 | BES | 1052 | AUC SN SP |
AUC 0.929 |
SN 94.4% |
SP 88.5% | Different types of retinal cameras do not affect the performance | |||
| Topcon | CUHK | 1254 | 0.948 | 99.3% | 83.1% | |||||||||
| Carl Zeiss | HKU | 7706 | 0.964 | 100% | 81.3% | |||||||||
| Fundus Vue | Guangdong | 15,798 | 0.949 | 98.7% | 81.6% | |||||||||
| Study type (testing) | Clinic-based | SiDRP | 76,370 | CUHK | 1254 | AUC SN SP |
AUC 0.948 |
SN 99.3% |
SP 83.1% |
The study type does not affect the performance in detection of disease | ||||
| Community-based | BES | 1052 | 0.929 | 94.4% | 88.5% | |||||||||
| Population-based | Guangdong | 15,798 | 0.949 | 98.7% | 81.6% | |||||||||
| Reference Standard (testing) | Retinal Specialists | SiDRP | 76,370 | CUHK | 1254 | AUC SN SP |
AUC 0.948 |
SN 99.3% |
SP 83.1% |
If minimally professional graders with ≥7 years’ experience grade, performance may not be affected | ||||
| Ophthalmologists | BES | 1052 | 0.929 | 94.4% | 88.5% | |||||||||
| Optometrists | HKU | 7706 | 0.964 | 100% | 81.3% | |||||||||
| Graders | RVEEH | 2302 | 0.983 | 98.9% | 92.2% | |||||||||
| Prevalence rate (testing) | 5.5% (BES) | SiDRP | 76,370 | BES | 1052 | AUC SN SP |
AUC 0.929 |
SN 94.4% |
SP 88.5% |
Lower prevalence rate does not greatly affect performance | ||||
| 8.1% (SCES) | SCES | 1936 | 0.919 | 100% | 76.3% | |||||||||
| 12.9% (AFEDS) | AFEDS | 1968 | 0.980 | 98.8% | 86.5% | |||||||||
| Concurrent diseases (testing) | Mixed pathologies | SiDRP | 76,370 | DR | 37,001 | AUC SN SP |
AUC 0.936 |
SN 90.5% |
SP 91.6% |
Concurrent ocular pathologies in the same image does not affect the model’s detection of either disease | ||||
| AMD | 773 | 0.942 | 96.4% | 87.2% | ||||||||||
| Glaucoma | 56 | 0.931 | 93.2% | 88.7% | ||||||||||
| Ethnicity (testing) | Malay | SiDRP | 76,370 | SIMES | 3052 | AUC SN SP |
AUC 0.889 |
SN 97.1% |
SP 82.0% |
Despite difference in the retina between ethnicities, this does not influence the performance in detection | ||||
| Indian | SINDI | 4512 | 0.917 | 99.3% | 73.3% | |||||||||
| Chinese | SCES | 1936 | 0.919 | 100% | 76.3% | |||||||||
| African American | AFEDS | 1968 | 0.980 | 98.8% | 86.5% | |||||||||
| White | RVEEH | 2302 | 0.983 | 98.9% | 92.2% | |||||||||
| Hispanic | Mexico | 1172 | 0.950 | 91.8% | 84.8% | |||||||||
| Bawankar31 | Mydriasis (testing) | Non-mydriasis (vs ETDRS mydriatic reference standard) | Eye-PACS1, India | 80,000 | India | 1084 | SN SP |
SN 91.2% |
SP 96.9% |
Despite no mydriasis of testing dataset, the DLS was able to perform highly when compared to mydriatic 7-field ETDRS grading reference standard | ||||
| Burlina33 | Dataset size (training) | Real | AREDS | 119,090 | AREDS | 13,302 | AUC AC |
AUC 0.971 |
AC 91.1% |
Creating proxy datasets using GANs may provide a solution to those with limited access to large number of images | ||||
| Synthetic | Image generated with GANs | 119,090 | 0.924 | 82.9% | ||||||||||
| Sahlsten25 | Image pixel size (training) | 256×256 | Digifundus Ltd (Finland) | 24,806 | Digifundus Ltd (Finland) | 7118 | AUC | AUC0.961 | Training with higher resolution images may improve performance | |||||
| 299×299 | 24,806 | 0.970 | ||||||||||||
| 512×512 | 24,806 | 0.979 | ||||||||||||
| 1024×1024 | 24,806 | 0.984 | ||||||||||||
| 2095×2095 | 24,806 | 0.987 | ||||||||||||
| Bellemo32 | Ethnicity (testing) | African | SiDRP | 76,370 | Zambia | 4504 | AUC SN SP |
AUC 0.973 |
SN 92.3% |
SP 89.0% |
Differences in ethnicity between training and testing dataset does not affect performance | |||
| Ting34 | Prevalence rate (testing) | 4.1% (VTDR) | SiDRP | 76,370 | Pooled dataset (SiDRP, SIMES, SINDI, SCES, BES, AFEDS, CUHK, DMP) | 93,293 | AUC |
AUC 0.950 |
Prevalence rate of diseases may be estimated accurately by DLS | |||||
| 6.5% (RDR) | 0.963 | |||||||||||||
| 15.9% (ADR) | 0.863 | |||||||||||||
AUC area under curve of receiver operating curve, AC accuracy, SN sensitivity, SP specificity, EyePACS Eye Picture Archive Communication System, SiDRP Singapore’s National Integrated Diabetic Retinopathy Screening Program, BES Beijing Eye Study, CUHK Chinese University Hong Kong, HKU Hong Kong University, RVEEH Royal Victoria Eye and Ear Hospital, AFEDS African American Eye Disease Study, SCES Singapore Chinese Eye Study, SIMES Singapore Malay Eye Study, SINDI Singapore Indian Eye Study, DMP Diabetes Management Project Melbourne, DLS Deep Learning System, ETDRS Early Treatment Diabetic Retinopathy Study, AREDS Age-Related Eye Disease Study, DR diabetic retinopathy, AMD age-related macular degeneration, VTDR vision threatening diabetic retinopathy, RDR reference diabetic retinopathy, ADR any diabetic retinopathy, GAN Generative Adversarial Network.