Table 1.
The summary of the major automated diabetic retinopathy (DR) detection tools, using feature-based and deep learning with regards to the training dataset, testing dataset, and diagnostic performance
| DR classification system | Year | Development dataset | Ground truth | Clinical validation | Mydriatic or non-mydriatic | n (gradable) | % ungradable | Total n (including ungradable) | Referable DR AUC | Referable DR sensitivity | Referable DR specificity |
|---|---|---|---|---|---|---|---|---|---|---|---|
| EyeArt | 2015 | Proprietary algorithm | EyePACS human graders | EyePACS | Mixed Mydriatic (~46%), Non-Mydriatic (~54%) | 101710 | 4.94% | 107001 | 0.965 | 91.30% | 91.10% |
| Full-time and part-time optometrist and non-optometrist graders, with arbitration | Homerton | Mydriatic | 19963 | 1.46% | 20258 | – | 86.00% | 54.00% | |||
| Abramoff et al. | 2016 | 10,000 to 1,250,000 unique samples of each lesion type graded by one or more experts | Adjudication by 3 retinal specialists until full consensus for all cases using a single 45 degree FOV image | Messidor-2 | Mydriatic | 840 | 4.0% | 874 | 0.980 | 96.8% | 87.0% |
| Gulshan et al. | 2016 | 128,175 images graded 3–7 times | Majority decision of 7 or 8 ophthalmologists for all cases using single macula-centred image with 45 degree FOV | EyePACS-1a | Mostly Non-Mydriatic | 8788 | 11.60% | 9963 | 0.991 | 97.5% | 93.4% |
| Messidor-2 | Mydriatic | 1745 | 0.17% | 1748 | 0.990 | 96.1% | 93.9% | ||||
| Gargeya and Leng | 2017 | 75,137 images from Kaggle competition graded by “a panel of retinal specialists” (with no additonal detail) | Not clearly described, likely the lesion-based grading that came with the public datasets using a single 45 degree FOV image | Messidor-2 | Mydriatic | 1748 | – | – | 0.940 (Any DR) | – | – |
| E-Ophtha | Likely Non-Mydriatic | 463 | – | – | 0.960 (Any DR) | – | – | ||||
| Ting et al. | 2017 | 72,610 images from multiple screening program and clinical studies graded by a minimum of 2 graders, often with a retinal specialist for arbitration | 2 trained graders for all cases, using 45 degree FOV a single image. If there is a disagreement, a retinal specialist generated final grade | SiDRP 14–15a | Non-mydriatic | 35055 | 1.1% | 35,948 | 0.936 | 90.5% | 91.6% |
| 2 graders; arbitration by 1 retinal specialist | Guangdong | Non-mydriatic | – | 1.4% | 15,798 | 0.949 | 98.7 | 81.6 | |||
| 1 grader; 1 retinal specialist | SIMES | Mydriatic | – | 1.8% | 3052 | 0.889 | 97.1 | 82 | |||
| 1 grader; 1 retinal specialist | SINDI | Mydriatic | – | 2.1% | 4512 | 0.917 | 99.3 | 73.3 | |||
| 1 grader; 1 retinal specialist | SCES | Mydriatic | – | 1.0% | 1936 | 0.919 | 100 | 76.3 | |||
| 2 ophthalmologists | BES | Mydriatic | – | 0.4% | 1052 | 0.929 | 94.4 | 88.5 | |||
| 2 retinal specialists | AFEDS | Mydriatic | – | 4.2% | 1968 | 0.98 | 98.8 | 86.5 | |||
| 2 graders | RVEEH | Mydriatic | – | 10.9% | 2302 | 0.983 | 98.9 | 92.2 | |||
| 2 retinal specialists | Mexican | Mydriatic | – | 0.5% | 1172 | 0.95 | 91.8 | 84.8 | |||
| 2 retinal specialists | CUHK | Mydriatic | – | 0.0% | 1254 | 0.948 | 99.3 | 83.1 | |||
| 2 optometrists | HKU | Mydriatic | – | 0.0% | 7706 | 0.964 | 100 | 81.3) | |||
| Bosch | 2017 | Nearly 80,000 images, 5000 from India verified by 3 ophthalmologists, the remainder from EyePACS-1 | Investigators following American Academy of Ophthalmology guidelines | Indian | Non-mydriatic | 560 | 3.9%c | 564 | – | 91.18% (Any DR) | 96.9% (Any DR) |
| Krause et al. | 2018 | 1.67 M images with clinical grades for train set 3737 fully adjudicated images for tune set | Adjudication by 3 retinal specialists until full consensus for all cases using a single 45 degree FOV image | EyePACS-2a | Mostly Non-Mydriatic | 1813 | 0% | – | 0.986 | 97.1% | 92.3% |
| Abramoff et al. | 2018 | 10,000 to 1,250,000 unique samples of each lesion type graded by one or more experts | Patient outcome based ETDRS and Clinically Significant DME grading from stereoscopic, 4-widefield field stereo images, and center-involved DME grading from macular OCT by three independent FPRC readers | Independent trial by contract research organisation Emmes, through a preregistered, prospective intention to screen trial protocol | 23.6% Mydriatic | 819 | 4.0% | 892 | – | 87.2% | 90.7% |
| Healgoo | 2018 | 71,043 images | Consensus grading of at least 3 out of the 21 ophthalmologists selected for the study | Internal | 4588 | 6.4% | 4900 | 0.989 | 97.00% | 91.40% | |
| 3 certified senior graders ( > 2 years experience) supervised by a retinal specialist | AusDiab | Non-mydriatic | – | – | 4349 | 0.9688 | 94.59% | 99.17% | |||
| 1 grader; 1 retinal specialist | SIMES | Mydriatic | – | – | 6431 | 0.9621 | 93.94% | 98.48% | |||
| 5 certified senior graders ( > 2 years experience) supervised by a retinal specialist | NIEHS | Mostly Non-Mydriatic | – | – | 2877 | 0.9367 | 89.76% | 97.57% | |||
| External validation total | 13394 | 1.9% | 13,657 | 0.955 | 92.50% | 98.50% | |||||
| Ruamviboonsuk et al. | 2019 | As of Gulshan et al. (2016) | A panel of international retinal specialists | Nationwide screening program of DR in Thailand | Both mydriatic and non-mydriatic | 25,326 | 15% | 29,800 | 0.987 | 0.968 | 0.956 |
aEstimated from given number of nonclassifiable eyes
bPrimary validation (e.g. validation set is drawn from population that overlaps with development set, but not the same patients). Lack of * means secondary validation (e.g. validation set from different population than development)
cImage-level