Table 5. External Validation Datasets Showing the Area Under the Curve, Sensitivity, Specificity, Concordant and Discordant Rates of the Deep Learning System in Detecting Referable Diabetic Retinopathy Among Populations With Diabetes, With Comparison to Retinal Specialists, General Ophthalmologists, Trained Graders, or Optometristsa.
| Datasets (No. of Images) | AUC (95% CI)b | % (95% CI) | Concordance Between DLS and Grader, No. (%)d | |||||
|---|---|---|---|---|---|---|---|---|
| Sensitivityc | Specificityc | DLS+ Graders+ |
DLS+ Graders− |
DLS− Graders+ |
DLS− Graders− |
Total Concordant Images | ||
| Community-based | ||||||||
| Guangdong (N = 15 798) |
0.949 (0.943-0.955) |
98.7 (97.7-99.3) |
81.6 (80.7-82.5) |
1785 (11.3) | 2575 (16.3) | 16 (0.1) | 11 422 (72.3) | 13 207 (83.6) |
| Population-based | ||||||||
| Singapore Malay Eye Study, (N = 3052) |
0.889 (0.863-0.908) |
97.1 (92.5-98.9) |
82.0 (79.4-84.4) |
282 (9.2) | 611 (20.0) | 3 (0.1) | 2156 (70.6) | 2438 (79.9) |
| Singapore Indian Eye Study, (N = 4512) |
0.917 (0.899-0.933) |
99.3 (95.1-99.9) |
73.3 (70.9-75.5) |
298 (6.6) | 1543 (34.2) | 0 | 2671 (59.2) | 2969 (65.8) |
| Singapore Chinese Eye Study (N = 1936) |
0.919 (0.900-0.942) |
100 (92.5-100.0)e |
76.3 (72.7-79.6) |
138 (7.1) | 560 (28.9) | 0 | 1239 (64.0) | 1377 (71.1) |
| Beijing Eye Study, (N = 1052) |
0.929 (0.903-0.955) |
94.4 (72.7-99.9) |
88.5 (85.4-91.2) |
35 (3.3) | 117 (11.1) | 1 (0.1) | 899 (85.5) | 934 (88.8) |
| African American Eye Disease Study (N = 1968) |
0.980 (0.971-0.989) |
98.8 (93.5-100.0) |
86.5 (84.1-88.7) |
171 (8.7) | 242 (12.3) | 2 (0.1) | 1553 (78.9) | 1724 (87.6) |
| Clinic-based | ||||||||
| Royal Victoria Eye and Ear Hospital (N = 2302) |
0.983 (0.972-0.991) |
98.9 (97.5-99.6) |
92.2 (89.5-94.3) |
1066 (46.3) | 198 (8.6) | 5 (0.2) | 1034 (44.9) | 2100 (91.2) |
| Mexican (N = 1172) |
0.950 (0.934-0.966) |
91.8 (88.4-94.4) |
84.8 (80.4-88.5) |
571 (48.7) | 83 (7.1) | 52 (4.4) | 466 (39.8) | 1037 (88.5) |
| Chinese University of Hong Kong (N = 1254) |
0.948 (0.921-0.972) |
99.3 (97.3-99.8) |
83.1 (77.9-87.3) |
576 (45.9) | 165 (13.2) | 4 (0.3) | 509 (40.6) | 1085 (86.5) |
| University of Hong Kong (N = 7706) |
0.964 (0.958-0.970) |
100 (99.0-100)e |
81.3 (80.0-82.6) |
701 (9.1) | 1310 (17.0) | 0 | 5695 (73.9) | 6396 (83.0) |
Abbreviations: AUC, area under the receiver operating characteristic curve; DLS, deep learning system.
For study locations and race/ethnicity data, see Table 1. Referable diabetic retinopathy was defined as moderate nonproliferative diabetic retinopathy, severe, proliferative diabetic retinopathy, and ungradable images.
Cluster-bootstrap, biased-corrected 95% CI was computed for each area under the curve, with individual patients as the bootstrap sampling clusters.
Asymptotic 95% CI was computed for the logit of each proportion and using the cluster sandwich estimator of standard error to account for possible dependency of eyes within each individual.
DLS+ and grader+ indicates positive concordance; DLS− and grader−, negative concordance. Last column reports total concordance (sum of these 2 values).
Exact Clopper-Pearson left-sided 97.5% CI was calculated owing to estimate being at the boundary.