Table 3.
Performance of the DeepDR system for diabetic retinopathy grading.
| Local validation | AUC (95% CI) | Sensitivity (95% CI) | Specificity (95% CI) | Positive likelihood ratio | Negative likelihood ratio | |
|---|---|---|---|---|---|---|
| SIM | Non-DR | 0.945 (0.943–0.947) | 0.912 (0.911–0.914) | 0.825 (0.818–0.831) | 5.201 (5.019–5.414) | 0.106 (0.104–0.109) |
| Mild NPDR | 0.943 (0.940–0.946) | 0.888 (0.877–0.897) | 0.839 (0.837–0.841) | 5.511 (5.409–5.614) | 0.134 (0.122–0.146) | |
| Moderate NPDR | 0.955 (0.953–0.957) | 0.937 (0.931–0.943) | 0.837 (0.834–0.839) | 5.736 (5.647–5.824) | 0.075 (0.069–0.082) | |
| Severe NPDR | 0.960 (0.956–0.964) | 0.920 (0.904–0.936) | 0.860 (0.858–0.862) | 6.578 (6.418–6.732) | 0.093 (0.074–0.112) | |
| PDR | 0.972 (0.966–0.977) | 0.930 (0.898–0.958) | 0.895 (0.893–0.897) | 8.868 (8.526–9.190) | 0.078 (0.046–0.113) | |
| DME | 0.946 (0.945–0.947) | 0.928 (0.924–0.931) | 0.813 (0.810–0.816) | 4.955 (4.878–5.037) | 0.089 (0.084–0.094) | |
| Referable DR | 0.973 (0.972–0.974) | 0.941 (0.936–0.946) | 0.897 (0.895–0.899) | 9.157 (8.980–9.335) | 0.066 (0.061–0.071) | |
| External validation | ||||||
| CNDCS | Non-DR | 0.937 (0.935–0.939) | 0.876 (0.874–0.878) | 0.831 (0.825–0.837) | 5.191 (5.019–5.382) | 0.149 (0.146–0.152) |
| Mild NPDR | 0.916 (0.912–0.920) | 0.914 (0.904–0.923) | 0.743 (0.741–0.746) | 3.562 (3.507–3.615) | 0.116 (0.103–0.129) | |
| Moderate NPDR | 0.927 (0.925–0.929) | 0.890 (0.884–0.896) | 0.793 (0.790–0.796) | 4.304 (4.239–4.370) | 0.138 (0.131–0.146) | |
| Severe NPDR | 0.962 (0.959–0.965) | 0.918 (0.905–0.929) | 0.880 (0.878–0.882) | 7.624 (7.460–7.789) | 0.094 (0.080–0.108) | |
| PDR | 0.955 (0.949–0.961) | 0.927 (0.900–0.950) | 0.855 (0.852–0.857) | 6.377 (6.168–6.573) | 0.086 (0.059–0.117) | |
| Referable DR | 0.970 (0.969–0.971) | 0.940 (0.936–0.944) | 0.883 (0.881–0.885) | 8.050 (7.896–8.208) | 0.068 (0.063–0.073) | |
| NDSP | Non-DR | 0.925 (0.915–0.934) | 0.907 (0.903–0.910) | 0.758 (0.729–0.787) | 3.750 (3.345–4.255) | 0.123 (0.117–0.129) |
| Mild NPDR | 0.929 (0.916–0.942) | 0.905 (0.866–0.941) | 0.770 (0.765–0.775) | 3.942 (3.752–4.121) | 0.123 (0.076–0.175) | |
| Moderate NPDR | 0.957 (0.951–0.963) | 0.933 (0.910–0.956) | 0.861 (0.857–0.865) | 6.692 (6.443–6.940) | 0.077 (0.051–0.104) | |
| Severe NPDR | 0.953 (0.942–0.964) | 0.875 (0.820–0.927) | 0.885 (0.881–0.888) | 7.591 (7.066–8.103) | 0.141 (0.083–0.204) | |
| PDR | 0.956 (0.933–0.974) | 0.894 (0.795–0.976) | 0.900 (0.896–0.903) | 8.898 (7.920–9.753) | 0.118 (0.027–0.227) | |
| Referable DR | 0.965 (0.960–0.969) | 0.945 (0.926–0.962) | 0.860 (0.856–0.865) | 6.768 (6.528–7.004) | 0.064 (0.045–0.086) | |
| EyePACS | Non-DR | 0.934 (0.932–0.936) | 0.900 (0.898–0.903) | 0.802 (0.797–0.807) | 4.546 (4.435–4.664) | 0.124 (0.121–0.127) |
| Mild NPDR | 0.937 (0.935–0.939) | 0.903 (0.896–0.911) | 0.804 (0.801–0.807) | 4.613 (4.539–4.688) | 0.120 (0.111–0.129) | |
| Moderate NPDR | 0.932 (0.930–0.934) | 0.897 (0.892–0.903) | 0.802 (0.799–0.805) | 4.532 (4.459–4.606) | 0.128 (0.121–0.134) | |
| Severe NPDR | 0.954 (0.951–0.957) | 0.924 (0.913–0.935) | 0.848 (0.846–0.851) | 6.094 (5.977–6.214) | 0.089 (0.076–0.102) | |
| PDR | 0.961 (0.958–0.964) | 0.932 (0.920–0.943) | 0.862 (0.859–0.864) | 6.742 (6.607–6.884) | 0.079 (0.066–0.093) | |
| Referable DR | 0.946 (0.945–0.947) | 0.928 (0.924–0.931) | 0.813 (0.810–0.816) | 4.955 (4.878–5.037) | 0.089 (0.084–0.094) |
For each level, AUC is report in a one-vs.-all manner, i.e., Mild NPDR means Mild NPDR vs. Non-DR, Moderate NPDR, Severe NPDR, and PDR. DR diabetic retinopathy, CNDCS China National Diabetic Complications Study, DME diabetic macular edema, NDSP Nicheng Diabetes Screening Project, NPDR non-proliferative diabetic retinopathy, PDR proliferative diabetic retinopathy, SIM Shanghai Integrated Diabetes Prevention and Care System (Shanghai Integration Model).