Table 2.
Comparison of classification performance between deep model and endoscopy doctors (per patient and per lesion).
| ResNet50 | Highest trainee endoscopist | P-value | Highest competent endoscopist | P-value | |
|---|---|---|---|---|---|
| Per patient analysis | |||||
| Accuracy (95%CI) | 0.991 (0.967–0.997) | 0.780 (0.720–0.830) | <0.001 | 0.922 (0.879–0.951) | <0.001 |
| Normal | |||||
| Sensitivity (95%CI) | 1.000 (0.955–1.000) | 0.951 (0.881–0.981) | 0.125 | 1.000 (0.955–1.000) | 1.000 |
| Specificity (95%CI) | 1.000 (0.973–1.000) | 0.890 (0.826–0.932) | <0.001 | 0.985 (0.948–0.996) | 0.500 |
| PPV (95%CI) | 1.000 (0.955–1.000) | 0.839 (0.751–0.900) | <0.001 | 0.976 (0.917–0.993) | 0.497 |
| NPV (95%CI) | 1.000 (0.973–1.000) | 0.968 (0.921–0.987) | 0.051 | 1.000 (0.972–1.000) | 1.000 |
| UC | |||||
| Sensitivity (95%CI) | 1.000 (0.949–1.000) | 0.694 (0.580–0.789) | <0.001 | 0.931 (0.848–0.970) | 0.219 |
| Specificity (95%CI) | 0.993 (0.962–0.999) | 0.870 (0.806–0.915) | <0.001 | 0.925 (0.870–0.957) | 0.002 |
| PPV (95%CI) | 0.986 (0.926–0.998) | 0.725 (0.610–0.816) | <0.001 | 0.859 (0.765–0.919) | 0.010 |
| NPV (95%CI) | 1.000 (0.974–1.000) | 0.852 (0.787–0.900) | <0.001 | 0.964 (0.919–0.985) | 0.197 |
| CD | |||||
| Sensitivity (95%CI) | 0.969 (0.893–0.991) | 0.656 (0.534–0.761) | <0.001 | 0.812 (0.700–0.889) | 0.001 |
| Specificity (95%CI) | 1.000 (0.976–1.000) | 0.909 (0.853–0.945) | 0.001 | 0.974 (0.935–0.990) | 0.375 |
| PPV (95%CI) | 1.000 (0.942–1.000) | 0.750 (0.623–0.845) | <0.001 | 0.929 (0.830–0.972) | 0.285 |
| NPV (95%CI) | 0.987 (0.954–0.996) | 0.864 (0.803–0.909) | <0.001 | 0.926 (0.875–0.957) | 0.006 |
| Per lesion analysis | |||||
| Accuracy (95%CI) | 0.904 (0.895–0.912) | 0.597 (0.583–0.610) | <0.001 | 0.699 (0.686–0.712) | <0.001 |
| Normal | |||||
| Sensitivity (95%CI) | 0.978 (0.971–0.984) | 0.831 (0.814–0.848) | <0.001 | 0.956 (0.946–0.964) | 0.090 |
| Specificity (95%CI) | 0.979 (0.973–0.984) | 0.757 (0.741–0.772) | <0.001 | 0.758 (0.743–0.773) | <0.001 |
| PPV (95%CI) | 0.967 (0.958–0.974) | 0.684 (0.665–0.703) | <0.001 | 0.714 (0.696–0.731) | <0.001 |
| NPV (95%CI) | 0.986 (0.981–0.990) | 0.877 (0.863–0.889) | <0.001 | 0.965 (0.956–0.971) | 0.001 |
| UC | |||||
| Sensitivity (95%CI) | 0.928 (0.914–0.939) | 0.675 (0.652–0.698) | <0.001 | 0.687 (0.663–0.710) | <0.001 |
| Specificity (95%CI) | 0.878 (0.866–0.888) | 0.669 (0.653–0.685) | <0.001 | 0.807 (0.793–0.820) | <0.001 |
| PPV (95%CI) | 0.778 (0.759–0.796) | 0.486 (0.464–0.507) | <0.001 | 0.622 (0.599–0.645) | <0.001 |
| NPV (95%CI) | 0.963 (0.956–0.969) | 0.817 (0.802–0.831) | <0.001 | 0.848 (0.835–0.860) | <0.001 |
| CD | |||||
| Sensitivity (95%CI) | 0.948 (0.935–0.958) | 0.206 (0.186–0.227) | <0.001 | 0.376 (0.352–0.402) | <0.001 |
| Specificity (95%CI) | 0.883 (0.872–0.893) | 0.960 (0.953–0.966) | <0.001 | 0.971 (0.964–0.976) | <0.001 |
| PPV (95%CI) | 0.773 (0.753–0.792) | 0.683 (0.638–0.725) | <0.001 | 0.844 (0.814–0.870) | 0.901 |
| NPV (95%CI) | 0.976 (0.970–0.980) | 0.742 (0.729–0.754) | <0.001 | 0.787 (0.774–0.799) | <0.001 |
UC, ulcerative colitis; CD, Crohn's disease; PPV, positive predictive value; NPV, negative predictive value; AUC, area under the receiver operating characteristic curve.