Table 2.
Quantitative evaluation of DLS in distinguishing normal versus abnormal CXRs across 6 datasets. (A) The DLS’s performance with the high-sensitivity operating point. (B) The DLS’s performance with the high-specificity operating point. The AUC is independent of the operating point and is identical to that in (A).
| (A) Scenario |
Dataset (reference label used for evaluation) | High-sensitivity operating point (optimizes for NPV) | AUC (95% CI) | |||||
|---|---|---|---|---|---|---|---|---|
| No. predicted negative (%) | NPV (95% CI) | Sensitivity (95% CI) | No. predicted positive (%) | PPV (95% CI) | Specificity (95% CI) | |||
| Abnormality detection | DS-1 (normal/abnormal) | 2313 (29.9%) | 0.98 (0.97–0.99) | 0.98 (0.97–0.98) | 5434 (70.1%) | 0.33 (0.32–0.34) | 0.38 (0.37–0.40) | 0.87 (0.87–0.88) |
| CXR-14 (normal/abnormal) | 194 (24.0%) | 0.85 (0.79–0.89) | 0.95 (0.93–0.97) | 616 (76.0%) | 0.89 (0.86–0.91) | 0.71 (0.65–0.76) | 0.94 (0.93–0.96) | |
| Unseen disease 1: TB | TB-1 (TB status) | 199 (43.1%) | 0.88 (0.84–0.93) | 0.90 (0.87–0.94) | 263 (56.9%) | 0.83 0.78–0.87 ) | 0.80 (0.74–0.85) | 0.95 (0.93–0.97) |
| TB-2 (TB status) | 51 (38.3%) | 0.98 (0.94–1.0) | 0.98 (0.94–1.0) | 82 (61.7%) | 0.63 (0.51–0.73) | 0.63 (0.51–0.73) | 0.97 (0.94–0.99) | |
| Unseen disease 2: COVID-19 | COV-1 (COVID-19 status) | 109 (5.9%) | 0.85 (0.78–0.92) | 0.97 (0.96–0.98) | 1710 (94.0%) | 0.33 (0.31–0.35) | 0.08 (0.06–0.09) | 0.68 (0.66–0.71) |
| COV-2 (COVID-19 status) | 59 (9.8%) | 0.56 (0.43–0.68) | 0.91 (0.87–0.94) | 546 (90.2%) | 0.48 (0.44–0.52) | 0.10 (0.07–0.14) | 0.65 (0.60–0.69) | |
| (B) Scenario |
Dataset (reference label used for evaluation) | High-specificity operating point (optimizes for PPV) | ||||||
|---|---|---|---|---|---|---|---|---|
| No. predicted negative (%) | NPV (95% CI) | Sensitivity (95% CI) | No. predicted positive (%) | PPV (95% CI) | Specificity (95% CI) | |||
| Abnormality detection | DS-1 (normal/abnormal) | 6027 (77.8%) | 0.89 (0.88–0.90) | 0.63 (0.61–0.65) | 1720 (22.2%) | 0.68 (0.65–0.70) | 0.91 (0.90–0.91) | |
| CXR-14 (normal/abnormal) | 715 (88.3%) | 0.32 (0.29–0.36) | 0.16 (0.13–0.20) | 95 (11.7%) | 0.99 (0.96–1.0) | 1.0 (0.99–1.0) | ||
| Unseen disease1: TB | TB-1 (TB status) | 260 (56.3%) | 0.81 (0.76–0.85) | 0.81 (0.74–0.84) | 202 (43.7%) | 0.95 (0.91–0.98) | 0.95 (0.92–0.98) | |
| TB-2 (TB status) | 80 (60.2%) | 0.94 (0.88–0.99) | 0.91 (0.82–0.98) | 53 (39.8%) | 0.91 (0.81–0.98) | 0.94 (0.88–0.99) | ||
| Unseen disease 2: COVID-19 | COV-1 (COVID-19 status) | 1558 (85.7%) | 0.72 (0.69–0.74) | 0.23 (0.20–0.27) | 261 (14.3%) | 0.52 (0.46–0.58) | 0.90 (0.88–0.92) | |
| COV-2 (COVID-19 status) | 537 (88.8%) | 0.55 (0.51–0.59) | 0.17 (0.12–0.21) | 68 (11.2%) | 0.71 (0.59–0.81) | 0.94 (0.91–0.96) | ||