Table 2.
Model performance metrics on internal and external test sets for label sets I and II.
| Label set I* | |||
|---|---|---|---|
| Internal test set‡ (n = 1,185) |
External test set (KM) (n = 350) |
External test set (YN) (n = 329) |
|
| AUROC (95% CI) | 0.862 (0.841–0.884) | 0.829 (0.785–0.873) | 0.835 (0.790–0.879) |
| Sensitivity (95% CI) | 0.833 (0.805–0.861) | 0.685 (0.623–0.748) | 0.706 (0.636–0.777) |
| Specificity (95% CI) | 0.754 (0.715–0.792) | 0.839 (0.778–0.901) | 0.870 (0.819–0.921) |
| PPV (95% CI) | 0.826 (0.798–0.854) | 0.869 (0.818–0.920) | 0.837 (0.775–0.899) |
| NPV (95% CI) | 0.763 (0.725–0.801) | 0.632 (0.562–0.702) | 0.758 (0.697–0.818) |
| Label set II† | |||
|---|---|---|---|
| Internal test set‡ (n = 842) |
External test set (KM) (n = 236) |
External test set (YN) (n = 229) |
|
| AUROC (95% CI) | 0.934 (0.911–0.957) | 0.883 (0.829–0.938) | 0.913 (0.876–0.951) |
| Sensitivity (95% CI) | 0.872 (0.847–0.897) | 0.779 (0.724–0.835) | 0.844 (0.787–0.900) |
| Specificity (95% CI) | 0.869 (0.816–0.923) | 0.870 (0.732–1.000) | 0.913 (0.847–0.980) |
| PPV (95% CI) | 0.968 (0.954–0.982) | 0.982 (0.962–1.002) | 0.957 (0.924–0.991) |
| NPV (95% CI) | 0.602 (0.537–0.666) | 0.299 (0.189–0.408) | 0.716 (0.622–0.810) |
*Label Set I categorizes images as ‘Diffusion-FLAIR Match’ for a binary comparison against a combined ‘Non-match’ group, which encompasses ‘Diffusion-FLAIR Mismatch,’ ‘Subtle FLAIR Change,’ and ‘Regional Heterogeneous FLAIR Change.’.
†Label Set II represents a binary categorization distinguishing only between ‘Diffusion-FLAIR Match’ and ' Diffusion-FLAIR Mismatch,’ excluding all other categories.
‡This number for internal test set represents the total number of patients aggregated from the results after performing a 5-fold validation on the 10% of patients allocated to the test set in the derivation cohort (Figure S2).
AUROC, Area Under the Receiver Operating Characteristic; PPV, Positive Predictive Value; NPV, Negative Predictive Value.