Table 2.
Error Metrics Values Across the Validation Experiments Performed
| Datasets | Features | AUC | AUC0.632+ | MCR % | FNR % | FPR % | ROC Figure |
||
|---|---|---|---|---|---|---|---|---|---|
| Train | Test | ||||||||
| A: Model external validation | |||||||||
| □ | A | B | Clinical | 0.59 | - | 50.4 | 45.0 | 51.5 | Figure 2a |
| Radiomic | 0.61 | - | 20.5 | 90.0 | 6.2 | ||||
| All | 0.70 | - | 22.2 | 90.0 | 8.2 | ||||
| ■ | B | A | Clinical | 0.59 | - | 43.9 | 44.4 | 43.8 | Figure 2b |
| Radiomic | 0.61 | - | 58.5 | 7.4 | 72.9 | ||||
| All | 0.62 | - | 56.1 | 14.8 | 67.7 | ||||
| B: Methodology external validation | |||||||||
| A (Bootstrap) |
Clinical | 0.67 (0.01) | 0.73 | 35.3 (1.4) | 46.3 (1.8) | 32.1 (1.3) | Figure 2c | ||
| Radiomic | 0.68 (0.01) | 0.74 | 32.4 (1.5) | 41.4 (2.1) | 29.9 (1.4) | ||||
| All | 0.69 (0.01) | 0.76 | 30.5 (1.5) | 39.9 (2.0) | 27.8 (1.3) | ||||
| B (Bootstrap) |
Clinical | 0.56 (0.01) | 0.64 | 31.8 (1.5) | 68.9 (2.4) | 24.1 (1.3) | Figure 2d | ||
| Radiomic | 0.73 (0.01) | 0.80 | 32.3 (1.7) | 37.0 (2.9) | 31.3 (1.4) | ||||
| All | 0.73 (0.01) | 0.80 | 30.9 (1.6) | 41.3 (3.0) | 28.8 (1.3) | ||||
| C: Pooled data validation | |||||||||
| A & B (Bootstrap) |
Clinical | 0.61 (0.01) | 0.67 | 36.1 (1.1) | 53.3 (1.5) | 31.9 (1.0) | Figure 2e | ||
| Radiomic | 0.71 (0.01) | 0.78 | 32.4 (1.1) | 38.7 (1.7) | 30.9 (1.0) | ||||
| All | 0.71 (0.01) | 0.78 | 31.3 (1.2) | 39.7 (1.8) | 29.3 (1.1) | ||||
| A & B (A Sub-Analysis) |
Clinical | 0.62 (0.01) | - | 37.2 (1.5) | 50.3 (1.9) | 34.0 (1.3) | Supplementary Figure S2a | ||
| Radiomic | 0.69 (0.01) | - | 33.6 (1.4) | 38.3 (2.0) | 32.5 (1.3) | Figure 2f | |||
| All | 0.70 (0.01) | - | 32.7 (1.4) | 39.1 (2.0) | 31.2 (1.3) | Supplementary Figure S2b | |||
| A & B (B Sub-Analysis) |
Clinical | 0.59 (0.02) | - | 33.8 (1.5) | 61.3 (2.4) | 27.1 (1.3) | Supplementary Figure S2a | ||
| Radiomic | 0.73 (0.01) | - | 29.2 (1.5) | 43.8 (2.5) | 25.6 (1.2) | Figure 2f | |||
| All | 0.72 (0.01) | - | 30.6 (1.5) | 41.0 (2.5) | 28.0 (1.3) | Supplementary Figure S2b | |||
| D: Feature harmonization | |||||||||
| □ | A | B | ComBat Radiomic | 0.63 | - | 37.6 | 50.0 | 35.1 | Supplementary Figure S2c |
| ■ | B | A | 0.61 | - | 43.9 | 44.4 | 43.8 | Supplementary Figure S2d | |
| E: Feature harmonization and consolidation | |||||||||
| A | B | n = 1 ComBat Radiomic | 0.66 | - | 42.7 | 35.0 | 44.3 | - | |
| B | A | 0.63 | - | 49.6 | 18.5 | 58.3 | - | ||
| A | B | n = 2 ComBat Radiomic | 0.61 | - | 48.7 | 30.0 | 52.6 | - | |
| B | A | 0.59 | - | 36.6 | 63.0 | 29.2 | - | ||
| A | B | n = 3 ComBat Radiomic | 0.70 | - | 33.3 | 40.0 | 32.0 | Supplementary Figure S2e | |
| B | A | 0.65 | - | 30.1 | 63.0 | 20.8 | |||
| □ | A | B | n = 4 ComBat Radiomic | 0.67 | - | 45.3 | 40.0 | 46.4 | Figure 4b |
| ■ | B | A | 0.70 | - | 26.0 | 33.3 | 24.0 | ||
| A | B | n = 5 ComBat Radiomic | 0.68 | - | 45.3 | 35.0 | 47.4 | Supplementary Figure S2f | |
| B | A | 0.65 | - | 30.1 | 51.9 | 24.0 | |||
| A | B | n = 6 ComBat Radiomic | 0.66 | - | 31.6 | 35.0 | 30.9 | - | |
| B | A | 0.65 | - | 36.6 | 37.0 | 36.5 | - | ||
| A | B | n = 7 ComBat Radiomic | 0.46 | - | 66.7 | 35.0 | 73.2 | - | |
| B | A | 0.61 | - | 46.3 | 37.0 | 49.0 | - | ||
| A | B | n = 8 ComBat Radiomic | 0.60 | - | 35.0 | 50.0 | 32.0 | - | |
| B | A | 0.62 | - | 37.4 | 44.4 | 35.4 | - | ||
| A | B | n = 9 ComBat Radiomic | 0.62 | - | 35.0 | 50.0 | 32.0 | - | |
| B | A | 0.58 | - | 47.2 | 40.7 | 49.0 | - | ||
| A | B | n = 10 ComBat Radiomic | 0.62 | - | 42.7 | 40.0 | 43.3 | - | |
| B | A | 0.58 | - | 39.0 | 63.0 | 32.3 | - | ||
The rows of the table are divided into 5 sections (A–E), corresponding to the 3 types of validation performed, along with 2 sections regarding the feature harmonization and consolidation results. The “Features” column indicates which features were available for use by the model. Comparisons between sections should therefore be made between rows with the same features indicated, with “Radiomic” features being used across all sections. For section E, the “Features” column value refers to using the top 1 to n consolidated radiomic features with ComBat harmonization also applied. Rows marked with “□” represent model external validation experiments for comparison with Center B as the test dataset using only radiomic features with no or different feature harmonization and consolidation techniques applied. Rows marked with “■” are represent a similar comparison, but with Center A as the test dataset. The “ROC Figure” column provides a means to cross-reference error metric values with the ROC curves from which they were derived (see Supplementary Figure S2). Where bootstrapped experiments were performed, 95% CI values are supplied in parentheses, along with AUC0.632+ Values