Table 1:
ImageNet results: For all models except VGG19, PEP achieves statistically significant improvements in calibration compared to baseline (BL) and temperature scaling (TS), in terms of NLL and Brier score. PEP also reduces test errors, while TS does not have any effect on test errors. Although TS and PEP outperform baseline in terms of ECE% for DenseNet121, DenseNet169, ResNet, and VGG16, the improvements in ECE% is not consistent among the methods. T* and σ* denote optimized temperature for TS and optimized sigma for PEP, respectively. Boldfaced font indicates the best results for each metric of a model and shows that the differences are statistically significant (p-value<0.05).
| σ* | Negative log-likelihood | Brier score | ECE% | Top-1 error % | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | T* | ×10−3 | BL | TS | PEP | BL | TS | PEP | BL | TS | PEP | BL | PEP |
| DenseNet121 | 1.10 | 1.94 | 1.030 | 1.018 | 0.997 | 0.357 | 0.356 | 0.349 | 3.47 | 1.52 | 2.03 | 25.73 | 25.13 |
| DenseNet169 | 1.23 | 2.90 | 1.035 | 1.007 | 0.940 | 0.354 | 0.350 | 0.331 | 5.47 | 1.75 | 2.35 | 25.31 | 23.74 |
| IncepttionV3 | 0.91 | 1.94 | 0.994 | 0.975 | 0.950 | 0.328 | 0.328 | 0.317 | 1.80 | 4.19 | 2.46 | 22.96 | 22.26 |
| ResNet50 | 1.19 | 2.60 | 1.084 | 1.057 | 1.023 | 0.365 | 0.362 | 0.350 | 5.08 | 1.97 | 2.94 | 26.09 | 25.18 |
| VGG16 | 1.09 | 1.84 | 1.199 | 1.193 | 1.164 | 0.399 | 0.399 | 0.391 | 2.52 | 2.08 | 1.64 | 29.39 | 28.83 |
| VGG19 | 1.09 | 1.03 | 1.176 | 1.171 | 1.165 | 0.394 | 0.394 | 0.391 | 4.77 | 4.50 | 4.48 | 28.99 | 28.75 |