Table 1.
Performance of different methods on identifying ambiguous in-distribution (AiD) samples. For all metrics, higher is better. The best results from the methods that require a single forward pass at inference time are highlighted.
| Train Dataset | Method/Model | Test-Set Accuracy | Identifying AiD | |
|---|---|---|---|---|
| AUC-ROC | TNR@TPR95 | |||
| AFHQ | Baseline | 99.44±0.02 | 0.87±0.04 | 48.93±10 |
| Baseline+TS [17] | 99.45±0.00 | 0.85±0.07 | 48.77±9.8 | |
| Baseline+TS+ODIN [38] | 99.45±0.00 | 0.85±0.06 | 35.72±1.26 | |
| Baseline+energy [42] | 99.44±0.02 | 0.87±0.06 | 49.00±1.64 | |
| Mixup [70] | 99.02±0.10 | 0.80±0.05 | 35.66±6.7 | |
| DUQ [65] | 94.00±1.05 | 0.67±0.01 | 26.15±4.5 | |
| DDU [47] | 97.66±1.10 | 0.74±0.02 | 19.65±4.5 | |
| Baseline+ACE | 99.52±0.21 | 0.91±0.02 | 50.75±3.9 | |
|
| ||||
| MC Dropout [9] | 98.83±1.12 | 0.87±0.04 | 51.56±1.2 | |
| 5-Ensemble [32] | 99.79±0.01 | 0.98±0.01 | 51.93±2.7 | |
|
| ||||
| Dirty MNIST | Baseline | 95.68±0.02 | 0.96±0.00 | 28.5±2.3 |
| Baseline+TS [17] | 95.74±0.02 | 0.94±0.01 | 27.90±1.3 | |
| Baseline+TS+ODIN [38] | 95.74±0.02 | 0.79±0.03 | 13.25±4.88 | |
| Baseline+energy [42] | 95.68±0.02 | 0.80±0.03 | 17.60±0.55 | |
| Mixup [70] | 94.66±0.16 | 0.94±0.02 | 25.78±2.1 | |
| DUQ [65] | 89.34±0.44 | 0.67±0.01 | 23.89±1.2 | |
| DDU [47] | 93.52±1.12 | 0.65±0.12 | 20.78±4.0 | |
| Baseline+ACE | 95.36±0.45 | 0.86±0.01 | 34.12±2.6 | |
|
| ||||
| MC Dropout [9] | 89.50±1.90 | 0.75±0.07 | 36.10±1.8 | |
| 5-Ensemble [32] | 95.90±0.12 | 0.98±0.02 | 34.87±3.4 | |
|
| ||||
| CelebA | Baseline | 89.36±0.96 | 0.73±0.01 | 17.18±1.6 |
| Baseline+TS [17] | 89.33±0.01 | 0.72±0.02 | 17.21±1.5 | |
| Baseline+TS+ODIN [38] | 89.33±0.01 | 0.57±0.01 | 6.34±0.38 | |
| Baseline+energy [42] | 89.36±0.96 | 0.57±0.28 | 4.87±0.32 | |
| Mixup [70] | 89.04±0.47 | 0.74±0.02 | 15.09±1.9 | |
| DUQ [65] | 71.75±0.01 | 0.65±0.01 | 14.20±1.0 | |
| DDU [47] | 70.15±0.02 | 0.67±0.06 | 11.39±0.4 | |
| Baseline+ACE | 86.8±0.79 | 0.74±0.06 | 22.36±2.3 | |
|
| ||||
| MC Dropout [9] | 89.86±0.33 | 0.73±0.03 | 19.78±0.7 | |
| 5-Ensemble [32] | 90.76±0.00 | 0.84±0.11 | 17.79±0.6 | |
|
| ||||
| Skin-Lesion (HAM10K) | Baseline | 85.88±0.75 | 0.82±0.06 | 20.52±3.7 |
| Baseline+TS [17] | 86.27±0.40 | 0.84±0.03 | 23.34±2.8 | |
| Baseline+TS+ODIN [38] | 86.27±0.40 | 0.78±0.01 | 15.87±4.33 | |
| Baseline+energy [42] | 85.88±0.75 | 0.77±0.12 | 18.40±0.51 | |
| Mixup [70] | 85.81±0.61 | 0.84±0.04 | 31.29±7.0 | |
| DUQ [65] | 75.47±5.36 | 0.81 ±0.02 | 30.12±4.4 | |
| DDU [47] | 75.84±2.34 | 0.79±0.03 | 26.12±6.6 | |
| Baseline+ACE | 81.21±1.12 | 0.84±0.05 | 71.60±3.8 | |
|
| ||||
| MC Dropout [9] | 84.90±1.17 | 0.85±0.06 | 43.78±1.9 | |
| 5-Ensemble [32] | 87.89±0.13 | 0.86±0.02 | 40.49±5.1 | |