Fig. 1. Unreliability scores are a function of class imbalance.

a Average unreliability as a function of the classifier prediction, $f\left(\stackrel{\rightharpoonup }{x}\right)$, when there is no class imbalance in the data, $P\left(y=1\right)=0.5$. b Average unreliability for different predictions, $f\left(\stackrel{\rightharpoonup }{x}\right)$, in the setting of large class imbalance, $P\left(y=1\right)=0.01$. Calculated expected values assume a uniform distribution for ${\beta }_{\stackrel{\rightharpoonup }{x}}$.