Fig. 1. Classical versus NP oracle classifiers in a binary classification example.

In this toy example, the two classes have equal marginal probabilities, that is, $\mathbb{P}(\mathit{Y}=0)=\mathbb{P}(\mathit{Y}=1)=0.5$. Suppose that a user prefers a type I error of ≤0.05. The classical classifier I(X > 1) that minimizes the risk would result in a type I error of 0.16. On the other hand, the NP classifier I(X > 1.65) that minimizes the type II error under the type I error constraint (α = 0.05) delivers a desirable type I error. This figure is adapted from Tong et al. (17) and Li and Tong (52), with permissions.