Algorithm 1 Algorithm of the optimal solution

Step 1. Determine the dataset $\mathrm{D}$, the classifier $h$, the learning algorithm $\mathrm{A}$, the required accuracy $\mathsf{\sigma }$, the proportion $\mathsf{\tau }$ in the holdout method.

Step 2. Initialize $\mathrm{n}=0$, $\mathrm{acc}({\mathrm{C}}^{(0)})$ = 0, ${\mathbf{D}}^{\ast }=\varnothing$;

Step 3. Execute the iteration below:

While $\mathrm{acc}({\mathrm{C}}^{(n)})<\mathsf{\sigma }$:

If $n<m$:

$n=n+1$;

For every sample $s$ in $\mathbf{D}$:

take $s[1,n]$ into ${\mathbf{D}}^{\ast }$

End

Learn the classifier ${\mathrm{C}}^{(n)}$ using the learning algorithm $\mathrm{A}$ on the dataset ${\mathbf{D}}^{\ast }$;

Estimate $\mathrm{acc}({\mathrm{C}}^{(n)})$ using the holdout method;

${\mathbf{D}}^{\ast }=\varnothing$

Else:

End while

Step 4. Output the optimal sampling number $n$