Table 1.
Summary of the reviewed evaluation strategies. Iterations: number of iterations, i.e. number of times a classifier is constructed and applied to data; u.d.= user-defined. Bias: Bias of the error estimation; ↑ means positive bias, i.e. underestimation of prediction accuracy and vice-versa. Principle: Gives the definition of the learning and test sets or the used combination of methods.
| Iterations | Bias | Principle | |
|---|---|---|---|
| Resubstitution | 1 | ↓ | l = t = {1, …, n} |
| Test | 1 | ↑ | {l, t} from a partition of {1, …, n} |
| LOOCV | n | – | t(j) = {j }, l(j) = {1, …, n}\{j },for j = 1, …, n |
| m-fold-CV | m | ↑ |
t(1), …, t(m) from a partition of {1, …, n}
l(j) = {1, …, n}\ t(j), for j =1, …, m} |
| MCCV | B (u.d.) | ↑ | {l(b), t(b)} from a partition of {1, …, n}, for b = 1, …, B |
| Bootstrap | B (u.d.) | ↑ |
l*(b) is a bootstrap sample drawn out of {1, …, n}
t*(b)= {1, …, n}\l*(b), for b = 1, …, B |
| 0.632,0.632+ | B (u.d.) | – | Weighted sum of resubstitution and bootstrap error rates. |
| Bootstrap-CV | nB (u.d) | – | LOOCV within B bootstrap samples. |