Table 4.

Mapping technical criteria and solutions for algorithm robustness and performance.

criterion	technical solution
expected generalization performance	cross-validation [43]: k-fold-cv, leave-one-out, etc. covariance-penalty [20]: Mallow’s Cp, Stein unbiased risk estimator concept drift [79,80]: gradual mitigation, abrupt correction and pre-emptive detection
adversarial robustness	evasion attacks: fast gradient sign method [81], DeepFool [82], etc. defence: label smoothing [76], variance minimization [83], thermometer encoding [84], etc.
formal verification	complete: satisfiability modulo theory [85,86], mixed integer programming [87], etc. incomplete: propagating bounds [88], Lagrangian relaxation [89], etc.
reliability and reproducibility	code versioning: Git (Github), Mercurial (BitBucket), etc. reproducible analysis: Binder, Docker, etc. automated testing: Travis CI, Scrutinizer CI, etc.