Table 8.

Overview of cross-validation techniques.

Dataset Cross-validation	Description
Hold out	Dataset is randomly split into a training and test set. Can suffer from sampling bias and overfitting to the training set.
k-fold	Data is split into k folds and the data is trained on k-1 folds and tested on the fold that was left out. Process is repeated and the result is averaged. The major advantage is that all observations are used for both training and validation.
Leave-one-user-out (LOUO)	Similar to k-fold validation. In LOUO validation, each surgeon’s trials are used as the test set in turn. Repeated until each surgeon’s trials are used for testing.
Leave-one-super-trial-out (LOSO)	Also a variation on k-fold validation. In LOSO validation, a trial from each surgeon’s set of trials is used as the test set. This process is repeated and the result is averaged. This tends to achieve better results compared to LOUO as the algorithm can learn on all surgeons’ trials.
Bootstrapping	Bootstrapping is similar to k-fold validation but resamples with replacement such that the new training datasets will always have the same number of observations as the original dataset. Due to replacement, bootstrapped datasets may have multiple instances or completely omit the original cases.