Figure 5.

The Bootstrap Bias Corrected Cross-Validation (BBC-CV) procedure

BBC-CV returns an estimate of the predictive performance of model M produced by the (meta)Learner CVT in Figure 3. The estimate is an out-of-bag bootstrap estimation of the Select-Best step performance in Figure 3: in each iteration, the input matrix Π produces a bootstrap version of itself using random selection of rows with replacement. The “Select-Best” step is applied to this matrix to find the winning configuration (f_∗). The performance of the winning configuration is estimated on the out-of-bag samples, i.e., the ones not selected by bootstrapping and not seen by the “Select-Best” step. The process is repeated 1,000 times and the average performance estimate is returned. BBC-CV estimates the performance of the model produced by CVT on all data without training any further models.