Algorithm 1: Model training and hyperparameter optimization for the standard AI models. |
Require: Sets of model hyperparameter values to evaluate ▹ Cartesian product |
Ensure: Average model performance across all folds |
-
1:
for each external fold iteration do ▹ External 5-fold CV
-
2:
Hold-out the samples of this fold (i.e., the test set)
-
3:
Use the rest of samples as calibration set
-
4:
for each hyperparameter set do
-
5:
for each internal fold iteration in the calibration set do ▹ Internal 5-fold CV
-
6:
Hold-out the samples of this fold (i.e., the validation set)
-
7:
Fit the model on the rest (i.e., the training set)
-
8:
Use the model to predict the held-out samples
-
9:
Calculate and store an accuracy metric for this fold
-
10:
end for
-
11:
Calculate the average performance across all folds
-
12:
end for
-
13:
Determine the optimal hyperparameter set
-
14:
Fit the final model using the calibration set with the optimal hyperparameters
-
15:
Use the model to predict the held-out samples
-
16:
Calculate and store accuracy metrics for this fold
-
17:
end for
-
18:
Calculate the average performance across all folds
|