|
Algorithm 1. APDHE-Pareto ensemble-selection procedure within each training partition |
Input: Training partition candidate classifier set ; allowable ensemble sizes
Output: Selected ensemble and decision threshold
For each candidate classifier, generate calibrated training-OOF probability predictions using the nested model-development procedure.
Using the OOF predictions, calculate ROC-AUC, balanced accuracy, F1-score, and MCC for each candidate classifier at a common probability threshold of 0.50.
Transform MCC to the [0, 1] interval and assign Pareto layers by sequential non-dominated sorting across ROC-AUC, balanced accuracy, F1-score, and transformed MCC.
Calculate model-level prediction diversity from pairwise disagreement and transformed Q-statistic measures.
Calculate model stability as , where is obtained from the fold-level OOF AUC values.
Rank candidate classifiers lexicographically by: lower Pareto-layer index, higher diversity, higher stability, higher ROC-AUC, higher balanced accuracy, higher F1-score, higher transformed MCC, and alphabetical model name as the final deterministic tie-breaker.
For each allowable ensemble size , define the candidate ensemble as the first classifiers in the ranked list. Non-prefix subsets are not evaluated.
For each candidate ensemble, uniformly average the calibrated OOF probabilities of its constituent classifiers.
Determine an OOF decision threshold for each candidate ensemble by maximizing Youden’s ; when equivalent thresholds occur, prioritize higher balanced accuracy and then the threshold closest to 0.50.
Compare candidate ensemble sizes lexicographically by higher OOF ROC-AUC, lower OOF Brier score, higher OOF balanced accuracy, and smaller ensemble size.
Select the highest-ranked candidate ensemble as .
Recalculate the fused OOF probabilities for and determine the final training-derived threshold using the same Youden’s rule.
Fit the corresponding calibrated candidate models on the complete current training partition.
Apply the fixed ensemble composition uniform probability averaging, and fixed threshold to the unseen validation or test observations.
|