Abstract
Background
Binary classification frequently arises in clinical and public health research, where accurate identification of rare outcomes is perticularly important. The Akaike information criterion (AIC) is a standard tool for model selection, yet it reduces the evaluation to a single global values. Building on our previous work, this study extends the three-level approach of the U-smile method, originally based on BA-RB-I coefficients, with ΔAIC into subclass-specific (level 1), class-specific (level 2), and global (level 3) insights.
Methods
Using Heart Disease dataset and synthetic data, 7 scenarios of 1%, 10%, 30%, 50%, 70%, 90%, 99% event prevalence (n = 300) were generated. A baseline logistic regression model was extended in a nested setting by adding candidate markers individually. Prediciton changes were categorized into 4 U-smile subclasses: improved and worsened predictions for events and non-events. ΔAIC values were computed at each level and visualized with U-smile and trend plots.
Results
The U-smile plots of level 1 differentiated informative markers (smile shapes) from non-informative ones (flatline shapes) across the whole imbalance range. At level 2, improvements in minority class fir (e.g., 2.23, 2.52) and reductions in majority class overfitting (e.g., -0.79, -0.92) under extreme imbalance were exposed. These effects were partially obscured at the global level 3, where ΔAIC suggested more modest improvements (e.g., 1.42, 1.91).
Conclusions
The three-level approach of the U-smile method with ΔAIC provides a structured and interpretable framework for model and feature selection in balanced and imbalanced settings. By decomposing a global model selection measure into its subclass- and class-specific components, it reveals pattern otherwise hidden in aggregate measures, supporting more precise and equitable evaluation.
Key messages
• In binary classification, the three-level extension of the U-smile method with ΔAIC provides subclass, class, and global insights, enabling structured evaluation beyond a single ΔAIC value.
• Class-level ΔAIC reveals improved model fit in the minority class and reduced overfitting to the majority class under extreme imbalance scenarios, often hidden by global aggregate measures.