|
Algorithm 2: Source-Only Reliability Fitting and External Evaluation |
Input: Trained primary diagnostic checkpoint; probability-calibration subset; conformal-calibration subset; source reliability-threshold partition; source test set; PAD-UFES-20. |
Output: Calibrated probabilities, conformal prediction sets, source-fitted selective decisions, and external reliability measurements. |
Method:
Freeze the trained primary diagnostic checkpoint.
Fit probability calibration using only the designated source calibration subset.
Apply the fixed calibration parameters to the independent conformal-calibration subset.
Estimate class-conditional conformal thresholds using this subset only.
Apply the fixed calibration and conformal pipeline to the source reliability-threshold partition.
Compute calibrated-confidence risk, predictive entropy, logit-derived energy, and conformal-set size.
Fit the source selective-classification rule and freeze all scaling parameters and decision thresholds.
Apply the complete fixed pipeline to the source test set and PAD-UFES-20 without adaptation.
Report discrimination, calibration, conformal, uncertainty-based error-detection, and selective-classification performance.
Perform DDI OOD and explanation-stability analyses separately using the prespecified fixed audit checkpoint.
Constraint: PAD-UFES-20 and DDI results are not used to modify the diagnostic model, calibration parameters, conformal predictor, or selective-classification thresholds. |