Translating machine learning predictions into meaningful risk estimates to support clinical decisions: a post hoc analysis of chronic obstructive pulmonary disease adverse outcomes using unified auto clinical scores

Abstract

Objective

The successful integration of Machine Learning (ML) models into clinical practice remains limited, as they often lack the standardized, quantifiable risk measures essential for clinical workflows. This study, therefore, aims to demonstrate the Unified Auto Clinical Scores (Uni-ACS) method as a means to translate ML predictions into interpretable biostatistical formats, a translation critical for clinical adoption and alignment with evidence-based practice guidelines.

Materials and Methods

We employed the Uni-ACS post hoc methodology to convert ML model outputs into clinical scores and odds ratios. Validation was performed on a retrospective cohort of Chronic Obstructive Pulmonary Disease patients admitted between 2016 and 2018, predicting adverse outcomes from 2019 to 2020. The performance of Uni-ACS was benchmarked against the original ML models and logistic regression.

Results

The successful application of Uni-ACS enabled the conversion of complex ML model outputs into readily interpretable clinical scores and metrics, like odds ratios. Clinical scores derived from the ML models’ SHAP values using Uni-ACS maintained a strong, interpretable predictive performance (AUROC 0.69-0.80), which was comparable to the original ML models (AUROC 0.71-0.83) and logistic regression (AUROC 0.72-0.81).

Discussion

Uni-ACS addresses the fundamental clinical requirement for standardized, meaningful risk stratification. It efficiently translates complex ML predictions into validated clinical scores with minimal computational burden.

Conclusion

In conclusion, this approach facilitates the widespread adoption of ML-driven risk assessment across diverse healthcare settings while ensuring compatibility with existing clinical guidelines and regulatory frameworks.

Introduction

While Machine Learning (ML) models have superior predictive performance for clinical outcomes as compared to traditional statistical methods,^1–3 successful integration of ML models’ outputs into clinical medicine remains sparse.⁴ Recent developments have introduced new possibilities for medical AI applications, yet have not resolved the fundamental barriers to clinical translation of ML models.⁵ This phenomenon could be attributed to the following significant barriers that persist even in the era of advanced AI systems.

First, there is a fundamental incompatibility of ML models with current clinical workflows. For example, an ML model to predict the mortality risk of Chronic Obstructive Pulmonary Disease (COPD) is unlikely to replace the current standard of care using clinical tools like the Global Initiative for Obstructive Lung Disease criteria⁶ for determining patient risk and the necessary clinical intervention. This is because granular risk prediction outputs from ML models are meaningless if they do not correlate to meaningful follow-up clinical actions. While Large Language Models (LLMs) can generate natural language explanations of predictions, they do not provide the standardized, meaningful risk stratification that clinical decision-making requires.⁵ Therefore, even when ML models far surpass conventional clinical scores in risk prediction, clinical scores continue to be developed or applied in parallel to ML techniques.^7–9

Second, ML models are notable for being black box models, where explanations of the prediction outcomes are opaque to users of the ML models.¹⁰ While there are advances in explainable ML methods such as Local Interpretable Model-agnostic Explanations¹¹ and SHapley Additive exPlainable (SHAP) values,¹² outputs from these techniques cannot be easily interpreted in the clinical context. Although LLMs and generative AI can provide verbose interpretations of ML models, these interpretations do not improve ML transparency to help clinicians in their evidence-based practice.¹³ The ability to express predictions as clinical scores with associated odds ratios remains critical for integrating AI predictions with existing medical knowledge and regulatory frameworks.

Third, even if the ML models could be clinically interpreted, there are differences in clinical feature risk interpretation between clinicians and ML developers. For instance, in clinical medicine and biostatistics, Logistic Regression (LogReg) models’ weights like beta values can be used to rank the feature importance’s and calculate the odds ratio associated with each of these features.¹⁴ On the other hand, there is no consistent methodology across ML models to rank the importance of specific features and calculate odds ratios associated with top features.¹⁵ Even newer AI architectures, like LLMs may generate plausible-sounding explanations without providing the quantitative risk measures that are standard in clinical epidemiology.¹³

Finally, there are challenges in deploying ML models in clinical workflows within modern Electronic Medical Record (EMR) systems. While EMRs such as Epic have developed integrations to incorporate ML models as part of clinical workflow,¹⁶ deployment of ML models is non-trivial as clinical workflows need to be reviewed to incorporate ML model outputs. Furthermore, with the advent of LLMs, it is increasingly difficult to set aside dedicated computing resources need for deployment of ML models within resource constrained health care settings.^13,17 Clinical scores, by contrast, can be calculated with minimal computational overhead and integrated seamlessly into existing workflows.

In a previous study, we developed an algorithm, Unified Auto Clinical Scoring (Uni-ACS) to address some of the aforementioned barriers.¹⁸ In this study, we introduce, demonstrate, and validate the application of the Uni-ACS framework. Specifically, we show how Uni-ACS can be applied to ML model outputs predicting adverse outcomes for patients with COPD. The core informatics innovation lies in Uni-ACS’ ability to translate the arbitrary, non-linear ML model output into a standardized, interpretable clinical risk score, guided by clinicians’ domain expertise. The resultant output can be presented as a point-based summary and a risk tier, allowing for immediate risk assessment at the point of care without requiring manual data entry or additional compute. Our focus is on the generalizable informatics methodology of Uni-ACS, with the COPD ML model serving as a relevant, but secondary, clinical application.

Methods

Study design

We used a retrospective cohort study of patients discharged with a diagnosis of COPD between 2016 and 2018, conducted at a hospital in Singapore, Tan Tock Seng Hospital (TTSH). Patients were included if they had an ICD-9 or ICD-10 diagnosis code of COPD, emphysema, or chronic bronchitis on discharge. (See Supplementary Material Table S1 Appendix A Table S1 for the list of codes used). We confirmed patients’ COPD diagnosis against spirometry results when available, using a cutoff of FEV₁/FVC ratio less than 0.7. Patients who died during the index admission of interest were excluded. A flow diagram to illustrate our cohort selection criteria is in Figure 1. In total, 1245 patients were eligible for analysis after applying our inclusion and exclusion criteria. They were followed for 365 days from discharge date. We further validated our final selected model by applying the model on 488 COPD patients who had an index admission from 2014 to 2015.

Figure 1.

Flow diagram showing COPD cohort selection. COPD, Chronic Obstructive Pulmonary Disease; ICD, International Classification of Diseases.

Our outcomes of interest were:

1. Early readmission: Hospital readmission for Acute Exacerbation of COPD (AECOPD) within 30 days of index discharge (codes in Table S2).

2. Frequent readmissions: Two or more AECOPD admissions within 365 days of index discharge (excluding the index admission).

3. Composite Outcome: Positive for either early or frequent readmission.

4. Mortality: Death within 365 days of index discharge.

5. Long-stayer: Cumulative length of stay greater than 21 days across all admissions within 365 days of index discharge.

The purpose of this study is to use the above cohort of COPD patients to demonstrate and validate the use of Uni-ACS to translate ML model predictions of adverse outcomes into clinical scores.

Data source

Data for the study was extracted from TTSH’s Electronic Health Records via the Healthcare Intelligence (HI) system, an institutional data warehouse that consolidates structured data from at least 25 different source systems including the EMR system, medication database and laboratory information system. Our study ethics approval covered data extraction between 2016 and 2018.

Uni-ACS algorithm

The Uni-ACS algorithm is a systematic framework designed to translate the complex predictive capabilities of a ML model into clinical scores. It accepts a trained ML model as its primary input and aims to achieve 2 critical outcomes: first, to derive robust measures of clinical risk association, such as odds ratios, directly from the ML model’s inherent feature importance; and second, to generate a simple, additive clinical risk score that closely approximates the ML model’s underlying mechanism for aggregating patient risk.

The operation of Uni-ACS is structured into a rigorous 5-step process (Figure 2), ensuring both fidelity to the original ML model’s prediction and transparency in the final score:

Figure 2.

Visual representation of Uni-ACS pipeline.

Step 1: calibration and feature contribution assessment

The initial phase focuses on establishing a reliable foundation for interpretation. Uni-ACS first ensures that the ML outputs match actual clinical risk by employing Isotonic Regression to calibrate the model’s sigmoid/softmax probabilities against observed outcomes, ensuring the “risk magnitude” is statistically grounded before SHAP decomposition. This is crucial for the subsequent derivation of clinically meaningful risk measures. Following calibration, the algorithm calculates SHAP values for every feature across all instances. SHAP values are a powerful tool from cooperative game theory that quantify the contribution of each feature’s value to the ML model’s final prediction, thereby providing a comprehensive, instance-level assessment of feature importance.

Step 2: optimal feature selection

With the individual feature contributions quantified, the next step involves identifying the most influential and clinically relevant features to be included in the final additive score. Uni-ACS primarily employs a method of selecting the best features based on the average absolute SHAP values calculated for each feature across the entire dataset. This quantitative measure ensures that features with the highest overall impact on the model’s predictions are prioritized. Importantly, the framework offers flexibility by allowing domain experts to manually include features from the original ML model that possess high clinical utility, even if they do not rank in the top tier of global SHAP importance. Because Uni-ACS is a post hoc analysis of a pre-trained model, this selection process does not alter the underlying model weights. Furthermore, as SHAP values are inherently additive, including or excluding a feature from the clinical score does not “skew” the risk contributions of the remaining features, ensuring the scoring system remains a faithful, modular representation of the base ML model’s logic.

Step 3: granular feature value categorization

The algorithm then moves to segment the continuous or high-cardinality categorical feature values into distinct, meaningful clinical groups. This step, known as feature value categorization, is achieved by identifying feature value ranges (or categories) that are consistently associated with unique patterns of SHAP values. By grouping feature values that exhibit similar risk contributions, Uni-ACS creates a set of clinically distinct categories for each selected feature, making the interpretation of the final score simpler and more intuitive for practitioners.

Step 4: aggregation and summative risk assignment

Once the feature value ranges are defined, Uni-ACS aggregates the SHAP values within each specified range. This aggregation process is performed to assign a summative risk value to every feature value range identified in Step 3. This risk value represents the typical, integrated contribution of that specific feature category to the overall risk prediction, directly translating the model’s internal logic into a quantifiable risk magnitude.

Step 5: clinical risk association and score finalization

The final step bridges the aggregated risk values back to established clinical metrics and generates the final score. For each defined feature value range, Uni-ACS performs 2 key calculations: (1) Odds Ratio Calculation: It calculates the associated odds ratio, a standard epidemiological measure, to quantify the increase or decrease in risk relative to a baseline category. Because SHAP values account for the marginal contribution of a feature in the context of all other features in the model, the resulting ORs effectively reflect the risk associated with a feature value range while controlling for other predictive variables. (2) Integer Clinical Score Determination: Based on the calculated odds ratios and the aggregated risk values, the algorithm determines a simple, non-negative integer clinical score value to be assigned to that feature value range. This process transforms the complex ML output into an easily calculable, transparent, and clinically usable additive score, enabling clinicians to quickly estimate a patient’s risk profile by simply summing the points associated with their clinical features.

Modeling methodology

Since Uni-ACS is ML model agnostic, any state-of-the-art ML model can be used. We employed 2 ML categories: (1) Ensemble decision trees (GBoost, RanFor, XGBoost^19–21) and (2) Deep learning (NN²²). We compared these to conventional Logistic Regression (LogReg). We then applied Uni-ACS to these ML and LogReg models to derive and compare clinical scores against their base models.

The cohort was split 80% (training) and 20% (held-out test). Hyperparameter tuning used grid search with 5-fold cross-validation, repeated 10 times, on the training data. Final evaluation on the held-out test set used AUROC and AUPRC to address class imbalance and achieve high positive predictive value. Baseline AUPRC was defined as the COPD prevalence in the cohort (non-discriminating model AUPRC).

Statistical methods

Categorical variables were reported as frequencies and percentages whereas continuous variables were reported as means with standard deviations (SD) for descriptive analyses. In bivariate analyses, Pearson’s χ² test was used for categorical variables whereas the student’s t-test was used for continuous variables. Any difference is deemed statistically significant if P < .05.

Software

Python version 3.13 software was used as the main programming code language. Scikit Learn package version 1.8 was used in the interpretation of ML models for Logistic Regression, Gradient Boosted Ensemble Decision Trees, Random Forest Ensemble Decision Trees as well as to train, test and validate dataset split model performance and metrics. XGBoost 3.1 software was used for ML model interpretation in extreme gradient boosted ensemble decision trees. Pytorch 1.11 software was used in interpretation of multilayer perception neural network MLs. SHapley Additive explanation (SHAP) package version 0.4 was used in application of tree and kernel SHAP values, and Uni-ACS (https://github.com/llja0112/Uni-ACS) was used in translation of ML models into clinical scores.

Results

There were a total of 1245 patients in this cohort of patients with COPD. Out of these patients, 1064 (88.5%) and 969 (77.8%) of these patients were males and Chinese, respectively. This was a predominantly elderly and frail cohort with a mean age of 72.6 years old, mean Hospital Frailty Score (HFS) of 4.2 and Charlson’s Comorbidity Index (CCI) score of 3.4. There were 253 (20.3%) patients living in blocks which contain rental flats, offered to Singapore citizens in the lowest income bracket.²³ There were no significant differences between the training and held-out test set (Table 1).

Table 1.

Descriptive statistics of the train and held out test dataset.

	Overall (N = 1245)	Training dataset (N = 996)	Held Out Test dataset (N = 249)	P value
Age, mean (SD)	72.6 (10.7)	72.4 (10.8)	73.2 (10.6)	.29
Male gender, n (%)	1064 (88.5%)	855 (85.8%)	209 (83.9%)	.51
Ethnicity, n (%)				.99
Chinese	969 (77.8%)	772 (77.5%)	197 (79.1%)
Malay	116 (9.3%)	92 (9.2%)	24 (9.6%)
Indian	88 (7.1%)	72 (7.2%)	16 (6.4%)
Others	72 (5.8%)	60 (6.0%)	12 (4.8%)
BMI, mean (SD)	22.0 (5.1)	22.0 (5.2)	21.9 (4.8)	.73
Housing: Staying in rental block, n (%)	253 (20.3%)	205 (20.6%)	48 (19.3%)	.71
CCI, mean (SD)	3.4 (2.5)	3.4 (2.5)	3.4 (2.4)	.96
HFS, mean (SD)	4.2 (4.7)	4.3 (4.9)	3.8 (4.1)	.12
Comorbiditiesa
Diabetes Mellitus, n (%)	347 (27.9%)	281 (28.2%)	66 (26.5%)	.65
Hypertension, n (%)	28 (2.2%)	23 (2.3%)	5 (2.0%)	.96
Chronic Heart Failure, n (%)	192 (15.4%)	158 (15.9%)	34 (13.7%)	.44
Ischaemic Heart Disease, n (%)	283 (22.7%)	231 (23.2%)	52 (20.9%)	.49
Depression, n (%)	42 (3.4%)	33 (3.3%)	9 (3.6%)	.97
Mortality, n (%)	157 (12.6%)	124 (12.4%)	33 (13.3%)	.73
Early readmission, n (%)	262 (21.0%)	207 (20.8%)	55 (22.1%)	.65
Frequent readmission, n (%)	315 (25.3%)	247 (24.8%)	68 (27.3%)	.42
Longstayer, n (%)	240 (19.3%)	198 (19.9%)	42 (16.9%)	.28

All figures denote levels at baseline.

^aDefined by presence of diagnosis codes reflecting each comorbidity.

CCI, Charlson comorbidity index; BMI, body mass index; HFS, hospital frailty score; SD, standard deviation.

Table 2 shows the comparative predictive performances of the best ML models and the LogReg model applied to the 5 outcomes of interest.

Table 2.

Comparison of predictive performance in the test set of the original cohort of COPD patients with index admissions between 2016 and 2018.

	AUROC	AUPRC	AUROC (Clinical Score)	AUPRC (Clinical Score)	AUPRC (Non-discriminating model)
Early Readmission (Baseline LogReg)	0.75 (0.74, 0.76)	0.40 (0.39, 0.41)	0.67 (0.67, 0.68)	0.33 (0.25, 0.34)	0.13
Early Readmission (GBoost)	0.71 (0.70, 0.72)	0.37 (0.36, 0.39)	0.69 (0.69, 0.70)	0.32 (0.33, 0.33)	0.13
Frequent Readmissions (Baseline LogReg)	0.75 (0.74, 0.75)	0.53 (0.52, 0.54)	0.74 (0.74, 0.75)	0.49 (0.47, 0.52)	0.21
Frequent Readmissions (GBoost)	0.80 (0.80, 0.81)	0.64 (0.64, 0.66)	0.80 (0.79, 0.81)	0.54 (0.53, 0.55)	0.21
Composite Outcome (Baseline LogReg)	0.72 (0.72, 0.73)	0.59 (0.58, 0.59)	0.69 (0.69, 0.70)	0.51 (0.50, 0.52)	0.25
Composite Outcome (XGBoost)	0.74 (0.74, 0.75)	0.60 (0.590, 0.61)	0.70 (0.69, 0.71)	0.58 (0.58, 0.59)	0.25
Mortality (Baseline LogReg)	0.74 (0.74, 0.75)	0.47 (0.46, 0.49)	0.72 (0.72, 0.73)	0.40 (0.36, 0.41)	0.19
Mortality (XGBoost)	0.77 (0.76, 0.77)	0.49 (0.48, 0.50)	0.71 (0.70, 0.72)	0.34 (0.34, 0.35)	0.19
Longstayer (Baseline LogReg)	0.81 (0.80, 0.81)	0.45 (0.44, 0.47)	0.80 (0.78, 0.81)	0.41 (0.40, 0,42)	0.11
Longstayer (RanFor)	0.83 (0.83, 0.84)	0.51 (0.50, 0.52)	0.80 (0.79, 0.81)	0.37 (0.366, 0.380)	0.11

AUPRC, area under precision-recall curve; AUROC, area under the receiver operating characteristic. Bold values highlight the models with better predictive performance for each of the 5 outcomes of interest.

The best ML model’s SHAP outputs for each outcome were converted into clinical scores with Uni-ACS. Table 2’s columns 3 and 4 illustrate the scores’ performances post conversion.

Across all outcomes, both LogReg and ML models performed satisfactorily with AUROCs ranging from 0.71 to 0.83, which are comparable to current state-of-the-art risk models for COPD (0.57-0.78) found in existing literature.^24,25 The predictive performance of the Uni-ACS derived clinical scores, AUROCs ranging from 0.67-0.80, was comparable to the original LogReg and ML models from which they were converted from (0.71-0.83).

For both AUROC and AUPRC, LogReg was found to be the best candidate model for early readmission (0.75 and 0.40, respectively), while GBoost was best for frequent readmissions (0.80, 0.64); XGBoost provided the best models for predicting composite readmission outcome (0.74, 0.60) and mortality (0.77, 0.49). AUPRCs of all models were significantly better than the AUPRCs that would have been obtained from a non-discriminating model.

The clinical scores for mortality converted from the best ML model, XGBoost, and the individual features’ odds ratio and SHAP values can be found in Table 3, while a comparison of SHAP score values and clinical score risk estimates in relation to features values is depicted in Figure 3. Table 4 shows the total Uni-ACS score can be stratified into risk tiers (Low, Moderate, High) and provides the corresponding observed probability of mortality for each tier. This allows a clinician to interpret a total score of, for example, “36” as a specific percentage risk of 50% mortality within 1 year.

Table 3.

Clinical score to predict mortality risk amongst COPD patients.^a

Feature	Scores	Mean |SHAP| values	Odds ratio (95% CI)
BMI < 18.8	7	0.28	1.49 (1.43, 1.54)
Age ≥ 75 years old	6	0.24	1.59 (1.49, 1.70)
RDW on discharge ≥ 15.9%	9	0.23	1.70 (1.64, 1.76)
Urea on discharge ≥ 5.6 mmol/dL	6	0.22	1.60 (1.56, 1.65)
Opioids prescribed	4	0.18	1.34 (1.33, 1.34)
HFS score ≥ 4.3	6	0.18	1.46 (1.40, 1.52)
CRP on admission < 47.7 mg/L	3	0.16	1.30 (1.27, 1.33)
Creatinine on discharge ≥ 119 micromole/L	4	0.14	1.24 (1.20, 1.28)
Haematocrit on discharge < 32.5%	8	0.12	1.75 (1.68, 1.81)
Procalcitonin on discharge ≥ 3.5	3	0.10	1.22 (1.17, 1.27)

^aInteger clinical scores are derived from the model’s internal non-linear logic (aggregated SHAP values), whereas Odds Ratios are post hoc summary statistics. A higher score reflects a greater relative contribution to the specific ML model’s prediction within this cohort, which may not always align linearly with the OR.

CI, confidence interval; BMI, body mass index; RDW, red-cell distribution width; HFS, hospital frailty score; CRP, C-reactive protein.

Figure 3.

Comparison of ML models’ underlying SHAP values, as indicated by the blue line, with the clinical score’s SHAP values, as indicated by the yellow line. BMI, Body Mass Index; CRP, C-Reactive Protein; HCT, Hematocrit; HFS, Hospital Frailty Score; PCT, Procalcitonin; RDW, Red cell Distribution Width; SHAP, SHapley Additive exPlainable values.

Table 4.

Risk stratification tiers for 1 year mortality (based on Uni-ACS).

Total score	Risk category	Observed mortality
0-10 points	Low	11%
11-35 points	Medium	24%
>35 points	High	50%

Clinical scores for other outcomes can be found in Appendix B Tables S3-S6.

The ML model interpretations of the various adverse outcomes of COPD are shown in Figure 4, where a larger mean absolute SHAP value suggests greater risk attribution. By applying Uni-ACS to the factors with the top 3 SHAP values for each outcome, we were able to derive clinically relevant odds ratios for these risk factors. The odds ratios of the individual risk factors contributing to mortality in the best ML model, XGBoost, can be found in the last column of Table 3.

Figure 4.

Global SHAP importances for (A) Early, (B) Frequent, (C) Composite, (D) Mortality, (E) Longstayer outcomes. AMI, Acute Myocardial Infarction; BMI, Body Mass Index; CCI: Charlson’s Comorbidity Index; CRP, C-Reactive Protein; ED, Emergency Department; ESR, Erythrocyte Sedimentation Rate; FEV1/FVC, Forced Expiratory Volume (FEV)1/Forced Vital Capacity (FVC); HFS, Hospital Frailty Score; Hx, History; LABA, Long Acting Beta Agonist; LAMA, Long Acting Muscarinic Antagonist; LBP, Look Back Period; LOS, Length Of Stay; RDW, Red cell Distribution Width; SHAP, SHapley Additive exPlainable values; SOC, Specialist Outpatient Clinics; Spiro, Spirometry; WBC, White Blood Cell count

We further validated the best ML models (trained and tested on the 2015-2018 cohort) on the 2014-2015 patient cohort (see Appendix B Tables S7 and S8). While there was a drop in AUROC between 0.02 and 0.08, most of the original ML models’ AUROC performances were above 0.7, consistent with what we found in our literature review.^26–30 While there was also a drop in AUPRC between 0.03 and 0.14, all models retained higher AUPRCs when compared with the AUPRCs that would be obtained from a non-discriminating model.

Despite being a simplified interpretation of the original ML model, Uni-ACS clinical scores’ risk attribution is consistent with their underlying base ML models’ risk attribution, hence, their predictive performance is comparable to the underlying base ML models as depicted in Figure 3, which shows the plot of the SHAP values of the original underlying ML model and the corresponding SHAP values of the scores, for the top 10 most important risk features.

From the multivariable analysis derived from Uni-ACS, high urea (>5.73 mmol/dL) and creatinine (>124.63 micromole/L) on discharge were found to be the strong predictors of mortality with odds ratios of 1.60 (95% CI: 1.56-1.65) and 1.24 (95% CI: 1.20-1.28), respectively. High RDW (OR: 1.70, 95% CI: 1.64-1.76), low hematocrit of <32.11% (OR: 1.75, 95% CI: 1.68-1.81), low BMI of <18.8 (OR: 1.49, 95% CI: 1.43-1.54), old age (OR: 1.59, 95% CI: 1.49-1.70) and high HFS (OR: 1.46, 95% CI: 1.40-1.52) were also found to be strongly associated with mortality. (Cutoffs used can be found in Table 3).

Discussion

A primary contribution in this study is the Uni-ACS clinical score (Tables 3 and 4), which can be calculated by hand and is well understood by clinicians. Furthermore, compared to SHAP values, the Uni-ACS clinical score’s accompanying odds ratios are well documented in medical literature and can be used to counsel laypeople on disease determinants and risk. While additional post processing steps and computational resources are required to develop the clinical scores during model development, users do not require a computer or the original ML model parameters to infer and explain risk.

To validate the advantage of Uni-ACS scores against existing clinical risk prediction methods, we used a real-world COPD dataset to benchmark the performance of Uni-ACS scores against robustly constructed ML models. The validation showed that Uni-ACS scores derived from the best-performing ML models achieved predictive performance that was only marginally lower compared to the original ML models, a drop in performance that is expected as Uni-ACS is an estimation based on aggregate SHAP values of the original models’ top risk factors. This suggests that Uni-ACS scores might have retained the non-linear patterns captured by ML models, thus maintaining a high level of predictive performance. For the specific outcome of early readmission, where a simpler LogReg approach sufficed for risk prediction, it suggested that there were no complex data patterns within the input feature set which require sophisticated modeling techniques. Finally, we completed a sensitivity analysis of Uni-ACS scores against an earlier patient cohort to confirm the reproducibility of their predictive performance.

While specific clinical predictors of poor outcomes among the COPD cohort were analyzed to understand the underlying ML models, the focus remains on illustrating how Uni-ACS standardizes these insights into an interpretable format. For instance, the analysis showed that length of stay (LOS) during the Look Back Period (LBP), defined as 1 year prior to admission, was the top feature contributing to the Uni-ACS score for the outcomes of frequent readmissions, composite outcome, and longstayers, while low Body Mass Index (BMI), old age, and high Urea levels were found to have higher contributions to the mortality score. The finding that high urea and creatinine on discharge are strong predictors of mortality is consistent with previous clinical studies on acute kidney injury in COPD exacerbation.³¹ Similarly, associations between High RDW, low hematocrit, low BMI, elderly status, high HFS and mortality correlate well with existing literature.^32–34 While Uni-ACS is primarily a predictive rather than a causal inference framework, the risk estimates generated align with the theoretical causal determinants of COPD adverse outcomes. For instance, the high ORs for urea and creatinine reflect the known physiological impact of renal impairment on COPD prognosis. The modular nature of Uni-ACS allows these variables to be viewed as nodes within a clinical risk network, where the score acts as a proxy for the cumulative burden of these physiological stressors.

While this study does not prospectively measure the effectiveness of clinical scores, the informatics innovation of Uni-ACS lies in providing a “translation layer.” By converting a black-box XGBoost model into a score that can be calculated manually on a clipboard, we remove the computational and transparency barriers that currently prevent ML models from entering clinical practice. This is essential for several reasons, particularly when contrasted with other methods. First, clinical scores offer simple, consistent, reproducible risk measures for clinical practitioners and patients to understand disease risk. Second, clinical scores, as a form of quantitative risk stratification, can be validated against traditional epidemiological standards. They can be seamlessly integrated into evidence-based decision-making, such as clinical guidelines and decision support systems. Third, regulatory authorities often require risk prediction tools to be transparent and reproducible. Uni-ACS clinical scores satisfy these criteria as they can be directly interrogated, compared with existing validated risk scores and rationalized against current clinical knowledge.

Finally, the mathematical stability of the Uni-ACS score is a key informatics advantage when incorporating human expert input to select features for clinical score construction. Unlike traditional regression models where adding a non-significant variable might destabilize the coefficients of other predictors (multicollinearity), the application of SHAP values post model training allows for a modular clinical score. Therefore, clinicians can customize the feature set to align with specific clinical settings without compromising the statistical validity of the individual risk points assigned to each feature.

Limitations and future work

We acknowledge the following limitations. First, the assumption that developed clinical scores and associated odds ratios offer greater interpretability than raw model explanations (eg, SHAP values); was not empirically validated with clinicians. Future work will collect structured feedback from clinical stakeholders to directly assess whether the Uni-ACS clinical scores and odds ratios are indeed perceived as more interpretable and useful for decision-making than model-agnostic explanations.

Second, our models excluded potentially important features—including symptom scores from COPD Assessment Tools, vital signs (respiratory rate, blood pressure), arterial blood gas results, and social determinants (living arrangements, caregiver availability)—due to the challenges of extracting unstructured data from electronic health records. Future work could leverage recent advances in natural language processing to address this. For instance, LLMs could extract structured information from clinical notes, while generative AI could synthesize multi-modal data to create more comprehensive feature sets.

Additionally, we included the elderly which might have distinct risk factors compared to the general adult population. This approach was intentional to reflect our planned real-world clinical application, where this risk stratification tool can be deployed on the general patient population at point of discharge. However, we note that it may introduce survivor bias and limit the utility of the clinical score’s predictive performance and real-time inpatient decision-making capability. Therefore, future work could develop separate models for different age groups, incident and prevalent populations or incorporate time-since-diagnosis as a feature to capture disease trajectory effects.

Conclusion

This paper demonstrates that Uni-ACS addresses fundamental gaps in clinical AI translation by providing standardized, quantifiable risk measures essential for clinical workflows. From a clinical informatics perspective, Uni-ACS is highly significant because it translates complex ML model outputs into clinical scores with validated odds ratios, a familiar interface to clinicians for estimating risk, thereby facilitating easy integration into clinical workflows. As a post hoc methodology applied to trained ML models, Uni-ACS offers flexibility to integrate domain knowledge during feature selection. It democratizes access to AI-driven risk stratification across diverse healthcare environments by enabling the deployment of the full ML model or the use of the derived clinical score in resource-constrained settings. This approach bridges the gap between sophisticated AI predictions and the meaningful, reproducible risk measures that are necessary for reliable integration into clinical practice.

Author contributions

Anthony Lian Jie Li (Formal analysis, Investigation, Methodology, Resources, Software, Validation, Writing—original draft, Writing—review & editing), Moses Lim (Project administration, Validation, Writing—review & editing), Weixiang Lian (Validation, Writing—review & editing), Lin Htun Htet (Data curation, Investigation, Project administration, Validation, Writing—review & editing), Hwee Pin Phua (Writing—review & editing), Ser Hon Puah (Writing—review & editing), Geak Poh Tan (Data curation, Validation, Writing—review & editing), Huiying Xu (Validation, Writing—review & editing), John Arputan Abisheganaden (Conceptualization, Methodology, Writing—review & editing), and Wei-Yen Lim (Conceptualization, Methodology, Supervision, Writing—review & editing)

Supplementary material

Supplementary material is available at Journal of the American Medical Informatics Association online.

Funding

No grant funding was received.

Conflicts of interest

None of the authors have conflicts of interest to declare.

Data availability

The data underlying this article will be shared on reasonable request to the corresponding author.

Prior publication

An abstract of this paper was previously presented at the Singapore Health and Biomedical Congress, held in Singapore in October 2022.

Ethics review

This study was approved by the hospital ethics review board (National Healthcare Group Domain Specific Review Board Ref: 2021/00894).

Guarantor statement

The guarantor for this manuscript is Li Lianjie Anthony, who accepts full responsibility for the finished work and the conduct of the study, had access to the data, and controlled the decision to publish.

Guarantor signature: Li Lianjie Anthony LianjiAnthonyAntony