Naïve Bayes is an interpretable and predictive machine learning algorithm in predicting osteoporotic hip fracture in-hospital mortality compared to other machine learning algorithms

Abstract

Osteoporotic hip fractures (HFs) in the elderly are a pertinent issue in healthcare, particularly in developed countries such as Australia. Estimating prognosis following admission remains a key challenge. Current predictive tools require numerous patient input features including those unavailable early in admission. Moreover, attempts to explain machine learning [ML]-based predictions are lacking. Seven ML prognostication models were developed to predict in-hospital mortality following minimal trauma HF in those aged ≥ 65 years of age, requiring only sociodemographic and comorbidity data as input. Hyperparameter tuning was performed via fractional factorial design of experiments combined with grid search; models were evaluated with 5-fold cross-validation and area under the receiver operating characteristic curve (AUROC). For explainability, ML models were directly interpreted as well as analysed with SHAP values. Top performing models were random forests, naïve Bayes [NB], extreme gradient boosting, and logistic regression (AUROCs ranging 0.682–0.696, p>0.05). Interpretation of models found the most important features were chronic kidney disease, cardiovascular comorbidities and markers of bone metabolism; NB also offers direct intuitive interpretation. Overall, NB has much potential as an algorithm, due to its simplicity and interpretability whilst maintaining competitive predictive performance.

Author summary

Osteoporotic hip fractures are a critical health issue in developed countries. Preventative measures have ameliorated this issue somewhat, but the problem is expected to remain in main due to the aging population. Moreover, the mortality rate of patients in-hospital remains unacceptably high, with estimates ranging from 5–10%. Thus, a risk stratification tool would play a critical role in optimizing care by facilitating the identification of the susceptible elderly in the community for prevention measures and the prioritisation of such patients early during their hospital admission. Unfortunately, such a tool has thus far remained elusive, despite forays into relatively exotic algorithms in machine learning. There are three major drawbacks (1) most tools all rely on information typically unavailable in the community and early during admission (for example, intra-operative data), limiting their potential use in practice, (2) few studies compare their trained models with other potential algorithms and (3) machine learning models are commonly cited as being ‘black boxes’ and uninterpretable. Here it is shown that a Naïve Bayes model, trained using only sociodemographic and comorbidity data of patients, performs on par with the more popular methods lauded in literature. The model is interpretable through direct analysis; the comorbidities of chronic kidney disease, cardiovascular, and bone metabolism were identified as being important features contributing to the likelihood of deaths. An algorithm-agnostic approach to machine learning model interpretation is also shown. This study shows the potential for Naïve Bayes in predicting elderly patients at risk of death during an admission for hip fracture.

1. Introduction

The osteoporotic hip fracture (HF) is a global issue with an estimated financial burden of 17 billion USD for the United States in 2002 and projected burden of £3.62 in 2023 for the United Kingdom [1,2]. Estimates for short-term (in-hospital) mortality following HF have been placed in the vicinity of 2–10%, with an estimated mortality rate of 2.7% for HF hospitalisation in Australia [3–5]. In developed countries, though preventative measures (targeting reduction of hip fracture risk factors such as osteoporosis and falls) have reduced the age-standardized incidence rate of hip fractures, the absolute rate is increasing due to the ageing population [6]. In Australia, for instance, hospitalisations for HF in the elderly increased by almost 20% between 2006–07 and 2015–16 from 15 900 to 18 700 respectively [5]. With the trend towards an aged population expected to continue, including in Australia, HFs in the elderly will remain a relevant, and increasingly pressing challenge in healthcare.

One key aspect in the management of HF is the prognostication of poor short-term outcomes. There exists a substantial amount of analysis from traditional statistical methods (such as logistical regression, LR) in identifying key risk factors for predicting poor outcomes, notably mortality, following HF and scoring tools that have risen to prominence are the Nottingham Hip Fracture Score (NHFS) and the orthopaedic- Physiological and Operative Severity Score for the enUmeration of Mortality and Morbidity (O-POSSUM) [7–10]. Most of these tools require a combination of both clinical, laboratory and intra-operative data; and the lack of laboratory and intra-operative data early during admission limits the use of such tools in early risk stratification.

Non-traditional mathematical algorithms, especially those associated with artificial intelligence (AI) and machine learning (ML), have become increasingly utilized in healthcare. A variety of ML algorithms, including regression-based methods, decision-tree based methods (i.e. decision trees [DT], Random Forests [RF], eXtreme Gradient Boosting [XGB] implementation), neural networks (NN), Naïve Bayes and support vector machines (SVM) have been used in the prognostication of patients in the general peri-operative [11–16] and peri-HF [17–20] period with varying degrees of success. However, most of these tools require data that are not readily available on admission (such as intra-operative data and laboratory data), much like the tools developed from traditional statistical methods, and most do not predict short-term in-hospital mortality following HF.

Moreover, there is a scarcity of studies that developed multiple different machine learning algorithms in HF prediction, and compared the end-results with one another. No study has trained and compared multiple machine learning algorithms for prediction of short-term outcomes (e.g. in-hospital mortality) following HFs. Indeed, even in task of long-term prognostication, only one study attempted to train and compare different classes of algorithms (SVM, NB and LR) in the in their ability to predict 1-year mortality post-HF [17]. Tree-based methods have received the majority of attention. An algorithm that has remarkable potential is Naïve Bayes, which is based on Bayes theorem, with the additional ‘naïve’ assumption that features are conditionally independent. It has been applied successfully across a wide variety of tasks in natural language processing (e.g. detection of spam email [21], text sentiment analysis, text/document classification) as well as in the medical field (e.g. the prognostication in cirrhotic patients following transjugular intrahepatic portosystemic shunt [22], prediction of 30-day mortality following HF [23], prediction of osteonecrosis of femoral head with cannulated screw fixation [24] and prediction of mortality in post-surgical intensive care unit patients [25]).

While predictive ability is an important characteristic of any prognostic tool, it is increasingly recognized that a desirable attribute of machine learning algorithm is that they are interpretable (or ‘explainable’) especially as ML models become increasingly complex [26,27]. Recognition of this issue has led to the development of the subfield of ‘interpretable’ ML and, in particular, the development and application of the SHapley Additive exPlanations (SHAP), an approach based on cooperative game theory [28–34].

The goal, was to train multiple ML models, specifically Bernoulli Naïve Bayes (NB), DT, RF, XGB, SVM, logistic regression (LR) and the multi-layer perceptron (MLP, a 3-layer NN) to predict in-hospital mortality for the elderly admitted with HF. The focus of this study is on using only those patient features that are readily available in the early phases during a hospital admission, i.e. sociodemographic and comorbidity data. The performances of each model would be compared to identify the most predictive algorithm. Finally, each predictive tool would be analysed via direct interpretation of model and with calculation of SHAP values.

2. Results

2.1. Patient cohort characteristics

Of the 3625 patients in the cohort, age was distributed non-normally with median age of 84 (interquartile range of 10 years) and females comprising 2730 (75.3%); 189 (5.2%) had in-hospital mortality. The most common comorbidity was hypertension (HTN, at 2045 [56.4%]). Details are present in Table 1 (with abbreviations defined below).

Table 1. Sociodemographic features, outcomes of HF cohort.

Variable	Total Cohort (N = 3625)	Female (N = 2730, 75.31%)	Male (N = 895, 24.69%)	p value(1)
Sociodemographic features
Age (median [IQR])	84 [10]	85 [10]	82 [12]	<0.001
Aged > 80 years (n,%)	2457, 67.8%	1937, 71.0%	520, 58.1%	<0.001
PRCF resident (n,%)	1208, 33.3%	950, 34.8%	258, 28.8%	0.001
Smoker (n,%)	180, 5.0%	124, 4.5%	56, 6.3%	0.050
Alcohol overuse (n,%) ( 2 )	144, 4.0%	60, 2.2%	84, 9.4%	<0.001
Walking aids user (n,%)	1300, 35.9%	999, 36.6%	301, 33.6%	0.116
Comorbidities features
HTN, (n,%)	2045, 56.4%	1606, 58.8%	439, 49.1%	<0.001
Anaemia (n,%)	1531, 42.2%	1051, 38.5%	480, 53.6%	<0.001
CKD (n,%)	1444, 39.9%	1106, 40.5%	338, 37.8%	0.152
Dementia (n,%)	1117, 30.8%	858, 31.4%	259, 28.9%	0.172
CAD (n,%)	1073, 29.6%	750, 27.5%	323, 36.1%	<0.001
History of AMI (n,%)	287, 7.9%	191, 7.0%	96, 10.7%	<0.001
AF (n,%)	702, 19.4%	513, 18.8%	189, 21.1%	0.139
COPD (n,%)	561, 15.5%	385, 14.1%	176, 19.7%	<0.001
T2DM (n,%)	482, 13.3%	325, 11.9%	157, 17.5%	<0.001
OP (n,%)	478, 13.2%	410, 15.0%	68, 7.6%	<0.001
CVA (n,%)	431, 11.9%	323, 11.8%	108, 12.1%	0.897
TIA (n,%)	309, 8.5%	227, 8.3%	82, 9.2%	0.474
PD (n,%)	172, 4.7%	97, 3.6%	75, 8.4%	<0.001
Malignancy (n,%)	82, 2.3%	52, 1.9%	30, 3.4%	0.017
PTH>6.8pmol/L	1684, 46.5%	1275, 46.7%	409, 45.7%	0.628
25(OH)vitamin D≤25nmol/L	610, 16.8%	467, 17.1%	143, 16.0%	0.464
25(OH)vitamin D≤50nmol/L	1659, 45.8%	1235, 45.2%	424, 47.4%	0.283
Outcome
Died (n,%)	189, 5.2%	130, 4.8%	59, 6.6%	0.040

¹Pearson’s Chi-squared test (Yates corrected).

²Use>3 times a week.

Abbreviations: PRCF, permanent residential care facility; HTN, hypertension; CKD, chronic kidney disease; CAD, coronary artery disease; AMI, acute myocardial infarction; AF, atrial fibrillation; COPD, chronic obstructive pulmonary disease; T2DM, type 2 diabetes mellitus; OP, osteoporosis; CVA, cerebrovascular accident; TIA, transient ischaemic attack; PD, Parkinson’s disease; PTH, parathyroid hormone.

2.2. Model performance–training

The model with the highest area under the receiver operating characteristic (AUROC) was MLP (AUROC 0.828) followed by LR, RF, XGB and NB (0.733, 0.730, 0.726 and 0.725 respectively, all p>0.05), then DT (AUROC of 0.697) and finally SVM (AUROC 0.533).

The model with greatest area under the precision-recall curve (AUPRC) was MLP (AUPRC 0.245), followed by LR, XGB and RF (AUPRCs of 0.134, 0.133 and 0.130 respectively, p>0.05), NB (AUPRC 0.124), DT (AUPRC of 0.094) and finally SVM (AUPRC of 0.058). Details are present in Table 2 and Table 3.

Table 2. Model performance (training phase).

	AUROC			AUPRC
	Mean	STD	95%CI	Mean	STD	95%CI
SVM	0.533	0.029	0.475–0.591	0.058	0.004	0.050–0.067
NB	0.725	0.007	0.711–0.739	0.124	0.003	0.117–0.131
LR	0.733	0.008	0.717–0.750	0.134	0.004	0.127–0.141
DT	0.697	0.004	0.690–0.704	0.094	0.001	0.093–0.095
RF	0.730	0.007	0.716–0.745	0.130	0.003	0.125–0.136
XGB	0.726	0.007	0.711–0.741	0.133	0.005	0.122–0.144
MLP	0.828	0.008	0.813–0.844	0.245	0.030	0.186–0.305

Table 3. Comparison of model performance during training.

(A)–AUROC (B)–AUPRC.

(A) AUROC
t-test statistic								t-test p-value
models	SVM	NB	LR	DT	RF	XGB	MLP	SVM	NB	LR	DT	RF	XGB	MLP
SVM	-	-14.391	-14.866	-12.527	-14.766	-14.466	-21.927	-	0.000	0.000	0.000	0.000	0.000	0.000
NB	-	-	-1.683	7.766	-1.129	-0.226	-21.666	-	-	0.131	0.000	0.291	0.827	0.000
LR	-	-	-	9.000	0.631	1.472	-18.776	-	-	-	0.000	0.546	0.179	0.000
DT	-	-	-	-	-9.153	-8.043	-32.750	-	-	-	-	0.000	0.000	0.000
RF	-	-	-	-	-	0.904	-20.614	-	-	-	-	-	0.393	0.000
XGB	-	-	-	-	-	-	-21.456	-	-	-	-	-	-	0.000
MLP	-	-	-	-	-	-	-	-	-	-	-	-	-	-
(B) AUPRC
t-test statistic								t-test p-value
models	SVM	NB	LR	DT	RF	XGB	MLP	SVM	NB	LR	DT	RF	XGB	MLP
SVM	-	-29.516	-30.042	-19.524	-40.249	-26.191	-13.816	-	0.000	0.000	0.000	0.000	0.000	0.000
NB	-	-	-4.472	21.213	-4.472	-3.451	-8.974	-	-	0.002	0.000	0.002	0.009	0.000
LR	-	-	-	21.693	2.236	0.349	-8.201	-	-	-	0.000	0.056	0.736	0.000
DT	-	-	-	-	-80.498	-17.103	-11.249	-	-	-	-	0.000	0.000	0.000
RF	-	-	-	-	-	-1.342	-8.572	-	-	-	-	-	0.217	0.000
XGB	-	-	-	-	-	-	-8.234	-	-	-	-	-	-	0.000
MLP	-	-	-	-	-	-	-	-	-	-	-	-	-	-

2.3. Model performance–test

The models with highest AUROC were RF, NB, XGB and LR (AUROCs of 0.696, 0.694, 0.689 and 0.682 respectively, all p>0.05;) followed by MLP and DT (AUROCs of 0.618, 0.616 respectively, p>0.05) and finally SVM (AUROC of 0.499).

The model(s) with highest AUPRC were NB, XGB, RF and LR (AUPRCs of 0.113, 0.113, 0.112, 0.112 respectively, p>0.05), MLP and DT (AUPRCs of 0.077, 0.070 respectively, p>0.05), and SVM (AUPRC of 0.054)–see Table 4, Table 5 and S1 Fig.

Table 4. Model performance during testing (5-fold cross-validation).

	AUROC			AUPRC—Average precision
	Mean	STD	95%CI	Mean	STD	95%CI
SVM	0.499	0.048	0.404–0.594	0.054	0.006	0.041–0.067
NB	0.694	0.024	0.646–0.742	0.113	0.019	0.076–0.151
LR	0.682	0.034	0.614–0.750	0.112	0.019	0.074–0.150
DT	0.616	0.028	0.560–0.671	0.070	0.006	0.058–0.081
RF	0.696	0.030	0.636–0.756	0.112	0.024	0.064–0.161
XGB	0.689	0.025	0.640–0.738	0.113	0.024	0.065–0.160
MLP	0.618	0.027	0.565–0.671	0.077	0.007	0.063–0.090

Table 5. Comparison of model performance on testing.

(A)–AUROC (B)–AUPRC.

(A) AUROC
t-test statistic									t-test p-value
models	SVM	NB	LR	DT	RF	XGB	MLP	SVM	NB	LR	DT	RF	XGB	MLP
SVM	-	-8.125	-6.957	-4.708	-7.782	-7.850	-4.832	-	0.000	0.000	0.002	0.000	0.000	0.001
NB	-	-	0.645	4.729	-0.116	0.323	4.704	-	-	0.537	0.001	0.910	0.755	0.002
LR	-	-	-	3.351	-0.690	-0.371	3.296	-	-	-	0.010	0.509	0.720	0.011
DT	-	-	-	-	-4.359	-4.349	-0.115	-	-	-	-	0.002	0.002	0.911
RF	-	-	-	-	-	0.401	4.321	-	-	-	-	-	0.699	0.003
XGB	-	-	-	-	-	-	4.315	-	-	-	-	-	-	0.003
MLP	-	-	-	-	-	-	-	-	-	-	-	-	-	-
(B) AUPRC
t-test statistic									t-test p-value
models	SVM	NB	LR	DT	RF	XGB	MLP	SVM	NB	LR	DT	RF	XGB	MLP
SVM	-	-6.621	-6.509	-4.216	-5.242	-5.333	-5.578	-	0.000	0.002	0.003	0.001	0.001	0.001
NB	-	-	0.083	4.826	0.073	0.000	3.976	-	-	0.936	0.001	0.944	1.000	0.004
LR	-	-	-	4.713	0.000	-0.073	3.865	-	-	-	0.002	1.000	0.944	0.005
DT	-	-	-	-	-3.796	-3.887	-1.698	-	-	-	-	0.005	0.005	0.128
RF	-	-	-	-	-	-0.066	3.130	-	-	-	-	-	0.949	0.014
XGB	-	-	-	-	-	-	3.220	-	-	-	-	-	-	0.012
MLP	-	-	-	-	-	-	-	-	-	-	-	-	-	-

2.4. Feature importance–Model interpretation

Feature importance rankings (1 being the most important) according to each model can be found in Table C in S3 Appendix. Corresponding coefficients for NB, LR, XGB and RF can be found in Fig 1.

Fig 1. Feature importance based on model interpretation.

The presence of CKD, cardiovascular comorbidities (either CAD or AF),deranged markers of bone metabolism (PTH>6.8pmol/Land vitamin D≤25nmol/L) and advanced age contributed the greatest amount to predictions across models.

For the LR model, the 5 most important patient features in prediction of mortality were presence of CKD, vitamin D deficiency (≤25nmol/L), advanced age (>80 years), COPD, and AF. In the SVM model the 5 most important patients features in prediction of mortality were advanced age (>80 years), CKD, vitamin D insufficiency (≤50nmol/L), anaemia, and use of walking aids. For the NB model, the 5 most important features in mortality prediction were history of MI, AF, CKD and CAD. For the DT model the 5 most important features in mortality prediction were presence of CKD, hyperparathyroidism (PTH>6.8pmol/L), CAD, dementia and advanced age (>80 years). For the RF model the 5 most important features in mortality prediction were CKD, hyperparathyroidism (PTH>6.8pmol/L), CAD, dementia and advanced age (>80 years). For the XGB model the 5 most important features in mortality prediction were CKD, CAD, advanced age (>80 years), PTH>6.8pmol/L and AF. Finally, for the MLP model the 5 most important features in mortality prediction were AF, CKD, male sex, dementia, and MI.

2.5. Feature importance–SHAP analysis

Features were also ranked by the mean absolute SHAP values (Figs 2–7).

Fig 2. Summary plot SHAP values for patient comorbidities–Logistic Regression.

Each point on the plot represents a SHAP value for an individual patient’s comorbidity (SHAP value is on x-axis, corresponding comorbidity on the y-axis. Positive SHAP values corresponds to a positive/additive contribution to the prediction (i.e. in-hospital mortality); conversely a negative SHAP value corresponds to a negative/subtractive contribution. Colours of points represents feature values: magenta/red corresponded to a value of ‘1’ (i.e. presence of the comorbidity) and blue corresponding to value ‘0’ (i.e. absence of comorbidity).

Fig 7. Summary plot SHAP values for patient comorbidities–Multi-Layer Perceptron.

Fig 3. Summary plot SHAP values for patient comorbidities–Naïve Bayes.

Fig 4. Summary plot SHAP values for patient comorbidities–Decision Tree.

Fig 5. Summary plot SHAP values for patient comorbidities–Random Forest.

Fig 6. Summary plot SHAP values for patient comorbidities–Extreme Gradient Boosting.

For the LR model, the 5 most predictive patient features for mortality in order from highest magnitude to lowest, based on mean SHAP values, were CKD, advanced age (>80 years), hyperparathyroidism (PTH>6.8pmol/L), CAD, and residency from PRCF. Absence of any of these features had a negative SHAP value (i.e. a negative contribution) on the model outcome (in-hospital mortality); the magnitude of this impact was consistent across all patients. Likewise, the presence of any of these features always had a positive SHAP value (i.e. an additive contribution) on in-hospital mortality. The magnitude of this effect was again consistent across all patients.

For the NB model, the 5 most predictive features were CKD, AF, MI, residency from PRCF, CAD. Again, absence of any of these features most commonly had a negative impact on in-hospital mortality; the magnitude of this effect varied among patients. The presence of any of the above 5 features had a positive contribution to the prediction of in-hospital mortality; similarly, the magnitude of this effect varied significantly among patients.

For the DT model, the 5 most predictive features were CKD, hyperparathyroidism (PTH>6.8pmol/L), advanced age (>80 years), presence of CAD and vitamin D deficiency (≤25nmol/L). Presence of these five comorbidities had a positive contribution to prediction of in-hospital mortality and, conversely their absence had a negative contributory effect on prediction. Interestingly, absence of T2DM had an additive effect and presence of T2DM had a negative effect on mortality prediction. The magnitude of contributions that each of the 5 variables had varied among different patients. Finally, it is noteworthy that all other comorbidities had little to no influence on patient outcomes.

For the RF model, the 5 most predictive features were CKD, hyperparathyroidism (PTH>6.8pmol/L), CAD, advanced age (>80 years) and residence from PRCF. The presence of these features increased likelihood of mortality and conversely absence decreased the likelihood of mortality; there was only a minor variation of contribution from each feature for each patient.

For the XGB model, the 5 most predictive patient features were advanced age (>80 years), vitamin D deficiency (≤25nmol/L), CKD, CAD and hyperparathyroidism (PTH>6.8pmol/L). The presence (and absence) of any of these features increased (or decreased) the likelihood of mortality. For each feature, there was only mild variation in the magnitude of contributions among patients.

Finally, from the MLP model, the 5 most predictive patient features were male sex, advanced age (>80 years), CVA, HTN and TIA. The presence (or, conversely, the absence) of any of these features except for HTN were associated with an increased (decreased) likelihood of mortality; presence (or absence) of HTN appeared to decrease (increase) the likelihood of mortality.

Across all models, the 5 comorbidities most consistently with the greatest influence on mortality prediction were: CKD, advanced age (>80 years), elevated PTH (>6.8pmol/L), cardiovascular disease (CAD, MI, AF or HTN) and PRCF residence.

3. Discussion

Seven ML models were derived to predict in-hospital mortality for hospitalized elderly minimal trauma HF patients using only categorical data and their performances compared. Overall, the models had reasonable to good performance. An analysis of each model and application SHAP analysis was also performed to gain insight into feature importance.

3.1. Model performance–training and test

Notably, but unsurprisingly, classification performance showed some variation among algorithms. The trained models, ordered in decreasing performance (based on both test AUROCs and test AUPRCs), were RF, NB, XGB and LR (all with no statistically significant difference in performance–see Table 4, Table 5 and S1 Fig) followed by MLP and DT (no statistically significant difference in performance) and finally SVM. AUROCs ranged from 0.500 (SVM) to almost 0.700 (good performance), while AUPRC values ranged from 0.050 (SVM) to 0.115; a reflection of using a simplified model (with binary input data) to perform predictions on a minority class in this imbalanced dataset. There was minimal difference between the training and cross-validation performance for the top 4 models (RF, NB, XGB and LR). A greater variation in training and cross-validation performance scores was noted for DT and MLP, an indicator of overtraining (an infamous tendency in machine learning). That overtraining has occurred despite systematic and meticulous hyperparameter tuning, is strongly suggestive of insufficient data.

To the investigator’s knowledge, most studies have focused on only training and applying one class of machine learning algorithm. Often there is no baseline model trained using traditional statistics (e.g. LR). Indeed, most studies have solely utilized tree-based methods (e.g. applying DT, XGB and RF methods) and this is reflected in a scoping study of ML usage in health economics and research (on 805 studies) which found the most frequent algorithms used were tree-based methods followed by regression-based (linear/logistic) methods, SVM, NN and finally NB [35]. However, it is known that performance on various tasks varies with different ML algorithms[36] and the finding that predictive performance varies among machine learning algorithms (for the same problem, using the same data) is consistent with this. It is thus ideal that in future applications of machine learning, a more comprehensive set of algorithms are trained, or some justification should be provided, if possible, when certain algorithms are not included.

The performance of NB in predicting mortality is on par with RF, XGB and LR which warrants further discussion here as it has received relatively little attention in the literature. Key to its success is the simplifying assumption of conditional independence among all patient input features. The most obvious advantage from this is that, by virtue of such a simplification, it is computationally inexpensive and is fast to train and run. However, with such a large, seemingly excessive, assumption (that is not strictly satisfied in the current database, with interdependent features such as vitamin D insufficiency and deficiency), it may seem surprising that this model performs so well. Contrary to intuition, its good performance is not a coincidental or even unexpected phenomenon; formal analysis of NBs has established it performs well because the interdependencies, when they do exist, occur in a manner which results in them ‘cancel[ling] each other out’ and impact only probability estimates, not overall classification performance [37].

3.2. Feature importance–model interpretation and SHAP analysis

Rankings of patient comorbidity importance in their role in mortality prediction were determined from all models from direct interpretation of feature coefficients (see Fig 1 and Table C in S3 Appendix). CKD was most consistently ranked as one of the 5 most important patient comorbidities in predicting mortality. The other most important patient features included markers reflective of bone metabolism (PTH, vitamin D levels) and cardiovascular disease (presence of either one of CAD, MI, AF). Similar trends were found via SHAP value analyses for each model, i.e. CKD, bone metabolism markers and presence of cardiovascular diseases had the strongest influence on prediction of mortality based on mean SHAP values (Figs 2–7). It is recognized in the literature that cardiovascular comorbidities and renal function are important for prognostication which is reflected in their inclusion as input parameters for non-cardiac surgery risk assessment tools such as the Revised Cardiac Risk Index and the American College of Surgeons—surgical risk calculator [38–42]. However, these features are not explicitly included in HF-specific risk assessment tools (e.g. in O-POSSUM only symptoms and clinical findings suggestive of cardiovascular disease are included and NHFS only the number of comorbidities is included as an input parameter) [7–10]. Moreover, neither PTH or vitamin D levels are included in any of the current tools, despite an increasing number of studies supporting the key role they play in bone metabolism and prevention of fracture [43–55] and, potentially, with increasing recognition of their importance in the immunity [56–58] prevention of post-operative complications such as hospital acquired infections.

3.3. Further insights from model analysis

Of the four most predictive models, NB and LR models offer intuitive, quantifiable insights into feature contributions to prediction: in LR, the odds ratio can be taken by calculating the exponent of the coefficients, while in NB, from the method of scoring input features (see S1 Appendix), each coefficient corresponds to the ratio of the rate of the comorbidity in those who experienced in-hospital mortality compared to the comorbidity rate in those who survived. So, for example, in predicting mortality, one can see from the LR model that CAD, with a score of 0.319 (95%CI 0.180–0.458) increased mortality risk by 37% (OR 1.37; 95%CI 1.20–1.58) and CKD, with a score of 0.711 (95%CI 0.505–0.918) increased mortality risk by 2.03 (95%CI 1.66–2.50). From the NB model, a score of 1.62 (for CAD), and a score of 1.67 (for CKD) indicated that the rate of each comorbidity was greater in mortality than in survival by 62% and 67% respectively.

For the other top predictive models, insights gained from direct interpretation of RF and XGB is not so straightforward. Both these methods are based on DTs, which is itself an interpretable and intuitive model. However, a major drawback of DTs is that they are very prone to bias and variance (overfitting). RF and XGB address this issue by constructing multiple DTs and the overall prediction is then made from an ensemble/collection of multiple trees (numbering in the hundreds) and, hence, increased predictive performance is obtained at the expense of interpretability. In this study, the coefficients for each feature correspond to the relatively abstract concept of mean decrease in (Gini) impurity (see S1 Appendix).

3.4. Further insights from SHAP values

SHAP values revealed that the presence of more ‘severe’ comorbidities in each ML model had a more important additive effect on mortality risk than less severe comorbidities, as one might expect. For instance, patients with a history of acute MI (a higher severity sub cohort of CAD patients) typically had the greatest SHAP value indicating that the presence of history of past MI had the greatest additive effect on mortality prediction. Similarly, the presence of vitamin D deficiency (≤25nmol/L) was correlated with greater SHAP values compared to vitamin D insufficiency (≤50nmol/L) (see Figs 2–7). In contrast, the absence of both MI and vitamin D deficiency in patients had less of a negative effect on mortality prediction compared to the other comorbidities (hence explaining their lower overall importance based on mean SHAP values). This likely is a reflection of their relatively low prevalence in the cohort (Table 1).

For LR, RF, and XGB the SHAP values had low variability and were highly concentrated–an indication that the corresponding input patient features were consistently strong contributors to mortality prediction; a corollary of this was that these models offered good population level insight into mortality risk. Of the top four models, NB was the only model in which the SHAP values themselves varied among individuals. This variability in SHAP values among patients suggested that the influence of each singular comorbidity was not constant, and that each prediction appeared to be tailored toward individual.

3.5. Clinical implications

This NB model could direct individualised HF prevention measures for patients in the community. Currently there exist osteoporotic fracture risk assessment tools, such as the FRAX and the Garvan tools [59–62], however, a reliable method to identify those at a high risk of in-hospital mortality remains elusive. A quantifiable and objective prognostic estimate of mortality risk following HF would guide clinicians in, firstly, triaging referrals to falls prevention clinics and, secondly, objectively appraise the need to commence anti-osteoporotic agents (based on in-hospital mortality risk following HF) against the risk of drug-related adverse effects [63,64]. The tool can also aid in identifying patients at risk of mortality early during their admission. It has been established that early surgical intervention reduces mortality risk in elderly HF patient admissions [65–67]. The model could be used by clinicians to assist in prioritisation of surgical intervention for those HF patients identified as high-risk for in-hospital mortality. By excluding the need for laboratory parameters, the ML model can be used by emergency physicians, orthopaedic surgeons to prioritise surgery for patients classified as high-risk with minimal delay. It can also prompt clinicians to set the expectations of patients and next of kin in the early phases of admission. Early discussions and clear communication with patients and family are a key element of clinical care and would facilitate better preparedness for end-of-life care in the event of rapid in-hospital deterioration, minimizing miscommunication, dissatisfaction and bereavement while maximising quality of life [68–70].

That the developed model does not depend on laboratory or intra-operative data confers a significant advantage over both the NHFS and O-POSSUM–it can be applied much earlier than either. The variables required for the calculation of NHFS are: age (<66, 66–85, ≥86 years), sex, admission haemoglobin, mini-mental test score, living in an institution, number of comorbidities (≥ 2), and presence of active malignancy [7,8]. The variables required for the calculation of the O-POSSUM are: age (in years), chest radiograph findings, respiratory symptoms, cardiac signs, vital signs (systolic blood pressure in mmHg, pulse in beats/min), Glasgow Coma Scale, full blood count (haemoglobin, white cell count), electrolytes (sodium, potassium, urea), electrocardiogram findings, operative severity, multiple procedures, total blood loss, peritoneal soiling, presence of malignancy, mode of surgery (emergent vs elective) [10]. For the NHFS, haemoglobin on admission is not immediately available, and the mini-mental test assessment, while possible to perform at the bedside, is not routinely performed as part of the initial assessment for a HF admission. For the O-POSSUM score, not only does it require laboratory test results (electrolytes, full blood count), but the status of certain features (i.e. respiratory symptoms, cardiac signs, chest radiograph, electrocardiogram) are victim to subjective (clinician dependent) interpretation and, finally, intraoperative data is required for the O-POSSUM model which precludes its potential use in the outpatient setting and early phases of hospital admission. While the NB model can clearly be applied in a more timely manner than either NHFS or O-POSSUM, a comparison of their discriminatory performances was not possible in this study as, due to limitations on the dataset, it was not possible to calculate the corresponding patient scores.

The results from the analysis of feature importance (both SHAP and direct approach) can also be used to guide treatment in the context of osteoporotic HF. Certain comorbidities, such as CKD, have been found in this study to have large contributions to patient mortality prediction across trained ML models. These findings could advance current clinical practice, if validated externally; they support early implementation of HF prevention strategies as standard of care for patients with these select, highly influential, comorbidities. On a case-by-case basis, SHAP values also offer local explanations, i.e. explanations specific to individualised patients and predictions, enabling stakeholders (primarily patients and their clinicians) to make informed decisions, expose underlying vulnerabilities and protect individuals from the potential pitfalls of automated decisions.

3.6. Limitations and future work

Internal nested cross-validation, though relatively rigorous, is no substitute for external validation. The initial step toward this would be temporal validation; patients meeting the same criteria as those defined in the ‘Methods–Data Collection’ section will have data collected in the period following 2019 and this data will be used to validate the developed algorithm. Following temporal validation, the aim would be to validate on a wider geographic region (inter-hospital, to inter-state and potentially international cohorts). Following rigorous validation, there remains the challenge of model dissemination and integration into clinical practice. A possible approach would be to implement the model into a web-based application making it readily accessible to any healthcare provider with internet access, though such an approach could result in improper use of the algorithm on non-validated populations. Alternatively, the model could be integrated into commercial Electronic Health Records used by hospitals, though this would be at the expense of limiting users to those with access to specific (proprietary) software.

It should also be noted that this was a study on a retrospective cohort, with members recruited from a single-centre. Though the dataset used here is not unreasonably small, it must be acknowledged that it may still be insufficient: firstly, because of the overfitting noted in MLP models and secondly because of the imbalance inherent to the dataset with a 5% mortality rate. With only 189 cases, the mortality population may be under-represented from a machine-learning perspective (which typically requires cohort sizes numbering in the 1000s or greater to be trained effectively). Moreover, inaccurate reports (probable under-reporting) on smoking and drinking habits by patients may bias findings. Furthermore, analysis and model derivation has been conducted using only categorical features which may negatively impact predictive ability. Model predictions were not calibrated, and it is known that certain machine learning models, particularly NB are notoriously poor at estimating probabilities despite being good classifiers.

3.7. Conclusion and final comments

In summary, NB was the most optimal model having the optimal virtues of strong predictive performance, model interpretability and potential for making individualized predictions. While RF, XGB and LR had similar performance capabilities, by nature they are not readily interpretable (i.e. RF and XGB) or are not optimal for individualized predictions (i.e. LR).

With ongoing development of digital infrastructure in the healthcare industry it is inevitable that machine learning algorithms will only become increasingly powerful and commonplace. As we await this reality, it is to be hoped that the findings here will provide physicians and clinicians with a tool that can be used to rapidly identify patients at higher risk of mortality early by knowledge of patient comorbidities; currently most prognostication tools can only be applied later in the admission. Moreover, hopefully this study provides valuable insights in applying ML models in healthcare for clinicians and researchers, in particular the advantages of the computationally inexpensive NB models highlighting its simplicity and interpretability with negligible compromise in performance.

4. Materials and methods

4.1. Ethics statement

The study was conducted in accordance with the Declaration of Helsinki (1964) and the Council for International Organisations of Medical Sciences International Ethic Guidelines and approved by the Australian Capital Territory Human Research Ethics Committee on the 31^st May 2023 (reference number: 2023.LRE.00063). Patients’ written informed consent was waived because analysis was performed on a digital anonymised database.

4.2. Data collection

Our cohort comprised 3625 elderly (i.e. aged ≥ 65 years of age) patients consecutively admitted to the Department of Orthopaedic Surgery at the Canberra Hospital between 1999–2019 with osteoporotic hip fracture. Patients admitted with hip fracture secondary to moderate-high energy trauma, or secondary to minimal trauma but with malignancy associated pathological fracture were excluded. Data on in-hospital mortality, sociodemographic features (age, sex, smoking status, active history of overuse of alcohol, use of walking aids, and if the patient was a resident of an permanent residential care facility [PRCF]) and comorbidities (presence of hypertension [HTN], coronary artery disease [CAD], previous history of acute myocardial infarction [MI], atrial fibrillation [AF], past history of stroke [cerebrovascular accident, CVA], transient ischaemic attack [TIA], dementia, Parkinson’s disease [PD], chronic obstructive pulmonary disease [COPD], type 2 diabetes mellitus [T2DM], chronic kidney disease [CKD], anaemia, history of solid organ malignancy, osteoporosis and hyperparathyroidism [parathyroid hormone/PTH>6.8pmol/L] and vitamin D insufficiency/deficiency; (25)OH vitamin D ≤ 50/25nmol/L) were collected. Binary variables were assumed to follow a Bernoulli distribution. For parathyroid hormone, the upper limit of the laboratory reference range (6.8 pmol/L) was chosen as the cutoff. The definition of vitamin D insufficiency and deficiency was based on those utilised previously in the literature [71,72].

The continuous variable of age was tested for normality (via visualisation using histogram, the Kolmogorov-Smirnov test, Shapiro-Wilke test and the Anderson-Darling test). Results suggested a non-normal distribution–hence median and interquartile ranges were used as measures of central tendency and spread respectively. The age cutoff was obtained by taking the median age of the cohort (84 years) and rounding to the nearest decade to 80 years.

There were few instances of missing data (see Table B in S3 Appendix). For this reason, patients with missing data were omitted from analysis.

4.3. Model development

Seven ML algorithms (LR, SVM, NB, DT, RF, XGB and MLP) were trained to predict mortality. For each of the algorithms, model selection and evaluation were performed using nested cross-validation. For each iteration of k-fold cross validation, data was shuffled and split into k stratified subsets (i.e. class proportions for mortality were maintained across all data partitions); this was performed due to the significant imbalance in the dataset. The same random seed was used for data shuffling in the development of each ML model.

Model selection was performed on the inner, 3-fold, cross-validation loop; a grid search on key hyperparameters (identified using an approach based on the fractional factorial design of experiments) was used to identify the optimal hyperparameter configuration using the mean area under the receiver operating characteristic (AUROC) on the validation set as the performance criteria. Computations were performed using the Python packages, sklearn and pandas [73,74].

4.4. Model performance (and comparisons)

Model performance was evaluated on the outer (5-fold) cross-validation loop using the mean of the validation AUROCs. The student t-test was used to compare mean AUROCs of different ML models against one other. Additionally, given the imbalance to the dataset, the mean area under the precision-recall curve (AUPRC) for performance on the validation set was calculated for each trained ML. Computations were performed using the Python package SciPy (in particular ‘scipy.stats’ routines) [75].

4.5. Feature importance–model interpretation

Each trained model was analysed directly. In general, the training of each model involved optimization of coefficients corresponding to each patient feature (comorbidity). The trained models were analysed; for each patient comorbidity a corresponding coefficient or score was computed (see S1 Appendix). Features were ranked by importance based on the values of these scores.

4.6. Feature importance–SHAP analysis

For each patient, the SHAP value allocates a quantifiable credit to each variable (i.e. patient comorbidity) in its contribution to the model output (i.e. the final prediction). Feature importance analysis with SHAP was performed using the Python implementation–further details can be found in S2 Appendix [30,33]. Features were ranked based on the mean SHAP values for each comorbidity.

Supporting information

S1 Fig. ML model test performance (area under the receiver operating characteristic, AUROC).

Only the test set AUCs evaluated from the 5-fold cross-validation for the four best-performing ML models are shown.

(PPTX)

S1 Appendix. Coefficient analysis.

(DOCX)

S2 Appendix. SHAP analysis.

(DOCX)

S3 Appendix. Supplementary Data.

(DOCX)

Data Availability

The data used for analysis in this study is available to researchers by written request to the approving Australian Capital Territory Human Research Ethics Committee at ethics@act.gov.au (ethics reference number: 2023.LRE.00063). Unfortunately, data cannot be made available on a public repository due to the conditions of approval set by the approving ethics committee as it contains potentially identifying and sensitive patient information.

Funding Statement

The author(s) received no specific funding for this work.