Artificial intelligence for predicting 30-day mortality after surgery for femoral shaft fractures: A retrospective study

Abstract

Background and Aims:

Surgical repair of femoral shaft fractures continues to have notable perioperative morbidity and mortality. The purpose of this study is to assess whether artificial intelligence (AI)-driven models can be utilised to predict 30-day mortality after surgery for femoral shaft fractures and to identify patient risk factors for mortality using AI.

Methods:

This retrospective study utilised data from the National Surgical Quality Improvement Program between 2015 and 2020. Five AI-driven models were developed and tested using patient clinical information to predict mortality within 30 days of surgery. Additionally, the most important variables for the best-performing model were identified.

Results:

A total of 1720 patients were identified, and the 30-day mortality rate after femoral shaft fracture surgery was 3.4% (n = 58). XGBoost demonstrated the best predictive performance, with an area under the curve (AUC) of 0.83, a calibration intercept of −0.03, a calibration slope of 1.17, and a Brier score of 0.02. The most important variables for prediction were age, preoperative white blood cell count, creatinine, haematocrit, platelets, blood urea nitrogen, and body mass index.

Conclusion:

This study is the first to internally validate an AI-driven model for predicting mortality within 30 days of surgery in an isolated population of femoral shaft fracture patients, demonstrating good performance. Further research is needed to develop an excellent-performing, AI-driven model that is externally validated prior to clinical translation to support anaesthesiologists and orthopaedic surgeons in perioperative risk stratification and patient education.

INTRODUCTION

Femoral shaft fractures continue to be a large source of morbidity and mortality, especially in the geriatric population, where a Medicare study found a 24.9% mortality rate within 1 year, a union failure rate of 2.52% after 2 years, and an infection rate of 2.22% after 2 years.[1] Intramedullary nailing (IM) or open reduction and internal fixation (ORIF) is often performed to promote early mobilisation and decrease hospital length of stay in patients with femoral shaft fractures.[2] However, there are risks of many postoperative complications, including infections, malunion and nonunion, neurovascular injury, pulmonary complications, and mortality. To enhance preoperative risk stratification, patient education, and other aspects of clinical decision-making, there is a need for clinical support tools that offer patient-specific risk predictions and guidance. This is especially important in the population of individuals with femoral shaft fractures, where morbidity and mortality are common.[3]

Artificial intelligence (AI) and machine learning (ML) have been extensively explored for their potential utility in preoperative risk prediction.[4,5,6,7,8,9,10] For example, Veeramani et al.[9] utilised ML algorithms for predicting unplanned intubations after anterior cervical discectomy and fusion surgery, achieving prediction accuracies between 72% and 100% and area under the curve (AUC) values ranging from 0.52 to 0.77. In another study by Hopkins et al.,[10] the team developed a deep neural network model using data from 4046 patients who underwent posterior spinal fusion, which performed well in predicting surgical site infections. These studies highlight the utility of AI-driven models for accurately and effectively predicting intraoperative and/or postoperative complications preoperatively.

However, there remains an unmet need to develop highly accurate, patient-specific preoperative risk prediction tools for perioperative mortality following surgery for femoral shaft fractures. A clinical decision support tool can augment the perioperative decision-making of both anaesthesiologists and orthopaedic surgeons. Therefore, the primary objective of this study is to develop and internally validate five AI-driven models for predicting mortality within 30 days of surgery for femoral shaft fractures. The secondary objective of this study is to identify important patient risk factors for 30-day mortality using AI algorithms. We hypothesised that one of the AI-driven models would be able to identify an optimal subset of features (i.e., the most relevant risk factors) that can produce the highest predictive ability for 30-day postoperative mortality.

METHODS

Data for this retrospective study were obtained from the American College of Surgeons—National Surgical Quality Improvement Program (NSQIP) database. The data includes but is not limited to patient demographics, comorbidities, laboratory values, and postoperative complications that occur within 30 days of surgery. All data used in this study were deidentified. This study was deemed exempt from the Institutional Review Board (IRB) approval of George Washington University.

The NSQIP participant use files from 2015 to 2020 were obtained. All patients with femoral shaft fractures were identified using International Classification of Diseases (ICD) codes, and patients who underwent surgical intervention were identified using Current Procedural Terminology (CPT) codes 27506 and 27507. Only patients who had an initial encounter with a closed fracture were included. The final analysis of this study cohort included 1720 patients.

The NSQIP dataset contains inputs for over 250 variables. Of these, a total of 31 preoperative variables, comprising patients’ demographics (e.g., race, gender, and age), preoperative comorbidities (e.g., diabetes, smoking, chronic obstructive pulmonary disease, and congestive heart failure), and laboratory measurements (e.g., creatinine, haematocrit, and platelets) were considered in the study [Table 1]. The initial variable selection was based on extensive literature analysis, prior NSQIP studies, and the author’s clinical expertise.[11,12,13,14] The primary outcome of interest was mortality within 30 days of surgery.

Table 1.

Baseline characteristics of the study population (n=1720)

Characteristics	Study Population (n=1720)	Missing Data
Gender, Female	1206 (70.1)	0 (0)
Race		0 (0)
White	1272 (73.9)	0 (0)
Black or African American	145 (8.4)	0 (0)
Asian	76 (4.4)	0 (0)
Unknown/not reported	215 (12.5)	0 (0)
Others	12 (0.8)	0 (0)
Body mass index, kg/m2	26.97 (10.08)	0 (0)
In/outpatient status, inpatient	1704 (99.1)	0 (0)
Age, years	69.47 (19.22)	0 (0)
Principal anaesthesia technique
General	1505 (87.5)	0 (0)
Spinal + epidural	156 (9.1)	0 (0)
MAC/IV sedation	55 (3.2)	0 (0)
Regional	4 (0.2)	0 (0)
Diabetes
No	1369 (79.6)	0 (0)
Insulin	177 (10.3)	0 (0)
Non-insulin	174 (10.1)	0 (0)
Smoker, yes	235 (13.7)	0 (0)
Dyspnoea status
No	1631 (94.8)	0 (0)
Moderate exertion	77 (4.5)	0 (0)
At rest	12 (0.7)	0 (0)
Functional status
Independent	1365 (79.4)	0 (0)
Partially dependent	253 (14.7)	0 (0)
Totally dependent	80 (4.7)	0 (0)
Unknown	22 (1.3)	0 (0)
History of severe COPD, yes	146 (8.5)	0 (0)
Ascites, yes	1 (0.1)	0 (0)
Congestive heart failure, yes	50 (2.9)	0 (0)
Hypertension requiring medication, yes	716 (41.6)	0 (0)
Currently requiring/on dialysis, yes	27 (1.6)	0 (0)
Disseminated cancer, yes	32 (1.9)	0 (0)
Steroid or immunosuppressant use for a chronic condition, yes	81 (4.7)	0 (0)
Greater than 10% loss of body weight, yes	15 (0.9)	0 (0)
Bleeding disorders, yes	196 (11.4)	0 (0)
Blood transfusion in the 72 hours before surgery, yes Preoperative laboratory values	110 (6.4)	0 (0)
Blood urea nitrogen, mg/dL	19.74 (10.90) (19.22, 20.27)	55 (3.2)
Creatinine, mg/dL	0.99 (0.72) (0.95, 1.02)	35 (2.0)
Haematocrit, %	34.73 (5.74) (34.46, 35.00)	30 (1.7)
Platelets, 103/μL	219.94 (77.36) (216.25, 223.64)	33 (1.9)
Serum sodium, mEq/L	138.02 (3.64) (137.85, 138.20)	35 (2.0)
White blood cell, 103/μL	9.78 (3.63) (9.61, 9.96)	35 (2.0)
Albumin, g/dL	3.51 (0.63) (3.51, 3.58)	633 (36.8)
Bilirubin, mg/dL	0.63 (0.61) (0.58, 0.65)	669 (38.9)
International normalised ratio	1.11 (0.22) (1.09, 1.11)	326 (19.0)
Partial thromboplastin time, seconds	28.92 (6.74) (28.49, 29.36)	797 (46.3)
Serum glutamic-oxaloacetic transaminase, U/L	31.00 (35.33) (28.85, 33.14)	676 (39.3)
Mortality
Alive	1662 (96.6)	0 (0)
Dead	58 (3.4)	0 (0)

Data expressed as mean (standard deviation) (95% confidence interval) or n (%), MAC=monitored anaesthesia care, IV=intravenous, COPD=chronic obstructive pulmonary disease

Data were collected and processed using structured query language (SQL) and the Python programming language. Out of the 31 variables initially selected for analysis, five variables (i.e., preoperative albumin, bilirubin, serum glutamic-oxaloacetic transaminase, partial thromboplastin time, and international normalised ratio) were removed due to a high percentage of missing data (i.e., >5%). In the remaining 26 variables, six variables (i.e., preoperative sodium, blood urea nitrogen (BUN), creatinine, white blood cell count (WBC), haematocrit, and platelets) had fewer than 4% of values missing at random. Missing data for the above six variables were imputed using a ML-based multiple imputation method, k-nearest neighbours.[15] The dataset used in this study was imbalanced, as a minority of patients experienced mortality events. Thus, we applied resampling techniques, combining the synthetic minority over-sampling technique and the Tomek link method, to help mitigate poor predictive performance.[16] Furthermore, ensemble learning methods and balanced evaluation metrics, such as precision, recall, F1-score, and area under the receiver operating characteristics (AUROC), were used in this study (as described in the following section), which are known to be suitable choices for imbalanced data.[17] The study population was divided into training and testing sets by using a stratified split in an 80:20 ratio.

The proposed risk prediction framework utilises ensemble learning methods, which combine the decisions from multiple models to enhance prediction accuracy [Figure 1]. The first component, ‘effective feature engineering’ [Figure 1a], handles raw data extraction, data cleaning, transformation, and study cohort preparation, which were explained in the preceding paragraph. The second component implements the ‘variable selection’ strategy, and an automatic feature selection algorithm, that is, recursive feature elimination with 10-fold cross-validation (RFECV, Figure 1b), handles this role. The third module [Figure 1c] trains and validates the final model by using ‘eXtreme Gradient Boosting (XGBoost)’, a scalable decision tree ML algorithm.[8] The XGBoost technique partitions the training data into subsets, evaluates the best split, validates the model using 10-fold cross-validation, and generates performance evaluation metrics, including feature importance scores, AUROC, calibration intercept, calibration slope, and Brier score. In addition, the model was tuned to further improve prediction accuracy with hyperparameter optimisation techniques.[18]

Figure 1.

Proposed machine learning framework. (a) model workflow, (b) automatic feature selection though recursive feature elimination with cross-validation, (c) model training and validation

RFECV, a greedy search feature selection algorithm, aims to identify the best possible feature subset that impacts the target outcome most. It repeatedly creates models by retaining more representative features (i.e., variables or risk factors) while removing irrelevant features at each iteration, thereby facilitating reduced computational overhead, higher learning accuracy, and better prediction.[19,20] It constructs the next model by using the remaining features until all features have been exhausted and ranks the features based on the order in which they are eliminated. The optimal subset of features (i.e., the most relevant risk factors associated with 30-day mortality after femoral shaft fracture surgery) was deployed to the ensemble learning model (i.e., XGBoost) for further evaluation. In the training phase with XGBoost, the model was trained using a training dataset, and hyperparameter tuning was applied to optimise the model’s parameters for improved performance. To determine the optimal parameters, a 10-fold cross-validation (CV) was performed on the tunable parameters of the model. The performance of the XGBoost classifier was further evaluated in the testing phase. Subsequently, four additional algorithms (random forest, support vector machine, neural network, and logistic regression) were constructed on the training data by using 10-fold cross-validation and validated on the test data. The relative importance of the risk factors was assessed using the AUC, calibration, and Brier score to quantify the performance of ML models.

RESULTS

A total of 1720 patients were identified, and the 30-day mortality rate after femoral shaft fracture surgery was 3.4% (n = 58) [Table 1]. Most patients were Caucasian (73.9%, n = 1272), followed by African Americans (8.4%, n = 145) and Asians (4.4%, n = 76). The mean age of the patients was 69.47 years. A total of 1206 (70.1%) patients were women. Approximately 50% (n = 861) were either overweight or obese, with BMI in the range of 25.0–34.99 kg/m². The most common preoperative comorbidities identified were hypertension requiring medication (41.6%, n = 716) and diabetes (20.4%, n = 251). The mean age of the patients was higher in the mortality group (82.31 years) than in the alive group (69.02 years). Similarly, the preoperative WBC count and creatinine levels had higher mean values in the mortality group (i.e. 10.83 10³/μL (SD: 4.67) (95% CI: 9.60, 12.06) and 1.40 mg/dL (SD: 1.51) (95% CI: 1.00, 1.80), respectively) than in the alive group (i.e. 9.75 10³/μL (SD: 3.58) (95% CI: 9.57, 9.92) and 0.97 mg/dL (SD: 0.67) (95% CI: 0.94, 1.01), respectively) [Table 2].

Table 2.

Patient characteristics of the ‘alive’ and ‘mortality’ groups in the study cohort

Variable	Alive (n=1662)	Mortality (n=58)
Gender, female	1167 (70.2)	39 (67.2)
Race
White	1223 (73.6)	49 (84.5)
Black or African American	144 (8.7)	1 (1.7)
Asian	73 (4.4)	3 (5.2)
Unknown/not reported	210 (12.6)	5 (8.6)
Other	12 (0.7)	0 (0)
Body mass index, kg/m2	25.22 (9.65)	29.34 (11.08)
In/outpatient status, inpatient	1646 (99.0)	58 (100)
Age, years	69.02 (19.31)	82.31 (9.99)
Principal anaesthesia technique
General	1454 (87.5)	0 (0)
Spinal + epidural	152 (9.1)	51 (87.9)
MAC/IV sedation	52 (3.1)	4 (6.9)
Regional	4 (0.2)	3 (5.2)
Diabetes
No	1327 (79.8)	42 (72.4)
Insulin	170 (10.2)	7 (12.1)
Non-insulin	165 (9.9)	9 (15.5)
Smoker, yes	1433 (86.2)	52 (89.7)
Dyspnoea status
No	1580 (95.0)	51 (87.9)
Moderate exertion	72 (4.3)	5 (8.6)
At rest	10 (0.6)	2 (3.4)
Functional status
Independent	1329 (80.0)	36 (62.1)
Partially dependent	240 (14.4)	13 (22.4)
Totally dependent	73 (4.4)	7 (12.1)
Unknown	20 (1.2)	2 (3.4)
History of severe COPD, yes	1522 (91.6)	52 (89.7)
Fluid accumulation, yes	1661 (99.9)	58 (100)
Newly diagnosed congestive heart failure, yes	1619 (97.4)	51 (87.9)
Hypertension requiring medication, yes	966 (58.1)	38 (65.5)
Currently requiring/on dialysis, yes	1637 (98.5)	56 (96.5)
Disseminated cancer, yes	1635 (98.3)	53 (91.4)
Steroid/immunosuppressant for a chronic condition, yes	1585 (95.4)	54 (93.1)
Greater than 10% loss of body weight, yes	1647 (99.1)	58 (100)
Bleeding disorder, yes	1476 (88.8)	48 (82.7)
Blood transfusion, yes	98 (5.9)	12 (20.7)
Preoperative lab values
Blood urea nitrogen, mg/dL	19.52 (10.51) (18.98, 20.04)	26.53 (14.10) (22.94, 30.10)
Creatinine, mg/dL	0.97 (0.66) (0.94, 1.01)	1.40 (1.51) (1.00, 1.80)
Haematocrit, %	34.83 (5.67) (34.56, 35.11)	31.77 (5.45) (30.34, 33.20)
Platelets, 103/μL	219.92 (75.81) (216.24, 223.61)	220.47 (113.39) (190.65, 250.28)
Serum sodium, mEq/L	138.03 (3.58) (137.86, 138.21)	137.72 (4.99) (136.41, 139.04)
White blood cell, 103/μL	9.74 (3.54) (9.57, 9.92)	10.83 (4.67) (9.60, 12.06)

Data expressed as mean (standard deviation) (95% confidence interval) or n (%). The bolded variables are the risk factors for 30-day mortality after surgery for femoral shaft fractures generated by the proposed AI-driven model. MAC=monitored anaesthesia care, IV=intravenous, and COPD=chronic obstructive pulmonary disease

In this study, we employed RFECV to determine the optimal number of features that yield the highest F1 score, which is a harmonic mean of precision and recall. The XGBoost RFECV model demonstrated the highest predictive ability with a cross-validated F1-score of 0.95 when the number of features (i.e., variables or risk factors) selected was 7 [Figure 2]. The resultant seven risk factors had importance scores of greater than 50. The seven risk factors identified were age, BMI, preoperative BUN, preoperative creatinine, preoperative haematocrit, preoperative platelet count, and preoperative WBC count [Figure 3].

Figure 2.

Predictive ability of the model: XGBoost classifier’s Recursive Feature Elimination with Cross-Validation (RFECV) model showing the number of features Vs cross-validation F1 score. (XGBoost = Extreme Gradient Boosting.)

Figure 3.

The relative variable importance plot demonstrating the relative importance of risk factors in the XGBoost model. (Preop = preoperative, BUN = Blood Urea Nitrogen, XGBoost = Extreme gradient Boosting.)

Table 3 presents the performance metrics for each model on the test dataset (n = 344), along with a 95% confidence interval. The AUC ranged from 0.44 to 0.83, the calibration intercept ranged from − 0.05 to − 0.14, the calibration slope ranged from 0.09 to 1.19, and the Brier score ranged from 0.02 to 0.03. Of the five ML models tested, the XGBoost demonstrated the best predictive performance, with AUC = 0.83 [Figure 4], calibration intercept = −0.03, calibration slope = 1.17, and Brier score = 0.02 [Table 3]. The most significant risk factors for predicting 30-day mortality after femoral shaft fracture surgery, in order of importance, were age, preoperative WBC count, creatinine, haematocrit, platelet count, BUN, and BMI.

Table 3.

Algorithm performance in testing dataset (95% confidence interval), n=344

Metric	XGBoost	Random Forest	Support Vector Machine	Neural Network	Logistic Regression
Area under the curve	0.83 (0.81, 0.86)	0.80 (0.80, 0.82)	0.751 (0.35, 0.86)	0.440 (0.23, 0.63)	0.789 (0.707, 0.821)
Calibration intercept	−0.03 (−0.17, 0.13)	−0.140 (−0.26, 0.05)	−0.032 (−0.57, 0.34)	0.047 (−0.06, 0.21)	0.059 (−0.191, 0.119)
Calibration slope	1.17 (0.81, 1.42)	1.29 (0.68, 1.64)	0.09 (−0.33, 1.27)	0.47 (−0.25, 4.29)	0.51 (0.2, 0.87)
Brier score	0.02 (0.02, 0.03)	0.02 (0.02, 0.02)	0.03 (0.02, 0.03)	0.03 (0.02, 0.03)	0.02 (0.02, 0.02)

Figure 4.

Area under the receiver operating characteristics curve, illustrating the discrimination of the XGBoost model, in testing set (n = 344)

DISCUSSION

This study found that the XGBoost model performed best, with an AUC of 0.83, indicating good discrimination ability.[21,22] Furthermore, the most significant patient risk factors for prediction by the XGBoost model were patient age, preoperative WBC count, and preoperative creatinine level.

The study’s findings, which indicate that AI-driven models such as XGBoost can effectively predict mortality following surgery for femur fractures, are consistent with previous studies. For example, Li et al.[23] found that a random survival forest model performed well in predicting 30-day mortality (AUC: 0.83) and 1-year mortality (AUC: 0.75) following surgical repair for femoral neck fractures and intertrochanteric fractures in patients over the age of 50 years. Similarly, Oosterhoff et al.[24] demonstrated that AI-driven models were effective in predicting 90-day and 2-year mortality following surgery for femoral neck fractures in patients aged 65 years or older, achieving the highest AUC of 0.74 for 90-day mortality and an AUC of 0.70 for 2-year mortality. Moreover, similar to our study, which found XGBoost to be the best-performing model, Mosfeldt et al.[25] found that their XGBoost model had the best overall performance, including calibration in predicting 1-, 3-, 6-, and 12-month mortality after surgery in patients over 60 years old with hip fractures.

This study also found that patient age was the most important predictive variable for the XGBoost model, highlighting its importance and role in 30-day mortality. This is consistent with prior studies showing the influence of patient age on 30-day mortality following surgical repair of femoral shaft fractures. For example, Tischler et al.[26] conducted a retrospective study analysing 30-day complications following IM nailing, ORIF, or external fixation of isolated, closed femoral shaft fractures using a national database. In their analysis of 1346 patients, the authors found a 30-day mortality rate of 2.7%, of which 80.5% of patients were over the age of 78.[26] Similarly, Bommireddy et al.[27] found a very high 30-day mortality rate of 13.2% in their single-institution, retrospective analysis of patients 60 years of age or older with a femoral shaft fracture, among whom the mean age was 78.7 years, suggesting a possible influence of age on mortality. The increasing risk of mortality with increasing age should be further investigated using both AI and non-AI methods.

This study also found that preoperative WBC was the second most important predictive variable for the XGBoost model. This is the first study, to our knowledge, that demonstrates the influence of preoperative WBC on 30-day mortality in patients undergoing surgery for femoral shaft fractures. An elevated preoperative WBC count suggests a systemic inflammatory state, which may be in response to trauma and does not imply an infection. Importantly, leucocytosis has been shown to be associated with an increased risk of mortality in patients undergoing colorectal surgery and gastrectomy for gastric cancer.[28,29] However, this is not always the case, as a retrospective study of patients 65 years or older who underwent surgery for femoral neck or trochanteric fractures did not find any association between leucocytosis and mortality.[30] Further studies evaluating preoperative WBC levels and mortality rates following femoral shaft surgery are warranted, given our study’s findings.

This study also found that preoperative creatinine was the third most important predictive variable for the XGBoost model, highlighting its influence on 30-day mortality. This is consistent with several studies that have shown associations between kidney disease or injury and postoperative mortality.[31,32,33] For example, Frisch et al.[31] found that patients with chronic kidney disease (CKD) had a higher risk of mortality at 90 days and 1 year after surgery for hip fractures compared to those without CKD. Similarly, Kim et al.[32] found that preoperative grade IV CKD was an independent risk for mortality within 1 year postoperatively in their retrospective study of surgically managed patients with intertrochanteric fractures. Further studies assessing the relationship between preoperative creatinine levels and outcomes following femoral shaft fracture surgery are warranted to improve perioperative risk stratification.

This study has certain limitations. The NSQIP database is a large national database that may have errors in data entry. However, these likely occur at a rate that is too low to significantly influence our results, given the stringent quality control measures in place. Furthermore, this database lacks certain socioeconomic variables, such as income and education levels, that should be included in future preoperative risk stratification models. Moreover, these models were only internally validated and, as such, cannot be used in the clinical setting immediately. There is a need for these models to be externally validated at multiple hospitals with additional datasets, including those from diverse settings and diverse populations, to substantiate their broader applicability.

CONCLUSION

This is the first study to internally validate an AI-driven model for predicting mortality within 30 days of surgery in an isolated population of patients with femoral shaft fractures, demonstrating good performance. Specifically, the XGBoost model demonstrated good discriminative ability with an AUC of 0.83, surpassing the performance of conventional predictive methods, such as logistic regression. Additionally, the model identified patient-specific risk factors that may increase the risk of mortality, including patient age, preoperative WBC count, and preoperative creatinine levels. This research serves as a groundwork for future research that will likely translate into an excellent performing, externally validated, AI-driven clinical decision support tool utilised by anaesthesiologists and orthopaedic surgeons to improve patient outcomes.

Study data availability

The de-identified data is available on the data repository and may be freely accessible from the website: https://www.facs.org/quality-programs/data-and-registries/acs-nsqip/participant-use-data-file.

Conflicts of interest

There are no conflicts of interest.

Funding Statement

Nil.