Development of a machine learning-based model for predicting the functional outcome of patients with proximal femur fractures

Abstract

Early-stage rehabilitation is crucial for the functional recovery of patients with proximal femur fractures. Predicting functional prognosis at such an early stage can simplify the process of planning for transfers and discharge destinations, as well as setting rehabilitation goals. The current study aimed to develop a model using machine learning to predict the functional prognosis of patients with proximal femur fractures based on parameters at the time of hospital admission. Our research utilized a dataset from 3,088 proximal femur fracture cases recorded in the Japan Association of Rehabilitation Database. The dependent variable was the level of independence in daily living (ADL) activities at discharge, categorized into nine classes. A regression model was implemented by approximating the dependent variable to a continuous value. Accuracy and Quadratic Weighted Kappa (QWK) were used to evaluate the prediction accuracy, and SHAP (SHapley Additive exPlanations) values were used to evaluate the model’s predictive explainability. The machine learning model exhibited an Accuracy of 0.340 and a QWK of 0.657, indicating solid agreement. The top parameters according to SHAP values were the total Functional Independence Measure (FIM) score and the level of independence in ADL in elderly with dementia. We developed a machine learning model capable of predicting the independence level in daily activities at discharge, utilizing patient data available at hospital admission for those suffering from proximal femur fractures.

Introduction

Proximal femur fractures, prevalent among the elderly, significantly impact patients’ quality of life (QOL) and have socioeconomic consequences¹. These fractures are prevalent among patients with osteoporosis², leading to difficulties in walking and bedridden states, significantly diminishing patients’ activities of daily living (ADL) and QOL³. Furthermore, these fractures are associated with a high mortality rate five years post-injury⁴, underscoring the need for rapid and appropriate treatment⁵. Surgical methods are often chosen for treatment, but patients rarely recover full function post-surgery⁶. Early-stage rehabilitation and physical therapy play crucial roles in recovery⁷. Implementing individualized exercise and nutrition programs can improve physical function and life quality⁸. Healthcare providers must consider the patient’s overall health status to develop a safe and effective rehabilitation plan.

Early prognosis prediction is essential for creating personalized treatment plans. However, multifactorial prognosis predictions can be challenging, where machine learning and deep learning techniques might prove beneficial. In recent years, the application of machine learning in healthcare has increasingly contributed to disease analysis, prognosis prediction, diagnosis, and improving treatment efficiency^9–11. Specifically, its utility has been demonstrated across diverse areas such as automated detection and classification of diseases in medical imaging (e.g., fracture detection from X-ray images, tumor segmentation from MRI images), electronic health record data-driven personalized treatment strategy formulation, accelerated drug discovery by shortening the screening time for new drug candidates, and prediction of infectious disease outbreaks. These advanced applications are essential as they significantly enhance diagnostic accuracy, optimize therapeutic interventions, and improve the efficiency of healthcare resource allocation.

Previous studies have identified age, comorbidities, functionality, and cognitive ability as prognostic factors for proximal femur fractures¹². The patient’s pre-injury living conditions also impact clinical outcomes¹³. However, most studies have focused on limited indicators or binary classifications, such as the ability to walk, which do not provide a comprehensive evaluation of the patient’s overall functional prognosis and ADL. Additionally, there are few studies on prognosis prediction using machine learning. A comprehensive assessment of a patient’s overall functional prognosis and ADL is critically needed because patients’ recovery extends beyond isolated functional metrics. To facilitate truly independent living post-discharge, it is essential to evaluate not only basic mobility but also a full spectrum of ADL, including self-care, transfers, and continence, alongside cognitive function and social participation. Such a holistic evaluation enables the development of highly individualized rehabilitation plans and provides precise information crucial for determining appropriate discharge destinations and required home care services, ultimately maximizing patient independence and QOL.

Given the limitations of existing research focused on narrow outcomes, and the inherent complexity of predicting multi-factorial functional recovery, there is a pressing need for objective and efficient prediction tools in clinical practice. Therefore, the current study aimed to develop a machine learning model capable of predicting the multi-level ADL independence at discharge for patients with proximal femur fractures, utilizing various clinical and demographic parameters collected at admission. This research is expected to significantly improve the accuracy of patient prognosis prediction, contribute to the formulation of personalized rehabilitation plans, and ultimately enhance the quality of medical care by providing a more granular understanding of anticipated functional outcomes.

Materials and methods

Patients

The study included data from the Japan Association of Rehabilitation Database (JARD) from 2005 to 2017. (Table 1) JARD enrolment included patients admitted for acute care and transferred due to other factors. The database project itself was approved by the Institutional Review Board of JARD. The requirement for informed patient consent was waived, as retracing is impossible because the data are anonymized. Demographic data, including age, sex, and comorbidities, were collected. Baseline patient characteristics, as shown in Table 1, include demographic and clinical data. The following characteristics of the proximal femur fractures were recorded: cause of injury; fracture side, fracture type, presence of surgical treatment, presence of prior fractures, number of days from injury to hospitalization, length of hospitalization, and scores on each item in the Functional Independence Measure (FIM) at admission, discharge, and rehabilitation hospital discharge. We excluded patients with unassessed and missing the level of independence in ADL at discharge. Furthermore, it should be noted that the admission and discharge dates differ for patients registered in the JARD.

Table 1.

Baseline characteristics at admission of patients (N = 2959) who were included in the prediction model for the level of independence in ADL at discharge.

Characteristics	Values	Missing data, n (%)
Sex (male)	629 (21.3)	6 (0.20)
Age at the time of admission	81.89 (10.68)	25 (0.84)
Comorbidities	1179 (43.55)	252 (8.52)
Number of days from fracture to admission	9.66 (189.29)	24 (0.81)
FIM score
Grooming	2.24 (1.78)	81 (2.74)
Bathing	3.66 (2.22)	78 (2.64)
Bladder management	3.14 (2.50)	78 (2.64)
Bowel management	3.55 (2.48)	81 (2.74)
Transfers—bed/chair/wheelchair	2.84 (2.02)	79 (2.67)
Transfers – bath/shower	1.79 (1.49)	85 (2.87)
Walk/wheelchair	1.97 (1.74)	134 (4.53)
Expression	5.15 (1.90)	83 (2.81)
Problem solving	4.05 (2.24)	81 (2.74)
Memory	4.15 (2.19)	82 (2.77)
Total FIM score	59.29 (27.78)	76 (2.57)
Pre-injury indoor mobility		614 (20.75)
Walking alone	1145 (38.70)
Walking with a cane	308 (10.41)
Walking while holding on to something (such as a cane or walking stick)	440 (14.87)
Silver car / walker	241 (8.14)
Wheelchair	167 (5.64)
Not implemented	43 (1.45)
Hand pull	1 (0.03)
Pre-injury outdoor mobility		763 (25.79)
Walking alone	814 (27.51)
Walking with a cane	426 (14.40)
Silver car / walker	350 (11.83)
Wheelchair	248 (8.38)
Not implemented	358 (12.10)
The activity status before surgery		720 (24.33)
Normal	531 (17.95)
J1	190 (6.42)
J2	317 (10.71)
A1	391 (13.21)
A2	317 (10.71)
B1	254 (8.58)
B2	134 (4.53)
C1	41 (1.39)
C2	64 (2.16)
The level of independence in ADL in elderly with dementia		303 (10.24)
Normal	839 (28.35)
1	396 (13.38)
2a	178 (6.02)
2b	341 (11.52)
3a	344 (11.63)
3b	175 (5.91)
4	324 (10.95)
M	59 (1.99)

For continuous variables, values are presented as mean (SD), and for categorical variables, values are presented as value (%).

FIM, Functional Independence Measure.

Model construction

The dependent variable in the current study is the level of independence in ADL at discharge, classified into eight stages from independent living to bedridden (Table 2), plus a category for complete independence, making nine classes in total. The level of independence in ADL at discharge from the hospital has the property of approximating a continuous severity of illness. Therefore, the current study introduces a machine learning regression model. Machine learning regression models were developed using Python 3.10.13, PyCaret 3.2.0, and scikit-learn 1.2.2 libraries. By rounding the regression model’s output values, the discharge-level daily living independence class for each patient was assigned. This method allows the application of continuous output to classification problems. The regression models used include Linear Regression, Ridge Regression, Bayesian Ridge, Random Forest Regressor, Extra Trees Regressor, Decision Tree Regressor, Gradient Boosting Regressor, Huber Regressor, Ada Boost Regressor, Light Gradient Boosting Machine, and Voting ensemble model, which combines predictions from multiple base models to improve overall performance. Each model contributes based on its predictive strength, ensuring robust and balanced predictions. We chose a regression model despite the categorical nature of the ADL independence levels (N-C2) because these levels represent an ordinal scale approximating a continuous spectrum of functional severity. This approach allows the model to capture the inherent ordering and progression of ADL dependency, potentially yielding more nuanced predictions than a direct multi-class classification that treats categories as discrete and unrelated. The continuous output of the regressor was then rounded to the nearest integer representing an ADL independence level. The best-performing model was selected based on the evaluation metrics discussed later. Finally, the predictive explainability of the selected model was assessed.

Table 2.

Level of independence in daily living at discharge.

Independence level	Class	Numerical transformation	Status
Independent Living	N	0	Independence in ADL
	J1	1	Go out using public transportation or other means
	J2	2	Go out to nearby places
Semi-Bedridden	A1	3	Go out with assistance, spending most of the day away from the bed
	A2	4	Rarely go out, spending the day alternating between lying down and being awake
Bedridden	B1	5	Transfer to a wheelchair for meals and toileting away from the bed
	B2	6	Transfer to a wheelchair with assistance
	C1	7	Able to turn over in bed independently
	C2	8	Unable to turn over in bed independently

Preprocessing

Primarily, information obtained at the time of admission was used, and features obtained at the time of discharge and rehabilitation hospital discharge were excluded. Clinically irrelevant parameters, such as the date of admission, or contain more than 30% missing values, were also excluded.

Missing values were imputed using scikit-learn’s Iterative Imputer. In this process, each variable with missing values is temporarily treated as a predictive variable, and a model is constructed to estimate its values from other variables. Through this iterative process, missing values are complemented more accurately and reliably.

For feature selection, the Boruta algorithm was employed. Boruta is a feature selection method based on random forests, used to reliably assess the importance of variables. Through Boruta’s feature selection, only variables that contribute to the performance of the model are selected, while variables that are noise are eliminated. This improves the accuracy and interpretability of the final predictive model.

One-Hot Encoding was applied to categorical features, and normalization was applied to numerical features.

Performance evaluation

The dataset was split into training and testing sets in an 8:2 ratio. As a result, 2,367 cases were used for training and 592 cases for testing. During training, tenfold cross-validation was conducted, and models were evaluated based on the Root Mean Square Error (RMSE) and the coefficient of determination (R²). The model with the highest R² value was selected and used to make predictions on the test data. The classification accuracy of the test data was evaluated by rounding the model’s predictive values and assessing them using Accuracy and Quadratic Weighted Kappa (QWK). QWK is an index evaluating the agreement of predictions, with values below 0 indicating no agreement and values closer to 1 indicating high agreement.

Interpretability evaluation

The predictive explainability of the selected model was evaluated using SHapley Additive exPlanations (SHAP). SHAP allows for a quantitative understanding of which features are most important when the model makes a specific prediction.

Results

Baseline patient characteristics

Among the 3088 patients in JARD, applying the exclusion criteria, 2959 patients were included for the prediction model for the level of independence in ADL at discharge. A flowchart of this process is presented in Fig. 1. Table 1 shows the baseline characteristics of the 2959 patients included in the prediction model for the level of independence in ADL at discharge. The 2959 patients included 629 males (21.3%) with a mean age of 81.89 ± 10.68 years.

Fig. 1.

Flowchart of patient selection.

Feature selection

Features obtained at the time of discharge and rehabilitation hospital discharge, clinically irrelevant features, and features with more than 30% missing values were excluded, resulting in 34 features. Further, by applying the Boruta algorithm, 18 features were used for model training (Table 3).

Table 3.

Parameters used for learning.

Features
Patient background
Age
Comorbidities
Characteristics of proximal femur fracture
Number of days from fracture to admission
Neurological status at admission
FIM score
Grooming
Bathing
Bladder management
Bowel management
Transfers—bed/chair/wheelchair
Transfers – bath/shower
Walk/wheelchair
Expression
Problem solving
Memory
Total FIM score
Pre-injury indoor mobility
Pre-injury outdoor mobility
The activity status before surgery
The level of independence in ADL in elderly with dementia

FIM, Functional Independence Measure; ADL, Activities of Daily Living.

Evaluation metrics

The breakdown of the level of independence in ADL at discharge for 2959 patients is 101 (3.41%) Normal, 116 (3.92%) J1, 207 (7.00%) J2, 512 (17.30%) A1, 439 (14.84%) A2, 726 (24.54%) B1, 577 (19.50%) B2, 122 (4.12%) C1, and 159 (5.37%) C2.

Prediction results

The RMSE and R² at validation are presented in Table 4. Gradient Boosting Regressor (GBR) demonstrated the highest R² value among the validated models. The predictive results of this model on the test data were an RMSE of 1.318, an R² of 0.497, an Accuracy of 0.340, and a QWK of 0.657, signifying solid agreement between predicted and true values. Figure 2 shows the confusion matrix for the GBR.

Table 4.

The performance of regression models for predicting the level of independence in ADL at discharge.

Model	RMSE	R2
Gradient Boosting Regressor	1.312	0.497
Voting ensemble model	1.315	0.495
Random Forest Regressor	1.328	0.485
Extra Trees Regressor	1.332	0.481
Ridge Regression	1.339	0.477
Bayesian Ridge	1.339	0.477
Light Gradient Boosting Machine	1.342	0.474
Huber Regressor	1.349	0.469
Linear Regression	1.371	0.451
Ada Boost Regressor	1.377	0.447

RMSE, root mean square error.

Fig. 2.

Confusion matrix for the prediction of the level of independence in ADL at discharge using test data for the GBR.

Critical factors of the model

Based on SHAP values, the most influential parameters in the predictions of GBR were the total FIM score, the level of independence in ADL in elderly with dementia, and the activity status before surgery (Fig. 3).

Fig. 3.

SHAP values for GBR. Impact of features on predicting the level of independence in ADL at discharge. Red and blue colors represent high and low levels of each predictor, respectively. The x-axis represents the SHAP value. A negative SHAP value indicates the patient is likely to secure independence in ADL at discharge, a positive value means they are less likely to secure independence in ADL at discharge. For example, age is a negative predictor since red falls on the positive SHAP value.

Discussion

In the current study, we used machine learning to predict the level of independence in ADL at discharge for patients with proximal femur fractures. GBR achieved a QWK of 0.657, suggesting a solid agreement between the predicted and actual levels of independence. Parameters identified as significant in our study include the total FIM score, the level of independence in ADL in elderly with dementia, and the activity status before surgery, highlighting their importance in understanding the functional prognosis of patients.

Using machine learning, the current study was able to predict the level of independence in ADL at discharge from the parameters at admission for patients with proximal femur fractures. Predicting the level of independence in daily living at discharge can provide more detailed information than simply predicting whether a patient can walk at discharge. For clinical utility, a QWK value of at least 0.7 is considered desirable, aligning with benchmarks from similar studies. While an accuracy of 0.340 might initially seem modest, it is important to consider the multi-class ordinal nature of the outcome variable. For such scales, a strict accuracy metric can undervalue the model’s utility. Therefore, the Quadratic Weighted Kappa (QWK) was employed as a more representative metric, which accounts for the proximity of misclassifications. The achieved QWK of 0.657 indicates a solid agreement, suggesting that the model provides clinically meaningful predictions, even if not perfectly aligned with the exact ADL level.

In similar previous studies focusing on patients with proximal femur fractures, the voting ensemble model was used to predict the motor FIM score at discharge, achieving an R² of 0.746¹⁴. Additionally, the random forest model was used to predict the one-year survival rate of patients, recording an Accuracy of 0.95¹⁵. However, our predictive model faced challenges due to the class imbalance in the dataset. The large number of classes, combined with the underrepresentation of some classes, can lead to difficulties in the learning process. Imbalanced datasets can cause machine learning algorithms to favor the majority class, potentially resulting in biased predictions. Addressing class imbalance is crucial to ensure that the model effectively learns and generalizes from the available data, especially for underrepresented classes. Techniques such as SMOTE (Synthetic Minority Over-sampling Technique) were employed, and their impact on model performance is discussed. Despite these challenges, our study’s results demonstrate the utility of machine learning in predicting the level of independence in ADL at discharge for patients with proximal femur fractures.

The current study revealed that the total FIM score, the level of independence in ADL in elderly with dementia, and the activity status before surgery are significant factors in predicting the level of independence in ADL at discharge for patients with proximal femur fractures. Analysis of SHAP values indicated that these factors significantly contribute to functional recovery, establishing them as essential indicators for predicting post-discharge independence in ADL. Our results align with previous studies^16,17, which also showed that factors such as age, pre-fracture physical ability, and post-surgical complications significantly impact patients’ functional prognosis. However, these studies indicated that gender and the type of fracture did not significantly affect functional prognosis. Similarly, our study found that gender and pre-incident residence were not significant predictors of outcome, suggesting that these elements have less impact on functional prognosis than other factors.

The regression models, particularly GBR, achieved good agreement (QWK = 0.657) on the test data. This outcome supports the applicability of regression models for medical data with categorized outcomes, especially when there is an order among the categories. This aligns with research^18,19, which indicated that regression models could outperform classification models under certain conditions. These findings underscore the effectiveness of our approach in predicting functional outcomes for patients with proximal femur fractures. The decision to employ a regression model (Gradient Boosting Regressor) for an ordinal categorical outcome variable, such as ADL independence, warrants further discussion. While a direct multi-class classification model or ordinal logistic regression might appear conventional, our approach was based on the premise that ADL levels represent an underlying continuous functional ability. Regression models can leverage this inherent ordering, often performing well when the categories are closely spaced and represent degrees of a continuous underlying construct. The achieved QWK of 0.657, which accounts for the ordinal nature, supports the practical utility of this approach, consistent with previous studies that have demonstrated the potential benefits of regression models.

The model developed in the current study provides useful information for planning rehabilitation programs for patients with proximal femur fractures. The validity of the predictive model is further supported by its alignment with common risk factors. However, identifying unexplored risk factors and addressing issues of data imbalance could contribute to a deeper understanding and improved accuracy of functional prognosis predictions. Future research will require more comprehensive datasets and different modeling techniques. For instance, the following researches are necessary to enhance the scientific impact and clinical applicability of this valuable research: (i) Application of Ordinal Logistic Regression or Deep Learning Models, (ii) More Comprehensive Handling of Class Imbalance, (iii) Conducting External Validation, and (iv) Incorporation of Time-Series Data.

The current study has several limitations that should be acknowledged. One limitation is that the final functional outcome was measured at the time of discharge, which varied among patients. Furthermore, it is important to note that this study was conducted in Japan, where the public health insurance system allows for relatively low patient out-of-pocket medical expenses (generally 30% or less, and even lower for the elderly). Additionally, specific healthcare payment systems such as the Diagnosis Procedure Combination/Per-Diem Payment System (DPC/PDPS) and the structure of convalescent rehabilitation wards facilitate comparatively longer inpatient rehabilitation periods than in some other countries where cost considerations may limit such durations. On average, patients were discharged around 49 days after admission, a time when most of the expected functional recovery had typically occurred. While this system provides ample opportunities for patients to receive sufficient necessary rehabilitation and potentially contributes to maximizing functional recovery, it is also recognized that unnecessarily prolonged hospital stays can lead to increased healthcare costs and may delay patients’ seamless reintegration into society. Nevertheless, the average length of stay for patients in this study was approximately 49 days post-admission, which aligns with the typical duration required for significant functional recovery in this patient population. Another limitation is that the study included patients in both acute and post-acute phases, making it challenging to perform standardized initial evaluations upon admission. To address this, the number of days from injury to admission was incorporated as a variable in the model, enabling its application to both acute and post-acute phases and increasing its usefulness for a broader patient population. The exclusion of deceased cases from the analysis may limit the model’s ability to accurately predict the prognosis of potentially fatal cases. Additionally, specific injury details, such as the timing of surgery, were not available in the database. While these factors could have potentially improved the prediction accuracy of the machine learning model, their absence was due to the retrospective nature of the study using a pre-existing database.

Despite these limitations, the current study provides valuable insights into predicting functional outcomes in patients with proximal femur fractures. While our regression model demonstrated solid agreement, exploring alternative modeling techniques and conducting external validation are essential for further improving model performance and practical application.

Conclusion

In conclusion, this study successfully developed a machine learning model utilizing only admission data to predict the multi-level ADL independence at discharge for patients with proximal femur fractures. The Gradient Boosting Regressor demonstrated a robust performance with a Quadratic Weighted Kappa of 0.657, indicating solid agreement with actual outcomes. Key predictive factors identified by the model included the total FIM score, the level of ADL independence in elderly patients with dementia, and the pre-surgical activity status. The significant implication of this research lies in its ability to provide more detailed and nuanced predictions of post-fracture functional outcomes compared to conventional binary classifications. This predictive model offers valuable support for healthcare professionals in clinical decision-making, aiding in the formulation of highly individualized rehabilitation plans, selection of appropriate discharge destinations, and setting realistic rehabilitation goals. Ultimately, it is expected to enhance patient functional recovery and overall quality of life. Future work will involve external validation and continuous refinement to further improve the model’s accuracy and establish its robust utility in diverse clinical settings.

Author contributions

KN, SM and IT conducted data collection and data entry, performed the statistical analysis, and wrote the manuscript. Other all authors, KI, YS, MI, YE, TF, JN, SH, YK, SeOh, and SuOr contributed to the recruitment of subjects and collected their data, and approved the final manuscript. SuOr integrated the project, working as the corresponding author.

Funding

This work was supported by a research grant funded by the JOA (Japanese Orthopaedic Association) Subsidized Science Project Research 2020–1 and JSPS KAKENHI Grant Number JP20K18052.

Data availability

The datasets used and/or during the current study available from the corresponding author on reasonable request.

Declarations

Competing interests

The authors declare that they have no competing interests. We did not receive payments or other benefits or a commitment or agreement to provide such benefits from any commercial entities.

Ethical approval

We declare that all protocols involving humans have been approved by the Chiba university Hospital and have been performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki and its later amendments. We declare that all participants provided written informed consent before their inclusion in the current study.