Random forest identifies predictors of discharge destination following total shoulder arthroplasty

Abstract

Background

Machine learning algorithms are finding increasing use in prediction of surgical outcomes in orthopedics. Random forest is one of such algorithms popular for its relative ease of application and high predictability. In the process of sample classification, algorithms also generate a list of variables most crucial in the sorting process. Total shoulder arthroplasty (TSA) is a common orthopedic procedure after which most patients are discharged home. The authors hypothesized that random forest algorithm would be able to determine most important variables in prediction of nonhome discharge.

Methods

Authors filtered the National Surgical Quality iImprovement Program database for patients undergoing elective TSA (Current Procedural Terminology 23472) between 2008 and 2018. Applied exclusion criteria included avascular necrosis, trauma, rheumatoid arthritis, and other inflammatory arthropathies to only include surgeries performed for primary osteoarthritis. Using Python and the scikit-learn package, various machine learning algorithms including random forest were trained based on the sample patients to predict patients who had nonhome discharge (to facility, nursing home, etc.). List of applied variables were then organized in order of feature importance. The algorithms were evaluated based on area under the curve of the receiver operating characteristic, accuracy, recall, and the F-1 score.

Results

Application of inclusion and exclusion criteria yielded 18,883 patients undergoing elective TSA, of whom 1813 patients had nonhome discharge. Random forest outperformed other machine learning algorithms and logistic regression based on American Society of Anesthesiologists (ASA) classification. Random forest ranked age, sex, ASA classification, and functional status as the most important variables with feature importance of 0.340, 0.130, 0.126, and 0.120, respectively. Average age of patients going to facility was 76 years, while average age of patients going home was 68 years. 78.1% of patients going to facility were women, while 52.7% of patients going home were. Among patients with nonhome discharge, 80.3% had ASA scores of 3 or 4, while patients going home had 54% of patients with ASA scores 3 or 4. 10.5% of patients going to facility were considered of partially/totally dependent functional status, whereas 1.3% of patients going home were considered partially or totally dependent (P value < .05 for all).

Conclusion

Of various algorithms, random forest best predicted discharge destination following TSA. When using random forest to predict nonhome discharge after TSA, age, gender, ASA scores, and functional status were the most important variables. Two patient groups (home discharge, nonhome discharge) were significantly different when it came to age, gender distribution, ASA scores, and functional status.

Total shoulder arthroplasty (TSA) is a common orthopedic procedure used to treat a variety of degenerative and traumatic pathologies of the shoulder. Over the past few decades, the volume of this procedure has increased drastically, notably with the Food and Drug Administration approval of reverse TSA procedure in 2004.¹² Between the years 2011 and 2017, the number of primary shoulder arthroplasties increased by 103.7% from 51,329 to 104,575, and it is expected to continue to increase.²³

As medicine continues to optimize the value of care, it has become increasingly important to understand the factors that contribute to the cost associated with TSA. Discharge location following surgery, whether to home or a rehabilitation center/skilled nursing facility (SNF), has been shown to influence both length of stay and overall cost.¹⁸ In addition, the current literature shows nonhome discharge is associated with poorer postoperative outcomes.^9,11 Preoperatively identifying the risk factors necessitating inpatient stay and nonhome discharge may aid orthopedic surgeons in properly determining the appropriate surgical setting (hospital vs. surgery center), prepare patients and family for postoperative transition process, and reduce potential length of stay for patients undergoing TSA.

Machine learning is a rapidly advancing field that focuses on computers processing data. Machine learning algorithms have a widespread use in orthopedic literature and are used to predict surgical outcomes for many different procedures. Compared to traditional statistical analyses which identify relationships and correlation between variables and outcomes, machine learning algorithms, such as random forest, offer a method to predict and classify data. Machine learning algorithms have been used to predict a variety of orthopedic outcomes from meniscus tears to amputations in trauma, outperforming multiple logistic regression, modified Charlson comorbidity score, American Society of Anesthesiologists (ASA) classification, etc.^10,13 In a study investigating clinical outcome scores in TSA, for example, random forest algorithms had the greatest area under curve (AUC) for predicting postoperative readmissions compared to other statistical methods.²²

Random forest is a specific type of machine learning algorithm based on the creation of multiple randomly generated decision trees that obtain gini impurity reduction at each node to calculate feature importance. In other words, feature importance is a measure of what variables the algorithm considers most relevant when classifying samples into groups of interest. Despite its high predictability even compared to other machine learning algorithms, random forest has not yet been used to predict discharge to facility after TSA.

Machine learning algorithms can be used to determine the variables that had the greatest effect on outcomes such as discharge destination. Understanding these variables may guide further investigative efforts toward improving patient counseling, outcomes, hospital workflow, and overall cost of treatments. Therefore, the aim of this study is to evaluate different machine algorithms in predicting and identifying which variables have the greatest effect on discharge destination (home vs. nonhome) following TSA.

Methods

In general, use of a larger dataset improves the predictability of a machine learning algorithm.²⁴ Because the National Surgical Quality Initiative Program (NSQIP) database includes more than 1 million cases of deidentified patients and their perioperative variables, it is a popular choice among medical scientists.¹⁶ The study was presented before our institutional review board and was determined to be exempt.

Using R Studio (R software; R Studio, Boston, MA, USA), the NSQIP database was screened to obtain a list of patients undergoing elective TSA (Current Procedural Terminology 23472) for diagnosis of primary degenerative arthritis between the years 2008 and 2018. Patients with a history of avascular necrosis, rheumatoid arthritis, other inflammatory arthritis, and concurrent fracture treatment were excluded from the study. The outcome of interest was discharge destination, which is listed in NSQIP database as “skilled care”, “unskilled facility”, “facility which was home”, “Home”, “Separate Acute Care”, “Rehab”, “Against Medical Advice”, etc. Patients whose discharge destinations were listed as “facility which was home”, “Home”, and “Against Medical Advice” were categorized as home discharge, while patients with other discharge destinations were grouped as nonhome discharge.

The machine learning and programming portion was written in Python 3 on Jupyter Notebook (Project Jupyter, New York NY, USA) where necessary packages were imported. The dataset was prepared for machine learning by excluding all patients who had invalid or null values for surgical and demographic variables. Next, LabelEncoder (scikit-learn, Paris, France) was used to one hot label categorical variables into dummy variables to prevent machine learning algorithms from inferring ordinal relationships where there is none. The scikit-learn scale function was then used to scale the continuous variables around the same mean and variance to optimize distance-based algorithms such as K-nearest neighbors.

Afterwards, scikit-learn package from Python was used to train 5 machine learning algorithms including random forest, gradient-boosted trees, artificial neural network, K-nearest neighbors, and Gaussian Bayes. Logistic regression based on ASA classification was also performed as a reference. Variables used to predict discharge destination were age, sex, body mass index, functional status in activities of daily living, smoking history, diabetes, ASA classification, operative time, history of congestive heart failure, history of bleeding disorders, history of steroid use, type of anesthesia used, and history of end-stage renal disease. The list of applied variables was then organized in order of feature importance as determined by the random forest algorithm. The variables ranked as the most important by the algorithm were then investigated further using t-test for continuous variables and chi-square test for categorical variables to see if there was a statistically significant difference between the home discharge and nonhome discharge group when it came to the variables of interest.

The algorithms were evaluated based on AUC of the receiver operating characteristic, accuracy, recall, and the F-1 score. Of these, AUC is considered to be the superior metric of discrimination and prediction of machine learning algorithms.⁸ Accuracy refers to the proportion of correct classifications out of total classifications. Recall is the fraction of total nonhome discharges that were correctly classified.⁴ F-1 score is the ratio of correctly classified nonhome discharges to a combination of all nonhome discharges and home discharges that were incorrectly classified as nonhome discharges.¹⁴ The AUC of algorithms was calculated to ensure it met a cutoff of 0.7. When it comes to evaluation of machine learning algorithms, AUC > 0.8 is considered excellent, > 0.7 is considered acceptable, and < = 0.5 is considered no discrimination.³

Results

Application of inclusion and exclusion criteria yielded 18,883 patients undergoing elective TSA between 2008 and 2018. Seventeen thousand seventy patients (90.4%) were discharged home, while 1813 patients (9.6%) were discharged to a nonhome destination. Demographics of the patients included in this study showed that 10,415 patients (55.2%) were female while 8468 patients (44.8%) were male, the average body mass index was 31.2 (standard deviation [SD] 6.9), the average age was 69 years (SD 9.6), and the average operative time was 109 minutes (SD 44.3). Eighteen thousand four hundred fifty nine patients (97.7%) were functionally independent, 403 patients (2.1%) were partially dependent, and 21 patients (0.1%) were totally dependent.

Random forest and gradient-boosted trees were the best performing algorithms (AUC 0.79 for both), with random forest marginally outperforming gradient boosted trees in accuracy (88.81% vs. 88.76%) and recall (1.00 vs. 0.99). The rest of the algorithms were ranked artificial neural network (AUC 0.73), Gaussian Bayes (AUC 0.65), and K-nearest neighbors (AUC 0.61) in that order. All studied machine learning algorithms had higher AUC than ASA-based logistic regression which had AUC of 0.55 (Table I).

Table I.

Metrics for algorithm evaluation.

	AUC of receiver operating characteristic	Accuracy	Recall	F-1 score
Random forest	0.79	88.81%	1.00	0.94
Gradient boosting trees	0.79	88.76%	0.99	0.94
Artificial neural network	0.73	78.52%	0.83	0.87
Gaussian Bayes	0.65	85.99%	0.99	0.94
K nearest neighbors	0.61	87.44%	0.98	0.93
ASA-based logistic regression	0.55	88.63%	1.00	0.94

AUC, area under the curve; ASA, American Society of Anesthesiologists.

Random forest ranked age, sex, ASA classification, and functional status as top 4 most important variables (Fig. 1) in order of feature importance of 0.340, 0.130, 0.126, and 0.120, respectively. Patients going to facility compared to going home were significantly older (age 76 years vs. 68 years), more females (78.1% vs. 52.7%), had higher ASA scores of 3 or 4 (80.3% vs. 54%), and had more partially/totally dependent functional status (10.5% vs. 1.3%) (Table II, P value < .05 for all). It should be noted that age, functional status, ASA classification, and sex were also ranked as some of the top variables by gradient-boosted trees, the second-best performing algorithm (Fig. 2).

Figure 1.

Feature importance ranked by random forest. ASA, American Society of Anesthesiologists; BMI, body mass index; HTN, hypertension; CHF, congestive heart failure; RF, random forest.

Table II.

Demographics and results.

	Home discharge	Nonhome discharge	P value
Total	17,070	1813
Gender (%)
Male	8071 (47.3)	397 (21.9)	< .05
Female	8999 (52.7)	1416 (78.1)
BMI	31.1	31.7	< .05
Age (years)	68	76	< .05
ASA (%)
1	294 (1.7)	4 (0.2)	< .05
2	7564 (44.3)	350 (19.3)
3	8823 (51.7)	1299 (71.6)
4	387 (2.3)	158 (8.7)
5	2 (0)	2 (0)
Functional status (%)
Independent	16,838 (98.6)	1621 (89.4)	< .05
Partially dependent	219 (1.3)	184 (10.1)
Totally dependent	13 (0)	8 (0.4)

BMI, body mass index; ASA, American Society of Anesthesiologists.

Figure 2.

Feature importance ranked by gradient-boosted trees. BMI, body mass index; hxchf, history of congestive heart failure; GBT, gradient boosting trees.

Discussion

Discharges to a nonhome facility after surgery can levy a financial burden to the healthcare system, especially as older patients with more comorbidities are getting surgery on an outpatient basis at higher rates than before. Prior literature using statistical analytic tools, including multivariate regression, have identified risk factors of nonhome discharge for various orthopedic procedures. Female gender, older age, and more comorbidities have been previously associated with nonhome discharge.^15,20

Machine learning technology has been proven useful as a tool in predicting postoperative outcomes via identification of specific variables, which ultimately aids in the improvement of outcomes and efficient utilization of healthcare resources. Research by Gowd et al showed that supervised machine learning algorithms were superior to logistic regression based on traditional comorbidity indices such as ASA classification in predicting adverse outcomes following TSA such as infection, readmissions, etc.⁷ Machine learning algorithms such as artificial neural networks have been previously used to make predictive models for outcomes following TSA.¹¹ Of various machine learning algorithms, random forest algorithms have been shown to outperform other algorithms such as adaptive boosted algorithms and neural networks in prediction of TSA outcomes such as readmission rates.¹ Although there has been previous work to predict nonhome discharge post TSA by use of algorithms such as boosted trees and neural networks, random forest has not yet been used for this purpose.¹⁵ Furthermore, previous works gauged algorithm performance by solely their AUC and accuracy, which may fall short of recall and F-1 score when classifying skewed datasets.⁴ By identifying the most important factors leading to nonhome discharge after TSA, machine learning algorithms such as random forest can assist orthopedic surgeons in assessing patients with risk factors for this and taking the appropriate steps to optimize their postoperative discharge.

The purpose of this study was 2-fold: to validate previous work in literature regarding random forest and other machine learning algorithm’s ability to predict TSA outcomes and also to investigate demographic variables predicting discharge destination following TSA by using the random forest algorithm, which has been shown to exceed other machine learning algorithms in its predictive value and discrimination.¹ As in previous studies, the studied machine learning algorithms had higher performance measured in AUC compared to ASA-based logistic regression⁷ (Fig. 3). Among the machine learning algorithms, random forest slightly outperformed gradient-boosted trees and artificial neural networks, which have previously recognized as reliable models of TSA outcome prediction¹⁵ (Table I). Using the random forest algorithm, we found that age, sex, ASA classification, and preoperative functional status were the most predictive variables of discharge destination following TSA.

Figure 3.

Algorithms ranked by AUC. AUC, area under the curve; ASA, American Society of Anesthesiologists.

Age was the most important variable for determining patients who had nonhome discharge to a nursing facility or rehab, with a feature importance of 0.340. Patients who were discharged home were on average 8 years younger compared to patients who were discharged to an SNF or rehab facility (68 vs. 76 years, P < .05). This finding could likely be attributed to increased frailty in the older patient population. As frailty has been associated with poorer outcomes following joint arthroplasty, discharge to a facility where a patient could receive supervised care and rehabilitation may help improve their surgical outcomes.¹⁹ These results are supported by other literature. Tang et al associated an increased age in the postoperative setting with a higher rate of rapid response events, mortality, and unforeseen intensive care unit stays for all surgery types.²¹ Another study by Gordon et al investigating nonhome destination risk factors for patients undergoing total hip arthroplasty using multivariate logistic regression models also identified age > 70 years as a risk for nonhome discharge (odds ratio 2.2).⁶ Given age as a risk factor of nonhome discharge, adequate preparation must be made preoperatively by patient counseling and incorporating case management early in the hospital admission for discharge to a an appropriate SNF/rehab facility.

ASA classification was the second most important variable identified as predictive of nonhome destination with a feature importance of 0.155. Patients who were discharged to nonhome facilities had a higher proportion of patients with high ASA scores compared to patients who were discharged home. Among patients discharged home, only 2.3% had an ASA classification of 4 or 5, whereas 8.7% of patients discharged to a facility had ASA 4 or 5. This finding is supported in other fields of orthopedics. In a study investigating elective adult spine deformity, Di Capua et al found that ASA class 3 or greater was associated with a 1.8 increased odds of a nonhome discharge.⁵ Additionally, ASA > 2 was independently associated with nonhome discharge in patients undergoing outpatient THA (odds ratio 2.74).⁶

Gender and functional status were also important variables in predicting nonhome discharge. Statistical analysis showed that a significantly higher proportion of patients with nonhome discharge were female (78.1%) compared to 52.7% female patients among those who were discharged home. While the direct relationship between the female sex and increased likelihood of nonhome discharge remains elusive, such findings are not unique to TSA. These findings are similar to other studies that investigated total knee and hip arthroplasty, identifying female gender as a risk factor for discharge to a facility.^2,11

Furthermore, only 1.3% of patients discharged home were classified by the NSQIP database as partially or totally dependent, while 10.5% of patients with nonhome discharge were classified as partially or totally dependent. Assuming preoperative functional status influences postoperative functional status; patients with lower functional status preoperatively would likely continue to have lower functional status postoperatively that could require nonhome rehabilitation at a facility.

Limitations

There are limitations to the study. First, participation in the NSQIP database is limited to hospitals and does not include outpatient surgery centers. As more TSA operations are being performed on an outpatient basis, the results of this study may not be a complete reflection of the total patient population undergoing TSA. The database may therefore be skewed toward having patients with higher risk of requiring nonhome discharge compared to patients in an outpatient surgery setting, inflating the rate of nonhome discharge. Additionally, patient variables are not independent from one another, with factors such as age and ASA likely influencing other baseline comorbidities. Our study did not distinguish between reverse TSA and TSA as they share the same Current Procedural Terminology code. Since these procedures can be done for different indications, it is possible that the results could be different depending on the indication for the shoulder arthroplasty. Finally, the NSQIP database did not contain marital or insurance status information which has been shown to influence discharge destination postoperatively.^17,25 Machine learning algorithms such as random forests are limited in that they are unable to establish causality. The important factors identified such as age and functional status are not independent from one another, with possible confounders.

Conclusion

Random forest outperformed other machine learning algorithms and ASA-based logistic regression in predicting discharge destination after TSA. When using a random forest algorithm to predict home versus nonhome discharge following TSA, age, ASA classification, sex, and functional status were found to be the most important variables in that order. Furthermore, 2 patient groups (discharged home, not discharged home) were significantly different when it came to distribution of ASA, sex, average age, and functional status. Machine learning algorithms continue to provide value in prediction of undesired outcomes following TSA. Given higher costs and risks of complications with nonhome discharge, orthopedic surgeons should risk stratify and counsel patients preoperatively for improved discharge planning based on identified risk factors.

Acknowledgments

The authors would like to acknowledge Elisabeth Clarke.

Disclaimers

Funding: No funding was disclosed by the authors.

Conflicts of interest: Wesley Phipatanakul is a consultant for Arthrex. The other authors, their immediate families, and any research foundation with which they are affiliated have not received any financial payments or other benefits from any commercial entity related to the subject of this article.