Machine learning used to determine features of importance linked to overnight stay after patellar tendon repair

Abstract

Purpose

The purpose of this study is to determine if machine learning is an effective method to identify features of patients who may need a longer postoperative stay following a patellar tendon repair.

Methods

The American College of Surgeons National Quality Improvement Program (ACS-NSQIP) was used to collect 1173 patients who underwent patellar tendon repair. Machine learning (ML) was then applied to determine features of importance in this patient population. Several algorithms were used: Random Forest, Artificial Neural Network, Gradient Boosting, and Support Vector Machine. These were then compared to the American Society of Anesthesiologists (ASA) classification system based logistic regression as a control.

Results

Random Forest (RF) was determined to be the best performing algorithm, with an AUC of 0.72, accuracy of 77.66 %, and precision of 0.79, and recall of 0.96. All other algorithms performed similarly to the control. RF gave the highest permutation feature importance to age (PFI 0.25), BMI (PFI 0.19), ASA classification (PFI 0.14), hematocrit (PFI 0.12), and height (PFI 0.11).

Conclusions

This study shows that machine learning can be used as a tool to identify features of importance for length of postoperative stay in patients undergoing patellar tendon repair. RF was found to be a better performing model than logistic regression at determining patients predisposed to longer length of stay as determined by AUC. This supported the study's hypothesis that ML can provide an effective method for identifying features of importance in patients requiring a longer postoperative stay after patellar tendon repair.

1. Introduction

While patellar tendon ruptures are relatively infrequent occurrences, only affecting approximately 0.5 % of the population of the United States annually,¹ their integral biomechanical role means this pathology must be urgently addressed. Patellar tendon ruptures often affect middle aged males (30–40 years old) and typically result from eccentric contraction of the quadriceps or high-energy impact to a flexed knee.² Surgical repair is considered the standard of care for treatment of patellar tendon rupture, and while these operations are often successful with excellent results after physical therapy,2, 3, 4 little has been done to study the acute postoperative time period or variables linked to increased length of stay.

Increased length of stay is known to lead to a higher chance of hospital acquired infections (HAI).⁵ The Centers for Disease Control and Prevention (CDC)⁶ identifies a range of HAIs, including pneumonia, surgical site infections, and urinary tract infections, among others. Such infections increase morbidity and mortality, as well as resource utilization.⁷ Each year, an estimated 2 million cases of HAI are identified,⁸ leading to an average of 90,000 deaths. Patients who experience postoperative complications also face a significant financial burden, with total payments ranging from $2436 to $5279⁹ higher than for those without complications. Furthermore, HAIs are a leading contributor to the ongoing antibiotic resistance crisis, with these multi-drug resistant infections being especially costly to the healthcare system.¹⁰

Patients who are required to stay in a hospital setting overnight also have increased risk for adverse drug reactions. A study performed in 2011 stated that patients carried approximately a 5.5 % increase for an adverse drug reaction for one overnight hospital stay, and another 0.5 % risk for each additional night.¹¹ Adverse drug events can increase the cost per case by up to $3420.¹² Furthermore, adverse drug events can increase patients’ hospital stays by 3.15 days, leading to a lack of mobility which increases the chance of thrombosis and blood clots.¹³

Machine learning (ML), a subfield of artificial intelligence, is a budding technique that can provide accurate and efficient predictions of clinically relevant outcomes through the analysis of large datasets. Classically, tabular patient data has been examined without ML using logistic regression, examining the character and strength of associated variables.¹⁴ Machine learning can perform better than traditional logistic regression because of its ability to extrapolate relationships based on datasets that were not used to train the original algorithm. The capabilities of ML have been previously illustrated in studies determining several pertinent patient outcomes including readmission, reoperation, and increased lengths of hospital stay.15, 16, 17 The extensive analysis provided by ML as well as its ability to quantify relative variable importance has led to its utilization throughout medical research. The purpose of this study was to employ various ML algorithms on a population that underwent patellar tendon repair to determine contributing factors to postoperative overnight stays. Our hypothesis is that ML is an effective method for identifying features of importance in patients requiring a longer postoperative stay after patellar tendon repair.

2. Methods

Entries from the American College of Surgeons National Quality Improvement Program (ACS-NSQIP) database with the CPT codes 27380 were included for data analysis in RStudio. The ACS-NSQIP database provided a robust patient population, capturing over 1 million cases from 700 hospitals annually.¹⁸ Patients who had traumatic injuries requiring concomitant surgery were excluded, with all other patients over the age of 18 years being included. The main outcome of interest for this screened population was overnight stays, which was defined as any patient who had a postoperative length of stay greater than or equal to 1 day. This was collected along with patient features including age, body mass index (BMI), American Society of Anesthesiologist (ASA) classification, hypertension requiring medication, preoperative lab values, hematocrit, height, smoking status, anesthesia, sex, diabetes, history of congestive heart failure, dialysis, ascites, dyspnea, renal failure, steroid use, weight loss, functional status in activities of daily living, history of bleeding disorders, and cancer. As a part of dataset preparation, patients with invalid or null values were excluded from the dataset for the ML algorithms to function properly.

This study employs supervised ML algorithms such as Random Forest (RF), Boosted Trees (BT), and Neural Networks (NN). Supervised ML algorithms in general can take an input of multiple variables such as demographic information and preoperative lab values to make outcome predictions, the specifics of which depend on the algorithm type. RF generates multiple independent decision trees and determine which trees most efficiently eliminate the impurity of a dataset. BT uses a similar method but builds sequential decision trees that attempt to correct errors of previous trees, allowing it to evaluate more complex relationships. NN is a deep learning algorithm that is inspired by the human brain, which uses connected nodes in a layered structure to identify underlying relationships in a set of data. Once a base model is created using one of the above techniques, permutation feature importance is used as a model inspection technique to determine how much a model depends on a certain feature. It achieves this by calculating the change in model error when a certain feature is omitted from analysis. If the model error is increased significantly, then the feature is considered crucial for algorithm performance and therefore given a higher feature importance.¹⁹

The above algorithms were coded and trained using Python 3 and the Scikit Learn package on Jupyter Notebook. To gauge the performance of the above algorithms, an ASA based logistic regression was used to predict overnight stays as a reference. Prior to the advent of machine learning algorithms, logistic regression was the basis of commonly used predictive models, such as ACS NSQIP mortality, TREAT, and the Mayo clinic model which provide a convenient baseline to test our ML algorithms against.²⁰

Algorithm performance was measured with metrics such as Area under the curve (AUC) of the receiver operating characteristic, accuracy, precision, recall, and F-1 scores. Recall is a metric that has an inverse relationship to precision and measures the proportion of actual positives identified correctly. The F-1 score is the mean of precision and recall, thus it represents both values in one metric. AUC, a value that considers both sensitivity and specificity, ranges in values from 0.0 to 1.0. Of these, AUC has been traditionally accepted as the most important measure²⁰. An AUC of less than 0.5 shows that the model cannot discriminate between correct and incorrect values, while values of 0.7–0.8 are considered acceptable; an AUC value of 0.8–0.9 is excellent, and above 0.9 is outstanding.²¹

Once the best performing algorithm was identified, preoperative variables considered most important by the algorithms of interest were further explored. Patient demographics and operative variables were analyzed based on order of permutation feature importance. The features determined to be most important by the algorithms were further tested for statistical significance using t-test for continuous variables and chi-squared for categorical variables. For t-tests, Cohen's d was used as effect size while for chi-square tests, the odds ratio was used as the effect size. For these calculations, alpha was set to 0.05 while beta was set to 0.80.

3. Results

After application of inclusion and exclusion criteria, a total of 1173 patients on the NSQIP database underwent patellar tendon repair between 2008 and 2018. Among these patients, 869 (74.1 %) were male and 304 (25.9 %) were female. The average BMI was 31.94 and average height of 68 inches. Average age was 45 years old. Only 213 (18.1 %) of the patients had a smoking history, while 960 (81.9 %) were non-smokers. 267 (22.8 %) patients required an overnight admission while 906 patients were discharged home the day of surgery. ASA classifications were divided as ASA 1 (n = 380, 32.4 %), ASA 2 (n = 577, 49.1 %), ASA 3 (n = 203, 17.3 %), and ASA 4 (n = 13, 1.1 %) (Table 1).

Table 1.

Demographics of the patient population broken down into individuals who did not stay overnight after patellar tendon repair and those who did. P-values are recorded as either >0.05 being non-significant and <0.05 being statistically significant.

	Outpatient	Overnight admission	P-value
Total
Gender (%)			<0.05
Male	790	79
Female	116	188
BMI	31.08	32.8	>0.05
Height	68.3	67.6	>0.05
Age (years)	39.8	49.3	<0.05
ASA (%)			<0.05
1	325	55
2	470	107
3	107	96
4	4	9
Smoking history			>0.05
No	738	222
Yes	168	45
Diabetes History			>0.05
No	854	228
Yes-no insulin use	24	20
Yes-insulin use	28	19

While logistic regression based on ASA classification did provide some predictive ability, all ML algorithms used had a similar or higher AUC. (Table 2). Of these algorithms, Random Forest was the best performing algorithm with an AUC of 0.72 (Fig. 1), followed by Artificial Neural Network (AUC 0.68), Gradient Boosting Trees (AUC 0.67), Support Vector Machine (AUC 0.65), and ASA based Logistic Regression (AUC 0.65) none of which met the 0.7 threshold for acceptable validity. Random Forest also had the highest accuracy (77.66 %) and precision (0.79 %) (Table 2).

Table 2.

All algorithms used to test for postoperative length of stay following patellar tendon repair and their outcomes. AUC-ROC was identified as the most important measurement, with a criteria of >0.7 signifying a functional algorithm.

	AUC of Receiver Operating Characteristic	Accuracy	Precision	Recall	F-1 score
Random Forest	0.72	77.66 %	0.79	0.96	0.85
MPClassifier	0.69	78.39 %	0.79	0.96	0.87
Artificial Neural Network	0.68	75.82 %	0.78	0.95	0.85
Gradient Boosting	0.67	77.29 %	0.79	0.94	0.86
Support Vector Machine	0.65	74.73 %	0.75	1.00	0.85
ASA Based Logistic Regression	0.65	74.73 %	0.75	1.00	0.85

Fig. 1.

Graphic showing the AUC-ROC for all ML algorithms used and logistic regression of ASA. The dotted red line represents a classifier that is no better than random inputs. RF was found to have the best AUC of 0.72 compared to logistic regression AUC of 0.65.

Random Forest assigned highest permutation feature importance to age (PFI 0.25), BMI (PFI 0.19), ASA classification (PFI 0.14), hematocrit (PFI 0.12), and height (PFI 0.11) (Fig. 2). Only 9.1 % of males had an overnight stay, compared to 61.8 % of females (p < 0.05). The inpatient group was on average 49.3 years old compared to 39.8 years old for patients who had outpatient surgery (p value < 0.05). ASA classifications were also statistically different between outpatient and inpatient groups, where 87.7 % of outpatient surgeries were ASA 1 or 2, compared to only 60.7 % of inpatient cases being ASA 1 or 2 (p value < 0.05).

Fig. 2.

Features ranked by importance by the random forest ML algorithm, with age as the most important feature. The rating given to each feature as seen on the x-axis represents the relative importance of each item. The three highest rated features are age (PFI 0.25), bmi (PFI 0.19), and ASA class (PFI 0.14).

4. Discussion

The present study employed ML algorithms identify factors that may lead to increased length of stay following patellar tendon repair and identify contributing factors. The results demonstrate the potential of ML algorithms, particularly RF, in identifying features of importance which would allow healthcare professionals to better optimize patient care and resource management.

Among the ML algorithms tested, RF emerged as the best-performing algorithm, achieving an area under the curve (AUC) of 0.72, an accuracy of 77.66 %, precision of 0.79, recall of 0.96, and an F-1 score of 0.85. As discussed previously, the AUC of a given algorithm is the most indicative metric of a predictive performance, with literature demonstrating an AUC of 0.7–0.8 being acceptable²¹. With RF validated against comparable analytic methods, data from NSQIP was inputted, with age, BMI, ASA class, hematocrit, and height found to be the most influential features of importance in this study regarding prolonged admissions.

Age has consistently been associated with increased risk of complications and extended hospital stays in various surgical procedures.^22,23 Our model supports this statement by showing a significant difference between the ages of the inpatient and outpatient groups, being 49.3 versus 39.8 years old on average. It is well documented that older individuals have a decreased wound healing ability.²⁴ The circulatory system also becomes less efficient with age, prolonging the time it takes for vital nutrients and resources to reach any postoperative damage. While these factors may contribute to longer monitoring periods for postoperative patients, the more likely explanation is a higher number of medical comorbidities in older patients that increase overall frailty, requiring more intensive inpatient management.²⁵

The influence of BMI on surgical outcomes is well-documented, with obesity being a known risk factor for postoperative complications.^26,27 BMI was not statistically different between inpatient and outpatient groups; however the RF algorithm classified it as the second most important feature for predicting overnight admissions. While the p-value for each variable studied can find possible relationships in data, this statistical process is not infallible. RF can see nonlinear relationships between feature and outcome by rewarding variables that optimize the gini score by reducing impurity. Therefore, some variables considered important may not necessarily be statistically significant. This does not indicate the feature is in fact not important, but instead demonstrates the power of ML to find important features leading to overnight admission that are missed by more classic statistical methods.

According to the American Society of Anesthesiologists, ASA class is a measure of preoperative physical status that provides valuable information regarding a patient's overall health and comorbidities, with class I indicating a healthy patient and class V indicating an individual who is not expected to survive without surgical intervention. This system was created to subjectively evaluate perioperative risk and has been in use for over 60 years. Since it has been widely accepted as a grading system of operative risk, it provided a strong benchmark for our ML algorithms through the ASA logistic regression. Beyond providing a control, RF determined ASA class to be a feature of importance. This is further evidence that ASA classification does work, as RF found it to be linked to postoperative length of stay. However, it should be noted that it was only the third most important feature behind age and BMI. There has been more recent speculation that the ASA classification system may not provide the most accurate representation of preoperative physical status due to its subjective nature. Several studies have demonstrated a discordance in selecting patient ASA status between anesthesiologists and surgeons.28, 29, 30, 31 It is also important to acknowledge the system was developed in 1941 and has since been used for a plethora of medical specialties and providers beyond its original intention. Several modifiers have been added to try and make the system more specific, such as “E” for emergency or “G” for pregnancy but are not consistently used.³² Our model finding ASA classification as a helpful part of determining postoperative length of stay provides further support that the system is subjective and may need to be updated by more modern solutions such as ML algorithms.

With patellar tendon rupture being a relatively rare occurrence, there are many gaps in knowledge regarding patient care. One of these gaps is regarding the acute inpatient recovery of patients after patellar tendon repair. The majority of studies focus on either outpatient rehabilitation³ or long-term recovery, examining Lysholm scores at follow up several years after surgery.^2,33 Because of this, there is a lack of similar studies to compare our data to.

5. Limitations

A weakness is that variables available in the NSQIP database may be missing key features such as ethnicity, race, geographical location, etc., all of which could affect the results and limits the generalizability of this study. Furthermore, the algorithms did not consider different surgical approaches, type of fixation, or hardware used during the procedure. Patients were selected based on CPT code 27380 which stipulates a primary repair of the infrapatellar tendon but does not further distinguish the techniques used to repair the patellar tendon. An additional limitation of using NSQIP is the wide array of participating facilities, from level 1 trauma centers to rural surgical centers, where resource or bed availability can play a significant factor in if patients are kept postoperatively. A final limitation is the inability to determine which, if any, patients had overnight stays that were planned before the surgery took place. A future study could further delineate between patients who had planned postoperative stays and those who were not, determining if there are any major differences in patient features between these two groups.

6. Conclusion

This study shows that machine learning can be used as an accurate tool determine risk factors for length of postoperative stay after patellar repair. RF was found to be a better performing model than logistic regression of ASA class at determining patients predisposed to longer length of stay as demonstrated by the higher AUC. It then evaluated numerous patient features and determined those that were most related to overnight admissions. This supported the study's hypothesis that ML can provide an effective method for identifying features of importance in patients requiring a longer postoperative stay after patellar tendon repair.

Statements and declarations

Declarations of interest: none. No outside funding was acquired.

Ethical statement

No direct experimentation or interaction with patients was necessary for this project. All data was collected ethically via the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) database.

Funding statement

No contributing authors received funding from any outside sources.

Guardian/patient consent

All data was collected from the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP), which is a deidentified data source. Project was approved by the Loma Linda School of Medicine IRB.

Conflict of interest

There are no conflicting interests to disclose for any authors involved in this submission. No financial disclosures.

CRediT authorship contribution statement

Luke Thomas: Conceptualization, Investigation, Project administration, Writing – original draft, Writing – review & editing, Visualization. Jun Ho Chung: Methodology, Software, Formal analysis, Validation. Sarah Lu: Writing – original draft, Writing – review & editing. Anthony Essilfie: Supervision, Project administration, Conceptualization.

Declaration of generative AI and AI-assisted technologies in the writing process

None used.