Prediction of Postoperative Delirium in Geriatric Hip Fracture Patients: A Clinical Prediction Model Using Machine Learning Algorithms

Abstract

Introduction

Postoperative delirium in geriatric hip fracture patients adversely affects clinical and functional outcomes and increases costs. A preoperative prediction tool to identify high-risk patients may facilitate optimal use of preventive interventions. The purpose of this study was to develop a clinical prediction model using machine learning algorithms for preoperative prediction of postoperative delirium in geriatric hip fracture patients.

Materials & Methods

Geriatric patients undergoing operative hip fracture fixation were queried in the American College of Surgeons National Surgical Quality Improvement Program database (ACS NSQIP) from 2016 through 2019. A total of 28 207 patients were included, of which 8030 (28.5%) developed a postoperative delirium. First, the dataset was randomly split 80:20 into a training and testing subset. Then, a random forest (RF) algorithm was used to identify the variables predictive for a postoperative delirium. The machine learning-model was developed on the training set and the performance was assessed in the testing set. Performance was assessed by discrimination (c-statistic), calibration (slope and intercept), overall performance (Brier-score), and decision curve analysis.

Results

The included variables identified using RF algorithms were (1) age, (2) ASA class, (3) functional status, (4) preoperative dementia, (5) preoperative delirium, and (6) preoperative need for mobility-aid. The clinical prediction model reached good discrimination (c-statistic = .79), almost perfect calibration (intercept = −.01, slope = 1.02), and excellent overall model performance (Brier score = .15). The clinical prediction model was deployed as an open-access web-application: https://sorg-apps.shinyapps.io/hipfxdelirium/.

Discussion & Conclusions

We developed a clinical prediction model that shows promise in estimating the risk of postoperative delirium in geriatric hip fracture patients. The clinical prediction model can play a beneficial role in decision-making for preventative measures for patients at risk of developing a delirium. If found to be externally valid, clinicians might use the available web-based application to help incorporate the model into clinical practice to aid decision-making and optimize preoperative prevention efforts.

Introduction

The most common complication following geriatric hip fracture surgery is a postoperative delirium.^1-5 Occurrence of a postoperative delirium is associated with reduced cognitive performance, other major postoperative complications, loss of functional independence, increased morbidity, higher patient costs, prolonged hospitalization, and decreased overall quality of life. ⁶ Patients developing postoperative delirium have 2.5 times greater costs than those without, with additional costs ranging $16,000 to $64,000 per patient-year. ⁷

Delirium is a complex neuropsychiatric syndrome characterized by acute and fluctuating course, inattention, altered level of consciousness, and evidence of disorganized thinking. ⁸ The results of a recent meta-analysis examining risk factors for delirium showed that patients with existing cognitive impairment, advancing age, living in an institution, heart failure, total hip arthroplasty, multiple comorbidities, and morphine use were more likely to experience delirium after hip surgery. ⁹

Preoperative risk stratification for delirium is needed to propose effective prevention measures. Previous studies have identified risk factors for delirium, but few have developed models for preoperative prediction of postoperative delirium. Numerous preventative strategies exist, including preoperative optimization of clinical condition, careful surgical and medical co-management, and appropriate postoperative support.^10,11 Despite this knowledge regarding prevention and management of this important and debilitating condition it is under-recognized and mis-managed, which is perturbing as delirium can be prevented with focused multicomponent interventions amongst at-risk hospitalized elderly patients.^12-14

To our knowledge, there are no studies that have explored machine learning (ML) for prediction of delirium in the hip fracture population, although such methodology has shown promise for other outcomes in orthopedic surgery.^15-22 In orthopedic trauma, there is growing interest in applying ML as a statistical method for developing a clinical prediction model. The computer trains an existing human-created algorithm to recognize patterns in the data and optimizes their own performance.

The aim of this study was to develop a clinical prediction model using ML algorithms for preoperative prediction of postoperative delirium in geriatric hip fracture patients. The secondary aim of this study was to deploy the best performing clinical prediction model as an open access web-application.

Materials and Methods

Guidelines

The study was conducted according to the Transparent Reporting of Multivariable Prediction Models for Individual Prognosis or Diagnosis Guideline (TRIPOD-Statement). ²³

Data Source

We utilized the 2016 through 2019 American College of Surgeons (ACS) and American Association of Orthopaedic Surgeons (AAOS) National Surgical Quality Improvement Program (NSQIP) Hip Fracture Procedure Targeted files. National Surgical Quality Improvement Program is a large clinical database that collects more than 150 variables (pre-, peri-, and postoperatively) up to 30 days following surgery of more than 680 US hospitals combined. The series undergoes routine auditing, which ensures high-quality data with reported inter-reviewer rate of less than 2%. ²⁴ Hip Fracture Procedure Targeted Files includes additional factors that are disease and procedure specific for hip fracture patients. These files were queried to identify patients older than 60 years of age who underwent surgery for femoral neck, intertrochanteric, and subtrochanteric hip fractures.

Primary Outcome—Outcome of Interest

The primary outcome was postoperative delirium within 30 days following hip fracture surgery as defined by the Hip Fracture Targeted ACS-NSQIP files. A chart-based method has been used for determining delirium; this method has been previously validated in surgical and non-surgical specialties.^25-28 Data abstractors from ACS-NSQIP routinely examine the entire medical record, and are instructed to assign postoperative delirium if the medical record consists phrases characteristic of an acute confusional state: “mental status change, confusion, disorientation, agitation, delirium, inappropriate behavior, inattention, hallucination, combative (eg, pulling out lines or tubes)”. ²⁵ The chart-based method has the advantage to detect delirium which may be missed when administrating with a standardized tool, for a delirium that may occur during nights or weekends. The first 2 h postoperative were excluded and not coded as delirium in order to separate from emergence delirium. ²⁹

Candidate Input Variables

Variables that were considered potentially important for predicting delirium included age (years), gender (female/male), body mass index (BMI) (kg/m²), fracture type (femoral neck Garden types 1 and 2—non-displaced/femoral neck Garden type 3 and 4-displaced/intertrochanteric/subtrochanteric), procedure (ORIF femoral neck, ORIF inter/subtrochanteric with plate/screw implant, ORIF inter/subtrochanteric with intramedullary implant), American Society of Anesthesiologist (ASA) class (I/II/III/IV), functional status (independent defined as patients do not require assistance from another person for any activities of daily living/partially depended defined as patients requires some assistance from another person for activities of daily living (including patients who utilize prosthetics, equipment, or devices but still requires some assistance from another person)/totally dependent if the patient requires total assistance for all activities of daily living), preoperative dementia as having cognitive impairment, dementia, or predefined descriptors consistent with dementia documented by a nurse or doctor stated (yes/no), preoperative delirium assessed by the chart-based method (yes/no), preoperative bone protection medication prescription (yes/no), preoperative need of mobility aid, for example, cane, walker, wheelchair, or scooter (yes/no), medical co-management (no/yes, co-management throughout stay/yes, partial co-management during stay), standardized hip fracture protocol (yes/no), diabetes (insulin dependent/non-insulin dependent/no), smoking (yes/no), dyspnea (at rest/moderate exertion/no), chronic obstructive pulmonary disorder (COPD) (yes/no), congestive heart failure (yes/no), hypertension requiring medication (yes/no), acute renal failure (yes/no), dialysis (yes/no), disseminated cancer (yes/no), wound infection (yes/no), preoperative steroid use (yes/no), weight loss >10% body weight in last 6 months (yes/no), bleeding disorder (yes/no), transfusion in 72 hours prior surgery (yes/no), systemic sepsis within 48 hours prior surgery (none/SIRS/sepsis), sodium (mg/dL), creatinine (mg/dL), white blood cell (×10³/µL) and hematocrit (%), platelet (×10³/µL). Categorical variables will be described as absolute numbers with frequencies, and continuous variables as medians with interquartile ranges (IQR).

Variable Selection

As a first step, variables potentially associated with postoperative delirium were identified using random forest (RF) algorithms with recursive selection.

Missing Data

After variable selection, variables with less than 30% missing values were imputed using multiple imputation with the missForest methodology. ³⁰ Rates of missing data were as followed: BMI 4247 (14.2%), ASA class 45 (.2%), preoperative delirium 345 (1.2%), functional status 198 (.7%) and preoperative need for mobility-aid 1259 (4.2%). Of preoperative laboratory values the rates of missing data resulted: preoperative sodium 187 (.6%), preoperative creatinine 199 (.7%), white blood cell count 175 (.6%), hematocrit 144 (.5%), and platelet 200 (.7%).

Development of the Clinical Prediction Model

We trained and internally validated several ML algorithms based on prior research: Stochastic Gradient Boosting (SGM), RF, Support Vector Machine, Neural Network (NN) and Elastic-Net Penalized Logistic Regression (PLR). First, the dataset was randomly split in 80:20 into a training and testing subset. The training set (n = 22 563) was used for model training to predict postoperative delirium (ie, the outcome of interest). Then, we trained each model using 10-fold cross-validation in the training set, and the model performance was assessed the test set (n = 5641). Cross-validation means dividing the data into a selected number of groups, named folds. Results are subsequently averaged across all repetitions of this sequence. ³¹

Model Performance

Model performance was evaluated according to a proposed framework for evaluation of a clinical prediction model ³² that includes: (1) discrimination with the c-statistic, (2) calibration with calibration slope and intercept (in line with the method by Cox ³³ ) and (3) the overall performance with the Brier score.

The c-statistic (area under the curve of a receiver operating characteristic curve) is a score ranging from .50 to 1.0 with 1.0 indicating the highest discrimination score and .50 indicating the lowest. The higher the discrimination score, the better the model’s ability to distinguish patients who got the outcome from those who did not. ³⁴

A calibration plot plots the estimated vs the observed probabilities for the primary outcome. A perfect calibration plot has an intercept of 0 (<0 reflects overestimation, >0 reflects underestimating the probability of the outcome) and a slope of 1 (model is performing similarly in training and test-sets).^32,35 In a small dataset, slope is often <1 reflecting model overfitting; probabilities are too extreme (low probability too low, high probability too high). ³⁴

The Brier score calculates a composite of discrimination and calibration, with 0 indicating perfect prediction and a Brier score of 1 the poorest prediction. ³²

In addition, a decision curve analysis was undertaken and visualized to investigate the net benefit (weighted average of true positives and false positives) of the conducted algorithms over the range of predicted probabilities. ³⁶

The best model was deployed as an open-access web-application accessible on desktops, tablets, and smartphones.

Software

Data pre-processing and analysis were performed using R Version 5.3 (“R: A Language and Environment for Statistical Computing” The R Foundation, Vienna, Austria 2013) and R-studio Version 1.2.1335 (R-Studio, Boston, MA, USA).

Level of Evidence

Prognostic Level III

Results

In total, 28 207 patients underwent surgery for a hip fracture of which 8030 (28.5%) developed a delirium within 30 days after surgery (Table 1). The majority of patients was female (n = 19 845, 70.4%) and aged 80 years or older (n = 18 809, 66.6%).

Table 1.

Baseline Characteristics of NSQIP Hip Fracture Population, n = 28 207.

Variable	n (%) | Median (IQR)
Age (years)
60+	3151 (11.2)
70+	6247 (22.1)
80+	11 691 (41.4)
90+	7118 (25.2)
Female sex	19 845 (70.4)
Body mass index (kg/m2)	24.3 (21.6–27.3)
Fracture type
Femoral neck fracture (sub capital, Garden types 1 and 2)-nondisplaced	2479 (8.8)
Femoral neck fracture (sub capital, Garden types 3 and 4)-displaced	8324 (29.5)
Intertrochanteric	15 761 (55.9)
Subtrochanteric	1642 (5.8)
ASA classification
I	126 (.4)
II	4162 (14.8)
III	17 631 (62.5)
IV	6288 (22.3)
Functional status
Independent	21 672 (76.8)
Partially dependent	5651 (20.0)
Totally dependent	884 (3.1)
Preoperative dementia	8668 (30.7)
Preoperative delirium	3714 (13.2)
Preoperative bone protective medication prescription	9047 (32.1)
Preoperative need for mobility aid	16 239 (57.6)
Preoperative pressure sore	971 (3.4)
Medical co-management
No	3071 (10.9)
Yes-co-management throughout stay	20 576 (72.9)
Yes-partial co-management during stay	4560 (16.2)
Standardized hip fracture protocol	15 808 (56.0)
Diabetes
Insulin dependent	2026 (7.2)
Non-insulin dependent	3035 (10.8)
No	23 146 (82.1)
Smoking	2848 (10.1)
Dyspnea
At rest	273 (1.0)
Moderate exertion	1880 (6.7)
No	26 054 (92.4)
Chronic obstructive pulmonary disorder	3035 (10.8)
Congestive heart failure	1051 (3.7)
Hypertension requiring medication	19 120 (67.8)
Acute renal failure	150 (.5)
Dialysis	522 (1.9)
Disseminated cancer	407 (1.4)
Wound infection	1095 (3.9)
Preoperative steroid use	1476 (5.2)
Weight loss >10% body weight in last 6 months	384 (1.4)
Bleeding disorder	4796 (17.0)
Transfusion	1205 (4.3)
Systemic inflammatory response syndrome (SIRS)
None	25 478 (90.3)
SIRS	2566 (9.1)
Sepsis	163 (.6)
Sodium (mg/dL)	138.0 (136.0–140.0)
Creatinine (mg/dL)	.88 (.70–1.13)
White blood cell (×103/µL)	9.60 (7.60–11.90)
Hematocrit (%)	35.0 (31.0–38.6)
Platelet (×103/µL)	196.0 (156.0–244.0)
Postoperative delirium	8030 (28.5)

n = number; IQR = interquartile range; ASA = American Society of Anesthesiologist.

Variables Selection

Variables selection identified the following predictive variables for prediction of postoperative delirium: (1) age, (2) ASA class, (3) functional status, (4) preoperative dementia, (5) preoperative delirium, and (6) preoperative need for mobility-aid (Figure 1).

Figure 1.

(A) Receiver operating curve and (B) global variable importance for the elastic-net penalized logistic regression for prediction of postoperative delirium in the testing set, n = 5641.

Model Performance

The model performance of the conducted algorithms varied as measured by c-statistic from .67 to .79. The calibration slopes ranged from .44 to 1.02, and the calibration intercepts ranged from −.01 to 1.06. The Null model Brier score was .20, calculated based on an incidence of 28.5%. The Brier scores ranged from .15 to .18 (Table 2).

Table 2.

Machine Learning Model Performance Assessment in the Testing Set, n = 5641.

Metric	Stochastic Gradient Boosting	Random Forest	Support Vector Machine	Neural Network	Elastic-Net Penalized Logistic Regression
C-statistic	.79 (.77, .80)	.71 (.73, .77)	.67 (.68, .71)	.79 (.77, .80)	.79 (.77, .80)
Intercept	−.01 (−.08, .06)	1.06 (.95, 1.17)	−.01 (−.01, .01)	−.01 (−.08, .06)	−.01 (−.07, .06)
Slope	.97 (.91, 1.03)	.44 (.39, .49)	.92 (.85, .98)	.96 (.90, 1.02)	1.02 (.96, 1.09)
Brier	.15	.18	.16	.15	.15

Values are given in with the 95% confidence interval in parentheses. Null model Brier score = .20.

Although model performance metrics were comparable, except the RF algorithm, the PLR model had the most clinically meaningful variable importance and was therefore chosen as the best algorithm and therefore the final model (Figure 1).

Decision curve analysis of the PLR model revealed that decision changes based on the model outperformed not only the default strategies of decision change looking at all patients or no patients, but also that the decision changes based on presence of only a preoperative delirium alone (Figure 2).

Figure 2.

(A) Calibration plot and (B) decision curve analysis for the elastic-net penalized logistic regression for prediction of postoperative delirium in the testing set, n = 5641. Decision curve analysis with net benefit achieve by management changes based on the PLR algorithm relative to default strategies and for those based on solely presence of preoperative delirium.

Available Web-Application

The PLR algorithm was incorporated into a web-based application and deployed as an open-access available tool for clinicians: https://sorg-apps.shinyapps.io/hipfxdelirium/

As an example, a patient older than 90 years of age is scheduled for hip fracture surgery. The patient is classified as ASA III, the patient lives dependent, and has preoperative need for mobile-aid. The patient has a history of dementia, but does not have preoperatively delirium. After filling out these values in the clinical prediction model in the available tool, this patient has a 42% chance of developing a delirium postoperatively. An explanation of the prediction is given by the available tool as shown in Figure 3, to overcome the “black box” of the developed model (How did this decision support tool came to this prediction?). In our example, factors increasing the likelihood were preoperative dementia, age, preoperative need for mobility-aid, dependent functional status, and ASA class. However, the lack of preoperative delirium reduced the likelihood of a delirium following hip fracture surgery.

Figure 3.

Example of individual patient-level explanation for postoperative delirium.

Discussion and Conclusions

We have developed a clinical prediction model using ML algorithms for prediction of delirium in geriatric patients undergoing hip fracture surgery with almost perfect performance metrics. The clinical prediction model is based on patient characteristics and may guide clinicians to identify high-risk patients at a preoperative stage. Variables predictive for postoperative delirium were the following: age; ASA class; functional status; preoperative dementia; preoperative delirium; and preoperative need for mobility-aid. The clinical prediction model with the best performance (PLR) showed good discrimination (c-statistic = .79), almost perfect calibration metrics (intercept = −.01, slope = 1.02), and good overall performance (Brier score = .15).

To the best of our knowledge, this is the first study applying ML algorithms for prediction of postoperative outcome following hip fracture surgery. In addition, this is the largest sample size thus far in predicting postoperative delirium in geriatric hip fracture surgery. ³⁷ Predicting postoperative delirium is a challenging problem, but multicomponent intervention strategies can be effective in reducing rates and length of postoperative delirium.^13,38 The decision curve analysis allows for the comparison of selectively changing management for patients based on the clinical prediction model’s predicted probabilities as compared to the default strategies of changing management for all patients or for no patients, or for patients screened based on the presence of postoperative delirium alone. In our opinion, the results of this study show that the developed ML prediction tool could play a beneficial role in decision-making for preventative measures for patients at risk of developing a delirium.

Our findings are in line with previous research. A meta-analysis from 2017 found moderate evidence for greater probability of the occurrence of postoperative delirium following hip fracture surgery for patients aged 80 years or older, higher ASA class (>2), functional status and presence of pre-admission diagnosis of dementia. ³⁷ A similar conclusion was reached by studies published more recently, where increasing age and cognitive impairment were also found as predictors.^1,13,39 Mossello et al ⁴⁰ have shown that moderate renal impairment (estimated by eGFR) was independently associated with the occurrence of delirium in older fracture patients (>75 years), based on serum creatinine and/or cystatin C in preoperative laboratory results.

When aiming to develop a prediction model that is applicable in daily practice, variables should be included in the trained algorithm that are readily available and use of definitions that are in line with daily practice should be followed. In this study, variables derived from variable selection are clinically readily available and in line with daily practice, where only preoperative creatinine level should be carried out which is mostly standardized collected prior surgery.

The results of this study should be viewed in light of several limitations. First, this was a retrospective study beholden to limitations inherent to such a research design. Second, the data was derived from the NSQIP Targeted Hip Fracture database and results may not be generalized to the (inter)national population. Third, the NSQIP Targeted Hip Fracture database does not provide information on patients treated with hemiarthroplasty or total hip arthroplasty which is often indicated in older and dependent patients and is associated with higher delirium rate. ⁹ This could lead to underestimation of the developed algorithm when applying to an independent population for prediction of postoperative delirium after hip fracture surgery in patients older than 60 years of age. Because of aforementioned limitations, external validation of the algorithm will support generalizability of results. Fourth, NSQIP collects data up to 30 days postoperatively and delirium may occur beyond this time-frame. Fifth, by nature of selection from participating ACS NSQIP hospitals, data may be subject to selection bias. Sixth, NSQIP is a large clinical database and collects more than 150 variables; however, the NSQIP does not account for specific factors such as preoperative use of medication and the use of cognitive screening instruments and/or interventions. Seventh, development of a prediction tool should be developed with the ability to enhance clinical care. Predicting patients at high risk is important, which may already be anticipated by the clinician, but maximum value may be obtained for those at moderate risk with the use of a prediction model. ⁴¹ However, treating all patients as at-risk with non-pharmacological and multicomponent interventions can do no harm, since use of medications to prevent delirium is not yet supported in prevention of developing a delirium following hip fracture surgery. ⁴² Eighth, preoperative risk stratification for delirium is needed to propose early effective prevention measures, although intraoperative and postoperative factors associated with postoperative delirium may be confounding with delirium prediction after surgery. ⁴³ Lastly, current AUC of .79 is considered good, suggesting 79% chance that the model will correctly distinguish between patients who developed a postoperative delirium from those who did not. ⁴⁴ In addition, the model showed almost perfect calibration which is considered more important in prognostic settings. ⁴⁵ External validation is essential before testing and implementing in clinical practice. External validation can be carried out with temporal, geographical or fully independent validation. ⁴⁶

Conclusion

In summary, the developed SORG model effectively predicted delirium in geriatric hip fracture patients with good model performance. Our model would likely improve the efficiency of a screening program aimed to identify patients at risk for delirium since it outperformed the default strategy of screening all patients, screening no patients, and screening based on the presence of preoperative delirium alone. We will seek to externally validate our model in an independent data set.

ORCID iD

Jacobien H. F. Oosterhoff https://orcid.org/0000-0002-3782-2791