Optimizing Predictive Strategies for Acute Kidney Injury after Major Vascular Surgery

Abstract

Background:

Postoperative acute kidney injury (AKI) is common after major vascular surgery, and is associated with increased morbidity, mortality and cost. High-performance risk stratification using a machine learning model can inform strategies that mitigate harm and optimize resource use. It is hypothesized that incorporating intraoperative data would improve machine learning model accuracy, discrimination and precision in predicting AKI among patients undergoing major vascular surgery.

Methods:

A single-center retrospective cohort of 1,531 adult patients who underwent non-emergent major vascular surgery including open aortic, endovascular aortic, and lower extremity bypass procedures was evaluated. The validated, automated MySurgeryRisk analytics platform used electronic health record data to forecast patient-level probabilistic risk scores for postoperative acute kidney injury using random forest models with preoperative data alone and perioperative data (preoperative plus intraoperative). The MySurgeryRisk predictions were compared with each other as well as with the American Society of Anesthesiologists (ASA) physical status classification.

Results:

Machine learning models using perioperative data had greater accuracy, discrimination and precision than models using either preoperative data alone or ASA physical status classification (accuracy: 0.70 vs. 0.64 vs. 0.62, AUROC: 0.77 vs. 0.68 vs. 0.61, AUPRC: 0.70 vs. 0.58 vs. 0.48).

Conclusions:

In predicting AKI after major vascular surgery, machine learning approaches that incorporate dynamic intraoperative data had greater accuracy, discrimination and precision than models using either preoperative data alone or ASA physical status classification. Machine learning methods have the potential for real-time identification of high-risk patients who may benefit from personalized risk-reduction strategies.

INTRODUCTION

Postoperative acute kidney injury (AKI) affects up to 30% of all surgical patients¹ and is independently associated with increased risk for other complications, short- and long-term mortality, hospital costs, and resource use.^2–5 In the vascular surgery population, even small changes in creatinine are associated with increased morbidity, mortality, and health care costs.^6–8 Several preoperative risk factors, including patient demographics, comorbidities, type of surgery, medication use, and biomarkers are associated with postoperative AKI.⁹ Recently, intraoperative variables, such as procedure duration, fluid balance, blood product transfusion, and dynamic changes in blood pressure and heart rate have also been shown to be important features contributing to the development of postoperative AKI.^10–12

Perioperative risk stratification can encourage high-risk patients to engage in personalized preoperative risk-reduction strategies and inform clinicians regarding the utility of targeted preventative strategies and surveillance for complications.¹³ However, traditional risk stratification processes are not uniform and depend greatly on surgeon preference and the patient’s baseline risk factors and history. Various clinical risk stratification models and tools have been developed to generate standardized, data-driven risk predictions. However, most of these models, including the American Society of Anesthesiologists (ASA) physical status classification system, rely on subjective assessments of preoperative health and do not use intraoperative physiologic data.¹⁴ The omission of intraoperative data may be particularly disadvantageous when predicting AKI after vascular surgery, as intraoperative hypotension is a major risk factor.^15–17 In addition, traditional models require time-consuming manual data entry, which is a major barrier to clinical integration.¹⁸ There is a critical need for efficient, high-performance models that predict AKI after major vascular surgery.

This longitudinal, retrospective cohort of 1,531 patients undergoing open aortic, endovascular aortic, and lower extremity bypass procedures, evaluated the ability of the validated MySurgeryRisk platform to predict postoperative AKI using machine learning models that incorporate both preoperative and intraoperative (i.e., perioperative) data. It is hypothesized that perioperative input data would improve machine learning model accuracy, discrimination and precision in predicting AKI after major vascular surgery compared with traditional risk models and machine learning models using preoperative input data alone. Additionally, it is hypothesized that intraoperative hypotension would make substantial contributions to AKI predictions, representing an important modifiable risk factor.

METHODS

Materials and Methods

A retrospective, single-center cohort of patients undergoing non-emergent vascular surgery was created with data from preoperative, intraoperative and postoperative phases of care. Random forest¹⁹ machine learning models were used to predict postoperative complications using preoperative data alone versus models using the same preoperative data plus intraoperative data (i.e., perioperative data). Intraoperative data included physiological vital signs and mechanical ventilator data. The University of Florida Institutional Review Board (#201600223) approved this study with waiver of informed consent. The statistical analysis and machine learning models were performed using Python and R (ggplot2, pROC) software.

Data Source and Participants

The University of Florida Integrated Data Repository was used to assemble a single-center cohort for all patients admitted to the University of Florida Health for longer than 24 hours following any type of procedure between June 1^st 2014 through March 1^st 2019 by integrating electronic health records (EHR) with other clinical, administrative and public databases, as previously described.²⁰ The 218,533 operations were filtered by vascular surgery attending and categorized as open aortic, endovascular aortic, or lower extremity bypass according to CPT codes (eTable 1).²¹ If patients underwent multiple surgeries during one admission, then only the first surgery was used in the analysis. A total of 1,531 adults undergoing 1,631 inpatient, non-emergent vascular surgeries were split into training and test cohorts chronologically, using surgeries from June 2014 to February 2018 for training and February 2018 to March 2019 for testing. Four hundred and twenty-seven index procedures were included in the test set (141 open aortic, 136 endovascular aortic and 150 lower extremity bypass) (eFigure 1). A chronologic split was performed to allow nonrandom variation between data sets, as described by Collins et al.²²

Outcomes

The primary outcome of interest was the development of AKI (defined using the consensus Kidney Disease: Improving Global Outcomes (KDIGO) criteria as at least a 50% or 0.3 mg/dl increase in serum creatinine relative to the reference creatinine after surgery and before discharge.²³ The preoperative (e.g. demographic, socioeconomic, comorbidities, neighborhood) and intraoperative (e.g. physiological time-series) characteristics used to build the predictive models have been previously defined²⁴ and are shown in eTable 2 stratified by training and testing cohort and in eTable 3 stratified by postoperative AKI. The prevalence of postoperative complications including mechanical ventilation (MV) and intensive care unit (ICU) admission for greater than 48 hours, cardiovascular (CV) complications, neurological complications, sepsis and venous thromboembolism (VTE) for patients with and without AKI was also evaluated.

Predictive Analytic Workflow

MySurgeryRisk is an automated EHR algorithm that is implemented using a perioperative platform.²⁵ The algorithm transforms data from EHR format to a processed dataset that is optimized for predictive modeling. Machine learning models in the validated MySurgeryRisk platform²⁰ were used to predict risk for developing postoperative AKI during the index hospitalization using perioperative data, each containing two cores, data transformer and data analytics, as previously described (eFigure 2).¹⁰ Briefly, the MySurgeryRisk platform uses a data transformer to integrate data from multiple sources, including the EHR with zip code links to US Census data for patient neighborhood characteristics and distance from the hospital. The platform optimizes the data for analysis through preprocessing, feature transformation, and feature selection techniques. In the data analytics core, the MySurgeryRisk PostOp algorithm was trained to calculate patient-level immediate postoperative risk probabilities for selected complications using 367 preoperative and intraoperative features with random forest classifiers.²⁶ Random forest methods¹⁹ were used to maintain consistency with previous work with the original MySurgeryRisk model. Random forest models are composed of an assembly of decision trees (i.e., a forest of trees), and are classified as ensemble algorithms. Each decision tree performs a classification or prediction task; the most common class (i.e., majority vote) or average prediction is then identified. Data dimensionality was reduced to 110 using an ANOVA approach,²⁷ as features with high variance tend to help ensemble algorithms like Random Forest to achieve optimum predictive behavior. The 50 most important features for the random forest perioperative model are listed in eTable 4.

Model Testing

Results are reported from application of the trained model on the test cohort. Nonparametric confidence intervals for each of the performance metrics were calculated using prediction results obtained from 1,000 bootstrap cohorts.

Model Performance

Each model’s discrimination was assessed using area under the operating curve (AUROC). For each complication, a Youden’s index threshold was calculated to identify the point on the receiver operating characteristic curve with the highest combination of sensitivity and specificity, using this point as the cut-off value for low versus high risk.²⁸ These cut-off values were used to determine the fraction of correct classifications as well as sensitivity, specificity, positive predictive value, and negative predictive value for each model. Model performance was also evaluated by calculating area under the precision-recall curve (AUPRC).²⁹ Wilcoxon’s Sign-Ranked tests were performed to assess the statistical significance of AUROC, AUPRC, and accuracy differences between models.³⁰ Bootstrap sampling and non-parametric methods were used to obtain 95% confidence intervals for all performance metrics. The Net Reclassification Improvement (NRI) index was used to quantify how well the perioperative model reclassified patients compared with the preoperative model.³¹

RESULTS

Participant Baseline Characteristics and Outcomes

A total of 1,531 adult patients undergoing 1,631 major vascular surgeries were included in the study. Baseline characteristics and outcomes of subjects are shown in Table 1. Median age was 68 years and 67% of the population was male. Stratified by type of surgery, patients undergoing open aortic surgery had a high rate of AKI (52%). This cohort also had a high rate of ICU admissions ≥48 hours (91%), CV complications (49%), and neurological complications (34%) as compared with endovascular and lower extremity bypass patients (all p<0.001). Stratified by the occurrence of postoperative AKI, patients who developed AKI had increased ICU admissions ≥48 hours (77% vs. 51%, p<0.001), CV complications (51% vs. 24%, p<0.001), wound complications (53% vs. 43%, p<0.001), and sepsis (29% vs. 11%, p<0.001) (Table 2). There were no differences in intraoperative total urine output when stratified by postoperative AKI. Total blood loss (mL) was higher in the AKI group as compared with the no-AKI group (927 ± 1,537 vs. 542 ± 1,324), p<0.001.

Table 1.

Cohort demographics and comorbidities stratified by the occurrence of postoperative acute kidney injury.

Demographics	Overall	AKI	No AKI
	(n = 1631)	(n = 599, 37%)	(n = 1032, 63%)
Age, median (25th,75th)	68.0 (59.0, 75.0)	69.0 (60.0, 76.0)	68.0 (58.0, 75.0)*
Male sex, n (%)	1076 (65.97)	382 (63.77)	694 (67.25)
Race, n (%)
White	1372 (84.12)	506 (84.47)	866 (83.91)
African American	178 (10.91)	57 (9.52)	121 (11.72)
Other	66 (4.05)	29 (4.84)	37 (3.59)
Hispanic	59 (3.62)	21 (3.51)	38 (3.68)
Comorbidities
PVD, n (%)	1583 (97.06)	583 (97.33)	1000 (96.90)
COPD, n (%)	780 (47.82)	300 (50.08)	480 (46.51)
CKD, n (%)	487 (29.86)	241 (40.23)	246 (23.84)*
CHF, n (%)	419 (25.69)	182 (30.38)	237 (22.97)*
CVD, n (%)	355 (21.77)	124 (20.70)	231 (22.38)
DM, n (%)	353 (21.64)	130 (21.70)	223 (21.61)
CCI, median (25th,75th)	5.0 (3.0, 7.0)	5.0 (3.0, 7.0)	5.0 (3.0, 7.0)

Abbreviations: AKI, acute kidney injury; PVD, peripheral vascular disease; COPD, chronic obstructive pulmonary disorder; CKD, chronic kidney disease; CHF, congestive heart failure; CVD, cardiovascular disease; DM, diabetes mellitus; CCI, Charlson comorbidity index.

^*p-value <0.05 when comparing patients with and without AKI.

Table 2.

Incidence of other complications in patients with and without AKI.

Complications, n (%)	AKI (n = 599, 37%)	No AKI (n = 1032, 63%)
Cardiovascular complications	288 (48.1%)	229 (22.2%)*
Venous thromboembolism	130 (21.7%)	153 (14.8%)*
Sepsis	161 (26.9%)	87 (8.4%)*
Wound complications	291 (48.6%)	405 (39.2%)*
Neurological complications	216 (36.1%)	171 (16.6%)
ICU ≥48 hours	462 (77.1%)	512 (49.6%)*
MV ≥48 hours	150 (25.0%)	41 (4.0%)*
In-hospital mortality	59 (9.8%)	30 (2.9%)*
30-day mortality	57 (9.5%)	33 (3.2%)*
90-day mortality	84 (14.0%)	51 (4.9%)*

The occurrence of postoperative AKI was associated with increased incidence of other complications.

Abbreviations: AKI, acute kidney injury; ICU, intensive care unit; MV, mechanical ventilation.

^*p-value <0.05 when comparing patients with and without AKI.

A higher percentage of total operating room time spent with mean arterial pressure <60 mmHg was associated with greater incidence of postoperative AKI (eTable 4).

Model Performance

Machine learning models incorporating perioperative data had greater accuracy, discrimination and precision than models using either preoperative data alone or ASA classification for predicting AKI (accuracy: 0.70 vs. 0.64 vs. 0.62, AUROC: 0.77 vs. 0.68 vs. 0.61, AUPRC: 0.70 vs. 0.58 vs. 0.48) (Table 3). Sensitivity was 0.78 (95% CI 0.72–0.84) in the preoperative model and 0.84 (95% CI 0.78–0.89) in the perioperative model. The perioperative model also had greater negative predictive value (0.85, 95% CI 0.79–0.89 vs. 0.78, 95% CI 0.71–0.84) (Figure 1A and 1B). The cutoff values of the models were outside the medically inapplicable area (designated by precision ≤0.20) in which models produce five or more false positives for one true positive. To assess incremental improvements in reclassifying patients into the correct risk groups by adding intraoperative data, net reclassification improvement (NRI) indices were calculated. Overall, the perioperative model correctly reclassified 5% of all patients from the low-risk group into the high-risk group (Figure 1C and 1D). The calculated NRI showed an improvement (0.10, 95% CI 0.02–0.18), although this was not statistically significant (p=0.06).

Table 3.

Performance metrics for models predicting postoperative acute kidney injury

Model	Sensitivity	Specificity	Accuracy	PPV	NPV	AUROC	AUPRC
ASA	0.56 (0.49, 0.64)	0.66 (0.60, 0.71)	0.62 (0.57, 0.66)	0.52 (0.45, 0.60)	0.69 (0.63, 0.75)	0.61 (0.57, 0.66)	0.48 (0.42, 0.55)
Preoperative model	0.78 (0.72, 0.84)	0.54 (0.48, 0.60)	0.64 (0.60, 0.69)	0.54 (0.48, 0.60)	0.78 (0.71, 0.84)	0.68 (0.63, 0.73)	0.58 (0.51, 0.65)
Perioperative model	0.84 (0.78, 0.89)	0.60 (0.54, 0.66)	0.70 (0.65, 0.74)	0.59 (0.53, 0.65)	0.85 (0.79, 0.89)	0.77 (0.73, 0.81)	0.70 (0.62, 0.76)

Abbreviations: ASA, American Society of Anesthesiologists physical status classification; PPV, positive predictive value; NPV, negative predictive value; AUROC, area under the operating receiver curve; AUPRC, area under the precision-recall curve; NRI, net reclassification improvement index

Figure 1. In predicting postoperative acute kidney injury, a model using both preoperative and intraoperative (i.e., perioperative) data outperformed a model using preoperative data alone.

(A) The perioperative model had greater area under the receiving operating curve (AUROC) (0.77 vs. 0.68, p<0.001). (B) The perioperative model had greater area under the precision-recall curve (AUPRC) (0.70 vs. 0.58, p<0.001). The perioperative model correctly reclassified negative cases of AKI (C) and positive cases (D). The red dots are patients at high-risk for AKI according to the perioperative model; the green dots are patients at low-risk. The y-axis on each C&D plots are the postoperative model acute kidney injury risk score, which ranges from zero to one.

DISCUSSION

In this large, single-center cohort of patients undergoing major vascular surgery, postoperative AKI was common and was associated with greater duration and severity of intraoperative hypotension as well as increased incidence of postoperative complications, consistent with prior studies.^{2, 3, 6, 32} A validated, dynamic machine learning algorithm that incorporates intraoperative physiological time-series data in addition to preoperative variables²⁰ was able to predict postoperative AKI with a greater accuracy, discrimination and precision than a similar model using preoperative data alone, and both machine learning models outperformed ASA classifications. As expected, the added value of intraoperative data was partially attributable to capturing intraoperative hypotension. The machine learning models used automated EHR data inputs, which have the potential for integration with real-time clinical workflows. MySurgeryRisk is designed as a plug-and-play analytic workflow that can adapt to different coding practices and data structures, but prior to implementation it requires training on a large dataset that is representative of the target population’s characteristics and local practice patterns.

To obtain prognostic information and decrease morbidity and mortality after major vascular surgery, patients are risk stratified preoperatively. However, this process is surgeon-dependent and many of the preoperative measures are subjective, including the commonly used ASA classification. First introduced in 1941, ASA physical status is commonly used to classify patient physical fitness prior to surgical intervention. It has been revised only slightly over time to decrease subjectivity, and substantial inter-observer variability persists.³³ Various risk stratification models incorporate ASA physical status to calculate postoperative morbidity and mortality, including the Physiological and Operative Severity Score for the Enumeration of Mortality and Morbidity (POSSUM), the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP), Surgical Outcome Risk Tool (SORT) and the National Emergency Laparotomy Audit.^34–37 Due to the frequency of adverse cardiac outcomes in vascular surgery patients, likely secondary to the systemic nature of the atherosclerotic process and overlapping risk factors for cardiac disease, various groups have sought to develop vascular surgery-specific models for predicting risk, with a focus on cardiac complications.³⁸ However, many of these online risk calculators require time-consuming manual data entry and are not readily available in EHRs.³⁶ Furthermore, most of these models rely on patients’ pre-existing health conditions, which do not take into account the acute physiologic stresses of surgery that can be represented by intraoperative data. For example, there are certain intraoperative events during surgery, such as hypotension (due to acute blood loss, vascular clamp application, and anesthetic administration), increased contrast load, or hypoxia, which may reclassify a patient from low to high risk for developing complications. By capitalizing on the availability of intraoperative vital signs, urine output, ventilator data and labs in EHRs, many patients were correctly re-classified by the perioperative model as high-risk.

Improved prediction performance using the MySurgeryRisk platform has been previously demonstrated using heterogenous surgical patient populations. In a study of 2,911 adults undergoing any type of major inpatient surgery, Adhikari and colleagues found that postoperative AKI predictive performance was improved by using a machine learning model that incorporates intraoperative physiological time-series variables.¹⁰ In another study of 43,943 adults undergoing major inpatient surgery, Datta and colleagues reported that the incorporation of intraoperative data improved accuracy, discrimination and precision in predicting in ICU stay ≥48 hours, MV duration ≥48 hours, neurological complications and hospital mortality.²⁴ Finally, in a prospective pilot study, Brennan and colleagues assessed the usability and accuracy of the MySurgeryRisk algorithm and found that in a simulated workflow, this model predicted postoperative complications with accuracy equal to or greater than that of clinicians.³⁹ Consistent with the above studies, in a test cohort of 427 major vascular surgeries, the development of postoperative AKI was predicted with greater accuracy, discrimination and precision than machine learning models using preoperative data alone or a traditional risk score using ASA classification. Compared with previous studies using perioperative data, the present study reports lower predictive accuracy, discrimination and precision, which is likely attributable to a smaller sample size inherent to using a select population of patients undergoing vascular surgery rather than a heterogeneous cohort of patients undergoing any type of major inpatient surgery. The authors anticipate that application of the perioperative model to a larger dataset of vascular surgery patients would yield greater predictive performance while maintaining the theoretical advantages of applying the model to a more homogenous cohort of patients.

Machine learning methods have the potential for real-time identification of high-risk patients who may benefit from personalized risk-reduction strategies. Improved prediction performance may have the ability to support decisions regarding postoperative triage, surveillance for complications and targeted prevention measures (e.g., renal protection bundles). This has the potential to inform clinical decision-making toward improved care and better patient outcomes.^{10, 40, 41}

Limitations of the MySurgeryRisk platform have been previously described, many of which are inherent to machine learning algorithms.^{20, 24, 39} While developing preoperative features, there is focus on information collected prior to surgery. For example, anesthesia records may be updated during the surgery but this model exploits preoperative suggestion of anesthesia type as a feature, which helps the model differentiate more serious cases. The model needs data transformed to certain numerical features, and relates on the numerical representation of the features. The system of data transformation, however, needs to be modified if non-EPIC data systems are used. Despite the internally validated results, the MySurgeryRisk platform has yet to be applied to a broader setting outside the University of Florida Health system. Additionally, only the first surgery was used for patients that underwent multiple surgeries in the same admission, which may not reflect the true incidence of complications as the more surgeries patients undergo, the more complications they are likely to incur. Although AKI is a common complication of emergency surgery, certain preoperative data from emergency cases was not available for real-time prediction, and therefore this study excluded emergency surgeries. Further, this is a single-center retrospective study, and was not designed to assess application of the algorithm in a real-time clinical setting. Future studies, such as a prospective, pilot study using the MySurgeryRisk algorithm to evaluate how individualized preoperative and intraoperative strategies tailored to individual vascular surgery patients can reduce postoperative complications, would be instrumental in understanding the clinical application of this machine learning model.

CONCLUSIONS

Acute kidney injury is common after major vascular surgery and is associated with intraoperative hypotension and increased incidence of postoperative complications. In predicting AKI, models using a machine learning approach that incorporate intraoperative data had greater accuracy, discrimination and precision than models using preoperative data alone or traditional risk scores. By capitalizing on the availability of automated EHR data, this method has the potential for real-time identification of high-risk patients who may benefit from personalized, targeted risk-reduction strategies that mitigate preventable harm and optimize resource use.