Abstract
Objective
To accurately calculate the risk for postoperative complications and death after surgery in the preoperative period using machine-learning modeling of clinical data.
Summary Background Data
Postoperative complications cause a two-fold increase in the 30-day mortality and cost and are associated with long-term consequences. The ability to precisely forecast the risk for major complications prior to surgery is limited.
Methods
In a single-center cohort of 51,457 surgical patients undergoing major inpatient surgery, we have developed and validated an automated analytics framework for a preoperative risk algorithm (MySurgeryRisk) that uses existing clinical data in electronic health records to forecast patient-level probabilistic risk scores for eight major postoperative complications (acute kidney injury, sepsis, venous thromboembolism, intensive care unit admission > 48 hours, mechanical ventilation > 48 hours, wound, neurologic and cardiovascular complications) and death up to 24 months after surgery. We used the area under the receiver characteristic curve (AUC) and predictiveness curves to evaluate model performance.
Results
MySurgeryRisk calculates probabilistic risk scores for eight postoperative complications with AUC values ranging between 0.82 and 0.94 (99% confidence intervals 0.81–0.94). The model predicts the risk for death at 1-, 3-, 6-, 12-, and 24-month with AUC values ranging between 0.77 and 0.83 (99% confidence intervals 0.76–0.85).
Conclusions
We constructed an automated predictive analytics framework for machine-learning algorithm with high discriminatory ability for assessing the risk of surgical complications and death using readily available preoperative electronic health records data. The feasibility of this novel algorithm implemented in real time clinical workflow requires further testing.
Keywords: Machine learning, predictive analytics, preoperative risk, postoperative complications, major surgery, mortality
INTRODUCTION
In the United States, where the average American can expect to undergo seven surgical operations during a lifetime, each year 1.5 million patients develop a medical complication and at least 150,000 patients die within thirty days after their surgery.1, 2 The risk for complications arises from the interactions between a patient’s preoperative health and physiologic capacity to withstand surgery-related stress, modulated by the type and quality of surgery and anesthesia that the patient undergoes.3 In the preoperative period, the accurate measurement of this risk can facilitate a discussion about the risks and benefits of surgery and can identify patients who would benefit from intraoperative strategies that could offset the risk.
Preoperative assessment of surgical risk requires integration and interpretation of the large amount of clinical information scattered throughout the healthcare system. A number of surgical risk scores have been developed to estimate postoperative mortality and less frequently specific complications. The most commonly used by anesthesiologists, the American Society of Anesthesiologists (ASA) physical status classification, relies on physicians’ subjective assessment of a patient’s preoperative health.4 Other scores are limited by the inclusion of intraoperative data, need for elaborate data extraction and specialized tests, applicability to only specific surgery types, inability to efficiently handle different data types found in electronic health records (EHR), and modest accuracy and precision for patient-level risk prediction.5–12 Although the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) risk score was developed as a universal surgical score, it requires data that is not readily available in EHR. 6 For prediction of some major complications, such as acute kidney injury and sepsis, current validated risk scores are limited when the complications are defined using contemporary definitions.12, 13
More importantly the existing surgical risk scores have not been developed as machine-learning algorithms with the potential for real-time automation.14, 15 Our objective was to develop an algorithm that could fulfill this role by being universally applicable for any type of surgery, while using all available data within any EHR platform, and by having the capacity for automation and implementation in real-time clinical workflow.16 Here we present the development and validation of an automated predictive analytics workflow for a preoperative risk algorithm MySurgeryRisk for major complications and death after surgery using resampling of a single-center perioperative longitudinal cohort.
METHODS
The University of Florida Institutional Review Board and Privacy Office approved this as an exempt study with waiver of informed consent. The analytical and writing plan followed the recommendations for the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) under the Type 1b analysis category (development and validation of the model using random data resampling)( Supplemental Digital Content (SDC) Table 1).17
Source of Data
Using the University of Florida Health (UFH) Integrated Data Repository as Honest Broker, we have created a single-center perioperative longitudinal cohort that integrated the EHR with public datasets.13 Using residency zip code, we linked the cohort with the United State Census data18 to calculate residing neighborhood characteristics and distance from hospital. We included all inpatient operative procedures requiring at least 24 hours hospital stay performed between January 1, 2000 and November 30, 2010. The date of death was determined using hospital records and the search of the Social Security Death Index and Florida Bureau of Vital Statistics in July 2014 to assess survival through January 31, 2014 using the full name, birth date, and social security number.
Participants
We included all patients with age greater or equal to 18 years admitted for longer than 24 hours following any type of inpatient operative procedure. We collapsed self-reported race categories to account for the association of African-American ethnicity with increased risk for kidney disease and to adjust for estimation of glomerular filtration rate. The final cohort consisted of 51,457 patients.
Outcomes
We modeled preoperative risk probabilities for eight major postoperative complications occurring anytime during hospitalization after the index surgery, including infectious and mechanical wound complications (wound complications), acute kidney injury (AKI), mechanical ventilation (MV) and intensive care unit (ICU) admission for greater than forty-eight hours, cardiovascular complications (CV), neurological complications and/or delirium (neurologic complications), sepsis, and venous thromboembolism (VTE). The algorithm also calculates risk probabilities for death at 1, 3, 6, 12 and 24 months after index surgery.
We used the exact dates to calculate the duration of MV and ICU stay. We defined AKI using consensus criteria while a set of previously described criteria was applied to annotate the remaining complications. 14, 19
Predictor Features
We have derived preoperative predictor features from 285 available preoperative demographic, socio-economic, administrative, clinical, pharmacy and laboratory variables (SDC Table 2) and use them all for each patient. Preoperative comorbidities were derived using up to fifty International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes as binary variables and with the Charlson comorbidity index.19–21 We extracted medications dispensed on the admission day using RxNorms data grouped into drug classes using existing ontologies.22
Sample Size
We included all patients in the cohort. The algorithm was trained on the development cohorts while the reported results were obtained from the validation cohorts. Using one fifth of the cohort as the validation cohort (n= 10,291) in each of the 50-time repeated 5-fold cross validation runs (resulting in 250 different cohorts), we estimated that the overall sample size allows a maximum 99% confidence interval for the area under the receiver operating characteristic curve (AUC) of 0.04 for each model when prevalence of predicted complication is 5% and 0.02 when prevalence is 40%.
Predictive Analytics Workflow
MySurgeryRisk (Figure 1A) is an automated EHR algorithm that will be implemented in real-time using the intelligent perioperative platform developed by our group.16 This platform resides in a secure environment and in real time integrates and transforms EHR data, runs predictive algorithms, produces outputs for physicians, inputs their feedback and prospectively collects data for the future retraining of the prediction models (Figure 1B). The MySurgeryRisk algorithm consists of Data Transformer and Data Analytics modules. The Data Transformer layer integrates data from various sources and then uses data preprocessing, feature transformation, and feature selection to optimize the data for analysis. The Data Analytics layer uses multiple computational algorithms to compute risk probabilities for postoperative complications and mortality for an individual patient.
Figure 1.
Figure 1A. The conceptual framework of MySurgeryRisk analytics platform. The diagram shows sequence of steps from aggregation of raw data, data engineering and data analytics to final output.
Figure 1B. The conceptual diagram of the Intelligent Perioperative Platform. This platform resides in a secure environment and in real time integrates and transforms electronic health records data, runs predictive algorithms, produces outputs for physicians, inputs their feedback and prospectively collects data for the future retraining of the prediction models
Data Transformer Layer
In this layer the algorithm transforms data from any native EHR format to the processed dataset optimized for use in predictive models. New complex variables are created (as described above in “Predictor Features”) and are used in data preprocessing, feature transformation and feature selection. For data preprocessing (SDC Table 2) we use a set of automatic rules to remove errors and outliers. We replace missing nominal variables with a distinct “missing” category while missing continuous variables are replaced by the mean value for a given variable.
Feature transformation is applied to reduce dimensionality of the data and to decrease overfitting. We optimize categorical and nominal variables with multiple levels (such as surgeon’s identities and zip codes) by calculating, for each postoperative complication separately, conditional probabilities for a particular variable value (such as each surgeon’s ID number or each zip code value in the dataset) to be associated with the occurrence of the complication. The probabilities are calculated as the log of the ratio of the prevalence of a particular variable value among cases with a complication (events) to cases without complication (nonevents) (SDC Methods). 23 Surgical procedure codes are optimized using a forest of trees approach to reduce the 4-digit primary procedure ICD-9-CM codes that are prefix-based on the anatomical location of surgery. Each node represents a group of procedures, with roots representing most general groups of procedures and leaf nodes representing specific procedures. 23 This grouping method reduces the number of discrete procedure codes from 1536 to 187 and improves the analysis of low frequency procedures (SDC Methods and SDC Table 2). Supervised feature selection uses variance inflation factors to evaluate collinearity and remove highly collinear predictors.
Data Analytics Layer
A set of algorithms was trained to calculate patient-level risk probabilities for each of eight complications. The calculated risk probabilities were subsequently used as input data for the algorithm trained to calculate mortality risk scores. The final output produces MySurgeryRisk, a personalized risk panel for eight major complications and mortality risk at 1, 3, 6,12 and 24 months after surgery (Figure 2A–B) together with a list of the top three features contributing to each of the calculated risk scores.
Figure 2.
MySurgeryRisk Output. The sample ouput for subjects with A, low mortality risk, and B, high mortality risk. Figure shows the predicted risks for eight postoperative complications for the given patient in eight equal-sized pies. The calculated cutoff values for AKI, ICU, MV, WND, CV, NEU, SEP, and VTE, were 0.35, 0.35, 0.13, 0.1, 0.07, 0.07, 0.06, and 0.03 respectively. Subjects are classified as high risk for a complication if calculated risk score exceeds the respective cutoff and respective pie is marked as red, and green otherwise. The size of the pie represents the proportion of the risk, scaled based on the cutoff for each complication. Green background color represents low mortality risk (Figure 2A) whereas red background color shows high mortality risk (Figure 2B).
Abbreviation: AKI, acute kidney injury, CV, cardiovascular complications, ICU, intensive care unit addmission > 48 hours, MV, mechanical ventilation > 48 hours, NEU, neurologic complications, SEP, sepsis, VTE, venous thromboembolism, WND, wound complications.
Patient-Level Risk Scores, representing the probability of each complication during hospitalization after index surgery, were calculated using a generalized additive model (GAM) with logistic link function as previously described. 14, 23 All models were adjusted for nonlinearity of all covariates using nonlinear risk functions fi estimated with cubic splines.23 For each complication separately, we used risk probabilities calculated by the GAM algorithm to define the optimal cutoff values that best categorize patients into low and high risk categories by maximizing the Youden index.24 The most important features contributing to the risk for an individual patient were derived based on how different she or he is from the patient with an “average” risk (Supplemental Methods).
Patient-Level Mortality Scores, representing the probability of death at 1, 3, 6, 12 and 24 months after index surgery, were calculated using a random forests classifier trained over the individual complication risk probabilities within a 5-fold cross validation design (SDC Figure 1).25 We automatically tuned the parameters for each classifier through maximizing accuracy as the cross validation performance score over searching a parameter space. We evaluated 675 random forest models to find the best performing one.
Validation
The results were reported based on a 5-fold cross validation procedure on 50 bootstrap samples, resulting in 250 different validation cohorts (with a total of 10,291 patients in each validation cohort). Data were randomly split into five disjoint folds in each run, taking one fold for validation and the other four folds for training the model. In each run, the data were reshuffled before splitting the data. Using the values obtained from the 250 validation cohorts, we calculated nonparametric confidence intervals for each of the performance metrics.
Model Performance
We assessed each model’s discrimination using the AUC and model accuracy by determining the fraction of correct classification for each model. Using the optimal thresholds for risk probabilities we built the classification table from which we calculated sensitivity, specificity, and positive and negative predictive values for each model. Model calibration was tested using the Hosmer-Lemeshow statistic and predictiveness curves were used to plot the distributions of risk scores for each complication. 26 Relative risk was calculated as the ratio of the absolute risk of the complication for high and low risk groups for each complication. We used bootstrap sampling and nonparametric methods to obtain 99% confidence intervals for all performance measures.
RESULTS
Participant Baseline Characteristics and Outcomes
Among 51,457 adult patients who underwent major inpatient surgery requiring longer than 24 hours inpatient admission in a quaternary-care academic center, all surgery types were well represented (Table 1 and SDC Table 3). The cohort included data for 520 operating surgeons with an average of 99 procedures per surgeon. The acuity of the patient population was high as 46% of surgeries were categorized as non-elective or associated with emergent/urgent hospital admissions and 52% had ICU admission with a median length of stay of 4 days (25th–75th percentiles 2–8 days) while median hospital length of stay was 7 days (25th–75th percentiles 4–12 days). The overall mortality was 3.4% at thirty days and 17% at two years after index admission (Table 2 and SDC Table 4). A wide range of comorbidities was documented on admission with cancer and diabetes mellitus being most prevalent. One third of the population were from rural areas while on average 10% of the patients resided in neighborhoods with household income below the poverty level.18 The prevalence of examined complications ranged from 3% for venous thromboembolism to 39% for acute kidney injury. As expected we observed a variation in the prevalence of complications among different surgery types likely reflecting the effect of the underlying primary disease process that may predispose to certain types of complications. Acute kidney injury, admission to ICU and mechanical ventilation for > 48 hours were the most common complications among all surgeries. The distribution of outcomes and preoperative clinical characteristics did not differ between training and validation cohorts.
Table 1.
Prevalence of complications by surgery type.
|
|
||||||||
|---|---|---|---|---|---|---|---|---|
| Acute kidney injurya | Intensive care unit admission > 48 hours | Mechanical ventilation > 48 hours | Wound complications | Neurological complications and delirium | Cardiovascular complications | Sepsis | Venous thromboembolism | |
| All surgeries (n=51457) | 20025 (40) | 16493 (32) | 7031 (14) | 6087 (12) | 4086 (8) | 3917 (8) | 2854 (6) | 1502 (3) |
| Cardiothoracic surgery (n=6890) | 4171 (63) | 4654 (68) | 1841 (27) | 970 (14) | 462 (7) | 1228 (18) | 491 (7) | 305 (4) |
| Non-cardiac general surgery (n=20756) | 9177 (48) | 6455 (31) | 2898 (14) | 3021 (15) | 870 (4) | 1479 (7) | 1503 (7) | 735 (4) |
| General gastrointestinal surgery (n=4151) | 1535 (38) | 919 (22) | 384 (9) | 680 (16) | 113 (3) | 243 (6) | 286 (7) | 124 (3) |
| General oncology surgery (n=2200) | 889 (41) | 577 (26) | 190 (9) | 338 (15) | 75 (3) | 139 (6) | 123 (6) | 64 (3) |
| General colorectal surgery (n=1841) | 716 (39) | 374 (20) | 133 (7) | 302 (16) | 47 (3) | 134 (7) | 99 (5) | 47 (3) |
| Vascular surgery (n=2789) | 1305 (53) | 919 (33) | 358 (13) | 506 (18) | 173 (6) | 286 (10) | 162 (6) | 157 (6) |
| Acute care and burn surgery (n=6369) | 3105 (49) | 2886 (45) | 1526 (24) | 543 (9) | 364 (6) | 439 (7) | 595 (9) | 253 (4) |
| Transplant surgery (n=3406) | 1627 (68) | 780 (23) | 307 (9) | 652 (19) | 98 (3) | 238 (7) | 238 (7) | 90 (3) |
| Neurologic surgery (n=8422) | 2580 (31) | 3270 (39) | 1528 (18) | 755 (9) | 2157 (26) | 583 (7) | 408 (5) | 283 (3) |
| Specialty surgery (n=14740) | 3965 (27) | 1982 (13) | 723 (5) | 1214 (8) | 574 (4) | 592 (4) | 426 (3) | 170 (1) |
| Urological surgery (n=2659) | 1312 (50) | 426 (16) | 107 (4) | 369 (14) | 71 (3) | 123 (5) | 107 (4) | 42 (2) |
| Orthopedics surgery (n=7495) | 1649 (22) | 517 (7) | 133 (2) | 326 (4) | 351 (5) | 272 (4) | 116 (2) | 59 (1) |
| Gynecologic surgery (n=2439) | 398 (16) | 113 (5) | 49 (2) | 236 (10) | 23 (1) | 67 (3) | 65 (3) | 26 (1) |
| Ear nose throat (n=2147) | 606 (29) | 926 (43) | 434 (20) | 283 (13) | 129 (6) | 130 (6) | 138 (6) | 43 (2) |
| Other surgery (n=649) b | 132 (21) | 132 (20) | 41 (6) | 127 (20) | 23 (4) | 35 (5) | 26 (4) | 9 (1) |
The numbers in the parentheses represent row percentages. All numbers are rounded.
Number of patents without end-stage renal disease were used to calculate proportions.
Other surgery includes ophthalmology and plastic surgery.
Table 2.
Short and long-term cumulative prevalence of mortality by surgery type.
|
|
|||||
|---|---|---|---|---|---|
| One-month mortality | Three-months mortality | Six-months mortality | Twelve-months mortality | Twenty-four-months mortality | |
| All surgeries (n=51457) | 1786 (3) | 3422 (7) | 4798 (9) | 6589 (13) | 8759 (17) |
| Cardiothoracic surgery (n=6890) | 358 (5) | 658 (10) | 894 (13) | 1143 (17) | 1449 (21) |
| Non-cardiac general surgery (n=20756) | 703 (3) | 1361 (7) | 1884 (9) | 2652 (13) | 3570 (17) |
| General gastrointestinal surgery (n=4151) | 108 (3) | 233 (6) | 343 (8) | 503 (12) | 687 (17) |
| General oncology surgery (n=2200) | 54 (2) | 155 (7) | 266 (12) | 416 (19) | 584 (27) |
| General colorectal surgery (n=1841) | 34 (2) | 92 (5) | 130 (7) | 209 (11) | 280 (15) |
| Vascular surgery (n=2789) | 48 (2) | 257 (9) | 344 (12) | 466 (17) | 653 (23) |
| Acute care and burn surgery (n=6369) | 294 (5) | 459 (7) | 565 (9) | 675 (11) | 806 (13) |
| Transplant surgery (n=3406) | 76 (2) | 165 (5) | 236 (7) | 383 (11) | 560 (16) |
| Neurologic surgery (n=8422) | 465 (6) | 783 (9) | 1044 (12) | 1297 (15) | 1588 (19) |
| Specialty surgery (n=14740) | 255 (2) | 601 (4) | 945 (6) | 1454 (10) | 2091 (14) |
| Urological surgery (n=2659) | 48 (2) | 96 (4) | 170 (6) | 261 (10) | 374 (14) |
| Orthopedics surgery (n=7495) | 117 (2) | 278 (4) | 405 (5) | 582 (8) | 850 (11) |
| Gynecologic surgery (n=2439) | 28 (1) | 61 (3) | 105 (4) | 183 (8) | 283 (12) |
| Ear nose throat (n=2147) | 62 (3) | 166 (8) | 265 (12) | 428 (20) | 584 (27) |
| Other surgery (n=649) a | 5 (1) | 19 (3) | 31 (5) | 43 (7) | 61 (9) |
The numbers in the parentheses represent row percentages. Cumulative mortality rates were calculated as the ratio of the cumulative number of deaths at a given time and number of patients in each surgery group in the preoperative cohort. All numbers are rounded.
Other surgery includes ophthalmology and plastic surgery.
Risk Score Stratification and Model Performance
For each patient in the validation cohort the MySurgeryRisk algorithm uses available preoperative clinical data to calculate the probability risk (range from 0 to 1) for having each of the eight complications and automatically determines the optimal threshold for stratifying patients into low and high-risk groups (Figure 2 and SDC Figure 2). The algorithm’s output provides a list of the most important features contributing to the risk for an individual patient based on how different she or he is from the “average” risk patient (the list of most important features for each model is provided in SDC Table 5). The predictive performance for each complication was very good with AUC values ranging between 0.82 and 0.94 and accuracy between 0.74 and 0.86 (Figures 3A–B). The cutoff values were similar to the prevalence of complications and ranged from 0.35 for the most prevalent complication AKI to 0.03 for least prevalent complication VTE. The calculated thresholds had sensitivity ranging between 0.74 and 0.86 while specificity ranged between 0.69 and 0.86. The risk models for the top three most common complications (AKI, ICU admission for > 48 hours duration and mechanical ventilation for > 48 hours duration) had the best positive (ranging from 0.37 to 0.72) and negative predictive values (0.85 to 0.98). The risk models for the complications with low prevalence had excellent negative predictive values but lower positive predictive values.
Figure 3.
Receiver operating characteristic curves and performance metrics for MySurgeryRisk algorithm in predicting A, more prevalent complications, B, less prevalent complications, and C, mortality.
In the second analytics step, for each patient in the validation cohort the MySurgeryRisk algorithm uses calculated risk probabilities for complications to calculate the probability (range from 0 to 1) for mortality up to two years after index admission and automatically determines the optimal threshold for stratifying patient into low and high-risk mortality groups (Figure 2). The performance metrics was tested for two approaches, the first using the cutoff at which the maximum accuracy was acquired (selected thresholds 0.26, 0.29, 0.3, 0.33, and 0.32 for 1, 3, 6, 12, and 24-month mortality, respectively, SDC Figure 3) and the reported one where the cutoff was based on the maximum Youden index at which both sensitivity and specificity were optimized (selected thresholds 0.12, 0.17, 0.20, 0.26, and 0.24 for 1, 3, 6, 12, and 24-month mortality, respectively, Figure 3C). Both approaches showed very good performance for specificity ranging between 0.91 and 0.99, accuracy ranging between 0.81 and 0.96 and AUC ranging between 0.75 and 0.83.
Comparison of Risk Groups
The observed absolute risk for each complication was distinctly different between high and low risk groups. Patients classified as high-risk for complications had a significant increase in relative risk compared to low-risk patients, ranging from 13.5 (99% CI: 11.4, 15.9) for the least prevalent complication (VTE) to 5.0 (99% CI: 4.8, 5.1) for the most prevalent complication (AKI) (Table 3). We used the integrated predictiveness and classification plots to demonstrate the distribution of patients with different risk probabilities in the cohort (SDC Figures 4A–H). For less common complications, such as sepsis, the predictiveness curve demonstrates that a majority of the cohort (82%) have risk scores below the high-risk cutoff of 0.06. By considering patients with risk scores at or above the cutoff value of 0.06 as the high-risk group for sepsis, we can identify 82% (99% CI: 78%, 86%) of subjects with sepsis while 14% (99% CI: 13%, 15%) of subjects without sepsis are falsely identified. In contrast, for the more prevalent complication such as AKI, almost half of the cohort (44%) had risk scores above the high-risk cutoff of 0.35. By considering patients with risk scores at or above the cutoff value of 0.35 as the high-risk group for AKI, we can identify 80% (99% CI: 78%, 82%) of subjects with sepsis while 21% (99% CI: 20%, 22%) of subjects without AKI are falsely identified.
Table 3.
Absolute and relative risk associated with high and low risk groups.
| Absolute risk % (99% Confidence Interval) | Relative risk (99% Confidence Interval) | ||
|---|---|---|---|
|
| |||
| Postoperative complication | Low risk groupa | High risk groupa | High vs low risk group |
| Acute kidney injury | 14.7% (14.1%, 15.2%) | 72.6% (71.8%, 73.4%) | 5.0 (4.8, 5.1) |
| Intensive care unit admission > 48 hours | 11.9% (11.5%, 12.4%) | 68.3% (67.5%, 69.2%) | 5.7 (5.5, 6.0) |
| Mechanical ventilation > 48 hours | 2.5% (2.3%, 2.7%) | 47.6% (46.5%, 48.8%) | 19.4 (17.8, 21.1) |
| Wound complications | 3.7% (3.5%, 4.0%) | 25.9% (25.1%, 26.7%) | 6.9 (6.4, 7.5) |
| Cardiovascular complications | 2.2% (2.0%, 2.3%) | 20.5% (19.7%, 21.3%) | 9.5 (8.6, 10.5) |
| Neurologic complications | 1.8% (1.7%, 2.0%) | 24.1% (23.2%, 25.1%) | 13.1 (11.8, 14.5) |
| Sepsis | 1.2% (1.0%, 1.3%) | 25.8% (24.6%, 26.9%) | 21.8 (19.3, 24.7) |
| Venous thromboembolism | 0.7% (0.6%, 0.9%) | 10.1% (9.4%, 10.8) | 13.5 (11.4, 15.9) |
| Mortality | Low risk groupb | High risk groupb | High vs low risk group |
|
| |||
| 1-month mortality | 0.4% (0.3%, 0.5%) | 52.6% (50.2%, 55.0%) | 133.3 (115.0, 154.7) |
| 3-months mortality | 0.7% (0.6%, 0.8%) | 60.6% (58.8%, 62.4%) | 86.2 (77.1, 96.5) |
| 6-months mortality | 1.2% (1.0%, 1.3%) | 67.3% (65.7%, 68.8%) | 58.3 (53.3, 63.7) |
| 12-months mortality | 1.7% (1.6%, 1.9%) | 78.8% (77.5%, 80.0%) | 45.1 (42.0, 48.5) |
| 24-months mortality | 2.1% (1.9%, 2.3%) | 69.9% (68.7%, 71.1%) | 32.7 (30.6, 35.1) |
Patients were classified as low risk if their prediction score was less than or equal to cutoff and high as otherwise where cutoff values were 0.35, 0.35, 0.13, 0.10, 0.07, 0.07, 0.06, and 0.03 for acute kidney injury, intensive care unit admission > 48 hours, mechanical ventilation > 48 hours, wound complications, cardiovascular complications, neurologic complications, sepsis, and venous thromboembolism, respectively.
Cutoff values were 0.12, 0.17, 0.20, 0.26, and 0.24 for 1, 3, 6, 12, and 24-month mortality, respectively.
DISCUSSION
In a large single-center cohort of surgical patients we have developed and validated an automated machine-learning algorithm MySurgeryRisk that uses existing clinical data in electronic health records to predict the risk for major complications and death after surgery with high sensitivity and high specificity. This algorithm will serve as an essential component of the intelligent perioperative platform designed by our group16 and will be deployed in a real-time clinical workflow for automated surgical risk prediction as a part of a prospective clinical trial.27 This automated system for surgical risk prediction offers several advantages including prediction based entirely on routinely available data prior to surgery, universal applicability to any surgical context and any type of surgery, exportability to other EHR systems and the ability to handle any data type in EHR (such as semi-structured data, missing or sparse data). The algorithm accounts for patient (characteristics of residing neighborhoods) and physician specific characteristics (the association between their case-mix and performance and postoperative complications in the past), provides consistency of interpretation (a machine makes the same prediction on a specific set of data every time), gives predictions with high sensitivity and specificity and has the potential for near instantaneous reporting of results. In addition, because an algorithm produces a precise probability of the risk, the thresholds for high-risk group can be set at different operating points so that sensitivity and specificity can be tuned to match the requirements for specific clinical settings, such as high sensitivity for a screening setting. In this study, sensitivities ranging between 0.74 to 0.86 were achieved for the single threshold maximized for the screening settings for the postoperative complications. In contrast, we maximized both sensitivity and specificity and negative predictive value when determining threshold for risk for mortality to achieve specificity ranging between 0.91 and 0.99. Furthermore, inclusion of personalized variables in the training dataset, such as surgeons’ previous performances in relation to his case-mix and patients’ residing neighborhoods allows the model to be tuned for a specific population and provides more personalized prediction. The social determinants of health such as income, poverty and inequality can be reflected in patients’ residing ZIP codes and their impact on health has been increasingly recognized. 28, 29
A number of surgical risk models have been developed to estimate postoperative risk for adverse outcomes but the development of a user-friendly, reliable model for individualized prediction across multiple surgery types has remained a challenge. 5–7, 30, 31 The ASA physical status classification relies on physicians’ subjective assessment of a patient’s preoperative health.4 Despite its wide inter-observer variability and limited utility for the quantitative assessment of surgical morbidity or mortality risk 32 it remains the most commonly used tool for preoperative risk assessment among anesthesiologists. The Physiologic and Operative Severity Score for the Enumeration of Mortality and Morbidity (POSSUM) predicts the probability of surgical mortality using twelve preoperative variables and six discharge variables. 10 The need for manual data collection beyond the EHR and the overestimation of the mortality risk among patients undergoing low-risk procedures are major limitations.33 A surgical APGAR score is a simple yet crude summary score of risk,11 but its widespread adoption has been slowed by skepticism.15 The ACS NSQIP risk score was developed as a universal surgical score utilizing population-based standardized surgical cases from participating institutions. The hierarchical linear regression models using twenty-three preoperative variables were developed to predict eight surgical outcomes occurring in the thirty postoperative days only.6 Although the model had good performance in validation studies (c statistics 0.81–0.94), its practical use is limited by the need for data not readily available in EHR, and limited accessibility through a web-based interface rather than through an automatic interface with the EHR. The NSQIP database does not utilize the contemporary consensus definitions for AKI and sepsis, leading to underestimation of the occurrence of these complications and questionable performance of the score when consensus definitions for sepsis and AKI are used in clinical practice.13 The Revised Cardiac Risk Index 9 has been widely used for cardiac risk prediction, although it had moderate performance for non-cardiac surgery patients in a systematic review. 34 The majority of AKI preoperative risk scores are limited to cardiac surgery and have modest accuracy. 35, 36 No validated risk scores exist for sepsis or ICU admission. Recent risk models for respiratory failure have improved accuracy but have not been evaluated fort the potential for automation with EHR and personalization. 37–39
In the preoperative period, knowing the extent to which preoperative health predisposes a patient for postoperative complications, even if not all predictors are modifiable, can facilitate a discussion about the risks and benefits of surgery, and thereby decrease uncertainty regarding outcomes. An accurate risk assessment allows physicians to identify patients who would benefit the most from strategies that can offset the risk. A patient with stage two chronic kidney disease with albuminuria, undergoing high risk surgery, is at increased risk for postoperative AKI. While his risk factors are not modifiable, knowing that he is at high risk allows providers to implements changes in perioperative management to lower that risk for this particular patient. Some of these strategies, like invasive monitoring, 42–44 are costly and carry their own risks while others, such as the avoidance of nephrotoxic medications and individualized blood pressure management, 41 are easy to implement if the risk is identified. On an institutional level, accurate risk assessment may help to quantify the complexity of work being undertaken and provide a method for documenting a risk-adjusted outcome for different health care providers. Our algorithm predicts risk for major complications with systemic effects and profound impact on patient outcomes thus potential interventions need to be multimodal, sequential and cross-disciplinary.40 Several interventions may reduce postoperative complications when applied to patients at risk, such as individualized intraoperative blood pressure management,41 hemodynamic optimization, 42–44 use of neuraxial anesthesia and volatile agents,45–47 glycemic control,48 non-invasive ventilation,49 remote ischemic preconditioning50–52, the use of standardized clinical protocols for prevention of AKI and sepsis. 12, 53, 54 Finally, the expansion of the use of EHR for the real-time tracking of systemic complications using computational algorithms as a higher-capacity and lower-cost information processing service is a logical next step for linking risk prediction with impact on healthcare outcomes.55
There are limitations to this system. The reference standard used for some of the complications, such as cardiovascular complications and sepsis, was based on administrative codes and is dependent on the institutional coding practice. Thus the algorithm may not perform as well for those cases with subtle findings that would not be identified in administrative codes. Another limitation arises from the nature of machine learning, in which the algorithm was provided with only the data and associated outcomes in the training dataset, without explicit definitions of features. Because the algorithm “learned” the features that were most predictive for the risk implicitly, it is possible that the algorithm is using features previously unknown to, or ignored by, physicians. The expansion of input features to include text notes may increase the accuracy but will require more elaborate computational approaches. The algorithm has been trained to work within the referral population of a large academic medical center in north-central Florida and can capture specific population characteristics as well as practice pattern for individual providers within that population. Further training and validation of the algorithm is necessary in a data set with different population characteristics and practice patterns. The algorithm was designed to be used by physicians and we are currently testing whether a simplified web version of the algorithm targeted for patients’ use provides comparable performance.
CONCLUSIONS
In a large single-center cohort of surgical patients we have developed and validated an automated machine-learning algorithm that uses existing clinical data in electronic health records in real-time to forecast the risk for major complications and death after any type of surgery with high sensitivity and specificity. Given the association between greater number of postoperative complications and increased adverse outcomes and costs, there is a critical need for accurate preoperative risk stratification for postoperative complications. Further research is necessary to externally validate this approach and to determine the feasibility of applying this algorithm in a real-time clinical setting in order to assess whether use of the algorithm could lead to improved care and outcomes compared with current practice.
Supplementary Material
SDC Table 1. Checklist for TRIPOD statement.
SDC Table 2. Characteristics of input variables.
SDC Table 3. Summary of overall cohort.
SDC Table 4. Cumulative incidence of mortality by surgery type.
SDC Table 5. Summary of features generalized additive models for each complication
SDC Figure 1. Illustration of a hypothetical random forest. The forest contains of multiple decision trees. In the figure, two decision trees are shown in detail. The decision tree branches at each internal node based on evaluating a single feature and each branch ends in a leaf node. The leaf nodes indicate the prediction for an outcome based on the conditions that were evaluated through the path taken in the branch of the decision tree. Colour of a leaf node indicates its predicted outcome. The final prediction of the random forest is obtained through aggregating all the predictions of the individual decision trees.
SDC Figure 2. Accuracy, positive and negative predicted values, and Youden index distribution using various values of probability of developing A, acute kidney injury, B, intensive care unit addmission > 48 hours, C, mechanical ventilation > 48 hours, D, wound complications, E, neurologic complications, F, cardiovascular complications, G, sepsis, and H, venous thromboembolism. Score which yielded the maximum Youden index value, was used to discriminate between low and high risk patients.
SDC Figure 3. Receiver operating characteristic (ROC) curves and performance metrics of MySurgeryRisk in predicting mortality calculated using the cutoff at which the maximum accuracy was acquired.
SDC Figure 4. The integrated predictiveness and classification plots for A, acute kidney injury, B, intensive care unit addmission > 48 hours, C, mechanical ventilation > 48 hours, D, wound complications, E, neurologic complications, F, cardiovascular complications, G, sepsis, and H, venous thromboembolism. The horizontal dashed black line in the top figure of each panel indicates the cutoff values and the vertical dashed red line shows the percentage of the patients in the cohort that falls below and above the cut-off. The solid dashed line represents the prevalence of the complications in the cohort. The bottom figure of the panel plots the observed true positive fraction or sensitivity (top curve intersecting with the vertical dashed line) and false positive fraction or 1 – specificity (bottom curve intersecting with the vertical dashed line) for the high-risk percentile of the respective complication.
Acknowledgments
Funding/Support: Conflicts of Interest and Source of Funding: A.B., A.E., T.O.B., P.R., P.P., P.M., G.L, X.L., D.W., and W.H. were supported by R01 GM110240 from the National Institute of General Medical Sciences. A.B. and T.O.B. were supported by Sepsis and Critical Illness Research Center Award P50 GM-111152 from the National Institute of General Medical Sciences. T.O.B. has received grant (97071) from Clinical and Translational Science Institute, University of Florida. This work was supported in part by the NIH/NCATS Clinical and Translational Sciences Award to the University of Florida UL1 TR000064. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. A.B., A.E., and T.O.B. had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. University of Florida and A.B., T.O.B., X.L., P.P, P.M., Z.W., P.R. have a patent pending on real-time use of clinical data for surgical risk prediction using machine learning models in MySurgeryRisk algorithm.
We would like to acknowledge our research colleagues Carmen T. Khorram, MD and Martin Rosenthal, MD who have assisted us in this project with literature review, and Paul Thottakkara, MS, Shivam Mittal, MS, Swati Sisodia, MS with coding, George Omalay for project management and Daniel W Freeman for graphical art.
Footnotes
Reprints will not be available from the author(s).
Conflict of Interest Disclosures: None reported.
Previous Presentation: Partial results from this research was presented at the University of Florida Research Day.
Author Contributions:
Azra Bihorac conceived the original idea for the study, and sought and obtained funding.
Bihorac, Ebadi and Ozrazgat-Baslanti had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Acquisition, analysis, or interpretation of data: All authors.
Analysis: Bihorac, Ebadi, Ozrazgat-Baslanti and Momcilovic.
The article was written by Bihorac, Ebadi, and Ozrazgat-Baslanti with input from all coauthors. Critical revision of the manuscript for important intellectual content: All authors.
All authors participated in critically revising the manuscript for important intellectual content and gave final approval of the version to be published.
Administrative, technical, or material support: Bihorac.
Study supervision: Bihorac, Momcilovic, Rashidi.
Azra Bihorac is guarantor for this Article.
References
- 1.Lee PH, Gawande AA. The number of surgical procedures in an American lifetime in 3 states. Journal of the American College of Surgeons. 2008;207(3):S75–S75. [Google Scholar]
- 2.Weiser TG, Regenbogen SE, Thompson KD, et al. An estimation of the global volume of surgery: a modelling strategy based on available data. The Lancet. 2008;372(9633):139–144. doi: 10.1016/S0140-6736(08)60878-8. [DOI] [PubMed] [Google Scholar]
- 3.Grocott MPW, Pearse RM. Perioperative medicine: the future of anaesthesia? British Journal of Anaesthesia. 2012;108(5):723–726. doi: 10.1093/bja/aes124. [DOI] [PubMed] [Google Scholar]
- 4.Lake AP, Williams EG. ASA classification and perioperative variables: graded anaesthesia score? Br J Anaesth. 1997;78(2):228–9. doi: 10.1093/bja/78.2.228-a. [DOI] [PubMed] [Google Scholar]
- 5.Moonesinghe SR, Mythen MG, Das P, et al. Risk stratification tools for predicting morbidity and mortality in adult patients undergoing major surgery: qualitative systematic review. Anesthesiology. 2013;119(4):959–81. doi: 10.1097/ALN.0b013e3182a4e94d. [DOI] [PubMed] [Google Scholar]
- 6.Bilimoria KY, Liu Y, Paruch JL, et al. Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg. 2013;217(5):833–42. e1–3. doi: 10.1016/j.jamcollsurg.2013.07.385. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Glance LG, Lustik SJ, Hannan EL, et al. The Surgical Mortality Probability Model: derivation and validation of a simple risk prediction rule for noncardiac surgery. Ann Surg. 2012;255(4):696–702. doi: 10.1097/SLA.0b013e31824b45af. [DOI] [PubMed] [Google Scholar]
- 8.Finks JF, Kole KL, Yenumula PR, et al. Predicting risk for serious complications with bariatric surgery: results from the Michigan Bariatric Surgery Collaborative. Ann Surg. 2011;254(4):633–40. doi: 10.1097/SLA.0b013e318230058c. [DOI] [PubMed] [Google Scholar]
- 9.Lee TH, Marcantonio ER, Mangione CM, et al. Derivation and prospective validation of a simple index for prediction of cardiac risk of major noncardiac surgery. Circulation. 1999;100(10):1043–9. doi: 10.1161/01.cir.100.10.1043. [DOI] [PubMed] [Google Scholar]
- 10.Copeland GP, Jones D, Walters M. POSSUM: a scoring system for surgical audit. Br J Surg. 1991;78(3):355–60. doi: 10.1002/bjs.1800780327. [DOI] [PubMed] [Google Scholar]
- 11.Gawande AA, Kwaan MR, Regenbogen SE, et al. An Apgar score for surgery. J Am Coll Surg. 2007;204(2):201–8. doi: 10.1016/j.jamcollsurg.2006.11.011. [DOI] [PubMed] [Google Scholar]
- 12.Hobson C, Ruchi R, Bihorac A. Perioperative Acute Kidney Injury: Risk Factors and Predictive Strategies. Crit Care Clin. 2017;33(2):379–396. doi: 10.1016/j.ccc.2016.12.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Bihorac A, Brennan M, Ozrazgat-Baslanti T, et al. National surgical quality improvement program underestimates the risk associated with mild and moderate postoperative acute kidney injury. Crit Care Med. 2013;41(11):2570–83. doi: 10.1097/CCM.0b013e31829860fc. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Thottakkara P, Ozrazgat-Baslanti T, Hupf BB, et al. Application of Machine Learning Techniques to High-Dimensional Clinical Data to Forecast Postoperative Complications. PLoS One. 2016;11(5):e0155705. doi: 10.1371/journal.pone.0155705. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Gawande AA, Regenbogen SE. Critical need for objective assessment of postsurgical patients. Anesthesiology. 2011;114(6):1269–70. doi: 10.1097/ALN.0b013e318219d76b. [DOI] [PubMed] [Google Scholar]
- 16.Feng Z, Rana Bhat R, Yuan X, et al. Intelligent Perioperative System: Towards Real-time Big Data Analytics in Surgery Risk Assessment. 2017;1709 doi: 10.1109/DASC-PICom-DataCom-CyberSciTec.2017.201. ArXiv e-prints. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Collins GS, Reitsma JB, Altman DG, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): The TRIPOD Statement. Ann Intern Med. 2015;162(1):55–63. doi: 10.7326/M14-0697. [DOI] [PubMed] [Google Scholar]
- 18.United States Census Bureau. [Accessed 05/16/2017];American FactFinder. 2010 Available at: http://www2.census.gov/
- 19.Hobson CE, Ozrazgat-Baslanti T, Kuxhausen A, et al. Cost and Mortality Associated With Postoperative Acute Kidney Injury. Ann Surg. 2015;261(6):1207–14. doi: 10.1097/SLA.0000000000000732. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Charlson ME, Pompei P, Ales KL, et al. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. Journal of Chronic Diseases. 1987;40(5):373–83. doi: 10.1016/0021-9681(87)90171-8. [DOI] [PubMed] [Google Scholar]
- 21.Elixhauser A, Steiner C, Harris DR, et al. Comorbidity Measures for Use with Administrative Data. Medical Care. 1998;36(1):8–27. doi: 10.1097/00005650-199801000-00004. [DOI] [PubMed] [Google Scholar]
- 22.U.S. Department of Veterans Affairs VHA. [Accessed 05/16/2017];National Drug File – Reference Terminology (NDF-RT™) Documentation. 2015 Available at: http://evs.nci.nih.gov/ftp1/NDF-RT/NDF-RTDocumentation.pdf.
- 23.Korenkevych D, Ozrazgat-Baslanti T, Thottakkara P, et al. The Pattern of Longitudinal Change in Serum Creatinine and 90-Day Mortality After Major Surgery. Ann Surg. 2016;263(6):1219–27. doi: 10.1097/SLA.0000000000001362. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Fluss R, Faraggi D, Reiser B. Estimation of the Youden Index and its associated cutoff point. Biometrical Journal. 2005;47(4):458–472. doi: 10.1002/bimj.200410135. [DOI] [PubMed] [Google Scholar]
- 25.Pardalos PM, Boginski VL, Vazacopoulos A. Data mining in biomedicine. New York, NY: Springer; 2007. [Google Scholar]
- 26.Pepe MS, Feng Z, Huang Y, et al. Integrating the predictiveness of a marker with its performance as a classifier. Am J Epidemiol. 2008;167(3):362–8. doi: 10.1093/aje/kwm305. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.ClinicalTrials.gov. [Accessed October 14, 2017];Integrating Data, Algorithms and Clinical Reasoning for Surgical Risk Assessment. Available at: https://clinicaltrials.gov/show/NCT02741986.
- 28.Gabert R, Thomson B, Gakidou E, et al. Identifying High-Risk Neighborhoods Using Electronic Medical Records: A Population-Based Approach for Targeting Diabetes Prevention and Treatment Interventions. PLoS One. 2016;11(7):e0159227. doi: 10.1371/journal.pone.0159227. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Goodwin AJ, Nadig NR, McElligott JT, et al. Where You Live Matters: The Impact of Place of Residence on Severe Sepsis Incidence and Mortality. Chest. 2016;150(4):829–836. doi: 10.1016/j.chest.2016.07.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Oliver CM, Walker E, Giannaris S, et al. Risk assessment tools validated for patients undergoing emergency laparotomy: a systematic review. Br J Anaesth. 2015;115(6):849–60. doi: 10.1093/bja/aev350. [DOI] [PubMed] [Google Scholar]
- 31.Gupta PK, Gupta H, Sundaram A, et al. Development and validation of a risk calculator for prediction of cardiac risk after surgery. Circulation. 2011;124(4):381–7. doi: 10.1161/CIRCULATIONAHA.110.015701. [DOI] [PubMed] [Google Scholar]
- 32.Moreno RP, Pearse R, Rhodes A, et al. American Society of Anesthesiologists Score: still useful after 60 years? Results of the EuSOS Study. Rev Bras Ter Intensiva. 2015;27(2):105–12. doi: 10.5935/0103-507X.20150020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Prytherch DR, Whiteley MS, Higgins B, et al. POSSUM and Portsmouth POSSUM for predicting mortality. Physiological and Operative Severity Score for the enUmeration of Mortality and morbidity. Br J Surg. 1998;85(9):1217–20. doi: 10.1046/j.1365-2168.1998.00840.x. [DOI] [PubMed] [Google Scholar]
- 34.Ford MK, Beattie WS, Wijeysundera DN. Systematic review: prediction of perioperative cardiac complications and mortality by the revised cardiac risk index. Ann Intern Med. 2010;152(1):26–35. doi: 10.7326/0003-4819-152-1-201001050-00007. [DOI] [PubMed] [Google Scholar]
- 35.Hobson CE, Singhania G, Bihorac A. Acute kidney injury in the surgical patient. Critical Care Clinics. 2015 doi: 10.1016/j.ccc.2015.06.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Huen SC, Parikh CR. Predicting acute kidney injury after cardiac surgery: a systematic review. Ann Thorac Surg. 2012;93(1):337–47. doi: 10.1016/j.athoracsur.2011.09.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Gupta H, Gupta PK, Fang X, et al. Development and validation of a risk calculator predicting postoperative respiratory failure. Chest. 2011;140(5):1207–15. doi: 10.1378/chest.11-0466. [DOI] [PubMed] [Google Scholar]
- 38.Brueckmann B, Villa-Uribe JL, Bateman BT, et al. Development and validation of a score for prediction of postoperative respiratory complications. Anesthesiology. 2013;118(6):1276–85. doi: 10.1097/ALN.0b013e318293065c. [DOI] [PubMed] [Google Scholar]
- 39.Kor DJ, Warner DO, Alsara A, et al. Derivation and diagnostic accuracy of the surgical lung injury prediction model. Anesthesiology. 2011;115(1):117–28. doi: 10.1097/ALN.0b013e31821b5839. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Bihorac A, Hobson CE. Acute kidney injury: Precision perioperative care protects the kidneys. Nat Rev Nephrol. 2017;14(1):8–10. doi: 10.1038/nrneph.2017.170. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Futier E, Lefrant JY, Guinot PG, et al. Effect of Individualized vs Standard Blood Pressure Management Strategies on Postoperative Organ Dysfunction Among High-Risk Patients Undergoing Major Surgery: A Randomized Clinical Trial. JAMA. 2017;318(14):1346–1357. doi: 10.1001/jama.2017.14172. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Rollins KE, Lobo DN. Intraoperative Goal-directed Fluid Therapy in Elective Major Abdominal Surgery: A Meta-analysis of Randomized Controlled Trials. Ann Surg. 2016;263(3):465–76. doi: 10.1097/SLA.0000000000001366. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Pearse RM, Harrison DA, MacDonald N, et al. Effect of a perioperative, cardiac output-guided hemodynamic therapy algorithm on outcomes following major gastrointestinal surgery: a randomized clinical trial and systematic review. JAMA. 2014;311(21):2181–90. doi: 10.1001/jama.2014.5305. [DOI] [PubMed] [Google Scholar]
- 44.Brienza N, Giglio MT, Marucci M, et al. Does perioperative hemodynamic optimization protect renal function in surgical patients? A meta-analytic study. Critical Care Medicine. 2009;37(6):2079–90. doi: 10.1097/CCM.0b013e3181a00a43. [DOI] [PubMed] [Google Scholar]
- 45.Zangrillo A, Musu M, Greco T, et al. Additive Effect on Survival of Anaesthetic Cardiac Protection and Remote Ischemic Preconditioning in Cardiac Surgery: A Bayesian Network Meta-Analysis of Randomized Trials. PLoS One. 2015;10(7):e0134264. doi: 10.1371/journal.pone.0134264. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Guay J, Choi P, Suresh S, et al. Neuraxial blockade for the prevention of postoperative mortality and major morbidity: an overview of Cochrane systematic reviews. Cochrane Database Syst Rev. 2014;1:CD010108. doi: 10.1002/14651858.CD010108.pub2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Popping DM, Elia N, Van Aken HK, et al. Impact of epidural analgesia on mortality and morbidity after surgery: systematic review and meta-analysis of randomized controlled trials. Ann Surg. 2014;259(6):1056–67. doi: 10.1097/SLA.0000000000000237. [DOI] [PubMed] [Google Scholar]
- 48.Haga KK, McClymont KL, Clarke S, et al. The effect of tight glycaemic control, during and after cardiac surgery, on patient mortality and morbidity: A systematic review and meta-analysis. J Cardiothorac Surg. 2011;6:3. doi: 10.1186/1749-8090-6-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Stephan F, Barrucand B, Petit P, et al. High-Flow Nasal Oxygen vs Noninvasive Positive Airway Pressure in Hypoxemic Patients After Cardiothoracic Surgery: A Randomized Clinical Trial. JAMA. 2015;313(23):2331–9. doi: 10.1001/jama.2015.5213. [DOI] [PubMed] [Google Scholar]
- 50.Thielmann M, Kottenberg E, Kleinbongard P, et al. Cardioprotective and prognostic effects of remote ischaemic preconditioning in patients undergoing coronary artery bypass surgery: a single-centre randomised, double-blind, controlled trial. Lancet. 2013;382(9892):597–604. doi: 10.1016/S0140-6736(13)61450-6. [DOI] [PubMed] [Google Scholar]
- 51.Zarbock A, Schmidt C, Van Aken H, et al. Effect of remote ischemic preconditioning on kidney injury among high-risk patients undergoing cardiac surgery: a randomized clinical trial. JAMA. 2015;313(21):2133–41. doi: 10.1001/jama.2015.4189. [DOI] [PubMed] [Google Scholar]
- 52.Meybohm P, Bein B, Brosteanu O, et al. A Multicenter Trial of Remote Ischemic Preconditioning for Heart Surgery. N Engl J Med. 2015;373(15):1397–407. doi: 10.1056/NEJMoa1413579. [DOI] [PubMed] [Google Scholar]
- 53.Croft CA, Moore FA, Efron PA, et al. Computer versus paper system for recognition and management of sepsis in surgical intensive care. J Trauma Acute Care Surg. 2014;76(2):311–7. doi: 10.1097/TA.0000000000000121. discussion 318–9. [DOI] [PubMed] [Google Scholar]
- 54.Meersch M, Schmidt C, Hoffmeier A, et al. Prevention of cardiac surgery-associated AKI by implementing the KDIGO guidelines in high risk patients identified by biomarkers: the PrevAKI randomized controlled trial. Intensive Care Med. 2017 doi: 10.1007/s00134-016-4670-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.American College of Physicians. EHR-Based Quality Measurement and Reporting: Critical for Meaningful Use and Health Care Improvement. 2010 [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
SDC Table 1. Checklist for TRIPOD statement.
SDC Table 2. Characteristics of input variables.
SDC Table 3. Summary of overall cohort.
SDC Table 4. Cumulative incidence of mortality by surgery type.
SDC Table 5. Summary of features generalized additive models for each complication
SDC Figure 1. Illustration of a hypothetical random forest. The forest contains of multiple decision trees. In the figure, two decision trees are shown in detail. The decision tree branches at each internal node based on evaluating a single feature and each branch ends in a leaf node. The leaf nodes indicate the prediction for an outcome based on the conditions that were evaluated through the path taken in the branch of the decision tree. Colour of a leaf node indicates its predicted outcome. The final prediction of the random forest is obtained through aggregating all the predictions of the individual decision trees.
SDC Figure 2. Accuracy, positive and negative predicted values, and Youden index distribution using various values of probability of developing A, acute kidney injury, B, intensive care unit addmission > 48 hours, C, mechanical ventilation > 48 hours, D, wound complications, E, neurologic complications, F, cardiovascular complications, G, sepsis, and H, venous thromboembolism. Score which yielded the maximum Youden index value, was used to discriminate between low and high risk patients.
SDC Figure 3. Receiver operating characteristic (ROC) curves and performance metrics of MySurgeryRisk in predicting mortality calculated using the cutoff at which the maximum accuracy was acquired.
SDC Figure 4. The integrated predictiveness and classification plots for A, acute kidney injury, B, intensive care unit addmission > 48 hours, C, mechanical ventilation > 48 hours, D, wound complications, E, neurologic complications, F, cardiovascular complications, G, sepsis, and H, venous thromboembolism. The horizontal dashed black line in the top figure of each panel indicates the cutoff values and the vertical dashed red line shows the percentage of the patients in the cohort that falls below and above the cut-off. The solid dashed line represents the prevalence of the complications in the cohort. The bottom figure of the panel plots the observed true positive fraction or sensitivity (top curve intersecting with the vertical dashed line) and false positive fraction or 1 – specificity (bottom curve intersecting with the vertical dashed line) for the high-risk percentile of the respective complication.