Abstract
Background
The objective of this study is to evaluate use of the American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) online risk calculator for estimating common outcomes after operations for gallbladder cancer and extrahepatic cholangiocarcinoma.
Methods
Subjects from the United States Extrahepatic Biliary Malignancy Consortium (USE-BMC) who underwent operation between January 1, 2000 and December 31, 2014 at 10 academic medical centers were included in this study. Calculator estimates of risk were compared to actual outcomes.
Results
The majority of patients underwent partial or major hepatectomy, Whipple procedures or extrahepatic bile duct resection. For the entire cohort, c-statistics for surgical site infection (0.635), reoperation (0.680) and readmission (0.565) were less than 0.7. The c-statistic for death was 0.740. For all outcomes the actual proportion of patients experiencing an event was much higher than the median predicted risk of that event. Similarly, the group of patients who experienced an outcome did have higher median predicted risk than those who did not.
Conclusions
The ACS NSQIP risk calculator is easy to use butrequires further modifications to more accurately estimate outcomes for some patient populations and operations for which validation studies show suboptimal performance.
Introduction
Gallbladder cancer and extrahepatic cholangiocarcinoma are rare.(1, 2) The American Cancer Society estimated 11,420 new cases of, and 3,710 deaths, from gallbladder cancer and extrahepatic cholangiocarcinoma in 2016.(3) Cure for both requires complete surgical resection with histologically negative margins.(1, 2)
Surgical removal of many gallbladder cancers and most extrahepatic cholangiocarcinomas are operations associated with significant risk. Surgical resection for extrahepatic cholangiocarcinoma is associated with perioperative mortality of 3-17% and morbidity of 31-85%. Post-operative complications may include anastomotic leak or other deep space infection, sepsis, liver insufficiency, bleeding, venous thromboembolism, renal failure and others.(4, 5) Factors associated with perioperative complications include preoperative biliary stent, intraoperative drain placement, advanced age (>70 years) and weight loss at presentation.(4) Postoperative complications are associated with inability to provide adjuvant therapies and worse survival.(6, 7)
The American College of Surgeons (ACS) recently established the National Surgical Quality Improvement Program (NSQIP) online risk calculator to help surgeons predict patient-specific risk for morbidity and mortality. The calculator was developed with data from 393 ACS NSQIP hospitals and was based on 1,414,006 patients including 1,557 unique CPT codes. The online risk calculator allows for input of 21 preoperative factors including patient demographics and comorbidities. Data entry is a rapid process on an open and unrestricted website, and estimates of adverse outcomes are provided immediately. The model was demonstrated to have good performance for mortality, morbidity and six additional complications and performs similarly to other procedure-specific risk calculators.(8) The objective of this study is to assess use of the ACS NSQIP calculator for patients undergoing operation for gallbladder cancer and extrahepatic cholangiocarcinoma.
Patients and Methods
The United States Extrahepatic Biliary Malignancy Consortium (USE-BMC) is a group of 10 U.S. academic medical centers (The Ohio State University, Columbus, Ohio; Emory University, Atlanta, Georgia; University of Wisconsin, Milwaukee, Wisconsin; Johns Hopkins University, Baltimore, Maryland; Stanford University, Stanford, California; New York University, New York, New York; Washington University, St. Louis, Missouri; Vanderbilt University, Nashville, Tennessee; University of Louisville, Louisville, Kentucky; Wake Forest University, Winston-Salem, North Carolina). A database of 1092 patients with gallbladder cancer, or distal or hilar cholangiocarcinoma, who underwent operation between January 1, 2000 and December 31, 2014 was compiled by the USE-BMC. The institutional review boards of all participating institutions approved the study.
The ACS NSQIP calculator was accessed online at http://riskcalculator.facs.org/ on August 26th and 27th, 2016. Individual patient data and Current Procedural Terminology (CPT) codes were entered into the calculator, and estimated risks were recorded in a database for each patient, specifically estimates for surgical site infection, death, reoperation, readmission and length of stay. Estimates of risk from the calculator are presented to the user as percentage risk of the event occurring.
At the time of analysis, the calculator required entry of the following factors: procedure type, age, sex, functional status, emergency case, ASA classification, steroid use, ascites, sepsis, ventilator status, disseminated cancer, insulin and non-insulin dependent diabetes mellitus, hypertension, congestive heart failure, dyspnea, current smoker, severe COPD, dialysis dependence, acute renal failure, height and weight. Definitions of each category may be found on the risk calculator website. For patients with a preoperative risk calculator factor missing, risk was assumed to be at the lowest risk level.
The estimated risk for each outcome was then compared to actual patient outcomes. The following calculator outcomes were included in the study: surgical site infection, mortality, readmission, return to the operating room and length of stay. Additional calculator outcomes were not analyzed as they were not included in the USE-BMC dataset. Demographics and clinical characteristics were summarized using descriptive statistics (median and range for continuous variables, frequency for categorical variables). Logistic regression models were used to determine the association between the actual outcome and the calculated risk using the ACS NSQIP calculator. The ability of the ACS NSQIP calculator to accurately predict a particular outcome was assessed using the c-statistic, also known as the area under the curve (AUC) of a receiving operating characteristic curve, and the Brier score. The Brier score simultaneously reflects discrimination and calibration and is calculated as the mean squared difference between a patient’s predicted probability and observed outcome. A Brier score of 0 represents perfect prediction and the cutoff at which a model is no longer informative is determined partially based on incidence in the sample.(9, 10) Two sub-group analyses were performed, one for patients undergoing liver surgery including partial lobectomy, total left lobectomy and total right lobectomy and trisegmentectomy, and one for patients who underwent pancreas surgery including standard and pylorus-sparing Whipple procedures.
Patients were excluded who did not have complete tumor resection for cure. Also, the following operations were excluded as they are not included in the ACS NSQIP risk calculator: liver transplantation and diagnostic laparoscopy.
All statistical analyses were conducted using SAS for Windows® Version 9.2 (SAS Institute Inc., Cary, NC), although ROC curves were plotted using the pROC package in R v. 3.0.1. (Vienna, Austria).
Results
Eight-hundred and fifty-four patients with gallbladder cancer, or distal or hilar cholangiocarcinoma, who underwent curative-intent, complete resection between January 1, 2000 and December 31, 2014 was compiled by the USE-BMC (Table 1). The majority of patients underwent hepatic resection including partial lobectomy (254, 29.7%), trisegmentectomy (118, 13.8%), total left lobectomy (62, 7.3%) or total right lobectomy (41, 4.8%) or pancreatic head resection including standard pancreaticoduodenectomy (132, 15.5%) and pylorus-preserving pancreaticoduodenectomy (105, 12.3%). Additionally, a minority of patients underwent resection of extrahepatic bile duct tumor (104, 12.2%), cholecystectomy alone (36, 4.2%) or exploratory laparotomy (2, 0.2%). Median age of included patients was 67 (range: 24-94) years old. There were 319 (37.4%) patients with gallbladder cancer, 281 (32.9%) with hilar cholangiocarcinoma and 254 (29.7%) with distal cholangiocarcinoma. There were 420 (49.2%) males and 434 (50.8%) females. The majority of patients were fully independent at the time of their operation (684, 80.1%). Only 8 (0.9%) underwent emergency surgery. The majority of patients were ASA Class 2 (205, 24%) and ASA Class 3 (377, 44.1%). The majority of patients were not on chronic steroids (759, 88.9%), did not have ascites (776, 90.9%), did not have systemic sepsis (808, 94.6%), were not ventilator dependent (764, 89.5%), did not have severe COPD (739, 89.5%), were not on dialysis (765, 89.6%) and did not have acute renal failure (759, 88.9%). There were 105 (12.3%) patients with diabetes on oral medication and 55 (6.4%) on insulin. Patients were median height of 169 (100-196) centimeters tall and weighed a median of 76 (37-164) kilograms.
Table 1.
Demographic Characteristics (n=854)
| Calculator Input | Type | Frequency |
|---|---|---|
| Procedure Type | Hepatectomy, resection of liver; partial lobectomy | 254 (29.7) |
| Standard pancreaticoduodenectomy with pancreaticojejunostomy | 132 (15.5) | |
| Hepatectomy, resection of the liver; trisegmentectomy | 118 (13.8) | |
| Pylorus-preserving pancreaticoduodenectomy; with pancreaticojejunostomy | 105 (12.3) | |
| Excision of bile duct tumor, with or without primary repair; extrahepatic | 104 (12.2) | |
| Hepatectomy, resection of the liver; total left lobectomy | 62 (7.5) | |
| Hepatectomy, resection of the liver; total right lobectomy | 41 (4.8) | |
| Cholecystectomy | 36 (4.2) | |
| Exploratory Laparotomy with or without biopsies | 2 (0.2) | |
| Age – Median (Range) | 67 (24-94) | |
| Cancer Type | Gallbladder Cancer | 319 (37.4) |
| Extrahepatic Cholangiocarcinoma - Hilar | 281 (32.9) | |
| MANUS Extrahepatic Cholangiocarcinoma - Distal | 254 (29.7) | |
| Gender | 0 | 420 (49.2) |
| 1 | 434 (50.8) | |
| Functional Status | Independent | 684 (80.1) |
| Partially Dependent | 32 (3.7) | |
| Dependent | 4 (0.5) | |
| Missing | 134 (15.7) | |
| Emergency Surgery | 0 | 846 (99.1) |
| 1 | 8 (0.9) | |
| ASA class | 1 | 10 (1.2) |
| 2 | 205 (24.0) | |
| 3 | 377 (44.1) | |
| 4 | 25 (2.9) | |
| Missing | 237 (27.8) | |
| Steroids | No | 759 (88.9) |
| Yes | 8 (0.9) | |
| Missing | 87 (10.2) | |
| Ascites | No | 776 (90.9) |
| Yes | 17 (2.0) | |
| Missing | 61 (7.1) | |
| Systemic Sepsis | No | 808 (94.6) |
| Yes | 31 (3.6) | |
| Missing | 15 (1.8) | |
| Ventilator | No | 767 (89.8) |
| Dependent | Yes | 1 (0.1) |
| Missing | 86 (10.1) | |
| Disseminated Cancer | No | 764 (89.5) |
| Yes | 4 (0.5) | |
| Missing | 86 (10.1) | |
| Diabetes | No | 609 (71.3) |
| Oral Medication | 105 (12.3) | |
| Insulin | 55 (6.4) | |
| Missing | 85 (10.0) | |
| Hypertension | No | 365 (42.7) |
| Yes | 403 (47.2) | |
| Missing | 86 (10.1) | |
| Congestive Heart Failure | No | 748 (87.6) |
| Yes | 21 (2.5) | |
| Missing | 85 (10.0) | |
| Dyspnea | No | 748 (87.6) |
| Yes | 19 (2.2) | |
| Missing | 87 (10.2) | |
| Smoking History | No | 596 (69.8) |
| Yes | 170 (19.9) | |
| Missing | 88 (10.3) | |
| Severe COPD | No | 739 (86.5) |
| Yes | 29 (3.4) | |
| Missing | 86 (10.1) | |
| Dialysis | No | 765 (89.6) |
| Yes | 4 (0.5) | |
| Missing | 85 (10.0) | |
| Acute Renal Failure | No | 759 (88.9) |
| Yes | 10 (1.2) | |
| Missing | 85 (10.0) | |
| Height – Median (Range) | 169 (100-196) | |
| Weight - Median (Range) | 76 (37-164) | |
Actual event occurrence (n and percent) for all patients is displayed alongside median predicted risk (median percent predicted risk and minimum and maximum predicted risk) in Table 2. Of note, this information is provided to help describe the cohort but these numbers should not be directly compared since one is a median and one a proportion. For the outcomes surgical site infection, death, reoperation and readmission, the percent of patients who actually had an event was higher than the median percent predicted risk in the entire cohort and in groups of patients who underwent liver and pancreas surgery (Table 2)
Table 2.
Comparison of Actual Events to Predicted Risk
| Entire Cohort (n=854) | ||
|---|---|---|
| Actual Events | Predicted Risk (%) | |
| Outcome | n (%) | Median Risk |
| Surgical Site Infection | 185 (21.7) | 9.6 (1.9-36.6) |
| Mortality | 18 (2.1) | 0.5 (0-19.4) |
| Reoperation | 57 (6.7) | 3.3 (0.9-13.2) |
| Readmission | 168 (19.7) | 9.1 (2.6-32.2) |
| Length of stay (Days) | Median: 8 (0-119) | 6.5 (2.5-25.5) |
| Liver Surgery (n=475) | ||
| Actual Events | Predicted Risk (%) | |
| Outcome | n (%) | Median Risk |
| Surgical Site Infection | 105 (22.1) | 7.4 (2.8 - 23.9) |
| Mortality | 12 (2.5) | 0.6 (0 - 19.4) |
| Reoperation | 34 (7.2) | 2.5 (0.9 - 9.3) |
| Readmission | 97 (20.4) | 9.4 (3.9 - 32.2) |
| Length of stay (days) | Median: 7 (0 - 91) | 6 (3.5 - 23.0) |
| Pancreas Surgery (n=237) | ||
| Actual Events | Predicted Risk (%) | |
| Outcome | n (%) | Median Risk |
| Surgical Site Infection | 62 (26.2) | 15.7 (8.1 - 36.6) |
| Mortality | 4 (1.7) | 0.5 (0.1 - 13.6) |
| Reoperation | 17 (7.2) | 4.6 (2.2 - 9.9) |
| Readmission | 42 (17.7) | 12.6 (7.2 - 30.8) |
| Length of stay (days) | 10 (0 - 119) | 9 (6 - 24) |
Percent predicted risk between those who did and did not have an event are compared for the entire cohort in Table 3. For all outcomes, the percent predicted risk was higher for the group of patients that did have the event than for those who did not have the event. The odds ratios presented here are the odds of having an actual event for each one percent increase in estimate of risk from the risk calculator. The odds ratios are significant for all measured outcomes: surgical site infection (1.08), death (1.24), reoperation (1.44) and readmission (1.03). The c-statistics for surgical site infection (0.635), reoperation (0.680) and readmission (0.565) are less than 0.7. The c-statistic for death is 0.740. The ROC curves are demonstrated in Figure 1. The upper limits at which the Brier scores are useful for interpretation are listed in the final column of Table 3. The Brier score for death is equal to the upper limit at which the score is useful (0.021), while the scores for surgical site infection (0.165 < 0.170), reoperation (0.059 < 0.063), and readmission (0.157 < 0.158) are close to the upper limit.
Table 3.
Risk Calculator Outcomes
| Entire Cohort (n=854) | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Did Not Have Event | Had Event | OR (Wald CI) | P-value | C-statistic (AUC) | Brier Score | Brier Score Cutoff | |||
| Outcome | n (%) | Median Risk | n (%) | Median Risk | |||||
| Surgical Site Infection | 669 (78.3) | 9 (1.9-36.6) | 185 (21.7) | 11.5 (3-30.1) | 1.08 (1.05 - 1.11) | <0.001 | 0.635 | 0.165 | 0.170 |
| Death | 836 (97.9) | 0.5 (0-19.4) | 18 (2.1) | 2.4 (0.1-10.9) | 1.24 (1.10 - 1.40) | 0.001 | 0.74 | 0.021 | 0.021 |
| Reoperation | 797 (93.3) | 3.2 (0.9-9.9) | 57 (6.7) | 5 (1.2-13.2) | 1.44 (1.26 - 1.64) | <0.001 | 0.68 | 0.059 | 0.063 |
| Readmission | 686 (80.3) | 8.6 (2.6-32.2) | 168 (19.7) | 10.2 (2.7-25.4) | 1.03 (1.00 - 1.07) | 0.042 | 0.565 | 0.157 | 0.158 |
| Liver Surgery (n=475) | |||||||||
| Did Not Have Event | Had Event | OR (Wald CI) | P-value | C-statistic (AUC) | Brier Score | Brier Score Cutoff | |||
| Outcome | n (%) | Median Risk | n (%) | Median Risk | |||||
| Surgical Site Infection | 370 (77.9) | 6.9 (2.8-21.6) | 105 (22.1) | 9.5 (3.0-23.9) | 1.14 (1.08 - 1.20) | <0.001 | 0.653 | 0.162 | 0.172 |
| Death | 463 (97.5) | 0.6 (0.0-19.4) | 12 (2.5) | 2.2 (0.5-10.9) | 1.20 (1.04 - 1.39) | 0.012 | 0.758 | 0.025 | 0.024 |
| Reoperation | 441 (92.8) | 2.5 (0.9-9.3) | 34 (7.2) | 4.1 (1.2-8.9) | 1.47 (1.23 - 1.75) | <0.001 | 0.686 | 0.063 | 0.067 |
| Readmission | 378 (79.6) | 8.9 (3.9-32.2) | 97 (20.4) | 11.1 (4.0-20.9) | 1.08 (1.03 - 1.13) | 0.002 | 0.625 | 0.16 | 0.162 |
| Pancreas Surgery (n=237) | |||||||||
| Did Not Have Event | Had Event | OR (Wald CI) | P-value | C-statistic (AUC) | Brier Score | Brier Score Cutoff | |||
| Outcome | n (%) | Median Risk | n (%) | Median Risk | |||||
| Surgical Site Infection | 175 (73.8) | 15.4 (8.136.6) | 62 (26.2) | 16.2 (8.8-30.1) | 1.02 (0.97 - 1.08) | 0.454 | 0.543 | 0.193 | 0.193 |
| Death | 233 (98.3) | 0.5 (0.1-13.6) | 4 (1.7) | 2.9 (0.1-8.5) | 1.35 (1.02 - 1.77) | 0.035 | 0.704 | 0.016 | 0.017 |
| Reoperation | 220 (92.8) | 4.6 (2.2-9.9) | 17 (7.2) | 5.6 (2.9-9.4) | 1.50 (1.12 - 2.00) | 0.006 | 0.702 | 0.065 | 0.067 |
| Readmission | 195 (82.3) | 12.6 (7.230.8) | 42 (17.7) | 13.3 (7.2-25.4) | 1.00 (0.93-1.07) | 0.947 | 0.505 | 0.146 | 0.146 |
Figure 1.
Receiver Operating Characteristic (ROC) curves for surgical site infection, mortality, reoperation and readmission.
For the liver surgery group percent predicted risk between those who did and did not have an event are compared Table 3. For all outcomes, the percent predicted risk was higher for the group of patients that did have the event than for those who did not have the event.. The odds ratios are significant for all measured outcomes: surgical site infection (1.14), death (1.20), reoperation (1.47) and readmission (1.08). The c-statistics for surgical site infection (0.653), reoperation (0.686) and readmission (0.565) are less than 0.7. The c-statistic for death is 0.758. The upper limits at which the Brier scores are useful for interpretation are listed in the final column of Table 3. The Brier score for death is just above the upper limit at which the score is useful (0.025 > 0.024), while the scores for surgical site infection (0.162 < 0.172), reoperation (0.063 < 0.067), and readmission (0.160 < 0.162) are close to the upper limit.
For the pancreas surgery group percent predicted risk between those who did and did not have an event are compared in Table 3. For all outcomes, the percent predicted risk was higher for the group of patients that did have the event than for those who did not have the event. The odds ratios are significant for death (1.35) and reoperation (1.50) and were not significant for surgical site infection (1.02) and readmission (1.00). The c-statistics for surgical site infection (0.543) and readmission (0.505) are less than 0.7. The c-statistic for death is 0.704 and for reoperation is 0.702. The upper limits at which the Brier scores are useful for interpretation are listed in the final column of Table 3. The Brier scores for readmission (0.146) and surgical site infection (0.193) are equal to the upper limit, while the scores for reoperation (0.065 < 0.67), and death (0.016 < 0.017) are close to the upper limit.
Discussion
This study compares estimated risk of surgical site infection, death, reoperation and readmission using an online risk calculator to actual outcomes for 854 patients from the United States Extrahepatic Biliary Malignancy Consortium. The estimates of risk were variable in terms of accuracy and generally calculator performance was poor… When percent predicted risk was compared between those patients who did and did not have an event, patients who did have an event had higher median percent predicted risk. For all outcomes, the Brier scores are either at, or close to, the upper limit at which the model can be considered useful. One might conclude the calculator is limited in usefulness for some operations and some patients The majority of patients in this cohort underwent hepatectomy (partial, trisegmentectomy, complete left, complete right) or Whipple procedures (standard or pylorus-preserving pancreaticoduodenectomy). The calculator performed similarly in these patient populations as in the entire cohort.
The utility of the c-statistic in the evaluation of risk-adjustment models in surgery has been previously questioned. It has been demonstrated that use of the c-statistic can lead to incorrect model conclusions as case mix is restricted and patient-level risk for morbidity and mortality becomes more similar.(11) While models are typically considered reasonable when the c-statistic is higher than 0.7 and strong when it is >0.8, as patient groups become more focused and homogenous the c-statistic decreases, but this does not coincide with a decrease in model performance.(11) On the other hand, the Brier score reflects both calibration and discrimination simultaneously and may be a more reasonable measure of model quality. A model that perfectly predicts the outcome would have a Brier score of 0 and the cutoff at which a model is no longer informative is determined partially based on incidence in the sample.(9, 10) When considering the Brier score this model performed poorly for most outcomes in the entire cohort and in the liver and pancreas surgery groups.
Risk of postoperative pancreatic fistula is one of the primary concerns after Whipple. Body mass index and pancreatic duct width are independently associated with postoperative pancreatic fistula, and a model incorporating these two preoperative factors has a c-statistic of 0.832 (p<0.001) for predicting postoperative pancreatic fistula.(12) A second model developed from single-center data incorporated other factors including main pancreatic duct index <0.25, away from portal vein on computed tomography, disease other than pancreatic cancer, male gender and intraabdominal thickness with a c-statistic of 0.834 for predicting postoperative pancreatic fistula.(13) Postoperative pancreatic fistula is not included in the ACS NSQIP risk calculator as it is designed to predict outcomes that are more universally relevant. Both surgical site infections and readmissions may be secondary to postoperative pancreatic fistulas, which are not captured by the risk calculator.
A score for 30-day mortality in pancreaticoduodenectomy patients using the NSQIP database incorporates hypertension with medication, history of cardiac surgery, age greater than sixty-two years old, bleeding disorder, albumin less than 3.5, disseminated cancer, use of steroids and systemic inflammatory response syndrome. Although this model is based on a similar data set and incorporates some of the same factors as the ACS NSQIP risk calculator, the reported c-statistic of this model is only 0.71.(14) The c-statistic of the ACS NSQIP risk calculator in all patients was reported to be 0.944 for mortality.(8) It has also been demonstrated that Whipple-specific complications, including postoperative pancreatic fistula, lead to an increase in actual length of stay beyond what the ACS NSQIP risk calculator predicts.(15)
Limitations of this study include the retrospective cohort design and modest sample size limiting statistical analysis for important subgroups. The USE-BMC cohort is substantially more homogenous than the population used to develop the risk calculator - all included patients have extrahepatic biliary malignancy and were treated at a large, academic medical center. These patients may have baseline higher risk of morbidity and mortality than the general population. It has been previously reported that morbidity and mortality in the NSQIP population varies based on etiology of disease – benign versus malignant – for hepato-pancreato-biliary (HPB) surgery.(16) A further limitation is that some patients (<5% for any variable) had missing data and the online calculator requires entry for all input variables. In these instances, we chose to use the lowest risk level for the variable since many patients with excess comorbid risks are not offered pancreatectomy or major hepatectomy. This will limit the ability to draw the conclusion that the calculator consistently underestimates risk.
Proper risk stratification for outcomes like readmission, SSI and mortality are critical to allow pre-operative optimization and to adequately counsel patients about risk and potential benefit.. Readmissions are an economic burden to the healthcare system.(17) SSIs increase morbidity and are associated with increased length of stay, increased readmissions and increased use of hospital resources.(18, 19) Surgical site infections are also “never events,” and associated care is not reimbursable by Medicare.(20) Furthermore, development of a surgical site infection is associated with decreased administration of adjuvant chemotherapy and worse oncologic outcomes after HPB surgery.(6, 7, 21, 22)
An ideal risk stratification tool uses easy to identify factors for accurate risk assessment, and the variables used in the ACS NSQIP risk calculator are commonly known. The low accuracy of the calculator for estimating risk of SSI and readmission underscores the importance of developing site- and procedure-specific risk calculators and for subsequent validation studies.. The number of complications seen in the USE-BMC database is sobering since this represents the collective experience of ten high volume academic centers. Given this, the authors suggest caution for instances in which low estimation of risk by the online calculator or any source may be inaccurate.. While the calculator allows input of multiple operations, it does not provide the user with any information about potential variable accuracy based on the operation one has queried. If there are operations for which the calculator is less accurate than others, a notification about this would be of benefit to users. For instance, solid organ transplant operations are not allowed to be input, and is the reason for this that calculator accuracy for these operations is low? Continued validation and modification of the calculator may improve performance for other outcomes and for other patient subgroups, especially those with malignancy and those undergoing higher risk operations. If median risk is used as the threshold of concern, using the calculator to estimate risk in this cohort would miss more than half of complications.. Our initial interest in validating the calculator stemmed from its’ use in the ambulatory setting. We were uncertain if the calculator’s classification of increased risk as estimated percentage risk higher than average was the best or most accurate threshold of concern. Perhaps a lower threshold should be used in the case of more intense pre-operative evaluation or “prehabilitation” and a higher threshold used to refuse operation in the case of potentially prohibitive risk. We recommend ACS-NSQIP consider providing information to calculator users regarding which operations have more or less accuracy in validation studies and consider offering different levels of concern for estimated risk.
Acknowledgments
Sources of Funding: There was no funding associated with the completion of this work.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Presentation: This manuscript was presented at the Americas Hepato-Pancreato-Biliary Association (AHPBA) as an oral presentation in March of 2017.
References
- 1.Kanthan R, Senger JL, Ahmed S, Kanthan SC. Gallbladder Cancer in the 21st Century. J Oncol. 2015;2015:967472. doi: 10.1155/2015/967472. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Esnaola NF, Meyer JE, Karachristos A, Maranki JL, Camp ER, Denlinger CS. Evaluation and management of intrahepatic and extrahepatic cholangiocarcinoma. Cancer. 2016;122(9):1349–69. doi: 10.1002/cncr.29692. [DOI] [PubMed] [Google Scholar]
- 3.Society AC. Cancer Facts and Figures 2016 Atlanta. 2016 Available from: http://www.cancer.org.proxy.lib.ohio-state.edu/acs/groups/content/@research/documents/document/acspc-047079.pdf.
- 4.Dixon JB, Bhathal PS, O’Brien PE. Nonalcoholic fatty liver disease: predictors of nonalcoholic steatohepatitis and liver fibrosis in the severely obese. Gastroenterology. 2001;121(1):91–100. doi: 10.1053/gast.2001.25540. [DOI] [PubMed] [Google Scholar]
- 5.Todoroki T, Kawamoto T, Takahashi H, Takada Y, Koike N, Otsuka M, et al. Treatment of gallbladder cancer by radical resection. Br J Surg. 1999;86(5):622–7. doi: 10.1046/j.1365-2168.1999.01085.x. [DOI] [PubMed] [Google Scholar]
- 6.Wu W, He J, Cameron JL, Makary M, Soares K, Ahuja N, et al. The impact of postoperative complications on the administration of adjuvant therapy following pancreaticoduodenectomy for adenocarcinoma. Ann Surg Oncol. 2014;21(9):2873–81. doi: 10.1245/s10434-014-3722-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Merkow RP, Bilimoria KY, Tomlinson JS, Paruch JL, Fleming JB, Talamonti MS, et al. Postoperative complications reduce adjuvant chemotherapy use in resectable pancreatic cancer. Ann Surg. 2014;260(2):372–7. doi: 10.1097/SLA.0000000000000378. [DOI] [PubMed] [Google Scholar]
- 8.Bilimoria KY, Liu Y, Paruch JL, Zhou L, Kmiecik TE, Ko CY, et al. Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg. 2013;217(5):833-42.e1–3. doi: 10.1016/j.jamcollsurg.2013.07.385. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Cohen ME, Ko CY, Bilimoria KY, Zhou L, Huffman K, Wang X, et al. Optimizing ACS NSQIP modeling for evaluation of surgical quality and risk: patient risk adjustment, procedure mix adjustment, shrinkage adjustment, and surgical focus. J Am Coll Surg. 2013;217(2):336–46.e1. doi: 10.1016/j.jamcollsurg.2013.02.027. [DOI] [PubMed] [Google Scholar]
- 10.Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128–38. doi: 10.1097/EDE.0b013e3181c30fb2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Merkow RP, Hall BL, Cohen ME, Dimick JB, Wang E, Chow WB, et al. Relevance of the c-statistic when evaluating risk-adjustment models in surgery. J Am Coll Surg. 2012;214(5):822–30. doi: 10.1016/j.jamcollsurg.2011.12.041. [DOI] [PubMed] [Google Scholar]
- 12.Roberts KJ, Hodson J, Mehrzad H, Marudanayagam R, Sutcliffe RP, Muiesan P, et al. A preoperative predictive score of pancreatic fistula following pancreatoduodenectomy. HPB (Oxford) 2014;16(7):620–8. doi: 10.1111/hpb.12186. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Yamamoto Y, Sakamoto Y, Nara S, Esaki M, Shimada K, Kosuge T. A preoperative predictive scoring system for postoperative pancreatic fistula after pancreaticoduodenectomy. World J Surg. 2011;35(12):2747–55. doi: 10.1007/s00268-011-1253-x. [DOI] [PubMed] [Google Scholar]
- 14.Gleeson EM, Shaikh MF, Shewokis PA, Clarke JR, Meyers WC, Pitt HA, et al. WHipple-ABACUS, a simple, validated risk score for 30-day mortality after pancreaticoduodenectomy developed using the ACS-NSQIP database. Surgery. 2016;160(5):1279–87. doi: 10.1016/j.surg.2016.06.040. [DOI] [PubMed] [Google Scholar]
- 15.Cusworth BM, Krasnick BA, Nywening TM, Woolsey CA, Fields RC, Doyle MM, et al. Whipple-specific complications result in prolonged length of stay not accounted for in ACS-NSQIP Surgical Risk Calculator. HPB (Oxford) 2016 doi: 10.1016/j.hpb.2016.10.015. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Kneuertz PJ, Pitt HA, Bilimoria KY, Smiley JP, Cohen ME, Ko CY, et al. Risk of morbidity and mortality following hepato-pancreato-biliary surgery. J Gastrointest Surg. 2012;16(9):1727–35. doi: 10.1007/s11605-012-1938-y. [DOI] [PubMed] [Google Scholar]
- 17.Raoof M, Lewis A, Goldstein L, Dumitra S, Warner SG, Singh G, et al. Timing and severity of post-discharge morbidity after hepatectomy. HPB (Oxford) 2017 doi: 10.1016/j.hpb.2016.12.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Tanner J, Padley W, Assadian O, Leaper D, Kiernan M, Edmiston C. Do surgical care bundles reduce the risk of surgical site infections in patients undergoing colorectal surgery? A systematic review and cohort meta-analysis of 8,515 patients. Surgery. 2015;158(1):66–77. doi: 10.1016/j.surg.2015.03.009. [DOI] [PubMed] [Google Scholar]
- 19.Ceppa EP, Pitt HA, House MG, Kilbane EM, Nakeeb A, Schmidt CM, et al. Reducing surgical site infections in hepatopancreatobiliary surgery. HPB (Oxford) 2013;15(5):384–91. doi: 10.1111/j.1477-2574.2012.00604.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Fry DE, Pine M, Jones BL, Meimban RJ. Patient characteristics and the occurrence of never events. Arch Surg. 2010;145(2):148–51. doi: 10.1001/archsurg.2009.277. [DOI] [PubMed] [Google Scholar]
- 21.Haruki K, Shiba H, Fujiwara Y, Furukawa K, Wakiyama S, Ogawa M, et al. Negative impact of surgical site infection on long-term outcomes after hepatic resection for colorectal liver metastases. Anticancer Res. 2013;33(4):1697–703. [PubMed] [Google Scholar]
- 22.Farid SG, Aldouri A, Morris-Stiff G, Khan AZ, Toogood GJ, Lodge JP, et al. Correlation between postoperative infective complications and long-term outcomes after hepatic resection for colorectal liver metastasis. Ann Surg. 2010;251(1):91–100. doi: 10.1097/SLA.0b013e3181bfda3c. [DOI] [PubMed] [Google Scholar]