Abstract
Background:
The ACS-NSQIP risk calculator predicts perioperative risk. This study tested the calculator’s ability to predict risk for outcomes following retroperitoneal sarcoma (RPS) resection.
Methods:
The United States Sarcoma Collaborative database was queried for adults who underwent RPS resection. Estimated risk for outcomes was calculated twice in the risk calculator, once using sarcoma-specific CPT codes and once using codes indicative of most comorbid organ resection (e.g. nephrectomy). ROC curves were generated, with area under the curve (AUC) and Brier scores reported to assess discrimination and calibration. An AUC < 0.6 was considered ineffective discrimination. A negative ▲ Brier indicated improved performance relative to baseline outcome rates.
Results:
In total, 482 patients were identified with a 42.3% 90-day complication rate. Discrimination was poor for all outcomes except “all complications” and “renal failure”. Baseline outcome rates were better predictors than calculator estimates except for “discharge to nursing or rehab facility” and “renal failure”. Replacing sarcoma-specific CPT codes with resection-specific codes did not improve performance.
Conclusion:
The ACS-NSQIP risk calculator poorly predicted outcomes following RPS resection. Changing sarcoma-specific CPT to resection-specific codes did not improve performance. Comorbidities in the calculator may not effectively capture perioperative risk. Future work should evaluate a sarcoma-specific calculator.
Keywords: Brier, Perioperative, Calibration, Discrimination, Outcomes
Introduction:
Surgical risk is an important concept for surgeons to understand and convey to patients. Risk informs a patient’s perioperative expectations and allows for informed consideration of risks and benefits of an operation, and this shared decision making with patients is an important part of patient autonomy.[1] The American College of Surgeons National Quality Improvement Program (ACS-NSQIP) risk calculator is a well-validated method for collating multiple preoperative variables to obtain individualized 30-day morbidity and mortality metrics and has been championed in aiding surgeons in this type of dialogue.[2]
Originally created as a procedure-specific tool, a universal ACS-NSQIP risk calculator was created in 2013 which involves input of Current Procedure Technology (CPT) codes.[2–4] Since its inception, the risk calculator has been applied to several procedures, including various complex oncologic operations.[5–10] However, there are limited data regarding the calculator’s accuracy in predicting outcomes for patients undergoing retroperitoneal sarcoma (RPS) resection.[11]
Generally, the study of sarcomas has been limited by the overall heterogeneity and the rarity of these tumors.[12] However, multi-institutional collaborations, such as the Transatlantic Australasian Retroperitoneal Sarcoma working group and the United States Sarcoma Collaborative (USSC) pool resources between high-volume institutions to answer pertinent questions and create guidelines.[5, 13] We sought to use the 8-institution USSC database to evaluate the predictive ability of the ACS-NSQIP calculator to predict postoperative outcomes for patients undergoing resection of a RPS.[14] Unfortunately, the USSC database primarily reports 90-day outcomes, while ACS-NSQIP reports 30-day outcomes, limiting direct comparison. Both 30-day mortality and readmissions were available from the USSC database and were considered the primary outcomes of the study. However, 90-day USSC outcomes were still compared to 30-day NSQIP outcomes as this information is critical for appropriate preoperative patient discussions on operative risk. Given the inherent heterogeneity of sarcomas, we hypothesized that the calculator would be a poor predictor of postoperative outcomes, regardless of the use of RPS- vs. organ-resection-specific CPT codes.
Methods:
Patient Population
The patient cohort utilized in this study was compiled from the United States Sarcoma Collaboration database, a multi-institutional collaborative of 8 institutions, including University of Wisconsin, Emory University, Stanford University, Medical College of Wisconsin, Wake Forest University, The Ohio State University, University of Chicago Medicine, and Siteman Cancer Center, Washington University in St. Louis.[14] Patients who underwent a primary or recurrent retroperitoneal sarcoma (RPS) resection between January 2000 and February 2016, were identified. Initially, 944 patients who underwent RPS resection were identified. Any patient with missing data was excluded from the analysis, leaving 482 patients for the final analysis cohort (Figure I). Patient characteristics were then identified for entrance into the ACS-NSQIP risk calculator.
Figure I:
Flowchart outlining study design and model comparisons. Abbreviations: RPS – retroperitoneal sarcoma, NSQIP – National Surgical Quality Improvement Program, CPT – Current Procedural Technology Code, AUC – Area under the curve
ACS-NSQIP Risk Calculator
The American College of Surgeons universal risk calculator is an online application that was developed in 2013 to aid in the preoperative risk stratification of patients undergoing a multitude of general and subspecialty procedures.[2] The risk calculator can be found at the following web address: https://riskcalculator.facs.org/RiskCalculator/, and currently supports the entrance of 19 preoperative variables at the time of the study, including a single procedural CPT code to adjust for the magnitude of the operation performed. These variables include age, sex, functional status, emergency operation, American Society of Anesthesiology (ASA) class, steroid use, ascites within 30 days, systemic sepsis within 48 hours prior to surgery, ventilator dependence, disseminated cancer, diabetes, hypertension requiring medication, congestive heart failure prior to surgery, dyspnea, current smoker within 1 year, history of severe chronic obstructive pulmonary disease (COPD), dialysis, acute renal failure and height/weight for Body Mass Index (BMI) calculation. The calculator then provides the estimated risk of a given outcome at 30 days, including “serious complications”, “any complication”, “pneumonia”, “cardiac complication”, “surgical site infection”, “urinary tract infection”, “venous thromboembolism”, “renal failure”, “readmission”, “return to the operating room”, “discharge to a nursing or rehabilitation facility”, “sepsis”, “death” and “length of stay” (LOS).
All preoperative variables, except preoperative ascites, were contained within the USSC database. Sepsis, diabetes and dyspnea were available as binary variables and thus were entered in as “None”/“No” or “Sepsis”, “Insulin” and “At rest” respectively. The CPT codes 49203 (Excision or destruction, open, intra-abdominal tumors, cysts or endometriomas, 1 or more peritoneal, mesenteric, or retroperitoneal primary or secondary tumors; largest tumor 5 cm diameter or less), 49204 (49204 - Excision or destruction, open, intra-abdominal tumors, cysts or endometriomas, 1 or more peritoneal, mesenteric, or retroperitoneal primary or secondary tumors; largest tumor 5.1-10.0 cm diameter) and 49205 (49205 - Excision or destruction, open, intra-abdominal tumors, cysts or endometriomas, 1 or more peritoneal, mesenteric, or retroperitoneal primary or secondary tumors; largest tumor greater than 10.0 cm diameter) were entered based on the reported size of the retroperitoneal tumor (Figure I).
Two variables, “death” and “readmission” were available as 30-day outcomes in the USSC database and were considered the primary outcomes of this study. “Death” and “readmission” as well as “any complication”, “serious complications”, “surgical site infection”, “renal failure”, “return to OR”, “discharge to nursing or rehab facility”, and “LOS” were also available as 90-day outcomes and were considered secondary outcomes of the study. Although, the NSQIP calculator does not predict 90-day outcomes, 90-day outcomes have been shown to more accurately reflects patients’ postoperative courses following a major oncologic resection and may not significantly change the calculator predictions.[15–17] Complications were graded based on the Clavien-Dindo grading system.[18] Serious complications were considered any grade III-V Clavien-Dindo complication.
Statistical Analysis
Risk estimates provided by the ACS-NSQIP risk calculator were presented as median values with an associated range (min-max) for each postoperative outcome. Actual outcome rates were then calculated for those available using the collaborative database and presented as the median value with an associated range. Receiver Operating Characteristic (ROC) curve analysis was then conducted and the area under the curve (AUC), also known as the c-statistic, sensitivity, specificity and p-value with associated 95% confidence interval (CI) were reported. Reported sensitivity and specificity were selected based on the point of the ROC graph corresponding to Youden’s Index (sensitivity + specificity − 1).[19] Models with an AUC < 0.6 were considered ineffective; AUCs can range from 0.5 (random discrimination) to 1 (perfect discrimination).[20, 21] While effective discrimination, or the ability to determine which patients have an outcome, is necessary for accurate model performance, it is not sufficient. Calibration—the agreement between observed and predicted outcomes—is also important to assess.[21] Both calibration and discrimination can be tested using the Brier score, also known as the mean squared difference between predicted and observed outcomes. In general, the Brier score is a range between 0 and 1, with values closer to 0 indicating better model predictiveness. A ‘null Brier score’ may also be calculated using the observed rate of an outcome in the entire population. A negative difference between a Brier score and a null Brier score (▲Brier) indicates that predictions based on the model are more accurate than baseline cohort event rates. Wilcoxon signed-rank test was used to compare actual and predicted LOS and results were considered significant on the two-tailed test if p < 0.05. ROC curve and Brier score statistics were generated for the 30-day “death” and “readmission” outcomes, and then repeated for all 90-day outcomes. All ROC curve models were considered significant if p <0.05.
Due to the potential for an RPS-resection specific CPT code to underrepresent the morbidity of a multivisceral RPS resection, these calculations were repeated using the following CPT codes, including pancreaticoduodenectomy (48150), distal pancreatectomy (48140), gastrectomy (43632), nephrectomy (50230), or hepatectomy (47120) (Figure I). The organ-resection-specific CPT codes were then used to repeat the ROC curve and Brier score analysis for the updated cohort.
The database was constructed in Excel (Microsoft, Redmond, WA) and analysis was carried out in SPSS 26 (IBM, Armonk, NY). Brier scores were calculated using R (R Foundation for Statistical Computing, Vienna, Austria). A p-value < 0.05 was considered statistically significant unless otherwise indicated. All institutions obtained Institutional Review Board approval with an approved waiver of consent before beginning any research efforts.
Results:
Data from 482 RPS patients were entered in the ACS-NSQIP risk calculator. Table I shows the preoperative characteristics entered in the risk calculator. Notably, the majority (53.5%) of patients had a tumor > 10 cm, were younger than age 65 (64.5%), were female (56.2%) and were ASA class 3 (49.6%). The most common comorbidities in the cohort included hypertension (46.9%), diabetes (16%) and smoking (10.8%).
Table I:
Cohort demographics entered in the ACS-NSQIP risk calculator
| Input Variable | Frequency (%) | |
|---|---|---|
| Procedure Type | Sarcoma-Specific Procedure | |
| Sarcoma < 5 cm | 77 (16) | |
| Sarcoma 5-10 cm | 147 (30.5) | |
| Sarcoma > 10 cm | 258 (53.5) | |
| Resection-Specific Procedure | ||
| Pancreaticoduodenectomy | 9 (1.9) | |
| Distal Pancreatectomy | 24 (5) | |
| Hepatectomy | 20 (4.1) | |
| Nephrectomy | 84 (17.4) | |
| Gastrectomy | 30 (6.2) | |
| Sarcoma < 5 cm | 58 (12) | |
| Sarcoma 5-10 cm | 97 (20.1) | |
| Sarcoma > 10 cm | 160 (33.2) | |
| Age | < 65 | 311 (64.5) |
| 65-74 | 106 (22) | |
| 75-84 | 57 (11.8) | |
| >85 | 8 (1.7) | |
| BMI (median with interquartile range) | 26.3 (23.5-30.9) | |
| Male Gender | 211 (43.8) | |
| Functional Status | Independent | 465 (96.5) |
| Partially Dependent | 2 (0.4) | |
| Dependent | 15 (3.1) | |
| Emergent Operation | 6 (1.2) | |
| ASA Class | 1 | 46 (9.5) |
| 2 | 179 (37.1) | |
| 3 | 239 (49.6) | |
| 4 | 18 (3.7) | |
| Chronic Steroid | 3 (0.6) | |
| Systemic Sepsis | 11 (2.3) | |
| Ventilator Dependence | 0 (0) | |
| Disseminated Cancer | 28 (5.8) | |
| Diabetes | Insulin | 77 (16) |
| Hypertension | 226 (46.9) | |
| Congestive Heart Failure | 6 (1.2) | |
| Dyspnea | At rest | 7 (1.5) |
| Smoking History | 52 (10.8) | |
| Severe Chronic Obstructive Pulmonary Disease | 6 (1.2) | |
| Dialysis | 4 (0.8) | |
| Acute Renal Failure | 4 (0.8) |
Abbreviations: BMI - Body Mass Index, ASA - American Society of Anesthesiology
The actual complication rate and median estimated complication rate for each complication can be found in Table II. The actual rate of 30-day “death” and “readmission” was 1.2% and 13.3% respectively. Utilizing sarcoma-specific CPT codes, the median estimated “death” was 0.4% and the estimated “readmission” rate was 8.5%. The 90-day “death” and “readmission” rates were 3.9% and 15.8% respectively. The 90-day actual rate of “serious” and “all complications” in the cohort was 20.5% and 42.3%, while the median estimated “serious” and “all complication” rates were 11.4% and 14% respectively. Other common complications included “surgical site infection” (16.2% versus 6.6% estimated) and “return to OR” (8.5% versus 3%). On ROC curve analysis (Table III), only the 90-day outcomes “any complication”, “renal failure” and “discharge to nursing or rehab facility” were found to be statistically significant models with an area under the curve (AUC) of 0.64, 0.26 and 0.68 respectively. Brier scores for 30-day “death” (0.0126) and 90-day “renal failure” (0.0125), “return to operating room” (0.0808) and “discharge to nursing or rehab facility” (0.0679) were all close to zero, however, only “renal failure” (−0.0018) and “discharge to nursing or rehab facility” (−0.0030) had a negative ▲ Brier, indicating better discrimination and calibration of the model when compared to the multi-institutional event-rates. The observed “LOS” (7 days) was significantly longer than the estimated 5 days by the calculator (p < 0.01).
Table II:
Estimated risk from the entrance of the sarcoma-specific and resection-specific CPT codes
| Actual Event |
Estimated Risk |
||||
|---|---|---|---|---|---|
| Complication | N (%) | Sarcoma-Specific CPT Median Risk (%) | Range | Resection-Specific CPT Median Risk (%) | Range |
| 30-Day Outcomes | |||||
|
| |||||
| Death | 6 (1.2) | 0.4 | 0-12.2 | 0.5 | 0-19.3 |
| Readmission | 64 (13.3) | 8.5 | 0.7-22.7 | 8.4 | 0.1-25.4 |
|
| |||||
| 90-Day Outcomes | |||||
|
| |||||
| Serious Complication | 99 (20.5) | 11.4 | 1.3-37.4 | 11.8 | 3.6-40.4 |
| Any Complication | 204 (42.3) | 14 | 4.5-43.1 | 14 | 4.5-47.9 |
| Pneumonia | - | 1.1 | 0.1-23.6 | 1.3 | 0.1-26.4 |
| Cardiac Complication | - | 0.4 | 0-6.8 | 0.5 | 0-8.3 |
| Surgical Site Infection | 78 (16.2) | 6.6 | 1-43 | 6.3 | 0.1-25.9 |
| Urinary Tract Infection | - | 2.2 | 0.4-6.9 | 2.3 | 0.4-25.1 |
| Venous Thromboembolism | - | 2 | 0.3-11.5 | 2.2 | 0.3-11.5 |
| Renal Failure | 7 (1.5) | 0.4 | 0-8.5 | 0.6 | 0-9.5 |
| Return to Operating Room | 41 (8.5) | 3 | 0.6-12.5 | 2.8 | 0.6-15.4 |
| Discharge to Nursing or Rehab Facility | 37 (7.7) | 2.2 | 0-48.7 | 2.2 | 0-54.5 |
| Sepsis | - | 2.9 | 0-20.5 | 2.7 | 0-25.2 |
| Length of Stay (median days, interquartile range) | 7 (0-173) | 5 | 2.5-17 | 5 | 0.6-19 |
Abbreviations: CPT - Current Procedural Technology Code
Table III:
Sarcoma-specific CPT code ROC curve and Brier score analysis
| Predicted Complication | AUC | p-value | Sensitivity (%) | Specificity (%) | 95% CI | Brier Score | Null Brier Score | ▲Brier† |
|---|---|---|---|---|---|---|---|---|
| Compared to USSC 30-day Outcomes | ||||||||
|
| ||||||||
| Death | 0.40 | 0.38 | 33 | 57 | 0.16-0.64 | 0.0126 | 0.0123 | 0.0003 |
| Readmission | 0.55 | 0.20 | 52 | 52 | 0.48-0.62 | 0.1174 | 0.1330 | 0.0046 |
|
| ||||||||
| Compared to USSC 90-day Outcomes | ||||||||
|
| ||||||||
| Serious Complication | 0.58 | 0.02 | 53 | 56 | 0.51-0.64 | 0.1674 | 0.1632 | 0.0042 |
| Any Complication | 0.64 | < 0.001 | 59 | 58 | 0.59-0.69 | 0.3097 | 0.2444 | 0.0653 |
| Surgical Site Infection | 0.55 | 0.15 | 53 | 56 | 0.49-0.62 | 0.1439 | 0.1356 | 0.0082 |
| Renal Failure | 0.26 | 0.05 | 33 | 44 | 0.08-0.44 | 0.0125 | 0.0143 | −0.0018 |
| Return to Operating Room | 0.58 | 0.08 | 51 | 65 | 0.49-0.67 | 0.0808 | 0.0783 | 0.0025 |
| Discharge to Nursing or Rehab Facility | 0.68 | < 0.001 | 65 | 67 | 0.58-0.78 | 0.0679 | 0.0709 | −0.0030 |
A negative ▲Brier indicates a better discrimination/calibration than multiinstitutional event rates
Abbreviations: USSC – United States Sarcoma Collaborative, AUC – Area under the curve, CI – Confidence Interval
Given the higher-than-predicted rate of “death” and “readmission” in this cohort, we hypothesized that the estimated post-operative risk may be poorly predicted during multivisceral resections requiring additional procedures. Therefore, we selected five morbid procedures that cooccurred with sarcoma resection including pancreaticoduodenectomy, distal pancreatectomy, gastrectomy, hepatectomy and nephrectomy and re-ran the analysis calculator following the entrance of new organ-resection-specific CPT codes into the risk calculator. Notably, any patient that did not undergo a multivisceral resection was still included in the analysis using their sarcoma-specific CPT code. Rates of these procedures can be found in Table I. In total, 34.6% of the patient population underwent one of these additional procedures, most commonly nephrectomy (17.4%) and gastrectomy (6.2%). On ROC curve analysis (Table IV), there were more significant models than with sarcoma-specific CPT codes, including the 90-day outcomes “serious complications” (AUC = 0.57), “any complication” (AUC = 0.63), “surgical site infection” (0.57) and “discharge to nursing or rehab facility” (0.68), however, AUC values were still poor with only “any complication” and “discharge to nursing or rehab facility” reaching above 0.60. Neither 30-day “death” or “readmission” was found to be statistically significant. Brier scores were again close to zero for 30-day “death” (0.0127) and 90-day “renal failure” (0.0146), “return to operating room” (0.0809) and “discharge to nursing or rehab facility” (0.0680), with the ▲ Brier for “discharge to nursing and rehab facility” (−0.0029) being the only outcome with better discrimination and calibration than multi-institutional event rates. The actual median “LOS” (7 days) was also significantly different than the estimated 5 days by the calculator (p < 0.01).
Table IV:
Resection-specific CPT code ROC curve and Brier score analysis
| Predicted Complication | AUC | p-value | Sensitivity (%) | Specificity (%) | 95% CI | Brier Score | Null Brier Score | ▲Brier† |
|---|---|---|---|---|---|---|---|---|
| Compared to USSC 30-day Outcomes | ||||||||
|
| ||||||||
| Death | 0.44 | 0.63 | 50 | 50 | 0.19-0.70 | 0.0127 | 0.0123 | 0.0004 |
| Readmission | 0.57 | 0.07 | 57 | 68 | 0.50-0.63 | 0.1375 | 0.1330 | 0.0045 |
|
| ||||||||
| Compared to USSC 90-day Outcomes | ||||||||
|
| ||||||||
| Serious Complication | 0.57 | 0.03 | 53 | 54 | 0.51-0.63 | 0.1676 | 0.1632 | 0.0044 |
| Any Complication | 0.63 | < 0.001 | 58 | 59 | 0.58-0.68 | 0.3086 | 0.2444 | 0.0643 |
| Surgical Site Infection | 0.57 | 0.05 | 53 | 53 | 0.50-0.64 | 0.1442 | 0.1356 | 0.0086 |
| Renal Failure | 0.3 | 0.09 | 33 | 47 | 0.04-0.55 | 0.0146 | 0.0143 | 0.0003 |
| Return to Operating Room | 0.56 | 0.20 | 56 | 53 | 0.47-0.65 | 0.0809 | 0.0783 | 0.0027 |
| Discharge to Nursing or Rehab Facility | 0.68 | < 0.001 | 65 | 68 | 0.58-0.78 | 0.0680 | 0.0709 | −0.0029 |
A negative ▲Brier indicates a better discrimination/calibration than multiinstitutional event rates
Abbreviations: USSC – United States Sarcoma Collaborative, AUC – Area under the curve, CI – Confidence Interval
Discussion:
The ACS-NSQIP risk calculator has been reported as an efficient way to produce personalized risk information for thousands of procedures.[2, 7] We sought to use a large, multi-institutional sarcoma database to determine the calculator’s performance for 482 patients who underwent retroperitoneal sarcoma resection (RPS). Our data demonstrate that the calculator underpredicted readmission and mortality rates at 30 days and did not improve outcome prediction over baseline event rates. The utilization of organ resection-specific CPT (rather than generic sarcoma resection CPT) codes did not improve model performance.
Although studies have not examined patients undergoing RPS resection, to date, prior literature detailing patient outcomes following operative management of other soft tissue sarcomas has suggested poor reliability of the ACS-NSQIP risk calculator. A smaller study examining the calculator’s accuracy for 265 patients who underwent extremity and truncal soft tissue sarcoma extirpation with pedicled or free flap reconstruction (with the entrance of free flap CPT codes) showed an AUC for any complication of 0.626 with a Brier score of 0.242.[11] Outcomes from this study were similar, with an AUC of 0.64 and Brier score of 0.3097. In both studies, overall complication rates were underestimated. Slump et al. found an overall estimated complication rate of 15.35% versus an actual complication rate of 32.5%. Likewise, in this study, an overall complication rate was estimated to be 14% while the observed was found to be 42.3%. Underestimation of ACS-NSQIP risk calculator event rates has been noted in other oncologic studies, including with resection of sacral chordomas, pancreatic neuroendocrine tumors and gastric cancer.[8, 10, 22] Although 90-day outcomes certainly contributed to the appearance of underestimation of complications by the calculator, it also occurred with 30-day mortality and readmission. Further, inclusion of 90-day data provides valuable prognostic information for patients undergoing complex oncologic operations and aids in painting a realistic picture of the patient’s postoperative course.[15, 16]
The current recommendation for the resection of RPS is to achieve grossly negative margins, which often involves a multivisceral resection.[23] Park and colleagues found in a large study of 846 patients undergoing resection of RPS that multivisceral resections increased the 30-day morbidity and serious morbidity rates on multivariable analysis with an odds ratio of 1.83 and 1.86 respectively. To attempt to account for any increase in surgical risk imposed by a multivisceral resection, five additional procedure codes were selected for their increased morbidity to replace existing RPS-specific CPT codes, including gastrectomy, nephrectomy, pancreatectomy and hepatectomy, and the analysis was repeated. Interestingly, although there were differences in the predicted risk, this was not likely to be clinically relevant (e.g. estimated mortality rate of 0.4% versus 0.5%), nor were the estimated rates consistently higher than those estimated by the sarcoma-specific CPT codes. This resulted in similar model predictiveness and likely reflects the calculator’s inherent inability to accommodate more than 1 CPT code; limiting its effectiveness in such multivisceral procedures.
Despite its merits, this study has several notable limitations. All NSQIP outcomes are 30-day estimates, while most USSC outcomes are reported at 90 days, except for mortality and readmission. While the calculator could be directly compared to USSC event rates for mortality and readmission rates, the remainder of the comparisons are less robust. When the calculator is shown to underestimate complication rates (as occurred in this study), it is impossible to know if this was caused by calculator performance (inaccurate estimation), high rates of complications in the 30- to 90-day period, or a combination of the two. Regardless, the knowledge that the calculator underpredicts adverse outcomes in the 90-day postoperative period can be useful for patient discussions of risk. Additionally, this study was limited to complete case analysis, (patients with complete data available). Despite an attempt to increase the reliability of the data being used, this may have introduced selection bias. Studies have demonstrated that missing data are more common in healthier patients, which in this study implies that by excluding those patients there may be an over-representation of highly complex cases.[24] However, no study, to date, has systematically analyzed the effects of missing data on the performance of the ACS-NSQIP risk calculator. Second, although data were present for all preoperative variables except the presence of ascites, postoperative outcomes including postoperative pneumonia, cardiac complications, urinary tract infection, venous thromboembolism and sepsis were not available for evaluation. This precludes making comments about the calculator’s ability to assess for these outcomes and is an inherent limitation in the design of the collaborative database. Finally, it is possible that other calculators, such as P-POSSUM, could offer better predictive power, however, the current collaborative database does not contain the requisite variables necessary to make such model comparisons.[25]
Given the calculator’s limited ability to handle multivisceral resections and the possible increased risk that comes with these procedures, future work could include considering an option to add more than one CPT code to the risk calculator. Furthermore, the poor performance of the calculator across two separate studies of sarcomas may indicate a need for a sarcoma-specific risk calculator. A Transatlantic Australasian Retroperitoneal Sarcoma study of over 1000 patients noted an increased multivisceral resection score, increased transfusion requirement and increased age to be predictive of increased morbidity on multivariable analysis.[26] Of the three independent predictors in this study, only age is currently included in the risk calculator. Given mortality in this study was found to triple (1.2% to 3.9%) between 30 and 90 days, consideration should be given to including an option to assess both 30- and 90-day outcomes. Additionally, the calculator gives an option for “Surgeon Adjustment of Risks”, which allows for manual risk adjustment based on the CPT code entered in the model. This was not adjusted during this study, however future studies could investigate the role of manual adjustment in cases, such as in RPS resection, where the calculator underestimates risk.
Conclusions:
This study utilized the 8-institution United States Sarcoma Collaborative database to evaluate the ACS-NSQIP risk calculator’s ability to predict risk for patient’s undergoing retroperitoneal sarcoma resection. These data suggest that the calculator has limited ability to predict risk in this patient population, even when accounting for higher morbidity procedures. The heterogeneity of sarcoma resection, which often involves complex multivisceral resections, likely limits the calculator’s predictive power, and inclusion of multiple CPT codes and the creation of a sarcoma specific-risk calculator should be considered in future iterations of the ACS-NSQIP risk calculator.
Synopsis:
The ACS-NSQIP risk calculator predicts perioperative risk. This study sought to test the calculator’s ability to predict risk for outcomes following retroperitoneal sarcoma resection. Using a multiinstitutional database, we found the risk calculator poorly predicted perioperative risk following RPS resection.
Acknowledgments:
Grant Support: This research was supported by the he National Institutes of Health under Award Number T32 CA090217 and Award Number T32 ES007015. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health
Abbreviations:
- USSC
United States Sarcoma Collaborative
- ACS-NSQIP
American College of Surgeons National Quality Improvement Program
- CPT
Current Procedural Technology
- RPS
Retroperitoneal Sarcoma
- COPD
Chronic Obstructive Pulmonary Disease
- LOS
Length of Stay
- OR
Operating Room
- BMI
Body Mass Index
- AUC
Area Under the Curve
- ROC
Receiver Operating Characteristic
- ASA
American Society of Anesthesiology
Footnotes
Data Availability Statement:
The data that support the findings of this study are available from the corresponding author upon reasonable request.
References:
- 1.Paruch JL, Ko CY, Bilimoria KY (2014) An opportunity to improve informed consent and shared decision making: the role of the ACS NSQIP Surgical Risk Calculator in oncology. Ann Surg Oncol 21:5–7. 10.1245/s10434-013-3345-3 [DOI] [PubMed] [Google Scholar]
- 2.Bilimoria KY, Liu Y, Paruch JL, et al. (2013) Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg 217:833-842.e1–3. 10.1016/j.jamcollsurg.2013.07.385 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Cohen ME, Bilimoria KY, Ko CY, Hall BL (2009) Development of an American College of Surgeons National Surgery Quality Improvement Program: morbidity and mortality risk calculator for colorectal surgery. J Am Coll Surg 208:1009–1016. 10.1016/j.jamcollsurg.2009.01.043 [DOI] [PubMed] [Google Scholar]
- 4.Parikh P, Shiloach M, Cohen ME, et al. (2010) Pancreatectomy risk calculator: an ACS-NSQIP resource. HPB 12:488–497. 10.1111/j.1477-2574.2010.00216.x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Johnson AC, Ethun CG, Liu Y, et al. (2018) Studying a Rare Disease Using Multi-Institutional Research Collaborations vs Big Data: Where Lies the Truth? J Am Coll Surg 227:357–366.e3. 10.1016/j.jamcollsurg.2018.05.009 [DOI] [PubMed] [Google Scholar]
- 6.Armstrong EA, Beal EW, Lopez-Aguiar AG, et al. (2019) Evaluating the ACS-NSQIP Risk Calculator in Primary GI Neuroendocrine Tumor: Results from the United States Neuroendocrine Tumor Study Group. Am Surg 85:1334–1340 [PMC free article] [PubMed] [Google Scholar]
- 7.Beal EW, Lyon E, Kearney J, et al. (2017) Evaluating the American College of Surgeons National Surgical Quality Improvement project risk calculator: results from the U.S. Extrahepatic Biliary Malignancy Consortium. HPB 19:1104–1111. 10.1016/j.hpb.2017.08.009 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Dave A, Beal EW, Lopez-Aguiar AG, et al. (2019) Evaluating the ACS NSQIP Risk Calculator in Primary Pancreatic Neuroendocrine Tumor: Results from the US Neuroendocrine Tumor Study Group. J Gastrointest Surg Off J Soc Surg Aliment Tract 23:2225–2231. 10.1007/s11605-019-04120-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Sahara K, Paredes AZ, Merath K, et al. (2020) Evaluation of the ACS NSQIP Surgical Risk Calculator in Elderly Patients Undergoing Hepatectomy for Hepatocellular Carcinoma. J Gastrointest Surg Off J Soc Surg Aliment Tract 24:551–559. 10.1007/s11605-019-04174-4 [DOI] [PubMed] [Google Scholar]
- 10.Beal EW, Saunders ND, Kearney JF, et al. (2018) Accuracy of the ACS NSQIP Online Risk Calculator Depends on How You Look at It: Results from the United States Gastric Cancer Collaborative. Am Surg 84:358–364 [PubMed] [Google Scholar]
- 11.Slump J, Ferguson PC, Wunder JS, et al. (2016) Can the ACS-NSQIP surgical risk calculator predict post-operative complications in patients undergoing flap reconstruction following soft tissue sarcoma resection? J Surg Oncol 114:570–575. 10.1002/jso.24357 [DOI] [PubMed] [Google Scholar]
- 12.Brennan MF, Antonescu CR, Moraco N, Singer S (2014) Lessons learned from the study of 10,000 patients with soft tissue sarcoma. Ann Surg 260:416–422. 10.1097/SLA.0000000000000869 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Trans-Atlantic RPS Working Group (2015) Management of primary retroperitoneal sarcoma (RPS) in the adult: a consensus approach from the Trans-Atlantic RPS Working Group. Ann Surg Oncol 22:256–263. 10.1245/s10434-014-3965-2 [DOI] [PubMed] [Google Scholar]
- 14.[dataset] From the Authors in the United States Sarcoma Collaborative (2016) Multiinstitutional Sarcoma Database
- 15.Mise Y, Vauthey J-N, Zimmitti G, et al. (2015) 90-day postoperative mortality is a legitimate measure of hepatopancreatobiliary surgical quality. Ann Surg 262:1071–1078. 10.1097/SLA.0000000000001048 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Damhuis R a M, Wijnhoven BPL, Plaisier PW, et al. (2012) Comparison of 30-day, 90-day and in-hospital postoperative mortality for eight different cancer types. Br J Surg 99:1149–1154. 10.1002/bjs.8813 [DOI] [PubMed] [Google Scholar]
- 17.Wingert NC, Gotoff J, Parrilla E, et al. (2016) The ACS NSQIP Risk Calculator Is a Fair Predictor of Acute Periprosthetic Joint Infection. Clin Orthop 474:1643–1648. 10.1007/s11999-016-4717-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Dindo D, Demartines N, Clavien P-A (2004) Classification of surgical complications: a new proposal with evaluation in a cohort of 6336 patients and results of a survey. Ann Surg 240:205–213. 10.1097/01.sla.0000133083.54934.ae [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Youden WJ (1950) Index for rating diagnostic tests. Cancer 3:32–35. [DOI] [PubMed] [Google Scholar]
- 20.Pencina MJ, D’Agostino RB (2015) Evaluating Discrimination of Risk Prediction Models: The C Statistic. JAMA 314:1063–1064. 10.1001/jama.2015.11082 [DOI] [PubMed] [Google Scholar]
- 21.Steyerberg EW, Vickers AJ, Cook NR, et al. (2010) Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiol Camb Mass 21:128–138. 10.1097/EDE.0b013e3181c30fb2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Houdek MT, Hevesi M, Griffin AM, et al. (2020) Can the ACS-NSQIP surgical risk calculator predict postoperative complications in patients undergoing sacral tumor resection for chordoma? J Surg Oncol. 10.1002/jso.25865 [DOI] [PubMed] [Google Scholar]
- 23.Fairweather M, Gonzalez RJ, Strauss D, Raut CP (2018) Current principles of surgery for retroperitoneal sarcomas. J Surg Oncol 117:33–41. 10.1002/jso.24919 [DOI] [PubMed] [Google Scholar]
- 24.Parsons HM, Henderson WG, Ziegenfuss JY, et al. (2011) Missing data and interpretation of cancer surgery outcomes at the American College of Surgeons National Surgical Quality Improvement Program. J Am Coll Surg 213:379–391. 10.1016/j.jamcollsurg.2011.05.010 [DOI] [PubMed] [Google Scholar]
- 25.Richards CH, Leitch FE, Horgan PG, McMillan DC (2010) A systematic review of POSSUM and its related models as predictors of post-operative mortality and morbidity in patients undergoing surgery for colorectal cancer. J Gastrointest Surg Off J Soc Surg Aliment Tract 14:1511–1520. 10.1007/s11605-010-1333-5 [DOI] [PubMed] [Google Scholar]
- 26.MacNeill AJ, Gronchi A, Miceli R, et al. (2018) Postoperative Morbidity After Radical Resection of Primary Retroperitoneal Sarcoma: A Report From the Transatlantic RPS Working Group. Ann Surg 267:959–964. 10.1097/SLA.0000000000002250 [DOI] [PubMed] [Google Scholar]