Abstract
Objective
The objective of this study was to evaluate the ability of the American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) surgical risk calculator to predict complications in gynecologic oncology patients undergoing laparotomy.
Methods
A chart review of patients who underwent laparotomy on the gynecologic oncology service at a single academic hospital from January 2009 to December 2013 was performed. Preoperative variables were abstracted and NSQIP surgical risk scores were calculated. The risk of any complication, serious complication, death, urinary tract infection, venous thromboembolism, cardiac event, renal complication, pneumonia and surgical site infection were correlated with actual patient outcomes using logistic regression. The c-statistic and Brier score were used to calculate the prediction capability of the risk calculator.
Results
Of the 1,094 patients reviewed, the majority were <65 years old (70.9%), independent (95.2%), ASA class 1-2 (67.3%), and overweight or obese (76.1%). Higher calculated risk scores were associated with an increased risk of the actual complication occurring for all events (p<0.05). The calculator performed best for predicting death (c-statistic=0.851, Brier=0.008) and cardiac complications (c-statistic=0.708, Brier=0.011). The calculator did not accurately predict most complications.
Conclusions
The NSQIP surgical risk calculator adequately predicts specific serious complications, such as postoperative death and cardiac complications. However, the overall performance of the calculator was worse for gynecologic oncology patients than reported in general surgery patients. A tailored prediction model may be needed for this patient population.
1. Introduction
In 2015 there will be an estimated 98,280 new gynecologic cancer diagnoses[1], and a majority of these patients will undergo surgery as part of their treatment. As the population ages and becomes more obese, gynecologic oncologists are operating on increasingly complex patients with multiple medical co-morbidities for both benign and malignant gynecologic conditions. Therefore, it is important for both patients and providers to thoroughly understand the unique risks for each patient, and identify which patients may benefit from medical rather than surgical management. Data on hospital outcomes, including surgical complication rates, are publically available, and the Centers for Medicare and Medicaid Services may begin to use the Physician Quality Reporting System to incentivize surgeons to discuss surgical risks prior to each elective operation[2]. While the importance of surgical risk counseling and stratification is growing in importance, our ability to accurately predict surgical complications is lacking.
The American College of Surgeons National Surgical Quality Improvement Project (ACS NSQIP) collects standardized clinical data on pre-operative risk factors and post-operative complications from over 500 hospitals in the United States[2-4]. In 2009, Cohen et al [5] developed a risk calculator specific to colorectal patients in order to inform surgical decision making and the informed consent process. More recently, this same group developed a universal surgical risk calculator[6]. Preoperative and postoperative data from more than 1.4 million patients at 393 ACS NSQIP hospitals were used to develop a regression model. The strengths of association identified between a preoperative variable and postoperative outcome in the dataset, as measured by the regression coefficient, were used to develop the risk calculation. This model was used to create a web-based tool which calculates the risk of nine post-operative complications within 30 days of surgery. Although data from all surgical specialties except for trauma and transplant were used to develop the calculator, the indication for surgery was not included in the model. It is conceivable that gynecologic cancer patients, who often present after weeks of debilitating symptoms and with suboptimal nutritional status, are at higher risk of postoperative morbidity than the general surgery population.
The objective of the current study was to evaluate the ability of the ACS NSQIP surgical risk calculator to accurately predict complications in gynecologic oncology patients undergoing laparotomy.
2. Materials & Methods
The study was approved by the University of Minnesota Institutional Review Board. All patients who underwent a laparotomy procedure with the gynecologic oncology service at the University of Minnesota Medical Center from January 1, 2009 through December 30, 2013 were identified through the gynecologic oncology surgical database. The surgical procedures were reviewed and the appropriate CPT codes assigned. Since the calculator allows only one CPT code entry, the calculator was run multiple times for multi-procedure surgeries without a bundled CPT code. For the statistical analysis, the calculator scores for the highest risk procedure were used. For instance, in surgical procedures including a hysterectomy, bilateral salpingo-oophorectomy and appendectomy, the calculator risk scores for the hysterectomy with bilateral salpingo-oophorectomy (CPT code 58150) was used. For patients undergoing a cytoreductive surgery for metastatic cancer, the initial analysis was performed using the calculated risk scores for a hysterectomy, bilateral salpingo-oophorectomy and debulking (CPT codes 58952-58954 or 58958-58958). However, for patients who underwent a bowel resection as part of their cytoreductive procedure, the calculator was run again using the CPT codes for the bowel resection procedure (CPT codes 44320, 44345, 49560, 58150, 58548, 58740, 48951-58953 or 58956) and the results were reported separately as a subgroup analysis.
The electronic health record was accessed to obtain the 21 preoperative risk factors required by the ACS NSQIP surgical risk calculator (Table 1): 1) age group (<65 years; 65-74 years; 75-84 years; >84 years); 2) sex (female); 3) functional status (independent; partially dependent; totally dependent); 4) emergent nature of the procedure (yes; no); 5) American Society of Anesthesiologists (ASA) class (I-V); 6) wound class (clean; clean-contaminated; contaminated; dirty-infected); 7) diabetes (no; yes, oral medications; yes, insulin); 8) hypertension requiring medication (yes; no); 9) previous cardiac event (yes; no); 10) congestive heart failure within 30 days of surgery (yes; no); 11) dyspnea with exertion (none; with moderate exertion; at rest); 12) chronic steroid use (yes; no); 13) smoking status within one year of surgery (yes; no); 14) severe chronic obstructive pulmonary disease (yes; no); 15) ascites within 30 days of surgery (yes; no); 16) sepsis within 48 hours of surgery (none; systemic inflammatory response syndrome (SIRS); sepsis; septic shock); 17) acute renal failure (yes; no); 18) dialysis dependence (yes; no); 19) ventilator dependence (yes; no); 20) body mass index (BMI) in kg/m2 and 21) presence of disseminated cancer as determined by preoperative imaging (yes; no).
Table 1. ACS NSQIP surgical risk calculator preoperative predictive variables and predicted postoperative outcomes.
| Preoperative Predictive Variables | Postoperative Predicted Outcomes |
|---|---|
| Age group (in years: <65, 65-74, 75-84, >84) | Death |
| Sex (female) | Pneumonia |
| Functional status (independent; partially dependent; totally dependent) | Cardiac event |
| Emergent nature of the procedure (yes; no) | Surgical site infection (SSI) |
| American Society of Anesthesiologists (ASA) class (I-V) | Urinary tract infection (UTI) |
| Wound class (clean; clean-contaminated; contaminated; dirty-infected) | Venous thromboembolic event (VTE) |
| Diabetes (no; yes, oral medications; yes, insulin) | Renal failure |
| Hypertension requiring medication (yes; no) | Return to the operating room |
| Previous cardiac event (yes; no) | Length of stay (in days) |
| Congestive heart failure within 30 days of surgery (yes; no) | |
| Dyspnea with exertion (none; with moderate exertion; at rest) | |
| Chronic steroid use (yes; no) | |
| Smoking status within 1 year of surgery (yes; no) | |
| Severe chronic obstructive pulmonary disease (yes; no) | |
| Ascites within 30 days of surgery (yes; no) | |
| Sepsis within 48 hours of surgery (none; SIRS*; sepsis; septic shock) | |
| Acute renal failure (yes; no) | |
| Dialysis dependence (yes; no) | |
| Ventilator dependence (yes; no) | |
| Body mass index (in kg/m2) | |
| Presence of disseminated cancer on preoperative imaging (yes; no) |
SIRS=Systemic Inflammatory Response Syndrome
The preoperative data were entered into the online ACS NSQIP surgical risk calculator (http:.//riskcalculator.facs.org), and the predicted risks of the eight post-operative complications (death, pneumonia, cardiac event, surgical site infection (SSI), urinary tract infection (UTI), venous thromboembolic event (VTE), renal failure, return to the OR) as well as predicted length of hospital stay were recorded (Table 1). Finally, actual postoperative complications experienced by the patients were abstracted from the medical record.
Baseline demographic and clinical characteristics were summarized. Surgical procedures were categorized as follows: <hysterectomy (e.g. adnexal surgery); hysterectomy ± bilateral salpingo-oophorectomy; staging (lymphadenectomy ±omentectomy± hysterectomy ±salpingo-oophorectomy); debulking (any of the preceding procedures plus removal of gross tumor); bowel resection; exenteration (anterior, posterior or total). The correlation between calculated risk and actual outcome was determined using logistic regression. The ability of the ACS NSQIP calculator to accurately predict a particular event was assessed using the c-statistic and Brier score. The c-statistic, also known as the area under the curve of a receiving operating characteristic curve, determines the probability that predicting the outcome is better than chance. The c-statistic ranges from 0.5 to 1.0, with 0.5 indicating the model is no better than chance and 1.0 indicating the model perfectly predicts the outcome. Models are typically considered reasonable when the c-statistic is higher than 0.7 and strong when it is greater than 0.8. The Brier score describes the mean squared differences between the predicted risk and the actual outcome. If the model perfectly predicts the outcome, the Brier score is 0. A pre-planned subgroup analysis was performed to compare the accuracy of the calculator using a general debulking code versus a more specific bowel resection code for patients undergoing cytoreductive surgery including a bowel resection. The Wilcoxon rank sum test compared the c-statistic and Brier score for each of the CPT codes. All statistical analyses were performed using SAS 9.3 (SAS Institute, Cary, NC).
3. Results
Between January 2009 and December 2013, 1,077 individuals underwent a total of 1,094 surgeries; 17 individuals underwent two separate surgeries at least one month apart and data from both surgeries were included. Demographic data are presented in Table 2. The majority of the patients were < 65 years old (70.9%), overweight or obese (76.1%), ASA class I-II (67.3%), and had an independent functional status (95.2%). A majority of patients (62.5%) had a cancer surgery, including a cancer staging procedure (36.2%), debulking procedure (23.8%) or exenteration (1.5%) (Table 3).
Table 2.
Preoperative demographic and clinical data (N=1,094).
| Variable | n (%) |
|---|---|
| Age Group | |
| <65 years | 776 (70.9) |
| 65-74 years | 213 (19.5) |
| 75-84 years | 94 (8.6) |
| >85 years | 11 (1.0) |
| Functional Status | |
| Independent | 1030 (95.2) |
| Partially dependent | 57 (5.2) |
| Totally dependent | 7 (0.6) |
| ASA Class | |
| Healthy patient | 219 (20.0) |
| Mild systemic disease | 517 (47.3) |
| Severe systemic disease | 353 (32.3) |
| Severe systemic disease | 5 (0.5) |
| BMI Category | |
| Under weight (<18.5 kg/m2) | 14 (1.3) |
| Normal (18.5-24.9 kg/m2) | 246 (22.7) |
| Over weight (25-29.9 kg/m2) | 260 (24.0) |
| Obese (≥30 kg/m2) | 565 (52.1) |
| Diabetes | |
| None | 947 (86.6) |
| Oral | 90 (8.2) |
| Insulin | 57 (5.2) |
| Hypertension | 457 (41.8) |
| Previous cardiac event | 67 (6.1) |
| CHF within 30 days | 17 (1.6) |
| Dyspnea | |
| None | 961 (87.8) |
| Moderate exertion | 113 (10.3) |
| At rest | 20 (1.8) |
| History of severe COPD | 37 (3.4) |
| Dialysis | 1 (0.2) |
| Acute renal failure | 7 (0.6) |
| Chronic steroids | 39 (3.6) |
| Ascites within 30 days | 11 (14.7) |
| Emergency Case | 7 (0.6) |
American Society of Anesthesiologists Classification
Table 3. Surgical data (N=1,094).
| Variable | n (%) |
|---|---|
| Type of Surgery | |
| < Hysterectomy* | 151 (13.8) |
| Hysterectomyϯ | 271 (24.8) |
| Staging§ | 396 (36.2) |
| Debulking¶ | 260 (23.8) |
| Exenteration# | 16 (1.5) |
| Wound Class | |
| Clean | 248 (22.7) |
| Clean/contaminated | 812 (74.3) |
| Contaminated | 33 (3.0) |
Adnexal surgery, no staging/debulking.
Hysterectomy with or without salpingo-oophorectomy, no staging/debulking.
Lymphadenectomy and/or omentectomy with or without any of the above procedures.
Removal of gross tumor including but not requiring bowel resection and/or upper abdominal procedures (diaphragm stripping or resection, splenectomy, distal pancreatectomy, liver resection, porta hepatis resection or cholecystectomy), with or without any of the above procedures.
Anterior, posterior or total.
Higher calculated risk scores were associated with increased risk of actual complications for all events (p <0.05 in all logistic regression models; Table 4). The calculator performed best for predicting death (c-statistic=0.851, Brier=0.008), renal failure (c-statistic=0.752, Brier=0.015) and cardiac complications (c-statistic=0.708, Brier=0.011). However, the scores did not discriminate well for most complications. This was illustrated by a large overlap in risk scores (Figure 1) and low c-statistic values and higher Brier scores (Table 4). The calculator performed the worst for overall complication risk (c-statistic=0.635, Brier=0.323), serious complication risk (c-statistic=0.644, Brier=0.148) and SSI risk (c-statistic=0.625, Brier=0.126).
Table 4.
Correlation between Calculated Risk and Outcomes, and Model Statistics for the NSQIP Risk Calculator in Gynecologic Oncology Patients (N=1,094).
| Did Not Have Event | Had Event | ||||||
|---|---|---|---|---|---|---|---|
|
| |||||||
| Outcome | n (%) | Median Risk (Min, Max) | n (%) | Median Risk (Min, Max) | Odds Ratio (95% CI)* | c-statisticϯ | Brier score¶ |
|
| |||||||
| Death | 1085 (99.2) | 0.5 (0.5-37.0) | 9 (0.8) | 6.0 (0.5-22.0) | 1.18 (1.08-1.29) | 0.851 | 0.007 |
| Serious complication | 884 (80.8) | 7.0 (2.0-37.0) | 210 (19.2) | 9.0 (2.0-37.0) | 1.09 (1.07-1.12) | 0.644 | 0.148 |
| Any complication | 726 (66.4) | 9.0 (0.5-51.0) | 368 (33.6) | 12.0 (1.0-54.0) | 1.06 (1.04-1.08) | 0.635 | 0.323 |
| Pneumonia | 1055 (96.4) | 0.5 (0.5-7.0) | 39 (3.6) | 1.0 (0.5-4.0) | 1.23 (0.91-1.66) | 0.591 | 0.034 |
| Cardiac | 1082 (98.9) | 0.5 (0.5-8.0) | 12 (1.1) | 1.0 (0.5-1.0) | 1.48 (0.83-2.63) | 0.708 | 0.011 |
| SSI | 927 (84.7) | 4.0 (1.0-36.0) | 167 (15.3) | 5.0 (1.0-26.0) | 1.14 (1.08-1.19) | 0.625 | 0.126 |
| UTI | 1003 (91.7) | 2.0 (0.5-29.0) | 91 (8.3) | 3.0 (1.0-24.0) | 1.14 (1.08-1.21) | 0.619 | 0.075 |
| VTE | 1074 (98.2) | 1.0 (0.5-10.0) | 20 (1.8) | 2.0 (0.5-6.0) | 1.42 (1.10-1.83) | 0.655 | 0.018 |
| Renal failure | 1070 (98.3) | 0.5 (0.5-7.0) | 18 (1.7) | 1.0 (0.5-4.0) | 1.91 (1.30-2.79) | 0.752 | 0.015 |
p-value <0.05 for all outcomes.
c-statistic: Range 0.5-1.0, where 0.5 is no better than chance and 1.0 indicates that the model perfectly predicts the outcome. Models are considered reasonable when the c-statistic is >0.7, and strong when the c-statistic is >0.8.
The Brier score describes the mean squared differences between the predicted risk and the actual outcome. The Brier score is 0 when the model perfectly predicts the outcome.
Figure 1.
The boxplots represent the ACS NSQIP Surgical Risk Calculator results for each outcome. The calculated risks for those who experienced the event (blue) are compared to those did not experience the event (red).
SSI=surgical site infection, UTI=urinary tract infection, VTE=venous thromboembolism.
The performance of the calculator in patients with disseminated cancer was compared to that of patients who did not have disseminated cancer (benign or malignant diagnoses) at the time of surgery. The calculator performed slightly better in patients without disseminated cancer (c-statistic 0.625 vs 0.582; Brier score 0.196 vs 0.239), however, did not reach the c-statistic threshold for a reliable model. A total of 62 patients underwent a cytoreductive procedure which included a bowel resection. With the exception of the risk for UTI, the calculated risks for all outcomes were equal to or higher than the calculated risks using the debulking code (p<0.0001 for all). Given the low occurrence of certain complications we were only able to compare the c-statistic and Brier score for serious complication, any complication, SSI and UTI (Table 5). The calculator did not predict complications significantly better when using the more specific bowel resection codes compared to the general primary debulking code.
Table 5.
Prediction of Events by CPT Code Entered for patients undergoing tumor debulking with bowel resections (N=62).
| Debulking CPT codes* | Bowel Resection CPT codesϯ | ||||||
|---|---|---|---|---|---|---|---|
|
| |||||||
| Outcome | Events n (%) | Odd Ratio (95% CI) | c-statistic | Brier score | Odds Ratio (95% CI) | c-statistic | Brier score |
| Death | 0 (0) | - | - | - | - | - | - |
| Serious complication | 25 (40.3) | 1.07 (0.98-1.18) | 0.568 | 0.232 | 1.07 (1.00-1.15) | 0.614 | 0.224 |
| Any complication | 38 (61.3) | 1.04 (0.98-1.12) | 0.574 | 0.235 | 1.04 (0.99-1.10) | 0.580 | 0.229 |
| Pneumonia | 3 (4.8) | - | - | - | - | - | - |
| Cardiac | 0 (0.0) | - | - | - | - | - | - |
| SSI | 9 (14.5) | 0.92 (0.70-1.20) | 0.546 | 0.134 | 1.02 (0.86-1.20) | 0.536 | 0.124 |
| UTI | 14 (22.6) | 1.04 (0.83-1.30) | 0.549 | 0.165 | 1.30 (1.01-1.67) | 0.703 | 0.161 |
| VTE | 3 (4.8) | - | - | - | - | - | - |
| Renal failure | 0 (0.0) | - | - | - | - | - | - |
CPT codes 58952-58954 or 58958-58958.
CPT codes 44320, 44345, 49560, 58150, 58548, 58740, 48951-58953 or 58956.
The risk calculator underestimated the length of hospital stay on average by approximately 1 day with a predicted mean hospital stay of 3.6 ± 2.0 days compared to actual mean hospital stay of 4.7 ±4.2 days (p<0.0001). The calculator underestimated the hospital stay for 55%, correctly predicted the hospital stay for 13.2% and over-estimated the length of stay for 31.8% of patients.
4. Discussion
The ACS NSQIP calculator was developed to help surgeons to better counsel patients for planned surgical procedures[6]. Although the calculator was developed with a large multi-institutional data set which included gynecologic oncology surgeries, gynecologic procedures (benign and malignant) comprised only 5.3% of the cohort[6, refer to Appendix]. The results of our study showed that the ACS NSQIP calculator accurately predicted specific severe complications such as cardiac events and death, but did not accurately predict more common complications. Thus, while the calculator does not help to individualize the risk/benefit counseling for complications in general, serious complications, or specific but common complications such as SSI, the ACS NSQIP surgical calculator may be used as a tool to help determine which patients would benefit from medical rather than surgical treatment due to high risk of life-threatening postoperative complications. Due to large overlaps in calculated risk, the calculator should not be used as the sole determinant for medical treatment, but higher than average calculated risks for death, cardiac complications and renal failure should be considered when the risk/benefit ratio is of surgery is otherwise unclear.
Healthcare providers, patients and payers are increasing their focus on quality of care. Therefore, we need a way to assess a patient's unique surgical risk so that we can appropriately counsel patients about surgical vs. non-surgical treatment options and appropriately compare postoperative outcomes to an individual patient's baseline risk. A recent study by Wright et al[14] evaluating compliance with Surgical Care Improvement Project (SCIP) measures (administration of appropriate antibiotics within 1 hour of surgery; discontinuation of antibiotics within 24 hours of surgery; appropriate ordering and administration of venous thromboembolism prophylaxis), patient satisfaction scores and hospital-reported 30-day mortality rates for medical conditions with perioperative complications and death showed poor correlation when comparing hospitals at the lowest and highest performance levels for each measure, suggesting that high-quality hospital care cannot completely overcome baseline patient risks. Several risk calculators have been developed for specific surgical subspecialties such as colorectal or bariatric surgery, specific procedures such as pancreatectomy and specific complications such as cardiac events[5, 7-10]. However, the ACS NSQIP calculator was the first universal surgical risk calculator. The universal calculator tends to overestimate risk for patients who do not have an adverse event while appropriately predicting the risk of those patients who do encounter a complication. In general, the universal surgical risk calculator performs better in the general surgery population, with c-statistics for all seven predicted postoperative complications of >0.8, indicating a strong prediction model [6]. In contrast, in our gynecologic oncology cohort, only death had a c-statistic of >0.8, and only cardiac events and renal failure met the threshold of a reliable model with a c-statistic of 0.7. No specific gynecologic surgical calculator is currently available for comparison to the universal calculator.
The results of our study are comparable to the results of another study evaluating the performance of the ACS NSQIP calculator in 628 gynecologic oncology patients [11]. This study showed high predictive value of the ACS NSQIP calculator for death (Brier score 0.004) and VTE (Brier score 0.003), but poor predictive value for cardiac complications, SSI and pneumonia. The differences in outcomes could be due to the differences in study design. In the previous study both laparoscopic and laparotomy procedures were included, while in this study only laparotomies were included. Additionally, for procedures requiring more than one CPT code, we used only the CPT code which yielded the highest risk scores, while the previous study used a composite score of all of the procedures. Despite these differences in study design, both studies concluded that a more specific calculator may be need for gynecologic oncology procedures.
There are several reasons why the ACS NSQIP surgical risk calculator may not perform as well in the gynecologic oncology population as in the greater surgery population. First, the intraperitoneal disease burden in ovarian and endometrial cancer patients varies, and thus the complexity and subsequent surgical risk of a cytoreductive procedure is likely to vary. Second, the calculator allows the entry of only one CPT code. However, for many of the staging and debulking surgeries several procedures are performed and it is difficult to determine a cause and effect relationship between a specific procedure and a specific complication, or to evaluate for an additive or synergistic effect. Third, many patients with advanced gynecologic malignancies experience anorexia in the weeks to months prior to surgery resulting in poor nutritional status. The ACS NSQIP calculator does not include laboratory values that surgeons may not have available at the time of the preoperative counseling visit. Thus, serum albumin level is not included as a preoperative variable, although it has been identified as a predictor of postoperative morbidity in other studies[12, 13]. Lastly, since the calculator performed only marginally better in patients without disseminated cancer, it is possible that under representation of gynecologic procedures in the initial model rather than cancer-specific variables may account for the difference in performance. However, is unclear what factors differentiate gynecologic procedures from other procedures in the same wound class.
Other indices have also been used in an attempt to predict perioperative morbidity. The Charlson comorbidity index assigns risk based on patient age plus severity of comorbidities, with a higher score indicating higher risk. In a study evaluating the correlation between the Charlson comorbidity index and outcomes following ovarian cancer cytoreductive surgery, the comorbidity index was found to be negatively correlated with progression-free and overall survival, but showed no correlation with the rate of minor or major perioperative complications [15]. The modified frailty index, which calculates a score based on 11 NSQIP-derived co-morbidity variables, has been shown to positively correlate with severe complications such as need for intensive care unit level of care and death within 30 days of surgery; however, to our knowledge its predictive value for more common but less severe complications has not been evaluated [13]. Another study used the NSQIP database to evaluate the effect of age, race, functional status, ASA class, preoperative albumin and number of cytoreductive procedures on morbidity and mortality of women undergoing surgery for ovarian cancer. Multivariate analysis showed that the number of cytoreductive procedures and preoperative albumin were the best predictors of morbidity [12].
The ACS NSQIP surgical calculator underestimated postoperative hospital stay in our patient population. While the surgical calculator estimated hospital stay is based on the surgical procedure and predicted risk of postoperative complications, there are social factors which also influence the length of hospital stay. Hospital discharge can be postponed due to need for bed availability in a transitional care unit for continued physical therapy and rehabilitation. These needs may be magnified in our patient population, which comes from a wide geographic region and includes many patients with limited or no health insurance. An updated version of the ACS NSQIP calculator estimates the risk of discharge to a nursing or rehabilitation facility. If this estimate is accurate, it may aid in preoperative discharge planning and decrease prolongation of hospital stay due to placement issues.
The strengths of our study are the relatively large number of surgical patients included and the availability of data on all of the preoperative variables and postoperative complications included in the ACS NSQIP calculator for 99% of patients. For comparison, we tested the calculator with the same statistical tools (c-statistic, Brier score) that were utilized in the initial publication[6]. The primary limitation of this study is that retrospective data was used to evaluate a tool which was designed to be used prospectively. We were unable to adjust risk based on the surgeon's overall impression of the patient (the surgeon risk adjustment function in the calculator), which may have improved correlation between predicted risk and surgical outcomes. All data were collected from patient medical records, and therefore data may have been misclassified or misinterpreted. Given the wide geographic distribution of our patients, it is also possible that we were missing information on postoperative complications managed at outside clinics and hospitals. However, a previous study performed by this group showed that we had follow-up data on 95% of patients through clinic visits and phone calls from patients or their outside providers[16]. Finally, given that this study was performed at a single institution there may be differences in outcomes based on practice patterns at the single institution.
In conclusion, the ACS NSQIP surgical calculator accurately predicts which patients would benefit from medical management, including neoadjuvant chemotherapy, in lieu of surgery due to high risk of cardiac complications or death. However, in the gynecologic oncology population it does not accurately predict the risk of more common complications such as SSI, UTI or pneumonia. A more specific surgical risk calculator may be needed for this unique and complex gynecologic oncology patient population.
Acknowledgments
Dr. Deanna Teoh's research is partially funded by the Masonic Cancer Center Woman's Health Scholarship sponsored by the University of Minnesota Masonic Cancer Center. She also has a KCI/Acelity grant for an unrelated investigator-initiated study.
Ms. Isaksson Vogel's work is supported by NIH grant P30 CA77598 utilizing the Biostatistics and Bioinformatics Core shared resource of the Masonic Cancer Center, University of Minnesota and by the National Center for Advancing Translational Sciences of the National Institutes of Health Award Number UL1TR000114.
Research reported in this publication was supported by NIH grant P30 CA77598 utilizing the Biostatistics and Bioinformatics Core shared resource of the Masonic Cancer Center, University of Minnesota and by the National Center for Advancing Translational Sciences of the National Institutes of Health Award Number UL1TR000114.
Research reported in this publication was supported by The Masonic Cancer Center Women's Health Scholarship sponsored by the University of Minnesota Masonic Cancer Center, a comprehensive cancer center designated by the National Cancer Institute, and administrated by the University of Minnesota Deborah E. Powell Center for Women's Health.
Footnotes
Presented at: Society of Gynecologic Oncology annual meeting, Chicago, IL 3/2015
American College of Surgeons Clinical Congress, Chicago, IL 10/2015
Conflict of Interest Statement: The remaining authors declare that they have no conflicts of interest.
References
- 1.Siegel RL, Miller KD, Jemal A. Cancer Statistics, 2015. CA Cancer J Clin. 2015 Jan;65(1):5–29. doi: 10.3322/caac.21254. [DOI] [PubMed] [Google Scholar]
- 2.Cohen ME, Ko CY, Bilimoria KY, Zhou L, Huffman K, Wang X, et al. Optimizing ACS NSQIP modeling for evaluation of surgical quality and risk: patient risk adjustment, procedure mix adjustment, shrinkage adjustment, and surgical focus. J Am Coll Surg. 2013 Aug;217(2):336–46 e1. doi: 10.1016/j.jamcollsurg.2013.02.027. [DOI] [PubMed] [Google Scholar]
- 3.Hall BL, Hamilton BH, Richards K, Bilimoria KY, Cohen ME, Ko CY. Does surgical quality improve in the American College of Surgeons National Surgical Quality Improvement Program: an evaluation of all participating hospitals. Ann Surg. 2009 Sep;250(3):363–76. doi: 10.1097/SLA.0b013e3181b4148f. [DOI] [PubMed] [Google Scholar]
- 4.Ingraham AM, Richards KE, Hall BL, Ko CY. Quality improvement in surgery: the American College of Surgeons National Surgical Quality Improvement Program approach. Adv Surg. 2010;44:251–67. doi: 10.1016/j.yasu.2010.05.003. [DOI] [PubMed] [Google Scholar]
- 5.Cohen ME, Bilimoria KY, Ko CY, Hall BL. Development of an American College of Surgeons National Surgery Quality Improvement Program: morbidity and mortality risk calculator for colorectal surgery. J Am Coll Surg. 2009 Jun;208(6):1009–16. doi: 10.1016/j.jamcollsurg.2009.01.043. [DOI] [PubMed] [Google Scholar]
- 6.Bilimoria KY, Liu Y, Paruch JL, Zhou L, Kmiecik TE, Ko CY, et al. Development and evaluation of the university ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg. 2013 Nov;217(5):833–42 e1-3. doi: 10.1016/j.jamcollsurg.2013.07.385. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Finks JF, Kole KL, Yenumula PR, English WJ, Krause KR, Carlin AM, et al. Predicting risk for serious complications with bariatric surgery: results from the Michigan Bariatric Surgery Collaborative. Ann Surg. 2011 Oct;254(4):633–40. doi: 10.1097/SLA.0b013e318230058c. [DOI] [PubMed] [Google Scholar]
- 8.Gupta PK, Franck C, Miller WJ, Gupta H, Forse RA, et al. Development and validation of a bariatric surgery morbidity risk calculator using the prospective, multicenter NSQIP dataset. J Am Coll Surg. 2011 Mar;212(3):301–9. doi: 10.1016/j.jamcollsurg.2010.11.003. [DOI] [PubMed] [Google Scholar]
- 9.Gupta PK, Gupta H, Sundaram A, Kaushik M, Fang X, Miller WJ, et al. Development and validation of a risk calculator for prediction of cardiac risk after surgery. Circulation. 2011 Jul;124(4):381–7. doi: 10.1161/CIRCULATIONAHA.110.015701. [DOI] [PubMed] [Google Scholar]
- 10.Parikh P, Shiloach M, Cohen ME, Bilimoria KY, Ko CY, Hall BL, et al. Pancreatectomy risk calculator: an ACS-NSQIP resource. HPB (Oxford) 2010 Sep;12(7):488–97. doi: 10.1111/j.1477-2574.2010.00216.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Szender JB, Frederick PJ, Eng KH, Akers SN, Lele SB, Odunsi K. Evaluation of the National Surgical Quality Improvement Program Universal Surgical Risk Calculator for a gynecologic oncology service. Int J Gynecol Cancer. 2015 Mar;25(3):512–20. doi: 10.1097/IGC.0000000000000378. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Patankar S, Burke WM, Hou JY, Tergas AI, Huang Y, Ananth CV, et al. Risk stratification and outcomes of women undergoing surgery for ovarian cancer. GynecolOncol. 2015 Jul;138(1):62–9. doi: 10.1016/j.ygyno.2015.04.037. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Uppal S, Igwe E, Rice LW, Spencer RJ, Rose SL. Frailty index predicts severe complications in gynecologic oncology patients. GynecolOncol. 2015 Apr;137(1):98–101. doi: 10.1016/j.ygyno.2015.01.532. [DOI] [PubMed] [Google Scholar]
- 14.Wright JD, Tergas AI, Ananth CV, Burke WM, Chen L, Neugut AI, et al. Relationship between surgical oncologic outcomes and publically reported hospital quality and satisfaction measures. J Natl Cancer Inst. 2015 Mar;107(3) doi: 10.1093/jnci/dju409. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Suidan RS, Leitao MM, Zivanovic, Gardner GJ, Long Roche KC, Sonoda Y, et al. Predictive value of the Age-Adjusted Charlson Comorbidity Index on perioperative complications and survival in patients undergoing primary debulking surgery for advanced epithelial ovarian cancer. GynecolOncol. 2015 Aug;138(2):246–51. doi: 10.1016/j.ygyno.2015.05.034. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Rivard C, Casserly K, Anderson M, Isaksson Vogel R, Teoh D. Factors influencing same-day hospital discharge and risk factors for readmission after robotic surgery in the gynecologic oncology patient population. J Minim Invasive Gynecol. 2015 Feb;22(2):219–26. doi: 10.1016/j.jmig.2014.10.001. [DOI] [PMC free article] [PubMed] [Google Scholar]