Evaluation of the American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator in Gynecologic Oncology Patients Undergoing Minimally Invasive Surgery

Abstract

Study Objective:

To evaluate the ability of the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) surgical risk calculator to predict discharge to postacute care and perioperative complications in gynecologic oncology patients undergoing minimally invasive surgery (MIS).

Design:

A retrospective chart review (Canadian Task Force classification II-1).

Setting:

A university hospital.

Patients:

All patients undergoing MIS on the gynecologic oncology service from January 1, 2009, to December 30, 2013.

Interventions:

Surgical procedures were reviewed, and appropriate Common Procedural Terminology codes were assigned. Twenty-one preoperative risk factors were abstracted from the chart and entered into the ACS NSQIP surgical risk calculator. The predicted risk of discharge to postacute care and 8 additional postoperative complications were calculated and recorded. Actual postoperative complications were abstracted from the medical record. The association between the calculated risk and the actual outcome was determined using logistic regression. The ability of the calculator to accurately predict a particular event was assessed using the c-statistic and Brier score.

Measurements and Main Results:

Of the 876 patients reviewed, a majority underwent hysterectomy (71.6%), with almost half of those patients undergoing additional cancer staging procedures (34.8%). Although the calculator was a poor predictor of postoperative complications, it was a strong predictor for discharge to postacute care (c-statistic = 0.91, Brier score = 0.02) with an odds ratio of 2.31 (95% confidence interval, 1.65–3.25; p < .0001).

Conclusion:

The ACS NSQIP surgical risk calculator does not accurately predict postoperative complications or length of stay in gynecologic oncology patients undergoing MIS. Although it was a strong predictor of need for discharge to postacute care, it vastly overestimated the number of patients requiring this service. Therefore, the calculator’s risk score for discharge to postacute care may be considered during preoperative counseling but should not be a predictor of whether or not the patient should proceed with surgery.

Minimally invasive surgery (MIS) has been used increasingly in gynecologic oncology since the presentation of a randomized controlled trial showing that laparoscopic surgery was feasible and safe for the staging of endometrial cancer [1]. MIS is currently preferred when possible because of decreased blood loss, earlier return of bowel function, fewer wound complications, shorter hospital stay, faster return to normal activities, and improved quality of life [1–6]. There are specific advantages in the gynecologic oncology population, including increased lymph node counts during lymphadenectomy and fewer overall complications, that reduce delays in adjuvant treatment, especially in obese patients [7–11]. Because of these favorable outcomes from MIS, many insurance companies are classifying MIS as an outpatient procedure. Simultaneously, hospital ratings are increasingly based on publicly available postoperative complication and readmission rates. These conflicting priorities make preoperative prediction of postoperative complications and the need for longer hospital stay or discharge to postacute care essential.

To address this need, the American College of Surgeons (ACS) developed a surgical risk calculator as part of the National Surgical Quality Improvement Project (NSQIP). This web-based tool, which predicts 8 postoperative complications within 30 days of surgery, length of stay, and need for discharge to postacute care, was developed using preoperative and postoperative data from more than 1.4 million patients at 393 NSQIP hospitals. These data were used to develop a regression model, and the strength of association between a preoperative variable and postoperative outcome in the data set, as measured by the regression coefficient, was used to develop the risk calculation. Data from all surgical specialties except for trauma and transplant were used; however, gynecologic surgeries contributed only 5.3% of the cases [12]. Furthermore, only a portion of these gynecologic surgeries were performed on oncology patients who were likely older and sicker; had more extensive surgical procedures; and had a higher risk of postoperative complications, longer hospital stays, and greater need for postacute care compared with their benign counterparts [13–16]. Additionally, it is unclear how many of these procedures were performed via a minimally invasive approach, and it is possible that the lower incidence of postoperative complications associated with MIS may decrease the predictive ability of the surgical risk calculator. The objective of this study was to evaluate the predictive ability of the ACS NSQIP surgical risk calculator to predict 8 postoperative complications, the length of hospital stay, and discharge to postacute care in gynecologic oncology patients undergoing MIS.

Methods

Study Population

University of Minnesota Institutional Review Board approval was obtained for this study. All MIS procedures, laparoscopic or robotic, performed at the University of Minnesota, Minneapolis, MN, by the gynecologic oncology service between January 1, 2009, and December 31, 2013, were identified through a query of the gynecologic oncology surgical database. A retrospective review of the electronic health record was performed to identify the 21 preoperative variables required by the surgical risk calculator (Table 1): (1) age group (<65 years, 65–74 years, 75–84 years, or >84 years), (2) sex (female), (3) functional status (independent, partially dependent, or totally dependent), (4) emergent nature of the procedure (yes or no), (5) ASA class (1–5), (6) wound class (clean, clean contaminated, contaminated, or dirty infected), (7) diabetes (no; yes, oral medications; or yes, insulin), (8) hypertension requiring medication (yes or no), (9) a previous cardiac event (yes or no), (10) congestive heart failure within 30 days of surgery (yes or no), (11) dyspnea with exertion (none, with moderate exertion, or at rest), (12) chronic steroid use (yes or no), (13) smoking status within 1 year of surgery (yes or no), (14) severe chronic obstructive pulmonary disease (yes or no), (15) ascites within 30 days of surgery (yes or no), (16) sepsis within 48 hours of surgery (none, systemic inflammatory response syndrome, sepsis, or septic shock), (17) acute renal failure (yes or no), (18) dialysis dependence (yes or no), (19) ventilator dependence (yes or no), (20) body mass index in kg/m², and (21) the presence of disseminated cancer as determined by preoperative imaging (yes or no). The surgery Common Procedural Terminology (CPT) code and 21 preoperative risk factors were entered into the online ACS NSQIP surgical risk calculator (http://riskcalculator.facs.org), and the risk of 8 postoperative complications (Table 1; death, pneumonia, cardiac event, surgical site infection, urinary tract infection, venous thromboembolic event, renal failure, and return to the operating room), risk of discharge to postacute care, and predicted length of hospital stay were calculated and recorded. The occurrence of the 8 postoperative complications, length of hospital stay, and disposition of the patient postoperatively were abstracted from the medical record for comparison with the predicted risks.

Table 1.

Preoperative variables and postoperative predicted outcomes of the American College of Surgeons National Surgical Quality Improvement Surgical Risk Calculator

Preoperative	Postoperative
predictive variables	predicted outcomes
Age group (in years: <65, 65–74, 75–84, >84)	Death
Sex (female)	Pneumonia
Functional status (independent, partially dependent, totally dependent)	Cardiac event
Emergent nature of the procedure (yes, no)	Surgical site infection
American Society of Anesthesiologists class (1–5)	Urinary tract infection
Wound class (clean, clean contaminated, contaminated, dirty infected)	Venous thromboembolic event
Diabetes (no; yes, oral medications; yes, insulin)	Renal failure
Hypertension requiring medication (yes, no)	Return to the operating room
Previous cardiac event (yes, no)	Any complication*
Congestive heart failure within 30 days of surgery (yes, no)	Serious complication†
Dyspnea with exertion (none, with moderate exertion, at rest)	Length of stay (in days)
Chronic steroid use (yes, no)	Discharge to postacute care
Smoking status within 1 year of surgery (yes, no)
Severe chronic obstructive pulmonary disease (yes, no)
Ascites within 30 days of surgery (yes, no)
Sepsis within 48 hours of surgery (none, SIRS, sepsis, septic shock)
Acute renal failure (yes, no)
Dialysis dependence (yes, no)
Ventilator dependence (yes, no)
Body mass index (in kg/m2)
Presence of disseminated cancer on preoperative imaging (yes, no)

SIRS = systemic inflammatory response syndrome.

^*Any complication defined as superficial incisional SSI, deep incisional SSI, organ space SSI, wound disruption, pneumonia, unplanned intubation, pulmonary embolus, ventilator >48 hours, progressive renal insufficiency, acute renal failure, urinary tract infection, stroke, cardiac arrest, myocardial infarction, deep vein thrombosis, or systemic sepsis.

^†Serious complications defined as death, cardiac arrest, myocardial infarction, pneumonia, progressive renal insufficiency, acute renal failure, pulmonary embolus, deep venous thrombosis, return to the operating room, deep incisional SSI, organ space SSI, systemic sepsis, unplanned intubation, urinary tract infection, or wound disruption.

Surgical Procedures

All operative reports were reviewed, and appropriate CPT codes were assigned. For surgeries that included more than 1 CPT code, each CPT code was entered into the calculator with the preoperative variables, and the highest-risk CPT code for each patient was used in the final analysis. The surgical procedures were categorized as (1) <hysterectomy (adnexal surgery and/or trachelectomy), (2) hysterectomy ± bilateral salpingo-oophorectomy, (3) staging (any of the previously mentioned procedures plus pelvic and/or para-aortic lymphadenectomy ± omentectomy ± appendectomy), or (4) debulking (any of the previously mentioned procedures plus procedures to remove gross metastatic disease). Cases that were initiated via a minimally invasive technique but that were converted to laparotomy were excluded. All surgeries were performed by 7 fellowship-trained gynecologic oncology surgeons.

Statistical Methods

Baseline demographic and clinical characteristics were summarized. The association between calculated risk and actual outcome was determined using logistic regression; odds ratios and 95% confidence intervals (CI) are presented. The ability of the ACS NSQIP calculator to accurately predict a particular event was assessed using the c-statistic and Brier score. The c-statistic, also known as the area under a receiving operating characteristic curve, determines the probability that predicting the outcome is better than chance. The c-statistic ranges from 0.5 to 1.0, with 0.5 indicating the model is no better than chance and 1.0 indicating the model perfectly predicts the outcome. Models are typically considered reasonable when the c-statistic is higher than 0.7 and strong when it is greater than 0.8. The Brier score describes the mean squared differences between the predicted risk and the actual outcome. If the model perfectly predicts the outcome, the Brier score is 0. All statistical analyses were performed using SAS 9.3 (SAS Institute, Cary, NC).

Results

Between January 1, 2009, and December 31, 2013, 876 patients underwent MIS on the gynecologic oncology service. The majority of patients were <65 years of age (73.3%), independent before surgery (98.9%), ASA class 1 to 2 (60.5%), and overweight or obese (73.8%) (Table 2). Four hundred ninety-two patients (56.2%) had a final diagnosis of cancer, and 34 patients (3.9%) had disseminated cancer per preoperative imaging. A majority of patients underwent minimally invasive hysterectomy (71.7%), with almost half of those patients undergoing additional cancer staging procedures (34.9%).

Table 2.

Patient and surgical characteristics

Variable	N	%
Age group
<65 years	641	73.3
65–74 years	166	19.0
75–84 years	55	6.3
>85 years	13	1.5
Functional status
Independent	865	98.9
Partially dependent	8	0.9
Totally dependent	2	0.2
Emergency case
No	869	99.3
Yes	6	0.7
ASA class
1: healthy patient	62	7.1
2: mild systemic disease	467	53.4
3: severe systemic disease	344	39.3
4: severe systemic disease/threat to life	2	0.2
Wound class
Clean	274	31.3
Clean/contaminated	601	68.7
Steroid use of chronic condition
No	852	97.4
Yes	23	2.6
Ascites within 30 days before surgery
No	874	99.9
Yes	1	0.1
Systemic sepsis within 48 hours before surgery
No	874	99.9
Yes	1	0.1
Ventilator dependent
No	875	100.0
Yes	0	0.0
Disseminated cancer
No	841	96.1
Yes	34	3.9
Diabetes
None	770	88.0
Oral	71	8.1
Insulin	34	3.9
Hypertension requiring medication
No	524	59.9
Yes	351	40.1
Type of surgery
<Hysterectomy*	246	28.1
Hysterectomy ± bilateral salpingo-oophorectomy	322	36.8
Staging	305	34.9
Debulking	2	0.2

ASA = American Society of Anesthesiologists.

^*Adnexal surgery without hysterectomy or other staging/debulking procedures.

Postoperative complications were rare among the entire cohort, with only 100 patients (11%) experiencing any complication. There were no statistically significant differences in the median risk scores between those experiencing a postoperative complication and those who did not, and the calculator was not an accurate predictor of postoperative complications with c-statistics of 0.49 to 0.57 for any complication, serious complication, surgical site infection, and urinary tract infection; risk scores could not be calculated for the other 4 complications (death, pneumonia, cardiac events, venous thromboembolic event, and renal failure) because of the low number of events (Tables 3 and 4).

Table 3.

Median risk scores by outcome

	Did not have event		Had event
Outcome	N	Median (minimum–maximum)	N	Median (minimum–maximum)
Death	875	0.5 (0.5–5.0)	0	—
Any serious complication	804	4.0 (1.0–13.0)	71	4.0 (2.0–7.0)
Any complication	775	5.0 (2.0–19.0)	100	6.0 (2.0–14.0)
Pneumonia	868	0.5 (0.5–1.0)	7	0.5 (—)
Cardiac	873	0.5 (0.5–1.0)	2	0.5 (—)
SSI	852	2.0 (0.5–12.0)	23	2.0 (1.0–3.0)
UTI	829	2.0 (0.5–7.0)	46	2.0 (1.0–4.0)
VTE	873	0.5 (0.5–3.0)	2	0.5 (—)
Renal failure	871	0.5 (0.5–2.0)	1	0.5 (—)
Postacute care discharge	857	0.5 (0.5–6.0)	18	1.0 (1.0–5.0)

SSI = surgical site infection; UTI = urinary tract infection; VTE = venous thromboembolic event.

Table 4.

Complication rates and model statistics

Outcome	Events, n (%)	Odds ratio (95% CI), p value	c-statistic	Brier score
Death	0 (0.0)	NE	NE	NE
Any serious complication	71 (8.1)	1.05 (0.85–1.30), p = .63	0.53	0.07
Any complication	100 (11.4)	1.08 (0.99–1.18), p = .09	0.57	0.10
Pneumonia	7 (0.8)	NE	NE	NE
Cardiac	2 (0.2)	NE	NE	NE
SSI	23 (2.6)	1.09 (0.73–1.62), p = .68	0.53	0.03
UTI	46 (5.3)	1.00 (0.70–1.41), p = .98	0.49	0.05
VTE	2 (0.2)	NE	NE	NE
Renal failure	1 (0.1)	NE	NE	NE
Discharge to postacute care	18 (2.1)	2.31 (1.65–3.25), p < .0001	0.91	0.02

NE = not evaluable; SSI = surgical site infection; UTI = urinary tract infection; VTE = venous thromboembolic event.

There was little variation in the predicted and actual length of stay, with a mean predicted length of stay of 1.1 ± 0.4 days and an actual mean hospital stay of 1.0 ± 1.4 days. The predicted length of stay was statistically significantly but only weakly correlated with the actual length of stay (r = .22, p < .0001). The length of stay was underestimated for 169 (19.3%), overestimated for 345 (39.4%), and accurately predicted for 361 (41.3%) patients.

In the study cohort, 18 patients (2.1%) were discharged to postacute care. Despite the calculator’s inability to accurately predict postoperative complications, the calculator was able to predict discharge to postacute care, with a c-statistic of 0.91 and a Brier score of 0.02 (Table 4). The median risk score for those discharged to postacute care was 1% (range, 1%–5%) compared with 0.5% (range, 0.5%–6%) for those who were discharged to home (odds ratio = 2.31; 95% CI, 1.65–3.25; p < .0001). Using a risk score of 1% as the cutoff, the sensitivity of the calculator was 100% (95% CI, 81.5%–100%), and specificity was 75.9% (95% CI, 72.9%–78.7%). Based on an event prevalence of 2.1% in this study, the negative predictive value was 100% (95% CI, 99.4%–100.0%), and the positive predictive value was 8.0% (95% CI, 4.8%–12.4%).

Discussion

Same-day discharge for MIS patients has previously been shown to be safe in gynecologic patients undergoing surgery for benign and malignant indications [17–27]. Postoperative complications have been shown to be the strongest predictor of 30-day postoperative readmission [15,16]. Unfortunately, the results of our study showed that the ACS NSQIP calculator was a poor predictor for postoperative complications within 30 days in gynecologic oncology patients undergoing MIS. Our previous evaluation of the ACS NSQIP calculator in gynecologic oncology patients undergoing laparotomy also showed poorer predictive capabilities in the gynecologic oncology population compared with general and colorectal surgery patients against whom the universal surgical calculator was previously compared [12,28]. However, our evaluation of patients undergoing laparotomy showed that the calculator was a good predictor of specific serious complications such as cardiac complications and death; the predictive ability of these complications could not be calculated in our MIS population because of the rare occurrence of these complications. The low incidence of all complications in this population may contribute to the overall poor predictive ability of the calculator in the MIS gynecologic oncology population. There is scant literature evaluating other complication prediction models for patients undergoing MIS. A clinical prediction model developed specifically for benign gynecologic surgery patients identified preoperative sepsis, ascites, unintentional weight loss, preoperative systemic inflammatory response syndrome, history of a cerebrovascular accident, morbid obesity, ASA class 3 or greater, current smoking status, and preoperative anemia to be risk factors for postoperative complications [29]. By assigning points to each variable and calculating a composite score, the investigators were able to identify patients at low (2.4%–2.7%), medium (6.3%–6.8%), and high (23.8%–29.5%) risk of complications. However, only 2.4% of the procedures in the entire study cohort were performed via an MIS approach.

Gynecologic oncology patients are at increased risk for readmission compared with their benign counterparts because of increased comorbidities, worse preoperative condition, and increased risk of postoperative complications [13–16]. Preoperative conditions that have been associated with an increased risk of postoperative readmission include higher ASA class, insulin-dependent diabetes, congestive heart failure, significant weight loss before surgery, tobacco use, use of corticosteroids, and preexisting renal failure [15,16]; these risk factors, with the exception of weight loss, are included as variables in the ACS NSQIP surgical risk calculator. Although a NSQIP study evaluating risk factors for 30-day postoperative readmission identified discharge to postacute care as a risk factor for readmission [16], this may have been a surrogate for sicker patients, and it is possible that discharge to appropriate postacute care may decrease an individual patient’s risk of readmission because of postoperative exacerbation of preexisting conditions. Additionally, setting realistic patient expectations has been shown to be associated with improved patient satisfaction [30], and the Center for Medicare and Medicaid Services requires that discussion of discharge expectations to postacute care be included in the informed consent process [31]. This component of the discussion can easily be missed in patients undergoing MIS because the majority of patients will have an uncomplicated postoperative course and be able to care for themselves in their own home. The c-statistic and Brier score suggest that the ACS NSQIP surgical risk calculator is a strong predictor of discharge to postacute care. Unfortunately, closer inspection of the data shows that the calculator achieves a high sensitivity by overestimating the number of patients who would require discharge to postacute care. Therefore, although the use of the ACS NSQIP surgical risk calculator can help identify those patients undergoing MIS with whom potential discharge to a postacute care facility should be considered during preoperative planning, patients need to be counseled that a majority of patients with an elevated risk score are still able to be discharged to home. Furthermore, this predicted risk should not be a strong determinant of whether or not to proceed with surgery.

The strengths of this study include the relatively large number of patients included for whom we had complete preoperative and postoperative data. The breadth of MIS procedures performed is typical of a gynecologic oncology service and included both cancer staging as well as advanced laparoscopic procedures performed for benign indications. The study was performed at an academic health center, which is a referral center for complicated medical patients of varying socioeconomic and insurance statuses. The calculator was also evaluated using the same statistical methods (c-statistic and Brier score) as the original validation article [12]. The primary limitation of the study is the fact that this was a retrospective evaluation of a calculator designed to be used prospectively. The calculator includes a “surgeon risk adjustment” tool that modifies risk scores based on the surgeon’s overall impression of the patient, and this risk adjustment could not be applied retrospectively. Only patients whose surgery was completed via MIS were included in the analysis in an attempt to truly evaluate the predictive ability of the surgical risk calculator for minimally invasive procedures. However, when used prospectively, some of these intended MISs will be converted to laparotomy, and this may change the complication event rate and predictive ability of the calculator. Further review of the data showed that during this time period, there were 24 surgeries (2.7% of all intended MISs) that were initiated via a minimally invasive approach but converted to laparotomy. This small number of converted surgeries is unlikely to significantly change the results, especially because previous evaluations of the surgical risk calculator’s predictive abilities in gynecologic oncology patients have shown poor performance overall [28,32]. As is a limitation of all retrospective studies, our data abstraction was limited to the information included in the medical record. Although we had complete preoperative data for all patients and complete postoperative information up to their postoperative clinic visit, there is the potential that complications diagnosed and treated at an outside facility were not captured in our data abstraction. However, a previous study performed on a subgroup of these patients showed that we had follow-up data on 95% of patients through clinic visits and phone calls from patients, families, or other health care facilities [24]. All of the patients included in this study were treated at a single academic health center, which standardized the postoperative care and evaluation and criteria for discharge to postacute care; however, this may limit the generalizability of our results.

In summary, the ACS NSQIP surgical risk calculator did not accurately predict postoperative complications in gynecologic oncology patients undergoing MIS. Although statistically the calculator was found to be a strong predictor of discharge to postacute care, it achieved this by overestimating the number of patients who would require additional posthospitalization care. Therefore, although the risk of discharge to postacute care can be considered in preoperative counseling and planning, the calculator should not be used to determine discharge location or whether or not a patient should proceed with surgery, even when nonsurgical treatment options are available.