Evaluating the ACS NSQIP Risk Calculator in Primary Pancreatic Neuroendocrine Tumor: Results from the US Neuroendocrine Tumor Study Group

Abstract

Background

In a changing health care environment where patient outcomes will be more closely scrutinized, the ability to predict surgical complications is becoming increasingly important. The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) online risk calculator is a popular tool to predict surgical risk. This paper aims to assess the applicability of the ACS NSQIP calculator to patients undergoing surgery for pancreatic neuroendocrine tumors (PNETs).

Methods

Using the US Neuroendocrine Tumor Study Group (USNET-SG), 890 patients who underwent pancreatic procedures between 1/1/2000–12/31/2016 were evaluated. Predicted and actual outcomes were compared using C-statistics and Brier scores.

Results

The most commonly performed procedure was distal pancreatectomy, followed by standard and pylorus-preserving pancreaticoduodenectomy. For the entire group of patients studied, C-statistics were highest for discharge destination (0.79) and cardiac complications (0.71), and less than 0.7 for all other complications. The Brier scores for surgical site infection (0.1441) and discharge to nursing/rehabilitation facility (0.0279) were below the Brier score cut-off, while the rest were equal to or above and therefore not useful for interpretation.

Conclusion

This work indicates that the ACS NSQIP risk calculator is a valuable tool that should be used with caution and in coordination with clinical assessment for PNET clinical decision-making.

Introduction

Pancreatic neuroendocrine tumors (PNETs) are rare, highly variable lesions with approximately 1000 new cases diagnosed per year in the USA.¹ Studies show the actual prevalence of these lesions may be higher, with up to 0.5–1.5% of individuals having PNETs on autopsy. PNETs are generally malignant with variable presentation—early due to secretory symptoms from functional tumors or late due to mass effect from nonfunctional tumors.² Five-year survival varies from 95% in well-differentiated pancreatic neuroendocrine tumors, to 44% in well-differentiated carcinomas, and 0% in poorly differentiated carcinomas.³

Surgical options for treatment of PNET include enucleation, distal pancreatectomy, pancreaticoduodenectomy (Whipple), and total pancreatectomy depending on PNET type and location within the pancreas. Morbidity and mortality rates vary significantly by procedure as well as by hospital volume; in one large-scale study, distal pancreatectomy, pancreaticoduodenectomy, and total pancreatectomy had in-hospital mortality rates of 3.5%, 6.6%, and 8.3% respectively.⁴ With the high mortality and variability in outcomes, perioperative surgical risk estimation is especially valuable for risk stratification, operative planning, and to inform discussions with patients and families.

Various published risk calculators have become available to address this growing need. The first risk model to evaluate cardiac risk in noncardiac surgery was published in 1977 and led to the revised cardiac risk index, published in 1999.^5,6 To address noncardiac morbidity and mortality, multiple disease-specific risk calculators were created. For pancreatic cancer patients, the Nationwide Inpatient Sample (NIS) was used to develop predictive models for pancreatic cancer.^7,8 The American College of Surgeons National Surgical Quality Improvement Project (ACS NSQIP) database was used to develop a pancreatectomy risk calculator in 2010.⁹ Then, in 2013, the ACS NSQIP universal risk calculator was developed to provide risk of 30-day morbidity and mortality using 21 patient-specific risk factors for any surgical procedure; this was made publically available online and is now widely used in clinical practice.¹⁰

The applicability of the ACS NSQIP risk calculator has been examined in a variety of procedures and diseases from head and neck cancer to cystectomy.^11–15 Use of the calculator has also been studied in other hepato-pancreato-biliary procedures and diseases including extrahepatic biliary malignancies and liver resection.^16,17 However, the calculator’s ability to predict complications after procedures for PNETs has not been studied. The objective of this study was to examine the ability of the ACS NSQIP calculator to estimate perioperative morbidity and mortality in patients undergoing surgery for PNETs.

Methods

ACS NSQIP Calculator

The ACS NSQIP risk calculator is a widely used tool originally developed in 2013 using data collected from 393 hospitals between 2009 and 2012. It allows surgeons and patients to estimate risk of 8 postoperative complications using 21 preoperative factors (spanning procedure, demographics, and comorbidities).¹⁰ The risk calculator has now evolved to include data from 740 hospitals and provides estimated risk of 12 postoperative complications along with a predicted length of stay. Open, online access to the risk calculator is available at the following web address: https://riskcalculator.facs.org/RiskCalculator/.

Patient Selection

The US Neuroendocrine Tumor Study Group (USNET-SG) is a multi-institutional collaborative including data on neuroendocrine tumors from the Ohio State University (Columbus, OH), Emory University (Atlanta, GA), Stanford University (Palo Alto, CA), Virginia Mason Medical Center (Seattle, WA), University of Wisconsin (Madison, WI), Washington University (St. Louis, Mo), Vanderbilt University (Nashville, TN), and the University of Michigan (Ann Arbor, MI). The institutional review board of each institution approved the study. The USNET-SG was used to identify patients who had undergone resection for PNET between January 1, 2000 and December 31, 2016. This study only included adult patients undergoing first-time resection for pancreatic neuroendocrine tumor. Each institution independently performed chart review to obtain patient information, which was entered into a standardized data collection form.

Patient data was entered in to the ACS NSQIP risk calculator to perform risk assessment on August 25th, 2017.¹⁰ At the time that the analysis was performed, the calculator included entry of the following factors: procedure type (CPT code), age, sex, functional status, emergency case, ASA classification, steroid use, ascites, sepsis, ventilator status, disseminated cancer, insulin and non-insulin dependent diabetes mellitus, hypertension, congestive heart failure, dyspnea, current smoker, severe COPD, dialysis dependence, acute renal failure, and BMI.

The following outcomes were included in the study: any complication; serious complication (cardiac arrest, myocardial infarction, pneumonia, progressive renal insufficiency, acute renal failure, PE, DVT, return to the operating room, deep incisional SSI, organ space SSI, systemic sepsis, unplanned intubation, UTI, wound disruption); pneumonia; cardiac complication; surgical site infection; urinary tract infection; venous thromboembolism; renal failure; return to the operating room; readmission; discharge to nursing/rehab; length of stay; and death.

Statistical Analysis

Patient demographics and clinical characteristics were summarized using descriptive statistics including median and interquartile range for continuous variables and frequency and percentage for categorical variables. Logistic regression models were used to determine the association between predicted and actual risk. The predicted rates of complications in these patients were then compared with actual patient outcome.

The performance of the ACS NSQIP calculator in predicting risk was evaluated using two metrics: C-statistic and Brier score. The C-statistic is the area under the curve (AUC) of a receiver operating characteristic (ROC) curve. It is a measure of discrimination and graphs the sensitivity (true positive rate) versus 1-specificity (false positive rate). If the variable under study, in this case, the ACS NSQIP calculated risk, perfectly predicts patients who will have complication versus those who will not, the AUC will be 1. If the variable, ACS NSQIP calculated risk, completely fails to distinguish between those who will have a complication and those who will not, the AUC will be 0.5. Generally, AUC > 0.7 is considered “fair” and AUC > 0.8 is considered “good”.¹⁸

The Brier score is a simultaneous measure of calibration and discrimination. The Brier score is reported as a score between 0 and 1—and is calculated as the mean squared difference between a patient’s predicted probability and observed outcome. A score of 0 indicates no difference between predicted and actual outcome, and thus indicates the best possible test. A score of 1 indicates that the test did not predict the outcome. The Brier score is compared to a Brier score cutoff, which is partially based on incidence in the sample and above which it is no longer useful.^10,19

All statistical analysis was completed using Stata MP 14.2 (StataCorp, College Station, TX).

Results

There were 890 patients who met inclusion criteria. Demographic and clinical data for these patients is shown in Table 1. The most commonly performed procedure was distal subtotal pancreatectomy (N = 559, 63%), followed by standard (N = 135; 16%) or pylorus-preserving pancreaticoduodenectomy (N = 110; 12%). The majority of included patients were under 65 years of age (N = 650; 73%), female (N = 458; 51%), were independent (N = 852, 96%), underwent elective surgery (N = 846; 99%), and were ASA class III with severe systemic disease (N = 507; 57%).

Table 1.

Demographic characteristics (n = 890)

Calculator input	Type	Frequency (%)
Procedure type	Distal subtotal pancreatectomy	559 (63)
	Pancreaticoduodenectomy	135 (16)
	Pylorus-sparing pancreaticoduodenectomy	110 (12)
	Other	83 (9)
Age	Under 65 years	650 (73)
	65–74 years	168 (19)
	75–84 years	64 (7)
	85 years or older	8 (1)
Body mass index, median (interquartile range)		28 (24–32)
Gender	Male	432 (49)
	Female	458 (51)
Functional status	Independent	852 (96)
	Partially dependent	37 (4)
	Dependent	1 (0.1)
Elective surgery	846 (99)
ASA class	Healthy patient	76 (8)
	Mild systemic disease	284 (32)
	Severe systemic disease	507 (57)
	Severe systemic disease/life threat	23 (3)
Steroids		21 (2)
Ascites		3 (0.3)
Systemic sepsis		1 (0.1)
Ventilator dependent		0 (0)
Disseminated cancer		70 (8)
Diabetes	Yes-no medication	19 (2)
	Oral medication	99 (11)
	Insulin	68 (8)
Hypertension		396 (45)
Congestive heart failure		10 (1)
Dyspnea	At rest	16 (2)
	With exertion	3 (0.3)
Smoking history		143 (16)
Severe COPD		17 (2)
Dialysis		2 (0.2)
Acute renal failure		1 (0.1)

Actual event rate among this patient cohort was then displayed alongside median predicted risk (Table 2). These figures are shown for demonstration purposes only and should not be directly compared using a statistical test as one (actual event rate) is a proportion and the other is a median (predicted risk). It is, however, noteworthy that the actual event rate was higher than the median predicted risk for any complication, serious complication, pneumonia, cardiac complication, surgical site infection, urinary tract infection, venous thromboembolism, return to the operating room, readmission, and death. The figures are comparable for discharge to nursing facility/rehabilitation (actual event rate N = 26; 3%; median predicted risk 2.9%; range, 0.5–52.4%). And, the actual event rate for renal failure (N = 9; 1%) is much lower than the median predicted risk of 7% (range, 1.0–50.0%).

Table 2.

Comparison of actual events versus predicted risk

Actual events		Predicted risk (%)
Outcome	N (%)	Median risk	Minimum	Maximum
Any complication	225 (25%)	22.0	10.2	51.9
Serious complication	215 (24%)	19.3	8.6	45.2
Pneumonia	25 (3%)	2.5	0.3	13.9
Cardiac complication	9 (1%)	0.7	0.0	11.7
Surgical site infection	156 (18%)	13.9	5.5	35.3
Urinary tract infection	39 (5%)	2.5	0.7	7.1
Venous thromboembolism	28 (3%)	2.7	1.0	7.8
Renal failure	9 (1%)	7.0	1.0	50.0
Return to the OR	41 (5%)	3.3	1.5	10.9
Readmission	201 (23%)	14.7	7.8	29.0
Discharge to nursing/rehab	26 (3%)	2.9	0.5	52.4
Length of stay (median days, range)	7 (5–10)	7.0	4.0	27.5
Death	12 (1%)	0.4	0.0	18.9

The median predicted risk for those who did and did not have an event were then compared (Table 3). For all outcomes, median predicted risk was higher among those who did have an event than among those who did not. ROC analysis was then performed (Fig. 1; Table 3). The C-statistics were highest and fair for discharge destination (0.79) and cardiac complications (0.71). The C-statistics were poor (< 0.7) for all other complications including any complication (0.54), serious complication (0.55), pneumonia (0.50), surgical site infection (0.57), urinary tract infection (0.55), renal failure (0.50), return to the OR (0.59), readmission (0.55), and death within 30 days of operation (0.63). Brier scores and Brier score cut-offs were then calculated and are listed in the final columns of Table 3. The Brier scores for surgical site infection (0.1441) and discharge to nursing/rehabilitation facility (0.0279) were below the Brier score cut-off, while the rest were equal to or above and therefore not useful for interpretation.

Table 3.

Risk calculator outcomes

Outcome	Did not have event		Had event		AUC	Brier score	Brier score cut-off
	N (%)	Median risk % (range)	N (%)	Median risk % (range)
Any complication	665 (75%)	21.7 (10.2–51.9)	225 (25%)	23.6 (10.2–46.7)	0.5424	0.1913	0.1889
Serious complication	675 (76%)	19.1(8.6–45.2)	215 (24%)	21.1 (8.7–43.2)	0.5518	0.1849	0.1832
Pneumonia	820 (97%)	2.6 (0.3–13.9)	25 (3%)	2.8 (0.4–6.1)	0.4969	0.0291	0.0287
Cardiac complication	836 (99%)	0.7 (0.0–11.7)	9 (1%)	1.7 (0.3–6.0)	0.7129	0.0105	0.0105
Surgical site infection	734 (82%)	13.7 (6.0–35.3)	156 (18%)	15.1 (5.5–32.7)	0.5707	0.1441	0.1446
Urinary tract infection	802 (95%)	2.7 (0.7–7.1)	39 (5%)	2.8 (0.7–4.9)	0.5491	0.0446	0.0442
Venous thromboembolism	862 (97%)	2.7 (1.0–7.8)	28 (3%)	3.2 (1.2–5.3)	0.5539	0.0305	0.0305
Renal failure	835 (99%)	0.6 (0.0–5.9)	9 (1%)	1.1 (0.1–3.7)	0.5000	0.0105	0.0105
Return to the OR	807 (95%)	3.3 (1.6–10.9)	41 (5%)	3.9 (1.9–10.1)	0.5889	0.0460	0.0460
Readmission	684 (77%)	14.6 (7.8–29)	201 (23%)	15.1 (7.8–23.3)	0.5459	0.1820	0.1818
Discharge to nursing/rehab	864 (97%)	2.8 (0.5–50.8)	26 (3%)	13.4 (0.5–52.4)	0.7912	0.0279	0.0284
Death	878 (99%)	0.4 (0.0–18.9)	12 (1%)	0.9 (0.1–9.7)	0.6328	0.0134	0.0133

Fig. 1.

Receiver operator characteristic curves comparing predicted to actual events for any complication, serious complication, pneumonia, cardiac complication, surgical site infection, urinary tract infection, venous thromboembolism, renal failure, return to the operating room, readmission, discharge destination, and death

Discussion

This study compares actual event rate and predicted risk for surgical complications within 30 days of operation for 890 patients with pancreatic neuroendocrine tumor from the USNET-SG. The ACS NSQIP calculator is a popular, convenient, easy-to-use tool to allow physicians to assess surgical risk. We evaluated the prediction capabilities of the ACS NSQIP risk calculator in patients undergoing pancreatic resection for PNET. In comparing actual event rate and median predicted risk, we found that actual event rate for all events except renal failure was higher than median predicted risk. We also showed that, based on ROC analysis with AUC > 0.7, the calculator was able to reasonably predict discharge destination and cardiac complication after resection for PNET, but that it failed to accurately predict other complications. Calculated Brier scores were also only useful for surgical site infection and discharge destination. These are not only quality benchmarks for hospitals and physicians, but also of significant importance to patients and families.

Previous work has demonstrated that in case mix restricted populations, the C-statistic may decrease with a decrease in model performance.²⁰ The Brier score, however, reflects both calibration and discrimination and may be a more useful statistical tool to evaluate model quality in homogenous populations.^19,21 In this study, we found that C-statistic was only considered “fair” for cardiac complication and discharge destination, while Brier score was only a useful measure for surgical site infection and discharge destination. Additionally, as our group has discussed previously, one important complication of pancreatic surgery not addressed by the ACS NSQIP risk calculator is postoperative pancreatic fistula.¹⁷ Others have developed models for predicting postoperative pancreatic fistula after pancreaticoduodenectomy. One such model includes body mass index and pancreatic duct width and has a C-statistic of 0.832, while another includes main pancreatic duct index < 0.25, away from portal vein on computed tomography, disease other pancreatic cancer, male gender and intra-abdominal thickness and reported a C-statistic of 0.834.^22,23 The ACS NSQIP risk calculator, which is designed to be more universally applicable, does not include any specific risk factors for postoperative pancreatic fistula. This may somewhat limit its utility in predicting postoperative complications for patients undergoing pancreatic resection. We might therefore conclude that the ACS NSQIP risk calculator, while remaining a valuable tool, should be used with caution in patients undergoing pancreatic resection for PNET.

The benefit of the universal calculator is its ability to be used for any surgical procedure. However, our results indicate that the ACS NSQIP risk calculator should be used with caution in patients undergoing resection for PNET. This may be because the ACS NSQIP calculator is limited by a lack of diagnosis discrimination. While it has been demonstrated that the indication for pancreatectomy—malignant versus nonmalignant—does not appear to be significantly associated with the risk of postpancreatectomy complications including leak abscess and fistula,²⁴ studies examining the use of the calculator in patients undergoing pancreatic resection for pancreatic adenocarcinoma and studies comparing the risks of complications and mortality between patients undergoing pancreatectomy for pancreatic neuroendocrine tumor versus pancreatic adenocarcinoma are lacking. This is an important area for future work.

Previous studies in a variety of fields including surgical oncology, urology, and gynecology have reported similar results using the ASC-NSQIP for risk prediction.^{11,13–15,17} Thus, an area of future research would be to compare the predictive capability of the original procedure-specific risk calculators developed by ACS with the universal risk calculator; perhaps, in certain niche areas, the procedure-specific calculator would yield higher predictive ability. Another area of work would be to include complications that are more specific to the procedure being performed. For example, in the case of PNETS, the ability to predict likelihood of pancreatic leak would be of immense value in surgical decision-making.

This study has many strengths and some limitations. The USNET-SG is a multi-institutional collaborative including data on neuroendocrine tumors from eight large academic medical centers. This allowed us to amass a large cohort of patients on which to perform this analysis. However, the use of patients undergoing resection at academic medical centers only does somewhat limit the generalizability of these results. Additionally, although we were able to include 890 patients, the sample size for each specific procedure is still too small to perform procedure-specific subgroup analysis. Furthermore, we were not able to separate benign and malignant or functional and nonfunctional PNETs—and it has been reported previously that the morbidity and mortality of the NSQIP population among patients undergoing hepato-pancreato-biliary surgery does vary based on disease etiology.²⁵

In conclusion, we found that actual event rate was higher than predicted risk for most surgical complications for 890 patients with pancreatic neuroendocrine tumor from the USNET-SG. In comparing patients who had an event versus those who did not, median predicted risk was higher among patients who had an event for all complications. Based on ROC analysis, we showed that the calculator reasonably predicted risk of discharge destination and cardiac complication, but failed to accurately predict other complications. Additionally, Brier scores were only useful for surgical site infection and discharge destination. Although the ACS NSQIP risk calculator remains useful for predicting morbidity and mortality among the NSQIP population as a whole, it should be used with caution in patients undergoing pancreatic resection for PNET.