Comparison of Observed to Predicted Outcomes Using the ACS NSQIP Risk Calculator in Patients Undergoing Pancreaticoduodenectomy

Abstract

Background

Postoperative outcomes predicted by the ACS NSQIP universal risk calculator have not been validated for specific procedures like pancreaticoduodenectomy (PD).

Methods

A random sample of 400 PD patients from the NSQIP database was analyzed. Patients were categorized into four groups of 100 each based on ICD-9 diagnosis (211.6, 157.0, 156.2, and 577.1). Estimated risks of postoperative outcomes recorded by the calculator were compared to observed outcomes using the Brier Score (BS). The calculated BS was compared to a null model BS. A BS of zero indicated perfect prediction, while a BS of one indicated the poorest prediction.

Results

BS for all groupings was generally low, reflecting good prediction. BS for any and major complications was higher (0.23 and 0.22, respectively). This was also seen within ICD-9 subgroups. For patients with ampullary cancer, BS for these outcomes was higher (0.27 and 0.26, respectively). Comparison to the null model BS (0.24 and 0.24, respectively) correlated lesser predictive accuracy of the calculator for this subgroup.

Conclusions

The ACS NSQIP risk calculator, although accurate in predicting outcomes in patients undergoing PD, shows variation when accounting for specific ICD-9 diagnoses. Incorporating the diagnosis may better guide surgeons and patients preoperatively in making informed decisions.

INTRODUCTION

The National Surgical Quality Improvement Program (NSQIP) was first developed in the 1990s at the Veterans Health Administration and has led to an improvement in surgical quality with a decline in morbidity and mortality outcomes [1–3]. Since opening to private institutions in 2004, the American College of Surgeons NSQIP has grown substantially with respect to institutional participation and now includes more than 600 hospitals [4]. High-quality clinical data collected by specially trained Surgical Clinical Reviewers (SCRs) are used to analyze and report 30-day risk-adjusted comparative outcomes to individual institutions as a means of benchmarking these measures and enhancing quality improvement efforts and care [5,6].

In response to a growing need for objective assessment of postoperative risks, procedure-specific risk calculators were developed using these data, including colectomy and pancreatectomy-specific calculators which incorporated standardized pre-operative clinical variables, to assist clinicians in counseling patients about procedure-specific post-operative complications [7,8]. In 2013, using data from more than 1.4 million cases in the NSQIP database, the universal risk calculator was developed [9]. This is an online user-friendly tool to estimate individualized, patient-specific risks for procedures in virtually every surgical subspecialty accounting for more than 2,500 Current Procedural Terminology (CPT) codes. During its development, the performance of the universal risk calculator was compared to that of other procedure-specific calculators developed by the NSQIP. Because the results between different operations were largely comparable, the universal calculator was validated against the colectomy risk calculator (which was the most utilized at the time) and shown to be accurate in predicting post-operative outcomes [9].

The NSQIP risk calculator has since been considered a risk assessment tool for clinicians to better understand risk of perioperative morbidity and mortality, and potentially help in pre-operative shared decision making. This is especially important when dealing with patients who are scheduled to undergo specific complex procedures like pancreaticoduodenectomy (PD), which carry a significant risk of postoperative complications [10–12]. However, the outcomes reported by the universal risk calculator have not been validated for these specific procedures. More so, the universal risk calculator does not take into account an indication variable or International classification of diseases, version 9 (ICD-9) diagnosis code in estimating risk of post-operative complications. Recent studies using the NSQIP database have shown that operative complications after PD differ based upon the indication for surgery [13,14]. Our aim was to validate the accuracy of the risk calculator in patients undergoing PD by comparing the predicted outcomes from the risk calculator to observed outcomes for patients in the NSQIP database. Additionally, we compared the outcomes estimated by the risk calculator for different indications, to specifically assess if inclusion of the ICD-9 diagnosis had any effect on the calculator’s predictive accuracy.

METHODS

This study was approved by the Wake Forest University Health Sciences Institutional Review Board. A simple random sample of 400 patients from the NSQIP Participant Use Data File (PUF) from 2005 to 2012 who underwent PD was analyzed. Patients were selected based upon the CPT codes 48150 (PD with distal gastrectomy) and 48153 (pylorus preserving PD) [15]. Patients undergoing PD were categorized into four groups of a 100 based on ICD-9 diagnosis (211.6—benign neoplasms, 157.0—pancreatic cancer, 156.2—ampullary carcinoma, and 577.1—chronic pancreatitis) [16]. These diagnoses were used, as they were broadly representative of the most common indications for performing PD. Twenty one pre-operative variables, as defined within the risk calculator, were entered for individual patients within the sample dataset (Fig. 1a). Estimated risks of postoperative outcomes were then calculated using the NSQIP universal risk calculator (Fig. 1b).

Fig. 1.

Preoperative variables (a) and predicted outcomes (b) in the National Surgical Quality Improvement Program (NSQIP) universal risk calculator.

Statistical Analysis

Since the calculator only provides an estimated percentage risk of developing an outcome and not whether or not the individual patient develops an outcome, comparing the performance of the calculator with actual outcomes poses a unique challenge. Therefore, the estimated outcomes were compared to the observed outcomes in the PUF using the Brier score (BS). The BS is calculated as the average squared difference between the predicted probabilities of events and the observed events (0 for non-events or 1 for events). A BS of zero indicated perfect prediction, while a BS of one indicated the poorest prediction. Brier scores were used for this analysis because of their ability to reflect both discrimination and calibration simultaneously, while c-statistics only reflect discrimination and Hosmer–Lemeshow statistics only reflect calibration [9]. The BS obtained was compared to a null model BS, which was calculated by assigning each patient the overall observed rate of each of the outcomes as the probability of experiencing an event. A BS less than the null model BS indicated greater accuracy of the calculator. All analyses were performed in SAS 9.4 (Cary, NC).

RESULTS

Comparison of Brier Scores for All Patients

Thirty-six percent (144/400) of patients developed a complication, while 31% (123/400) developed a major complication and 21% (84/400) developed a surgical site infection (SSI). Other complications were noted in less than 10% of patients. BS for patients in the entire cohort was generally low, reflecting good prediction of the risk calculator (Table I). For patients within the entire sample, the BS for any complications, major complications, and SSI was higher (0.23, 0.22, and 0.17, respectively) than that for other outcomes. Box plots comparing the predicted and observed outcomes for patients with or without an event occurrence showed less separation, demonstrating less predictive accuracy of the calculator for these outcomes (Fig. 2). Cardiac complications and renal complications had the lowest BS (0.01 and 0.003, respectively); however, the total number of events in these categories was very small (three and six, respectively). Urinary tract infection (UTI), venous thromboembolism (VTE), return to Operating Room, and death also had lower BS (0.04, 0.03, 0.06, and 0.02, respectively) and box plots showed good separation indicating better predictability of the calculator (Fig. 3).

TABLE I.

Brier Score for All Patients

Outcomes	n	%	Brier score	Brier score (null model)
Major complications	123	30.75	0.2175	0.2129
Any complications	144	36.00	0.2265	0.2304
Pneumonia	24	6.00	0.0552	0.0564
Cardiac complications	3	0.75	0.0079	0.0074
SSI complications	84	21.00	0.1658	0.1659
UTI	18	4.50	0.0427	0.0431
VTE	12	3.00	0.0292	0.0291
Renal complications	6	1.50	0.0029	0.0150
Return to OR	26	6.50	0.0603	0.0609
Death	8	2.00	0.0196	0.0196
Discharge to NH or rehab	21	12.80	0.1037	0.1117

SSI, Surgical site infection; UTI, Urinary tract infection; VTE, Venous thromboembolism; OR, Operating room; NH, Nursing home; Brier score lower than null model Brier score indicates better predictability for that outcome.

Fig. 2.

Comparison of observed to predicted outcomes using box plots of (a) Any complications, (b) Major complications, and (c) SSI for all patients. Boxes indicate 1^st to 3^rd Interquartile ranges. Horizontal bold lines indicate the median. Increased separation of plots for patients with and without an event occurrence suggests good prediction of the risk calculator for that outcome.

Fig. 3.

Comparison of observed to predicted outcomes using box plots of (a) UTI, (b) VTE, (c) Return to OR, and (d) Death for all patients. Boxes indicate 1^st to 3^rd Interquartile ranges. Horizontal bold lines indicate the median. Increased separation of plots for patients with and without an event occurrence suggests good prediction of the risk calculator for that outcome.

Comparison of Brier Scores by Specific ICD-9 Diagnosis

When stratified by ICD-9 diagnosis, the incidence of complications ranged from 32% to 47% and was highest in patients with ampullary carcinoma. This trend was also noted for major complications (38%), SSI (31%), pneumonia (8%), and UTIs (7%). The BS was generally low for patients within all specific ICD-9 subgroupings. The relatively higher BS observed for any and major complications for all patients were also observed when patients were classified based on their indication for surgery. The difference was most pronounced for patients in the ampullary carcinoma subgroup. BS for any complications, major complications, and SSI was 0.27, 0.26, and 0.24, respectively, which is higher than the BS for the same outcomes in the other three ICD-9 categories (Table II). Comparison of the calculated and null model BS corroborated this finding with BS being higher than null model BS, indicating less predictive accuracy of the calculator in this subgroup.

TABLE II.

Brier Score Comparisons by ICD-9 Diagnosis

	Benign neoplasms			Pancreatic adenocarcinoma			Ampullary carcinoma			Chronic pancreatitis
Outcomes	n	BS	BS–NM	n	BS	BS–NM	n	BS	BS–NM	n	BS	BS–NM
Major complication	33	0.2322	0.2211	27	0.1924	0.1971	38	0.2617	0.2456	25	0.1837	0.1875
Any complication	35	0.2176	0.2275	30	0.2040	0.2100	47	0.2730	0.2491	32	0.2114	0.2176
Pneumonia	6	0.0559	0.0564	6	0.0542	0.0564	8	0.0726	0.0736	4	0.0379	0.0384
Cardiac complication	0	0.0003	0.0000	2	0.0194	0.0520	0	0.0021	0.0000	1	0.0099	0.0099
SSI complication	19	0.1478	0.1539	19	0.1496	0.1539	31	0.2363	0.2139	15	0.1294	0.1275
UTI	5	0.0476	0.0475	4	0.0379	0.0384	7	0.0657	0.0651	2	0.0192	0.0196
VTE	2	0.0195	0.0196	1	0.0101	0.0099	3	0.0294	0.0291	6	0.0577	0.0564
Renal complication	2	0.0103	0.0198	2	0.0005	0.0198	1	0.0005	0.0100	1	0.0003	0.0100
Return to OR	6	0.0565	0.0001	5	0.0470	0.0002	7	0.0646	0.0001	8	0.0733	0.0001
Death	2	0.0198	0.0196	3	0.0286	0.0291	0	0.0006	0.0000	3	0.0292	0.0291
Discharge to nursing home or rehab	4	0.0916	0.0912	8	0.1420	0.1709	8	0.1503	0.1631	1	0.0252	0.0203

BS, Brier score; NM, Null model.

DISCUSSION

With advances in surgical technique, anesthesia, and perioperative care, the morbidity and mortality from PD reported from high volume centers has declined over the last few decades [10,13,17–20]. This has contributed to an increase in the frequency of this operation being performed for other benign and premalignant indications besides cancer [13]. With a growing need for objective models that predict postoperative risks, predictive nomograms that use computer-based algorithms to estimate the probability of an outcome have therefore been developed and used [21,22]. These nomograms are however limited in their applicability because they only estimate risk of in-hospital mortality and use administrative databases that can reliably capture only complications that result in reoperation or other procedural intervention, and fail to track events that occur after hospital discharge.

The NSQIP universal risk calculator was developed and validated to provide estimated risks of post-operative outcomes using a defined set of pre-operative variables. This tool can be used in counseling and making decisions prior to considering surgery, particularly in the elective setting. With the emphasis on multimodality therapy and the resultant complex decision-making involved in the care of cancer patients in particular, the risk calculator offers unique opportunities in these patients to improve overall care [23]. However, the predictive accuracy of the calculator has not been validated in specific patient cohorts.

This study was performed with the objective of validating the universal risk calculator in patients who undergo PD. Our data shows, that the universal risk calculator is generally accurate in predicting outcomes for patients undergoing PD. However, the calculator’s predictive accuracy appears somewhat diminished when estimating the risk of any and major complications and SSIs. Previous studies have used NSQIP data to determine outcomes of patients undergoing PD and compare it to large series of PD patients in the literature. However, none of these studies have used the NSQIP universal risk calculator specifically in PD patients. In a recent study, Greenblatt et al. [12] used the NSQIP database to determine the 30-day incidence of serious complications and mortality after PD. Multivariable models of 30-day morbidity and mortality outcomes were then used to create and validate a PD specific risk-prediction tool which is available and accessible online [12,24]. However, outcomes such as SSI and UTI were not included, as they were not considered serious complications. Additionally, since VTE, renal complications, cardiac complications, and pneumonia were all included together under serious complications, estimates for these individual adverse outcomes were not specifically obtained.

This is the first study that attempts to validate the predictive accuracy of the universal risk calculator for multiple outcomes after PD, using patients from the NSQIP database.

A difference in accuracy of the risk calculator was noted when patients were classified based on their ICD-9 diagnosis. Patients with ampullary cancer and benign neoplasms had higher rates of any (47% and 35%, respectively) and major complications (38% and 33%, respectively) compared to patients with pancreatic adenocarcinoma or chronic pancreatitis. This is not entirely unexpected given that the major cause of postoperative morbidity after PD relates to the incidence of postoperative pancreatic fistula, which develops less frequently when the pancreatic duct is enlarged and the pancreatic texture is firm as is noted in patients with pancreatic adenocarcinoma or chronic pancreatitis. Specifically, the predictive accuracy appeared to be somewhat diminished in ampullary carcinoma PD patients, likely secondary to the effect of diagnosis on the calculator’s predictability. In a recent study, Kimura et al. investigated 8,575 PD patients from the Japanese National Cancer Database and developed a risk prediction tool for complications after PD [11]. They noted that the 30-day and in-hospital mortality rates for pancreatic cancer were significantly lower than those for non-cancer PD patients. Similarly, using patients from the NSQIP database, Newhook et al. demonstrated a significant difference in the complication profile for patients undergoing PD based on whether their diagnosis was benign/pre-malignant or malignant [13]. The authors concluded that there was a need for collection of pancreatectomy-specific variables in addition to the standard pre-operative variables in the NSQIP database, and for documenting pancreatectomy-specific diagnoses in order to truly estimate outcomes of patients undergoing PD for different indications. Shubert et al. further demonstrated that patients in the NSQIP database undergoing PD can be categorized as high-and low-risk groups for major postoperative morbidity based solely upon their ICD-9 diagnosis, and concluded that the specific diagnosis should be included as a predictive parameter in measuring outcomes and counseling patients for informed consent [14]. The NSQIP Pancreatectomy Demonstration Project, a collaborative of multiple institutions that collect pancreatectomy specific parameters may mitigate some of these limitations by including these variables in estimating the risk of complications [25–28].

Estimating perioperative risk has several advantages. Firstly, it is a key component of safe, high-quality patient care as it not only facilitates informed consent but also helps surgeons set realistic expectations of outcomes for specific procedures. If fully informed of the potential risks, some patients may choose to avoid surgery and the risk of complications or prolonged recovery, and instead choose non-operative treatment or palliative care. Discussion of the risks derived from a validated prediction tool therefore makes this process a truly shared and informed process between the patient and the surgeon. Secondly, even in the pre-consultation phase, public reporting of outcomes through web-based platforms like Hospital compare, essentially enables patients to compare the quality of care delivered for similar services across multiple institutions [29]. However, given their relative complexity and the fact that adjustment of these outcomes based on individualized risk may not be immediately apparent to patients, their use at the patient-level has not found wide applicability. Hence, the availability of a user-friendly and accurate tool to predict their individualized risk from different operations, particularly major procedures, proves invaluable for patients. Thirdly, the National Quality Forum (NQF) incentivizes surgeons through the Physician Quality Reporting System to report high quality clinical performance data and encourages eligible professionals to provide consumers with quality of care information that will help them make informed decisions about their health [30]. Based upon recommendations by the NQF, the Centers for Medicare & Medicaid Services may soon financially incentivize surgeons for discussing patient-specific risks prior to elective procedures [31]. This underscores the need for use of objective models that accurately predict individualized perioperative risks for patients undergoing major procedures like PD, rather than the traditionally used empiric physician-specific data or population averages. Fourthly, identifying and estimating the probability of developing certain postoperative complications may provide an opportunity for care providers to implement risk-specific measures or clinical care pathways to improve postoperative outcomes and decrease costs.

The major limitation of our study was its small sample size. This could contribute to the low event rates that were noted with specific outcome measures like cardiac and renal complications and preclude drawing specific conclusions about the predictive accuracy of the calculator for these outcomes. However, within the limitations of this confined dataset, the outcomes measured in our study with respect to major morbidity and mortality are comparable to those reported in most recent studies, including those that use data comprising all the PD patients within the NSQIP PUF [11–14]. Hence, this randomly selected dataset could be considered largely representative of the entire NSQIP PD database. In addition, even though analysis may have been impacted by the smaller overall sample size, the comparison of the calculator’s predictability between subgroupings was done using similar sample sizes. This enables the study to objectively assess the validity of the calculator among the different ICD-9 diagnoses.

CONCLUSION

The NSQIP universal risk calculator although largely accurate in predicting postoperative outcomes in patients undergoing PD, shows some variation in predictability for certain outcomes and when accounting for specific ICD-9 diagnoses. Incorporating the ICD-9 code within the risk calculator alters its predictive accuracy, thus underscoring the need for the use of pancreatectomy-specific variables in estimating risk of complications. We recommend including preoperative diagnosis as a predictive variable within future iterations of the calculator to improve risk estimation and therefore better guide surgeons and patients in making shared, informed decisions prior to surgery.