Abstract
Background:
Head and neck cancer (HNC) is the seventh most prevalent cancer worldwide, usually requiring a multidisciplinary approach. The National Surgical Quality Improvement Program (NSQIP) risk calculator is a tool for predicting postoperative complications that possibly can help in decision-making, support patient education, and guide the choice between surgical and nonsurgical treatment options. The aim of this study was to assess the reliability of the NSQIP risk calculator as a predictor of postoperative complications following head and neck surgery with microvascular reconstruction.
Methods:
This retrospective study included 99 patients diagnosed with HNC who underwent microvascular reconstruction from January 2016 to February 2021. The observed complications were compared with those predicted by the NSQIP calculator. To assess the discriminatory power of the NSQIP estimates, receiver operating characteristic statistics, logistic regression, and the overall Brier score were used.
Results:
Forty-four percent of the patients experienced at least 1 postoperative complication, and 27.2% developed serious complications. The receiver operating characteristic analysis for any complications revealed an area under the curve (AUC) of 0.62 (95% confidence interval 0.51–0.73, P = 0.046). For serious complications, the AUC was 0.65 (95% confidence interval, 0.52–0.79, P = 0.021). Both AUC values fell short of the threshold for “acceptable discrimination” (0.7–0.8). The overall Brier score was 0.32, with scores less than 0.09 considered to have good accuracy.
Conclusions:
The results suggest that the NSQIP risk calculator tends to underestimate the likelihood of postoperative complications in patients who are undergoing resection for HNC accompanied by microvascular reconstruction.
Takeaways
Question: Can the National Surgical Quality Improvement Program risk calculator serve as a dependable predictor of significant postoperative complications in a cohort of older patients undergoing head and neck cancer (HNC) surgery with microvascular reconstruction?
Findings: The risk estimates provided by the National Surgical Quality Improvement Program demonstrated limited predictive capability and seemed to underestimate the likelihood of postoperative complications following HNC resection accompanied by microvascular reconstruction.
Meaning: To enhance the accuracy of this tool, it is essential to conduct larger studies aimed at assessing the calculator’s precision and developing a more reliable instrument for forecasting complications and mortality in patients undergoing HNC resection with microvascular reconstruction.
INTRODUCTION
Head and neck cancer (HNC) is the seventh most prevalent cancer type worldwide, including malignancies of the lip, oral cavity, oropharynx, nasopharynx, hypopharynx, nasal cavity, sinuses, larynx, and major salivary glands.1,2 Over the past decade, the incidence of HNC has increased.3 Surgery and radiotherapy remain the primary treatment modalities for HNC, either alone or in combination. In cases of recurrent HNC following multimodal treatment, salvage surgery remains the sole curative option. Surgical resection for HNC can be both extensive and challenging, often requiring reconstruction with free tissue transfer. In our institution, an increased frequency of postoperative complications in patients older than 60 years has been observed, indicating a need for better preoperative prediction of complications to help inform the patients and choose the right treatment.
This need has been recognized in several surgical specialties, and in 2013, the American College of Surgeons introduced the National Surgical Quality Improvement Program (NSQIP) surgical risk calculator, which uses a database of North American patients.4 The calculator predicts the probability of 19 postoperative complications within the first 30 days after surgery, based on 20 patient predictors and 1 surgical procedure code. It has been evaluated for several procedures in many surgical specialties. Prior evaluations of the risk calculator in HNC patients undergoing surgical resection and reconstruction have reported that the predictive accuracy of postoperative complications was greater for surgical resection than for microvascular reconstruction.5–9 This study aimed to evaluate the NSQIP risk calculator, especially for serious complications, in a cohort of HNC patients older than 60 years undergoing surgery with microvascular reconstruction, motivated by the increased number of complications previously observed in this age group.
MATERIALS AND METHODS
In this retrospective study, all patients 60 years and older with primary HNC who underwent tumor resection with concurrent microvascular reconstruction at Örebro University Hospital between January 2016 and February 2021 were considered for inclusion. The patients were found in the Örebro Head and Neck Cancer Register, which has documented all HNC cases treated at our institution since 1988.10 Patients whose medical records lacked the necessary data for the NSQIP risk assessment11 and postoperative complications were excluded. The input predictor data for the calculator included surgical procedure code, age, sex, functional status, emergency status, American Society of Anesthesiologists classification, steroid usage, presence of ascites, systemic sepsis occurring within 48 hours postsurgery, ventilator dependency, disseminated cancer, diabetes, hypertension, congestive heart failure, dyspnea, smoking history, severe chronic obstructive pulmonary disease, dialysis, acute renal failure, and body mass index.
The predictor data and the postoperative complication data within 30 days following surgery were gathered from the medical records. Serious complications in this study were defined as cardiac arrest, myocardial infarction, acute renal failure, pneumonia, deep venous thrombosis, pulmonary embolism, reoperation, sepsis, readmission to the intensive care unit, and death. Additional complications estimated by the calculator were superficial incisional infection (SSI), deep incisional SSI, organ space SSI, wound disruption, ventilator dependence greater than 48 hours, unplanned intubation, stroke, progressive renal insufficiency, and urinary tract infection.
The surgical procedural codes along with the specific characteristics of each patient were inputted into the NSQIP calculator to estimate the probability of postoperative complications. The Current Procedural Terminology codes used were either free skin flap with microvascular anastomosis (15757) or bone grafting with microvascular anastomosis (20955). The study was approved by the Swedish Ethical Review Authority (DNR: 2022-03047-01).
Statistical Analysis
Statistical analyses were conducted using SPSS Statistics 27.0 (IBM Corp, Armonk, NY). The Student t test and the chi-square test were used to compare the baseline characteristics of patients who experienced serious complications with those who did not. A P value less than 0.05 was considered statistically significant.
The Brier score was used to evaluate the accuracy of the NSQIP complication predictions. The score is the sum of the mean squared differences between binary outcomes and predicted events. Patients who experienced a specific complication were assigned a value of 1, whereas those who did not experience the complication were assigned a value of 0. In light of similar research, we chose a threshold of 0.09 for good accuracy.
The NSQIP risk calculator estimates of postoperative complications were evaluated for both “any complication” and “serious complication” using receiver operating characteristics (ROC) analysis. To evaluate the discriminatory usefulness of the risk calculator, the area under the curve (AUC) was computed, with an AUC value greater than 0.70 deemed acceptable, aligning with common thresholds observed in similar studies.12
RESULTS
Patient Characteristics
A total of 99 patients were included (mean age 71 y, 57% men). The general medical characteristics of these patients are detailed in Table 1. The majority were categorized as having mild systemic disease (class 2, 66.7%) according to the American Society of Anesthesiologists classification system, as presented in Table 1. Nearly half of the patients were receiving treatment for hypertension. The most common tumor locations were the tongue and gingiva.
Table 1.
Baseline Characteristics of the Study Population Revealed No Significant Differences Between Patients Who Experienced Serious Complications and Those Who Did Not (P > 0.05)
| Patients (n = 99) | Patients Without Serious Complications (n = 72) | Patients With Serious Complications (n = 27) | P | |
|---|---|---|---|---|
| Male, n (%) | 56 (56.6) | 42 (58.3) | 17 (62.9) | 0.389 |
| Mean age (SD), y | 71 (5.5) | 72 (5.8) | 70 (5.0) | 0.447 |
| BMI mean (SD), kg/m2 | 25.0 (3.5) | 24.8 (3.8) | 24.9 (3.6) | 0.255 |
| Age group, n (%) | 0.237 | |||
| 60–65 | 15 (15.2) | 5 (6.9) | 2 (7.4) | |
| 65–74 | 54 (54.6) | 39 (54.1) | 17 (52.9) | |
| 75–84 | 30 (30.3) | 28 (38.8) | 8 (29.6) | |
| Tumor localization, n (%) | 0.322 | |||
| Gingival cancer | 28 (28.2) | 19 (26.3) | 9 (33.3) | |
| Tongue cancer | 16 (16.1) | 11 (15.2) | 5 (18.5) | |
| Floor of the mouth | 7 (7) | 3 (4.1) | 4 (14.8) | |
| Buccal cancer | 3 (3) | 2 (2.7) | 1 (3.7) | |
| Other (oral cavity, throat, and neck) | 16 (16.1) | 8 (11.1) | 8 (29.6) | |
| Steroid treatment, n (%) | 6 (6.1) | 4 (5.5) | 2 (7.4) | 0.402 |
| Hypertension treatment, n (%) | 46 (46.5) | 32 (44.4) | 14 (51.8) | 0.558 |
| COPD, n (%) | 6 (6.1) | 6 (8.3) | 1 (3.7) | 1.0 |
| Patients who smoked within 1 y of treatment, n (%) | 17 (17.1) | 6 (8.3) | 7 (25.9) | 0.065 |
| ASA* class before operation, n (%) | 0.193 | |||
| 1 | 8 (8.1) | 9 (12.5) | 2 (7.4) | |
| 2 | 66 (66.7) | 52 (72.2) | 17 (62.9) | |
| 3 | 25 (25.3) | 11 (15.2) | 8 (29.6) | |
| Diabetes, n (%) | 0.236 | |||
| None | 87 (87.9) | 59 (81.9) | 24 (88.8) | |
| Treated with pills | 5 (5.1) | 7 (9.7) | 1 (3.7) | |
| Treated with insulin | 7 (7.1) | 6 (8.3) | 2 (7.4) | |
| Flap type, n (%) | 0.058 | |||
| Radial forearm | 41 (41.4) | 33 (45.8) | 13 (48.1) | |
| Fibula | 41 (41.4) | 32 (44.4) | 11 (40.7) | |
| Anterolateral thigh | 17 (17.2) | 7 (9.7) | 3 (11.1) | |
| Functional status, n (%) | 0.444 | |||
| Independent | 98 (98.9) | 72 (100) | 26 (96.2) | |
| Partially dependent | 1 (1.0) | 0 | 1 (3.7) | |
| Disseminated cancer, n (%) | 1 (1.0) | 0 | 1 (3.7) | 0.444 |
| Dyspnea before surgery, n (%) | 5 (5.1) | 1 (1.3) | 2 (7.4) | 0.168 |
| Congestive heart failure, n (%) | 2 (2.0) | 1 (1.3) | 1 (3.7) | 0.873 |
| Acute renal failure, n (%) | 2 (2.0) | 1 (1.3) | 1 (3.7) | 0.195 |
American Society of Anesthesiologists (ASA) physical status classification system.
BMI, body mass index; COPD, chronic obstructive pulmonary disease.
Observed complications
Postoperative complications were observed in 44 (44.4%) patients, and 27 (27.2%) patients experienced severe complications (Fig. 1). Several patients developed more than 1 complication. The predominant observed complication was surgical site infection (56.8%). Among serious complications, reoperation was the most frequent (17.2%), followed by readmission to the intensive care unit (10.1%). Sepsis, pneumonia, and deep vein thrombosis were observed in 2 patients each. Additionally, 1 patient experienced a cardiac complication, and another had a pulmonary embolism. One patient died within 30 days after surgery. There were no statistically significant differences in the distribution of patient characteristics and risk factors between patients who had serious complications (n = 27) and those who did not (n = 72).
Fig. 1.
Boxplot of NSQIP value in patients with or without serious complication (P < 0.05).
Flap-related complications included partial necrosis (n = 3), venous thrombosis (n = 1), venous stasis (n = 1), and postoperative bleeding next to the flap (n = 3). Reconstruction was performed using free fibula flaps (n = 41), radial free flaps (n = 41), and anterolateral thigh flaps (n = 17).
NSQIP Complication Risk Estimate
The NSQIP risk estimate for any complication was determined to be 28% (SD = 6.9), whereas the estimate for serious complications was 23.1% (SD = 5.5), and the risk of mortality was 0.4% (SD = 0.8). The comparison between the NSQIP risk estimate and the observed postoperative complications is shown in Figure 2. The patient who died had the highest risk estimate for any complication (48.5%) and death (6%).
Fig. 2.
NSQIP-predicted vs observed postoperative complications in the study group (n = 99).
The ROC analysis revealed an AUC of 0.62 (95% confidence interval 0.51–0.73, P = 0.046) for any complications, whereas the AUC for serious complications was 0.65 (95% confidence interval 0.52–0.79, P = 0.021). Both values fell short of the threshold for useful discrimination (Fig. 3).
Fig. 3.
ROC curves illustrating the predictive performance of the NSQIP risk calculator for serious complications (A) and any complications (B) in older patients undergoing HNC surgery with microvascular reconstruction. The line indicates the baseline performance level, represented by an AUC of 0.5. The AUC for any complication was 0.62 (95% CI 0.51–0.73, P = 0.046), whereas for serious complications it was 0.65 (95% CI 0.52–0.79, P = 0.021). CI, confidence interval.
The Brier score associated with each serious complication is presented in Table 2. The overall Brier score for serious complications was computed to assess the accuracy of the NSQIP. The overall score for serious complications was determined to be 0.32, indicating low accuracy.
Table 2.
Brier Scores for All Serious Complications
| Type of Complication | Brier Score |
|---|---|
| Reoperation | 0.311 |
| Readmission to intensive care unit | 0.285 |
| Pneumonia | 0.275 |
| Deep venous thrombosis | 0.537 |
| Lung emboli | 0.397 |
| Sepsis | 0.358 |
| Acute renal failure | 0.0 |
| Cardiac complication | 0.336 |
| Death | 0.365 |
The cumulative score for serious complications was determined to be 0.32. Scores less than 0.09 were considered accurate.
DISCUSSION
A significant proportion of HNC patients are older, typically aged 60 and older, and often present with comorbid conditions.3 Consequently, there is a clinically motivated demand for improved predictive tools, especially for predicting serious postoperative complications in these patients before performing major surgical procedures involving microvascular reconstruction. This is particularly important when nonsurgical treatment alternatives may be available. Nonetheless, the primary finding of this study indicates that the estimates provided by the NSQIP risk calculator were generally limited in their accuracy for predicting serious complications following resection and microsurgical reconstruction in our patient cohort.
The overall complication rate observed in patients with HNC undergoing microvascular reconstruction is approximately 44%, which aligns with findings from prior research.13,14 Although the overall prediction of serious complications was not far from the observed complication rate (23.1% versus 27.2%), the risk calculator’s individual predictions of serious complications fell short of being useful, underestimating the risk that was observed. The ROC analysis yielded an AUC of 0.65 for serious complications, indicating a level of discrimination that falls below acceptable standards. Our findings revealed a discrepancy between the NSQIP estimated risk and the overall accuracy of the calculator. The threshold value for the Brier score varies across studies, with some identifying 0.01 as predictive, whereas others do not incorporate the score in their analyses.15–17 Additional studies have reported threshold values of 0.14, 0.16, and 0.09.6,18,19 The overall Brier score for serious complications was determined to be 0.32, indicating the risk calculator’s lack of accuracy.
The NSQIP risk calculator has been previously evaluated in various contexts.6,20,21 Previous studies corroborate our findings, indicating that the aforementioned program tends to underestimate the risk of postoperative complications.22,23 A possible factor contributing to this discrepancy could be that our patient cohort has a higher complication frequency compared with the North American patients on which the NSQIP data are based. Additionally, another potential reason may be that the NSQIP risk calculator only uses 1 Current Procedural Terminology code and thereby does not account for the risks associated with multiple surgical procedures. However, this is frequently encountered in the surgical treatment of HNC patients, for instance, tracheotomy, neck dissection, primary tumor resection, and microvascular reconstruction. Ma et al5 have previously pointed out the limitation of the program in considering only a single procedure code. The calculator also does not include other possibly important predictor variables, including estimated operation time, nutritional status, tumor location, tumor stage, and the patient’s history of radiotherapy and chemotherapy. Previous studies have, for instance, indicated that prolonged operation time, along with an abnormal white blood cell count, is linked to an increased overall risk of complications.24 Perhaps the accuracy of predictions could improve if these and other variables were included and if more patients in different parts of the world were included in the database.
LIMITATIONS
There are multiple factors that constrain the findings of our study. This is a relatively small retrospective study conducted at a single institution. The patients included in this study are older than 60 years and thus not representative of all HNC patients. It is plausible that the true incidence of complications may be underestimated, as the data regarding these complications were obtained from medical records in a retrospective fashion. For instance, information on conditions such as pneumonia or urinary tract infections proved challenging to locate. A study that incorporates all national outcome data should be the next step for a more thorough and fair assessment of the NSQIP calculator.
CONCLUSIONS
The risk estimates provided by the NSQIP calculator demonstrated limited predictive capability and seemed to underestimate the likelihood of postoperative complications for older patients following HNC resection with microvascular reconstruction.
DISCLOSURE
The authors have no financial interest to declare in relation to the content of this article.
Footnotes
Published online 26 June 2025.
Disclosure statements are at the end of this article, following the correspondence information.
REFERENCES
- 1.Sung H, Ferlay J, Siegel RL, et al. Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. 2021;71:209–249. [DOI] [PubMed] [Google Scholar]
- 2.Gormley M, Creaney G, Schache A, et al. Reviewing the epidemiology of head and neck cancer: definitions, trends and risk factors. Br Dent J. 2022;233:780–786. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Confederation of Regional Cancer Centres. The Swedish Head and Neck Cancer Register (SweHNCR). 2020:11. Available at https://cancercentrum.se/diagnosbehandling/cancerdiagnoser/huvudochhals/kvalitetsregister.7158.html.
- 4.Bilimoria KY, Liu Y, Paruch JL, et al. Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg. 2013;217:833–842. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Ma Y, Laitman BM, Patel V, et al. Assessment of the NSQIP surgical risk calculator in predicting microvascular head and neck reconstruction outcomes. Otolaryngol Head Neck Surg. 2019;160:100–106. [DOI] [PubMed] [Google Scholar]
- 6.Vosler PS, Orsini M, Enepekides DJ, et al. Predicting complications of major head and neck oncological surgery: an evaluation of the ACS NSQIP surgical risk calculator. J Otolaryngol Head Neck Surg. 2018;47:21. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Prasad KG, Nelson BG, Deig CR, et al. ACS NSQIP risk calculator: an accurate predictor of complications in major head and neck surgery? Otolaryngol Head Neck Surg. 2016;155:740–742. [DOI] [PubMed] [Google Scholar]
- 8.Tierney W, Shah J, Clancy K, et al. Predictive value of the ACS NSQIP calculator for head and neck reconstruction free tissue transfer. Laryngoscope. 2020;130:679–684. [DOI] [PubMed] [Google Scholar]
- 9.Arce K, Moore EJ, Lohse CM, et al. The American College of Surgeons National Surgical Quality Improvement Program surgical risk calculator does not accurately predict risk of 30-day complications among patients undergoing microvascular head and neck reconstruction. J Oral Maxillofac Surg. 2016;74:1850–1858. [DOI] [PubMed] [Google Scholar]
- 10.Lundin E, Bergqvist M, Ahlgren J, et al. Validation of a clinical cancer register at the Head and Neck Oncology Center in Örebro. Anticancer Res. 2019;39:285–289. [DOI] [PubMed] [Google Scholar]
- 11.ACS NSQIP. ACS NSQIP Surgical Risk Calculator. Available at https://riskcalculator.facs.org/RiskCalculator/. Accessed February 21, 2024. [Google Scholar]
- 12.Wu YC, Lee WC. Alternative performance measures for prediction models. PLoS One. 2014;9:e91249. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Eskander A, Kang S, Tweel B, et al. Predictors of complications in patients receiving head and neck free flap reconstructive procedures. Otolaryngol Head Neck Surg. 2018;158:839–847. [DOI] [PubMed] [Google Scholar]
- 14.Vieira L, Isacson D, Dimovska EOF, et al. Four lessons learned from complications in head and neck microvascular reconstructions and prevention strategies. Plast Reconstr Surg Glob Open. 2021;9:e3329. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Samson P, Robinson CG, Bradley J, et al. The National Surgical Quality Improvement Program risk calculator does not adequately stratify risk for patients with clinical stage I non-small cell lung cancer. J Thorac Cardiovasc Surg. 2016;151:697–705.e1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Cohen ME, Liu Y, Ko CY, et al. An examination of American College of Surgeons NSQIP Surgical Risk Calculator accuracy. J Am Coll Surg. 2017;224:787–795e1. [DOI] [PubMed] [Google Scholar]
- 17.Cohen ME, Bilimoria KY, Ko CY, et al. Development of an American College of Surgeons National Surgery Quality Improvement Program: morbidity and mortality risk calculator for colorectal surgery. J Am Coll Surg. 2009;208:1009–1016. [DOI] [PubMed] [Google Scholar]
- 18.Hu WH, Chen HH, Lee KC, et al. Assessment of the addition of hypoalbuminemia to ACS-NSQIP surgical risk calculator in colorectal cancer. Medicine (Baltim). 2016;95:e2999. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Peterson B, Ghahramani M, Harris S, et al. Usefulness of the myocardial infarction and cardiac arrest calculator as a discriminator of adverse cardiac events after elective hip and knee surgery. Am J Cardiol. 2016;117:1992–1995. [DOI] [PubMed] [Google Scholar]
- 20.Larsen AMG, Cote DJ, Karhade AV, et al. Predictors of stroke and coma after neurosurgery: an ACS-NSQIP analysis. World Neurosurg. 2016;93:299–305. [DOI] [PubMed] [Google Scholar]
- 21.Zielsdorf SM, Kubasiak JC, Janssen I, et al. A NSQIP analysis of MELD and perioperative outcomes in general surgery. Am Surg. 2015;81:755–759. [PubMed] [Google Scholar]
- 22.Veeravagu A, Li A, Swinney C, et al. Predicting complication risk in spine surgery: a prospective analysis of a novel risk assessment tool. J Neurosurg Spine. 2017;27:81–91. [DOI] [PubMed] [Google Scholar]
- 23.Adegboyega TO, Borgert AJ, Lambert PJ, et al. Applying the National Surgical Quality Improvement Program risk calculator to patients undergoing colorectal surgery: theory vs reality. Am J Surg. 2017;213:30–35. [DOI] [PubMed] [Google Scholar]
- 24.Cannady SB, Hatten KM, Bur AM, et al. Use of free tissue transfer in head and neck cancer surgery and risk of overall and serious complication(s): an American College of Surgeons-National Surgical Quality Improvement Project analysis of free tissue transfer to the head and neck. Head Neck. 2017;39:702–707. [DOI] [PubMed] [Google Scholar]