Abstract
Background:
Several risk calculating tools have been introduced into clinical practice to provide patients and clinicians with objective, individualised estimates of procedure-related unfavourable outcomes. The currently available risk calculators (RCs) have been developed by well-endowed health systems in Europe and the USA. Applicability of these RCs in low-middle income country (LMIC) settings with wide disparities in patient population, surgical practice and healthcare infrastructure has not been adequately examined.
Patients and Methods:
Through this single tertiary care, LMIC-centre, retrospective cohort study, we investigated the accuracy of the two most widely validated RCs - American College of Surgeons-National Surgical Quality Improvement Program (ACS-NSQIP) RC and ColoRectal Physiological and Operative Severity Score for the Enumeration of Mortality and Morbidity (CR-POSSUM) - for the prediction of mortality in patients undergoing elective and emergency colorectal surgery (CRS) from March 2013 to March 2020. Online RCs were used to predict mortality and other outcomes. Accuracy was assessed by Brier score and C statistic.
Results:
Of 105 patients, 69 (65.71%) underwent elective and 36 (34.28%) underwent emergency CRS. The 30-day overall mortality was 12 - elective 1 (1.4%) and emergency 11 (30.5%). ACS-NSQIP RC performed better for the prediction of overall (C statistic 0.939, Brier score 0.065) and emergency (C statistic 0.840, Brier score 0.152) mortality. However, for elective CRS mortality, Brier scores were similar for both models (0.014), whereas C statistic (0.934 vs. 0.890) value was better for ACS-NSQIP.
Conclusions:
Both ACS-NSQIP and CR-POSSUM were accurate for the prediction of CRS mortality. However, compared to CR-POSSUM, ACS-NSQIP performed better. The overall performance of both models is indicative of their wider applicability in LMIC centres also.
Keywords: Benchmarking, colorectal surgery, complications, mortality, outcomes assessment, post-operative
INTRODUCTION
Over last two decades, several risk calculating tools have been introduced into clinical practice in the USA American College of Surgeons-National Surgical Quality Improvement Program Risk Calculator (ACS-NSQIP RC) and UK (POSSUM group of RC) with an aim to provide patients and clinicians with objective, individualised estimates of procedure-related unfavourable outcomes for shared decision-making and informed consent.[1] They also serve as valuable audit tools for comparison of risk-adjusted outcomes between different patient populations.
The risk scoring models can broadly be classified as procedure-specific and universal. Arguably, a procedure-specific model designed for specific surgery should be more accurate than universal calculator.[2] This has been true for the POSSUM group where the procedure-specific CR-POSSUM (6 pre-operative and 4 intra-operative parameters) is reported to be more accurate than POSSUM and P-POSSUM for colorectal surgery (CRS) mortality.[3,4,5] On the other hand, universal ACS-NSQIP RC requires 20 pre-operative variables and makes risk-adjusted predictions in 15 domains, including mortality being found to be similarly robust to procedure-specific RC, e.g., colorectal ACS-NSQIP, and being used to estimate risks of most operations including those on the digestive tract.[2,6] Therefore, while CR-POSSUM is more useful for surgical audits, ACS-NSQIP RC may be useful for both individual prognostication and surgical audits.
Low-middle income countries (LMICs) with wide disparities in patient population, surgical practice and healthcare infrastructure lack resources such as large data registries to generate similar RCs. The applicability of existing RCs - ACS-NSQIP or the CR-POSSUM - in LMIC patients remains underestimated. Through this retrospective, single-centre study, we evaluated the performance of the two most extensively validated risk scoring tools: ACS-NSQIP RC and CR-POSSUM in patients undergoing CRS, and also compared which of the two is more accurate for prediction of 30-day mortality.
PATIENTS AND METHODS
Study design and population
This was a single-centre, retrospective study performed at the department of surgical gastroenterology of a tertiary care private teaching hospital, New Delhi, India. The study was duly approved by the institutional scientific and ethics committees.
The study group comprised consecutive adult (≥18 years) patients who underwent elective or emergency, resective and/or reconstructive CRS from March 2013 to March 2020. Patients who underwent appendicectomy, creation of diverting colostomy alone and rectopexy were excluded from this analysis.
Management protocol
For elective procedures, mechanical bowel preparation was prescribed to all patients. Intravenous cefuroxime and metronidazole were the preferred antibiotics. All anastomoses were hand sewn (ileocolic: side to side, continuous two-layered; colocolic and colorectal: single-layered, interrupted). In the post-operative period, early enteral nutrition was initiated and parenteral nutrition was only sparingly used. The institution follows a policy of universal venous thromboembolism (VTE) prophylaxis as per risk assessment (after 48 h of admission, no new progress note can be entered without completing the VTE data). Our management protocol for the conduct of emergency abdominal surgery has already been reported earlier.[7]
Data collection and parameters analysed
Data were accessed from the institutional electronic health record system/departmental database/pre-anaesthetic check-up sheets and entered in a specifically created Microsoft Access database. The recorded parameters included demographic data, comorbidities, indications for surgery, urgency, procedure performed and access (minimally invasive or open). Outcome measures included length of stay (LOS), surgical site infection (SSI), anastomotic leak, re-exploration and 30-day mortality. All the other complications that are part of ACS-NSQIP system were also recorded. Complications were also recorded by the Clavien–Dindo System.
The online ACS-NSQIP RC was used to calculate scores for each patient as described by Lubitz et al.[6] Briefly, only the principle current procedural terminology code was used when multiple codes were present. The codes used included 44140, 44141, 44143, 44144, 44145, 44150, 44160, 44204, 44207, 44208, 44210, 44604, 44605, 44620, 44626, 45111 and 45113. When prompted by the RC whether any other potential appropriate treatment options were available, ‘none’ was chosen for all cases. For the surgeon adjustment of risks, ‘1 - no adjustment necessary’ was chosen for all cases.
For CR-POSSUM, as described by Tekkis et al., the procedures were classified as intermediate (reversal of stoma), major (all colectomies, Hartmann’s procedure and its reversal) and complex major (anterior resection, subtotal or total colectomy, ileal pouch anal anastomosis).[3] Similarly, CR-POSSUM mortality score for each patient was calculated using the online RC.
Statistical analysis
For the purpose of statistical analysis, the data were converted to Microsoft EXCEL spreadsheet and analysis was done using Statistical Package for the Social Sciences (SPSS) IBM SPSS Statistics for Windows, Version 21.0. (Armonk, NY, US: IBM Corp.). Categorical variables were presented in number and percentage (%) and continuous variables were presented as mean ± standard deviation and median. Univariate logistic regression was used to assess the relationship between observed and predicted value, and the predictive accuracy was determined by Brier score (for both discrimination and calibration). Univariate linear regression was used to assess the relationship between actual and predicted LOS. Receiver operating characteristic (ROC) curve was used to find out cut-off point of parameters for predicting mortality, and DeLong and Parker’s test was used for the comparison of area under the curve (AUC).[8] P < 0.05 was considered statistically significant. C statistic values were used for quantifying the discriminatory capacity and C statistic values were considered excellent (0.9–1), good (0.8–0.89), fair (0.7–0.79), poor (0.6–0.69) and failed or no discriminatory ability (0.50–0.59).
RESULTS
Demographics and brief overview
Over a period of 7 years, 105 patients underwent elective 69 (65.71%) and emergency 36 (34.28%) CRS. Laparoscopy-assisted CRS was exclusively performed in elective scenario only 28/69 (40.5%). Complications of Clavien–Dindo grade 3 and 4 occurred in 7 (6.6%) and 6 (5.7%) patients, respectively. The 30-day mortality following elective CRS was 1 (1.4%) and after emergency CRS was 11 (30.5%). The median LOS was 8 (range 1–40) days. An overview of demographic data and outcomes is provided in Table 1.
Table 1.
Results at a glance (n=105)
| Parameter | n (%) |
|---|---|
| Age (years) (median) | 57 (18-93) |
| Sex (male:female) | 73:32 |
| Presentation | |
| Elective | 69 (65.71) |
| Emergency | 36 (34.28) |
| Procedure | |
| Open | 77 (73.33) |
| Minimally invasive (laparoscopic - 25, robotic - 3) | 28 (26.66) |
| LOS (days) (median) | 8 (1-40) |
| Complications (Clavien-Dindo system) | |
| None | 47 (44.76) |
| 1 | 18 (17.14) |
| 2 | 15 (14.28) |
| 3 | 7 (6.66) |
| 4 | 6 (5.70) |
| 5 | 12 (11.42) |
| Unplanned return to operating room | 6 (5.70) |
| SSI | 15 (14.28) |
| Readmission within 30 days of discharge | 8 (7.61) |
| Mortality | 12 (11.42) |
| Elective surgery | 1 (1.44) |
| Emergency surgery | 11 (30.55) |
SSI: Surgical site infection, LOS: Length of stay
Indications for surgery
The surgery was performed for entire spectrum of elective colorectal diseases including tumours, inflammatory bowel disease (IBD) and complicated diverticular disease. The apparent low numbers in diverticular disease, IBD and colorectal cancers reflect overall low incidence of these diseases in India. In emergency setting, the spectrum included right colon gangrene along with small bowel in superior mesentery artery thrombosis, fulminant amoebic colitis besides lower GI bleed, obstruction and perforation [Table 2].
Table 2.
Colorectal surgery by diagnosis (n=105)
| Diagnosis (procedure) | n (%) |
|---|---|
| Tumours | 41 (39) |
| Colorectal cancers | 32 |
| Appendiceal tumours | 5 |
| Mucocele | 3 |
| Cystadenocarcinoma | 2 |
| Benign tumours | 4 |
| Lipoma | 3 |
| Giant villous adenoma | 1 |
| Inflammatory bowel disease | 9 (8.57) |
| Ulcerative colitis | 7 |
| Crohn’s disease | 2 |
| Complicated diverticular disease | 5 (4.766) |
| Mesenteric ischaemia | 7 (6.6) |
| Colonic ischaemia | 4 |
| Superior mesenteric artery thrombosis | 3 |
| Non-malignant obstruction | 6 (5.71) |
| Strangulated hernia containing right colon | 2 |
| Ileocaecal tuberculosis | 2 |
| Stoma reversal | 11 (10.47) |
| Loop colostomy closure | 4 |
| Hartmann’s reversal | 4 |
| Ileocolic reconnection | 3 |
| Perforation | 7 (6.66) |
| Anastomotic leak (index surgery outside hospital) | 4 (3.80) |
| Others | 15 (14.28) |
| Fulminant amoebic colitis | 3 |
| Recurrent rectal prolapse (resection rectopexy) | 3 |
| Sigmoid volvulus (sigmoid colectomy) | 3 |
| Lower GI bleed | 5 |
| Post-gunshot injury rectal fistula (low anterior resection) | 1 |
GI: Gastrointestinal
Accuracy of mortality prediction by risk scoring systems
Both ACS-NSQIP (Brier score 0.065, AUC ROC 0.939) and CR-POSSUM (Brier score 0.089, AUC ROC 0.912) were accurate and excellent predictors for overall mortality. The statistical details are provided in Figure 1 and Table 3.
Figure 1.
Comparison of AUC ROC for overall colorectal surgery mortality by American College of Surgeons-National Surgical Quality Improvement Program risk calculator and Colorectal POSSUM score. AUC: Area under the curve, ROC: Receiver operating characteristic, ACS: American College of Surgeons, CRP: ColoRectal Physiological and Operative Severity Score for the Enumeration of Mortality and Morbidity
Table 3.
Colorectal surgery - Brier score and C statistic for mortality by ACS-NSQIP RC and by CR-POSSUM score
| Scores | ACS-NSQIP RC | CR-POSSUM RC | |
|---|---|---|---|
| Overall | Brier score | 0.065 | 0.089 |
| mortality | C statistic (95% CI) | 0.939 (0.874-0.976) | 0.912 (0.841-0.959) |
| Elective | Brier score | 0.014 | 0.014 |
| mortality | C statistic* | 0.934 | 0.890 |
| Emergency mortality | Brier score | 0.152 | 0.201 |
| mortality | C statistic (95% CI) | 0.840 (0.680-0.940) | 0.773 (0.603-0.895) |
| (95% CI) |
*95% CI: Not calculated because of only 1 mortality in elective CRS. CRS: Colorectal surgery, CI: Confidence interval, RC: Risk calculator, ACS-NSQIP: American College of Surgeons-National Surgical Quality Improvement Program, CR-POSSUM: ColoRectal Physiological and Operative Severity Score for the Enumeration of Mortality and Morbidity
For elective CRS mortality, the accuracy of ACS-NSQIP was excellent (Brier score 0.014, C statistic 0.934), while it can be termed as good for CR-POSSUM (Brier score 0.014, C statistic 0.890) [Table 3]. AUC ROC could not be drawn as there was a single mortality following elective CRS.
The difference in accuracy between the two RCs was more discernible for emergency CRS mortality; accuracy was good for ACS-NSQIP (Brier score 0.152, AUC ROC 0.840) and fair for CR-POSSUM (Brier score 0.201, AUC ROC 0.773). The statistical details are provided in Figure 2 and Table 3.
Figure 2.
Comparison of AUC ROC for emergency colorectal surgery mortality by American College of Surgeons-National Surgical Quality Improvement Program risk calculator and Colorectal POSSUM score. AUC: Area under the curve, ROC: Receiver operating characteristic, ACS: American College of Surgeons, CRP: ColoRectal Physiological and Operative Severity Score for the Enumeration of Mortality and Morbidity
Analysis of other predicted outcomes by American College of Surgeons-National Surgical Quality Improvement Program
The observed and predicted outcomes by ACS-NSQIP RC are provided in Figure 3. Of the 15 domains, besides mortality, ACS-NSQIP RC had excellent accuracy for pneumonia (Brier score 0.037, C statistic 0.94), while the RC had good accuracy (C statistic value 0.8–0.89) for any, serious and cardiac complications and fair accuracy (C statistic value 0.70–0.79) for anastomotic leak and renal failure. With C statistic values ranging 0.6–0.69, the ACS-NSQIP RC had poor accuracy for readmission, return to operation room and ileus (over-prediction) as well as urinary tract infection (under-prediction). For SSI, RC was no better than flip of a coin (Brier score 0.122, C statistic 0.513) in our study.
Figure 3.
Actual versus predicted outcomes by the ACS-NSQIP risk calculator, ACS-NSQIP: American College of Surgeons-National Surgical Quality Improvement Program, CRP: ColoRectal Physiological and Operative Severity Score for the Enumeration of Mortality and Morbidity, UTI: Urinary tract infection, SSI: Surgical site infection, LOS: Length of stay, OT: Operation theatre
The ACS-NSQIP RC significantly under-predicted the LOS, actual 8 days versus predicted 6.5 days (P = 0.004). As discharge to a nursing home facility is not available in India, all patients were discharged to home; hence, no calculations were made for this parameter. Similarly, no incidence of VTE was recorded in our patients; this may be due to a combination of modest number of patients, proactive policy for deep vein thrombosis prophylaxis and overall low incidence of VTE in Indian patients.
DISCUSSION
For the estimation of surgery-related mortality, a specialist CRS unit may be adequately served by a procedure-specific, easy-to-use risk scoring system such as CR-POSSUM. However, a comprehensive universal RC such as ACS-NSQIP that obviates the use of multiple procedure-specific risk scoring systems is preferable for umbrella digestive surgery unit such as ours that performs a wide range of surgical procedures.
A procedure-specific RC however would still find wider acceptance in comprehensive digestive surgery units if it can be demonstrated to be more accurate than universal RC. Both CR-POSSUM and ACS-NSQIP RC have not been adequately evaluated in LMIC settings. We therefore analysed the accuracy of both risk scoring tools for the prediction of mortality in our patients undergoing elective and emergency CRS and also examined which of the two models is more accurate.
Mortality
In our study, the elective and emergency CRS mortality of 1.4% and 30.5%, respectively, is comparable to that reported in other similar studies.[6,9] Our results suggest that the two models have excellent accuracy for overall CRS mortality. However, ACS-NSQIP outperformed CR-POSSUM for the prediction of overall and emergency CRS mortality based on both Brier score and C statistic. For elective mortality, by Brier score, the predictive accuracy was similar for both models; however, based on C statistic, ACS-NSQIP performed better. Single mortality in the elective CRS subgroup however remains a limitation for assessing the predictive accuracy of both models in this cohort. Further, both models have better accuracy for elective CRS as compared to emergency surgery. Other studies have also reported that ACS-NSQIP has better accuracy for mortality prediction for elective CRS when compared to emergency surgery.[6,10]
The accuracy for overall CRS mortality predicted by ACS-NSQIP in our study (Brier score 0.065, C statistic 0.939) is comparable to that reported by Bilimoria et al. (Brier score 0.011, C statistic 0.944).[2] A possible explanation for this excellent accuracy could be that we have included consecutive CRS patients; therefore, the study population is not homogenous and is closer to clinical practice. Other studies have also shown that ACS-NSQIP RC performs better when the case mix is heterogeneous.[11,12]
Similarly, for CR-POSSUM score, the case mix in our study (elective 68.2% vs. emergency 31%; cancer 35.9% vs. non-cancer 62.8%) and the accuracy of mortality prediction (Brier score 0.0898, C statistic 0.912) were comparable to the study by Tekkis et al. wherein CR-POSSUM RC was developed.[3] Our CR-POSSUM mortality for emergency CRS (AUC ROC 0.773) is also comparable to that reported by Kwan et al. (864 patients, 15 hospitals, in-hospital mortality 18.9%, AUC ROC 0.808).[13]
Evaluation of American College of Surgeons-National Surgical Quality Improvement Program risk calculator for parameters other than mortality
We analysed outcomes for ACS-NSQIP in other domains also. In addition to mortality, ACS-NSQIP RC had excellent accuracy for pneumonia, while for 5 other parameters including any and serious complications, cardiac complications, anastomotic leak and renal failure, the accuracy ranged from fair to good.
In our patients, ACS-NSQIP RC significantly underpredicted LOS (actual 8 vs. predicted 6.5 days, P = 0.004). This may well be because in India, integrated facilities for ‘discharge to nursing home’ are not available making it imperative for all patients to be ‘discharge to home’.
An important inference of our study is universal applicability of ACS-NSQIP RC outside the USA. This in our view is particularly relevant to LMICs like ours which lack resources such as data registries with a large number of patients to generate RCs similar to ACS-NSQIP or the CR-POSSUM. Centres in LMICs may also utilise ACS-NSQIP RC for clinical benchmarking.
Surgical site infections
For SSI (Brier score 0.122, C statistic 0.513), ACS-NSQIP RC had poor accuracy in our study. The overall SSI rate in our study was 14.2%, which is comparable to reported SSI rate of 15.1% after standard care as reported in a recent systematic review of 16 studies.[14] Another study that evaluated SSI following elective CRS reported that none of the prediction models including ACS-NSQIP model (C statistic 0.62) accurately predicted SSI after CRS.[15] There is an emerging evidence that the introduction of CRS SSI prevention bundles significantly reduces the risk of SSI. Since the completion of the study, we have introduced strategies including sterile closure trays and pre-closure glove changes for all patients undergoing elective as well as emergency CRS.[16]
Limitations
Surgical volumes have historically been used as one of the indicators of quality of surgery; the number of patients in our study may be considered relatively modest.[17] However, emerging data indicate that volume is not the only indicator of quality of surgery.[18] Further higher colectomy skills appear to be associated with lower complication rates not only for colectomy but also for all other operations performed by a surgeon.[19] Another possible limitation of the current study is that it was conducted at a tertiary care centre and as such applicability of our results to the general population and other less-endowed centres need to be further validated at more LMIC centres.
CONCLUSIONS
Both ACS-NSQIP and CR-POSSUM were accurate for the prediction of CRS mortality. However, compared to CR-POSSUM, ACS-NSQIP performed better. The overall performance of both models is indicative of their wider applicability in LMIC centres for patient counselling and clinical benchmarking.
Financial support and sponsorship
Nil.
Conflicts of interest
There are no conflicts of interest.
Acknowledgements
Statistical analysis was performed by Ms Bhawna Garg, M.Sc. Statistics, CFA (Member of Ethical Committee of Holy Family Hospital).
REFERENCES
- 1.Mansmann U, Rieger A, Strahwald B, Crispin A. Risk calculators-Methods, development, implementation, and validation. Int J Colorectal Dis. 2016;31:1111–6. doi: 10.1007/s00384-016-2589-3. [DOI] [PubMed] [Google Scholar]
- 2.Bilimoria KY, Liu Y, Paruch JL, Zhou L, Kmiecik TE, Ko CY, et al. Development and evaluation of the universal ACS NSQIP surgical risk calculator: A decision aid and informed consent tool for patients and surgeons. J Am Coll Surg. 2013;217:833–42.e1. doi: 10.1016/j.jamcollsurg.2013.07.385. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Tekkis PP, Prytherch DR, Kocher HM, Senapati A, Poloniecki JD, Stamatakis JD, et al. Development of a dedicated risk-adjustment scoring system for colorectal surgery (colorectal possum) Br J Surg. 2004;91:1174–82. doi: 10.1002/bjs.4430. [DOI] [PubMed] [Google Scholar]
- 4.Leung E, Ferjani AM, Stellard N, Wong LS. Predicting post-operative mortality in patients undergoing colorectal surgery using P-POSSUM and CR-POSSUM scores: A prospective study. Int J Colorectal Dis. 2009;24:1459–64. doi: 10.1007/s00384-009-0781-4. [DOI] [PubMed] [Google Scholar]
- 5.Bromage SJ, Cunliffe WJ. Validation of the CR-POSSUM risk-adjusted scoring system for major colorectal cancer surgery in a single center. Dis Colon Rectum. 2007;50:192–6. doi: 10.1007/s10350-006-797-6. [DOI] [PubMed] [Google Scholar]
- 6.Lubitz AL, Chan E, Zarif D, Ross H, Philp M, Goldberg AJ, et al. American College of Surgeons NSQIP risk calculator accuracy for emergent and elective colorectal operations. J Am Coll Surg. 2017;225:601–11. doi: 10.1016/j.jamcollsurg.2017.07.1069. [DOI] [PubMed] [Google Scholar]
- 7.Vashistha N, Singhal D, Budhiraja S, Aggarwal B, Tobin R, Fotedar K. Outcomes of emergency laparotomy (EL) care protocol at tertiary care center from low-middle-income country (LMIC) World J Surg. 2018;42:1278–84. doi: 10.1007/s00268-017-4333-8. [DOI] [PubMed] [Google Scholar]
- 8.Parker CB, DeLong ER. ROC methodology within a monitoring framework. Stat Med. 2003;22:3473–88. doi: 10.1002/sim.1580. [DOI] [PubMed] [Google Scholar]
- 9.Ingraham AM, Cohen ME, Bilimoria KY, Feinglass JM, Richards KE, Hall BL, et al. Comparison of hospital performance in nonemergency versus emergency colorectal operations at 142 hospitals. J Am Coll Surg. 2010;210:155–65. doi: 10.1016/j.jamcollsurg.2009.10.016. [DOI] [PubMed] [Google Scholar]
- 10.Hyder JA, Reznor G, Wakeam E, Nguyen LL, Lipsitz SR, Havens JM. Risk prediction accuracy differs for emergency versus elective cases in the ACS-NSQIP. Ann Surg. 2016;264:959–65. doi: 10.1097/SLA.0000000000001558. [DOI] [PubMed] [Google Scholar]
- 11.Hyde LZ, Valizadeh N, Al-Mazrou AM, Kiran RP. ACS-NSQIP risk calculator predicts cohort but not individual risk of complication following colorectal resection. Am J Surg. 2019;218:131–5. doi: 10.1016/j.amjsurg.2018.11.017. [DOI] [PubMed] [Google Scholar]
- 12.Cohen ME, Liu Y, Ko CY, Hall BL. An examination of American College of Surgeons NSQIP surgical risk calculator accuracy. J Am Coll Surg. 2017;224:787–95.e1. doi: 10.1016/j.jamcollsurg.2016.12.057. [DOI] [PubMed] [Google Scholar]
- 13.Kwan TL, Lai F, Lam CM, Yuen WC, Wai A, Siu YC, et al. Population-based information on emergency colorectal surgery and evaluation on effect of operative volume on mortality. World J Surg. 2008;32:2077–82. doi: 10.1007/s00268-008-9632-7. [DOI] [PubMed] [Google Scholar]
- 14.Tanner J, Padley W, Assadian O, Leaper D, Kiernan M, Edmiston C. Do surgical care bundles reduce the risk of surgical site infections in patients undergoing colorectal surgery? A systematic review and cohort meta-analysis of 8,515 patients. Surgery. 2015;158:66–77. doi: 10.1016/j.surg.2015.03.009. [DOI] [PubMed] [Google Scholar]
- 15.Bergquist JR, Thiels CA, Etzioni DA, Habermann EB, Cima RR. Failure of colorectal surgical site infection predictive models applied to an independent dataset: Do they add value or just confusion? J Am Coll Surg. 2016;222:431–8. doi: 10.1016/j.jamcollsurg.2015.12.034. [DOI] [PubMed] [Google Scholar]
- 16.Zywot A, Lau CS, Stephen Fletcher H, Paul S. Bundles prevent surgical site infections after colorectal surgery: Meta-analysis and systematic review. J Gastrointest Surg. 2017;21:1915–30. doi: 10.1007/s11605-017-3465-3. [DOI] [PubMed] [Google Scholar]
- 17.Birkmeyer JD, Siewers AE, Finlayson EV, Stukel TA, Lucas FL, Batista I, et al. Hospital volume and surgical mortality in the United States. N Engl J Med. 2002;346:1128–37. doi: 10.1056/NEJMsa012337. [DOI] [PubMed] [Google Scholar]
- 18.Chiu AS, Arnold BN, Hoag JR, Herrin J, Kim CH, Salazar MC, et al. Quality versus quantity: The potential impact of public reporting of hospital safety for complex cancer surgery. Ann Surg. 2019;270:281–7. doi: 10.1097/SLA.0000000000002762. [DOI] [PubMed] [Google Scholar]
- 19.Stulberg JJ, Huang R, Kreutzer L, Ban K, Champagne BJ, Steele SR, et al. Association between surgeon technical skills and patient outcomes. JAMA Surg. 2020;155:960–8. doi: 10.1001/jamasurg.2020.3007. [DOI] [PMC free article] [PubMed] [Google Scholar]