Abstract
Objective To develop a mathematical model that will predict the probability of death after surgery for colorectal cancer.
Design Descriptive study using routinely collected clinical data.
Data source The database of the Association of Coloproctology of Great Britain and Ireland (ACPGBI), encompassing 8077 patients with a new diagnosis of colorectal cancer in 73 hospitals during a 12 month period.
Statistical analysis A three level hierarchical logistic regression model was used to identify independent predictors of operative mortality. The model was developed on 60% of the patient population and its validity tested on the remaining 40%.
Results Overall postoperative mortality was 7.5% (95% confidence interval 6.9% to 8.1%). Independent predictors of death were age, American Society of Anesthesiology (ASA) grade, Dukes's stage, urgency of the operation, and cancer excision. When tested the predictive model showed good discrimination (area under the receiver operating characteristic curve = (0.775) and calibration (comparison of observed with expected mortality across different procedures; Hosmer-Lemeshow statistic = 6.34, 8 df, P = 0.610).
Conclusions Clinicians can predict postoperative death by using a simple numerical table derived from the statistical model of the ACPGBI. The model can be used in everyday practice for preoperative counselling of patients and their carers as a part of multidisciplinary care. It may also be used to compare the outcomes between multidisciplinary teams for colorectal cancer.
Introduction
Surgeons and the units in which they work are now clearly accountable for clinical outcomes.1 Consent to surgery cannot be truly informed unless operative risk is estimated by considering the patient's comorbid condition, extent of disease, and complexity of the proposed treatment. Patients and carers may then make decisions with greater awareness of the risks involved.
Currently no dedicated system exists in the United Kingdom that predicts operative mortality, adjusted for risk, in surgery for colorectal cancer. We describe the development of a dedicated model for colorectal cancer that estimates the operative risk for individual patients while providing an example of the general problem of quantifying surgical risk.
Data and methods
Data sources
The colorectal cancer study of the Association of Coloproctology of Great Britain and Ireland (ACPGBI) was conducted prospectively in 73 hospitals; participating surgeons joined the study on a voluntary basis. Data were collected locally, either by data managers dedicated to colorectal cancer or by participating surgeons, using the standardised ACPGBI dataset2 or equivalent on all patients presenting to the local hospitals with a new diagnosis of colorectal cancer. Data were submitted centrally by using electronic media and recorded on a Microsoft Access 2000 database (Microsoft Corporation, Seattle, USA). The data in the patients' records were edited extensively to check for missing values or values that were out of range and inconsistencies between data fields. The contributing hospitals reported a total of 8077 new cases of colorectal cancer over a 12 month period between 1 April 1999 and 31 March 2001.
Inclusion and exclusion criteria
We included data on patients undergoing emergency or elective surgery for colorectal cancer. We excluded patients with a diagnosis of colorectal cancer who did not undergo surgery and patients of whom only the name and demographic details were recorded on the database, without information on outcomes or risk factors.
Study end point and risk factors
The primary outcome was operative mortality, defined as death occurring within 30 days of an operative procedure, from whatever cause, occurring either in hospital or after discharge from hospital. The risk factors related to patients and procedures in the ACPGBI database included age; sex; American Society of Anesthesiology (ASA) grade3; cancer site in accordance with the World Health Organization's international classification of diseases and related health problems; surgical procedure categorised according to the UK classification of operative procedures (OPCS-4)4; operative urgency classified as emergency, urgent, scheduled, or elective according to the classification of the National Confidential Enquiry into Perioperative Deaths (NCEPOD)5; cancer staging according to the clinical findings and histological Dukes's classification as modified by Turnbull et al6; timing of surgery during a 24 hour period; cancer excision; and annual case volume.
Statistical analysis
We used unifactorial logistic regression to identify risk factors related to the operative mortality within 30 days. We grouped continuous variables into subcategories of increasing operative risk and used unifactorial logistic regression to compare these with a reference category (table 1). To maximise the information extracted from the predictor variables we used a median imputation technique to substitute for incomplete data.7 This method allows each case to be used by the multifactorial model, thereby reducing the 95% confidence intervals around the model estimates and odds ratios.
Table 1.
Patients' demographic characteristics, univariate and multivariate analysis for postoperative mortality for patients undergoing surgery for colorectal cancer
| Variable | Subgroup | No of patients (% of total) | No of deaths (% mortality) | Unadjusted odds ratio (95% CI)* | Adjusted odds ratio (95% CI)* |
|---|---|---|---|---|---|
| Age (years) | <65 | 2025 (27.5) | 67 (3.3) | 1 | 1 |
| 65-74 | 2337 (31.7) | 138 (5.9) | 1.81 (1.36 to 2.47) | 1.79 (1.30 to 2.47) | |
| 75-84 | 2248 (30.5) | 240 (10.7) | 3.49 (2.65 to 4.61) | 2.84 (2.09 to 3.86) | |
| 85-94 | 597 (8.1) | 83 (13.9) | 4.94 (3.64 to 7.04) | 3.67 (2.55 to 5.30) | |
| 95+ | 26 (0.4) | 9 (34.6) | 18.3 (7.70 to 43.4) | 13.3 (5.01 to 35.3) | |
| Missing | 141 (1.9) | 13 (9.2) | |||
| Sex | Male | 3843 (52.1) | 302 (7.9) | 1 | |
| Female | 3080 (41.8) | 207 (6.7) | 0.85 (0.70 to 1.02) | ||
| Missing | 451 (6.1) | 41 (9.1) | |||
| American Society of Anesthesiology (ASA) grade | I | 1055 (14.3) | 21 (2.0) | 1 | 1 |
| II | 2517 (34.1) | 116 (4.6) | 2.38 (1.49 to 3.80) | 2.18 (1.31 to 3.61) | |
| III | 1348 (18.3) | 173 (12.8) | 7.24 (4.57 to 11.47) | 5.05 (3.00 to 8.50) | |
| IV-V | 238 (3.2) | 74 (31.1) | 22.19 (13.30 to 37.01) | 13.31 (7.38 to 24.01) | |
| Missing | 2216 (30.1) | 161 (7.3) | |||
| Cancer staging | Dukes's A | 795 (10.8) | 36 (4.5) | 1 | 1 |
| Dukes's B | 2276 (30.9) | 140 (6.2) | 1.38 (0.95 to 2.01) | 0.98 (0.66 to 1.46) | |
| Dukes's C | 2189 (29.7) | 150 (6.9) | 1.55 (1.07 to 2.25) | 1.20 (0.82 to 1.76) | |
| Dukes's D | 987 (13.4) | 117 (11.9) | 2.84 (1.92 to 4.17) | ||
| Missing | 1127 (15.3) | 107 (9.5) | 1.87 (1.23 to 2.84) | ||
| Operative urgency | Elective | 5521 (74.9) | 307 (5.6) | 1 | 1 |
| Urgent | 1227 (16.6) | 172 (14.0) | 2.77 (2.27 to 3.38) | 2.25 (1.80 to 2.80) | |
| Emergency | 238 (3.2) | 46 (19.3) | 4.07 (2.90 to 5.73) | 2.81 (1.90 to 4.15) | |
| Missing | 388 (5.3) | 25 (6.4) | |||
| Time of day | 8 00-17 00 | 4138 (56.1) | 261 (6.3) | 1 | |
| 17 00-24 00 | 278 (3.8) | 35 (12.6) | 2.14 (1.47 to 3.11) | ||
| 24 00-8 00 | 65 (0.9) | 14 (21.5) | 4.07 (2.23 to 7.46) | ||
| Missing | 2893 (39.2) | 240 (8.3) | |||
| Cancer site | Rectum | 3255 (44.1) | 206 (6.3) | 1 | |
| Colon | 3929 (53.3) | 321 (8.2) | 1.32 (1.10 to 1.58) | ||
| Missing | 190 (2.6) | 23 (12.1) | |||
| Cancer excision | Resected | 6598 (89.5) | 443 (6.7) | 1 | 1 |
| Not resected | 433 (5.9) | 74 (17.1) | 2.86 (2.19 to 3.74) | 5.23 (1.41 to 19.41) | |
| Missing | 343 (4.7) | 33 (9.6) | |||
| Procedure | Right hemicolectomy | 2078 (28.2) | 153 (7.4) | 1 | |
| Transverse colectomy | 70 (0.9) | 9 (12.9) | 1.86 (0.91 to 3.81) | ||
| Left hemicolectomy | 481 (6.5) | 31 (6.4) | 0.87 (0.58 to 1.29) | ||
| Sigmoid colectomy | 685 (9.3) | 32 (4.7) | 0.62 (0.42 to 0.91) | ||
| Subtotal or total colectomy | 262 (3.6) | 28 (10.7) | 1.51 (0.98 to 2.30) | ||
| Anterior resection | 1780 (24.1) | 87 (4.9) | 0.65 (0.49 to 0.85) | ||
| Abdomino-perineal excision of rectum | 579 (7.9) | 31 (5.4) | 0.71 (0.48 to 1.06) | ||
| Hartmann's procedure | 420 (5.7) | 62 (14.8) | 2.18 (1.59 to 2.99) | ||
| Palliative stoma | 239 (3.2) | 46 (19.2) | 3.00 (2.09 to 4.30) | ||
| Enteroenteric bypass | 55 (0.7) | 11 (20.0) | 3.15 (1.59 to 6.21) | ||
| Examination under anaesthesia, laparotomy only | 115 (1.6) | 15 (13.0) | 1.89 (1.07 to 3.32) | ||
| Other | 267 (3.6) | 18 (6.7) | |||
| Missing | 343 (4.7) | 33 (9.6) | |||
| Volume | Per 10 unit increase | 1.00 (0.98 to 1.02) | 1.01 (0.99 to 1.03) | ||
| Interaction variables | ASA II×no resection | 0.54 (0.14 to 2.09) | |||
| ASA III×no resection | 0.30 (0.07 to 1.24) | ||||
| ASA IV×no resection | 0.16 (0.03 to 0.74)+ | ||||
| Total | 7374 (100) | 550 (7.5) |
Odds ratios and 95% confidence intervals (CI) calculated by using univariate and multivariate logistic regression analysis. Each subgroup is compared separately with the reference category (odds ratio of one).
ACPGBI colorectal cancer model—We used multifactorial logistic regression analysis to adjust for multiple risk factors, their interactions, and the clustering of adverse outcomes in hospitals (see appendix 1 on bmj.com). We subsequently used the coefficients derived from the multivariate analysis as weights in the derivation of the ACPGBI colorectal cancer score (table 2).
Table 2.
The colorectal cancer model of the Association of Coloproctology of Great Britain and Ireland (ACPGBI) and conversion chart of ACPGBI score to predicted 30 day operative mortality for patients undergoing surgery for colorectal cancer
|
ACPGBI colorectal cancer model
|
Conversion chart
|
|||
|---|---|---|---|---|
| Risk factor | Score | ACPGBI colorectal cancer score | Predicted mortality % | |
| Age group | <65 | 0 | 0 | 0.8 |
| 65-74 | 0.7 | 0.1-0.4 | 0.9-1.1 | |
| 75-84 | 1.1 | 0.5-0.8 | 1.3-1.7 | |
| 85-84 | 1.3 | 0.9-1.2 | 1.9-2.5 | |
| 95+ | 2.6 | 1.3-1.6 | 2.8-3.7 | |
| Cancer resected | ASA* I | 0 | 1.7-2.0 | 4.1-5.4 |
| ASA II | 0.8 | 2.1-2.4 | 6.0-7.9 | |
| ASA III | 1.6 | 2.5-2.8 | 8.6-11.3 | |
| ASA IV-V | 2.5 | 2.9-3.2 | 12.3-16.0 | |
| Cancer not resected | ASA I | 1.7 | 3.3-3.6 | 17.4-22.1 |
| ASA II | 1.8 | 3.7-4.0 | 23.9-29.8 | |
| ASA III | 2.1 | 4.1-4.4 | 31.9-38.7 | |
| ASA IV-V | 2.4 | 4.5-4.8 | 41.1-48.5 | |
| Cancer staging | Dukes's A | 0 | 4.9-5.2 | 51.0-58.4 |
| Dukes's B | 0 | 5.3-5.6 | 60.8-67.7 | |
| Dukes's C | 0.2 | 5.7-6.0 | 69.9-75.8 | |
| Dukes's D or any metastases | 0.6 | 6.1-6.4 | 77.6-82.4 | |
| Operative urgency | Elective | 0 | 6.5-6.8 | 83.8-87.4 |
| Urgent | 0.8 | |||
| Emergency | 1.1 | |||
The probability of death R can be calculated by using the ACPGBI equation as follows: In(R/1-R)=−4.859+(ACPGBI colorectal cancer score)
American Society of Anesthesiology grade.
Model validation—We validated the model internally by dividing the data into two distinct sets. We developed the model on 60% of the study population, selected randomly, and the remaining 40% of patients used for testing the model. We used the Hosmer-Lemeshow statistic8 to evaluate the performance of the ACPGBI colorectal cancer model to assess calibration or goodness of fit (the ability of the model to assign the correct probabilities of outcome to individual patients). To obtain this statistic we computed the estimated probability of death for each patient on the basis of the model, ranked them into 10 equal groups of ascending operative mortality, and then statistically evaluated the expected and observed number of outcomes in each tenth. Smaller values represent better calibration of the model. Model discrimination refers to the ability of the model to assign higher probabilities of death to patients who actually die than to patients who live: we measured this by the area under the receiver operator characteristic curve or c-index. Values ranging from 0.7 to 0.8 represent reasonable discrimination, and values exceeding 0.8 represent good discrimination.9 We analysed subgroups by comparing the observed and expected 30 day operative mortality for individual procedures on the validation set of 3000 patients.
Statistical software—We used the following software packages for our analyses: Intercooled STATA 6.0 for Windows (STATA Corporation, College Station, Texas, USA), Statistical Package for the Social Sciences, version 11 for Windows (SPSS, Chicago, Illinois, USA), and MLwiN Version 1.2 (University of London, United Kingdom).
Results
Altogether, 7374 (91.3%) of 8077 patients presenting with colorectal cancer satisfied the inclusion criteria. We excluded from the analysis 499 (6.2%) patients who did not have surgical treatment and 204 (2.5%) patients whose records were incomplete. Operated cases were submitted by 73 hospitals scattered across all 11 geographical regions in the United Kingdom. A total of 4491 patients (60.9%) had complete data for the risk factors included in the final ACPGBI model. We found no evidence of systematic under-reporting of risk factors, and missing data were distributed evenly among hospitals, with no significant differences in outcomes between missing and non-missing data. Unadjusted postoperative mortality was 7.5% (range 2-14.5%). Table 1 shows the patients' demographic characteristics, 30 day postoperative mortality, unifactorial and multifactorial analysis.
Figure 1 shows the relation between age, ASA grade, and 30 day operative mortality R (plotted on the log [R/1-R] scale). Hospitals are depicted by individual prediction lines, parallel to each other but with different intercepts on the vertical axis. Increasing age was found to be associated with a higher operative mortality among the four ASA grades; adjusted odds ratio of 1.53 per 10 years' increase in age (95% confidence interval 1.36 to 1.73). Operations performed on patients with ASA grade II, III, and IV were associated with a 2.0-fold (1.2 to 3.2), 5.3-fold (3.2 to 8.5), and 15.8-fold (9.2 to 27.0) increase in operative mortality compared with patients with ASA grade I, after adjusting for age and hospital effect.
Fig 1.
Variation of 30 day operative mortality with age across the strata of ASA grade. Individual prediction lines are provided for individual hospitals parallel to each other but with different intercepts on the vertical axis. The variability in the intercept represents the “hospital effect,” which is an indirect proxy of the structure and process of care in each hospital
Figure 2 shows the variation of postoperative mortality by age, ASA grade, and cancer excision. Increasing ASA grade was associated with a higher operative mortality, but this effect differed according to whether or not patients had their cancer resected (first order interaction). Cancer excision seemed to confer an advantage in terms of predicted outcome in three of the four ASA categories. This effect diminished with increasing ASA grade, and for patients with ASA grade IV-V the two lines overlapped, indicating a similar outcome between patients who underwent cancer resection and those who did not. On further testing we observed no other significant interactions between any other risk factors in the dataset.
Fig 2.
Variation of operative mortality by age, ASA grade, and cancer resection. The regression lines have been adjusted for age, ASA, and the interaction term ASA×cancer resection by using a three level logistic regression model
Development of the ACPGBI colorectal cancer model—Table 1 shows the risk factors of the ACPGBI model for colorectal cancer, their adjusted odds ratios, and 95% confidence intervals. On multivariate analysis, the patient's age, ASA grade, urgency of procedure, Dukes's stage, cancer excision, and the product of ASA grade and cancer excision (interaction term) were found to be independent predictors of outcome. Table 2 shows the ACPGBI colorectal cancer model, which is a simple additive score. We categorised each risk factor into relevant subgroups with weights derived directly from the β coefficients (log odds ratios) of the multifactorial analysis shown in table 1. The probability of 30 day operative mortality for each of the ACPGBI scores is also shown on a conversion chart in table 2.
Model performance—The model fitted the data well, as evidence from by the calibration Hosmer-Lemeshow statistic shows: development set = 5.98, 8 df, P = 0.649; validation set = 6.069, 8 df, P = 0.640. Figure 3 shows the calibration chart for the validation set. The area under the receiver operating characteristic curve was 0.775 (95% confidence interval 0.744 to 0.806). On subgroup analysis, the predicted mortality for various types of operations for the validation set (n = 3000), as calculated by the ACPGBI colorectal cancer model, was well within the confidence limits of the observed outcome as shown in figure 4 (Hosmer-Lemeshow statistic: 6.34, 8 df, p = 0.610). Observed and expected mortality did not show any significant differences among the strata of age, ASA grade, Dukes's stage, and urgency of the procedure.
Fig 3.
Calibration chart of the observed and model predicted operative mortality for the ACPGBI colorectal cancer model based on the validation set (n=3000)
Fig 4.
Comparison of observed and predicted 30 day operative mortality by type of procedure on the validation set of 3000 cases (Hosmer-Lemeshow statistic 6.34, 8 df, P=0.610). For each procedure the number of cases, the observed 30 day operative mortality (%), and its 95% confidence intervals are displayed
Discussion
Clinicians can predict postoperative death in patients with colorectal cancer by using a simple numerical table derived from the statistical model of the ACPGBI, which may provide patients and carers with an estimated numerical probability of survival from surgery as part of the decision making process in the perioperative setting. Our study represents a nationwide attempt to provide accurate risk adjusted outcomes in colorectal cancer care. Review of the observed performance statistics (validity and accuracy) of the model indicates that this was adequate for the prediction of risk to individual patients. A further use for the ACPGBI model might be to compare the results of treatment between units. However, before such comparative studies can be undertaken it will be essential to ensure inclusion of all patients and to have robust methods of data validation.
We used mortality in hospital as an indirect measure of the quality of care. In comparison with 30 day operative mortality, in-hospital mortality can be quantified more reliably. With the current tendency to discharge patients early, in the future the need might arise to include a combination of in-hospital and 30 day operative mortality. It is important to note that if surgeons are simply governed by achieving the lowest death rates they may fail to understand that well informed patients may trade off a short term risk in exchange for “cancer cure.”10 Patients with advanced cancers may not be suitable for cancer resection as the risks of resectional surgery outweigh the benefits of a simpler and possibly safer palliative operation. This is particularly true for obstructed, metastatic cancers of the large bowel, where palliation with colonic stenting offers an alternative definitive treatment to potentially high risk surgery.
Validity of the data
An essential prerequisite for the evaluation of healthcare outcomes is the provision of comprehensive, accurate, clinically valid, and reliable information. Although currently the quality of the data may be limited by the observational nature of this study, implementation of such a system should increase awareness of the minimum required data, with subsequent improvement in quality. The study was based on data that we had carefully and voluntarily collected from hospitals throughout the United Kingdom. Since these hospitals were self selected, inferences cannot be made about how representative the outcomes are for the overall population. Although the model was internally validated for the study population, it would require external validation by testing the model on further prospective series in different hospitals. As data on an increasing number of cases are collated centrally the colorectal cancer model is expected to evolve from year to year, with individual “hospital effects” becoming more important in modelling the variability of outcome between hospitals or regions. At present the sample size of the cases submitted by individual hospitals is not sufficient to draw any meaningful inferences about the quality of care between providers. In the future, model reliability (the ability to obtain similar values for the same variable in specific patients at different times or in different locations) can be formally assessed, both centrally and at the local level.
Quality of statistical analysis
The ACPGBI colorectal cancer model used a relatively new technique called hierarchical regression analysis. This is particularly useful in modelling observations that have a hierarchical or clustered structure, as they allow for the possibility that patients from the same hospital have more similar outcomes than patients chosen at random from different hospitals. This is often found in studies of patients within hospitals or pupils within schools.11 Multilevel models have been shown to provide more accurate predictions of outcome, and similar techniques have been adopted to display the comparative performance of organizations in cardiac surgery,12 and clinics for in vitro fertilisation.13
What is already known on this topic
Risk factors associated with postoperative mortality in colorectal cancer surgery are well established
Predictive models are available for surgical patients in general but are not applicable for predicting individual risk and analysis of subgroups in patients with colorectal cancer
What this study adds
A dedicated model has been developed to predict operative mortality for patients undergoing surgery for colorectal cancer
This modified model is presented in a format that is suitable for frontline clinicians
Usefulness of the ACPGBI colorectal cancer model and future developments
In the model adjustment was made for age, ASA grade, Dukes's stage, urgency of the procedure, and cancer excision, which are well established, independent, prognostic factors in colorectal cancer.14 In addition, we introduced an interaction term, which was clinically relevant and clearly improved performance of the model. Although ASA grade is simple to use and has been incorporated by other predictive indices,15 it is open to subjectivity and manipulation. In future years the ACPGBI dataset can be augmented with other measures of patients' comorbidity, such as the risk factors from the POSSUM physiological score16 or from the Veterans Affairs surgical risk model.17 The drawback of using specialised investigations such as albumin, prealbumin, or liver function tests is that, although they may be readily, prospectively, collected in elective situations, in the emergency setting data collection and score generation may be incomplete and lead to incorrect assessment of operative risk.18 Further work is therefore required to evaluate the accuracy, reliability and cost involved in collecting such data in addition to the ASA grade. Although POSSUM and p-POSSUM systems have been widely applied in general and colorectal surgery,19,20 disease specific models have been shown to outperform models designed to predict over a range of different conditions. The weights assigned to clinical variables depend on the specific disease context, such as colorectal cancer.21
We describe an accurate and validated model for providing risk adjusted outcomes in patients undergoing surgery for colorectal cancer. The study represents a valuable tool in the multidisciplinary management of cancer patients. The method used in the development of the ACPGBI colorectal cancer model can be applied to other surgical specialties for the objective quantification of operative risk in patients with and without cancer.
Supplementary Material
The statistical model used by the Association of Coloproctology of Great Britain and Ireland and a list of hospitals participating in data collection appear on bmj.com
The research on which this article is based was funded by the Hue Falwasser fellowship of the Royal College of Surgeons of England. The authors thank all the consultants who contributed patients to the study and the data collection officers, managers, and audit facilitators for their invaluable assistance. For a list of hospitals that took part in data collection for the ACPGBI study see bmj.com
Contributors: JDS and MRT initiated the study; data harvesting and data aggregation were undertaken by PPT and JDS; data analysis and risk modelling were performed by PPT and JDP; drafting of this report was undertaken by PPT and edited by JDP, MRT, and JDS. All investigators contributed comments and corrections on the final draft. JDS and PPT are guarantors of the study.
Funding: PPT is funded by the Hue Falwasser research fellowship of the Royal College of Surgeons of England.
Competing interests: None declared.
Ethical approval: The study was approved by the multicentre ethics research committee for Wales, 10 January 2001.
References
- 1.Spiegelhalter DJ, Aylin P, Best NG, Evans SJW, Murray GD. Commissioned analysis of surgical performance by using routine data: lessons from Bristol inquiry. J R Statist Soc A 2002;165: 1-31. [Google Scholar]
- 2.The Association of Coloproctology of Great Britain and Ireland Database and Dataset [computer program]. Version 2 for Access 2000. (UK):ClinIT 2000. www.cancernw.org.uk/clinit/products.htm?productindex = 1 (accessed 23 Mar 2003).
- 3.Dripps RD, Lamont A, Eckenhoff JE. The role of anesthesia in surgical mortality. JAMA 1963;178: 261-6. [DOI] [PubMed] [Google Scholar]
- 4.Department of Health. Hospital episode statistics, Main operations 2000/01. www.doh.gov.uk/hes/standard_data/available_tables/main_operations/index.html. (accessed 23 Mar 2003).
- 5.Callum KG, Gray AJG, Hoile RN, Ingram GS, Martin IC, Sherry KM, et al. Appendix A. In: Then and now: the 2000 report of the national confidential enquiry into perioperative deaths. London: NCEPOD, 2000: 125-6.
- 6.Turnbull RB Jr, Kyle K, Watson FR, Spratt J. Cancer of the colon: the influence of the no-touch isolation technique on survival rates. Ann Surg 1967;166: 420-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Harrell FE, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996;15: 361-87. [DOI] [PubMed] [Google Scholar]
- 8.Hosmer DW, Lemeshow S. Applied logistic regression. 2nd ed. New York: John Wiley, 2000.
- 9.Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982;143: 29-36. [DOI] [PubMed] [Google Scholar]
- 10.Treasure T. Whose lung is it anyway? Thorax 2002;57: 3-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Goldstein H, Thomas S. Using examination results as indicators of school and college performance. J R Statist Soc A 1996;159: 149-63. [Google Scholar]
- 12.Aylin P, Alves B, Best N, Cook A, Elliott P, Evans SJ, et al. Comparison of UK paediatric cardiac surgical performance by analysis of routinely collected data 1984-96: was Bristol an outlier? Lancet 2001;358: 181-7. [DOI] [PubMed] [Google Scholar]
- 13.Marshall EC, Spiegelhalter DJ. Reliability of league tables of in vitro fertilisation clinics: retrospective analysis of live birth rates. BMJ 1998;316: 1701-4 (discussion 5). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Holme T, Johansson H, Cedermark B, Ekelund G, Rutqvist LE. Influence of hospital and surgeon-related factors on outcome after treatment of rectal cancer with or without preoperative radiotherapy. Br J Surg 1997;84: 657-63. [PubMed] [Google Scholar]
- 15.Khuri SF, Daley J, Henderson W, Hur K, Demakis J, Aust JB, et al. The Department of Veterans Affairs' NSQIP: the first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care. National VA Surgical Quality Improvement Program. Ann Surg 1998;228: 491-507. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Copeland GP, Jones D, Walters M. POSSUM: a scoring system for surgical audit. Br J Surg 1991;78: 355-60. [DOI] [PubMed] [Google Scholar]
- 17.Khuri SF, Daley J, Henderson W, Hur K, Gibbs JO, Barbour G, et al. Risk adjustment of the postoperative mortality rate for the comparative assessment of the quality of surgical care: results of the National Veterans Affairs Surgical Risk Study. J Am Coll Surg 1997;185: 315-27. [PubMed] [Google Scholar]
- 18.Copeland GP. Surgical scoring, risk assessment and the surgeon. J R Coll Surg Edinb 1992;37: 145-8. [PubMed] [Google Scholar]
- 19.Sagar PM, Hartley MN, MacFie J, Taylor BA, Copeland GP. Comparison of individual surgeon's performance. Risk-adjusted analysis with POSSUM scoring system. Dis Col Rectum 1996;39: 654-8. [DOI] [PubMed] [Google Scholar]
- 20.Tekkis PP, Kocher HM, Bentley AJ, Cullen PT, South LM, Trotter GA, et al. Operative mortality rates among surgeons: comparison of POSSUM and p-POSSUM scoring systems in gastrointestinal surgery. Dis Col Rectum 2000;43: 1528-32. [DOI] [PubMed] [Google Scholar]
- 21.Daley J, Henderson WG, Khuri SF. Risk-adjusted surgical outcomes. Annu Rev Med 2001;52: 275-87. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.