Comparison of Prospective Risk Estimates for Postoperative Complications: Human vs Computer Model

Abstract

Background

Surgical quality improvement tools such as NSQIP, are limited in their ability to prospectively impact individual patient care by the retrospective audit and feedback nature of their design. We hypothesized that statistical models using patient preoperative characteristics could prospectively provide risk estimates of postoperative adverse events comparable to risk estimates provided by experienced surgeons, and could be useful for stratifying preoperative assessment of patient risk.

Study Design

Prospective observational cohort. Using previously developed models for 30- day postoperative mortality, overall morbidity, cardiac, thromboembolic, pulmonary, renal, and SSI complications, model and surgeon estimates of risk were compared to each other and to actual 30-day outcomes.

Results

The study cohort included 1791 general surgery patients operated between June, 2010 and January, 2012. Observed outcomes were: mortality(0.2%), overall morbidity(8.2%) {pulmonary(1.3%), cardiac(0.3%), thromboembolism(0.2%), renal(0.4%), SSI(3.8%)}. Model and surgeon risk estimates showed significant correlation (p<0.0001) for each outcome category. When surgeons perceived patient risk for overall morbidity to be low, the model predicted risk and observed morbidity rates were 2.8% and 4.1%, respectively, compared to 10% and 18% in perceived high risk patients. Patients in the highest quartile of model predicted risk accounted for 75% of observed mortality and 52% of morbidity.

Conclusions

Across a broad range of general surgical operations, we confirmed that the model risk estimates are in fairly good agreement with risk estimates of experienced surgeons. Using these models prospectively can identify patients at high risk for morbidity and mortality who could then be targeted for intervention to reduce postoperative complications.

Introduction

Current quality assessment programs for surgery such as the voluntary American College of Surgeons National Surgical Quality Improvement Program (NSQIP) have led to improvement in surgical outcomes.[1–4] These programs are limited in their ability to impact individual patient care by the retrospective audit and feedback nature of their design. A more optimal strategy for patient perioperative risk mitigation might be to prospectively identify risk at the individual patient level preoperatively to allow enough time to engage in strategies to prevent specific surgical complications. While there is abundant literature on the risk factors for adverse perioperative events[5–8], few available decision aid tools assess the patient and procedure risk variables for a broad group of operative procedures and surgical outcomes. Furthermore, minimal knowledge is available on the accuracy or precision of surgeon risk assessment with or without decision aid tools.

The purpose of this study was to compare risk estimates from statistical models previously developed and evaluated (9) to risk estimates from the patient’s surgeon for 30-day postoperative mortality, overall morbidity, and cardiac, pulmonary, thromboembolic, renal, and SSI complications in a diverse group of elective general surgical patients. In so doing, we sought to evaluate the predictive validity of the DS3 model in predicting periorperative risk for specfic complications and the face validity of this model by correlating the model risk predictions to those of experienced surgeons. We hypothesized that the statistical models using patient preoperative characteristics could provide risk estimates of postoperative adverse events comparable to risk estimates provided by experienced surgeons, and that the models could be useful for the prospective, preoperative assessment of patient risk.

Methods

Approvals

The study was approved by the Institutional Review Boards at the University of Colorado Denver, the University of Utah, the University of Alabama at Birmingham and the New England IRB for QC Metrix, Inc.

Statistical Prediction Models

The development of the statistical prediction models is described in detail elsewhere,[9] and will only briefly be described here. We used National Surgical Quality Improvement Program (NSQIP) data on 60,411 patients undergoing elective general and vascular surgical operations from the Michigan Surgical Quality Collaborative[10] between 2003 and 2008 to develop prediction models for 30-day postoperative mortality, overall morbidity, cardiac, thromboembolic, pulmonary, renal, and surgical site infection (SSI) complications using logistic regression analysis. Only data that would routinely be available prior to the surgical procedure such as patient demographics, selected patient preoperative comorbidities, and operative variables for the planned procedure were considered in the model development. The models were developed using a random sample of 80% of the surgical cases and were tested on the remaining 20% of the sample. The c-indices for the models were generally good to excellent, ranging from 0.763 for SSI to 0.893 for mortality. There was very little change in the c-indices from the development to the test datasets, ranging from a decrease of 0.058 for thromboembolic events to an increase of 0.015 for renal events. The most important predictor variables across all of the models included some operative variables—work RVU of the operation, inpatient operation, CPT category of the operation, and some patient characteristics—age, ASA class, chronic steroid use, race, functional status, wound classification, on dialysis, history of congestive heart failure, BMI, and current smoker.

Study Variables for Model Prediction

As part of this grant, we developed a software system, called Decision Support for Safer Surgery (DS3), which involves entry of patient level data about demographics, general medical condition, comorbidities, and operative variables and which outputs risk calculations for individual patients regarding selected postoperative adverse events. Demographic variables included patient age, gender, ethnicity, and race. General medical condition variables included functional status, weight, height, BMI, and ASA class. Operative variables included whether or not the surgery was inpatient or outpatient, wound classification, CPT codes of the primary and secondary operations, and the work relative value unit of the primary operation. Patient preoperative comorbidities included on dialysis, disseminated cancer, peripheral vascular disease, hypertension, history of congestive heart failure, history of COPD, open wound, chronic steroid use, history of percutaneous coronary intervention, previous cardiac surgery, bleeding disorder, and current smoker. The DS3 data entry form is shown in Appendix 2.

Study Cohort

To compare risk estimates from the statistical models and surgeons, we prospectively collected model and surgeon predicted risk scores as well as actual 30 day morbidity and mortality outcomes on patients undergoing elective general surgical operations at the University of Utah and the University of Alabama Birmingham during the period June, 2010 to January, 2012. Only patients being seen in an outpatient clinic who were being scheduled for elective surgery were included; emergency, current inpatient and transfer patients were excluded. Participating experienced, attending surgeons at each institution were fellowship trained and/or had a narrow scope of practice in specific disciplines of general surgery including foregut and bariatric, hepatobiliary, pancreatic, colorectal, breast, hernia, and endocrine surgery.

Risk Prediction

Model risk prediction

A research assistant at each hospital entered the risk data into the web-based software developed for the project. For any missing data (e.g., height, weight, ASA class), chart review and electronic record review were conducted to complete the data collection.

Surgeon risk prediction

The attending surgeon estimated risk of post-operative morbidity and mortality for each patient after consultation and prior to the surgical procedure. The surgeons were blinded to the model prediction. Surgeons were instructed to give a probability assessment for each adverse outcome (e.g., 1%, 5%, 10%, etc.) and to rate their perception of patient’s risk for each adverse outcome as low (bottom 25%ile), average (25–75%ile), or high (top 25%ile). These percentiles were only considered to be guidelines for the surgeons in terms of their risk estimation of low, average, or high.

Surgical Outcomes

Thirty-day postoperative adverse outcomes included mortality, overall morbidity, cardiac event, venous thromboembolism, pulmonary event, renal event, and surgical site infection. For the surgical patients included in the American College of Surgeons’ National Surgical Quality Improvement Program (ACS-NSQIP), the postoperative adverse outcomes were obtained from the hospital’s ACS-NSQIP database. For the surgical patients not in the ACS-NSQIP, the adverse outcomes were collected by nurse review of the patient’s medical record using the standard ACS-NSQIP definitions of postoperative events. The nurse assessors did not have knowledge of the model or surgeon risk prediction.

Statistical Analysis

Descriptive statistics using means, standard deviations, and frequency distributions were computed for patient demographic characteristics, comorbidities, and characteristics of the operations that they underwent. Risk estimates determined by the statistical models and surgeons for each of the postoperative adverse events were compared using the signed rank test. Also, the statistical models and the surgeons’ risk estimates for patients with and without adverse events were compared using the Wilcoxon test. Spearman rank correlations were computed between the risk estimates of the statistical models and the surgeons as a measure of agreement, and the correlations were tested to determine if they were significantly different from zero. P-values of 0.05 and below were considered statistically significant. SAS version 9.3 was used to conduct all statistical analyses.

Results

The study sample consisted of 1791 patients who underwent elective general surgical operations at the University of Utah (n=987) and the University of Alabama, Birmingham (n=804) during the period between June, 2010 to January, 2012. Data collection was nearly complete for all variables, ranging from a low of 96% for ASA class to a high of 100% for patient comorbidities. Patient preoperative variables (demographics, comorbidity and planned procedure) are summarized in Table 1. The vast majority of the surgical procedures involved the alimentary tract (foregut, hepatopancreaticobiliary, gallbladder and colorectal surgery) followed by hernia, and integumentary cases. The case mix for the top 25 most common CPT codes for each site as well as the case mix for the MSQC data used for the statistical modeling with associated RVU are shown in Appendix 3 (online only).

Table 1.

Demographics, Comorbidities, and Surgical Characteristics of the Study Population with Median Model and Surgeon Predicted Risk of Morbidity and Observed Overall Morbidity Rates for each Variable Category (n=1,791)

			Model estimate		Surgeon estimate
	n	Mean (SD) or %	Median, % (mean)	IQR, %	Median, % (mean)	IQR, %	p Value*	Observed overall morbidity rate (%)
Overall	1,791		5.5 (9.0)	2.0 ′11.8	5.0 (7.7)	2.0–10.0	<.0001	8.2
Demographics
Age, y
≤40	390	21.8	2.5 (5.0)	1.4–6.9	2.0 (4.9)	1.0–5.0	0.0117	5.4
41–55	513	28.6	4.5 (7.5)	1.9–10.6	5.0 (6.9)	2.0–10.0	0.0013	8.8
56–65	436	24.3	7.1 (10.0)	2.6–13.0	5.0 (8.4)	2.0–10.0	<0.0001	9.4
65+	452	25.2	8.6 (13.2)	3.5–17.4	5.0 (10.6)	3.0–15.0	<0.0001	8.9
Race/ethnicity
Caucasian	1468	82.0	5.8 (9.4)	2.0–12.0	5.0 (7.8)	2.0–10.0	<0.0001	8.8
African-American	163	9.1	6.9 (8.9)	2.8–13.9	5.0 (8.1)	2.0–10.0	0.0408	5.5
Hispanic	81	4.5	2.5 (6.8)	1.6–7.0	2.0 (5.6)	1.0–5.0	0.0533	4.9
Other	79	4.4	2.8 (5.2)	1.7–7.5	4.0 (7.2)	2.0–10.0	0.3291	6.3
Sex (1 UTD)
Male	845	47.2	5.1 (9.0)	1.9–11.7	4.0 (7.7)	1.5–10.0	<0.0001	8.6
Female	945	52.8	5.8 (9.0)	2.1–12.1	5.0 (7.8)	2.0–10.0	<0.0001	7.8
BMI, kg/m2
Underweight (<18.5)	53	3.0	11.3 (14.3)	4.0–21.7	5.0 (8.5)	2.0–12.0	<.0001	11.3
Normal (18.5–25)	501	28.0	5.1 (9.1)	2.0–11.7	5.0 (8.5)	2.0–10.0	<.0001	8.2
Overweight (25– 29.9)	570	31.8	5.1 (8.7)	1.8–11.3	5.0 (7.2)	2.0–10.0	<.0001	7.5
Obese Class 1 (30– 34.9)	348	19.4	5.6 (8.2)	2.2–11.0	5.0 (7.0)	2.0–10.0	<.0001	7.2
Obese Class 2 (35– 39.9)	179	10.0	7.1 (10.5)	2.8–14.4	5.0 (8.1)	2.0–10.0	0.0011	10.1
Obese Class 3 (40+)	123	6.9	5.3 (7.4)	1.6–11.9	5.0 (7.8)	2.0–10.0	0.6973	9.8
Selected comorbidities
Hypertension	643	35.9	9.1 (12.3)	3.4–16.0	5.0 (9.6)	2.0–15.0	<0.0001	9.3
Current smoker	211	11.8	6.5 (11.0)	2.2–15.1	5.0 (9.5)	2.0–15.0	0.0563	12.8
Partially/totally dependent functional status	114	6.4	10.4 (15.7)	4.8–20.1	10.0 (12.9)	5.0–20.0	0.0091	8.8
COPD	109	6.1	13.6 (17.3)	6.3–24.2	10.0 (11.9)	5.0–15.0	<0.0001	14.7
History of PCI	84	4.7	8.1 (12.9)	3.2–18.3	5.0 (10.1)	2.0–15.0	0.0297	9.5
Chronic steroid use	76	4.2	12.3 (14.7)	6.3–20.2	5.0 (10.6)	3.0–13.5	<0.0001	13.2
Previous cardiac surgery	69	3.9	10.0 (15.7)	4.2–22.5	8.0 (11.5)	4.0–15.0	0.0121	11.6
Metastatic cancer	58	3.2	13.5 (17.6)	6.0–29.1	10.0 (11.6)	4.0–15.0	0.0020	15.5
Peripheral vascular disease	45	2.5	8.7 (15.5)	3.9–22.5	5.0 (10.6)	3.0–15.0	0.0045	17.8
Congestive heart failure	39	2.2	13.5 (21.0)	6.3–31.3	5.0 (11.1)	3.0–15.0	<0.0001	15.4
Bleeding disorder	32	1.8	9.3 (11.9)	3.6–17.7	5.0 (8.9)	2.5–12.5	0.0092	15.6
On dialysis	22	1.2	6.3 (10.8)	3.7–14.5	5.0 (10.3)	5.0–15.0	0.5184	9.1
ASA Class
1-Healthy	413	23.1	1.8 (3.9)	1.3–4.8	2.0 (3.8)	1.0–5.0	0.0014	4.6
2-Mild systemic disease	638	35.6	4.5 (7.2)	2.0–9.3	5.0 (6.7)	2.0–10.0	0.0006	6.1
3-Severe systemic disease	615	34.3	10.6 (14.0)	5.4–17.8	7.0 (11.2)	3.0–15.0	<0.0001	12.9
4-Constant threat to life	29	1.6	17.6 (24.2)	10.8–45.6	10.0 (17.6)	5.0–20.0	0.0053	20.7
5-Moribund	1	0.1	30.4 (30.4)	30.4–30.4	30.0 (30.0)	30.0–30.0	n/a	0.0
None assigned/missing	95	5.3	3.5 (6.5)	1.6–8.2	2.0 (6.3)	1.0–10.0	0.5212	4.2
Operative characteristics
Surgery type
Inpatient	1011	56.5	10.7 (14.0)	6.6–17.3	7.5 (11.1)	4.0–15.0	<0.0001	12.4
Outpatient	780	43.6	1.9 (2.5)	1.4–2.8	2.0 (3.4)	1.0–5.0	0.0711	2.8
Work RVU of operation
0–6.9	478	26.7	2.1 (3.1)	1.3–4.2	2.0 (4.4)	1.0–5.0	0.0011	4.8
7–11.9	427	23.8	2.4 (3.7)	1.7–4.8	2.0 (3.6)	1.0–5.0	<0.0001	4.0
12–21.9	462	25.8	8.0 (8.8)	4.9–11.5	5.0 (7.1)	2.0–10.0	<0.0001	8.4
22+	424	23.7	17.7 (21.3)	12.1–29.0	15.0 (16.4)	6.0–23.5	<0.0001	16.0
Wound class
Clean	965	53.9	3.4 (5.1)	1.6–6.8	3.0 (4.4)	1.0–5.0	<0.0001	4.5
Clean Contaminated	704	39.3	11.2 (14.2)	4.5–18.5	10.0 (11.9)	3.0–15.0	<0.0001	12.9
Contaminated	86	4.8	9.2 (11.9)	2.4–17.3	5.0 (10.1)	1.0–15.0	0.0280	14.0
Dirty/Infected	32	1.8	3.2 (7.2)	2.1–9.9	6.5 (11.1)	1.0–15.0	0.0751	3.1
CPT codes by category
Foregut	316	17.6	7.9 (9.3)	5.1–11.4	5.0 (6.1)	2.0–5.0	<0.0001	5.4
Hepatopancreatico biliary	191	10.7	25.6 (27.0)	15.3–36.8	20.0 (19.4)	10.0–30.0	<0.0001	16.2
Cholecystectomy	210	11.7	2.2 (3.0)	1.8–3.1	2.0 (2.6)	1.0–3.0	<0.0001	2.9
Colorectal	329	18.4	11.3 (12.5)	6.7–16.3	10.0 (12.0)	5.0–15.0	0.0046	16.1
Vascular	45	2.5	5.2 (9.2)	2.9–12.8	2.0 (6.6)	2.0–10.0	0.0017	4.4
Integumentary	71	4.0	1.4 (3.8)	0.9–3.0	2.0 (4.7)	1.0–10.0	0.0051	7.0
Hernia	441	24.6	2.5 (4.2)	1.5–6.0	2.0 (4.5)	1.0–5.0	0.0283	5.4
Location
UAB	804	44.9	8.4 (11.7)	3.9–15.2	5.0 (9.4)	3.0–15.0	<0.0001	10.2
U of U	987	55.1	3.1 (6.8)	1.6–8.8	2.0 (6.4)	1.0–8.0	<0.0001	6.6

^*Signed-rank test to test difference between surgeon and model prediction for the category.

The University of Utah sample consisted of more outpatient operations of lower workRVU and in patients with less comorbidity and who had lower adverse event rates. However, the main outcomes of the study (statistical model risk estimation vs. surgeon risk estimation) were not significantly different between the two institutions. Therefore, the results of the study are presented for the two institutions combined for ease of presentation and permitting analyses with a larger sample size.

The median model risk estimates, surgeon predicted risk estimates, and observed overall morbidity rates for each patient characteristic are shown in Table 1. Mortality results are not shown since there were only four (0.2%) deaths in the sample. Just over 8% of patients experienced morbidity. The median model prediction of overall morbidity was 5.5% versus 5.0% median surgeon prediction (p<0.0001). In comparing the predicted (model or surgeon) to observed overall morbidity for each variable category, both the model and surgeon consistently underestimated the overall morbidity with the exception of functional status where both model and surgeon predicted a higher associated morbidity than observed. For the majority of the patient characteristics, the model risk prediction was closer to the actual observed overall morbidity rate than the prediction of the surgeons. For some of the categories of operations (foregut and hepatopancreaticobiliary), the statistical model and surgeon overestimated risk of morbidity, while for the remaining categories model and surgeon underestimated risk. Table 2 presents median model and surgeon estimates of risk for patients with and without postoperative adverse events, including mortality, overall morbidity, and pulmonary, cardiac, thromboembolic, renal, and SSI complications. For all events except thromboembolic, both the model and surgeons predicted a significantly higher risk of the complication in those patients who had a complication than in those who did not.

Table 2.

Thirty-day Morbidity and Mortality Occurrences with Model and Surgeon Estimates of Risk (n=1,791 Patients)

			Model estimate					Surgeon estimate
			Event		No event		p Value *	Event		No event		p Value*
Outcomes	n	%	Median, %	IQR, %	Median, %	IQR, %		Median, %	IQR, %	Median, %	IQR, %
Mortality	4	0.2	1.3	0.4–2.3	0.1	0.03–0.5	0.0468	1.5	0.8–3.5	0.5	0.1–0.8	0.0145
Morbidity	147	8.2	12.5	6.4–21.4	5.1	1.9–11.1	<.0001	10.0	5.0–20.0	5.0	2.0–10.0	<0.0001
Pulmonary	23	1.3	5.3	1.0–16.7	0.7	0.2–2.2	<.0001	2.0	1.0–3.0	0.5	0.1–2.0	0.0009
Cardiac	5	0.3	1.5	1.0–2.9	0.1	0.02–0.3	0.0008	5.0	2.0–10.0	0.5	0.1–1.0	0.0009
DVT	4	0.2	1.7	0.3–4.6	0.5	0.2–0.9	0.2342	1.5	0.7–2.0	0.5	0.1–1.0	0.1884
Renal	7	0.4	0.6	0.3–0.9	0.1	0.04–0.4	0.0200	1.0	0.2–5.0	0.5	0.1–1.0	0.0463
SSI	68	3.8	6.9	3.9–12.6	3.1	1.5–6.3	<.0001	6.0	5.0–10.0	2.0	1.0–5.0	<0.0001

^*Wilcoxon test to test difference between event and no-event for the outcome category.

DVT, deep venous thromboembolism (deep venous thrombosis and/or pulmonary embolism); SSI, surgical site infection.

As shown in figure 1, post-operative morbidity and mortality events predominantly occurred in both the model and surgeon highest predicted risk quartile. Figure 1A shows the occurrences by quartile of model prediction of risk. Patients in the highest quartile of model predicted risk for an event accounted for 75% of the mortality, 52% of overall morbidity, 100% of cardiac, 50% of VTE, 61% of pulmonary, 57% of renal, and 51% of SSI. If model predicted risk was not associated with actual events, we would expect that each quartile of risk would contain about 25% of observed adverse events. Figure 1B shows the occurrences by quartiles of surgeon predicted risk. Patients in the highest quartile of surgeon predicted risk of an event accounted for 75% of the mortality, 55% of overall morbidity, 100% of cardiac, 50% of VTE, 61% of pulmonary, 42% of renal, and 49% of SSI, Thus both model and surgeon predicted risk placed similar numbers of patients who actually had events in the top quartile of risk. When comparing the model to surgeon estimate of risk for overall morbidity and mortality as well as each category of postoperative complication, a highly significant correlation was observed. The correlation between model prediction and surgeon prediction of risk was greatest for overall morbidity (Spearman 0.66, 95% CI 0.64–0.69, p<0.0001), mortality (Spearman 0.59, 95% CI 0.56–0.62, p<0.0001), and risk of surgical site infection (Spearman 0.58, 95% CI 0.54–0.61,p<0.0001). Significant correlations were also present for cardiac (Spearman 0.54,95% CI 0.51–0.58, p<0.0001), pulmonary (Spearman 0.53, 95% CI 0.49–0.56, p<0.0001), renal complications (Spearman 0.47, 95% CI 0.44–0.51, p<0.0001), and thromboembolism (Spearman 0.43, 95% CI 0.39–0.47, p<0.0001).

Figure 1.

(A) Morbidity, mortality and complication occurrences by model prediction of risk. (B) Morbidity, mortality and complication occurrences by attending prediction of risk. DVT, deep vein thrombosis; Pred, predicted; SSI, surgical site infection.

In addition to estimating a specific numerical value for risk, surgeons were asked to rate their perception of the individual patient’s risk for specific complications on an ordinal scale of low risk (bottom 25% of patients), average (25–75^th percentile of patients), or high risk (top 25% of all patients). The percentile groups were used as a suggested guideline for the surgeons. The association between surgeon perception (Low, Average, and High) and median model prediction and observed event rate for specific postoperative outcomes is shown in Table 3. The surgeons actually rated more of their patients as low and average risk and fewer of their patients as high risk compared to the suggested guidelines. The percentage of patients rated by the surgeons to be at the highest quartile risk for a specific complication ranged from 5.8% of patients for renal complications to 17.5% for overall morbidity. For each outcome, as surgeon perception of risk increased, median model prediction of risk and observed rate of the specific outcome measure increased as well. For example, when surgeons estimated the risk of morbidity to be low, the risk prediction model estimated the risk of overall morbidity to be 2.8% compared to 6.5% for average risk, and 10.0% for the highest quartile of perceived risk patients. Similarly, overall observed morbidity was 4.1% for low risk, compared to 7.6% for average, and 18.0% for patients perceived as high risk for morbidity by the surgeon.

Table 3.

Association between Surgeon Perception of Risk (Low, Average, and High) and Median Model Prediction and Actual Event Rate for Postoperative Outcomes (n=1,791 Patients)

		Distribution of surgeon perceived risk			Median model prediction of risk			Actual event rate
	Events	Low perceived risk, %	Average perceived risk, %	High perceived risk, %	Low perceived risk, %	Average perceived risk, %	High perceived risk, %	Low perceived risk, %	Average perceived risk, %	High perceived risk, %
Mortality	4	47.2	42.7	10.1	0.06	0.17	0.71	0.12	0.14	0.57
Morbidity	147	35.6	47.0	17.5	2.75	6.48	10.02	4.08	7.56	18.00
Pulmonary	4	46.2	41.3	12.5	0.33	0.90	2.09	0.75	0.84	5.09
Cardiac	23	47.8	41.0	11.2	0.05	0.12	0.27	0.00	0.42	1.55
DVT	5	44.9	46.2	8.9	0.27	0.56	0.71	0.13	0.25	0.65
Renal	7	50.2	44.0	5.8	0.08	0.16	0.34	0.23	0.40	2.02
SSI	68	35.1	50.1	14.8	1.85	3.61	5.11	2.31	3.94	6.30

Surgeon perception of risk: low perceived risk (bottom 25% of patients), average perceived risk (25–75^th percentile of patients), high perceived risk (>75^th percentile of patients) for perioperative morbidity and mortality.

Discussion

The purpose of the current study was to compare risk estimates from statistical models with operating surgeon’s estimates of risk as well as observed outcomes for a broad range of general surgical patients and postoperative adverse events. We found that the risk prediction models and surgeons could identify those patients who were more likely to develop specific surgical complications. Both the model and surgeons were also able to quantitatively predict the risk for specific complications for their patients. For each category of postoperative complications except for thromboembolism, the model and the surgeons predicted a higher risk for event occurrence in those patients who went on to have an occurrence than in those patients who did not.

We also found a fairly substantial correlation between model and surgeon risk prediction at the individual patient level and also good agreement between median model and surgeon estimates using only patient information available preoperatively for a diverse collection of general surgical procedures. Highly significant correlations were observed between the model and surgeon risk prediction for each outcome measure. This study supports the hypothesis that the risk prediction model preforms as well as experienced surgeons in estimating risk. Our findings also support that the model can identify those patients at highest risk for complications, allowing a system to target those patients for intervention who will experience the majority of morbidity. This applies to both high risk patients and patient undergoing higher risk surgeries. The extensive body of surgical literature on risk prediction primarily focuses on how well statistical models perform on predicting risk. However, it is primarily surgeons that predict risk on a daily basis, yet little work has been done to study how well surgeons accurately predict risk for their patients. We found one other study in the literature (11) that compared risk estimates from statistical models (the POSSUM models from Great Britain) with those from experienced surgeons for postoperative complications in 1,077 patients after major hepatobiliary or gastrointestinal surgery (11). That study found that the surgeon estimates of risk for overall morbidity (32.1%) was closer to the observed morbidity rate (29.5%) compared to that of the statistical model (46.4%). The statistical model also overestimated risk of mortality (6.9% vs. 3.4% observed, risk of mortality was not assessed by the surgeons). In contrast to our study, the surgeons estimate risk for postoperative complications after the operation was completed likely biasing their results in favor of surgeon prediction.

Current large scale strategies for improving surgical outcomes include the Surgical Care Improvement Project (SCIP) and the ACS NSQIP. The SCIP measures processes thought to be important for improving outcomes, but most evaluations of the program have not demonstrated that adherence to these processes has yielded improved outcomes.[12–14] The ACS NSQIP is voluntary and primarily involves feedback of risk adjusted outcomes to participating institutions on a semiannual basis. Hospitals participating in ACS-NSQIP have demonstrated improvement in outcomes from their baseline measures.[1] With the feedback of accurate, actionable data regarding outcomes, quality improvement then follows if the institution recognizes and reacts to the need to improve specific outcomes. This is the basis for quality improvement derived from participation in current large national databases like the ACS-NSQIP and University Healthcare Consortium (UHC). These programs are designed to allow comparison of outcomes at the hospital level after risk adjustment. An additional approach to quality improvement might be to predict risk at the individual patient level prior to the surgery. This would potentially allow for mitigation of risk of specific surgical complications in a prospective fashion. While several such risk calculators are currently available, to our knowledge, none have included a broad. spectrum of both different types of operations and different surgical outcomes.[15–17]

The greatest utility of this risk prediction model lies in the potential usefulness to practicing surgeons, clinicians, and other members of the perioperative home in their efforts at reducing morbidity and mortality. If broadly applied, this model would provide all clinicians with real time estimates of patient risk as the patient is seen preoperatively. A reliable estimate of a patient’s specific risk of an event could be useful in the informed consent process. More importantly as this risk estimate is available to the surgical team prior to surgery, the team would have time to trigger additional efforts to mitigate risk to the patient. Incorporating this decision support tool into the existing electronic medical record or on a smart phone application, for example, could allow for real time risk prediction and mitigation at the time of the initial preoperative consultation. IIn patients found to have an elevated risk of a perioperative cardiac event, further cardiac evaluation, risk stratification, and mitigation strategies could be pursued.[18] In patients found to be at increased risk for DVT, a strategy of administering preoperative low molecular weight heparin to decrease risk could be activated.[19] In cases of elective surgery, the operation might be postponed to allow for preoperative conditioning or alternative non-surgical management might be entertained if the surgeon and/or patient and family feel the risk for a postoperative event outweighs the potential benefits of the operation. Or, if no potential mitigating intervention is available, escalation of care including intensive and post-operative care setting may improve care and assist with allocation of these resources.[20]

The advantage of the model risk prediction tool is that it could be used by any clinical member of the surgical team, including the preoperative clinic or the anesthesiologist to provide an objective, quantifiable estimate of an individual patient’s risk for undergoing a particular procedure. It would be advantageous for hospitals in which many different surgeons operate to have a standardized approach to preoperative risk prediction and mitigation that is not dependent on individual practitioners’ specialty or experience, thus a system-based solution.

Limitations

Overall the patients in this study experienced relatively few complications and very few deaths. The lack of statistical significance for thromboembolism, for example, is possibly a function of the infrequency by which the events occurred in this cohort. For example, the model predicted a thromboembolism risk of 1.7% in those patients who had an event compared to 0.5% in those who did not, over a three fold difference yet not statistically significant as only four patients actually had an event. Furthermore, while the model predicted an increased risk of adverse events in patients who went on to have the adverse event, the absolute value of predicted risk and difference in risk over those who did not have the event was relatively low. This is possibly related to the omission from the model of other preoperative variables that might more specifically predict risk of a specific complication. For example, the inclusion of family history of deep venous thrombosis (DVT) might add to a patient’s predicted risk of DVT, but is not currently collected in the DS3 software.[21] Of course, other as yet to be determined variables might exist that could allow for increased sensitivity of the model for specific risk prediction. Finally, as this was a prospective evaluation of the utility of the DS3 model, none of the surgeons in this study had prior experience using this DS3 model for risk prediction as done in this study. That said, many of the surgeons participating in this study have had extensive experience using NSQIP data for quality improvement purposes and were familiar with the potential influence of specific prospective variables on patient outcomes. For this reason, surgeon estimates of risk in this study could be more accurate compared to surgeons less familiar with the utilitiy of these data and this methology. In other words, the model estimates of risk might be more accurate and the differences between model estimates and surgeon estimates of risk might be understated by this study.

Conclusion

This study has confirmed that there is good agreement between risk estimation by a statistical model and surgeons across a broad spectrum of surgical operations and outcomes. Further studies are warranted to determine if implementation of this tool with appropriate interventions in the clinical setting will result in a reduction of surgical complications.

Appendix 1: DS3 Study Group

University of Alabama Birmingham

John Christein, MD

Jamie Cannon, MD

Melanie Morris, MD

John Porterfield, MD

Richard Stahl, MD

Jayleen Grams, MD

Martin Heslin, MD

Marc Passman, MD

Laura Altom MD

Emily Roberson RN

University of Utah

Robert Andtbacka, MD

Cherisse Davis, RN

Toby Enniss, MD

John Langell, MD

Sean Mulvihill, MD

Edward Nelson, MD

Raminder Nirula, MD

William Peche MD

Joyce Pell, RN

Amber Ryckaert, RN

Courtney Scaife, MD

Bradford Sklow, MD

John Sorensen, MD

Daniel Vargo, MD

Eric Volckmann, MD

QC Metrix, Inc

Babar N. Rao, System Architect

Appendix 2 Top 25 Procedures for Each Site by CPT Code and Associated RVU (University of Utah, University of Alabama, MSQC)

Appendix 3 DS3 QCM|Surgery DS3 Grant Data Collection Form

Surgeon: ____________________________________ Visit/Exam Date:__________________________

Hospital ID: _______________________ Clinic ID: ________________________ QCMetrix ID: ___________ (system generated)

Pt. Info. : First Name: ____________________ MI: ______ Last Name: __________________________ DOB: __________

Gender: Male | Female | UTD

Ethnicity: Hispanic or Latino | Non-Hispanic and Non-Latino | UTD

Race: Unknown | American Indian/Alaska Native | Asian | Black/African American | White | Native Hawaiian /Pacific Islander

Inpatient Surgery? Yes | No Planned Principal Procedure (Description)_________________________(CPT Code): _______

Wound Classification: Clean | Clean Contaminated | Contaminated | Dirty/Infected Currently has open wound? Yes | No

Medical History
Condition	Yes	No	Date of Onset	Most Recent Date
Metastatic Cancer
Peripheral Vascular Disease (PVD)
Hypertension
Chronic Pulmonary Disease (COPD)
Bleeding Disorder
Congestive Heart Failure

Chronic Steroid Use? Yes | No Percutaneous Coronary Intervention (PCI)? Yes | No

Current Smoker? Yes | No Previous Cardiac Surgery? Yes | No Dialysis? Yes / No

Functional Status? Independent | Partially Dependent | Totally Dependent | Unknown

Weight: ________ lbs | kgs Height: ________ inches | cm

ASA Class: O-None Assigned | 1-Healthy | 2-Mild Systemic Disease | 3-Severe Systemic Disease 4- Severe Systemic Disease that is a constant threat to life | 5-Not expected to survive the operation

Other factors influencing your estimates of risk for this patient: ____________________________________________________________________________________________________________ ____________________________________________________________________________________________________________

	Resident Estimate of risk(%) (Define to 0.1% for risk < 1%)	Resident Perception of Risk (check one per row)			Attending Estimate of risk(%) (Define to 0.1% for risk < 1%)	Attending Perception of Risk (check one per row)
		High	Avg	Low		High	Avg	Low
Mortality
1+ Complications
Cardiac
DVT
Pulmonary
Renal
SSI

Select One

Completed By: ______________ NP PA Resident PGY _______ Completed By: ______________

CONFIDENTIAL: This material is prepared pursuant to Utah Code Annotated Section 26-25-1 et seq. for the purpose of evaluating health care rendered by hospitals and/or physicians and is NOT PART of the medical record. It is also classified as "protected" under the Government Records Access and Management Act, Utah Code Annotated Section 63-2-101 et seq

Definitions for DS3

Current Smoker: cigarettes within one year. Does not include cigars, pipes, or chewing tobacco.

COPD: functional disability from COPD, past hospitalization for COPD, chronic bronchodilator therapy, FEV <75% of predicted on PFTs. Does not include asthma or interstitial fibrosis or sarcoidosis.

PCI: patient has undergone balloon dilatation or stent placement at any time, or has had it attempted.

Previous Cardiac Surgery: CABG, valve repairs/ replacements, atrial or ventricular septal defects, great thoracic vessel repair, transplantation, left ventricular aneurysmectomy, LVAD insertions, etc. Does not include pacemaker or AICD insertions.

CHF: only “current” if new diagnosis or new signs and symptoms within 30 days prior to surgery.

Hypertension: persistent SBP > 140 or DBP > 90 or requires antihypertensive treatment within 30 days prior to surgery.

Chronic Steroid Use: oral or parenteral corticosteroid in the 30 days prior to surgery for a chronic medical condition. Does not include topical corticosteroids applied to the skin or administered by inhalation or rectally. Does not include short course steroids (duration 10 days or less) in the 30 days prior to surgery.

Bleeding Disorders: any condition that places the patient at risk for excessive bleeding, requiring hospitalization due to a deficiency of blood clotting elements. Does not include aspirin therapy. Does include anticoagulants not discontinued within sufficient time to have worn off.

Wound Classification:

1. Clean: uninfected operative wound in which no inflammation is encountered and the respiratory, alimentary, genital, or uninfected urinary tract is not entered. Wound is closed and, if necessary, drained with closed drainage.

2. Clean/Contaminated: an operative wound in which the respiratory, alimentary, genital, or urinary tracts are entered under controlled conditions and without unusual contamination. Operations involving the biliary tract, appendix, vagina, and oropharynx are included in this category.

3. Contaminated: Open, fresh, accidental wounds. Also includes operations with major breaks in sterile technique or gross spillage from the GI tract, and incisions in which acute, nonpurulent inflammation is encountered including necrotic tissue without evidence of purulent drainage.

4. Dirty/Infected: Old traumatic wounds with retained devitalized tissue and those that involve existing clinical infection or perforated viscera.