Development and Validation of a Multivariable Prediction Model for Postoperative Intensive Care Unit Stay in a Broad Surgical Population

Paul D Rozeboom; William G Henderson; Adam R Dyas; Michael R Bronsert; Kathryn L Colborn; Anne Lambert-Kerzner; Karl E Hammermeister; Robert C McIntyre, Jr; Robert A Meguid

doi:10.1001/jamasurg.2021.7580

. 2022 Feb 16;157(4):344–352. doi: 10.1001/jamasurg.2021.7580

Development and Validation of a Multivariable Prediction Model for Postoperative Intensive Care Unit Stay in a Broad Surgical Population

Paul D Rozeboom ^1,², William G Henderson ^2,^3,⁴, Adam R Dyas ^1,², Michael R Bronsert ^2,³, Kathryn L Colborn ^1,^2,⁴, Anne Lambert-Kerzner ^2,⁴, Karl E Hammermeister ^2,^4,⁵, Robert C McIntyre Jr ^1,², Robert A Meguid ^1,^2,^3,^✉

¹Department of Surgery, University of Colorado School of Medicine, Aurora

²Surgical Outcomes and Applied Research Program, University of Colorado School of Medicine, Aurora

³Adult and Child Center for Health Outcomes Research and Delivery Science, University of Colorado School of Medicine, Aurora

⁴Department of Biostatistics and Informatics, Colorado School of Public Health, Aurora

⁵Division of Cardiology, Department of Medicine, University of Colorado School of Medicine, Aurora

Accepted for Publication: December 7, 2021.

Published Online: February 16, 2022. doi:10.1001/jamasurg.2021.7580

^✉

Corresponding Author: Robert A. Meguid, MD, MPH, Department of Surgery, University of Colorado School of Medicine, Anschutz Medical Campus, 12631 E 17th Ave, C-310, Aurora, CO 80045 (robert.meguid@cuanschutz.edu).

Author Contributions: Drs Rozeboom and Meguid had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Rozeboom, Henderson, Dyas, Colborn, Hammermeister, McIntyre, Meguid.

Acquisition, analysis, or interpretation of data: Rozeboom, Dyas, Bronsert, Colborn, Lambert-Kerzner, Hammermeister, Meguid.

Drafting of the manuscript: Rozeboom, Henderson, Dyas, Meguid.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Rozeboom, Henderson, Bronsert, Meguid.

Obtained funding: Dyas, Meguid.

Administrative, technical, or material support: Colborn, McIntyre, Meguid.

Supervision: Henderson, McIntyre, Meguid.

Conflict of Interest Disclosures: Dr McIntyre reported receiving a research grant from Genentech during the conduct of the study. No other disclosures were reported.

Funding/Support: This work was supported by Surgical Outcomes and Applied Research program funding from the Department of Surgery, University of Colorado School of Medicine.

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The American College of Surgeons National Surgical Quality Improvement Program and participating hospitals are the source of these data; they have not verified and are not responsible for the statistical validity of the data analysis or the conclusions derived by the authors.

^✉

Corresponding author.

PMCID: PMC8851361 PMID: 35171216

This decision analytical model study investigates if postoperative intensive care unit admission can be accurately predicted using the 8 variables of the Surgical Risk Preoperative Assessment System.

Key Points

Question

Can postoperative intensive care unit (ICU) admission be accurately predicted using the 8 preoperative variables of the Surgical Risk Preoperative Assessment System?

Findings

In this decision analytical model study including data from 34 568 patients, the Surgical Risk Preoperative Assessment System accurately predicted postoperative ICU admission.

Meaning

Predicting postoperative ICU admission allows planning for use of a valuable and limited resource, particularly in settings where ICU beds are scarce.

Abstract

Importance

Despite limited capacity and expensive cost, there are minimal objective data to guide postoperative allocation of intensive care unit (ICU) beds. The Surgical Risk Preoperative Assessment System (SURPAS) uses 8 preoperative variables to predict many common postoperative complications, but it has not yet been evaluated in predicting postoperative ICU admission.

Objective

To determine if the SURPAS model could accurately predict postoperative ICU admission in a broad surgical population.

Design, Setting, and Participants

This decision analytical model was a retrospective, observational analysis of prospectively collected patient data from the 2012 to 2018 American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) database, which were merged with individual patients’ electronic health record data to capture postoperative ICU use. Multivariable logistic regression modeling was used to determine how the 8 preoperative variables of the SURPAS model predicted ICU use compared with a model inputting all 28 preoperatively available NSQIP variables. Data included in the analysis were collected for the ACS NSQIP at 5 hospitals (1 tertiary academic center, 4 academic affiliated hospitals) within the University of Colorado Health System between January 1, 2012, and December 31, 2018. Included patients were those undergoing surgery in 9 surgical specialties during the 2012 to 2018 period. Data were analyzed from May 29 to July 30, 2021.

Exposure

Surgery in 9 surgical specialties, including general, gynecology, orthopedic, otolaryngology, plastic, thoracic, urology, vascular, and neurosurgery.

Main Outcomes and Measures

Use of ICU care up to 30 days after surgery.

Results

A total of 34 568 patients were included in the analytical data set: 32 032 (92.7%) in the cohort without postoperative ICU use and 2545 (7.4%) in the cohort with postoperative ICU use (no ICU use: mean [SD] age, 54.9 [16.6] years; 18 188 women [56.8%]; ICU use: mean [SD] age, 60.3 [15.3] years; 1333 men [52.4%]). For the internal chronologic validation of the 7-variable SURPAS model, data from 2012 to 2016 were used as the training data set (n = 24 250, 70.2% of the total sample size of 34 568) and data from 2017 to 2018 were used as the test data set (n = 10 318, 29.8% of the total sample size of 34 568). The C statistic improved in the test data set compared with the training data set (0.933; 95% CI, 0.924-0.941 vs 0.922; 95% CI, 0.917-0.928), whereas the Brier score was slightly worse in the test data set compared with the training data set (0.045; 95% CI, 0.042-0.048 vs 0.045; 95% CI, 0.043-0.047). The SURPAS model compared favorably with the model inputting all 28 NSQIP variables, with both having good calibration between observed and expected outcomes in the Hosmer-Lemeshow graphs and similar Brier scores (model inputting all variables, 0.044; 95% CI, 0.043-0.048; SURPAS model, 0.045; 95% CI, 0.042-0.046) and C statistics (model inputting all variables, 0.929; 95% CI, 0.925-0.934; SURPAS model, 0.925; 95% CI, 0.921-0.930).

Conclusions and Relevance

Results of this decision analytical model study revealed that the SURPAS prediction model accurately predicted postoperative ICU use across a diverse surgical population. These results suggest that the SURPAS prediction model can be used to help with preoperative planning and resource allocation of limited ICU beds.

Introduction

Intensive care unit (ICU) beds are an expensive and limited resource, which was particularly demonstrated during the COVID-19 pandemic.^1,2,3,4 With roughly 10% of surgical patients requiring postoperative ICU care⁵ and 22 million surgical procedures in the US annually,⁶ there are more than 2 million postoperative ICU admissions each year. Unanticipated surges in ICU admissions, like those experienced during the COVID-19 pandemic, present problems for patients, health care professionals, and hospital systems alike.³

Accurate preoperative prediction of surgical patients requiring postoperative ICU care could prove useful in allocating this precious resource. We identified 26 studies^{7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32} over the past 20 years related to predicting postoperative ICU use. Most of the studies (77%) covered specific types of surgery and thus were not generalizable to broad surgical populations.^{7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26} Six studies^{27,28,29,30,31,32} included broader surgical populations, but 3 studies^27,28,29 had small sample sizes (100 to 6000 patients), and 1 study included both preoperative and intraoperative predictors, making it not useful for prediction in the preoperative period.³⁰ One study³¹ from Singapore General Hospital developed a surgical risk calculator for the prediction of postsurgical mortality and ICU admission in 79 914 noncardiac and nonneurosurgical patients that could be useful for a broad surgical population.³² There have been a paucity of studies using the data collected by the American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP), probably because the database does not include ICU admission as an outcome. Kongkaewpaisan and colleagues²⁰ used the ACS NSQIP database to predict ICU need, which was a composite of outcomes that probably required ICU admission, but this does not necessarily equate to actual ICU use.

Meguid and colleagues^33,34,35 and Henderson et al³⁶ developed the Surgical Risk Preoperative Assessment System (SURPAS), which is a universal surgical risk calculator that uses 8 preoperative variables to estimate the risk of 12 postoperative adverse outcomes.^37,38,39 The models were developed using the large ACS NSQIP database encompassing more than 6 million surgical cases from 9 surgical specialties. However, postoperative ICU use is currently not one of the outcomes predicted by SURPAS. The purpose of this study was to determine if the 8-variable SURPAS risk model accurately estimates postoperative ICU admission.

Methods

Study Design and Patient Selection

This was a retrospective, observational, decision analytical model study of the prospectively collected ACS NSQIP database from January 1, 2012, to December 31, 2018. Because the ACS NSQIP does not collect postoperative ICU use, we restricted the study to the University of Colorado Health System (UCHealth) ACS NSQIP data so that we could obtain postoperative ICU use from the local electronic health record software, Epic (Epic Systems Corporation). The UCHealth System hospitals participating in the ACS NSQIP included the University of Colorado Hospital and 4 academic-affiliated hospitals in the Colorado Springs and Fort Collins metropolitan areas of the state. This research was approved by the Colorado Multiple Institutional Review Board with a waiver of informed consent owing to the use of deidentified patient data. The Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guidelines were followed in the analysis and reporting of this study.

Exposures

The ACS NSQIP collects a systematic sample of surgical cases in 9 specialty areas (general, gynecology, orthopedic, otolaryngology, plastic, thoracic, urology, vascular, and neurosurgery). Data collection for each surgical case includes preoperative demographic characteristics and comorbidities, selected variables about the operation, and 30-day postoperative outcomes. Data are collected by trained, certified, surgical clinical reviewers and are periodically audited for accuracy and completeness. NSQIP collects 40 preoperative variables, including 28 patient characteristics and comorbidities and 12 laboratory values. Previous work has shown that the laboratory variables do not add to the predictive ability of the 28 patient characteristics and comorbidities³⁴; therefore, these were not used in the analysis. The race and ethnicity variables came from the ACS NSQIP database. These were self-assigned or assigned by institutional personnel per the practice of the institution. Race and ethnicity were used to characterize the sample and to study their association with postoperative ICU use. The specific races and ethnicities identified included American Indian or Alaska Native, Asian or Pacific Islander, non-Hispanic Black, Hispanic, non-Hispanic White, or unknown if that information was missing from the patient’s record. The 8 preoperative variables included in the SURPAS prediction model were 4 related to the operation (Current Procedural Terminology [CPT]–specific event rate of the outcome of interest calculated from the ACS NSQIP database, work relative value unit [RVU] of the primary operation, specialty of the primary surgeon, and inpatient/outpatient setting) and 4 related to the patient (age, American Society of Anesthesiology [ASA] physical status classification I-V, functional health status, and emergency status). In this study, CPT-specific event rate was not used because it was believed that the estimates using only UCHealth data would not be very accurate owing to small sample size and data origin from only 1 health care system.

Primary Outcome

Patient records in the UCHealth ACS NSQIP sample (2012-2018) were matched to their electronic health record data from Epic using the patient medical record numbers and operative dates to obtain 30-day postoperative ICU use (yes or no). The data and linkage were provided by the University of Colorado’s Health Data Compass Data Warehouse project.

Statistical Analysis

If a patient had 1 or more operations in the database, only the first operation was used. Patients were also excluded if they had any missing variables (complete-case analysis). An analysis of the bivariable association of each of the 28 preoperative predictor variables with postoperative ICU use was performed using a χ² test for categorical predictor variables or an unpaired t test for continuous variables.

A stepwise forward-selection logistic regression analysis was performed including all 28 ACS NSQIP preoperative predictor variables, using P values of .05 for entry into and exiting the prediction model. All P values were 2-sided with significance set as P < .05. The established SURPAS model was then performed using 7 of the 8 SURPAS preoperative predictor variables. The 2 models were compared for performance using the C statistic for discrimination, Hosmer-Lemeshow (H-L) for goodness of fit, and Brier scores for overall fit. In the SURPAS model, variables were ordered by their χ² test statistic and not by stepwise forward selection.

An internal chronologic validation of the 7-variable SURPAS model was done by temporally splitting the data into a training set (2012-2016) and test set (2017-2018). The model was developed using the training set and applied to both the training and the test data sets. The C statistic, H-L graphs, and Brier scores were then compared between the training and test data sets. All statistical analyses and the H-L graphs were performed from May 29 to July 30, 2021, using SAS, version 9.4 (SAS Institute).

Results

Patient Characteristics

There were 35 073 observations in the UCHealth NSQIP database and 34 587 unique patient identification numbers. For those patients with more than 1 operation, the first operation was selected. Nineteen patients were excluded for missing work RVU, resulting in a total of 34 568 patients: 32 032 (92.7%) in the cohort without postoperative ICU use and 2545 (7.4%) in the cohort with postoperative ICU use (no ICU use: mean [SD] age, 54.9 [16.6] years; 18 188 women [56.8%]; 13 835 men [43.2%]; ICU use: mean [SD] age, 60.3 [15.3] years; 1212 women [47.6%]; 1333 men [52.4%]) in the analytical data set. In the group without ICU use, 98 patients (0.3%) were identified as American Indian or Alaska Native, 428 (1.3%) as Asian or Pacific Islander, 1196 (3.7%) as non-Hispanic Black, 3143 (9.8%) as Hispanic, 25 625 (80.0%) as non-Hispanic White, and 1533 (4.8%) as unknown race. In the group with ICU use, 11 patients (0.4%) were identified as American Indian or Alaska Native, 30 (1.2%) as Asian or Pacific Islander, 120 (4.7%) as non-Hispanic Black, 234 (9.2%) as Hispanic, 2023 (79.5%) as non-Hispanic White, and 127 (5.0%) as unknown race.

eFigure 1 in the Supplement presents the frequency distribution of days from operation to ICU admission for the 2545 patients admitted to the ICU: 1467 of 2545 patients (57.7%) were admitted on the day of surgery (postoperative day [POD] 0), 423 of 2545 (16.6%) on POD 1 (cumulative 74% within 1 day after surgery), 2172 of 2545 (85.2%) within POD 4, and 2309 of 2545 (90.6%) within POD 8. Table 1 presents the bivariable association between the 28 ACS NSQIP preoperative characteristics and postoperative ICU use. The group with postoperative ICU use had more men than women (1333 [8.8%] vs 1212 [6.2%]), more patients with underweight (113 [17.7%]) than patients with normal weight (777 [8.0%]) or patients with overweight (752 [6.7%]), and patients with each of 17 different comorbidities. The group with postoperative ICU use had more patients who were partially (188 [25.4%]) or totally (25 [28.7%]) dependent than independent (2332 [6.9%]) of activities of daily living, more patients who were transferred from another acute care hospital (349 [38.2%]) or chronic care facility (34 [23.1%]) than from home (2162 [6.4%]) before surgery, more inpatients (2501 [13.8%]) than outpatients (44 [0.3%]), more patients undergoing emergency operations (385 [16.6%]) than not (2160 [6.7%]), more patients with higher ASA class than lower ASA class (I: 13 [0.4%]; II: 577 [3.3%]; III: 1460 [11.7%]; IV: 468 [33.6%]; V: 27 [84.4%]), and more patients undergoing more complex operations than in the group that did not use postoperative ICU (ICU use: mean [SD] work RVU, 26.1 [12.0] units vs no ICU use: mean [SD] work RVU, 15.6 [7.6] units). There were also large differences in ICU use by surgical specialty: the highest users were in vascular surgery, 46.9% (n = 328); neurosurgery, 28.0% (n = 732); and thoracic surgery, 26.5% (n = 242). ICU use in the other 6 specialties ranged from 0.9% (n = 35) in gynecology to 8.1% (n = 789) in general surgery.

Table 1. Association Between Preoperative Patient and Operative Characteristics and Postoperative ICU Use.

Characteristic	No. (%)
Characteristic	No ICU use (n = 32 023)	ICU use (n = 2545)
Sex
Female	18 188 (93.8)	1212 (6.2)
Male	13 835 (91.2)	1333 (8.8)
Age, mean (SD), y	54.9 (16.6)	60.3 (15.3)
Race and ethnicity
American Indian or Alaska Native	98 (89.9)	11 (10.1)
Asian or Pacific Islander	428 (93.5)	30 (6.5)
Non-Hispanic Black	1196 (90.9)	120 (9.1)
Hispanic	3143 (93.1)	234 (6.9)
Non-Hispanic White	25 625 (92.7)	2023 (7.3)
Null/unknown	1533 (92.4)	127 (7.7)
Body mass index category^a
Underweight (<18.5)	524 (82.3)	113 (17.7)
Normal weight (18.5-24.9)	8940 (92.0)	777 (8.0)
Overweight (25.0-29.9)	10 437 (93.3)	752 (6.7)
Obesity class
I (30.0-34.9)	6413 (93.3)	460 (6.7)
II (35.0-39.9)	2955 (93.5)	204 (6.5)
III (≥40.0)	2220 (92.7)	175 (7.3)
Null/unknown	534 (89.3)	64 (10.7)
Ascites (within 30 d)
No	31 953 (92.7)	2510 (7.3)
Yes	70 (66.7)	35 (33.3)
Bleeding disorder requiring hospitalization
No	30 971 (93.2)	2251 (6.8)
Yes	1025 (78.2)	294 (21.8)
Diabetes
No	28 396 (93.2)	2083 (6.8)
Yes
Oral	2420 (91.2)	234 (8.8)
Insulin	1207 (84.1)	228 (15.9)
Dialysis or hemofiltration (within 2 wk)
No	31 829 (92.7)	2497 (7.3)
Yes	194 (80.2)	48 (19.8)
Disseminated cancer
No	31 369 (93.3)	2257 (6.7)
Yes	654 (69.4)	288 (30.6)
Dyspnea (within 30 d)
None	29 766 (93.2)	2185 (6.8)
Moderate exertion	2148 (87.8)	298 (12.2)
At rest	109 (63.7)	62 (36.3)
Functional health status before surgery
Independent	31 408 (93.1)	2332 (6.9)
Partially dependent	553 (74.6)	188 (25.4)
Totally dependent	62 (71.3)	25 (28.7)
Congestive heart failure (within 30 d)
No	31 917 (92.8)	2469 (7.2)
Yes	106 (58.2)	76 (41.8)
Severe chronic obstructive pulmonary disease
No	30 856 (93.2)	2263 (6.8)
Yes	1167 (80.5)	282 (19.5)
Blood pressure >140/90 mm Hg or taking antihypertension medication
No	20 385 (94.1)	1288 (5.9)
Yes	11 638 (90.3)	1257 (9.7)
Systemic sepsis (within 48 h)
None	30 475 (93.7)	2046 (6.3)
Systemic inflammatory response syndrome	1007 (83.9)	193 (16.1)
Sepsis	522 (71.3)	210 (28.7)
Septic shock	19 (16.5)	96 (83.5)
Acute kidney failure (rising creatinine to >3 mg/dL within 24 h)
No	31 983 (92.7)	2522 (7.3)
Yes	40 (63.5)	23 (36.5)
Cigarette smoker (within 1 y)
No	26 791 (93.1)	1979 (6.9)
Yes	5232 (90.2)	566 (9.8)
Steroid use for chronic condition
No	30 887 (93.1)	2274 (6.9)
Yes	1136 (80.7)	271 (19.3)
Transfusion of pack red blood cells (within 72 h)
No	31 903 (92.8)	2468 (7.2)
Yes	120 (60.9)	77 (39.1)
Ventilator dependent (within 48 h)
No	32 003 (92.9)	2444 (7.1)
Yes	20 (16.5)	101 (83.5)
Open wound with or without infection
No	31 551 (93.0)	2382 (7.0)
Yes	472 (74.3)	163 (25.7)
>10% Loss of body weight (within 6 mo)
No	31 683 (92.9)	2405 (7.1)
Yes	340 (70.8)	140 (29.2)
Transfer status
Admitted directly from home	31 345 (93.6)	2162 (6.4)
Acute care hospital	565 (91.8)	349 (38.2)
Chronic care facility	113 (76.9)	34 (23.1)
Inpatient/outpatient
Inpatient	15 681 (86.2)	2501 (13.8)
Outpatient	16 342 (99.7)	44 (0.3)
Primary surgeon specialty
General surgery	8939 (91.9)	789 (8.1)
Gynecology	3677 (99.1)	35 (0.9)
Neurosurgery	1884 (72.0)	732 (28.0)
Orthopedics	11 248 (98.3)	194 (1.7)
Otolaryngology	1656 (94.0)	105 (6.0)
Plastic	803 (65.7)	36 (4.3)
Thoracic	670 (73.5)	242 (26.5)
Urology	2775 (97.1)	84 (2.9)
Vascular	371 (53.1)	328 (46.9)
Emergency surgery
No	30 090 (93.3)	2160 (6.7)
Yes	1933 (83.4)	385 (16.6)
American Society of Anesthesiology class
I (Patient without systemic disease)	3145 (99.6)	13 (0.4)
II (Patient with mild systemic disease)	16 888 (96.7)	577 (3.3)
III (Patient with severe systemic disease)	11 062 (88.3)	1460 (11.7)
IV (Patient with severe systemic disease that is a constant threat)	923 (66.4)	468 (33.6)
V (Moribund patient not expected to survive)	5 (15.6)	27 (84.4)
Work relative value unit,^b mean (SD)	15.6 (7.6)	26.1 (12.0)

Open in a new tab

Abbreviation: ICU, intensive care unit.

SI conversion factor: To convert creatinine to micromole per liter, multiply by 88.4.

^{^a}

Body mass index calculated as weight in kilograms divided by height in meters squared.

^{^b}

Work relative value unit is provided by the American Medical Association and is an estimate of procedure complexity in units.

Outcomes

Table 2 presents the stepwise logistic regression analysis using the 28 ACS NSQIP preoperative predictor variables to predict postoperative ICU use. Twelve of the 28 variables were statistically significant predictors. The first 4 variables (3 related to the operation: surgical specialty, work RVU, inpatient/outpatient operation; and 1 related to the patient: ASA class) accounted for 98.8% (C statistic, 0.923; 95% CI, 0.918-0.927) of the total C statistic of the 28-variable model (C statistic, 0.934; 95% CI, 0.930-0.939). Figure 1, which displays the H-L graphs of observed and expected events (applying the 12-variable model to the full data set), showed good calibration. The Brier score was 0.044 (95% CI, 0.043-0.048).

Table 2. Stepwise Multiple Logistic Regression Analysis of Postoperative Intensive Care Unit Stay (Dependent Variable) and Preoperative Patient and Operative Characteristics (Independent Variable) Applied to Entire Data Set of 34 568 Surgical Patients^a.

Characteristic	Odds ratio (95% CI)^b	C statistic
Characteristic	Odds ratio (95% CI)^b	Cumulative (95% CI)	Maximal %
1. Primary surgeon specialty vs general surgery
Gynecology	0.26 (0.18-0.38)	0.802 (0.794-0.811)	85.9
Neurosurgery	2.55 (2.22-2.93)
Orthopedics	0.17 (0.14-0.20)
Otolaryngology	2.45 (1.88-3.19)
Plastic	0.81 (0.55-1.20)
Thoracic	1.85 (1.53-2.24)
Urology	0.27 (0.21-0.35)
Vascular	4.43 (3.62-5.42)
2. Work relative unit^c	1.09 (1.08-1.09)	0.865 (0.857-0.874)	92.6
3. Inpatient/outpatient operation	18.40 (13.41-25.24)	0.909 (0.904-0.915)	97.3
4. ASA physical status classification
I (Patient without systemic disease)	1 [Reference]	0.923 (0.918-0.927)	98.8
II (Patient with mild systemic disease)	2.97 (1.67-5.30)
III (Patient with severe systemic disease)	5.86 (3.30-10.42)
IV (Patient with severe systemic disease that is a constant threat) and V (patient not expected to survive)	11.96 (6.61-21.61)
5. Disseminated cancer	2.14 (1.78-2.57)	0.924 (0.919-0.928)	98.9
6. Functional health status before operation vs independent
Partially dependent	2.06 (1.62-2.62)	0.925 (0.921-0.930)	99.0
Totally dependent	2.97 (1.62-5.45)	0.925 (0.921-0.930)	99.0
7. History of congestive heart failure	3.13 (2.09-4.68)	0.926 (0.922-0.931)	99.1
8. Transfer status vs home
Acute care hospital	2.36 (1.95-2.85)	0.928 (0.923-0.932)	99.4
Chronic care facility	1.22 (0.73-2.02)	0.928 (0.923-0.932)	99.4
9. Bleeding disorder requiring hospitalization	1.58 (1.32-1.90)	0.928 (0.924-0.933)	99.4
10. Steroid use for chronic condition	1.49 (1.24-1.78)	0.929 (0.924-0.933)	99.5
11. Dyspnea vs none
Moderate exertion	1.08 (0.92-1.28)	0.929 (0.924-0.933)	99.5
At rest	2.39 (1.58-3.63)	0.929 (0.924-0.933)	99.5
12. Open wound with or without infection	2.08 (1.61-2.67)	0.929 (0.925-0.934)	99.5

Open in a new tab

Abbreviation: ASA, American Society of Anesthesiology.

^{^a}

Inputting all 28 nonlaboratory National Surgical Quality Improvement Program characteristics, only 12 were statistically significant.

^{^b}

All P values were <.001, and the y-intercept was −8.2035.

^{^c}

Work relative unit is provided by the American Medical Association and is an estimate of procedure complexity in units.

Table 3 presents the 7-variable SURPAS model. The total C statistic was 0.925 (95% CI, 0.921-0.930), just slightly smaller than the C statistic of the 12-variable model of 0.929 (95% CI, 0.925-0.934). The H-L graphs of observed and expected events (applying the SURPAS model to the full data set) also showed good calibration (Figure 2). The Brier score was 0.045 (95% CI, 0.042-0.046), just slightly larger than the Brier score of the 12-variable model.

Table 3. Stepwise Multiple Logistic Regression Analysis of Postoperative Intensive Care Unit Stay (Dependent Variable) and the 7 Surgical Risk Preoperative Assessment System Predictor Variables Applied to Entire Data Set of 34 568 Surgical Patients.

Characteristic	Odds ratio (95% CI)^a	C statistic
Characteristic	Odds ratio (95% CI)^a	Cumulative (95% CI)	Maximal %
1. Primary surgeon specialty vs general surgery
Gynecology	0.31 (0.22-0.44)	0.802 (0.794-0.811)	86.7
Neurosurgery	2.76 (2.40-3.17)
Orthopedics	0.19 (0.16-0.22)
Otolaryngology	2.74 (2.11-3.55)
Plastic	0.92 (0.62-1.36)
Thoracic	2.35 (1.94-2.84)
Urology	0.29 (0.22-0.37)
Vascular	5.30 (4.33-6.48)
2. Work relative unit^b	1.09 (1.08-1.09)	0.865 (0.857-0.874)	93.5
3. Inpatient/outpatient operation	19.68 (14.36-26.97)	0.909 (0.904-0.915)	98.3
4. ASA physical status classification
I (Patient without systemic disease)	1 [Reference]	0.923 (0.918-0.927)	99.8
II (Patient with mild systemic disease)	3.49 (1.96-6.19)
III (Patient with severe systemic disease)	8.13 (4.58-14.44)
IV (Patient with severe systemic disease that is a constant threat) & V (patient who is not expected to survive)	18.27 (10.14-32.92)
5. Emergency operation	2.33 (1.97-2.75)	0.924 (0.919-0.928)	99.9
6. Functional health status before operation vs independent
Partially dependent	2.80 (2.23-3.51)	0.925 (0.921-0.930)	100
Totally dependent	4.16 (2.31-7.49)	0.925 (0.921-0.930)	100
7. Age^c	1.00 (0.99-1.00)	0.925 (0.921-0.930)	100

Open in a new tab

Abbreviation: ASA, American Society of Anesthesiology.

^{^a}

All P values are <.001.

^{^b}

Work relative unit is provided by the American Medical Association and is an estimate of procedure complexity in units.

^{^c}

P = .14. The y-intercept was −8.3515.

For the internal chronologic validation of the 7-variable SURPAS model, 2012-2016 were used as the training data set (24 250 of 34 568 [70.2%]) and 2017-2018 were used as the test data set (10 318 of 34 568 [29.8%]). eFigure 2 in the Supplement shows the SURPAS model applied to the training data set, and eFigure 3 in the Supplement shows the SURPAS model applied to the test data set. The C statistic improved in the test data set compared with the training data set (0.933; 95% CI, 0.924-0.941 vs 0.922; 95% CI, 0.917-0.928), whereas the Brier score was slightly worse in the test data set compared with the training data set (0.045; 95% CI, 0.042-0.048 vs 0.045; 95% CI, 0.043-0.047). These results showed good internal chronologic validation of the SURPAS model.

We also examined the agreement between actual ICU use and the ICU need composite variable as used by Kongkaewpaisan and colleagues.²⁰ Of 443 patients who were classified as needing ICU care, 371 of 443 (83.8%) actually had ICU use; therefore, the composite outcome variable did identify very sick patients, most of whom ended up in the ICU. However, the composite variable also missed a large percentage of the patients who did use the ICU (2178 of 2549 [85.4%]).

Discussion

This decision analytical model study demonstrated that the 8-variable SURPAS model had excellent discrimination (C statistic, 0.925; 95% CI, 0.921-0.930) and calibration (H-L graphs) in predicting a postoperative ICU stay in a broad surgical population. It also performed well in an internal chronologic validation test, with no important changes in the C statistic, Brier score, or H-L graphs between the model applied to the training and test data sets.

The results of this study are unique because most of the literature on predictors of postoperative ICU stay has been done for specific operations and not for broad surgical populations. In the 6 studies^{27,28,29,30,31,32} using broad surgical populations, 3 studies^27,28,29 had relatively small sample sizes (100-6000 patients), and 1 study³⁰ was done using both preoperative and intraoperative variables. We found only one other research group, from the Singapore General Hospital, that used a large, broad surgical population^31,32 to develop a preoperative prediction model, the Combined Assessment of Risk Encountered in Surgery (CARES) model, for postoperative ICU use. In the CARES study, 4 of 7 predictor variables were identical or similar to the SURPAS variables (patient age, ASA class, emergency surgery, and a measure of surgical complexity). In addition, the CARES study used anemia status, sex, and congestive heart failure, which were not used in SURPAS, whereas SURPAS used functional health status of the patient, surgeon specialty, and inpatient/outpatient setting, which the CARES model did not use. The magnitudes of the C statistic values also differed; the SURPAS model had a C statistic of 0.925 (95% CI, 0.921-0.930) compared with a C statistic of 0.837 (95% CI, 0.808-0.868) in the CARES model. Another factor that makes our study unique is that the ACS’ surgical risk calculator, which is also based on the ACS NSQIP data set and probably the most widely used of the surgical risk calculators, does not have ICU stay as one of its outcome variables.⁴⁰

We believe that the SURPAS prediction model performed at the patient’s preoperative visit could be useful for surgeon and patient decision-making and for efficient planning of the use of this limited and expensive resource. One use of the prediction model might be to identify elective surgical patients at high risk for a postoperative ICU stay who might have their surgery delayed when ICU resources are scarce, eg, as in the COVID-19 pandemic. We recently published a report on using SURPAS to identify the high-risk surgical patient, which may aid in this application.⁴¹ Another potential use might be the estimation of the number of ICU beds needed for operating on a group of surgical patients. In this case, one would need the SURPAS variables for the group as well as the timing after surgery when the ICU beds might be needed and for how long they would be occupied.

The characteristics of the SURPAS model, including the minimal number of predictor variables, which are well known at the preoperative visit and cover a broad surgical population, should make the tool attractive to health care professionals. We have made the logistic regression equations in predictive model markup language code freely available for others to use,⁴² we have provided SURPAS online,⁴³ and we are developing a mobile application of the SURPAS risk prediction tool in order that health care professionals at other institutions can use the tool. In the SURPAS online tool, 7 variables are entered (CPT code of primary operation, patient age, functional health status, ASA class, surgeon specialty, emergency operation [yes, no], and inpatient/outpatient setting), and then risks of all outcomes are calculated. There is a search bar that uses a CPT code search engine when the user types the name of the operation. Once the CPT code is specified, there is a table lookup for the variables work RVU and CPT-specific event rate for each outcome (not used for the ICU use outcome, as previously mentioned). Two additional variables (tier and the need for an ICU bed) are collected for purposes of triaging surgical patients during the COVID-19 pandemic in the UCHealth System but are not used in calculating the SURPAS risks.

We also found that, although most of the patients (371 of 443 [83.8%]) who experienced the cluster of ACS NSQIP outcomes to label them as positive for ICU need did have a postoperative ICU stay, this surrogate outcome also missed 85% (2178 of 2549) of the patients who did use the ICU postoperatively. Reasons for this poor agreement between ICU need and actual ICU use might include that the ICU need variable does not include several important indications for admission to intensive care, including hourly neurologic checks, hourly neurovascular checks, use of certain medication infusions, or other nursing interventions required hourly. Additionally, postoperative ICU admission often is a complex decision that surgeons make that varies depending on factors other than explicit admission indications, like surgeon experience, practice setting, operative course, and comfort. Thus, this outcome being used in the literature²⁰ is not a good substitute for actual ICU use.

Strengths and Limitations

The inclusion of data from several hospitals representing different hospital types strengthens the applicability of our findings to other hospital systems. A major limitation of this study was that it was from a single health care system, although the UCHealth ACS NSQIP and electronic health record data did come from 5 separate hospitals, 1 being a major tertiary university hospital and 4 smaller community hospitals in the same state. Based on local practice patterns, we suspect that patients who were transferred to ICU immediately after surgery were usually provided a bed based on surgeon request and less frequently based on intraoperative physiology changes necessitating increased monitoring. However, for the patients transferred to an ICU on POD 1 or greater, this would have been due to some change in patient status that required increased monitoring or support. Local variation in practice patterns at each of the participating hospitals dictates this change in level of care, and we cannot control for this subjective variability. It would be advantageous to repeat this study in the future using multiple health care systems from different states in the US. There are likely institutional variations in ICU use, which are based on surgeon and anesthetist experience and preferences, and do not necessarily conform to strict indicators for ICU need.

Conclusions

This decision analytical model study found that the 8-variable SURPAS risk prediction model performed well at predicting postoperative ICU use in a broad surgical population across a diverse hospital system. This tool may be of use to institutions where ICU bed resources are limited, especially during circumstances such as the COVID-19 pandemic.

Supplement.

eFigure 1. Frequency Distribution of Days From Surgery to ICU Admission (n=2545)

eFigure 2. Hosmer-Lemeshow Graph of Observed and Expected ICU Use by Deciles of Risk Using the 7 Variable SURPAS Model Applied to the Developmental Data Set (n=24 250)

eFigure 3. Hosmer-Lemeshow Graph of Observed and Expected ICU Use by Deciles of Risk Using the 7 Variable SURPAS Model Applied to the Test Data Set (n=10 318)

Click here for additional data file.^{(237.7KB, pdf)}

References

1.Wunsch H, Wagner J, Herlim M, Chong DH, Kramer AA, Halpern SD. ICU occupancy and mechanical ventilator use in the United States. Crit Care Med. 2013;41(12):2712-2719. doi: 10.1097/CCM.0b013e318298a139 [DOI] [PMC free article] [PubMed] [Google Scholar]
2.Halpern NA, Pastores SM. Critical care medicine beds, use, occupancy and costs in the United States: a methodological review. Crit Care Med. 2015;43(11):2452-2459. doi: 10.1097/CCM.0000000000001227 [DOI] [PMC free article] [PubMed] [Google Scholar]
3.Moghadas SM, Shoukat A, Fitzpatrick MC, et al. Projecting hospital utilization during the COVID-19 outbreaks in the United States. Proc Natl Acad Sci U S A. 2020;117(16):9122-9126. doi: 10.1073/pnas.2004064117 [DOI] [PMC free article] [PubMed] [Google Scholar]
4.Dasta JF, McLaughlin TP, Mody SH, Piech CT. Daily cost of an intensive care unit day: the contribution of mechanical ventilation. Crit Care Med. 2005;33(6):1266-1271. doi: 10.1097/01.CCM.0000164543.14619.00 [DOI] [PubMed] [Google Scholar]
5.International Surgical Outcomes Study group . Global patient outcomes after elective surgery: prospective cohort study in 27 low-, middle- and high-income countries. Br J Anaesth. 2016;117(5):601-609. doi: 10.1093/bja/aew316 [DOI] [PMC free article] [PubMed] [Google Scholar]
6.Steiner CA, Karaca Z, Moore BJ, Imshaug MC, Pickens G. Surgeries in hospital-based ambulatory surgery and hospital inpatient settings, 2014. Accessed March 29, 2021. https://www.hcup-us.ahrq.gov/reports/statbriefs/sb223-Ambulatory-Inpatient-Surgeries-2014.jsp [PubMed]
7.Brunelli A, Ferguson MK, Rocco G, et al. A scoring system predicting the risk for intensive care unit admission for complications after major lung resection: a multicenter analysis. Ann Thorac Surg. 2008;86(1):213-218. doi: 10.1016/j.athoracsur.2008.03.063 [DOI] [PubMed] [Google Scholar]
8.Kim SH, Na S, Park SY, et al. Perioperative factors for predicting the need for postoperative intensive care after major lung resection. J Clin Med. 2019;8(5):744. doi: 10.3390/jcm8050744 [DOI] [PMC free article] [PubMed] [Google Scholar]
9.Pinheiro L, Santoro IL, Faresin SM. Who needs to be allocated in ICU after thoracic surgery? an observational study. Can Respir J. 2016;2016:3981506. doi: 10.1155/2016/3981506 [DOI] [PMC free article] [PubMed] [Google Scholar]
10.Okiror L, Patel N, Kho P, et al. Predicting risk of intensive care unit admission after resection for non–small cell lung cancer: a validation study. Interact Cardiovasc Thorac Surg. 2012;14(1):31-33. doi: 10.1093/icvts/ivr060 [DOI] [PMC free article] [PubMed] [Google Scholar]
11.Courtney PM, Melnic CM, Gutsche J, Hume EL, Lee GC. Which patients need critical care intervention after total joint arthroplasty? a prospective study of factors associated with the need for intensive care following surgery. Bone Joint J. 2015;97-B(11):1512-1518. doi: 10.1302/0301-620X.97B11.35813 [DOI] [PubMed] [Google Scholar]
12.Klausing A, Martini M, Wimmer MD, Gravius S, Wirtz DC, Randau TM. Postoperative medical complications and intermediate care unit/intensive care unit admission in joint replacement surgery: a prospective risk model. J Arthroplasty. 2019;34(4):717-722. doi: 10.1016/j.arth.2018.12.034 [DOI] [PubMed] [Google Scholar]
13.Kamath AF, McAuliffe CL, Gutsche JT, et al. Intensive care monitoring after total joint replacement. Bone Joint J. 2013;95-B(11)(suppl A):74-76. doi: 10.1302/0301-620X.95B11.33093 [DOI] [PubMed] [Google Scholar]
14.Franko LR, Hollon T, Linzey J, et al. Clinical factors associated with ICU-specific care following supratentorial brain tumor resection and validation of a risk prediction score. Crit Care Med. 2018;46(8):1302-1308. doi: 10.1097/CCM.0000000000003207 [DOI] [PubMed] [Google Scholar]
15.Hanak BW, Walcott BP, Nahed BV, et al. Postoperative intensive care unit requirements after elective craniotomy. World Neurosurg. 2014;81(1):165-172. doi: 10.1016/j.wneu.2012.11.068 [DOI] [PMC free article] [PubMed] [Google Scholar]
16.Peng L, Luo Z, Liang L, et al. Comparison of the performance of 24 early warning scores with the updated National Early Warning Score (NEWS2) for predicting unplanned intensive care unit (ICU) admission in postoperative brain tumor patients: a retrospective study at a single center. Med Sci Monit. 2021;27:e929168. doi: 10.12659/MSM.929168 [DOI] [PMC free article] [PubMed] [Google Scholar]
17.Garfinkle R, Abou-Khalil M, Salama E, et al. Development and validation of a clinical risk score for intensive care resource utilization after colon cancer surgery: a practical guide to the selection of patients during COVID-19. J Gastrointest Surg. 2021;25(1):252-259. doi: 10.1007/s11605-020-04665-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
18.Hicks CW, Alshaikh HN, Zarkowsky D, Bostock IC, Nejim B, Malas MB. Intensive care unit admission after endovascular aortic aneurysm repair is primarily determined by hospital factors, adds significant cost, and is often unnecessary. J Vasc Surg. 2018;67(4):1091-1101.e4. doi: 10.1016/j.jvs.2017.07.139 [DOI] [PubMed] [Google Scholar]
19.Morgan DJR, Ho KM, Armstrong J, Baker S. Incidence and risk factors for intensive care unit admission after bariatric surgery: a multicentre population-based cohort study. Br J Anaesth. 2015;115(6):873-882. doi: 10.1093/bja/aev364 [DOI] [PubMed] [Google Scholar]
20.Kongkaewpaisan N, Lee JM, Eid AI, et al. Can the Emergency Surgery Score (ESS) be used as a triage tool predicting the postoperative need for an ICU admission? Am J Surg. 2019;217(1):24-28. doi: 10.1016/j.amjsurg.2018.08.002 [DOI] [PubMed] [Google Scholar]
21.Naar L, El Hechi M, Kokoroskos N, et al. Can the Emergency Surgery Score (ESS) predict outcomes in emergency general surgery patients with missing data elements? a nationwide analysis. Am J Surg. 2020;220(6):1613-1622. doi: 10.1016/j.amjsurg.2020.02.034 [DOI] [PubMed] [Google Scholar]
22.Ross MS, Burriss ME, Winger DG, Edwards RP, Courtney-Brooks M, Boisen MM. Unplanned postoperative intensive care unit admission for ovarian cancer cytoreduction is associated with significant decrease in overall survival. Gynecol Oncol. 2018;150(2):306-310. doi: 10.1016/j.ygyno.2018.06.008 [DOI] [PubMed] [Google Scholar]
23.Lin YC, Chen YC, Yang CH, Su NY. Surgical Apgar score is strongly associated with postoperative ICU admission. Sci Rep. 2021;11(1):115. doi: 10.1038/s41598-020-80393-z [DOI] [PMC free article] [PubMed] [Google Scholar]
24.Sobol JB, Gershengorn HB, Wunsch H, Li G. The surgical Apgar score is strongly associated with intensive care unit admission after high-risk intraabdominal surgery. Anesth Analg. 2013;117(2):438-446. doi: 10.1213/ANE.0b013e31829180b7 [DOI] [PMC free article] [PubMed] [Google Scholar]
25.Ruskin R, Urban RR, Sherman AE, et al. Predictors of intensive care unit utilization in gynecologic oncology surgery. Int J Gynecol Cancer. 2011;21(8):1336-1342. doi: 10.1097/IGC.0b013e31822d0ed0 [DOI] [PubMed] [Google Scholar]
26.Bentrem DJ, Yeh JJ, Brennan MF, et al. Predictors of intensive care unit admission and related outcome for patients after pancreaticoduodenectomy. J Gastrointest Surg. 2005;9(9):1307-1312. doi: 10.1016/j.gassur.2005.09.010 [DOI] [PubMed] [Google Scholar]
27.Iden T, Caliebe A, Renner J, et al. The Extended Postoperative Care—score (EXPO-score)—an objective tool for early identification of indication for extended postoperative care. J Clin Med. 2019;8(10):1666. doi: 10.3390/jcm8101666 [DOI] [PMC free article] [PubMed] [Google Scholar]
28.Silva JM Jr, Rocha HMC, Katayama HT, et al. SAPS 3 score as a predictive factor for postoperative referral to intensive care unit. Ann Intensive Care. 2016;6(1):42. doi: 10.1186/s13613-016-0129-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
29.Weissman C, Kleln N. Preoperative evaluation using therapeutic intensity scoring. Eur J Anaesthesiol. 2011;28(1):20-28. doi: 10.1097/EJA.0b013e32833ddf6e [DOI] [PubMed] [Google Scholar]
30.Wanderer JP, Anderson-Dam J, Levine W, Bittner EA. Development and validation of an intraoperative predictive model for unplanned postoperative intensive care. Anesthesiology. 2013;119(3):516-524. doi: 10.1097/ALN.0b013e31829ce8fd [DOI] [PubMed] [Google Scholar]
31.Chan DXH, Sim YE, Chan YH, Poopalalingam R, Abdullah HR. Development of the Combined Assessment of Risk Encountered in Surgery (CARES) surgical risk calculator for prediction of postsurgical mortality and need for intensive care unit admission risk: a single-center retrospective study. BMJ Open. 2018;8(3):e019427. doi: 10.1136/bmjopen-2017-019427 [DOI] [PMC free article] [PubMed] [Google Scholar]
32.Chiew CJ, Liu N, Wong TH, Sim YE, Abdullah HR. Utilizing machine learning methods for preoperative prediction of postsurgical mortality and intensive care unit admission. Ann Surg. 2020;272(6):1133-1139. doi: 10.1097/SLA.0000000000003297 [DOI] [PMC free article] [PubMed] [Google Scholar]
33.Meguid RA, Bronsert MR, Juarez-Colunga E, Hammermeister KE, Henderson WG. Surgical Risk Preoperative Assessment System (SURPAS) I. parsimonious, clinically meaningful groups of postoperative complications by factor analysis. Ann Surg. 2016;263(6):1042-1048. doi: 10.1097/SLA.0000000000001669 [DOI] [PubMed] [Google Scholar]
34.Meguid RA, Bronsert MR, Juarez-Colunga E, Hammermeister KE, Henderson WG; Parsimonious Risk Models for Postoperative Adverse Outcomes Addressing Need for Laboratory Variables and Surgeon Specialty-specific Models . Surgical Risk Preoperative Assessment System (SURPAS) II. parsimonious risk models for postoperative adverse outcomes addressing need for laboratory variables and surgeon specialty-specific models. Ann Surg. 2016;264(1):10-22. doi: 10.1097/SLA.0000000000001677 [DOI] [PubMed] [Google Scholar]
35.Meguid RA, Bronsert MR, Juarez-Colunga E, Hammermeister KE, Henderson WG. Surgical Risk Preoperative Assessment System (SURPAS) III. accurate preoperative prediction of 8 adverse outcomes using 8 predictor variables. Ann Surg. 2016;264(1):23-31. doi: 10.1097/SLA.0000000000001678 [DOI] [PubMed] [Google Scholar]
36.Henderson WG, Bronsert MR, Hammermeister KE, Lambert-Kerzner A, Meguid RA. Refining the predictive variables in the “Surgical Risk Preoperative Assessment System” (SURPAS): a descriptive analysis. Patient Saf Surg. 2019;13:28. doi: 10.1186/s13037-019-0208-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
37.Hammermeister KE, Henderson WG, Bronsert MR, Juarez-Colunga E, Meguid RA. Bringing quantitative risk assessment closer to the patient and surgeon: a novel approach to improve outcomes. Ann Surg. 2016;263(6):1039-1041. doi: 10.1097/SLA.0000000000001668 [DOI] [PubMed] [Google Scholar]
38.Gibula DR, Singh AB, Bronsert MR, et al. Accurate preoperative prediction of unplanned 30-day postoperative readmission using 8 predictor variables. Surgery. 2019;166(5):812-819. doi: 10.1016/j.surg.2019.05.022 [DOI] [PubMed] [Google Scholar]
39.Singh AB, Bronsert MR, Henderson WG, Lambert-Kerzner A, Hammermeister KE, Meguid RA. Accurate preoperative prediction of discharge destination using 8 predictor variables: a NSQIP analysis. J Am Coll Surg. 2020;230(1):64-75.e2. doi: 10.1016/j.jamcollsurg.2019.09.018 [DOI] [PubMed] [Google Scholar]
40.American College of Surgeons. Welcome to the ACS NSQIP surgical risk calculator. Accessed April 3, 2021. https://riskcalculator.facs.org/RiskCalculator/
41.Dyas AR, Bronsert MR, Meguid RA, et al. Using the surgical risk preoperative assessment system to define the “high-risk” surgical patient. J Surg Res. 2022;270:394-404. doi: 10.1016/j.jss.2021.08.045 [DOI] [PubMed] [Google Scholar]
42.Colborn K. Code repository for SURPAS risk model katiecolborn/SURPAS. GitHub. Published January 14, 2020. Accessed April 3, 2021. https://github.com/katiecolborn/SURPAS
43.Surgical Risk Preoperative Assessment System . Home page. Accessed March 29, 2021. https://surpas.agilemd.com

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplement.

eFigure 1. Frequency Distribution of Days From Surgery to ICU Admission (n=2545)

eFigure 2. Hosmer-Lemeshow Graph of Observed and Expected ICU Use by Deciles of Risk Using the 7 Variable SURPAS Model Applied to the Developmental Data Set (n=24 250)

eFigure 3. Hosmer-Lemeshow Graph of Observed and Expected ICU Use by Deciles of Risk Using the 7 Variable SURPAS Model Applied to the Test Data Set (n=10 318)

Click here for additional data file.^{(237.7KB, pdf)}

[soi210115r1] 1.Wunsch H, Wagner J, Herlim M, Chong DH, Kramer AA, Halpern SD. ICU occupancy and mechanical ventilator use in the United States. Crit Care Med. 2013;41(12):2712-2719. doi: 10.1097/CCM.0b013e318298a139 [DOI] [PMC free article] [PubMed] [Google Scholar]

[soi210115r2] 2.Halpern NA, Pastores SM. Critical care medicine beds, use, occupancy and costs in the United States: a methodological review. Crit Care Med. 2015;43(11):2452-2459. doi: 10.1097/CCM.0000000000001227 [DOI] [PMC free article] [PubMed] [Google Scholar]

[soi210115r3] 3.Moghadas SM, Shoukat A, Fitzpatrick MC, et al. Projecting hospital utilization during the COVID-19 outbreaks in the United States. Proc Natl Acad Sci U S A. 2020;117(16):9122-9126. doi: 10.1073/pnas.2004064117 [DOI] [PMC free article] [PubMed] [Google Scholar]

[soi210115r4] 4.Dasta JF, McLaughlin TP, Mody SH, Piech CT. Daily cost of an intensive care unit day: the contribution of mechanical ventilation. Crit Care Med. 2005;33(6):1266-1271. doi: 10.1097/01.CCM.0000164543.14619.00 [DOI] [PubMed] [Google Scholar]

[soi210115r5] 5.International Surgical Outcomes Study group . Global patient outcomes after elective surgery: prospective cohort study in 27 low-, middle- and high-income countries. Br J Anaesth. 2016;117(5):601-609. doi: 10.1093/bja/aew316 [DOI] [PMC free article] [PubMed] [Google Scholar]

[soi210115r6] 6.Steiner CA, Karaca Z, Moore BJ, Imshaug MC, Pickens G. Surgeries in hospital-based ambulatory surgery and hospital inpatient settings, 2014. Accessed March 29, 2021. https://www.hcup-us.ahrq.gov/reports/statbriefs/sb223-Ambulatory-Inpatient-Surgeries-2014.jsp [PubMed]

[soi210115r7] 7.Brunelli A, Ferguson MK, Rocco G, et al. A scoring system predicting the risk for intensive care unit admission for complications after major lung resection: a multicenter analysis. Ann Thorac Surg. 2008;86(1):213-218. doi: 10.1016/j.athoracsur.2008.03.063 [DOI] [PubMed] [Google Scholar]

[soi210115r8] 8.Kim SH, Na S, Park SY, et al. Perioperative factors for predicting the need for postoperative intensive care after major lung resection. J Clin Med. 2019;8(5):744. doi: 10.3390/jcm8050744 [DOI] [PMC free article] [PubMed] [Google Scholar]

[soi210115r9] 9.Pinheiro L, Santoro IL, Faresin SM. Who needs to be allocated in ICU after thoracic surgery? an observational study. Can Respir J. 2016;2016:3981506. doi: 10.1155/2016/3981506 [DOI] [PMC free article] [PubMed] [Google Scholar]

[soi210115r10] 10.Okiror L, Patel N, Kho P, et al. Predicting risk of intensive care unit admission after resection for non–small cell lung cancer: a validation study. Interact Cardiovasc Thorac Surg. 2012;14(1):31-33. doi: 10.1093/icvts/ivr060 [DOI] [PMC free article] [PubMed] [Google Scholar]

[soi210115r11] 11.Courtney PM, Melnic CM, Gutsche J, Hume EL, Lee GC. Which patients need critical care intervention after total joint arthroplasty? a prospective study of factors associated with the need for intensive care following surgery. Bone Joint J. 2015;97-B(11):1512-1518. doi: 10.1302/0301-620X.97B11.35813 [DOI] [PubMed] [Google Scholar]

[soi210115r12] 12.Klausing A, Martini M, Wimmer MD, Gravius S, Wirtz DC, Randau TM. Postoperative medical complications and intermediate care unit/intensive care unit admission in joint replacement surgery: a prospective risk model. J Arthroplasty. 2019;34(4):717-722. doi: 10.1016/j.arth.2018.12.034 [DOI] [PubMed] [Google Scholar]

[soi210115r13] 13.Kamath AF, McAuliffe CL, Gutsche JT, et al. Intensive care monitoring after total joint replacement. Bone Joint J. 2013;95-B(11)(suppl A):74-76. doi: 10.1302/0301-620X.95B11.33093 [DOI] [PubMed] [Google Scholar]

[soi210115r14] 14.Franko LR, Hollon T, Linzey J, et al. Clinical factors associated with ICU-specific care following supratentorial brain tumor resection and validation of a risk prediction score. Crit Care Med. 2018;46(8):1302-1308. doi: 10.1097/CCM.0000000000003207 [DOI] [PubMed] [Google Scholar]

[soi210115r15] 15.Hanak BW, Walcott BP, Nahed BV, et al. Postoperative intensive care unit requirements after elective craniotomy. World Neurosurg. 2014;81(1):165-172. doi: 10.1016/j.wneu.2012.11.068 [DOI] [PMC free article] [PubMed] [Google Scholar]

[soi210115r16] 16.Peng L, Luo Z, Liang L, et al. Comparison of the performance of 24 early warning scores with the updated National Early Warning Score (NEWS2) for predicting unplanned intensive care unit (ICU) admission in postoperative brain tumor patients: a retrospective study at a single center. Med Sci Monit. 2021;27:e929168. doi: 10.12659/MSM.929168 [DOI] [PMC free article] [PubMed] [Google Scholar]

[soi210115r17] 17.Garfinkle R, Abou-Khalil M, Salama E, et al. Development and validation of a clinical risk score for intensive care resource utilization after colon cancer surgery: a practical guide to the selection of patients during COVID-19. J Gastrointest Surg. 2021;25(1):252-259. doi: 10.1007/s11605-020-04665-9 [DOI] [PMC free article] [PubMed] [Google Scholar]

[soi210115r18] 18.Hicks CW, Alshaikh HN, Zarkowsky D, Bostock IC, Nejim B, Malas MB. Intensive care unit admission after endovascular aortic aneurysm repair is primarily determined by hospital factors, adds significant cost, and is often unnecessary. J Vasc Surg. 2018;67(4):1091-1101.e4. doi: 10.1016/j.jvs.2017.07.139 [DOI] [PubMed] [Google Scholar]

[soi210115r19] 19.Morgan DJR, Ho KM, Armstrong J, Baker S. Incidence and risk factors for intensive care unit admission after bariatric surgery: a multicentre population-based cohort study. Br J Anaesth. 2015;115(6):873-882. doi: 10.1093/bja/aev364 [DOI] [PubMed] [Google Scholar]

[soi210115r20] 20.Kongkaewpaisan N, Lee JM, Eid AI, et al. Can the Emergency Surgery Score (ESS) be used as a triage tool predicting the postoperative need for an ICU admission? Am J Surg. 2019;217(1):24-28. doi: 10.1016/j.amjsurg.2018.08.002 [DOI] [PubMed] [Google Scholar]

[soi210115r21] 21.Naar L, El Hechi M, Kokoroskos N, et al. Can the Emergency Surgery Score (ESS) predict outcomes in emergency general surgery patients with missing data elements? a nationwide analysis. Am J Surg. 2020;220(6):1613-1622. doi: 10.1016/j.amjsurg.2020.02.034 [DOI] [PubMed] [Google Scholar]

[soi210115r22] 22.Ross MS, Burriss ME, Winger DG, Edwards RP, Courtney-Brooks M, Boisen MM. Unplanned postoperative intensive care unit admission for ovarian cancer cytoreduction is associated with significant decrease in overall survival. Gynecol Oncol. 2018;150(2):306-310. doi: 10.1016/j.ygyno.2018.06.008 [DOI] [PubMed] [Google Scholar]

[soi210115r23] 23.Lin YC, Chen YC, Yang CH, Su NY. Surgical Apgar score is strongly associated with postoperative ICU admission. Sci Rep. 2021;11(1):115. doi: 10.1038/s41598-020-80393-z [DOI] [PMC free article] [PubMed] [Google Scholar]

[soi210115r24] 24.Sobol JB, Gershengorn HB, Wunsch H, Li G. The surgical Apgar score is strongly associated with intensive care unit admission after high-risk intraabdominal surgery. Anesth Analg. 2013;117(2):438-446. doi: 10.1213/ANE.0b013e31829180b7 [DOI] [PMC free article] [PubMed] [Google Scholar]

[soi210115r25] 25.Ruskin R, Urban RR, Sherman AE, et al. Predictors of intensive care unit utilization in gynecologic oncology surgery. Int J Gynecol Cancer. 2011;21(8):1336-1342. doi: 10.1097/IGC.0b013e31822d0ed0 [DOI] [PubMed] [Google Scholar]

[soi210115r26] 26.Bentrem DJ, Yeh JJ, Brennan MF, et al. Predictors of intensive care unit admission and related outcome for patients after pancreaticoduodenectomy. J Gastrointest Surg. 2005;9(9):1307-1312. doi: 10.1016/j.gassur.2005.09.010 [DOI] [PubMed] [Google Scholar]

[soi210115r27] 27.Iden T, Caliebe A, Renner J, et al. The Extended Postoperative Care—score (EXPO-score)—an objective tool for early identification of indication for extended postoperative care. J Clin Med. 2019;8(10):1666. doi: 10.3390/jcm8101666 [DOI] [PMC free article] [PubMed] [Google Scholar]

[soi210115r28] 28.Silva JM Jr, Rocha HMC, Katayama HT, et al. SAPS 3 score as a predictive factor for postoperative referral to intensive care unit. Ann Intensive Care. 2016;6(1):42. doi: 10.1186/s13613-016-0129-5 [DOI] [PMC free article] [PubMed] [Google Scholar]

[soi210115r29] 29.Weissman C, Kleln N. Preoperative evaluation using therapeutic intensity scoring. Eur J Anaesthesiol. 2011;28(1):20-28. doi: 10.1097/EJA.0b013e32833ddf6e [DOI] [PubMed] [Google Scholar]

[soi210115r30] 30.Wanderer JP, Anderson-Dam J, Levine W, Bittner EA. Development and validation of an intraoperative predictive model for unplanned postoperative intensive care. Anesthesiology. 2013;119(3):516-524. doi: 10.1097/ALN.0b013e31829ce8fd [DOI] [PubMed] [Google Scholar]

[soi210115r31] 31.Chan DXH, Sim YE, Chan YH, Poopalalingam R, Abdullah HR. Development of the Combined Assessment of Risk Encountered in Surgery (CARES) surgical risk calculator for prediction of postsurgical mortality and need for intensive care unit admission risk: a single-center retrospective study. BMJ Open. 2018;8(3):e019427. doi: 10.1136/bmjopen-2017-019427 [DOI] [PMC free article] [PubMed] [Google Scholar]

[soi210115r32] 32.Chiew CJ, Liu N, Wong TH, Sim YE, Abdullah HR. Utilizing machine learning methods for preoperative prediction of postsurgical mortality and intensive care unit admission. Ann Surg. 2020;272(6):1133-1139. doi: 10.1097/SLA.0000000000003297 [DOI] [PMC free article] [PubMed] [Google Scholar]

[soi210115r33] 33.Meguid RA, Bronsert MR, Juarez-Colunga E, Hammermeister KE, Henderson WG. Surgical Risk Preoperative Assessment System (SURPAS) I. parsimonious, clinically meaningful groups of postoperative complications by factor analysis. Ann Surg. 2016;263(6):1042-1048. doi: 10.1097/SLA.0000000000001669 [DOI] [PubMed] [Google Scholar]

[soi210115r34] 34.Meguid RA, Bronsert MR, Juarez-Colunga E, Hammermeister KE, Henderson WG; Parsimonious Risk Models for Postoperative Adverse Outcomes Addressing Need for Laboratory Variables and Surgeon Specialty-specific Models . Surgical Risk Preoperative Assessment System (SURPAS) II. parsimonious risk models for postoperative adverse outcomes addressing need for laboratory variables and surgeon specialty-specific models. Ann Surg. 2016;264(1):10-22. doi: 10.1097/SLA.0000000000001677 [DOI] [PubMed] [Google Scholar]

[soi210115r35] 35.Meguid RA, Bronsert MR, Juarez-Colunga E, Hammermeister KE, Henderson WG. Surgical Risk Preoperative Assessment System (SURPAS) III. accurate preoperative prediction of 8 adverse outcomes using 8 predictor variables. Ann Surg. 2016;264(1):23-31. doi: 10.1097/SLA.0000000000001678 [DOI] [PubMed] [Google Scholar]

[soi210115r36] 36.Henderson WG, Bronsert MR, Hammermeister KE, Lambert-Kerzner A, Meguid RA. Refining the predictive variables in the “Surgical Risk Preoperative Assessment System” (SURPAS): a descriptive analysis. Patient Saf Surg. 2019;13:28. doi: 10.1186/s13037-019-0208-2 [DOI] [PMC free article] [PubMed] [Google Scholar]

[soi210115r37] 37.Hammermeister KE, Henderson WG, Bronsert MR, Juarez-Colunga E, Meguid RA. Bringing quantitative risk assessment closer to the patient and surgeon: a novel approach to improve outcomes. Ann Surg. 2016;263(6):1039-1041. doi: 10.1097/SLA.0000000000001668 [DOI] [PubMed] [Google Scholar]

[soi210115r38] 38.Gibula DR, Singh AB, Bronsert MR, et al. Accurate preoperative prediction of unplanned 30-day postoperative readmission using 8 predictor variables. Surgery. 2019;166(5):812-819. doi: 10.1016/j.surg.2019.05.022 [DOI] [PubMed] [Google Scholar]

[soi210115r39] 39.Singh AB, Bronsert MR, Henderson WG, Lambert-Kerzner A, Hammermeister KE, Meguid RA. Accurate preoperative prediction of discharge destination using 8 predictor variables: a NSQIP analysis. J Am Coll Surg. 2020;230(1):64-75.e2. doi: 10.1016/j.jamcollsurg.2019.09.018 [DOI] [PubMed] [Google Scholar]

[soi210115r40] 40.American College of Surgeons. Welcome to the ACS NSQIP surgical risk calculator. Accessed April 3, 2021. https://riskcalculator.facs.org/RiskCalculator/

[soi210115r41] 41.Dyas AR, Bronsert MR, Meguid RA, et al. Using the surgical risk preoperative assessment system to define the “high-risk” surgical patient. J Surg Res. 2022;270:394-404. doi: 10.1016/j.jss.2021.08.045 [DOI] [PubMed] [Google Scholar]

[soi210115r42] 42.Colborn K. Code repository for SURPAS risk model katiecolborn/SURPAS. GitHub. Published January 14, 2020. Accessed April 3, 2021. https://github.com/katiecolborn/SURPAS

[soi210115r43] 43.Surgical Risk Preoperative Assessment System . Home page. Accessed March 29, 2021. https://surpas.agilemd.com

PERMALINK

Development and Validation of a Multivariable Prediction Model for Postoperative Intensive Care Unit Stay in a Broad Surgical Population

Paul D Rozeboom, MD

William G Henderson, PhD, MPH

Adam R Dyas, MD

Michael R Bronsert, PhD, MS

Kathryn L Colborn, PhD, MSPH

Anne Lambert-Kerzner, PhD, MSPH

Karl E Hammermeister, MD

Robert C McIntyre Jr, MD

Robert A Meguid, MD, MPH

Key Points

Question

Findings

Meaning

Abstract

Importance

Objective

Design, Setting, and Participants

Exposure

Main Outcomes and Measures

Results

Conclusions and Relevance

Introduction

Methods

Study Design and Patient Selection

Exposures

Primary Outcome

Statistical Analysis

Results

Patient Characteristics

Table 1. Association Between Preoperative Patient and Operative Characteristics and Postoperative ICU Use.

Outcomes

Table 2. Stepwise Multiple Logistic Regression Analysis of Postoperative Intensive Care Unit Stay (Dependent Variable) and Preoperative Patient and Operative Characteristics (Independent Variable) Applied to Entire Data Set of 34 568 Surgical Patientsa.

Figure 1. Hosmer-Lemeshow Graph of Observed and Expected Intensive Care Unit Use by Deciles of Risk Using the 12-Variable Model Applied to the Entire Data Set of 34 568 Surgical Patients.

Table 3. Stepwise Multiple Logistic Regression Analysis of Postoperative Intensive Care Unit Stay (Dependent Variable) and the 7 Surgical Risk Preoperative Assessment System Predictor Variables Applied to Entire Data Set of 34 568 Surgical Patients.

Figure 2. Hosmer-Lemeshow Graph of Observed and Expected Intensive Care Unit Use by Deciles of Risk Using the 7-Variable Surgical Risk Preoperative Assessment System Model Applied to the Entire Data Set of 34 568 Surgical Patients.

Discussion

Strengths and Limitations

Conclusions

References

Associated Data

Supplementary Materials

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases

Table 2. Stepwise Multiple Logistic Regression Analysis of Postoperative Intensive Care Unit Stay (Dependent Variable) and Preoperative Patient and Operative Characteristics (Independent Variable) Applied to Entire Data Set of 34 568 Surgical Patients^a.