This cross-sectional study investigates the association of undertriage to hospital wards after surgical procedures with mortality, morbidity, and resource use.
Key Points
Question
Is postoperative undertriage associated with increased mortality, morbidity, and resource use?
Findings
In this cross-sectional study of 14 890 postoperative admissions, undertriage to hospital wards was associated with increased mortality and morbidity compared with admissions that had similar risk profiles and were triaged to intensive care units. Postoperative undertriage was identifiable using automated preoperative and intraoperative data as features for real-time machine-learning models.
Meaning
These findings suggest that there is a rationale and framework for clinical decision support platforms to augment postoperative triage decisions.
Abstract
Importance
Undertriaging patients who are at increased risk for postoperative complications after surgical procedures to low-acuity hospital wards (ie, floors) rather than highly vigilant intensive care units (ICUs) may be associated with risk of unrecognized decompensation and worse patient outcomes, but evidence for these associations is lacking.
Objective
To test the hypothesis that postoperative undertriage is associated with increased mortality and morbidity compared with risk-matched ICU admission.
Design, Setting, and Participants
This longitudinal cross-sectional study was conducted using data from the University of Florida Integrated Data Repository on admissions to a university hospital. Included patients were individuals aged 18 years or older who were admitted after a surgical procedure from June 1, 2014, to August 20, 2020. Data were analyzed from April through August 2021.
Exposures
Ward admissions were considered undertriaged if their estimated risk for hospital mortality or prolonged ICU stay (ie, ≥48 hours) was in the top quartile among all inpatient surgical procedures according to a validated machine-learning model using preoperative and intraoperative electronic health record features available at surgical procedure end time. A nearest neighbors algorithm was used to identify a risk-matched control group of ICU admissions.
Main Outcomes and Measures
The primary outcomes of hospital mortality and morbidity were compared among appropriately triaged ward admissions, undertriaged wards admissions, and a risk-matched control group of ICU admissions.
Results
Among 12 348 postoperative ward admissions, 11 042 admissions (89.4%) were appropriately triaged (5927 [53.7%] women; median [IQR] age, 59 [44-70] years) and 1306 admissions (10.6%) were undertriaged and matched with a control group of 2452 ICU admissions. The undertriaged group, compared with the control group, had increased median [IQR] age (64 [54-74] years vs 62 [50-73] years; P = .001) and increased proportions of women (649 [49.7%] women vs 1080 [44.0%] women; P < .001) and admitted patients with do not resuscitate orders before first surgical procedure (53 admissions [4.1%] vs 27 admissions [1.1%]); P < .001); 207 admissions that were undertriaged (15.8%) had subsequent ICU admission. In the validation cohort, hospital mortality and prolonged ICU stay estimations had areas under the receiver operating characteristic curve of 0.92 (95% CI, 0.91-0.93) and 0.92 (95% CI, 0.92-0.92), respectively. The undertriaged group, compared with the control group, had similar incidence of prolonged mechanical ventilation (32 admissions [2.5%] vs 53 admissions [2.2%]; P = .60), decreased median (IQR) total costs for admission ($26 900 [$18 400-$42 300] vs $32 700 [$22 700-$48 500]; P < .001), increased median (IQR) hospital length of stay (8.1 [5.1-13.6] days vs 6.0 [3.3-9.3] days, P < .001), and increased incidence of hospital mortality (19 admissions [1.5%] vs 17 admissions [0.7%]; P = .04), discharge to hospice (23 admissions [1.8%] vs 14 admissions [0.6%]; P < .001), unplanned intubation (45 admissions [3.4%] vs 49 admissions [2.0%]; P = .01), and acute kidney injury (341 admissions [26.1%] vs 477 admissions [19.5%]; P < .001).
Conclusions and Relevance
This study found that admitted patients at increased risk for postoperative complications who were undertriaged to hospital wards had increased mortality and morbidity compared with a risk-matched control group of admissions to ICUs. Postoperative undertriage was identifiable using automated preoperative and intraoperative data as features in real-time machine-learning models.
Introduction
Approximately 15 million major inpatient surgical procedures are performed in the United States annually.1 Immediately after surgical procedures, surgeons must determine whether patients can be triaged safely to hospital wards (ie, floors) with high patient-to-clinician ratios and infrequent assessments. When patients with high acuity are undertriaged to hospital wards, postoperative complications may progress to critical illness and cardiac arrest between patient assessments that are typically spaced every 4 hours.2,3,4,5,6 Approximately 41% of hospital arrests occur on wards, at 0.1 per 1000 bed-days, with 80% to 85% hospital mortality.4,6,7 The underlying disease process is often reversible, suggesting opportunities for prevention with appropriate surveillance and early intervention.8
Despite the intuitive importance of undertriage, there is a paucity of evidence regarding associations between postoperative undertriage and patient outcomes. Among hospital admissions involving surgical procedures in the United Kingdom, 85% of all patients undergoing high-risk procedures (ie, those with mortality rates ≥5%) were admitted to hospital wards; patients with subsequent intensive care unit (ICU) admission had a 37% incidence of mortality.2 In a similar analysis, 74% of postoperative deaths occurred outside the ICU.3 These findings are compelling, but they use procedure-associated mortality alone to identify patients at increased risk. Granular, patient-level acuity assessments may further elucidate the sequelae associated with postoperative undertriage.
Using a longitudinal cohort of postoperative admissions, we tested the hypothesis that postoperative undertriage would be associated with increased mortality and morbidity compared with risk-matched ICU admission. Undertriaged ward admissions were identified by a validated machine-learning model using automated electronic health record (EHR) data to generate risk estimations at surgical procedure end time, when triage decisions were finalized.9,10 We compared mortality and morbidity among appropriately triaged ward admissions, undertriaged wards admissions, and risk-matched ICU admissions.
Methods
The University of Florida Institutional Review Board approved this cross-sectional study with full waiver of informed consent with the justification that this retrospective study involved no interventions and it would be difficult to contact thousands patients who received care many years prior to enrollment. This study complies with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.
Study Design
We generated a longitudinal cohort of patients aged 18 years or older undergoing inpatient surgical procedures at a university hospital. Ward admissions were considered undertriaged if their estimated risk for hospital mortality or prolonged ICU stay (ie, ≥48 hours) was in the top quartile among inpatient surgical procedures according to a validated machine-learning model using preoperative and intraoperative estimator features extracted directly from EHRs, generating estimations at surgical procedure end time; risk thresholds were applied to these estimations.9,10 A nearest neighbors algorithm was used to identify a risk-matched control ICU admissions. Matching was performed on composite risk for hospital mortality or prolonged ICU stay represented by the mean z score across both outcomes. The primary analysis compared mortality and morbidity among admissions that were appropriately triaged to hospital wards or undertriaged to hospital wards and members of a risk-matched control group.
Data Source
The University of Florida Integrated Data Repository functioned as honest broker in assembling the study cohort, consisting of 71 065 hospital admissions for inpatient surgical procedures at University of Florida Health in Gainesville, Florida, from June 1, 2014, to August 20, 2020. Derivation of the study population is illustrated in eFigure 1 in the Supplement. Briefly, we included inpatient surgical admissions and excluded organ donation surgical procedures, anesthesia outside the operating room, and admissions lasting less than 24 hours, which are often considered observation only, failing to represent major inpatient surgical procedures. Deaths within 24 hours were excluded to avoid excessively narrow estimation windows. The study cohort was split chronologically into development and validation cohorts to allow nonrandom variation between data sets and mitigate potentially adverse outcomes associated with data set drift for algorithm estimations, consistent with Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline recommendations under the type 2b analysis category.11 The final data set included granular information from preoperative, intraoperative, and postoperative phases of care.9,10 Descriptions of all data elements and their preprocessing steps are provided in eMethods in the Supplement.
Model Development and Validation
The model used a data transformer layer to perform preprocessing, feature transformation, and feature selection techniques.9,10 A data analytics layer computed separate, patient admission–specific risk probabilities for hospital mortality and prolonged ICU stay using a random forest algorithm.10 The model was trained on a development cohort of 49 687 postoperative admissions, using only preoperative and intraoperative data as input features. The trained model made estimations on 20 940 validation cohort admissions; results are reported exclusively from this separate cohort.
Identifying Undertriaged Ward Admissions
In identifying undertriage, the top quartile (ie, ≥75th percentile) cutoff for hospital mortality risk was chosen because it corresponds to 5% incidence of mortality, consistent with definitions of high-risk surgical procedures in peer-reviewed literature.2,3,12 Regarding risk of prolonged ICU stay, for which there is no precedent in peer-reviewed literature, the top quartile cutoff was chosen to maintain consistency with the mortality cutoff.
Identifying Risk-Matched Control ICU Admissions
To identify a control population of postoperative ICU admissions, we fit a nearest neighbors, brute-force algorithm13 on estimations for hospital mortality and prolonged ICU stay among ICU admissions that did not receive immediate postoperative mechanical ventilation or vasopressor support (ie, within 2 hours of surgical procedure end time) and applied the fit model to the undertriaged group using a maximum Minkowski distance of 1 to identify the 3 nearest neighbors for each admission. Matching was performed on composite risk for hospital mortality or prolonged ICU stay represented by the mean z score across these outcomes. This method matched 2452 control admissions with 1306 undertriage admissions.
Hospital Stations
We extracted EHR-embedded hospital station labels with entrance and exit date times. Level of care assignments allowed ICU admission under intensive status, requiring patient-to-nurse ratios of 2:1 or less, or intermediate status, requiring patient-to-nurse ratios of 3:1 or less. The ICU was considered as a single destination that provided low patient-to-nurse ratios and consistent availability of critical care resources and personnel.
Statistical Analysis
We performed a power analysis using data from a peer-reviewed study of 121 290 inpatient surgical procedures.14 Planned surgical complexity grade 4 was observed among 35 931 hospital admissions and was associated with hospital mortality among 1863 admissions (5.18%), corresponding to high-risk surgical procedures definitions.2,3,12 Planned surgical complexity grade 1 was observed among 13 484 hospital admissions and was associated with hospital mortality among 49 admissions (0.36%). Accounting for imbalanced cohorts and hospital mortality as low as among 16 admissions (1.24%) in the high-risk group, our sample would detect a difference in hospital mortality of 1.19 percentage points, with 80% power and α = .05. Other outcomes reported were exploratory. While conducting this study, we observed that the undertriaged group had increased area deprivation indices and increased proportions of patients identifying as Black or African American and with preoperative do not resuscitate (DNR) orders; we performed exploratory, post hoc, secondary analyses for these 3 variables. Race data were collected from electronic health records using self-reported classifications. Race was assessed to determine whether there were associations among race, postoperative triage decisions, and patient outcomes. All primary outcome analyses were adjusted for multiple comparisons using the Benjamini-Hochberg procedure.
For model training, missing values were replaced with a distinct missing indicator; for reporting results, missing values were imputed with medians. Continuous variables were reported as median values with IQR and compared by the Kruskal-Wallis test. Discrete variables were reported as raw numbers with percentages and compared by Fisher exact test. Model performance was assessed by calculating area under the receiver operating characteristic curves (AUROCs) and area under the precision recall curves (AUPRCs) with 95% CIs. All hypothesis tests were 2-sided with α = .05. The power analysis was performed using PASS statistical software (NCSS). Other statistical analyses were performed with Python programming language version 3.8.8 (Python Software Foundation). Data were analyzed from April through August 2021.
Results
Study Population Characteristics
Patient characteristics are summarized in Table 1. Among 12 348 postoperative ward admissions, 11 042 admissions (89.4%) were appropriately triaged and 1306 admissions (10.6%) were undertriaged and matched with a control group of 2452 ICU admissions. The undertriaged group, compared with the appropriately triaged group, had a decreased proportion of women (649 [49.7%] women vs 5927 [53.7%] women; P = .007), increased median (IQR) age (64 [54-74] vs 59 [44-70] years; P < .001), and an increased proportion of admitted patients self-identifying as Black or African American (261 admissions [20.0%] vs 1627 admission [14.7%]; P < .001), with a decreased proportion of admitted patients self-identifying as White (956 admissions [73.2%] vs 8483 admissions [76.8%]; P = .004) and a similar proportion of admitted patients self-identifying as Hispanic (2 admissions [0.2%] vs 20 admissions [0.2%]; P > .99). Area deprivation indices were increased among undertriaged admissions, suggesting socioeconomic vulnerability. The undertriaged group had increased median [IQR] Charlson Comorbidity Index score (2.0 [1.0-4.0] vs 1.0 [0-2.0]; P < .001) and increased proportions of admitted patients undergoing emergent admission (868 admissions [66.5%] vs 3369 admissions [30.5%]; P < .001), preoperative red blood cell transfusion (87 admissions [6.7%] vs 84 admissions [0.8%]; P < .001), and emergent surgical procedures (298 admissions [22.8%] vs 1352 admissions [12.2%]; P < .001).
Table 1. Patient Characteristics.
| Characteristic | Admissions, No. (%) | P valuea | |||
|---|---|---|---|---|---|
| Appropriate triage (n = 11 042) | Undertriage (n = 1306) | Control (n = 2452) | Appropriate triage vs undertriage | Undertriage vs control | |
| Demographics | |||||
| Women | 5927 (53.7) | 649 (49.7) | 1080 (44.0) | .007 | <.001 |
| Men | 5115 (46.3) | 657 (50.3) | 1372 (56.0) | ||
| Age, median (IQR), y | 59.0 (44.0-70.0) | 64.0 (54.0-74.0) | 62.0 (50.0-73.0) | <.001 | .001 |
| Race | |||||
| American Indian or Alaska Native | 17 (0.2) | 0 | 5 (0.2) | .25 | .17 |
| Asian | 101 (0.9) | 9 (0.7) | 20 (0.8) | .53 | .85 |
| Black or African American | 1627 (14.7) | 261 (20.0) | 331 (13.5) | <.001 | <.001 |
| Hispanic | 20 (0.2) | 2 (0.2) | 2 (0.1) | >.99 | .61 |
| Native Hawaiian or other Pacific Islander | 2 (<0.1) | 0 | 0 | >.99 | >.99 |
| White | 8483 (76.8) | 956 (73.2) | 1946 (79.4) | .004 | <.001 |
| Otherb or multiracial | 605 (5.5) | 63 (4.8) | 113 (4.6) | .37 | .81 |
| Unknown | 187 (1.7) | 15 (1.1) | 35 (1.4) | .17 | .55 |
| ADI score, median (IQR) | |||||
| State rank | 7.0 (4.0-9.0) | 8.0 (5.0-9.0) | 7.0 (5.0-9.0) | <.001 | .004 |
| National rank | 69.0 (46.0-85.0) | 73.0 (53.0-85.0) | 68.0 (47.0-85.0) | <.001 | .004 |
| Illness severity, median (IQR) | |||||
| ASA score | 3.0 (2.0-3.0) | 3.0 (3.0-3.0) | 3.0 (3.0-3.0) | <.001 | .16 |
| Charlson Comorbidity Index score | 1.0 (0-2.0) | 2.0 (1.0-4.0) | 2.0 (0-3.0) | <.001 | <.001 |
| SOFA score | 0 (0-0) | 0 (0-0) | 1.0 (0-2.0) | <.001 | <.001 |
| Admitted from emergency department | 3319 (30.1) | 810 (62.0) | 850 (34.7) | <.001 | <.001 |
| Red blood cell transfusion | |||||
| Preoperative | 84 (0.8) | 87 (6.7) | 68 (2.8) | <.001 | <.001 |
| Intraoperative | 23 (0.2) | 5 (0.4) | 13 (0.5) | .21 | .63 |
| Admission priority | |||||
| Elective | 7158 (64.8) | 305 (23.4) | 1354 (55.2) | <.001 | <.001 |
| Urgent | 130 (1.2) | 49 (3.8) | 64 (2.6) | <.001 | .06 |
| Emergent | 3369 (30.5) | 868 (66.5) | 848 (34.6) | <.001 | <.001 |
| Trauma activation or transfer | 383 (3.5) | 84 (6.4) | 186 (7.6) | <.001 | .21 |
| Unknown | 2 (<0.1) | 0 | 0 | >.99 | >.99 |
| Surgical procedure priority | |||||
| Elective | 9070 (82.1) | 920 (70.4) | 1919 (78.3) | <.001 | <.001 |
| Urgent | 620 (5.6) | 88 (6.7) | 89 (3.6) | .10 | <.001 |
| Emergent | 1352 (12.2) | 298 (22.8) | 444 (18.1) | <.001 | <.001 |
Abbreviations: ADI, area deprivation index; ASA, American Society of Anesthesiologists; SOFA, Sequential Organ Failure Assessment.
P values are from significance tests comparing appropriate triage, undertriage, and control groups by each variable.
Other was a category for self-reporting race that patients could select.
The undertriaged group, compared with the control group, had increased median [IQR] age (vs 62 [50-73] years; P = .001) and increased proportions of women (vs 1080 [44.0%] women; P < .001) and admitted patients self-identifying as Black or African American (vs 331 admissions [13.5%]; P < .001), with a decreased proportion of admitted patients self-identifying as White (vs 1946 admissions [79.4%]; P < .001) and a similar proportion of admitted patients self-identifying as Hispanic (vs 2 admissions [0.1%]; P = .61). Area deprivation indices were increased among undertriaged admissions compared with admissions in the control group. American Society of Anesthesiologists class distributions were similar between these 2 groups, but the undertriaged group had an increased median (IQR) Charlson Comorbidity Index score (vs 2.0 [0-3.0]; P < .001) and decreased median (IQR) Sequential Organ Failure Assessment score on admission (0 [0-0] vs 1.0 [0-2.0]; P < .001). The undertriaged group had increased proportions of admitted patients undergoing emergent admission (vs 848 admissions [34.6%]; P < .001) and emergent surgical procedures (vs 444 admissions [18.1%]; P < .001) and with preoperative red blood cell transfusions (vs 68 admissions [2.8%]; P < .001). Primary surgical services for each admission and service-specific mortality rates are listed in eTable 1 and eTable 2 in the Supplement, respectively.
Triage Classifications
In the validation cohort, hospital mortality estimations had an AUROC of 0.92 (95% CI, 0.91-0.93) and an AUPRC of 0.26 (95% CI, 0.23-0.30); prolonged ICU stay estimations had an AUROC of 0.92 (95% CI, 0.92-0.92) and an AUPRC of 0.85 (95% CI, 0.85-0.86). According to these estimations, 11 042 postoperative ward admissions were appropriate and 1306 admissions were undertriaged, representing 10.6% of all ward admissions. The 20 most important features (as defined by weight, which was derived by calculating change in node impurity and the probability of reaching each node) for estimating hospital mortality and prolonged ICU stay are listed in eTable 3 and eTable 4 in the Supplement, respectively. Primary procedure, scheduled postoperative location, and intraoperative minimum alveolar concentration measurements or inhalational anesthetic duration were among the 5 most important features for both estimations.
Perioperative Factors
Perioperative factors are summarized in Table 2. Increased proportions of undertriaged admissions, compared with appropriately triaged admissions, were planned for postoperative ICU admission under intensive status (91 admissions [7.0%] vs 284 admissions [2.6%]; P < .001) and intermediate status (19 admissions [1.5%] vs 66 admissions [0.6%]; P = .002). Immediately after the surgical procedure, undertriaged admissions had increased median (IQR) heart rate (82.1 beats per minute [bpm; 72.2-91.4 bpm] vs 79.8 bpm [71.2-89.0 bpm]; P < .001) and decreased median (IQR) systolic blood pressure (124.3 [111.0-138.9] mm Hg vs 126.1 [114.5-138.0] mm Hg; P < .001) but similar respiratory rate, oxygen saturation, and Glasgow coma scale eye opening response. An increased proportion of undertriaged admissions had cardiac telemetry ordered prior to ward admission (225 admissions [17.2%] vs 460 admissions [4.2%]; P < .001) compared with appropriately triaged admissions, while the proportion with a continuous pulse oximetry order was similar between these 2 groups (1094 admissions [83.8%] vs 9408 admissions [85.2%]; P = .18). The undertriaged group had an increased proportion of admitted patients with DNR orders placed preoperatively (53 admissions [4.1%] vs 55 admissions [0.5%]; P < .001) and during admission (91 admissions [7.0%] vs 78 admissions [0.7%]; P < .001) compared with the appropriately triaged group.
Table 2. Admitted Patient Perioperative Factors.
| Perioperative factorb | Admissions, No. (%) | P valuea | |||
|---|---|---|---|---|---|
| Appropriate triage (n = 11 042) | Undertriage (n = 1306) | Control (n = 2452) | Appropriate triage vs undertriage | Undertriage vs control | |
| Planned postoperative destination | |||||
| ICU, intensive status | 284 (2.6) | 91 (7.0) | 1570 (64.0) | <.001 | <.001 |
| ICU, intermediate status | 66 (0.6) | 19 (1.5) | 237 (9.7) | .002 | <.001 |
| Postanesthesia care unit and ward | 10 665 (96.6) | 1173 (89.8) | 630 (25.7) | <.001 | <.001 |
| Unknown | 27 (0.2) | 23 (1.8) | 15 (0.6) | <.001 | .002 |
| Immediate postoperative vital signs, median (IQR)c | |||||
| Heart rate, bpm | 79.8 (71.2-89.0) | 82.1 (72.2-91.4) | 80.5 (70.2-91.2) | <.001 | .01 |
| Blood pressure, mm Hg | |||||
| Systolic | 126.1 (114.5-138.0) | 124.3 (111.0-138.9) | 131.0 (116.4-145.4) | <.001 | <.001 |
| Diastolic | 70.3 (62.8-78.0) | 66.4 (58.2-74.8) | 67.0 (58.2-76.5) | <.001 | .23 |
| Respiratory rate, breaths per minute | 16.0 (14.6-17.6) | 16.2 (14.3-17.9) | 16.4 (14.5-18.6) | .31 | <.001 |
| Oxygen saturation, % | 96.0 (94.5-97.5) | 95.9 (94.4-97.6) | 96.2 (94.8-97.8) | .28 | <.001 |
| Temperature, °C | 37.2 (37.1-37.4) | 37.3 (37.1-37.5) | 37.2 (37.0-37.5) | .07 | <.001 |
| GCS eye opening response | |||||
| Median (IQR) | 4.0 (4.0-4.0) | 4.0 (4.0-4.0) | 4.0 (4.0-4.0) | .42 | <.001 |
| <4 | 1058 (9.6) | 133 (10.2) | 398 (16.2) | .49 | <.001 |
| Surveillance during ward admission | |||||
| Continuous pulse oximetry | 9408 (85.2) | 1094 (83.8) | NA | .74 | NA |
| Cardiac telemetry | 460 (4.2) | 225 (17.2) | NA | <.001 | NA |
| DNR order status | |||||
| Ordered before first surgical procedure | 55 (0.5) | 53 (4.1) | 27 (1.1) | <.001 | <.001 |
| Ordered during admission | 78 (0.7) | 91 (7.0) | 59 (2.4) | <.001 | <.001 |
| Canceled during admission | 6 (0.1) | 18 (1.4) | 15 (0.6) | <.001 | .03 |
Abbreviations: bpm, beats per minute; DNR, do not resuscitate; GCS, Glasgow coma scale; ICU, intensive care unit; NA, not applicable.
P values are from significance tests comparing appropriate triage, undertriage, and control groups by each perioperative factor.
Postoperative variables, including postoperative vital signs, were not included as features in the estimation models used for cohort derivation.
Within 4 hours after surgical procedure end time.
The undertriaged group, compared with the control group, had similar immediate postoperative diastolic blood pressure but decreased median (IQR) systolic blood pressure (vs 131.0 [116.4-145.4] mm Hg; P < .001), respiratory rate (16.2 breaths per minute [14.3-17.9 breaths per minute] vs 16.4 breaths per minute [14.5-18.6 breaths per minute]; P < .001), and oxygen saturation (95.9% [94.4%-97.6%] vs 96.2% [94.8%-97.8%]; P < .001) and increased median (IQR) heart rate (82.1 bpm [72.2-91.4 bpm] vs 80.5 bpm [70.2-91.2 bpm], P = .01) and temperature (37.3 °C [37.1 °C-37.5 °C] vs 37.2 °C [37.0 °C-37.5 °C ]; P < .001). A decreased proportion of admitted patients who were undertriaged had a Glasgow coma scale eye opening response score less than 4 (133 admissions [10.2%] vs 398 admissions [16.2%]; P < .001) compared with admitted patients in the control group. An increased proportion of admitted patients who were undertriaged had a DNR order placed preoperatively (vs 27 admissions [1.1%]; P < .001) and during admission (vs 59 admissions [2.4%]; P < .001).
Patient Outcomes
Patient outcomes are summarized in Table 3. The undertriaged group, compared with the appropriately triaged group, had increased incidence of ICU transfer (207 admissions [15.8%] vs 298 admissions [2.7%]; P < .001) at later median (IQR) postoperative times (71.0 [31.5-144.0] hours vs 35.0 [21.2-70.0] hours; P < .001). Undertriaged admissions had increased median (IQR) hospital length of stay (8.1 [5.1-13.6] days vs 3.0 [1.5-5.1] days; P < .001) and increased incidence of prolonged ICU admission (165 admissions [12.6%] vs 212 admissions [1.9%]; P < .001), prolonged mechanical ventilation (32 admissions [2.5%] vs 19 admissions [0.2%]; P < .001), postoperative red blood cell transfusion (325 admissions [24.9%] vs 475 admissions [4.3%]; P < .001), arterial catheter placement (173 admissions [13.2%] vs 370 admissions [3.4%]; P < .001), central venous catheter placement (37 admissions [2.8%] vs 49 admissions [0.4%]; P < .001), second surgical procedure during admission (360 admissions [27.6%] vs 536 admissions [4.9%]; P < .001), hospital mortality (19 admissions [1.5%] vs 5 admissions [0.05%]; P < .001), cardiac arrest (7 admissions [0.5%] vs 7 admissions [0.1%]; P < .001), unplanned intubation (45 admissions [3.4%] vs 48 admissions [0.4%]; P < .001), and acute kidney injury (341 admissions [26.1%] vs 965 admissions [8.7%]; P < .001) compared with appropriately triaged admissions. The undertriaged group had increased median (IQR) total charges per admission ($101 300 [$70 300-$155 000] vs $65 700 [$47 600-$88 300]; P < .001) and total costs per admission ($26 900 [$18 400-$42 300] vs $15 900 [$11 300-$22 500]; P < .001).
Table 3. Postoperative Outcomes.
| Outcome | Admissions, No (%) | P valuea | |||
|---|---|---|---|---|---|
| Appropriate triage (n = 11 042) | Undertriage (n = 1306) | Control (n = 2452) | Appropriate triage vs undertriage | Undertriage vs control | |
| Resource use | |||||
| Transfer from ward to ICU | 298 (2.7) | 207 (15.8) | NA | <.001 | NA |
| Time from surgical procedures end to ICU transfer, median (IQR), hr | 35.0 (21.2-70.0) | 71.0 (31.5-144.0) | NA | <.001 | NA |
| ICU admission ≥48 h | 212 (1.9) | 165 (12.6) | 1407 (57.4) | <.001 | <.001 |
| Mechanical ventilation ≥48 h | 19 (0.2) | 32 (2.5) | 53 (2.2) | <.001 | .60 |
| Postoperative red blood cell transfusion | 396 (3.6) | 275 (21.1) | 337 (13.7) | <.001 | <.001 |
| Red blood cell transfusion during admission | 475 (4.3) | 325 (24.9) | 380 (15.5) | <.001 | <.001 |
| Hospital length of stay, median (IQR), d | 3.0 (1.5-5.1) | 8.1 (5.1-13.6) | 6.0 (3.3-9.3) | <.001 | <.001 |
| Postoperative procedure | |||||
| Arterial catheter | 370 (3.4) | 173 (13.2) | 261 (10.6) | <.001 | .02 |
| Central venous catheter | 49 (0.4) | 37 (2.8) | 51 (2.1) | <.001 | .20 |
| Bronchoscopy | 9 (0.1) | 8 (0.6) | 23 (0.9) | <.001 | .38 |
| Chest tube | 8 (0.1) | 3 (0.2) | 8 (0.3) | .13 | .76 |
| Electric cardioversion | 4 (<0.1) | 3 (0.2) | 4 (0.2) | .04 | .71 |
| Second surgical procedure during admission | 536 (4.9) | 360 (27.6) | 353 (14.4) | <.001 | <.001 |
| Emergent second surgical procedure | 107 (1.0) | 92 (7.0) | 91 (3.7) | <.001 | <.001 |
| Time between surgical procedures, median (IQR), d | 2.6 (1.8-4.7) | 3.8 (2.0-6.0) | 3.3 (1.6-6.3) | <.001 | .23 |
| Complication | |||||
| Hospital mortality | 5 (0.1) | 19 (1.5) | 17 (0.7) | <.001 | .04 |
| Discharge to hospice | 13 (0.1) | 23 (1.8) | 14 (0.6) | <.001 | <.001 |
| Cardiac arrest | 7 (0.1) | 7 (0.5) | 10 (0.4) | <.001 | .64 |
| Unplanned intubation | 48 (0.4) | 45 (3.4) | 49 (2.0) | <.001 | .01 |
| Acute kidney injury | 965 (8.7) | 341 (26.1) | 477 (19.5) | <.001 | <.001 |
| Rapid reversal | 459 (4.2) | 153 (11.7) | 248 (10.1) | <.001 | .16 |
| Persistent | |||||
| With kidney recovery | 100 (0.9) | 97 (7.4) | 91 (3.7) | <.001 | <.001 |
| Without kidney recovery | 406 (3.7) | 91 (7.0) | 138 (5.6) | <.001 | .14 |
| Charges and costs, median (IQR), thousands of $ | |||||
| Professional service charges | 13.7 (10.0-18.5) | 18.5 (13.1-30.2) | 24.5 (16.9-37.9) | <.001 | <.001 |
| Charges for hospital admission | 65.7 (47.6-88.3) | 101.3 (70.3-155.0) | 120.2 (84.5-163.7) | <.001 | <.001 |
| Costs for hospital admission | 15.9 (11.3-22.5) | 26.9 (18.4-42.3) | 32.7 (22.7-48.5) | <.001 | <.001 |
Abbreviations: ICU, intensive care unit; NA, not applicable.
P values were adjusted for multiple comparisons using the Benjamini-Hochberg procedure. P values are from significance tests comparing appropriate triage, undertriage, and control groups by each outcome.
The undertriaged group, compared with the control group, had decreased proportions of admitted patients with prolonged ICU admission (vs 1407 admissions [57.4%]; P < .001), similar proportions of admitted patients with prolonged mechanical ventilation (vs 53 admissions [2.2%]; P = .60), decreased proportions of admitted patients receiving postoperative red blood cell transfusion (275 admissions [21.1%] vs 337 admissions [13.7%]; P < .001), and increased median (IQR) hospital length of stay (vs 6.0 [3.3-9.3] days; P < .001). There was also an increased proportion of admitted patients who were undertriaged undergoing postoperative arterial catheter placement (vs 261 admissions [10.6%]; P = .02), second surgical procedure during admission (vs 353 admissions [14.4%]; P < .001), and emergent second surgical procedure (vs 91 admissions [3.7%]; P < .001) compared with admitted patients in the control group. Intervals between first and second surgical procedures were similar between undertriaged and control groups (median [IQR], 3.8 [2.0-6.0] days vs 3.3 [1.6-6.3] days; P = .23). The undertriaged group, compared with the control group, had increased incidence of hospital mortality (vs 17 admissions [0.7%]; P = .04), discharge to hospice (23 admissions [1.8%] vs 14 admissions [0.6%]; P < .001), unplanned intubation (vs 49 admissions [2.0%]; P = .01), and acute kidney injury (vs 477 admissions [19.5%]; P < .001), which was primarily associated with increased incidence of persistent acute kidney injury with kidney recovery (97 admissions [7.4%] vs 91 admissions [3.7%]; P < .001). Undertriaged admissions had decreased median (IQR) total charges per admission (vs $120 200 [$84 500-$163 700]; P < .001) and total costs per admission (vs $32 700 [$22 700-$48 500]; P < .001) compared with the control group.
Secondary Analyses
Exploratory secondary analyses investigating whether there were associations between self-identification as Black or African American, area deprivation indices, and preoperative DNR orders and outcomes are illustrated in eFigure 2, eFigure 3, and eFigure 4 in the Supplement, respectively; we provide analytic details in figure legends. Appropriately triaged admissions were excluded from these analyses owing to substantial differences in baseline characteristics and outcomes. Secondary analysis outcomes were hospital mortality or discharge to hospice, unplanned intubation, and acute kidney injury because these outcomes differed between undertriaged and control groups in primary analyses.
There were no associations of area deprivation indices or self-identification as Black or African American with selected outcomes. Admitted patients with preoperative DNR orders in the undertriaged and control groups had increased incidence of hospital death or discharge to hospice (5 admissions [6.2%] vs 68 admissions [1.8%]; P = .02) and acute kidney injury (32 admissions [40.0%] vs 786 admissions [21.4%]; P < .001) compared with admitted patients without DNR orders. The undertriaged group had an increased proportion of admitted patients with both preoperative DNR orders and postoperative acute kidney injury (19 admissions [1.5%] vs 13 admissions [0.5%]; P = .005) compared with the control group; the proportion of admitted patients with both preoperative DNR orders and death or hospice were similar between the undertriage and control groups (3 admissions [0.2%] vs 2 admissions [0.1%]; P = .35). There were no associations between preoperative DNR orders and unplanned intubation.
Discussion
This cross-sectional study found that approximately 11% of postoperative ward admissions had increased risk of postoperative complications and had increased short-term mortality and morbidity compared with risk-matched ICU admissions. The observed increased incidence of preoperative DNR orders among undertriaged admissions may have been associated with decreased escalation of care in response to impending or evolving postoperative complications; deeper analyses of palliative care consultations and clinical notes are needed to address this hypothesis. Undertriaged admissions had increased incidence of cardiac telemetry; among patients admitted to wards who were at increased risk of complications, continuous monitoring may be associated with increased early detection of decompensation without requiring ICU resources.15,16,17 Additionally, we found that patients who were at increased risk of postoperative complications and undertriaged were accurately identified using automated EHR data from preoperative and intraoperative phases of care, suggesting opportunities for data-driven decision support. These results could be operationalized by generating EHR advisories triggered by ward admission orders placed for patients who have received surgical procedures who are at increased risk of complications.
Previous work has characterized postoperative triage patterns by classifying patients at increased risk of complications using a single parameter: high-risk procedures with mortality rates of 5% or greater. Using this approach in a National Health Service study, Pearse et al2 provided a sentinel report that patients undergoing high-risk surgical procedures accounted for 13% of surgical procedures and 84% of deaths. Notably, fewer than 15% of patients at increased risk of complications were triaged to ICUs postoperatively. In a subsequent study by Jhanji et al,3 35% of patients at increased risk of complications were admitted to an ICU at any point after their surgical procedures. Overall, the high-risk mortality rate was 12%, but among patients at increased risk of complications who were initially triaged to a ward and subsequently transferred to the ICU, mortality was 30%. Others have assessed the ability of physicians to estimate adverse events when making triage decisions. In a multicenter study of patients admitted to ICUs for medical and surgical procedures who were transferred to hospital wards, physicians surveyed at the time of ICU discharge exhibited mediocre performance in estimating adverse events, ICU readmissions, and hospital mortality.18 A similar phenomenon may occur at surgical procedure end time, although further investigation is needed to address this hypothesis.
The methods presented in our study are unique, which hinders direct comparisons with previous work, but our results are generally consistent with those in peer-reviewed literature. Advanced age, demographic variables, comorbidities, American Society of Anesthesiologists physical status classifications, and organ dysfunction have all been previously described as major risk factors associated with postoperative mortality and morbidity.9,10,19,20,21,22,23,24,25,26,27,28 Analyses including area deprivation indices in postoperative outcomes are sparse, but Carmichael et al29 found that increased area deprivation indices, along with other indicators associated with social vulnerability, were associated with increased odds of undergoing emergency surgical procedures. We observed increased area deprivation indices among undertriaged admissions, which may be associated with overlap with emergency admission priority and surgical procedure priority or an odious association with biased care or unobserved disease; further investigation of these potential etiologies is needed. We observed that DNR orders were associated with increased incidence of postoperative mortality, as was previously described by Kazaure et al.30 Overall, although our postoperative acuity assessment methods are unique, results based on our methods are similar to results from similar studies.
Limitations
This study has several limitations. Our study was limited by its single institution design and may not be generalizable to other practice settings. In addition, the retrospective analysis of EHR data is associated with risk for selection bias. We sought to minimize selection bias by including all consecutive admissions meeting inclusion criteria. These analyses excluded decisions for postoperative admission vs discharge home (ie, ambulatory surgical procedures), which require further investigation. Time-splitting development and validation cohorts allow bias from practice changes over time but may also mitigate the potential for model performance degradation from data set drift when applied prospectively.11 Although model discrimination was strong, precision in estimating hospital mortality was poor, as is often observed when estimating rare outcomes. Additionally, our power analysis was targeted loosely, owing to the lack of similar studies in peer-reviewed literature. This underscores a greater problem: despite the severe consequences associated with postoperative undertriage, there is sparse evidence regarding this topic, which hinders the development of decision-support tools. In the absence of efficient, effective decision-support platforms to augment postoperative triage decisions, surgeons must rely on individual judgement and hypothetical-deductive reasoning, which is highly variable and error-prone and may contribute to variability in surgical care costs and the incidence of failure to rescue across US hospitals.31,32
Conclusions
This study found that patients at increased risk for postoperative complications who underwent surgical procedures and were undertriaged to hospital wards had increased mortality and morbidity compared with patients in a risk-matched control group admitted to ICUs. Postoperative undertriage was identifiable using automated preoperative and intraoperative EHR data as features in machine-learning models that made data-driven, patient-level risk assessments. These findings may provide a framework and rationale for real-time clinical decision support platforms to augment postoperative triage decisions.
eFigure 1. Derivation of Study Population
eTable 1. Primary Surgical Service
eTable 2. Hospital Mortality Rates for Each Primary Surgical Service
eTable 3. 20 Most Important Features for Estimating Hospital Mortality
eTable 4. 20 Most Important Features for Estimating Prolonged Intensive Care Unit Stay
eFigure 2. Subgroup Analyses of Associations Between Outcomes and Self-identification as Black or African American
eFigure 3. Subgroup Analyses of Associations Between Outcomes and Area Deprivation Index (ADI) National Ranks
eFigure 4. Subgroup Analyses of Associations Between Outcomes and Preoperative Do No Resuscitate (DNR) Orders
eMethods.
References
- 1.Elixhauser A, Andrews RM. Profile of inpatient operating room procedures in US hospitals in 2007. Arch Surg. 2010;145(12):1201-1208. doi: 10.1001/archsurg.2010.269 [DOI] [PubMed] [Google Scholar]
- 2.Pearse RM, Harrison DA, James P, et al. Identification and characterisation of the high-risk surgical population in the United Kingdom. Crit Care. 2006;10(3):R81. doi: 10.1186/cc4928 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Jhanji S, Thomas B, Ely A, Watson D, Hinds CJ, Pearse RM. Mortality and utilisation of critical care resources amongst high-risk surgical patients in a large NHS trust. Anaesthesia. 2008;63(7):695-700. doi: 10.1111/j.1365-2044.2008.05560.x [DOI] [PubMed] [Google Scholar]
- 4.Skogvoll E, Isern E, Sangolt GK, Gisvold SE. In-hospital cardiopulmonary resuscitation: 5 years’ incidence and survival according to the Utstein template. Acta Anaesthesiol Scand. 1999;43(2):177-184. doi: 10.1034/j.1399-6576.1999.430210.x [DOI] [PubMed] [Google Scholar]
- 5.Merchant RM, Yang L, Becker LB, et al. ; American Heart Association Get With The Guidelines-Resuscitation Investigators . Incidence of treated cardiac arrest in hospitalized patients in the United States. Crit Care Med. 2011;39(11):2401-2406. doi: 10.1097/CCM.0b013e3182257459 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Perman SM, Stanton E, Soar J, et al. ; American Heart Association’s Get With the Guidelines—Resuscitation (formerly the National Registry of Cardiopulmonary Resuscitation) Investigators . Location of in-hospital cardiac arrest in the United States–variability in event rate and outcomes. J Am Heart Assoc. 2016;5(10):e003638. doi: 10.1161/JAHA.116.003638 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Sandroni C, Nolan J, Cavallaro F, Antonelli M. In-hospital cardiac arrest: incidence, prognosis and possible measures to improve survival. Intensive Care Med. 2007;33(2):237-245. doi: 10.1007/s00134-006-0326-z [DOI] [PubMed] [Google Scholar]
- 8.Schein RM, Hazday N, Pena M, Ruben BH, Sprung CL. Clinical antecedents to in-hospital cardiopulmonary arrest. Chest. 1990;98(6):1388-1392. doi: 10.1378/chest.98.6.1388 [DOI] [PubMed] [Google Scholar]
- 9.Bihorac A, Ozrazgat-Baslanti T, Ebadi A, et al. MySurgeryRisk: development and validation of a machine-learning risk algorithm for major complications and death after surgery. Ann Surg. 2019;269(4):652-662. doi: 10.1097/SLA.0000000000002706 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Datta S, Loftus TJ, Ruppert MM, et al. Added value of intraoperative data for predicting postoperative complications: the MySurgeryRisk postop extension. J Surg Res. 2020;254:350-363. doi: 10.1016/j.jss.2020.05.007 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Finlayson SG, Subbaswamy A, Singh K, et al. The clinician and dataset shift in artificial intelligence. N Engl J Med. 2021;385(3):283-286. doi: 10.1056/NEJMc2104626 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Boyd O, Jackson N. How is risk defined in high-risk surgical patient management? Crit Care. 2005;9(4):390-396. doi: 10.1186/cc3057 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.sklearn.neighbors.NearestNeighbors. Scikit Learn. Accessed October 15, 2021. https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.NearestNeighbors.html
- 14.Cutti S, Klersy C, Favalli V, et al. A multidimensional approach of Surgical Mortality Assessment and Stratification (SMATT score). Sci Rep. 2020;10(1):10964. doi: 10.1038/s41598-020-67164-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Subbe CP, Williams E, Fligelstone L, Gemmell L. Does earlier detection of critically ill patients on surgical wards lead to better outcomes? Ann R Coll Surg Engl. 2005;87(4):226-232. doi: 10.1308/003588405X50921 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Taenzer AH, Pyke JB, McGrath SP, Blike GT. Impact of pulse oximetry surveillance on rescue events and intensive care unit transfers: a before-and-after concurrence study. Anesthesiology. 2010;112(2):282-287. doi: 10.1097/ALN.0b013e3181ca7a9b [DOI] [PubMed] [Google Scholar]
- 17.Brown H, Terrence J, Vasquez P, Bates DW, Zimlichman E. Continuous monitoring in an inpatient medical-surgical unit: a controlled clinical trial. Am J Med. 2014;127(3):226-232. doi: 10.1016/j.amjmed.2013.12.004 [DOI] [PubMed] [Google Scholar]
- 18.Sauro KM, Soo A, de Grood C, et al. Adverse events after transition from ICU to hospital ward: a multicenter cohort study. Crit Care Med. 2020;48(7):946-953. doi: 10.1097/CCM.0000000000004327 [DOI] [PubMed] [Google Scholar]
- 19.Finlayson EV, Birkmeyer JD. Operative mortality with elective surgery in older adults. Eff Clin Pract. 2001;4(4):172-177. [PubMed] [Google Scholar]
- 20.Teh SH, Nagorney DM, Stevens SR, et al. Risk factors for mortality after surgery in patients with cirrhosis. Gastroenterology. 2007;132(4):1261-1269. doi: 10.1053/j.gastro.2007.01.040 [DOI] [PubMed] [Google Scholar]
- 21.Pan Y, Wang W, Wang J, Yang L, Ding F; ISN AKF 0by25 China Consortium . Incidence and risk factors of in-hospital mortality from AKI after non-cardiovascular operation: a nationwide survey in China. Sci Rep. 2017;7(1):13953. doi: 10.1038/s41598-017-13763-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Oh TK, Jeon YT, Do SH, Hwang JW. Pre-operative assessment of 30-day mortality risk after major surgery: the role of the quick sequential organ failure assessment: a retrospective observational study. Eur J Anaesthesiol. 2019;36(9):688-694. doi: 10.1097/EJA.0000000000000957 [DOI] [PubMed] [Google Scholar]
- 23.Forte ML, Virnig BA, Swiontkowski MF, et al. Ninety-day mortality after intertrochanteric hip fracture: does provider volume matter? J Bone Joint Surg Am. 2010;92(4):799-806. doi: 10.2106/JBJS.H.01204 [DOI] [PubMed] [Google Scholar]
- 24.Kim TH, Um SH, Yim SY, et al. The risk of perioperative adverse events in patients with chronic liver disease. Liver Int. 2015;35(3):713-723. doi: 10.1111/liv.12529 [DOI] [PubMed] [Google Scholar]
- 25.Nouraei SAR, Middleton SE, Hudovsky A, et al. A national analysis of the outcome of major head and neck cancer surgery: implications for surgeon-level data publication. Clin Otolaryngol. 2013;38(6):502-511. doi: 10.1111/coa.12185 [DOI] [PubMed] [Google Scholar]
- 26.Wolters U, Wolf T, Stützer H, Schröder T. ASA classification and perioperative variables as predictors of postoperative outcome. Br J Anaesth. 1996;77(2):217-222. doi: 10.1093/bja/77.2.217 [DOI] [PubMed] [Google Scholar]
- 27.Brennan M, Puri S, Ozrazgat-Baslanti T, et al. Comparing clinical judgment with the MySurgeryRisk algorithm for preoperative risk assessment: a pilot usability study. Surgery. 2019;165(5):1035-1045. doi: 10.1016/j.surg.2019.01.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Gawande AA, Kwaan MR, Regenbogen SE, Lipsitz SA, Zinner MJ. An Apgar score for surgery. J Am Coll Surg. 2007;204(2):201-208. doi: 10.1016/j.jamcollsurg.2006.11.011 [DOI] [PubMed] [Google Scholar]
- 29.Carmichael H, Moore A, Steward L, Velopulos CG. Disparities in emergency versus elective surgery: comparing measures of neighborhood social vulnerability. J Surg Res. 2020;256:397-403. doi: 10.1016/j.jss.2020.07.002 [DOI] [PubMed] [Google Scholar]
- 30.Kazaure H, Roman S, Sosa JA. High mortality in surgical patients with do-not-resuscitate orders: analysis of 8256 patients. Arch Surg. 2011;146(8):922-928. doi: 10.1001/archsurg.2011.69 [DOI] [PubMed] [Google Scholar]
- 31.Ghaferi AA, Birkmeyer JD, Dimick JB. Complications, failure to rescue, and mortality with major inpatient surgery in Medicare patients. Ann Surg. 2009;250(6):1029-1034. doi: 10.1097/SLA.0b013e3181bef697 [DOI] [PubMed] [Google Scholar]
- 32.Ghaferi AA, Osborne NH, Birkmeyer JD, Dimick JB. Hospital characteristics associated with failure to rescue from complications after pancreatectomy. J Am Coll Surg. 2010;211(3):325-330. doi: 10.1016/j.jamcollsurg.2010.04.025 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
eFigure 1. Derivation of Study Population
eTable 1. Primary Surgical Service
eTable 2. Hospital Mortality Rates for Each Primary Surgical Service
eTable 3. 20 Most Important Features for Estimating Hospital Mortality
eTable 4. 20 Most Important Features for Estimating Prolonged Intensive Care Unit Stay
eFigure 2. Subgroup Analyses of Associations Between Outcomes and Self-identification as Black or African American
eFigure 3. Subgroup Analyses of Associations Between Outcomes and Area Deprivation Index (ADI) National Ranks
eFigure 4. Subgroup Analyses of Associations Between Outcomes and Preoperative Do No Resuscitate (DNR) Orders
eMethods.