A Scoring System to Predict Unplanned Intubation in Patients Having Undergone Major Surgical Procedures

Abstract

Background

Unplanned tracheal intubation after surgery has been associated with high mortality. Few studies have examined the risk factors for this complication.

Methods

The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) is a multicenter, prospective, outcome-oriented database for patients having undergone major surgical procedures. Using the NSQIP data for the years 2005–2007 (n=231,548) and Cox proportional hazards modeling, identified risk factors and used them to derive a scoring system to stratify patients' risk of having an unplanned intubation outcome. NSQIP data for the year 2008 (n=176,031) were then used to validate the scoring system.

Results

The variables most predictive of unplanned intubation were patient age (0–4 points), ASA class (0–7 points), the presence of preoperative sepsis (3 points) and total operative time (0–4 points). The Unplanned Intubation Risk Index based on the adjusted hazard ratios for these variables, ranging from 0 (lowest risk) to 18 (highest risk), had a 79% accuracy in distinguishing patients requiring unplanned intubation from those not requiring [area under the receiver operating characteristic curve (AUC) 0.79, 95% confidence interval (CI) 0.79 – 0.80]. When the scoring system was applied to the validation cohort data, its discriminative performance remained virtually unchanged (AUC 0.79, 95% CI 0.79–0.80).

Conclusions

A scoring system based on clinical risk factors was able to accurately predict unplanned intubation after surgery. Further investigation is needed to assess the utility of the Unplanned Intubation Risk Index in reducing the incidence of unplanned intubation through improved risk stratification and management in perioperative care.

Introduction

Within the surgical realm, efforts have focused on identifying preventable complications to reduce postoperative mortality, which ranges between 3.5–6.9%.¹ One postoperative predictor for increased mortality is the need for unplanned tracheal intubation, which is defined as requiring postoperative placement of an endotracheal tube in the 30 days after surgery. In a study of patients undergoing general and vascular surgery, the rate of unplanned intubation was shown to be approximately 3%, but its occurrence was associated with a mortality rate ranging from 31–71%, with the most common indications for intubation being sepsis and cardiopulmonary events.² Another study of postoperative respiratory failure, which most often necessitates intubation, showed a difference in 30-day mortality of 26.5% for patients with respiratory failure versus 1.4% for patients without.³ Although unplanned intubation is associated with a sicker patient population, this by no means indicates that the complication is inevitable. Indeed, in a study of hospital mortality after surgery, the only complication that differed significantly between very-high-mortality and very-low-mortality hospitals was the rate of unplanned intubation (4.6% versus 3.6% respectively).¹

The high mortality rate associated with unplanned intubation underscores the importance of more objective, clinical data-based risk stratification and management. Studies quantifying the risk of postoperative respiratory complications have either partially captured unplanned intubation patients (i.e., analyses of patients age > 65 years or only early unplanned intubation) or included them into a larger patient population of postoperative respiratory failure, and it is not clear that their results can be applied to all patients at risk for unplanned intubation.^3–6 Prior studies of unplanned intubation identified several major risk factors including chronic obstructive pulmonary disease (COPD), dependent functional status, emergent operation and reoperation.^2,5 Lastly, a recent study evaluated risk factors for unplanned intubation, but their analysis was limited to events occurring within the first three postoperative days.⁶ Thus, our study aims to quantify risk factors associated with unplanned intubation and develop a valid and practical tool based on the identified risk factors for assessing the likelihood of requiring unplanned intubation in patients undergoing major surgical procedures.

Methods

Patients and Data Collection

The study protocol was reviewed and approved by the IRB of Columbia University Medical Center (New York, NY). Written informed consent was waived. Data for this study came from the multicenter, prospective, outcome-oriented database of the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) for the years 2005–2008.^a These data were collected from 251 participating hospitals for patients who underwent major surgical procedures. Major surgical procedures included any case performed under general, spinal or epidural anesthesia, as well as the following procedures regardless of anesthetic technique: carotid endarterectomy, inguinal herniorrhaphy, parathyroidectomy, thyroidectomy, breast lumpectomy and endovascular abdominal aortic aneurysm repair. In order to participate, hospitals must submit a minimum of 900 cases annually. Surgical cases are sampled in 8-day cycles, with the first 40 consecutive general or vascular cases performed under general, spinal and epidural anesthesia included. Excluded from the ACS NSQIP were cases performed under monitored anesthesia care, peripheral nerve block or local anesthesia, patients younger than 16 years, trauma cases, and transplant cases. No more than three breast lumpectomies, inguinal herniorrhaphies, laparoscopic cholecystectomies, transurethral resections of the prostate or transurethral resections of the bladder are included in any 8-day sampling period, because these are considered to be low-risk but high-volume cases. For subspecialty surgeries, NSQIP samples using both a high-volume model, where hospitals must submit 20% of their cases, and a low-volume model, where a minimum of 900 cases are submitted annually. Gynecologic, neurologic, orthopedic, otolaryngologic, plastic, cardiac, thoracic, urologic and vascular surgeries are included. In each participating hospital, a trained surgical nurse abstracted information for 135 variables, including demographic characteristics, preoperative and intraoperative variables and 30-day postoperative morbidity and mortality outcomes from medical records using standard protocols. Case selection and case mix is monitored weekly to ensure proper sampling. Further detailed information regarding the database and its methods has been published.⁷

There are several quality assurance measures to ensure that only data of the highest quality are recorded in the participant use data file. Hospitals with a 30-day follow-up rate under 80% and whose surgical volume does not meet eligibility criteria are excluded. Furthermore, the consistency in data recording and reporting is checked with the Inter-Rater Reliability Audit, which is a process involving the review of 20 charts, with some cases selected randomly and some cases selected based on predetermined criteria; an inter-rater agreement rate of 95% or more is deemed acceptable. Combined results of the audits for the 2005–2008 data revealed an inter-rater agreement rate of 98%.^b

These data comprise the ACS NSQIP participant data use file. After exclusion of patients with preoperative ventilator dependence (n = 3,901) and outpatients (n = 128,488), the derivation cohort consisted of 231,548 patients in the NSQIP database for the years 2005–2007. After applying the same exclusion criteria, data for the year 2008 were used as the validation cohort (n = 176,031).

Unplanned intubation was the primary outcome measure, which is operationally defined in the NSQIP database as requiring placement of an endotracheal tube secondary to the onset of respiratory or cardiac failure as evidenced by severe respiratory distress, hypoxia, hypercarbia or respiratory acidosis within 30 days of the operation. For patients who were intubated for surgery, any intubation after extubation was considered an unplanned intubation event; in patients who were not intubated during surgery, any postoperative intubation was considered to be unplanned.

Variable Selection

Variables thought to be predictive of the primary outcome were broadly selected based on the methods of prior studies of postoperative respiratory complications.^2,3–6 Demographic variables included age, race and gender. Lifestyle variables included alcohol use (defined as > 2 drinks per day in the 2 weeks before admission) and smoking (current smoking within one year of surgery). General factors included ASA classification (ASA 1 - normal healthy patient, ASA 2 - patient with mild systemic disease, ASA 3 - patient with severe systemic disease, ASA 4 -patient with severe systemic disease that is a constant threat to life, ASA 5 - moribund patient who is not expected to survive without the operation), transfer status (admitted from home, acute care facility or chronic care facility), functional status (independent, partially dependent or totally dependent), emergency status and body mass index. Laboratory values included preoperative hematocrit, white blood cell count, platelet count, serum sodium blood urea nitrogen, creatinine, albumin, bilirubin, serum glutamic oxaloacetic transaminase, prothrombin time and partial thromboplastin time.

Preoperative comorbidities included in the NSQIP database were recoded into a comorbidity index modified from the Charlson comorbidity index.⁸ The following comorbidities were assigned a score of 1: history of COPD, history of chronic heart failure, history of myocardial infarction, peripheral vascular disease, any diabetes and cerebrovascular disease. Dialysis, patients with radiation and chemotherapy without disseminated cancer and hemiplegia were coded as a 2. Patients with ascites received a score of 3 and patients with disseminated cancer received a 6. A comorbidity score was tabulated for each patient. Other preoperative comorbidities included in the model that were not part of the Charlson index were sepsis (which includes the systemic inflammatory response syndrome, sepsis, severe sepsis and septic shock), dyspnea (at rest, moderate or with exertion), and weight loss (defined as unintentional loss of 10% of body weight in the 6 months before surgery), chemotherapy, and transfusion requirement (requiring > 4 units of packed red blood cells in the 72 hours before surgery).

With regards to surgical variables, the Current Procedural Terminology codes were identified and grouped by surgical specialty. Because there is an increased incidence of respiratory complications with incisions in closer proximity to the diaphragm,^4,9 general surgery was separated into abdominal and nonabdominal categories. Vascular surgery was separated into abdominal and nonabdominal cases as well as an endovascular category. Total operative time was defined as surgical start to surgical stop. This variable was highly correlated with all other times reported in the NSQIP database (duration from anesthesia start to surgery start, duration from surgery stop to anesthesia stop, duration patient is in room and duration of anesthesia). To determine categories for total operative time, groups were broken down into 60-minute intervals and groups with like odds ratios grouped together.

Statistical Analysis

To develop a scoring system, predictors of unplanned intubation were identified as risk factors for unplanned intubation using χ² tests. A Cox Proportional Hazards model was used to determine association and strength of independent predictors of time until unplanned intubation. Risk factors statistically significant at an α of 0.05 in univariate tests and potential confounders (age, sex, race) identified a priori were evaluated in Cox models. In order to simplify the scoring system, variables with the highest p-values were sequentially eliminated and at each step, the ROC curve was assessed to maintain the model's discriminative power. A scoring system was created based on hazard ratios (HR) from the final Cox model. Points for each category were generated as follows: a HR between 1.00 and 1.20 was dropped, while a HR between 1.21 and 1.49 for a given variable was given 1 point, and HR of 1.50–2.49 would yield 2 points, and so forth. Points for each variable were summed to create a total score. A logistic regression model of unplanned intubation status as predicted by the total score generated above was used to generate a Receiver Operator Characteristics (ROC) curve. Total score was modeled as a continuous variable. The performance of the scoring system was then validated using the NSQIP data for the year 2008. Discrimination of the model was assessed using the c-statistic and ROC curve. Calibration of the model was assessed using the Hosmer-Lemeshow goodness-of-fit-test. The overall performance of the model was assessed using the Brier score.¹⁰ Statistical analysis was performed using SAS version 9.2 (SAS Institute, Cary, NC) and R version 2.14.0 (2011-10-31 Copyright (C) 2011 The R Foundation for Statistical Computing).

Results

In the derivation cohort, 5,028 (2.2%) patients had an unplanned intubation event. Unadjusted 30-day mortality was 28.1% for patients who experienced unplanned intubation versus 1.5% for those who did not (p < 0.0001). Approximately 50% of the unplanned intubations occurred within the first 3 days, and 70% of the unplanned intubation events occurred within the first 7 days after surgery. In the validation cohort, there were 3,327 unplanned intubations of 176,031 patients (1.9%). The incidence rate of unplanned intubation in the validation cohort was significantly lower than in the derivation cohort (p < 0.001). Thirty-day mortality was similar to that of the derivation cohort (28.0% for patients who experienced unplanned intubation versus 1.5% for those who did not, p < 0.0001). Patient characteristics in the derivation and validation cohorts were similar with regard to demographic makeup, ASA classification, comorbidity, transfer status, and surgical variables (Table 1).

Table 1.

Baseline Characteristics of the Derivation and Validation Cohorts, American College of Surgeons National Surgical Quality Improvement Program, 2005–2008

	Derivation Cohort (Years 2005–2007)		Validation Cohort (Year 2008)
	Frequency	Percent	Frequency	Percent
Sexc
Female	130,407	56.32	100,305	57.02
Male	101,126	43.67	75,617	42.98
Race
White, Not of Hispanic Origin	166,219	71.79	129,870	73.82
Black, Not of Hispanic Origin	23,350	10.08	17,954	10.21
Hispanic	16,123	6.96	10,708	6.09
Asian or Pacific Islander	4,264	1.84	3,591	2.04
Other/Unknown	21,592	9.33	13,802	7.85
Age, (years)*
16 – 39	41,836	18.07	29,542	16.79
40 – 49	36,617	15.81	26,671	15.16
50 – 59	46,565	20.11	35,431	20.14
60 – 69	45,218	19.53	36,056	20.5
70 – 79	38,654	16.69	29,621	16.84
≥80	22,656	9.78	18,604	10.57
ASA Classification*
1	15,768	6.81	10,796	6.14
2	92,072	39.76	68,624	39.01
3	103,929	44.88	80,202	45.59
4	19,098	8.25	15,654	8.9
5	529	0.23	407	0.23
General Anesthesia*
No	9,221	3.98	8,532	4.85
Yes	222,315	96.01	167,376	95.14
Surgery Type
General Surgery	182,625	78.87	122,734	69.76
Vascular	38,495	16.63	27,587	15.68
Other	10,428	4.5	25,604	14.55
Emergency case
No	191,751	82.81	146,966	83.54
Yes	39,797	17.19	28,959	16.46
Transfer Status
Acute Care Hospital	6,218	2.69	4,109	2.34
Admitted directly from home	221,078	95.48	168,309	95.67
Chronic Care Facility	3,386	1.46	2,878	1.64
Other	866	0.37	629	0.36
Dyspnea
No	199,299	99.86	152,484	99.87
Yes	32,249	13.92	23,441	13.32
>10% loss body weight in last 6 months
No	222,928	96.28	170,415	96.87
Yes	8,620	3.72	5,510	3.13
Systemic Inflammatory Response Syndrome, Sepsis, Severe Sepsis or Septic Shock
No	202,699	87.54	156,362	88.88
Yes	28,849	12.46	19,563	11.12
Modified Charlson Comorbidity Index
0	150,258	64.89	115,654	65.74
1	43,594	18.83	34,795	19.78
2–3	25,182	10.88	17,477	9.93
4–5	4,853	2.1	2,993	1.7
≥6	7,661	3.31	5,006	2.85
Total operation time, (minutes)*
0 – 119	125,778	54.32	97,058	55.17
120 – <299	90,773	39.2	67,617	38.44
300 – 359	6,825	2.95	5,167	2.94
≥360	8,153	3.52	6,047	3.44

^cTotals within variables may vary due to missing data.

Development and Validation of the Unplanned Intubation Risk Index

Variables that were significantly associated with unplanned intubation included age in years (grouped 16–29 and in 10 year increments), general anesthesia, surgical specialty, emergency status, transfer status, modified Charlson comorbidity index, total operative time (< 120 minutes, 120–299 minutes, 300–359 minutes, ≥ 360 minutes), surgical procedure type (cardiothoracic, vascular abdominal and other), ASA classification, preoperative weight loss, any preoperative sepsis and any preoperative dyspnea (Table 2).

Table 2.

Independent Predictors of Unplanned Intubation, American College of Surgeon National Surgical Quality Improvement Program, 2005–2007

Variable	Adjusted Hazard Ratio	95% Confidence Interval
Age, (years)
16–29	0.80	0.60–1.06
40–49	1.27	1.05 – 1.54
50–59	1.66	1.39 – 1.99
60–69	2.15	1.81 – 2.57
70–79	2.67	2.26 – 3.21
≥80	3.30	2.76 – 3.94
General Anesthesia	2.12	1.77 – 2.55
Emergency status	1.64	1.52 – 1.76
Transfer status	1.43	1.31 – 1.55
Comorbidity score
1	1.13	1.05 – 1.22
2–3	1.36	1.25 – 1.47
4–5	1.83	1.62 – 2.06
≥6	1.74	1.56 – 1.95
ASA class
2	2.19	1.51 – 3.19
3	6.16	4.25 – 8.91
4–5	9.75	6.70– 14.17
Preoperative weight loss	1.63	1.48 – 1.80
Any sepsis	2.09	1.95 – 2.25
Any dyspnea	1.46	1.37 – 1.56
Cardiothoracic surgery	1.84	1.64 – 2.06
Vascular abdominal surgery	1.75	1.57 – 1.94
Total operative time, (minutes)
120 – 259	1.56	1.47 – 1.66
300 – 359	2.52	2.22 – 2.85
≥360	3.57	3.23 – 3.95

The four variables most predictive of unplanned intubation (age, ASA class, preoperative sepsis and total operative time) were retained in the parsimonious model to create the unplanned intubation risk index (UIRI) (Table 3). Starting at age 40 years, each age decile was associated with increased hazard for unplanned intubation, with patients age > 80 years having the greatest hazard (adjusted HR 3.93, 95% confidence interval (CI) 3.38 – 4.58). With regards to the presence of preoperative comorbidities, the hazard of unplanned intubation was increased for ASA class 3 patients (adjusted HR 3.46, 95% CI 3.15 – 3.79), ASA class 4–5 patients (adjusted HR 7.74, 95% CI 6.63 – 8.14) and patients with sepsis (adjusted HR 2.81, 95% CI 2.64 – 3.00). Finally, total operative time significantly increased the hazard for unplanned intubation, with the greatest hazard occurring for patients with operative times more than 360 minutes (adjusted HR 4.00, 95% CI 3.63 – 4.41). Based on these adjusted HRs, the UIRI was created to measure the risk of unplanned intubation within the first 30 postoperative days (Table 3). Ranging from 0 (lowest risk) to 18 (highest risk), the UIRI had a 79% accuracy in distinguishing patients who did and did not require unplanned intubation (area under the receiver operating characteristic curve (AUC) 0.79, 95% CI 0.79 – 0.80) (Figure 1). Adding other risk factors to the scoring system did not improve the performance of the scoring system to any meaningful degree. When the risk index was applied to the validation cohort data, its diagnostic performance remained virtually unchanged (AUC 0.79, 95% CI 0.79 – 0.80) (Figure 1). The incidence of unplanned intubation increased progressively with the UIRI score for both the derivation cohort and the validation cohort (Figure 2). With regards to calibration of the model, the Hosmer-Lemeshow goodness of fit test was significant (p < 0.0001); however, this test has been shown to perform poorly with large sample sizes.^11–13 We measured the overall model performance, based on the overall model discrimination and calibration, using the Brier score, where a score of 0 indicates a perfect model and a score of 0.25 indicates a noninformative model. The Brier score was 0.021, indicating good model performance.

Table 3.

Selected Independent Predictors of Unplanned Intubation and The Unplanned Intubation Risk Index, American College of Surgeons National Surgical Quality Improvement Program, 2005–2007

Variable	Estimated Hazard Ratio	95% Confidence Interval	Points Allotted
Age, (years)*
16 – 39	1.00	-	0
40 – 49	1.47	1.24 – 1.74	1
50 – 59	2.01	1.72 – 2.34	2
60 – 69	2.67	2.30 – 3.09	3
70 – 79	3.33	2.87 – 3.87	3
80+	3.93	3.38 – 4.58	4
ASA Class*
1 – 2	1.00	-	0
3	3.46	3.15 – 3.79	3
4 – 5	7.34	6.63 – 8.14	7
Any Sepsis*
No	1.00	-	0
Yes	2.81	2.64 – 3.00	3
Total Operative Time, (minutes)*
< 120	1.00	-	0
120 – 259	1.61	1.51 – 1.71	2
300 – 359	2.74	2.42 – 3.10	3
≥360	4.00	3.63 – 4.41	4

^*p <0.0001

Figure 1. The Unplanned Intubation Risk Index.

The performance of the Unplanned Intubation Risk Index was assessed for both the derivation and the validation cohorts using a Receiver Operating Characteristic curve and quantified as the area under the curve.

Figure 2. Percentage of Patients with Unplanned Intubation by the Unplanned Intubation Risk Index (UIRI) Score.

The percentage of patients who experienced unplanned intubation for each scoring category in the UIRI is shown for both the derivation and validation cohorts. T bars indicate standard error. *Scores 17 and 18 were combined because of the small numbers of unplanned intubation cases in the UIRI 18 group (2 in the derivation cohort and 1 in the validation cohort).

Discussion

In this study of patients having undergone major surgical procedures, we confirm that unplanned intubation occurs at a rate of approximately 2% and is associated with heightened mortality. Furthermore, we have developed a scoring system to identify patients at greatest risk for this complication. The UIRI accurately predicts a patient's likelihood of having an unplanned intubation event within 30 days of a surgical procedure. A previous analysis of unplanned intubation in surgical patients age > 65 years identified numerous risk factors with modest levels of association, with reoperation being the most significant one, but did not create a scoring system. A recent study by Ramachandran et al. improved upon this previous effort; after identifying 17 risk factors, the presence of individual risk factors were added together to separate patients into risk classes.⁶ However, their analysis focused on early unplanned intubation within the first 3 postoperative days, which only accounts for half of the population at risk. Although the risk of unplanned intubation is highest in the early postoperative period, 50% of all events occur after the first 3 postoperative days, with events still occurring weeks after surgery (Figure 3). The novel features of the UIRI are its inclusiveness, its good discrimination and its simplicity, which ultimately makes it more useful in a clinical setting.

Figure 3. Number of Unplanned Intubations by Postoperative Day.

The number of unplanned intubation events occurring on each postoperative day for the cohort is shown.

We found that age, ASA class, the presence of preoperative sepsis and total operative time were the factors that were most predictive of unplanned intubation. High ASA class was associated with the greatest hazard, followed by total operative time > 6 hours and age > 80 years. While it is intuitive that sicker patients are at higher risk for this complication, the UIRI highlights the factors that matter most for risk stratification of the “sick patient.” Furthermore, several variables which have previously been associated with increased frequency of postoperative respiratory complications did not significantly improve the discriminative power of the UIRI. Preoperative comorbidities, laboratory abnormalities and the type of surgical procedure have been identified as significant risk factors for respiratory failure.^3,4 Although the modified Charlson comorbidity index and type of surgery were both associated with a significantly increased hazard of unplanned intubation, their inclusion into the risk index did not add significantly to its discriminative power. This is likely because the inclusion of ASA class, which is a clinician's assessment of a patient's comorbidities and overall health status, was able to capture the effect of major comorbid conditions.

The risk of unplanned intubation increases linearly with the UIRI score; thus, the UIRI can be used to prognosticate for individual patients and may be useful for both clinicians and patients to better understand the risks inherent in surgery and the postoperative period. For patients at high-risk for unplanned intubation, clinicians may choose to intensify monitoring or optimize pulmonary function in the perioperative period. The UIRI can aid planning and allocation of the necessary resources for this monitoring (e.g., ensuring adequate staffing for recovery room nursing and respiratory therapists, ensuring the availability of monitored intensive care unit or step-down beds, and planning for surgical admissions and surgical scheduling). However, the UIRI should not be used in isolation to determine whether or not patients should remain intubated after surgery. The decision to extubate at the end of surgery should balance the risk of unplanned intubation with the risks of prolonged mechanical ventilation, such as ventilator-associated pneumonia, sepsis and a longer period of immobility.

Most importantly, the UIRI can also be used for risk adjustment of rates of unplanned intubation across the different contributing NSQIP hospitals. In 2009, an analysis by Ghaferi et al. of variations in hospital mortality found that unplanned intubation was the only postoperative complication whose incidence was greater in very-high-mortality hospitals in comparison to very-low-mortality hospitals.¹ Use of the UIRI would help determine if hospitals with higher rates of unplanned intubation were simply taking care of sicker patients or if they had an appropriate UIRI-adjusted rate of unplanned intubation. This information could be used at an individual hospital level for quality assurance purposes.

There are several limitations to our study. First, although the method of data collection in the NSQIP has been well validated,¹⁴ cases of unplanned intubation may have been missed. Second, the components of the UIRI are not modifiable risk factors, and thus, they themselves are not appropriate targets for interventions. Also, in choosing to simplify the risk index to make it more convenient to use, we lost some discriminative power by not including every possible risk factor for unplanned intubation; our AUC, at 0.79, has only moderate diagnostic accuracy.¹⁵ Although we have simplified the risk index, its relative complexity (4 variables with different point values assigned) may limit its clinical usefulness unless it is automatically computed and incorporated into the electronic medical records. Finally, participation in the NSQIP is voluntary and thus the study sample is unlikely representative of all surgical patients. As a result, it might be unwise to extrapolate the findings of this study into other study populations and geographical regions.

Nevertheless, the UIRI developed in this study appears to be a valuable tool for accurately predicting unplanned intubation after surgery and may potentially be used for preoperative risk stratification. Further research is needed to refine this risk index and assess its utility in improving the management of unplanned intubation and related adverse consequences in the perioperative setting.