The Development of Tracheal Intubation Proficiency Outside the Operating Suite during Pediatric Critical Care Medicine Fellowship Training: A Retrospective Cohort Study using Cumulative Sum Analysis

Abstract

Objective

Tracheal intubation (TI) is a core technical skill for pediatric critical care medicine (PCCM) fellows. Limited data exist to describe current PCCM fellow TI skill acquisition through the training. We hypothesized both overall and first attempt TI success rates by PCCM fellows improve over the course of training.

Design

Retrospective cohort study at a single large academic children’s hospital.

Materials and Methods

The National Emergency Airway Registry for Children (NEAR4KIDS) database and local QI database were merged for all TIs outside the Operating Suite by PCCM fellows from July 2011 to January 2015. Primary outcomes were TI overall success (regardless of number of attempts) and first attempt success. Patient-level covariates were adjusted in multivariate analysis. Learning curves for each fellow were constructed by cumulative sum (CUSUM) analysis.

Results

A total of 730 TI courses performed by 33 fellows were included in the analysis. The unadjusted overall and first attempt success rates were 87% and 80% during the first 3 months of fellowship respectively and 95% and 73% respectively during the last 3 months of fellowship. Overall success, but not first attempt success, improved during fellowship training (Odds ratio for each 3 months: 1.08, CI⁹⁵ 1.01–1.17, p=0.037) after adjusting for patient level covariates. Large variance in fellow’s TI proficiency outside the Operating Suite was demonstrated with a median number of TI=26 (range: 19–54) to achieve a 90% overall success rate. All fellows who completed 3-years of training during the study period achieved an acceptable 90% overall TI success rate.

Conclusions

TI overall success improved significantly during the course of fellowship, however, the TI first attempt success rates did not. Large variance existed in individual TI performance over time. Further investigations on a larger scale across different training programs are necessary to clarify intensity and duration of the training to achieve TI procedural competency.

Introduction

Tracheal intubation (TI) is one of the core interventions in the stabilization of critically ill or injured children. It is a lifesaving procedure for many patients with respiratory failure or shock. However, adverse events are associated with TI in both adults and children.^1–10 Proficiency in airway management skills by providers is one of the key factors determining both TI success and occurrence of adverse events.^8,11 Airway management skill, including TI, is one of the Accreditation Council for Graduate Medical Education (ACGME) requirements for pediatric critical care medicine (PCCM) fellows.¹² However, there is no existing data demonstrating how TI skills are acquired overtime throughout PCCM fellowship, and the number of TIs PCCM fellows perform throughout the training. It is also unknown how many TIs are recommended for PCCM fellows to achieve competency. While Anesthesiology literature suggested 43–57 TIs^13,14, and Emergency Medicine literature suggested 74 TIs¹⁵ to achieve competency defined as 90% success, PCCM fellows might require more TI experience considering that pediatric ICU patient population includes various TI difficulty levels from a wide range of patient age, frequent difficult airway features, and urgent and emergent nature of TIs with limited apneic time allowed for TI procedure. Therefore, objective assessment of trainee’s performance specific to each specialty is desired.

Learning curves have been used with various statistical approaches to demonstrate skill acquisition process.¹⁶ Historically, improvement in provider performance was analyzed as aggregates over the period of time. At the group level, overlaid learning curves show the full variation of a group of learners’ paths through a given learning domain. Newer methods focus more on improvement in individual performance. Individual learning curves map a single person’s progress towards competence. Cumulative sum (CUSUM) method has been recently applied to generate control charts to monitor individual procedural skills.^14,17 It allows early detection of deviations from pre-established standards by monitoring the difference of the quality characteristic from the set target level leading to tighter control of a given process.¹⁸

In this study, we described the aggregate TI success rates for the PCCM fellows with the duration of training period, and also assess individual TI performance using the CUSUM method. We also sought to determine the median number of TI courses required by PCCM fellows to achieve competency for 90% success rate. We set our primary outcome as the TI overall success (provider success regardless of number of attempts) as a learning outcome by providers. We also evaluated the first attempt success as a secondary outcome because it is important from a patient safety perspective, because recent reports demonstrated that number of TI attempts on patients is closely associated with desaturation and adverse TI associated events.^6,10,19 We hypothesized that PCCM fellow’s individual and aggregate TI overall success rates improve over time, with substantial variation across individual providers.

Method

Setting

This study was conducted in a large tertiary children’s hospital in the United States with 55-bed pediatric ICU (PICU), 26-bed cardiac ICU (CICU) and a large Emergency Department (ED) with approximately 90,000 annual visits. The hospital has an ACGME accredited PCCM fellowship with a total of 15 fellows over 3 years.

Design

The study period was from July 01, 2011 to January 31, 2015. The current study utilized two databases: The National Emergency Airway Registry for Children (NEAR4KIDS) and local TI quality improvement database with Institutional Review Board approval. The NEAR4KIDS database was developed as a quality improvement tool by members of the Pediatric Acute Lung Injury and Sepsis Investigators (PALISI) Network in conjunction with the NEAR investigators in 2004.^3,10 When TI took place within our PICU, CICU and ED, the bedside airway providers completed a data collection sheet. The data accuracy was verified by designated personnel. A site-specific compliance plan and monitoring were in place to ensure >90% TI capture in the database.^3,8 The data included patient and practitioner demographics, indication, difficult airway evaluation, intubation events (including adverse TI associated events), and outcomes.^3,8–10,19 Tracheal intubation experience of PCCM fellows in the Operating Suites was collected using fellows’ self-reported procedure logs. This data was not included in CUSUM analysis due to the focus of the investigation on intubations by PCCM fellows outside the Operating Suite and lack of objective data in TI success.

Subject

Tracheal intubation data of all PCCM fellows who were in training during the study period were included in the analysis. Therefore, some fellows had TI performance data collected for only part of their PCCM fellowship training. We only included initial oral TI courses in which fellows used direct laryngoscopy. Nasal TI courses or courses performed by non-direct laryngoscopy (e.g., video laryngoscopy, bronchoscopy, cricothyrotomy, or laryngeal mask) were excluded. Elective endotracheal tube change was also excluded. Only PCCM fellows with TI data collected from the beginning of their fellowship outside the Operating Suite were included in the CUSUM analysis.

Definitions and outcome measures

The previously defined terms, “Encounter”, “Course” and “Attempt” were used to describe the TI event.^3,8,10 Briefly, an “Encounter” is defined as one sequence of completed advanced airway management (e.g., TI, laryngeal mask placement, or cricothyrotomy). A “Course” is defined as one method or approach to secure an airway (e.g., oral, or nasal) and one set of medications including premedication and induction. An “Attempt” is defined as a single advanced airway maneuver with the insertion of a device into patient mouth or nose. This study only evaluated the first course of each encounter. A “Success” was defined as successful placement of the tracheal tube in the trachea confirmed by primary and secondary methods (i.e., colorimetric end-tidal carbon dioxide detector or quantitative capnography waveform), and chest radiograph. We defined two outcome measures: overall success was defined as a successful endotracheal tube placement with one or more attempts by a single fellow, regardless of number of attempts, and first attempt success was defined as a successful tracheal tube placement on fellow’s first attempt.

Statistical analysis

Statistical analysis was performed using STATA 11.2 (Stata Corp. College station, TX). Summary statistics were described with mean and standard error of mean for parametric variables. For non-parametric variables median with interquartile range was reported. For categorical variables with a dichotomous outcome, a contingency table method was used with Chi-square test. For multivariable analysis, a logistic regression model was developed to adjust for known factors associated with difficult TI, which included a history of difficult airway, upper airway obstruction, limited mouth opening and patient age. Those covariates were chosen a priori, based on our previous literature.¹⁹ Generalized estimate equation models were developed to describe PCCM fellow provider’s learning curve over the time while accounting for covariates listed above. Training experience was analyzed with 3 months (1 quarter) as a unit. A p-value of <0.05 was considered as statistically significant.

Our CUSUM calculation and sample size estimation were performed using Microsoft Excel (Microsoft, Inc., Redmond, WA). Standard CUSUM charting requires specification of acceptable (p0) and unacceptable (p1) intubation failure rates for trainees along with type I (α) and type II (β) error. Type I error is the probability of crossing the acceptable failure rate limit when the true failure rate is not in the acceptable range. Type II error is the probability of failing to cross the acceptable limit when the true failure rate is in the acceptable range.¹⁷

The formulae used for CUSUM calculation are provided in Supplemental Digital Content - Appendix A, which is similar to those in the study by Komatsu et al.¹⁷ From values p0, p1, α and β, the variable s is calculated. The resulting chart begins at 0, and for each success, the amount s was subtracted from the previous CUSUM score. For each failure, the amount 1-s was added to the previous CUSUM score. Individual CUSUM charts were then formed by connecting these values for every TI for each fellow from the start of fellowship. A negative trend of the CUSUM line indicates consecutive success (i.e., progression towards proficiency), whereas a positive trend indicates consecutive failure (i.e., deviation away from proficiency).

When the CUSUM line crosses the upper decision limit (h1) from below, the actual failure rate is significantly greater than the acceptable failure rate (not meeting competency criteria), and when the line crosses the lower decision limit (h0) from above, the true failure rate does not differ significantly from the acceptable failure rate (meeting competency criteria). When the CUSUM line is kept between the decision limits, no statistical inference can be made.

Acceptable failure rate (p0) in this study was arbitrarily set at 10% (i.e. 90% success rate) for both overall and first attempt TI success, while unacceptable failure rate (p1) was set at 20% (2 times of p0) or 80% success rate for both outcomes.^14,15,19 The probability of type I (α) and II (β) error was set at 10%. Upper and lower decision limits (h1 and h0, respectively) corresponded to unacceptable and acceptable failure rates while integrating the type I and II errors. An average number of courses needed to achieve the acceptable failure rate (p0) of 10%, and unacceptable failure rate (p1) of 20% was estimated as 48 and 40 TI courses respectively for every individual provider.

Results

A total of 880 first TI courses were performed by 33 PCCM fellows in PICU, CICU and ED during the study period. Seven hundred twenty-one courses (82%) were included in the analysis (Figure 1). Nine courses had 2 fellows as an intubator, therefore 730 course-provider pairs were analyzed in this study.

Figure 1.

Study enrollment diagram

Twenty-three fellows had TI data available from their beginning of fellowship, while ten fellows started their fellowship before study period and had TI data from some points during their fellowship training. Four fellows had TI data for entire 3 years of fellowship. Among all 33 fellows, the median number of TI courses included in the analysis per PCCM fellow provider was 20 (interquartile range: IQR: 11–31) over 6 (IQR 4–8) quarters (18 months). Among 4 fellows who had TI data for entire 3 years of fellowship, the median number of TI courses was 42 (range:39–60). In addition, during the first 2 months of PCCM fellowship, 1^st year fellows performed TIs in the Operating Suites at median 26 (IQR 22–28) times in year 2013 and 2014.

Fellow Characteristics

The median age at the beginning of fellowship was 31 years old (IQR: 29–33). Thirty (91%) fellows completed residency training in pediatrics, 2 (6%) completed an internal medicine/pediatrics residency, and 1 (3%) completed residency in child neurology. No fellows previously completed residency training in anesthesiology or emergency medicine.

Patient Characteristics

The patient demographic data are presented in Table 1. Median age was 3 year (IQR: 0–9), median weight was 14.3 kg (IQR: 8–30). History of difficult airway was reported in 110 courses (15%). History of difficult airway was negatively associated with first attempt success (p=0.003), and sign of upper airway obstruction was negatively associated with overall attempt success (p=0.025). Neither history of difficult airway, sign of upper airway obstruction, nor limited mouth opening was associated with fellow’s experience at the time of TI course (p>0.1 for all difficult airway features).

Table 1.

Fellow experience and patient characteristics and TI success rates (n=721)

	Overall success			First attempt success
	Success	Failure	p-value	Success	Failure	p-value
Fellow training duration (quarter‡) (Median, IQR)	5 (3–9)	4 (2–7)	0.07	5 (3–9)	5 (3–7)	0.27
Age			0.41*			0.18*
< 12 month	166/185 (89.7%)	19/185 (10.3%)		144/185 (77.8%)	41/185 (10.3%)
1–7 year	296/323 (91.6%)	27/323 (8.4%)		252/323 (78.0%)	71/323 (22.0%)
≥ 8year	199/213 (93.4%)	14/213 (6.6%)		179/213 (84.0%)	34/213 (16.0%)
Difficult airway features
History of difficult airway (%)	97/110 (88.2%)	13/110 (11.8%)	0.15	76/110 (69.1%)	34/110 (30.9%)	0.003
Upper airway obstruction (%)	46/55 (83.6%)	9/55 (16.4%)	0.025	41/55 (74.6%)	14/55 (25.5%)	0.32
Limited mouth opening (%)	93/106 (87.7%)	13/106 (12.3%)	0.11	80/106 (75.5%)	26/106 (24.5%)	0.24

A total of 721 patient encounters were included (730 patient-provider pair)

A p-value for age was calculated with chi-square test.

^‡Duration from the beginning of PCCM fellowship. Each quarter equals 3 months.

Outcomes of TI courses

The average overall and first attempt success rates were 91±1% and 79±2% respectively. The average overall and first attempt success rates for the first 3 months of PCCM fellowship were 87±4% and 80±4% respectively. The average overall and first attempt success rates for the last 3 months of PCCM fellowship were 95±4% and 73±7% respectively, as shown in Figure 2. The overall success rate increased significantly during training: odds ratio 1.08 (95% confidence interval: CI; 1.00–1.16, p=0.048, unadjusted), but not first attempt success rate: unadjusted odds ratio 1.04 (95% CI: 0.98–1.09, p=0.21, unadjusted).

Figure 2.

Unadjusted overall and first attempt success rate

Multivariate analysis for TI outcomes

After adjusting for patient-level covariates, the overall attempt success was significantly associated with training duration (OR: 1.08, CI⁹⁵ 1.01–1.17, p=0.037), however, first attempt success was not (OR: 1.04, CI⁹⁵ 0.98–1.10, p=0.21), shown in Table 2. Based on these multivariate models, the adjusted first attempt and overall success rates with 95% confidence interval are presented in Table 3.

Table 2.

Multivariable analysis for overall and first attempt success

	Overall success		First attempt success
Variables	Odds Ratio (95% CI)	P-value	Odds Ratio (95% CI)†	P-value
Fellow training duration (quarter‡)	1.08 (1.01–1.17)	0.037	1.04 (0.98–1.10)	0.21
Age
Infant	baseline		baseline
1–7 year	1.42 (0.80–2.53)	0.24	1.08 (0.71–1.64)	0.71
≥ 8yr	1.80 (0.77–4.19)	0.18	1.60 (0.99–2.59)	0.054
History of difficult airway	0.75 (0.37–1.50)	0.41	0.53 (0.31–0.93)	0.025
Upper airway obstruction	0.42 (0.19–0.93)	0.033	0.75 (0.37–1.53)	0.43
Limited mouth opening	0.63 (0.29–1.37)	0.25	0.83 (0.51–1.33)	0.44

Based on multivariable logistic regression model with generalized estimated equation. Both models were statistically significant (p=0.028 for first attempt success, p=0.0006 for overall attempt success).

^†CI denotes confidence interval.

^‡Duration from the beginning of PCCM fellowship. Each quarter equals 3 months.

Table 3.

Adjusted TI success rate for each quarter since the beginning of Pediatric Critical Care Medicine fellowship

Fellow training duration		Overall success rate		First attempt success rate
		(%)	(95%CI)	(%)	(95%CI)
1st year	1st quarter	88.5	(82.3–92.7)	77.1	(71.2–82.1)
	2nd quarter	89.3	(84.2–92.9)	77.7	(72.7–82.0)
	3rd quarter	90.0	(85.9–93.1)	78.3	(74.1–82.1)
	4th quarter	90.8	(87.2–93.4)	78.9	(75.2–82.2)
2nd year	5th quarter	91.4	(88.4–93.7)	79.5	(76.0–82.6)
	6th quarter	92.0	(89.3–94.1)	80.1	(76.5–83.2)
	7th quarter	92.6	(89.9–94.6)	80.6	(76.8–84.0)
	8th quarter	93.2	(90.4–95.1)	81.2	(76.9–84.8)
3rd year	9th quarter	93.7	(90.8–95.7)	81.7	(76.9–85.8)
	10th quarter	94.1	(91.1–96.2)	82.3	(76.7–86.7)
	11th quarter	94.5	(91.3–96.6)	82.8	(76.5–87.6)
	12th quarter	95.0	(91.5–97.1)	83.3	(76.3–88.5)

Fellow training duration demonstrates the time since a fellow started Pediatric Critical Care Medicine fellowship.

Each quarter equals 3 months.

CI denotes confidence interval.

Individual performance

Twenty-three fellows had data available for CUSUM analysis from the beginning of PCCM fellowship with a total of 573 courses (median 23; IQR 11–37). Ten fellows had data collected from the middle of their fellowship training and were therefore excluded. Figure 3 shows CUSUM charts for individual fellow performance for the 7 fellows who had data for more than 2 years from the beginning of fellowship. (Appendix B displays CUSUM charts for the remainder of the 23 fellows). Eight fellows achieved acceptable failure rate for 90% overall success after the median of 26 (range 19–54) courses. All 4 fellows (A, B, C and D) who completed 3-year fellowship during the study period had median of 42 TI courses (range 39–60) outside the Operating Suites, and achieved acceptable failure rate for 90% overall success after the median of 22 (range 19–40) courses.

Figure 3.

cumulative sum (CUSUM) charts for TI overall and first attempt success

For overall success, only 2 fellows crossed upper limit line after the median of 18 (range 14–22) courses. However, none of 23 fellows achieved acceptable failure rate for 90% first attempt success. Eight fellows crossed the upper limit line for unacceptable first attempt success rate of 80% after median of 18 (range 5–32) courses.

Discussion

We described PCCM fellows’ success rates and progression towards proficiency in pediatric TI as a group and individually. Length of time within fellowship training was significantly associated with overall TI success rate, but not with first attempt success rate, even after adjusting for patient-level covariates. All fellows who completed their 3-year fellowship during the study period achieved an acceptable overall TI success rate, providing evidence of the acquisition of TI proficiency as a result of PCCM fellowship training.

The ACGME requires PCCM fellows to become proficient in critical care procedures, including TI.¹² To our knowledge this is one of the first reports to assess the learning curve of PCCM fellows in TI skills. Surprisingly, their first attempt success rate remained around 80% without statistically significant improvement over time. We suspect this may be from the fact that senior fellows experience more difficult TI courses. For instance, senior fellows might have more challenging cases for their first attempts, such as being tasked to intubate a patient anticipated to have a difficult airway or after multiple failed TI attempts by residents or junior fellows. Interestingly, there was no significant association between the difficult airway features and fellow’s experience at the time of TI course. We suspect that these difficult airway features were not sufficient to describe overall difficulty of TI encounter. Therefore, although we attempted to address this confounding by TI difficulty in our multivariate model, under-adjustment is quite possible. On the other hand, if these difficult airway features truly represent the patient difficult TI status, supervising physicians should be more cognizant to match fellow’s skills to anticipated TI difficulty to increase the chance for first attempt success (i.e., minimize multiple attempts) and adverse TI events.

Our outcomes of interest in this study were both overall TI success and first attempt success. Multiple TI attempts, even successful, have been associated with adverse TI events and desaturation,^6,10,21 which highlights the necessity of future educational interventions to improve first attempt success. On the other hand, the overall TI success is very important from fellowship training perspective as PCCM fellows will soon graduate and become independent providers. Our study demonstrated the estimated overall TI success rate at the last 3 months of fellowship was 95%, surpassing our criteria of 90% success. This provides evidence that TI proficiency was attained during PCCM fellowship training.

Using the CUSUM method, we were able to assess the individual learning process of TI skills for 23 PCCM fellows, including 4 fellows who completed 3-year fellowship. Although no fellow achieved 90% first attempt success rate, 8 fellows achieved 90% overall success rate after a median of 26 courses. This data analysis did not include intubation experience in the Operating Suite. Considering the median 26 of TIs first year PCCM fellows performed in the Operating Suites during their initial 4 weeks in fellowship, a total of 52 TI courses were needed to achieve 90% overall success.

Limited literature is available regarding number of TI courses required to achieve a pre-defined TI success rate. Oliveira utilized the CUSUM method to assess TI performance by individual first year anesthesiology residents, and showed that 80% first attempt success rate was achieved after a mean of 43±33 procedures.¹⁴ Je et al did similar studies among ED residents and reported a mean number of 75 TI’s were required to achieve 90% success rate within 2 attempts.¹⁵ Konrad et al explored recommended number of experience for TI for first year anesthesiology residents. Ninety percent success rate within 3 attempts was achieved after a mean of 57 procedures.¹³ Our PCCM fellows’ achievement of 90% overall success rate after 26 TI courses in PICU, CICU and ED (a total of 52 TIs with TIs in the Operating Suites) seems to be consistent with what have been reported in the above literature. Although our PCCM fellows could have had some TI experience prior to PCCM fellowship, recent studies suggested these are relatively small.^20,21

Our overall TI success was defined as the first provider TI success regardless of the number of attempts. Therefore, a maximum of four attempts were performed by a fellow in some cases, resulting in a less stringent determination of proficiency than the previous reports that used one,¹⁴ two¹⁵ or three¹³ attempts as the cut off. All 4 fellows who completed 3-year fellowship achieved > 90% overall TI success rates, suggesting that our current PCCM fellowship training allowed fellows adequate TI experience.

Contrary to our expectations, more fellows crossed the limit line for unacceptable failure rate than the limit line for acceptable failure rate for their first attempt success. This may be explained if fellows start with easier cases and move to more challenging cases as they get more experience. Another possible explanation is our threshold of 90% first attempt success might be too stringent. Compared to TIs in the Operating Suite, TIs in PICU clearly have more challenging factors such as frequent difficult airway features, unstable hemodynamics, existing respiratory failure, and urgent and emergent nature.³ These patient factors make TIs in PICU more difficult, however, also require high skill proficiency of providers for better patient outcomes. Therefore, appropriate criteria for PCCM fellow performance need to be further determined with the TI data from PCCM practice such as these described in this study. The CUSUM method was a promising approach for assessing an individual fellow’s performance. If used prospectively in the future, this method will allow us to visualize fellows learning process, providing the ability to give timely feedback and early remediation, if necessary.

Our study needs to be interpreted with important limitations. The study was done in the PICU at a large tertiary children’s hospital, and results may not be generalizable. The study population was small and TI data for the entire 3-year fellowship was available for only 4 fellows. A future multicenter study will need to evaluate the generalizability of these findings and set expectation for fellowship milestones in TI proficiency. This report focused only on TI experience outside the Operating Suite for PCCM fellows. We were not able to include TIs performed in the Operating Suite in our analyses. For our PCCM fellows, this included 4 weeks during the first 2 months of fellowship, followed by additional two weeks during their second and third year. We attempted to take these numbers into consideration to estimate a median number of TI to achieve 90% of overall TI success, although we lacked TI performance data in the Operating Suites for each individual fellow. Further investigations would be needed to examine the effect of TI experience in the Operating Suite on performance in the PICU, CICU and ED. We also did not collect, nor account for, TI experience before fellowship training. The number of maximum attempts allowed per fellow provider was not protocolized; therefore it was largely affected by the supervisor’s comfort level or the patient’s condition, resulting in maximum of 4 attempts in our study. Lastly our analyses did not include the adverse TI associated events during the procedure. Although these events were collected through the NEAR4KIDS registry, this was not possible because the events were not linked to each provider. Future studies should strongly consider the TI success without adverse events as the true success.

Conclusion

We described PCCM fellows’ progression towards proficiency in pediatric TI in aggregate, as well as individual performance utilizing CUSUM method. Overall TI success improved significantly during the course of training. Large variance existed in individual TI performance over time. All fellows who completed their 3-year fellowship during the study period achieved an acceptable overall TI success rate. Further investigation on a larger scale is necessary to determine the TI skill level required for fellows in order to improve current PCCM fellowship program and ensure both trainee’s skill competency and patient safety.