Do you see what I see? A randomised pilot study to evaluate the effectiveness and efficiency of simulation-based training with videolaryngoscopy for neonatal intubation

Abstract

Introduction

Direct laryngoscopy (DL) and airway intubation are critical for neonatal resuscitation. A challenge in teaching DL is that the instructor cannot assess the learners’ airway view. Videolaryngoscopy (VL), which allows display of a patient's airway on a monitor, enables the instructor to view the airway during the procedure. This pilot study compared deliberate practice using either VL with instruction (I-VL) or traditional DL. We hypothesised that I-VL would improve the efficiency and effectiveness of neonatal intubation (NI) training.

Methods

Participants (students, paediatric interns and neonatal fellows) were randomised to I-VL or DL. Baseline technical skills were assessed using a skills checklist and global skills assessment. Following educational sessions, deliberate practice was performed on mannequins using the Storz C-MAC. With I-VL, the instructor could guide training using a real-time airway monitor view. With DL, feedback was based solely on technique or direct visual confirmation, but the instructor and learner views were not concurrent. During summative assessment, procedural skills checklists were used to evaluate intubation ability on a neonatal airway trainer. The duration of attempts was recorded, and recorded airway views were blindly reviewed for airway grade. ‘Effectiveness’ reflected achievement of the minimum passing score (MPS). ‘Efficiency’ was the duration of training for learners achieving the MPS.

Results

58 learners were randomised. Baseline demographics were similar. All participants had a significant improvement in knowledge, skills and comfort/confidence following training. There were no significant differences between randomised groups in efficiency or effectiveness, but trends towards improvement in each were noted. Fellows were more likely to achieve ‘competency’ postinstruction compared to non-fellows (p<0.001).

Conclusions

This educational intervention to teach NI increased the learner's knowledge, technical skills and confidence in procedural performance in both groups. I-VL did not improve training effectiveness. The small sample size and participant diversity may have limited findings, and future work is indicated.

Introduction

Neonatal intubation (NI) is a critical skill for paediatric providers, yet clinical success rates are poor.^1–3 A recent multisite study described a 58% failure rate for all providers (80% for residents and 30–56% for fellows).⁴ This is a matter of concern as repeated intubation attempts may lead to airway trauma,⁵ and prolonged attempts are associated with physiological deterioration, morbidity and mortality.^{6 7} The most common causes of failure involved difficulty visualising the airway (25%), patient decompensation during prolonged attempts (22%) and oesophageal intubation (19%). These three factors contributed to 66% of failures and are all related to an inadequate view during direct laryngoscopy (DL).⁴ Therefore, if providers’ ability to visualise the airway is improved, success rates should increase.

Effectively teaching visualisation using traditional methods can be challenging. Supervision of learners has customarily involved the presence and verbal support of an expert instructor. Feedback is provided based on reported observations of the learner, rather than independent visual confirmation.⁸ It is virtually impossible to know what the learner is seeing in real time during DL, and this challenge is enhanced given the small size of the neonatal airway.

Given the high NI failure rates, efforts to improve training remain a high priority. Videolaryngoscopy (VL) utilises a fibre-optic camera embedded in the laryngoscope blade to project a real-time magnified image of the airway on a monitor. VL allows instructors to guide learners using concurrent visualisation.^{8 9} VL with instruction (I-VL) may address the challenges of inadequate visualisation. The Storz C-MAC (Karl Storz, Tuttingen, Germany) is a VL device with a blade very similar to a Miller blade and a monitor view that is comparable to what is seen when performing DL.^{8 10} This device can either be used to intubate directly (in the same manner as DL) or indirectly, utilising the image on the screen. The instructor can assess the learner's real-time view of the airway, and provide formative feedback to adjust the technique accordingly. This technology can be used to facilitate NI skill development during simulated experiences, and ideally these skills will translate into improved visualisation and increased success rates during clinical intubations.

The current pilot study aimed to compare a standardised training session utilising best educational practice including deliberate practice (DP) using either I-VL or DL. During DP, learners received immediate formative feedback from an instructor with the goal of achieving a predetermined level of performance. We hypothesised that using I-VL to concurrently see the learners’ real-time airway view would allow the instructor to better guide skill development, thus improving the efficiency and effectiveness of NI training compared to traditional DL methods.

Methods

The Human Investigations Committee at Yale University determined that the present study was exempt from review (HIC# 1205010282). A convenience sample of physicians-in-training at various levels (3rd year medical students, paediatric interns and first year neonatal-perinatal medicine fellows) was recruited to participate in this study between July 2012 and June 2013. There were no additional exclusion criteria. Yale medical students were enrolled during their paediatric clerkship and paediatric interns during their neonatal intensive care unit rotation. Neonatal fellows participated during the regional neonatal boot camp (RNBC) attended by all first year fellows from the participating programmes (from Yale, University of Pennsylvania, St. Christopher's Hospital, Thomas Jefferson University, Pennsylvania State University, University of Maryland, Children's National Medical Center, Johns Hopkins University, West Virginia University and University of Medicine and Dentistry of New Jersey).

Prior to the training sessions, learners watched the New England Journal of Medicine Videos in Clinical Medicine on Orotracheal Intubation¹¹ to introduce the airway anatomy and basic procedural technique. Training was facilitated by one of two investigators (LCJ or CB) with extensive experience in neonatal and infant airway management. Learners gave verbal consent for participation, and underwent block randomisation stratified by level of training (medical student, paediatric intern and neonatal fellow). Group assignment, intervention (I-VL) or control (DL) was dictated using a random number generator programme (figure 1). All learners completed an initial 13 question demographic survey, which included questions about prior training or experience with NI, frequency of exposure to video games (due to potential impact on hand-eye coordination), as well as their baseline attitudes (comfort and confidence) regarding performance of NI in a term infant rated on a four-point scale (strongly agree, agree, disagree and strongly disagree). Learners answered nine baseline knowledge questions about airway anatomy and technical aspects of the NI procedure. Next, learners demonstrated their baseline NI technique on a bench-top airway trainer (Neonatal Resuscitation Baby (Laerdal Medical, Wappinger Falls, New York, USA)) without any coaching or assistance. The instructor scored this performance using a Provider Skills Checklist and Global Skills Assessment (GSA) for NI (see online supplementary appendix A). This 11-point Provider Skills Checklist and GSA were developed by five attending neonatologists for the study by Haubner et al⁴ in 2010 using a modified Delphi process.

Figure 1.

Randomisation strategy. Participants (students, paediatric interns and neonatal fellows) were randomised to VL with instruction (I-VL) or direct laryngoscopy (DL). Baseline technical skills were assessed using a skills checklist and global assessment. Following educational sessions, deliberate practice was performed on mannequins using the Storz C-MAC. With I-VL, the instructor could guide training using a real-time airway monitor view. With DL, feedback was based solely on technique or direct visual confirmation, but the instructor and learner views were not concurrent.

After the baseline skill assessment, a scripted educational session started (see online supplementary appendix B). First, the instructor reviewed the key steps in the intubation procedure (equipment selection, preparation, patient positioning and discrete procedural steps of laryngoscopy and endotracheal tube (ETT) insertion). Expert modelling was provided to demonstrate proper technique, and learners were provided with clear objectives for expected performance. Learners engaged in a process of active learning through coached DP. Learners reflected on their own performance and received specific formative feedback from the instructor.^12–14 Feedback compared observed performance to the predefined standard and was given with the intent to improve performance.¹⁵ Training was modified to the learners’ needs in an iterative process of practice, feedback and correction. Using this strategy, all learners would ideally be expected to continue training until the uniform educational result of competency was achieved, but the time required for each individual to reach that end point is variable.^16–18 The overall duration of session was timed by a research assistant and included scripted teaching, expert modelling and DP. Learners were informed that they had a maximum of 20 min for the session (predefined based on β-testing of the intervention). They could continue coached DP until either they reached a point where they felt confident in their skills or the 20 min elapsed.

Both groups of learners utilised the VL device and all NI attempts were videorecorded. In the intervention group (I-VL), the instructor was able to concurrently view the monitor and could provide immediate formative feedback based on the real-time airway view and observed psychomotor technique. Learners focused primarily on their direct airway view, but viewed the monitor image to confirm their understanding of the anatomy or proper procedural technique. Learners in the control group participated in the ‘traditional teaching method’ using DL, where the instructor provided feedback based on assessment of external procedural technique and the learner's reported observations. The instructor would intermittently confirm the airway view obtained during the learners’ DL attempt, but the VL monitor was concealed so that neither the learner nor instructor could view it. Following each NI attempt, the learner reported their airway view at the time the ETT was being inserted compared to the standardised images of the Cormack-Lehane (C-L) scale. A single attending neonatologist (OLL), blinded to the randomisation group, scored the recordings of each learner's final intubation attempt using the C-L scale.¹⁹ An assessment form (see online supplementary appendix C) was completed by a second investigator to document the total number of attempts and duration of each attempt. The duration of an intubation attempt was defined as the time (seconds) from insertion of the laryngoscope into the mouth until its removal. A successful intubation was defined as an ETT placed in the airway between the vocal cords that could be used to provide ventilation.

After completion of the educational session, the learners underwent summative assessment. This included a repeat skills assessment with the procedural skills checklist and GSA, which were completed in real time by the instructor. ‘Effectiveness’ of training was assessed by whether or not learners achieved the minimum passing score (MPS) that defined competency during summative assessment (yes/no). To achieve the MPS, a learner must have successfully completed all 11 steps on the skills checklist, achieved a rating of ‘5’ or ‘mastery’ on the GSA, completed their final intubation attempt in <30 s, and achieved a grade I C-L score on blinded review. ‘Efficiency’ of training was measured as the duration of training sessions for providers achieving competency. The nine-item knowledge questionnaire was repeated, and the proportion of correctly answered questions was measured before and after training for each learner. Attitudes towards intubation were reassessed using the same four-point scale.

Statistical analysis was performed using SPSS (V.22.0, IBM, Chicago, Illinois, USA). The clinical experience surveys were compared using the Pearson's χ² statistic, the Student's t test and the Wilcoxon-Mann-Whitney U two-sample test where appropriate, with a significance level of p=0.05. Median values with IQR are reported for non-parametric data, and mean values with SDs are reported for normally distributed data. To test agreement between the self-reported C-L level and blind reviewer C-L level, a κ statistic was calculated. Differences in baseline characteristics between the two groups were evaluated with Pearson's χ² test and the Wilcoxon-Mann-Whitney U two-sample test.

Results

A total of 58 participants were randomised and completed the study intervention (table 1). No participants dropped out of the study after enrolment, and all completed training in their assigned study group as per the study protocol. Of note, all 31 neonatal fellows were randomised prior to study enrolment, but only 24 were eligible and completed the training due to time limitations. There was no significant difference noted between the control and intervention groups with respect to any of the provider factors.

Table 1.

Baseline characteristics of study participants

	Randomisation
	Control (DL) n=25 (%)	Intervention (VL) n=33 (%)	p Value*
Gender			0.769
Male	6 (25.0)	10 (30.3)
Female	18 (75.0)	23 (69.7)
Training			0.738
Fellow	9 (36.0)	15 (45.5)
Resident	9 (36.0)	11 (33.3)
Student	7 (28.0)	7 (21.2)
Previous NRP training	18 (72.0)	26 (78.8)	0.550
Previous PALS training	18 (72.0)	26 (78.8)	0.550
Prior intubation exposure	24 (96.0)	31 (93.9)	0.726
Prior videolaryngoscopy experience	10 (41.7)	13 (40.1)	0.938
Self-reported level of understanding to intubate infant (of 4)	1.7±0.8	1.9±0.8	0.245
Comfort level performing intubation on an infant (of 4)	1.3±0.9	1.5±0.9	0.391
Confidence level performing intubation on an infant (of 4)	1.2±0.9	1.3±1.0	0.496
Video gaming	3 (12.0%)	11 (33.3%)	0.060

*p Values were calculated using Pearson's χ² test (ratios) and the Wilcoxon-Mann-Whitney U two-sample test (non-parametric data).

DL, direct laryngoscopy; NRP, neonatal resuscitation; PALS, Pediatric Advanced Life Support; VL, videolaryngoscopy.

Effectiveness of knowledge acquisition

Table 2 and figure 2 report the improvement in knowledge, before and after the training session and between the intervention and control groups. For all participants, the proportion of knowledge questions answered correctly before and after training increased significantly (p<0.001). However, there was no difference in this change between the intervention and control groups (p=0.552).

Table 2.

Variables presimulation and postsimulation-based training session

	Randomisation
	Control (DL) n=25 (%)	Intervention (VL) n=33 (%)	p Value*
Knowledge acquisition median (IQR)
Pretraining	89.0 (44.0)	89.0 (33.0)	0.957
Post-training	100.0 (0.0)	100.0 (5.0)	0.552
Pretraining vs post-training	–	–	<0.001
Effectiveness
Psychomotor skill acquisition (achieving mastery)
Pretraining	0/25 (0.0)	0/33 (0.0)	–
Post-training	6/25 (24.0)	15/33 (45.5)	0.092
Fellows	5/9 (55.6)	11/15 (73.3)	0.371
Residents	1/9 (11.1)	3/11 (27.2)	0.369
Students	0/7 (0.0)	1/7 (14.3)	0.299
Pretraining vs post-training	–	–	<0.001
Efficiency
Median time of psychomotor skill acquisition (IQR)	1180 (197)	1128 (236)	0.279
Fellows	832 (205)	759 (317)	0.411
Residents	1200 (31)	1200 (60)	0.822
Students	1200 (18)	1170 (63)	0.165
Mean number of intubation attempts	5.8±1.5	5.8±1.3	0.896
Median CL score postintervention (IQR)	2.0 (1.0)	2.0 (1.0)	0.279

*p Values were calculated using Pearson's χ² test (ratios), the Student's t test (normal continuous variables), the Wilcoxon-Mann-Whitney U two-sample test (non-parametric data) and the Kruskal-Wallis one-way analysis of variance test (non-parametric data).

DL, direct laryngoscopy; VL, videolaryngoscopy.

Figure 2.

Effectiveness outcomes before and after simulation. Differences in participants’ knowledge (black) and mastery (grey) are shown. There were significant improvments in knowledge and mastery (p<0.001), and there was a non-significant increase in percentage of learners achieving mastery in the intervention group (45.0% compared to 24.0% control, p=0.092).

Effectiveness of skill acquisition

Figure 2 depicts the proportion of participants achieving the MPS, when comparing the control and intervention groups before and after training. On baseline procedural testing, no participants attained the MPS. A total of 21 of 58 participants achieved the MPS at the end of the training session, including 16 fellows, 4 residents and 1 medical student. This included 6/25 (24.0%) of the control group, and 15/33 (45.5%) of the intervention group, but did not reach statistical significance (table 2, figure 2; p=0.092; 95% CI −0.49 to 0.06). Two-thirds (16/24) of the fellows achieved the MPS, compared to only 5/34 (14.7%) of the non-fellows (p<0.01, 95% CI 0.26 to 0.78).

Individual metrics of the skills checklist after the intervention are represented in table 3. There was a statistically significant increase in the intervention group in the domain “ensuring necessary equipment assembled before the procedure” (difference 24.0%; 95% CI 7.7% to 43.4%). There was a non-statistically significant increase in learners who preoxygenated (difference 23.9%; 95% CI −3.6% to 33.6%), positioned head appropriately (difference 5.9%, 95% CI −9.7% to 24.4%), inserted ETT to appropriate depth (difference 6.8%; 95% CI −14.7% to 29.0%), and checked position of ET tube at level of lip (difference 21.3%; 95% CI −1.2% to 40.1%).

Table 3.

Checklist items post-training

	Randomisation
	Control (DL) (n=25)	Intervention (VL) (n=33)	p Value*
1. Selects appropriate size ETT	25 (100.0)	25 (100.0)	–
2. Ensures that the necessary equipment is assembled, available and in working order before the procedure	19 (76.0)	33 (100.0)	0.004
3. Preoxygenates	20 (80.0)	31 (93.9)	0.114
4. Positions head appropriately	22 (88.0)	31 (93.9)	0.368
5. Inserts blade smoothly using left hand	25 (100.0)	25 (100.0)	–
6. Lifts handle forward	23 (92.0)	28 (84.8)	0.343
7. Inserts ETT into the oropharynx and vocal cords in one smooth motion	24 (96.0)	32 (97.0)	0.681
8. Inserts ETT to appropriate depth	18 (72.0)	26 (78.8)	0.384
9. Checks position of ETT at level lip	3 (12.0)	11 (33.3)	0.182

*p Values were calculated using Pearson's χ² test.

DL, direct laryngoscopy; ETT, endotracheal tube; VL, videolaryngoscopy.

Efficiency of skill acquisition

Overall, neonatal fellows required significantly less instructional time compared to the paediatric interns and students (median (IQR) 810 s (282) vs 1200 (32) s, p<0.001). Median instructional time for the control group participants was 1180 sec (IQR 197) and 1128 s (IQR 236) for the intervention group, respectively, (p=0.279, table 2). An increased number of participants achieved the MPS over time in the intervention group compared to the control group, but this did not reach statistical significance (p=0.092) (figure 2).

In order to examine the correlation between C-L scores given by learner self-report and blinded expert review C-L score, we calculated a kappa value measuring inter-rater reliability. We found κ=0.020 (p=0.812, 95% CI −0.133 to 0.173), indicating a very low level of agreement given by learners' self-report and blinded expert review.

With regard to attitudes, we found a significant increase in participants’ self-reported comfort level with the NI procedure (p<0.001) as well as confidence in their ability to perform NI in a term infant (p<0.001) following completion of the educational session.

Discussion

In this pilot study evaluating the effects of I-VL and DP on efficiency and effectiveness of training, we noted an increase in the proportion of participants achieving the predefined MPS denoting competency after completing the simulation-based educational session. Learners’ knowledge of the procedure and airway anatomy significantly improved after the training, as did their confidence and comfort in approaching the procedure. However, no difference was noted between the intervention and control groups. There was improved efficiency of teaching in the intervention group, although this did not reach statistical significance.

The use of simulation in the development of procedural skills training has been well described,²⁰ and simulation is routinely utilised to train novices to perform NI. Using traditional methods of training, the instructor cannot see the learner's real-time airway view and must give corrective feedback based on the learner's external positioning and self-reported observations. It stands to reason that using I-VL, where the instructor can independently assess the learner's airway visualisation and offer immediate feedback to optimise performance, would improve NI success rates.

Previous authors have assessed VL in simulated neonatal and paediatric intubations, with VL as the investigative method (ie, to assess if VL was useful in intubation attempts).²¹ Donoghue et al²² noted improved first-attempt success rates and improved airway views in a cross-sectional study evaluating VL in paediatric emergency medicine providers on neonatal, infant and adult simulators. Sylvia et al²³ performed a randomised trial of paediatric intubation for emergency medicine providers using either DL or VL with no additional benefit conferred by the latter. Others have studied intubation performance of paediatric residents²⁴ and students²⁵ using VL and found no improvement in success rates over DL.

In this study, VL was utilised to determine if this technology leads to improvements in training. This is a very different methodology to that of the studies cited above, as we used I-VL to assess the efficacy of this simulation-based method of training.²¹ Use of the VL did not confer a statistically significant benefit in the efficiency of training in a heterogeneous group of participants. There were notable improvements in effectiveness, however, with a large absolute difference in participants achieving the MPS (24% vs 45.5% in the DL and I-VL groups, respectively), although the results were not statistically significant. One of the checklist items, ensuring proper assembly and function of equipment, was significantly improved in the I-VL group. This checklist item is associated with competency with borderline statistical significance (p=0.051). Other checklist items were improved in the intervention group but did not reach statistical significance. The lack of significant improvement in efficiency and effectiveness is most likely related to the small sample size, which may have limited the ability to detect significant differences between the groups.

To the best of our knowledge, this study is the first to apply DP to neonatal airway management (including the use of VL). Wayne et al¹⁶ have demonstrated that utilisation of a training paradigm including DP has been shown to improve outcomes in several adult procedures, such as central line placement, thoracentesis and Advanced Cardiac Life Support performance.^26–35 In this study, participation in an individualised session of NI instruction involving DP led to significant improvement in knowledge, procedural performance of intubation in a mannequin model and comfort/confidence in performing this procedure in a term infant for all learners. Since learners were limited to 20 min with the educator, not all reached the MPS to denote competence on summative evaluation. This is certainly an area that could be addressed in future work. The positive effects of utilising this best educational practice in this study may suggest a role for this educational paradigm in the training for many other neonatal and paediatric procedures.

The lack of correlation between the self-rating of the airway grade compared to that of the blinded reviewer is also striking. This speaks to the previously identified difficulty that learners have in visualising the neonatal airway, and may be related to a lack of familiarity with the airway anatomy. Given this finding, the use of VL in intubation training and clinical intubation procedures may be beneficial in that it removes the subjectivity of the learner's self-reported view to some extent. An experienced supervisor can trust their direct, real-time visual confirmation, and will ideally be able to offer corrective strategies to increase success rates.

Limitations

There were several limitations in this pilot study. First, the arbitrary limit of 20 min of educational time most likely prevented a significant proportion of participants from reaching the MPS on summative evaluation. Consistent with the work of previous authors,^{12 36} these participants would ideally have had unlimited time to continue DP, and in future studies, participants will be permitted to continue practice for an unrestricted time period until the MPS is attained.

The second limitation is the heterogeneity of the study population. Medical students and paediatric interns have had little previous clinical exposure to NI, and typically needed to spend more time learning cognitive information and reviewing the procedural steps. In subsequent studies, a more homogeneous population with some previous exposure to NI should be utilised (eg, current neonatal fellows and senior paediatric residents). Additionally, a pretraining module will be developed for future studies to review the relevant cognitive information, allowing the face-to-face time with the instructor to be spent in expert modelling and massed DP.

The number of participants enrolled in this study was lower than desired. Initial sample size calculations estimated that 65 learners per group would be sufficient to detect a 20% difference in success rates between the I-VL and DL groups at a power of 80%. Given the lower than expected enrolment and diversity in learner experience, the number of participants may have been insufficient to detect a difference in this study. Additionally, the number of fellows randomised to the control and intervention groups was unequal, since randomisation was performed prior to study enrolment, and not all participants at the RNBC were able to consent to and participate in training due to time limitations. In the future, individuals will undergo randomisation at the time of enrolment, and a multicentre investigation will increase the number of participants.

Another limitation was that the primary investigators acting as the instructors (LCJ or CJB) completed the procedural skills checklist and GSA during the training sessions, and were therefore not blinded to the participants’ randomisation group. It would be preferable to videotape training sessions, and to have blinded reviewers rate the participants’ performance.

Skill retention was not addressed in this study as the learners only underwent one session of massed practice. Additional sessions of distributed practice would be optimal to assess for changes in proficiency due to skill decay. Finally, this study did not assess for translation of skills demonstrated on a simulator to success in clinical intubations.

Conclusion

In conclusion, no significant difference was noted in the efficiency or effectiveness of training in this pilot study comparing I-VL to DL for teaching NI. However, the absolute difference in the achievement of the MPS for competency (24% in the control group vs 46% in the intervention group) and improvements in checklist items is fairly striking, and suggests that further investigation is indicated. Similarly, time to achieve the MPS was shorter in the I-VL group, but did not reach statistical significance. Both of these findings may be attributable to the small sample size and diversity among participants. Overall, participation in an individualised session of NI instruction with DP did lead to a significant improvement in knowledge, procedural performance in a mannequin model and levels of comfort and confidence towards performing this procedure for participants in both study groups.

Data from this study will form the basis of a multicentre randomised study investigating the effect of DP and I-VL on NI instruction in neonatal fellows and senior paediatric residents planning to subspecialise in critical care through the INSPIRE Network. This study will be appropriately powered to detect potential differences between study groups. Ultimately, we aim to link learner performance in simulation-based procedural training sessions to success rates in clinical intubations.