Abstract
Background:
Fiberoptic intubation (FOI) is key in managing difficult airways. Good scope control increases efficiency and patient safety. Understanding the gap between novices and experts in scope control would help medical educators develop a feedback-based teaching approach for novices. We designed and used a checklist for evaluating the gap in fiberoptic scope control between novices and experts.
Methods:
Twelve first-year anesthesiology residents (novice group) attended a lecture, followed by hands-on practice with a fiberoptic scope on a manikin. Five staff anesthesiologists (expert group) only did the hands-on practice. After practice, each participant was video-recorded while conducting an FOI on the manikin. Two senior anesthesiologists developed and used a 7-item checklist to assess the FOIs. Checklist scores and total times for FOIs were compared between groups using the Mann-Whitney U test. Internal consistency of the checklist items, interrater reliability, and the relationship between checklist score and total time for FOI were assessed with Cronbach alpha, Cohen kappa, and the Pearson correlation coefficient, respectively.
Results:
Experts had higher checklist scores than novices (P = .0016). The item with the lowest success rate for novices (50%) was keeping the scope straight. Novices spent more time on the FOI than experts (P = .0005). Cronbach alpha, Cohen kappa, and the Pearson correlation coefficient were 0.8699, 0.75, and −0.9454, respectively.
Conclusions:
Our checklist was used to detect differences in fiberoptic scope control skills between novices and experts. With a video-based assessment method, it can be used to develop a feedback-based teaching method for fiberoptic scope control.
Keywords: Fiberoptic intubation, checklist, fiberoptic intubation teaching
Introduction
Fiberoptic intubation (FOI), or intubation using a flexible video bronchoscope, is one of the key techniques in difficult airway management.1 It is often needed emergently or electively and should be performed as quickly as possible to prevent adverse outcomes, such as desaturation, morbidity, and mortality. Therefore, the ability to control the scope to perform an FOI in a timely manner is important for patient safety. Because of the scope’s flexible design, fiberoptic scope control during intubation is different from handling other types of laryngoscopes and requires practice.2 The wide use of video laryngoscopes has reduced FOI practice opportunities for trainees.3 The current predominant teaching model is hands-on practice in the clinical setting, but this setting can be psychologically difficult for trainees who feel under pressure in the acute setting. Teaching in this setting also can put patient safety in danger. Therefore, medical educators have been focusing on applying different simulation methods, from a simple “through-the-hole” model4 to a complex virtual reality simulator5 to facilitate FOI teaching and learning in a safe nonclinical setting so that trainees can focus on dexterity training. To gain proficiency in FOI requires novices to overcome a substantial learning curve, even on simulators.6 Along with developing new simulation methods, it is also important for medical educators to understand the gap between novices and experts in FOI skills, particularly skills in controlling the scope. Good scope control would enable novices to not only perform FOIs successfully in patients with normal anatomy, but also adjust to anatomic variations in patients. Understanding the gap in these skills between novices and experts would help medical educators to develop a feedback-based teaching approach for novices.
A checklist is an assessment tool with a list of items providing binary (Yes/No) answers. It has been used to reliably assess anesthesia procedural skills, including FOI in real patients.4,7 It also can be used as a framework to identify differences in fiberoptic scope control between different user groups.
Swift successive maneuvers during an FOI make it challenging to conduct checklist assessments in real time. Video recordings provide a solution by allowing for asynchronous, repetitive, and slow-speed visualization of scope-operator maneuvers afterward. This approach has been successfully used in both sports8 and surgical skills coaching.9 In this study, we set 3 objectives: (1) to design a checklist for evaluating fiberoptic scope control on a manikin model, (2) to use video recordings for data collection and analysis, and (3) to identify the gap in fiberoptic scope control between novices and experts using our checklist. We hypothesized that we could identify differences in fiberoptic scope control between novices and experts through video-recording analysis. As a secondary hypothesis, we anticipated that there would be an association between fiberoptic scope control and total time to perform an FOI.
Materials and Methods
The medical education research committee at the Peking Union Medical College Hospital (PUMCH) provided institutional review board approval for this prospective cohort study. Oral and written consent was obtained from each participant. The consent had an opt-out clause indicating that participation was voluntary.
Participants
We recruited 2 groups of participants. For the novice group, we recruited a convenience sample of first-year anesthesiology residents from PUMCH. All 12 first-year residents at the time of the study were included in the study. They all had performed fewer than 3 FOIs before this study. For the expert group, we recruited 5 staff anesthesiologists who had more than 3 years’ experience after graduation and perform an average of at least 10 FOIs per year. Three staff anesthesiologists were from PUMCH. Two staff anesthesiologists were from Beth Israel Deaconess Medical Center (BIDMC).
Data Collection
All participants in the novice group attended a 15-minute standardized didactic lecture in a medical education laboratory given by 1 staff anesthesiologist from PUMCH. The lecture covered airway anatomy, FOI setup and orientation, fiberoptic scope handling, and troubleshooting. Each novice participant then had 15 minutes of hands-on practice with a fiberoptic scope, including orienting and handling the scope, on a manikin model with normal airway anatomy. The participants in the expert group did not attend the lecture but practiced for 15 minutes with the same fiberoptic scope on the same manikin model to become familiar with the manikin. They were also provided with the lecture slides to review beforehand. Because this study focused on dexterity for scope control after the scope enters the mouth, no endotracheal tube was loaded over the fiberoptic scope throughout the study.
After practice, each participant conducted 1 summative FOI attempt on the manikin model and was video-recorded. The FOI was considered complete when the participant stopped the scope right above the carina in the manikin model. The FOI was recorded from the side with a smart phone. The video from the scope was also recorded with the scope’s video-recording function.
After every participant’s FOI was recorded, the side view and the scope view from the same participant were synchronized in time, and a picture-in-picture video was created for each participant (see Supplemental Online Material, Appendix A). The video was used to assess each participant’s performance. To ensure anonymity, the participants’ faces were masked in the videos. They were also wearing standard operating room attire and did not wear accessories like jewelry and watches that would identify them. Audio was disabled to further ensure anonymity.
A 7-item checklist to assess FOI scope control on a manikin airway model (Table 1) was developed using a predefined process. Two senior anesthesiologists who were specialized in FOI training (1 from PUMCH and 1 from BIDMC) reviewed checklists previously used to assess FOI skills in a clinical setting.4,7 They selected items for the checklist in the current study based on the following criteria: (1) the item reflected skill of fiberoptic scope control, (2) the item could be assessed by using video recordings, and (3) the item received consensus agreement from the 2 senior anesthesiologists. The final checklist for the current study included items similar to the previously used checklists, except for the following (because the focus of this study was on scope control and did not include loading the endotracheal tube):
Table 1.
Seven-item Checklist for Fiberoptic Intubation Assessment in Supine Manikin Model
| Item ID | Item Content |
|---|---|
| 1 | Hold control section correctly with thumb positioned on control |
| 2 | Stabilize tip of scope with other hand |
| 3 | Introduce scope into mouth centered |
| 4 | Anteflex scope to see epiglottis |
| 5 | Slightly retroflex after passing the vocal cord |
| 6 | Advance scope in trachea centered |
| 7 | Keep scope straight throughout the procedure |
(1) The items related to loading the tube and checking for CO2 were removed.
(2) For 3 checklist items (items 4, 5, and 6 in the final checklist), assessment of scope control was emphasized by including evaluation of motions such as anteflexion, retroflexion, and keeping the scope centered in the trachea when it is advanced.
(3) An additional item was added as an emphasis on scope control: “Keeping the scope straight throughout the procedure.”
The 2 specialized senior anesthesiologists conducted pilot FOIs on an airway manikin model and reviewed their own FOI recordings.
Checklist Assessment
Checklist assessment of the video recordings was done by the same 2 senior anesthesiologists who developed the checklist. The assessors were not aware of which attempts were performed by experts and which were performed by novices. Each item in the checklist was worth a point. For each participant, each assessor reviewed the videos and gave 1 point for each checklist item if passed, 0 if not. All points were added to calculate each participant’s total score from each reviewer. The total scores from the 2 reviewers were averaged to calculate the checklist score for that participant.
In addition, for each checklist item, the reviewers’ points were averaged for each participant and added across the participants for each group. This number was divided by the total number of possible points each group could have achieved, which was the number of participants in each group (because each participant could have earned a maximum of 1 point), to determine the success rate for each checklist item per group.
Total Elapsed Time
The total elapsed time for each participant’s FOI was measured from the fiberoptic scope entering the mouth to it stopping above the carina in the airway model. We did not include the time of handling the scope before insertion into the mouth because we focused on dexterity for scope control after the scope enters the mouth.
Statistical Analysis
Statistical analysis was conducted using Stata version 15.1 (StataCorp, College Station, Texas). A 2-tailed P value less than .05 was considered statistically significant.
Checklist Performance
Cronbach alpha was used to assess internal consistency of the checklist scores. Each rater’s z-score was calculated for each group. Cohen kappa was used to assess the interrater reliability between the 2 raters for the checklist scores; agreement between the 2 raters was calculated for each checklist item. Checklist scores are summarized as medians (interquartile ranges) and were compared between groups using the Mann-Whitney U exact test. Success rates are reported as percentages.
Total Elapsed Time
Total elapsed times for the FOIs are reported in seconds as medians (interquartile ranges). They were compared between groups using the Mann-Whitney U exact test.
Relationship Between Checklist Performance and Total Elapsed Time
The relationship between participants’ checklist scores and total elapsed times for the FOI was assessed using the Pearson correlation coefficient.
Results
All 12 novice participants were included in the checklist performance analysis. One of the 12 novice participants was excluded from the total elapsed time analysis because the participant did not reach the endpoint of identifying and stopping at the carina and performed a main stem intubation instead. All 5 expert participants were included in both the checklist performance and total elapsed time analyses.
Checklist Performance
Cronbach alpha for the checklist scores across participants and items was 0.8699, indicating good internal consistency. The 2 raters’ z-scores for the novice group were 0.051 and −0.057; z-scores for the expert group were not calculated because the SD for both raters was 0. Cohen kappa between the 2 raters’ checklist ratings was 0.75 (95% confidence interval: 0.61, 0.89), indicating moderate to substantial interrater reliability for the checklist. Agreement for checklist items 1 to 7 was 82%, 94%, 94%, 94%, 94%, 88%, and 88%, respectively.
The median checklist score for the novice participants was 6 (2.75–6.5), whereas all experts achieved a 7 for a checklist score (Figure 1, Table 2). Experts’ checklist scores were significantly higher than novices’ checklist scores (P = .0016).
Figure 1.
Checklist scores in novice and expert groups. The median (interquartile range) checklist score for novice participants was 6 (2.75–6.5), whereas all experts achieved a checklist score of 7. Experts scored significantly higher on the checklist than novices (P = .0016).
Table 2.
Median (Interquartile Range [IQR]) of Checklist Scores and Total Elapsed Times
| Novice (N = 12 for checklist performance analysis; N = 11 for total elapsed time analysis) | Expert (N = 5) | P Value | |||||
|---|---|---|---|---|---|---|---|
| Median | IQR | Range | Median | IQR | Range | ||
| Checklist score | 6 | 2.75–6.5 | 0.5–7 | 7 | 7–7 | 7–7 | .0016 |
| Total elapsed time | 109 | 76–165 | 58–236 | 44 | 44–48 | 36–48 | .0005 |
Unlike the expert group, the novice group was not able to pass each checklist item with a 100% success rate (Figure 2). The lowest success rate for the novice group was checklist item 7: Keep scope straight throughout the procedure, with a success rate of 50%. The success rates for the other items were also lower in the novice group.
Figure 2.
Success rate for each checklist item for participants in novice and expert groups. The novice group was not able to pass each checklist item with a 100% success rate. The lowest success rate for the novice group was checklist item 7: Keep scope straight throughout the procedure, with a success rate of 50%.
Total Elapsed Time
The median total time to complete the FOI for novices was 109 (76–165) seconds, and the median time was 44 (44–48) seconds for experts (Figure 3, Table 2). Experts completed the FOI in significantly less time (P = .0005).
Figure 3.
Total elapsed fiberoptic intubation (FOI) time for participants in novice and expert groups. The median (interquartile range) total time to complete the FOI for novices was 109 (76–165) seconds, whereas the median (interquartile range) time was 44 (44–48) seconds for experts. Experts completed the FOI in significantly less time (P = .0005).
Relationship Between Checklist Performance and Total Elapsed Time Among Novices
The checklist score and the total elapsed time for the FOI on the manikin had a negative correlation for the novice participants (Figure 4): the higher an individual participant’s checklist score was, the less time the participant tended to need for the FOI. The Pearson correlation coefficient was −0.9454 (P < .001) between the checklist score and the total elapsed time for the FOI for novices.
Figure 4.
Relationship between checklist performance and total elapsed time among novices. The checklist score and the total elapsed time for the fiberoptic intubation had a negative correlation for the novice participants. The Pearson correlation coefficient was −0.9454 (P < .001) between checklist score and total elapsed time for novices.
Discussion
In this study, we designed a 7-item checklist to evaluate fiberoptic scope control on a manikin model between novices and experts. The face validity of the checklist was ensured through domain expert participation and a predefined process of literature review and pilot testing. This checklist showed satisfactory internal consistency among the checklist items and moderate interrater reliability when used by raters to assess participants’ video recordings. By using this checklist, we were able to differentiate fiberoptic scope control skills between novices and experts. Essentially, the novices lagged significantly behind experts in the checklist criteria. Based on the novice group’s success rates for the individual checklist items, we identified the most challenging checklist item for the novice group: to keep the scope straight throughout the procedure. Keeping the scope straight is important for maneuvering the fiberoptic scope.10 A curved scope would not permit wrist motion to rotate its tip, reducing the efficiency of the FOI and likely increasing the total time needed to perform the FOI.
Besides fiberoptic scope control, there are many other criteria that can be used to assess FOI skills. Total elapsed time to complete an FOI was used in many studies as one criterion of success for an FOI because an FOI should be performed as quickly as possible to prevent adverse patient outcomes. With time being an important consideration for patient safety in FOIs, good scope control is a necessary skill to perform an FOI in a timely manner. Without the skills to control the scope, novices would not be able to handle difficult anatomic variations in real patients. Our study showed that novices spent significantly more time on their FOIs than experts did. This suggests that novices can decrease their time to perform an FOI with targeted practice as they become experts. More interestingly, when we examined the relationship between checklist performance and total elapsed time for the FOI among novices, we detected an excellent negative correlation: a higher checklist score was associated with a shorter intubation time. This supports our view of the importance of teaching novices the skills to control a fiberoptic scope. If novices have better control of the scope, their time to perform an intubation would likely decrease.
By identifying the gaps between novices and experts, medical educators can direct their attention to the novices’ deficiencies and develop a feedback-based teaching method for novices. According to the Dreyfus education developmental model,11 one of the characteristics at the novice stage is rule following. An instructional strategy for this stage is to provide straightforward rules. The checklist, besides being used as an assessment tool, can work as a framework of teaching points for novices. A medical educator can provide the checklist to novices as performance criteria and conduct formative assessments on their fiberoptic scope control. Feedback also can be directed to specific items on the checklist where individual novices show weakness. This directed feedback approach was shown to be more effective than generalized feedback.12,13
Video assessment has been shown to be a reliable and valid assessment tool for basic surgery skills.14 In this study, raters used video recordings to assess participants’ skills in controlling a fiberoptic scope. The picture-in-picture approach to synchronize the side and scope views was useful to facilitate rater assessment. The rater was able to see both external scope handling from the side view and internal scope view simultaneously. In addition, compared with direct observation, this approach gave raters more time to assess the details. These factors may help provide more accurate assessments than direct observation. This demonstrates the benefit of video recording and playback for assessment of performance by instructors. This method also may be useful for self-assessment of performance by learners. When combined with debriefing, video recordings may prove to be a valuable tool for enhancing deliberate practice.
The major limitation of our study was the small convenience sample size. Nonetheless, our study still demonstrated significant differences between experts and novices in scope control based on our modified checklist and total time to complete an FOI. A larger sample size would increase the power for further analyses of checklist performance. Another limitation was the lack of higher interrater reliability for novices, which may have contributed to the difference we detected in checklist scores between novices and experts. This lack of interrater reliability may imply that novices’ FOI skills are not as refined and clear to assess as experts’ skills. Although we mitigated bias in the checklist scores by averaging the 2 raters’ scores and blinding the raters to whether each video was from an expert or a novice, future studies should include training of the raters to increase interrater reliability. Checklist items 1, 6, and 7 had the lowest agreement between the raters, indicating that when training raters, clear definitions should be made for how to hold the control section correctly, how centered the scope should be in the trachea when advancing the scope, and how straight the scope should be throughout the procedure. In addition, a global rating scale should be used to further assess FOI skills. The other limitation of our study stemmed from the single-center design: although the experts and the checklist developers were from 2 institutions, the novices were from 1 institution. The differences we detected between this group of novices and the experts reflected only 1 institution and may not be generalizable to other institutions. In the future, we plan to expand our study to multiple centers to both increase the sample size and improve generalizability. The results from this study give us a theoretical basis to calculate the sample size needed for future studies.
Conclusion
Although checklists and, to a lesser extent, video recordings, have been used before to assess fiberoptic intubations and compare FOI skills between novices and experts, we have modified previously used checklists to focus on scope control, an important skill to master to complete a fiberoptic intubation in a timely and accurate manner. We used this checklist to identify differences in scope control between experts and novices as well as identify checklist items that are more challenging for novices than others. In addition, scores on our modified checklist correlate with total elapsed time for FOI, providing further evidence of the checklist’s validity. Our checklist, combined with the novel picture-in-picture video-based assessment method we used in this study, provides the foundation for developing a feedback-based teaching method for novices in controlling a fiberoptic scope.
Acknowledgments
We thank the residents and attending anesthesiologists who participated in this study at Peking Union Medical College Hospital. We also thank the Department of Anesthesiology at the PUMCH and the Department of Anesthesia, Critical Care and Pain Medicine at BIDMC for their support of this study.
Appendix
Appendix A.
Synchronized Videos from Side View and Scope View After Video Editing
Footnotes
Authorship: Haobo Ma and Xia Ruan are co-first authors.
Financial support: Departmental funding.
References
- 1.Apfelbaum JL, Hagberg CA, Caplan RA, et al. Practice guidelines for management of the difficult airway: an updated report by the American Society of Anesthesiologists Task Force on Management of the Difficult Airway. Anesthesiology. 2013;118(2):251–70. doi: 10.1097/ALN.0b013e31827773b2. [DOI] [PubMed] [Google Scholar]
- 2.Latif RK, Bautista A, Duan X, et al. Teaching basic fiberoptic intubation skills in a simulator: initial learning and skills decay. J Anesth. 2016;30(1):12–9. doi: 10.1007/s00540-015-2091-z. [DOI] [PubMed] [Google Scholar]
- 3.Wanderer JP, Ehrenfeld JM, Sandberg WS, Epstein RH. The changing scope of difficult airway management. Can J Anaesth. 2013;60(10):1022–4. doi: 10.1007/s12630-013-9999-2. [DOI] [PubMed] [Google Scholar]
- 4.Naik V, Matsumoto E, Houston P, et al. Fiberoptic orotracheal intubation on anesthetized patients: do manipulation skills learned on a simple model transfer into the operating room? Anesthesiology. 2001;95(2):343–8. doi: 10.1097/00000542-200108000-00014. [DOI] [PubMed] [Google Scholar]
- 5.Goldmann K, Steinfeldt T. Acquisition of basic fiberoptic intubation skills with a virtual reality airway simulator. J Clin Anesth. 2006;18(3):173–8. doi: 10.1016/j.jclinane.2005.08.021. [DOI] [PubMed] [Google Scholar]
- 6.Dalal PG, Dalal GB, Pott L, et al. Learning curves of novice anesthesiology residents performing simulated fibreoptic upper airway endoscopy. Can J Anaesth. 2011;58(9):802–9. doi: 10.1007/s12630-011-9542-2. [DOI] [PubMed] [Google Scholar]
- 7.Chandra DB, Savoldelli GL, Joo HS, et al. Fiberoptic oral intubation: the effect of model fidelity on training for transfer to patient care. Anesthesiology. 2008;109(6):1007–13. doi: 10.1097/ALN.0b013e31818d6c3c. [DOI] [PubMed] [Google Scholar]
- 8.Oyama S, Sosa A, Campbell R, Correa A. Reliability and validity of quantitative video analysis of baseball pitching motion. J Appl Biomech. 2017;33(1):64–8. doi: 10.1123/jab.2016-0011. [DOI] [PubMed] [Google Scholar]
- 9.Soucisse ML, Boulva K, Sideris L, et al. Video coaching as an efficient teaching method for surgical residents - a randomized controlled trial. J Surg Educ. 2017;74(2):365–71. doi: 10.1016/j.jsurg.2016.09.002. [DOI] [PubMed] [Google Scholar]
- 10.Leslie D, Stacey M. Awake intubation. Contin Educ Anaesthesia Crit Care Pain. 2015;15(2):64–7. [Google Scholar]
- 11.Batalden P, Leach D, Swing S, et al. General competencies and accreditation in graduate medical education. Health Aff (Millwood) 2002;21(5):103–11. doi: 10.1377/hlthaff.21.5.103. [DOI] [PubMed] [Google Scholar]
- 12.Bienstock JL, Katz NT, Cox SM, et al. To the point: medical education reviews - providing feedback. Am J Obstet Gynecol. 2007;196(6):508–13. doi: 10.1016/j.ajog.2006.08.021. [DOI] [PubMed] [Google Scholar]
- 13.Archer JC. State of the science in health professional education: effective feedback. Med Educ. 2010;44(1):101–8. doi: 10.1111/j.1365-2923.2009.03546.x. [DOI] [PubMed] [Google Scholar]
- 14.Driscoll PJ, Paisley AM, Paterson-Brown S. Video assessment of basic surgical trainees’ operative skills. Am J Surg. 2008;196(2):265–72. doi: 10.1016/j.amjsurg.2007.09.044. [DOI] [PubMed] [Google Scholar]