Retention of emergency cricothyroidotomy skills: A multicenter randomized controlled trial

Abstract

Objectives

Emergency cricothyroidotomy is the final approach to establishing a secure airway. The procedure is acute and highly infrequent, making it difficult to achieve and maintain competence in the clinic. Simulation‐based training in emergency cricothyroidotomy is effective but it is unknown how often training should be repeated to maintain skills. This study aimed to assess novices’ retention of technical skills in emergency cricothyroidotomy after completing SBT.

Methods

Novices in emergency cricothyroidotomy completed a structured, simulation‐based training program and were randomized to retention tests after 1, 3, or 6 months. Participants completed two emergency cricothyroidotomy tests at end‐of‐training and follow‐up retention testing. Test performances were video recorded and evaluated by two experienced blinded raters using a structured assessment tool. Differences in the performances and the pass/fail rates were analyzed.

Results

Eighty‐two medical students from two different Danish universities were included from April 2021 to February 2022. Paired t‐tests showed skills decay significantly after 1 month (mean loss 6.7 points, p < 0.001). The mean loss of points, representing the difference in global score points, from the end‐of‐training to retention test was 6.7 points (95% confidence interval [CI] 4.5–8.8) for the 1‐month group, 8.2 points (95% CI 5.8–10.0) for the 3‐months group, and 9.9 points (95% CI 8.1–11.7) for the 6‐months group. Six participants in both the 1‐month group (23.1%) and the 3‐month group (24%) passed the first retention test, but no one in the 6‐months group had a passing performance.

Conclusions

Novices’ technical skills performance in emergency cricothyroidotomy decay significantly already after 1 month. This initial loss of skill seems to be stable until 3 months, after which there is a further significant loss of skills. Recurring training should be implemented for the benefit of patient safety and outcomes.

INTRODUCTION

Airway evaluation is an essential task when a critically ill patient arrives at the emergency department or in any prehospital setting. ¹ Emergency physicians and anesthesiologists should master the technique because time is an essential factor to prevent brain damage and death in case of compromised airway. ¹

Unfortunately, even experienced physicians struggle with the emergency cricothyroidotomy procedure, which has a reported failure rate of 25%. ² , ³ This indicates the need for structured training to acquire and maintain an adequate level of competency to minimize preventable harm and thus increase patient safety. ⁴ As cricothyroidotomy is a low‐frequency procedure, it makes it difficult to rely on skills development during routine clinical care, ² resulting in insufficient experience and procedural knowledge for physicians. ⁵

Simulation‐based training (SBT) is a well‐established tool for the development and maintenance of technical skills. ⁵ , ⁶ , ⁷ , ⁸ It is a supplement to real‐time training in the clinical setting and offers a safe training environment by providing an opportunity to practice medical procedures without compromising the safety of patients. ⁹ Little is known about the best strategies to prevent or minimize skill decay, ¹⁰ and how often SBT of emergency cricothyroidotomy should be repeated to maintain skills remains an important question. This study aimed to determine the retention of skills in emergency cricothyroidotomy by exploring the loss of technical skills at 1, 3, and 6 months after novices (represented by medical students) received structured SBT in the procedure.

METHODS

Setting

This multicenter randomized controlled trial took place at the Simulation Center (SimC) at Odense University Hospital, Denmark, and at the Copenhagen Academy for Medical Education and Simulation (CAMES), Copenhagen, Denmark. Data were collected from April 2021 to February 2022.

Materials

A TruCorp airway and tracheotomy phantom was used (TruCorp). Available equipment was a standard 6.0 endotracheal tube, a scalpel size 10, an Iterson tracheal hook 16 cm, a 10‐mL Luer‐Lock syringe, Leukoplast sleek LF adhesive tape, and a ventilation balloon.

Participants

Medical students with no prior experience with emergency cricothyroidotomy were invited to participate. Participation was voluntary and participants were invited by email, face‐to‐face meetings, and social media posts. No compensation, study credits, or other incentives were offered.

Before inclusion, participants received information regarding the study and consented to study participation. Participants could be included if they were actively enrolled at a Danish medical school and performed both the initial and the retention tests. The participants received no compensation or salary.

Sample size calculation

The sample size calculation was based on the results of a previous study. ⁸ The sample size aimed to detect a 2‐point difference between the groups with an anticipated standard deviation of two points. At least 22 participants in each group were required (alpha = 0.05 and power = 0.90). To allow an estimated dropout of 20%, 27 participants were required in each group, resulting in a total number of 81 participants.

Randomization

Participants were randomized into one of three retention groups using block randomization in blocks of 9. The block randomization list was produced by an online random number service (www.random.org) as a central masked allocation. It was administered by a person not otherwise involved in the data collection (ABN). Neither the instructor (MSN or FNRL) nor the participant received information regarding which group the participant was allocated to, until after the end‐of‐training tests.

Data collection

Participants received theoretical and hands‐on training in emergency cricothyroidotomy in individual sessions by an instructor (MSN or FNRL). The training was based on the standardized rapid four‐step technique (RFST) course. ⁸ An introduction video was shown to introduce the RFST. ⁸ , ¹¹ The steps were repeated verbally and demonstrated on the phantom. Participants were then allowed to practice individually until they felt ready for the end‐of‐training tests in which two consecutive emergency cricothyroidotomy procedures on the phantom were video recorded using a tablet. The faces of the participants were not visible on the recordings to anonymize them to the experienced raters. No interaction or feedback between instructor and participant was allowed during the tests.

Participants returned for follow‐up retention tests after approximately 1, 3, or 6 months depending on the allocated group (Figure 1). At follow‐up, participants received no further training before performing the tests, which were recorded similarly to the end‐of‐training tests.

FIGURE 1.

Data collection flowchart.

Evaluation of performances

An evidence‐based, structured assessment tool of technical skills performance has been developed to evaluate physicians’ skills in the RFST procedure and a pass/fail standard has been set. ⁸ All recordings were evaluated by two experienced raters (SA, KW) using this assessment tool. Both raters are experienced otorhinolaryngologists and experienced instructors in teaching cricothyroidotomy. Further, they work as consultants at the Copenhagen Academy for Medical Education and Simulation and routinely assess performance in SBT of cricothyroidotomy using the structured assessment tool. The raters were blinded and did not have access to any information regarding the participants. Recordings were blinded and tagged with a unique identification number and were presented to the raters in a random order so that the raters were also blinded to whether recordings were from end‐of‐training or retention tests.

The assessment tool consists of four items:

• Positioning of the head

• Palpation

• Appropriate employment of instruments

• Stepwise progression

Each parameter could be assigned a score from 0 to 4 points with 4 being the maximum score (Table 1). The total score was divided by the time spent in minutes, resulting in a final score per minute. The pass/fail standard is 12 points per minute as established for the assessment tool. ⁸

TABLE 1.

Evidence‐based assessment tool.

	Points
	0	1	2	3	4
Positioning of the head	Failed to perform		Performed, but insufficiently		Performed successfully
Palpation	Failed to perform		Performed, but insufficiently		Performed with determination
Appropriate employment of instruments	Incorrect and clumsily		Correct but insecurely		Correct and determined
Stepwise progression, namely, flow	Chaotic and hesitantly		Nonlinear but deliberate		Linear and deliberate

Statistical analysis

Statistical analyses were performed in SPSS V28 (IBM). Pearson's r was used to determine the correlation between the total score given by the raters. A paired t‐test was used to examine the difference in the mean scores per time at the end‐of‐training tests and the retention tests. Unpaired t‐tests were performed to compare the performances of the three groups for the end‐of‐training tests and the retention tests. An unpaired t‐test was performed to examine the loss of points (i.e., the delta value) between the groups from the end‐of‐training tests to the retention tests. Fisher's exact test was conducted to examine the number of participants failing the test in each group. p‐values of <0.05 were considered statistically significant.

Ethics

Data was pseudo‐anonymized as a correct pairing of end‐of‐training test videos and retention test videos was essential. Only MSN and FNR had access to information about the participant and the associated video IDs. All data were entered and handled in an online database: Research Electronic Data Capture (REDCap). All tests were recorded on a tablet and immediately uploaded to a secure server hosted by the independent research support unit Open Patient Data Explorative Network (OPEN), and the recording was deleted from the tablet. MSN and FNR had access to the server, and logins were tracked and supervised. An application was sent to the regional Scientific Ethics Committee in the Region of Southern Denmark, case number 20212000‐40, where it, by March 18, 2021, was concluded exemption for further applications.

RESULTS

Eighty‐two participants were included in the study and 78 participants (95%) completed the retention tests. The mean age of the participants was 23.7 years (range 19–33 years). Participants in the 1‐ and 6‐month groups were a mean of 3.0 years into medical school, and the 3‐months group were a mean of 2.8 years in.

Paired t‐tests were conducted to examine whether a significant change in score was observed between initial tests and retention tests for each group. The paired t‐tests demonstrated a significant reduction in the score for each group when comparing the last initial test to the first retention test: the 1‐month group had a mean loss of 6.7 points (p < 0.001), the 3‐months group had a mean loss of 8.2 points (p < 0.001), and the 6‐months group had a mean loss of 9.9 points (p < 0.001).

The mean score and comparison of the groups in performances in both end‐of‐training tests and retention tests are summarized in Table 2. Table 3 supplements this, by presenting pass/fail data for the groups.

TABLE 2.

Performances of the tests based upon the assessment tool.

	First end‐of‐training test			Second end‐of‐training test			First retention test			Second retention test
	Mean score	SD	p‐value	Mean score	SD	p‐value	Mean score	SD	p‐value	Mean score	SD	p‐Value
1‐month group	13.8	5.9	vs. 3‐months group = 0.039	15.6	5.3	vs. 3‐months group = 0.11	8.9	4.1	vs. 3‐months group = 0.61	11.2	4.1	vs. 3‐months group = 0.55
			vs. 6‐months group = 0.73			vs. 6‐months group = 0.63			vs. 6‐months group, < 0.001			vs. 6‐months group, <0.001
3‐months group	17.0	4.8	vs. 6‐months group = 0.006	17.7	3.8	vs. 6‐months group = 0.012	9.5	4.1	vs. 6‐months group < 0.001	11.9	4.3	vs. 6‐months group <0.001
6‐months group	13.3	4.6		15.0	3.7		5.1	3.5		6.8	3.9
All participants	14.6	5.3		16.1	4.4	vs. first post test, < 0.001	7.8	4.3		9.9	4.6	vs. first retention test, <0.001

TABLE 3.

Tests pass/fail.

		First end‐of‐training test	Second end‐of‐training test	First retention test	Second retention test
1‐month group
	Pass	17 (63)	19 (70.4)	6 (23.1)	9 (34.6)
	Fail	10 (37)	8 (29.6)	20 (76.9)	17 (65.4)
3‐months group
	Pass	20 (74.1)	22 (81.5)	6 (24)	12 (48)
	Fail	7 (25.9)	5 (18.5)	19 (76)	13 (52)
6‐months group
	Pass	17 (60.7)	21 (77.8)	0 (0)	6 (22.2)
	Fail	11 (39.3)	8 (28.6)	27 (100)	21 (77.8)
Total
	Pass	54 (65.9)	62 (76)	12 (15.4)	27 (34.6)
	Fail	28 (34.1)	20 (24.3)	66 (84.6)	51 (65.4)

Note: Data are reported as n (%).

Table 4 presents the results of unpaired t‐test to examine the loss of points (i.e., the delta value) between the groups from the end‐of‐training tests to the retention tests. For all groups, a significant loss of skills was found between end‐of‐training tests and retention tests (Table 4). The loss seemed to be greater the more time had passed between end‐of‐training and retention testing, but this was only statistically significant when comparing the 1‐ and 6‐months groups (p = 0.024) (Table 4).

TABLE 4.

Difference between end‐of‐training and retention performance score per time mean.

	Mean score	SD	p‐value
1‐month group	−6.7	−5.4	vs. 3‐months group = 0.32
			vs. 6‐months group = 0.024
3‐months group	−8.2	−5.1	vs. 6‐months group = 0.21
6‐months group	−9.9	−4.7

The 3‐months group performed significantly better and had more passing performances at end‐of‐training than the 1‐ and 6‐months group (Tables 2 and 3). This statistical difference was not found when comparing the second end‐of‐training tests for the 1 and the 3‐months groups but remained significantly different for the 3‐ and 6‐months groups (Table 2).

Regardless of group and time of testing (end‐of‐training or retention), participants statistically significantly improved their performance between the first and the second test (p < 0.001, Table 2). The same pattern was also reflected in more participants across groups having a passing performance in the second end‐of‐training and retention test compared with the first test (Table 3). A high and positive correlation between the total score per minute assigned by the two raters was found for both the end‐of‐training tests (r = 0.90, p < 0.001) and retention tests (r = 0.90, p < 0.001).

DISCUSSION

In this study, we explored the retention of skills in emergency cricothyroidotomy after a structured SBT course. We found that technical skills performance decreased more as time passed even though this only reached statistical significance when comparing the differences between retention performances of the 1‐ and 6‐months groups. This rapid loss of skills in the emergency cricothyroidotomy procedure substantiates the importance of retraining skills that are performed infrequently in the clinic to ensure that adequate skills are maintained.

This is further substantiated in a study investigating important skills for emergency medicine residents: only 4% of the residents had performed a cricothyroidotomy and 82% of their senior colleagues felt uncomfortable with the procedure, despite acknowledging it as lifesaving with important skills to perform it with speed and accuracy. ¹²

The finding in this study, with rapid loss of skills are furthermore in accordance with a systematic review on critical procedural skills after simulation training where the study concludes that there is a loss of score immediately after simulation training ends. ¹³ In the literature on SBT of medical and technical skills, a substantial decay of performance has been demonstrated from 2 weeks to 14 months for procedures such as electrocardiography interpretation, critical care skills, and resuscitation. ¹⁴ , ¹⁵ , ¹⁶ , ¹⁷ , ¹⁸ These findings show the duration of skill retention is related to the intensity of training, as an expression of massed training compared with distributed practice, and which procedure that is learned. ¹³ , ¹⁵ , ¹⁶ , ¹⁷

Other studies found that anesthesiologists’ skills in emergency cricothyroidotomy overall were maintained at 6 ¹⁹ and 12 months ²⁰ although there were indications of skills deterioration. This is different from our findings but might be a result of the fact that anesthesiologists are exposed to the theory and tools more than novice medical students. However, none of these studies have used an evidence‐based assessment tool for skills assessment. Another study also found that skills were maintained for up to 15 months, which again contrasts our findings. ²¹ The reason the results in these studies might be different from our finding is the inclusion of experienced participants and the use high‐fidelity simulation training with multiple‐hour interventions. This could also indicate that high‐intensity training results in longer skills retention.

Another study on advanced airway management focusing on skills retention of pediatric and neonatal intubation showed that experienced participants had increased posttraining retention compared to novice providers—highlighting the need for research into novice skills retention to ensure a high level of competencies after the initial skill certification. ²² This difference in skill retention underlines the challenge of making recommendations on standardized training intervals. Currently, the management guideline by the Difficult Airway Society (DAS) guidelines on the approach to difficult airways recommend repeating training at 6‐month intervals but this is not based on evidence. ²³ The American Society of Anesthesiologists altogether refrains from making recommendations on training intervals in their 2022 guidelines. ²⁴

The participants’ performance in this study improved between the first and second end‐of‐training tests. This indicates that even though participants felt ready to perform the tests, their maximum learning potential was not yet reached. In other words, they were inadequate in self‐assessing their skills because 29% did not practice sufficiently to pass the established benchmark. Also, this could indicate learning from testing and therefore a continued development of skills. A similar observation was made in a study investigating skills retention in basic intubation. The first intubation at the retention test was significantly slower than the second test—highlighting the ability to learn from testing. ²⁵

A study found that anesthesiologists had to conduct at least five consecutive tests in emergency cricothyroidotomy to reach a learning plateau. ²⁶ It remains uninvestigated if the results of skills retention in our study could have been retained longer and/or better if it was ensured that each individual participant had trained to their maximal potential (i.e., a plateau in their score).

This highlights the importance of implementing mastery learning to ensure that a certain level of competency is attained. ²⁷ As learners differ regarding experience with a given skill and learning pace, ideally, brush‐up training would be determined by the individual learner's characteristics and how they retain skills. Consequently, future studies and courses should implement mastery learning so that participants keep practicing until a predefined performance standard is obtained. ²⁷

Different methods are suggested to structure repeated training and secure skills retention through continuous exposure. Maintenance training is a way to prevent skill decay and maintain skills over time. ¹⁰ It involves frequent but short session training on an ongoing basis. Another approach is booster training, which is used when competencies have started to wane, due to lack of use. ¹⁰ It has more distributed training compared with maintenance training but longer training sessions. Lastly, refresher training is defined as an intervention that aims to reestablish a specific skill level after a period of nonuse, in which the competency has decayed to an unsatisfactory level. ¹⁰

However, to even structure training, it is important to be aware of potential barriers to regular implementation. One of the barriers to SBT is time and, as a consequence, long training sessions might be a disadvantage in a busy clinical schedule. ²⁸ Instead, we focused on a briefer approach, which allows participants to train skills using >30 min. This might be easier to implement in a busy clinical schedule as a regular refresher for skills maintenance and requires only access to the phantom, equipment, and the short instructional video.

LIMITATIONS

We enrolled medical students, and this might be a limitation as they, compared with experienced physicians, are not as used to the equipment and other procedures. However, by enrolling medical students we ensured that participants were true novices in the procedure as physicians in relevant specialties early receive training in cricothyroidotomy. It is debatable whether medical students can act as proxies for physicians, but physicians without any prior experience in the procedure would likely perform like the novices. This is nevertheless a limitation; however, the power calculation showed that many participants were needed in this randomized study and busy clinicians are difficult to recruit and a considerable drop‐out rate also had to be expected. Including medical students made our study feasible but considerations about the novice status and limited procedural skills in general are necessary before generalizing our results. Further studies examining physicians’ skills retention in low‐frequency procedures should be conducted.

Another important limitation is the limited background knowledge of the participants as a difference in initial test performances was found between the 3‐months group and versus the 1‐month group and the 6‐months group. We anticipated the participants to be a homogenous group of novices, but it remains unknown whether this finding is a result of coincidence or perhaps due to experience in their clinical rotations.

Experienced physicians, who have been exposed to the theory and practice training of emergency cricothyroidotomy multiple times, might retain their skills for a longer period. ²² Future studies with a focus on the retention of experienced physicians’ skills could include specialist‐trained physicians to investigate skill retention for more advanced learners.

We acknowledge performance on phantoms differs from real patients, as complications like bleeding are not replicated on low‐fidelity phantoms. Stress factors which in a real clinical setting might affect performance are difficult to imitate in a low‐fidelity simulation‐based setting. ²⁹ Despite this, the literature suggests a transfer of skills from SBT to a real clinical setting. ³⁰ , ³¹

As to the used assessment tool, it is important to discuss how suited it is for this study. Validity evidence was systematically gathered according to the framework of Messick. Evidence regarding relationship to other variables was based on the fact that the assessment tool could discriminate between novices and experienced participants in a sample of only 16 participants. This standardized assessment tool is currently used as the standard to assess skills in emergency cricothyroidotomy in Denmark.

FURTHER PERSPECTIVES

The results of this study implicate the need for structured practice to maintain technical skills. An increased focus could be placed on brush‐up interventions, as our findings of skill decay are not exceptional. It is important to implement recurring training in structured training programs when skills are initially obtained to secure sufficient skills for the benefit of patient safety and outcomes. To secure the implementation, it is important to be aware of the barriers to SBT, why maintenance training programs should be structured as brief brush‐ups with a low need for resources.

CONCLUSIONS

This study found novices’ technical skills in emergency cricothyroidotomy following a structured simulation‐based training program significantly decay after 1 month, in this low‐frequency high‐stakes procedure. This highlights the importance of repeated training following the initial skill acquisition to maintain technical skills. Brush‐up training should be an integrated part of a structured learning program since skills are not retained when rarely used.

AUTHOR CONTRIBUTIONS

Martine Siw Nielsen: study concept and design, acquisition of the data, analysis and interpretation of the data, drafting of the manuscript, critical revision of the manuscript for critical important intellectual content. Felix Nicolai Raben‐Levetzau: acquisition of the data and critical revision of the manuscript for critical important intellectual content. Steven Arild Wuyts Andersen and Kasper Wennervaldt: study concept and design, acquisition of the data, and critical revision of the manuscript for critical important intellectual content. Lars Konge and Anders Bo Nielsen: study concept and design, analysis and interpretation of the data, drafting of the manuscript, critical revision of the manuscript for critical important intellectual content, statistical expertise, administrative, technical and material support, study supervision.

CONFLICT OF INTEREST STATEMENT

The authors declare no conflicts of interest.

Nielsen MS, Raben‐Levetzau FN, Andersen SAW, Wennervaldt K, Konge L, Nielsen AB. Retention of emergency cricothyroidotomy skills: A multicenter randomized controlled trial. AEM Educ Train. 2023;7:e10900. doi: 10.1002/aet2.10900