Optimizing simulator‐based training for emergency transesophageal echocardiography: A randomized controlled trial

Abstract

Background

Resuscitative clinician‐performed transesophageal echocardiography (TEE) is a relatively novel ultrasound application; however, optimal teaching methods have not been determined. Previous studies have demonstrated that variable practice (VP), where practice conditions are changed, may improve learning of procedural skills compared with blocked practice (BP), where practice conditions are kept constant. We compared VP and BP for teaching resuscitative TEE to emergency medicine residents using a simulator.

Methods

Emergency medicine residents with no prior TEE experience were randomized to the BP or VP groups. The BP group practiced 10 repetitions of a fixed five‐view TEE sequence, while the VP group practiced 10 different random five‐view TEE sequences on a simulator. Participants completed a performance assessment immediately after training and a transfer test 2 weeks after training. Ultrasound images and transducer motion metrics were captured by the simulator for blinded analysis. The primary outcome was the percentage of successful views on the transfer test.

Results

Twenty‐eight participants completed the study (14 in the BP group, 14 in the VP group). The BP group had a higher rate of successful views compared with the VP group on the transfer test (93.6% vs. 77.6%; p = 0.002). The BP group also had higher image quality on a 5‐point scale (3.3 vs. 2.9; p = 0.01) and fewer probe angular changes (2982.5 degrees vs. 4239.8 degrees; p = 0.04). There were no statistically significant differences between the groups for the rate of correct diagnoses, confidence level, or scan time.

Conclusions

Practicing a fixed sequence of views was more effective than a variable sequence of views for learning resuscitative TEE on a simulator. These results should be validated in TEE scans performed in the clinical environment.

INTRODUCTION

Transesophageal echocardiography (TEE) is a relatively new point‐of‐care ultrasound (POCUS) modality that is increasingly being used in emergency departments (ED). ¹ TEE scans performed in the ED are most commonly used to assist with managing critically ill patients, including those with hemodynamic instability and cardiac arrest. In these patients, TEE can help to identify important causes of shock such as left ventricular dysfunction, cardiac tamponade, and massive pulmonary embolism. ¹ Emergency physician–performed TEE has been shown to be feasible, can assist with patient diagnosis, and leads to changes in management in a significant proportion of patients. ² In 2017, the American College of Emergency Physicians introduced practice guidelines for the use of TEE in the ED. ³ These guidelines acknowledged that TEE has the potential to identify life‐threatening conditions in the setting of cardiac arrest, where traditional transthoracic echocardiography (TTE) may be technically challenging.

Given the relatively novel aspect of resuscitative TEE, there is limited evidence on best practices for teaching this skill to emergency physicians. Unlike most other POCUS applications, teaching TEE using healthy volunteers is not practical. It is also not feasible to bring large numbers of learners to the operating room to gain experience. As an alternative, simulation‐based training using high‐fidelity simulators has been shown to be highly effective for teaching resuscitative TEE to emergency physicians. ⁴ , ⁵ Skills acquired using high‐fidelity TEE simulators have been shown to translate well into real‐world scan performance. ⁶

Despite its effectiveness, simulator‐based training for TEE is resource intensive, often requiring one‐on‐one instructor and learner time. It is therefore important to structure simulator training sessions to maximize efficiency. To date, no studies have compared different teaching methods for simulator‐based training for resuscitative TEE. One factor that can affect learning is whether the practice is done through repetition of the same steps each time (blocked practice) or if the practice conditions are changed with each subsequent attempt (variable practice). While blocked practice can be effective at reinforcing learning, it may lead to difficulties when the situation requires modifications, which is common with the goal‐directed nature of emergency POCUS. ⁷ The objective of this study was to compare blocked and variable practice for teaching emergency medicine residents how to perform resuscitative TEE using a high‐fidelity simulator.

METHODS

Study design

This was a single‐center prospective randomized controlled trial using a high‐fidelity ultrasound simulator to assess TEE learning and application. The study was reviewed and approved by the local institutional review board. All data were deidentified and kept confidential. All participants provided written informed consent to participate in the study.

Study setting and population

This study was conducted at an academic health sciences center that is a training site for an emergency medicine residency training program. The hospital is the primary ultrasound training site for the residency program and has an active emergency ultrasound fellowship program. All emergency medicine residents who were on their emergency medicine rotation at the time of the study were invited to participate. Inclusion criteria included having completed a core and advanced ultrasound rotation that includes training in TTE. Exclusion criteria included previous training in TEE.

Study protocol

The study flow diagram is outlined in Figure 1. Participants completed a baseline questionnaire to assess their familiarity with TTE and TEE. They then attended a 30‐min lecture on emergency TEE. The lecture covered the function of the equipment, indications and contraindications to performing the scan, the scan views, and common pathologies. The scan views were presented in a unique order that was not used again in the study to avoid bias. Following the lecture, participants completed an orientation to the simulator. The simulator was a Vimedix ultrasound simulator (CAE Healthcare), which has previously been validated for assessing TEE skill level. ⁸ Participants all completed a baseline skill assessment consisting of acquiring five TEE views (midesophageal four‐chamber, midesophageal long axis, bicaval, transgastric short axis, descending aorta) once in fixed sequence and once in random order. Half of the participants in each group performed the fixed sequence first, and the other half performed the random sequence first. The view sequences were provided to the participants by the study staff as they completed the scan. These views have previously been shown to be feasible and helpful for guiding management in ED patients. ² , ⁹

FIGURE 1.

Study flow diagram. TEE, transesophageal echocardiography

Participants were subsequently randomized into either the variable or blocked practice groups using a random‐number generator (www.random.org). The blocked practice group completed 10 repetitions of the same sequence of five TEE views. The variable practice group completed 10 different sequences of five TEE views in random order. During this practice period, participants were allowed to refer to a reference card. In addition, an instructor provided feedback on their scan technique during image acquisition as well as on the quality of their final image.

To test skill acquisition, participants completed a posttraining skill assessment. This consisted of acquiring two additional sequences of the five images, once in fixed sequence and once in random order. Due to the potential confounding effect of additional learning during the first sequence, half of the participants in each group performed the fixed sequence first, and the other half performed the random sequence first.

To test knowledge retention, participants returned 2 weeks after training for a skill transfer test. They were provided two clinical patient scenarios and were asked to obtain the best TEE views possible and provide a diagnosis based on their scan. These scenarios included a patient in cardiogenic shock and a patient with a massive pulmonary embolism. The scans were performed once in a previously practiced sequence and once in a new random order that they had not previously practiced. Half of the participants in each group performed the practiced sequence first, and the other half performed the random sequence first. Following the transfer test, a questionnaire was completed to determine their self‐rated confidence in performing TEE.

Measurements and outcomes

Assessment of learning was performed using an Objective Structured Assessment of Technical Skills (OSATS) as well as analysis of simulator metrics. For the OSATS, the final image for each view was captured and stored for subsequent review. The OSATS tool was adapted from a previously validated echocardiography scoring tool. ¹⁰ In addition, participants were asked to provide a final diagnosis for the two clinical scenarios provided to them. Two expert reviewers blinded to the participant data reviewed each scan image for quality using an anchored 5‐point scale (Appendix S1). A score of 3 or higher was defined as an image that could be used for clinical decision‐making purposes and was the definition of a successful view.

Simulator metrics were automatically recorded for each TEE view obtained. Metrics included the number of probe accelerations from rest, the total distance the probe traveled, the angular distance the probe completed, and the total time spent performing the scan. Simulator metrics have previously been shown to be a valid measure of TEE skill level. ¹¹ , ¹² These metrics were exported into an Excel spreadsheet (Microsoft Corp.).

The primary outcome of this study was percentage of successful views on the transfer test. Secondary outcomes included efficiency of motion as determined by simulator metrics, percentage of correct patient diagnoses, and confidence levels performing TEE.

Data analysis

We selected an effect size of 1.2 standard deviations (SDs) to be significant based on previous procedural skills training literature. ¹³ , ¹⁴ , ¹⁵ Using a power of 0.8 and a two‐tailed alpha of 0.05, we required 11 participants in each group. OSATS scores and simulator metrics were compared using paired t‐tests, and categorical data were compared with the chi‐square test. Significance was set at an alpha of 0.05.

RESULTS

A total of 33 residents were invited to participate in the study, and 28 (84.8%) completed the study (14 in the blocked practice, 14 in the variable practice groups). Baseline characteristics of the participants is listed in Table 1. All participants had completed prior training in bedside TTE. No participants reported having any training in TEE, and none had performed a TEE scan prior to the study. Prior to training, there were no significant differences in rates of successful views or simulator metrics between the groups. There was no significant difference in the time spent practicing the technique on the simulator between the groups (29 min vs 30 min; p = 0.75). No participants in either group had any further exposure or practice of TEE during the washout period.

TABLE 1.

Baseline characteristics of participants

	Blocked practice (n = 14)	Variable practice (n = 14)
Age (years)	34.3 (±5.5)	31.4 (±2.0)
Women	6 (42.9)	5 (35.7)
Men	8 (57.1)	9 (64.3)
Year of training
PGY‐3	3 (21.4)	3 (21.4)
PGY‐4	4 (28.6)	2 (14.3)
PGY‐5	7 (50.0)	9 (64.3)
Confidence with TTE, 10‐point scale	7.2 (±1.7)	6.5 (±1.5)
Baseline successful TEE views (%)	46.4 (±28.7)	52.9 (±28.1)

Note: Data are reported as mean (±SD) or n (%).

Abbreviations: PGY, postgraduate year; TEE, transesophageal echocardiography; and TTE, transthoracic echocardiography.

Performance on the OSATS

Prior to training, there were no significant differences in successful TEE views between the blocked and variable practice groups (46.4%, 95% confidence interval [CI] 31.7%–61.1% vs. 52.8%, 95% CI 24.8%–67.5%; p = 0.55). After training, both groups achieved successful views in almost every attempt (100%, 95% CI 99.5%–100% vs. 97.8%, 95% CI 95.7%–99.9%; p = 0.07). On transfer testing, the blocked practice group had a higher rate of successful views than the variable practice group (93.6%, 95% CI 89.9%–97.3% vs. 77.6%, 95% CI 69.7%–85.4%; p = 0.002; Figure 2).

FIGURE 2.

Percentage of successful views in the blocked practice and variable practice groups. Error bars represent 95% CIs

The mean image score on a 5‐point scale was higher in the blocked practice group compared with the variable practice group (3.3 vs. 2.9; p = 0.01). These results remained consistent for both a previously practiced and a novel sequence of TEE views (Table 2). There was no significant difference in the proportion of correct diagnoses on the simulated patient encounter between the groups (78.5% correct in the blocked practice group, 71.4% in the variable practice group; p = 0.54).

TABLE 2.

Results of the Objective Structured Assessment of Technical Skills (OSATS) on the transfer test

	Blocked practice (n = 14)	Variable practice (n = 14)	p‐value
Successful views (%)	93.6	77.6	0.002
Overall image score (5‐point scale) a	3.3 (±0.7)	2.9 (±0.8)	0.01
Image score for novel scan sequence (5‐point scale) a	3.4 (±0.7)	3.0 (±0.8)	0.01
Image score for previously practiced scan sequence (5‐point scale) a	3.2 (±0.6)	2.9 (±0.9)	0.06
Correct diagnosis (%)	78.5	71.4	0.54

^a Data are reported as mean (±SD).

Performance on simulator metrics

Prior to training, there were no significant differences between the groups for any of the efficiency metrics captured by the simulator. On transfer testing, the blocked practice group outperformed the variable practice group in all four simulator measures of efficiency; however, only the number of angular changes reached statistical significance (Table 3). The number of probe accelerations from rest, total probe distance traveled, and time to complete the scan were all lower in the blocked practice group; however, these differences were not statistically significant.

TABLE 3.

Results from simulator metrics on the transfer test

	Blocked practice (n = 14)	Variable practice (n = 14)	p‐value
Number of accelerations from rest	292.5 (±126.6)	352.5 (±157.6)	0.12
Total probe distance traveled (cm)	149.2 (±87.9)	208.1 (±146.2)	0.07
Total angular changes (degrees)	2982.5 (±1752.5)	4239.8 (±2769.7)	0.04
Time to complete scan (sec)	171.6 (±51.5)	196.1 (±68.1)	0.16

Note: Data are reported as mean (±SD).

Confidence

There was no significant difference between the groups in self‐reported confidence level for TEE on a 10‐point scale (6.5 vs. 5.8; p = 0.35; Table 4). Similarly, there were no significant differences in self‐reported confidence for any specific TEE views.

TABLE 4.

Confidence in performing TEE on 10‐point scale

	Blocked practice (n = 14)	Variable practice (n = 14)	p‐value
Baseline confidence in TEE	1.1 (±0.3)	1.0 (±0)	0.15
Confidence after training session	7.4 (±1.4)	6.6 (±2.1)	0.28
Confidence after transfer test	6.5 (±1.9)	5.8 (±1.8)	0.35

Note: Data are reported as mean (±SD).

Abbreviation: TEE, transesophageal echocardiography.

DISCUSSION

In this study, we found that a blocked practice teaching strategy outperformed variable practice for teaching TEE to emergency medicine residents using a simulator. Participants who practiced the same sequence of views each time had a significantly higher rate of successful views on a delayed transfer test. These results remained consistent even when participants completed a completely novel sequence of TEE views. The blocked practice group also demonstrated improved efficiency of motion, as evidenced by better scores on simulator motion metrics and shorter time to completion of the scan. Both groups exhibited similar levels of confidence after the training period.

Clinician‐performed focused TEE is emerging as a potentially useful tool for managing critically ill patients. TEE can provide valuable information on the presence of important findings such as left ventricular dysfunction, right ventricular dilation, regional wall motion abnormalities, pericardial effusion, and aortic dissection. In unstable patients, the ability to quickly rule in and rule out these conditions can save time and expedite management for life‐threatening diagnoses. In contrast to conventional TTE, TEE is able to generate higher quality images and the transducer can be left in place to help monitor the resuscitation in real time. TEE is not affected by common barriers to TTE such as large body habitus, hyperinflated lungs, and subcutaneous emphysema. Studies have shown that after a brief training session, emergency physicians are able to learn a simplified focused TEE protocol and use it to guide resuscitation in the ED. ¹

One of the challenges associated with implementing TEE in the ED is how best to train emergency physicians on performing this technique. Simulator‐based training has been shown to be highly effective for teaching TEE; however, there is a lack of data to guide how to use this technology most efficiently. Traditionally, an elective comprehensive TEE is performed using a sequence of different views completed in a specific order. However, unlike elective TEE scans, focused resuscitative TEE scans performed in the ED are goal directed, and views may be obtained in varying orders depending on the clinician's suspicion of pathology. It is therefore possible that learning the different views in a variable order may promote retention of skills for focused TEE scans.

There is evidence that introducing variability into the practice method may have a positive effect on learning and facilitate transfer to new clinical situations. In a landmark study of 72 participants, Shea et al. ¹⁶ compared random and blocked practice methods for learning and retention of three motor skills. They found that on a transfer test 10 days later, the random practice group outperformed the blocked practice group, with the effect being most pronounced for more complex transfer tasks. They reasoned that random practice created a more difficult learning environment, which forced the participants to use multiple processing strategies during the learning process, leading to better skill performance and retention. Hatala et al. ¹⁷ evaluated the effect of mixed practice compared with blocked practice on electrocardiogram (ECG) learning by medical students. Students who practiced using a broad variety of ECGs outperformed those who practiced using multiple examples of similar ECGs.

Other studies have not found a benefit to a variable practice strategy for learning and retention. Drummond et al. ¹⁸ compared fixed and variable practice for teaching pediatric asthma scenarios to 85 medical students. In that randomized trial, they found no differences between the groups in performance on long‐term retention of skills. Rivard et al. ¹⁵ randomized 36 medical students and residents to blocked or random practice for laparoscopic surgical tasks. They found no difference between the groups on retention test performance and hand motion metrics analysis.

Some studies have found a benefit to a blocked practice strategy over variable practice. Tofil et al. randomized 23 pediatric residents to repeated practice of a single patient clinical scenario versus mixed scenarios. ⁷ They found that the repeated practice group demonstrated improved performance for tasks in similar patient scenarios. Brydges et al. ¹⁹ evaluated the effect of variable practice on learning a specific orthopedic procedure. They found a trend toward improved performance using variable practice; however, performance was higher in the blocked practice group on a delayed transfer test.

The results from our study show that practicing a fixed block of TEE views leads to improved performance on subsequent scans, including on never‐before practiced sequences of views. These results may be explained due to a mastery learning effect, where repetition of the same steps allows for a rapid achievement of competency. Mastery learning has been shown to be effective for a wide variety of technical skills including thoracentesis, lumbar puncture, and central venous access. ²⁰ , ²¹ , ²² In these studies, the same steps are repeated until competency is achieved. It is likely that in the variable group, the practice conditions were changed before mastery was achieved, leading to worse performance on the transfer test. The TEE probe manipulations required are relatively new in emergency medicine, and it is likely that the complexity of the skill leads itself to favor repetition over variation.

LIMITATIONS

This study has several important limitations. Residents who agreed to participate were likely those who were more motivated to learn TEE, leading to a potential for selection bias. However, participants appeared to have similar baseline characteristics in each study group. Due to limited resident availability, we were only able to assess skill retention 2 weeks after training. We therefore do not know if these results would remain consistent after a longer washout period. Our sample size of 28 participants was small; however, we had adequate power to detect an effect size of 1.2 in this study. Finally, this study was conducted on a simulator; therefore, we cannot comment on whether these findings translate into real‐world scan performance. However, previous studies have demonstrated that performance on this simulator translates well into scan performance on actual patients.

CONCLUSIONS

When training emergency physicians how to perform TEE scans on a simulator, repeating a fixed sequence of views appears to be superior to practicing variable sequences of views. A fixed sequence strategy resulted in higher image quality and increased probe efficiency on delayed transfer testing. Future larger studies using real‐world scans are needed to validate these results.

AUTHOR CONTRIBUTIONS

Jordan Chenkin was responsible for study concept and design, acquisition of data, analysis and interpretation of the data, drafting of the manuscript, critical revision of the manuscript, and statistical expertise. Tomislav Jelic was responsible for study concept and design and critical revision of the manuscript. Edgar Hockmann was responsible for analysis and interpretation of the data and critical revision of the manuscript.

CONFLICT OF INTEREST

The authors declare no potential conflict of interest.

Supporting information

Appendix S1

Chenkin J, Jelic T, Hockmann E. Optimizing simulator‐based training for emergency transesophageal echocardiography: A randomized controlled trial. AEM Educ Train. 2023;7:e10845. doi: 10.1002/aet2.10845