Resident performance in basic perioperative transesophageal echocardiography

Abstract

Background:

Training in transesophageal echocardiography (TEE) is based on hands-on training in the operating room, which is time consuming and therefore limits its experience among anesthesiologists. Medical simulations have been successfully used for training of invasive procedures in many areas.

This paper compares the difference in effectiveness of teaching the 11 basic TEE views using either e-learning, simulation based training or hands-on training in the operating room in 3 groups of residents.

Methods:

We included 51 anesthesia and intensive care residents of all training levels but no prior training in echocardiography in this prospective randomized single-center study.

Residents received a tutorial about theoretical knowledge followed by 2 practical study sessions either by e-learning using an online simulator (www.pie.med.utoronto.ca/TEE), with the simulation mannequin (CAE Vimedix Simulator) or in the operating room. Both, a theoretical multiple choice test (0–50 points) and a practical exam test (0–110 points) on the simulation mannequin had to be completed.

The primary endpoint was the post-training scores in the practical and theoretical exams after all training sessions.

Results:

Residents received significantly higher test scores in both practical and theoretical examinations after training with the simulation mannequin (108.41 ± 2.09, 40.6 ± 5.23, n = 17) compared with e-learning (106.88 ± 4.53, 36 ± 4.76, n = 17) or hands-on training (106.82 ± 2.01, 34.94 ± 4.72, n = 17).

Conclusions:

Simulation based TEE training provides more effective training than other teaching methods. It is therefore especially suitable for the initial stages of TEE training to acquire psychomotor skills and knowledge of echo-anatomy.

1. Introduction

Intraoperative transesophageal echocardiography (TEE) was first used in the 1980s and has increased in importance and usage over the last 3 decades. Today it is no longer confined to cardiac surgery as its benefits are well known for non-cardiac surgery as well as for critical care medicine.^[1–3] Consequently, recent guidelines recommend the use of TEE in both cardiac and non-cardiac surgery and implement the basic perioperative transesophageal echocardiography examination with 11 TEE views for non-cardiac surgery.^[4–6] According to the American Society of Echocardiography (ASE) and the Society of Cardiothoracic Anaesthesiologists (SCA) the main goal of the basic TEE examination is intraoperative monitoring. The basic TEE examination focuses on the 11 most relevant views, which enable the evaluation of general hemodynamic instability. According to guidelines a complete basic perioperative TEE examination should be performed on each patient as a standard examination. Once completed and stored, a more focused examination can be used for monitoring and to track changes in therapy.^[6]

Increasing interest and demand in TEE is leading to a growing need for TEE education and training opportunities. These have traditionally been based on didactic lectures and hands-on training in the operation theatre. Teaching in the operation room is considered as the “golden-standard” but poses many challenges, including a limited number of available cases and experienced teaching faculty, working patterns of the trainees, high stress environment as well as a potential risk to patients when beginners are performing TEE examinations.^[2,3] Studies have shown that simulation based medical education can be used as an alternative teaching method. It enhances not only patient safety and comfort but is also superior to conventional didactic training (e.g., self study, e-learning).^[7–14] So far there are only 2 studies evaluating simulation based training versus hands-on training in the operation room,^[3,14] and no study comparing all 3 teaching methods.

The aim of our study was to compare the effectiveness of teaching practical and theoretical skills of the 11 basic TEE views with 3 different methods: the traditional didactic training with e-learning, simulation based training, and hands-on training in the operating room. The primary endpoint of the study was the scores in the practical and theoretical exams after completion of the training.

2. Methods

2.1. Trial design and participants

This prospective randomized controlled single-center study was approved by the Ethics committee of the Medical University of Vienna (Ref: 1057/2016). The data that support the findings of this study are available in anonymized form from the corresponding author on reasonable request and in consultation with the data clearing agency of medical university of Vienna. Study data are further publicly available at clinicaltrials.gov (accession number NCT02842632, registrar: Ulrike Weber). The study was registered before start of participant recruitment and study data and results were added when the study was completed. Written informed consent was obtained from all subjects. We included anesthesia and intensive care residents of all training levels (year 1–6) who had no prior training in echocardiography and had not completed a cardiac anesthesia rotation. Residents were invited via e-mail to participate in the study.

2.2. Randomization

Upon enrolment, each resident was assigned a unique identification number which was used in randomization procedures and data collection forms. The random allocation sequence was generated by a statistician prior to study commencement using block randomization with a predefined randomization list.

2.3. Interventions and outcomes

We collected demographic data about age, sex, year of training, specialty of training, and asked subjects about their prior experience and usage of video games in their childhood and currently, as this seemed to be related to performance on simulation based models in previous studies.^[15]

2.4. Baseline assessment

First, all subjects undertook a theoretical multiple-choice pre-test (theoretical test 1) designed by a TEE-certified anesthetist. The test assessed basic knowledge about normal anatomy, physics, probe handling, and standard imaging views. Subjects could achieve up to 50 points. Following this, all residents received a 2-hour tutorial with a theoretical part about normal anatomy, physics, and the 11 basic views according to the American Society of Echocardiography (ASE)^[6] as well as a practical part introducing the TEE simulation model (CAE Vimedix Simulator, CAE Health Care, Montreal, Canada).

2.5. Training

Afterwards residents completed 2 individual training sessions corresponding to their randomization:

The first group took part in a traditional didactic training using e-learning with the freely accessible online TEE simulator (http://pie.med.utoronto.ca/TEE/) and studying the guidelines for the 11 basic views recommended by the ASE. The internet-based learning resource has free online access and was developed by the Perioperative Interactive Education group of the Department of Anesthesia at Toronto General Hospital. It allows viewing standard TEE image planes and has a steerable 3D heart model.^[16] It has been used in previous studies to successfully evaluated TEE teaching methods.^[9,17] Residents in this group were only allowed to study for 1 hour with the online TEE simulator per study session and were not allowed to do any additional studying besides.

The second group received a one-on-one teaching session using the mannequin-based TEE simulator.

The third group had a one-on-one teaching session in the operating theatre during actual operations.

Both the simulation and the hands-on group were led by 1 of 3 participating national TEE-board certified instructors. Each training session lasted 1 hour.

Three months after completion of the study we invited the residents to a follow-up test to evaluate long-term memory. This included a multiple-choice test and a practical examination on the simulation mannequin (third theoretical test and fourth practical test). Residents were asked not to access additional learning resources in between the pre-, post-, and follow-up tests.

2.6. Data acquisition

After each training session the residents completed a practical examination on the simulation mannequin (practical test 1 and 2). This practical examination was completed directly after the training session in the operation room or on the simulation mannequin and after studying with the online TEE simulator. After final completion of the 2 training sessions, residents did a second theoretical multiple-choice test and another practical examination (third practical test). The flow diagram illustrates the time frames between the training sessions and examinations. The theoretical examination included different questions to the pre-test but also had a maximum score of 50 points. In the practical examination each resident was asked to demonstrate the 11 basic views on the simulation-mannequin (to standardize conditions for all groups) within a 10-minute time limit. A grading scale applied successfully by Ferrero et al^[9] in previous investigations was used for the evaluations. In this grading system each of the 11 basic echocardiographic views is evaluated on a scale of 0 to 10 according to predefined criteria including: imaging angle, overall clarity, and visualization of the 3 major anatomic structures pertinent to each view (Tables 1 and 2). Each view could receive a maximum score of 10 and therefore each examination a maximum score of 110. The exams were scored by a national TEE-board certified anesthetist who was blinded to the resident group allocation.

Table 1.

11 Basic views in practical examination with anatomic structures scored.

Table 2.

Transoesophageal image scoring system used in the study.^[9].

We also evaluated the satisfaction of the residents with their training. Therefore, we used a questionnaire comprised of 6 statements designed to focus on different aspects of the training, such as: overall satisfaction and usefulness, perceived benefit, duration of the respective sessions, relevance to the current level of training, expectations of improvement, and whether the residents would recommend the training they received for others. Satisfaction was assessed using a 5 point Likert scale (1 = strongly disagree, 5 = strongly agree). Responses were deemed negative (Likert scale 1 and 2), neutral (Likert scale 3), or positive (Likert scale 4 and 5). The maximum score a resident could award their training was 30 points.^[3]

3. Statistics

3.1. Primary outcomes

The primary endpoint of the study was the scores in the practical and theoretical exams after the training was completed (practical exam test 3 and theoretical exam test 2 in Table 3).

Table 3.

Test scores of residents in the 3 intervention groups.

3.2. Sample size calculation

To account for multiple testing and control the overall type I error rate, a hierarchical testing procedure was applied. The null hypothesis on the theoretical test score was tested first. Only if this test was significant at the 5% level, the null hypothesis on the practical exam would be tested in a confirmative way, also at a significance level of 5%. The order of the 2 hypotheses in the hierarchical testing approach was chosen, because information required for sample size planning was available for the theoretical exam score only, and the first test in the hierarchical procedure should have sufficient power. Hence, the sample size was planned for the between groups comparison of the theoretical test score using the analysis of covariance model. Based on recent literature, the standard deviation within 1 group was assumed as 5 points^[3,12,17] and a difference of 5 points in the theoretical test (10% of the maximal achievable exam score results) was considered as minimally relevant difference. Assuming a true difference of 5 points between 1 training group and the other 2 groups, 17 students should be included in each group to achieve 80% power at a significance level of 5%.

3.3. Statistical methods

The effect of training group on the theoretical test score after training was analyzed using analysis of covariance: a linear model was fit explaining the test score after training through the factor training group and the baseline theoretical test score. The null hypothesis of no mean differences between the training groups in the theoretical test score after training was tested from this model using an F-test for the factor training group.

A Kruskal-Wallis test was used to test the null hypothesis of identical distributions of the practical test score after the training in all 3 training groups. This non-parametric approach was preferred over a parametric test due to the skewed distribution of the practical test scores.

All other comparisons are regarded as secondary analyses. For comparisons between training groups we used analysis of variance for the theoretical score at baseline, analysis of covariance, similar to the main analysis, for the follow up theoretical test, and Kruskal-Wallis tests for the practical test scores. For comparisons between time-points we used paired t tests for the theoretical test score and Wilcoxon signed rank tests for the practical test scores. To analyze potential effects of prior experience of video games in childhood (>1/wk) on the test scores we calculated t tests and Wilcoxon rank sum tests for the theoretical and practical test scores, respectively.

For descriptive purposes we calculated the mean, standard deviation, minimum, and maximum for metric variables. For categorical variables we report absolute frequencies.

4. Results

Fifty-five residents were included in the study, 51 completed the protocol, 4 residents were excluded after the initial tutorial due to schedule difficulties (flow diagram) (Fig. 1).

Figure 1.

Flow diagram: number of residents included in each group and time schedule for training sessions and examinations.

Demographic data is presented in Table 4.

Table 4.

Demographics of residents in the 3 intervention groups.

Table 3 summarizes the scores of the practical and theoretical examinations in the groups. For the theoretical score, the mean difference adjusted for the baseline test score was 4.2 (95% confidence interval [CI] [0.9–7.5]) between the simulation group and the e-learning group and 5.3 (95% CI [2.0 to 8.5]) between the simulation group and the operating room group.

We analysed potential differences between the test scores of residents with and without childhood experience with video games. There was a significant difference in the theoretical test scores at baseline (playing video games vs no video games: 30.3 ± 4.1 vs 27.1 ± 4.8, P = .014), but no significant difference in the theoretical test scores at the second theoretical test (37.5 ± 5.1 vs 36.8 ± 5.9, P = .668), the first (101.8 ± 7.7 vs 95.3 ± 14.9, P = .071), second (107.5 ± 1.9 vs 105 ± 8.8, P = .268), or third (107.7 ± 2.3 vs 107 ± 3.9, P = .557) practical exam between the groups.

The average time to complete the practical examination decreased from the first practical test (6.54 ± 2.15 minutes) throughout the second (5.53 ± 1.86 minutes), the third exam (5.21 ± 2.03 minutes), and the follow up (5.11 ± 1.86 minutes) in all groups.

There was no significant difference in the average score of satisfaction with the training on the Likert scale in the didactic (27.47 ± 4.37; range 13–30), the simulation (29.41 ± 1.23; range 26–30), and the operating room group (29.18 ± 2.04; range 23–30).

5. Discussion

Our study shows that residents with no prior knowledge of echocardiography received significantly higher scores in the post-training practical and theoretical exams in the simulation group compared with the other two groups. All groups showed significant improvement from the pre- to the post-training exams.

This is the first study comparing three different teaching methods for transesophageal echocardiography. Previous studies either evaluated the difference between simulation training and conventional didactic training and e-learning^{[9,10,12,13,18,19]} and only two authors compared simulation training with hands-on training in the operating room,^[3,14] which is considered as the “golden-standard” for teaching TEE. The majority of these studies revealed a superiority of simulation training compared with the other methods.^{[3,9,10,12,14,18,19]}

Only Sohmer et al^[13] reported no difference between the groups. They evaluated 10 commonly used transesophageal views after an instructor guided lesson with the simulator or after a self-directed slide presentation of TEE psychomotor skills. Similar to our study the residents performed both a theoretical and practical examination and both groups improved their results significantly, but did not show differences between the teaching methods. Compared with our study they spent less time on the simulation mannequin. Our residents had two study sessions of one hour with a 1:1 instructor to student ratio, while residents in the study of Sohmer et al. only received 30 minutes of hands-on experience with a 4:1 instructor to student ratio.

We demonstrated that simulation is superior to didactic learning, but especially to e-learning, in both theoretical and practical examinations. There was only one study that evaluated e-learning using a web-based TEE simulation system like our study. This study of Ferrero et al^[9] was also able to demonstrate that simulation training is superior to traditional didactic training using a practical examination for evaluating 10 commonly used TEE views.^[9] In a study conducted by Bose et al^[12] the simulator-trained group performed substantially better than their counterparts in the cognitive aspects of TEE when evaluated by written testing. This study did not assess technical skill performance in the clinical setting. However, literature and our study findings imply that hands on practice are the key for the initial stages of mastering TEE skills and TEE knowledge.

Further, literature also suggests an advantage for trainees who engage in video games. They reported better visual, spatial, and motor coordination and showed that residents who played video games improved significantly in their laparoscopic skills compared with their peers.^[15] When we analyzed potential differences between the test scores of residents with and without childhood experience with video games, we were able to demonstrate that there was a significant difference only in the theoretical test scores at baseline. The practical tests demonstrated higher test scores in those residents with childhood experience with video games, but were not significant. We observed the highest difference between the test scores in the first practical test (101.8 ± 7.7 vs 95.3 ± 14.9, P = .071), and decreasing differences from the second (107.5 ± 1.9 vs 105 ± 8.8, P = .268) to the third exam (107.7 ± 2.3 vs 107 ± 3.9, P = .557).

However, our study was the first to evaluate the 11 views recommended by the ASE and SCE guidelines for performing a basic TEE examination.^[6] We have chosen the 11 views included in the basic perioperative TEE examination protocol since it is the goal of our department to train every anesthesiologist and intensivist to be able to perform a basic TEE examination. The authors believe that the results of the study are therefore more relevant to general anesthesia departments. Previous studies either assessed the 20 view model or just different standard views that are commonly used in clinical practice.^[9,11,13] The American Society of Echocardiography recommends the 11 basic views in the care and treatment of an unstable surgical patient as a monitoring tool.^[6] The 11 basic views make for a comprehensive examination on their own and are easier and faster to perform than a 20 + 8 view standard examination and therefore more suitable for beginners. The basic examination can facilitate the understanding of hemodynamic assessment in beginners. Prat et al^[19] demonstrated in their study that simulation training improves the learning curve of ICU TEE hemodynamic assessment. In the simulation group they reported a significant improvement in the ability of trainees to record the main views and in technical skills, like measuring simple hemodynamic parameters accurately.

In our study the residents first received a theoretical tutorial of 2 hours followed by 2 practical sessions of 60 minutes each with a one-on-one teaching setting, which was longer than in other studies.^[3,9,10,14]

Residents not only improved their practical test score, but also achieved shorter examination times after each training session. This suggests that they gained more confidence with each training session.

We performed a follow up examination on the trainees 3 months after they completed their TEE training to assess possible long term benefit. This follow up demonstrated that both theoretical and practical test scores decreased only slightly and that the scores were still higher than the initial scores. This implies that the 11 basic views can be mastered only with a few training sessions.

6. Limitations

We observed more senior residents in the simulation group, but despite their clinical experience there was no difference in baseline theoretical echocardiography knowledge (theoretical test 1) between the 3 groups.

All residents did their practical examination on the simulation mannequin. This might be a potential benefit for those residents who were assigned to the simulation group because they were used to the setting and spent most time with the simulation model. Especially after the first training session residents in the simulation group received significantly higher scores with a more obvious difference than their colleagues in the other groups. But this could also be a result of the superiority of this teaching method. Scores in the 3 groups became closer in the second and the third practical exam, but still remained significant higher.

Another possibility would have been to conduct the practical exams in the operating room. Other authors compared simulation training with training in the operating room, but they did not evaluate the skills of the trainees on actual patients. The instructors took the pictures while the trainees were only responsible for answering theoretical questions.^[3,14] Ferrero et al^[9] were the only investigators who assessed the simulation training skills on actual patients in the operating room. One explanation is of course time factors and the difficulty to find patients with acceptable image quality (e.g., without any major pathologies) suitable for beginners. On the other hand patient safety is an issue. Early TEE training involves multiple probe positions which can be potentially traumatic to the patient. Kinematic studies demonstrated that there is a significant decrease in TEE probe accelerations over time with simulator training, but trainees still have more transitions and “jerking motions” when compared with experts.^[11,20]

We evaluated knowledge and demonstration of normal images only. But proficiency in TEE requires knowledge of the machine, image acquisition skills and the ability to differentiate between normal and abnormal images for image interpretation and decision making.^[3] Therefore, further studies are needed to evaluate training in TEE with abnormal images.

However, image acquisition skills can only be gained at the bedside and require repetitive exposure and practice.^[11]

To date, there has been little development of curricula geared specifically at the psychomotor component of TEE. Skills training for TEE requires a 1:1 ratio of instruction on patients and much repetition.^[7,8,13] Combined with the limited clinical indications and opportunities, the time consuming process of acquisition of psychomotor skills are the bottleneck of gaining TEE competence.^[13]

7. Conclusion

Our study suggests that simulation based TEE training may be more effective than theatre based teaching or traditional didactic training and is especially suitable for the initial stages of TEE training to acquire psychomotor skills and knowledge of echo-anatomy.

Acknowledgments

The authors would like to thank all the medical staff at the Division of Cardiothoracic and Vascular Anaesthesia and Intensive Care Medicine for their collaboration and assistance.

Author contributions

Conceptualization: Ulrike Weber, Eva Base, Bruno Mora.

Data curation: Ulrike Weber, Bernhard Zapletal, Eva Base, Michael Hambrusch, Bruno Mora.

Formal analysis: Michael Hambrusch, Robin Ristl.

Investigation: Ulrike Weber, Bernhard Zapletal, Michael Hambrusch, Bruno Mora.

Methodology: Ulrike Weber, Bernhard Zapletal, Bruno Mora.

Project administration: Ulrike Weber, Bernhard Zapletal, Michael Hambrusch, Bruno Mora.

Resources: Ulrike Weber.

Software: Robin Ristl.

Supervision: Eva Base, Bruno Mora.

Validation: Ulrike Weber, Robin Ristl.

Visualization: Bruno Mora.

Writing – original draft: Ulrike Weber.

Writing – review & editing: Bernhard Zapletal, Eva Base, Robin Ristl, Bruno Mora.