Quantitative Echocardiography Improves Resident Assessment of Left Ventricular Systolic Function

Abstract

Background:

The use of echocardiography to assess left ventricular ejection fraction (LVEF) is an important component of anesthesiology resident education; however, there is no consensus on the most effective method for teaching this skill set. This study investigates the impact and feasibility of teaching a quantitative LVEF assessment method to anesthesiology residents, compared with teaching visual estimation techniques.

Methods:

We included all anesthesiology residents rotating through cardiac anesthesia at our institution from August 2020 through March 2021. Participants completed a pretest to assess baseline ability to accurately estimate LVEF. All tests consisted of transthoracic echocardiography images with standard views from 10 patients. Participants were assigned to either a control group that received teaching on visual estimation of LVEF or an intervention group that was taught quantitative LVEF assessment with the Simpson biplane method of discs. After 4 weeks, all participants were administered a postteaching exam. A retention exam was administered an additional 4 weeks later. LVEF accuracy was measured as the absolute difference between their LVEF estimation and the reference value.

Results:

Control and intervention groups performed similarly on the preteaching exam of LVEF estimation accuracy. Intervention-group residents demonstrated significantly improved accuracy in LVEF assessment on the postteaching exam (3.6% improvement in accuracy, confidence interval [CI], 1.23–5.97; P = .03) compared with the control group (0.60% improvement inaccuracy, CI, −1.77–2.97; P = .62). The observed improvement was not maintained through the retention exam.

Conclusions: Addition of quantitative LVEF assessment to traditional teaching of visual LVEF estimation methods significantly improved the diagnostic accuracy of anesthesiology residents’ left ventricular systolic function assessment.

Introduction

Perioperative echocardiography has been well established as an essential diagnostic and monitoring tool that can provide anesthesiologists with detailed information about cardiorespiratory stability in both cardiac and noncardiac surgery.^1,2 As a rapidly available, portable, and noninvasive diagnostic tool, the clinical application of point-of-care ultrasound (POCUS) has increased exponentially in the past decade.³ Focused cardiac ultrasound protocols have been developed for use in the perioperative arena, providing clinically useful information that changes the course of patient care.^4–6 Effective July 2021, the American College of Graduate Medical Education (ACGME) has highlighted POCUS as a key component of anesthesiology education through its inclusion in the Anesthesiology Milestone Project.⁷ In addition, the American Board of Anesthesiology will add “Interpretation of Echocardiograms” and “Application of Ultrasound” sections to the skill set tested during the Objective Structured Clinical Examination beginning in the spring of 2022.⁸ Despite these efforts, the implementation of an ultrasound curriculum in anesthesiology residency remains program specific and nonstandardized.⁹

Whereas estimation of the left ventricular ejection fraction (LVEF) is one of the most helpful tools in the evaluation of patients in the perioperative setting, this skill also happens to be one of the most difficult to master. As part of a focused cardiac ultrasound exam, simulation-based curricula have been successfully used to teach visual approximation of LVEF.^10–13 Structured teaching interventions in visual estimation techniques led to improvement in resident ability to qualitatively categorize LVEF with improved image quality and structure identification and a decrease in interobserver variability.^10–13 The improvement in image interpretation was most pronounced in images representing extremes of systolic function.^10,14 Although it is the cornerstone of current cardiac ultrasound education, qualitative assessment of LVEF has been shown to be less accurate and less reproducible than quantitative assessment by the biplane method of discs or modified Simpson’s rule.¹⁵ The efficacy of teaching quantitative methods of LVEF and the incorporation of these techniques into a curriculum has not been explored in studies of anesthesiology resident education.

Another important feature of ultrasound education to consider is the attrition of knowledge following an educational intervention. Whereas studies have shown that participants exhibit improvement and competency in echocardiogram skills immediately following an educational intervention, these skills wane over time if learners do not regularly use ultrasound in their practice.¹⁶

The goal of this study was to evaluate the feasibility and the impact of teaching quantitative methods for evaluating LVEF compared with visual estimation techniques in a cohort of anesthesiology residents. We hypothesized that implementation of structured education in quantitative LVEF estimation techniques taught to anesthesiology residents would result in improved LVEF diagnostic accuracy compared with residents taught qualitative estimation techniques alone.

Materials and Methods

Study Design

This study was approved by the UCLA David Geffen School of Medicine institutional review board. From August 2020 to March 2021, we prospectively enrolled second-year (CA-2) and third-year (CA-3) clinical anesthesiology residents during their cardiothoracic anesthesia rotation. For CA-2 residents this was an 8-week rotation and their first exposure to cardiac anesthesia during residency. The CA-2 residents had not received structured cardiac ultrasound education as part of the residency curriculum prior to this rotation. The CA-3 residents spent 4 weeks on cardiac anesthesia, having previously completed the 8-week rotation during the CA-2 year. Due to differences in cardiac ultrasound experience inherent to residents enrolled in the study, the CA-2 and CA-3 residents were divided as equally as possible into control and intervention groups.

Both groups were administered a preteaching exam to evaluate baseline skill in LVEF assessment. Each examination consisted of transthoracic echocardiography (TTE) clips from 10 different patients using 4 standard TTE views each: parasternal long-axis, parasternal short-axis, apical 4-chamber, and apical 2-chamber. The TTE images were all acquired by a single cardiac ultrasonographer using the GE Vivid S70 ultrasound machine on awake, spontaneously ventilating patients. All exams were administered in person and one-on-one with a proctor in the echocardiography laboratory using the TTE clips saved on the ultrasound machine. During each exam, participants were asked to assess LVEF after review of all of the clips from a single patient by providing a single value (eg, LVEF of 45%). Accuracy of LVEF assessment was determined as the absolute difference between the LVEF value chosen by the resident and the reference value. Reference values were determined as the average LVEF obtained by 2 cardiac anesthesiologists, board certified in echocardiography, using the Simpson biplane method of discs. The 10 patients to be included in each of the 3 exams were chosen by the same cardiac anesthesiologists to ensure uniformity in image quality and inclusion of patients with a wide range of LVEFs. The TTE clips in which the reference attending’s interpretation of LVEF differed widely were not included.

Control Group

After administration of the preteaching exam, residents in the control group received one 60-minute, one-on-one teaching session on visual estimation of LVEF by 1 of 3 cardiac anesthesiologists board certified in echocardiography. As part of the qualitative assessment of LVEF, residents were taught to evaluate regional and global myocardial wall thickening (normal > 30% thickening; mildly reduced was 10%–30% thickening; severely reduced < 10% thickening and akinesis), regional wall motion abnormalities, and overall change in chamber size throughout the cardiac cycle. The TTE image datasets from 5 preselected patients with varying LVEFs were used in the teaching session. All teaching sessions occurred in the echocardiography laboratory using the GE S70 ultrasound machine during the first week of the cardiac anesthesia rotation. Control-group residents were encouraged to continue practicing qualitative LVEF assessment in the operating room during their cardiac anesthesia rotation. Four weeks following the structured teaching session, residents were administered the postteaching exam, in which they were asked to evaluate LVEF on 10 new TTE image datasets in the same format as the preteaching exam. After the postteaching exam, there was no further structured education on LVEF estimation with the designated cardiac anesthesiologists. Approximately 4 weeks after the postteaching exam, a retention test with 10 new TTE image datasets was administered in the same format.

Intervention Group

After the preteaching exam, residents in the intervention group received one 60-minute, one-on-one teaching session on quantitative measurement of LVEF using the Simpson biplane method of discs by 1 of the same 3 cardiac anesthesiologists. The teaching session consisted of explanation of the concept, identification of left ventricular (LV) end-diastolic frame, identification of LV end-systolic frame, and endocardial border tracing on the same 5 TTE image datasets used with the control group. Participants were given the opportunity to perform a quantitative assessment of LVEF using the Simpson biplane method of discs under direct supervision during the tutorial. Intervention-group residents were given 2 additional 15-minute opportunities to practice endocardial border tracing and knobology on the GE S70 ultrasound machine prior to the postteaching exam. Postteaching and retention exams were administered in an identical time frame and format as those given the control group. On the postteaching and retention exams, intervention-group residents were encouraged to use the Simpson method for quantitative assessment of LVEF, although they could override the final LVEF answer with their own visual estimation if they believed it to be more accurate for the given TTE clip. The test proctor was permitted to help with knobology as needed, but not with identification of appropriate LV frames or endocardial border tracing.

Intraoperative Teaching

All cardiac anesthesia attendings and fellows were aware of the grouping for each participant during their rotation. The attendings were able to teach visual estimation techniques of LV systolic function to all participants. However, they were prohibited from teaching quantitative methods to participants in the control group. Residents in both groups otherwise had comparable intraoperative echocardiography teaching experience, including going over basic transesophageal echo views, qualitative wall motion abnormality evaluation, and evaluation of valvular pathologies if clinically indicated.

Statistical Analysis

To assess the impact of the teaching intervention on improving LVEF assessment accuracy, we ran generalized linear and logistic mixed effects models for LV (linear) systolic function. The terms of the models were fixed effects for intervention (yes/no), time (preteaching, postteaching, retention), and random effects for exam and participant. Pairwise contrasts within or between groups (with 95% confidence intervals [CI] and P values) were extracted from the models. Analyses were conducted using SAS version 9.4 (SAS Inc, Cary, NC) and P values < .05 were considered statistically significant.

Results

Demographic data of the study population is presented in Table 1. All enrolled residents (28) completed the study protocol in its entirety. Table 2 shows the mean LVEF accuracy compared with the reference value for control and intervention groups at each study time point, as well as the mean difference in accuracy between control and intervention groups. There was no statistically significant difference in LVEF accuracy on the preteaching exam between the control group (9.75% ± 7.49%) and the intervention group (10.19% ± 7.54%), with a mean difference of −0.46% (CI, −2.10%–1.19%; P = .59). The intervention group performed significantly better than the control group on the postteaching exam (6.76% ± 6.20% vs 9.30% ± 6.90%, respectively), with a mean difference of 2.54% (CI, 0.90%–4.19%, P = .003). This treatment effect was not maintained through the retention exam, given that both groups again demonstrated no significant difference in LVEF accuracy (control group: 9.06% ± 6.93% vs intervention group: 8.79% ± 7.07%), with a mean difference of 0.26% (CI, 1.42%–1.94%, P = .76).

Table 1.

Resident Breakdown by Year of Training

Training Level	Control Group (n = 14)	Intervention Group (n = 14)
CA-2	9	8
CA-3	5	6

Abbreviations: CA-2, second-year clinical anesthesiology resident; CA-3, third-year clinical anesthesiology resident.

Table 2.

Mean LVEF Accuracy Compared With the Reference Value for Control and Intervention Groups (Absolute Value in %LVEF) ^a

Exam	Control (n = 14)	Intervention (n=14)	Differential intervention vs control (95% CI)
	Mean ± SD	Mean ± SD
Pretest	9.75 ± 7.49	10.19 ± 7.54	−0.46 (−2.10–1.19)
Posttest	9.30 ± 6.90	6.76 ± 6.20	2.54 (0.90–4.19)b
Retention	9.06 ± 6.93	8.79 ± 7.07	0.26 (−1.42–1.94)

Abbreviations: CI, confidence interval; LVEF, left ventricular ejection fraction.

^aPositive differential signifies improvement in accuracy.

^bP value = .003.

Within the control group, accuracy in LVEF assessment did not significantly change across the study time points (Table 3). Residents in the intervention group, however, had a statistically significant improvement in LVEF accuracy on the postteaching exam compared with preteaching exam, with a mean difference of 3.60% (CI, 1.23%–5.97%, P = .003). Again, this treatment effect was not maintained through the retention exam, because there was no statistically significant increase in LVEF accuracy for intervention group residents between the preteaching and retention exams (mean difference = 1.33%, CI, −0.76–3.42; P = .21).

Table 3.

Change in Performance Between Phases of Exam Within Group ^a

Study Arm	Pre vs Post (CI)	Pre vs Retention (CI)	Post vs Retention (CI)
Control (n = 14)	0.60 (−1.77–2.97)	0.61 (−1.46–2.68)	0.01 (−2.35–2.37)
	P = .62	P = .56	P = .99
Intervention (n = 14)	3.60 (1.23–5.97)	1.33 (−0.76–3.42)	−2.27 (−4.66–0.11)
	P = .003	P = .21	P = .06

Abbreviations: CI, confidence interval; Post, posttest; Pre, pretest.

^aPositive number means improvement in performance on subsequent exams.

Discussion

The primary aim of this study was to compare the effect on diagnostic accuracy in teaching a quantitative method of LVEF assessment to anesthesiology residents versus the traditional qualitative assessment techniques. Implementation of a structured education program in quantitative LVEF assessment using the Simpson biplane method of discs led to a statistically significant improvement in LVEF assessment accuracy for anesthesiology residents in the intervention group. On the other hand, residents who were taught qualitative visual estimation techniques did not show any improvement in LVEF estimation accuracy over the course of the study. After structured teaching sessions ended, however, the performance of the intervention group quickly regressed back to baseline.

As the perioperative care arena expands beyond the walls of the operating room, so too does the need for a portable and quick diagnostic tool to be used in the clinical differentiation of unstable patients. Focused cardiac ultrasound, as part of a larger POCUS curriculum, fills this need and has been established as a core component of anesthesiology resident education by the American Board of Anesthesiology and ACGME.^3,7–8 Facility with cardiac ultrasound is an expected diagnostic skill for graduating residents as they transition to independent practice, and accurate assessment of global and regional cardiac function and assessment of LVEF is arguably the most important component of this skill set.

Despite these expectations and recommendations, integration of cardiac ultrasound curricula into anesthesiology training programs has been inconsistent.³ Furthermore, the methodology, duration, and frequency of teaching LVEF assessment remains quite variable. Current training environments typically incorporate some combination of hands-on perioperative instruction, self-guided didactics, or simulator-based education with a focus on qualitative visual estimation techniques.^10–13 Visual estimation techniques, however, suffer from significant interobserver variability and may yield agreement as low as 50% even in the hands of experienced echocardiographers.⁶ The addition of quantitative LVEF assessment with the Simpson biplane method of discs was shown to improve diagnostic accuracy in a cohort of critical care physicians,¹⁴ but the positive impact of incorporating this technique into an anesthesiology residency curriculum has never been studied.

To our knowledge, ours is the first study to demonstrate an improved LVEF diagnostic accuracy in anesthesiology residents as a result of structured teaching that focuses on quantitative rather than qualitative LV systolic performance. In fact, sticking with the traditional visual estimation technique did not lead to any significant change in performance over time. The improvement in systolic function assessment was achieved rapidly in the intervention group, with only a single 60-minute tutored session required to significantly affect performance. This may be of particular interest in residency curriculum development because new educational objectives must be feasible and efficient within the time constraints of clinical care, work-hour restrictions, and existing didactics. Although the impact of teaching quantitative LVEF assessment may be seen rapidly, this study also highlights the rapid attrition of a new skill or knowledge that occurs without frequent, deliberate practice. Residency training programs must walk a tightrope between effective implementation of new knowledge and oversaturation or burnout of trainees. The results of this study reinforce that structured ultrasound education is effective but exposure must be longitudinal and frequent. Program directors must find creative ways to incorporate ultrasound education and practice throughout the duration of residency. Simulation, online learning and practice modules, and ultrasound review conferences may all be effective avenues to increase exposure to and familiarity with ultrasound techniques and interpretation.

The added value to anesthesiology residents of using quantitative LVEF assessment can only be measured by the impact it would have on clinical care and patient outcomes. Whereas our current study is unable to comment on this big-picture impact, we have shown that the first steps are achievable. Teaching relatively ultrasound-naïve residents a quantitative tool to assess LV systolic function is feasible with little time cost and is an improvement over the current standard qualitative assessment techniques.

There are a few important considerations in the interpretation of the results of this study. First, intervention group residents were not required to use the LVEF from their quantitative assessment as the final answer on the postteaching and retention exams. They could fall back on a visual estimation if they felt this to be more accurate for the given patient’s TTE clips. As such, the results of this study do not demonstrate that quantitative LVEF assessment techniques are more accurate than visual estimation per se, but rather that teaching residents the concepts, mechanics, and limitations of the Simpson method of discs is feasible and improves resident accuracy in LVEF estimation regardless of chosen technique. In addition, ultrasound imaging platforms are complex and residents in both control and intervention arms received assistance with knobology during the exams. This, coupled with the rapid attrition rate in LVEF accuracy in the intervention group, suggests that additional teaching and more frequent practice, especially for technically cumbersome quantitative techniques, is required for durable learning. Finally, the magnitude of the improvement in LVEF accuracy generated by this teaching intervention was small (absolute difference of 3.6%) and likely clinically insignificant. Using cardiac ultrasound to triage patients into normal, abnormal, and severely abnormal LVEF groups is more clinically relevant to an anesthesiology resident, but the effect of any teaching intervention on large LVEF groups such as this would require a significantly larger study to demonstrate any efficacy. Regardless of the size of the effect, educators should be encouraged by these results and continue to pursue innovations in ultrasound education to create durable and clinically significant improvements in the application of this technology.

We would like to acknowledge additional limitations of this study. Although we tried to make the study groups as homogeneous as possible, we could not control for several resident variables that could have affected performance. Residents and faculty were not blinded to group assignment, and knowledge of their study group may have altered the content of their intraoperative education and structure of their independent learning. The inherent variability of call and case assignments leads to variable exposure to intraoperative echocardiography and cardiac pathology. As previously noted, study groups consisted of residents from different years with varying prior exposure to echocardiography during internship and medical school. The impact of these differences, however, may be mitigated to some extent by the relatively equal balance of junior and senior residents in each group. One additional notable limitation is the short duration of the experimental teaching intervention. Whereas the magnitude and duration of the intervention effect may be significantly greater with additional teaching sessions, this study emphasizes that an effective teaching intervention can be quickly incorporated into an already rigorous cardiac anesthesia rotation without significant time burden added. The study protocol is a reasonable approximation of clinical faculty’s capacity to teach and evaluate residents in a busy training program during normal clinical workflow.

Funding Statement

Research Support: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.