Simulation for competency assessment in vascular and cardiac ultrasound

Abstract

Healthcare providers who use peripheral vascular and cardiac ultrasound require specialized training to develop the technical and interpretive skills necessary to perform accurate diagnostic tests. Assessment of competence is a critical component of training that documents a learner’s progress and is a requirement for competency-based medical education (CBME) as well as specialty certification or credentialing. The use of simulation for CBME in diagnostic ultrasound is particularly appealing since it incorporates both the psychomotor and cognitive domains while eliminating dependency on the availability of live patients with a range of pathology. However, successful application of simulation in this setting requires realistic, full-featured simulators and appropriate standardized metrics for competency testing. The principal diagnostic parameter in peripheral vascular ultrasound is measurement of peak systolic velocity (PSV) on Doppler spectral waveforms, and simulation of Doppler flow detection presents unique challenges. The computer-based duplex ultrasound simulator developed at the University of Washington uses computational fluid dynamics modeling and presents real-time color-flow Doppler images and Doppler spectral waveforms along with the corresponding B-mode images. This simulator provides a realistic scanning experience that includes measuring PSV in various arterial segments and applying actual diagnostic criteria. Simulators for echocardiography have been available since the 1990s and are currently more advanced than those for peripheral vascular ultrasound. Echocardiography simulators are now offered for both transesophageal echo and transthoracic echo. These computer-based simulators have 3D graphic displays that provide feedback to the learner and metrics for assessment of technical skill that are based on transducer tracking data. Such metrics provide a motion-based or kinematic analysis of skill in performing cardiac ultrasound. The use of simulation in peripheral vascular and cardiac ultrasound can provide a standardized and readily available method for training and competency assessment.

Introduction

In the past three decades, medical education has undergone two revolutions that, although converging from different disciplines, are directed towards the goal of improving patient safety that has been spurred by revelations of medical error.¹ The need to focus on competency-based medical education (CBME) was voiced in 1978 by McGaghie in a World Health Organization report: ‘The intended output of a competency-based program is a health professional who can practice medicine at a defined level of proficiency, in accord with local conditions, to meet local needs’.² At about the same time, simulation technology, which was initially developed for aviation and industrial safety, was adapted for medical applications.³ This review will report on the use of simulation to enable CBME in diagnostic ultrasound of blood vessels and the heart.

Competency-based medical education (Table 1)

Table 1.

Competency-based medical education.

Definition of competencies as outcomes of the curriculum: ‘The competency based approach consists of functional analysis of occupational roles, translation of these roles into outcomes, and assessment of trainees’ progress on the basis of their demonstrated performance of these outcomes.’62 Description of how competencies should be assessed: ‘Assessments are based on a set of clearly defined outcomes so that all parties concerned, including assessors and trainees, can make reasonably objective judgments about whether or not each trainee has achieved them.’62

Methods suggested by Accreditation Council for Graduate Medical Education (ACGME) for assessment of medical procedures:5 ‘Most Desirable’: Checklist, Simulations and Models ‘Next Best Method’: 360 Degree Global Rating ‘Potentially Applicable’: Global Rating, Procedure or Case Logs

To evaluate competence based on proficiency rather than case volume it is necessary to define competence and describe how it should be assessed. The Accreditation Council for Graduate Medical Education (ACGME) defined six core competencies common to all programs, developed detailed Milestones to be achieved in each specialty and subspecialty program, and provided a collection of assessment tools to assist program directors in implementing the core competencies.^4,5 Organizations in other countries worked in parallel.⁶

Assessment may be summative (judging a candidate’s ability to perform for certification or promotion) or formative (providing feedback on progress, accomplishment, and areas needing further practice). In addition to providing feedback to learners and progress reports to program directors, assessment metrics are valuable for assessing or comparing new curricula. In diagnostic ultrasound, assessment must encompass technical (psychomotor) skill in image acquisition in addition to cognitive skill in image interpretation. There are many tools for assessing cognitive skill,⁷ but technical skill competency has always been difficult to assess in a manner that is objective and reproducible between instructors when there is no standardized assessment method. Instead, certification has traditionally been, and still is, based on faculty observation of performance. The ACGME ranked several evaluation methods for performance of medical procedures.⁵ The methods have advantages as well as limitations.^8–16 For diagnostic ultrasound the pertinent question is how best to assess technical skill in image acquisition.

Simulation (Table 2)

Table 2.

Advantages of simulation for competency-based medical education.

Advantages for training Enables repetitive practice in a safe learning environment Allows experience with a wide range of clinical scenarios or pathology Avoids concerns regarding patient safety and comfort Compensates for inadequate patient volume Reduces use of clinical resources for teaching Preserves faculty time for advanced training Advantages for assessment Provides feedback Enables standardized testing (metrics are objective and cases are simulated) Informs trainee and faculty of trainee progress Saves faculty time spent recruiting and scheduling patients

The ACGME requires simulation in multiple specialties due to its many important advantages in medical training, which include enabling practice in a safe environment and providing exposure to rare conditions that might not be encountered clinically.^17–20 Since the Agency for Healthcare Research and Quality’s report in 2007 that simulation is especially effective for training in psychomotor skills,²¹ many studies have confirmed that simulation with deliberate practice is more effective in improving skills than the traditional ‘watch one, do one, teach one’ approach, and improves clinical outcomes.^22,23

Simulation is also a very useful tool for assessment. During training, incorporation of metrics for competency testing can provide formative assessment or feedback and thereby help trainees reach proficiency. The ability to provide feedback was cited as a key educational feature of simulation in two reviews.^24,25 This is an important advantage of simulation because feedback leads to the most effective learning and ‘appears to slow the decay of acquired skills’.²⁴ Frequent feedback was recommended as a key component of CBME.² Collection of skill metric data can inform program directors of the rate at which trainees reach competence in specific procedures.²⁶ Training time can be increased in areas where poorly answered questions reflect potential learning needs. Skill metrics have shown that some trainees may never achieve competence.²⁷ Simulation may also assist in training to proficiency, which is advocated by some educators because the traditional approach penalizes fast learners.²⁸

Simulation provides an opportunity to perform rigorous competency testing on cases that have suitable pathology and are standardized not only for all trainees in a program, but for all trainees being considered for certification within a specialty. A survey by the American Association of Medical Colleges in 2011 found that most medical schools and teaching hospitals use simulation for education and/or assessment of psychomotor tasks.²⁹ However, the employment of simulation for summative assessment or certification, credentialing, and accreditation remains in its infancy despite the clear theoretical advantages. ‘Only two specialty boards, anesthesiology and surgery, have required the use of simulation for primary certification.’³⁰ Some authors criticize validity evidence for simulation-based assessments as being sparse and suboptimal.³¹ However, 28 studies that evaluated associations between simulator-based performance and performance with real patients showed that ‘higher simulator scores were associated with higher performance in clinical practice’.³¹

Simulation of vascular Doppler ultrasound

The need for skills assessment

In most applications of medical ultrasound the diagnosis is made from two-dimensional (2D) B-mode (gray scale) images alone. However, classification of disease severity in peripheral vascular applications is based primarily on alterations in velocities and flow patterns detected in the imaged vessels. In duplex ultrasound scanning, these flow patterns are characterized by Doppler spectral waveforms and color-flow imaging.³² The criteria for classification of arterial stenosis are based on peak systolic velocity (PSV) obtained from the Doppler spectral waveforms.³³

Because the Doppler flow detector can only measure the component of velocity that is parallel to the ultrasound beam, the true velocity is calculated by assuming a flow-to-insonation angle. Accurate velocity calculation is therefore dependent on proper imaging technique by the examiner. Errors in Doppler velocity measurements can be introduced and exacerbated by improper alignment of the angle cursor, incorrect placement of the Doppler sample volume within the vessel lumen, utilizing too large a Doppler angle, or changing the Doppler angle between examinations.³² Since duplex ultrasound interpretation criteria for stenosis are based primarily on velocities, errors in the Doppler velocity measurements can lead to errors in diagnosis and treatment. The sensitivity of carotid duplex scanning in grading stenosis is only 65% to 71%.³⁴ One study reported erroneous assignment of 15 of 35 lesions to non-surgical intervention solely on the basis of incorrect Doppler insonation angle.³⁵ These findings are of particular concern because Doppler ultrasound is used as the primary, and sometimes the only, diagnostic test for vascular pathology.³⁶

Skilled use of Doppler ultrasound by sonographers, physicians, and nurses for evaluating vascular access sites is an essential component of caring for patients on hemodialysis.³⁷ Duplex ultrasound, which displays the B-mode tissue image together with Doppler spectral waveforms and color-flow, plays a major role in minimizing the morbidity and costs of hemodialysis access by determining the adequacy of arterial inflow and venous outflow, assessing fistula maturation, and determining the reasons for maturation failure.³⁸ Duplex scanning can also diagnose problems related to maintenance of fistulas, including arterial steal, pseudoaneurysms, and stenoses in prosthetic or native venous conduits.^38–40 Accurate assessment of dialysis access fistula patency is important because as many as 60% of access sites require an intervention within 1 year to maintain functional patency.^41,42

Several studies have highlighted the crucial need for improved training methods in Doppler ultrasound. The Intersocietal Accreditation Commission (IAC, formerly ICAVL) reported that up to 35% of the applications for vascular laboratory accreditation demonstrated improper Doppler angle correction techniques.⁴³ Stenosis classification error has been linked to lack of experience in Doppler ultrasound,⁴⁴ and clinical management decisions differed between examiners in 45% of severe internal carotid artery stenoses.⁴⁵ Accredited vascular laboratories that adhere to training and practice guidelines have been shown to perform better than non-accredited laboratories in terms of accuracy of treatment decisions. Stanley showed that IAC-accredited vascular laboratories correlated 83% of the time with reference studies performed at a central site, while facilities without IAC accreditation had only 45% correlation.⁴⁶ Disagreement in the examination outcome indicated cases for which an incorrect treatment decision would have been made based solely on the non-accredited laboratory examination. Similarly, Brown et al. found that 35% of vascular examinations from non-accredited vascular laboratories had clinically significant errors when compared with reference studies at an accredited facility.⁴⁷ These variations in practice outcome underline the need to not only improve the initial training of physicians and sonographers but also to monitor skill retention in the clinical setting.

Current Doppler ultrasound simulation technology (Table 3)

Table 3.

Current Doppler ultrasound simulators with skill metrics.

Type	Doppler source	Metrics
Flow phantom	Real ultrasound machine	Psychomotor: flow velocity – Transducer manipulation skill – Sample volume location
Mannequin with mock transducer	Generated in real time from the computational flow model as the mannequin is scanned	Psychomotor and cognitive – Transducer manipulation skill – Sample volume location – Flow velocity – Volume flow – Image location in vessel – Vessel segment identification – Spectral waveform analysis – Stenosis assessment

There are several types of vascular ultrasound simulators that provide representations of the color-flow Doppler images and Doppler spectral waveforms with varying realism. Flow phantoms have a tissue-mimicking substrate with one or more vessels connected to a pump containing blood-mimicking fluid. An advantage for training is that the learner can acquire skill and experience with the operation (sometimes referred to as ‘knobology’) of the duplex ultrasound system because the phantom is examined with an actual duplex scanner. These simulation products can also be used to practice ultrasound-guided vascular access. However, flow phantoms present vascular flow abnormalities in an idealized fashion, and the number of cases is limited by the need for a separate physical model for each flow abnormality.

Some commercial vascular ultrasound simulators provide case review from recorded images when a mannequin is scanned (UltraSim, MedSim, Inc., Kfar Sava, Israel; SonoSim, SonoSim Inc., Santa Monica, CA). One product includes a simulated ultrasound machine for training in ‘knobology’ (MedSim, Inc.). Another product enables hands-on practice without a mannequin, either by tracking the mock transducer alone as it is moved in the air, or by tracking the transducer on a live subject instrumented with skin tags (SonoSim, Inc.). None of these vascular ultrasound simulators measures technical skill.

The University of Washington and Sheehan Medical LLC (Mercer Island, WA, USA) jointly developed a computer-based duplex ultrasound simulator that presents real-time color-flow Doppler images and Doppler spectral waveforms along with the corresponding B-mode images. The hardware consists of a personal computer, a mannequin, and a mock transducer whose spatial location and orientation are measured using a tracking device. As the examiner scans a mannequin the computer displays real patient B-mode images that morph according to the position and orientation of the mock transducer. The simulated B-mode images are derived from a three-dimensional (3D) volume of ultrasound image data previously acquired from a normal volunteer or patient. The vessel is reconstructed as a 3D lumen volume and computational fluid dynamics modeling is used to populate the lumen with time-varying velocity vectors that define the blood flow at all points within the vessel. This velocity field is sampled along with the image data to create a color-flow and spectral waveform display that responds in real time to the control panel settings selected by the examiner, including beam steering, pulse repetition frequency, Doppler angle, and sample volume size and depth (Figure 1).

Figure 1.

Computer display of the interface for the duplex ultrasound simulator developed at the University of Washington showing an examination of a dialysis access fistula in the left arm. (A) Left panel: duplex display showing the B-mode image with color-flow Doppler on top and Doppler spectral waveforms on the bottom. The Doppler spectral waveform has been acquired at a beam angle of 60 degrees and the cursor has been placed on the waveform for measurement of peak systolic velocity. Middle panel: controls for the ultrasound display mode and Doppler settings (including Doppler beam angle, sample volume depth, sample volume size, and PRF or scale). Right panel: the 3D display shows the locations of the transducer (‘star’ and cone) and B-mode image (translucent rectangle) on the vessel in the mannequin; this display can be zoomed. (B) Report page. The learner performs a scan and measures velocities that are entered on this table. When the learner ‘submits’ his/her answers, the cells are colored to indicate whether each response is correct (green), close (yellow), or definitely wrong (red). As the mouse is hovered over each cell, the correct value is displayed for feedback. In practice mode, the learner can ‘unsubmit’ the answers and try again. (Note: figure is in color online.)

Technical skill is not just assessed from the PSV that the trainee measures because the correct value could be obtained despite using incorrect technique. Instead, the trainee is presented with case studies on which his/her skill is assessed based on correctly identifying vessel anatomy, selecting appropriate ultrasound parameter settings, measuring blood flow velocity (and, when indicated, measuring volume flow), and making the diagnosis. The validity of the simulation has been demonstrated based on the accuracy of blood flow velocity measurements by examiners compared to the correct values in the flow model. When three experts made 36 PSV measurements on two carotid artery models, the mean deviation from the actual PSV was 8 ± 5%, and was similar between vessel segments and between examiners.⁴⁸ Measurements of PSV in dialysis access shunt models had similar accuracy.⁴⁹ These results demonstrate that an examiner can measure PSV from the spectral waveforms using the settings on the simulator with a mean absolute error in the velocity measurement of less than 10%.

The simulator developed at the University of Washington provides an absolute measure of examiner performance because the blood velocity values are known at all locations in the computer models. This is not possible with the scanning of live subjects or even with flow phantoms because their exact velocity values are not known precisely. With the addition of cases with a range of pathologies, this duplex ultrasound simulator may become a useful tool for training healthcare providers in vascular ultrasound applications and for assessing their skills in an objective and quantitative manner.

Simulation of cardiac ultrasound

The need for skills assessment

In the past, echocardiography (echo) was only performed by cardiologists with the assistance of cardiac sonographers. With the advent of low-cost, portable ultrasound machines, physicians in many specialties are adopting bedside or ‘point of care’ ultrasound, and medical schools and residencies are now adding ultrasound training to their curricula.^50,51 Because physicians performing a focused cardiac ultrasound (FoCUS) examination acquire the images themselves rather than relying on a sonographer, they require hands-on training to develop the requisite skill. Assessment during training is essential to provide feedback, since frequent feedback is one of the pillars of CBME.² Indeed, the American Society of Echocardiography (ASE) guidelines recommend that a portion of hands-on studies be proctored in real time to provide this important feedback.⁵²

Assessment of competence is important because ‘Optimization of cardiac views is critical to obtaining a correct diagnosis’.⁵² There is concern that diagnostic ultrasound will be substandard when performed by practitioners with only brief training compared with dedicated ultrasound physicians or experienced sonographers.^53–55 Recent ASE guidelines emphasize the breadth of the experience, that the trainees have ‘acquired and interpreted ultrasound images that represent the full range of diagnostic possibilities’, rather than just the number of patients scanned.⁵⁶ At the time these guidelines were published (2013), objective metrics of competence in cardiac ultrasound were just being developed.

Current cardiac ultrasound simulation technology (Table 4)

Table 4.

Current echocardiography simulators with skill metrics.

Image source	Mode	Feedback to trainee	Metrics
Synthetic	TEE	Visual	Economy and efficiency (time, path length, velocity, acceleration) Probe handling fluidity and smoothness (jerk, peak, spectral arc length)
Patients	TEE and TTE	Quantitative and visual	Angle between acquired view and anatomically correct view Probe location: distance from correct location

TEE, transesophageal echocardiography; TTE, transthoracic echocardiography.

Echo simulators, which typically consist of a mannequin, mock transducer, and computer, were first marketed in the 1990s, before computer technology could provide realistic assessment of technical skill employing simulation. Modern echo simulators offer 3D graphic displays to assist the learner in visualizing the relationship between the transducer position, the location of the image sector in the heart’s anatomy, and the resulting 2D image. In January 2017, one of us (FHS) invited each simulation vendor to submit publications on research utilizing their echo simulators for skills assessment for inclusion in a presentation at the American Institute of Ultrasound in Medicine. The replies of vendors who responded are presented here. There are three echo simulators with the technology necessary to measure the technical skill of the examiner: one simulates transthoracic echo (TTE) and two simulate transesophageal echo (TEE). All of them generate skill metrics using transducer tracking data collected during scanning. The simulators differ in realism of the images, with some displaying synthetic images and some displaying real patient images.

Investigators at the University of Washington were the first to develop an echo simulator with metrics of technical skill. Working initially with a graphic interface intended for tele-ultrasound applications, they showed that novices performed significantly better at visualizing specified anatomy with rather than without the visual guidance afforded by 3D reconstruction of the organ being imaged.⁵⁷ In the echo simulator, numeric as well as visual feedback is provided to trainees (Figure 2). As the mock transducer is manipulated over the mannequin, the computer displays ultrasound images from a previously acquired 3D data set in a 2D view that changes in real time appropriately for the transducer’s position and orientation. The anatomically correct plane can be displayed during training as a guide. When the trainee ‘acquires’ a cine loop, its plane is also displayed for comparison, and the trainee’s technical skill is measured in terms of the angular deviation between his/her acquired view plane and the correct plane.⁵⁸ The metric was validated by demonstrating its ability to discriminate trained novices from experts, and is applicable to TEE as well as TTE training.⁵⁹

Figure 2.

User interface for the echocardiography simulator developed at the University of Washington. (A) The image ‘acquired’ by the trainee appears in a dual display (upper inset) and appears side by side and beating in synchrony with the anatomically correct view. The angular deviation and other metrics are displayed for additional feedback (arrow indicates the enlarged metrics in the inset). This image was very close to correct. (B) Acquisition of a poorly positioned image in which the aortic valve is obscured and the left ventricle is foreshortened; note the difference in visualization of anatomy and the magnitude of the angular deviation.

Both the CAE Healthcare (Sarasota, FL, USA) and HeartWorks (Inventive Medical Ltd, London, UK; recently acquired by Medaphor) simulators perform kinematic or motion-based analysis of TEE performance. CAE’s metrics measure the smoothness and fluidity of transducer manipulation. Five of their metrics differed significantly between experts and untrained novices on univariate analysis; two (peak movement and path length) showed independent discriminatory value.⁶⁰ HeartWorks’ metrics ‘represent economy and efficiency … as well as motion smoothness and fluidity’ (Figure 3). Six of their eight metrics showed significant differences between experts and untrained novices.⁶¹

Figure 3.

Kinematic metrics of skill in transesophageal echo (TEE) in the HeartWorks simulator.⁶¹ Probe tip trajectory and depth of the transesophageal echo probe are smoother and more fluid for the expert than for the novice. Reproduced from ref. 61 (© Springer International Publishing Switzerland 2016) with permission of Springer.

In summary, metrics for simulator-based assessment of technical skill have been developed and validated for both TTE and TEE. All three of these simulators display 3D graphic feedback, and all provide metrics that can be applied to inform program directors of trainee progress. The simulator developed at the University of Washington is the only one to provide immediate quantitative feedback to learners.

Conclusion

In medical ultrasound, accuracy of diagnosis is critically dependent on the skill of the examiner. Thus, a standardized and readily available method for competency assessment provides an important link between efforts to improve patient safety and the need for better training of health professionals. Simulator-based competency testing in diagnostic ultrasound is likely to contribute to the paradigm change in medical education referred to as CBME because it can: (1) permit quantitative and objective assessment of skill competence for training, certification, and continuing education; (2) promote standardization of examinations; (3) remove the dependency of testing and training on availability of live patients; and (4) improve training efficiency by giving immediate and frequent feedback to the trainee and by informing the instructor on areas of inadequacy.

Simulators have been developed for both vascular and cardiac ultrasound applications. For peripheral vascular ultrasound, the measurement of flow velocities is critical for diagnosis, and realistic display of Doppler spectral waveforms presents unique technical challenges. Flow phantoms allow use of a standard duplex ultrasound system but depict flow abnormalities in an idealized manner that does not provide a realistic scanning experience. The computer-based duplex ultrasound simulator developed at the University of Washington uses a 3D model of the vessel lumen and computational fluid dynamics to create a realistic color-flow Doppler image and Doppler spectral waveforms. At this time, simulators for echocardiography are generally more advanced than those for peripheral vascular ultrasound. Computer-based echocardiography simulators are available that offer 3D graphic displays that provide feedback to the learner as well as metrics for assessment of technical skill.