Development and implementation of competency-based assessment for urological ultrasound training using SonoSim: A preliminary evaluation

Abstract

Introduction:

Urology residents are encouraged to learn ultrasound (U/S) imaging, yet there are few tools available for teaching and assessing a resident`s competence. The aim of this study was to test the new SonoSim LiveScan^® and to propose a competency-based assessment model for the urology graduate medical education.

Materials and Methods:

Urology residents attended an interactive training session covering the urological U/S techniques guided by the assessment model developed by the authors. Faculty members evaluated the residents using defined objectives, and the residents were surveyed on their comfort level for performing each of the model tasks. A subset of the residents then underwent a structured testing using the SonoSim LiveScan device 6 months following the training. The model developed assessed: general U/S setup, structure identification, and pathologic clinical scenarios.

Results:

The residents felt most comfortable in identifying the bladder (4.73/5) and the kidneys (4.53/5) during the training sessions. They felt least comfortable while testing for total ureteric obstruction (3.13/5). All the residents were confident that additional U/S training sessions would improve their comfort level in performing the assessed objectives. Resident`s assessment performed at 6 months had a median test score of 15.5/20 and the assessment scores increased with resident seniority. Self-reported comfort, however, did not seem to correlate with seniority. In general, the residents felt that the SonoSim device was highly functional (4.4/5) and the pathologic assessments in particular were very helpful (4.4/5).

Conclusions:

Through pilot testing, we propose that a competency-based assessment used with the SonoSim LiveScan could guide the resident`s education through the acquisition of U/S skills and warrants testing in a larger cohort.

INTRODUCTION

Ultrasound (U/S) is commonly used in the field of urology, and it is often the first-line imaging examination in obtaining or confirming a diagnosis. It can be used to evaluate a variety of urological pathologies, including those involving the kidney, bladder, prostate, genitalia, or the pelvic floor, and has the added benefit of being safe and cost-effective relative to the other imaging modalities.[1] Despite the recognized importance of familiarity with U/S in the urological practice, there are only a few requirements pertaining to the U/S training in the residency program. Beginning July 2012, the Urology Residency Review Committee of the Accreditation Council for Graduate Medical Education (ACGME) requested the residents' to log cases involving urologic U/S procedures, but the only index case with a minimum requirement is the transrectal U/S for prostate biopsy (25 cases).[2] The Urology Milestone project also lists interpretation of the office-based genitourinary U/S as a metric for evaluating residency competency but fails to detail how to assess the residents.[3] In practice, 43% of the practicing urologists perform nonprostate U/S;[4] therefore, residents should receive structured training in U/S so that they are prepared to perform a variety of procedures.

Other specialties, such as emergency medicine and obstetrics and gynecology, have developed detailed objectives and criteria to guide U/S training and assessment during the residency.[5,6] While there are many recommendations and generalizations about the urological U/S technique, to the authors' knowledge, there is no standardized list of objectives to evaluate the residents' U/S ability in the field of urology. Utilization of simulators has been limited outside of the emergency situations and prostate biopsy.[7,8] Testing the knowledge of the trainees will require more than the documentation of normal anatomy and maneuvering the common artifacts. Therefore, the purpose of this study was to develop a competency-based assessment for urological U/S procedures and test its usefulness in the residency training using the SonoSim LiveScan^®, an U/S simulation device.

MATERIALS AND METHODS

This study was reviewed and approved by our institutional review board with a waiver of informed consent.

Development of assessment guidelines

The American Institute of Ultrasound in Medicine (AIUM) collaborated with the American Urological Association (AUA) to develop parameters for U/S examination in the urologic practice.[9] The parameters included specifics for kidney and bladder examination, which were used in part to design the assessment guidelines used in this study. This included obtaining longitudinal (long axis) and transverse views of the kidney, measuring the kidney, assessing the kidney for abnormalities, and measuring the bladder volume. The other competencies were determined using the U/S training recommendations created by the United Kingdom's Royal College of Radiologists (RCR).[10] In addition to the objectives outlined in the AIUM parameters, the RCR recommended urology trainees to be able to evaluate the liver, recognize and assess the degree of hydronephrosis, and use the color Doppler to assess ureteric jets. The remaining objectives were determined by the urology faculty based on their clinical experiences.

Equipment

The SonoSite M-Turbo U/S system (Fujifilm, Bothell, WA, USA) is a mobile U/S unit stored at our Center for Clinical U/S Training facility. The SonoSim (Santa Monica, CA, USA) is an U/S training platform that uses real sonographic images imported to a screen for viewing, while the probe is detected in space [Figure 1a]. The SonoSim LiveScan^® replicates pathology seen using an actual U/S probe so that we can scan human models or mannequins. Radiofrequency identification (RFID) tags are applied to particular anatomic locations designated for scanning. When the U/S probe is placed on the RFID tag, a video of U/S images obtained from a real patient plays on the screen. The images may depict normal anatomy or pathologic conditions depending on the goal of the training session [Figure 1b-d]. Motion-sensing technology detects the position of the probe and changes the image based on the user's movements, allowing the user to practice scanning through the pathologic images.

Figure 1.

(a) SonoSim LiveScan® simulates real pathologies on manikins and live volunteers for teaching the integration of ultrasound findings into clinical decision-making. Credit: SonoSim, Inc. (b) representative images from the SonoSim live pathology session: bladder stone, (c) representative images from the SonoSim live pathology session: hydronephrosis, (d) representative images from the SonoSim live pathology session: polycystic kidney

Survey and data collection

The urology residents were taught at the Center for Clinical U/S Education at our institution using the SonoSite M-Turbo U/S system. The session began with an introduction to U/S physics and knobology and then addressed the items listed in the assessment guidelines. The residents were given 30 min to practice the various tasks before being evaluated by the faculty members on their ability to complete them. After the training session, the residents completed a 5-point Likert questionnaire to report their comfort in performing individual portions of the U/S scan [Table 1].

Table 1.

Distribution of comfort level in performing competency-based objectives (n=15)

Objective	Extremely uncomfortable	Uncomfortable	Neutral	Comfortable	Extremely comfortable	Mean Likert Score
1. Understanding the principles of U/S technology for clinical use	0	1	1	12	1	3.87
2. Identifying the bladder	0	0	0	4	11	4.73
3. Measuring bladder volume	0	0	2	7	6	4.27
4. Noting artifacts such as posterior acoustic enhancement	0	1	5	8	1	3.60
5. Using color flow mode to identify ureteric jets	0	0	3	9	3	4.00
6. Testing for total ureteric obstruction	0	4	6	4	1	3.13
7. Identifying the kidneys	0	0	0	7	8	4.53
8. Measuring the kidney (longitudinal and transverse)	0	0	0	8	7	4.47
9. Distinguishing the ureter	0	1	3	11	0	3.67
10. Determining the presence of hydronephrosis	0	0	0	10	5	4.33
11. Identifying the liver	0	0	0	8	7	4.47
12. Identifying the spleen	0	0	0	10	5	4.33

U/S: Ultrasound

Creation and application of the skill test

We chose areas of probe identification and basic utilizations skills (questions 1 and 2), knobology (questions 3–6), and pathology identification (question 7 a–h). There are a variety of tools for testing, but most of these seem to be specialty specific or targeted for the U/S technologists.[11] The total possible score for our assessment tool was 20. Six months after the initial training session, the residents underwent structured U/S testing using the SonoSim device to perform general U/S setup, structure identification, and clinical scenarios including pathology. The residents were unaware that they would be tested that day and completed a 5-point Likert questionnaire to report their assessment of their performance as well as the utility of the SonoSim [Table 2].

Table 2.

Resident assessment of SonoSim testing session (n=6)

Survey Question	Mean Likert score
How prepared did you feel? (1=very unprepared, 5=very prepared)	2.8
How well did you feel you performed? (1=poorly, 5=very well)	3.2
How did this training session compare to standard sessions? (1=much less informative, 5=much more informative)	3.6
How fair did you feel the questions were? (1=very unfair, 5=very fair)	4.4
Functionality of SonoSim (1=poor, 5=excellent)	4.2
How realistic was the simulation in representing pathologic findings? (1=very unrealistic, 5=very realistic)	4
How well did the session simulate real-life scenarios? 1=poorly, 5=very well)	4
How helpful was the pathology for learning U/S utilization? (1=very unhelpful, 5=very helpful)	4.4
How likely are you to recommend SonoSim incorporation for training? (1=very unlikely, 5=very likely)	3.8

U/S: Ultrasound

RESULTS

Training session evaluation of skills

Out of the 20 urology residents, 15 (75%) attended the interactive session. Assessment by the faculty members using the created guidelines [Figure 2] showed that the majority of the residents were able to perform the tasks on the testing sheet aside from adrenal identification.

Figure 2.

Proposed competency-based assessment tool used to guide teaching sessions and to objectively evaluate urologic ultrasound skills in the subsequent sessions

Training session post survey

Fifteen residents completed the questionnaire. One resident completed the questionnaire but did not attend the session. Respondents ranged from postgraduate year (PGY)-1 to Urology year 4 (U4). As assessed with the Likert scale, most of the residents felt confident in their understanding of the principles of U/S technology for the clinical use (mean Likert 3.87/5, range 2–5). The majority of the residents (11/15, 73%) reported an exposure to U/S education in medical school, and the individual who was not confident in the principles of U/S technology did not have an exposure to U/S education prior to the residency.

The residents felt most comfortable in identifying the bladder (4.73/5) and the kidneys (4.53/5). They also reported comfort in identifying the liver (4.47/5), measuring the kidney in the longitudinal and transverse planes (4.47/5), identifying the spleen (4.33/5), and determining the presence of hydronephrosis (4.33/5). The residents were less comfortable in measuring the bladder volume (4.27/5, range 3–5) and using the color flow mode to identify the ureteric jets (4.0/5, range 3–5). They were least comfortable in performing the following tasks: distinguishing the ureter (3.67/5, range 2-5); noting artifacts such as posterior acoustic enhancement (3.60/5, range 2–5); and testing for total ureteric obstruction (3.13/5, range 2–5). Stratifying the data into junior (PGY-1, U1, and U2) and senior residents (U3 and U4) demonstrated that the junior residents were less comfortable in completing the last three objectives than the residents in U3 and U4. In addition, residents who had received U/S education in medical school were slightly more comfortable meeting the U/S objectives overall (4.14/5) as compared to those who did not learn U/S techniques in the medical school (4.04/5). All the residents were “confident” or “extremely confident” that additional training sessions would improve their comfort in performing the U/S tasks.

Assessment of content retention

Six residents (training experience ranging from U2 to U4) were tested using the SonoSim LiveScan device with a median test score of 15.5/20 and a range of 13–19. Resident selection was limited by their availability and the access to the SonoSim device. All the residents first adjusted the scanning depth before using the probe, and all of them identified rib shadowing, hydronephrosis, normal kidney, and normal bladder. Most were able to identify polycystic kidney, dilated ureter, bladder stone, and uterus. None of the residents correctly identified the adrenal mass pathology. The assessment scores increased with resident seniority: the median score for junior residents was 13, whereas the median score for senior residents was 17 [Figure 3]. Self-reported comfort, however, did not seem to correlate with seniority. On an average, the residents reported feeling unprepared for the testing session (2.8/5, range 1–4), with senior residents feeling more prepared than the junior residents; however, the residents as a whole felt that they performed neither above nor below the average with no appreciable difference between the junior and the senior residents' responses (3.2/5, range 3–4). The residents felt that the questions were fair (4.4/5) and judged the functionality of the SonoSim to be good (4.4/5). They felt that the session simulated real-life scenarios well (4/5), that the pathology was very helpful for U/S education (4.4/5), and said that they would recommend the incorporation of the SonoSim into the resident training program (3.8/5).

Figure 3.

Residents' individual ultrasound testing scores as determined using the competency-based assessment tool

DISCUSSION

In this single-center evaluation study of a novel, competency-based assessment for U/S training involving 15 urology residents ranging from PGY-1 to U4, we found that the assessment tool allowed for objective evaluation of the residents' scanning abilities, that the SonoSim LiveScan has face validity for urologic U/S simulation, and that the residents were receptive to our instructional and testing methods. Initial assessment showed that the residents were able to complete all the tasks on the testing sheet except adrenal identification. Although not all the residents felt comfortable in performing the tasks, all the residents agreed that further training sessions would improve their confidence (Likert mean 4.47/5). A subsequent session assessing six residents by the faculty using the SonoSim LiveScan device showed that all the residents were proficient in identifying rib shadowing, hydronephrosis, normal kidney, and normal bladder. Most were able to identify polycystic kidney, dilated ureter, bladder stone, and uterus. None was able to correctly identify adrenal mass pathology. This is not surprising as the residents had difficulty in identifying the adrenals in the initial assessment and there was no additional training before the simulation assessment. This indicates that initial results could be used to target areas of weakness during the subsequent training sessions. As the seniority of the resident increased, so did their SonoSim assessment score (median score 13/20 vs. 17/20). This suggests that performance on the simulator may correlate with performance in the actual patient care, which is consistent with the findings of the other studies exploring the use of simulators in U/S education.[12,13] Interestingly, self-reported comfort level did not correlate with the seniority of the residents. It is possible that with regular interval assessment during the training, residents will be better able to gauge their abilities, and their comfort level will improve along with their experience level.

Clinical use of U/S without formal instruction is insufficient for the residents to become competent in the point-of-care U/S,[11,14] and the simulation-based training has been shown to improve ultrasonography abilities considerably more than the standard teaching methods.[15,16] Until recently, however, high-fidelity simulation in U/S education had been largely underutilized due to its high cost, limitations in imaging quality, small size of the scanning sections, lack of anatomical landmarks, inability to examine the moving objects, and the required presence of an experience teacher for feedback.[17,18] With the improvement of the simulation technology and the resolution of these problems, the use of simulation has increased dramatically and is expected to continue to grow.[19] Simulation-based U/S education has the advantage of allowing the residents to practice a realistic U/S examination in a low-risk environment, without requiring an experienced physician to be present to provide the feedback. Simulators also allow for the assessment of competency, as cases can be standardized for all the residents, and the resident education is further enhanced by simulator-provided pathologies that may be rarely seen in practice but must be learned nonetheless.[20] In the field of urology, however, the majority of the studies involving simulators focus on their role in learning to perform prostate biopsies or to gain percutaneous renal access. For this reason, we chose to evaluate the use of SonoSim LiveScan in imaging a variety of urologic pathologies.

A review of U/S teaching methods showed an extensive variety between and within the specialties, including a structured 30-h course during intern orientation using scanning models and simulators;[21] a formal curriculum integrated into didactics over the course of residency with hands-on training;[22] and a 3-h session covering the basics of U/S and extended focused assessment with sonography for trauma (EFAST) in a lecture followed by a hands-on session.[23] The vast difference in training time and methodology reflects the lack of formal requirements for the resident's U/S education. Although many specialties are recognizing the growing importance of U/S education in residency, only emergency medicine has recommended the minimum requirements for the U/S curricula, as outlined by the ACGME.[24] In addition, the Council of Emergency Medicine Residency-Academy of Emergency U/S consensus document provides further guidelines for the resident training including an introductory session to ultrasonography, 2 weeks of a dedicated U/S rotation, a minimum of 20 h of scheduled educational U/S sessions throughout the residency, and at least 150 emergency U/S examinations.[25] Curricula for emergency medicine residency programs in the USA are varied,[26] but the provided guidelines offer a framework for emergency medicine programs to design a suitable U/S training program. We sought to provide a similar framework for the field of urology through the development of our competency-based assessment tool. To the best of our knowledge, there have been no other studies that created a tool that objectively evaluates urologic U/S abilities, nor have there been any that evaluate a curriculum for urologic U/S training.

Ultrasonography is a critical skill for the practice of urology, as demonstrated by consistent enrollment in AUA U/S courses[4] and by the training guidelines released by the AIUM.[27] The AIUM proposes that the physicians who evaluate and interpret urologic U/S must have completed a residency that includes a training in U/S after 2009. Specific components of an U/S curriculum have not been described, however. If they were not taught ultrasonography during their residency, physicians must have performed at least 100 diagnostic urologic U/S examinations within the past 36 months to demonstrate proficiency in U/S. Despite this minimum number for the practicing physicians, the ACGME only requires residents to document 25 prostate U/S during their training.[2] There are no requirements for nonprostate ultrasonography.

Standards are necessary in order to evaluate and compare the levels of competency. Although there are recommended guidelines for teaching urological U/S, there is no widely accepted standard for evaluating performance during the training. It is, therefore, unclear if the residents are graduating with the recommended U/S skillset and even more unclear what that skillset entails. By utilizing the assessment tool developed in this study, urology residents will have a better understanding of what a basic urological U/S examination should include, and the programs will have a more objective method to evaluate U/S competency.

Prior to the current study, we have had two urology U/S training sessions per year since 2016. During the general sessions, we discuss knobology and had human subjects to practice kidney and bladder U/S. Our residents enjoyed the additional standardized training sessions and found the assessment to be useful. Therefore, we plan to incorporate competency-based assessment using the SonoSim LiveScan into the residency curriculum at least every 6 months. However, a large missing piece of U/S training is “in-the-field” use and documentation. Utilization of handheld U/S devices with the ability to document and log encounters with intermittent competency testing will likely make the largest impact.

Our study is limited by the small sample size and will require future studies to strengthen the validity of our results; however, there remains a void of data regarding how to incorporate U/S into urologic residency and test competency. The assessment by residents was also based on their subjective impressions. Of note, residents continued to perform U/S examinations as a part of their clinical duties throughout the course of this study, including kidney, bladder, and prostate. The results of the SonoSim assessment may, therefore, be confounded by residents' ongoing training. In addition, those results may be further influenced by attrition bias, as only a subset of the available residents underwent assessment on the SonoSim. Although this study proposes an itemized assessment method, continued investigation is needed to determine how performance during U/S training correlates to the clinical performance. This study would also benefit from the input of other institutions and leaders in the field of urology around the world in determining what constitutes competency, as there is currently no established score cutoff, and there may be disagreement regarding the objectives that are assessed. Perhaps as the involvement increases, the urology community can reach a consensus and expand this competency-based assessment into a competency system for a standardized U/S curriculum.

CONCLUSIONS

This study supports a valid method for objectively evaluating urologic U/S skills among the residents using a novel competency-based assessment tool and SonoSim LiveScan. The training and testing sessions were well received by the residents, objective assessment scores correlated with the seniority of the residents, and the assessment objectives provided a framework for urologic U/S education. The combination of the assessment tool and SonoSim LiveScan facilitates evaluating the residents' ability to identify pathological conditions throughout their training. Further testing is needed to establish the content and construct validity of the SonoSim LiveScan and to determine the effect of the periodic assessment with this system on the clinical performance.