Point-of-care ultrasound in urology: Design and evaluation of a feasible introductory training program for Canadian residents

Abstract

Introduction

Point-of-care ultrasound (POCUS) is an increasingly used bedside tool. Applications in urology include the assessment of an undifferentiated acute scrotum, renal colic, and the guidance of suprapubic catheter placement. However, the user-dependent nature of this modality necessitates appropriate use and competence. The objective of this study was to develop and evaluate a low-cost, feasible, and guideline-based introductory POCUS program for Canadian urology residents.

Methods

Residents from McMaster University’s urology program completed a three-hour online course, followed by a three-hour hands-on seminar. Course material was developed by ultrasound educators based on national guidelines. Low-cost testicular phantoms and suprapubic catheter insertion models were constructed. Pre- and post-course surveys focused on participant skill confidence, while multiple-choice questionnaires assessed theoretical knowledge.

Results

Fourteen residents participated in the course. Theoretical knowledge in POCUS improved significantly (p<0.001, d=2.2) and mean confidence scores improved for all skills, including performing kidney, bladder, and testicular POCUS (all p<0.001; d=3.4, 1.9, 2.9, respectively). Participants indicated that the course increased their confidence and likelihood of using POCUS in clinical practice, and that POCUS training should be integrated into urology training curricula.

Conclusions

This novel study included the development of an inexpensive, feasible, guideline-based introductory training program for urological POCUS, developed in collaboration with ultrasound educators. Participants significantly improved in theoretical knowledge and skill confidence. Although this study was limited to one residency program, the basis of this course may serve as a foundation for the development of competency-based training for urological POCUS in Canada.

Introduction

Point-of-care ultrasound (POCUS) is a widely used bedside tool that aids in time-sensitive clinical diagnoses.^1,2 This modality uses non-ionizing ultrasonic waves that generate real-time images, which allow the user to correlate findings with a patient’s presentation, thus aiding clinical decision-making. ^2,3 The development of more compact and portable machines has resulted in POCUS becoming relatively inexpensive and readily available.² POCUS has become increasingly popular in multiple fields of medicine, including surgical disciplines.^4–6

Urology is among the top five non-radiological specialties that use ultrasound in practice.⁷ A recent study found that 43% of urologists registered under the American Board of Urology performed non-prostate ultrasound for their patient management.⁸ There are a multitude of clinical applications for POCUS in the acute urological setting, including, but not limited to, the timely diagnosis of an undifferentiated acute scrotum,^9,10 the assessment of hydronephrosis in renal colic,¹¹ and guidance for suprapubic catheter placement.^12,13

POCUS is a useful adjunct tool that urologists can use in patient care; however, the user-dependency of this modality necessitates appropriate use and operator competence. Several committees, including the Canadian Association of Radiologists (CAR) and the Canadian Association of Emergency Physicians (CAEP), have stated that the use of POCUS by untrained users has the potential for misdiagnosis, poor clinical management, and negative patient outcomes. ^3,14 Fortunately, it has been shown that with proper educational training, medical practitioners and learners are able to acquire the skill and confidence to use POCUS.^15,16

To our knowledge, there are no formal guideline-based Canadian POCUS programs for urological applications. There is one unique Canadian POCUS course for both emergency medicine and urology residents in its early stages; however, the program focuses only on the diagnosis of testicular torsion.¹⁷ Given the lack of POCUS training options for urology residents, the objective of this pilot study was to develop and evaluate a low-cost, feasible, and guideline-based introductory POCUS curriculum for Canadian urology residents, with a focus on familiarization of foundational technical skills and theoretical knowledge through the application of several urology-specific examinations.

Methods

Study design and participants

This prospective study was conducted in November 2019 and followed a pre- and post-course intervention design. Participants were recruited from McMaster University’s urology residency program. All urology residents were invited to participate, and the study only excluded participants if they had previous formal education or training in ultrasound. The study was exempt from the approval of the Hamilton Integrated Research Ethics board, as it was considered program implementation and evaluation.

Course design and content rationale

Development of this course was achieved in collaboration with licensed sonographers and ultrasound educators from McMaster University and Mohawk College’s School of Medical Radiation Sciences. Course material was modelled from the guidelines used to train Canadian sonographers (Sonography Canada National Competency Profiles),¹⁸ and content recommendations made by CAR.³

The course was comprised of two main components: 1) a pre-course, three-hour, online module with associated assessments; and 2) a three-hour, hands-on session at the Mohawk College Imaging Research Center.

The pre-course online content was created by licensed sonographers and included recorded lectures with associated presentation slides and assessment forms. Learning objectives based on national competency profiles were chosen based on the input from sonography and urology stakeholders (ultrasound educators from McMaster University’s School of Medical Radiation Sciences and the McMaster University urology residency program director). These objectives primarily focused on POCUS definitions and indications, sonographic image comprehension, knobology, image acquisition, and specific kidney, bladder, and testicular topics. Detailed learning objectives can be found in Appendix A (available at cuaj.ca). Furthermore, the information presented in the online lectures had an emphasis on practical sonographic skills, including image interpretation, optimization, and critique. Content was hosted on an established online learning platform (AvenueToLearn, Desire2Learn, Kitchener, ON, Canada).¹⁹ To allow for flexibility, participants were given two weeks to finish the online component on their own time.

The hands-on session was facilitated by licensed sonographers and began with a brief review of the online content. The participants were then distributed into small groups of 3–4 residents, each with an ultrasound machine. Under the guidance and observation of sonographers, residents completed ultrasound image optimization and interrogations of the kidney in several patient positions (using simulated patients): left lateral decubitus, right lateral decubitus, supine, and prone. Residents then interrogated the bladder with the patient in the supine position. Finally, testicular imaging and ultrasound-guided suprapubic catheter insertion skills were practiced by using handmade, low-cost, and reproducible imaging simulators (Fig. 1).^20,21 These simulators decreased the barrier to course implementation by avoiding the cost associated with commercial simulators. The detailed scanning protocols created for kidney, bladder, and testicular imaging can be found in Appendix B (available at cuaj.ca).

Fig. 1.

(A) Simulated testicular phantoms inside scrotal case. (B) Corresponding ultrasonic appearances of testicular phantoms. (C) Suprapubic catheter insertion training simulator components. (D) Suprapubic catheter insertion training simulation example.

Analysis of confidence and theoretical knowledge

All participants completed a pre- and post-course survey and multiple-choice questionnaire (MCQ) assessment. The survey used a five-point Likert scale to determine self-rated user confidence and interest in POCUS prior to and after the course. This scale was adapted from a previous ultrasound study.²² To evaluate knowledge, the MCQ assessments used questions from a bank produced by ultrasound experts from the Sonography Canada National Competency Profiles,¹⁸ and topics focused on relevant urological clinical applications (Appendix A; available at cuaj.ca). To control for practice bias, an additional 15 unique questions from the same question bank were added to the post-course MCQ assessment. These additional questions were created to assess resident learning objective completion without the potential bias of recall from writing the previous pre-course MCQ assessment. We further randomized the question order to control for order bias.

The Wilcoxon signed rank test was used to compare means of pre- and post-course MCQ assessments and Likert scale responses. For the initial comparative assessment, to ensure true pre- and post-course analysis, we only compared questions that were present in both the pre- and post-course test. The additional 15 unique questions on the post-course test were marked independently and reported separately as another indicator of knowledge acquisition. Effect size (Cohen’s d) was used to calculate the magnitude of improvement, which can range from 0.2 (small) to 0.8 (large), while 1.2 is very large and >2.0 is considered a huge effect size.²³ Due to the small sample size, it was determined a priori that only effect sizes of >0.8 were determined clinically meaningful. Feedback was elicited via open-ended questions to assist in guiding any future changes to the curriculum. The α-level was set at 0.05 for statistical significance for all tests. Statistical analysis was performed using IBM SPSS Statistics version 26.0 (Armonk, NY, U.S.).

Results

A total of 14 urology residents were eligible and participated in the course. All participants completed the preand post-course survey, as well as the MCQ assessment. Three residents were not locally available at the time of running the course and, thus, were not able to participate. The cohort consisted of residents of all post-graduate years (PGY) (Three PGY1s, four PGY2s, two PGY3s, one PGY4, and four PGY5s), with a mean age of 28.4±2.6 years old and 85.2% being male.

Theoretical knowledge

Statistically significant improvements were observed in participants’ theoretical knowledge (Table 1). Scores (%, standard deviation [SD]) of the 15 additional questions on the post-course MCQ were similar to the baseline questions (77.6±8.2 vs. 72.4±6.5, p=0.066). Confidence in theoretical knowledge of POCUS significantly improved (all p=0.000), with clinically meaningful effect sizes (d=1.5–2.5) (Table 1).

Table 1.

Theoretical knowledge of POCUS

	Pre-course	Post-course	p	d
Quantitative assessments
MCQ assessment (%)	52.4 (10.9)	72.4 (6.5)	0.000	2.2
Additional questions (%)		76.6 (8.2)
Theoretical confidence*
Indications for POCUS	2.6 (0.6)	4.1 (0.6)	0.000	2.5
Terminology	2.4 (0.7)	4.0 (0.6)	0.000	2.5
Image physics	2.4 (0.9)	3.9 (0.8)	0.000	1.8
Probe selection	1.9 (0.9)	3.1 (0.7)	0.000	1.5

Standard deviation in brackets.

^*Confidence assessed via 5-point Likert data: 1=very unskilled (little to no experience); 2=unskilled (beginner proficiency); 3=intermediate performer (proficient); 4=skilled user (comfortable with use); 5=very skilled (expert).

MCQ: multiple-choice questionnaire; POCUS: point-of-care ultrasound.

Skill confidence

Post-course mean confidence scores significantly improved for all skills (p<0.001–0.003). Meaningful changes were assessed by effect size (all d>0.8). The largest effect size changes were observed in the technical skills of performing kidney POCUS, image-guided suprapubic catheter insertion, and performing testicular POCUS (d =3.4, 2.2, 2.9, respectively) (Table 2).

Table 2.

POCUS skill confidence scores

	Pre-course	Post-course	p	d
Performing kidney POCUS	1.7 (0.7)	4.1 (0.7)	0.001	3.4
Normal appearance	2.2 (1.1)	4.1 (0.5)	0.002	2.2
Detect hydronephrosis	2.4 (1.2)	4.0 (0.7)	0.002	1.6
Detect stones	2.1 (1.1)	3.8 (0.8)	0.003	1.8
Performing bladder POCUS	2.6 (1.0)	4.2 (0.6)	0.001	1.9
Normal appearance	2.6 (1.1)	4.1 (0.6)	0.003	1.8
Suprapubic catheter insertion	2.4 (0.9)	4.1 (0.6)	0.001	2.2
Performing testicular POCUS	1.7 (0.9)	4.0 (0.7)	0.001	2.9
Normal appearance	1.7 (1.1)	3.7 (0.7)	0.002	2.2
Detect torsion	1.9 (1.2)	3.6 (0.6)	0.002	1.8
Doppler imaging	1.6 (0.8)	3.7 (0.7)	0.001	2.8

Standard deviation in brackets; confidence assessed via 5-point Likert scale data: 1=very unskilled (little to no experience); 2=unskilled (beginner proficiency); 3=intermediate performer (proficient); 4=skilled user (comfortable with use); 5=very skilled (expert).

POCUS: point-of-care ultrasound.

Course evaluation

Before and after the course, all participants noted that they were interested in POCUS, believed it is a helpful adjunct to their physical examination skillset, will improve their clinical practice, and that POCUS training should be integrated into Canadian urology residency programs (all responses were above 4.3 on the five-point Likert scale). The post-course comments revealed that participants appreciated hands-on training with experts, while one participant noted that “[This course] should be part of the Royal College’s [of Physicians and Surgeons of Canada] competencies for a modern urologist.”

Discussion

As the availability and accessibility of ultrasound increases, so will the need for its appropriate use and training. We developed a novel introductory POCUS program, designed as the first step for urology residents to gain experience and eventual competency in the modality. This study found that, after a blended online and hands-on course created by ultrasound educators, participants were able to meaningfully improve theoretical knowledge and self-confidence in urological POCUS. Overall, the study succeeded in creating an inexpensive, feasible, guideline-based training program for Canadian urology residents. The basis of this course may serve as a model for eventual competency-based training for urological POCUS in Canada.

There was an overwhelmingly positive response to the course in our cohort. Not only were there significant improvements in participants’ theoretical knowledge (Table 1), there were also meaningful improvements in skill confidence in all taught applications, involving kidney, bladder (including suprapubic catheter insertion), and testicular POCUS (Table 2). These improvements in knowledge address two of the main barriers to safe POCUS implementation: understanding the principles of the tool and how to use ultrasound appropriately. As defined by CAR and CAEP, POCUS should be limited in its scope of use for specific clinical questions, and the tool should enhance the safety of a procedure.^3,14 These principles are of utmost importance for inclusion in any POCUS training program. Furthermore, we used confidence in skills as an early marker for competency development, as has been done in other studies. For example, Kotagal et al performed a similar sized (n=16) POCUS curriculum for general surgery residents and found comparable improvements in self-efficacy and confidence levels in their specialty-specific applications.¹⁶ Overall, the future incorporation of this skill into surgical fields is very likely, and our course represents the first step in competency training for POCUS in Canadian urology.

A key strength of this study comes from the use of national POCUS training guidelines, as well as collaboration with ultrasound educators, to serve as a model for the course’s development.¹⁸ Specifically, the curriculum followed topic guidelines created by several steering groups for appropriate POCUS instruction, including specialty-specific indications for POCUS, ultrasound physics and controls, and anatomy-specific image acquirement and interpretation (Appendix A; available at cuaj.ca).^3,14,24 These topics were supplemented with learning objectives from Sonography Canada’s National Competency Profiles,¹⁸ and the content was hosted on a robust and validated e-learning system.¹⁹ Further, the course enlisted licensed Canadian sonographers from a university-based School of Medical Radiation Sciences for the instruction of the course, lending their expertise attributed to years of training and clinical practice. Collaboration from local ultrasound training institutions, academic centers, and diagnostic imaging centers are resources available to most Canadian urology programs. Inclusion of these experts in consultation for course development can ensure safe implementation of POCUS.

Another unique aspect of this study includes the use of low-cost, homemade imaging simulators (Fig. 1). Use of imaging “phantoms” are common strategies in ultrasound training to simulate real-life pathology and anatomy.²⁵ For example, Fillipou et al used abdominal phantoms to teach urology residents how to perform ultrasound-guided percutaneous needle placement.²⁶ In our setting, we built testicular phantoms based on descriptions in a previously published article.²⁰ This allowed for training of a sensitive exam without the need for recruiting simulated patients, which are typically expensive and require ethics approval. Additionally, we modified our previously published, validated suprapubic catheter insertion training simulator to allow for ultrasound-guided needle access practice.²¹ These simple and cost-effective teaching adjuncts allow for this course and its principles to be reproducible at any academic urology center.

Limitations

Though the results of this study are encouraging and hypothesis-generating, there are some limitations to the study.

First, the course itself does not prove competence in POCUS, nor should it attempt to. The purpose of the study was to develop an introductory course for urology residents to familiarize them to the indications, implications, and use of POCUS in the clinical setting and evaluate its utility and feasibility. Further development with accredited bodies, such as CAR, will need to be pursued if competency-based training and assessment is desired. This is specifically relevant within the Competence by Design (CBD) initiative of the Royal College of Physicians and Surgeons of Canada, where entrustable professional activities (EPAs) will need to be considered for certain procedures.

Second, the implementation of any such training will also require a thorough needs assessment to explore how this training will fit into the current Canadian urology training environment. We recognize that resident workhour restrictions are a barrier to implementing extra training courses; however, within our urology program, we have weekly protected academic half-days, which is what allowed us to provide this training. This could be an option for other residency programs that have protected educational time.

Third, as with any training for competency evaluation, there will be a need for repeat training to maintain a knowledge base and technical skills. The general consensus is that training and assessment needs to be done consistently, especially due to potential advancements in the field.²⁷ Future work is required to determine the validity of any POCUS education program, the subsequent proficiency gained by participants, and the frequency of training.

Finally, this study was limited to one urology residency training program with no study controls, therefore, our findings may not be generalizable to other similar training programs. The implication of POCUS in the clinical setting and its translation into practical use for clinical management still requires further investigation in a Canadian urological training setting, and our program can serve as a model that may be adaptable to other urology residency programs.

Conclusions

Our novel study has provided a new model for introducing POCUS to Canadian urology residents, with an overall goal of eventually building towards a validated competency-based curriculum. This inexpensive, feasible, guideline-based training program, designed in conjunction with ultrasound educators, significantly improved theoretical knowledge and skill confidence in our urology resident cohort. Future directions for urological POCUS bedside imaging include: collaborating and sharing the course constructs with other academic institutions, implementation of skills in clinical practice with validation studies, and collaborating with diagnostic imaging and governing bodies to construct a competency-based curriculum. As POCUS becomes more prevalent in the field of urology, it will likely become a crucial component of the tool kit of the modern urologist.