Creation of a novel course for the advanced emergency medicine ultrasound focused practice designation examination

Abstract

Introduction

The American Board of Emergency Medicine recently established a new certification pathway in advanced emergency medicine ultrasound (AEMUS). Eligible applicants come from a wide range of experience but must pass the same examination to become certified. This study sought to evaluate a novel review course targeting a wide range of learners for a new examination.

Methods

This was a cross‐sectional study evaluating the outcomes of the American College of Emergency Physicians (ACEP) AEMUS review course. The program evaluation survey was designed to collect outcomes across multiple Kirkpatrick levels, including overall satisfaction, topic‐specific satisfaction, topic‐specific coverage, topic‐specific confidence before and after the course, topic‐specific perceived knowledge before and after the course, and changes in practice.

Results

Seventy‐four of 79 participants (93.7%) completed the survey. Mean course satisfaction was 4.59/5.00 (95% CI 4.47–4.72). Confidence to pass examination components increased for the following topics: administration, advanced left ventricle (LV), education, head and neck, hepatobiliary, male genitourinary, musculoskeletal, nonobstetric gynecology, pediatrics, physics, procedures, research, right ventricle, and venous/arterial. Perceived knowledge increased for the following topics: administration, advanced LV, education, head and neck, male genitourinary, musculoskeletal, pediatrics, physics, procedures, renal/bladder, research, right ventricle, and venous/arterial. Fifty‐three of 74 participants (71.6%) stated they would change their practice based on the course. Examples of changes in practice include increased use of ultrasound for advanced abdominal, musculoskeletal, pediatric, and procedural applications. Future work will determine the impact on longer‐term outcomes and focused practice designation (FPD) examination pass rates.

Conclusion

The ACEP AEMUS FPD review course demonstrated high levels of satisfaction, increased participant confidence, increased perceived knowledge, and several self‐reported changes in participants' ultrasound practice.

INTRODUCTION

The use of clinical ultrasound within emergency medicine has grown substantially over the past several decades. While initially limited to a few select examinations among a small group of users, clinical ultrasound has expanded to become a core component of residency training. ¹ As the field advanced, ultrasound fellowships were established to address the increasing need for advanced training in clinical ultrasound. ² , ³ With continued growth of the specialty, there was a need to ensure greater consistency in fellowship training programs and fellowship graduate expertise. ⁴ This led the American Board of Emergency Medicine to establish an advanced certification referred to as a focused practice designation (FPD) in advanced emergency medicine ultrasound (AEMUS) in 2022. ⁵

Participants are eligible for an FPD if they have completed an accredited ultrasound fellowship or met a set of practice‐based criteria and passed the FPD certification examination. Consequently, eligible applicants for the certification examination have a wide range of experience, including variation in rates of fellowship training and postfellowship careers. ⁶ Moreover, variations in fellowship curricular content and incomplete standardization across fellowship can lead to potential knowledge gaps that can vary across participants. Despite this, there were scant preparatory resources available to guide participants in preparing for this novel examination. Therefore, we created the first and only review course for the AEMUS FPD examination. In this study, we sought to conduct a program evaluation of this novel course.

METHODS

This was a cross‐sectional study evaluating the outcomes of the American College of Emergency Physicians (ACEP) AEMUS review course. We utilized the Kirkpatrick program evaluation framework to inform our outcomes. ⁷ , ⁸ This study was deemed exempt by the Rush University Medical Center institutional review board.

The ACEP AEMUS review course was a 2‐day, intensive review course, covering a broad array of topics for the FPD examination. There was no limit on attendees, and they were not required to be eligible to sit for the FPD examination this year. The course did have a small cost and speakers were reimbursed for their time. To develop the content for the course, we followed the CLAIM framework (competitive analysis, literature review with thematic analysis, ask stakeholders, internal review by experts, and mapping the curriculum). ⁹ Our initial competitive analysis demonstrated no similar courses available on the topic. We then performed a literature search. This included reviewing national guidelines, ¹ core content materials, ² , ¹⁰ textbooks, ¹¹ , ¹² , ¹³ and gray literature and seeking input from both junior faculty and national experts. We determined major categories and then filled in all potential clinical ultrasound applications under each category. From this more expansive list, we reached out again to ultrasound leaders who were not involved in the exam writing process to prioritize items as likely versus unlikely to be on the examination. A full list of topics is included in the Appendix S1. Finally, we identified speakers based on national reputation, prior speaking experience, and content expertise with an emphasis on diversity, equity, and inclusion, as well as gender parity.

Individual sessions were designed with an emphasis on active learning and informed by Mayer's Theory of Multimedia Learning. ¹⁴ Due to the COVID‐19 pandemic, all sessions were run using a video conferencing platform (Zoom Video Communications, Inc.). All sessions were performed live, but speakers were asked to send prerecorded versions of all talks in case of technical issues. Chat was enabled and moderated by two dedicated course faculty throughout all sessions. Breaks were scheduled after every two to three didactic sessions. Live question‐and‐answer sessions with speakers and course faculty were held at the midway point and at the end of each day.

Upon completion of the course, participants were sent a survey (Appendix S2). The survey was developed specifically for the needs of this program evaluation. Members of the research team reviewed items in detail to optimize content validity. ¹⁵ This included assessment of comprehension, retrieval, judgment, and response. ¹⁵ To optimize response process validity, we also piloted the survey among ultrasound faculty who were not course attendees and sought feedback. In addition to basic demographics, we collected outcomes for Kirkpatrick Levels 1–3. For Kirkpatrick Level 1, we evaluated participants' overall satisfaction, topic‐specific satisfaction, and topic‐specific coverage. For Kirkpatrick Level 2, we evaluated participants' topic‐specific perceived knowledge and topic‐specific confidence using a retrospective pre/post methodology, which compares participant baseline and postcourse perception of each item after completion of the course. ¹⁶ , ¹⁷ For Kirkpatrick Level 3, we evaluated self‐reported changes in practice resulting from the course. Finally, we sought open‐ended feedback about areas for course improvement.

Surveys were distributed via individualized emails to each participant. Following the modified Dillman method, we sent a minimum of three emails to each participant. ¹⁸ All participants completing the course received a survey (i.e., there were no exclusion criteria).

We reported our data using descriptive statistics (Microsoft Excel 2020, Microsoft Corp.). We conducted thematic analyses of the qualitative feedback using an iterative approach informed by an interpretive paradigm. ¹⁹

RESULTS

Out of 79 attendees, we received 74 total responses (93.7%). Participants had been in practice a mean of 12 years (range 1–33 years) and 62 (83.8%) had completed an ultrasound fellowship. The majority were from academic medical centers (55.4%) or academic hybrid programs (39.2%) compared with community programs (5.4%). Seventy participants (94.6%) were eligible for the FPD examination.

Mean course satisfaction was 4.59/5.00 (95% CI 4.47–4.72). The degree of coverage and satisfaction for each topic was relatively high across all topics (Table 1). There were statistically significant improvements in the confidence to pass the examination components for the following topics: administration, advanced left ventricle (LV), education, head and neck, hepatobiliary, male genitourinary, musculoskeletal, nonobstetric gynecology, pediatrics, physics, procedures, research, right ventricle, and venous/arterial. There were statistically significant improvements in perceived knowledge for the following topics: administration, advanced LV, education, head and neck, male genitourinary, musculoskeletal, pediatrics, physics, procedures, renal/bladder, research, right ventricle, and venous/arterial.

TABLE 1.

Program evaluation survey outcomes

Topic	Degree of coverage	Satisfaction	Precourse confidence	Postcourse confidence	Precourse knowledge	Postcourse knowledge
Admin/QA/infection control	4.64	4.81	3.50	4.05	3.82	4.27
	(4.50–4.77)	(4.71–4.91)	(3.24–3.76)	(3.84–4.27)	(3.58–4.07)	(4.08–4.46)
Adult abdomen	4.76	4.85	3.76	4.15	4.14	4.45
	(4.64–4.88)	(4.76–4.94)	(3.54–3.98)	(3.95–4.35)	(3.95–4.32)	(4.30–4.59)
Advanced LV	4.80	4.78	3.51	4.03	3.89	4.32
	(4.68–4.91)	(4.68–4.89)	(3.28–3.75)	(3.82–4.24)	(3.66–4.13)	(4.14–4.50)
Aorta	4.80	4.82	4.11	4.43	4.43	4.65
	(4.68–4.91)	(4.72–4.93)	(3.90–4.31)	(4.25–4.61)	(4.29–4.58)	(4.53–4.77)
Basic cardiac	4.73	4.82	4.23	4.46	4.62	4.72
	(4.61–4.85)	(4.72–4.93)	(4.02–4.44)	(4.28–4.64)	(4.48–4.76)	(4.60–4.83)
Education	4.70	4.74	3.01	3.73	3.23	4.01
	(4.58–4.83)	(4.60–4.88)	(2.75–3.27)	(3.50–3.96)	(2.95–3.51)	(3.82–4.21)
Head and neck	4.82	4.85	3.47	4.11	3.73	4.35
	(4.72–4.93)	(4.76–4.94)	(3.25–3.69)	(3.92–4.29)	(3.54–3.92)	(4.21–4.49)
Hepatobiliary	4.59	4.72	3.93	4.35	4.30	4.54
	(4.44–4.75)	(4.58–4.86)	(3.71–4.16)	(4.17–4.53)	(4.13–4.47)	(4.40–4.68)
Lung/thoracic	4.66	4.85	4.12	4.43	4.46	4.64
	(4.52–4.80)	(4.76–4.94)	(3.91–4.33)	(4.25–4.62)	(4.30–4.62)	(4.51–4.76)
Male GU	4.86	4.82	3.64	4.20	3.95	4.43
	(4.78–4.95)	(4.72–4.93)	(3.41–3.86)	(4.01–4.40)	(3.75–4.15)	(4.29–4.58)
Musculoskeletal	4.84	4.81	3.59	4.09	3.91	4.38
	(4.74–4.94)	(4.69–4.93)	(3.35–3.84)	(3.89–4.30)	(3.69–4.12)	(4.20–4.56)
Non‐OB gynecology	4.65	4.77	3.63	4.11	4.00	4.36
	(4.50–4.80)	(4.65–4.89)	(3.38–3.87)	(3.90–4.31)	(3.77–4.23)	(4.19–4.54)
OB	4.73	4.82	3.92	4.27	4.24	4.49
	(4.61–4.85)	(4.72–4.93)	(3.69–4.15)	(4.06–4.48)	(4.04–4.44)	(4.32–4.65)
Ocular	4.76	4.86	4.12	4.46	4.50	4.72
	(4.63–4.88)	(4.77–4.96)	(3.90–4.34)	(4.27–4.64)	(4.35–4.65)	(4.60–4.83)
Pediatrics	4.84	4.86	3.35	4.05	3.54	4.27
	(4.74–4.94)	(4.77–4.96)	(3.11–3.59)	(3.85–4.26)	(3.32–3.76)	(4.12–4.42)
Physics	4.72	4.76	2.95	3.70	3.36	3.93
	(4.56–4.87)	(4.63–4.89)	(2.72–3.18)	(3.48–3.92)	(3.13–3.6)	(3.74–4.12)
Procedural/nerve block	4.72	4.81	3.35	3.95	3.70	4.22
	(4.56–4.87)	(4.71–4.92)	(3.10–3.60)	(3.73–4.16)	(3.48–3.93)	(4.03–4.40)
Renal/bladder	4.76	4.82	3.96	4.27	4.16	4.53
	(4.62–4.90)	(4.72–4.93)	(3.75–4.17)	(4.07–4.47)	(3.98–4.35)	(4.40–4.65)
Research	4.72	4.82	2.99	3.68	3.23	3.92
	(4.57–4.86)	(4.73–4.92)	(2.73–3.25)	(3.43–3.92)	(2.99–3.47)	(3.71–4.13)
Right ventricle	4.76	4.77	3.61	4.11	3.97	4.41
	(4.64–4.87)	(4.66–4.88)	(3.37–3.85)	(3.90–4.32)	(3.74–4.21)	(4.24–4.57)
Skin/soft tissue	4.68	4.82	4.15	4.45	4.50	4.66
	(4.53–4.82)	(4.70–4.95)	(3.93–4.37)	(4.26–4.63)	(4.34–4.66)	(4.54–4.79)
Trauma/resuscitative	4.66	4.77	4.15	4.42	4.58	4.66
	(4.53–4.79)	(4.65–4.89)	(3.94–4.36)	(4.23–4.60)	(4.44–4.72)	(4.54–4.78)
Venous/arterial	4.62	4.73	3.42	4.04	3.78	4.23
	(4.47–4.78)	(4.61–4.85)	(3.20–3.64)	(3.85–4.23)	(3.57–4.00)	(4.06–4.40)

Note: Data are reported as course satisfaction score (95% CI).

Abbreviations: GU, genitourinary; LV, left ventricle; OB, obstetrics; QA, quality assurance.

Of the 74 respondents, 53 (71.6%) stated they would change their practice based on the course. Examples of changes in practice include increased use of ultrasound for advanced abdominal, musculoskeletal, pediatric, and procedural applications. Several participants also highlighted that they would change their approach to studying based on new identification of knowledge gaps as a result of the course.

Participants generally felt that the course was comprehensive, organized, and engaging and had a good pace given the content. They highlighted the following strengths: short session length with frequent breaks, high‐quality videos and images, audience engagement (e.g., polls, chat with moderator, question‐and‐answer sessions), speaker diversity, use of live lectures with prerecorded lectures as back‐up or for later review, and the convenience of an online format. Areas for improvement included providing handouts in advance, delivering the more complex topics more slowly or over a longer time period, and shortening or removing more basic content.

DISCUSSION

This program evaluation study describes the outcomes of the first review course for the novel advanced emergency ultrasound FPD examination. We found that course satisfaction was very high both overall and for each of the included topics. Additionally, participants reported increased knowledge and confidence with passing the FPD examination as well as self‐reported changes in practice among nearly three‐quarters of attendees.

One unique facet of this course was the wide range of learner experience, including years in practice and rates of formal ultrasound fellowship completion. Despite this, a large number of participants commented that they felt it was comprehensive and the degree of coverage was very high for all topics. We also received suggestions for improvement, such as the provision of handouts, devoting more time to complex topics, and shortening the more basic material. Based on this feedback, we created a review handout and list of resources that we provided to all attendees after the course. We will also revise the structure of future sessions to allow more time for advanced topics and shorten the basic components.

Interestingly, we found that several topics were associated with reduced confidence rates or baseline knowledge among participants. These included both clinical (e.g., advanced cardiac, head and neck, pediatrics, procedural) and nonclinical topics (e.g., administration, education, physics, research). This highlights a need for further training in these areas during fellowship and can serve to inform fellowship programs and future review courses.

While this work shows promise, outcomes were limited to perceived knowledge and confidence with passing the FPD examination. We intend to conduct future research on subsequent iterations of the program capturing short‐term knowledge acquired, longer‐term outcomes (e.g., perceived confidence, perceived knowledge, knowledge acquired, sustained changes in practice), and FPD examination pass rates of attendees.

LIMITATIONS

It is important to consider several limitations to the study. First, the survey was voluntary and subject to reporting and recall bias. While this was initially distributed immediately after the course, participants were able to complete it up to 1 month later, which may have influenced their recall of specific components. Additionally, reliance on self‐assessment may have overestimated what participants had actually learned. ²⁰ However, the retrospective pre/post assessment may reduce this risk by allowing them to situate their perceived knowledge in the context of their prior knowledge deficits. ¹⁶ , ¹⁷ Similarly, changes in practice were based on self‐report and external data were not available to confirm that these changes occurred or were sustained postcourse. Finally, the timing of the survey did not allow us to capture their FPD examination pass rates.

CONCLUSION

The American College of Emergency Physicians advanced emergency medicine ultrasound review course demonstrated high levels of satisfaction, increased participant confidence and perceived knowledge, and several self‐reported changes in participants' ultrasound practice. Through this program evaluation, we were able to identify several strengths to apply to future programs, areas for improvement, and knowledge gaps for programs and fellows alike.

AUTHOR CONTRIBUTIONS

MG served as Executive Course Director for the ACEP AEMUS review course. PDG served as Assistant Course Director for the ACEP AEMUS review course. DC served as Assistant Course Director for the ACEP AEMUS review course. PL served as Senior Course Director for the ACEP AEMUS review course. MG, PDG, DC, and PL received a small stipend for their role as course directors.

CONFLICT OF INTEREST

The authors declare no potential conflict of interest.

Supporting information

Appendix S1

Appendix S2

Gottlieb M, Duran‐Gehring P, Coneybeare D, Lema P. Creation of a novel course for the advanced emergency medicine ultrasound focused practice designation examination. AEM Educ Train. 2022;6:e10810. doi: 10.1002/aet2.10810