Teaching emergency ultrasound to emergency medicine residents: a scoping review of structured training methods

Abstract

Background

Over the past 2 decades, emergency ultrasound has become essential to patient care, and is a mandated competency for emergency medicine residency graduation. However, the best evidence regarding emergency ultrasound education in residency training is not known. We performed a scoping review to determine the (1) characteristics and (2) outcomes of published structured training methods, (3) the quality of publications, and (4) the implications for research and training.

Methods

We searched broadly on multiple electronic databases and screened studies from the United States and Canada describing structured emergency ultrasound training methods for emergency medicine residents. We evaluated methodological quality with the Medical Education Research Study Quality Instrument (MERSQI), and qualitatively summarized study and intervention characteristics.

Results

A total of 109 studies were selected from 6712 identified publications. Publications mainly reported 1 group pretest–posttest interventions (38%) conducted at a single institution (83%), training in image acquisition (82%) and interpretation (94%) domains with assessment of knowledge (44%) and skill (77%) outcomes, and training in cardiac (18%) or vascular access (15%) applications. Innovative strategies, such as gamification, cadaver models, and hand motion assessment are described. The MERSQI scores of 48 articles ranged from 0 to 15.5 (median, 11.5; interquartile range, 9.6–13.0) out of 18. Low scores reflected the absence of reported valid assessment tools (73%) and higher level outcomes (90%).

Conclusions

Although innovative strategies are illustrated, the overall quality of research could be improved. The use of standardized planning and assessment tools, intentionally mapped to targeted domains and outcomes, might provide valuable formative and summative information to optimize emergency ultrasound research and training.

1. INTRODUCTION

1.1. Background

In the United States, the leading organizations in emergency medicine, including the American College of Emergency Medicine (ACEP), the Society for Academic Emergency Medicine (SAEM), the Council of Emergency Medicine Residency Directors (CORD), the Accreditation Council for Graduate Medical Education (ACGME), and the American Board of Emergency Medicine (ABEM), released collaborative guidelines in 2001 that listed “bedside ultrasonography” as 1 of the procedures and skills integral to the practice of emergency medicine. ¹ In 2012, the ACGME designated emergency ultrasound as an essential patient care skill and mandated that all emergency medicine residents attain competency in emergency ultrasound by the completion of residency training. ² Subsequently, this was endorsed by ABEM, and additional framework for defining competency was provided by the consensus guidelines from the CORD‐SAEM emergency ultrasound milestones project in 2013. ³ , ⁴

To meet expanding competency requirements, emergency medicine residency programs in the United States have been tasked with providing residents adequate emergency ultrasound instruction. Since the first model emergency ultrasound curriculum was developed by Mateer et al ⁵ in 1994, guidelines for emergency ultrasound training have evolved significantly. In 2002, the Scope of Training Task Force recommended that best practice was to teach applications in “discrete sessions” as a 2‐day course with both didactic (lecture) and hands‐on (laboratory) components. ⁶ In 2008, ACEP published comprehensive guidelines, subsequently recognized by the SAEM and the American Institute of Ultrasound in Medicine, that recommended a 1‐day orientation course early in residency training and a standard 2‐day course with lectures and technical components. ⁷ , ⁸ , ⁹ Also in 2008, CORD published a model emergency ultrasound curriculum with minimum education standards and a framework for the integration of emergency ultrasound into resident education. ¹⁰ Revised guidelines were published by ACEP in 2017, which further define the components of emergency ultrasound competency and detail expanded core applications and competency assessment. ¹¹

The Canadian Association of Emergency Physicians (CAEP) first published a general position statement on the use of ultrasound in the emergency department (ED) in 1999. ¹² , ¹³ This was updated in 2006 and 2012 to include training recommendations, including that all practicing emergency medicine physicians be competent in the core applications of focused assessment with sonography for trauma (FAST), abdominal aorta aneurysm identification, first trimester pregnancy, thoracic ultrasound, focused cardiac ultrasound, and guided vascular access. ¹² , ¹³ The 2014 guidelines from the Royal College of Physicians and Surgeons of Canada also included 6 targeted ED ultrasound examinations as a core competency. ¹⁴ In 2018, the Canadian Association of Emergency Physicians working group discussed expanding the core ultrasound applications, and although there was not agreement, they concluded that there was a need for frequent review and reassessment of core emergency ultrasound curriculum. ¹⁵

1.2. Importance

The translation of these guidelines into published literature related to emergency ultrasound education during residency training is largely unknown. We conducted a comprehensive scoping review of the emergency ultrasound education literature with the express purpose of collating, critically appraising, and highlighting quality structured training interventions to better understand the state of emergency ultrasound education research. The scoping review method was selected for its rigor and transparency, with the potential to map primary research, identify gaps in the evidence base, summarize findings, and facilitate use by policy makers and practitioners.

1.3. Goals of this investigation

Our objectives were framed by the following questions:

1. What are the reported range and characteristics of structured emergency ultrasound interventions that have been used to train emergency medicine residents?

2. What types of outcome evidence support the effectiveness of these published methods?

3. What is the quality of the selected emergency ultrasound publications?

4. What are the implications for emergency ultrasound research and training?

2. METHODS

2.1. Protocol and registration

We registered our study protocol on the Center for Open Science (OSF) registry. ¹⁶ Our study followed the Preferred Reporting Items for Systematic Reviews and Meta‐Analyses extension for scoping reviews (PRISMA‐ScR) standards, ¹⁷ which incorporates the guidelines of Arksey and O'Malley, ¹⁸ Levac and colleagues, ¹⁹ and the Johanna Briggs Institute. Our checklist is presented as Supporting Information Table S1. Our protocol included the critical appraisal of individual studies (optional items 12 and 16, PRISMA‐ScR). This scoping review was exempt from institutional ethics approval.

2.2. Eligibility criteria

We included studies that met the following criteria: (1) described an educational intervention (systematic instruction as a program, course, curriculum or pedagogical technique), (2) trained emergency medicine residents at any post‐graduate level, (3) trained in the United States or Canada, (4) were described in English, (5) were prospective studies, surveys, and descriptions, and (6) trained in any of the core 12 emergency ultrasound applications (focused assessment of sonography in trauma, gallbladder, aorta, renal, cardiac, airway/thoracic, gastrointestinal, lower extremity venous, ocular, soft tissue/musculoskeletal, pelvic/obstetric, procedures) or in adjunct emergency ultrasound applications (including but not limited to trans‐esophageal echocardiogram, transcranial Doppler). ¹¹ , ¹⁵ Mixed learning groups that included emergency medicine residents were included.

We excluded studies that: (1) did not describe an educational intervention, (2) whose primary focus was not education, (3) was published in a non‐English language, or (4) were review articles, clinical practice guidelines, or editorials.

2.3. Information sources

The searches were developed and conducted by a health science librarian (HH) trained in systematic searching. A broad set of search terms were identified. Search strategies, using both subject heading and keyword methods, were created for PubMed, Cochrane CENTRAL (Wiley), ERIC (EBSCOhost), and Embase (Elsevier). The initial search was conducted in November of 2017, and was updated in May of 2019. No date restrictions were applied. The PubMed and Embase searches were limited to English. Studies were de‐duplicated using the method described by Bramer and colleagues. ²⁰

2.4. Search

Our PubMed search strategy is presented in Supporting Information Table S2. The search strategies for the other databases are available on request.

2.5. Selection of sources of evidence

All retrieved studies were transferred to Covidence software. ²¹ First, reviewers had several discussions to confirm concepts and definitions relevant to article selection. Then, 2 of 4 reviewers (SL, GAB, LLP, and BMW) independently screened studies by title and abstract and identified articles for full text review. Articles were assigned to a single exclusion category based on a predetermined ordered list of categories. At several group meetings of 3 reviewers, the third reviewer objectively mediated discordance after discussion. Second, full text articles were reviewed by 2 reviewers (LLP and SL). Discordance was mediated by reviewer discussion. The data for both screenings is archived on the Covidence software and is available on request.

2.6. Data charting process

Select data items were charted on custom forms (Microsoft Excel software) by a single reviewer (LLP) and checked for accuracy by other reviewers (SL and BMW). Disagreements were resolved by discussion. Form 1 collected data on study characteristics—(year, country of implementation, design, sample size, number of institutions, and learner type), Form 2 collected data on intervention characteristics (application taught, training domains [indications, image acquisition, image interpretation, clinical integration, and documentation/reimbursement] and learner outcomes [benefit to patients, behavior change, skills, knowledge, self‐efficacy, attitudes, and reaction]), and Form 3 collected data on educational strategies (pre‐intervention [asynchronous learning], intervention [design, learner assessment], and post‐intervention [program and learner evaluation survey]). Individual studies were matched into 1 or several domains.

2.7. Critical appraisal of individual studies

A total of 48 studies were critically appraised for methodological quality using the Medical Education Research Study Quality Instrument (MERSQI) by 2 reviewers (LP, BW). Because of brevity and concision of abstracts, a quantitative analysis with the MERSQI was not performed on the 61 abstracts. The MERSQI is a 10‐item instrument, organized into 6 domains (study design, sampling, type of data, validity, data analysis, and outcomes). Each domain has a maximum score of 3, for a total score of 18. ²² , ²³ There are no published (print or on‐line) training modules available for the MERSQI. Both reviewers (LLP and BMW) self‐trained with reading of primary articles on MERSQI, ²² , ²³ , ²⁴ and item and domain definitions were thoroughly discussed before scoring. Constructs for the 3 validity measures: (1) content, (2) internal structure, and (3) relationship to other variables, followed those presented by Beckman and colleagues. ²⁵ The wide variability in reporting made it necessary for both reviewers (LLP and BMW) to meet frequently and review their independent scores for each of the 48 studies. All items were discussed to consensus.

2.8. Synthesis of results

Studies were grouped by general study characteristics, including design, sample size, learner type and response rate, number of institutions, and by educational intervention characteristics, including emergency ultrasound application, techniques, training domains, outcomes, learner assessment, and program evaluation.

Training strategies were grouped as: (1) pre‐intervention asynchronous learning, (2) intervention design, (3) learner assessment, and (4) post‐intervention survey. Training domains were grouped as: (1) indications, (2) image acquisition, (3) image interpretation, (4) clinical integration, and (5) quality, documentation, and reimbursement.

Training outcomes were grouped as: “Reaction to educational experience,” “Attitudes,” “Self‐efficacy,” “Knowledge,” “Skills,” “Behaviors,” and “Benefit to patients.” The outcome categories were adapted from Kirkpatrick's Hierarchy of Levels of Outcomes. ²⁶ , ²⁷ , ²⁸

3. RESULTS

3.1. Selection of studies

Our PRISMA‐ScR flow diagram is shown as Figure 1. A broad search on multiple electronic databases yielded 6712 studies. For the first screening, a total of 4852 unique studies were screened by title and abstract, yielding 1320 studies. For the second screening, these 1320 studies were screened by full text, and 1211 studies were excluded for the following reasons: 558 studies had the wrong study intention (not an educational focus), 192 studies targeted the wrong learners (not emergency medicine residents), 133 studies were located outside the United States and Canada, 141 studies had the wrong intervention (116 studies were not structured training methods and 25 studies were not emergency ultrasound), 104 studies were not emergency medicine based, and 83 studies were the wrong study design (reviews, guidelines, editorials). The remaining 109 studies, 48 articles ⁵ , ²⁹ , ³⁰ , ³¹ , ³² , ³³ , ³⁴ , ³⁵ , ³⁶ , ³⁷ , ³⁸ , ³⁹ , ⁴⁰ , ⁴¹ , ⁴² , ⁴³ , ⁴⁴ , ⁴⁵ , ⁴⁶ , ⁴⁷ , ⁴⁸ , ⁴⁹ , ⁵⁰ , ⁵¹ , ⁵² , ⁵³ , ⁵⁴ , ⁵⁵ , ⁵⁶ , ⁵⁷ , ⁵⁸ , ⁵⁹ , ⁶⁰ , ⁶¹ , ⁶² , ⁶³ , ⁶⁴ , ⁶⁵ , ⁶⁶ , ⁶⁷ , ⁶⁸ , ⁶⁹ , ⁷⁰ , ⁷¹ , ⁷² , ⁷³ , ⁷⁴ , ⁷⁵ and 61 abstracts, ⁷⁶ , ⁷⁷ , ⁷⁸ , ⁷⁹ , ⁸⁰ , ⁸¹ , ⁸² , ⁸³ , ⁸⁴ , ⁸⁵ , ⁸⁶ , ⁸⁷ , ⁸⁸ , ⁸⁹ , ⁹⁰ , ⁹¹ , ⁹² , ⁹³ , ⁹⁴ , ⁹⁵ , ⁹⁶ , ⁹⁷ , ⁹⁸ , ⁹⁹ , ¹⁰⁰ , ¹⁰¹ , ¹⁰² , ¹⁰³ , ¹⁰⁴ , ¹⁰⁵ , ¹⁰⁶ , ¹⁰⁷ , ¹⁰⁸ , ¹⁰⁹ , ¹¹⁰ , ¹¹¹ , ¹¹² , ¹¹³ , ¹¹⁴ , ¹¹⁵ , ¹¹⁶ , ¹¹⁷ , ¹¹⁸ , ¹¹⁹ , ¹²⁰ , ¹²¹ , ¹²² , ¹²³ , ¹²⁴ , ¹²⁵ , ¹²⁶ , ¹²⁷ , ¹²⁸ , ¹²⁹ , ¹³⁰ , ¹³¹ , ¹³² , ¹³³ , ¹³⁴ , ¹³⁵ , ¹³⁶ were included in this review.

FIGURE 1.

Preferred reporting items for systematic reviews and meta‐analyses extension for scoping reviews (PRISMA‐SCR): study selection process

The chance–adjusted interrater agreement (Cohen's κ with 95% confidence interval [CI]) of each set of paired reviewers in the first screening were 0.33 (0.28–0.38), 0.39 (0.33–0.44), 0.48 (0.37–0.58), and 0.57 (0.51–0.62) and in the second screening was 0.36 (0.26–0.47).

3.2. Characteristics of studies

The 109 studies (48 articles and 61 abstracts) were published from 1994 to 2019. Although 9 (8.2%) studies were published prior to 2005, 10 (9.1%) studies were published between 2006 and 2010, 55 (50.4%) studies were published between 2011 and 2015, and 35 (32.1%) studies have been published from 2016 to 2019. Most studies (86, 78.8%) were published in the United States and 23 (21.1%) were published in Canada.

The study design of 109 studies was analyzed, and there were 41 (37.6%) one group pretest and posttest studies, 34 (31.1%) cross‐sectional or one group posttest studies, 16 (14.7%) randomized studies, and 5 (4.6%) descriptive studies. The majority (91, 83.4%) were conducted at a single institution, and 12 (11%) were conducted at 2 or more sites. The median sample size in the 48 articles was 30 (range, 0–99), the median sample size in the 61 abstracts was 18 (range, 0–900). A total of 57 (52.2%) studies taught emergency medicine residents exclusively.

3.3. Critical appraisal of quality

The MERSQI composite scores of the 48 articles, at the domain and item level, are presented in Table 1. Four (8%) of the 48 studies were descriptive in nature and were included in the MERSQI calculations, but received a score of 0 for all items. Of a total of 18, the MERSQI scores of individual studies ranged from 0 to 15.5 (median, 11.5; interquartile range, 9.6–13.0). At the individual domain level, studies scored best at data measurement and analysis. Common reasons for lower scores were omitting a control group (77%), studying a single institution (78%), not reporting validity of assessment tools (73%), and evaluating skills or attitudes rather than behaviors or patient outcomes (90%). The MERSQI scores of individual studies are presented in Supporting Information Table S4.

TABLE 1.

Summary of MERSQI domain and item scores for 48 selected studies ^a

Domain	MERSQI item	Max. possible score		Studies no. (%) b	Median score (Q1–Q3) c
		Item	Domain		Item	Domain
Study design			3			1.5 (1.0–1.5)
	1. Study design				1.5 (1.0–1.5)
	Descriptive	0	[0]	4 (8)
	Single group cross‐sectional or single group posttest only	1		13 (27)
	Single group pre‐ and post‐test	1.5		20 (42)
	Non‐randomized, 2 group	2		5 (10)
	Randomized controlled trial	3		6 (13)
Sampling			3			2.0 (1.0–2.0)
	2. No. of institutions studied				0.5 (0.5–0.5)
	None	0		4 (8)
	Single institution	0.5		38 (79)
	Two institutions	1		3 (6)
	More than 2 institutions	1.5		3 (6)
	3. Response rate, %				1.5 (0.5–1.5)
	Not applicable	0		6 (13)
	<50 or not reported	0.5		8 (17)
	50–74	1		4 (8)
	>75	1.5		30 (63)
Type of data			3			3.0 (3.0–3.0)
	4. Type of data				3.0 (3.0–3.0)
	No assessment	0		4 (8)
	Assessment by study participant	1		7 (15)
	Objective measurement	3		37 (77)
Validity of evaluation instrument		3			0 (0–2.0)
	5. Internal structure				0 (0–1.0)
	Not reported	0		35 (73)
	Reported	1		13 (27)
	6. Content				0 (0–1.0)
	Not reported	0		34 (71)
	Reported	1		14 (29)
	7. Relationship to other variables				0 (0–1.0)
	Not reported	0		35 (73)
	Reported	1		13 (27)
Data analysis			3			3.0 (3.0–3.0)
	8. Appropriateness of analysis				1.0 (1.0–1.0)
	Data analysis inappropriate for study design or type of data	0		5 (10)
	Data analysis appropriate for study design and type of data	1		43 (90)
	9. Complexity of analysis				2.0 (2.0–2.0)
	No analysis	0		4 (8)
	Descriptive analysis only	1		7 (15)
	Beyond descriptive analysis	2		37 (77)
Outcomes			3			1.5 (1.5–1.5)
	10. Outcomes				1.5 (1.5–1.5)
	None	0		4 (8)
	Satisfaction, attitudes, perceptions, opinions, general facts	1		7 (15)
	Knowledge, skills	1.5		32 (67)
	Behaviors	2		2 (4)
	Patient/health care outcome	3		3 (6)
Total			18			11.5 (9.6–13.0)

MERSQI, medical education research study quality instrument; Max, maximum.

^aTable adapted from Reed et al.²²

^bPercentages may not total 100 due to rounding.

^cInterquartile range reported as Q1–Q3.

An example of a study ⁴² with a high MERSQI score (15.5/18) was a randomized trial with a control group (3/3) that had >75% learner response rate (1.5/1.5). Learners were assessed with a written knowledge pretest and posttest, as well as a skill assessment (3/3). Multiple, blinded evaluators performed a validated observed structured clinical examination (OSCE) test (3/3). Data analysis was appropriate and beyond descriptive analysis (3/3). This study lost points, however, because it was conducted at a single institution (0.5/1.5) and only assessed knowledge and skill outcomes (1.5/3).

This is in contrast to a study ³⁰ with a low MERSQI score (5/18) that was a single group cross‐sectional study (1/3) conducted at a single institution (0.5/1.5), and the sampling rate was not reported (0.5). No validated evaluation instrument was used (0/3) and learners assessed themselves (1/3). Data analysis was descriptive and inappropriate for study design (1/3) and only satisfaction, attitudes, and perception outcomes were assessed (1/3).

3.4. Individual study characteristics

Detailed characteristics of each of the 109 studies are presented in Supporting Information Table S3.

3.5. Synthesis of results

For all 109 studies, training strategies are presented in Table 2, training domains are presented in Table 3, and training outcomes are presented in Table 4.

TABLE 2.

Pre‐intervention, intervention, learner assessment, and post‐intervention educational strategies performed in 109 selected studies

Domains	Studies	N (%)
Pre‐intervention
Asynchronous learning	Amini, 32 Arntfield, 33 Caffery, 39 Chenkin, 42 Gable, 97 Hafez, 101 Hall, 103 Jang, 49 Jang, 50 Laack, 53 Lewiss, 56 Liteplo, 57 McGraw, 63 Minnigan, 114 Norris, 119 Parks, 67 Parks, 123 Parks, 124 Stolz, 74 Stolz, 132 Woodcroft 136	21 (19)
Intervention
Model curriculum	Adhikari, 77 Alkhalifah, 30 Amini, 32 Amini, 31 Bahner, 78 Bayci, 34 Boulger, 37 Chenkin, 82 Field, 94 Gable, 97 Grall, 45 Hall, 103 Hayward, 48 Jones, 51 Lall, 54 Lanoix, 55 Lee, 112 Leung, 113 Mahler, 60 Mandavia, 62 Mateer, 5 McGraw, 63 Noble, 66 Norris, 119 Shah, 70 Stolz 74 , Stolz, 132 Woodcroft 136	28 (26)
Large scale institutional training	Grudziak, 47 Sessler 128	2 (2)
Simulation
Human models	Amini, 31 Amini, 32 Bayci, 34 Berg, 35 Chao, 81 Chenkin, 42 De Lucia, 90 Dulani, 91 Duran Gehring, 92 Hall, 103 Hrymak, 109 Jones, 51 Lewiss, 56 Liteplo, 57 Noble, 66 Salen, 69 Shah, 70 Shah, 129 Shokoohi, 71 Williams 135	20 (18)
Cadaver models	Adan, 76 Ghosh, 99 Herring, 107 Laack, 53	4 (4)
Animal models	Berg, 35 Bloch, 36 Campanella, 40 Ferre, 93 Nguyen‐Phuoc 117	5 (5)
Patients	Jang, 49 Jang, 50 Lanoix, 55 MacVane, 59 Mandavia, 62 Miller, 64 Nguyen, 116 Nguyen‐Phuoc, 117 Norris, 119 Shokoohi, 71 Smalley 72	11 (10)
Mannequins and phantoms	Adan, 76 Akhtar, 29 Alkhalifah, 30 Arntfield, 33 Bayci, 34 Bayers, 38 Caffery, 39 Chenkin, 84 Chenkin 83 , Chenkin, 41 Chenkin, 42 Cho, 86 Corujo, 88 Furman, 96 Girzadas, 44 Godbout, 100 Greenstein, 46 Grudziak, 47 Hakmeh, 102 Hall, 103 Haydel, 106 Hayward, 48 Hrymak, 108 Hrymak, 109 Jagneaux, 110 Laack, 53 Lall, 54 Lewiss, 56 Liteplo, 57 Lobo, 58 Mallin, 61 McGraw, 63 Minnigan, 114 Nguyen, 118 Norris, 119 Olson, 121 Olszynski, 122 O'Keefe, 120 Parks, 67 Parks, 123 Parks, 124 Runde, 127 Salen, 69 Sessler, 128 Sommerkamp, 73 Staum, 131 Woo, 75 Woodcroft 136	48 (44)
Novel educational techniques	Chenkin, 83 Clinton, 87 Field, 95 Gelabert, 98 Kerwin, 52 Kluger, 111 Mallin, 61 Miller, 64 Morse, 65 Nelson, 115 Nguyen, 116 O'Keefe, 120 Olszynski, 122 Shokoohi, 71 Sommerkamp, 73 Williams 135	16 (15)
Case‐based learning	Adhikari, 77 Alkhalifah, 30 Amini, 31 Amini, 32 Bharati, 79 Byars, 38 Byars, 80 Chao, 81 Chenkin, 41 Datta, 89 Duran Gehring, 92 Field, 95 Gable, 97 Girzadas, 44 Grall, 45 Greenstein, 46 Hall, 103 Hayward, 48 Jones, 51 Laack, 53 Lall, 54 Leung, 113 MacVane, 59 Minnigan, 114 Parks, 67 Parks, 123 Parks, 124 Rohra, 126 Stolz, 74 Stolz 132	31 (28)
Social media: blog, Twitter, Facebook, YouTube	Bahner, 78 Hafez, 101 Tyler 133	3 (3)
Multimedia/online modules	Amini, 31 Amini, 32 Bayci, 34 Bharati, 79 Byars, 80 Chao, 81 Chenkin, 42 Chenkin, 43 Chenkin, 85 Datta, 89 Dulani, 91 Field, 94 Field, 95 Gable, 97 Hafez, 101 Hall, 103 Hassani, 105 Kerwin, 52 Laack, 53 McGraw, 63 Minnigan, 114 Nguyen, 116 Norris, 119 Peterson, 125 Platz, 68 Rohra, 126 Shah, 70 Smalley, 72 Stolz, 74 Tyler, 133 Wadhawan, 134 Williams 135 , Woo 75 , Woodcroft 136	35 (32)
Gamification	Lewiss, 56 Liteplo, 57 Lobo, 58 Olson 121	4 (4)
Novel track/elective/rotation/shifts	Boulger, 37 Chenkin, 82 Lee, 112 Haney, 104 Hayward, 48 Mahler, 60 Smalley 72	7 (6)
Deliberate practice, blocked practice, mastery learning	Chenkin, 83 Chenkin, 85 Chenkin, 43 Hayward, 48 McGraw, 63 Smalley, 72 Smith, 130 Woodcroft 136	8 (7)
Learner assessment
Pretest: knowledge, skills, confidence	Akhtar, 29 Alkhalifah, 30 Bayci, 34 Bharati, 79 Campanella, 40 Chenkin 83 , Chenkin 83 , Chenkin 41 , Chenkin, 42 Chenkin, 43 Chenkin, 85 Clinton, 87 Corujo, 88 Datta, 89 Dulani, 91 Ferre, 93 Gable, 97 Gelabert, 98 Greenstein, 46 Grudziak, 47 Hassani, 105 Haydel, 106 Jagneaux, 110 Jones, 51 Kerwin, 52 Kluger, 111 Laack, 53 Lee, 112 Leung, 113 Lewiss, 56 Lobo, 58 Mahler, 60 Mandavia, 62 McGraw, 63 Morse, 65 Nelson, 115 Nguyen, 116 Noble, 66 Parks, 67 Parks, 123 Peterson, 125 Platz, 68 Rohra, 126 Sessler, 128 Shah, 70 Shah, 129 Stolz, 132 Stolz, 74 Wadhawan, 134 Williams, 135 Woo 75	51 (47)
Posttest: knowledge	Akhtar, 29 Amini, 31 Amini, 32 Bayci, 34 Bharati, 79 Byars, 38 Campanella, 40 Chao, 81 Chenkin, 82 Chenkin, 42 Chenkin, 43 Chenkin, 85 Cho, 86 Clinton, 87 Corujo, 88 Datta, 89 Gable, 97 Gelabert, 98 Grall, 45 Grudziak, 47 Hafez, 101 Hassani, 105 Haydel, 106 Jagneaux, 110 Jones, 51 Kerwin, 52 Kluger, 111 Lee, 112 Leung, 113 Lewiss, 56 Lobo, 58 MacVane, 59 Mahler, 60 Mandavia, 62 McGraw, 63 Morse, 65 Nelson, 115 Nguyen, 116 Noble, 66 Norris, 119 Olson, 121 Parks, 67 Parks, 123 Peterson, 125 Platz, 68 Rohra, 126 Salen, 69 Sessler, 128 Shah, 70 Shah, 129 Smalley, 72 Stolz, 132 Wadhawan, 134 Woo 75	54 (49)
Posttest: skills (including OSCE, SDOT, OSATS, GRS, checklist, video review) a	Akhtar, 29Alkhalifah, 30 Amini, 31 Amini, 32 Arntfield, 33 Bayci, 34 Bharati, 79 Byars, 80 Caffery, 39 Chao, 81 Chenkin, 82 Chenkin, 83 Chenkin, 84 Chenkin, 41 Chenkin, 42 Cho, 86 Clinton, 87 Corujo, 88 Datta, 89 De Lucia, 90 Dulani, 91 Duran Gehring, 92 Ferre, 93 Gable, 97 Ghosh, 99 Girzadas, 44 Godbout, 100 Grall, 45 Greenstein, 46 Grudziak, 47 Hall, 103 Hayward, 48 Hrymak, 108 Hrymak, 109 Jagneaux, 110 Jang, 49 Jang, 50 Jones, 51 Laack, 53 Lall, 54 Lanoix, 55 Lee, 112 Leung, 113 Lewiss, 56 Lobo, 58 MacVane, 59 Mahler, 60 Mallin, 61 McGraw, 63 Miller, 64 Nguyen, 116 Norris, 119 O'Keefe, 120 Olson, 121 Parks, Salen, 69 Sessler, 128 Shah, 70 Shah, 129 Smalley, 72 Smith, 130 Sommerkamp, 73 Stolz, 74 Stolz, 132 Williams, 135 Woo, 75 Woodcroft 136	67 (61)
Long‐term assessment	Akhtar, 29 Amini, 32 Arntfield, 33 Bahner, 78 Bayci, 34 Bharati, 79 Chao, 81 Chenkin, 83 Chenkin, 84 Chenkin, 41 Chenkin, 42 Cho, 86 Clinton, 87 Datta, 89 Ferre, 93 Furman, 96 Gable, 97 Godbout, 100 Grall, 45 Haydel, 106 Hayward, 48 Jang, 49 Jang, 50 Kluger, 111 Laack, 53 Lanoix, 55 Leung, 113 MacVane, 59 Mallin, 61 McGraw, 63 Miller, 64 Morse, 65 Noble, 66 Rohra, 126 Smith 130	36 (33)
Assessment: hand motion analysis	Chenkin, 84 Chenkin, 83 McGraw, 63 Woodcroft 136	4 (3.7)
Post‐intervention
Subjective program or learner assessment survey	Adan, 76 Adhikari, 77 Alkhalifah, 30 Amini, 32 Arntfield, 33 Bahner, 78 Bayci, 34 Berg, 35 Bloch, 36 Caffery, 39 Chenkin, 84 Chenkin, 42 Chenkin, 85 Cho, 86 Clinton, 87 Corujo, 88 Dulani, 91 Furman, 96 Ghosh, 99 Girzadas, 44 Grall, 45 Greenstein, 46 Grudziak, 47 Hakmeh, 102 Haney, 104 Hassani, 105 Haydel, 106 Hrymak, 108 Hrymak, 108 Leung, 113 Liteplo, 57 Lobo, 58 Mallin, 61 Nguyen‐Phuoc, 117 Nguyen, 118 Noble, 66 O'Keefe, 120 Olson, 121 Parks, 67 Parks, 123 Parks, 124 Runde, 127 Salen, 69 Sessler, 128 Shah, 70 Shah, 129 Sommerkamp, 73 Staum, 131 Stolz, 74 Stolz, 132 Woo 75	52 (48)

OSCE, Objective Structured Clinical Examination; SDOT, Standardized Direct Observational Assessment Tool; OSATS, Objective Structured Assessment of Technical Skills; GRS, Global Rating Scale.

^aChecklist and video review are bolded.

TABLE 3.

Training domains of 109 selected articles

Domain	Studies	No. (%) a
Recognition of indications/contraindications	Adhikari 77 , Akhtar, 29 Alkhalifah, 30 Amini, 31 Amini, 32 Arntfield, 33 Berg, 35 Byars, 38 Byars, 38 Caffrey, 39 Chao, 81 Field, 94 Gable, 97 Grall, 45 Girzadas, 44 Greenstein, 46 Grudziak, 47 Herring, 107 Kluger, 111 Lall, 54 Lanoix, 55 Mateer, 5 Nelson, 115 O'Keefe, 120 Parks, 123 Sessler, 128 Shah, 129 Woo, 75 Woodcroft 136	29 (27)
Image acquisition	Adan, 76 Akhtar, 29 Alkhalifah, 30 Amini, 31 Amini, 32 Arntfield, 33 Bayci, 34 Berg, 35 Bharati, 79 Bloch, 36 Boulger, 37 Byars, 80 Byars, 38 Caffrey, 39 Chao, 81 Chenkin 41 , Chenkin, 42 Chenkin, 43 Chenkin, 83 Chenkin, 84 Cho, 86 Clinton, 87 Corujo, 88 Datta, 89 De Lucia, 90 Dulani, 91 Duran Gehring, 92 Ferre, 93 Field, 94 Furman, 96 Gable, 97 Ghosh, 99 Girzadas, 44 Godbout, 100 Grall, 45 Greenstein, 46 Grudziak, 47 Hakmeh, 102 Hall, 103 Haney, 104 Hayward, 48 Haydel, 106 Herring, 107 Hrymak, 108 Hrymak, 109 Jagneaux, 110 Jang, 49 Jang, 50 Jones, 51 Laack, 53 Lall, 54 Lanoix, 55 Lee, 112 Leung, 113 Lewiss, 56 Liteplo, 57 Lobo, 58 MacVane, 59 Mahler, 60 Mallin, 61 Mandavia, 62 Mateer, 5 McGraw, 63 Miller, 64 Nguyen, 116 Nguyen, 118 Nguyen‐Phuoc, 117 Noble, 66 Norris, 119 O'Keefe, 120 Olson, 121 Parks, 67 Parks, 123 Parks, 124 Runde, 127 Salen, 69 Sessler, 128 Shah, 70 Shah, 129 Shokoohi, 71 Smalley, 72 Smith, 130 Sommerkamp, 73 Staum, 131 Stolz, 74 Stolz, 132 Williams, 135 Woo, 75 Woodcroft 136	89 (82)
Image interpretation	Adan, 76 Adhikari, 77 Akhtar, 29 Alkhalifah, 30 Amini, 31 Amini, 32 Arntfield, 33 Bayci, 34 Berg, 35 Bharati, 79 Bloch, 36 Boulger, 37 Byars, 38 Byars, 80 Caffrey, 39 Campanella, 40 Chao, 81 Chenkin, 41 Chenkin, 42 Chenkin, 43 Chenkin, 82 Chenkin, 83 Chenkin, 84 Cho, 86 Clinton, 87 Corujo, 88 Datta, 89 De Lucia, 90 Dulani, 91 Duran Gehring, 92 Ferre, 93 Field, 95 Furman, 96 Gable, 97 Gelabert, 98 Ghosh, 99 Godbout, 100 Girzadas, 44 Grall, 45 Greenstein, 46 Grudziak, 47 Hafez, 101 Hakmeh, 102 Hall, 103 Haney, 104 Hassani, 105 Haydel, 106 Hayward, 48 Herring, 107 Hrymak, 108 Hrymak, 109 Jagneaux, 110 Jang, 49 Jang, 50 Jones, 51 Kerwin, 52 Kluger, 111 Laack, 53 Lall, 54 Lanoix, 55 Lee, 112 Leung, 113 Lewiss, 56 Liteplo, 57 Lobo, 58 MacVane, 59 Mallin, 61 Mandavia, 62 Mateer, 5 McGraw, 63 Miller, 64 Minnigan, 114 Morse, 65 Nelson, 115 Nguyen, 116 Nguyen‐Phuoc, 117 Nguyen, 118 Noble, 66 Norris, 119 O'Keefe, 120 Olson, 121 Olzsynski, 122 Parks, 67 Parks, 123 Parks, 124 Peterson, 125 Platz, 68 Rohra, 126 Runde, 127 Sessler, 128 Salen, 69 Shah, 70 Shah, 129 Shokoohi, 71 Smalley, 72 Smith, 130 Staum, 131 Sommerkamp, 73 Stolz, 74 Stolz, 132 Williams, 135 Woo, 75 Woodcroft 136	103 (94)
Clinical integration	Adhikari, 77 Alkhalifah, 30 Amini, 31 Amini, 32 Bayci, 34 Byars, 80 Boulger, 37 Byars, 38 Chao, 81 Clinton, 87 Datta, 89 Dulani, 91 Duran Gehring, 92 Field, 94 Furman, 96 Gable, 97 Girzadas, 44 Godbout, 100 Grall, 45 Greenstein, 46 Hafez, 101 Hayward, 48 Herring, 107 Hrymak, 108 Hrymak, 108 Jang, 49 Jang, 50 Lanoix, 55 Leung, 113 Lewiss, 56 Lobo, 58 MacVane, 59 Mahler, 60 Mandavia, 62 Mateer, 5 Miller, 64 Minnigan, 114 Nelson, 115 Norris, 119 Okeefe, 120 Olson, 121 Olszynski, 122 Parks, 67 Platz, 68 Sessler, 128 Shah, 70 Stolz, 74 Stolz, 132 Woo 75	49 (45)
Accuracy, documentation, quality assurance, reimbursement	Boulger, 37 Lanoix, 55 Mateer 5	3 (3)

^aNumbers (percentages) total >109, as studies may train >1 domain.

TABLE 4.

Outcomes assessment of 109 selected studies (adapted from Kirkpatrick's hierarchy of levels of outcomes ^a )

Assessment category	Method of assessment	Studies	N (%) b
Benefit to patients	Patient‐oriented outcomes	Furman, 96 Jang, 49 Jang, 50 Lanoix, 55 MacVane, 59 Mandavia, 62 Miller, 64 O'Keefe, 120 Sessler 128	9 (8)
Behaviors	Activity monitoring	Amini 31 , Arntfield, 33 Furman, 96 Godbout, 100 Jang, 49 Jang, 50 Laack, 53 Lanoix, 55 MacVane, 59 Mandavia, 62 Nelson, 115 O'Keefe, 120 Tyler 133	13 (12)
Skills (image acquisition, image interpretation)	Performance assessment	Adan, 76 Adhikari, 77 Akhtar, 29 Amini, 31 Amini, 32 Arntfield, 33 Bayci, 34 Bharati, 79 Bloch, 36 Byars, 38 Caffrey, 39 Campanella, 40 Chao, 81 Chenkin, 83 Chenkin, 84 Chenkin, 85 Chenkin, 41 Chenkin, 42 Cho, 86 Clinton, 87 Corujo, 88 Datta, 89 De Lucia, 90 Dulani, 91 Duran Gehring, 92 Ferre, 93 Gable, 97 Gelabert, 98 Ghosh, 99 Godbout, 100 Greenstein, 46 Haydel, 106 Hakmeh, 102 Hassani, 105 Jagneaux, 110 Jang, 49 Jang, 50 Jones, 51 Kerwin, 52 Laack, 53 Lall, 54 Lanoix, 55 Lee, 112 Leung, 113 Liteplo, 57 Lobo, 58 MacVane, 59 Mahler, 60 Mandavia, 62 McGraw, 63 Miller, 64 Minnigan, 114 Morse, 65 Nelson, 115 Nguyen 113 , Nguyen‐Phuoc, 117 Norris, 119 Noble, 66 O'Keefe, 120 Olson, 121 Parks, 67 Parks, 123 Parks, 124 Peterson, 125 Platz, 68 Salen, 69 Sessler, 128 Shah, 70 Shah, 129 Shokoohi, 71 Smith, 130 Sommerkamp, 73 Stolz, 74 Stolz, 132 Smalley, 72 Williams, 135 Woo, 75 Woodcroft 136	84 (77)
Knowledge	Cognitive testing	Adhikari, 77 Akhtar, 29 Amini, 31 Amini, 32 Bayci, 34 Bharti, 79 Bloch, 36 Campanella, 40 Chao, 81 Chenkin, 42 Chenkin, 43 Cho, 86 Clinton, 87 Corujo, 88 Datta, 89 Gable, 97 Grudziak, 47 Haydel, 106 Jagneaux, 110 Jones, 51 Kluger, 111 Laack, 53 Lall, 54 Lee, 112 Leung, 113 Liteplo, 57 Lobo, 58 MacVane, 59 Mahler, 60 Mandavia, 62 McGraw, 63 Minnigan, 114 Nelson, 115 Nguyen, 116 Noble, 66 Norris, 119 Olson, 121 Parks, 123 Parks, 124 Platz, 68 Rohra, 126 Sessler, 128 Shah, 70 Shah, 129 Stolz, 74 Stolz, 132 Tyler, 133 Wadhawan 134	48 (44)
Self‐efficacy: “confidence,” “comfort”	Self‐report/opinion	Adan, 76 Arntfield, 33 Bayci, 34 Berg, 35 Caffrey, 39 Chenkin, 42 Chenkin, 43 Chenkin, 84 Chenkin, 83 Clinton, 87 Corujo, 88 Dulani, 91 Furman, 96 Grall, 45 Greenstein, 46 Grudziak, 47 Hakmeh, 102 Haney, 104 Haydel, 106 Hrymak, 108 Hrymak, 109 Kerwin, 52 Lall, 54 Leung, 113 Lobo, 58 Mallin, 61 Noble, 66 Nguyen, 118 Parks, 67 Parks, 123 Parks, 124 Runde, 127 Sessler, 128 Shah, 70 Shah, 129 Staum, 131 Stolz, 74 Stolz, 132 Woo 75	39 (36)
Attitudes: “useful,” “valuable,” “effective,” “easy”	Self‐report/opinion	Adan, 76 Adhikari, 77 Alkhalifah, 30 Amini, 31 Arntfield, 33 Bahner, 78 Berg, 35 Bloch, 36 Chenkin, 42 Dulani, 91 Grall, 45 Grudziak, 47 Haney, 104 Lall, 54 Liteplo, 57 Lobo, 58 Mallin, 61 Noble, 66 Olson, 121 O'Keefe, Runde, 127 Salen,69 Shah 70 , Shah, 129 Stolz, 74 Woo 75	26 (24)
Reaction to educational experience: “satisfied,” “enjoyed”	Self‐report/opinion	Adhikari, 77 Amini 31 , Alkhalifah, 30 Arntfield, 33 Berg, 35 Chenkin, 42 Corujo, 88 Girzadas, 44 Grall, 45 Greenstein, 46 Grudziak, 47 Leung, 113 Liteplo, 57 Lobo, 58 Mallin, 61 Noble, 66 Salen, 69 Sessler, 69 Shah, 70 Stolz, 74 Stolz, 132 Woo 75	22 (20)
Not applicable	Description	Boulger, 37 Byars, 80 Hayward, 48 Lewiss, 56 Mateer, 5 Chenkin, 82 Field, 94 Field, 94 Hafez, 101 Hall, 103 Herring, 107 Olszynski 122	12 (11)

^aAdapted from Tilson et al.²⁶

^bTotals do not equal 109 because as a study may have measured >1 outcome. Total percentage may not equal 100 because of rounding.

The emergency ultrasound applications taught in the published literature are presented in Supporting Information Table S5. The most common applications were cardiac (20, 18.3%), followed by ultrasound guided vascular access (17, 15.6%). Studies also reported training in the pelvic application (9, 8.2%), undifferentiated hypotension (8, 7.3%), and nerve blocks (6, 5.5%). Twenty‐five (22.9%) studies reported training in multiple (4 or more) applications.

Cadaver models were used to train residents in ultrasound guided peripheral ¹⁰⁷ or regional nerve blocks, ⁷⁶ ultrasound guided vascular access, ⁵³ and for the diagnosis of Achilles tendon rupture. ⁹⁹ Collaborative learning, through team work and gamification, was reported by 4 studies (SonoGames, ⁵⁶ , ⁵⁷ Sound Games, ⁵⁸ and UltraSimageddon ⁴⁶ ). Additionally, 4 studies, which taught ultrasound guided vascular access ⁶³ , ¹³⁶ and transesophageal echocardiogram, ⁸³ , ⁸⁴ assessed their learners with transducer motion metrics (hand motion analysis).

4. LIMITATIONS

We are confident that this scoping review provides a representative range of published work. Along with articles, we have intentionally included conference abstracts. Although these may lack the methodological rigor of articles, we feel that the inclusion of these was necessary to fully describe the vast range of research in emergency ultrasound training. Our review focused on the United States and Canada, and we do not believe that the restriction of our review to research published in the English language negatively affected our results.¹⁴⁷ We performed comprehensive searches, applying a broad search strategy, of the most relevant databases; however, despite our best intentions, the sheer volume of publications precluded a search of the grey literature. Another concern was that hand‐searching of relevant journals and reviewing of reference lists might impair the reproducibility of our results.

Our review has several limitations. First, we recognize that published training methods may not reflect the practice of emergency ultrasound training. Much training is performed during clinical shifts and through unstructured learning processes, which is difficult to capture with planned research. Second, we encountered a number of studies investigating feasibility and test characteristics of emergency ultrasound. Often, these included an educational component, and many included residents. These studies were excluded because their focus was not that of evaluating a structured educational intervention. However, consideration of these studies may provide another facet of emergency ultrasound education. Third, although the use of the MERSQI enabled us to quantitatively evaluate the quality of research, there might be contradictions between what is reported in studies and practice. Fourth, our reviewer training of the MERSQI may be seen as a limitation. However, reviewers felt very comfortable with the concepts and definitions before screening. For our MERSQI analysis, our final decisions were reached by discussion and consensus. We recognize that others might not arrive at the same consensus as us. Last, despite our best efforts, our interrater agreement is low. When screening, we were faced with a large array of literature, beset with inherent heterogeneity and the lack of clear definitions. We were concerned that using overly strict criteria might exclude important evidence. We used a conservative approach during our first and second screening process, by being more inclusive of studies than exclusive. Some reviewers were more conservative than others, and this issue was resolved by consensus.

5. DISCUSSION

To the best of our knowledge, this is the first scoping review that evaluates publications on structured methods for emergency ultrasound training of emergency medicine residents in the United States and Canada. This review presents >2 decades of relevant research, from the first published article in 1994 to mid‐2019. A total of 109 publications, 48 articles, and 61 abstracts describe an exhaustive range of educational characteristics, strategies, and outcomes of structured training methods that has evolved with the development of the field.

There is substantial evidence that emergency ultrasound education has developed significantly in the last 2 decades. An early curriculum, exemplified by that of Mandavia ⁶² et al describes a 2‐day, 16‐hour course that teaches “7 indications,” workshop‐style with lectures followed by an ultrasound “lab.” A contemporary curriculum that blends modern technology with traditional methods is exemplified by that of Stolz et al, ⁷⁴ which describes a 1‐day course consisting of flipped classroom didactics with asynchronous learning, case‐based interactive teaching, and goal‐oriented skills training using checklists. Emergency ultrasound training is observed to have progressed from the teaching of basic emergency ultrasound applications (focused assessment with sonography of trauma [FAST]) ⁷ , ⁶⁹ to more advanced applications (such as transesophageal echocardiogram) ³³ , ³⁸ , ⁴¹ , ⁸³ , ⁸⁴ and clinical integration of skills with protocols and algorithms (such as undifferentiated hypotension). ⁶⁷ , ⁸¹ , ¹⁰⁸ , ¹⁰⁹ , ¹²³ , ¹²⁴ , ¹³¹

There is also strong evidence on the remarkable innovations that showcase emergency ultrasound education over this period. Although many are not new to the field of general medical education, several classic instructional techniques have been creatively adapted to provide novel and fresh approaches to emergency ultrasound training. Twenty‐eight unique curricula are described. On‐line learning, with multimedia modules and through social media, ⁷⁸ , ¹⁰¹ , ¹³³ are reported. Sixty‐eight studies report training with simulation using phantoms and mannequins, human models, animal models, ³⁶ , ⁴⁰ , ⁹³ , ¹¹⁷ and cadavers. ⁷⁶ , ⁹⁹ , ¹⁰⁷ Other successful innovations include large scale multi‐institution initiatives, ⁴⁷ , ¹²⁸ collaborative learning through gamification, ⁵⁶ , ⁵⁷ , ⁵⁸ , ¹²¹ case‐based learning, and learning through deliberate practice, blocked practice, and mastery learning.

The published literature leaves pronounced gaps in our knowledge of training domains, learner assessment, long term learning retention, and the translation of training into practice.

The most recent ACEP guidelines ¹¹ recommend 5 emergency ultrasound training domains: image acquisition, image interpretation, recognition of indications, clinical integration and quality, documentation, and reimbursement. However, the majority of studies report training in image acquisition and interpretation; training in the last 3 domains is only marginally reported. Although we may hope that training in the latter domains takes place in the larger clinical arena, all 5 domains are important separate educational needs for emergency ultrasound in emergency medicine and should be an integral part of any focused emergency ultrasound training.

The majority of reported interventions assess learners on outcomes at the lower Kirkpatrick levels; behavior change and beneficial patient outcomes have not been often reported. The majority of publications report assessment with surveys, subjective self‐assessment, and single observer ratings. Only 28 (26%) of the studies report using validated and standardized assessment instruments, such as the OSCE (Objective Structured Clinical Examination), SDOT (Standard Direct Observational Assessment Tool), OSATS (Objective Structured Assessment of Technical Skills), GRS (Global Rating Scale), and checklists. Only 36 studies (one‐third) report long term assessment of learners, and >10% of the studies reported the translation of training into practice.

Our review reflects selective research interest in the training of applications and procedures. Based on the number of studies, there appears to be a strong interest in procedural guidance (nerve blocks and vascular access) and cardiac applications. However, publications in gallbladder, lower extremity venous, musculoskeletal, and renal applications are limited to single studies, and there is no dedicated gastrointestinal study. Temporal publication numbers also indicate a strong interest in emerging applications, like transesophageal echocardiography. This publication bias makes the evaluation of North American content training guidelines challenging.

We have intentionally focused our review on studies conducted in the United States and Canada. Given the diversity of country‐specific emergency ultrasound training requirements, this geographic focus was necessary, if only to enable the authors to better evaluate training methods against a familiar contextual backdrop of emergency ultrasound guidelines and recommendations. Although it is likely that emergency ultrasound education is influenced by processes particular to individual countries, we believe that our study can inform and empower astute researchers and educators on an international level. In our search, we did find several noteworthy international studies; however, a discussion of these is beyond the scope of this article.

The MERSQI was selected to critically appraise our 48 articles because of its objective methodological rigor and growing body of validity evidence. ²² , ²³ Although the MERSQI has been used to evaluate general education research in internal medicine, ²³ obstetrics and gynecology, ¹³⁷ and surgery, ¹³⁸ and to evaluate research on echocardiography teaching to cardiology fellows, ¹³⁹ this is the first time that the MERSQI has been used to evaluate research on emergency ultrasound training of emergency medicine residents.

Our critical analysis revealed several common weaknesses in study design, including a low number of sampled institutions (usually only a single site), using assessment instruments with unknown or unreported validity, and assessing low‐level outcomes. These weaknesses negatively affected the overall quality of emergency ultrasound education research.

Significant quality improvement would require the careful selection of assessment instruments and reference standards. An ideal reporting structure would use a randomized design at >2 institutions, a learner response rate of more 75%, objective assessment of learners with validated evaluation instruments (at least internal structure, content and relationship to other variables), appropriate and inferential data analysis, and the assessment of patient and healthcare outcomes.

Based on our review, there are several shortcomings that need consideration including how to (1) overcome the heterogeneity in research, (2) gather high level outcome data, and (3) best assess learner proficiency. Of these, we believe that a structured and validated learner assessment strategy should be considered a priority. A practical approach suggested by Hamstra is a 7‐step checklist that includes validating content with experts from multiple institutions, inter‐rater training and assessment, and ongoing item writing development, pilot testing, and construct validity reassessment. ¹⁴⁰ Alternately, SDOT checklists for 10 common emergency ultrasound applications are included in the supplement of the CORD‐AEUS 2013 consensus guidelines. ³

Emergency ultrasound might benefit from the experience of other medical education fields, such as evidence‐based medicine, which has developed guidelines for the development of assessment tools, defined a taxonomy, and created a framework called the Classification Rubric for Evidence‐based Practice Assessment Tools in Education (CREATE) framework that ties the modified Kirkpatrick outcomes levels to intentional instrument design. ²⁶ , ²⁷ , ²⁸ Consideration should be given to the formation of a focused group or a collaborative network dedicated to enhancing the quality of emergency ultrasound education research through the development of robust reporting guidelines and frameworks. We recognize that there may be practical considerations of cost and funding; however, this should not preclude the development of quality standards.

Daily training of emergency ultrasound consists of apprentice‐type encounters in the ED. These are poorly represented in the literature and vary widely with the individual styles of attending emergency physicians and the workload of the ED. Although structured training methods merely represent a fragment of a larger educational system, they play an important role in ensuring that all learners get a modicum of high‐quality, standardized training and assessment.

A robust emergency ultrasound education program requires considerable faculty expertise, dedicated faculty time, training resources, and departmental support, ¹⁰ and programs are faced with the challenge of creating curricula that meet training goals and are time and cost‐effective. Our review suggests that the evaluation of a curriculum or intervention during the planning phase, using the MERSQI or another validated quality instrument, is likely to be valuable to educators.

In summary, this scoping review covers >2 decades of structured emergency ultrasound training and illustrates several innovative advances that mirror the rapid expansion of emergency ultrasound to its current status as an essential component of emergency medicine training and practice. Overall, we found a dearth of rigorous, high‐quality studies. Instead, we found many articles on novel interventions conducted as small, single‐institution studies using unvalidated assessment tools.

Our findings have several important implications for educators and researchers. Research in emergency ultrasound structured training methods would benefit from careful consideration of several areas: underrepresented emergency ultrasound applications, higher‐level outcomes assessment of behavior change and benefit to patients, and measures of instrument and content validity. The use of standardized and intentionally developed planning and assessment tools, mapped to targeted content and outcome domains might provide valuable formative and summative assessments, that would not only benefit research, but also training.

CONFLICTS OF INTEREST

The authors declare no conflicts of interest.

AUTHOR CONTRIBUTIONS

GB, Conceptualization; Formal analysis. HH, Data curation (lead), Visualization, Writing‐original draft, review and editing. NB, Formal analysis. SL and LP, Conceptualization, Formal Analysis, Supervision, Validation, Writing‐original draft, review and editing. BW, Formal analysis, Validation, Visualization, Writing‐original draft, review and editing. LLP takes final responsibility for all contents of this manuscript.

Supporting information

Supporting Information

Supporting Information

Supporting Information

Supporting Information

Supporting Information

PoSaw LL, Wubben BM, Bertucci N, Bell GA, Healy H, Lee S. Teaching emergency ultrasound to emergency medicine residents: a scoping review of structured training methods. JACEP Open. 2021;2:e12439. 10.1002/emp2.12439