Core Content for Pediatric Emergency Medicine Ultrasound Fellowship Training: A Modified Delphi Consensus Study

Abstract

Background

Pediatric emergency medicine (PEM) point‐of‐care ultrasound (POCUS) fellowships exist to provide learners with expertise in ultrasound (US) education, administration, and research oversight. Currently, there are no standardized goals or objectives for these programs, resulting in considerable variability in PEM POCUS fellowship training.

Methods

A modified Delphi survey of PEM and general emergency medicine (EM) POCUS experts in Canada and the United States was conducted to obtain consensus regarding the most important curricular components of a PEM POCUS fellowship training program. Participants were solicited from the P2 Network mailing list and from PEM and EM POCUS fellowship directors listed on the Society of Clinical Ultrasound Fellowships and the Canadian Society of POCUS‐EM Fellowships websites. Curricular components considered as part of the survey included US skills, educational skills, administrative skills, and research requirements. Consensus was considered to have been reached when ≥80% of respondents agreed to either include or exclude the component in fellowship training.

Results

Round 1 of the survey was sent to 311 participants. A total of 118 (37.9%) completed eligibility for the survey, and 92 (78.0%) met eligibility criteria. Of those, 80 (67.8% of eligible participants) completed the first round of the survey. Round 2 of the survey was sent to those who completed part 1, and 64 (80.0%) completed that round. During Round 1, consensus was achieved for 15 of 75 US applications, seven of seven educational skills, nine of 11 administrative skills, and four of six research requirements. In Round 2 of the survey, consensus was reached on two additional US skills, but no additional administrative skills or research requirements.

Conclusions

With a consensus‐building process, the core content for PEM POCUS fellowship training was defined. This can help POCUS educators formulate standardized curricula to create consistent training in POCUS fellowship graduates.

Over the past two decades, the use of point‐of‐care ultrasound (POCUS) has increased exponentially in the emergency care setting.1, 2, 3, 4 Training programs have recognized the importance of ultrasound (US) in the care of both adult and pediatric patients.5, 6, 7, 8, 9, 10, 11 The Accreditation Council for Graduate Medical Education (ACGME) has designated POCUS as a core competency for general emergency medicine (EM) residency graduates.12, 13 POCUS is also listed in the core objectives of the pediatric emergency medicine (PEM) fellowship training guidelines by the American Board of Pediatrics and the Royal College of Physicians and Surgeons of Canada.6 Additionally, a consensus curriculum for core POCUS skills to be taught as part of PEM fellowship were recently published as a guide for PEM fellowship directors and US directors.14

Point‐of‐care US fellowship programs were created within the field of EM with goals to refine POCUS skills and develop proficiency in US education, program development, and oversight.7 There are over 100 EM POCUS fellowship training programs across North America and core content guidelines have been published to guide general EM POCUS fellowship directors.7 Because general EM POCUS fellowships are currently not ACGME certified, these core content guidelines are important for providing a framework for US educators. Prior to 2011, there were few training opportunities in POCUS specific to PEM. Many of the early adopters of PEM POCUS obtained POCUS training through EM POCUS fellowships or through self‐directed learning, using general EM POCUS courses, third‐party courses, or online educational modalities. There are now 13 PEM POCUS fellowship programs listed on the Society of Clinical Ultrasound Fellowships (SCUF) website and one PEM POCUS fellowship listed on the Canadian Society of POCUS‐EM Fellowship website. Several other advanced POCUS training programs are offered, either independent of SCUF or in conjunction with PEM fellowship training. Current PEM US fellowships are loosely based on general EM US fellowships in that they usually require completion of 1000 scans during fellowship as well as an US‐based research project. However, there are no standardized educational objectives within these PEM POCUS advanced training programs, leading to significant variability in PEM POCUS fellowship curricula.15, 16 To create a standardized framework for PEM POCUS fellowship training, we conducted a modified Delphi consensus process to delineate the core content for PEM POCUS fellowships and advanced PEM US training programs.

Methods

The Modified Delphi Process

The Delphi process is an iterative process that aims to achieve consensus opinion among a group of experts based on their responses to a series of questionnaires.17 The process was chosen for this particular study because it allowed for the solicitation of anonymous opinions from a broad group of experts in the field of POCUS. Additionally, the process is more cost‐effective than in‐person meetings, allowing for greater participation for a geographically diverse cohort.

Participants were presented with the various proposed curricular components and were asked to rate their importance for acceptability for inclusion in the curriculum on a scale of 1 to 5, 1 being “not at all important” and 5 being “very important.” Consensus was achieved when ≥80% of respondents agreed on a particular item (either for inclusion or for exclusion from the curriculum). This level of agreement has been used in previous Delphi studies, including studies designed to better define POCUS curriculum for PEM fellows.14 Between rounds, the data were analyzed and the survey was modified to include the data and comments. Modified surveys were then presented to the participants for consideration in the subsequent round. Participants were informed of which curricular components had reached consensus, and these components were removed from consideration for subsequent rounds. Precise data regarding overall group response for nonconsensus items were not provided to survey respondents between rounds to ensure the independence of responses. Additionally, because this was an anonymous survey, there was no mechanism to provide participants with their own Round 1 response to allow them to make an informed decision in Round 2. The decision to pursue further rounds was based on whether there were any significant changes in nonconsensus items between rounds.

Survey Tool Development

The content of the survey tool used for this study was designed by a group of 16 experts in PEM POCUS and general EM POCUS education. All are members of the P2 Network. The P2 Network is an international group of Pediatric POCUS experts and enthusiasts, open to all who are interested. It was formed in 2014 with the intention of sharing expertise, building research collaborations, and offering mentorship in the field of pediatric POCUS.18

The survey was developed using the same framework as a previously published curriculum for general EM POCUS fellowships.7 The survey was divided into four main sections: 1) US applications, 2) educational skills, 3) administration skills, and 4) research skills. Twenty‐one basic US skills, previously determined to be core skills required as part of a PEM fellowship, were not included in the survey, and participants were informed of this at the beginning of the survey.14 Additional basic and advanced POCUS skills deemed relevant to emergency care of the pediatric patient were included in the survey. A broad definition of what was considered relevant to pediatric practice was used. The survey included questions relevant to the structure of POCUS training and skills required for completion of PEM POCUS fellowship training. The survey also solicited suggestions for additional components that should be considered as part of a formal PEM US fellowship curriculum. The survey was reviewed and approved by all members of the research team prior to its implementation. The study was approved by the institutional review board (IRB) of the primary study site, Rhode Island Hospital, in Providence, Rhode Island.

Survey Participants

Participation in the survey was solicited from individuals who practiced in Canada or the United States, had personally completed more than 1500 US scans, and had PEM POCUS leadership positions or training. PEM POCUS leadership or training was defined as meeting one of the following criteria: 1) has completed a PEM POCUS fellowship, 2) serves as a PEM POCUS lead or director, 3) serves as a PEM POCUS fellowship director, or 4) serves as a general EM POCUS fellowship director and teaches PEM POCUS skills as a part of this role. This definition of “expert” was agreed upon by the study authors because it acknowledges the variety of training backgrounds in the current field of PEM POCUS and requires more scans than is typically required of a newly graduating PEM POCUS fellow. Study authors and contributing authors who met inclusion criteria for survey participation were included among those invited to participate. All had an expertise similar to that required for participation in the survey. Potential survey participants were solicited from the P2 Network mailing list and from PEM and EM POCUS fellowship directors listed on the Society of Clinical Ultrasound Fellowships and the Canadian Society of POCUS‐EM Fellowships websites,15, 16 which to our knowledge, offers the most comprehensive list of general and pediatric POCUS fellowships offered in Canada and the United States.

Survey Administration and Data Analysis

REDCap (v. 7.1.2, Vanderbilt University), a secure Web application for building and managing online surveys and databases, was used to distribute the survey and collect the results. Initial participation was solicited via e‐mail, and up to three reminders, 1 week apart, were sent to nonresponders. All questions within the survey required a response, although participants could choose not to finish the survey. Data were analyzed using SAS (v.9.4, SAS Institute).

Results

We conducted Round 1 of the survey in September 2017. A total of 311 thirteen individuals were invited to participate. Of the 280 people from this list with known affiliations, 73.9% indicated PEM as their specialty. A total of 118 responded (37.9%) and 92 (78.0% of respondents) met eligibility criteria. Eighty participants (87% of eligible) completed Round 1. We conducted Round 2 of the survey in March 2018 and 64 experts (80% of eligible) completed this portion of the survey (Figure 1).

Figure 1.

Recruitment of experts to the PEM POCUS fellowship modified Delphi. PEM = pediatric emergency medicine; POCUS = point‐of‐care ultrasound.

The majority of study participants reported working in the United States, and 59.7% of respondents saw only children as part of their practice. Experts were largely involved in POCUS curriculum development for EM residency, PEM fellowship, and EM POCUS fellowships. Many also reported involvement in medical school and pediatrics residency POCUS training. Most of the survey participants reported having a PEM fellowship, an EM POCUS fellowship, or a PEM POCUS fellowship at their institution. A total of 2.6% of experts reported having no PEM or US fellowship at all at their institutions (Table 1).

Table 1.

Round 1 Expert Participant Demographics

Characteristic	n (%)
Country of work (n = 77a)
United States	63 (81.8)
Canada	14 (18.2)
Years of practice since residency (n = 77a)	Median = 6 (IQR = 3.8)
Primary patients seen in ED (n = 77a)
Children only	46 (59.7)
Adults only	24 (31.1)
Children and adults	7 (9.1)
US curriculum development involvement (n = 80)
Medical school	30 (37.5)
Pediatric residency	22 (27.5)
EM residency	41 (51.2)
PEM fellowship	58 (72.5)
PEM POCUS fellowship	32 (4.0)
EM POCUS fellowship	38 (47.5)
Other (APPs/global health/critical care/surgical subspecialties)	3 (3.8)
Fellowships available at expert institution (n = 80)
PEM	53 (66.3)
PEM POCUS	27 (33.8)
EM POCUS	47 (58.8)
None	2 (2.5)

^a n = 77 due to three participants choosing not to complete the demographic portion of the survey.

APP = advanced practice provider; PEM = pediatric emergency medicine; POCUS = point‐of‐care ultrasound; US = ultrasound.

After the first round, there was consensus to include 15 PEM POCUS applications and 20 PEM POCUS fellowship applications in the domains of research, education, and administration. In Round 2, consensus was reached for two additional PEM POCUS applications (POCUS for long‐bone fracture identification and US guided arterial line placement), but no additional consensus was achieved for skills in any of the other domains (Table 2). The applications and skills for inclusion in a PEM POCUS fellowship curriculum are summarized in Figure 2. There was no expert consensus reached on the remaining 58 PEM POCUS applications, two administrative skills, and two research requirements. Percent agreement for these items can be found in Table 3.

Table 2.

Number of Items Receiving ≥80% Consensus in Each Round

Domain (Number of Items Rated)	Round 1	Round 2	Total Items Reaching Consensus, n (%)
PEM POCUS applications (75)	15	2	17 (22.7)
Educational skills (7)	7	0 (none asked)	7 (100)
Administration skills (11)	9	0	9 (81.8)
Research requirements (6)	4	0	4 (66.7)

PEM = pediatric emergency medicine; POCUS = point‐of‐care ultrasound.

Figure 2.

Skills and requirements recommended upon completion of a PEM POCUS fellowship. *Consensus on POCUS curriculum as part of PEM fellowship are outlined by Shefrin et al.14 PEM = pediatric emergency medicine; POCUS = point‐of‐care ultrasound.

Table 3.

POCUS Curriculum Items With no Agreement to Include or Exclude by Modified Delphi: Percentage of Experts Who Ranked Item “Important/Very Important” After Round 2

US applications
Cardiac applications
Cardiac output assessment	64.1
Assess IVC/aorta ratio	37.5
Cardiac valve assessment	29.7
Identify regional wall motion abnormality	28.1
Aortic root assessment	23.4
Gastrointestinal applications
Identify small bowel obstruction	65.6
Identify pneumoperitoneum	46.9
Identify ileus	45.3
Identify gallbladder polyps	25.0
Identify gallbladder masses	15.6
Genitourinary applications
Assessment for renal stones	78.1
Identify testicular torsion	71.9
Identify renal cysts	64.1
Identify congenital renal abnormalities	59.4
Identify hydrocele	59.4
Fetal dating	54.7
Identify epidydimoorchitis	54.7
Identify adnexal torsion	51.6
Assessment for scrotal abscess and cellulitis	51.6
Assessment for adnexal pathology (abscess, cysts)	48.4
Identify renal masses	45.3
Assessment for renal jets	45.3
Identify varicocele	45.3
Identify testicular masses	43.8
Identify retained products of conception	35.9
Assessment of renal parenchyma	28.1
Identify uterine masses or cysts	26.6
Identify testicular cysts	14.1
Renal Doppler assessment	7.8
Ocular applications
Identify retinal detachment	78.1
Identify ocular foreign body	70.3
Identify lens dislocation	62.5
Identify vitreous hemorrhage	60.9
Identify vitreous detachment	57.8
Pupillary assessment	45.3
Extraocular muscle assessment	31.3
Identify retrobulbar hematoma	31.2
Musculoskeletal applications
Identify necrotizing fasciitis	76.6
Identify skull fracture	75.0
Identify rib/sternal fracture	53.1
Identify noninfectious lymph node pathology	46.7
Identify tendon injury	43.8
Identify muscle injury	39.1
Identify ligamentous injury	29.7
Identify myositis	26.6
Soft tissue applications
Identify peritonsillar abscess	76.6
Identify soft tissue masses	59.4
Salivary gland assessment	34.4
Vascular applications
Identify deep vein thrombosis	70.3
Identify aortic dissection	37.5
Identify abdominal aortic aneurysm	35.9
US‐guided procedures
Endotracheal tube placement confirmation	70.3
Thoracentesis	68.8
Suprapubic bladder aspiration	68.1
Child lumbar puncture	64.1
Infant lumbar puncture	62.5
Fracture reduction	60.9
Paracentesis	57.8
Intraosseus needle confirmation	56.3
Peritonsillar abscess drainage	53.1
Administration skills
Develop hospital‐based POCUS program relationships	74.6
Understand vendor/industry relationships	69.8
Research requirements
Dedicated instruction on biostatistics and epidemiology	41.2
Dedicated instruction on grant preparation and submission	41.2

IVC = inferior vena cava; POCUS = point‐of‐care ultrasound; US = ultrasound.

Our survey also included questions regarding general fellowship structure that could serve as a guideline for those developing a PEM POCUS fellowship. In Round 1, experts reached consensus that the completion of an IRB‐approved research project that can lead to publication would meet the requirements for scholarly activity. In Round 2, this was broadened to include the completion of a non–IRB‐reviewed project, such as a podcast or blog, non–peer‐reviewed publication, or other quality improvement or educational initiative. Submission of a case report or case series to fulfill the scholarly activity requirement in fellowship did not reach consensus.

Experts reached consensus (90% agreement) in the first round that a PEM POCUS fellowship should be a 1‐year program completed after PEM fellowship is successfully completed. Combined PEM/PEM US fellowships completed within a 2 year for EM residents or 3‐ or 3.5‐year time frame for pediatrics residents did not reach consensus.

Experts were asked in Round 1 whether third‐party certification should be a requirement for completion of PEM POCUS fellowship, with a request for a “yes,” “no,” or “I don't know” response. This item did not reach consensus (13.8% “yes,” 77.5% “no,” and 8.8% “I don't know) in Round 1. In Round 2, this question was revisited because study authors believed that the “I don't know” option allowed survey respondents to remain undecided, potentially preventing consensus on this topic. As such, the question was changed to remove the “I don't know” response option, and an additional question, “Should eligibility for third‐party certification be a requirement for completion of PEM POCUS fellowship,” was added to further characterize participants’ thoughts on third party certification. In Round 2, 88.9% of respondents answered “no” to the question of whether third party certification should be required as part of fellowship completion. This was the sole item of the survey that was rejected by experts. However, for the question regarding eligibility for certification, consensus was still not reached (46.0% “yes,” 54.0% “no”).

Finally, our survey solicited comments from participants regarding additional elements to be considered as part of a fellowship curriculum. There was one suggestion for leadership training and another for interdisciplinary collaboration and scanning opportunities with various experts such as cardiology, radiology, or anesthesia. These were not felt to be a strong enough response to consider including in Round 2 of the survey. No other suggestions for curriculum content were offered.

The Delphi process was terminated after two rounds as only two additional items reached consensus out of 64 remaining items surveyed that had not achieved consensus on the first round.

Discussion

To our knowledge, this is the first modified Delphi consensus study outlining core content for PEM POCUS fellowships. Through a consensus process, we were able to outline 17 PEM POCUS applications and 20 PEM POCUS fellowship skills in the domains of research, education, and administration, recommended as part of PEM POCUS fellowship training.

Currently, there is a lack of a standardized curriculum or structure for PEM POCUS fellowships, with each program crafting its own goals and objectives. In October 2018, the American Board of Medical Specialties approved a Designation of Focused Practice in Advanced Emergency Medicine Ultrasonography under the American Board of Emergency Medicine,19 but no similar designation exists for the American Board of Pediatrics. The resulting flexibility in the structure of PEM POCUS fellowships has resulted in significant variability in the outcome of training. By outlining core elements of PEM POCUS training that should be included in all US fellowship programs, new PEM US fellowship graduates should demonstrate uniformity in knowledge and expertise, regardless of where they trained.

In designing our survey tool, we were deliberately broad in our inclusion of US applications to be considered for fellowship training, primarily because uses for POCUS are continually expanding and clinical applicability varies by hospital setting. Although many items included in the survey did not meet criteria for inclusion in a formal PEM US fellowship curriculum, this does not preclude individual programs from expanding their curriculum based on local disease patterns, clinical needs, and local expertise. Because POCUS is a rapidly expanding field, and new modalities and applications for POCUS frequently arise, the proposed curriculum will need ongoing reassessments to maintain consistency among training programs to accommodate changes in the uses of POCUS.

There were a number of items included as part of our survey that offer some guidance regarding duration of fellowship, scholarly work requirements, and need for third‐party certification eligibility. Although we achieved consensus that fellowship duration should be 1 year after completion of a PEM fellowship, no consensus was achieved regarding the possibility of a combined PEM/PEM POCUS fellowship. Although a 3‐year PEM fellowship can be tailored to include many components of a PEM POCUS fellowship, whether an individual can complete all components of both PEM and PEM POCUS fellowships in a 3‐year time frame, while still satisfying ACGME requirements for a PEM fellowship needs further evaluation.

Limitations

Our study has several limitations. First, it is possible that there were POCUS experts that were not included in our electronic mailing because they did not belong to the organizations from which we solicited our mailing list. Second, the survey tool we generated contained items that are current to the practice of POCUS in the ED setting. Other items specific to local practice variation may have been appropriate to consider as part of the modified Delphi process. Third, expert participants had a median of 6 years in practice since residency. While this makes for a relatively young group of experts, this is the current reality for PEM POCUS because training in its use is relatively recent. Finally, although the initial mailing list included a large number of experts, the response rate was 37.9%, which may have affected survey results.

Our decision to design this study as a modified Delphi, without providing more precise data on where the group was leaning after Round 1, may have affected the results of our study. In particular, for those skills that almost reached consensus (for example, identifying renal stones or identifying retinal detachment), knowing that the group was leaning toward agreement to include these items may have swayed respondents toward an “important/very important” response. However, because we were seeking more independent responses from participants, we did not feel that providing this data was warranted.

Conclusions

Through a modified Delphi process, we created a guideline for core content to be included in a pediatric emergency medicine point‐of‐care ultrasound fellowship curriculum that excludes previously published14 competencies for an ultrasound curriculum for pediatric emergency medicine fellows. National adoption and implementation of the core components in pediatric emergency medicine ultrasound fellowships will allow for standardization of education of our future pediatric emergency medicine point‐of‐care ultrasound leaders.

P2 Network Point‐of‐care Ultrasound Fellowship Delphi Group

Alyssa Abo, MD, Division of Emergency Medicine, George Washington University School of Medicine and Health Sciences, Washington, DC

Alex Arroyo, MD, Division of Pediatric Emergency Medicine, Department of Emergency Medicine, Maimonides Medical Center, Brooklyn, NY

Kiyetta H. Alade, MD, Department of Pediatrics, Section of Emergency Medicine, Baylor College of Medicine, Houston, TX

Erika Constantine, MD, Division of Pediatric Emergency Medicine, Department of Emergency Medicine, Alpert Medical School of Brown University, Providence, RI

David O. Kessler, MD, MSc, Department of Emergency Medicine, Division of Pediatric Emergency Medicine, Columbia University Vagelos College of Physicians and Surgeons, New York, NY

Charisse Kwan, MD, Division of Emergency Medicine, Department of Pediatrics, University of Toronto, Toronto, Ontario, Canada

Samuel Lam, MD, Department of Emergency Medicine, Sutter Medical Center, Sacramento, CA

Stephanie K. Leung, MD, Department of Pediatrics, Section of Emergency Medicine, Baylor College of Medicine, Houston, TX

Marla Levine, MD, Department of Pediatrics, University of Texas at Austin, Dell Medical School, Dell Children’s Medical Center of Central Texas, Austin, TX

Resa E. Lewiss, MD, Department of Emergency Medicine, Thomas Jefferson University Hospital, Philadelphia, PA

Lorraine Ng, MD, Department of Emergency Medicine, Division of Pediatric Emergency Medicine, Columbia University Vagelos College of Physicians and Surgeons, New York, NY

Allan Shefrin, MD, Department of Pediatrics and Emergency Medicine, University of Ottawa, Ottawa, Ontario, Canada

Adam B. Sivitz, MD, Department of Emergency Medicine, Newark Beth Israel Medical Center, Children's Hospital of New Jersey, Newark, NJ

Amanda Toney, MD, Department of Emergency Medicine, Denver Health Medical Center, University of Colorado School of Medicine, Aurora, CO

Atim Uya, MD, Department of Pediatrics, University of California at San Diego, Rady Children's Hospital, San Diego, CA

Fred Warkentine, MD, Department of Pediatrics, Division of Pediatric Emergency Medicine, University of Louisville/Norton Children's Hospital, Louisville, KY

AEM Education and Training 2020;4:130–138