Evaluating a Web‐based Point‐of‐care Ultrasound Curriculum for the Diagnosis of Intussusception

Abstract

Objectives

Intussusception is a pediatric medical emergency that can be difficult to diagnose. Radiology‐performed ultrasound is the diagnostic study of choice but may lead to delays due to lack of availability. Point‐of‐care ultrasound for intussusception (POCUS‐I) studies have shown excellent accuracy and reduced lengths of stay, but there are limited POCUS‐I training materials for pediatric emergency medicine (PEM) providers.

Methods

We performed a prospective cohort study assessing PEM physicians undergoing a primarily Web‐based POCUS‐I curriculum. We developed the POCUS‐I curriculum using Kern’s six‐step model. The curriculum included a Web‐based module and a brief, hands‐on practice that was developed with a board‐certified pediatric radiologist. POCUS‐I technical skill, knowledge, and confidence were determined by a direct observation checklist, multiple‐choice test, and a self‐reported Likert‐scale survey, respectively. We assessed participants immediately pre‐ and postcourse as well as 3 months later to assess for retention of skill, knowledge, and confidence.

Results

A total of 17 of 17 eligible PEM physicians at a single institution participated in the study. For the direct observation skills test, participants scored well after the course with a median (interquartile range [IQR]) score of 20 of 22 (20–21) and maintained high scores even after 3 months (20 [20–21]). On the written knowledge test, there was significant improvement from 57.4% (95% CI = 49.8 to 65.2) to 75.3% (95% CI = 68.1 to 81.6; p < 0.001) and this improvement was maintained at 3 months at 81.2% (95% CI = 74.5 to 86.8). Physicians also demonstrated improved confidence with POCUS‐I after exposure to the curriculum, with 5.9% reporting somewhat or very confident prior to the course to 76.5% both after the course and after 3 months (p < 0.001).

Conclusion

After a primarily Web‐based curriculum for POCUS‐I, PEM physicians performed well in technical skill in POCUS‐I and showed improvement in knowledge and confidence, all of which were maintained over 3 months.

Ileocolic intussusception is a common cause of gastrointestinal obstruction in children under 36 months and happens when one portion of the bowel, the ileum, telescopes into another portion, the colon. Diagnosis can be difficult, because children with intussusception often present with nonspecific symptoms. The classic triad of colicky abdominal pain, palpable mass, and bloody stool is present in less than 50% of patients with intussusception. ¹ Radiology‐performed ultrasound is typically the test of choice for intussusception and has been shown to have excellent accuracy. ² Intussusception ultrasound is technically easy to learn and can be accurately performed by junior radiology trainees after a brief training period. ³ However, barriers exist in obtaining a radiology‐performed ultrasound, including that it may require transport out of the emergency department (ED) or to another health facility, depending on availability, which can delay diagnosis, leading to increased morbidity and mortality. ⁴

Studies evaluating the use of point‐of‐care ultrasound for intussusception (POCUS‐I) by emergency providers have shown promising results. The test accuracy of POCUS‐I performed by pediatric emergency medicine (PEM) physicians is comparable to that of radiology‐performed studies. ⁵ , ⁶ , ⁷ , ⁸ In addition, the use of POCUS‐I results in improved hospital and patient‐centered outcomes including improved length of stay in the ED and expedited time to diagnosis. ⁹ , ¹⁰ , ¹¹ , ¹²

Still, because POCUS‐I is not commonly performed by PEM providers, the best method of training in this application is unclear. Prior POCUS‐I studies typically describe a 1‐ to 4‐hour, in‐person didactic session often taught by pediatric radiologists. ⁵ , ⁶ , ⁹ , ¹⁰ , ¹¹ , ¹² However, this may not always be feasible, and in general, limited educational materials exist for POCUS‐I training. Compared to these in‐person, didactic sessions, online POCUS training allows for increased accessibility, including availability to anyone with an internet connection, flexibility in timing including on‐shift and longitudinal learning, and access to curricula when a local expert is not available. ¹³ Indeed, various online training modules for different POCUS applications have been developed. ¹³ Studies have found that online curricula for POCUS applications, such as ultrasound‐guided venous access and ultrasound assessment of jugular venous pressure, can be effective. ¹⁴ , ¹⁵ However, the data on efficacy of online curricula for POCUS are limited, and furthermore, online curricula for POCUS‐I have not been evaluated. In our study, we developed a novel, Web‐based curriculum for POCUS‐I and sought to assess its efficacy longitudinally by examining technical skill, knowledge, and confidence of PEM providers performing POCUS‐I.

METHODS

This was a prospective evaluation of a primarily Web‐based POCUS‐I educational intervention for PEM faculty and fellows with assessments for technical skill, knowledge, and confidence. This study received institutional review board approval and was supported by a University Innovations Funding in Education Seed Grant.

Curricular Development

We used Kern’s six‐step model to develop the POCUS‐I educational curriculum. This model includes problem identification, targeted needs assessment, goals and objectives, educational strategies, implementation, and evaluation/feedback. ¹⁶ We performed a literature search and identified that POCUS‐I could improve the clinical care of pediatric patients with suspected intussusception. We further noted that while the majority of our faculty had experience with some POCUS applications, they did not know how to perform POCUS‐I. We then performed a needs assessment of POCUS‐I curricular elements, through informal interviews seeking feedback from faculty and fellows in PEM, and created five learning objectives:

1. Describe the indications for performing POCUS‐I.

2. Describe and demonstrate the technique for performing POCUS‐I.

3. Distinguish anatomical landmarks for POCUS‐I.

4. Interpret signs of intussusception on POCUS‐I.

5. List the limitations of POCUS‐I.

In considering the best educational strategies to teach POCUS‐I, we included a combination of educational methods:

1. Web‐based learning: to allow for on‐demand, visual and interactive learning to promote reflection.

2. Case‐based learning: to promote reinforcement of learning using multiple examples to promote retention.

3. Brief, supplemental, in‐person hands‐on session with POCUS experts: to facilitate finetuning of skills necessary to perform POCUS‐I.

4. Multimodal evaluation methods: to assess confidence, knowledge, and technical proficiency.

Study Design and Setting

Our study participants were PEM faculty and fellows from a single academic institution who work in three different urban, pediatric EDs with patient volumes of approximately 9,000, 18,000, and 50,000. At these sites, some shifts have multiple learners including fellows and residents, while other shifts are covered only by attendings. Eligible participants had not completed an ultrasound fellowship. Participation in the curriculum was voluntary. To recruit participants, announcements were made in person at the PEM division’s faculty meeting and followed by an invitation by e‐mail. Participants were enrolled in the study during July 2019 and first accessed the online course and performed their brief in‐person training within that month.

Enrolled participants were given access to a Web‐based curriculum for intussusception (https://www.aliemu.com/courses/point‐of‐care‐ultrasound‐intussusception/, ALiEMU registration required, but free and open to all). The curriculum was primarily Web‐based in that all instructional materials were online, and participants’ only in‐person learning was a brief (less than 15 minutes) hands‐on, practice ultrasound session. The online curriculum consists of an introductory case (text), step‐by‐step ultrasound technique with labeled images, multiple video examples of intussusception ultrasounds and mimics, a brief literature review on POCUS‐I, and a case resolution. The online course further includes a 10‐item knowledge acquisition test, described further below. This curriculum was created and modified with input from our institutions’ PEM POCUS experts (all PEM and ultrasound fellowship trained) as well as from a pediatric radiologist. The curriculum was piloted with residents and attendings not involved in the study, with modifications made for clarity and functionality. Participants also completed a series of assessments prior to the curriculum (pretest), after the curriculum (immediately after the brief hands‐on session; posttest), and 3 months later (Figure 1).

Figure 1.

Timeline of curriculum and assessment. Assessments: 1 = survey on confidence with POCUS‐I; 2 = knowledge and image quiz; 3 = direct observation POCUS‐I skills checklist. POCUS‐I = point‐of‐care ultrasound for intussusception.

Outcome/Measurements

Outcome measurements were assessed in three areas: (1) performance of technical skill (direct observation of POCUS‐I performance on a standardized patient with 22‐point checklist), (2) knowledge acquisition (score on a 10‐question multiple‐choice, online knowledge and image test), and (3) confidence (survey instrument using a 5‐point Likert scale). Because no validated tools for POCUS‐I assessment were available, we developed the above three tools based on a literature search of current conceptual frameworks for general POCUS learning and competency ¹⁷ , ¹⁸ , ¹⁹ and information obtained during needs assessment. These tools were iteratively refined by consensus of three local experts in PEM POCUS and a pediatric radiologist.

The direct observation checklist for POCUS‐I scanning included points for patient positioning and comfort, probe selection, scanning technique, recognition of anatomic landmarks, and identification of findings of ileocolic intussusception and mimics (Data Supplement S1, Appendix S1, available as supporting information in the online version of this paper, which is available at http://onlinelibrary.wiley.com/doi/10.1002/aet2.10526/full). The direct observation checklist was scored by one observer. The knowledge and image test (Data Supplement S1, Appendix S2) and confidence survey (Data Supplement S1, Appendix S3) similarly focused on aspects of POCUS‐I such as image acquisition, image interpretation, identification of landmarks, ultrasound technique, identification of classic findings of ileocolic intussusception, and mimics of ileocolic intussusception. The knowledge and image test included written questions as well as multiple video images of ileocolic intussusceptions, mimics, and anatomic landmarks relevant to POCUS‐I.

As per usual practice, POCUS studies performed by PEM physicians were recorded and underwent quality assurance (QA) in our ED POCUS database (QPath E, Telexy). PEM POCUS faculty reviewed studies for adequacy of images acquired and recorded if studies were true‐positive, true‐negative, false‐positive, or false‐negative based on image review, final read on radiology studies (if performed), and chart review. True positives typically have a characteristic “target sign” (Figure 2) in the transverse view or “sandwich” or “pseudokidney sign” in the longitudinal view. They are also often >2 cm in diameter and have central echogenic mesenteric fat and/or lymph nodes. True negatives do not show the above findings. Technically limited studies are limited by image acquisition (did not obtain all views or images unclear or unlabeled). Due to technical limitations, QA data were only readily available at two of the three sites where participants practice; thus, participants were also surveyed on self‐reported number of POCUS‐I performed before and after the curriculum. Participants were also surveyed on the number of times they accessed the Web‐based module and approximate time spent on the Web‐based curriculum.

Figure 2.

Example of a true positive for intussusception. This is a characteristic target sign lesion, with central mesenteric fat and lymph node, and the diameter measured > 2 cm (not shown). Initially published online by author on ALIEMU.com. Permission given by author to reproduce.

Data Analysis

Standard descriptive statistics were reported for each outcome measure. Performance between direct observation assessments were compared by paired t‐test. The proportion of correct responses on the knowledge and image tests was compared at different time points using Fisher’s exact tests. The confidence survey responses used a 5‐point Likert scale (very unconfident, unconfident, neutral, confident, and very confident). Confidence survey scores for each question were calculated using the proportions of participants responding “confident” or “very confident.” These proportions were compared between assessments with a proportional odds model and “within‐subject” random effects. For all comparisons, a two‐tailed p‐value less than 0.05 was considered statistically significant and all analysis were performed using R version 3.5 or SAS version 9.4.

RESULTS

Seventeen PEM faculty and fellows (13 faculty and four fellows) participated in the study, constituting 100% enrollment of invited participants. Seven of the participants (7/17, 41%) had not received formal POCUS training during fellowship. The majority had been practicing PEM for over 6 years, had not received hands‐on training or lectures on POCUS‐I prior to the start of the study, and had not previously performed POCUS‐I (Table 1).

Table 1.

Demographics and Prior POCUS Experience of Participants (n = 17)

Training Level	n (%)
Fellow	4 (24)
Faculty	13 (76)
Years practicing in PEM (years)
0–3	4 (24)
4–6	2 (12)
7–10	9 (53)
>10	2 (12)
Received formal POCUS training during fellowship
Yes	10 (59)
No	7 (41)
Reported performed POCUS‐I prior to study
0	12 (71)
1–5	3 (18)
6–10	1 (6)
11–25	1 (6)
Received previous hands‐on training with POCUS‐I
Yes	4 (24)
No	13 (76)
Received previous lectures on POCUS‐I
Yes	2 (12)
No	15 (88)

PEM = pediatric emergency medicine; POCUS‐I = point‐of‐care ultrasound for intussusception.

After taking the course, participants scored well on technical proficiency, based on the direct observation checklist, with a median (interquartile range [IQR]) score of 20 of 22 (20–21). Participants were able to maintain this high score (median score of 20) at the 3‐month follow‐up evaluation (IQR = 20–21; p = 0.7; Figure 3). In the initial evaluation, participants most commonly lost points for checklist items involving the identification of the transition from small bowel to large bowel and identification of the ileocecal valve, maintaining awareness of patient comfort, and measuring the diameter of the intussusception and rescanning for the intussusception after viewing all four abdominal quadrants. At the follow‐up evaluation, participants results were overall similar; however, maintaining awareness of patient comfort improved (p = 0.01) and saving representative clips of all four abdominal quadrants decreased (p = 0.03; see Data Supplement S1, Appendix S4, for more detailed results on the direct observation checklist).

Figure 3.

PEM physician performance on POCUS‐I direct observation checklist immediately after the course and 3 months later. PEM = pediatric emergency medicine; POCUS‐I = point‐of‐care ultrasound for intussusception.

On the knowledge and image test, there was significant improvement from a baseline of 57.4% (95% CI = 49.8% to 65.2%) to 75.3% (95% CI = 68.1% to 81.6; p < 0.001), and this improvement was maintained at 3 months at 81.2% (95% CI = 74.5% to 86.8; Figure 4).

Figure 4.

PEM physician performance on knowledge and image test. *Statistically significant difference (p < 0.05) from precourse. PEM = pediatric emergency medicine.

A significantly larger proportion of participants reported confidence in response to the question: “How confident do you feel in performing a point‐of‐care ultrasound for intussusception?” after the curriculum, with only 5.9% (1/17) initially reporting “confident” or “very confident” prior to the course to 76.5% (13/17) after the course (p < 0.001). This increased proportion was maintained at 3 months (76.5%, 13/17; Figure 5). Similarly, for questions examining particular aspects of POCUS‐I, an increased proportion of participants reported confidence in obtaining ultrasound images for POCUS‐I (p < 0.001), interpreting POCUS‐I images (p < 0.001), and displacing bowel gas (p < 0.001) after the curriculum and at 3 months (Figure 5). This increased proportion of participants reporting confidence also was noted for identifying relevant anatomical landmarks such the psoas and rectus muscles after the curriculum and at 3 months (p < 0.001).

Figure 5.

Proportion of PEM physicians responding “confident” or “very confident” to questions on POCUS‐I. *Statistically significant difference (p < 0.05) from precourse. PEM = pediatric emergency medicine; POCUS‐I = point‐of‐care ultrasound for intussusception.

On self‐report, only 29.4% (5/17) participants reported ever performing POCUS‐I prior to the study. In comparison, in the 3 months after the curriculum, 76.5% (13/17) participants reported performing POCUS‐I (nine performed one or two studies, one performed three or four, one performed five or six, two performed seven or more). These findings were similar to our available QA data. Prior to the study, only three participants had ever recorded POCUS‐I studies (a total of seven studies) in our POCUS database and in the 3‐month study period, eight participants recorded 15 total POCUS‐I studies, a 114% increase. Of these three were technically limited studies, 11 were true negatives, and one was true positive. Study participants reported accessing the curriculum a median of two times and spending a median of 15 to 30 minutes on the website.

DISCUSSION

This study demonstrates that a primarily Web‐based POCUS‐I curriculum with a brief, in‐person session resulted in a measurable increase in technical skill, knowledge, and confidence of novice PEM providers in POCUS‐I. These improvements were maintained even after 3 months. To our knowledge this is the first study to evaluate a specific POCUS‐I curriculum and the first to incorporate multiple evaluations to assess learning longitudinally.

Participants were assessed on multiple levels of the Kirkpatrick model for evaluation. ²⁰ While we did not assess level 1 (reaction), participants’ direct reactions to the course, we did assess level 2 (learning) using multiple methods including a survey on participant POCUS‐I confidence, an online multiple‐choice knowledge test with videos and images, and a direct observation assessment of their technical ultrasound skills. We also assessed level 3 (behavior) by examining their change in practice through the number of POCUS‐I performed for patient care. For these levels, participants increased their knowledge and practice of POCUS‐I after exposure and access to the curriculum.

Participants overall felt more confident with performing POCUS‐I and aspects of performing POCUS‐I after the curriculum and at 3 months. Interestingly, there was trend toward decrease in confidence in interpreting POCUS‐I images at the 3‐month follow‐up; however, this was not statistically significant. Furthermore, participant scores on the knowledge and image test, which included questions requiring interpretation of POCUS‐I images, continued to be high, with a trend toward improvement, though not to the level of statistical significance, at the 3‐month follow up.

This study had excellent enrollment with 100% of eligible faculty and fellows participating in this voluntary curriculum. Factors that may have influenced enrollment included strong motivation among participants to learn this skill as well as relatively low time commitment to complete the curriculum. Rather than a lengthy, scheduled, didactic session, the curriculum was Web‐based, self‐directed, and available on demand. Furthermore, participants only needed to complete only one, brief (less than 15 minute) in‐person session. Participants were either PEM fellows or faculty and most had prior formal POCUS training. Thus, they already had familiarity with POCUS and this self‐directed approach may have been appealing; indeed, autonomy has been positively associated with medical specialists’ lifelong learning motivation. ²¹

The benefits of Web‐based curricula have led to its incorporation into ultrasound programs as blended learning, with components of online learning and in‐person learning. ²² , ²³ As was found in this study, incorporation of Web‐based learning has been shown to be useful for learning efficiency, skill acquisition, and operator confidence. ²² , ²⁴ , ²⁵ , ²⁶ , ²⁷ , ²⁸ Especially given the COVID‐19 era of physical distancing, this Web‐based curricular approach is encouraging in teaching a procedural skill, such as POCUS. ²⁹ , ³⁰ , ³¹ , ³²

In this study, most participants reported accessing the website more than once. They also increased the number of POCUS‐I studies they performed in clinical practice during the study period, both in self‐reported data and in QA data. This indicates that they reviewed the Web‐based resource longitudinally and continued to perform the technical skill after the initial educational experience. The Web‐based curriculum was designed to be easy to navigate and includes a visual database of normal and abnormal findings that participants could review on shift as needed. This ease of accessibility and review is a benefit of online POCUS learning ¹³ and likely contributed to the participants’ retention of skills, knowledge, and confidence in this study. Indeed, just‐in‐time training with specific, immediate, and focused training has been shown to increase learner engagement, retention, and success with procedural skills. ³³ , ³⁴ , ³⁵

LIMITATIONS

Our study has several limitations. First, participants were evaluated performing POCUS‐I on a volunteer model rather than actual patients. However, educational ultrasounds on practice patients are known to be an important part of ultrasound learning, and given the relative rarity of intussusception, it is not feasible to use patients for all POCUS‐I skill evaluations. Furthermore, participant‐reported and QA data showed an increase in the number of studies performed on patients with overall adequate and accurate POCUS‐I on our limited QA review. Additional studies are needed to further assess the accuracy of PEM performed POCUS‐I after educational interventions and its effect on clinical decision making.

Second, only one observer scored participants on the direct observation checklist, which could introduce bias. However, there were clearly defined criteria for the checklist items that were determined a priori with a panel of experts to help standardize the evaluation. We also did not perform a preassessment of participants for comparison using the direct observation checklist because the majority of participants had limited or no experience performing POCUS‐I.

Third, participants reported performing a higher number of POCUS‐I studies than the recorded number in our QA database. While this could be due to reporting bias, it is important to note the recorded number is only from two clinical sites and our participants practice at three different sites. We were unable to obtain data from the other site due to technical limitations. Furthermore, participants may have performed studies that they did not officially record. Of the recorded studies, most were true‐negative studies, with one true‐positive and three technically limited studies. Technically limited studies were usually due to lack of certain images or the presence of unlabeled images. The majority of studies performed by our participants were adequate and accurate on review, indicating the participants’ ability to adequately perform the POCUS‐I study.

Finally, this study included a convenience sample of PEM faculty and fellows at a single institution, and thus results may not be generalizable to other populations. Participants were relatively junior with only 11.8% (2/17) practicing PEM for more than 10 years. Participants had a variety of prior experience with POCUS and POCUS‐I. Fifty‐nine percent of participants had had formal POCUS training during fellowship, although few had specific training in POCUS‐I. A recent survey of PEM fellows and attending physicians at four major pediatric academic health centers showed that even less (46.9%) had a formal ultrasound education program during medical school, residency, or fellowship. ³⁶ Our population size was too small to stratify POCUS‐I performance by level of prior POCUS training. Future studies could expand the curriculum to additional learner types, other POCUS applications, and other institutions. Future studies are also needed to evaluate clinical outcome changes after exposure to a primarily Web‐based POCUS curriculum.

CONCLUSIONS

Our study demonstrates that a primarily Web‐based curriculum for a point‐of‐care ultrasound application may be effective at supplementing learning for faculty and fellows. Furthermore, with a Web‐based curriculum, retention of participant skill, knowledge, and confidence may be maintained over time.

Supporting information

Data Supplement S1. Supplemental material.

AEM Education and Training 2021;5:1–9