A Longitudinal Curriculum In Point-Of-Care Ultrasonography Improves Medical Knowledge And Psychomotor Skills Among Internal Medicine Residents

Abstract

Purpose

Despite its growing popularity and clinical utility among hospital-based physicians, there are no formal competency requirements nor training standards for United States based Internal Medicine Residencies for learning point-of-care ultrasonography (POCUS). The purpose of this investigation was to study the impact and effectiveness of a novel POCUS curriculum for an Internal Medicine (IM) residency program.

Patients and methods

This was a Single-Group Educational Quasi-Experiment involving Categorical and Preliminary Internal Medicine Residents in Post-Graduate Years 1 through 3 at a single United States academic tertiary center. The study period was from January 1, 2017, through June 30, 2017, during which time the residents participated in monthly modules including didactics and hands-on ultrasound scanning skills with live models. Participants completed a comprehensive knowledge examination at the beginning and end of the six-month period. Participants were also tested regarding hands-on image acquisition and interpretation immediately before and after the hands-on skills labs. The primary outcome measure was performance improvement in a comprehensive medical knowledge assessment.

Results

In total, 42 residents consented for participation. The residents’ monthly rotations were adjusted in order to accommodate the new educational process. Among 29 participants with complete data sets for analysis, the mean (SD) comprehensive knowledge examination score improved from 60.9% before curriculum to 70.2% after curriculum completion (P<0.001). Subgroup analysis determined that improvement in medical knowledge required attending at least 2 out of the 6 (33%) educational sessions. Attendance at hands-on skills labs correlated significantly with improvement; didactics alone did not.

Conclusion

A longitudinal POCUS curriculum consisting of both didactic sessions and hands-on skills labs improves knowledge, image acquisition, and interpretation skills of residents. Having this curriculum span at least 6 months provides learners the opportunity to attend multiple classes which strengthens learning through repetition while also providing learners flexibility in schedule.

Introduction

Point-of-care ultrasonography (POCUS) is a diagnostic modality that allows skilled clinicians of nearly all disciplines to answer focused clinical questions at the bedside.^1–3 A robust body of evidence shows the effect of POCUS on multiple patient-oriented outcomes, such as decreasing time to diagnosis, reduction of procedural complications, and decreased cumulative exposure to ionizing radiation.^4–8 As a result, numerous medical specialties, including but not limited to emergency medicine, obstetrics/gynecology, sports medicine, and rheumatology, have incorporated POCUS into the core content of their graduate medical education curricula as an adjunct to the standard physical examination.^9–12 In fact, most US medical schools have begun to incorporate POCUS into both clinical and preclinical years as part of undergraduate medical education.^13–15

Investment in POCUS curricula for residency programs has proven to be effective for IM residents in specific modalities.^16–19 Residents recognize the value of ultrasonography when performing invasive procedures and generally desire further training for not only procedural but also diagnostic indications.²⁰ Educational leadership in IM training programs appears to view incorporation of ultrasonography for diagnostic and procedural purposes as valuable. Yet, neither competency requirements nor training standards exist for IM residency programs in the United States.^21,22 Research shows that although many IM trainees use POCUS clinically, most have not received formal training in image acquisition and interpretation.^22,23 Similarly, no broadly accepted board certification exists for practicing hospitalists and POCUS competencies are not included in certification by the American Board of Internal Medicine.²⁴

Prior published works investigating the implementation of POCUS training into IM residency programs largely describe single protocols or clinical indications, such as renal sonography or musculoskeletal or cardiac applications, over more comprehensive curricula.^25–28 In addition, although many published reports describe effectiveness of novel ultrasonography training programs among IM residents, few have reported a longitudinal curriculum. Most of this literature focuses on either 1-time seminars of single-organ systems or single-day workshops. However, longitudinal ultrasonography curricula may improve residents’ long-term competency and knowledge retention compared with a single workshop.²⁹ Implementation of regular competency assessments and brief reviews of online material also results in modest improvement in retesting scores.³⁰

Herein, we describe the design and implementation of a novel longitudinal pilot curriculum in POCUS for IM residents and its subsequent effectiveness in improving knowledge content and hands-on sonographer skills. We hypothesized that this educational intervention would result in a statistically significant improvement across all categories of data collection for knowledge and skills competencies. We considered whether a difference would be seen in the observed effect size between didactic lectures and hands-on skill training. In addition, we sought to collect limited qualitative survey data regarding the individual resident’s current attitudes and feedback about the implementation of such a curriculum.

Participants And Methods

The present prospective, single-group educational quasi-experiment was conducted at an academic tertiary care center in Florida. Study inclusion criteria were all categorical IM resident postgraduate years (PGYs) 1 through 3 and all PGY-1 residents in their preliminary IM year. A baseline comprehensive knowledge assessment was conducted in January 2017. The residents all received a protected 30-min session to take the assessment. No outside materials were allowed during the assessment, and participants were not allowed to discuss the assessment with each other during the testing session. This knowledge assessment was composed of 30 multiple-choice and short-answer questions regarding ultrasonography physics, standardized instrumentation, and organ system-specific protocols (eg, limited renal, cardiac, pulmonary, deep venous system) and focused assessment using sonography in trauma (FAST) (Appendix A). This instrument was developed by the authors for this study after literature/database review failed to identify an externally validated instrument deemed appropriate for the target participants with scope of protocols. The content and language of each item were developed over a six-month period following discussion and revision among local subject matter experts in an iterative process to ensure logical validity. Pilot testing with directed feedback was subsequently performed on upper level IM residents who were not subsequently included as study participants, after which final selection and revision of items were completed.

A survey was included with the pretest and posttest examinations to collect such background information as training level, attitudes, and experience with POCUS. Respondents self-reported their level of experience, frequency of use, and perceived value of POCUS as it relates to clinical care. The posttest survey also collected program evaluation and faculty feedback information. All content was unique and was developed specifically for this protocol by the study investigators. The Mayo Clinic Institutional Review Board determined the protocol to be exempt; nevertheless, written informed consent was obtained from all participants.

The pilot curriculum was implemented on January 1, 2017, and data collection continued through June 30, 2017. All study-eligible residents were encouraged to participate, but enrollment was strictly voluntary. Once monthly, a 30- to 45-min traditional didactic lecture was administered to all participants during the regularly scheduled morning report. Morning reports are considered mandatory by the residency program, but off-service rotations or urgent clinical duties may impact attendance. Each monthly didactic session and each skills station focused on a different application: ultrasound physics and equipment instrumentation; FAST examination; and cardiac, pulmonary, renal, and deep venous systems. These applications were selected after discussion and consensus among the investigators to represent the highest relevance to the IM discipline. Morning lectures began with a 5-question multiple-choice knowledge test (Appendix A). These too were developed through identical processes, but they represented questions different from the comprehensive knowledge assessment (Appendix A). Immediately following the lecture, participants again submitted the same 5 questions as a posttest. All knowledge testing, which included the comprehensive as well as the same-day modality-specific, was administered electronically and the data collected through Google Forms (Alphabet).

Following the lecture, participants were divided into small groups for hands-on skills stations in the Mayo Clinic Simulation Center. Participants scheduled themselves for a 45- to 60 min block offered throughout the afternoon in order to provide greatest flexibility with their clinical duties given that their time was not strictly protected. The simulation center utilizes compensated standardized patients for residents to learn and practice the sonographic techniques and protocols that correlated with the didactics presented earlier the same day. This educational model has been shown to be effective in teaching POCUS.³¹ Standardized patients are part-time employees of the simulation center representing diverse ages, genders, and body mass index, though having been previously identified as possessing adequate anatomy for demonstration.

The hands-on skills stations were led by the same instructor who taught the morning lecture, with assistance from the core POCUS faculty. These stations began with an individual skills assessment before demonstration or teaching, during which the learner was asked to perform 5 sonography tasks within a finite period (ie, 30–60 s, dependent on task complexity). Examples of these tasks are optimization of image depth or gain, identification of anatomical structures, the obtaining of standardized imaging windows, and interpretation of normal or abnormal image findings (Appendix A). Participants’ performance was recorded in real time into a master electronic spreadsheet.

Participants then were divided into small groups and taught equipment operation, image acquisition, standardized sonographic windows, normal anatomy, image interpretation, and abnormal anatomy. Sonosite (Fujifilm, Bothell, Washington) models M-Turbo and X-Porte were the ultrasound machines used for all examinations. Pathological examples, when appropriate, were demonstrated on Vimedix high-fidelity ultrasound simulator (CAE Healthcare). The participants were encouraged to independently scan, with moderation and feedback from the instructor, during the remainder of their allotted time. Following the hands-on instruction and practice, each person was evaluated on ability to perform the same 5 tasks, with their performance again recorded for comparison.

Following completion of the 6-month pilot curriculum, a comprehensive knowledge assessment was administered that was identical to the one at the beginning of the pilot. Only participants who completed the comprehensive examination before and after the study period were included for this analysis of overall improvement.

Statistical Analysis

Statistical analysis compared the before and after data of the cumulative knowledge tests, module-specific knowledge test, and module-specific skills test. A statistics software package, JMP Pro 13.0.0 (SAS Institute Inc), was used for all analyses. Continuous parametric data were analyzed with paired t-test and nonparametric data were evaluated with Wilcoxon signed-rank test as appropriate.

To analyze how attendance may have affected performance, we conducted a post hoc subgroup analysis by stratifying participants into groups on the basis of their attendance. A cutoff value of attending 2 or more of the 6 sessions (33%) was selected to most evenly distribute participants into 2 groups. A Pearson correlation coefficient was calculated for attendance for each educational modality, to determine whether didactic lectures or hands-on skills sessions had greater impact on medical knowledge improvement. P values less than 0.05 were considered statistically significant.

Results

Of 44 eligible participants, 43 consented for enrollment and participated in at least some of the educational sessions. Thirty-six participants completed the baseline comprehensive knowledge assessment before any educational session and 34 completed the same knowledge test post study. Overall, 29 participants completed both the pre- and posttest, which could be paired. Most participants were men and in PGY 1. Survey data collected at the beginning of the study also showed that although most residents perceived POCUS as valuable, they self-reported low levels of baseline experience and frequency of use and mediocre satisfaction with the existing quantity of available POCUS educational content (Table 1).

Table 1.

Baseline Characteristics Of The Participants Who Completed Both Comprehensive Knowledge Assessments

		Likert Scalea,b					Value
Characteristics	Participantsa	1	2	3	4	5	Mean (Range)	SD
Sex
Male	19 (65.5)
Female	10 (34.5)
Residency year
PGY 1	13 (44.8)
PGY 2	9 (31.0)
PGY 3	7 (24.1)
Categorical resident	20 (69.0)
POCUS
Level of experience		8 (27.6)	14 (48.3)	5 (17.2)	2 (6.9)	0 (0.0)	2.03 (1.71–2.36)	0.87
Frequency of use		6 (20.7)	13 (44.8)	9 (31.0)	1 (3.4)	0 (0.0)	2.17 (1.87–2.48)	0.80
Perception of value		0 (0.0)	2 (6.9)	2 (6.9)	6 (20.7)	19 (65.5)	4.45 (4.10–4.79)	0.91
Adequacy of current training		2 (6.9)	9 (31.0)	10 (34.5)	7 (24.1)	1 (3.4)	2.86 (2.49–3.24)	0.99

Notes: ^aCategorical data are presented as number (percentage) of participants or responses. ^bIncreases with number (1, none/low; 5, often/high).

Abbreviations: PGY, postgraduate year; POCUS, point-of-care ultrasonography.

At the completion of the study’s 6 months, overall performance on the comprehensive knowledge assessment improved from 60.9% to 70.2% (P<0.001). A statistically significant improvement was also observed during each monthly module in both knowledge-based questions (Table 2) and competency of performing hands-on tasks (Table 3). Among these monthly modules, performance in knowledge-based questions demonstrated a minimum improvement of 21.2% (renal) and a maximum improvement of 44.4% (physics and instrumentation). Hands-on skills competency showed similar effects, with a minimum task completion improvement of 28.8% (physics and instrumentation) and a maximum of 65.8% (deep venous thrombosis).

Table 2.

Pretest And Posttest Scores For Monthly Modules: Didactics

		Score, Mean (SD), %
Session Module	No. Of Participants	Pretest	Posttest	Mean Difference, %	P-Valuea
Physics	23	44.4 (22.6)	88.6 (11.8)	44.4	<0.001
FAST	15	48.0 (24.8)	69.4 (18.4)	21.4	0.01
Cardiac	12	48.4 (19.8)	90.0 (10.4)	41.6	<0.001
Renal	16	46.2 (14.0)	67.6 (21.8)	21.2	0.003
Pulmonary	18	50.0 (19.8)	77.8 (15.2)	27.8	<0.001
DVT	12	31.6 (18.0)	63.4 (14.4)	31.6	0.001

Note: ^aAll P values show statistical significance.

Abbreviations: DVT = deep venous thrombosis; FAST = focused assessment using sonography in trauma.

Table 3.

Pretest And Posttest Scores For Monthly Modules: Skills Sessions

		Score, Mean (SD), %
Session Module	No. Of Participants	Pretest	Posttest	Mean Difference, %	P-Valuea
Physics	25	60.8 (21.2)	89.6 (11.8)	28.8	<0.001
FAST	8	35.0 (17.8)	80.0 (15.2)	45.0	<0.001
Cardiac	8	30.0 (18.6)	85.0 (14.2)	55.0	<0.001
Renal	6	60.0 (12.6)	96.6 (8.2)	36.6	<0.001
Pulmonary	10	52.0 (27.0)	86.0 (16.4)	34.0	0.004
DVT	7	20.0 (2.8)	85.8 (9.8)	65.8	<0.001

Note: ^aAll P values show statistical significance.

Abbreviations: DVT, deep venous thrombosis; FAST, focused assessment using sonography in trauma.

Attendance was highly variable throughout the curriculum, with greater attendance observed at morning report didactic sessions than for afternoon hands-on skills sessions. Mean attendance at morning didactics was 16 participants (range 12–23) and at hands-on skills stations was 10.7 participants (range 7–25). Subgroup analysis showed that individual attendance of 33% or greater was associated with significant improvement in the comprehensive examination and attendance below this threshold was not (Table 4). In addition, attendance at the hands-on skills stations correlated positively with performance improvement in the comprehensive knowledge assessment (P=0.02), whereas attendance at didactic lectures did not show a significant correlation (Figure 1).

Table 4.

Effect Of Attendance And Educational Modality On Medical Knowledge Improvement

Effect	Improvement	ra	P-Valueb
Mean (range), %
≥33% participation	12.08 (6.40 to 17.77)		<0.001
<33% participation	5.64 (–1.31 to 12.59)		0.10
By attendance modality, mean (SD)
Total		0.2579	0.18
Didactic		0.0697	0.72
Hands-on skills		0.4411	0.02

Notes: ^aPearson correlation coefficient. ^bBold type indicates statistical significance.

Figure 1.

Didactic and hands-on attendance vs improvement in comprehensive knowledge assessment.

Review of program evaluation feedback responses indicated a significantly increased frequency and level of comfort with which residents used POCUS in their clinical practice. Most participants indicated a desire to either maintain or expand curriculum content and its dedicated time. Specifically, 47.1% wanted “exactly this much”; 50.0%, “more”; and 2.9%, “less.” Although all instructional methods were believed to be valuable, participation in hands-on skills sessions was the only method of instruction seen as more valuable than others. Most feedback was positive and focused on the value of the training. Nevertheless, participants did report concerns about the accessibility to the curriculum; most trainees (93.75%) reported that clinical obligations were a barrier that limited their participation and attendance.

Discussion

The present study describes a longitudinal POCUS curriculum encompassing varied clinical indications and protocols. Longitudinally distributed training—as opposed to intense single-encounter training—has shown superiority in skills acquisition, retention, and application.³² Our results support the hypothesis that a structured longitudinal curriculum improves the ability of IM trainees to accurately use and interpret POCUS. The cumulative assessment at the end of the six-month curriculum demonstrated modest but statistically significant improvement in medical knowledge. Although there was no cumulative hands-on skills assessment, the ability to perform ultrasound image acquisition tasks before and after each monthly module consistent showed immediate improvement. As one might expect, those residents with better overall attendance generally demonstrated the greatest improvement on the cumulative knowledge assessment, supportive of the concept that repetition of training is necessary for proficiency in POCUS. Previous studies on repetition have shown that trainees exposed to repeated use of POCUS had better retention of basic anatomy on ultrasonography.²⁹ Our work went beyond the previous literature in finding that this improvement may correlate more so with participation in hands-on sessions, indicating that didactics alone may be insufficient when teaching POCUS.

Our results restate the body of evidence that POCUS can be taught effectively to IM trainees within a short period. Most previous studies used 1 session—didactic or hands-on, or both together—to teach 1 specific body system or POCUS protocol. In contrast, our study taught 6 different systems with continued structured practice in the hands-on sessions over a 6-month course.

Our study describes an example of how a POCUS curriculum can be designed and implemented for IM trainees. The POCUS curriculum should offer dedicated time for trainees such that they have no other obligations to prevent their attendance at educational programs. The sessions should provide didactic and hands-on learning. Didactic sessions are necessary to learn the fundamentals of ultrasonography and imaging technique and to provide opportunity for learners to assess comparison images of normal and abnormal anatomy. Hands-on sessions are fundamentally important because the learner can apply the didactic knowledge through active practice of proper ultrasonographic technique. This type of learning, known as experiential learning, has been shown to be superior to passive didactic learning in regard to focused tasks.³³ Finally, repetition of the hands-on sessions, especially when paired with instructor feedback, is crucial for optimal initial learning and long-term retention.³⁴

The study’s strengths were its longitudinal nature and the integration of hands-on skills training and supervised practice paired with traditional didactic lectures. The size of the resident pool was relatively small but allowed for well-controlled and well-monitored initial training and more one-on-one teaching opportunities. One final finding was that no resident participated in more than 4 of the 6 offered sessions. This was likely due to the busy schedules of the residents which made it extremely difficult to attend all sessions. As previously stated, attending at least two sessions correlated with a statistically significant increase in retained medical knowledge at the six-month cumulative posttest. This indicates the crucial nature that repetition plays in the learning of POCUS. These combined findings show the importance of training programs providing protected time to attend the learning sessions. Ideally, resident curriculums would be structured with at least one day monthly where residents were free from patient and administrative obligations in order to attend dedicated ultrasound teaching. Only through this structured repetition of experiential training of POCUS will training programs make the most impact on their learners.

This study has several limitations. First, it was performed at a single academic medical center and only with IM trainees. Second, no protected time was available for trainees to attend all the optional educational sessions, such that the data of some trainees could be neither adequately captured nor included in the final analysis. Third, we could not control for other confounding variables, such as different amounts of additional instruction that participants may have received during clinical rotations. For example, the participant with the greatest improvement attended few sessions because of clinical obligations in multiple intensive care unit rotations (where ultrasonography is also taught). Fourth, some degree of self-selection bias likely existed wherein the trainees with greatest attendance were probably the most independently motivated to learn POCUS. This motivation may contribute to improved performance rather than strictly attendance itself. Fifth, the nature of the study design is subject to regression to the mean, maturation, history, and test effects. It is also widely recognized that without a comparison group, participants in new educational and curricula studies will almost always show improvement from baseline.^35,36

Our study showed that a combination of didactic lectures and hands-on skills modules performed in repetition over time was an effective method of teaching IM residents. This teaching method not only improved their technical ability to acquire and interpret images but also was positively received by IM trainees and significantly increased their satisfaction of ultrasonography training. Internal Medicine programs can optimize their teaching of POCUS by both structuring their curriculum with at least once-monthly didactic and hands-on teaching sessions as well as ideally providing their learners with dedicated time to maximize resident attendance.

Abbreviations

FAST, focused assessment using sonography in trauma; IM, internal medicine; PGY, postgraduate year; POCUS, point-of-care ultrasonography.

Disclosure

The authors report no conflicts of interest in this work.