Internal Medicine Residents' Retention of Knowledge and Skills in Bedside Ultrasound

ABSTRACT

Background 

The long-term retention of knowledge and skills in bedside ultrasound by internal medicine residents after ultrasound training is not well understood.

Objective 

We sought to determine whether knowledge and skills acquired from focused training in bedside ultrasound are retained over time, and whether retention is related to independent practice.

Methods 

We conducted a prospective observational trial of 101 internal medicine residents at an academic medical center who participated in a bedside ultrasound workshop followed by 12 months of independent practice. Performance was measured on image-based knowledge and skills assessment using direct observation, both before the workshop and 12 months later. Individual usage data were obtained along with a survey on attitudes toward bedside ultrasound.

Results 

Participants' mean knowledge assessment score increased from a baseline of 63.7% to 84.5% immediately after training (P < .001). At 12 months, mean knowledge score fell to 73.0%, significantly different from both prior assessments (P < .001). Despite knowledge decline, the mean skills assessment score improved from a baseline of 30.5% to 50.4% at 12 months (P < .001). Residents reporting more ultrasound use (> 25 examinations) had higher scores in baseline knowledge and skills assessments than those with lower usage (< 25 examinations). Change in knowledge and image acquisition skills between assessments was equal in both subgroups.

Conclusions 

Residents' knowledge of ultrasound improved after brief training but decayed over time, whereas skills showed marginal improvement over the study, with minimal support. Growth and retention of ultrasound abilities were not impacted by usage rates.

What was known and gap

Little is known about how well residents retain focused learning, such as training in bedside ultrasound.

What is new

A prospective study of internal medicine residents' retention of knowledge and skills over 12 months showed a sizable reduction in knowledge and somewhat better retention of skills.

Limitations

Single site, single specialty study limits generalizability; assessment instrument lacks validity evidence.

Bottom line

Ultrasound knowledge from brief training was not sustained, while skills showed some improvement, with neither affected by residents' clinical use of ultrasound.

Introduction

Bedside ultrasound is associated with reduced cost^1,2 and increased procedural safety,^1,3 thus becoming the standard of care for many procedures. Diagnostically, bedside ultrasound is valuable in evaluating hypotension,⁴ dyspnea,⁵ and cardiac pathology.⁶

Accrediting bodies of several non–internal medicine (IM) training programs require competency in specialty-specific ultrasound skills.^7–9 Surveys of IM residents and program leaders demonstrate a need for ultrasound training,^10,11 and some residencies^10,12,13 and medical schools^14,15 have ultrasound training programs. IM residents can acquire ultrasound knowledge and interpret images,^10,12,13 yet there is no consensus about optimal methods for training, determination of competency, or level of trainee supervision. Prior studies have shown a decline in ultrasound knowledge and confidence after training.^16,17 We investigated whether IM residents would learn, use, and retain image acquisition and interpretation skills after focused ultrasound training.

Methods

We conducted a prospective, observational study from May 2013 through June 2014 on the outcomes of an ultrasound workshop for the University of Chicago's IM residency program. All participating residents graduated from US allopathic medical schools. Medicine-pediatrics (n = 15) and preliminary track (n = 7) residents were excluded from the analysis. Before the workshop, the residency had no formal ultrasound training and resident use was primarily for procedural guidance. All course instructors had participated in professional continuing medical education programs for bedside ultrasound and were faculty or fellows who use this technology in the intensive care unit (ICU).

The workshop consisted of a 20-minute lecture, two 1-hour ultrasound practice sessions, and a 15-minute case-based discussion. The lecture reviewed ultrasound principles, image optimization, and specific instructions on performing the target skills: 2-point compression testing to assess for deep venous thrombosis, internal jugular vein, and inferior vena cava identification. Practice sessions were faculty guided, emphasizing hands-on time examining standardized patients (SPs) with a 4:1:1 learner-SP-faculty ratio using bedside ultrasound machines (Sonosite M-Turbo, Bothell, WA) equipped with linear (Sonosite L25x) and phased array (Sonosite P21x) transducers. Faculty coached participants provided individualized feedback; the time spent on each skill varied based on learners' needs. The case discussion occurred between hands-on sessions and reviewed the evidence for deep venous thrombosis screening and volume status estimation using bedside ultrasound. Learners rotated through at least 4 SP stations to experience anatomic variation.

We chose these ultrasound skills because other educators have taught these skills to IM residents successfully,^12,13,18 and they are clinically applicable for internists.^2,4,5 After the workshop, ultrasound machines were made available to IM residents for use during inpatient rotations. All residents were required to participate in the workshop and encouraged to log ultrasound studies, but subsequent use was not mandated. Course faculty were accessible for feedback by personal communication (eg, pager, phone, or e-mail) or appointment at the learner's initiative. The clinical application of resident-performed ultrasound examinations was left to the judgment of the attending physician on service.

Two authors (J.F.M. and J.A.T.) developed the knowledge and skills assessments, which were reviewed and modified by an ICU physician practiced in bedside ultrasound and an educator with expertise in assessment. The assessments were aligned with an ultrasound assessment scale developed via Delphi consensus¹⁹ to include knowledge of examination indications, equipment, image optimization, systematic approach to examination, and interpretation of images. Knowledge assessments were administered before (13 multiple-choice questions), immediately after (13 multiple-choice questions), and 12 months after (12 multiple-choice questions) the workshop. Results were reported as percentage correct. A timed skills assessment of the 3 ultrasound examinations was performed before the workshop and at 12 months. Participants had 3 minutes to (1) identify the internal jugular vein and measure its diameter; (2) perform a 2-point compression examination on 1 leg to assess for deep venous thrombosis; and (3) locate the inferior vena cava using a subcostal cardiac window, measure its diameter, and qualitatively assess for collapsibility on the SP. Participants received no guidance during assessments. Results were reported as the percentage of steps performed correctly from an 18-item checklist, scored by direct faculty observation. We timed the assessment because each task should take under 1 minute to perform, and we reasoned that efficient task completion implies facility with the skill. A written survey assessing attitudes toward bedside ultrasound, prior exposure, and individual usage was administered at 12 months. “High” utilizers were residents reporting ≥ 25 studies and “low” utilizers < 25 studies.²⁰

The Institutional Review Board exempted this study from review.

Results were tabulated (Excel, Microsoft, Redmond, WA) and deidentified prior to analysis. Mean test scores were compared using paired t tests (STATA version 13.1, StataCorp, College Station, TX). Differences in mean score and changes in score among subgroups were compared using 2-tailed t tests with unequal variance.

Results

A total of 101 residents participated in the workshop. See table 1 for participant characteristics and data on ultrasound use.

Table 1.

Participant Characteristics (N = 101)

Ultrasound knowledge increased after the intervention from a mean pretest score of 63.7% to a mean posttest score of 84.5% (P < .001). The mean knowledge score fell to 73% at 12 months, different from prior assessments (P < .001 for both). High utilizers had a higher mean pretest score than low utilizers (66.2% versus 60.5%, P = .029). Otherwise, there was no difference between groups in posttest, follow-up test, or change in score between assessments (figure 1).

Figure 1.

Knowledge Assessment by User Group

Note: Mean knowledge scores at pretest, posttest, and follow up-test for high (> 25 self-performed examinations) and low (< 25 self-performed examinations) utilizers. Mean scores for all users were significantly different at all 3 assessments (P < .001). High utilizers scored significantly higher than low utilizers at the pretest assessment (66.2% versus 60.5%, P = .029), but mean knowledge scores between groups were not significantly different at any other assessment.

The mean skills assessment score increased from the baseline of 30.5% to 50.4% at 12 months (P < .001). High utilizers had higher mean skills scores than low utilizers at both assessments (figure 2). Both groups showed significant improvement between assessments, with no difference in change in skills score between groups (P = .24).

Figure 2.

Skills Assessment by User Group

Note: Mean skills scores at pretest and posttest for high and low utilizers. The difference in scores between high and low utilizers was significant at both assessments (pretest, P = .003; posttest, P = .006), as was the difference between pretest and posttest scores for both groups (P < .001). The change in score between assessments was not significantly different between groups (P = .24).

Discussion

After a brief workshop, resident knowledge of and skills with bedside ultrasound improved, yet knowledge waned and skills progression was limited over time, despite unlimited access to ultrasound machines. Self-reported ultrasound use over the study period was not associated with knowledge retention or skills progression.

Knowledge increased after training and fell over the subsequent 12 months. The posttest may have reflected transient improvement rather than durable learning. The decline in knowledge and image interpretation over time did not differ between high and low utilizers. A cohort of IM residents also saw their scores wane over time,¹⁶ like our subjects. These residents' opportunities for continued practice were limited, whereas our subjects had unfettered access to ultrasound machines. A recent study showed that regular, faculty-led “ultrasound rounds” helped IM residents retain image recognition skills.¹⁷ Additionally, a longer ultrasound rotation (4 versus 2 weeks)²¹ and 1-on-1 proctoring²² were associated with greater short-term knowledge improvements, but only the longer rotation was associated with improved retention.

While knowledge diminished over time, image acquisition skills increased between assessments. We suspect some improvement was due to cumulative practice and resident-to-resident teaching. Kimura et al¹³ described resident-to-resident transmission of knowledge as critical to their ultrasound curriculum's full implementation. Residents can readily demonstrate the mechanics of image acquisition to one another, whereas integrating knowledge and interpretation skills requires exposure, study, and practice. The disparity in knowledge and skills over time underscores the multiple domains of learning required for effective bedside ultrasound use.

Given the knowledge decline and anemic skills progression shown here, one may ask, What constitutes an effective ultrasound curriculum? Results from high-quality reports of ultrasound training^{10,12,13,16,17} and studies teaching cardiac ultrasound to IM residents,^6,20,23,24 and extrapolating from emergency medicine⁷ literature may provide educators with a reasonable starting point. We suggest a curriculum that includes 4 elements: (1) review of ultrasound basics and evidence-based scope of practice for particular examinations^{6,7,12,13,16,23}; (2) a combination of supervised and independent image acquisition using actual patients for a total of approximately 20 to 30 examinations for each particular ultrasound examination^{6,7,13,17,23,24}; (3) review of a catalogue of normal and pathologic findings for each particular examination^6,7,23; and (4) continued demonstration of proficiency over time.^7,13 See table 2 for details.

Table 2.

Proposed Framework for Ultrasound Curricula

The American Society of Echocardiography recognizes that an “acceptable level of skill” in focused cardiac ultrasound is obtainable in residents after 20 to 30 studies “if the scope of acquisition and interpretation were limited”²³; however, some skills are easier to learn,^13,20 and some learners are more adept. Thus, a specific score or study volume for determining “competency” may not be useful. Our subjects' skills score of 50% at 12 months is poor and probably does not meet a standard of “competence,” yet a crucial missing piece to determining competence is an assessment tool with contextual validity. Our timed assessment artificially reproduced pressure but was conducted in an examination setting on an SP. Thus, while the assessment can show learner progression, insufficient validity evidence exists to use this instrument to determine competence.

Limitations of this study include the lack of a control group, the fact that real-time ultrasound usage was not tracked, and the use of estimated logs for tracking the specific numbers of studies performed, which may have introduced error. Our study was conducted in a single academic IM program, and the assessment instruments were developed internally, with lack of validity evidence. Another limitation is the lack of an immediate post-workshop skills assessment, so we cannot state whether ultrasound skills increased after the workshop and waned over time, like knowledge, or increased slowly over the study despite the workshop's intensive faculty oversight. Future studies and curricula should include robust usage documentation and regular faculty supervision to correlate high-quality practice with learner progression.

Conclusion

The combination of an ultrasound workshop and easy access to ultrasound machines, without an ongoing curriculum, temporarily increased ultrasound knowledge in IM residents but resulted in suboptimal learner progression over time. Programs should consider building ongoing supervision and skills maintenance into their ultrasound curricula.