Lessons Learned from a Study of the Integration of a Point-of-Care Ultrasound Course into the Undergraduate Medical School Curriculum

Abstract

Background

Point-of-care ultrasound (POCUS) diagnosis, performed by a physician at the patient bedside, is routinely used in emergency medicine and critical care. Although training in ultrasonography has become part of the medical school curriculum, POCUS can be challenging for medical students. This study aimed to assess the effectiveness of a one-day POCUS course in a group of final-year medical students by pre-course and post-course assessment of both diagnostic ability and changes in student confidence levels in making a diagnosis.

Material/Methods

A prospective study recruited 57 final-year medical students who participated in a one-day POCUS course. Improvement in making decisions and levels of confidence were evaluated before and two weeks after the course, using image-based testing and a self- evaluation questionnaire.

Results

All 57 final-year medical students attended the course and completed the tests and surveys. The pre-training and post-training test scores of medical students improved from 41.78±12% to 58±13%. Student confidence scores in the post-training assessment significantly increased. The post-training confidence scores were significantly increased compared with the pre-training scores regardless of whether or not the questions were answered correctly (p<0.001). The Dunning-Kruger effect, or cognitive bias, might partially explain this phenomenon.

Conclusions

One day of POCUS training integrated into anesthesia curriculum for final-year medical students improved performance in the post-training test scores and improved their confidence scores. Further studies are needed to evaluate this effect and to develop adequate tools to assess medical students.

Background

Point-of-care ultrasonography (POCUS) [1–7] is performed at the patient bedside in real-time by a physician. Due to more portable and affordable ultrasound machines, POCUS is used routinely [8]. As POCUS has become an integral diagnostic component in critical care and emergency medicine [9,10], training of ultrasonography skills has become part of the medical school curriculum for undergraduate medical students. Although POCUS training is now part of professional development, its role in undergraduate medical education is still controversial [11].

Several studies have compared the efficacy of POCUS training for novice learners [4,12,13]. Several approaches are available for training in POCUS that are used individually or in combination, including lectures, e-learning, simulator training, and the practical sessions [14]. In medical schools, the duration of each course varies between one day to a few days. The effectiveness of the training is usually evaluated with written or practical testing. However, student assessment can also include an evaluation of the degree of confidence in decision-making based on ultrasound images. Although the role of ultrasonography training in medical education is established [15], there remains a need for the evaluation of student ability and confidence following training.

The aims of this study were to assess the effectiveness of a one-day POCUS course in a group of final-year medical students by pre-course and post-course assessment of both diagnostic ability and changes in student confidence levels in making a diagnosis.

Material and Methods

Study design, setting, and population

In this prospective study, a one-day POCUS course was conducted in a simulation center in the Central Teaching Hospital in Warsaw. The study group consisting of 57 final-year medical students were included in the study. Students had no prior experience in ultrasound imagining, and participation in the study was voluntary. The knowledge of the students on POCUS was assessed before and two weeks after the course with image-based testing, as well as with self-evaluation surveys. The instructors were staff anesthesiologists or anesthesiology trainees from the Second Department of Anesthesiology and Intensive Care in the Central Teaching Hospital in Warsaw. Practical sessions were performed on healthy individuals.

Ultrasound equipment

During the practical sessions, the training was conducted using SonoSite X-porte ultrasound unit (SonoSite Inc, Bothell, WA, USA) with a curvilinear transducer (2–5 MHz) phased array transducer probe (1–5 MHz) and linear array transducer (6–15 MHz). Another ultrasound unit was Philips Sparq (Philips Healthcare, Andover, MA, USA) with a curvilinear transducer (2–6 MHz) phased array transducer probe (2–4 MHz) and linear array transducer (4–12 MHz).

Assessment of student confidence in their diagnostic ability

For this study, a confidence scoring system was designed to rate the subjective certainty of a given answer for each student. Students could grade how confident they felt with their answer from 1, the lowest confidence score to 10, the highest confidence score.

Study protocol: pre-course tests and post-course tests

A two-week medical student anesthesia training course in the final year of undergraduate medical school began with a one-day POCUS course. At the beginning of the course, after a short introduction, students were asked to complete a pre-training test, which consisted of nine multiple-choice questions presenting ultrasound cine loops and pictures and simple clinical scenarios. The individual students taking the test were anonymized and each student was de-identified and given a code to use. Students also graded their level of confidence for their answer to each question using a 10-point confidence score.

The one-day POCUS training consisted of four lectures, seven practical skill training sessions on models, and one computer-based ultrasound interpretation session (Table 1). For the practical ultrasound sessions, students were divided into groups consisting of five participants who rotated between training stations. Each station was run by the instructor who supervised the training and provided real-time feedback. The purpose of each training station was to learn about the acquisition of four basic transthoracic echocardiography views (parasternal long-axis and short-axis views, apical four-chamber view, and subcostal view), using the Extended Focused Assessment Sonography for Trauma (EFAST) [16] and Bedside Lung Ultrasound Examination (BLUE) protocols [17,18]. In the last training station, the medical students were taught ultrasound-guided vascular cannulation. During the computer-based session, participants were taught how to interpret life-threatening conditions that could be identified with POCUS.

Table 1.

The one-day point-of-care ultrasound (POCUS) medical student training program, including a description of the training sessions.

Training session	Topic	Duration (min)
Introduction (all participants)	Introduction to the course Pre-training test and self-evaluation survey	20
Lecture I (all participants)	Basics of transthoracic echocardiography	15
Lecture II (all participants)	Application of ultrasonography in trauma using the Extended Focused Assessment Sonography for Trauma (EFAST) protocol	15
Practical session in groups (rotation)	Parasternal view in the long-axis and short-axis (PLAX, PSAX)	30
Practical session in groups (rotation)	Four-chamber apical view (4CH)	30
Practical session in groups (rotation)	Subcostal view	30
Practical session in groups (rotation)	Extended Focused Assessment Sonography for Trauma (EFAST)	30
Lecture III (all participants)	Basics of lungs ultrasonography, using the Bedside Lung Ultrasound Examination (BLUE) protocol	15
Lecture IV (all participants)	Ultrasound-guided vessel cannulation	15
Practical session in groups (rotation)	Lung ultrasonography (BLUE protocol)	30
Practical session in groups (rotation)	Point-of care ultrasound (POCUS)	30
Computer station in groups (rotation)	Virtual station: POCUS, lung ultrasonography, EFAST	30
Practical session in groups (rotation)	Vascular cannulation	30
Summary (all participants)	Summary	20

Anonymized post-course testing, using a de-identified code for each student, was completed by all students at the end of their two-week anesthesia training. The level of student confidence for each answer, using a 10-point confidence score, was assessed again (Table 1).

Statistical analysis

Statistical analysis was performed using Statistical Analysis Software (SAS) version 9.4. McNemar’s test was used to evaluate the results of the pre-training test and post-training test. Wilcoxon rank-sum test was used to assess the difference in confidence scores of individual students in the pre-training test and post-training test. Parametric data were presented as the mean ± standard deviation (SD), whereas nonparametric data were presented as the median (Me) and interquartile range (IQR). Significance was represented as a P-value <0.05.

Results

Pre-course and post-course test results

All 57 final-year medical students recruited to the study attended the one-day training course in point-of-care ultrasound (POCUS) and completed all tests and surveys. There was a statistically significant improvement in test results when compared with the pre-training test results two weeks’ previously, 41.78±12% vs. 58±13% (p<0.001). However, the improvement in test responses was not evenly distributed across all nine questions. There were no statistically significant differences in answers for four questions (1, 2, 5, 8) between the pre-training test and the post-training test (Figure 1).

Figure 1.

Bar graph of the pre-training and post-training test results following the one-day course on point-of-care ultrasound (POCUS).

Student confidence evaluation scores pre-training and post-training

Overall, the confidence scores of given answers in the post-training test were significantly higher when compared with the pre-training test scores. The median and (IQR) for the pre-training test and post-training test were median=6 (range, 4–7) vs. median=7 (range, 6–9) (p<0.001). There was no significant difference in student degree of confidence scores before and after training for one question (p=0.09) (Figure 2). The proportion of correct answers did not correlate with the scores in student confidence (data not shown). Post-training student confidence scores were significantly increased compared with the pre-training scores regardless of whether the question being answered was right or wrong (p<0.001) (Figure 3).

Figure 2.

Box plot of pre-training and post-training test scores in the degree of student confidence for each question following the one-day course on point-of-care ultrasound (POCUS).

Figure 3.

Box plot of the comparison between the pre-training and post-training test scores in the degree of student confidence according to the correct or wrong answer for each question following the one-day course on point-of-care ultrasound (POCUS).

Discussion

Previous studies on ultrasound training in medical education have focused on stand-alone ultrasound courses and assessment shortly after the course. In the present study, point-of-care ultrasound (POCUS) training was only a part of a two-week anesthesia rotation. This approach provides benefits in assessing how the incorporation of POCUS training affects student clinical decision-making in a more real-world situation. Also, the post-training assessment took place two weeks after the initial pre-training assessment, which allowed enough time to pass for students to partially forget their ultrasound training and also introduced an element of surprise because students were not aware when they would be tested. All tests were anonymous without recording any personal data, and the students were made aware of this, which made it more likely that students replied honestly to the questions used in the assessment of their confidence scores.

Although the findings of this study showed a significant increase in the test scores after the course, the size of this effect was only moderate (41.78±12% vs. 58±13%) (Figure 1). This finding may be explained by the fact that testing after the one-day POCUS course was undertaken two weeks later. Testing at two weeks following the one-day training course evaluated long-term memory rather than repetition of newly-acquired information. The other factor that could have affected scoring was that students were not graded based on their test scores. In previous studies, because different methodology was used, the size of the scores from testing students before and after training varies from 57±0.2% vs. 90±0.1% [13], to 66±13% vs. 83±9% [12].

The results of the present study also showed an increase in scores for student confidence in their diagnostic ability, with a median pre-training test score of 6 (range, 4–7) compared with the median post-course test score of 7 (range, 6–9) (p<0.001) (Figure 2). This finding was supported by those of previous studies [13,19]. An unexpected finding was the high pre-training test scores for student confidence that coexisted with the poor initial mean rate of correct answers (Figure 1). Additional analysis was undertaken to try to explain this phenomenon. However, student confidence scores did not correlate with test scores for the given answers, and it was concluded that student confidence scores increased regardless of whether a correct or incorrect answer was given. The Dunning-Kruger effect might partially explain this finding. The Dunning-Kruger effect of cognitive bias was described by social psychologists David Dunning and Justin Kruger (Figure 4) [20–23]. This effect is the inability to acknowledge the lack of individual competence. Based on such assumptions, the lack of experience makes medical students more confident in clinical decision-making. However, lack of awareness gained by clinical experience impairs their ability to assess the level of their individual competence appropriately. The group of students included in this study could be especially prone to this effect because the study group included final-year medical students with limited clinical experience and lack of exposure to professional diagnostic challenges.

Figure 4.

Graph of the Dunning-Kruger effect on the confidence of medical students in their diagnostic ability.

This study had several limitations. The method of enrolment of study participants limited the number of students included in the study. A previous power analysis was not undertaken to determine the optimum size of the study group. Because the POCUS program was conducted within a tightly organized teaching schedule with limited available time, student testing included only nine questions, which limited the sample size and the variety of answers. In the future, a better way to assess the ability of each student to make a diagnosis with POCUS examination would be an ultrasound assessment at the patient’s bedside. Other limiting factors were the availability of instructors and patients with certain types of disease. The recent improvements in patient simulation technology in medical training can provide tools to facilitate more individual assessment of ultrasound and other clinical diagnostic skills in medical education.

Conclusions

A one-day point-of-care ultrasound (POCUS) training course integrated into the anesthesia training curriculum for undergraduate medical students improved performance in post-training test scores and the scores for the student levels of confidence in their diagnostic ability. Confidence scores increased regardless of the accuracy of the answers given on testing. Further studies are needed to evaluate this effect and to develop adequate tools to assess medical students during their training.