Does Test‐enhanced Learning Improve Success Rates of Ultrasound‐guided Peripheral Intravenous Insertion? A Randomized Controlled Trial

Abstract

Introduction

Teaching procedural skills in medicine is time‐ and resource‐intensive, and it is therefore important to determine which educational strategies are most effective. Test‐enhanced learning (TEL) has been demonstrated to be effective in improving learner retention; however, there is little research evaluating the testing effect on the acquisition of procedural skills in medicine. The objective of this study was to examine the impact of TEL on learning ultrasound‐guided intravenous (USG IV) insertion on a simulated model.

Methods

We conducted a prospective randomized controlled trial of medical students at a single tertiary care academic hospital. Participants were randomized to either the TEL group (TEG) or control group (CG). Each group received an USG peripheral IV teaching session that included a didactic portion and hands‐on skills training. The training sessions were identical except that the TEG was informed at the outset that there would be an assessment at the end of the session. The TEG then received a formal assessment of the skill during the last 15 minutes of the session, whereas the CG had continued practice time. Subjects in both groups were evaluated 10–14 days later to compare skill performance using a simulation‐based assessment tool consisting of a global rating scale (GRS) and checklist items.

Results

Thirty medical students completed the study, 15 in the TEG and 15 in the CG. There were no significant differences between the two groups at baseline based on year of medical training or prior IV or ultrasound experience. The overall procedural success rate was 93.3% (95% CI = 79.0%–100.0%) in the TEG and 80.0% (95% CI = 57.1%–100%) in the CG (p = 0.60). The first‐attempt failure rate was 13.3% (95% CI = 0.0%–32.8%) in the TEG and 33.3% (95% CI = 6.3%–60.4%) in the CG (p = 0.39). There were no statistically significant differences between the means of the GRS (TEG = 4.7, CG = 4.2; p = 0.53, r = 0.11) or checklist scores (TEG = 78.6%, CG = 75.3%; p = 0.20, r = 0.24).

Conclusions

In this study of novice learners, both TEL with structured practice and structured practice alone lead to high success rates performing USG IV insertion on a simulated model. While we noted a trend toward higher procedural success rates and lower first‐attempt failure rates in the TEG, these did not meet statistical significance. Further studies with larger sample sizes are required to determine whether the beneficial effects of TEL can be transferred to procedural skills teaching.

Establishing peripheral intravenous (IV) access is a critical skill for patient care that can be technically challenging. The traditional landmark technique has a failure rate that ranges from 12% to 26% in adults and 24% to 54% in pediatric patients.1, 2, 3, 4 Failure rates can be even higher in patients with obesity, chronic IV drug users, peripheral edema, and patients who have had long‐term hospitalization.5, 6, 7 Ultrasound‐guided (USG) IV insertion allows for access to veins not directly or easily visible within the dermis.7 It is increasingly being used in situations where traditional techniques such as surface landmarks and palpation are expected to be challenging or have failed. USG peripheral IV cannulation has been demonstrated to be both safe and effective.8 Multiple studies have demonstrated that the use of ultrasound improves the ability for practitioners to establish IV access in patients with suspected or proven difficult access.6, 9, 10, 11

Being a relatively new procedure, there is little evidence to guide best training practices when teaching USG IV insertion to novices. Test‐enhanced learning (TEL) is a teaching technique that may be an effective strategy for teaching USG IV insertion. TEL involves the administration of a formative assessment during the training phase of learning. There are several proposed mechanisms whereby the use of TEL during training may be effective. Knowledge of having a test increases student motivation and attentiveness, and frequent testing allows for spaced repetition known to beneficial for memory.12 Testing also leads to structured instructor feedback, which clarifies learning objectives and improves learner confidence in responses.13 The act of data retrieval during a test has been shown to increase retention beyond the act of just reading and reviewing material, referred to as the direct effect of testing.14 This “testing effect” has been shown to be effective for improving learning of clinical reasoning, resuscitation skills, and forming an approach to management of clinical conditions.15, 16, 17

While there is a growing body of evidence supporting the use of TEL in factual learning, there is a lack of studies evaluating its impact on learning complex motor tasks. Procedural skills require a component of retention of both knowledge and motor skill, both of which may be influenced by the introduction of testing during practice. The objective of this study was to examine the impact of the addition of TEL on teaching medical students the procedure of USG IV insertion on a simulated model.

Methods

Study Design

This study was a prospective randomized controlled trial using an educational intervention to assess the impact of TEL on USG IV insertion. The study was approved by the institutional review board of Sunnybrook Health Sciences Center, and all study participants provided written informed consent.

Study Setting and Population

The study was performed at a single academic tertiary care hospital in Toronto, Ontario, from November 2015 to May 2016. All medical students affiliated with the hospital were eligible for participation. Medical students were recruited by e‐mail notification and the first 32 to reply were enrolled. Participants were randomized using a computer‐generated random number sequence into either the TEL group (TEG) or the control group (CG). Participants received a $25 gift card for participating in the study.

Study Protocol and Educational Intervention

Each participant completed an initial questionnaire assessing their level of education and baseline experience with ultrasound and USG IV insertion (see Data Supplement S2, available as supporting information in the online version of this paper, which is available at https://doi.org/onlinelibrary.wiley.com/doi/10.1002/aet2.10044/full). Participants in both the CG and the TEG received a 30‐minute didactic lecture and demonstration of the technique. The lectures were identical except that the TEG was informed at the outset that there would be a formal assessment at the end of the training session. All the teaching sessions were taught by the same instructor (AS) using a standardized slide set and script to maintain consistency. The teaching material was developed by the local ultrasound fellowship director and based on published descriptions of the technique.7, 18, 19 It was pilot‐tested at several vascular access workshops and modified based on participant feedback. Topics covered during the session included image generation, vein selection, and dynamic needle guidance under ultrasound.

Following the lecture, participants practiced the procedure for 1 hour. The practice session included two stations: a live model and a simulated model. At the live model station, participants practiced using ultrasound to identify an appropriate target arm vein and optimize the image on the screen. At the simulated model station, participants practiced the actual cannulation of a simulated vein using a short‐axis approach with a 25‐mm 20‐gauge IV catheter under direct ultrasound guidance. A commercially available IV simulator was used for practicing the real‐time cannulation (CAE Healthcare). Simulated training has been previously demonstrated in central‐line vascular access to be an effective method for teaching the procedure and is transferable to real‐world performance.20, 21 Sonosite M‐Turbo (FujiFilm SonoSite Inc.) and Zonare Z.one (Mindray Medical Ltd) portable ultrasound machines were used with 10–5MHz linear array transducers. Feedback on image generation, vein identification, needle visualization, and catheter insertion technique was provided to the participants in both groups during the training session.

During the final 15 minutes of training, the TEG completed a formative assessment while the CG continued with more practice. For the formative assessment, the session instructor directly observed the TEG participants completing an entire USG IV landmarking and insertion as described above without any guidance. Upon completion, verbal feedback was provided by the same instructor on the participant's performance for each step of the procedure, highlighting potential areas of improvement. The CG did not undergo a formal assessment and completed 15 more minutes of procedural practice with feedback.

Procedural learning was assessed using a transfer task after a 10‐ to 14‐day washout period, in keeping with previous studies.17, 22 Participants were informed ahead of time that they would be assessed performing an USG IV; however, they were not provided any further opportunity to practice the procedure between the two sessions. The transfer task consisted of performing the complete procedure using a live model and a different simulated tissue model from the one used during the initial teaching sessions. The simulated tissue model for the transfer task consisted of chicken breasts with water balloons inserted to mimic veins. All assessments were videotaped for subsequent blinded review using a structured assessment tool as described below.

Measurements

The primary outcome for this study was the rate of successful IV cannulation, defined as flashback in the catheter and ability to flush the catheter with saline. Secondary outcomes included performance on a simulation‐based assessment tool, and total number of cannulation attempts. Videos of each participant completing the procedure were scored by reviewers blinded to the training group using the assessment tool.

The USG IV assessment tool was developed by modifying previously validated central venous catheter insertion tools.23, 24 Similar to these prior studies, the assessment tool combined an anchored 7‐point global rating scale (GRS) and a binary 17‐point checklist (see Data Supplement S1, available as supporting information in the online version of this paper, which is available at https://doi.org/onlinelibrary.wiley.com/doi/10.1002/aet2.10044/full). A GRS was chosen to accompany the checklist due to their previously demonstrated ability to discriminate expertise.24, 25, 26, 27 We gathered validity evidence for our assessment tool using Messick's framework.28 Content validity was assessed by having the scoring tool reviewed independently by three emergency physicians with significant expertise in USG IV. These experts have been routinely using USG IV insertion in their practice for over 5 years and have taught the procedure at multiple national ultrasound courses. The checklist items and the GRS anchors were modified based on their feedback. Response process validity was assessed by training the reviewers, calibrating the scoring tool, and then having them score a series of five sample recordings to confirm similar scores. Internal structure validity was assessed through testing inter‐rater reliability using a second blinded reviewer who independently one‐third of the videos.

Data Analysis

We calculated a required sample size of 28 participants to detect an effect size of 1.1, with a power of 80% and a significance level of 0.05. (GraphPad statmate 2.0).29 GRS and checklist scores were compared between the two groups using the Wilcoxon rank sum test. The number of attempts at cannulation and success rate of cannulation were compared between the two groups using Fisher's exact test. A multiple linear regression analysis was used to adjust for any differences between participants who had prior ultrasound training or were in the clinical years of their medical education. To measure inter‐rater reliability, intraclass correlation coefficients (ICCs) were calculated for the checklist and GRS using a random sample of one‐third of the video assessments by a second blinded reviewer. We predefined the interpretation of our ICC values as follows: <0.20 slight, 0.21–0.40 fair, 0.41–0.50 moderate, 0.61–0.80 substantial, and 0.81–1.00 almost perfect agreement.30

Results

Thirty‐two medical students agreed to participate in the study, with 16 participants in the TEG and 16 in the CG (Figure 1). Two students (one in each group) did not attend an assessment session and therefore were excluded from the final analysis. There were no significant differences between the two groups based on year of medical training or prior ultrasound or IV experience (Table 1). The majority had received some prior introductory ultrasound training; however, none of the participants had previous exposure to USG peripheral IV insertion.

Figure 1.

Participant enrollment and randomization flow chart. CG = control group; TEG = test‐enhanced group.

Table 1.

Baseline Characteristics of Study Participants

	CG (n = 15)	TEG (n = 15)
Year of medical training, mean (±SD)	2.3 (±0.8)	2.5 (±0.7)
Years 1–2 of medical school, n (%)	8 (53)	7 (47)
Years 3–4 of medical school, n (%)	7 (47)	8 (53)
Prior ultrasound training, n (%)	12 (80)	10 (66)
Baseline comfort with ultrasound, mean score/10 (±SD)	4.2 (±2.1)	3.5 (±1.6)
Prior USG IV training, n (%)	0 (0)	0 (0)
Baseline comfort with IV insertion, mean score/10 (±SD)	2.5 (±1.8)	3.1 (±2.4)

CG = control group; IV = intravenous; TEG = test‐enhanced learning group; USG = ultrasound‐guided.

The results from the simulation‐based assessment are summarized in Table 2. The procedural success rate was 93.3% (95% CI = 79.0%–100.0%) for the TEG and 80.0% (95% CI = 57.1%–100%) for the CG (p = 0.60). The first‐attempt failure rate was 13.3% (95% CI = 0.0%–32.8%) for the TEG and 33.3% (95% CI = 6.3%–60.4%) for the CG (p = 0.39). The total number of needle insertions was 1.1 (95% CI = 0.9–1.3) in the TEG and 1.4 (95% CI = 1.0–1.7) in the CG (p = 0.16). There were no statistically significant differences between the groups on performance measured by the GRS (TEG = 4.7, 95% CI = 4.2–5.2; CG = 4.2, 95% CI = 3.4–5.0; Z = 0.63, p = 0.53, r = 0.11) or the checklist scores (TEG = 78.6%, 95% CI = 73.1%–84.1%; CG = 75.3%, 95% CI = 68.9–81.7%; Z = 1.3, p = 0.20, r = 0.24).

Table 2.

Comparison of Assessment Scores for USG IV Insertion Between Study Groups

	CG (n = 15)	TEG (n = 15)	p‐value
Successful procedure, n (%)	12 (80)	14 (93)	0.60
Failed first attempt, n (%)	5 (33)	2 (13)	0.39
Total number of attempts, mean (±SD)	1.4 (±0.6)	1.1 (±0.3)	0.16
GRS scores, mean (±SD)	4.2 (±1.52)	4.7 (±0.88)	0.53
Checklist score, % (±SD)	75.2 (±11.6)	78.5 (±9.9)	0.19

GRS = global rating scale; IV = intravenous; TEG = test‐enhanced learning group; USG = ultrasound‐guided.

A sensitivity analysis after adjusting for year of clinical training and for previous ultrasound training did not find any difference in the two groups for GRS or checklist scores. The ICC was 0.90 (95% CI = 0.63–0.97) for the GRS and 0.91 (95% CI = 0.67–0.98) for the checklist, indicating a high inter‐rater reliability.

Discussion

Test‐enhanced learning is a training technique that has shown promise in previous studies evaluating knowledge acquisition; however, there is limited evidence on the effectiveness of TEL for procedural skills. In this study, we found that both the CG and the TEG were highly successful in achieving USG IV cannulation on a simulated model. While the TEG had a higher rate of successful cannulation and a lower first‐attempt failure rate, these differences did not meet statistical significance. Performance scores using the procedure checklist and GRS were similar between the two groups.

There are several factors that could have contributed to this result. The first to consider is that while most studies explored the effect of TEL on knowledge acquisition, our study evaluated the performance of a procedural skill. Unlike pure memory and recall, procedural skills require complex motor learning and it is possible that the effects of TEL are not directly translatable to these tasks.

Another possible explanation for the lack of benefit seen for TEL in this study is the complexity of the procedure. Some authors have challenged the idea that TEL has an effect in complex learning environments, demonstrating that the effect may be diminished as complexity increases.31 While it may be true that complexity diminishes its effect, there are studies demonstrating the benefit of TEL in other complex medical environments. TEL has been shown to be superior to routine practice in complex tasks such as making a diagnosis from a radiograph, resuscitation simulation, presenting an initial approach to managing clinical conditions, and clinical reasoning.15, 17, 22, 32 The procedural skill of USG IV insertion requires a high degree of element interactivity, combining recall and application of knowledge of anatomy, ultrasound use, steps of the procedure, and motor skills.33 It may be this complexity that prevented us from detecting a significant benefit from TEL.

It is possible that the method and type of testing that we used in this study was not optimized to enhance learning of procedural skills. The design included a single test and so could not include spaced repetition of testing or increased time spent by students practicing the material in expectation of a test. An experimental study where the test results do not affect academic standing in any way is also unlikely to motivate students in the same way as formalized testing as part of a medical curriculum. Further studies are needed to determine whether modification of the delivery of testing could lead to enhanced learning of technical skills.

The small sample size of this study may have contributed to the inability to detect a small but clinically significant difference between the two groups. While this study was unable to demonstrate an advantage to TEL compared with structured practice, there is enough evidence supporting the theory of TEL in medical education that its potential value in the realm of procedural skills teaching should continue to be explored. The high success rate of both groups demonstrated that the teaching model used was effective in training students in the procedure.

Limitations

This study has a number of limitations. Due to resource constraints, the study was powered based on the detection of a large effect size. While not achieving statistical significance, the TEG outperformed the CG on all components of the simulation‐based assessment. It is possible that with a larger sample size we may have detected a statistically significant difference between the groups. The study population had limited clinical experience with both ultrasound and IV insertion. It is possible that a more clinically experienced group would have higher baseline success rates and therefore less potential benefit from TEL. This study also used simulated models to teach USG peripheral IV insertion and it is possible that use of simulated models contributed to the high success rate and that the skills may not transfer directly to patient care. To avoid contaminating the CG, we did not perform baseline skill testing. While it is possible that the groups were unevenly balanced in technical skill proficiency, they otherwise appeared to have similar baseline characteristics.

Conclusion

In this study of novice learners, both test‐enhanced learning with structured practice and structured practice alone lead to high success rates when performing ultrasound‐guided peripheral intravenous insertion on a simulated model. While the test‐enhanced learning group had higher procedural success and lower first‐attempt failure rates, these findings did not meet statistical significance. Additional studies with larger sample sizes are required to determine if the effects of test‐enhanced learning can be transferred to areas of medical education such as procedural skills teaching.

Supporting information

Data Supplement S1. Ultrasound‐guided peripheral intravenous simulation‐based assessment tool.

Data Supplement S2. Participant baseline questionnaire for comfort with ultrasound and intravenous insertion.

AEM Education and Training 2017;1:310–315.