Generative Retrieval Does Not Improve Long-Term Retention of Regional Anesthesia Ultrasound Anatomy in Unengaged Learners

Abstract

Background:

Ultrasound-guided regional anesthesia is increasingly used in the perioperative period but performance requires a mastery of regional ultrasound anatomy. We aimed to study whether the use of generative retrieval to learn ultrasound anatomy would improve long-term recall.

Methods:

Fourth-year medical students without prior training in ultrasound techniques were randomized into standard practice (SP) and generative retrieval (GR) groups. An initial pre-test consisted of 74 regional anesthesia ultrasound images testing common anatomic structures. During the study/learning session, GR participants were required to verbally identify an unlabeled anatomical structure within 10 seconds of the ultrasound image appearing on the screen. A labeled image of the structure was then shown to the GR participant for 5 seconds. SP participants viewed the same ultrasound images labeled with the correct anatomical structure for 15 seconds. Retention was tested at 1 week and 1 month following the study session. Participants completed a satisfaction survey after each session.

Results:

Forty-five medical students were enrolled with forty included in the analysis. There was no statistically significant difference in baseline scores (GR = 11.5 ± 4.9; SP = 11.2 ± 6.2; P = 0.84). There was no difference in scores at both the 1-week (SP = 54.5 ± 13.3; GR = 53.9 ± 10.5; P = 0.88) and 1-month (SP = 54.0 ± 14.5; GR = 50.7 ± 11.1; P = 0.42) time points. There was no statistically significant difference in learner satisfaction metrics between the groups.

Conclusions:

The use of generative retrieval practice to learn regional anesthesia ultrasound anatomy did not yield significant differences in learning and retention compared with standard learning.

Introduction

Regional anesthesia is being used with increasing frequency in the perioperative period, both as the primary anesthetic for an increasingly sick patient population that may be unable to tolerate general anesthesia, as well as for postoperative pain control as part of a balanced, opioid-sparing anesthetic. The addition of ultrasound guidance has greatly improved the use of regional techniques, increasing both the safety and efficacy of regional nerve blocks.¹ The use of ultrasound to guide regional anesthesia procedures is now considered standard of care, and requirements for training in regional anesthesia are determined by the Anesthesiology Review Committee of the Accreditation Council for Graduate Medical Education.^2,3 Safe and effective use of ultrasound requires extensive knowledge of ultrasound anatomy. This proficiency is often gained “hands on/on the job” during residency training, and may be difficult to achieve while simply observing the procedures. Indeed, even the training of anesthesiology residents in ultrasound-guided procedures is varied, with international societies forming joint committees to guide training programs in the education of resident physicians.^4,5

There is a need for effective, time efficient teaching in postgraduate education when an increasing amount of information is needed to be mastered in an ever decreasing period of time. A study by Kornell et al⁶ demonstrated that when given the opportunity to attempt to retrieve items from memory (even prior to being formally taught the material), learners were more likely to retain information when compared with those who were asked to simply read the given information. Generative retrieval (GR) is a technique in which there is an attempt to retrieve (and produce) an answer from memory, based on cues, as part of the learning process. Previous work from our research group showed that the use of generative retrieval resulted in improved retention of transesophageal echocardiography (TEE) anatomy both 1 week and 1 month following a study session in a group of 4th-year medical students, PGY1 and PGY2 anesthesiology residents when compared with standard practice technique.⁷ A follow-up study using transthoracic echocardiography (TTE) anatomy in second year medical students confirmed these findings with improved recall of TTE anatomy using generative retrieval compared with standard practice 1 week, 1 month and 6–9 months after the study session, though only the latter time point reached statistical significance.⁸ Survey results showed that participants in the generative retrieval group found the activity more effective, enjoyable, and satisfying than did participants in the standard practice group.⁸ These positive emotions associated with generative retrieval makes the technique particularly appealing for self-study as it may result in more of a desire to spend time learning.

Generative retrieval has the potential to be relevant to learning ultrasound anatomy during preclinical training. In this study, we sought to further evaluate the impact of generative retrieval by applying it to learning regional anesthesia ultrasound anatomy in senior medical students with no prior training in regional anesthesia. We hypothesized that the use of generative retrieval would result in better retention of regional ultrasound anatomy defined as a higher score on the 1-month posttest and higher learner satisfaction compared with standard practice.

Materials and Methods

Following Institutional Review Board approval (University of Virginia Health System, Charlottesville, VA, USA) and written informed consent, 45 fourth-year medical students at the University of Virginia without prior training in regional anesthesia ultrasound were enrolled to participate in the study. Participants were randomized to GR or Standard Practice (SP) groups using a computer-generated randomization scheme. Figure 1 shows a CONSORT diagram illustrating enrollment, randomization, and data analysis for the study. Following enrollment in the study, participants were asked not to review any regional anesthesia anatomy for the 1-month duration of the study period. All ultrasound images were obtained from healthy volunteers and de-identified. Images were captured from 7 anatomical views. A total of 37 structures were marked by a white arrow for identification. Table 1 shows the views and structures tested. Figure 2 shows an example of each of the 7 regional ultrasound views presented to study participants.

Figure 1.

Consort diagram illustrating organization of the study, including enrollment, randomization, and analysis of participants.

Table 1.

Seven Anatomical Views and 37 Structures Tested

View	Structure
Interscalene	Anterior scalene muscle
	Middle scalene muscle
	Sternocleidomastoid muscle
	Brachial plexus roots
Supraclavicular	First rib
	Subclavian artery
	Brachial plexus divisions
	Lung
	Pleura
	Omohyoid muscle
Infraclavicular	Axillary artery
	Axillary vein
	Medial cord of brachial plexus
	Lateral cord of brachial plexus
	Posterior cord of brachial plexus
	Pectoralis major muscle
	Pectoralis minor muscle
Femoral	Femoral artery
	Femoral vein
	Femoral nerve
	Fascia iliaca
	Iliopsoas muscle
Adductor canal	Superficial femoral artery
	Superficial femoral vein
	Sartorius muscle
	Adductor longus muscle
	Vastus medialis muscle
	Saphenous nerve
Sciatic at popliteal fossa	Common peroneal nerve
	Tibial nerve
	Biceps femoris muscle
	Semimembranosus muscle
Transversus abdominus plane	Subcutaneous tissue
	Internal oblique muscle
	External oblique muscle
	Transversus abdominus muscle
	Peritoneum

Figure 2.

Images of the 7 regional ultrasound views tested.

We employed a procedure similar to that which we have used in our previous studies.^7,8 All study procedures were conducted in a quiet library environment under direct supervision by a trained proctor. At the beginning of the study, each participant completed a 74-question pretest consisting of 2 sets of the 37 study images. A score of greater than 70% was defined as the cutoff for exclusion from the study. Following the pretest, participants underwent a study session using either the Generative Retrieval or Standard Practice method according to the randomization scheme. The study session consisted of 5 sessions of 37 images for a total of 185 questions (each structure reviewed 5 times). Subjects in the GR group were shown an unlabeled ultrasound image for 10 seconds and asked to verbally identify the structure indicated by the white arrow. The study proctor was continuously present to confirm adherence to the protocol and ensure subjects in the GR group attempted to verbally identify each structure. They were then shown a labeled version of the image for 5 seconds. Subjects in the SP group were shown the same ultrasound image with a labeled anatomic structure for a total of 15 seconds and were not asked to verbally name the structure. The total time to complete the study session was 47 minutes. Both short-term (1 week) and long-term (1 month) retention of regional anesthesia ultrasound anatomy were tested for all participants using a 74-question posttest, which was similar to the baseline test. The images differed between 1-week and 1-month posttests. To assess learner satisfaction, a 3-question survey was administered to all participants following the study session and both post-tests: “How effective a tool was this to learn regional ultrasound?” “How enjoyable was this tool for learning regional ultrasound?” and “How satisfied are you with this tool for learning regional ultrasound?” Participants were provided with a $10.tift certificate after both the initial session and 1-week posttest. They were given a $50.tift certificate after the final session. Total compensation for completion of the study was $70.

Statistical Analysis

The exam and survey scores from baseline, 1-week, and 1-month were considered as the longitudinal responses in the study. Following a Shapiro-Wilk test to assess for normality, the data was analyzed using repeated measures ANOVA. Sphericity was assessed via Mauchly's Test of Sphericity. Comparisons were made using two-tailed tests and P < .05 was considered significant. Statistical analyses were performed with SPSS Statistics Version 24.0 (IBM Corporation, Armonk, New York).

A power analysis revealed that enrolling 20 participants per group would give > 95% chance of detecting an absolute difference of 10% in examination score between the groups at the 1-month time point, assuming an alpha of 0.05 and a standard deviation of 15%.

Results

A total of 45 participants were enrolled and completed the pretest and study session. Five participants were excluded from data analysis. One became unavailable to complete study procedures within the defined time period due to travel. Four were excluded for failure to follow exact study session protocols. No participants scored above the 70% threshold on the pretest to be excluded from the study. Descriptive data of the 40 participants included in the analysis are presented in Table 2. Test result data is summarized in Table 3. Figure 3 shows average test scores for each group throughout the course of the study period. All scores are based on a possible 74 questions correct. Baseline pretest scores were similar for both groups (GR [11.5 ± 4.9]; SP [11.2 ± 6.2]; P = 0.84). There was no statistically significant difference in exam scores at both the 1-week (GR [53.9 ± 10.5]; SP [54.5 ± 13.3]; P = 0.88) and 1-month (GR [50.7 ± 11.1]; SP [54.0 ± 14.5] P = 0.42) time points.

Table 2.

Descriptive Data From All Study Participants

	Male (n)	Female (n)	Total (n)
Generative retrieval	10	10	20
Standard practice	12	8	20

Table 3.

Test Result Data

	Pretest	1 Week	1 Month
SP	11.2 [4.9]	54.5 [13.3]	54.0 [14.5]
GR	11.6 [6.20]	53.9 [10.5]	50.7 [11.1]
P value	0.84	0.88	0.42
Observed Power	0.054	0.053	0.124

SP = standard practice; GR = generative retrieval

Results of the 3-question survey given to participants after all 3 sessions are summarized in Table 4. The participants were asked to rate the method on a scale of 1 to 10. There was no statistically significant difference in learner satisfaction metrics between the groups.

Table 4.

Survey Responses

	Question 1: Effectiveness			Question 2: Enjoyment			Question 3: Satisfaction
	Study	1 Week	1 Month	Study	1 Week	1 Month	Study	1 Week	1 Month
SP	7.78	6.85	6.75	5.50	6.15	5.70	7.00	6.90	6.60
GR	8.10	7.70	7.20	7.00	7.20	6.75	7.53	7.48	7.05
P value	.57	.18	0.54	0.07	0.16	0.21	0.45	0.37	0.56

Results of 3-question survey administered to all participants following each session. Subjects were asked to rate their responses on a 10-point scale.

Discussion

Contrary to our hypothesis and findings from previous studies, generative retrieval did not result in improved recall of regional ultrasound anatomy compared with standard practice at either the short-term (1 week) or long-term (1 month) time points tested. There were no significant differences in responses to the learning satisfaction survey between the 2 groups, although generative retrieval participants tended to rate their experience more favorably than standard practice participants. These results are in contrast to prior studies evaluating the effectiveness of generative retrieval in learning TEE (improved retention at 1 week and 1 month) and TTE (trend to improved retention at 1 week and 1 month, statistically significant improvement in retention at 6 to 9 months with greater learning satisfaction).^7,8

Although negative, this study provides us with valuable insights into learning and the critical importance of learner engagement and motivation. Interest and engagement in the study material is of great importance to learning, influencing long-term commitment to learning, retention, and achievement.^9,10 We speculate that the learning material and anatomy used were complex, beyond that typically currently expected of 4th-year medical students. In this way, the participants had a low expectation of success. An association between a low expectation of success and low motivation is explained by the Expectancy Valence Theory, which posits that the motivation behind a given choice of behavior is guided by the favorability of the expected outcome.¹¹ The theory involves distinct elements including expectancy and valence. Expectancy involves the belief that effort will result in attainment of a desired performance while valence refers to the value placed on the rewards of an outcome. A desirable outcome will elicit a high degree of motivation. Conversely, as seen in this study, an unfavorable outcome (low expectation of success paired with a high amount of effort and complexity) will be approached with a low degree of motivation.

It is also possible that this complexity created an overwhelming cognitive load that rendered the type of learning (GR vs SP) irrelevant. In this study, regional anesthesia ultrasound anatomy was presented and tested in a static fashion (ie, still 2-dimensional ultrasound images that represent 3-dimensional anatomy structures). This is in contrast to the previous studies, which used dynamic TTE and TEE clips. It may be that learning the anatomy of the heart using echocardiography videos is conceptually easier than viewing still images of peripheral nerve locations.

The choice of subjects (4th-year medical students instead of anesthesiology residents) likely influenced study results. Senior medical students were chosen for several reasons in an attempt to decrease study heterogenicity. First, all 4th-year medical students would have completed the same gross and applied anatomy course work, unlike anesthesiology residents who come from a variety of different medical schools with different curricula. At our institution, anesthesiology residents gain exposure to regional anesthesia anatomy early in their training, as early as PGY1 or CA1 years. Potential confounders in knowledge base were felt too likely to adequately control for in anesthesia residents. Despite representing a more homogeneous knowledge base, the use of 4th-year medical students introduced several limitations. Many of the medical student subjects were not interested in anesthesiology as a career and we speculate that their participation was more motivated by financial remuneration ($70) than an honest interest in learning ultrasound regional anatomy. “Genuine participation” as opposed to participation guided by a desire for secondary gain or to avoid punishment helps to intensify motivation as well as the rate of learning which may have played a role in this study.¹² A key tenant of adult learning theory, as opposed to childhood learning, is that adult learners require an applicability of the knowledge acquired in order to have motivation to learn and retain knowledge, no matter the acquisition technique.¹³ McGrath believes adult learners must know how the information or skill will benefit them otherwise they will not be ready to learn as well as internalize the subject matter leading to poor retention.¹⁴ The learning must not only be stimulating but also relevant to the learner's own desired learning outcomes and objectives.^13,15

As a pioneer in adult learning, Malcolm Knowles suggests in The Adult Learner: A Neglected Species, “The best time to learn anything is when whatever to be learned is immediately useful to us.”¹² Because the material being studied by the participants in the study was not immediately, or perhaps at all, useful to them as learners, this likely affected significantly the effort of the participants and explains why generative retrieval was not beneficial in this study.

Likewise, the time demands of 4th-year medical students in the winter and spring, when this study was conducted, likely contributed to a lack of engagement. While adult learners are more self-motivated than children, they often have competing demands on their time. To maximize the learning effect, educators must design a task that is engaging enough to seize the learner's enthusiasm but is efficient, applicable, and does not waste energy.¹⁴ We further hypothesize that participants became overwhelmed with the complexity and volume of the subject matter along with the lack of immediate applicability, resulting in a reduced effort during the study and testing sessions, negating the potential benefit of generative retrieval in the encoding of knowledge. To put it simply, the best technique in the world is not going to help students learn if they are not motivated to do so.

The lack of statistically significant differences in the survey data compared with those seen previously also suggest a lack of impact of the study method on learners. A theory used to explain the effectiveness of generative retrieval is the idea of “desirable difficulties” in learning.¹⁶ This concept suggests that long-term retention is increased when cognitively difficult environments are incorporated into the acquisition phase of learning in engaged participants. Un-engaged learners learning highly complex material that is not relevant to their current study may not benefit from the effect of “desirable difficulties” resulting in less long-term retention.

Despite the results of the current study, generative retrieval has been shown to be effective in learning both anatomy and radiology techniques including ultrasound, suggesting a role for the technique in more engaged learners.^7,8 Several studies by Dobson have shown improved recall of anatomy and physiology concepts when self-testing was used as a study tool.^17–22 Likewise, a study of predoctoral dental students showed improved identification of confusable radiographic abnormalities with generative retrieval.^22,23 Our study method employed a completely automated, digital platform that could be easily distributed online to learners. Unlike other teaching methods, the presence of a knowledgeable instructor was not required making it not only economical but also ideal for use in self-study.

The study had several limitations. Unlike our previous study using TTE, we did not test subjects at the 6–9 month time point; thus, it is unknown how the same subjects would have performed at this later and likely more important time point. Since both groups were tested during the study, both groups participated in a form of retrieval practice; however, as both groups were treated similarly, this should have minimal effect on the results, as we previously demonstrated. While we asked participants not to study any regional anatomy information, it is impossible to ensure that no participants studied the material. Lastly, the survey questions were not validated prior to administration due to the complexity of questions related to the impact of educational tools.

The use of generative retrieval for learning regional anesthesia ultrasound anatomy did not result in better short- or long-term recall compared with standard practice in a study population of 4th-year medical students who were not specifically interested in a career in anesthesiology. Likewise, there was no difference in learner satisfaction between the two groups, suggesting that poor engagement may have been responsible for lack of statistical difference seen with generative retrieval.