A Pilot Study Comparing Cued Versus Recognition Recall Question Design on Medical Student Utilization, Effectiveness, and Perceptions of Pharmacology Educational Games

Abstract

Although games are used for active learning, research is lacking on how design impacts learning, engagement, and utilization in medical students. This pilot study compared pharmacology educational games designed with cued recall questions with a recognition-recall trivia multiple choice format. Learning was determined by comparing quiz performance pre- and post-game, and perceptions were measured by post-game survey. Pharmacology performance improved regardless of game design, although the trivia game produced greater post-game performance improvement. Medical students positively perceived both pharmacology games to be interesting, effective, and engaging tools, supporting that they are a feasible method to promote engagement and active pharmacology learning.

Background

Educational games have been developed for use in healthcare education as engaging tools that reinforce basic science using active learning principles to encourage self-learning, collaborative interactions, and gamification of retrieval-based practice [1, 2]. Limited research on how to design and use games to facilitate medical learning has limited their use as serious educational tools despite potential to promote thinking, problem-solving, remote learning, and interest in dry foundational sciences critical to medical practice [3, 4]. Cognitive literature supports that activities, like games, promoting recall are powerful learning tools, although few have studied how games are designed or implemented in medical education to facilitate central processing mechanisms essential to retention [5–7]. The literature also supports those different types of recall and cueing aspects within assessments impact learning and retrieval-based practice differently [8–10]. However, data is lacking on how different recall designs within games impact learning or engagement.

A recent review of medical game-based learning showed that types of pharmacotherapy games varied, and outcomes evaluated were generally overall satisfaction with the games as opposed to an analysis of game features or learning outcomes [3]. Limited data from recent studies supports increased post-game knowledge for an anti-infectives card game, a drug-dosing game, and a Jeopardy-style psychotherapeutics game [3, 11–13]. Interpretation of two studies was limited by methodological issues given that identical pre- and post-quiz items were used and testing effects of repeating questions may have contributed to results [11, 13]. Further, while many studies show medical learners enjoy games, few have examined specific design components critical to optimizing participation and feasibility as a medical education tool for millennial learners.

Activity

This pilot study was developed to determine whether pharmacology games utilizing different designs and recall questioning demonstrate differential impacts on medical student pharmacology learning, and perception of utility, engagement, and collaborative learning. Both pharmacology games were designed to cover diverse second year (M2) pharmacology topics, and distributed electronically for individual or group play to M2 students who had completed systems and were beginning study for the United States Medical Licensing Exam (USMLE) Step 1. Both games contained immediate feedback, clinical vignettes, humor, and clinical pearls as reminders of relevance. All participants played the games in the same order as game 2 was embedded at the end of game 1. One game was designed using a Jeopardy© template with limited cue recall and open-ended questions. Per this format, no answer options were presented to players, so participants had to answer based on their memory or prior knowledge related to the description. Jeopardy employed a categorical organization. The second game used a “Who Wants to be a Millionaire©” trivia-style template with a variety of topics, a classical clinical vignette stem with one-right-answer-type questions, and 5 answer options in a recognition-recall format. Different pharmacology multiple choice questions (MCQ) (N = 12 each), vetted to be of similar difficulty based on prior p-scores (for similar or identical MCQ items presented on formative quizzes 6 months previously), were selected for the pre-game baseline and post-game quizzes (drugs reinforced in games) to avoid improved performance due to a pre-post testing effect of repeating the same items. The post-game questions were not identical to MCQ items in the Trivia game, the pre-game quiz, or the open-ended questions in Jeopardy game, although the post-game quiz items tagged to the games covered the drugs and content addressed in the games. A post-quiz survey used a Likert scale and queried perceptions on themes of utilization, preferences, and engagement. The post-quiz and survey were embedded at the end of game 2 (trivia) to ensure completion of both games. The study was approved by the University of Central Florida (UCF) human subjects IRB and is a medical student research project funded by the UCF College of Medicine (COM).

Results

Twenty-nine female (66%) and 15 male (34%) M2’s from UCF COM completed the pre-quiz and electronic games. Forty-two participants (95%) completed the post-game quiz and survey. Pharmacology knowledge pre- and post-game is represented in Fig. 1. In comparison with pre-game scores, Student’s independent t tests revealed significantly improved post-game performance on pharmacology MCQs tagged to topics in the Trivia game (p = 0.01) but not the Jeopardy game (p = 0.11). Participants played both games, and there was a significant effect of combination game exposure on post-game performance (p = 0.03).

Fig. 1.

Data represented as mean ± SD for scores on pre-game (gray) and post-game quiz (black). N = 11 items on pre-quiz (1 item dropped from pre-test due to use on recent test) and N = 12 on post-game quiz (N = 6 tagged to Jeopardy, 6 tagged to Trivia). The combined game category reflected the mean score when MCQ items tagged to both games were combined (N = 12 items). *p < 0.05 designated significant difference determined by Student’s independent t test comparing pre- and post-tests

Survey data is summarized in Fig. 2. Most participants agreed that the games were efficient, engaging, and potentially useful for reinforcing M2 systems or learning new topics. Participants were most positive (selecting 5 or 4) in their perception that the games were engaging (Jeopardy, 83%; Trivia, 72%), and useful (> 70%) for reinforcing M2 systems pharmacology, and a majority also perceived utility for learning new topics. Less than 10% selected Likert responses of disagreement, although some selected a 3 (somewhat agree) and suggestions for improvement included time limits of 2 min per question or options to review feedback again at the game end. A median Likert of “4” (mostly agree) was determined for all items queried except usefulness for clerkship where the median was “3” (somewhat agree).

Fig. 2.

Data are represented as % selecting answers for the themes on the x-axis. N = 42 participants completed the survey. Answers to questions on Jeopardy (section A) are on the left with answers to trivia (section B) on the right. Likert anchors: 5 = strongly yes; 4 = mostly yes; 3 = somewhat yes; 2 = mostly no; 1 = strongly no. black bar = total % strongly to somewhat agreeing (sum of % selecting 5, 4, 3); Other bars = % per Likert response: dark black stripes = 5; gray = 4; dots = 3; light gray stripes = 2; diamond = 1

Other preferences are summarized in Table 1. Participants preferred playing games alone and reported clinical pearls to be the most rewarding feature. Most agreed that the Jeopardy made them think more than the Trivia game (data not shown), but more indicated a preference for the Trivia game.

Table 1.

Game preferences. Summary of categorical preference data from the post-game survey. N = 42 participants

Results of 4 non-Likert categorical items
Survey questions summarized	N (%)	N (%)	N (%)
How did you use Jeopardy-style game?	1 time, self 36 (82%)	> 1 time, self 3 (7%)	1 time, group 4 (9%)
How did you use the trivia-style game?	1 time, self 38 (87%)	> 1 time, self 1 (2%)	1 time, group 4 (9%)
In addition to feedback, what feature did you find most rewarding?	Clinical relevance 23 (52%)	Humor, cartoons 10 (23%)	Feedback correct answer reward 8 (18%)
Overall, what game style did you prefer?	Jeopardy 9 (20%)	Trivia 22 (50%)	No preference 13 (30%)

Discussion

In this pilot study, we compared the cued recall Jeopardy with recognition-recall trivia in terms of improving performance, engagement, and usefulness. The Trivia game improved performance more, although the MCQ format in Trivia was similar to that in the post-game quiz and improvement for Trivia-tagged items may have been associated with retrieval-based practice. In fact, this data suggests designing games with recognition-recall may be ideal if improved MCQ performance is desired. The M2 learners preferred the Trivia game, and suggested creating timing limits similar to those for assessments. Thus, they seemed motivated to use gamification to facilitate recognition retrieval-based practice, and this may have been amplified by their proximity to their USMLE assessment. Participants played both games prior to taking the post-quiz and benefits may have derived from both given each covered different drugs but both stimulated thinking about similar pharmacology topics, and contained clinical vignettes and features to promote engagement. Evidence suggests that differing recall cueing methods within games can facilitate different types of cognitive transference and processing, and some studies have shown that free or limited recall questions can also improve performance on related recognition-recall MCQ questions by stimulating thinking and cognitive retrieval processes [8, 9].

Most preclinical learners desired to use both pharmacology games as efficient study tools to reinforce M2 systems topics, although some participants were ambivalent about using games for active learning of new topics. Many were less certain of their usefulness in future clerkship experiences. The games were valued most highly for promoting engagement and interest. Interestingly, they perceived the cued recall Jeopardy game to be more engaging, but most preferred is the recognition-recall trivia game. While we didn’t find a large increase in knowledge solely for Jeopardy-tagged items, other studies have shown that free or limited cue recall promotes more thinking and retention [9, 10]. A limitation of this pilot was a small number of post-game quiz questions, as a larger number might have yielded significant differences. Given the study was optional, our participants (39% of the class) may have self-selected for those preferring games. The majority of volunteers were female, suggesting gender differences may influence preferences for pharmacology games. Our data also suggested that the Jeopardy and trivia formats may not be the best for facilitating collaborative learning, as most played alone.

Our results support literature of improved initial performance after playing pharmacology games, although our interpretation was limited to a retrieval practice benefit because this pilot study looked at an MCQ post-quiz. The post-game quiz most closely resembles the Trivia-style game questions format so this is a limitation in the interpretation of this study. However, we note that Karpicke has shown in several studies that such retrieval practice can also lead to deeper learning [6, 7]. Consistent with the literature, our data show high satisfaction with medical education games [1, 3, 11–14]. Specifically, our data support that gamification of recognition retrieval practice and engagement are the greatest M2 perceived game values. Participants recognized the importance of feedback, and this supports findings that immediate feedback is critical to retention after using retrieval-based formative assessments [15, 16]. Future topics for study include the organizational design of topics as well as the use of alternative style assessments at different times post-game to determine if retention or deeper learning occurred. Our data support improved short-term performance, but literature suggests that active learning methods and retrieval-based practices strongly improve retention [5, 6, 17, 18]. Thus, our data in this pilot study add to the body of literature supporting the use of educational games for pharmacology reinforcement, and provide useful data on question and game design that are likely applicable to other foundational medical disciplines.

Funding information

This research was funded by a University of Central Florida College of Medicine Focused Inquiry & Research Experience (FIRE) medical student research grant.