Information overdose: Student performance and perceptions of pharmacology resources on exams

Abstract

Physicians of the future will be expected to synthesize new knowledge and appropriately apply it in patient care. Here, we report on the effects of and student attitudes towards resource‐enhanced exams by comparing student performance on closed‐book exams with or without access to pharmacology flashcards. Setting: the University of California, San Francisco (UCSF) School of Medicine (SOM), class of 2021 (N = 149), followed over 4 years. We provided pharmacology flashcards for studying purposes in all blocks; flashcards were only accessible during closed‐book exams in 2 of 5 blocks. We collected pharmacology open‐ended question (OEQ) scores and analyzed results using repeated measures ANOVA (SPSS). We collected MS4 survey data using Qualtrics and conducted a thematic content analysis. Performance on pharmacology questions on exams was not higher with access to pharmacology flashcards during exams. The number of students who passed pharmacology questions without flashcards on exams was as follows: 137 ± 3.7, 132 ± 5.0, and 134 ± 7.9 (average ± SEM). The number of students who passed pharmacology questions with flashcards on exams was as follows: 132 ± 6.6 and 120 ± 7.5. Survey comments revealed several themes. Access to pharmacology flashcards during exams allowed learners to focus on understanding the bigger picture and reduced stress. A subset of students reported having access to flashcards on pre‐clerkship exams hurt their preparation for clerkships. Flashcards as exam resources were received well by approximately half the class, who reported benefits including more time to focus on understanding bigger picture concepts and reduced stress.

Overview of curriculum and timeline of data collection. Data collection started in F1 in 2017 and the study cohort completed the final survey in 2021.

Abbreviations

ABC

Airways, Blood, Circulation block

BMB

Brain, Movement, Behavior block

CNS

Central nervous system

F1

Foundations 1 curriculum

GI

Gastrointestinal

GS

Ground School block

IRB

Institutional Review Board

MCQ

Multiple‐choice question

OBE

Open‐book exam

OEQ

Open‐ended question

PHD

Pathogens, Host Defense block

REGN

Renal, Endocrine, Gastrointestinal, Nutrition block

SOM

School of Medicine

SPSS

Statistical Product and Service Solutions software

UCSF

University of California, San Francisco

USMLE

United States Medical Licensing Examination

1. BACKGROUND

The physician of the future will be expected to rapidly synthesize new knowledge, often from electronic resources, and appropriately apply it in a patient care setting. ¹ , ² This requires deep learning via the development of core knowledge structures ³ , ⁴ , ⁵ and the ability to actively construct and expand conceptual frameworks. ⁶ There are several evidence‐based strategies that can contribute to the development of these skills, ⁷ , ⁸ , ⁹ , ¹⁰ , ¹¹ and at the University of California, San Francisco (UCSF) School of Medicine (SOM), we use elaboration during active learning sessions to provide students with the opportunity to apply pharmacology content in each block. ¹²

Assessment is an important part of any curriculum. ¹³ Our institutional assessment strategy consists of weekly formative quizzes (including open‐ended questions (OEQs, Appendix S1) and multiple‐choice questions (MCQs) to facilitate retrieval practice and distributed learning) and closed‐book summative examinations that are fully OEQ and application based, requiring elaboration. ¹⁴ , ¹⁵ However, learning and memorizing the sheer volume of medical knowledge, especially when it comes to pharmacology content, is impractical especially as the field continues to evolve. This requires reframing the approach to assessment. It has been suggested that open‐resource assessments provide learners with the opportunity to demonstrate application and elaboration ¹⁶ , ¹⁷ without the burden of memorizing a large body of knowledge in a limited amount of time. The constantly evolving body of pharmacological agents supports the approach of focusing on enduring concepts of pathophysiology while allowing students to look up more detailed drug information and apply it correctly to clinical scenarios.

Open‐book exams (OBE) are assessments in which textbooks, notes, and/or all other reference materials are allowed. ¹⁸ Reported benefits of OBEs are reduced anxiety, decreased emphasis on memorization, ¹⁹ more profound engagement with the content, ²⁰ and use of the material at a higher Bloom's level. ²¹ OBEs may serve a catalytic function by enhancing student understanding in a more realistic context ²² as it more closely mimics real life where providers can refer to resources to make clinical decisions. ²³ However, despite evidence showing that the OBE format tests knowledge application whereby students use their own understanding to guide searches and apply knowledge to clinical scenarios, ²⁴ students may need a more solid framework regarding the foundational sciences to facilitate this independent searching. ²⁵ Therefore, we postulate that complete OBEs may not be ideal for pre‐clerkship medical education.

This led us to investigate if a hybrid model of resource‐enhanced assessments, as described for biochemistry ²⁶ and for pharmacology, ²⁷ could prepare early learners for clinical practice without the need to perform fully independent searches during an exam, as with OBEs. To do so, we provided medical students with in‐house, course‐specific pharmacology flashcards containing the drug name, mechanism of action, and side effects. These pharmacology flashcards were provided during all in‐class pharmacology activities throughout years 1 and 2 and were accessible during summative OEQ assessments in select blocks. Thus, while at UCSF summative exams are mostly closed‐book OEQ exams, we have created exceptions for certain blocks in which the pharmacology content was open book in the form of pharmacology flashcards. This repeated‐measures design allowed us to compare the same cohort's performance on exams without flashcards on autonomic, musculoskeletal, cardiovascular, and central nervous system (CNS) pharmacology, versus exams with flashcards on endocrine, gastrointestinal, antimicrobial, and anti‐inflammatory pharmacology. Here, we report on the effect of and student attitudes toward resource‐enhanced exams by comparing student performance on closed‐book exams with or without access to pharmacology flashcards in the first 2 years and by surveying the same cohort during their 4th year of the medical school curriculum.

2. METHODS

2.1. Setting

The curriculum at the University of California, San Francisco (UCSF) School of Medicine (SOM) is a 4‐year, integrated curriculum in which the first 18 months, called foundations 1 (F1), are devoted primarily to the foundational sciences and health systems sciences (in dedicated Health & Society and Health & Individual blocks), and inquiry and clinical skills are threaded throughout. Students take the United States Medical Licensing Examination (USMLE) Step 1 exams after clerkships in the 3rd year. ¹⁴ F1 is divided into the following organ systems‐based blocks, in chronological order: (1) Ground School (GS); (2) Airways, Blood, Circulation (ABC); (3) Renal, Endocrine, Gastrointestinal, Nutrition (REGN); (4) Pathogens, Host Defense (PHD); (5) Life Stages; and (6) Brain, Movement, Behavior (BMB). Throughout these F1 blocks, students learn via multiple educational settings including live lectures, online videos, and online PowerPoints. Learners actively apply knowledge in small groups and in large group case‐based wrap‐up sessions (Figure 1), and in formative open‐ended questions (OEQs) on weekly checkpoints (Appendix S1), linked to specific session objectives. Block summative exams are closed book and consist of OEQs that are based on clinical scenarios and emphasize application of knowledge. ¹⁵ Mid‐term and end‐of‐block summative exams occur roughly every 4 weeks and consist of 16 OEQs per exam. The number of summative exams per block varies by length of block. OEQ questions (formative and summative) are written by faculty who deliver the content. OEQs are reviewed and improved in an iterative process during block team meetings by discipline leads and clinicians. Block exams are then reviewed by a team of OEQ experts including the Dean of Assessment prior to their use as summative exams.

FIGURE 1.

Overview of curriculum and timeline of data collection. Data collection started in F1 in 2017 and the study cohort completed the final survey in 2021. The pre‐clerkship component, Foundations 1 (F1), was 18 months in duration and consisted of 10 Foundational Sciences (FS) Blocks, Clinical Microsystems Clerkship, Inquiry Curriculum, and Assessment Reflection Coaching & Health weeks. Each F1 block included foundational sciences (i.e., anatomy, physiology biochemistry, pharmacology, genetics, pathology, immunology, microbiology, epidemiology & biostatistics, histology, etc.). In F1, pharmacology was delivered via multiple educational settings as shown per block (GS: Ground School, ABC: Airways, Blood, Circulation, BMB: Brain, Movement, Behavior, REGN: Renal, Endocrine, Gastrointestinal, Nutrition, PHD: Pathogens, Host Defense). Pharmacology content was taught by the same instructors in all blocks except in PHD, where infectious disease clinicians taught antimicrobials. The number of summative exams varied by length of block (GS: 1, ABC: 3, BMB: 2, REGN: 2, PHD: 2). Pharmacology flashcards were provided as study materials in each block but were only available on summative exams in REGN and PHD.

Authors created in‐house pharmacology flashcards and summary tables based on textbooks, USMLE board review materials, and UCSF curricular guidance. These pharmacology flashcards are provided during all pharmacology sessions and include the mechanism of action, adverse effects of prototype drugs, and names of similar drugs in the class (Figure 2). The total number of drugs provided in flashcards and/or summary tables per block are GS: 6 (cholinomimetics & cholinolytics); ABC: 54 (cardiovascular pharmacology); REGN: 55 (renal, endocrine, & GI pharmacology); PHD: 62 (anti‐inflammatory, anti‐cancer pharmacology, & antimicrobials); and BMB: 46 (CNS pharmacology). Students are encouraged to use the cards as study tools with each pharmacology session. No additional instructions on whether or how to use the cards are given.

FIGURE 2.

Sample pharmacology flashcards from Ground School (GS). Pharmacology flashcards include the mechanism of action, adverse effects of prototype drugs and names of similar drugs in the class. Analogous flash cards were used across all F1 blocks.

In this study of a single cohort of medical students, we compare pharmacology exam performance on F1 closed‐book summative block exams without (GS, ABC, and BMB) and with (REGN and PHD) access to pharmacology flashcards during the exam.

2.2. Data collection and analysis

This study was approved by the UCSF Institutional Review Board (IRB #: 19‐27346). Data were collected from students in the UCSF SOM class of 2021 (N = 149). Deidentified pharmacology summative OEQ answers were evaluated by trained faculty and assigned a score from 1 to 6. To study the impact of access to flashcards during the exam, the number of students scoring a passing score of >4 was averaged per block and compared using a Chi‐squared test with Yates correction. Deidentified scores were averaged and compared using repeated measures ANOVA to study the effect of different educational settings (i.e., large group vs. small group) on performance. After students completed their clerkships and USMLE Step 1 examination, we collected follow‐up data using an author‐created, 9‐item Qualtrics (Provo, UT) survey. The survey was based on surveys performed by Spicer et al. ²⁶ who investigated the effect of biochemistry resources on exams. The survey was not reverse coded.

The survey consisted of Likert‐like questions and free‐response prompts with no character limitations. All students were invited to participate in the study by e‐mail, and 44 students responded (30%) although not every student completed every question in the survey. Students participated in the survey with informed consent and without compensation. Following data collection, the two investigators (MK & RLT) conducted a thematic content analysis on the student comments. The investigators independently reviewed 10–15 responses to identify initial codes and then met to achieve consensus on an initial codebook of common themes. Following independent analysis of all student responses using the codebook, the investigators met to discuss coding, reconcile differences, and discuss new or redundant codes. Frequency of codes was counted to determine dominant themes.

3. RESULTS

3.1. Effect of flashcard access during exam on students' pharmacology OEQ performance

Students were provided flashcards with each pharmacology session and were encouraged to use them as study tools. There were no additional instructions on whether or how to use them in educational settings or during formative assessments. Thus, students were able to utilize these resources as study tools in the GS, ABC, and BMB blocks, where exams were fully closed‐book OEQ exams, and were able to use them as both a study tool and an exam resource in the REGN and PHD blocks. Regardless of whether the pharmacology component of the exam was closed book or resource enhanced, students were reminded regularly in class to familiarize themselves with the flashcards. To study the impact of access to flashcards during the exam, we compared the number of students achieving a passing score (>4) on pharmacology OEQs per F1 block. The number of students who scored >4 on pharmacology questions on exams was not higher when students had access to pharmacology flashcards on exam (see Table 1). The number of students who passed pharmacology questions without flashcards on exams in three courses was as follows: 137 ± 3.7, 132 ± 5.0, and 134 ± 7.9 (average ± SEM). The number of students who passed pharmacology questions with flashcards on exams was as follows: 132 ± 6.6 and 120 ± 7.5 (average ± SEM). The number of students who had a pharmacology score of >4 was significantly lower in PHD (with flashcards on exam) compared to GS (no flashcards on exam) (p = .008, Chi‐Squared test with Yates correction). However, exam performance overall in PHD was lower, suggesting PHD was a more difficult block (Table 1).

TABLE 1.

Student performance on summative pharmacology OEQs with and without pharmacology flashcards. The number of pharmacology OEQs in each F1 block exam was proportional to the total amount of pharmacology in each block.

F1 Block a	% Students who passed block exams (scored >70%) (N = 149 students)	Average number of students who passed (scored >4) on pharmacology OEQs (N = 149 students)	Average pharmacology OEQ answer score on a scale of 1–6 (class average per question, N = 149 students)	Flashcards during exam?	Educational methods designed for pharmacology content delivery in block	Educational methods designed for pharmacology knowledge application in block
GS	99%	137 ± 3.7	5.25; 5.49; 5.71; 5.79	No	Live Lectures Online video	Checkpoints Wrap up (large group) Small group
ABC	99%	132 ± 5.0	4.94; 4.96; 5.43; 5.51	No	Live lecture Online video Online ppt	Checkpoints Small group
BMB	99%	134 ± 7.9	4.97; 5.17; 5.79; 5.86	No	Online videos	Checkpoints Wrap up (large group)
REGN	95%	132 ± 6.6	4.97; 4.99; 5.25	Yes	Live lecture Online video	Checkpoints Wrap up (large group) Small group
PHD	97%	120 ± 7.5 b	4.55; 4.57; 4.68; 5.03; 5.06; 5.12; 5.13; 5.17; 5.59	Yes	Live lecture Online video	Checkpoints Wrap up (large group) Small group

^a Ground School (GS); Airways, Blood, Circulation (ABC); Renal, Endocrine, Gastrointestinal, Nutrition (REGN); Pathogens, Host Defense (PHD); and Brain, Movement, Behavior (BMB).

^b GS vs. PHD: p = .008, all other values in this column were not significantly different from each other, Chi‐Squared test with Yates correction.

3.2. Does application in large group versus small group affect pharmacology exam performance?

To rule out that the differences in exam performance were caused by differences in opportunities to apply knowledge in that block, we compared the educational settings across blocks. Each block incorporated educational methods designed for pharmacology knowledge application, including small group problem‐based learning sessions, large group case‐based wrap‐up sessions, and weekly checkpoints that students completed independently (Table 1). Each session had specific session objectives which were directly linked to the formative checkpoints and summative OEQ exam.

Table 1 shows that there is no direct relationship between application format and pharmacology exam performance. Table 1 also shows that OEQ answer quality was high in ABC (no flashcards on closed‐book exam), which did not have large group wrap‐up sessions, and high in BMB (no flashcards on closed‐book exam), which did not have pharmacology small groups, suggesting no impact of application format. Average pharmacology OEQ answer quality was high across all blocks (>5 on a scale of 1–6), even though they were statistically significantly different from each other due to larger sample size (GS: p = .000, ABC: p = .000, BMB: p = .000, REGN: p = .000, PHD: p = .000; repeated measures ANOVA, see Appendix S2).

3.3. Post‐clerkship survey

Forty‐four students responded (30%) to the post‐clerkship survey although not all respondents answered each question. Thirty‐three students responded to the question with a yes or no answer: “Would you have preferred to have had flashcards on all exams in all blocks in F1?” Most respondents (58%) would have preferred not to have access to flashcards on exams, while the remaining 42% chose that they would. Table 2 shows student preferences for pharmacology exam resources. Eleven respondents preferred the provided flashcards with generic drug names, mechanism of action, adverse effects, and similar drugs in the class on the card. Eleven respondents preferred cards with only the generic drug name but no further information about the drugs on the card. The statement respondents agreed most with was that having flashcards available changed the way they studied for OEQ exams (Figure 3).

TABLE 2.

Which of the following would have been the most optimal for your learning of pharmacology in F1? (Consider what helped in your preparation for clerkships/Step1).

	Number of responses a
Flashcards on exam Full flashcards with mechanism of action, similar drugs, drug–drug interactions, and adverse effects	11
Only drug names on exam Flashcard with only the drug name and no additional details to help relieve anxiety around spelling	11
Nothing on exam No flashcards provided on the exam	10
Other pharm resources on exam	0

^a 32 of the 44 respondents completed this question.

FIGURE 3.

MS4 response to survey questions shown. Student responses regarding the impact of pharmacology flashcards on their performance in F1 blocks, USMLE Step 1 exam and in clinical practice. Quantitative responses on a scale of 1–5 (strongly disagree [1] to strongly agree [5]), error bars represent SEM. Note that 3 is neither agree nor disagree.

Analysis of the comments revealed several themes (Table 3). While most students reported that pharmacology flashcards helped them pass OEQ exams and allowed them to focus on understanding the bigger picture, responses were divided regarding the usefulness of pharmacology flashcards in mimicking the real world.

TABLE 3.

Qualitative analysis of survey comments.

Survey Question 1: “Having flashcards available during the F1 OEQ exams changed the way I prepared for the UCSF OEQ exams”

Dominant themes (n = number of responses) Spent less studying/memorizing pharmacology (n = 21) Allowed focus big picture concepts (n = 7)

Sample quotes: “I spent more time on concepts and integrating the material rather than spending so much time memorizing individual drugs, which would have taken me a LONG time!” “did not focus on memorizing the info on flashcards but instead focused more on big picture concepts” “I focused less on memorizing. Though, I think while Step 1 and ‘the real world’ still continues to emphasize memorization (e.g., because attendings still would ask those questions), this may not ‘beneficial’ in the long run to exempt students from having to memorize during F1 OEQ exams.” “I still tried my best to learn the drugs, but the pressure was off to memorize every detail. Unfortunately, students need to learn those details anyway for 3rd‐year shelf exams and STEP's beyond that. It made the OEQ exams even easier, and did us no favors in the long‐term.”

Survey Question 2: “Having flashcards helped me pass UCSF OEQ exams”

Dominant themes (n = number of responses) Yes, they helped me pass (n = 7) Yes, they helped due to reduced memorization load (n = 7) They were helpful but would have pass anyway (needed to study differently) (n = 12)

Sample quotes: “I think the pharm flashcards helped me get full credit on some UCSF open‐ended questions when I may have gotten partial credit on that question without them. I feel like the pharm flashcards were a helpful supplement/reference on the exams. They helped me build on the information that I had studied on my own. I do not think the pharm flash cards had enough information to help me pass a UCSF OEQ exam without drawing upon the other information that I had studied.”

Survey Question 3: “Having flashcards mimics the clinic (real world) where I can look things up”

Dominant themes (n = number of responses) No, it does not—you needed working knowledge (n = 22) Yes, it does—you can look things up/focus on big picture (n = 12) No, it does not— the real world is high paced/stress and you cannot look things up (no time) (n = 7)

Sample quotes: “I am constantly looking things up in clinical situations, and being able to understand the underlying concepts has proved more useful than memorizing a specific drug.” “I think this is generally true! I do think that as I became more confident in my role as a medical student and more confident as a person in general, I stopped worrying about what my team might think of me if I pulled out my cell phone, and instead just started looking lots of things up during rounds. I found this really helped me feel more a part of the team because I was much better able to follow along.” “I do not think having flashcards mimics real‐world scenarios. I think flash cards benefit exam preparation specifically because of the spaced repetition. In real‐life clinic work, learning is real‐time from your patients and real‐time feedback from attendings which is different than the spaced repetition model of flash cards. I think flash cards are great tool for exam prep but not applicable in a clinic model.” “It is not feasible to look up the mechanism of every drug that you come across. Some memorization is needed, but the amount required should be targeted towards the high‐priority clinically relevant drugs. For example, just as it would not be feasible to look up the definition of every word you read, it would not be feasible to look up every drug you come across.” “Absolutely it does. But the real world does not matter to MS1's. Standardized exams matter.”

Survey Question 4: “I believe that having flashcards available on the F1 exams hurt my understanding of pharmacology”

Dominant themes (n = number of responses) Yes, it hurt my understanding (n = 9) No, I studied it later in clerkship (did not hurt) (n = 5) Flash cards helped my understanding (n = 3) No, I studied it later for Step I (did not hurt) (n = 6)

Sample quotes: “Between F1 and studying for step 1/2, I learned pharmacology sufficiently. I'm do not think having flashcards had an impact on my knowledge in the long term.” “My understanding of pharmacology was more impacted by the fire hose of studying in medical school than the presence of flash cards on exams. I needed a more digestible way of approaching studying pharmacology.” “Evaluations are probably the strongest motivator of learning in F1 students. If something explicitly is not required knowledge for the exam (i.e. something that will be given to us on a flashcard), students are much less inclined to learn it. Beyond that, relegating the drug info to flashcards put pharmacology in a sort of ‘second‐class citizen’ status in the curriculum where it was often short‐ changed in more complex discussions in small groups, lectures, and student study sessions as something ‘we do not need to worry about for the exam’. In this way, we not only did not memorize the basics, but we did not grapple with the complexities of clinical pharmacology either, so F2 was definitely a bit jarring since the nuances of treatment plans are often the main topic of discussion during rounds/presentations/etc.” “I agree that it made me struggle in pharmacology in the future because I never had to develop an in‐ depth understanding and memorization of the material (because some of pharmacology is truly memorization ‐ not all side effects make sense or can be explained). That being said, I still understand the general points of pharm (for example, what a sulfonylurea is) even if I did not remember every aspect of side effects, etc.”

Survey Question 5: “I believe having flashcards available on the F1 exams hurt my preparation for Step 1”

Dominant themes (n = number of responses) Yes, it hurt and took longer to study (n = 14) No, I used other resources/skills for Step 1 prep anyways (n = 10) Reduced load in F1 but increased in Step 1 (n = 8)

Sample quotes: “Not having fully learned the pharmacology during F1 due to the use of flash cards left me having to learn what I should've learned during F1 to learn during Step 1 prep time.” “I used Sketchy Pharm and a Sketchy Pharm Anki deck to prepare for Step 1 and it was unbelievably time consuming. I cannot imagine having spent that kind of time during F1 ‐ I do not think I would have had the time, nor do I think it would have been necessary.”

Survey Question 6: “I believe having flashcards available on the F1 exams hurt my preparation for clerkships”

Dominant themes (n = number of responses) Yes, it hurt clerkships (n = 15) No, because the focus different (n = 9) Cards were a study tool (n = 2)

Sample quotes: “As stated above, my experience in Pisces was going to see a patient, then leaving the room and having to present to your preceptor right away. So preceptors aren't giving you to time to look up all the pharmacology mechanism and side effects you may have forgotten (or not learned due to relying heavily on flashcards). It definitely reflected poorly on me when I had not learned those things and could not present them. And I felt preceptors held on to those early first impressions of me struggling in pharmacology and it negatively affected my longitudinal relationship with them.” “Honestly, I feel like the focus of clerkships is more diagnosis rather than management, so most attendings were impressed when we knew the appropriate pharm but it did not seem like it was expected of an MS3. I think studying pharm for Step 1 did really help me for my medicine Sub‐I and other Sub‐Is because during 4th year management become a larger focus of the medical student role.”

We found most students reported that pharmacology flashcards on their exams allowed them to focus on understanding the bigger picture, with less memorization and less stress.

I don't think the pharm flashcards had enough information to help me pass a UCSF OEQ exam without drawing upon the other information that I had studied.

I am constantly looking things up in clinical situations and being able to understand the underlying concepts has proved more useful than memorizing a specific drug.

Anxiety about Step 1 prep was not universal, since students report relying on external resources to relearn the pharmacology for Step 1 after clerkships:

I used Sketchy Pharm and a Sketchy Pharm Anki deck to prepare for Step 1 and it was unbelievably time consuming. I can't imagine having spent that kind of time during F1 ‐ I don't think I would have had the time, nor do I think it would have been necessary.

Interestingly, we did find that a subset of students reported that having access to flashcards on their exams hurt them in their preparation for clerkships (Table 3):

It definitely reflected poorly on me when I hadn't learned those things and couldn't present them. And I felt preceptors held on to those early first impressions of me struggling in pharmacology and it negatively affected my longitudinal relationship with them.

While this may not reflect the opinion of the whole class, we consider this an important observation as our intent is to set our learners up for success. The comment below lamented lack of basics and clinical application when flashcards were provided on exams:

In this way, we not only didn't memorize the basics, but we didn't grapple with the complexities of clinical pharmacology either, so F2 was definitely a bit jarring since the nuances of treatment plans are often the main topic of discussion during rounds/presentations/etc.

Finally, our data revealed that responses were divided regarding usefulness of pharmacology flashcards in mimicking the real world where healthcare providers can access relevant information electronically. Comments ranged from agree:

I am constantly looking things up in clinical situations and being able to understand the underlying concepts has proved more useful than memorizing a specific drug.

to disagree:

I don't think having flashcards mimics real‐world scenarios. I think flashcards benefit exam preparation specifically because of the spaced repetition. In real‐life clinic work, learning is real‐time from your patients and real‐time feedback from attendings which is different than the spaced repetition model of flashcards. I think flashcards are great tool for exam prep but not applicable in a clinic model.

4. DISCUSSION

4.1. Pharmacology exam resources do not improve performance

Given that learning and memorizing the sheer and expanding volume of pharmacology content is impractical, we set out to study a resource‐enhanced approach to teaching and assessing pharmacology. In this study, students were always provided with pharmacology flashcards as they learned the material and, for half of their first‐year courses, these same flashcards were allowed as resources during exams, which were closed‐book otherwise. The purpose of offering pharmacology flashcards on exams was to reduce rote memorization, ¹⁹ to encourage students to think about the course material at higher cognitive levels, ¹⁸ and to provide a more realistic approach and mimic a clinical setting. Furthermore, resource‐enhanced exams can be a learning experience in that they provide students with correct information and can therefore minimize commission errors that may be perpetuated long term. ²⁸ Previous studies have found mixed results when assessing student performance on full OBEs: some report an improvement, ²⁰ a decline, ²⁹ or no effect ²⁷ , ³⁰ of OBE versus closed‐book exams on student performance. In this study, we find that having access to pharmacology resources on otherwise closed‐book exams did not increase exam performance on pharmacology test questions. This is in contrast with a prior meta‐analysis which found that student‐generated notes provide a moderate benefit on exam performance ³⁸ but is in line with reports that students perform better without resources. ²⁴ Given that the flashcards did not contain additional information beyond pharmacology, our findings are in line with reports showing that access to notes on an exam resulted in little difference in exam performance unless the notes contained substantial information on the specific question topic. ³⁰

Prior work has shown that assessments at our institution promote deeper engagement with and application of medical knowledge, ¹⁴ , ³¹ contradicting the concern that expecting resources on an exam may reduce student preparation for the content and reduce the effort students put forth to build their knowledge. ²⁴ , ³²

To measure students' ability to demonstrate application of pharmacology knowledge, we analyzed average pharmacology OEQ answer scores on exams. Because different blocks utilized different teaching methods, but all students performed well across all blocks, we postulate that if there is an opportunity to practice applying knowledge, students can apply pharmacology to clinical scenarios on OEQ assessments. In fact, while the two blocks with lower average scores (REGN and PHD) had pharmacology content in both the large group and small group settings, BMB had higher average scores despite only having pharmacology in the large group setting. We note that these may only be statistically significant due to the large sample size (149 students and 3–9 OEQs per block). Overall, the data do not suggest a preference for educational format when it comes to the ability of students to apply pharmacology knowledge. This conclusion is limited by comparisons of non‐matched topics (i.e., cardiovascular vs. endocrine pharmacology), however, since all active learning sessions have clear learning objectives and all assessments are based on these objectives, we were interested in exploring the analysis of this relationship. Another variable is that the students are being provided flashcards at a later point in their training when they are further along the track of becoming expert learners. However, if students were more advanced when they received the flashcards, we would expect their performance to be higher, not lower.

We delivered a survey post‐clerkships and post‐Step 1 to the same cohort of students, which allowed students to respond with a long‐term perspective. Analysis of the comments in response to the questions revealed several themes (Table 3). In our survey, we found most students reported that pharmacology flashcards on their exams allowed them to focus on understanding the bigger picture, with less memorization and less stress, in line with prior studies. ²⁶ , ³¹ However, it has been reported that students tend to overestimate the impact of anxiety reduction. ²⁴

The quotes alleviate the reported concern that students who expect resources on an exam may reduce the effort students put forth to build their knowledge. ¹⁷ , ³² , ³³ , ³⁴ Rather, they show that students focus their efforts on larger concepts and do not memorize those facts they know they will have access to on an exam. This suggests it is important to select which topic(s) will be accessible via resources and which may need to remain closed book on an exam to ensure building of core concepts. ³⁵

Fear of under preparation for Step 1 is a known obstacle to implementing OBEs, ²⁶ which we also found in our survey data but was not a universal stance among the respondents. Moreover, students report relying on external resources to relearn pharmacology for Step 1 after clerkships.

Given that our institution moved Step 1 post‐clerkships the same year, we introduced flashcard access on exams; we cannot compare overall Step 1 performance to prior cohorts as the move of Step 1 post‐clerkships increased overall Step 1 performance, as described before. ³⁶ The change to pass/fail reporting for Step 1 may alleviate some of these fears as well. The effect of resources on block exams and Step 1 performance requires further study.

One aspect not reported prior in literature was the effect on clerkships. We did find that a subset of students reported that having access to flashcards on their exams hurt them in their preparation for clerkships. While there was a spread in what students perceived to be expected in different clerkships with respect to pharmacology, this is an important finding that deserves further exploration of the need to define what pharmacology knowledge and skills are required to be successful in clerkships. Some work has gone into this already ³⁷ but our data suggest the need for a more detailed roadmap.

Our data do align with the studies by Heijn‐Penninga, ³⁵ which showed that a combination of closed and open‐book exams was synergistic in reinforcing the content tested on closed‐book exams. In our case, students learned physiology and pathology in the F1 curriculum (closed book) and memorized less details for pharmacology (open book for pharmacology only). Survey responses showed this as a benefit: this approach allowed students to truly learn and understand the enduring concepts of physiology and pathology. Heijn‐Penninga's study showed that a combination of open‐book and closed‐book exams increased the long‐term retention of core knowledge; in our case, physiology and pathology were reinforced by having to apply pharmacology. In other words, a combination of closed and open‐book formats will allow testing of core knowledge (closed book) which will serve as a framework to apply other knowledge (open book) on OEQ assessments.

Finally, our data revealed that responses were divided regarding usefulness of pharmacology flashcards in mimicking the real world where healthcare providers can access relevant information electronically.

Taken together, this generates interesting research questions to see if open‐book assessment of clinical management—a true real‐world clinical skill—will reinforce pharmacology knowledge. Other questions to be explored include: What do core clerkships consider core pharmacology knowledge? What pharmacology knowledge can or should be accessible during exams at which phase of training, and what should remain closed book?

In conclusion, we found that use of pharmacology flashcards during otherwise closed‐book exams in the pre‐clerkship curriculum did not increase exam performance. The exam resource was received well by approximately half the class, who reported benefits including more time to focus on understanding the bigger picture concepts and less stress. Perceived negatives were fear of underperformance on Step 1 and underperformance in clerkships. Future research is needed to further explore our findings.

5. LIMITATIONS

This study was done at a single institution with a single cohort. The impact of our study would have been stronger if we had evaluated performance of matched cohorts. Future research may explore comparisons between cohorts where certain groups receive flashcards on specific topics (mixture of concept heavy drug classes and memorization heavy drug classes) and other groups who do not receive flashcards.

While the class served as its own control, the course content and exam difficulty across blocks are inherently different. Moreover, the setting in which students applied pharmacology knowledge and utilized flashcards varied between large group or small group sessions, which may have introduced additional variation. Finally, the response rate to the survey overall was 30%, however, not every participant replied to every question in the survey, making the response rate lower for a subset of questions. However, we did receive a wide range of comments, suggesting that responses were not limited to a biased subset of students with the same opinion. Keeping these limitations in mind, this study provides interesting insights for future studies and adds to the literature on resource‐enhanced exams, where findings remain contradictory and inconclusive. Elucidating this further will be relevant for a rapidly expanding field such as pharmacology and the evolving relationship between the practice of medicine and the use of clinical resources.

FUNDING INFORMATION

This study did not receive any external funding.

CONFLICT OF INTEREST STATEMENT

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

ETHICS APPROVAL

The Institutional Review Board (IRB) of our institution approved the study.

Supporting information

Appendix S1.

Appendix S2.

Kruidering‐Hall M, Tuan RL. Information overdose: Student performance and perceptions of pharmacology resources on exams. Pharmacol Res Perspect. 2023;11:e01087. doi: 10.1002/prp2.1087

DATA AVAILABILITY STATEMENT

Data sharing does not apply since all relevant data are summarized in the tables and figures.