Determining Whether Pharmacy Students Can Accurately Recall Items on an Examination Using Their Collective Memory

Abstract

Objective. To determine the extent to which students can recreate a recently completed examination from memory.

Methods. After two mid-term examinations, students were asked, as a class, to recreate recently completed examinations. Students were given 48 hours to recreate the examination, including details about the questions and answer choices. The results were compared to the original examination to determine the accuracy of students’ recall and reproduction of content.

Results. The students were able to collectively recreate 90% of the questions on the two examinations. For the majority of questions (51%), students also recreated the question as well as the correct response and at least one incorrect response. The majority of questions that the students recreated were of medium to high accuracy as they contained detailed phrasing that aligned with the original question on the examination.

Conclusion. The collective memory of a group of students may allow them to accurately recreate the majority of a completed examination from memory. Based on the findings of this study and tenets of social psychology, faculty should consider the potential implications for examination security, whether to provide feedback to students on examinations, and whether completed examinations should be released to students following the examination.

INTRODUCTION

When it comes to releasing a completed examination to students, faculty generally fall into one of three categories: those that release the examination, those that do not release the examination but offer reviews/provide feedback, and those that do not release the examination and provide no review. Often, students want to see their completed examination so they can learn from their mistakes and focus their efforts on clear gaps in their knowledge, consistent with the discrepancy reduction model.^{1 -3} Additionally, given the time and effort students put into preparing for an examination, they naturally want to know the results of this effort.^4,5 As educators, faculty should provide feedback as evidence suggests it is an essential part of the learning process.^{6 -8} However, faculty often protect their examinations and do not release them because of the time and resources required to construct quality examinations. Faculty believe that by not releasing completed examinations they can safely reuse questions and ensure that future examinations are secure and learning is accurately evaluated. Faculty often see students discussing answers to examination questions or learn that students recreate the examination in their study groups. Their inherent curiosity combined with the investment they have made in their learning inclines students to seek answers. The question then arises whether completed, graded examinations that are not released to students as a form of feedback are truly safe from reproduction.

In theory, a group of students should be able to recreate a shared experience such as an examination, because of collective memory. The social psychology literature is replete with examples of collective memory, which is the phenomenon in which a group of people can recall information about an event better than any one individual from the group can.^{9 -12} Brown and colleagues asked college aged students to individually recall advertisements seen on local mass transit.¹³ As a whole, the 175 participants recalled close to 900 items and over 200 different advertisements. However, the average person could only recall five advertisements. If this phenomenon were applied to recalling examination questions and each student could recall five questions, then it would only take approximately 20 students to recall all items on a 100-question examination, assuming no students recall the same items. Additionally, due to a phenomenon called collaborative facilitation, recall of one item by one person who then shares that item with a group may cue recall of other items by other members of the group, allowing students to remember more items as they discuss the examination questions.¹²

As mentioned earlier, students compare their responses after an examination to determine the “correctness” of their responses and to communicate. Humans communicate to remember and share experiences. Harber and colleagues conducted a study where 33 students completed a field trip to a morgue.¹⁴ In three days, over 800 people had heard about the field trip. The spread of the news of the field trip was attributed to the compulsion people have to disclose traumatic experiences. In other words, people tend to share emotional events with others with little prompting. Examinations may represent an emotional event for students, and discussing details about the event may be a natural human response.¹⁵ Thus, if students can recall information from an examination, they may naturally want to share it.

To date, there is negligible research on the ability of students to recreate assessments. Anderson and McDaniel examined students’ memory of a quiz they had recently completed.¹⁶ Students were asked to write down questions from the quiz that they remembered, and based on the results, the authors concluded that students had poor memory of the quiz. While individual students in the study did not have strong memories about the content of the quiz, that does not mean that their collective memory of the entire quiz would not have been better.

The objective of this study was to test a faculty observation that students can recreate a recently completed examination from memory. By answering this question, faculty members’ views about releasing examinations to students may change, which in turn may persuade them to focus more on using assessment to promote learning than to evaluate learning.

METHODS

Participants were 125 first-year, first semester student-pharmacists enrolled in a core course within a Doctor of Pharmacy program. This course had three mid-term examinations, each of which assessed new content, and a cumulative final. The three examinations included 40 to 70 single-answer, multiple-choice questions, ranging in difficulty from lower level (knowledge/comprehension) to middle level (application/analysis) on Bloom’s Taxonomy.¹⁷ Each question had one correct response and two or three distractors. Examinations were administered using ExamSoft. Because of the COVID-19 pandemic, examinations were virtually proctored to minimize academic dishonesty.¹⁷ Immediately after completion of the second and third midterm examinations, the instructor emailed the students a link to a single GoogleDoc with instructions for them to work as a group to recreate the examination. For the second examination, students were instructed to be as accurate as possible with content, order, and phrasing. For the third examination, the students were asked to collaborate on recreating the examination in GoogleDocs, but were given even more explicit instructions directing them to include both correct and incorrect answer choices. Students were given 48 hours after each examination to complete the group activity and offered extra credit (one extra percentage point on their final examination grade) for participation.

In previous years, course examinations were not released to students. Students only received their examination score and a report of their performance on question categories. That is, they did not have access to examination questions from previous years. In the study year, the same procedures were used: students only received their score and overall performance within question categories.

The primary outcome measures were the number of questions students were able to recreate, the number of questions recreated for which the correct answer was recalled, the number of questions recreated for which at least one distractor answer was correctly recalled, and the ordinal scoring of question specificity. For this latter metric, recreated questions that captured the gist of the target question were considered low specificity, recreated questions that contained more accurate phrasing of the target question were considered medium specificity, and recreated questions that captured specific or exact phrasing of the target question, parts of the case vignette, and/or other details were considered high specificity.

In the analysis, we included the item difficulty of the examination question and its discrimination index to assess whether question difficulty impacted students’ ability to recreate the examination. Based on the data distribution, we tested for differences between generated vs non-generated items using a Mann-Whitney U test. Statistical significance was set at p<.05. This was a retrospective study deemed exempt by the University of North Carolina Institutional Review Board.

RESULTS

Within 48 hours of completing two midterm examinations, the class collectively recreated 90% of the questions and the respective correct answers (103/115, 90% for examination 2 and 89% for examination 3). On average, students individually recreated 51% of these questions with the correct answer and at least one distractor. However, the percentage of questions for which distractors were recreated varied between the two examinations, with 31% recreated on the second examination and 82% recreated on the third examination. Of the 51% of questions with the correct answer in combination with at least one distractor, 10% had one distractor, 12% had two distractors, and 29% had all three distractors recreated correctly.

Cumulatively, 8% of the questions recreated were deemed low specificity, meaning the student captured a minimal amount of the general content or gist of the question but no specifics; 38% were deemed medium specificity, meaning the student captured the stem of the question, and the remaining 54% were deemed high specificity, meaning the student provided details about the question. Examples of low and high specificity questions are presented in Table 2.

Table 2.

Questions That Were Recreated From Memory by Pharmacy Students Who Had Recently Completed an Examination

When examining item difficulty, discrimination index, and question recreation, no notable trends were identified (Table 1). The median difficulty for questions not recreated vs recreated was 93% vs 89% (p=.13); the median discrimination index for not recreated vs recreated was .14 and .17, respectively (p=.21).

Table 1.

Summary of Question Specificity and Question Psychometrics for Items Recreated From Memory by Pharmacy Students Who Had Recently Completed an Examination

DISCUSSION

The goal of this study was to assess students’ collective memory of a recently completed examination by having them recreate the questions from memory. The results demonstrate that soon after completing an examination, students can collectively recall a very large portion of the material.

The students were not able to recreate the entire examination. Interestingly, the 10% of questions that were not recreated seemed to be less difficult or challenging questions. There are several explanations for this phenomenon: encoding variability, attentional failure, and/or recollection failure. Generally, recall of information is enhanced when there are multiple routes to the target information rather than a singular route.^19,20 Encoding variability theory suggests that the more cues a person has to recall information, the better the recall.¹⁹ Therefore, an easy question, which may have less cues, may not elicit a strong enough recall. The second theory is attentional failure, which proposes that questions that do not garner sufficient attention are less memorable. For example, if the question is easy, it may not warrant the attentional processing, and as such, may not be memorable. The final explanation is recollection failure. When people successfully recollect information, it is because that recollection is associated with retrieval of specific contextual details about an event. Recollection failure occurs when contextual details are lacking, in part by increased familiarity. For example, if a question was discussed in class, it may be familiar, while “old” and older items in memory may be harder to recall.^21,22

In this study, the examination was recalled as a collaborative group, asynchronously. If a group of people can divide a memory task among themselves, the burden of remembering is distributed as each person uses the input of others as a memory aid. This is referred to as transactive memory.⁹ For this reason, when several people come together as a group to work on a task or brainstorm, they can perform better than they could as individuals. However, this advantage is contextual. For instance, one student’s input could serve as a cue for another student and promote recall of other facts, and this would be consistent with the idea of collaborative facilitation.⁹ However, the reverse could also be true as collaborative brainstorming and recall often lead to collaborative inhibition.⁹

We did note some differences between the two examinations that may reflect the specificity of instructions. For the third examination, more specific instructions were provided to students, asking them to recreate question distractors. This resulted in an increased number of distractors listed (32% vs 81% of questions). Therefore, the instructions provided to students for recreating their collective memory may influence the outcome. This is consistent with the literature on eyewitness testimony in that how questions are asked leads to varying degrees of answer specificity.^{23 -25}

One strength of this study is that it was conducted in an authentic classroom environment. Additionally, students were unaware they were going to be asked to reproduce the examinations and therefore did not prepare or plan ahead to memorize the questions. Since these were first year, first semester Doctor of Pharmacy students, they were unfamiliar with the instructors and their question writing style. Finally, examinations from previous years had not been released and no practice examinations had been administered; therefore, it is unlikely students had additional exposure to the examination questions. A potential limitation of this study is that it is unknown exactly how many students were involved in recreating the examination. If only a few students participated, the accuracy of the cohort’s overall ability to recreate these questions may have been underestimated. A final potential limitation was that students with higher academic ability, such as those in a professional program, may have had a higher ability to retain information and thus recreate examinations more easily than those with lower academic abilities.

An argument could be made that recalling questions from an examination and later sharing them with other students falls within the realm of academic dishonesty in the same sense that unauthorized use or attempted use of material, information, notes, study aids, devices or communication in the learning environment is considered academic dishonesty.²⁶ However, students simply recalling an experience does not constitute academic dishonesty. The real concern is what such students do with the recalled information. Some might argue if they generate an examination and share it with future classes, that might constitute academic dishonesty. However, if a student generated an examination from memory and months later tutored a lower classman by providing helpful tips to be successful in class, is this cheating or peer tutoring? We know that peer and near peer tutoring is a powerful learning strategy.^{27 -30} Additionally, sharing of experiences should deepen learning based on the theory of shared in-group attention.³¹ This theory suggests adults are more likely to engage in elaborative processing of material and make linkages to the broader knowledge structure when that object or experience is believed to be co-attended with one’s social group.³¹ Thus, the sharing of examination experiences should promote learning.

Assuming students can collectively reproduce examinations from memory and that their natural tendency is to share information, which also may facilitate learning, we offer the following recommendations. Academicians should reduce the high stakes nature of examinations and/or minimize the transactive nature of education (ie, the expectation for students to reproduce what faculty teach). In doing so, we may minimize some students’ inclination to cheat and, hopefully, promote learning.^31,33 In the learning process, we need to remember feedback is important. As such, we recommend that learning and feedback be the priority for assessments. If the focus is using assessments to promote learning, then the issue of assessment security is minimized as the goal is to promote learning rather than evaluate learning. Anderson and McDaniel found that if quizzes are returned so students can study them, there is minimal need for feedback. However, if quizzes are not returned, corrective feedback is necessary.¹⁶ Thus, simply returning students’ assessments would reduce the need to provide corrective feedback. If assessments are to be used for summative judgements of learning that impact student progression through a curriculum, there should be a larger programmatic assessment plan. Best-practices suggest that this larger programmatic assessment plan should focus less on examinations and more on the whole student.³⁴ However, when high-stakes assessments are necessary (eg, licensure), then best practices for examination development should be followed.³⁵

CONCLUSION

By understanding the concept of collective memory, instructors’ refusal or reluctance to return completed examinations to students or provide feedback to them because of concerns about examination security may be challenged. The results of this study should not be surprising as many students already practice recalling examinations. However, the findings may further encourage the use of assessments to aid in the learning process by either providing examinations back to students or offering feedback.