A Subgroup Analysis of the Impact of Self-testing Frequency on Examination Scores in a Pathophysiology Course

Abstract

Objective: To determine if the frequency of self-testing of course material prior to actual examination improves examination scores, regardless of the actual scores on the self-testing.

Methods: Practice quizzes were randomly generated from a total of 1342 multiple-choice questions in pathophysiology and made available online for student self-testing. Intercorrelations, 2-way repeated measures ANOVA with post hoc tests, and 2-group comparisons following rank ordering, were conducted.

Results: During each of 4 testing blocks, more than 85% of students took advantage of the self-testing process for a total of 7042 attempts. A consistent significant correlation (p≤0.05) existed between the number of practice quiz attempts and the subsequent examination scores. No difference in the number of quiz attempts was demonstrated compared to the first testing block. Exam scores for the first and second testing blocks were both higher than those for third and fourth blocks.

Conclusion: Although self-testing strategies increase retrieval and retention, they are uncommon in pharmacy education. The results suggested that the number of self-testing attempts alone improved subsequent examination scores, regardless of the score for self-tests.

INTRODUCTION

Assessment, in its many forms, permeates pharmacy curricula. The Accreditation Council for Pharmacy Education (ACPE) places particular emphasis on assessment of programmatic and student outcomes and encourages use of multiple types of assessments in its standards and guidelines.¹ From a student perspective, examinations comprised of multiple-choice items are perhaps one of the most commonly employed means by which individual performance is assessed, both in a formative and summative manner (eg, taking a practice examination and licensure examination, respectively). Examinations can be short, long, low-stakes (or no-stakes), high-stakes, and can be called by several names, including quizzes, tests, and assessments. Pharmacy students have opined that testing serves primarily to assess the amount of material learned in pharmacy courses and to assign grades.² While valid examinations can be used to assess student outcomes, research indicates examinations serve another purpose less commonly perceived by test givers and test takers alike: examinations can improve learning.^3,4

The testing effect has been one of the topics of interest for improving learning in the cognitive psychology arena for centuries.^5,6 Specifically, researchers have examined multiple aspects of testing as it relates to retrieval (accessing stored information) and retention (keeping information in memory).^3,4,7-15 Karpicke, Roediger, Bjork, among others, noted increased long-term retention after repeated retrieval (eg, self-testing) compared to repeated studying (eg, rereading notes).^4,8,10,14 Intermittently testing events in learning environments also facilitated learning more than repeated studying did.^7,9,16,17

Despite the ubiquitous nature of examinations in schools and colleges, to our knowledge, no research has been conducted to evaluate the relationship between self-testing attempts and course outcomes in pharmacy curricula. Hagemeier and Mason examined student pharmacists’ perceptions of testing and found that only 7.3% of respondents perceived tests as events that improve learning.² The same students, when given scenarios to gauge the extent to which they would employ self-testing strategies as opposed to restudying strategies, indicated a lack of awareness in their responses that self-testing can improve learning more than restudying can. Overall, their research indicated many students did not employ strategies that optimally promoted learning, a finding similar to what was reported by Karpicke.^18,19

ACPE Standard 11 indicates colleges and schools should employ teaching and learning strategies that “enable students to transition from dependent to active, self-directed, lifelong learners.” ¹ Structuring student opportunities to self-test through development of a practice self-test item bank is one method of promoting strategies that optimize learning. Frequent testing can mediate student learning by providing students with formative feedback from the examinations and can result in material retention for longer periods.^20,21 Therefore, self-test banks allow students to test frequently and simultaneously encourage the use of retrieval techniques that promote retention.

Previously, we developed and implemented a structured self-test bank of questions for a required first professional year (P1) pathophysiology course. Published results documented benefits of self-testing, specifically showing a positive correlation between performance on self-testing activities and performance on subsequent course examinations.²² To our knowledge, no data exists evaluating the frequency of self-testing attempts, irrespective of student performance, and the relationship to course outcomes as measured by examination scores. The purpose of this manuscript is to report an analysis of the frequency of self-testing compared to examination scores.

METHODS

Pathophysiology was a 17-week, 4-credit hour, required course taken by doctor of pharmacy students during their first professional year. The course had 4 examinations, equally spaced and weighted throughout the semester. The material covered on each examination only pertained to that presented after the previous examination, and no examination was comprehensive. There were also 8 required quizzes, equally spaced and weighted throughout the course, accounting for 20% of the final grade. In 2011, a self-testing/quizzing component was added to the course to allow students to practice retrieval of course information. After content was discussed in class, a graded quiz was given. Following the quiz, a subsequent practice pool of questions was released to the students using the university’s course management software, Desire-2-Learn (D2L) (Version 8.3, Kitchener, ON, Canada). During this time, students had unlimited access to the question pool, and each practice quiz took a maximum of 30 minutes and consisted of 15 questions drawn randomly from the pool. When the practice quiz was submitted to D2L, the quiz was scored immediately, and the students were allowed to see the percentage of questions answered correctly. To provide feedback on areas of student deficiency prior to the examination, students were able to view incorrectly answered questions with all potential answer choices, but the correct answer choice was not indicated. While taking the practice quizzes and reviewing incorrect quiz responses, students used the Respondus Lockdown Browser (Respondus, Redmond, WA). This browser prevents copying and pasting or similar activities. The D2L software proctoring the quizzes was set for auto-submit when the time period for the quiz ended. Thus, the D2L system would submit the quiz at the end of the 30 minutes even if a student had not completed it. All questions for the examinations, required quizzes, and practice quiz pools were written by one of the investigators, who was also the course director and lecturer. As this was not a prospective investigation, the questions were written in real time as the content was covered during lecture or in the appropriate sections of the textbook. All questions were written prior to the practice quizzes and only grammatical or formatting changes were made to questions while the practice quiz was available to students. A discussion board on D2L provided clarification to students on content issues related to the questions, and highlighted “bad” questions, and gave correct information to students. The investigation received approval from the East Tennessee State University Institutional Review Board prior to data collection and analysis.

All data were organized using Excel 2007. Data analyses were completed using IBM SPSS Statistics 19 (IBM Inc., Armonk, NY).^23,24 Two-way repeated measures Analysis of Variance (ANOVA) with interaction was conducted. The 2 main effects for the 2-way ANOVA were the variables “time” with 2 blocks and “performance” with 4 blocks. Two time blocks were compared: the first time block when the practice quizzes were available to the students and the second time block when the examinations were taken. Each performance block represented the number of attempts at the practice quizzes and the examination score, during each of the 4 examination periods. The 2-way ANOVA determined if there were any differences in number of practice quiz attempts using the question pool prior to each examination and if there was a difference in the examination scores that followed the practice quiz interval. The interaction between main effects showed the relationship between the number of attempts prior to the examination and the subsequent examination score for each of the 4 examination periods. Mauchly’s test for sphericity was conducted on the performance main effect and interaction between performance and time effects. No test for sphericity was conducted on the main effect of time as only 2 blocks represented this analysis. A Greenhouse-Geisser correction was used to correct the F statistic for lack of sphericity when appropriate. Subsequently, only differences between number of practice quiz attempts or examination scores were evaluated. Group differences were examined using a 1-way ANOVA with a Welch’s F test for lack of homogeneity, followed by Games-Howell post hoc multiple comparisons. A Pearson’s correlation was conducted to examine individual associations between the number of practice quiz attempts and score on the subsequent examination. Finally, a 2-group comparison was conducted to determine whether there were differences in examination grades based on number of practice quiz attempts. For each examination, the students’ scores and number of practice quiz attempts were rank-ordered based on the number of practice quiz attempts prior to the examination. The rank orders were then divided into cohort upper and lower 50th percentiles based on number of attempts. Comparison of examination scores between the upper and lower 50^th percentiles were conducted with a 2-tailed t test for independent samples with equal variance not assumed. All graphs were created using Slide Write Plus for Windows Version 5.0 (Advanced Graphics Software Inc., Encinitas, CA), and data was reported as arithmetic mean plus or minus standard error of the mean.

RESULTS

Seventy-nine students were enrolled in the course in the spring semester of 2011. Table 1 provides the descriptive statistics associated with the number of practice quiz attempts using the question pool on D2L. The number of questions in each self-testing quiz bank varied by examination from 247 to 431 questions. The number of days the practice quiz question pool was open also varied from 14 to 19. Total number of self-attempts by the students for the practice quizzes prior to the examination increased from the first to the second testing block, and subsequently declined. The range of attempts by students also varied, and some students—not necessarily the same students—never accessed the practice quizzes during a given testing block, which was indicated by a zero at the bottom of the range for each examination period. However, as Table 1 documents, during testing blocks 1 through 3 more than 95% of students accessed the practice quizzes at least once, and in testing block 4 more than 85% accessed the practice quizzes.

Table 1.

Descriptive Statistics and Multiple Group Comparisons for Number of Attempts and the Corresponding Examination Score

The 2-way, repeated measures ANOVA with a Greenhouse-Geisser adjustment for lack of sphericity documented significant differences in the number of practice attempts and subsequent examination scores during the 4 testing blocks with all reported main effects for the ANOVA being significant (p<0.05). Additionally, during the 4 testing blocks, the number of practice quiz attempts did not result in a consistent equivalent score on the subsequent examination as documented by the significant interaction between performance and time variables. This was observed in the increase of number of attempts during the second testing block without a proportional increase in examination score, as compared to the other 3 examination blocks (Table 1).

Each of the subsequent 1-way ANOVAs was also significant (p<0.002) for the main effects of “Average Examination Score” and “Quiz Attempts/Student,” after Welch’s F test adjustment. The variable “Average Examination Score” determined the differences between the examination scores during the 4 testing blocks, and “Quiz Attempts/Student” determined the number of quiz attempts during the same 4 blocks. As there was no control in this study, the student cohort’s first exposure to the self-testing concept during the first testing time block was viewed as the control (Table 1). Compared to the first time block, there were no significant differences in the number of practice quiz attempts in blocks 2 through 4; however, the number of attempts in block 3 was significantly lower than in block 2. In contrast, the scores for the first and second examinations were not significantly different, but both were significantly higher than scores for examinations 3 and 4.

A Pearson’s correlation was conducted to compare practice quiz attempts to subsequent scores on examinations 1 through 4 (Table 2). A significant correlation existed between the number of practice quiz attempts and the subsequent examination score in all four testing blocks. However, this association decreased continuously as the testing blocks (ie, the semester) progressed, and was most precipitous for the last block.

Table 2.

Pearson’s Correlations Between Number of Practice Quiz Attempts and Examination Score

Figure 1 shows the 2-group comparison for number of attempts at the practice question pool and subsequent examination score after separating the students into the upper and lower 50^th percentile based on number of attempts. Prior to all examinations, the upper 50^th percentile demonstrated significantly more practice quiz attempts compared to the lower 50^th percentile. On examinations 1 and 3, the lower half, based on number of practice quiz attempts, scored lower on the subsequent examination. No difference was observed between the 2 groups on examination 2, and examination 4 differences only trended toward significance (p<0.06).

Figure 1.

Comparison of upper and lower 50th percentile of class on the number of practice quiz attempts (upper graph), and subsequent examination score (lower graph) during the 4 testing blocks. The star represents when the upper and lower percentiles were significantly different (p<0.05) in either the upper or lower panels.

DISCUSSION

Examinations can be viewed by students and even faculty members as a necessary evil in education, representing major components of grades in coursework at all levels of the educational process. The common opinion is that examinations are required to assess student knowledge of course material content. Yet testing has the possibility of providing a learning process by improving retrieval and retention and increasing the efficacy of study time by showing students content requiring further study.^{3,4,7-15,25-27} To evaluate the concept of self-testing in this analysis, we chose the number of quiz attempts by the students for comparison to examination performance. Choosing quiz attempts as the variable minimized confounding factors, such as differences in previous academic success among students taking the course.

The value of such a learning process can go unnoticed by students. When surveyed, the majority of students both within and outside pharmacy education admitted to not using self-imposed or institutionalized self-testing compared to rereading, practice recall, or concept mapping.^2,19 Yet when compared to these alternative methods, testing provides improved retrieval and retention.^{4,7,13,14,28,29} Even more surprising, students using these methods were more confident than students who self-tested, even though scores on subsequent examinations was lower for students opting not to self-test.^8,10,19,30 In contrast to previous surveys, students in this investigation did take advantage of the opportunity to self-test prior to an examination on the related content area. Seventy-nine students accessed the practice quizzes a total of 7042 times during the 17-week semester for an overall average of 22 attempts per examination per student. The difference between what students stated when surveyed about self-testing and what occurred in the present investigation may be related to ease of accessibility of such self-testing formats.

Admittedly, there were students who never accessed the practice quizzes prior to each examination. This may have been related to the way the software program recorded access. If 2 or more students accessed the practice quizzes as a group, only 1 student would have logged into the system and would have been recorded as having taken the quiz. This obviously affected not only the potential total number of attempts by students but all subsequent analyses as well. However, when students access a testing program on their own time outside of class, there is no way to control such group participation.³¹ No survey was conducted at the end of the course to determine student satisfaction with the self-testing quizzes. However, the use of these quizzes for the 4 examinations suggested that students found them beneficial. Moreover, data from a survey of medical students reported 90% of the students felt the self-testing process was beneficial for learning clinically related material.³²

Assessment of testing on retention and retrieval has been conducted using different formats, including word pairs, free recall, and multiple choice questions.^{3,4,7-11,13-15,28,29,31,33-36} Multiple choice questions have historically been thought to evaluate only recognition, and in turn result in decreased retention as compared to retrieval tests.^37-39 However, Little et al conducted research specific to multiple-choice examinations and the extent to which this commonly employed format can induce retrieval compared to recognition.¹⁵ The researchers noted that multiple-choice examinations can induce retrieval if incorrect alternatives, or distractors, are plausible. Additionally, randomized multiple-choice questions have the advantage of facilitating recall of incorrect alternatives, which is missing from free recall and word pair assessments. In the present investigation, multiple-choice questions were used with the D2L testing program as they gave students immediate feedback when they used the practice quizzes for self-testing outside of class.

Although many investigations have examined testing under controlled laboratory settings, fewer have attempted such an investigation as a component of an actual class.^3,31,40,41 Roediger and Karpicke noted a lack of controls in the educational setting,³ a limitation that existed in this investigation as well. Despite the lack of controls, Roediger and Karpicke concluded that the testing effect does extend to the classroom setting. In our investigation, a significant correlation existed for all 4 testing blocks between the number of attempts at self-testing via the practice quizzes and subsequent examination scores. Additionally, the upper 50th percentile, based on number of quiz attempts, scored higher on the subsequent examinations in 2 of 4 testing blocks, (examinations 1 and 3) and trended higher on examination 4 (p<0.06). In aggregate, these results support the value of testing, or in this case self-testing, as a learning tool in the academic setting.

Previously, investigators noted that technology limitations prevented tracking the frequency of use when self-testing outside the class room.³¹ Using the Internet-based D2L program eliminated this problem. Lee, Nagel, and Gould also used D2L to assess self-testing by students in a human anatomy course in professional dentistry education.⁴⁰ The investigators concluded that self-testing, as assessed by multiple-choice quizzes, provided no consistent benefit to students on subsequent examinations; however, in that analysis only scores on the first attempt at the practice quizzes were compared to subsequent examination scores.

Previously, our team documented an association between the average practice quiz score and the related examination score.²² The current analysis showed a similar, albeit weaker, association between the number of practice quiz attempts and subsequent examination performance irrespective of the students practice quiz scores. Both these analyses seem to be more logical predictors of students’ eventual performance than simply how they performed on the first iteration of an exercise that should have been repeated multiple times.

Additionally, unsuccessful attempts, recognizing incorrect answer choices, and temporal spacing between testing attempts demonstrated a significant impact on retrieval, with equally spaced retesting superior to either retesting at expanded intervals or mass testing.^3,4,9,11 Although equally spaced testing optimizes recall, the current investigation was unable to control the interval between self-tests by students using the D2L testing program. Our results did document that more attempts correlated with a higher subsequent examination score, and that the self-testing opportunity was always open for a minimum of 2 weeks during any of the 4 separate pre-exam periods. Unfortunately, the software does not allow for minimal intervals between practice quiz attempts. Nor is systematic collection of individual attempts for statistical analysis available with the current version, so determination of intervals between attempts for each student would be difficult.

Incorrect answers during testing is also thought to enhance learning and retention, and this phenomenon may be further enhanced by delaying the feedback correcting the error.^3,13,34-36 Repeated testing of material correctly identified also increased learning and retention compared to dropping the item once correctly identified.¹⁸ The positive benefit of continuously testing already correctly identified material is especially important, as students often stop self-testing on material once correctly identified.⁴ In the current investigation, the software program parameters were set to allow students to review failed question attempts. Specifically, students were allowed to review incorrectly answered questions along with all the potential choices; however, the correct answer choice was not identified. Previous correctly and incorrectly answered questions were also kept within the random question pool, and the students had no control themselves over which questions were selected for their quizzes. This prevented students from removing correctly answered questions from the pool and repeatedly exposed students to these questions with randomization of the correct answer’s position. Additionally, greater retrieval on examination occurs with testing compared to restudying when interfering tasks exist between the preparation, testing or restudying, and the examination. For students in academia such interfering tasks would be other courses.¹⁴

There were limitations to this investigation, some of which are unique. The number of days the self-testing practice quizzes were open prior to the examination was dictated by several factors. Each practice quiz window was open only after the course content was covered and prior to the examination date. The former was dictated by the class schedule and the latter was coordinated with the Office of Academic Affairs for the college. The variation of the self-testing practice quiz window was thus partially out of the investigators’ control. The variation for number of days the self-testing practice quizzes were open was 5 days. As this investigation was not a priori, one of the investigators wrote all the questions for the practice quizzes in real time during the semester, which accounted for part of the difference in the number of questions during each testing block. Separately, these practice quizzes were an independent self-testing activity outside the class but within a larger academic curriculum for that semester. Thus, there were competing requirements from other courses within the curriculum, which may have interfered with student use of the practice quizzes during each of the 4 testing blocks. Moreover, study time for competing courses during the semester was completely out of the control of the investigators. Alternatively, students may have felt more self-confident in their ability to master the content of the course as the semester progressed. This failure to recognize their mastery of the course content may have resulted in slightly lower self-testing attempts during the third and fourth testing blocks and the subsequent scores on examinations 3 and 4.

CONCLUSION

In conclusion, in both the laboratory setting and the classroom, evidence supports the value of self-testing for retrieval and retention. This investigation presented additional data of the application of these concepts in the professional pharmacy education environment. The investigation also provided further documentation of the value and applicability of self-testing as a learning process by specifically showing a correlation between the number of self-testing attempts and subsequent examination scores.

APPENDIX 1

Tests of Within-Subjects Effect