Effect of an Individualized Post-Examination Instructor Remediation on Pharmacy Student Performance in a Biochemistry Course

Abstract

Objective. To assess the effectiveness of a post-examination, one-on-one instructor remediation process on student performance in a pharmacy biochemistry course by measuring the degree of score improvement on a subsequent examination.

Methods. Students who scored below 70% on any examination were encouraged to meet with the course coordinator. A typical remediation session lasts about 30 minutes, and covers academic preparation, study habits, concept understanding, application, critical thinking, time management, and stress control. Scores in two consecutive examinations were compared between students who underwent remediation and those who did not. All scores were adjusted for level of difficulty.

Results. At-risk students with relatively lower scores are more likely to seek remediation. After receiving a score below 70%, students perform better on the next examination regardless of remediation. However, the remediation process results in a statistically significant 43% increase in the degree of improvement in student performance on the next examination.

Conclusion. A post-examination, one-on-one remediation is effective in enhancing student performance in the biochemistry course. As this course is one of the two with the highest failure rates in the PharmD program, current intervention might improve student retention.

INTRODUCTION

A strong foundation in the basic sciences is crucial for students to develop critical thinking skills necessary to deliver patient-centered care.^1,2 Rigorous summative assessment of students’ knowledge in biomedical and pharmaceutical sciences is essential to the training of students in professional pharmacy programs. Early identification of poor-performing and at-risk students and effective intervention are integral to student academic success and self-directed learning.³

The Accreditation Council for Pharmacy Education (ACPE) Standards 2007 mandate that pharmacy schools have a course remediation policy in place. These standards, however, did not explicitly mention or describe an early intervention process/policy. The most recent ACPE Standards 2016 have addressed this issue and now require that schools have an early intervention process for academically struggling students.⁴ This change in standards serves to highlight the importance of addressing student performance at an early stage of professional training. Prevention of course failure is beneficial to all stakeholders involved and necessitates further research into best practices. Timely intervention by course coordinators during the semester that facilitates improved academic performance on an individual basis (ie, remediation) may provide an effective solution to academically struggling students.

A study on remediation of medical students has shown that self-assessment, reflection and faculty feedback significantly improve the performance of at-risk students.⁵ Moreover, a teacher-led, multi-ingredient remediation process supporting students’ emotional needs and improving self-regulation and critical thinking is highly effective.⁶ Minimal research has been published related to one-on-one instructor remediation, especially in the field of pharmacy. Most of these studies have occurred in the fields of nursing and medicine. In 2006, Stark reported that the use of student-directed post-examination review was far more effective than traditional faculty-led review sessions, suggesting the importance of active learner participation in the process.⁷ In 2013, Bachman concluded that individual attention with a faculty member provided meaningful remediation to struggling college students compared to group reviews.⁸ In 2014, Corrigan-Magaldi and colleagues reported on a faculty-facilitated remediation program, where a combination of formative assessment and motivational emails were used to improve student performance.⁹ In a 2015 article, Wiles described an individualized, faculty-facilitated examination feedback procedure, which improved student performance by identifying students’ specific weaknesses.¹⁰ The procedure involved the use of a so-called “feedback grid” that allowed the faculty member to identify specific weaknesses of a particular student among various areas, including content-specific knowledge, comprehension, application, analysis, synthesis and evaluation.

The above studies highlight the importance of individualized intervention and remediation by an instructor, and lay a solid foundation for the development of effective early intervention programs in pharmacy education. Since fall 2009, approximately half of the students who failed their first pharmacy course at an individual US pharmacy school did so in the fall semester of the first professional (P1) year (Table 1). The Cellular and Molecular Biochemistry (CMB) course is one of the two courses that has the highest failure rates in the P1 year. For at least seven years, a standard practice, as stated in the course syllabi from the pharmaceutical sciences department, is that students who receive a grade below 70% on any examination are strongly encouraged to meet with the course coordinator to discuss possible avenues for improvement. During the meeting, the course coordinator asks the student to provide a self-assessment of his/her examination performance and explain potential causes for their poor performance. This is then followed by discussions on academic preparation, study habits, content understanding, concept application, critical thinking, time management, and other life issues that might have adversely affected their performance on the examination. Specific questions that the student missed are also discussed, and reasons for choosing the wrong answers explored. In the end, the course coordinator provides several suggestions for the student. This study was designed to assess the effectiveness of such a remediation measure in CMB, by evaluating the degree of improvement in student performance on a subsequent examination.

Table 1.

Distribution of First-Time Course Failures at a US Pharmacy School^a

METHODS

This study spans five consecutive years from 2010 to 2014. The CMB course is offered in the fall semester of the P1 year. Four examinations were administered throughout the course. The content of the course is roughly divided into three and a half blocks. Each of the first three examinations (Exams 1-3) covers one of the first three distinct blocks, and the Final Exam covers the last half block plus a comprehensive component that covers the material taught throughout the semester. The course grade is the average of the scores from the four examinations. Students receiving a course grade <67% fail the course. After each of the first three examinations, all students who receive a grade below 70% are advised (but not required) to meet with the course coordinator for one-on-one remediation.

Before the post-examination, one-on-one remediation, all students have had the opportunity to complete an examination question review, where they view the questions they missed, and the correct answers to these questions. A typical one-on-one remediation session lasts 30 minutes. The student provides information on their prior academic preparation. Information collected includes how many years of college education the student has completed prior to pharmacy school, what biology and chemistry courses the student has taken, whether the student has taken biochemistry before, in which educational institutions the student took these courses, and how long the student was out of school prior to entering the pharmacy program. Next, the student is asked to describe how he/she studied for the course, how he/she prepared for the examination, and how much time he/she spent studying for the examination.

At least five randomly selected examination questions that the student missed are used as examples to probe specific deficiencies the student has regarding the material. The student is asked to re-answer the questions. His/her answers are compared with the answers he/she gave during the examination and then with the correct answers. The student is asked to describe why he/she selected a particular answer. Specific references are made to the corresponding lecture handouts. Based on the above information, a “diagnosis” can be made regarding the difficulties the student has in effective learning in the course. These can include deficiencies in concept understanding, knowledge application, and critical thinking.

Aside from content-related questions, the student is also asked about their time management skills and level of stress. In the end, the student receives suggestions on how they can improve. During the five-year period, the same remediation approach has been used by two different course coordinators: three years for one and two years for the other.

To quantitatively assess the effectiveness of the remediation procedure, the scores of the students on the examinations before and after all three windows of remediation (ie, between Exam 1 and Exam 2, Exam 2 and Exam 3, and Exam 3 and Final Exam) were compared between the “remediation” and the “no remediation” groups, as were the points of improvement. To make the scores among different examinations comparable, student scores on each individual examination were normalized to the class average of the same examination, and further normalized to the average of all four examinations in percentage. Unpaired, two-tailed t-test was used to perform statistical analysis. This study was approved by Samford University Institutional Review Board.

RESULTS

There were 632 students enrolled in the CMB course during the 2010-2014 academic years. There were 225 cases (after Exams 1, 2 and 3) that qualified for remediation, among which students went through the post-examination, one-on-one instructor remediation process in 111 cases (49.3%) and did not in 114 cases (50.7%).

To understand what kind of students are more likely to seek help through the optional remediation process, the 225 student scores below 70% on any of the first three examinations were collected, and then adjusted based on the relative level of difficulty of the examinations. The adjusted scores were divided into two groups: the “no remediation” group (114 student scores) and the “remediation” group (111 student scores). The average adjusted percentage score for the “no remediation” group was 64.1% whereas that for the “remediation” group was 61.9%. The adjusted examination scores of the two groups were compared using an unpaired, two-tailed t-test. There was a small but statistically significant difference between the adjusted scores of the two groups (p<.01) (Figure 1). The average scores for both groups were well below the passing grade of 67%. However, a further reduction of 2.2% in the mean percentage score seems to have motivated the students to seek help from the one-on-one remediation process. These data suggest that at-risk students who scored “slightly better” are less likely to seek help than those who scored “slightly worse.”

Figure 1.

Lower Scoring Students Are More Likely to Seek Remediation. A total of 225 student scores below 70% on any of the first three examinations of the Cellular and Molecular Biochemistry (CMB) course from 2010 to 2014 were collected. Scores were normalized to adjust for level of difficulty. The adjusted scores were then divided into two groups based on whether the students went through the post-examination, one-on-on instructor remediation. Shown are the means (64.1 and 61.9) and the standard errors of the mean (SEMs) of the “No Remediation” and “Remediation” groups, respectively. *** p<.001, unpaired, two-tailed t-test.

For students who scored below 70% on a given examination, their scores were compared between the initial examination and the subsequent examination. Surprisingly, in both the “no remediation” and “remediation” groups, the scores on the second examination were significantly higher than those on the first examination, with a p value less than .001 in both cases (Figure 2). The adjusted score rose from a mean of 64.1% and a standard error of mean (SEM) of 0.6% to 72.7% (1.0%) for the “no remediation” group, and from 61.9% (0.6%) to 74.3% (0.9%) for the “remediation” group. The percentage of students who achieved at least a 10% increase in the adjusted scores in the subsequent examination was 59% for the “no remediation” group and 74% for the “remediation” group. These results suggest that a score below 70% alone motivates students to improve on the next examination regardless of whether they go through the post-examination, one-on-one instructor remediation.

Figure 2.

A Score Below 70% Alone Is Sufficient to Stimulate Student Improvement. A total of 225 pairs of student scores on two consecutive examinations of the CMB course were collected, which spans five consecutive years from 2010 to 2014. The normalized score pairs were divided into the “No Remediation” and “Remediation” groups. Shown are the means and the SEMs of the scores on both examinations in the two groups. *** p<.001, unpaired, two-tailed t-test.

To further assess the degree of improvement with and without remediation, the percentages of improvement in the adjusted score of each pair of two consecutive examinations from all qualified students were compared. They increased from 14.6% (1.9%) (without remediation) to 20.9% (1.8%) (with remediation), with a p value of .017 (Figure 3). This represents an increase in the magnitude of improvement by 43%. These results suggest that while self-motivated improvement after receiving a score below 70% may on its own increase the level of student performance on the next examination, post-examination, one-on-one instructor remediation further enhances the extent of such improvement.

Figure 3.

The Post-Examination, One-on-One Remediation Enhances the Extent of Improvement in Student Performance on the Next Examination. The percentage increase in the adjusted percentage scores from examination N to examination N+1 were calculated for the 225 pairs of student scores, and were divided into the “No Remediation” and “Remediation” groups. The means and the SEMs of the percentage increase in examination scores are shown. * p<.05, unpaired, two-tailed t-test.

The study covered a five-year period from 2010 to 2014. As the study progressed, the study investigators realized the importance of more complete documentation and started to take detailed notes of the remediation sessions. As a result, the details of each of the 22 one-on-one remediation sessions conducted in fall 2014 were well documented. A review of the notes from the 22 remediation sessions revealed that 95% of the students reported problems with learning habits, 36% reported insufficient amount of time to study, and 5% reported anxiety during the examination.

Students struggling with learning techniques and study habits were unable to accurately grasp key concepts, failed to think critically, or were less capable of applying knowledge to problem solving in a new context. These were often reflected in the inability of some students to critically read an examination question, digest and capture the key elements of the question, and provide an unambiguous and precise answer. Successful adjustment in learning habits should result in a long-term improvement in the students’ academic performance.

Another major contributor to poor student performance is insufficient study time. This can result from problems in study-life balance. Some students worked part-time while taking a full load of course work without realizing that their academic performance was jeopardized until it was too late. Occasionally, students had family issues that prevented them from spending the amount of time needed to succeed in the course. Reducing work hours or making life adjustments should improve their academic performance dramatically. A more common theme observed was that some struggling students were unable to tell the difference between essential concepts and “for your information” materials. They would spend an unnecessary amount of time on the latter, being unable to finish studying all the materials before the examination. In such cases, improving study habits is able to free up more time for studying essential materials.

DISCUSSION

Post-examination, one-on-one instructor remediation is effective in enhancing the performance of struggling students (examination score below 70%) in the CMB course. This is demonstrated by a statistically significant increase of 43% in the degree of improvement in student performance for students who underwent such a remediation as compared with those who did not. The improvement in student performance is not content-dependent as, for Exams 1 and 2, the subsequent examination covers completely different materials. Even for Exam 3, no more than half of the Final Exam cover old material tested in the previous three examinations. This translates into ∼17% of the material from a given remediation session. Moreover, as the remediation session usually covers only 5 out of at least 15 missed questions, less than 6% of the content-related advantage can be attributable to the increase in score from Exam 3 to the Final Exam.

A similar outcome was reported in a nursing program by Wiles in 2015, where students who did not pass the first course examination were strongly encouraged to meet with faculty for review within one week of the examination.¹⁰ During these reviews, faculty provided students with an individual summary report, a paper copy of the examination, and a feedback grid. Faculty and student reviewed each missed question, the student was prompted to determine the rationale for the correct answer and talk through their selection reasoning. Students were also asked to identify time management, classroom distractions and life issues that may have affected their examination performance. Such a remediation process increased the average performance of at-risk students by 6.5 percentage points.

In addition, the results from this study also provides insights to student learning behavior. One interesting finding is that struggling students with lower scores are more likely to seek help. In this course, the passing examination score is 67%. On average, the group of students who sought remediation had obtained a score 5 percentage points below the passing grade prior to the remediation, whereas the group of students who did not seek remediation had obtained on average a score only 3 percentage points below the passing grade. Possible embarrassment at the need for remediation on the part of the student might explain their unwillingness to meet with the course coordinator. However, consistent with a recent study,⁸ such unwillingness might have been outweighed by the urgent need for remediation as prompted by a lower score.

Another interesting observation from this study is that a score below 70% appears to be sufficient to stimulate improved student performance on the next examination regardless of whether they underwent the remediation. A similar phenomenon was also observed in another study involving a nursing program.¹⁰ Both underscore students’ inherent motivation to improve on their own once they realize the existence of a deficiency. Therefore, bringing the deficiency to the students’ attention plays a critical role in their self-driven improvement.

Consistent with the above notion, at-risk nursing students participated in a faculty-facilitated intervention program that included online adaptive quizzing program to reinforce course content, encouraging and motivational emails from faculty, and individual reminders to those who exhibited low participation. Following program completion, 91% of participants passed the course, progressed to the final semester and graduated. Providing further evidence that the intervention was beneficial, 80% of the program participants passed the higher standard NCLEX-RN examination on the first attempt.⁹

The quantitative analysis conducted in this study reveals that the remediation process enhances the degree of improvement in student performance. This might be accomplished through improving students’ learning habits, time management, and capability to cope with various life challenges. A similar remediation process has been implemented in other courses at this pharmacy school but its effectiveness has not been studied. As such a remediation process goes beyond the course content itself, it can also positively impact student performance in other courses.

The CMB course accounts for roughly 21% of all first-time course failures. Improving student performance in this course will likely have a significant impact not only on student learning but also on student retention. Early identification of at-risk students and timely intervention can not only improve the retention of P1 students but also enhance their academic performance in the subsequent years. Some P1 students struggling with the CMB course are not very well prepared in certain biology coursework. A study by McCall and colleagues showed that advanced biology coursework such as genetics, cell biology, immunology, biochemistry, and molecular biology are significantly associated with academic success in pharmacy school.¹¹ More studies need to be conducted before it can be ascertained that strengthening in these areas will improve student performance in the CMB course.

Based on these observations, a two-pronged approach should be used. On one hand, student awareness of their poor academic performance can be enhanced to stimulate their self-motivation. This study indicates that students have significant potential for self-motivated improvement. On the other hand, a well-designed intervention process can be instituted. Consistent with the results of this study, the latter approach will provide significant additional benefits to those students who are willing to seek help from their instructors. Proper combinational use of these two measures can dramatically enhance student performance and retention.

The above approach can also be used at multiple levels. Aside from the course-level monitoring and early intervention as described in this study, there is a school-level early intervention program implemented through the offices of the associate dean of academic affairs and the associate dean of student affairs. In addition, the school uses a university-level midterm grades reporting and early intervention program through the Academic Success Center.

Based on information gathered between 2010 and 2014, there are roughly 45 qualified cases for remediation per semester. Only half of these students sought remediation. On average, each remediation session took about 30 minutes, which translates into about 11 hours of time commitment on the part of the course coordinator. Should student participation increase to 100%, the total time commitment is estimated at 22 hours. While this is a significant time commitment, such a personalized remediation process, which focuses on study habits, time management, and stress control, offers the struggling students a unique opportunity to grow intellectually, personally, and emotionally in the first semester of their professional pharmacy program, and lays a strong foundation for their academic success in their subsequent semesters at the pharmacy school.

While this study demonstrates that the one-on-one, instructor remediation process enhances the academic performance of struggling students in the CMB course, leading to 43% more improvement as compared with those who did not participate in such remediation, the direct impact of such remediation on student retention has not been assessed. Future research aimed at addressing the latter issue will provide more information to guide educators in their effort to effectively improve student retention.

CONCLUSION

This study demonstrates that the post-examination, one-on-one instructor remediation is effective in enhancing the performance of struggling students in the CMB course. Both self-motivated efforts on the part of the students and the remediation process contribute to improved student performance. The individualized nature and the whole-person approach of the remediation process are expected to improve student retention, nurture their intellectual and personal growth, and help prepare them for a pharmacy career that transforms lives.