Student Performance in a Pharmacotherapy Oncology Module Before and After Flipping the Classroom

Abstract

Objective. To determine if a flipped classroom improved student examination performance in a pharmacotherapy oncology module.

Design. Third-year pharmacy students in 2012 experienced the oncology module as interactive lectures with optional case studies as supplemental homework. In 2013, students experienced the same content in a primarily flipped classroom. Students were instructed to watch vodcasts (video podcasts) before in-class case studies but were not held accountable (ie, quizzed) for preclass preparation. Examination questions were identical in both cohorts. Performance on examination questions was compared between the two cohorts using analysis of covariance (ANCOVA), with prior academic performance variables (grade point average [GPA]) as covariates.

Assessment. The students who experienced the flipped classroom approach performed poorer on examination questions than the cohort who experienced interactive lecture, with previous GPA used as a covariate.

Conclusion. A flipped classroom does not necessarily improve student performance. Further research is needed to determine optimal classroom flipping techniques.

INTRODUCTION

Flipping the classroom is an educational approach that aims to transform the learning process from in-classroom knowledge acquisition followed by out-of-classroom knowledge use to out-of-classroom knowledge acquisition followed by in-classroom knowledge use and can be facilitated by faculty members.¹ This approach is increasingly used in all educational settings, including pharmacy education.^2-4 Despite increased use, until recently the majority of data supporting this approach in pharmacy education is based on students’ perceptions of the approach.⁵

When academic performance is evaluated, the results are not consistent. Wong and colleagues evaluated examination scores after flipping the classroom in three different courses (basic sciences, pharmacology, and therapeutics) covering cardiac arrhythmias. Student examination scores were significantly higher after the flipped classroom approach in pharmacology and therapeutics, but not in basic sciences.⁶ Persky and Dupuis found that flipped and active-learning teaching methods were associated with better academic performance in foundational and clinical pharmacokinetic courses.⁷

The Accreditation Council for Pharmacy Education (ACPE) Standards have evolved to stress more inclusion of active-learning strategies, and active learning has been adopted in almost every college and school of pharmacy in the United States.^8,9 To that end, the East Tennessee State University (ETSU) Bill Gatton College of Pharmacy has supported faculty members’ desire to incorporate more active-learning strategies into the classroom. In recent years, faculty development sessions that describe the pedagogy of active-learning strategies have been offered to faculty members.

The objective of the study was to determine student performance in the oncology module before and after flipping the classroom in consecutive classes. Assessment of this flipped approach was important institutionally because a previous attempt to utilize more active-learning strategies did not show a benefit in student learning and appeared to harm students in the lower fiftieth grade point average (GPA) percentile.¹⁰ The majority of the oncology module was flipped in hope of improving student learning. The oncology module occurs during the final pharmacotherapy course in the spring semester of the third professional (P3) year.

DESIGN

Similar to other flipped classroom work published, a design experiment methodology was used.¹¹ The oncology module was delivered to P3 students in 2012 using an interactive lecture format. Lectures incorporated anywhere from 2-5 audience response questions (TurningPoint) to allow for anonymous classroom assessment and rapid feedback on key points from the lecture. All lectures were delivered in a large classroom setting that allowed for laptop use and Internet access. Lecture capture technology was not used, but students were allowed to use digital voice recorders. Lecture material was supplemented with two formative quizzes administered via the online learning support system, Desire-2-Learn (D2L), v8.3 (Kitchener, ON, Canada). The practice quizzes covered lecture objectives at the Bloom’s Taxonomy levels of knowledge and comprehension.¹² Each 10-question quiz could be taken an unlimited number of times and was drawn from a bank of 25 questions to allow students to repeatedly self-test comprehension of lecture material.

Nine practice cases also were available on D2L. The practice cases were designed to allow students to self-study at the level of application and analysis.¹² These cases were ungraded homework. In lieu of grading the homework, answer keys were posted from 4-7 days after presentation of the material. Students were free to self-assess their own performance on practice cases with subsequent review of the keys.

The oncology module was delivered to P3 students in 2013 using a mostly flipped approach. Of the 15 hours of lecture material, 10 hours were flipped. The introductory two hours on basic chemotherapy concepts, and the final three hours covering traditionally difficult concepts of solid organ and hematopoietic stem cell transplantation, were provided exactly as they were in 2012. The remaining 10 hours of interactive lecture were condensed to eight hours of 25-50-minute vodcasts without cutting meaningful content.

Faculty preparation for the vodcasts was minimal because they were developed from materials used for interactive lecture in previous years. It took approximately 10 hours to create the vodcasts using Camtasia Studio, v8.0 (TechSmith, Okemos, MI, normal cost $270). These were uploaded to the university’s iTunesU website. Students were assigned to watch the vodcasts prior to class but were not quizzed on the material or otherwise held accountable for this assignment. As with the 2012 cohort, two formative online quizzes could be taken an unlimited number of times to allow self-testing.

To balance the addition of eight hours of preclass work, four hours of face-to-face time were eliminated. This resulted in a full six hours of face-to-face interaction facilitating case studies over the material from the vodcasts. As with the 2012 online case studies, the face-to-face case studies were designed for students to learn at the levels of application and analysis. In most instances, practice cases used in 2012 were repurposed for the 2013 cohort with faculty facilitation. Unlike the 2012 students who had the option of completing the case studies independently and self-facilitating learning once the keys were posted, the 2013 students all experienced faculty facilitated, yet ungraded, cased-based instruction.

Both cohorts of students took the same end-of-module examination, which consisted of 15 matching questions asking students to associate chemotherapy drugs with unique toxicities, and 60 multiple-choice questions. The multiple-choice questions were designed to assess student learning at the knowledge, comprehension, application, and analysis levels of Bloom’s Taxonomy.¹² For assessment of the flipped approach, only the 46 examination items pertaining to flipped material were evaluated in the analysis. The remaining 29 nonflipped material examination items, which included the 15 matching questions, were evaluated separately.

As part of the college’s assessment plan, individual faculty members are encouraged to submit custom summative student evaluation questions for students. While student evaluation of faculty instruction is strongly encouraged, it is not required. For the 2013 cohort, custom evaluation questions asked students about their perceptions of vodcasts and in-class case studies. Student summative evaluation performance using a primarily flipped classroom was compared to previous student performance on the same examination a year earlier after interactive lecture delivery of content. Assessments were administered with paper and pencil in a proctored setting using Scantron to analyze scores. Approval for this study was obtained from the ETSU Institutional Review Board (IRB). All data were extracted from D2L and organized using Excel 2007.

For the practice quizzes, the following independent variables were collected for each student: number of attempts, high and low scores, scores on first and last attempt, range between high and low score, and number of days the practice quiz question pool was used. Grade point average (GPA) in undergraduate school, cumulative GPA within the college of pharmacy (COP), cumulative GPA in previous pharmacotherapy courses, and total score for the Pharmacy College Admissions Test (PCAT) also were included in the database. All data analyses were completed using SPSS, v19 (IBM Inc., Armonk, NY).

Oncology module examination scores, undergraduate GPA, COP GPA, pharmacotherapy series GPA and total PCAT scores were analyzed by one-way analysis of variance (ANOVA), with testing for homogeneity of variances. The cohort year was used as the grouping variable. Analyses of covariance (ANCOVA) were conducted to determine if cohort performance was influenced by extraneous factors (GPA and PCAT scores).

All analyses were conducted with a single covariate. Finally, the oncology module examination scores for both cohorts were rank-ordered from low to high score regardless of year. The rank order was divided into upper and lower fiftieth percentiles. After separating, the cohorts were again broken into 2012 and 2013 cohorts. Two group comparisons of the oncology module examination scores were then conducted separately on the upper and lower fiftieth percentiles with cohort year as the grouping variable. All data are presented as average plus/minus standard deviation.

EVALUATION AND ASSESSMENT

Students who received the interactive lecture oncology module in 2012 had a mean examination score of 89.5 (SD=6.8) of subsequently flipped material; the 2013 students had a numerically lower mean examination score of 88.0 (6.9) on the flipped material (Table 1). Potential confounders that might affect the outcome of examination scores for the oncology module also were examined with ANOVA. There were no differences in the undergraduate GPA, COP GPA, or GPA in previous sections of the pharmacotherapy curriculum. However, 2013 cohort PCAT scores were higher than those of the 2012 cohort (p=0.003).

Table 1.

One Way ANOVA on Variables for Both Cohorts (2012 n=72; 2013 n=76)

Using the ANOVA results, an ANCOVA was performed to adjust examianation scores for undergraduate GPA, composite COP GPA, and pharmacotherapy GPA (Table 2). After adjusting for student academic performance while in pharmacy school, students receiving the flipped classroom approach did significantly worse (p=0.02) compared with the previous cohort. Similarly, adjusting for academic performance within the pharmacotherapy course series revealed that students experiencing the flipped classroom performed poorer (p=0.01) than students receiving the same material via interactive lecture. Because ANOVA revealed a significant difference between the cohorts, PCAT scores were not used in the ANCOVA analysis.

Table 2.

Comparison of Cohorts’ Examination Performance on Flipped Material by ANCOVA

There was no difference (p=0.82) in student performance on the 29 examination items covering nonflipped material between the 2012 [85.9 (8.8)] and 2013 [86.2 (9.9)] cohorts. There was no significant difference in the oncology examination scores between the 2012 and 2013 cohorts in either the upper fiftieth percentile [p=0.49, 94.2 (3.6), n=42; 93.9 (3.3), n=33], or the lower fiftieth percentile [p=0.94, 83.4 (4.8), n=32; 83.5 (5.5), n=43], respectively.

Summative teaching evaluations for students in the 2013 class included two questions about the oncology module. Thirty students (39.4%) completed parts of this evaluation. Using a 5-point Likert scale, students were asked the degree to which they agreed that “The vodcasts were helpful to my learning oncology material.” Six strongly agreed and seven agreed (43.3% agreement) with the statement, while two strongly disagreed and six disagreed (26.7% disagreement). Seven students (23.3%) answered neutral and two students did not respond to the item. Using the same response scale, when asked the degree to which they agreed that, “Case discussions in class were helpful to my learning of oncology material,” 12 strongly agreed and 13 agreed (83.3% agreement) with the statement, while one strongly disagreed and two disagreed (10% disagreement). One student answered neutral (3.3%), and one student not answer the question.

Despite the low response rate on the summative evaluation, 17 of the students provided commentary and qualitative feedback. In general, students seemed to appreciate the case discussions: “Completing the cases in class was helpful;” “The cases and in-class exercises were very informative and fun as well”). Students had mixed feelings about the vodcasts. Some enjoyed the vodcasts for their repeat viewing availability, but also felt they were “too long and detailed.” Others said the “vodcasts are fine, but an interactive lecture is preferred.”

There were numerous comments made about not liking “having lectures outside of class” and the added time commitment. Perhaps the most constructive summary of feedback came from one student comment, “I have mixed emotions about the podcasts. The only nice part about them was that I could rewind and re-listen. … They were beneficial when I finally had time to sit down and listen to them all. However, I almost never came to class fully prepared because I just did not have enough time to listen to all of the podcast lectures before class.”

DISCUSSION

Discrepancies exist in the literature over the effect of flipped classrooms on student learning. While the general consensus is that active learning is beneficial, this study provides the second data set from the institution with negative results for flipping the classroom. While flipping the classroom can be a reasonable tool to engage students and promote higher-level critical-thinking skills, it is important for faculty members to appropriately implement the approach.

The oncology module was flipped following Pierce and Fox’s publication describing improved student learning with flipping the classroom in a renal pharmacotherapy module; however, the majority (67%) of the oncology module was flipped, compared with only ∼17% of the renal module (four of 23 hours) in their analysis.¹¹ The extreme change from 0% flipped to 67% flipped is just one of several explanations for the apparent negative effect of the flipped classroom in our experience.

In both this analysis (flipping the oncology module) and the prior work from the institution (using active-learning strategies in a cardiovascular pharmacotherapy module), faculty members failed to ensure students completed the out-of-class assignments.¹⁰ In this higher-level learning environment, students who come in without proper preparation will not learn at the same level as those who do. Unfortunately, we did not collect data on how many students watched the vodcasts prior to class. In fact, data from our previous research demonstrated a positive correlation between previous poor academic performance and likelihood to not perform well in the flipped environment when the class was rank ordered and separated into upper and lower fiftieth percentiles based on programmatic GPA.¹⁰

While we did not observe this correlation in the current analysis, this should be a significant concern when requiring students to prepare for high-level discussions and then not holding them accountable for completing the work. Consideration must be given to ensure students in the lower-performance percentiles, who may be less likely to complete ungraded assignments, are not left behind.

Additionally, in the previous analysis, a difference in examination performance was noted both before and after the ANCOVA; however, in the current analysis, only after accounting for the covariates of COP GPA and pharmacotherapy series GPA did the difference become significant. The 2013 cohort’s significantly higher average PCAT score could bias these results. If bias were present, one would expect the 2013 cohort to perform as well or better than the 2012 cohort. However, since the 2013 cohort performed poorer, such bias is not suspected.

Differences that exist between this current analysis and the prior analysis, which could have affected the results, are the course material and instructor. Oncology is generally viewed by many students to be more difficult than cardiovascular material, likely because of the lack of prior exposure to the highly specialized area of oncology. Also a different instructor taught the oncology material than the cardiovascular material without using a consistent standardized approach in the two experiences. In fact, this was being done simultaneously without knowledge between instructors, who only collaborated after the fact.

That the same negative effect was seen on examination score, despite correcting for baseline intellect using the covariates of prior course GPA and prior college GPA, provides strong evidence that flipping the classroom without preclass preparation accountability, while well perceived by the students, may not result in higher performance on examinations.

To maximize resources, vodcasting only “core” material increases the longevity of the media material. This approach can be done with minimal time and cost expenditure if adapting old material and using trial versions of vodcasting or podcasting software. This may be especially true in the area of oncology, which has seen dramatic changes and new drug approvals, especially pertaining to the treatment of metastatic melanoma, prostate, and breast cancers. Subsequent oncology modules have used shorter vodcasts (10 to 15 minutes) over more static material (eg, pathophysiology and presentation of disease, risk factors, burden of disease). These vodcasts were created using Tegrity Campus 2.0 (Tegrity USA, Santa Clara, CA), which was licensed for use campus-wide by the university.

Additionally, the latest attempts to use vodcasts in the oncology module have required students to take an electronic quiz via D2L at the beginning of class to hold them accountable for the preclass viewing. This should ensure student preparation for class, which will allow better assessment of flipping the classroom in oncology. Announced quizzes following preclass lecture recordings have been used in successful flipped classroom approaches covering cardiac arrhythmias, with significant improvements in student examination performance in a therapeutics course.⁶

A renewed appreciation for the interactive lecture format and its cost-benefit ratio has also been considered.¹³ Moving forward, achieving the appropriate balance of interactive lecture and facilitated case discussion will be sought by carefully selecting learning objectives that require greater depth of knowledge. For example, learning which breast cancer patient should receive adjuvant chemotherapy requires consideration of many patient and disease state factors, include tumor receptor status, patient age, comorbid conditions, and other risk factors for disease recurrence. This requires an in-depth and nuanced discussion more amenable to a facilitated case-discussion. Conversely, the initial treatment of metastatic prostate cancer is more straightforward and thus conducive to interactive lecture.

SUMMARY

Flipping the classroom as a technique to promote active-learning approaches in the classroom does not guarantee improved student examination performance. Pharmacy educators considering flipping the classroom should use an “evidence-based” approach when planning flipped classroom experiences. The results of our flipped classroom in an oncology module suggest that the composite of flipping a majority of a module or course at one time, not holding students accountable for out-of-class assignments, and flipping traditionally difficult subject matter does not improve academic performance compared to interactive lecture.