Abstract
Objective. To examine the evidence of the effectiveness of flipped classroom compared to traditional lecture.
Methods. Experimental and observational studies were included and obtained through searches of PubMed, Education Resources Information Center (ERIC), and Google Scholar. Publications from January 1, 2000 through July 1, 2017 were included. Studies were eligible for this research if: (a) the study compared student outcomes using flipped classroom versus lecture and (b) at least one outcome measure was final examination score or final course score. This analysis used a random effects model with weighted mean difference (WMD) as the outcome.
Results. Six studies were included in the qualitative synthesis and five were included in the quantitative synthesis. To date, there has only been one prospective randomized comparison of flipped classroom to lecture in student pharmacist education. When comparing final examination scores, there was no significant difference between flipped classroom and lecture based instruction. Only two studies examined the effect of flipped classroom compared to lecture on final course score. This analysis also found no significant difference.
Conclusion. Despite a lack of prospective randomized studies, findings from this meta-analysis suggest that flipped classroom may be associated with minimal gains in student knowledge compared to lecture. These findings are important because previous research has estimated that the flipped classroom requires more time to develop and implement. Future studies using prospective randomized designs need to be conducted before widespread adoption.
Keywords: flipped classroom, lecture, student outcomes
INTRODUCTION
The flipped classroom is an innovative teaching methodology that is growing in popularity.1 Formally introduced in 1998, the flipped classroom “flips” the lecture from occurring within the classroom to being delivered outside of class meetings. This design allows time for active learning and “homework” to be completed during class meetings.2-4 The in-class activities are meant to focus students on content application for them to gain a better understanding of the material being taught. These activities could be individual or collaborative and move the instructor from being a knowledge source to a facilitator of student learning.1,4
The flipped model has recently become a popular instructional model within medical, nursing, and pharmacy education because of the promise of enhancing student understanding of material through active learning.1,5,6 However, there are several practical problems associated with this technique, the most pressing of which is faculty time.5 Existing literature, although limited, suggests that flipping a class requires considerably more faculty time and resources in order to be effectively delivered.5,7 McLaughlin and colleagues estimated that in order to “flip” a class, a professor would have to invest 127% more time for course development and management. After initial development, the same group found that after the initial development time, the flipped classroom requires 57% more time to maintain compared to a lecture course. These problems may be compounded in health profession programs without access to graduate students, teaching assistants, and instructional designers.5 Further, the evidence is mixed on the effectiveness of the flipped classroom when compared to lecture in individual studies.1,8
Recent systematic reviews in medical and nursing education have shown that lecture may be just as effective as the flipped classroom.9,10 However, there have been no such reviews reported in pharmacy education. Further, there has not been a formal meta-analysis of the effectiveness of flipped classroom compared to lecture in any health profession. Therefore, the objective of this study is to answer this research question in student pharmacist education: what is the published evidence regarding the effectiveness of flipped classroom versus traditional lecture on student pharmacist educational outcomes?
METHODS
The authors developed and refined a review protocol using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.11 The following databases were searched to find eligible studies: PubMed, Education Resource Information Center (ERIC), and Google Scholar. Because educational literature is much more dispersed than biomedical literature, two investigators of this study examined the databases of the most widely read pharmacy and medical journals. The journals selected were Academic Medicine, Medical Education, Medical Teacher, American Journal of Pharmaceutical Education, and Currents in Pharmacy Teaching and Learning. The PubMed search strategy was: team-based learning [All Fields] OR “flipped classroom” [All Fields] OR “blended learning” [All Fields] AND (“pharmacy” [MeSH Terms] OR “pharmacy” [All Fields] OR “pharmacies” [MeSH Terms] OR “pharmacies” [All Fields]). The publication dates for the included studies were between January 1, 2000 and July 1, 2017. The search was restricted to studies published in English and studies that included student pharmacists. The databases were last searched for this study on August 11, 2017. Study authors were contacted for information that was not presented in the included study, most often the standard deviation of exam scores.
Eligible studies in this research consisted of experimental and observational studies. Studies were eligible for this research if: the study compared the flipped classroom to traditional lecture, the study population was student pharmacists, and at least one student outcome was final examination score, average examination grade, and/or final numeric course score. To ensure accurate comparisons, studies were excluded from this research if the study compared more than one year of the same course in the same study without differentiating between years. Additionally, to reduce the possibility that one study would carry more weight and bias the results of the meta-analysis, author lists, course subject matter, and years examined to the other included studies were compared. When there were multiple publications from the same study, the larger study was included. Studies were excluded from this research if: the examination(s) were a subset of larger examinations, did not compare a flipped classroom approach, examination questions were combined by Bloom’s taxonomy, which makes it impossible to compare two years of the same exam, or did not compare student pharmacist outcomes with traditional lecture. Finally, studies that examined a lecture or a very short sequence of lectures that were a small part of a larger course were excluded.
Two investigators independently screened study titles using the above inclusion criteria. Studies whose titles indicated the study may be eligible for inclusion then had the abstract screened. Finally, if the abstract indicated the study may be eligible for inclusion, then the full text was reviewed. After identifying eligible studies, the same investigators independently searched through the included studies’ reference lists to identify studies that were not included in the original database search. Disagreements were resolved through consensus.
The following information were extracted from each study: study design (experimental or observational), course topic, course length, number of semester credit hours, number of student pharmacists included in the study, student pharmacist outcome(s) measured, description of the intervention, and student population. The two investigators also met continuously throughout the data extraction process to ensure consistency and reliability of findings.
Two investigators independently conducted a bias assessment for each study using the risk of bias tool from the Cochrane Handbook for Systematic Reviews of Interventions.12 This tool assesses selection bias, detection bias, attrition, and reporting bias. In each category, studies were identified as having low risk of bias, high risk of bias, or unclear risk of bias. All studies were included in this research regardless of risk of bias to ensure the most comprehensive research of the literature possible. The investigators met to discuss the risk of bias to ensure consistent measurement and resolved discrepancies through discussion.
All analyses were conducted in Review Manager v5.3 (Copenhagen, Denmark). For both primary outcomes, a qualitative review was performed to summarize the evidence. The outcomes were stratified by type of outcome examined. Further, a random effects model was used to estimate the impact of flipped classroom versus lecture. A random effects model was chosen instead of a fixed effects model because of the investigators’ belief that due to various factors in education (eg, heterogeneity of courses and instructor teaching skill) there is no common effect size across all studies. For all analyses, a weighted mean difference (WMD) based on the mean final examination score (percentage out of 100%), and mean final numeric course score (percentage out of 100%) were used.
RESULTS
There were 208 titles, 55 abstracts, 23 full-text articles, and six studies screened for the qualitative synthesis (Figure 1).1,5,6,13-15 Six studies compared flipped classroom to lecture on final examination scores, of which two examined the final numeric course score. Among the six included studies, one was a prospective, parallel-group randomized controlled trial that compared flipped classroom to lecture.1 The number of student pharmacists in each study ranged from 70 to 316. All but two reports indicated that the lecture sections consisted of lectures accompanied by active learning.
Figure 1.
PRISMA Search Strategy.
The first study that examined the flipped model in pharmacy education that was included in the research was in 2014, while 2016 had the most publications examining flipped classroom in pharmacy education. The flipped classroom and lecture comparison has been examined in pharmaceutical calculations (two studies), drug information (one study), drug information and literature evaluation (one study), pharmaceutics (one study), and over-the-counter pharmacotherapy (one study). Table 1 presents the extracted information for each study and Table 2 presents the Cochrane Risk of Bias assessment.
Table 1.
Included Studies’ Data Extraction
Table 2.
Cochrane Risk of Bias
Six studies compared the final examination scores of student pharmacists using flipped classroom and lecture. One study used a prospective, parallel-group randomized controlled trial to evaluate flipped classroom to lecture.1 The other five studies used observational designs to compare the two models, so the decision was made to include the Anderson study in the qualitative synthesis but not the quantitative synthesis.1,5,6,24-26 Anderson and colleagues found that students in the flipped model performed significantly better on a high-stakes final examination by 9.5 percentage points (p=.02).1 However, when comparing the same outcome six months post-course completion, Anderson and colleagues found no significant difference between those in the flipped classroom and lecture (FC=82%, TL=78.2%, p=.19).1 Suda and colleagues found a decrease of 2.35 points on the final examination after instituting the flipped classroom in their drug information and literature evaluation course, which was not statistically significant.6 However, students scored 3 points higher on their overall course grade compared to the previous lecture course (p<.01). McLaughlin and colleagues found an increase of 2.71 percentage points on the final examination grade after implementing the flipped classroom (p<.01). Hughes and colleagues found that mean final exam scores increased by more than 4 percentage points (p<.05) after introducing the intervention, but found a small, non-significant decrease in final course grade (-0.4%).13 Cotta and colleagues showed that after implementing flipped classroom intervention, students scored significantly higher on the final exam than students in lecture (4.2 point increase).15 Finally, Wilson and colleagues found that students in the flipped model scored 6.1 percentage points higher than students in the lecture course (p<.01).14
Meta-analysis findings (Table 3 and Figure 2) indicate there was not a statistically significant difference on final examination scores comparing the two educational models in observational study designs (WMD=2.90, 95% Confidence Interval (CI): -0.02-5.81, p=.05). The I2 statistic (I2=91%) indicates high heterogeneity among the studies and the funnel plot indicates asymmetry in the analysis (results not shown). However, an ad hoc analysis which added the Anderson and colleagues’ study to the quantitative synthesis found a significant difference in favor of flipped classroom (WMD=3.44, 95% CI=0.60-6.27, p=.02).
Table 3.
Quantitative Synthesis Comparing Flipped Classroom to Lecture on Mean Final Exam Score
Figure 2.
Forest Plot of Quantitative Synthesis Comparing Flipped Classroom to Traditional Lecture on Mean Final Exam Score.
Similar to the meta-analytic results for final examination scores, there was no significant difference (Table 4 and Figure 3) between final numeric course scores when comparing the flipped classroom to lecture (WMD=1.26, 95% CI:-2.07-4.59, p=.46). The I2 statistic (I2=94%) indicates high heterogeneity among the studies and the funnel plot indicated asymmetry (results not shown).
Table 4.
Quantitative Synthesis Comparing Flipped Classroom to Traditional Lecture on Final Numeric Course Score
Figure 3.
Forest Plot of Quantitative Synthesis Comparing Flipped Classroom to Traditional Lecture on Final Numeric Course Score.
DISCUSSION
This meta-analysis is the first to examine the effectiveness of the flipped classroom compared to lecture and the first to also examine student pharmacist educational outcomes. Final examination scores and final numeric course scores were considered. There was no statistically significant difference for either outcome among included studies. However, there was a consistently small positive overall effect for using flipped classroom compared to lecture in pharmacy education, usually representing a 1.3% to 3.4% increase in student pharmacist performance in individual studies. Examination of study heterogeneity and the funnel plot on both outcomes should moderate the inferences that can be made about the effectiveness of the flipped classroom in pharmacy education. Systematic reviews in medical and nursing education have found similar results to our study.9,10 The strength of our findings however should be examined with caution, as only one study used a prospective, randomized controlled trial to examine student pharmacist outcomes and this study was not included in the initial analysis.1
The results of educational studies in the health sciences and undergraduate literature assessing the effectiveness of the flipped classroom may be disappointing to pharmacy educators who may have invested significant time and resources while expecting a greater return on their investment. Previous studies in undergraduate research similarly found that there was no statistically significant effect between lecture with active learning and the flipped classroom.16 All but two studies in our research indicated that active learning was used in conjunction with lecture. Similar to Jensen and colleagues’ findings, this analysis found that there does not appear to be a significant difference between lecture and flipped classroom in pharmacy education, in terms of student outcomes.
Other factors may attenuate the potential gains from more active learning in the flipped classroom, such as student preparation. Student preparation is a key moderator of the effectiveness of the flipped classroom because the model assumes that students have prepared for the material by completing and reviewing all assigned pre-work. If students fail to prepare, the instructor loses class time (and active learning time) by having to review or teach those concepts that were already addressed through the pre-work. Moreover, it is unlikely that students will be able to derive much benefit from active learning exercises with little to no background knowledge.
While results comparing flipped classroom to lecture are inconclusive based on student pharmacist outcomes, the concerns about faculty and student time remain. In almost all pharmacy schools and colleges, faculty are required to conduct research, engage in clinical care (if applicable), and provide service to the profession and school. If a faculty member wishes to “flip” the classroom, the school’s administrators would likely need to accept trade-offs in terms of reduced research productivity (including less grant funding) and reduced clinical and service time. This is especially true in regards to promotion and tenure. Alternatively, schools could invest in additional faculty or a combination of faculty/instructors and graduate assistants in order to manage faculty teaching load.
This study has limitations. First, the funnel plot for both final examination score and final course score shows asymmetry. This is the result of multiple factors, chiefly the low number of included studies and high heterogeneity. However, it should be recognized that with the limited number of studies in the meta-analysis, the power to detect asymmetry is low.17 Second, the results of this study confirm previous studies in science, engineering, and mathematics wherein the flipped model may be more effective in smaller classes, as evidenced by the greatest difference in scores in the smallest studies, also influencing the lack of symmetry in the funnel plot.18 This study did not examine student preferences with regards to flipped classroom compared to lecture. The literature shows that students report being receptive to the concept of the flipped classroom, but the same concerns from our analysis (workload and lack of time to prepare) are consistently reported by students across multiple studies.5,13 There is speculation that traditional assessment methods may not accurately reflect gains from the flipped classroom, which may cause the reported effect to be understated.19,20 However, adding essays and other long-answer types of questions to examinations only adds to instructor effort. Another limitation is the type of courses that have so far compared traditional lecture to flipped classroom have been mostly foundational with one study in a therapeutics course (over-the-counter pharmacotherapy). Therefore, it cannot be ruled out that the flipped classroom may be more effective in therapeutics courses. Future research should examine if the flipped classroom is more effective than traditional lecture in therapeutics courses. Finally, some lecturers may be effective teachers and their teaching may be more effective than others regardless of the teaching modality.
CONCLUSION
This study is the first meta-analysis comparing the flipped classroom to lecture in any health profession. Much like similar studies in medical and nursing education, the flipped classroom may result in small gains in student learning in pharmacy education. The extensive amount of faculty time to develop and implement a flipped course compared to lecture further exacerbates the controversy surrounding utilization of this technique. More research using prospective, randomized designs with larger classes should be conducted before widespread adoption of this teaching methodology.
ACKNOWLEDGMENTS
The authors would like to thank Avishek Mallick, PhD, of Marshall University for assistance with statistics, the authors who were contacted for additional information and provided responses, Ms. Lisa Harrell and Dr. Mark Bush from Wingate University School of Pharmacy for their assistance with table and figure layout, and Debra Hargett, MLIS, for her assistance with helping develop the search.
REFERENCES
- 1.Anderson HG, Jr., Frazier L, Anderson SL, et al. Comparison of pharmaceutical calculations learning outcomes achieved within a traditional lecture or flipped classroom andragogy. Am J Pharm Educ. 2017;81(4) doi: 10.5688/ajpe81470. Article 70. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Brame CJ. Flipping the classroom. Vanderbilt University Center for Teaching website.https://cft.vanderbilt.edu/guides-sub-pages/flipping-the-classroom/ Published 2013. Accessed July 31, 2017. [Google Scholar]
- 3.Lambert C. Twilight of the lecture. Harvard Magazine website. http://harvardmagazine.com/2012/03/twilight-of-the-lecture Published 2012. Accessed July 31, 2017.
- 4.Miller A. 5 best practices for the flipped classroom. https://www.edutopia.org/blog/flipped-classroom-best-practices-andrew-miller Published Feb. 24, 2012. Accessed July 31, 2017.
- 5.McLaughlin JE, Roth MT, Glatt DM, et al. The flipped classroom: a course redesign to foster learning and engagement in a health professions school. Acad Med. 2014;89:236–243. doi: 10.1097/ACM.0000000000000086. [DOI] [PubMed] [Google Scholar]
- 6.Suda KJ, Sterling JM, Guirguis AB, Mathur SK. Student perception and academic performance after implementation of a blended learning approach to a drug information and literature evaluation course. Curr Pharm Teach Learn. 2014;6(3):367–372. [Google Scholar]
- 7.Koo CL, Demps EL, Farris C, Bowman JD, Panahi L, Boyle P. Impact of flipped classroom design on student performance and perceptions in a pharmacotherapy course. Am J Pharm Educ. 2016;80(2) doi: 10.5688/ajpe80233. Article 33. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Stewart DW, Panus PC, Hagemeier NE. An analysis of student performance with podcasting and active learning in a pharmacotherapy module. Curr Pharm Teach Learn. 2013;5(6):574–579. [Google Scholar]
- 9.Chen F, Lui AM, Martinelli SM. A systematic review of the effectiveness of flipped classrooms in medical education. Med Educ. 2017;51(6):585–597. doi: 10.1111/medu.13272. [DOI] [PubMed] [Google Scholar]
- 10.Betihavas V, Bridgman H, Kornhaber R, Cross M. The evidence for “flipping out”: a systematic review of the flipped classroom in nursing education. Nurs Educ Today. 2016;38:15–21. doi: 10.1016/j.nedt.2015.12.010. [DOI] [PubMed] [Google Scholar]
- 11.Moher D, Liberati A, Tetzlaff J, Altman DG. The PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264–269. doi: 10.7326/0003-4819-151-4-200908180-00135. [DOI] [PubMed] [Google Scholar]
- 12.Assessing risk of bias in included studies. Cochrane Methods website. http://methods.cochrane.org/bias/sites/methods.cochrane.org.bias/files/public/uploads/6.%20Assessing%20risk%20of%20bias%20in%20included%20studies%20v1.0%20Standard%20author%20slides.pdf Accessed July 31, 2017.
- 13.Hughes PJ, Waldrop B, Chang J. Student perceptions of and performance in a blended foundational drug information course. Curr Pharm Teach Learn. 2016;8(3):359–363. doi: 10.1016/j.cptl.2016.02.013. [DOI] [PubMed] [Google Scholar]
- 14.Wilson JA, Waghel RC, Free NR, Borries A. Impact of team-based learning on perceived and actual retention of over-the-counter pharmacotherapy. Curr Pharm Teach Learn. 2016;8(5):640–645. [Google Scholar]
- 15.Cotta KI, Shah S, Almgren MM, Macias-Moriarity LZ, Mody V. Effectiveness of flipped classroom instructional model in teaching pharmaceutical calculations. Curr Pharm Teach Learn. 2016;8(5):646–653. [Google Scholar]
- 16.Jensen JL, Kummer TA. d M Godoy PD. Improvements from a flipped classroom may simply be the fruits of active learning. CBE-Life Sci Educ. 2015;14(1):1–12. doi: 10.1187/cbe.14-08-0129. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Sterne JAC, Sutton AJ, Ioannidis JPA, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomized controlled trials. BMJ. 2011;343 doi: 10.1136/bmj.d4002. d4002. [DOI] [PubMed] [Google Scholar]
- 18.Freeman S, Eddy SL, McDonough M, et al. Active learning increases student performance in science, engineering, and mathematics. Proc Natl Acad Sci USA. 2014;111(23):8410–8415. doi: 10.1073/pnas.1319030111. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Persky AM, McLaughlin JE. The flipped classroom – From theory to practice in health professional education. Am J Pharm Educ. 2017;81(6) doi: 10.5688/ajpe816118. Article 118. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.White PJ, Naidu S, Yuriev E, Short JL, McLaughlin JE, Larson IC. Student engagement with a flipped classroom teaching design affects pharmacology examination performance in a manner dependent on question type. Am J Pharm Educ. 2017;81(9) doi: 10.5688/ajpe5931. Article 5931. [DOI] [PMC free article] [PubMed] [Google Scholar]