Flipping a Single Lecture in a Survey Course to Active Learning: Do the Benefits Justify the Costs?

Abstract

Health education has seen a surge of interest in active learning strategies like the flipped classroom. In response to the need for physical distancing in the age of COVID-19, schools are rapidly shifting to web-based and video technology, sometimes without being able to predict the outcomes of this change. The objectives of this pilot experiment were to (1) compare active learning (AL) methods versus traditional lecture for transmitting and retaining knowledge in the introductory pre-clinical medical school curriculum and (2) weigh whether the costs required to flip instruction were justified by learning gains. The authors took a 2 h lecture for first-year medical students and converted half of it into an AL format. In-person lecture and active learning groups were compared in terms of student knowledge at pre-intervention, immediately post-intervention, and 6 months post-intervention. Costs for first-time delivery and anticipated costs for repeat delivery of each format were calculated. Students’ gains in knowledge increased in both groups, though more by lecture (control) than via AL. Delivering a single hour of new AL costs 3.4 times that of a new lecture. Repeat offerings of the AL intervention were estimated to cost 5.4 times that of the repeat lecture. The 1 h AL session was less effective than the 1 h lecture for knowledge acquisition and retention at 6-month follow-up. The AL was more expensive to produce and to repeat. Future research needs to evaluate the impact of AL with a larger N, control group, structured faculty/resident procedures, and assessment of gaining and applying attitudes and skills in addition to knowledge.

Introduction

Active learning (AL) strategies — based on adult learning theories and philosophies (i.e., andragogy) — have become a fundamental part of undergraduate and graduate medical education (Prober & Khan, 2013; Mehta et al., 2013; Thompson et al., 2015; Liu & Beaujean, 2017). These strategies go by the names of case-, team-, and problem-based learning (PBL), or flipped classroom, among others. These formats share a number of overlapping elements, such as pre-class preparation, group process, and in-the-moment problem solving (Savery, 2006; Jensen et al., 2015; Krupat et al., 2016). AL has shown impressive results in student gains in life-long learning and professionalism (Koh et al., 2008; Thrall et al., 2016; Liu & Beaujean, 2017). In addition to remembering and understanding, these activities involve applying, analyzing, and evaluating levels of Bloom’s taxonomy (Bloom, 1956). In recent years, AL curricula and methods increasingly incorporate web-based and video technology, partly to appeal to a generation of learners who habitually teach themselves via digital sources such as YouTube tutorials (Kamei et al., 2012) and who are experiential learners (Flynn et al., 2015).

AL methods are commonly used for students by educators in psychiatry and behavioral health (Thrall et al., 2016; Skokauskas et al., 2012; Morreale et al., 2012; Liu & Beaujean, 2017; Madson et al., 2020). Psychiatry clerkships emphasize skills (e.g., interviewing and therapeutic engagement) and attitudes (e.g., compassion and teamwork) as much as knowledge (Morreale et al., 2012; Skokauskas et al., 2012; Thrall et al., 2016). With AL and PBL, students independently seek out basic science knowledge and then integrate it with clinical reasoning while discussing a case with peers (French et al., 2020; Koh et al., 2008; Ramnanan & Pound, 2017). The teacher serves as a facilitator and guide rather than only as a source of knowledge (Peters et al., 2000; Savery, 2006). These AL methods in undergraduate medical education parallel future practice-based learning opportunities in continuing medical education (CME) after they complete their medical training (see Table 1). Increasingly, quality CME incorporates ideas from adult learning theory, incorporates AL methods, and is facilitated by technology (Cullen et al., 2019).

Table 1.

Components and outcomes of active learning (AL) better prepare learners for professional technology-based learning than lecture

Component of active learning	AL qualities and outcomes	Lecture qualities and outcomes compared to al	Professional technology-based learning tools
Flipped classroom learning
Cognitive	Facilitates application and creativity Emphasizes longitudinal problem-solving and continuity	Transmits ideas rather than application Emphasizes cross-sectional learning	Cues and queues processes to focus and continue tasks Promotes clinical decision support
Social	Engages in meaningful interaction and presence Helps learners at all levels with skills and role development	Uses individual rather than a team-based focus Fails to help different learner levels within and across professions	Emphasizes social- and team-centered practice Facilitates engagement across distance
Physical	Helps participants build, learn and share resources in session	Offers fewer formats for interactive activities; often causes fatigue	Offers concrete feedback on behavior and tracks longitudinally
Emotional	Builds on interests and passions Provides support, empathy & coping	Offers less flexibility to adapt to interests Gives fewer opportunities to share and receive	Gives individual options and virtual team options for engagement and communication
Self-directed learning
Case-based reading	Promotes self-efficacy and knowledge on challenging issue	Case discussions are brief, rote, and usually superficial	Journals offer case series, and continuing medical education (CME)
Individual journal clubs, webinars	Applies knowledge and resources on a new issue or recap of one	Focuses on knowledge with limited toward skill and attitude adjustment	American Psychiatric Association and subspecialty organizations offers
Peer consultation: individual and/or group	Helps develop skills to seek advice perspective; shared understanding	Does not offer these unless small group activities and tables are built in	Commonly offers advice via e-mail, e-consult, telephone, or video
In-depth discussion and/or course	Scope is limited but applies knowledge, tips, and best practices in-depth	Does not offer much time to apply to knowledge to cases or develop skills	Online courses offer interactive formats and topics in-depth
Self-and peer-assessment/reflection
Assessment of errors or things to do differently	Uses pre- and post-class time to weigh decisions and outcomes and shift approach	Discusses or outlines examples of bad outcomes	Employs electronic health record alerts and offers feedback on performance
Feedback (simulated) from peers, patients, and faculty	Offers direct engagement (e.g., simulated patients) to inform care decisions and learn interpersonally	Discusses concepts, practices, and approaches to therapeutic relationship	Avails video case conferences, virtual and augmented reality (in the future)
Examination of clinical practices	Verifies attitudes and skills related to applying, analyzing, and evaluating	Focuses on understanding more than applying, analyzing, and evaluating	Offers Psychiatry In Practice Examination (PIPE) from American Board of Psychiatry and Neurology
Data-based feedback	Provides individual and aggregate outcomes on decisions to learn and change practices	Does not connect the relationship of knowledge to skills and other clinical dimensions	Avails decision support, electronic health record tools in time

Integrating technologies into AL may provide further benefits beyond text-based cases and group discussions. Mobile phones, tablets, and even video games may be used for role play exercises, collaboration, and engaging users in learning (Hilty et al., 2019; Collis & Winnips, 2002). AL and technology also make dimensions of learning theories and their common denominators more explicit (e.g., assumptions about learning) (Flynn et al., 2015). The use of technologies early in medical school may promote competencies for using video, social media, mobile health, and asynchronous technologies in clinical care (Maheu et al., 2019; Hilty et al., 2020), particularly in the COVID-19 era. The use of these technologies also inculcates learners into a culture of e-learning for lifelong learning via webinars and CME/maintenance of certification. Creating an e-culture for learning includes fitting the technology to medical expertise, practice, and professional roles; technology training on hardware and software; adjustments for learning, teaching, and evaluation based on learning theory; and assessment and development of learning styles and competencies (Fig. 1).

Fig. 1.

Technology’s role in health education curricula: key steps to build an e-culture for learning

Paper-based AL methods have costs in terms of faculty training, time facilitating, meeting space for groups, technology, and creating an AL experience (French et al., 2020; Koh et al., 2008; Ramnanan & Pound, 2017). However, technology-based AL comes with additional costs, as it may require substantial help from an audiovisual (AV) technician to investigate and select the best software, help create videos or other interactive materials, and stage/deliver on the day of small groups (Abdelkhalek et al., 2010).

The authors of this paper undertook an experiment to investigate two questions. First, are AL methods that use technology as effective at transmitting and retaining knowledge for child development topics as a traditional lecture? Second, are the costs of developing and delivering AL methods justified by the benefits they bring in improving student knowledge?

Methods

Context

The Keck School of Medicine (KSOM) is a private allopathic medical school situated just outside of the metropolitan area of downtown Los Angeles. This pilot experiment involved 2 h of lecture on child and adolescent development embedded in an annual survey course for all members of the first-semester medical students. The content of the survey course included genetics, statistics, and human development over the lifespan. In its usual form, about 90% of the didactic content in this course was traditional lecture with assigned readings, and 10% small group discussion sections. This 2 h lecture had been part of the MS1 curriculum for over 15 years.

Participants

All 189 members of the KSOM first-year medical student class of 2015–2016 were invited to participate. This class consisted of 48% women and 52% men. Attendance at these didactics was voluntary (as was the case for all lectures). Participation in the study was voluntary and students were told that quiz results would not impact their exam grade. Making participation in the study mandatory would not have been approved either by the curriculum committee or the IRB of KSOM. The recruitment effort was coordinated by the medical student researcher on this research team (CJ) through public announcements at town hall meetings and large gatherings as well as reminders delivered via SOM email account to students; no incentives were offered. In order to reassure students that their quiz results were truly anonymous, no demographic data on participants was collected.

Procedures and Measures

Design, Intervention, and Outcome

This study is a randomized comparison study with two groups and a 6-month follow-up specific to each group to examine the impact of different modalities (in-person lecture vs. AL) on students’ immediate recall and their knowledge retention on delayed recall after 6 months. The control intervention was a traditional in-person lecture in which the only technological component was PowerPoint slides. The active intervention was small groups, facilitated by psychiatry residents, who incorporated VideoScribe slides for knowledge acquisition and YouTube videos of children as discussion prompts for application of new knowledge. See Table 2 for a description of subject material, content, and instructors for the lecture (control) and active learning small group + 2 types of video (experimental intervention). Psychiatry residents were chosen as facilitators in order to provide a large enough group of facilitators to keep groups small.

Table 2.

Content and structure of active-learning intervention versus traditional lecture

Component	Traditional lecture (control)	Active learning (intervention)
Subject material	Development of: School-Age Children Adolescents	Development of: Infants Toddlers Preschoolers
Time/method of delivery	50 min: •in-person lecture •technological component: PowerPoint slides •Typed lecture notes (knowledge acquisition)	15 min: •technological component: Videoscribe animated slide presentations •Typed lecture notes (knowledge acquisition) 35 min: •small group discussions •technological component: 3 clinical vignettes with YouTube video prompts •Instructor question prompts (knowledge integration)
Instructor(s)	One professor in child and adolescent psychiatry	10 third-year psychiatry residents •Each received 3 h training in being facilitator by external medical educator
Instructor to learner ratio	1:120	1:8–10
Space/room needs	One large lecture hall	Ten small group rooms

Delivery

The medical student class was split into two equal-sized groups by last name alphabetically: lecture first followed by AL sessions, and AL sessions first followed by lecture. All study participants were offered both the lecture (control) and the AL small group (experimental intervention). In this way, students served as their own control group. This design was chosen because the medical school would not have supported two groups of students getting different educational content. Having one group receiving only the experimental intervention and one group receiving only the control intervention would have raised concerns among student and school leadership about fairness, especially fairness in grading.

Evaluation

Student medical knowledge of child development was measured using online quizzes with the same questions administered through Qualtrics at three different time points: just before the session; immediately after the session (before the next session); and at 6-month follow-up. Prompts to complete quizzes in Qualtrics were delivered through students’ SOM email accounts. We interpreted scores on quizzes as an indicator of intervention effectiveness. The 3 quizzes each contained 10 multiple choice questions that were written by ES and CF in alignment with National Board of Medical Examiner (NBME) formatting and standards for clarity (Case & Swanson, 2002). Some questions were designed for factual recall, others for application of material and clinical reasoning.

Cost

The time/cost resources to prepare and deliver curricula were compiled.

Faculty and medical students reviewed their calendars for meetings, emails, reviewing drafts of scripts and videos, as well as project development/training time. Total numbers of hours per instructor/staff member were broken out by type of work: content expertise, development of the educational product, and administrative troubleshooting. The Association of American Medical College (AAMC) listed mean salary for an associate professor in child and adolescent psychiatry in 2014–2015 in the western region of the United State of America was $207,000 (AAMC, 2019) in US dollars. Given that fringe benefits costs are estimated at 30%, the total cost to the university for that professor was $260,260. If one divides $260,260 by 52 weeks and again by 50 h per week of work, the rough cost of the associate professor’s time was $100/h (AAMC, 2019) (this is a medium range for physicians, though if this hourly rate was used for all hours of the year, it may project high for some physicians). Salary for third-year psychiatry residents in our program and the AV technician were both prorated at $30/h. These calculations for residents and AV technicians likewise assume a 50 h workweek and assume a 30% cost of fringe benefits (AAMC, 2019). The cost for medical student time making Videoscribe videos was calculated at $15/h. Number of hours for repeats of AL and lecture (for example, if the AL intervention was used after the year in which it was piloted) were estimated based on past experience and projections for AL (Bleichrodt & Quiggin, 1999; Jensen et al., 2015; McPherson & Talbot, 2018). The cost per student for each educational intervention was based on an estimated class size of 200 students. It was not feasible to quantify the cost of lecture hall versus small group room space. A return on investment analysis over time has not been conducted.

Data Analysis

The two groups were compared using a paired-sample t-test to compare knowledge scores of the lecture-based content versus AL-based content. A repeated-measure one-way ANOVA was conducted to see the effect of teaching modality on students’ medical knowledge at three different time points. Effect size was measured using Cohen’s d. A rudimentary cost comparison analysis is used to compare the AL and lecture groups.

Institutional Review Board (IRB)

This study was reviewed and approved by the Health Sciences Campus IRB (ID# HS-14–00,838).

Results

Medical Knowledge

The response rate for the quizzes/evaluation of medical knowledge was 22% (41/189). To set baseline data, a pair-sample t-test was conducted to compare knowledge scores on the pre-test between lecture-based content (m = 2.05, sd = 0.81) and flipped-classroom-based content (m = 2.00, sd = 0.77). The difference did not reach statistical significance. This established that at baseline, students’ knowledge in both content areas were at the same level. Upon immediate post-test, the paired-sample t-test revealed that students’ exam score on the lecture content (m = 3.24, sd = 0.73) was significantly higher than their exam scores on the content taught in small-group flipped classroom (m = 2.95, sd = 0.97) (t = 2.056, df = 40, p < 0.05). The difference continued to be significant even on the 6-month follow-up, where the exam score on the lecture content (m = 4.78, sd = 1.19) was significantly higher than the exam score on the small-group/AL content (m = 2.80, sd = 0.85) (t = 14.863, df = 40, p < 0.01). This also resulted in a large effect size of 1.95 as measured using Cohen’s d.

A repeated-measure one-way ANOVA was conducted to see the effect of teaching modality on students’ knowledge at three different time points. The results show that students’ performance on the knowledge test was significantly affected by both the lecture, F (2, 80) = 15.118, p < 0.05 and by small group/flipped classroom, F (2, 80) = 80.691, p < 0.05. This significant effect was observed between the pre-test and immediate post-test. Students’ knowledge learned from the lecture continue to grow while their knowledge learned from the small group/flipped classroom setting tapered off after the immediate post-test (Fig. 2).

Fig. 2.

Student performance on content taught by lecture and by active learning/flipped classroom (based on 10 questions)

Cost

The cost of converting a single hour of instruction from lecture to AL in this study was 3.4 times that of a lecture, and the projected cost of giving a repeat of the AL intervention was 5.4 times the cost of giving a repeat lecture (Table 3).

Table 3.

Estimated hours of work and cost for producing lecture versus active learning

	Lecture		Active learning
Personnel	1st time	Recurring	1st time	Recurring
Faculty $100/h
Content development	20	2	20	2
Product develelopment	6	1	20	2
Institutional buy-in	0	0	6	1
Administration (communication, review, other)	1	1	20	2
Training facilitators	0	0	4	4
Classroom	1	1	0	0
Total faculty hours	28	5	70	11
Total faculty cost	$2800	$500	$7000	$1100
Residents $30/h
Administration	0	0	2 h × 10 res = 20	2 h × 10 res = 20
Facilitator training	0	0	2 h × 10 res = 20	2 h × 10 res = 20
Classroom time	0	0	2 h × 10 res = 20	2 h × 10 res = 20
Total resident hours	0	0	60	60
Total resident cost	0	0	$1800	$1800
Med student $15/h
Product development	0	0	52	0
Administration	0	0	3	0
Total medical student hours	0	0	55	0
Total medical student cost	0	0	$825	0
AV Tech $30/h
Administration	1	1	4	2
On-site coordination	1	1	2	2
Total AV hours	2	2	6	4
Total AV cost	$60	$60	$180	$120
Total per class	$2860	$560	$9,805	$3,020
Total per learner (assume 200 learners)	$14.30	$2.80	$49.05	$15.10

Discussion

The results of this study found no measured benefit in medical knowledge in using a flipped classroom model over the traditional lecture model for a single lecture. Although students’ knowledge increased immediately after the teaching session in both modalities, results show that 6 months out, students retained more content taught via lecture than via small group/flipped classroom in this study. Furthermore, the AL modality — which required substantial labor from faculty and resident physician facilitators — we estimated as many times more expensive to produce than the traditional lecture (of note, had the faculty facilitator been from a high-paying procedural/surgical specialty, the faculty costs of the creation of the AL session would have been even higher). The benefits of this study’s pilot AL intervention in may not appear to justify its increased costs; however, the study did not measure the impact of students gaining and applying skills and attitudes longitudinally in clinical and non-clinical practice. This paper continues an ongoing discussion on the cost/dose of AL for SOMs considering curriculum changes that incorporate AL (Bleichrodt & Quiggin, 1999; Jensen et al., 2015; McPherson & Talbot, 2018).

Our results were unexpected. Systematic reviews and a meta-analysis of flipped classrooms in medical education and health professions have shown significant benefits (Chen et al., 2017; Hew & Lo, 2018), as have studies in psychiatry comparing small group case-based learning (Colton et al., 2013) or case discussions to lectures (Simmons & Wilkinson, 2012). However, much of this benefit comes in the form of improved professionalism or habits of lifelong learning. By restricting our measurement outcomes to medical knowledge, and by neglecting to measure changes in students’ attitudes and skills, we may have missed positive outcomes. From a broader adult learning theory or evidence-based educational practice perspective, a re-evaluation of instructional strategies, learning objectives, and assessment and evaluation approaches may be in order for such shifts in context and the environment for learning (Mukhalalati & Taylor, 2019). Educators at the course, department, and school level need be able to integrate learning theories, subject matter, and student understanding to improve student learning — to understand surprising outcomes and the impact of individual student differences on their learning outcomes.

Even more important, however, may be the factor of scale of this intervention. This AL intervention was 1 h embedded in hundreds of hours of lecture-based instruction. Students were not used to learning the material this way. AL requires learners to prepare for the activities prior to the session; pre-class preparation was in place for this study as for other school courses but needs to be better operationally defined for assessing an educational intervention. The curriculum that we delivered was a pilot and would certainly require more testing in order to fully examine its potential impact.

It is now 5 years after the date we conducted this experiment. Under pressure from the COVID pandemic and the need for learning at a distance, the Keck SOM leadership and faculty are currently joining together to convert most of the non-clinical curriculum to AL methods using pre-recorded video, teleconferencing, and other interactive technologies. This change will be curriculum-wide, and the leadership of KSOM is committed to improving this renewed curriculum. With such a large scale of investment in technology and pedagogical approach, the anticipated outcome may look very different from our pilot study. It may also help with knowledge, attitudes, and skills in learners, resident facilitators, and faculty facilitators (Liaison Committee on Medical Education, 2015; Accreditation Council for Graduate Medical Education, 2019; AAMC, 2020), and the experience with residents may also stimulate recruitment in psychiatry (Ghatavi & Waisman, 2006; Hickie et al., 2013; Spollen et al., 2017).

Our study had several limitations in design and methodology, which limits its generalizability. First, the sample size was small, and it may not be representative of all students at Keck SOM or other institutions. Second, we were lacking baseline data across graduate medical courses regarding percentage of lecture vs. AL across courses, locally and nationally. Third, the design did not feature a traditional control group. Fourth, the study evaluated only changes in knowledge, not changes in attitudes or skills, nor the application of these in clinical and non-clinical practice; unfortunately, student satisfaction, impressions, and feedback were not collected. Fifth, the questions used to evaluate knowledge may not have been sufficiently sensitive to detect differences in learning between conditions and how things were conducted by faculty/residents (i.e., need structured procedures). Sixth, the cost analysis was rudimentary, as it depended on estimates and retrospective data (i.e., subject to recall bias) and excluded important variables (e.g., lost clinical productivity by residents in the psychiatry clinic, cost of lecture hall versus small group rooms, and faculty time for writing or revising multiple-choice questions); a return on investment analysis would also be prudent to assess costs over time. Most importantly, the very small “dose” of AL delivered in this intervention greatly limits the generalizability of our results to schools that may be committed to larger-scale curriculum renewal.

The 1 h AL session was more expensive and less effective than the 1 h lecture for knowledge acquisition and retention at 6-month follow-up, and future research needs to evaluate the impact of AL with a larger N, control group, structured faculty/resident procedures, and assessment of gaining and applying attitudes and skills in addition to knowledge.

Declarations

Competing Interests

The authors declare no competing interests.