Effect of a Flipped Classroom on Knowledge Acquisition and Retention in an Internal Medicine Residency Program

ABSTRACT

Background

The flipped classroom is a teaching approach with strong evidence for effectiveness in undergraduate medical education. Objective data for its implementation in graduate medical education are limited.

Objective

We assessed the efficacy of the flipped classroom compared with standard approaches on knowledge acquisition and retention in residency education.

Methods

During academic year 2016–2017, 63 medical interns in a large academic internal medical residency program on their ambulatory block were randomized to a flipped classroom or standard classroom during a 6-hour cardiovascular prevention curriculum. The primary outcome was performance on a 51-question knowledge test at preintervention, immediate postintervention, and 3- to 6-month postintervention (delayed postintervention). Secondary outcomes included satisfaction with the instructional method and preparation time for the flipped classroom versus standard approach. We also examined feasibility and barriers to the flipped classroom experience.

Results

All 63 interns (100%) responded during the preintervention period, 59 of 63 (94%) responded during the postintervention period, and 36 of 63 (57%) responded during the delayed postintervention. The flipped classroom approach significantly improved knowledge acquisition immediately after the curriculum compared with the standard approach (knowledge test scores 77% versus 65%, P < .0001). This effect was sustained several months later (70% versus 62%, P = .0007). Participants were equally satisfied with the flipped classroom and standard classroom.

Conclusions

A flipped classroom showed greater effectiveness in knowledge gain compared with a standard approach in an ambulatory residency environment.

Introduction

Modern health care systems necessitate more efficient, evidence-based methods for teaching in graduate medical education.^1–7 The flipped classroom achieves this by introducing an asynchronous learning experience where learners access basic knowledge at their own pace, preserving face-to-face time with instructors to strengthen advanced concepts through active learning.^8,9

Most studies on the flipped classroom rely on lower-order outcomes, such as descriptive data and learner satisfaction, rather than objective assessments of performance.^10–13 Studies evaluating knowledge are pre-post studies involving a single lecture.^8,12,14–16 More evidence evaluating a comprehensive flipped classroom curriculum with higher-order outcomes is needed,^5,17,18 in addition to testing its efficacy within residency education.^{12–14,16,19–22}

The aim of our study was to compare the knowledge acquisition and retention of the flipped classroom method with that of standard didactic teaching. We hypothesized that the flipped classroom would result in similar satisfaction and knowledge acquisition and would improve knowledge retention.

Methods

Setting and Participants

Our study took place in academic year 2016–2017. We invited all 63 medicine interns to participate during their first 2-week ambulatory training block, using a standard script describing the voluntary nature of the study and the confidentiality of the data collection. We assigned each block of interns to either the standard classroom or flipped classroom in an alternating fashion. Program directors made assignments to ambulatory blocks randomly except when accommodating vacation preferences and efforts to distribute interns by clinic location.

Intervention

The structure of the blocks involved in-classroom curriculum in the mornings. Our intervention took place within the cardiovascular prevention curriculum, composed of three 2-hour sessions: (1) assessment of cardiovascular risk and use of chemoprevention, including statins and aspirin; (2) diagnosis and management of type 2 diabetes; and (3) diagnosis and management of hypertension. Overall, there were 6 cohorts, each receiving the 2-hour curriculum, with 3 assigned to the standard arm and 3 to the flipped arm. By the end of 12 weeks, all interns had completed this curriculum. Interns randomized to the standard approach received a lecture-based format with slides and classroom discussion. Interns in the flipped classroom group had an alternative teaching schedule with protected time for prework on the first morning of the block. Prework materials, consisting of information normally contained in the lecture, were curated by the instructor and posted on a wiki website. The in-person session used problem-based learning and occurred during regularly scheduled teaching time. The instructor (K.L.G.) was the same for both approaches to ensure standardization of content delivery and teaching style across groups. A representation of how content was addressed by the flipped and standard classrooms is provided as online supplemental material.

Data Collection

During orientation, interns completed a preintervention survey to collect demographic information and baseline characteristics. We distributed a postintervention survey to all participants, soliciting preparation time in minutes and satisfaction. Interns in the flipped classroom group received additional questions about barriers to prework and feasibility of the flipped classroom on inpatient services. We developed a 51-question knowledge test based on learning objectives, and we piloted it on residents and faculty in primary care, endocrinology, and nephrology, assessing for clarity, length, and difficulty. We created a standardized grading rubric to score free-text responses. The knowledge test was distributed preintervention, postintervention, and 3 to 6 months postintervention (delayed postintervention), the last of which occurred at the end of the academic year (surveys provided as online supplemental material). To link data across repeated measures, interns supplied a unique identifier known only to themselves.

Our Institutional Review Board approved the protocol as exempt from further review.

Outcomes and Analysis

We compared mean scores on the assessment test immediately postintervention and delayed postintervention between groups using unpaired t tests to assess knowledge retention. We performed a paired analysis for each knowledge outcome, comparing the change in scores from preintervention to postintervention, and from preintervention to the delayed postintervention time frames. A Wilcoxon rank sum test was used to compare preparation time and satisfaction between groups. We performed content analysis on responses to the open-ended questions to identify emerging themes about attitudes and feasibility.

Results

Response rates were 63 of 63 (100%) for the preintervention period, 59 of 63 (94%) for the postintervention period, and 36 of 63 (57%) for the delayed postintervention. The average age of participants was 26.9 years, and 41% (26 of 63) were female. Baseline characteristics and performance on the preintervention knowledge test did not differ between the flipped classroom and standard classroom groups (Table 1).

Table 1.

Intern Characteristics in Flipped Classroom Versus Standard Approach^a

Characteristics	Flipped Classroom (n = 29)	Standard Classroom (n = 30)	P Valueb
Age (y), mean (SD)	27.3 (2.2)	26.6 (6.2)	.60
Female sex, n (%)	11 (38)	12 (40)	.60
Career interest in primary care, n (%)			.50
Yes	1 (3)	2 (7)
Unsure	9 (31)	14 (47)
No	19 (66)	14 (47)
Exposure to primary care prior to internship, n (%)			.60
Little to none	1 (3)	4 (13)
Some	24 (83)	23 (77)
A lot	4 (14)	3 (10)
Baseline knowledge assessment score,c mean (SD, range?)	51.1 (2.4)	51.3 (2.0)	> .99

^aNote: Missing data for all variables were < 5% (age 5%, ambulatory experience 2%, all other variables had no missing data).

^bP values are derived using a t test for continuous variables and a χ² test for categorical variables.

^cMean cohort score on our knowledge assessment tool during the preintervention time frame, at the start of the academic year.

The average preintervention knowledge test score was 51%. Preintervention scores were not significantly different for the 2 groups (51% for both, P > .99). The flipped classroom group performed significantly better than the standard classroom group in unpaired analysis (77% versus 65%; P < .0001; Table 2; Figure). For the delayed postintervention, the flipped classroom group had significantly higher scores than the standard classroom (70% versus 62%; P = .0007; Table 2; Figure). These differences also were significant in the paired analysis (table 3).

Table 2.

Knowledge Acquisition and Retention in Flipped Classroom Versus Standard Classroom Cohorts (Unpaired Analysis)

	Baseline Knowledge Assessmenta (n = 63)	Postintervention Knowledge Assessmentb (n = 59)	Delayed Postintervention Knowledge Assessmentc (n = 36)
Flipped classroom, mean (SD) [n]	51.1 (11.5)	77.0 (0.07) [29]	69.7 (8.5) [15]
Standard classroom, mean (SD) [n]	51.3 (10.0)	65.7 (0.07) [30]	62.4 (6.9) [21]
P valued (unpaired)	> .99	< .0001	.0007

Figure.

Knowledge Acquisition and Retention in Standard Versus Flipped Classroom Groups^a

^a Group mean scores on knowledge assessment test at baseline (preintervention, summer 2015), immediately after the curriculum (postintervention, fall 2015), and several months later for both cohorts (flipped classroom and standard curriculum delivery, spring 2016). The change in scores between baseline and immediately after the curriculum represents knowledge acquisition and is significantly better using a flipped classroom. The change in scores from immediately after the intervention until the spring represents knowledge retention and is significantly better using a flipped classroom.

Table 3.

Knowledge Acquisition and Retention in Standard Versus Flipped Classroom Cohorts (Paired Analysis)

	Baseline Knowledge Assessmenta	Postintervention Knowledge Assessmentb	Difference in Scores	Delayed Postintervention Knowledge Assessmentc	Difference in Scores
Flipped classroom, mean (SD) [n]	51.6 (11.4)	77.8 (7) [21]	26.2 (13)	68.7 (10.8) [9]	21.9 (12.3)
Standard classroom, mean (SD) [n]	50.6 (10.1)	67.2 (6.5) [23]	16.6 (11.9)	63.6 (6.7) [18]	12.2 (11)
P valued (paired)	> .99		.014		.048

^aAdministered during the first month of the academic year prior to any curricular intervention.

^bAdministered immediately after curriculum was delivered.

^cAdministered at the end of the academic year.

^dBased on a t test comparing mean scores between groups on a 51-point knowledge assessment test.

Interns spent more time preparing for the flipped classroom compared with the standard classroom (23 minutes versus 11 minutes; P = .001; Table 4). All interns reported high satisfaction with their respective teaching modalities (range of satisfaction scores 4.7–4.9 out of 5).

Table 4.

Preparation Time and Satisfaction With Learning Experience in Flipped Classroom Versus Standard Classroom Groups

Category	Flipped Classroom (n = 29)	Standard Classroom (n = 30)	P Valuea
Preparation time for class, min, mean (SD)	23.1 (14.2)	11 (19.4)	.001
Satisfaction with learning experienceb
Part I: cardiovascular risk assessment and management	4.8 (0.4)	4.8 (0.5)	.40
Part II: management of type 2 diabetes mellitus	4.8 (0.4)	4.9 (0.4)	.10
Part III: management of hypertension	4.7 (0.5)	4.9 (0.4)	.028

^aP values derived from a Wilcoxon rank sum test.

^bMean score on a 5-point scale: 5, very satisfied; 4, satisfied; 3, neutral; 2, dissatisfied; 1, very dissatisfied.

Interns reported the following barriers towards prework: (1) interference with clinical work (14%, 4 of 29); (2) lack of motivation (7%, 2 of 29); and (3) trouble accessing assignments (10%, 3 of 29). A majority of interns (66%, 19 of 29) felt that the flipped classroom would not be feasible on inpatient rotations. These results are displayed in Table 5.

Table 5.

Intern Attitudes Toward Implementation of the Flipped Classroom^a

Item	No. (%)
Barriers to completing prework
None	4 (14)
Trouble accessing prework	3 (10)
Interference of administrative clinical work	4 (14)
Lack of motivation	2 (7)
Would this method of teaching be feasible on inpatient rotations?
Yes	6 (21)
No	19 (66)
Unsure	4 (14)
Perceived barriers to the flipped classroom on inpatient rotationsb
Not enough protected time	7 (24)
Clinical demands too high	2 (7)
Time required not achievable within workday	1 (3)

^an = 29.

^bRepresents a count of total responses for each theme.

Discussion

In this randomized trial evaluating the flipped classroom within an ambulatory curriculum, we found significant improvement in knowledge acquisition and retention compared with a standard approach in both paired and unpaired analyses. This indicates that the flipped classroom may have achieved the aim of “deep learning.” Learning satisfaction was not compromised, and the additional time commitment was manageable. Our study fills an important gap by evaluating higher-order learning outcomes with a randomized trial in an understudied population.

The finding that learners were satisfied despite increased preparation time with the flipped classroom was encouraging, as time was an important barrier. Restructuring the teaching schedule to provide protected time for prework was likely essential. A noteworthy result was that interns felt that the time barrier would be insurmountable in the inpatient setting. We recognize that an ambulatory teaching schedule, which resembles a “classroom,” more easily permits this teaching method than an unpredictable inpatient schedule does.

This study has limitations, including its scope as a single center study evaluating interns only in the ambulatory environment, which limits generalizability. Using the same instructor for both groups to limit heterogeneity and establish proof of principle represented a potential source of bias and obviated the possibility of blinding. We had a lower response rate for our delayed postintervention measurement, which renders our findings as speculative rather than definitive. Finally, our results do not delineate whether the impact of the flipped classroom derived from the prework or the active learning experience; however, both are considered essential components to implementation of this type of instruction.

Next steps include an expansion of the flipped classroom to the entire ambulatory medicine curriculum. This curriculum includes approximately 900 hours of teaching by 8 faculty members across a large residency program, and it will require faculty development, technological support, and restructuring of the curriculum to implement this teaching approach on a broader scale.

Conclusion

A flipped classroom in a controlled residency setting improved knowledge gain without sacrificing satisfaction or introducing significant time constraints. Residency programs wanting to simulate the principles outlined in this study will need to address the logistical challenges of the flipped classroom related to access and time constraints in addition to providing faculty development to ensure its success.