A Flipped Classroom in Graduate Medical Education

Summary:

Background:

The role of the flipped classroom model in graduate medical education (GME) is not yet defined. We set out to evaluate feasibility, acceptability, and outcomes of a flipped classroom instructional model in an internal medicine curriculum.

Methods:

This pilot study was carried out in an academic medical center in the United States with 43 second year internal medicine post-graduate trainees. Trainees watched videos about pharmacologic treatment of type 2 diabetes outside of the classroom, followed by an in-class session in which they engaged in case-based discussions. The intervention was evaluated using surveys and a knowledge test before, immediately after, and six months after the intervention.

Results:

The mean number of correct answers on a ten-question knowledge test was 5.25 on the pretest, 8.00 on the immediate post-test, and 7.10 on the 6-month post-test (p<0.001). Six months after the intervention, 57% of participants reported prescribing an antidiabetic medication discussed at the session. In a focus group, trainees reported their preference for interactive, case-based learning, concern about time required for preparation, and interest in incorporating real patient cases.

Discussion:

Trainees preferred the flipped classroom, which also resulted in increased knowledge and self-reported prescribing changes. However, the required preparatory time may limit its feasibility in GME.

Introduction

Instructional methods that encourage interactive learning and applied clinical reasoning are increasingly favored over classic lecture-based methods. The “flipped classroom” is one such model that has gained popularity throughout health professions education, referring to an instructional model in which classroom and self-study time are reversed.¹ Students prepare for class by reading and/or watching pre-recorded content, and class time is then devoted to applying new knowledge through interactive activities such as problem solving and discussion.¹ Proposed benefits of this model include improved self-directed learning skills, more efficient student-teacher interactions, and increased learner engagement and retention.¹

The flipped classroom approach for health professions students is well established, and its use in medical subspecialty training (graduate medical education - GME) is also increasing.² However, concerns have been raised about implementing the flipped classroom in GME.³ To date, only several small studies have reported implementing flipped classroom curricula in the GME context, with inconclusive data on knowledge acquisition and retention and little data regarding feasibility.^4-7 Further, the impact of learning in the flipped classroom model on trainees’ clinical practice has not been studied and no prior studies have examined the use of the flipped classroom in internal medicine residency teaching sessions.

We conducted a pilot study to assess the feasibility, acceptability, and effectiveness of a flipped classroom strategy in teaching outpatient diabetes management to internal medicine trainees, specifically focusing on pharmacotherapy of type 2 diabetes.

Methods

Setting

The intervention was delivered to 43 post-graduate year (PGY)-2 internal medicine residents at a large teaching hospital in Boston, Massachusetts in the 2017–2018 academic year. Residents are trainees who have completed medical school and are pursuing subspecialty training in internal medicine. The Partners Healthcare Human Research Committee determined that the project met criteria for institutional review board exemption because it was research conducted in established educational settings. Completion of the survey was considered implied consent of participation, and participation in surveys was optional. Only deidentified data were stored.

Teaching Innovation

We designed a flipped classroom intervention focused on pharmacotherapy for type 2 diabetes based on a needs assessment of the residents and supervising physicians. The self-study material in the intervention consisted of five concept videos, each 5 to 10 minutes in length (total duration of 35 minutes). There was an optional introductory video about commonly used medications, three videos that reviewed the pharmacology, indications, and adverse effects for several newer classes of medications used in the treatment of type 2 diabetes, as well as a mandatory summary video. Each resident was assigned to watch one of the three videos covering the medication classes (thus, the minimum required preparation time was 10–15 minutes). The students were emailed links to the videos one week prior to the in-class session and expected to watch the videos on their own. The videos were all created by one of the authors (OPRH) (see Box 1 for further details).

Box 1: Detailed curriculum tools.

• Knowledge Test: Knowledge questions used with permission from the NEJM Knowledge+ publisher. This resource is a board review and life-long learning tool that consists of case-based questions. Questions are written by content experts and then subject to several layers of review before being incorporated into the platform.

• Videos: Whiteboard animation videos in which a lecturer narrates while drawing and writing key points on the slide in real time.

  • The videos were created using Explain Everything software and an Apple iPad.
  • The videos were uploaded to YouTube and the students were provided with the links to the videos using the Moodle learning management system.

The in-class session was 45 minutes long and consisted of case-based discussions in two small groups of approximately 4–8 students each. The in-class session was repeated four times with different groups of residents; each resident attended only one in-class session. Residents were able to access other resources while working on the cases and two facilitators circulated among the groups.

Feasibility and Acceptability Assessments

For the purposes of this study, we defined feasibility as to what extent our findings support implementation of the flipped classroom in GME and acceptability as how participants reacted to the pilot.⁸ Immediately after the in-class session, residents were asked about their opinions of the flipped classroom approach using a questionnaire. We also conducted a focus group to better understand participant attitudes and acceptability of this flipped classroom pilot. All PGY-2 residents were invited to take part and three residents participated in the focus group. Two of the researchers (RAB, JBC) moderated the focus group, using a semi-structured approach that started with a list of open-ended questions generated in advance by the researchers. The focus group was recorded and transcribed. The same two researchers independently performed an inductive thematic analysis using emergent coding.⁹ Each researcher read through the transcript of the focus group and attempted to articulate emerging themes. Subsequently, the two researchers met to discuss and compare identified themes to ensure agreement.

Effectiveness Assessments

Prior to engaging in the intervention, residents completed an online knowledge test and an attitudinal survey. The knowledge test consisted of ten questions from the New England Journal of Medicine (NEJM) Knowledge+ question bank and were used with permission from the publisher. The attitudinal survey consisted of two 5-point Likert scales rating comfort choosing and prescribing diabetes medications. The knowledge and attitudinal surveys were disseminated using Qualtrics survey software (Qualtrics, Provo, Utah). Residents were asked to repeat the knowledge test and attitudinal survey immediately after the in-class session, and again approximately six months later. Finally, in the 6-month post-test, they were also asked if they had considered prescribing or actually prescribed any antidiabetic medications in the drug classes covered in the session.

We used a linear mixed effects model with random intercept for repeated measures over time to analyze the change in residents’ self-reported comfort with choosing and prescribing antidiabetic agents as well as changes in their knowledge test scores. This analysis prevented list-wise deletion due to missing data, thus allowing all data to be analyzed. All analyses were performed in Stata Version 15 (Statacorp, College Station, Texas) using the mixed command.

Results

Feasibility and Acceptability

A total of 43 residents were assigned to watch the videos and attend the in-class session. The majority of residents who replied to the survey (82.8%, 24/29 post-test respondents) reported that watching the videos was a valuable use of their time and 89.7% (27/29) reported that they preferred the format of the in-class session to a traditional lecture. Three key themes were evident in the focus group: preference for interactive, case-based learning; concerns regarding motivation and time for preparation outside of scheduled sessions; and interest in involving real, resident-generated cases in future sessions (see Table 1).

Table 1:

Resident perceptions of the flipped classroom approach grouped by theme with illustrative quotes

Theme	Quotes
Preference for interactive, case-based learning	-“I’m really tired of lecture-based for this process, and so I really prefer when it’s either more interactive or flipped”
Concerns regarding motivation and time for preparation outside of scheduled sessions	-“It’s hard to motivate us to do stuff outside the classroom because our time is so limited” -“I was thinking that if you want to increase the number of people that will actually watch the video and come prepared, you [should] actually block a time prior to the session, half-an-hour, 45 minutes, whatever it is, that is just dedicated to watching those videos”
Interest in real patient cases	-“You know what’d be cool is if…we could discuss our real primary care cases in [the] didactic sessions. That would make it feel really tangible”

Effectiveness

Twenty-four participants completed the attitudinal questions on the pre-test and at least one post-test. Of these, 22 participants took the immediate post-test (51.2% response rate) and 14 participants took the 6 month post-test (32.5% response rate). The results are shown in Table 2. On a 5-point Likert scale, participants’ rated comfort in prescribing the newer antidiabetic agents increased from a mean of 2.29 preintervention, to 4.04 immediately post-intervention (mean increase 1.7, 95% CI 0.44–1.40, p<0.001) and remained higher at 3.1 six months post-intervention (mean increase 0.91 from baseline, 95% CI 1.97–2.61, p=0.001).

Table 2:

Self-rated comfort with choosing and prescribing advanced antidiabetic medications

Question	Number who agree or strongly agree (n (% of total responses))
	Preintervention (n=24)	Immediate post- intervention (n=22)	6 months post- intervention (n=14)
I am comfortable choosing another antidiabetic agent if a diabetic patient remains hyperglycemic on metformin/lifestyle interventions alone	13 (54.2%)	21 (95.5%)	9 (64.3%)
I am comfortable prescribing antidiabetic medications other than metformin, sulfonylureas and insulin.	1 (4.2%)	19 (86.3%)	9 (64.3%)

Twenty-four participants completed the knowledge pre-test and at least one knowledge post-test. Of these, 22 completed the immediate post-test (51.2% response rate) and 10 completed the 6 month post-test (23.3% response rate). The mean score on the pretest was 5.25 (95% CI 4.68–5.81). The mean score on the immediate post-test improved by 2.72 points to 8.00 (95% CI 7.35–11.33, p<0.001), and on the 6-month post-test by 1.80 points to 7.10 (95% CI 6.24–7.86, p<0.001) when compared with the baseline test (Figure 1)

Figure 1: Scores on knowledge test at each time point.

Graphical summary showing the scores of residents on a 10-question knowledge test on diabetes pharmacology at three separate time points. The box indicates the 25th-75th percentile; the whiskers indicate 10th-90th percentile and the dots represent outliers.

At six months after the intervention, 11 of the 14 respondents (78.6%) reported that they had considered prescribing a medication from the drug classes discussed at this session and 8 of 14 (57%) reported that they had actually started one of these medications. Of the six participants who had not started a medication, four reported that none of their patients had needed additional agents.

Discussion

To our knowledge, this one of the first studies to examine implementation of a flipped classroom approach within an internal medicine residency. Furthermore, it is the first published report to consider how and to what extent the introduction of flipped classroom pedagogy might impact residents’ clinical practice.

Feasibility and Acceptability

We found that, overall, residents reported that they enjoyed the curricular model, which is consistent with existing literature on the flipped classroom outside of this context.¹ However, the residents in our group interview expressed significant concerns about the time and motivation to watch required videos outside of scheduled teaching sessions. Our data provide new evidence that the flipped classroom might not be a feasible approach if implemented widely in GME given the need for protected time for preparation, as Cooper and colleagues suggested.³

Effectiveness

We examined both knowledge and self-reported prescribing practices to assess the effectiveness of the flipped approach. We found an improvement in the mean knowledge scores on the immediate post-test and six-month post test. The decrease in mean knowledge score at six-months compared with the immediate post-test is consistent with existing literature on knowledge retention in the health professions.¹⁰ We found that six months after the session the majority of respondents reported starting a patient on one of the classes of antidiabetic medications covered in this curriculum. This data suggests an impact of this intervention on knowledge application and clinical practice.

Our study has several limitations. First, all residents in our program were exposed to the intervention and thus we did not have a comparable control group. There were only a small number of participants in the focus group, who may have had stronger opinions than other residents about the format of teaching sessions. Next, our survey response rates were only modest, though not substantially different from other surveys of physicians.¹¹ There may have been some self-selection bias in those who did answer surveys. Additionally, while our data captures resident self-report of prescribing behavior, we did not measure prescribing behavior or residents’ patient outcomes directly. We also did not collect information about baseline prescribing practices of residents, though the reported comfort in prescribing novel antidiabetic agents was low prior to the teaching session.

Conclusion

While residents expressed great satisfaction and engagement with the flipped classroom curricular model, and also reported changes in clinical practice, the ability to prepare for the in-class sessions is constrained by long residency work-hours. Therefore, preparation time may need to be incorporated into the didactic schedule if this model is to be feasible on a larger scale. However, increased engagement and satisfaction may translate into improved clinical behavior and patient outcomes which should be the subject of future studies of this curricular model in GME.