Evaluation of the flipped classroom approach in a veterinary professional skills course

Abstract

Background

The flipped classroom is an educational approach that has had much recent coverage in the literature. Relatively few studies, however, use objective assessment of student performance to measure the impact of the flipped classroom on learning. The purpose of this study was to evaluate the use of a flipped classroom approach within a medical education setting to the first two levels of Kirkpatrick and Kirkpatrick’s effectiveness of training framework.

Methods

This study examined the use of a flipped classroom approach within a professional skills course offered to postgraduate veterinary students. A questionnaire was administered to two cohorts of students: those who had completed a traditional, lecture-based version of the course (Introduction to Veterinary Medicine [IVM]) and those who had completed a flipped classroom version (Veterinary Professional Foundations I [VPF I]). The academic performance of students within both cohorts was assessed using a set of multiple-choice items (n=24) nested within a written examination. Data obtained from the questionnaire were analyzed using Cronbach’s alpha, Kruskal–Wallis tests, and factor analysis. Data obtained from student performance in the written examination were analyzed using the nonparametric Wilcoxon rank sum test.

Results

A total of 133 IVM students and 64 VPF I students (n=197) agreed to take part in the study. Overall, study participants favored the flipped classroom approach over the traditional classroom approach. With respect to student academic performance, the traditional classroom students outperformed the flipped classroom students on a series of multiple-choice items (IVM mean =21.4±1.48 standard deviation; VPF I mean =20.25±2.20 standard deviation; Wilcoxon test, w=7,578; P<0.001).

Conclusion

This study demonstrates that learners seem to prefer a flipped classroom approach. The flipped classroom was rated more positively than the traditional classroom on many different characteristics. This preference, however, did not translate into improved student performance, as assessed by a series of multiple-choice items delivered during a written examination.

Background

Medical educators are regularly presented with new techniques and technologies that have the potential to support student learning. The flipped classroom is one idea that has had much recent coverage in both the popular and scientific educational literature. The flipped classroom describes an approach whereby course material is first presented to students prior to class, usually via an online learning environment. Class time is then used for learning-centered activities that build on the preclass work, rather than traditional instructor-led lecture sessions. The overall effect is to reverse or flip the way in which material is presented to students.

If we choose a loose definition of the term, the basic concept of the flipped classroom is not new; educators across the ages have set preclass reading assignments for their students. However, the idea has seen a recent revitalization due to the possibilities of new educational technologies. The pioneers of the modern flipped classroom include US high school science teachers Jon Bergmann and Aaron Sams, who decided to use video-recorded lectures to reach a proportion of their students who had missed class.¹ Bergmann and Sams found that students were generally positive about the method; they believed that it allowed more effective use of class time and challenged them to take responsibility for their own learning. In recent years, the idea has expanded, and the flipped classroom approach has been reported in a wide variety of educational settings, including medical education.²

Advantages of the flipped classroom

One of the main advantages proposed for the flipped classroom is that it creates opportunities for individualized education.³ If course material is presented to learners via an online learning environment, this means that they can access information at a time and location of their choosing. If asynchronous video-recorded lectures are used, these can also be paused and rewound; thus, learners can move through material at their own pace. The approach also allows learners to access online resources repeatedly. In one study of engineering students, the participants viewed each video-recorded lecture three times, on average, and returned to the resources to prepare for examinations.⁴ An online learning environment can also support presentation of material to students in a variety of different ways, eg, podcasts, computer-assisted learning modules, and online whiteboards. It is possible that utilizing a range of these formats to present course content can support students’ individual learning styles and preferences.

Another proposed advantage of the flipped classroom is that it allows instructors and students to make better use of face-to-face time.^1,5 Removal of didactic course material to an online environment means that class time is freed for other, more active forms of learning. The use of small-group, case-based or team-based exercises in flipped classrooms has been documented, and it is thought that such active learning teaching techniques aid learner retention and understanding.⁶ Such active learning exercises also facilitate instructor–student interaction. With traditional courses, the instructor delivers material in class, but it will not be present when the learner attempts to apply the material, eg, during an assignment. With the flipped classroom, the instructor is physically present during this application-of-knowledge stage, a feature of the teaching approach that appeals to both educators and students.⁵

Many educators and students also agree that the flipped classroom is a valuable way of improving learner engagement and promoting a sense of responsibility for one’s education.³ Further, studies suggest that the teaching method improves their academic performance. McLaughlin et al found that the approach enhanced student performance and learning on the first-year basic pharmaceutics course offered at the University of North Carolina, USA; cumulative final examination performance rose and overall grades increased from 80% to 82.5% when a flipped classroom approach was adopted.⁷

Disadvantages of the flipped classroom

The flipped classroom approach has a number of perceived drawbacks. Although the flipped classroom is thought to encourage learners to take responsibility for their own learning, this will only happen if they are willing and able to do so.⁸ For example, at a simple level, instructors using an online learning management system must ensure that students have access to the required technology and to a reliable Internet connection. In addition, it may be necessary to help students to buy into the value of the flipped classroom. This is especially relevant when students are used to a passive, didactic lecture environment and are being asked to adapt to one that centers on active learning. Giving structure and meaning to the flipped classroom is vital. In a study involving US college-level students, Strayer found that:

[…] students in the flip classroom were less satisfied with how the structure of the class oriented them to the learning tasks in the course. The analysis showed that the variety of learning activities in the flip classroom contributed to an unsettledness among students (a feeling of being “lost”) that students in the traditional classroom did not experience.⁹

Students are not the only ones to benefit from careful orientation. Educators who are not familiar with the flipped classroom are also likely to require support because they take on the responsibility of preparing a variety of preclass materials and assignments, eg, video lectures and active learning classroom exercises.¹⁰ Educators have reported a concern about the significantly increased workload involved with migrating from traditional lectures to a flipped classroom.⁴ However, the time commitment involved may not be as severe as first thought. As Talbert explains, “The expense of creating materials for class is largely a one-time startup expense, since videos and activities can be reused and updated over time.”¹⁰

Purpose of the study

Despite much discussion in the literature, there are few studies that use objective assessment of student performance to measure the impact of a flipped classroom approach.² The flipped classroom has shown potential in medical education; two studies that have looked at student performance in a pharmacy course recorded an improvement in exam results with a flipped classroom approach.^7,11 However, it is clear that more evidence is required as to the effectiveness of the flipped classroom.

The purpose of this study was to evaluate the use of a flipped classroom approach within a veterinary professional skills course to the first two levels of Kirkpatrick and Kirkpatrick’s effectiveness of training framework.¹² The two specific objectives were to:

1. evaluate student perceptions of a flipped classroom in comparison to a traditional, lecture-based course; and

2. assess the impact of the flipped classroom approach, if any, on student academic performance.

To the authors’ knowledge, this is the first time that use of the flipped classroom in veterinary education has been investigated.

Methods

Description of flipped classroom approach

This study examined the use of a flipped classroom approach within a veterinary professional skills course offered to students enrolled on a Doctor of Veterinary Medicine program at Ross University School of Veterinary Medicine (RUSVM), St Kitts, West Indies. In 2013, a traditional, lecture-based professional skills course, “Introduction to Veterinary Medicine” (IVM), underwent a redesign. Traditional large group lectures on a variety of topics, including communication skills, financial planning, career success and research skills, were replaced with a flipped classroom approach. This latter approach comprised two key educational components:

1. Pre-class activities: Content material was presented to students through the online learning management system eCollege (Pearson Education Inc., Centennial, CO, USA). Each topic consisted of a number of components: a list of learning outcomes, a narrated PowerPoint lecture, a written article, and a short multiple-choice quiz. The online course work was designed so that the majority of students would take less than 20 minutes to complete each topic. The preclass activities were made available to students 1 week prior to class and were freely accessible thereafter.

2. In-class activities: Each workshop opened with an overview of the topic and a brief discussion of the online, preclass work. The rest of the workshop was then devoted to active learning exercises including role play, case discussions, and small-group activities. Each workshop was 2 hours in length, and student activity during class was documented using an audience response system (TurningPoint; Turning Technologies, Youngstown, OH, USA).

Following the redesign, the flipped classroom course was given the name “Veterinary Professional Foundations I” (VPF I). During the course redesign, an attempt was made to preserve course content; that is, identical learning objectives, topics for discussion, and assessment methods were presented to students enrolled on both IVM and VPF I courses. In addition, both courses were delivered by the same instructors and face-to-face activities were held in the same classroom.

Sample

The sample consisted of two cohorts of students: those enrolled on the final IVM class (January–April 2013 cohort) and those enrolled on the first VPF I class (May–August 2013 cohort). There were 168 students (132 females, 36 males) in the IVM class and 146 students (112 females, 34 males) in the VPF I class. Members of each cohort were comparable in that all students had met minimum entry requirements to RUSVM, which include a minimum of 48 college credits in a number of specified, prerequisite courses. In addition, data from the RUSVM examination office revealed no statistically significant difference in overall academic performance between the two cohorts within their first semester examinations (C Fuentealba, personal communication, September 9, 2013). All students were in the fourth month of their veterinary education at the time of sampling.

Although both IVM and VPF I courses were required for students enrolled on the Doctor of Veterinary Medicine program, participation in the study was voluntary. Participants in the study completed a written informed consent document that communicated that the confidentiality of all participants would be protected and that only collated data from the study would be presented in research outputs.

Instruments

The Course Evaluation Questionnaire (CEQ), a 25-item survey instrument which has been validated for evaluation of short courses in medical education,¹³ was used to explore student perceptions of their classroom experience. Each item on the questionnaire was answered using a 5-point Likert scale ranging from a score of 1 (“strongly agree with the statement”) to 5 (“strongly disagree with the statement”). The questionnaire was administered to all study participants using an online survey tool, E*Value (Advanced Informatics, Minneapolis, MN, USA), and was made available for a 30-day period at the end of their respective courses.

Student performance on each course was measured using a set of multiple-choice items (n=24) nested within a written multiple-choice examination, which was held at the end of each course. The items were written in alignment with the course learning objectives and designed to assess student performance to “knows” (knowledge) and “knows how” (application) levels.^14,15 Identical items were presented to both the IVM and VPF I class cohorts. The items consisted of ten application-level multiple-choice items and 14 knowledge-level multiple-choice items. There were six questions for each of four topics: communication skills, financial planning, career success, and research skills. There were four possible response options for each question.

The research project, including the questionnaire and use of exam data, was approved by the Institutional Review Board of Ross University School of Medicine, Dominica.

Statistical analysis

Data obtained from the CEQ were analyzed using Cronbach’s alpha to determine the internal consistency of the responses.¹⁶ Student perceptions of the teaching models were examined through comparison of their responses to the CEQ. Differences in responses between cohorts to individual Likert items were analyzed using Kruskal–Wallis tests. Following the construct validity approach of Broomfield and Bligh, factor analysis with varimax rotation was used to determine the number of factors that may account for the variance in the overall sample.¹³ Here, the summed responses to the items within each identified factor were also compared using a Kruskal–Wallis test to determine if particular factors were perceived to be different between the cohorts. To account for the negative style of questioning for items 13 and 14 in the CEQ, the responses were reversed prior to analysis.

Student performance in the written examination (multiple-choice items) was compared between cohorts. Both the total number of correct multiple-choice items and correct results grouped by knowledge and application type were analyzed using the nonparametric Wilcoxon rank sum test as the exam results were non-normally distributed.

All analyses were carried out in the open source statistical software R, using packages “likert” and “plyr”.^17–19

Results

A total of 133 IVM students and 64 VPF I students (n=197) agreed to take part in the study and completed both a CEQ and a written examination. This represents an overall participation rate of 63% (79% of IVM students and 44% of VPF I students).

Student perceptions

One student from the IVM cohort responses was omitted from the perception analysis as he/she had responded “strongly disagree” to all items. Cronbach’s alpha on the remaining 196 responses for the questionnaire was 0.833, which suggested that the survey tool had a good level of internal consistency and reliability.

Overall, study participants favored the flipped classroom approach (VPF I) over the traditional classroom approach (IVM). When asked to rate the statement “Overall, I am satisfied with the quality of this course”, study participants gave a higher rating to the flipped classroom than the traditional classroom (IVM mean = ± standard deviation [SD]; VPF I mean = ± SD; Kruskal–Wallis z=11.39; P<0.05) (question [Q] 25 on Figure 1).

Figure 1.

Factor 1 (“generic skills”): Likert responses to items grouped in Factor 1.

Notes: Percentage values refer to the proportion of positive (orange), neutral (gray), and negative (green) responses. The difference in response between cohorts was tested for each question, and the Kruskal–Wallis results are presented alongside each question. There is a significant difference in mean response between cohorts (Kruskal–Wallis z=8.45; P=0.003).

Abbreviations: VPF I, Veterinary Professional Foundations I; IVM, Introduction to Veterinary Medicine.

Factor analysis revealed four broad factors related to “generic skills” (Factor 1), “workload and content” (Factor 2), “staff and teaching” (Factor 3), and “student expectations” (Factor 4), that explained the variance in responses (Table 1). Some items corresponded with two or more factors, and these were only included in the grouping that the correlations were the strongest. The summed responses to the items in each of the four factors were significantly different between cohorts; the VPF I cohort gave slightly more positive answers for factors related to “generic skills” (Figure 2), “staff and teaching” (Figure 3) and “student expectations” (Figure 4). For individual items in the “workload and content” factor (Figure 5), the VPF I cohort disagreed with the negative statements more often than the IVM cohort, overall being more positive about their experience than the IVM students.

Table 1.

Summary table of the principal components analysis of the CEQ with veterinary students revealing four broad factors related to “generic skills” (Factor 1), “workload and content” (Factor 2), “staff and teaching” (Factor 3), and “student expectations” (Factor 4)

CEQ item	Factor 1	Factor 2	Factor 3	Factor 4
5. The course sharpened my analytic skills	0.807
2. The course developed my problem-solving skills	0.801
9. The course helped me to develop my ability to work as a team member	0.793
22. My course helped me to develop the ability to plan my own work	0.768
11. The course improved my skills in written communication	0.686
10. As a result of my course, I feel confident about tackling unfamiliar problems	0.685		0.284
25. Overall, I am satisfied with the quality of this course	0.667		0.316	0.304
3. The teaching staff on this course motivated me to do my best work	0.576		0.383
23. The sheer volume of work to be got through in this course means you can’t comprehend it all thoroughly		0.77
4. The work was too heavy		0.595
19. Too many staff on this course ask us questions just about facts		0.585
12. Staff on this course seem more interested in testing what you’ve memorized than what you’ve understood		0.511
16. The course is overly theoretical and abstract		0.412
14. I was generally given enough time to understand the things we have to learn		0.501	−0.253	−0.356
8. To do well in this course all you really need is a good memory	0.338	0.343
21. There was a lot of pressure on me to do well in this course		0.251
15. The staff on this course make a real effort to understand difficulties students may be having with their work			0.687	0.28
20. Teaching staff on this course work hard to make their subjects interesting			0.597
7. Staff on this course put a lot of time into commenting on student’s work	0.27		0.584
17. Teaching staff on this course normally give helpful feedback on how you’re doing			0.54	0.264
18. Our lecturers are extremely good at explaining things to us			0.527	0.289
6. You usually have a clear idea of where you’re going and what’s expected of you in this course			0.299	0.774
1. It is always easy here to know the standard of work expected				0.63
13. It is often hard to discover what’s expected of you in this course		−0.566		0.632
24. The staff on this course make it clear right from the start what they expect of students			0.446	0.586

Note: The values in bold are factors with a loading >0.4.

Abbreviation: CEQ, Course Evaluation Questionnaire.

Figure 2.

Factor 3 (“staff and teaching”): Likert responses to items grouped in Factor 3.

Notes: Percentage values refer to the proportion of positive (orange), neutral (gray), and negative (green) responses. The difference in response between cohorts was tested for each question, and the Kruskal–Wallis results are presented alongside each question. There is a significant difference in mean response between cohorts (Kruskal–Wallis z=42.93; P<0.001).

Abbreviations: VPF I, Veterinary Professional Foundations I; IVM, Introduction to Veterinary Medicine.

Figure 3.

Factor 4 (“student expectations”): Likert responses to items grouped in Factor 4.

Notes: Percentage values refer to the proportion of positive (orange), neutral (gray), and negative (green) responses. The difference in response between cohorts was tested for each question, and the Kruskal–Wallis results are presented alongside each question. There is a significant difference in mean response between cohorts (Kruskal–Wallis z=21.49; P<0.001).

Abbreviations: VPF I, Veterinary Professional Foundations I; IVM, Introduction to Veterinary Medicine.

Figure 4.

Factor 2 (“workload and content”): Likert responses to items grouped in Factor 2.

Notes: Percentage values refer to the proportion of positive (orange), neutral (gray), and negative (green) responses. The difference in response between cohorts was tested for each question, and the Kruskal–Wallis results are presented alongside each question. There is a significant difference in mean response between cohorts (Kruskal–Wallis z=5.82; P=0.016).

Abbreviations: VPF I, Veterinary Professional Foundations I; IVM, Introduction to Veterinary Medicine.

Figure 5.

Overall student performance on a series of multiple-choice items.

Notes: The proportion of IVM (traditional classroom; red points) and VPF I (flipped classroom; blue points) cohorts correctly answering individual exam questions.

Abbreviations: VPF I, Veterinary Professional Foundations I; IVM, Introduction to Veterinary Medicine.

There was also a significant difference in response to some individual items between cohorts. VPF I students gave a significantly higher rating than IVM students to eleven items (Figures 1–4). Conversely, IVM students gave a significantly higher rating than VPF I students on three items (Figures 3 and 4).

Student performance

Overall, the students enrolled on the IVM course performed better in the multiple-choice items than the students enrolled on the VPF I course (IVM mean =21.4±1.48 SD; VPF I mean =20.25±2.20 SD; Wilcoxon test, w=7,578; P<0.001). There were also differences in performance at a topic level (Figure 6). IVM students outperformed VPF I students on items that tested the topics of communication skills (IVM mean =5.45±0.73 SD; VPF I mean =4.89±1.11 SD; Wilcoxon test, w=7,465; P<0.001) and financial planning (IVM mean =5.81±0.41 SD; VPF I mean =5.42±0.71 SD; Wilcoxon test, w=7,546; P<0.001). There was no significant difference in student performance between cohorts on items which tested career success IVM mean =5.17±0.64 SD; VPF I mean =5.01±0.85 SD; Wilcoxon test, w=5,829; P=0.962) and research skills (IVM mean =4.97±0.84 SD; VPF I mean =4.93±0.93 SD; Wilcoxon test, w=6,262; P=0.275).

Figure 6.

Topic-specific student performance.

Notes: Proportional frequency histograms of the number of correct answers achieved by IVM (traditional classroom) students and VPF I (flipped classroom) students in each of the four topic areas.

Abbreviations: VPF I, Veterinary Professional Foundations I; IVM, Introduction to Veterinary Medicine.

IVM students also performed better than VPF I students overall on both the knowledge-level (IVM mean =8.75±0.94 SD; VPF I mean =8.20±1.34 SD; Wilcoxon test, w=7,075; P<0.005) and application-level (IVM mean =12.65±0.94 SD; VPF I mean =12.04±1.41 SD; Wilcoxon test, w=7,075; P<0.005) items.

Discussion

This study demonstrates that learners seem to prefer a flipped classroom approach. The flipped classroom was rated more positively than the traditional classroom on many different characteristics. This finding relates favorably to current evidence. In a review of flipped classroom educational research, eleven studies examining student perceptions of the flipped classroom were identified and all reflected overall positive opinion.² From the data, we can see that flipped classroom students believed specifically that the approach sharpened their analytic skills and also increased their confidence levels in tackling unfamiliar problems.

An interesting finding was that students participating in the flipped classroom rated the teaching staff higher than those participating in the traditional classroom course. Teaching staff remained the same for both VPF I and IVM courses, yet the students’ perceptions of the staff were different. It is possible that the format of the flipped classroom, which allows greater opportunities for feedback to students and educator–student interaction, meant that the teaching staff were perceived to be more understanding of, and engaged with, their students. Alternatively, it is possible that the course redesign had an effect on teaching staff engagement and enthusiasm. It has been recognized in the literature that educators find flipped learning courses more stimulating to teach.²⁰ Flipped classroom students may have picked up on an increased level of positivity from the teaching staff, which was reflected in their responses to the questionnaire. Certainly, it would help to explore these findings further through a qualitative research method such as a focus group or interview process.

The increased educator–student interaction of the flipped classroom may also have had an effect on students’ perceptions of the clarity of the course offered and the expectations of teaching staff. Our results show a significant difference between the two cohorts; flipped classroom students were more likely to report that they had a clear idea of what was expected of them on the course. With more opportunities to talk to the educator, and ask questions, in class, it is likely that this provided a forum for staff and students to develop shared expectations.

Although the students demonstrated a preference for the flipped classroom, this did not translate into improved performance, as assessed by a series of multiple-choice items delivered during a written examination. This finding is in contrast to some flipped classroom studies, which have demonstrated improved examination performance.^7,21 In this study, students on the traditional course outperformed the flipped classroom students. There were also topic-level differences. For example, the flipped classroom students performed comparably to traditional students on the topics of career success and research skills. However, traditional students did outperform flipped classroom students on the topics of communication skills and financial planning. It is likely that this difference arises due to the level of detail covered in each of these topics. For example, the communication skills topic, representing a detailed introduction to an evidence-based consultation approach known as the Calgary-Cambridge model, covers a greater level of detail than the career success topic, which has a broader, more general discussion about being a veterinarian, setting your career goals, etc. The authors theorize that a greater quantity of content material can be covered in a simple, didactic lecture, as compared to a workshop of similar length, which centers on active learning.

If we consider these results alongside those from the CEQ, it is possible that the students’ learning changed in different, unexpected ways, outside the scope of the assessment used. For example, the students of the flipped classroom reported that they felt this teaching approach improved their analytical skills. It would be useful to check the accuracy of their predictions by using an alternative assessment method that could evaluate analytical skills to a “shows how” level.¹⁴

Limitations of the study

There are several limitations to this study, which should be highlighted. One of the main challenges is that the study reflected the use of two different teaching methods with two different cohorts. It was considered that the two first semester classes were sufficiently comparable in profile, but it is possible that there were inherent differences between the two class cohorts that were not controlled for. The framework of the veterinary curriculum operated at RUSVM made it logistically difficult to offer a course via two different teaching methods during one semester, which would have been a preferable route.

It should also be noted that there was a difference in study uptake between both cohorts; that is, 79% of IVM students agreed to take part, as opposed to 44% of VPF I students. For the purposes of this study, we have assumed that both samples are representative of their respective cohort because the opportunity to participate was offered in identical methods (ie, at the same stage of the academic calendar, through the same methods, with consistent informed consent). It is possible, however, that the difference in level of participation could contribute to some of the findings.

Recommendations for building on this study would be to include a more open-ended, qualitative study of student perceptions, eg, through focus groups. Also, as discussed above, it may be that a different assessment tool, eg, one that measures students’ communication and analytic and teamwork skills, would be more appropriate to measure the impact of the flipped classroom on student performance.

Conclusion

This study demonstrates that learners seem to prefer a flipped classroom approach. The flipped classroom was rated more positively than the traditional classroom on many different characteristics. This preference, however, did not translate into improved student performance, as assessed by a series of multiple-choice items delivered during a written examination.