The Case-Based Collaborative Learning Peer Observation Worksheet and Compendium: An Evaluation Tool for Flipped Classroom Facilitators

Abstract

Introduction

In a flipped classroom, students learn basic concepts before class, allowing them time during class to apply newly gained knowledge to problem sets and cases. Harvard Medical School (HMS) has introduced a form of flipped classroom, called case-based collaborative learning (CBCL), during preclinical curricula. Finding few published resources, the HMS Academy's Peer Observation of Teaching Interest Group developed a guide for observations and feedback to CBCL facilitators.

Methods

After conducting an extensive literature search, speaking to flipped classroom methodology experts, and observing 14 facilitators using CBCL methods, the interest group identified specific teaching behaviors that optimize student interaction and knowledge application. The group next engaged in several rounds of the modified Delphi method to develop the CBCL peer observation worksheet and compendium and then tested these materials' effectiveness in capturing CBCL teaching behaviors and providing feedback to CBCL faculty facilitators.

Results

Seventy-three percent of faculty rated the worksheet and compendium as extremely helpful or helpful in identifying new teaching techniques. Moreover, 90% found the CBCL peer observation and debriefing to be extremely helpful or helpful, and 90% were extremely likely or likely to incorporate peer suggestions in future teaching sessions.

Discussion

Medical schools have begun to embrace flipped classroom methods to eliminate passive, lecture-style instruction during the preclinical years of the MD curriculum. This tool identifies specific in-classroom approaches that engage students in active learning, guides peer observers in offering targeted feedback to faculty on teaching strategies, and presents consensus-based resources for use during CBCL faculty development and training.

Educational Objectives

By using this tool, faculty members will be able to:

• 1.Compare and contrast the flipped classroom approach with traditional, lecture-style instruction.
• 2.Identify seven teaching categories associated with the case-based collaborative learning (CBCL) method.
• 3.Provide feedback to a peer on specific teaching behaviors relating to the CBCL instructional method.

Introduction

The medical education community has slowly begun to incorporate flipped classroom pedagogy into medical school classrooms.¹ In a flipped classroom, students are responsible for learning basic concepts and foundation knowledge outside of class time by reading materials, reviewing concept videos, watching prerecorded lectures, and/or completing written assignments. By learning basic concepts before class, students, under the guidance of faculty facilitators, are then able to engage in application, analysis, synthesis, and evaluation of new knowledge, skills, and behaviors.² As students move from passive to active participation, they develop critical thinking, inquiry-oriented strategies, and problem-solving skills.^3–5 Through this process, students alternately work alone or together to devise conceptual frameworks, apply knowledge to clinical cases, and engage in hypotheses generation and testing. Emerging literature on flipped classrooms has been mostly descriptive and theoretical in nature,^5,6–11 but early research has shown that the use of this pedagogy significantly improves students' performance when compared with that of students who learned the same material in a traditional format the year prior.¹²

Building on the success of the flipped classroom format, Harvard Medical School (HMS) created a novel pedagogy called case-based collaborative learning (CBCL). CBCL is a method of instruction that puts profound importance on student learning strategies as it fuses elements of problem-based, case-based, and team-based learning with peer instruction. Similar to the flipped classroom method, students are expected to prepare content-based assignments and answer readiness assessment questions prior to their class. In class, students work in small groups to review the assignment; consider new cases and problem sets; answer focused, open-ended questions; and achieve consensus on tasks emphasizing active learning and critical thinking.

In February 2013, HMS piloted CBCL with a group of first-year students enrolled in a 6-week Integrated Human Physiology (IHP) course.¹³ A randomized controlled trial found that students whose prior exam performance was below the median of all participants had a significantly higher mean score on the course's final exam if they had taken the CBCL curriculum instead of the problem-based learning curriculum.¹³

Simultaneous to HMS's piloting the CBCL method, we, the authors of this publication and members of the HMS Academy's¹⁴ Peer Observation of Teaching Interest Group, began to investigate teaching behaviors that define the successful application of CBCL methodology. Our aim was to describe behaviors we could use to conduct peer observation of a CBCL classroom. Given the limited use of the flipped classroom approach in medical school teaching, we felt it necessary to create our own teaching resources. Following a 3-year process of development, implementation, and testing, we created the CBCL peer observation worksheet and compendium presented here.

This tool is intended for health professional leaders who plan to initiate flipped classroom instruction and those who have already converted their curriculum. In either instance, this worksheet and compendium should be used as part of faculty development efforts to help faculty consider how effectively they are implementing techniques necessary to lead a successful flipped classroom.

Methods

Instrument Development

Members of the HMS Academy's Peer Observation of Teaching Interest Group developed this tool over a 3-year period in order to create a peer observation instrument and compendium of CBCL behaviors. In September 2013, we conducted an extensive literature search (see included bibliography, Appendix A) to devise a list of teaching behaviors that encourage student-centered, interactive learning that in turn fosters critical thinking, collaboration, and reflection. We then conducted an observational study of the IHP course, which was using flipped classroom methods. These observations helped to further identify specific teaching behaviors that encourage active learning, knowledge application, and problem solving. To begin to refine our list of teaching behaviors, we engaged the other interest group members and IHP course faculty in several rounds of the modified Delphi method to rate the importance of each teaching criterion in the proposed CBCL peer observation worksheet. Two Delphi rounds were completed before we reached consensus on the worksheet. We spent a year observing 14 faculty during the first year of their new CBCL-based pathways curriculum. This enabled us to test the effectiveness of the worksheet and compendium in capturing the teaching behaviors demonstrated during CBCL classes as well as in providing feedback to CBCL faculty facilitators in real time.

The Instrument

Based on the feedback from participants in the modified Delphi method group, the CBCL peer observation worksheet (Appendix B) is divided into two parts: (1) the seven CBCL-specific instructional criteria and (2) the seven elements of effective instruction. The latter are the elements a peer observer would expect to see demonstrated during any session (traditional or flipped classroom). The seven CBCL-specific instructional strategies included in the worksheet and compendium promote learner-centered instruction within the classroom. These CBCL-specific strategies include the following:

• 1.Connect prior learning and preclass assignment to in-class activities.
• 2.Prompt deeper learning by using one or more active-learning strategies.
• 3.Respond to students' questions in ways that promote further learning.
• 4.Use a variety of learner-centered activities to engage students in the application, transfer, or generation of knowledge.
• 5.Conduct frequent, nongraded assessment of students' understanding to allow for immediate feedback and inform real-time instruction.
• 6.Co-teach with other faculty in a coordinated and collaborative manner.
• 7.Balance instruction with students' self-directed learning.

The compendium (Appendix B) further identifies demonstrable behaviors associated with each of the CBCL categories in order to facilitate a shared understanding among faculty of the instructional strategies and teaching behaviors that optimize student engagement, collaboration, critical thinking, reflection, and self-directed learning.

While there are no prerequisite skills or experience needed to use the worksheet, we include a detailed description of the reasoning behind each of the seven CBCL instructional strategies (Appendix C) and also encourage faculty to review the previously published “Peer Observation of Teaching Handbook.”¹⁵

Instrument Implementation

The CBCL peer observation worksheet and compendium can be used for faculty employing any flipped classroom pedagogical method. Prior to the session to be observed, both the observer and the faculty member should read through the entire worksheet and compendium (Appendix B) to familiarize themselves with effective criteria of flipped classroom teaching. Both should pay particular attention to the provided examples that describe behaviors associated with each criterion. At this time, the faculty member who is being observed should notify the observer about the three or four teaching criteria on which the faculty member would like to receive feedback. By paying attention to a limited number of teaching strategies in a single session, the observer is able to provide more in-depth accounting of what occurred during the postobservation debrief.

During the observation, the observer should take note of specific times when students are actively engaging with and applying medical knowledge in classroom discussion or activities. We also suggest making note of how much time students spend working individually or in groups to solve problems, the types of questions the faculty member asks to encourage critical thinking, and the ways students express their understanding of a particular topic or concept. Furthermore, the observer should make note of the seven elements of effective instruction included on the CBCL worksheet, being sure to identify best practices or areas of instruction that need further development.

Following the observation, the observer and faculty member should reserve a minimum of 20 minutes to debrief the teaching encounter. The peer observation debrief should follow standard feedback practice: Allow the faculty member to first share his or her feelings about the teaching encounter, focus the debrief on the areas for which the faculty member has requested feedback, and have the observer mirror the classroom behaviors he or she witnessed back to the faculty member. This should be a mutually beneficial conversation to identify best classroom methods and to generate solutions for difficult learning situations. The observer should conclude the debrief by creating an action plan with the faculty member.

Results

To determine if we had successfully met the three educational objectives given above, we had 10 HMS Academy's Peer Observation of Teaching Interest Group members observe 14 faculty members leading classes using the CBCL method during the first year of the new HMS Pathways Curriculum. The 10 observers were HMS clinician educators who practice across specialties including internal medicine, pediatrics, obstetrics and gynecology, adolescent medicine, hospital medicine, gastroenterology, physical and rehabilitation medicine, and women's health. All were experienced observers who had received advanced training in peer observation of teaching. The 14 faculty members who led the CBCL sessions were clinician educators practicing in the fields of cardiology, dermatology, endocrinology, gastroenterology, immunology, nephrology, pathology, pulmonology, and rheumatology. These were mid- to senior-level faculty members, including an associate dean, course directors, and course instructors.

After each observation, we collected quantitative and qualitative feedback from both the peer observers and the faculty members to determine if the CBCL observation resources successfully captured relevant instructional strategies and teaching behaviors needed to lead a medical school course using flipped classroom methods. Furthermore, we surveyed the 14 faculty members after the first implementation of the CBCL curriculum and peer observation experiences and received mostly positive responses.

When asked to compare CBCL flipped classroom methods to more traditional, lecture-style methods, 100% of faculty members completely agreed or agreed that with the CBCL method:

• •They encouraged more critical thinking during a teaching session.
• •They more frequently encouraged students to use higher-order thinking skills (i.e., application, analysis, evaluation).
• •They used more learner-centered activities during a session.
• •They were more likely to stop and assess the students' understanding of the content.

In addition, 90% of the faculty members completely agreed or agreed that they encouraged more student self-directed learning, which resulted in students being more actively engaged during a CBCL teaching session than a traditional, lecture-based session.

Survey responses showed that faculty found the CBCL observation resources to be helpful—73% rated the CBCL peer observation worksheet and compendium to be extremely helpful or helpful in identifying new teaching techniques to use when leading a session. Specific comments about the compendium included the following:

• •“It reinforced the behaviors associated with effective teaching in CBCL—now that I have been doing this for 6 weeks, it was helpful to review my own behaviors and it served as a reminder of some techniques that I had not been utilizing.”
• •“I went through the compendium several times before the session; useful to think through teaching behaviors ahead of time.”
• •“It was specific and I could clearly see when I was and was not using the techniques appropriately.”
• •“The worksheet and compendium are very comprehensive for training on using CBCL in my classroom and for feedback on how well I led the sessions.”

Faculty who participated in the peer observation reported a positive experience. When asked how helpful the experience was, 90% found the CBCL peer observation and debriefing to be an extremely helpful or helpful experience. The majority of faculty (87%) were also satisfied with the type of feedback they were offered, and 90% were extremely likely or likely to incorporate peer suggestions in their future teaching sessions.

Last, we asked the faculty to identify what new teaching strategies they used in the CBCL classroom and what they noticed about the impact these strategies had on students' engagement during their course. The new teaching techniques most frequently named included the following:

• •Starting the session with a case instead of a lecture.
• •Having students solve problems in small groups at their tables.
• •Having students come to the whiteboard to teach the rest of the class or share explanations of illustrations/representations with the group as a whole.
• •Asking students to commit to an answer before sharing with the class.
• •Having each table of students work on a piece of a problem and then integrating these responses during the class discussion.
• •Having students peer-teach when others do not understand a concept.
• •Asking open-ended questions or questions designed so that students will generate their own hypotheses.

Based on this participant feedback, we believe these CBCL peer observation resources successfully capture relevant instructional strategies and teaching behaviors needed to lead an effective medical school course using flipped classroom methods.

Discussion

Although medical schools have begun to embrace flipped classroom methods to eliminate passive, lecture-style instruction during the preclinical years, there are few publications that define strategies for engaging students in active learning during a flipped classroom session. Furthermore, for those who do implement flipped classroom techniques, there have been no existing resources to evaluate the efficacy of instruction for formative assessment and faculty development purposes. To ensure that we were able to provide standardized feedback to those implementing flipped classroom methodologies, we developed this instrument and resources for conducting peer observation of flipped classroom teaching.

After a lengthy development, implementation, and evaluation process, we believe these CBCL peer observation resources successfully capture relevant instructional strategies and teaching behaviors needed to lead an effective medical school course using flipped classroom methods. Other benefits of this work have been to identify specific, in-classroom pedagogic approaches that engage students in active knowledge application, problem solving, and critical thinking; to offer detailed resources that allow peer observers to determine if faculty members are using flipped classroom methods effectively; to guide peer observers in providing targeted feedback to faculty on their CBCL strategies; and to present consensus-based resources for use during CBCL faculty development and training.

As of 2017, 50 HMS faculty who teach in the preclinical curriculum have been taught about the CBCL method in faculty development sessions that use the peer observation worksheet and compendium as training resources.

The generalizability of this work is limited by the fact that the development, implementation, and evaluation of the CBCL peer observation worksheet and compendium were done at a single institution. Moreover, the resources were tested during the initial year of the medical school's new curriculum with a small group of faculty members.

Appendices

A. CBCL Bibliography.docx

B. CBCL Workseet & Compendium.docx

C. CBCL Constructs.docx

All appendices are peer reviewed as integral parts of the Original Publication.