Choosing Wisely in Pre-clinical Medical Education: a Direct Comparison of Active Learning Methods for Teaching Metabolic Liver Disease

Abstract

Background

With the movement away from lectures and towards active learning sessions, medical school faculty must choose a format that fits the learning objectives and is achievable with available resources. Small groups bring organizational challenges including additional faculty and classrooms. Large groups are an opportunity to conduct case-based active learning exercises with fewer faculty. Our study compares the learning effectiveness of an active learning session on metabolic liver disease conducted in both the small group and large group formats.

Methods

All MS1 students at University of Vermont Larner College of Medicine (LCOM) (n = 120) were randomized to participate in either the small or large group session. The same pre-learning videos and computer-based active learning module were used for all students in both session types. A post-session questionnaire was administered, and student exam performance was analyzed to evaluate each session type on the following criteria: (1) student preparedness for the session, (2) student perceptions of the session’s learning effectiveness, (3) learner knowledge upon completion of the session, and (4) knowledge retention at 10 days.

Results

No statistically significant differences were found between the large and small group cohorts on student perceptions of learning effectiveness or knowledge assessment, both immediate and delayed. Students assigned to the large group did choose their collaborators differently than those in the small group, tending more often to work with their friends.

Conclusion

When organized well, a large group active learning format may be used in place of a small group without diminishing the learning effectiveness of the activity.

Electronic supplementary material

The online version of this article (10.1007/s40670-019-00821-2) contains supplementary material, which is available to authorized users.

Introduction

Medical education has undergone a pedagogical transformation with movement away from the classic lecture format and towards active learning activities. Among the noted faculty challenges in approaching the task of converting lectures is choosing which active learning method to use [1]. The small group format is well established for both case-based learning and problem-based learning and is familiar to most faculty educators. Learning in small groups has been shown to improve learner engagement and achievement, especially for female and underrepresented minority students [2]. The format offers students opportunities for peer-to-peer teaching, higher level application of concepts, and direct faculty facilitation.

Compared with lecture, however, the small group format is far more resource-intensive, as it requires greater numbers of faculty to conduct each session [3]. Small groups are among the most expensive types of active learning sessions, as they utilize large numbers of faculty and classroom spaces. In academic centers, it is not uncommon for small group sessions to draw the majority of the faculty from the same department. This effectively renders a substantial cluster of faculty members unavailable for clinical duties during the session, which potentially strains a department during busy clinical periods.

Applying small group curriculum development to a large group setting may be one way to reduce both cost and logistic burdens. However, this should not come at the expense of effective, objective-driven learning. To directly compare learning effectiveness, we developed a 90-min active learning session on metabolic liver diseases and conducted the session using both the small group and large group models. The primary objective of this study was to examine whether a small group/case-based learning session can be adapted to a large group/case-based learning format without diminishing the overall learning effectiveness of the session. A secondary objective was to better understand student preferences regarding group size and choice of collaborators in active learning environments.

Materials and Methods

We developed an active learning session on metabolic liver disease and implemented it in March 2018, during the required, integrated MS1 course “Nutrition, Metabolism, and Gastrointestinal System” (NMGI). Standard to the Larner College of Medicine (LCOM) active learning curriculum, the session consisted of pre-session preparation work followed by an in-person active learning session focusing on the application of basic science knowledge and clinical decision-making. Students were randomized to participate in the session in either a small or large group format. In this study, a small group was defined as 15 learners based in one classroom with a single faculty facilitator. A large group was defined as 60 learners based in one classroom with three faculty facilitators. Using a post-session participant questionnaire, multiple choice knowledge assessment questions, and exam performance data, the effectiveness of each session format was evaluated on the following parameters: (1) student preparedness for the session, (2) student perceptions of the session’s learning effectiveness, (3) learner knowledge upon completion of the session, and (4) short-term (10 days) knowledge retention.

This topic was previously taught through lectures developed and delivered by a hepatologist. With assistance from The LCOM Educational Technologies (EdTech) Team, this faculty lecturer condensed and converted approximately 2 ½ hours of lecture material into four pre-session independent learning Camtasia video modules totaling 36 min in length. The teaching objectives from previous years remained unaltered with a single addition pertaining to writing an assessment and plan for the case patient (Fig. 1). All MS1 participants viewed the same pre-session content videos and were required to participate in a 9-question online readiness quiz (RQ) prior to the in-class session. The RQ questions were written to assess the lower order Bloom’s taxonomy session objectives with the purpose of optimizing student preparedness for the in-class active learning session. The timed RQ allowed for collaboration and a total of 3 possible attempts.

Fig. 1.

Participants were asked to view four videos covering basic content prior to the active learning session. This knowledge was applied to a patient case during the active learning session

The 90-min in-class active learning activity consisted of a computer-based patient assessment exercise. Using hyperlinks to progress through a non-linear slide show (created in Microsoft PowerPoint), participants were guided through a patient assessment exercise that required them to choose the direction of their information gathering. The patient in the case is a college-age female who presents to the student health center with vague, non-specific symptoms. Although the students were aware that the etiology of her symptoms is most likely one of 4 metabolic liver diseases, they were instructed to develop broad differentials. They narrowed their differential diagnoses as they worked through the module, gathering patient history, review of systems, physical exam data, imaging, and laboratory results (Fig. 2). Near the end of the module, they were asked to write an assessment and plan (one per group) that they submitted to an instructor via email. Feedback on these submissions was later provided.

Fig. 2.

The interactive module used during the active learning session allowed students to gather subjective and objective patient assessment data by clicking on hyperlinks to view results. Results were often accompanied by questions, prompting students to take notice of key teaching points

Half of the class of 2021 (n = 60) was randomized to participate in this session in the small group format (15:1 student to faculty ratio). The other half of the class (n = 60) was randomized to participate in the session in a large group format utilizing 3 roving faculty (20:1 student to faculty ratio). In both the small and large group sessions, students were encouraged to work through the module in groups of any number up to four students. Working independently was permitted, though not encouraged.

In the small group sessions, a single faculty facilitator was available to answer questions and help students with time management. The facilitator led periodic discussions with the whole group throughout the session, soliciting findings and contributions from the smaller sub-groups. In the large group session, each student sub-group was provided with a green card that they could place in a wire stand to request the help of a faculty facilitator. Three faculty facilitators roamed the room, responding to student questions. A single faculty facilitator in the large group session brought the whole group together after 30 min and again in the final 10 min of the session to explain the answers to commonly asked student questions and highlight key teaching points for the session.

A post-session, mixed multiple choice open-ended questionnaire was administered (using Lime Survey) immediately following the sessions. In this questionnaire, students in both cohorts were asked to answer questions designed to assess the following: (1) perceptions of the session’s effectiveness, (2) information about how students chose their collaborators for the activity, and (3) how they self-identify on a scale of introversion/extroversion. In addition, this questionnaire contained four multiple choice questions that immediately assessed their knowledge acquisition. These questions were written to a level of difficulty comparable with USMLE Step 1. Students were required to complete the questionnaire within 24 h of the end of the session. Ten days later, a cumulative exam was administered. Student performance on five exam questions pertaining to the content of the session was analyzed to assess knowledge retention. All data from the post-session questionnaire, as well as the exam, was de-identified prior to being analyzed for inclusion in the study.

The student responses to the questionnaire were analyzed using cross tabulation and the Pearson chi-square test. A t test was also performed on the student exam data to compare the mean percent correct between the small and large group cohorts. Analyses were conducted in SPSS. This study was reviewed by the IRB and did not meet the definition of a “research” activity under the regulatory definition adopted by the University of Vermont.

Results

Student Preparation

Due to the allowance of multiple attempts, nearly all 120 students received a final score of 100% on the pre-session RQ. The post-session questionnaire item measuring how students prepared (“Which answer best describes the primary method you used to prepare for this active learning session?”) resulted in the following: 84.7% of small group participants and 87.3% of large group participants reported watching all four pre-learning videos one time. In the small group cohort, 8.5% watched all four videos more than once, as compared with 7.9% of the large group. No participants in the large group reported using a text book or other outside resource, while four participants (6.8%) assigned to a small group used a resource other than the videos. Three students (4.8%) in the large group used the videos but did not watch all four videos in their entirety. No one from the small group selected this response. There was no statistically significant difference between the small and large group cohorts in how they chose to prepare for the active learning session (p = 0.068).

Student Perceptions of Learning Effectiveness

Five out of nine questions on the post-session questionnaire were dedicated to student perceptions of the effectiveness of the activity. Nearly all students in both cohorts (93.2% of small group participants and 98.4% of the large group) responded that the session met the stated objectives, revealing no significant difference between cohorts (p = 0.148). When asked, “Did you have enough time to finish the planned activity of the session,” 93.2% of small group participants and 88.9% of large group participants responded “yes.” This difference was also not significant (p = 0.404) (Table 1).

Table 1.

Evaluating student perceptions of learning effectiveness, time management, and access to learning resources during the active learning session. No statistically significant difference was found between the small group and large group session types

	Yes	No	p value
Did the session meet the stated objectives?
Small group (n = 59)	55 (93.2%)	4 (6.8%)	0.148
Large group (n = 63)	62 (98.4%)	1 (1.6%)
Total (n = 122)	117 (95.9%)	5 (4.1%)
Did you have enough time to finish the planned activity of the session?
Small group (n = 59)	55 (93.2%)	4 (6.8%)	0.404
Large group (n = 63)	56 (88.9%)	7 (11.1%)
Total (n = 122)	111 (91.0%)	11 (9.0%)
Did you receive satisfactory answers to your questions/explanations of confusing concepts during this session?
Small group (n = 59)	49 (83.1%)	10 (16.9%)	0.352
Large group (n = 63)	56 (88.9%)	7 (11.1%)
Total (n = 122)	105 (86.1%)	17 (13.9%)
Who provided these answers?
Small group (n = 59)			Faculty, 0.844 Classmate, 0.709
Faculty	45 (91.8%)	4 (8.2%)
Classmate	37 (75.5%)	12 (24.5%)
Large group (n = 63)
Faculty	52 (92.9%)	4 (7.1%)
Classmate	44 (78.6%)	12 (21.4%)

For the question, “Did you receive satisfactory answers to your questions/explanations of confusing concepts during this session,” an answer of “yes” prompted an additional question to appear about who provided these answers. Respondents could choose multiple answers to the supplemental question, with “classmate(s)” and “faculty” provided as options. The vast majority of students (83.1% of small group participants and 88.9% of large group participants) responded “yes” to this question. Of those who responded “yes,” faculty provided answers for 91.8% of those who were small group participants and 92.9% of those in the large group. Classmates answered questions for 75.5% of those in the small group and 78.6% of those in the large group. There was no statistically significant difference between cohorts on either component of this question, revealing that most students had questions answered and explanations provided by both faculty and classmates, and students in the large group did not experience less access to faculty, despite having a higher student to faculty ratio (Table 1).

One measure of learning effectiveness was the amount of study time that students expected to spend reviewing the session material after the active learning session, in order to prepare for the cumulative exam. Students were asked, “How much time do you expect to spend studying hemochromatosis, Wilson’s disease, alpha-1 antitrypsin liver disease, and steatosis/steatohepatitis prior to the next exam?” Out of all respondents, only one student (from the small group cohort) responded that they would not study these topics again before the exam. The greatest proportion of respondents (47.5% of the small group participants and 52.4% of the large group) was expected to spend more than 1 h, but not more than 2 h, studying these topics. Similar numbers of students fell on either side of this response, with 27.1% of small group students and 30.2% of large group students expecting to spend 1 hour or less, and 23.7% of small group students and 17.5% of large group students expecting to spend more than 2 h. Although there was a wide distribution of students across responses to this question, there was no statistically significant difference between the large and small group cohorts (p = 0.594).

The final question that was used to evaluate student perceptions of learning effectiveness was, “Were there any concepts taught in the prep materials or active learning session that you still ‘don’t get’ and will need to look up/further clarify after the session has ended?” Nearly one-third of all respondents replied “yes” (30.5% of small group participants and 25.4% of large group participants), with no significant difference between cohorts (p = 0.529). For those responding yes to this question, a supplemental free text question was offered to further assess which concepts remained unclear. Most of the free text responses described minor concepts or details that were not integral to the objectives of the session.

Knowledge Assessment and Retention

Four formative assessment questions were written in the style of USMLE Step 1 questions and included in the post-session questionnaire. As students were required to complete the questionnaire within 24 h of completing the active learning session, these questions were designed to measure knowledge acquisition and immediate comprehension, as opposed to retention. All were two-step questions with relatively long question stems that contained several pieces of clinical data. For three out of the four questions asked, the percentage of students answering correctly ranged from 83.9–89.3% of the small group participants and 75.0–86.7% of the large group participants. One out of the four questions asked had a greater percentage of students responding incorrectly, with only 60.7% of small group participants and 65.0% of large group participants answering correctly. There was no significant difference in performance on any of these questions between the large and small groups (p = 0.046–0.931).

Short-term knowledge retention was measured using five multiple choice questions pertaining to the content of the session on a cumulative exam, which was administered 10 days post-session. These questions were similar in the level of difficulty to those used in the post-session questionnaire. The percentage of students answering all five exam questions correctly was 70.0% for small group participants and 76.7% for large group participants. Four out of five questions were answered correctly by 21.7% of the small group participants and 18.3% of the large group participants. The number of students who answered only two or three out of five questions correctly was negligible. There was no significant difference in exam performance on these questions between the small and large group cohorts (p = 0.452).

Additional Questions of Interest

To address the secondary objective of the study, three questions in the post-session questionnaire were written with the intent of gathering data regarding student behavior and preferences in a group learning environment. When asked how many peers students chose to collaborate with, only one student in each cohort elected to work alone. Most students (66.1% in the small group cohort and 54.0% in the large group) chose to work in groups of the maximum recommended size of four. The remainder of the students worked either in groups of three (20.3% of small group participants and 14.3% of large group participants) or ignored the recommendations and worked in groups that were larger than four (11.9% of small group participants and 22.2% of large group participants). Generally, participants showed a preference for working in larger sub-groups, independent of the cohort to which they were assigned (p = 0.092).

Students were asked how they chose their collaborators for the active learning session, and their responses to this question revealed a statistically significant difference between the small and large group cohorts (p = 0.005). Among the large group participants, 33.3% reported “I worked with my friend(s),” while 60.3% selected “I worked with the person(s) seated closest to me.” Among the small group participants, a greater proportion of students (88.1%) reported working with the person(s) seated closest to them, with only 8.5% choosing to work with friends. Respondents also had the option of choosing “I worked with the person(s) that I know are well-matched with my pace or level of preparedness,” but only 4.8% of large group participants and 1.7% of small group participants selected this option.

Since a transition to active learning involves an increase in the amount of group and cooperative learning experienced by students, we were interested in how students self-identify on the introvert-extrovert spectrum. Students were randomly assigned to either the small or large group cohort, so there was no possibility of self-selection into either session type. As expected, there was no significant difference between cohorts. However, within the class as a whole, 47.6% of respondents identified as either “slightly introverted” or “totally introverted.” A smaller proportion of the class (27.1%) identified as either “slightly extroverted” or “totally extroverted,” with 25.4% responding “neutral.”

Discussion

Student preparedness was examined due to a concern that the social pressure and close proximity to faculty that occur in a small group may motivate students to prepare more thoroughly for small group sessions than large group activities. Our study found an insignificant difference in how thoroughly students prepared, which may be due to chance. However, the small differences that we observed between the large and small group cohorts may suggest that, for a minor number of students, the intimacy of a small group may encourage more thorough preparation for the active learning session.

The vast majority of students in both cohorts believed that the active learning activity met the stated objectives. Since the same pre-learning materials and active learning modules were used in both the small and large group sessions, it is predictable that no significant difference between cohorts was found on this question. However, this finding is helpful in further evaluating the delivery modalities of the material, as there is wide and consistent agreement between both cohorts that the content of the session is effective for meeting the learning objectives.

Time management is one of the most challenging aspects of active learning for a faculty facilitator. Students often complain that there is either too much or too little material in a session. If there is too little content, students perceive that their time is being wasted and they would be able to accomplish more learning by studying independently. If there is too much content, students may not finish the activity and often become overwhelmed by the amount of information that they will have to learn independently, without the benefit of faculty facilitation. Since students all work at different paces, it is challenging to create a group learning activity that is the perfect length for all learners. Whether in a large group or a small group, it is important that the facilitator assist students in moving through the activity at a pace that supports completion within the time allotted. One might expect that this would be easier to do in a small group setting, as the facilitator is in a better position to monitor the progress of all students. However, when we asked students about their ability to complete the activity within the time allotted, we found no significant difference between the small and large group cohorts. This supports the notion that, although time management may be more challenging in a large group setting, it is quite possible to do it successfully. In our large group session, a facilitator made two planned disruptions in the student-guided activity (one at 30 min and one at 80 min) for the purpose of answering common student questions and summarizing key teaching points for the entire group. These pauses also served to help students with time management and ensure completion of the module within the time allotted.

Small groups may have a perceived advantage over large groups due to increased faculty attention. However, when we asked students whether they received satisfactory answers to their questions and explanations of confusing concepts during the session, between 80 and 90% of all students responded “yes,” with no significant difference between cohorts. Though it was not surprising that classmates answered students’ questions with similar frequency in both the large and small groups, it is revealing that students also received answers from faculty with similar frequency in both cohorts. This shows us that, in the large group format, even with a larger student to faculty ratio, students are still receiving comparable opportunities for discussion with faculty. This finding suggests that the increased faculty to student ratios enabled by small groups may not be worth the significantly higher costs. According to our 2017 institutional data, the faculty cost for a small group active learning session involving ten LCOM faculty (1 faculty:12 students) was $17,900 per hour of instruction. By comparison, a team-based learning (TBL) session of the same length conducted by two faculty members in a large group setting cost $7166. If a large group session can be designed to provide ample opportunity for students to ask questions directly to faculty, its value to students may be comparable with a small group at a much lower cost to the institution.

Active learning is designed to provide medical students with opportunities to apply basic science concepts in clinical scenarios, with the goal of enhancing critical thinking and knowledge retention. When successfully executed, students should not have to spend as much time independently learning material as they do in a lecture-based curriculum. Although there was no significant difference between the large and small group participants when they were asked how much time they would spend studying the session content prior to the next exam, we believe that this question is an important overall measure of the learning effectiveness of an active learning activity. There is no question that active learning takes up more class time than lecturing, and the best way to justify that time increase is by reducing the time that students spend returning to material already covered in active learning sessions. Our goal for this active learning activity was that most students would have learned these entities so well during the session that they would spend no more than 1 h reviewing them prior to the next exam. However, half of the class expected to spend between 1 and 2 h studying them. Considering the volume of material, this may be acceptable, but it should remain a primary goal of faculty to focus on maximizing the efficiency of in-class learning and work to reduce this number.

Knowledge assessment using multiple choice questions was the most objective measurement of learning effectiveness used in this study. We found no significant difference between cohorts on either immediate or delayed (10 days post-session) knowledge assessment. It is reassuring that student performance is consistent in both the immediate and delayed settings, as it supports the assumption that the consistency in exam performance that we observed 10 days post-session is not due to variations in the amount of independent studying between cohorts. The immediate knowledge assessment results confirm that there were no knowledge gaps that emerged with greater frequency in one group more than the other.

The additional questions that we posed to students about the size of their work group, their rationale for selecting collaborators, and their identity as introverts versus extroverts were meant to help us better understand possible social barriers to learning in an active learning environment. Although we cannot draw many conclusions from these results, they indicate that the topic may be worthy of further exploration. It is common for students to express a preference for independent learning over group learning. However, when they were given the choice of working independently versus in a group during our active learning activity, nearly all participants chose to collaborate with peers. Furthermore, they tended to work in groups of the maximum recommended size, with a notable proportion of students collaborating with more than the maximum recommended number of collaborators.

When students were asked how they chose their collaborators, their responses revealed the only statistically significant difference between the large and small group cohorts in our entire investigation. In small groups, most students worked with the people seated closest to them, while in the large group, most people worked with their friends. This result is intriguing because students often claim that collaboration slows down the pace of learning. However, when given the opportunity to choose their collaborators, very few people selected other students for their pace or preparedness.

Nearly half of all participants identified as either slightly or totally introverted. With one quarter of respondents identifying as “neutral” (i.e., neither introverted nor extroverted), extroverts were in the minority in our study. Although one might expect introverts to choose smaller group sizes, they tended to do the opposite in our study. It is difficult to speculate about whether students’ perceptions about their personalities affect their preferences and performance in group active learning activities. Nonetheless, this perception may be playing a role in student resistance to active learning. Collaboration with peers can be both valuable and also quite draining for students. In our active curriculum, we are expecting students to expend a lot of energy on the social aspects of active learning. There should be flexibility and choice built into most active learning sessions so that the greatest amount of energy can be spent on learning. When developing an active learning curriculum, it is important to remember that, while group learning plays an important role in active learning, not all active learning must be experienced in groups. Indeed, it is just as important for students to develop active learning skills that they can employ while learning independently. Independent learning remains the primary way that students prepare for USMLE exams, and independent learning will continue to play an important role in their continued learning as working physicians. Using active techniques during independent studying stands to benefit learners just as much as learning to work on teams and peer-to-peer teaching.

The findings of this study may not be generalizable, due to several institution-specific factors. The facilities available for large group active learning sessions may have a profound effect on how students experience the active learning session and whether it is effective for their learning. Student expectations of active learning and past experiences with similar session types likely affect their perceptions, as well.

An additional limitation of our study was a discrepancy between the number of students participating in the study (n = 120) and the number of post-session questionnaires that were completed (n = 122). We believe that this occurred because one student from the small group cohort mistakenly accessed the survey link for the large group, and two students partially completed the survey for the large group activity twice, increasing the survey count for this cohort.

Conclusions

Our study found that active learning in a large group setting is no less effective for learning than the same activity experienced in a small group setting. The next step in this investigation of active learning modalities may be to determine whether there are actually advantages to large group learning. As faculty, we have anecdotally observed a few encouraging advantages to teaching in this format.

The student-driven nature of active learning is a critical aspect of its effectiveness. Since all students work and learn at a different pace, the ability to individualize the pace of an activity may help students to learn more effectively in these sessions. In a small group setting, there can be a greater tendency for the faculty facilitator to do more teaching, thereby dictating the pace of the activity. In the large group setting, students may be in a better position to direct the activity themselves. When they request assistance from a facilitator, it is more likely that they will be receiving information in response to a question they came up with independently. This “timely” learning optimizes students’ readiness to receive critical information, which helps to minimize the all-too-common med student experience of being flooded with information that they were not yet prepared to organize or contextualize.

Faculty members are sometimes nervous about being the “expert” in the room during a small group session. For example, this may occur when a basic science faculty member is facilitating a clinical case-based exercise, or vice versa. Perhaps the activity requires specialized knowledge and the facilitator is minimally familiar with this area of science or medicine. In the large group setting, faculty can consult each other for support when answering difficult questions. This format is also advantageous for integrating teaching assistants, who may also prefer to be in close proximity to a faculty expert for help with addressing difficult student questions or clarifying teaching points. One of the most common concerns that students cite about small groups is the fear of inconsistent learning experiences between small groups. The large group format may help to allay this concern due to the ability of faculty to work more closely as a team.

One limitation to our study is that we did not specifically question students about their preference for small group versus large group learning. This was intentional because of the potential for student preference to confound our findings about the learning effectiveness of each format. A study similar to ours was conducted to compare small group learning with team-based learning, and their findings, like ours, showed no significant difference in students’ exam performance between the two formats. However, they did identify a student preference for the small group setting [4].

Noise control is one of the most challenging aspects of conducting a large group. It is necessary to have a classroom that is well-equipped with technology that minimizes ambient sound, amplifies individuals when they are addressing the whole group, and enables students to sit comfortably in their groups in a configuration that promotes discussion. Technology that enables screen sharing is also helpful. Depending on the size and layout of the available classroom, course directors may consider splitting the class in half for a large group activity and conducting the session twice.

Standard operating procedures should be developed so that faculty members are using facilitation and group management techniques that are consistent throughout sessions. A method for requesting the assistance of a facilitator, such as placing a card in a stand to signal the request, should be used so that students are not raising their hands for a long period of time. Hand-raising brings the group’s work to a halt and wastes time, in addition to frustrating students. Using the card-signaling method, many groups move on with the activity until a facilitator arrives, which may help to allay student frustration with the format.

Regardless of the group size used, the effectiveness of an active learning activity depends on quality facilitation. As faculty develop successful techniques for time and group management in the delivery of a well-designed active learning curriculum, we anticipate that large group activities will provide an increasingly popular and effective means of training future doctors.

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical Approval

Under the regulatory definition adopted by the University of Vermont, it was determined by the UVM Research Protections Office/IRB that this project does not meet the definition of a “research” activity.

Informed Consent

All participants were informed of the structure and intent of the project via email in advance of the session. Due to the determination of “not research,” signed consent was not required by the UVM Office of Research Protections.