Medical Student Engagement in a Virtual Learning Environment Positively Correlates with Course Performance and Satisfaction in Psychiatry

Abstract

Introduction

There has been a move to a “flipped classroom” (FC) in medical education. The FC promotes active learning and utilizes independent preparation prior to in-class sessions. Few studies have evaluated the effectiveness of the FC approach in medical education, specifically via virtual learning. The purpose of this study evaluates student and faculty perceptions of the FC approach and relationships between student engagement and performance.

Method

The first-year medical student psychiatry curriculum was redesigned with an FC approach and subsequently altered by COVID-19 to a virtual learning environment. A mixed-method approach was used to examine both qualitative assessment and quantitative performance data. Students and facilitators were invited to participate in surveys regarding the curriculum changes. Student performance data was collected via quizzes and examinations. Engagement was evaluated by student participation in National Board of Medical Examiners–style multiple-choice questions delivered via Top Hat®. Correlational analyses were used to evaluate associations between engagement and performance. T-tests were used to compare student satisfaction across 2019 and 2020.

Results

Performance on in-class questions was positively associated with class rank and performance (p < 0.005). More students were either satisfied or strongly satisfied (91.5%) in 2020 compared to 85.7% in 2019 (two-tailed t-test, p = 0.04). Most students (81.3%) preferred in-class questions to lectures. In 2020, 62.6% of student comments were positive regarding the psychiatry curriculum vs 33.3% in 2019. Over 61.5% of facilitators felt positive towards the changes.

Conclusion

Our results demonstrate a positive relationship between engagement and class performance. Students and facilitators positively perceived the approach, with students preferring in-class questions compared to lectures. Future research should evaluate overall performance on standardized tests, third-year clerkships, and number of students matching into psychiatry.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-021-01287-x.

Introduction

The “flipped classroom” (FC) has become a widespread blended approach to learning that has gained significant traction in medical student education [1]. The FC approach assigns coursework outside of the classroom and focuses in-classroom interactions toward “active learning” [2, 3]. The FC remains an area of active research and has been utilized at multiple levels of education.

There have been several studies of higher education FC outcomes in the Science, Technology, Engineering, and Mathematics fields [4–6] including medical education [7–11]. These studies demonstrate effectiveness in improving procedural skill development, increasing understanding of complex ideas, and improving material application. Furthermore, studies have shown a decreased failure rate among low-performing students as well as student and faculty preference for the FC approach versus traditional teaching methods [4, 12–15]. Recently, there has been an increased interest in the FC learning modality in medical school education due to a focus of the Liaison Committee on Medical Education standard 6.3, which emphasizes self-directed learning [16].

The psychiatry curriculum for first-year medical students (MS1) at the Indiana University School of Medicine (IUSM) was revitalized for the 2019–2020 academic year using FC principles. IUSM is one of the largest allopathic medical schools in the USA and, as of 2019, accounted for 1.67% of the total US Medical School matriculants [17]. IUSM includes nine campuses distributed across the state. Prior to the 2019–2020 academic year, psychiatry content was delivered by psychiatrists, psychologists, or basic scientists depending on availability. The planned revitalization was developed by the Psychiatry Curriculum Redesign Committee (PCRC) consisting of psychiatry residents and faculty with support from the Department of Psychiatry and approved by the Neuroscience and Behavior (N&B) course management team (CMT). The proposed changes included pre-recorded lectures that students watched before engaging in locally facilitated cases delivered to small groups using National Board of Medical Examiners (NBME)–style questions. Delivery of the curriculum in May 2020 was impacted by the Coronavirus disease 2019 (COVID-19) pandemic and required rapid transition to virtual delivery of all content. Additionally, locally delivered clinical content was replaced with single statewide presentations. Psychiatry material was taught by content experts delivering clinical questions and case discussion to all students simultaneously.

Overall, few studies have evaluated the effectiveness of virtual learning on medical student education, a delivery modality that was once an option turned into necessity driven by the COVID-19 crisis. Even fewer studies have looked at active learning in psychiatry education [11, 18]. Thus, this manuscript aims to (1) detail the development and evaluation of FC MS1 psychiatry curriculum, (2) evaluate the relationship between in-class student engagement and performance, and (3) evaluate the perceptions and feedback of medical students on the curriculum revision.

Methods

Course-related data review, analysis, and post-course survey creation were approved by the Indiana University Institutional Review Board. A mixed-method approach to analysis was applied given the nature of the collected data.

Course Development and Format

The psychiatry portion of the N&B course was the focus of revitalization for the 2019–2020 academic year. The course provides a comprehensive introduction to the structure, functions, and disorders of the human nervous system. Using an organ system approach, it integrates a strong foundational basic science framework (regional and systems neuroanatomy, neuroembryology, and neuropharmacology) with the appropriate clinical disciplines (neurology, psychiatry, and neuropathology). The 6-week, 120-h course occurs at the end of the MS1 curriculum. Prior to the course redesign, psychiatry content was delivered in 13 h of didactic lectures and small group sessions. The small group sessions constituted one-third of the psychiatry series and utilized a Case-Based Learning format with approximately 1–2 facilitators per 30 students. During these sessions, groups of 6–8 students answered 5–10 case-based multiple-choice questions followed by a facilitator-guided discussion. Student learning objectives and supplemental readings from Introductory Textbook of Psychiatry by Black and Andreason [19] were provided. There were minimal other requirements outside of class. At the end of the course, students were asked to participate in a standardized anonymous evaluation.

As a result of the revitalization, the N&B psychiatry content was revised and integrated with pharmacology instruction with FC model restructuring. Short lecture videos (10–30 min) for each of the following psychiatry topics were created by the PCRC for students to watch in preparation for the accompanying in-class sessions: Anxiety and Trauma, Childhood and Adolescent Disorders, Eating Disorders, Mood Disorders, Personality Disorders, Psychotic Disorders, Substance Use Disorders, Somatic Symptom Disorders, and Sexual Dysfunction/Gender Dysphoria. In addition, a final review and question-based session covering all psychiatry material was planned. Student learning objectives and supplemental readings, similar to prior course iterations, were provided. Please see Supplemental Table 1 for a visual comparison of curricular changes that occurred between the 2019 and 2020 academic years.

Prior to COVID-19 social distancing precautions, the updated material was to be delivered in-person at each of the 9 IUSM campuses. In-class time was scheduled for approximately 1–2 h per topic for a total of 13 h. Each session was to consist of two parts: a case with open-ended questions for group discussion, followed by NBME-style multiple-choice questions, which were written by the PCRC. After each question, important takeaways including answer explanations for the questions were provided. Question material with answers and explanations were made available to students after each session.

The pandemic further challenged the redesign. Given limited local expert availability, the N&B CMT decided to deliver the neurology, neuropathology, and psychiatry clinical content from the largest campus, Indianapolis, by live virtual sessions to all 373 students simultaneously using virtual meeting software (Zoom®). Session attendance was mandatory but not strictly enforced. Students were divided randomly into smaller groups for case discussion with content experts consisting of psychologists and psychiatry residents, fellows, attendings, and community practitioners. Small groups consisted of 20–30 students. Following the group cases, students returned to the larger Zoom® room for the NBME-style question and answer session, delivered using the Top Hat® software platform (Top Hat® Monocle Corp.). Real-time student answer distribution for each question was used to guide subsequent discussion. During these sessions, students could either ask the primary facilitator questions or post questions in the text chat, which was monitored by members of the PCRC and IUSM faculty volunteers. Most chat questions were answered in chat, engaging the primary facilitator as needed. At the end of each pre-recorded lecture and in-class session, students were invited to participate in an anonymous survey (see Supplemental Material). The post-session survey data was analyzed after each instructional day and the results were discussed with the class at the beginning of the next day. Changes were made as possible throughout the course with the aim of improving content delivery.

Assessment of Student Performance and Engagement

Assessment of student performance was also impacted by the pandemic. Historically, N&B course performance was evaluated using three block exams created by the CMT and delivered in-person using ExamSoft®, as well as a cumulative customized NMBE final examination. Because NBME rules at the time precluded virtual exam delivery, a new final exam was created by the CMT. The block exam development and content remained consistent with that of prior years, but traditional proctoring was not feasible during the 2020 academic year. Students completed the exam remotely and had to endorse the Honor Code.

Student engagement was assessed using Top Hat® electronic records. Top Hat® is a teaching platform and student response system that was used to deliver the NBME-style multiple-choice questions. To participate, students must use a web-enabled device. The software creates participation records (attendance, response time, accuracy etc.). Questions were coded with 1, 0.5, and 0 for correct, incorrect, and missing responses, respectively. Students who did not register or respond to questions for an individual session were classified as non-responders.

Student and Facilitator Perspectives

Course feedback for the N&B course has historically been obtained by a standardized, anonymous End of Course Evaluation. One PCRC member extracted psychiatry content relevant comments from the 2019 and 2020 standardized End of Course Evaluations and categorized them as positive (1), neutral (0), or negative (−1). The comments were independently reviewed by another PCRC team member and ultimately a faculty member.

During the 2019–2020 academic years, students were also asked to complete a separate, anonymous Psychiatry-Specific Global Survey after completion of all psychiatry sessions (see Supplemental Material). This questionnaire was adapted from prior studies with permission from the authors [20]. Facilitators of the psychiatric portion of the course were also invited to participate in an anonymous Facilitator Curriculum Redesign Survey.

Data Analysis

Student Performance and Engagement

Student performance data, including quiz and exam scores, were recorded individually for standard academic recording purposes. This data was matched to student engagement data and anonymized by a faculty member. Correlational analyses were performed using Microsoft Excel® including Solver and Analysis ToolPak (Office 365®, Microsoft Corporation).

Student and Facilitator Perspectives

Objective Assessment: Psychiatry-Specific Global Survey and Facilitator Curriculum Redesign Survey data was exported to Microsoft Excel®. The total number of each response type was totaled for each question, excluding non-responders.

To compare student course perspectives from 2019–2020, a two-tailed t-test of the average number of responses per question type was performed with the null hypothesis being there was no difference in student satisfaction between the academic years.

Subjective Assessment: The total number and percentage of positive, neutral, and negative student comments from the standardized End-of-Course Evaluation were determined with Microsoft Excel®.

Results

Student Performance and Engagement

A majority of students participated in the Top Hat®–based sessions. More than 80% of students engaged in at least 9 of the 10 sessions (See Supplemental Table 2).

Each student’s cumulative course exam performance was placed in rank order and compared to cumulative Top Hat® performance (Fig. 1). Class Rank 1 is the highest performing student. Overall, increasing Top Hat® performance was significantly associated with class rank (p = 3.3 e−4) although the overall explained variance was low. Participation and performance in the course by session is shown in Table 1 and Supplemental Table 3.

Fig. 1.

Cumulative Top Hat® score vs. final class rank, n = 372. Linear correlation and R2 value shown on the graph. Increasing Top Hat® cumulative score was significantly related to higher course performance (lower class rank)

Table 1.

Each day’s topic and response by Top Hat® Table 1. Top Hat® performance by day and topic. The high-yield review topic contained one question erroneously coded as 0.5 total points for the correct answer instead of 1 pt which left the total points per session as 7.5

Session day	Day 1	Day 2	Day 3			Day 4				Day 5
Topic	Mood disorders	Psychotic disorders	Anxiety and trauma	Substance use disorders	Personality disorders	Somatic symptom disorders	Sexual dysfunction/gender dysphoria	Eating disorders	Childhood and adolescent disorders	High yield review	Overall
Total points	21	24	17	17	10	8	10	8	21	7.5	143.5
Average	10.5	14.4	12.2	11.6	7.1	5.6	6.6	5.2	14.5	5.1	93.1
Standard deviation	5.7	7.0	4.8	5.9	3.0	2.3	3.1	2.5	6.8	2.5	36.2
Mode	0	0	15	0	8.5	7	8.5	7	0	6.5	0
Total non-responders	66	49	39	69	49	46	53	61	59	66	114

A significant positive relationship was also found between the cumulative Top Hat® score and the Exam 3 scores (p = 4.8 e−5, Fig. 2). When correlations were evaluated by class percentiles, there was a relationship between Top Hat® score and overall class performance for the top 25th percentile (p = 2.5 e−4) (Supplemental Fig. 1). There was no correlation between intermittent and poor performers and engagement (Supplemental Figs. 2 and 3).

Fig. 2.

Exam 3 score (covering psychiatry content; maximum score 80) vs Top Hat® cumulative score (p = 4.8 e−5). Increasing Top Hat® cumulative score was significantly related to higher Exam 3 performance

Fig. 3.

Student comments about psychiatry from the end-of-course evaluation. N = 433 in 2020 and N = 39 in 2019. The percentage of positive, neutral, and negative comments were N = 271 (62.6%), N = 34 (7.8%), and N = 128 (29.6%) in 2020 and N = 7 (17.8%), N = 6 (15.5%), and N = 26 (66.7%) in 2019

Student and Facilitator Perspectives

Survey response rates were adequate and comparable, when applicable, to prior years. In 2020, 75.3% (N = 280) of students completed the End of Course Evaluation, 30.3% (N = 113) completed the Psychiatry-Specific Global Survey, and 66.7% (N = 14) of facilitators completed the Facilitator Curriculum Redesign Survey. In 2019, 69.8% (N = 257) of students provided feedback on the End of Course Evaluation. In 2020, a higher percentage (91.5%, N = 256) of MS1s were either Satisfied (40.4%, N = 113) or Strongly Satisfied (51.1%, N = 143) with the psychiatry content as compared to 2019 where 85.7% were either Satisfied (44.8%, N = 115) or Strongly Satisfied (40.9%, N = 105), (two-tailed t-test, p = 0.04; Supplemental Figs. 4 and 5). In 2020, most students (81.3%, N = 91) either Somewhat or Strongly Agreed that they preferred in-class sessions driven by questions rather than the traditional lecture format (Supplemental Fig. 6; Fig. 3 ).

The number and type of psychiatry-related comments on the Standardized End of Course Evaluation increased substantially in 2020 (433 in 2020 as compared to 39 in 2019). In 2019, only 17.8% (N = 7) of comments were positive, while in 2020 62.6% (N = 271) were positive. Of the negative comments in 2020, 28/128 were related to inconsistent student group membership (data not shown).

The Facilitator Curriculum Redesign Survey revealed that 61.5% (N = 8) of facilitators felt Positive/Very Positive towards the in-class question session and 76.9% (N = 10) felt the question difficulty was Just Right (Supplemental Table 4). The facilitators were split on perceived student preparation with 30.8% (N = 4) Somewhat Agreeing, 30.8% (N = 4) Somewhat Disagreeing, and the remaining Neither Agreeing nor Disagreeing. Regarding student engagement, over 53.9% (N = 7) of facilitators Somewhat Agreed while 23% (N = 3) Neither Agreed nor Disagreed, and 23% (N = 3) Somewhat Disagreed.

Discussion

Despite the challenges presented by a significant change in approach and the COVID-19 pandemic, the IUSM MS1 psychiatry content revitalization was successful. The FC changes were positively perceived by students, with the majority preferring in-class questions to traditional lectures. While the explained variance was low, there was a significant positive correlation between synchronous student engagement and both overall performance and performance specifically on Exam 3, covering psychiatry portion of the course, and overall performance. This pattern suggests that engagement and performance on the active learning portion, the latter dependent upon asynchronous material review, was clearly related to overall class performance, but as expected, not the sole predictor. Slightly more than one-third of students did not respond in at least one question session, and almost one-fifth did not respond in 2 or more sessions. Because the vast majority of these students passed the course, active learning was clearly not the only successful strategy.

The confluence of the changes necessitated by the COVID-19 pandemic and our curriculum revitalization changes makes simple comparisons impossible but our dataset unique. We cannot clearly determine any year-to-year change in knowledge or performance as attributable to the design versus mandated course changes. In the literature, results are mixed regarding overall efficacy in knowledge gained or performance outcomes among medical students compared to traditional lectures [21]. Studies have revealed that there may be a negligible or positive effect with the FC approach as compared to traditional lectures [14, 22]. No studies in this review showed a negative impact of FC methodologies. However, most of these studies evaluated were in undergraduates. Notably, the literature supports that most FC effects are in lower-performing students and the FC approach reduces class failure rates by these students [4]. To become a medical student, undergraduates must demonstrate a pattern of high achievement and test-taking abilities, which is associated with consistent study habits and motivation. Thus, the benefit attributable to the FC transition in medical students is likely smaller than in undergraduate students. In conclusion, given the medical student sample and amid the unclear impact of the pandemic on study habits (particularly during lockdown), it is difficult to conclude that our FC curriculum revision made a clear impact on overall knowledge and performance. More research with improved rigor will be needed to ultimately reach a conclusion regarding the efficacy of FC approaches to medical student education, specifically in the behavioral sciences.

The curriculum changes were positively perceived by students. Between 2019 and 2020, student satisfaction for the psychiatry portion of the course increased by more than 6%. Impressively, there was a threefold increase in positive comments regarding the psychiatric content. This finding is consistent with the literature. A systematic review recently reported that FC approaches are often viewed positively by medical students, reduce boredom, and bring about a greater task value when compared to traditional lectures [21]. Consistent with the literature, the curriculum changes were positively perceived by students—there was a threefold increase in the positive comments about the psychiatric content.

Interestingly, there was a discrepancy between facilitator and student attitudes about the active learning approach. Over 85% of students reported feeling positive/very positive about the experience, compared to only 61% of facilitators. The discrepancy may be multifactorial. First, the sample sizes between the two groups were vastly different. Second, the FC approach was a new experience for many facilitators; some only participated in one session and many only one instructional day. In contrast, by the psychiatry portion of the N&B course, the students had at least 5–10 × more experience with the FC approach, and day-by-day student feedback and text-based comments subjectively indicated increasing comfort and appreciation over time. We suspect the abrupt transition to virtual instruction, with facilitators unfamiliar with the methodology, exacerbated perceived challenges with student engagement.

The daily feedback approach also proved informative, if not strictly analyzable. We used response rate and accuracy-based thresholds (~ 80% of registered respondents per question and majority correct/incorrect answers) to guide pacing. The responses were mixed, with many daily complaints of “too fast” or “too slow” regarding question progression. Allowing students to ask questions verbally versus via the chat was also a source of concern/comment. The team attempted both approaches but ultimately allowed any questions in any manner. Unfortunately, the planned student-to-facilitator ratio was dramatically increased due to COVID. Many students reported difficulty engaging in the case discussions due to the large number of unfamiliar peers in each breakout room, which may have been further hindered by the virtual meeting rooms versus in-person instruction. These concerns were a recurrent topic of discussion for the N&B course management committee throughout the psychiatry portion of the course (and was discussed as part of the daily feedback response). However, software limitations surrounding large group breakout room assignment and practical considerations precluded a change to group size. Based on our experience, we suggest keeping the student-to-instructor ratio small and, if possible, keeping group membership consistent to facilitate student engagement and maximize satisfaction.

The limitations of our work must be considered. The course was delivered in the midst of a global pandemic which, as above, confounded many planned analyses. Not only did the delivery method change, but examination content and procedures were altered. For example, IUSM students usually take a customized but standard proctored final examination; however, virtual proctoring was financially prohibitive and the customized examination unavailable. Consequently, our analyses became limited as there was no baseline. However, we assume the pandemic will eventually end and the impact of the curriculum redesign on the standardized course exams may be accomplished. Further, eventually this and future cohorts of first-year medical students will take standardized licensing examinations, allowing for comparison across teaching modalities. Finally, the IUSM campus is somewhat unique in that there are 9 campuses spanning the state. While N&B instruction was centralized during the 2020 academic as a result of the pandemic, this structure may limit the generalizability of our results. However, future studies could also include cost analysis of running the curriculum from one centralized campus versus running the course on each specific campus.

A secondary goal of the MS1 psychiatry content redesign was to utilize FC advantages, specifically the reported increase in engagement, knowledge, and enjoyment of material delivered in this fashion, to spur interest in psychiatry and mental health. Of additional import to medical student education is the escalating psychiatry and mental health provider shortage and concurrent increasing prevalence of mental health conditions in society. Suicide is now a top 10 cause of death in the USA. Reducing the stigma associated with mental illness and/or inspiring medical students to become psychiatrists through engaging education is one way to address the crisis. There have been limited studies on the impact of the flipped classroom on choosing psychiatry as a specialty, with one study revealing the learning methodology was without impact [23]. Post-pandemic, with increasing reports of anxiety, depression, and substance-related challenges, increasing the capacity to treat mental health is paramount. Eventually, we hope to examine the impact of our curriculum change on students’ perception of psychiatry.

Conclusion

In summary, our results demonstrate a positive relationship between student engagement and class performance. Additionally, the FC approach was well perceived by both students and faculty, with most students preferring in-class questions to lectures. Although limited studies have found no correlation between USMLE Step 1 scores and FC approaches [23], our future research will evaluate performance on these exams and during third-year clerkships before and after the psychiatry curriculum change.

Supplementary Information

Below is the link to the electronic supplementary material.

Author Contribution

L. Grant: obtained the IRB approval; developed the facilitator and student surveys; gathered the student comment data; performed the statistical analysis and interpretation of the survey data; drafted, edited, and reviewed the manuscript; and consented for approval of publication. M. J. Opperman: edited the IRB submission, performed the statistical analysis and interpretation of student engagement and performance data; drafted, edited, and reviewed the manuscript; consented for approval of publication. B. Schiller: reviewed the IRB submission; evaluated the student comment data; drafted, edited, and reviewed manuscript; and consented for approval of publication. J. Chastain: reviewed the IRB submission; evaluated the student comment data; drafted, edited, and reviewed manuscript; and consented for approval of publication. J. Richardson: reviewed the IRB submission; anonymized all student data prior to statistical analysis performance; drafted, edited, and reviewed manuscript; and consented for approval of publication. C. Eckel: reviewed the IRB submission; drafted, edited, and reviewed manuscript; and consented for approval of publication. M. Plawecki: edited the IRB submission; assisted in developing facilitator and student surveys; evaluated discrepancies of student comment data; performed statistical analysis and interpretation of survey data and student engagement and performance data; drafted, edited, and reviewed manuscript; and consented for approval of publication.

Data Availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Declarations

Ethics Approval

Survey creation, analysis, and course-related data review was approved by the Indiana University Institutional Review Board.

Conflict of Interest

The authors declare that there is no conflict of interest.