Exploring a novel approach to surgery clerkship didactics during the COVID-19 pandemic: A qualitative study

Abstract

Background

The COVID-19 pandemic brings unforeseen challenges in medical education. The current study aims to: 1) describe third-year medical students’ experiences with the novel Shelf-Exam-Type Question Didactics (SET QD) before (in-person) and during (virtual) COVID-19.

Methods

In this qualitative study using grounded theory, we conducted purposive sampling and used 23 in-depth semi-structured interviews. Audio recordings were transcribed verbatim and the Atlas.Ti software was used to manage the thematic analysis.

Results

There are three themes and eight subthemes that emerged: 1) The SET QD Framework (sub-themes: questions as learning opportunities; interleaving; notable clinical scenarios; team learning; accountability). 2) Experienced Educator (subtheme: transformational teaching). 3) Virtual Accessibility (sub-themes: alleviating time constraints, and mitigating life-stressors).

Conclusions

Medical students regarded SET QD as impactful for shelf exam preparation, clinical preparation, and long-term retention of the material. This novel virtual didactic method may be used in non-surgical clerkships as well.

Introduction

The COVID-19 pandemic has brought unforeseen challenges in medical education, especially for medical students entering their clinical years who rely on learning from hands-on clinical experience with patients.¹ On March 17, 2020, the Association of American Colleges (AAMC) issued strong guidance advising institutions that medical students should not be involved in direct patient contact activities due to the risk of medical students contracting the COVID-19 disease and the limited supply of Personal Protective Equipment (PPE) to prevent infection.² As of August 14, 2020, the AAMC continues to advise that proper and sufficient personal protective equipment (PPE) be available for students seeking to complete graduation requirements, as many medical schools struggle to adjust their curriculum in response to current health, economic, social, racial and political unrest, and now educational crisis.^{3 , 4} As a result of the restrictions and uncertainty of in-person clinical education, there has been an increasing need for medical schools to develop effective and timely virtual education curricula for medical students, particularly during their clinical years.^{1 , 5 , 6}

In the clerkships years, patient care has historically been supplemented, to a limited and variable extent, with traditional lecture didactics and other forms of team learning to enhance medical students’ clinical education and preparation for their end-of-clerkship exam or National Board of Medical Examiners (NBME) Subject Exams or shelf exams. To our knowledge, few active learning virtual pedagogical strategies are being used in the clerkships, especially in surgery where students have even less of an active role now due to a decrease in the number of elective surgeries.7, 8, 9, 10 The lack of readily available and effective virtual didactics could potentially pose a threat to student clerkship learning outcomes.

At our institution, an educational program was developed for the 3rd year surgery clerkship termed the Shelf-Exam-Type Question Didactics (SET QD), using the processes that drive long-term learning and active engagement. This weekly session was seamlessly transitioned into a virtual format during the COVID-19 pandemic, to offer an effective and timely solution to remote education. To assess the impact of the SET QD, we sought to learn from medical students' perspectives how SET QD, as an innovative pedagogical framework during the surgery clerkship, contributed to their education and learning. Ultimately, studying effective virtual pedagogical strategies may benefit medical students’ clinical preparation for the course learning outcomes.

Methods

Study design and participants

This was an interview-based qualitative study utilizing a grounded theory analytical approach. Grounded theory is a research methodology used to explore social processes to produce a descriptive theory.¹¹ We used this approach to explore the perceived impact that SET QD sessions had on the students' educational experience. By interviewing students, as opposed to gathering survey data or test scores, we were able to take a deeper exploration of the students’ subjective experience, and allow them to evaluate the didactic sessions and share suggestions for improvement. A sample of 3rd-year medical students was recruited using a non-probability, purposive sampling method to select participants who may have insight into the SET QD framework through their completion of the surgical clerkship during the 2019–2020 academic year. Medical students were recruited from two medical schools in Southern California (David Geffen School of Medicine at UCLA and Charles R. Drew University of Medicine and Science) and were eligible to participate if they attended at least one of the weekly SET QD sessions. We explored how students perceived the SET QD before (in-person) and during (virtual) the COVID-19 pandemic.

Recruitment

We recruited the study participants by sending an e-mail to medical students in the class of 2021 (see Fig. 1 for conceptual framework). The email included the study flyer and some general information about the study's aim. Students interested in participating would contact the first author (J.A.N.M.), be screened, and subsequently enrolled if eligible. Following the enrollment, all participants received an email with a unique Zoom (San Jose, California, United States) link for the interview confirming the day and time. Students that completed an interview were offered a $20 gift card to Starbucks. All recruitment and interviews occurred in September 2020. We stopped interviews when thematic saturation was reached (i.e., when interviews yielded no new information).¹²

The SET QD pedagogical framework

The weekly SET QD sessions were structured following the principles of Soderstrom and Bjork's “Learning vs Performance” and the flipped classroom model to facilitate long-term learning through active engagement, as opposed to the less reliable method of assigning grades during the learning process, which is defined as performance.13, 14, 15 The 12-week surgery clerkship is offered 4 times per year and SET QD sessions were held 8 times during each 12-week surgery clerkship. Sessions were led by an experienced clinician and surgical educator (senior author) for about 2 h. The sessions were optional for students going through the surgery clerkship and typically consisted of 10–20 students attending regularly. Each week an email was sent to the students announcing the surgical topic that would be discussed.

Before the COVID-19 pandemic, sessions were held in person. During the COVID-19 pandemic, the sessions were effortlessly transitioned to a virtual format, using Zoom. Both in-person and virtual formats used a PowerPoint (Microsoft Corporation, United States) presentation combined with shelf-exam-type questions created by the instructor to guide a focused discussion covering the weekly assigned topic. These PowerPoint slides (without the correct answers to the multiple-choice questions) were made available to students the week prior. Students were encouraged to read about the topic and review the slides throughout the week in preparation for the SET QD session, mirroring the evidence-based work of de Virgilio et al. and others.16, 17, 18 The SET QD framework served the purpose to guide students through a targeted didactic session that would build on their foundational knowledge of the material gained from reading and studying the weekly PowerPoint slides and help them apply and analyze content during the sessions as a way to improve long-term retention well beyond their clerkship training.¹⁹

Data collection

The first author (J.A.N.M.) conducted all the interviews, after having undergone training by one of the co-senior authors of this study (S.B.), who has extensive experience in the conduct of qualitative studies. Emphasis was placed on how to help keep the discussion on track, generate lively and productive discussion, and obtain balanced input from participants. All interviews were conducted via Zoom and were recorded. Each interview was approximately 30-min in duration. The questions for the individual in-depth interview were designed to explore medical students’ insight and experiences with the in-person and virtual SET QD, depending on when the student had their surgery clerkship. The interviewer (J.A.N.M.) asked probing questions when necessary to encourage additional information. Before beginning the interviews, the first author (J.A.N.M.) pilot-tested the interview questions with the co-author (S.B.) and made modifications according to this experience (see Table 1 for sample list of interview questions). Notably, the interviewer (J.A.N.M.) was a medical student from the participating medical schools who completed the surgery clerkship in 2019–2020 and attended multiple SET QD sessions during the COVID-19 pandemic. These noteworthy characteristics highlight the experience of the interviewer to engage medical students in the discussion and help to balance the power dynamic between interviewer and interviewee since both were medical students in the same year. Also, it was imperative to obtain a wide range of experiences while being mindful to include students from groups that are historically underrepresented in medicine. At the end of each interview, participants filled out a demographic survey. The study was approved by the Charles R. Drew University of Medicine and Science Institutional Review Board (IRBNet# 1630369–1).

Data analysis

The first author (J.A.N.M.) transcribed the recorded conversations verbatim after each interview using the application “Ava: Best Live Captions” (San Francisco, California, Unites States) and added any relevant field notes. Team meetings occurred about 1–2 times per week. The data analysis occurred concurrently with data collection. Based on each interview, the interviewer (JANM) would add new probes to the interview guide and consult with the team regarding changes. To conduct thematic analysis and augment data saturation, transcripts were read independently by three members of the research team (J.A.N.M., S.B., and S.N.),^{12 , 20} who looked for common themes and sub-themes within responses for each question. To achieve this, the data were segmented into significant analytical components, marked the segment with descriptive words (coding), and generated a list of the codes (code list) to reapply to new segments of data.^{20 , 21} Once the initial coding was completed for all segments of the transcribed text, the contents were compared and discussed among the reviewers to achieve consensus in regards to the best themes for organizing the data.²¹ Narrow themes were condensed into broader, general ones based on their similarities and differences. Then, the relationship among general themes and their value in identifying important thematic domains through interview discussions were considered. This iterative process of data analysis yielded three different major themes and eight subthemes.¹¹ Of note, any coding, and thematic issues were discussed and any disagreement arbitrated by the senior qualitative researcher (S.B.) and resolved.^{22 , 23} Our team included a diverse group of individuals in an effort to capture all perspectives during the analytic process. The researchers on the study team had training in qualitative research, psychology, philosophy, learning and memory, medical student affairs, medical and surgical education, and quantitative research. Author (C.d.V.) was excluded from any part of the participant recruitment, data gathering, and analysis to avoid any conflict of interests as (C.d.V.) was the instructor who developed the SET QD framework. We used ATLAS.ti (version 9.0.4)—a qualitative data analysis software—to extract, organize and manage, and code the data.

Results

The recruitment email included 196 students. Twenty-nine students responded and one was deemed not eligible since this student did not complete the surgery clerkship during the 2019–2020 academic year. Data saturation was reached after interviewing twenty-three eligible participants and no further data was collected from the remaining five eligible respondents. The age of the participants ranged from 24 to 30 years (mean = 26.5 years; SD = 1.6) (see Table 2 for demographics). Slightly over half of the participants self-identified as male (n = 12; 52%). Participants self-identified as Asian American (n = 8; 35%), Caucasian (n = 6; 26%), African American (n = 5; 22%), Latinx (n = 2; 9%), and African American/Latinx (n = 2; 9%). About half of the participating students reported being first-generation college students (n = 12; 52%) while all 23 identified as first-generation medical students. Approximately half of the participants attended the SET QD sessions before COVID-19 (n = 12; 52%) and the other half during COVID-19 (n = 11; 48%). The SET QD participants had completed between 0 and 7 clerkships (mean = 4 clerkships; SD = 3) before the surgical clerkship.

We identified three major themes and eight subthemes from the student's experiences with the SET QD pedagogical framework: 1) The SET QD framework (sub-themes: questions as learning opportunities; interleaving; notable clinical scenarios; team learning; accountability). 2) Experienced educator (subtheme: transformational teaching). 3) Virtual accessibility (sub-themes: alleviating time constraints, and mitigating life-stressors). Table 3 illustrates the themes and subthemes with corresponding representative quotations from participants labeled arbitrarily as participants 1 to 23.

Theme 1: The SET QD Framework

The SET QD framework was well-received by students. Most enjoyed the learning atmosphere created by the SET QD format of asking targeted shelf-like questions in a small group setting and spoke about their success on the shelf exam and long-term retention of the material for application to other clerkships as a result of the SET QD sessions.

Subtheme 1: Questions as learning opportunities

Many students attributed their success on the shelf exam and clinical experience to the sample questions that were reviewed by the instructor during the SET QD session. The practice questions were created based on the educator's experience as a clinician and educator working with students. One student stated:

“I like question-based more. When you referenced Problem Based Learning (PBL), PBL I think goes too slow … I would rather just do more questions. I really liked [the instructor's] approach” (Participant 10).

Furthermore, beyond simply using questions to do well on the shelf exam, one student highlighted the essence of the SET QD structure by saying:

“And [SET QD] really promotes not thinking like UWorld, but being able to proficiently answer UWorld questions. One is for purely taking the shelf [UWorld], and the other one is for your learning [SET QD]” (Participant 6).

Subtheme 2: Interleaving

Students described the instructor's approach as unique and effective in being able to use their experience to discuss related topics and integrate information. Others enjoyed the way the educator discussed why the incorrect answers were incorrect and why the correct answers were correct as a method to integrate learning and demonstrate the interconnectedness of different topics in medicine. For example, one student recalled:

“One thing [the instructor] does very well is not being formulaic with how [they] present stuff. So, even though it was a surgery shelf, which has a lot of medicine in it, he did a good job of kind of asking us pimping questions about other things that would be going on like other medical questions or maybe an OB[Obstetrics] question here or there” (Participant 6).

Subtheme 3: Notable clinical scenarios

Students felt that the use of anecdotes and life experiences from the instructor provided them with tangible and engaging information. Students explained that the use of examples from real life allowed them to remember the material better and made the sessions more stimulating. For instance, one student reflected on their experience as a trained medical educator and said:

“We just learned that medical students as an adult learner, things that you really want to incorporate, are tying the content to actual experiences, whether it's your own or the students, and allowing the student to develop a thirst for learning and be self-motivated. And I think both of those are hit with these sessions” (Participant 14).

Another student shared a memorable example that summarizes this principle well when they said:

“[The instructor] obviously has a wealth of experience … which I think I learn a little bit better with stories and anecdotes. [For example, the instructor shared the following:] ‘When I saw this gunshot victim at [the hospital] 15 years ago, this is what we had to do for this patient.’ He is able to sort of put information together in a way that I think a med student would not really be able to see, but someone who has been taking care of these types of patients for so long would be able to see” (Participant 6).

Subtheme 4: Team learning

Students were grateful for the opportunity to learn from their peers and were appreciative of the richness that diversity brought to the learning experience. Students felt encouraged to express their thought process while answering questions and this enabled other students to learn from their peers. One student reflected on their experience:

“[The SET QD sessions] really felt like it was more of a team learning environment, but without trying too hard” (Participant 1).

Another student expressed this concept to be especially meaningful to them when they said,

“[The instructor] does a really good job of having us explain our thought processes before he actually gives his own two cents, and I feel that kind of discussion along with having him overseeing the entire conversation is super helpful, I learned a lot a lot from my peers and from him during those sessions” (Participant 5).

Subtheme 5: Accountability

Participants relished the opportunity to attend a weekly learning session, which gave them the accountability to keep up with the surgical material during the long rotation. Having the weekly sessions, students felt at ease that they were not falling behind and learning the important concepts they needed to do well on the shelf exam and in the clinical setting. One student recounted the following:

“The weekly [SET QD] is designed to kind of go through all the learning points that you are supposed to learn for the rotation. So, I used [SET QD] as a starting point and then [went] home and learned more about it, versus at the end [end-of-clerkship review] is more like okay, I this point I should know all of it. If most of it does not sound familiar, then something is wrong; it's a different function” (Participant 8).

Another student felt the sessions provided reassurance, accountability, and structure when they said:

“I was really worried about the shelf and I wasn't studying a lot during the week because the hours were so long. So, I needed some structure to my studying. So, going there in person showing up, provided some accountability as well” (Participant 18).

Theme 2: Experienced Educator

As third-year medical students, participants generally felt two main responsibilities during their clerkship that led to cognitive dissonance—the need to study and do well on their standardized shelf exams (performance) and the need to learn and practice clinical medicine and apply it appropriately to patient care (learning). Participants regarded the SET QD sessions as valuable for facilitating an effective integration into expected learning and performance. Many regarded the educator's experience, enthusiasm, care, and passion for teaching as an essential component for creating a conducive learning environment. Students generally felt the sessions prepared them for the expected learning outcomes and also exposed them to a learner-centered training strategy, that could benefit them as future academic clinicians.

Subtheme 1: Transformational teaching

The participants described that the weekly SET QD sessions facilitated building a caring and supportive relationship between the students and the instructor. Therefore, students felt comfortable asking questions, and actively engaging with the material. Through this positive relationship, students felt empowered during learning and applying the material. Students indicated that the instructor's teaching approach was “transformational”. One student reflected on their experience by saying:

“[The instructor] became a really nice person in general and very approachable … and I felt, at least if I thought I wanted to go into surgery, I could approach him and ask him to be a mentor. I think that he gives off this persona that he is definitely there for students” (Participant 14).

Similarly, participants commented on the educator's ability to anticipate where students historically struggle the most with the material, such as biliary topics, and personalize attention and feedback to enhance mastery of key concepts while enabling improved attitudes and beliefs. According to one student:

“What I really like about the sessions is that I think it's frankly unique to [the instructor]. He could tell when people didn't get it. I think he has been teaching for so long that he had such a sense of the room. When people were confused on a subject, he would speed up or go slower … So, he really tailored his teaching to the unique needs of whoever was there that day and that really stood out to me because I am not the type of person who is going to be like, ‘yes, can you please slow down?‘, but even if I just looked confused, he would pick up on it and go slower, and I was just amazed that he was able to do that” (Participant 18).

Theme 3: Virtual Accessibility

The COVID-19 pandemic created an unprecedented challenge for medical student education in the clinical setting. Even after one year since the start of the pandemic, many medical schools are still trying to adjust to the mutating COVID-19 virus and identify ways to supplement in-person clinical experiences for medical students and mitigate the loss of learning. As our study highlights, students generally preferred having didactic sessions be virtual and voluntary during the pandemic.

Subtheme 1: Alleviating time constraints

Students who were attending in-person SET QD sessions before the COVID-19 pandemic spoke about the difficulty of trying to get time off the rotation to attend the weekly sessions. Once the SET QD sessions transitioned to an online format, students had more time to attend the sessions. One student mentioned the following:

“When [SET QD] was in-person, it was a little more difficult to attend because it depended on which day I got off” (Participant 21).

Another student expanded on this concept when they said:

• “I actually liked how when we transitioned to Zoom curriculum, his lectures almost became integrated into the actual curriculum” (Participant 11).

Subtheme 2: Mitigating life-stressors

Students enjoyed the optional and virtual format of the SET QD sessions. The SET QD sessions gave students the control and flexibility to manage their learning according to their unique life circumstances. One student mentioned the following:

“I think if I was in rotations [in-person], I don't think I would have gone to the sessions as much. I would just be too gassed and maybe it won't be so helpful or maybe I won't pay attention, but now that I had extra time [with the virtual format], let me go and see what it’s about and it can't hurt” (Participant 9).

Another student emphasized the following:

• “One thing that really stuck out was how interactive they were. I think maybe it was due to the fact that both attendance and participation were voluntary” (Participant 17).

Discussion

The SET QD framework

In this study, we found that the weekly SET QD sessions were viewed as an optimal method for conducive learning in the clinical setting during the COVID-19 pandemic. Our results are consistent with evidence-based research on the science of learning from Robert Bjork that highlights the effectiveness of the weekly SET QD structure.^{13 , 15} First, the spacing effect or distributing out learning opportunities, in this case through weekly active learning virtual sessions, may lead to more long-term retention of the material. This is opposite to massing learning opportunities as is common for end-of-clerkship exam reviews.¹⁵ Second, unique to the SET QD framework is the use of targeted shelf-specific questions for learning. Using questions as learning opportunities is at the core of the SET QD structure. Students felt this method helped them learn the material and be more successful in the clerkship than regular didactic sessions. It has been demonstrated that the use of questions creates a testing effect, or retrieval practice that renders information to be more easily retrievable in the future.^{15 , 24} Third and synergistic with the testing effect, is the generation effect, where students were asked to generate an answer and explanation for their responses, which activates the brain processes that lead to long-term learning.¹⁵ Fourth, the students spoke about the uniqueness of interleaving and the benefits of this technique to help them prepare for the exams and learn the material. Interleaving, or contextual interference, was used when the instructor was bringing in different, but related topics as they pertained to the learning objective of the question. This was done through the use of multiple answer choices and a subsequent explanation of why incorrect answers were incorrect, which stimulated discussion of different but related topics.¹⁵ Fifth, students found the use of notable clinical scenarios from the instructor's experience interesting, engaging, and improving information retention.²⁵ Sixth, students received the team learning environment of SET QD to be less constraining than previous team-oriented learning formats. Seventh, students were satisfied and grateful for the interactive and focused nature of the virtual didactic session. It helped them understand the core learning objectives and instill a sense of accountability with the material.

Transformational Teaching

The SET QD framework through a knowledgeable educator in transformational teaching (i.e., creating deep and versatile relationships with students), leads to enhanced learning and personal growth.²⁶ It is important to note that about half of the participants identified as first-generation college students, and all identified as first-generation medical students. Therefore, the SET QD framework may be especially helpful for students who lack academic mentors, professional support systems, and feel intimidated by medical school. Multiple students mentioned the instructor's ability to connect with them on an individual level, provide career advice, and empower them to succeed in both their personal and academic lives, well beyond the surgery clerkship.

Virtual Accessibility

The transition of the weekly SET QD sessions into a virtual format provided an innovative and equitable solution to mitigating the effects of limited in-person clinical experiences due to the pandemic. The virtual format enhances the students’ attendance at the session, which was more difficult to do when the sessions were in-person. The students felt that they were able to learn from their peers while the observational learning component was still present.²⁷ Also, students attending the SET QD sessions found that voluntary attendance gave them control and flexibility over their education, especially during the COVID-19 pandemic.

Limitations

There are several limitations to note in this study. First, our study did not compare the shelf-exam score of students who did and did not attend the SET QD sessions. Therefore, it is difficult to quantify the impact the sessions have had on students' performance. However, the qualitative approach of our study highlights the importance of understanding student perceptions regarding their exam preparation and clinical learning. It has been reported that focusing solely on grades and scores (performance) to estimate learning misleads both learner and the evaluator.28, 29, 30 Test scores often highlight the difference in learners’ socioeconomic status (SES) and access to opportunities.28, 29, 30, 31, 32, 33, 34 Second, our results may be susceptible to social desirability bias.³⁵ To address social-desirability bias, the interviewer was a medical student in the same academic cohort as the interviewees, who were trained to remain task-oriented and obtained balanced input.³⁵ Third, the recruitment of students during the 2019–2020 academic year could have introduced selection bias, given their opinions may be influenced by the unexpected transition to virtual education, and therefore not representative of all medical students. However, our findings remind us of the importance to structure teaching and curricula to embrace trauma-informed pedagogy strategies.³⁶ It is imperative to be cognizant of the disproportionate effect the pandemic has had on vulnerable student populations, especially those who are underrepresented in medicine, first-generation, and those with lower socioeconomic status. There is increasing evidence that many students at all levels of educational achievement have experienced extenuating circumstances, which has negatively impacted their learning.36, 37, 38 Furthermore, given our current lack of evidence for test scores as a proxy for success and learning, our findings set a precedent to reassess and look beyond the central role of test scores in assessing successful completion of clerkship learning outcomes.36, 37, 38, 39

Conclusion

We found that the weekly SET QD sessions in the virtual setting offered a timely solution to the clinical learning crisis introduced by the COVID-19 pandemic for medical students. Our findings help narrow the knowledge gap by introducing a learner-center training strategy that suggests a positive application for other clerkships during the third year of medical school. There is currently a critical need for virtual education strategies that are evidence-based, tested, and well-received by students.

Funding

This research was supported by NIH National Center for Advancing Translational Science (NCATS) UCLA CTSI Grant Number UL1TR001881. Charles Drew University of Medicine and Science; Kermanshah University of Medical Sciences.

Appendix A. Tables and Figures

Fig. 1.

Recruitment flow chart.

Table 1.

Sample List of Interview Questions

Interview Questions
(1) Can you walk me through what influenced your decision to attend the SET QD sessions?
(2) If you were inviting a friend to the SET QD sessions, what would you say to convince them to attend?
(3) Based on your previous learning experiences while in medical school, how would you describe how the weekly SET QD sessions have contributed to your attainment of knowledge during the surgical clerkship?
(4) In what ways have the weekly SET QD sessions prepared you for the surgery shelf exam?
(5) What are your thoughts on the implementation of similar weekly SET QD sessions in other non-surgical clerkships?
(6) How would you describe your relationship with the instructor after attending these SET QD sessions?
(7) Thinking back about these SET QD sessions and your experience during medical school, have we missed anything you think would be important for us to know?

Table 2.

Sample demographics.

Demographics	Number (% out of 23)
Age in Yearsa
24–25	6 (26)
26–27	11 (48)
28–29	5 (22)
30	1 (4)
Genderb
Male	12 (52)
Female	11 (48)
Race/ethnicityc
Asian American	8 (35)
Caucasian	6 (26)
African American	5 (22)
Latinx	2 (9)
African American and Latinx	2 (9)
No. Clerkships Prior to Surgery
0–3	9 (39)
4–7	14 (61)
FG College
Yes	12 (52)
No	11 (48)
FG Medical School
Yes	23 (100)
No	0 (0)
ESL
Yes	12 (52)
No	11 (48)
Immigrant
Yes	6 (26)
No	17 (74)
Clerkship Impacted by COVID-19
Yes	11 (48)
No	12 (52)
Abbreviations	FG=First-Generation: the first in their family to attend.
	ESL = English as a Second Language

^aThe mean age of the 23 participants was 26.5 years (standard deviation = 1.6).

^bParticipants were given the choice to specify gender, so as to not propagate the rigid gender binary of “Male, Female” and embrace diversity and freedom of expression.

^cPercentage may not equal 100 due to rounding.

Table 3.

Subthemes and representative quotations.

Theme 1: The SET QD framework
Subthemes:
• Questions as learning opportunities
o “I like question-based more. When you referenced PBL, PBL I think goes too slow … I would rather just do more questions. I really liked [the instructor's] approach” (Participant 10).
o And [SET QD] really promotes not thinking like UWorld, but being able to proficiently answer UWorld questions. One is for purely taking the shelf [UWorld], and the other one is for your learning [SET QD]” (Participant 6).
• Interleaving
o “One thing [the instructor] does very well is not being formulaic with how [they] present stuff. So, even though it was a surgery shelf, which has a lot of medicine in it, he did a good job of kind of asking us pimping questions about other things that would be going on like other medical questions or maybe an OB [Obstetrics] question here or there” (Participant 6).
o “I like his format where like if I remember correctly, there is a bunch of questions with like ABCDE answer choices. I think that is really effective and I think it is very geared towards doing well on the shelf” (Participant 13).
• Notable clinical scenarios
o “We just learned that medical students as an adult learner, things that you really want to incorporate, are tying the content to actual experiences, whether it's your own or the students, and allowing the student to develop a thirst for learning and be self-motivated. And I think both of those are hit with these sessions” (Participant 14).
o “[The instructor] obviously has a wealth of experience … which I think I learn a little bit better with stories and anecdotes. [For example, the instructor shared the following:] ‘When I saw this gunshot victim at [the hospital] 15 years ago, this is what we had to do for this patient.’ He is able to sort of put information together in a way that I think a med student would not really be able to see, but someone who has been taking care of these types of patients for so long would be able to see” (Participant 6).
• Team learning
o “[The SET QD sessions] really felt like it was more of a team learning environment, but without trying too hard” (Participant 1).
o “[The instructor] does a really good job of having us explain our thought processes before he actually gives his own two cents, and I feel that kind of discussion along with having him overseeing the entire conversation is super helpful, I learned a lot a lot from my peers and from him during those sessions” (Participant 5).
• Accountability
o “The weekly [SET QD] is designed to kind of go through all the learning points that you are supposed to learn for the rotation. So, I used [SET QD] as a starting point and then [went] home and learned more about it, versus at the end [end-of-clerkship review] is more like okay, I this point I should know all of it. If most of it does not sound familiar, then something is wrong; it's a different function” (Participant 8).
o “I was really worried about the shelf and I wasn't studying a lot during the week because the hours were so long. So, I needed some structure to my studying. So, going there in person showing up, provided some accountability as well” (Participant 18).
Theme 2: Experienced educator
Subthemes:
• Transformational teaching
o “What I really like about the sessions is that I think it's frankly unique to [the instructor]. He could tell when people didn't get it. I think he has been teaching for so long that he had such a sense of the room. When people were confused on a subject, he would speed up or go slower … So, he really tailored his teaching to the unique needs of whoever was there that day and that really stood out to me because I am not the type of person who is going to be like, ‘yes, can you please slow down?‘, but even if I just looked confused, he would pick up on it and go slower, and I was just amazed that he was able to do that” (Participant 18).
o “[The instructor] became a really nice person in general and very approachable … and I felt, at least if I thought I wanted to go into surgery, I could approach him and ask him to be a mentor. I think that he gives off this persona that he is definitely there for students” (Participant 14).
Theme 3: Virtual accessibility
Subthemes:
• Alleviating time constraints
o “When [SET QD] was in-person, it was a little more difficult to attend because it depended on which day I got off” (Participant 21).
o “I actually liked how when we transitioned to Zoom curriculum, his lectures almost became integrated into the actual curriculum” (Participant 11).
• Mitigating life-stressors
o “One thing that really stuck out was how interactive they were. I think maybe it was due to the fact that both attendance and participation were voluntary” (Participant 17).
o “I think if I was in rotations [in-person], I don't think I would have gone to the sessions as much. I would just be too gassed and maybe it won't be so helpful or maybe I won't pay attention, but now that I had extra time [with the virtual format], let me go and see what it’s about and it can't hurt” (Participant 9).