Third year medical student knowledge gaps after a virtual surgical rotation

Abstract

Introduction

This study describes perceived knowledge gaps of third-year medical students after participating in a virtual surgical didactic rotation (EMLR) and shortened in-person surgery rotation during the COVID-19 Pandemic.

Methods

Open-ended and Likert questions were administered at the end of the virtual rotation and inperson-surgical rotation to medical students. Three blinded coders identified themes by semantic analysis.

Results

82 students (51% of all MS3s) participated in the EMLR. Semantic analysis revealed gaps in perioperative management (Post-EMLR:18.4%, Post-Inpatient:26.5%), anatomy (Post-EMLR:8.2%, PostInpatient:26.5%). and surgical skills (Post-EMLR: 43.0%, Post-Inpatient: 44.1%). Students also described gaps related to OR etiquette (Post-EMLR: 12.2%, Post-Inpatient: 8.8%) and team dynamics/the hidden curriculum (Post- Inpatient:26.6%). There was a significant improvement in perceived confidence to perform inpatient tasks after completing the inpatient clinical experience (p ≤ 0.01).

Conclusion

Virtual interactive didactics for cognitive skills development cannot replace a full clinical surgical experience for third-year medical students. Future curricula should address perceived gaps.

1. Introduction

Since the World Health Organization declared COVID-19 a pandemic, there have been great efforts at the institutional level to promote social distancing to limit the spread of COVID-19 while maximizing efforts to maintain quality education for medical trainees.¹ Following the AAMC's strong recommendation in March 2020 to temporarily pause all in-person medical student rotations, medical school responses varied from complete closure of schools to continued clinical rotations for fourth-year medical students.¹ The COVID-19 pandemic has substantially impacted medical education, with many medical schools transitioning from in-person to virtual learning.^{2 , 3}

Early in the COVID-19 pandemic, between March 9, 2020 and July 13, 2020, education leaders at the University of California, San Francisco (UCSF) replaced in-person clinical clerkships for third-year medical students with a virtual three-week Extended Mastery Learning Rotation (EMLR) for each of the core clerkships.⁴ Given the temporary suspension of in-person clerkships, the EMLRs were designed to continue the solidification of foundational and clinical science knowledge through virtual teaching. After the completion of all EMLRs, students returned to a one month in-person surgical clerkship.

As virtual learning becomes a greater contributor to surgical education, further assessment of the current landscape in online learning is warranted. This study aimed to evaluate perceived knowledge gaps of third-year medical students in the domains of baseline surgical knowledge, clinical skills, and professional behavior after participating in the EMLR and after participating in a shortened in-person surgery rotation. We aimed to determine the change in perceived confidence to perform various clinical tasks after the EMLR and after students’ one-month in-patient surgical clerkship. We hypothesize this study will serve as a curricular needs assessment for identifying the strengths and weaknesses of virtual surgical education.

2. Materials and methods

2.1. Participants

This study included third-year medical students who participated in a three-week virtual surgical didactic rotation followed by a one-month shortened in-person surgical rotation. The EMLR was taught by faculty, residents, and near-peer medical students and covered the American College of Surgeon/Association of Surgical Education Medical Student Core Curriculum.⁵ This course was composed of a series of virtual didactics and interactive sessions that reviewed common surgical patient presentations with associated management plans, fundamental anatomy and pathophysiology. The EMLR was implemented as a result of the COVID-19 pandemic to continue medical student learning outside of the in-patient clinical setting. Shortened in-patient surgery rotations were performed within one specialty at six different clinical sites. Students who completed their in-person surgery rotation prior to the COVID19 pandemic and those who participated in a longitudinal surgery rotation were not included in this study.

2.2. Data sources

Student-perceived knowledge gaps were identified using a modified version of the Readiness for.

Clerkship Student Survey.⁶ Survey modification was described in a prior study.¹ The Readiness for Clerkship Student Survey was previously validated for use in program evaluation.⁶ However, the modified survey that is used in this study has not been formally validated.

Students completed two different surveys: 1) the post-EMLR survey and 2) the post-in person surgical clerkship survey. Both open-ended and Likert questions were utilized in both surveys. Open-ended and Likert questions both evaluated gaps in clinical skills; however, open-ended questions were evaluating gaps in clinical skills after performing the EMLR, while Likert questions were evaluating students' perceived preparedness to perform specific clinical tasks at the time that they completed the survey. Open-ended questions were utilized to evaluate students’ perceived knowledge gaps after the EMLR to allow students to freely express their perceptions of their virtual experiences. Given that it was the first time the EMLR curriculum was utilized, we wanted to allow for a full range of responses that would otherwise be limited by Likert questions.

Open-ended questions asked students to articulate perceived gaps in their knowledge base, clinical skills, and professional behavior that they felt should be addressed before beginning their in-patient surgical clerkship. Students also were asked to describe what areas of functioning as a third-year medical student they felt the EMLR course prepared (or did not prepare) them for their in-person surgery clerkship. The survey asked students to determine their specialty of choice. Therefore, students who responded to these questions in the post-in person third year clerkship survey were asked to retrospectively evaluate what knowledge gaps they had before their in-person rotation. This allowed us to determine what students found to be important after they experienced and knew what was required of them during their in-person rotation and therefore allowed us to identify gaps within the EMLR curriculum.

Questions with a Likert response scale asked students to articulate how prepared they felt to perform various tasks, such as performing a full physical examination of a surgical patient. Therefore, students who responded to the post-EMLR survey were asked to evaluate how prepared they felt to perform various clinical tasks after performing the EMLR. Students who responded to the post-in person surgical clerkship survey were asked to evaluate how prepared they felt to perform various clinical tasks after performing their in-person surgery clerkship. These questions were evaluated on a five-point scale (1- an unacceptable level of competence and 5- an extremely high level of competence). Students also evaluated their current interest in surgery, whether COVID-19 influenced their decision to pursue a surgical specialty, and whether they believed the EMLR should be implemented in future surgical rotations. These questions were evaluated on the following five-point Likert scale: 1-strongly disagree to 5-strongly agree.

Students were surveyed after completing the EMLR rotation and after completing all their third year in-person clinical rotations. This was done to evaluate perceived knowledge gaps after the students had a better understanding of what was required of them on their in-patient surgery rotation, and to identify how the in-patient surgery rotation improved or changed medical student perceived competence to perform various clinical tasks. The times from surgery clerkship completion to post-inpatient survey administration varied from 1 to 5 months. Students who performed their surgery clerkship earlier in their third year were surveyed later than those who had their surgery rotation at the end of their third year.

2.3. Data analysis-open ended questions

Three coders (S.H., S.S., O.N.S.) separately identified concepts among open ended questions by semantic analysis in a blinded fashion. The coders were then unblinded, concepts were discussed, and themes were identified to better conceptualize student perspectives. The number of individuals who described a particular theme were summed, and percentages of the overall cohort were calculated.

2.4. Data analysis-likert questions

Average Likert scores and standard deviations were obtained using Microsoft Excel (OS 10.3) and compared by two sample t-test using Stata Version 16 (College Station, TX). P-values ≤ 0.05 were considered statistically significant. While a paired t-test would have been preferred, given the anonymity and voluntary nature of our survey, we were unable to pair the responses. Our Institutional Review Board deemed this study exempt.

3. Results

3.1. Participants

Overall, 82 students (51% of all MS3s) participated in the EMLR and the shortened in-person surgery clerkship. Forty-nine (60%) students completed the post- EMLR survey, and 34 (41%) students completed the post in-person surgery clerkship survey (Fig. 1 ). All students who completed the open-ended questions also completed the Likert questions.

Fig. 1.

Timeline of survey distribution.

3.2. Perceived knowledge gaps after the EMLR

Semantic analysis for baseline surgical knowledge after completing the EMLR revealed gaps in perioperative management and anatomy (Table 1 ). After their inpatient surgery rotation, more students identified these themes as gaps of the EMLR. Additional themes were identified, with students describing gaps in anatomy related to surgical operations and surgical equipment knowledge. Notably, students seemed amenable to filling these gaps with more virtual content. For example, one student stated “Bring back anatomy lab before the rotation. Could it be an interactive zoom course?“.

Table 1.

Perceived knowledge gaps of the EMLR.

	Post-EMLR Survey N = 49	Post-In-person Surgical Clerkship Survey N = 34
Baseline Surgical Knowledge	Perioperative management N = 9 (18.4%) Anatomy N = 4 (8.2%)	Perioperative management N = 9 (26.5%) Anatomy N = 9 (26.5%) Surgical Anatomy N = 4 (11.8%) Surgical equipment N = 4 (11.8%)
Clinical Skills	Surgical skills N = 21 (43.0%) Obtaining a history/performing a physical exam N = 14 (28.6%) Presenting findings to a surgical team N = 9 (18.4%)	Surgical skills N = 15 (44.1%) Obtaining a history/performing a physical exam N = 6 (17.6%) Presenting findings to a surgical team N = 5 (14.7%) Wound care N = 5 (14.7%) Drain management N = 4 (13.3%)
Professional Behavior	OR etiquette N = 6 (12.2%) Student role and expectations while on a surgical service N = 6 (12.2%)	Team dynamics/Hidden Curriculum N = 9 (26.5%) Student role and expectations while on a surgical service N = 9 (26.5%) OR etiquette N = 3 (8.8%)

N=Number of individuals who identified each theme within each survey (%) = percentage of individuals that identified a specific theme among those who took the survey at that time (after the EMLR or after their in-person surgery clerkship).

Perceived gaps in clinical skills were identified in surgical skills, obtaining a history and performing a physical examination, and presenting findings to a surgical team after the EMLR course. The post inpatient survey had fewer individuals describing gaps in obtaining a history and performing a physical examination, and presentations to a surgical team. Instead, students described gaps in knowledge about wound care and drain management (Table 1). Suturing skills remained the most prevalent theme of perceived deficiency. One student noted, “Wish I had a surgical skills lab before the rotation; COVID cancelled ours”, suggesting that their surgical skills sessions were negatively impacted by the COVID-19 pandemic. Another student noted, “Suture practice skills should be mandatory before entering the OR, with practicing at home. I was (very) unprepared when asked to do my first suturing.” Like anatomy, students seemed amenable to virtual surgical skills teaching with one student stating, “Videos and tutorials on how to suture should be provided so that (these skills) can be reinforced and practiced.”

When students were asked about gaps in knowledge regarding professional behavior after performing the EMLR, they identified gaps in OR etiquette and the student role and expectations while on a surgical service. There was an increase in the proportion of students who recognized a gap in EMLR teaching of student roles and expectations on the surgery service after students performed their inpatient rotation. Themes of the “hidden curriculum” and team dynamics were identified as gaps in the EMLR rotation after the students performed their inpatient experience. In this study, the hidden curriculum is defined as a set of unspoken professional expectations that are influenced by surgeons' social, cultural, and belief systems that may influence learning more than the formal curriculum.^{7 , 8} One student described:

“[I] did not know proper OR etiquette with respect to when it's okay to dab and when it's not, nor when it's okay to move tools or hand tools or how to hand them among different individuals.” Another noted,

“It would have been very helpful to know these differences [between medicine and surgery rounds] to prevent miscommunication – I thought that my resident didn't trust me enough to do a physical exam[s] during rounds and wasn't sure how to bring it up to [them].”

Additional themes were identified outside of baseline surgical knowledge, clinical skills, and professional behavior, such as the students' appreciation for the EMLR's role in developing fundamental surgical knowledge. One student stated, “The course gave me a good foundation for thinking through different surgical pathologies and techniques.” Students also described how the COVID-19 pandemic impacted their overall surgery experience. For example, some students felt that the timing of the EMLR in their actual clerkship impacted their learning experience, with one student stating “the EMLR was 6–7 months prior to my actual surgery clerkship, so it was challenging to remember and apply things I learned in the EMLR to the clerkship.”

3.3. Surgical career interest

Surgical career interest remained the same before and after the students performed their inpatient surgery rotation (p = 0.7, Table 2 ). After students performed their inpatient rotations, fewer believed that the COVID-19 pandemic impacted their decision to pursue a surgical career (p = 0.001, Table 2). This suggests that students anticipated the COVID19 pandemic would impact their surgical experience more than it ultimately did, possibly due to the prompt response from surgical education faculty to ensure continued medical student surgical experiences, the effectiveness of virtual didactic teachings, and adequate exposure to the inpatient surgery setting during their shortened inpatient clerkships.

Table 2.

Student interest in surgical specialties.

		Post-EMLR Survey	Post-In person Surgical Clerkship Survey
		N = 49	N = 34
Question		Average Likert (±Std)	Average Likert (±Std)	P-Value
Q1) How do you rate your current interest in surgery?		3.7 (1.1)	3.8 (1.3)	0.7
Q2) COVID19 has influenced my decision to go into a surgical specialty.		2.8 (0.9)	2.1 (1.0)	0.001*
Q3) I would like to see the EMLR implemented into all future surgical clerkships.		3.4 (1.1)	2.6 (1.4)	0.005*

* = statistically significant, Q1 Likert Scale: 1- strongly uninterested to 5-strongly interested, Likert scale: 1-strongly disagree to 5- strongly agree; Q2-3: Likert scale: 1-strongly disagree to 5- strongly agree.

3.4. Perceived confidence to perform tasks

There was a significant improvement in perceived confidence to perform more inpatient tasks

after students completed their inpatient clinical experience. For example, students felt more confident in taking a full history, performing a physical exam, interpreting imaging findings, presenting a patient to a surgical team, and understanding anatomy for surgical procedures (Table 3 ). However, there was no significant difference in perceived confidence to perform the other tasks described in Table 3. This suggests that the EMLR prepared students as well as the inpatient surgery rotation for these more foundational, cognitive tasks.

Table 3.

Student perceived preparedness for the in-patient surgery clerkship after performing a 3-week virtual didactic rotation and after performing a shortened in-patient surgery rotation.

	Post-EMLR Survey N = 49	Post-In person Surgical Clerkship Survey N = 34
Question	Average Likert (±Std)	Average Likert (±Std)	P-Value
Q1-Take an appropriate history of the current surgical problem.	2.5 (0.8)	2.7 (1.0)	0.32
Q2-Take a full surgical medical history.	2.1 (0.8)	2.6 (1.0)	0.01*
Q3-Formulate a problem list in a surgical patient.	2.6 (0.9)	2.8 (0.9)	0.32
Q4-Perform a full physical examination in a surgical patient.	1.9 (0.7)	2.8 (0.9)	<0.001*
Q5-Interpret relevant key lab results obtained on surgical patients.	2.8 (0.7)	3.1 (0.8)	0.07
Q6-Interpret relevant imaging reports for common health problems of surgical patients.	2.6 (0.7)	3.0 (0.9)	0.03*
Q7-Explain the underlying pathology and pathophysiology of key surgical problems.	2.8 (0.7)	2.6 (0.9)	0.26
Q8-Verbally present your findings to the resident or preceptor.	1.9 (0.9)	3.3 (0.8)	<0.001*
Q9-Demonstrate a clear understanding of anatomy in the context of physical exams and interventions.	2.4 (0.8)	3.0 (0.7)	<0.001*
Q10-Propose a differential diagnosis consisting of more than one reasonable alternative based on the history, laboratory, and other test, and the results of the physical examination in a surgical patient.	2.9 (0.7)	2.9 (0.8)	1.0
Q11-Identify medications appropriate to treat a surgical patient's problems.	2.3 (0.7)	2.5 (0.9)	0.26
Q12-Propose a basic short-term management plan for a surgical patient's major problems.	2.5 (0.8)	2.7 (0.8)	0.27
Q13 - Explain the short- intermediate- and long-term management plans that were developed for the surgical patients under your care.	2.2 (0.8)	2.8 (1.0)	0.003*
Q14 - Identify if a surgical patient is seriously ill and requires immediate assessment and treatment.	3.2 (0.8)	3.1 (0.9)	0.60
Q15- Pursue opportunities to learn the required surgical technical skills.	1.7 (0.8)	3.0 (1.2)	<0.001*

* = statistically significant, Likert scale: 1-strongly disagree to 5- strongly agree.

4. Discussion

This article is the logical extension of our prior study entitled “Where Do We Go From Here? Assessing Medical Students' Surgery Clerkship Preparedness During COVID-19”.¹ In this previous study, we surveyed student perceived preparedness to perform various clinical tasks before the start of any third year clerkship and again at the end of the EMLR. The current study reports additional qualitative survey responses as well as responses obtained after the completion of all third-year virtual and in-person rotations.

In this cohort study, third year medical students who participated in a three-week virtual surgical didactic rotation and a shortened one-month in-person surgical rotation were surveyed to identify student perceived knowledge gaps of the EMLR, and to identify how the inpatient surgery rotation impacted medical student self-reported competence to perform various clinical tasks. In-person requirements to pass the surgery rotation during the pandemic were the same as those before the pandemic. Students’ perceived knowledge gaps after completing the EMLR became more nuanced after they understood what was expected of them on their surgery rotation. For example, surgical anatomy, surgical equipment, wound care, drain management, and how to navigate team dynamics, or the “hidden curriculum”, were all identified as knowledge gaps only after completion of the inpatient surgical clerkship. This discrepancy in identified themes likely is occurring because students could describe these knowledge gaps once they recognized their existence in clinical practice. Students felt more prepared to perform in-patient surgical clinical tasks after performing their in-person surgery rotation. Therefore, this study helps to identify the impact of a virtual surgical didactic rotation for cognitive skills development and identifies how the.

4.1. COVID-19 pandemic impacted students’ surgical learning experience

While the EMLR improves fundamental surgical knowledge and students’ perceived confidence to perform in-person clinical tasks, gaps remain in EMLR didactics that students were amenable to having supplemented with virtual content. Previous studies have suggested that virtual supplementation of surgical teaching is effective. For example, in a study by Lindeman et al., surgical clerkship students who participated in virtual learning had similar NBME exam and clinical evaluation scores compared to students who had in-person didactics.⁹ Another study performed by Gormley et al. reported that medical students felt the use of virtual patients in the surgical clerkship to be a useful learning tool for improving clinical reasoning and preparing for in-person cases.¹⁰ Therefore, given the effectiveness of virtual surgical didactics, it is reasonable that gaps in surgical knowledge be filled with virtual content per student request. The usage of virtual modalities to supplement inpatient learning is a viable option to support surgical clerkship student education. These results delineate the positive effects that inpatient clinical experiences can have on surgical learning.

However, not all gaps can be adequately supplemented with virtual learning. Clinical skills gaps that are more difficult to teach in a virtual format were learned and improved upon during the in-patient rotation, while skills relating to the hidden curriculum were not taught well in a virtual nor in-person setting. Therefore, further attention must be given to dismantling the hidden curriculum within the surgery clerkship, possibly through in-person small groups or near-peer discussions.

While this study demonstrates that virtual interactive surgical didactics cannot supplant an inpatient clinical experience, we know from our previous study, Nnamani Silva & Hernandez et al., 2021, that the EMLR improved student perceived preparedness for the surgery clerkship compared to preclinical students.¹ The students’ desire to see the EMLR or virtual didactics implemented into all future surgery clerkships decreased after the students performed their inpatient rotation. This could be secondary to pre-clerkship student anxiety resulting in a falsely elevated desire for more surgical didactic teaching. Additionally, this could be falsely elevated due to the uncertainty of how the COVID-19 pandemic would impact their surgical experience. Lastly, this discrepancy could be a direct result of the length of the EMLR and the resulting shortening of the inpatient rotation. While many students found the EMLR helpful, it seemed too long and was completed too far before their actual surgery rotation to be optimally effective. Based on these results, our surgery department has transitioned to a one-week virtual interactive EMLR rotation that will be followed by a seven-week in-patient surgical clerkship with weekly virtual didactics. By doing this, students will be able to develop surgical clinical frameworks that can be further solidified and refined on their in-patient surgery rotation. This will allow for uniformity of didactic teaching among all medical students at various clinical rotation sites. Further studies must be performed to determine the effectiveness of a one-week EMLR on student perceived preparedness for the surgical clerkship.

This study has some limitations. First, potential outcome bias threatens the validity of the survey

results. Students received their grade for the surgical clerkship before responding to the post-inpatient survey, and their overall performance may have impacted their perceptions of both the EMLR and the surgical clerkship as a whole. Furthermore, the time at which the students undertook their surgical clerkship also may have impacted their perceived competence as students who completed their surgical rotation later in the year had more time to develop clinical frameworks. For this reason, Block 5 students, for example, may have perceived more competence at the end of their rotation than Block 1 students due to their exposure to other specialties and practice with clinical skills independent of the student's experience with the EMLR. Additional limitations pertain to the match of post-EMLR and post-inpatient surgical clerkship survey responses. The anonymous and voluntary nature of the surveys precluded matching of post-EMLR and post-inpatient surgical clerkship responses and thus prohibited use of paired t-testing to evaluate pairwise differences in mean Likert scores. Additionally, this study did not evaluate the student's knowledge base or clinical skills to identify gaps in the curriculum. Therefore, student perceived gaps could be biased by students' preference for in-person vs. virtual learning.

5. Conclusion

As in previous studies, our research suggests that virtual teaching is effective for surgical didactics. However, it cannot replace an in-person surgery rotation. The results of this study are important given the increased utilization of virtual didactics and surgical rotations by many medical schools throughout the world during the COVID-19 pandemic.^{10 , 11} This work can be used to improve virtual curricula. Student-identified gaps in this didactic rotation can be further supplemented with virtual learning to better prepare them for their in-person rotation. Further studies should be performed to confirm perceived knowledge gaps through quantitative evaluation of surgical knowledge and skills. Additionally, decisions to continue virtual surgical didactics after in-person teaching is safe should be further informed by data comparing hybrid virtual and in-person teaching compared to that of in-person teaching alone. Determining the optimal balance of virtual clinical didactics and in-person surgical clerkship training merits further investigation.

Author contributions

All authors made a significant contribution to this work. S.H., O.N.S., and S.A.R. participated in the project design. S.H., S.S., O.N.S. participated in data analysis and drafting the manuscript. S.H., S.S., O.N.S., C.A., A.S.K., A.R.C., E.H.K, A.A, E.C.W., J.A.S., J.G., M.Y.C.L., and S.A.R. participated in course development, teaching, data acquisition, data interpretation, and manuscript revision. S.H., S.S., O.N.S., C.A., and S.A.R. wrote the manuscript.

Abbreviations

Third-year medical student (MS3), extended medical learning rotation (EMLR)

Funding

Sophia Hernandez was supported by the National Center for Advancing Translational Sciences, National Institutes of Health, through UCSF-CTSI Grant Number TL1 TR001871. The manuscript's contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH.

Declaration of competing interest

Julie Ann Sosa, MD, MA, is a member of the Data Monitoring Committee of Medullary Thyroid Cancer Consortium Registry supported by GlaxoSmithKline, Novo Nordisk, Astra Zeneca, and Eli Lilly. She receives institutional research funding from Exelixis and Eli Lilly. Ogonna N. Nnamani Silva, MD, is a medical consultant for Calyan Technologies Inc. and will receive equity compensation from the company for her work.