Virtual Surgery Oral Board Examinations in the Era of COVID-19 Pandemic. How I Do It!

Abstract

INTRODUCTION

Traditional in-person Mock Oral Examinations (IP-MOEs) are utilized by surgery residency programs to prepare trainees for the American Board of Surgery Certifying Exam (ABS-CE). However, the COVID-19 Pandemic has led to a profound disruption of on-campus and in-person educational activities, with subsequent instantaneous revolutionization of educational systems all over the world, including a massive switch to virtual platforms. Many in-person didactics and examinations were canceled or rescheduled, including the ABS-CE. The study aims to evaluate Virtual MOEs’ (V-MOEs) feasibility as a potential alternative to in-person MOEs in residency programs.

METHODS

Twenty-five participants–16 general surgery residents (7 females, 9 males) and 9 faculty – in the inaugural Department of Surgery Virtual Mock Oral Examination completed an anonymous, voluntary online survey via Microsoft Forms. Faculty was given 24 questions, and residents 28, with 9 questions common between both residents and faculty. Participants were asked about the accessibility to virtual examination rooms, V-MOE effectiveness, resident's preparation for the exam, resident's stress, diversity, and number of clinical scenarios, and possible future implementation of, and barriers to, V-MOEs.

RESULTS

All participants have participated in IP-MOEs in the past. All faculties were very satisfied or satisfied with IP-MOE, compared to 93.8% of residents. All participants were very satisfied or satisfied with the orientation and instructions before V-MOE. Only 66.6% of faculty, compared to all residents, was satisfied with time allocation for sessions. While 88.9% of faculty felt the V-MOE was less stressful on residents, only 68.8% of residents felt so. Additionally, 87.5% of residents said they prepared for the V-MOE similarly to the IP-MOE. As a future platform, only 22.2% of faculty compared to 43.8% of residents preferred V-MOE over the IP-MOE. Both faculty (88.9%), and residents (81.3%) preferred immediate feedback at the end of sessions. All faculty recommend collaboration with other programs to enhance the resident's preparation. Time constraints, lack of experience with the format, and availability were the top 3 barriers.

CONCLUSION

V-MOE is feasible, accessible, and a potential alternative for IP-MOEs at a program level for ABS-CE preparation. Given the time constraints and costs associated with IP-MOEs, it is an opportunity to collaborate with other residency programs.

INTRODUCTION

Successful completion of the General Surgery Qualifying and Certifying Exams (QE and CE) is required for board certification in general surgery. While the QE is a multiple-choice computer-based exam, the CE is an in-person face-to-face oral exam consisting of 3 consecutive 30-minute sessions, each conducted by a team of 2 examiner.¹ Successful performance on the CE is also an important indicator of residency training program quality in accreditation standards defined by the Accreditation Council for Graduate Medical Education.²

On March 11, 2020, the World Health Organization declared that the novel Coronavirus (COVID-19) outbreak had reached a global pandemic level.³ The COVID-19 pandemic has led to an unprecedented disruption of academic teaching worldwide and has forced institutions and hospitals to devise and revolutionize new platforms of education, training, and practice. While academic and training programs were able to swiftly transition to virtual learning for lecturing and grand rounds, it is not clear whether in-person Mock Oral Examinations (IP-MOEs) for certifications are to be transitioned to virtual platforms. At the program level, there have been numerous studies that suggest the benefits of in-person Mock Oral Examinations (MOEs),4, 5, 6 however, to our knowledge, there has been no study addressing utilization of Virtual Mock Oral Examinations (V-MOEs) in general surgery residency programs. This study evaluated the effectiveness of V-MOEs as an alternative format of American Board of Surgery Certifying Exam (ABS-CE) preparation for surgical residents and serves as a starting point for the discussion of utilizing V-MOEs on a national level.

METHODS

Traditionally, MOEs are conducted 3 times per year in our program: 2 local exams for the entire program, and 1 for senior/chief residents in collaboration with regional surgery residency programs. The IP-MOEs are conducted similarly to ABS in that there are 3 exam rooms, 30 minutes allotted for each exam, 2 examiners per room, and 4 to 5 cases per examinee. In our local exam, residents are also provided ten minutes of immediate verbal feedback in each exam room. Residents are evaluated using a scoring sheet modeled after that used for the ABS-CE. Upon exam completion, the examiners meet as a group to discuss the performance of each resident and decide on the final pass/fail grade. Resident score reports, including additional comments from examiners, are then distributed to residents within 1 to 2 weeks of MOEs.

Due to the COVID-19 pandemic, our planned local and regional IP-MOEs were cancelled. While cognizant of the challenges, in order to continue administration of the MOE, we decided to utilize a virtual teaching platform to administer the exams. To evaluate this new approach, we conducted an anonymous, voluntary online survey of participants at the conclusion of the V-MOE. The survey contained 28 questions for residents and 24 questions for faculty. Nine questions were shared between the faculty and resident surveys. Detailed instructions were emailed separately to faculty and residents 1 week prior to the exam. One faculty examiner was assigned per Virtual Exam Room. On the day of V-MOE, a brief virtual orientation took place between the PD, proctor, and faculty using Zoom. All participants (faculty and residents) then joined a single Zoom session before being assigned to “breakout rooms” by the proctor. Breakout rooms enabled residents and faculty to leave the main Zoom session and meet in a separate virtual “room” for their one-on-one exam. To ensure exams began on time, all residents were instructed to sign in to the Zoom meeting at least 10 minutes before their scheduled exam time. In order to stay on schedule, breakout sessions were formatted to end automatically at the 20-minute mark. Each room was provided with a 2-minute warning before the session closed. Upon breakout room closure, examiners and examinees were all automatically returned to the main Zoom session and asked to remain there until their next assigned session. A 10-minute break was scheduled between each exam, during which the examiners were asked to complete an online evaluation form before being assigned to their next breakout room. Our evaluation form was modeled after the ABS instrument for scoring resident performance. Examiners completed 1 evaluation form for each resident (up to 4 scenarios can be evaluated on a single form). In addition to numerical scoring, the examiner provided comments on resident performance. Clinical scenarios were prepared in advance and selected based on the Surgical Council on Resident Education (SCORE) curriculum and the ABS booklet of information.^{7 , 8} Each examiner was given 5 clinical cases to choose from. At the conclusion of the V-MOE, data was extracted from the online evaluation forms, and a detailed report (pass/fail) with comments was generated for each resident. This is in contrast to the traditional reporting method which required faculty comments be manually transcribed and added to the report. Turnaround time for the traditional process was 1 to 2 weeks, and V-MOE reports were distributed to residents within 2 days. Therefore, in V-MOE, time efficiency was notable.

A link to an anonymous, voluntary survey was provided to all participants through the organization's secure email. The survey focused on the following: participation in previous IP-MOEs, number of previous exams, satisfaction with V-MOE access, perception of stress during V-MOE, and potential implementation of V-MOEs in the future. Additionally, the survey asked about participant and scenario characteristics (e.g., postgraduate level, local/community vs. academic faculty examiner, feedback strategies, future V-MOE collaborations, and perceived implementation barriers). Two optional survey questions allowed respondents to provide comments on barriers to exam participation. This study was reviewed and deemed exempt by our Institutional Review Board.

RESULTS

Twenty-five participants–16 general surgery residents (7 females, 9 males), and 9 faculty–participated in this survey. We had a 100% response rate. Table 1 demonstrates both faculty and resident responses to questions common to both. Table 2 demonstrates faculty-specific responses. Table 3 shows resident specific responses. Senior/chief residents were assigned to 3 virtual exam rooms, while juniors had 1 or 2 rooms. Clinical scenarios were preselected, with 3 scenarios presented in 62.5% of rooms. There was only 1 examiner per virtual room.

TABLE 1.

Questions Shared Between the Faculty (N = 9) and Residents (N = 16)

Variable	Faculty Responses n (%)	Resident Responses n (%)
Gender
Male	9 (100)	9 (56.3)
Female	0 (0)	7 (43.8)
Previous participation in IP-MOEs
Yes/No	Yes, 9 (100)	Yes, 16 (100)
Frequency	<10 (44.4)	<4: 9 (56.3)
	>10 (55.6)	>4: 7 (43.8)
Satisfaction with IP-MOE
Very satisfied/satisfied	9 (100)	15 (93.8)
Accessibility/orientation for V-MOE
Very satisfied/satisfied	9 (100)	16 (100)
Satisfaction with time allocation per sessions
Very satisfied/satisfied	6 (66.6)	16 (100)
Perception of stress on residents
Little less stressful/ Lot less stressful	8 (88.9)	11 (68.8)
Future preference for MOE
Prefer in-person	2 (22.2)	3 (18.8)
Prefer virtual	2 (22.2)	7 (43.8)
No preference	5 (55.6)	6 (37.5)
Preference to immediate feedback
Yes	8 (88.9)	13 (81.3)
No	NA	1 (6.3)

TABLE 2.

Faculty-specific Responses (N = 9 Faculty Responses)

Variable	Responses n (%)
Numbers of years working with residents
1-5 years	2 (22.2)
6-10 years	3 (33.3)
>10 years	4 (44.4)
Faculty designation
Academic Faculty	4 (44.4)
Community Faculty	5 (55.6)
Faculty's scope of surgical practice
General surgery with/without subspecialty	6 (66.6)
Subspecialty only	3 (33.3)
Effectiveness of V-MOE compared to IP-MOE
Less effective	4 (44.4)
More effective	1 (11.1)
Neutral	4 (44.4)
Satisfaction with diversity of scenarios
Very satisfied/satisfied	9 (100)
Is number of cases appropriate for allocated time?
Yes	6 (66.7)
No	3 (33.3)
Should number of cases in V-MOE should be same as MOE?
Strongly agree/ Agree	6 (66.7)
Disagree	1 (11.1)
Recommendation to conduct all oral boards as V-MOE
Highly/moderately recommended	4 (44.4)
Less recommended	2 (22.2)
Neutral	2 (22.2)
Not recommended	1 (11.1)
Efficiency (time utilization)
Highly efficient/moderately efficient	8 (88.9)
Less efficient	1 (11.1)
Collaborate with other programs
Highly/moderately recommend	9 (100)

TABLE 3.

Residents-specific Response (N = 16 Resident Responses)

Variable	Responses n (%)
Postgraduate level (PGY)
PGY1	4 (25)
PGY2	3 (18.8)
PGY3	4 (25)
PGY4	2 (12.5)
PGY5	3 (18.8)
Number of virtual exam rooms
1 room	4 (25)
2 rooms	4 (25)
3 rooms	8 (50)
Previous evaluation by any examiners in previous exams
Yes	9 (56.3)
No	7 (43.8)
Perception of passing V-MOE
Yes	10 (62.5)
Unsure	4 (25)
No	2 (12.5)
Degree of preparation for V-MOE compared to IP-MOEs
Similar to in-person	14 (87.5)
Better than in-person	2 (12.5)
Level of distraction during V-MOE
No distraction	14 (87.5)
Mild distraction	2 (12.5)
Satisfaction with the technology available (internet)
Very satisfied/satisfied	16 (100)
Satisfaction with the V-MOE format
Very satisfied/satisfied	16 (100)
Ability to complete your scenarios in Room 1 before your time was up
Yes	14 (87.5)
No	2 (12.5)
# scenarios completed in room 1
2 cases	4 (25)
3 cases	10 (62.5)
4 cases	1 (6.3)

The most frequently cited barrier for implementing or continuing V-MOEs was a lack of time and faculty availability to serve as examiners and lack of experience with the format. Table 4 demonstrates participants' comments and suggestions about the V-MOE.

TABLE 4.

Summary of Summary Comments

Faculty suggestions:

▪Combination of in-person and virtual mock orals. ▪Coordinating with other programs sounds great and is plausible virtually. ▪Early implementation of Mock Orals for PGY1. ▪Allow for feedback time after the 20 minutes with the residents to ask questions etc. ▪Dual examiners would add more stress to interviewee. ▪More scenarios, images, ability to have 2 examiners in room.

Pros of virtual

▪A benefit of the break out rooms is the timer on the screen and that they end automatically so that you can't go over on time and delay other sessions. I would also think this would be very convenient for the attending who can take a break from their own clinical duties in their own office with less time commitment for travel. ▪Feel virtual format is most efficient way to prepare for oral exam. ▪Residents in virtual exam were less dressed up and seemed much more relaxed than they do when it's in person with 2 examiners. ▪Virtual format allows wider participation with faculty and I feel preparation for the oral exam is improved due to enhanced efficiency, however the experience of face-to-face exam is quite different, and as such there will be some reluctance to replace it with the virtual modality.

Pros of in person MOE

▪The inherent nature of face-to-face interrogation allows improved insight into residents’ level of comprehension. ▪IP-MOEs have 2 examiners that make evaluation more contemporaneous.

DISCUSSION

The COVID-19 pandemic emerged as the greatest threat and challenge to educational and healthcare systems in recent history. However, the agile digital world plays a pivotal and critical role in mitigating the pandemic impact. A variety of virtual interactive communication platforms and apps have been used in an attempt to seal and heal the unprecedented educational chasm. Such platforms include; Microsoft Skype, Microsoft Teams, Zoom, and Cisco WebEx. These platforms allow real-time communications that enable viva voce and actual clinical examinations to be conducted online, with the examiner observing the clinical encounter remotely via telemedicine.^{9 , 10} While this innovation eliminates many of the inconveniences of traditional clinical evaluation ¹¹ it has not yet been established for use in high stake board exams.

Over the last 4 months (March-June 2020), virtual education has proven to be an effective alternative and has been increasingly adopted and implemented by almost all institutions for distance learning and remote work. Our study evaluates the potential benefits of V-MOE in a general surgery residency program. In this survey, taken at the completion of the first V-MOE in May 2020, we found that this platform has a high satisfaction rate by both faculty and residents. It is less stressful and less intimidating than the IP-MOE along with being highly efficient (Fig. 1 A)

FIGURE 1.

(A): Faculty (V-MOE). (B): Faculty and resident preference for future mock oral examinations (MOE). and resident perception of resident stress during virtual mock oral examinations.

While all participants were satisfied with both IP-MOE and V-MOE formats, the residents prefer the virtual format twice (43.8%) as much as faculty (22.2%) as a future platform, and more than half of the faculty (55.6%) have no preference for 1 or the other (Fig. 1B). Additionally, while almost all faculty feel the V-MOE is efficient, only half of them recommend all oral exams be conducted virtually. This originates from the faculty feeling that the experience may not be representative of the in-person ABS-CE. The format of the ABS-CE may change in the future, but research by the ABS is needed to better understand the possibility of utilizing V-MOEs on a national level. The V-MOE results observed in this study could be helpful in informing such efforts.

Adopting the V-MOE could potentially offer additional benefits, especially in regard to the expense of preparation and travel to the testing site. However, to implement V-MOEs, standards for validity, reliability, acceptability, and cost-effectiveness must be met.¹² in addition to Cybersecurity

This survey study has limitations. First, it is primarily descriptive, cross-sectional and was not designed to identify specific characteristics of V-MOEs; however, the overall results are favorable for the utilization of V-MOEs at small general surgery programs. While there was reasonably good gender diversity among residents (43.5% females), the faculty cohort was 100% males. Women surgeons constitute 20% of our faculty (9/45). Unfortunately, one of the challenges in organizing MOE exams is the availability of faculty. In the past several years, we have had participation from both male (70%-80%) and female (10%-30%) examiners. In the future, a multi-institutional trial will be needed to validate the generalizability of the V-MOE as an alternative standardized exam. Second, this study was conducted at a single academic training program and therefore has a small sample size. Third, while the passing rate in the V-MOE was higher than in IP-MOE (93.4% vs. 47%-69%), the study was not designed to assess the pass rate. This significant improvement could be partly explained by less stress or potential biases with only 1 examiner in each virtual room. Because only 1 examiner evaluated each examinee, inter-rater reliability was not assessed; therefore, inter-rater reliability remains an issue to be studied in future research.

This study showed that barriers to the implementation of V-MOEs included a time commitment for brief training of faculty to learn the system used to administer the examinations. Technology has evolved tremendously to offer robust, scalable, authentic, and auditable solutions to authorities and institutions that are willing to take the step to test new-age technologies and usher in the age of digital examination.¹³

CONCLUSION

V-MOEs are feasible, accessible, and a viable alternative for in-person MOEs at a program level for ABS-CE preparation. Given the time constraints and costs associated with IP-MOEs, it is also an opportunity to collaborate with residency programs.