Feasibility and acceptability of virtual mock oral examinations for senior vascular surgery trainees and implications for the certifying exam

Abstract

Objectives

The COVID-19 pandemic resulted in the cancellation of in-person testing across the country. We sought to understand the feasibility of conducting virtual oral examinations as well as solicit opinions of vascular surgery program directors (PD) regarding the use of virtual platforms to conduct both low stakes mock oral examinations with their trainees and potentially “real” high stakes certifying examinations (CE) moving forward.

Methods

Forty-four senior vascular surgery trainees from 17 institutions took part in a virtual mock oral examination conducted by 38 practicing vascular surgeons via Zoom. Each examination lasted 30 minutes with four clinical scenarios. An anonymous survey pertaining to the conduct of the examination and opinions on feasibility of using virtual examinations for the vascular surgery CE was sent to all examiners and examinees. A similar survey was sent to all vascular surgery program directors.

Results

The overall pass rate was 82% (36/44 participants) with no correlation with training paradigm. 32/44 (73%) of trainees, 29/38 (76%) of examiners and 49/103 (48%) of PDs completed the surveys. Examinees and examiners thought the experience was beneficial and PDs also thought the experience would be beneficial for their trainees. While the majority of trainees and examiners believed they were able to communicate and express (or evaluate) knowledge and confidence as easily virtually as in person, PDs were less likely to agree confidence could be assessed virtually. The majority of respondents thought the CE of the Vascular Surgery Board of the American Board of Surgery could be offered virtually, although no groups thought virtual exams were superior to in person exams. While cost benefit was perceived in virtual examinations, the security of the examination was a concern.

Conclusions

Performing virtual mock oral examinations for vascular surgery trainees is feasible. Both vascular surgery trainees as well as PDs feel that virtual CEs should be considered by the Vascular Surgery Board.

INTRODUCTION

Distance learning has been increasingly utilized during the COVID-19 pandemic.¹ The established paradigm of in-person meetings and conferences focused on surgical education have shifted to virtual platforms in order to increase social distancing among participants in hopes of decreasing the spread of the virus. Along with scientific meetings, other educational forums, such as Webinars, teleconferences, and remote procedural simulations have become more popular.²

The advantages of Technology-Enhanced Learning (TEL) have been established. It eliminates costs associated with lodging and transportation, can be done in any place with internet connectivity at dates and times convenient for all participants, and can provide learners with recordings to review for enhanced memorization and learning.² However, with increasing utilization of these technologies, several disadvantages of TEL have been made apparent. Disruptions in clinical work, duty hours violations for trainees, eye and body fatigue associated with screened device use, lack of hands-on skills training, and audio video streaming issues are some of the most common disadvantages.^{2 , 3} Despite this, TEL has continued to evolve and become a valuable tool in our current education environment. Furthermore, the application of TEL to the assessment of learning has become increasingly important.

Prior to the current pandemic, board examinations have been structured to be taken on an in-person basis.³ These high-stake examinations require assessment of a candidate's suitability for certification by discussion of typical cases the participants should be expected to be able to handle. Preparation for this certifying examination (CE) may occur through Mock Orals Exams (MOE), which commonly take place at national and regional society meetings or at the trainees’ home institutions. MOEs have been deemed to be an important tool in preparing vascular surgery trainees for their oral CEs.⁴ These exercises allow trainees to get used to the pace of the test, increase comfort level with the test, and decrease anxiety levels about the testing format.⁵ Lack of perceived need, examination content protection, confidentiality issues, lack of technological platforms, and organizational ease might have been potential reasons for the absence of virtual options rather than in-person board examinations and MOEs despite rapid technological advancements over the past two decades.

The impact of MOEs delivered in a remote, computer-based platform have not yet become popular among program directors and teaching institutions as of the writing of this manuscript. Further, the usefulness of these board certification virtual practicing tools from a trainee and faculty perspective has not been investigated. Restrictions placed on in-person gatherings and examinations due to the COVID-19 pandemic presented the opportunity to trial MOEs virtually with vascular surgery trainees from across the country. As a result of these virtual MOEs, we sought to gather the perceptions and feedback from both faculty and trainees and to gauge the feasibility of using virtual examinations to prepare vascular surgery trainees to take the CE moving forward.

MATERIALS AND METHODS

Virtual MOEs were administered in June of 2020 via Zoom. Anonymous electronic surveys were sent out shortly after the virtual mock oral exams to trainees and examiners who took and administered the mock orals to collect their opinions on various aspects of the virtual MOE. Program directors of all United States vascular surgery fellowship and/or integrated vascular residency programs were also sent a survey to assess their interest and opinions on conducting virtual CEs. All surveys were created and sent using REDCap. All responses were anonymous and analyzed by the author using Microsoft Excel. Statistical significance between the responses of examinees, examiners, and/or PDs was calculated using Chi-Square analysis due to answers to survey questions being compared classified as categorical variables (Table 1, Table 2, Table 3 , Figs. 2,5). The Institutional Review Board of St. Louis University deemed this study exempt from review.

Table 1.

Demographics of respondents.

	Examinees (n=32)	Examiners (n=29)	Program Directors (n=49)	P-value
Gender
Male	22 (68%)	24 (83%)	38 (78%)	0.422
Female	10 (32%)	5 (17%)	9 (18%)	0.306
Prefer not to answer	0 (0%)	0 (0%)	2 (4%)	0.281
Vascular Training Program Description
Academic/University Affiliated	28 (88%)	27 (93%)	36 (73%)	0.059
Hybrid (Work with University and Private Practice/Hospital Based Surgeons)	4 (12%)	2 (7%)	9 (18%)	0.353
Community-Based Program	0 (0%)	0 (0%)	3 (6%)	0.147
Military	0 (0%)	0 (0%)	1 (2%)	0.534
Other	0 (0%)	0 (0%)	0 (0%)	N/A
Training Paradigm(s) Offered at Institution
Integrated Vascular Surgery Residency	21 (66%)	15 (52%)	32 (65%)	0.451
Vascular Surgery Fellowship	28 (88%)	28 (97%)	40 (82%)	0.161

Table 2.

Strongly agree/agree opinions on virtual MOEs.

	Examinees (n=32)	Examiners (n=29)	PD (n=49)	P-value
Virtual Mock Oral Examination Was (would be) Beneficial for Trainees	31 (97%)	28 (97%)	48 (98%)	0.922
Examinee Traits Expressed Virtually as well as in Real Life
Knowledge Base	30 (94%)	29 (100%)	44 (90%)	0.204
Confidence	26 (82%)	25 (86%)	29 (59%)	0.015
Examinees were/would be able to communicate thoughts/answers virtually just as well as they would have been able to in person	21 (66%)	29 (100%)	38 (78%)	0.003
Audio Quality was (would be) Adequate	29 (91%)	28 (97%)	28 (57%)	<0.001
Video Quality was (would be) Adequate	31 (97%)	29 (100%)	38 (78%)	0.002
Virtual Oral Exams are Superior to In Person Exams	4 (13%)	2 (7%)	0 (0%)	0.049
Certifying Examination of VSB Could be Done Virtually	22 (69%)	23 (79%)	27 (55%)	0.085
Virtual Certifying Examinations Would Not be as Secure as In Person Examinations	13 (41%)	12 (41%)	21 (43%)	0.979
Examinees Would be More Able to Cheat on Virtual Certifying Examinations as In Person Examinations	9 (28%)	11 (38%)	13 (27%)	0.548
The Cost of a Virtual Certifying Examination Would Be Cheaper than In Person Examination	31 (97%)	26 (90%)	37 (76%)	0.021

Table 3.

Residents vs. fellow opinions.

	Integrated Vascular Surgery Residents (n=19)	Vascular Surgery Fellows (n=13)	P-value
I was able to express my fund of knowledge just as well virtually as I would have in person (Agree & Strongly Agree)	18 (95%)	12 (92%)	0.780
I was able to demonstrate confidence as well virtually as I would have in person (Agree & Strongly Agree)	14 (74%)	12 (92%)	0.185
Performing oral examinations virtually was no different than performing oral examinations in person (Agree & Strongly Agree)	13 (68%)	6 (46%)	0.208
The certifying examination of the vascular surgery board could be done virtually rather than in person (Agree & Strongly Agree)	12 (63%)	10 (77%)	0.409
Virtual oral examinations are superior to live in person oral examinations (Agree & Strongly Agree)	4 (21%)	0 (0%)	0.077

Fig. 2.

Radar plot of ranked attributes.

Fig. 5.

Which trainees should participate in virtual mock oral examinations?.

VIRTUAL MOCK ORALS

Integrated vascular residents (PGY 4-5) and vascular fellows (junior and senior) (n=44) at 17 U.S. institutions were assigned to an individual “zoom room” to be examined by two board-certified vascular surgeon volunteers. Examiners were given an exam book with 30 vascular scenarios from which to give four to the examinee similar to the CE. Scenario stems were projected in the zoom room via standardized PowerPoint given to all examiners which also contained the images for the scenarios (if requested by the examinee). Examiners graded the examinees’ responses using a standardized scoring sheet (Fig. 1 ) with the scoring based on whether they thought the examinee would pass the CE if these scenarios were selected. This scoring sheet was developed based off the essential attributes of a certifiable surgeon identified by the American Board of Surgery.⁶

Fig. 1.

Standardized scoring sheet.

RESULTS

32/44 (73%) of trainees who participated in the virtual mock exams, 29/38 (76%) of examiners and 49/103 (48%) of PDs completed the surveys. The majority of examinees (N=22, 68%), examiners (N=29, 83%), and PDs (N=38, 78%) were male. In addition, most of the examinees (N=28, 88%), examiners (N=27, 93%), and PDs (N=36, 73%) were training or practicing vascular surgery at an academic or University affiliated program. Table 1 shows demographic information and further information about the programs of respondents. Trainees and examiners surveyed were all involved in the virtual mock oral exams, while PDs surveyed were not involved and their answers were based on their own experiences and preconceptions with virtual exams.

There was no statistically significant difference between examinees, examiners, and PDs surveyed about how well the knowledge base of those being interviewed could be assessed virtually compared to in person. However, examiners tended to agree more than examinees and PDs that trainees could communicate that knowledge virtually as well as they could/would have been able to in person (Table 2 , P=0.003). In regard to the confidence and nervousness that could be assessed over a virtual mock interview compared to an in-person interview, PDs did not believe confidence could be assessed well virtually while trainees who participated in the virtual mock orals and examiners believed confidence could be expressed just as well virtually as in person (Table 2, P=0.015). PDs also overwhelmingly expressed concerns that audio (P<0.001) and video (P=0.002) quality would be inadequate while examinees and examiners did not see this as a limitation (Table 2). A small but significant (P=0.049) number of examinees and examiners agreed that virtual oral exams are superior to in person exams whereas no PDs agreed with the statement (Table 2). The only other significant difference between the three groups was in regard to the cost of virtual certification being cheaper than in person with PDs believing virtual exams would not be cheaper while examinees and examiners believed cost could be reduced (P=0.021). Table 3 depicts the breakdown of opinions related to perspectives and perceptions from examinees and examiners as well as PDs.

Both integrated vascular resident trainees and vascular fellow trainees responded similarly to questions regarding their performance in the virtual mock orals and opinions regarding virtual assessments in surgical education. Both groups largely believed they were able to adequately project their confidence and share their knowledge virtually. Notably, although both groups believed that the certifying examination of the vascular surgery board could be done virtually rather than in person, neither group believed that virtual examinations were superior to in-person examinations (Table 3).

Participants in all three groups ranked cost, convenience, security of examination, protection from cheating, fairness/equity of test grading, and fairness/equity of test questions of the CE of the Vascular Surgery Board oral examinations 1–6 (1 being the most important, 6 being the least important). The averaged ranked attributes among all three groups are plotted against each other in Fig. 2 . All three groups on average agreed fairness/equity of test grading and fairness/equity of test questions were the first and second most important attributes, respectively. Examinees tended to rank cost as a more important attribute than examiners and PDs (P = 0.037). Examinees also ranked security of exam lower than examiners and PDs (P = 0.046).

All three groups were surveyed on suggestions that would be beneficial and improve the experience for trainees (Fig. 3 ). No significant differences were noted among the three groups. Responses from trainees indicate they would prefer to have feedback at the end of session from examiners rather than from other trainees watching with live feedback. Trainees also saw more benefit in a larger zoom meeting with other trainees to discuss oral board strategies compared to examiners and PDs. In addition, the participants of the mock oral examinations were surveyed on any difficulties they encountered. Difficulties with the zoom program itself were relatively absent, with audio issues and accessing the zoom-rooms being the most common among both examinees and examiners. Keeping track of time was the most common difficulty noted among examiners while examinees had more difficulty with their examiner being late.

Fig. 3.

Which of the following would be beneficial and improve the virtual mock oral experience for trainees?.

20/29 (69%) of examiners believed the virtual MOEs were about the same difficulty compared to “real” oral examinations, with the remaining 9/29 (31%) believing the virtual mock orals were easier. No examiner perceived the virtual MOE as being more difficult than the “real” oral examination. Asked whether performing oral examinations virtually was no different than performing oral examinations in person, 19/32 (59%) of examinees responded they “Strongly Agreed” or “Agreed”, while 13/29 (45%) of examiners and 18/49 (37%) of PDs responded similarly; notably, no one from any of the three groups responded they “Strongly Disagreed” with the statement. 32/32 (100%) of examinees expressed interest in participating in a future virtual mock oral examination, while 29/29 (100%) of examiners and 45/49 (92%) of PDs responded they would serve as an examiner in future virtual mock oral exams. 29/29 (100%) of examiners and 47/49 (96%) of PDs responded they would have their trainees do virtual MOEs in the future.

DISCUSSION

Rapid changes initiating a transition of traditional medical licensing exams from an in person to a remote, virtual platform have been promulgated by the COVID pandemic. Electronic, remote teaching and learning systems have also become a reality in the training process of surgical trainees based on the impact of the pandemic and social distancing requirements. The use of virtual platforms for administering MOEs for healthcare trainees has been slowly and sporadically tried and studied for years, but the COVID-19 pandemic has accelerated these efforts in an unprecedented way. The rapid transition to using virtual platforms for MOEs has led to increased data and feedback which, coupled with rapid advancements in technology, has allowed for greater flexibility and more rapid integration of improvements, largely leading to positive reviews of virtual avenues for administering MOEs.

The positive-trending feedback of MOEs, especially those that are done virtually, in turn has led to an increased interest in the use of virtual platforms for administering MOEs.7., 8., 9. MOEs are just one example of traditionally in-person events being shifted to a virtual paradigm due to the pandemic. Other educational forums, such as meetings and conferences, have had to adapt to the changing landscape of medical education, as well.^{1 , 2 , 10 , 11} With rapid changes and unforeseen technical challenges, however, the adoption of virtual platforms for high-stakes medical exams has already been a source of great controversy. In July 2020, the virtual American Board of Surgery General Qualifying Exam had issues with their online exam process, and the examination was unable to be administered last minute due to issues with the proctoring software.¹² Since that time, however, there has been rapid embracing of virtual platforms for board examinations. The Vascular Surgery Board recently administered a very successful virtual certifying examination earlier this year with positive results and no significant known flaws.

With the long-lasting pandemic, the timeline of returning to “normal” in person meetings and events is uncertain, cementing virtual paradigms of medical education and conferences as a feasible alternative that are becoming increasingly more common.^{8 , 10 , 11} However, the virtual experience and perspective of vascular surgery virtual mock oral examinations have not yet been assessed on a larger scale. Our study assessed the perceptions and results of such an initiative in a pioneering manner in the US.

While examiners and examinees did not encounter any significant technical challenges during the virtual MOEs (Fig. 4 ), questions and concerns remain for the administration of CEs virtually, especially among PDs surveyed. As perceived by the examinees, the convenience of virtual exams outweighed the fact that trainees overwhelmingly disagreed that virtual exams are superior to in person examinations. Notably, trainees who participated in the MOE virtually viewed security and the possibility of cheating as less of a concern compared to examiners and PDs. From security and cheating concerns to questions on the logistics of conducting virtual examinations and any associated technical problems, PDs tended to believe that virtual examinations could not be adequately done to assess trainee knowledge and performance when compared to examinees and examiners who took part in the virtual MOEs. Although this is not entirely unsurprising as PDs are responsible for the education and preparation of their trainees for future board certification, the participation of more PDs in mock/real virtual examinations may change their perceptions as they become more familiar with the format and security/technical challenges are addressed and remedied.

Fig. 4.

Difficulties encountered during the mock oral examinations for examinees and examiners.

There are also questions as to when virtual MOEs are helpful throughout the vascular surgery training residency/fellowship. In regard to which trainee(s) should participate in virtual mock oral examinations, the three groups surveyed all agreed trainees from senior vascular integrated (forth year) residents through senior/graduating fellows should participate more than interns and junior vascular integrated (Second-Third year) residents. However, compared to examinees and examiners, PDs consistently responded that fewer fourth year senior vascular integrated residents (P = 0.041), 5^th year senior/graduating vascular integrated residents (P = 0.017), junior fellows (P = 0.003) and senior/graduating fellows (P = 0.017) should participate in virtual mock oral examinations (Fig. 5 ). Taken in the context of the new untested virtual administration of the mock oral exams, PDs seem to trust the effectiveness of virtual exam administration less than trainees and examiners, especially considering the many security and privacy concerns raised with the software used for the failed virtual American Board of Surgery Qualifying Exam.¹² Considering the length of the COVID-19 pandemic and rapid shift towards virtual paradigms, virtual examinations will likely continue to be developed and refined so attitudes of PDs towards the use of virtual CEs may change over time as they gain more exposure and a better understanding of the virtual methods available for administering oral certifying exams. Notably, almost all PDs surveyed believed virtual MOEs would be beneficial for trainees, indicating that there may be a place for virtual methods of assessing vascular surgery trainees that are more convenient than what is currently offered.

The administration of a MOE virtually for vascular surgery trainees was conducted successfully with few reported technical difficulties. Although questions and concerns remain about certain aspects of virtual oral examinations, it is clear from survey responses from examinees and examiners who participated in the virtual MOE that the feasibility of future certifying exams in vascular surgery can be done virtually. PDs seem to be less convinced of this possibility, indicating more work needs to be done to refine and develop virtual examinations to assuage fears and uncertainties surrounding virtual examinations. Virtual methods of assessing trainee knowledge and administering certifying exams are on track to potentially replace in person oral examinations entirely. It would therefore be important to continue assessing the perceptions and opinions of PDs as virtual formats for oral board examinations are developed further.

The Vascular Surgery Certifying Exam was just recently administered virtually for the first time and there are already plans to have the next oral CE in July 2021 administered virtually again. With the adoption of a virtual format for this CE, it seems that we are headed towards implementation of virtual certifying exams as a standard model. As indicated by our survey results, there is support among trainees and vascular surgeons alike for the transition to virtual modes of administering the vascular oral certifying exam.

Limitations

The results of this study should be interpreted in the context of several limitations. The survey questions on cheating and security of conducting virtual certifying examinations were vague and not well defined. This may have led to survey respondents not considering the same factors when answering questions in regard to “cheating” and “security”. Clarification, such as explaining possible methods of cheating and asking about specific aspects of security of software used to administer virtual examinations, was not provided. Because PDs are heavily invested in accurately assessing trainees’ abilities and knowledge in vascular surgery, they may have thought more about the possible ways of cheating and security flaws while trainees partaking in virtual mock oral exams were more concerned with demonstrating their knowledge and passing the exam. This discrepancy may have led to the stark differences in opinion on the use of virtual exams between PDs and examinees. PDs surveyed about the feasibility of virtual certifying examinations were asked about a new evaluation paradigm that they may not be familiar with, especially if they were not involved as examiners in the mock oral exams administered. Virtual methods for medical trainee examination and assessment have become increasingly more prevalent due to COVID-19 and initial reviews have shown them to be adequate alternatives to traditional in-person educational paradigms.8., 9., 10., 11. The absence of major technological issues with administration of both the mock oral exam and recent vascular surgery oral board exam virtually is evidence of the practical utilization of alternative methods to in-person assessment and educational training.⁶ As a result, some discrepancy between PDs and trainees in this study might represent different points of view instead of major issues with VMO per se.

CONCLUSIONS

Trainees and examiners who participated in the mock oral exams were more confident on the pros of the virtual format outweighing the cons compared to PDs, who expressed more hesitancy. However, performing virtual MOEs for vascular surgery trainees on a national level is both feasible and worthwhile, as perceived by surveyed vascular surgery trainees, volunteer examiners, and PDs. Both vascular surgery trainees as well as PDs feel that virtual CEs should be considered by the Vascular Surgery Board. However, it remains to be evaluated the impact of such virtual platforms on the performance of examinees on achieving success in the vascular surgery CE and whether virtual paradigms will persist as in person gatherings resume and the pandemic winds down.

Conflict of Interests

The authors have no competing interests to declare