The Impact of a Virtual Radiology Medical Student Rotation: Maintaining Engagement During COVID-19 Mitigation

ABSTRACT

Introduction

The purpose of this study is to introduce a virtual radiology rotation curriculum that is being used to safely maintain medical student and intern education and engagement with the Department of Radiology at Walter Reed National Military Medical Center during coronavirus disease 2019 (COVID-19) mitigation.

Materials and Methods

The curriculum is designed as a 4-week block with each week representing one of the four highest yield radiology subspecialties for medical students; neuroradiology, thoracic radiology, body radiology, and musculoskeletal radiology. A subspecialist radiologist from each section was identified as a primary mentor and tasked with designing a daily schedule for medical students and interns. The first 2 months of virtual rotators were surveyed to assess the effectiveness of the course.

Results

Thirty-five of 41 rotators responded to the survey, a response rate of 85%. Thirty-one of 35 (89%) of the rotators gave this virtual elective rotation a positive rating, with 16 trainees scoring the course as 4 out of 5 and 15 trainees selecting 5 out of 5. Four respondents selected 3 out of 5. Five out of 5 respondents who had experienced an in-person radiology elective rotation before this virtual rotation rated the virtual elective as more educational than the in-person rotation. We found the 4-week subspecialist mentor-based structure to be highly versatile, allowing us to simultaneously accommodate multiple groups of full or partial block rotators throughout COVID-19 mitigation.

Conclusion

A virtual rotation curriculum is a viable method of maintaining medical student and intern education and engagement with the department of radiology during COVID-19 mitigation.

INTRODUCTION

“All in-person medical student rotations are cancelled, effective today.” This is the first sentence of the email from the Uniformed Services University that confirmed our fears at Walter Reed National Military Medical Center about the potential impact of the coronavirus disease 2019 (COVID-19) pandemic on medical student education at our institution. The total effect on medical student and graduate medical education (GME) will only be fully evaluated in retrospect, but it will be significant. The impact is being felt not only in lost clinical hours because of quarantine, but also in decreased patient exam volume. In our radiology department, we experienced up to a 60% decrease in overall examinations compared with 1 year prior. We have curtailed nonurgent outpatient examinations and limited the number of in-house staff and in-house trainees, which has the potential for significant negative impact on GME if not mitigated. To answer this challenge, our institution designed and implemented an all-virtual radiology rotation targeted at the level of third-year medical students (MS3s) and fourth-year medical students (MS4s). Our primary considerations in mitigating these negative impacts with a virtual rotation were the safety of our rotators and the need to maintain adequate training of our medical students. A secondary consideration was our desire to maintain recruiting efforts for our residency and to use the virtual curriculum to advertise the benefits of radiology as a specialty. This is an important endeavor given the decline in the number of applicants to diagnostic radiology from 2009 through 2015, with a more recent rebound in from 2016 to 2018.¹

The concept of virtual or computer-assisted education is not new. Articles were published as early as the 1960s about computer integration into education, which would later come to be known as e-learning.² The use of e-learning is now common in GME. A recent survey of 368 internal medicine program directors revealed that e-learning was routinely utilized in 71.5% of programs.³ Many articles have been published about the benefits of blended learning, defined as the combination of traditional face-to-face learning and asynchronous or synchronous e-learning. For example, a recent meta-analysis found blended learning to be more effective than, or at least as effective as nonblended traditional instruction.⁴ These findings add to an earlier analysis of 201 studies comparing internet-based instruction with traditional instruction showing the two methods to be similarly effective.⁵ A separate meta-analysis from the same author dealing with efficiency of e-learning found that on average, internet-based instruction and noncomputer instruction require a similar amount of learning time.⁶

Learner-centered asynchronous instruction has been recognized formally by the Accreditation Council for Graduate Medical Education (ACGME). The ACGME refers to these materials as individualized interactive instruction. Emergency medicine has been on the forefront of this advancement and has been approved by the ACGME to utilize individualized interactive instruction, such as web-based learning, for up to 20% of the planned educational experiences or didactics provided three criteria are met: the program director must monitor resident participation, there must be an evaluation component, and there must be faculty oversight.⁷ We decided to follow these three criteria from emergency medicine when designing our virtual elective for medical students, as there is no such reference to individualized interactive instruction in the ACGME Diagnostic Radiology requirements.⁸

Radiology, in particular, is well suited as a specialty for blended learning. Our images are easily accessed via the internet and generally do not require dedicated interaction with a patient. In fact, an example of successfully utilizing a virtual read-out for radiology residents as part of COVID-19 mitigation has been published recently.⁹ This ability to utilize radiology’s electronic flexibility also allows continuation of medical education using online formats, even in the absence of new patients. Radiologists as a group are distinctly well versed in mitigating the technical challenges of image interpretation and interaction over the internet. In a recently published survey of 936 radiologists, 85.6% reported performing teleradiology within the past 10 years.¹⁰ The skills we have learned in performing teleradiology directly translate into radiology education in an e-learning environment. In this paper, we discuss the implementation of our blended e-learning based rotation and explore the survey results obtained from our initial participants.

MATERIALS AND METHODS

In the second half of March, because of the COVID-19 pandemic, the Uniformed Services University of the Health Sciences recalled all medical students from clinical rotations and back to the National Capital Region. On April 3, it was announced that the class of 2021, i.e., MS4s, would not return to in-person clinical rotations until at least late June, and therefore, each department was solicited for proposals for virtual electives for May and June.

A curriculum document for a 4-week diagnostic radiology virtual elective rotation was rapidly designed then submitted for MS4 enrollment. An early decision in our developmental process was to embrace a weekly modular format, where each of the 4 weeks would focus on one subspecialty of diagnostic radiology and would be assigned to a fellowship-trained “mentor”. This is a different model from most physical radiology rotations, where a medical student typically rotates and observes through various reading rooms on a day-by-day basis. Changing to a weekly modular format decreased variety of exposure to only four subspecialties in exchange for increased depth of exposure and continuity that was predicted to maximize student engagement in a virtual environment.

Weekly Modules

Each fellowship-trained weekly mentor was granted substantial autonomy in the choice and design of educational material for their week. The decentralized approach was helpful in the rapid construction of our virtual curriculum and was also intended to encourage a diversity of approaches to “solve the problem” of radiology education. Only three guidelines were provided: (1) provide approximately 3 to 5 hours of content per day including reading assignments, (2) specify objectives and requirements to pass each week, and (3) host at least one interactive conference including a quiz. In total, this diversity of approach and use of materials exposed the rotators to sources ranging from textbooks such as Learning Radiology: Recognizing the Basics 4e and Felson’s Principles of Chest Roentgenology 5e to commonly used websites such as radiopaedia.org, radiologyassistant.nl and thoracicrad.org to primary journal articles such as Radiographics and the ACR Appropriateness Criteria (Appendix). An example of a single day’s assignments is seen in Fig. 1.

FIGURE 1.

A daily assignment example from our 4-week diagnostic radiology virtual elective rotation. This example from the chest radiology week depicts our liberal use of hyperlinks and checkboxes in this Adobe PDF document for students’ accessibility and convenience. Each week’s daily assignments included reading, video lectures, and module/website tasks; they were listed under Monday to Friday, but there was flexibility to complete tasks earlier or later as needed.

• Week 1: Neuroradiology (Dr. Shih)

• Week 2: Thoracic Radiology (Dr. Peterson)

• Week 3: Abdominal Radiology (Dr. Lee)

• Week 4: Musculoskeletal Radiology (Dr. Heltzel)

Email Listserv

For simplicity of adding and removing rotators to email dissemination of information, we used a group email address to send out course details. The email listserv contained all current rotators as well as weekly mentors and course directors. Group membership was simple to update whenever there was a student addition or departure and whenever a member preferred to receive with a different email address.

Cloud Drive

Course materials were centralized in a cloud drive folder, which was shared with the group email address. Whenever a student was scheduled to join our rotation, he or she would receive an invitation to our shared folder, which includes the curriculum PDF, a brief “READ ME” orientation, and lists of enrolled students by block. It also includes a folder for each weekly module, which houses assigned material that is not freely accessible online at home (e.g., downloaded articles, PowerPoint slides). On the other hand, assigned material that is freely accessible online (e.g., modules, videos, websites) was embedded as web hyperlinks within the curriculum PDF document (see Appendix). Administrative access to the Drive was given only to the rotation directors. Rotators were not given administrative privileges to alter the contents of the Drive.

Video Conferences

Interactive educational sessions were conducted on a video conferencing platform. Each weekly mentor was able to send invitations to the group email listserv containing all of the current rotators’ emails. Although the minimum requirement was one interactive conference with the radiologist at the end of the week on Friday, three mentors opted to schedule conferences more frequently (one had daily recurring invitations from Monday to Friday at 3 p.m.). The virtual meetings ranged in length from 15 minutes (e.g., quick video chat orientations) to 1 hour (e.g., PowerPoint case conferences). Students could unmute their microphone to ask or answer questions, however the chat box proved to be a fast and efficient method to incorporate audience interactivity (see Fig. 2). Using the built-in record function, MP4 video files could be shared on the cloud drive, either for review purposes or for students with a home or schedule conflict precluding live attendance.

FIGURE 2.

Screen capture from an interactive video conference on the afternoon of elective Week 1 Tuesday. Calendar invitations were sent to the group’s group email address beforehand. This hour-long conference started with a brief open-ended Q&A session on the day’s assignments and then progressed to a case conference that was designed to reinforce the assigned material. On the right, the chat box proved to be a fast and efficient method to incorporate audience interaction. In the upper left, the small REC icon indicates that recording is activated (attendees receive an alert and can see the icon).

Rotating medical students and interns were also invited to join our radiology residency’s morning and noon lectures, also conducted via video conference. Although most physical radiology rotations would require in-person attendance at department conferences, we decided to encourage but not mandate participation in our radiology residency’s virtual lectures for two reasons: (1) we believed that student engagement could be limited in conferences targeted to a radiology resident’s level of understanding especially in a virtual setting and (2) we wanted to provide students maximal flexibility outside of our core weekly module material as they and their families adapted to the COVID-19 restrictions.

End-of-Rotation

At the end of the rotation, students sent their completed curriculum documents to the rotation supervisor (Dr. Adams), which satisfied the passing requirement for this pass/fail elective. These PDF documents used fillable checkboxes as student attestation on daily assignment completion and participation in each mentor’s end-of-week quiz. At the end of these documents, there was a free text block for positive or negative rotation feedback (to be received by the rotation supervisor, who is not a weekly mentor) and a digital signature block. The rotation supervisor also distributed a separate end-of-rotation questionnaire with Likert scale answers (see Fig. 3). Departing students’ email addresses were removed from the radiology elective’s email listserv.

FIGURE 3.

This chart shows survey responses to three critical questions. Rotators indicated that they were likely to recommend the virtual course to colleagues, that the materials were easy to access, and that the course increased their confidence integrating imaging into their future practices.

RESULTS

In total, we accommodated 75 rotators in the 2 months of the virtual rotation comprised of 36 MS4s, 5 transitional year (TY) interns, and 34 MS3s. Our prior record high number of rotators for a single in-person block was 12 (Fig. 4.)

FIGURE 4.

Graph showing the total number of department rotators over the course of academic year 2019 to 2020. April and May 2020 reflect the significant increase in number of rotators in our department on the virtual rotation during the time of COVID-19 mitigation.

• April 27 to May 22: 17 MS4s enrolled (4 scheduled as partial rotations)

• May 4 to May 29: 2 TY interns enrolled (offset from MS4s by 1 week)

• May 25 to June 19: 19 MS4s enrolled (4 scheduled as partial rotations)

• June 1 to June 26: 3 TY interns enrolled (offset from MS4s by 1 week)

• June 8 to Jun 19: 34 MS3s enrolled (partial rotations because of quarantine)

A postrotation survey was sent to the 41 MS4 and TY intern rotators. The 34 MS3s were excluded from the survey as many of them decided to quarantine in place at military bases in Hawaii and Washington and were unable to attend live sessions as they were five and three time zones away, respectively. Thirty-five out of 41 trainees responded, a response rate of 85%. The survey, designed to be completed in an average of 60 seconds, was limited to 7 total questions with free text boxes provided for additional comments. Survey highlights are shown in Fig. 3. When asked how likely our students were to recommend this course to a colleague, responses were nearly universally positive, quantified by 34 agree or strongly agree responses, with one neutral response. Thirty-two of 35 (91%) of respondents agreed or strongly agreed that the elective rotation virtual materials were easy to access with only three neutral responses. Thirty-five out of 35 respondents selected agree or strongly agree that our virtual elective increased their confidence integrating imaging with their future practice of medicine.

All five of the respondents who had experienced an in-person radiology elective rotation before this virtual rotation rated the virtual elective as more educational or much more educational than the traditional in-person rotation. Thirty-one of 35 (89%) of the rotators gave this virtual elective rotation a positive rating, with 16 trainees scoring the course as 4 out of 5 and 15 trainees selecting 5 out of 5. Four respondents selected neutral (3 out of 5) for the overall course rating. Thirty-four of 35 (97%) of the rotators rated the amount of assigned material as appropriate, with 1 response rating the assigned material as too much. Twenty-nine of 35 rotators rated the course material as “at my level” with 6 responders rating the material above my level.

The survey comments focused on the added value of the live in-person conferences, for example, “The video conference meetings were the most helpful part of the course” and “Weekly sessions with mentors were very helpful.” The constructive feedback offered also largely concerned the video lectures, for example, “I would like at least 2 to 3 virtual meetings each week, since the instruction and interaction I got through those made me more confident in my knowledge of the material” and “I would value more frequent check in/virtual, live knowledge checks!”

DISCUSSION

We were pleasantly surprised by the large jump in the number of rotators in our department (Fig. 4). Most importantly, this large number meant we were able to help ensure the safety of many of our intern and medical student trainees during COVID-19 mitigation by reducing their chances of exposure to the virus. We previously averaged five to six medical student and intern rotators per month before the addition of the virtual rotation. After transitioning to a virtual rotation, this number increased to greater than 20 rotators during the 2 months we have offered the elective. The immediate effect is that many trainees now have additional experience with radiology with positive survey results attesting to the benefits of this exposure. Although the downstream effects will not be known until we look back in 2 to 3 years, this increased attendance and exposure of medical students to radiology will hopefully pay dividends in increased applications to our residency, as well as to the other radiology residencies throughout Army GME.

The survey results, as summarized above, were nearly universally positive. The rotators expressed overall satisfaction with the volume and complexity of the assigned material. All responders indicated that the course helped them integrate radiology into their future practice and agreed that they would recommend the course to colleagues. Significantly, all rotators who had previously completed an “in-person” radiology rotation agreed that the virtual rotation was more educational. We suspect this is at least partially because of the increased staff interaction as well as the increased assigned material culminating in a weekly quiz.

As part of the virtual rotation curriculum, we decided to invite medical students and trainees to our morning and noon conference didactic residency lectures, which were also converted to virtual delivery via video conference. There were several unanticipated benefits of our decisions to deliver our GME residency lectures via a virtual platform and inviting our rotators to participate. In February 2020, before we transitioned to a virtual rotation and video conferences for our platform for morning and noon conferences, we averaged 2 to 6 faculty, 12 to 15 residents, and 2 to 4 rotators; typically never exceeding 25 to 30 individuals in our resident library/lecture hall. Since transitioning to virtual rotations during the ongoing pandemic, our total attendance has more than tripled with 6 to 18 faculty, 20 to 25 residents, and 20 to 30 rotators attending regularly. This marked increase in trainees across the medical training spectrum accessing our resident didactics has been confirmed via recorded daily attendance, reaching a peak of 72 participants at one noon lecture. Interestingly, word of mouth from participants has led to several medical students that are not currently enrolled in the virtual elective to ask for access to our virtual materials. This increased interest and engagement with a virtual curriculum may also translate well to other image-driven specialties such as pathology or dermatology, among others.

The weekly modular format, essentially comprising four mini-electives, proved advantageous for administrative reasons, both by distributing the responsibilities of execution across four experienced attendings and by allowing for flexibilities of scheduling. For example, in both blocks, there were a few MS4s with other military officer training requirements who requested partial rotations: 1, 2, or 3 weeks. There were also a few TY interns who had been pulled off other electives because of the pandemic and who asked to join the MS4 elective. During the second block, 34 MS3s who were scheduled to return from out of state learned that they would be required to complete a mandatory 2-week quarantine before returning to work. We were able to add them immediately into 2 weeks of our virtual rotation. Even though the rotation schedules of MS3s and TY interns were offset from the schedule of MS4s by 1 week, all of these requests were fairly easy to accommodate because of the weekly modular format.

There are multiple limitations to this study of our new virtual radiology rotation. We were only able to run the curriculum for 2 months before Uniformed Services University of the Health Sciences decided to re-implement in-person rotations, which limits the size of our dataset. In particular, there were only 5 of 41 surveyed rotators who had previously completed an in-person rotation to adequately compare the two. Direct comparison of the virtual to in-person formats is also limited by the switch to a weekly format from a day-to-day format. There are also multiple limitations of the virtual format itself, the most significant of which is the lack of exposure to more ancillary radiology services such as nuclear medicine, mammography, and interventional radiology. Additionally, the rotators do not get a sense of the day-to-day life of a radiologist or the feeling of being in the reading room. This may be a considerable limitation, particularly for a potential interviewer.

CONCLUSION

Transitioning to a virtual curriculum because of the COVID-19 pandemic has been a successful tool for maintaining engagement and training for our medical students and interns during periods of viral mitigation. We found many unanticipated benefits such as increased attendance and participation in residency lectures, increased imaging confidence in trainees, and a significantly higher overall number of rotators in the department. Given the ability to reach a larger number of medical students and interns via a virtual platform, we posit that residency programs should consider a virtual curriculum not only to maintain rotator engagement during COVID-19 mitigation but also as a way to increase trainee exposure to radiology.

APPENDIX

Full version of the virtual rotation curriculum is possibly for upload as an online supplement.

FUNDING

None declared.

CONFLICT OF INTEREST STATEMENT

None declared.