Medical student education in the time of COVID-19: A virtual solution to the introductory radiology elective

Abstract

Rationale and objectives

During the COVID-19 pandemic, medical educators and students are facing unprecedented challenges while navigating the new virtual landscape that social-distancing policies mandate. In response to these challenges, a new virtual introduction to radiology elective was established with unique online resources and curriculum.

Materials and methods

A previously in-person 2-week introductory radiology elective was converted into a completely virtual experience using an internally developed, open-source, peer-reviewed, web-based teaching modules combined with virtual lectures, interdisciplinary conferences, and readout sessions of de-identified cases loaded to a DICOM viewer. Students from the first four months of course enrollment completed a multiple choice pre- and post-course knowledge assessments and a 5-point Likert Scale survey as part of their educational experience.

Results

In total, 26 4th-year medical students participated over 4 separate 2-week sessions from July to October of 2020. This included 12 students from the home intuition and 14 visiting students. On average, students scored 62.2% on the 55-question pre-test and 89.0% on the same test upon completion of the course, a statistically significant increase (p < 0.001). All 26 students felt engaged throughout the course. All 26 agreed (23 “strongly agreed”) that they were more comfortable looking at imaging studies following the course. All 26 also agreed (21 “strongly agreed”) that the course helped them prepare for their future clinical rotations and careers.

Conclusion

Initial pilot program using unique web-based resources and student encounters during a two-week virtual introductory radiology elective proved to be a positive educational experience for the first 26 students enrolled.

1. Introduction

The coronavirus disease 2019 (COVID-19) pandemic has profoundly impacted nearly all facets of medicine. While patient care and community health initiatives have been at the forefront during this time, the current and probable long-standing effects on medical education cannot be understated.1., [2], [3]

The Association of American Medical Colleges published guidelines urging medical schools to suspend in-person clinical rotations for medical students in mid-March of 2020.⁴ While the decision to remove students from clinical duties varied geographically, all in-person medical school electives were cancelled at our academic institution in New York City during the initial US peak of the pandemic. There are several justifications for the cancellation of in-person clerkships, including: to help flatten the curve in the local general population, to decrease the risk of exposure to medical students, and to preserve personal protective equipment.^{5 , 6} The introductory radiology elective, which had grown in popularity among home institution and visiting medical students in recent years, was included in these cancellations.

During the time of COVID-19, medical educators are tasked with addressing the ever present need to train highly competent and assured students within the confines of safety and social distancing policies. With this in mind, the introductory radiology elective was converted to a completely virtual, two-week course utilizing the “flipped classroom” model with synchronous and asynchronous learning activities.^{7 , 8} The goal was to establish an online curriculum, including interactive modules, dedicated lectures, and small group case discussions, encapsulating all of the core components of the in-person elective. This manuscript shares the curriculum, resources, and outcomes from the first four months of course enrollment in this virtual introductory radiology elective.

2. Methods and materials

The virtual radiology elective at our academic institution was designed by the radiology faculty course director for the prior in-person introductory elective in radiology. The elective was offered during a two-week block each month from July through October 2020. During this time, a total of 26 students (all in their fourth-year of medical school) enrolled in the course (see Table 1 for additional student demographics). Due to the virtual nature of the elective, no cap was placed on the total number of students allowed. The structure of the two-week elective allowed for a total of 10 class days. The “flipped-classroom” method was used, meaning that students completed the online learning assignments independently at their own pace (asynchronous learning) prior to the real-time in-person (synchronous learning) teaching activities on the same topic. Major components of the course were: the daily online learning assignments, two daily 45-minute radiology resident conferences via Zoom,⁹ daily 60-minute medical student lectures via Zoom, five 60-minute virtual readout sessions using a web-based viewer hosted in the cloud (AMBRA),¹⁰ and interdisciplinary conferences via Zoom (Table 2 ). Course objectives were the same as those of the prior in-person elective: to learn about the different diagnostic imaging exams and therapeutic procedures that radiology offers and understand their appropriate uses, to learn the indications and contraindications for specific imaging tests and associated safety considerations, and to review radiologic anatomy and learn basics of image interpretations including “must know” diagnoses. The WhatsApp Messenger was used to communicate with the students in real time throughout the workday outside of the in-person sessions.¹¹ Specifically, course directors used this application to greet students each morning, send assignment reminders, and respond in real time to student questions about online assignments. Following institutional review board approval, students completed pre- and post-course knowledge assessments and an anonymous post course survey as part of their educational experience. The knowledge assessment and survey questions were the same for each of the four course enrollments. These surveys were optional and administered via SurveyMonkey email invitations (Appendix A).¹²

Table 1.

Student demographics.

Total number of student enrolled	26
Enrollment per session
July session	5 (19.2%)
August session	7 (26.9%)
September session	6 (23.1%)
October session	8 (30.8%)
Graduate year
3rd year medical student	0 (0%)
4th year medical student	26 (100%)
Sex
Male	18 (69.2%)
Female	8 (30.8%)
Institution
Home institution student	12 (46.2%)
Visiting/outside institution student	14 (53.8%)

Table 2.

Virtual elective assignments and daily schedule.

Day 1 (Introduction)	Day 2 (Abdominal)	Day 3 (Chest)	Day 4 (Pediatrics)	Day 5 (MSK/ER)
Assignments
RT: Modalities & Safety	GoU: Abd Edition	RC: CXR	RT: Peds Patient	RC: Fracture Imaging
RC: MRI	ATC: Abd Signs	RT: Tubes and Lines	GoU: Peds	RC: Knee MRI
RC: Abd x-rays	ICARUS: GI 1–4	GoU: Chest Edition	ICARUS: GI 5	RT: Arthritis
RC: Abd CT	Preview Abd Cases	ATC: Chest Signs	ICARUS: GU 5	ATC: MSK signs
		ICARUS: CP 1–4	ICARUS: CP 5	ICARUS: MSK 1–4
		Preview Chest Cases	ICARUS: MSK 5	Preview MSK Cases
			Preview Peds Cases
Daily schedule
9 am - orientation/pre-test
	10 am - body interesting case conference
12:30-2 pm - resident lectures	12:30-2 pm - resident lectures	12:30-2 pm - resident lectures	12:30-2 pm - resident lectures	12:30-2 pm - resident lectures
3-4 pm - medical student lecture	3-4 pm - medical student lecture	3-4 pm - medical student lecture	3-4 pm - medical student lecture	3-4 pm - medical student lecture
4 pm - GU Tumor Board	4 pm - virtual readout	4 pm - virtual readout	4 pm - virtual readout	4 pm - virtual readout

Day 6 (Neuroradiology)	Day 7 (Ultrasound)	Day 8 (Women's Imaging)	Day 9 (IR/Nuclear Medicine)	Day 10 (Wrap Up)
Assignments
RC: Head CT	RT: US Scrotum	RT: Breast Parts 1 & 2	RT: Nuclear Medicine	RadResources
RT: Stroke	RT: US Thyroid	GoU: Breast Edition	RT: Oncology Pt	ATC: Food Signs
RT: C-Spine trauma	RT: US Trans Kid	RT: Female Pelvic Pain	GoU: Oncology
GoU: Neuro Edition	GoU: US Edition	RT: Preg/Postpartum Pt	iRad Series
ICARUS: BS 1–5	ICARUS: GU 1	ICARUS: GU 2–3
Preview Neuro Cases
Daily schedule
				8 am - GYN/ONC Board
12:30-2 pm - resident lectures	12:30-2 pm - resident lectures	12:30-2 pm - resident lectures	12:30-2 pm - resident lectures	12:30-2 pm - resident lectures
3-4 pm - medical student lecture	3-4 pm - medical student lecture	3-4 pm - medical student lecture	3-4 pm - medical student lecture	2:30 pm - post-test 3:30 pm - test review
4 pm – virtual readout	4 pm - Neuro Tumor Board	4 pm - Chest Tumor Board		4 pm - post course survey

www.create-rad.com.

RC: RadCast; RT: RadTorial; Abd: Abdominal; GoU: Game of Unknown; ATC: Ace the Case; ICARUS: Interactive Clinical Anatomy and Radiology Utilization Simulator; CP: Cardiopulmonary; GI: Gastrointestinal; GU: Genitourinary; MSK: Musculoskeletal; BS: Brain and Spine; Trans Kid: Transplant Kidney; Preg: Pregnant; GYN/ONC: Gynecology Oncology.

2.1. Daily online learning assignments

The schedule design allowed students to focus on a different area or subspecialty of radiology each day. The daily assignments were sent to the students via email on the first day of the course. Within each area of focus, students were assigned online learning modules from an educational platform designed by the authors. This online platform, the Clinical Radiologist Educator Alliance for Teaching Excellence (www.create-rad.com) is an open-source peer reviewed repository of learning modules designed by radiology faculty and trainees at the home institution of the virtual elective. The different types of modules include: short podcast-type lectures, linear self-directed learning modules with pre- and post-test questions, and branching self-directed learning modules with real-time assessment and feedback. The students were also assigned modules from the Interactive Clinical Anatomy and Radiology Utilization Simulator (www.ICARUS-rad.com), which was also designed by the authors. This platform is also open-source and offers peer reviewed modules designed to encourage students to think critically about the use of imaging during patient encounters while reviewing important concepts in radiologic anatomy, basic image interpretation, and safety. Curricular design for module content and diagnoses was based on the Alliance of Medical Student Educators in Radiology (AMSER) national curriculum in radiology.¹³ On the last day of the elective, students were given the opportunity to explore some of the other web-based educational radiology resources highlighted in the RadResources page of our website.

2.2. Radiology resident conference

The students participated in two 45-minute virtual radiology resident conferences per day given by radiology faculty (total 20 conferences over the 2-week period). These conferences were a mix of didactic and interactive case presentations and were presented over Zoom to the residents and students.

2.3. Medical student lectures

The students also virtually attended daily 60-minute lectures by subspecialty radiology faculty, fellows, and senior residents. These lectures were designed specifically for them and included student level case-based interactive conferences and introductory didactic lectures on different subspecialty topics in radiology. Lecture topics and diagnoses were chosen with guidance from the AMSER National Medical Student Curriculum in Radiology (https://www.aur.org/en/affinity-groups/amser/curriculum).

2.4. Virtual readout sessions

A web-based radiology viewer hosted in the cloud (AMBRA) was used to provide elective students with their own virtual readout sessions with subspecialty radiology faculty. Cases from the core radiology rotations (chest, abdominal, musculoskeletal, neuroradiology, and pediatric radiology) were compiled by the authors in conjunction with the “must know” diagnoses suggested in the AMSER curriculum. Whenever possible, the diagnoses were shown in multiple imaging modalities (Table 3 ). These cases were de-identified and sent to the AMBRA image viewer where unique URLs to each case were generated. Worksheets were created for the students that provided the link to each case with questions regarding pertinent findings, potential complications, and diagnosis. Similarly, answer sheets were created for the faculty member assigned to each readout session, covering teaching points and pearls for each case. The students were asked to preview the cases during the day along with their other assignments, and to answer the questions for each case to the best of their ability. The 60-minute interactive readout session was then completed with radiology faculty at the end of the day.

Table 3.

Virtual readout diagnoses.

Abdominal	Chest	Pediatrics	Musculoskeletal/Emergency	Neuroradiology
Acute appendicitis	Misplaced NGT	Buckle fracture	Tibial plateau fracture	Acute stroke
Acute pancreatitis	Pneumomediastinum	Swallowed coin	Colles' fracture	Hydrocephalus
Sigmoid diverticulitis	Pneumothorax/tension	Croup	Trimalleolar fracture	Subdural hematoma
Cecal mass with perforation	Right middle lobe pneumonia	Non-accidental trauma	Radial head fracture	Epidural hematoma
Small bowel obstruction	Saddle pulmonary embolism	Toddlers fracture	Anterior shoulder dislocation	Intra-axial mass
Splenic laceration	Diaphragm rupture	Supracondylar fracture	Scaphoid fracture	Lumbar spine compression fracture
Sigmoid volvulus	Aortic dissection	Pneumonia	Subcapital femoral fracture	Disc herniation
Hydronephrosis with calculus	Cardiomegaly	Wilms tumor	Pelvis fracture	Type II dens fracture
Ruptured abdominal aortic aneurysm	Pleural effusion	Neuroblastoma	Osteomyelitis of toe
Hepatic metastases	Emphysema	Transient synovitis	Distal femur enchondroma
	Pneumoperitoneum	Acute appendicitis	Multiple myeloma
	Misplaced endotracheal tube
	Chest tube for pneumothorax

2.5. Interdisciplinary conferences

The students virtually attended what were considered by section chiefs to be some of the most high-yield departmental interdisciplinary conferences. These conferences were led by radiology faculty and covered genitourinary tumor board, neuroradiology tumor board, chest case conference, abdominal interesting case conference (with a neighboring academic institution), and gynecology tumor board.

2.6. Assessment and grading

On the first day of the elective, the students received a handout outlining the course objectives and expectations. The virtual course would have the same pass/fail grading policy as the prior in-person elective. Their participation in all components of the course would result in a passing grade. All online assignments and zoom conferences were considered mandatory unless discussed with the course director beforehand. Assignment completion was based on the honor system. The students also had to demonstrate improvement from the pre-course test to the post-course test. The test was made up of questions across multiple subspecialties and modalities as well as imaging physics, all of which were covered in the educational materials and lectures provided. The students were also asked to complete an anonymous and voluntary 20-question post course survey on SurveyMonkey to provide feedback on the virtual elective. A five-point Likert Scale was used to assess student perceptions of the various course components. Their participation was not mandatory and in no way impacted their grade in the course.

2.7. Data analysis

Student pre- and post-course knowledge assessment scores were compared via an unpaired t-test. Survey data Likert scale score averages calculated.

3. Results

A total of 26 fourth-year medical students were enrolled in the two-week virtual elective between the months of July and October 2020.

3.1. Knowledge assessment

All students were given a 55-question pre-course test on the first morning of the course. Raw scores ranged from 26/55 to 42/55 with an average of 34.42/55 or 62.6%. The same test was administered at the conclusion of the course. The scores on the post-course test ranged from 40/55 to 54/55 with an average of 48.9/55 or 89%. An unpaired t-test showed this increase in score to be statistically significant (p < 0.001).

3.2. Student experience and perceptions

There was a 100% response rate for the anonymous voluntary 20-question post-course survey. Twenty-one of the 26 students reported that they had not taken a virtual course similar to this one in medical school. They all reported that they would recommend this virtual elective to other students.

Overall, students felt that the course was educational, engaging, and well organized with effective delivery of material (Table 4 ). All 26 students agreed (23 of the 26 “strongly agreed”) that they felt much more comfortable looking at imaging studies after completing the course. All 26 students agreed (21 of the 26 “strongly agreed”) that the amount of radiology exposure during the virtual elective had helped to prepare them for their clinical rotations and future career.

Table 4.

Course evaluation.

Course component	Avg score	Scale
Course engagement & structure
Overall educational value	4.80/5	5 = extremely valuable – 1 = not at all valuable
Course organization	4.35/5	5 = extremely effective – 1 = not at all effective
Delivery of material	4.58/5	5 = extremely effective – 1 = not at all effective
Level of engagement	3.96/5	5 = completely engaged – 1 = not at all engaged
Lectures		5 = very high quality – 1 = very low quality
Resident lectures	4.35/5
Medical student lectures	4.92/5
AMBRA virtual readout sessions	4.85/5	5 = excellent – 1 = poor
Interdisciplinary conferences	4.50/5	5 = excellent – 1 = poor
CREATE modules		5 = strongly agree – 1 = strongly disagree
Interactive	4.73/5
Informative	4.96/5
Enjoyable	4.77/5
Encouraged self-directed learning	4.81/5
Fostered critical thinking	4.62/5
Effective in delivery of material	4.81/5
Met course goals and objectives	4.96/5

Prior to the elective, 17 of the students were considering a career in radiology and felt that the course had reaffirmed their choice. Four students had not considered a career in radiology prior to the course but were considering it after taking the course. Five students were not considering a career in radiology prior to the course and that did not change after taking the course. When asked to comment on the most valuable part of the virtual elective, the student responses cited each of the various course components with the virtual readout sessions and CREATE modules receiving multiple mentions. Open text responses were collected regarding any change in perception of the field of radiology after taking the course (Table 5 ). Overall, responses were positive, and many emphasized the important role of radiology in patient care.

Table 5.

Anonymous student feedback.

Has your perception of the field of radiology changed after the virtual elective, if yes, how so? •Yes, it has shown how complex and difficult of a field it is to become proficient in, and how valuable the contribution of a radiologist is to medical care. •I have learned a lot and radiology is less intimidating. •Radiology is much more nuanced than I thought, and I really enjoyed it! •I've gained more clarity on how radiology interacts with other departments, what imaging is available to those departments, and where radiology is going as a field. •It gave me a nice exposure to the various sub-specialties in radiology and helped me better appreciate the ubiquitous role of radiology in managing and guiding patient care. •Radiology is such a vast field and radiologists are so knowledgeable, invested in teaching, supportive and fun! •It absolutely confirmed that I want to go into radiology! •Yes- radiology is awesome and I want to pursue this path. •I know much more now about the field of radiology. This virtual elective felt like we were on an actual radiology rotation- between the resident lectures, med student lectures, and ability to view cases- and I cannot thank you enough for that. •Yes, I am now much more interested in radiology and I wish I was exposed to this elective before I submitted my ERAS application. •I have a much better understanding of the day-to-day tasks of a radiologist.

3.3. Evaluation of course components

3.3.1. Lectures

The daily virtual radiology resident lectures received an overall score of 4.35/5 and the virtual medical student lectures received an overall score of 4.92/5 (5 = very high quality).

3.3.2. AMBRA

All of the students reported the teaching cases via virtual PACS AMBRA viewer to be extremely valuable. The quality of the virtual readout sessions had a reported average of 4.85/5 with a score of 5 as “excellent.”

3.3.3. CREATE

Fourteen of the students completed 100% of the assigned online learning modules on the CREATE website, while ten students reported completing 75–100% of the assigned modules. Overall, the students reported that they thought the CREATE modules were interactive, informative, enjoyable, and encouraged self-directed learning. They all reported that they would like to view additional modules on other topics in radiology. When asked what they enjoyed most about the CREATE modules, multiple students commented that they loved the “interactive” and “self-paced” nature of the modules and thought that concepts were taught in a “clear and informative” manner, specifically citing the ICARUS and Game of Unknown modules as exemplary. One student commented that the modules “integrated clinical workup and had far more advanced topics than my home radiology rotation.” Several students reported that they liked the variety of different formats used in the different types of modules and one student commented that “they were well thought out and centered on bread and butter topics that are useful for everyone regardless of career choice.”

3.3.4. Interdisciplinary conferences

The quality of the interdisciplinary conferences had a reported average of 4.50/5 with a score of 5 as “excellent.” Student comments included: the “tumor board sessions really highlighted the multidisciplinary approach to patient care and how important radiology plays a role in next steps” and “these were great for seeing how different specialties come together to answer important questions.”

3.3.5. Recommendations

When asked about areas for improvement the students mentioned some of the technical difficulties that we encountered with the Zoom sessions, which we were able to resolve in real-time, but certainly impacted the flow of the daily schedule. One student suggested promoting more engagement with the students during virtual readout sessions by allowing them to “take” cases in more of a hot-seat format. All of the students commented that they appreciated the responsiveness from faculty when issues arose. No substantive changes were made to the course based on student feedback during the July–October course enrollment period.

4. Discussion

Social distancing measures related to the COVID-19 pandemic have prompted a revamping of current teaching methods in medical education. In radiology, technological advances in image archiving and viewing have lent themselves well to remote learning for medical students.^{14 , 15} Advances in web-conferencing and student response system software have also contributed greatly to our ability as radiologist educators to offer students a robust and interactive virtual learning experience.^{16 , 17} Using these innovative teaching strategies, other investigators have demonstrated great success with virtual radiology electives and virtual core clerkship in radiology.^{18 , 19}

At our institution, the introductory radiology clerkship had been replaced with a vertically integrated radiology component within the general medical school curriculum. We offered an in-person elective for those who wished to learn more about radiology. Using the “flipped-classroom” model, the prior two-week in-person radiology elective was successfully converted to a completely virtual format. This introductory radiology elective often attracts medical students who are assessing their interest in radiology as a future career path, in addition to the inherent learning objectives of the clerkship. It was paramount in designing this clerkship to mirror a typical workday in radiology as much as possible, including virtual readout sessions on PACS. By utilizing a web-based image viewer hosted in the cloud, the students in this elective were not bound by institutional VPN access. The students previewed all of the cases prior to the readout session with faculty just as a radiology resident would do on rotation. The students enjoyed the autonomy of looking at studies on their own, despite reports of feeling a bit overwhelmed at first. The readout sessions were interactive with screen sharing among the participants so that they could ask specific questions and point out key findings. The feedback provided by the students that they would like more of a leading role in the readout sessions with more of a “hot-seat” format will be incorporated into future courses.

Student engagement was something we thought about a great deal in designing this virtual course. We wanted the students to feel as engaged in the work as they would if they were sitting next to us in the reading room. We were very pleased with the engagement responses from the post course survey. The high level of engagement that the students reported during the course mirrored that of the course directors and faculty. We made sure to stay in contact with the students throughout the day via WhatsApp. Course directors checked in with them each morning, made them aware of any last-minute changes to the schedule or Zoom links, and made sure they knew we were available for questions. For all of the virtual readout sessions and medical student lectures we asked that all participants turn their video on, so that we could all see one another. The level of engagement that the students experienced is also reflected in the number of students newly considering radiology as a career after completing the elective. Many comments regarding the breadth and depth of radiology as a subspecialty, and the importance of radiology in guiding patient care reflects a critical shift in perspective for many of the medical students after taking the course. One of the areas we would like to work on in the future is to increase the amount of interaction the students have with the radiology residents. This has always been a highlight of the in-person elective and it is something we would like to encourage in the virtual elective as well.

One of the major limitations of our study is the small sample size. We decided against collecting additional data over the coming year of elective enrollment at our institution, because we wanted to expeditiously disseminate our resources and what we have learned from our early experience, in the hope that other institutions may benefit from our methodology during this challenging time. The open source nature of our online modules, daily assignments and diagnosis list, combined with broadly used videoconferencing technology makes this course accessible and reproducible for other radiology departments. Additionally, we see a growing role for the virtual elective in the coming months as travel restrictions continue and visiting students may not be able to do in-person radiology electives at other institutions. We feel that a virtual elective such as this one, with robust online resources combined with individual institutional virtual lectures and conferences may serve as an important recruitment tool for residency programs.

Future directions may also include international virtual electives in radiology providing radiology education in areas of the world that need it. This would also allow for exposure to a very diverse student population and offer a different perspective into global imaging practices and diagnoses. Looking beyond the pandemic when the constraints of social distancing are lifted, this pilot virtual elective format could serve as a model with improved hands-on experience and clearer educational milestones.

5. Conclusion

Our initial experience piloting a completely virtual radiology elective has been a positive one. The first 26 students participating in the early months of this course offering enjoyed the experience and reported feeling more comfortable looking at imaging studies and more prepared for their future rotations after taking the course. This two- week elective will continue to be offered at our institution. Future directions include development of an international elective for students unable to visit our institution due to travel restrictions. Sharing our course curriculum and resources may allow educators at other institutions to adopt similar practices in response to the challenges related to educating medical students during the COVID-19 pandemic.

Appendix A. Survey questions

Q1. How would you describe the overall educational value of this Virtual Elective in radiology?

Answer Choices:

5- Extremely valuable

4- Very valuable

3- Neutral

2- Somewhat valuable

1- Not valuable at all

Q2. Have you ever completed a virtual course similar to this one during medical school?

Answer Choices:

Yes

No

I don't recall

If yes, please describe below: (open text)

Q3. Would you recommend this virtual elective to other students?

Answer Choices:

Yes

No

I'm not sure

Q4. Please select which of the below statements most applies to you:

Answer Choices:

I was considering a career in radiology prior to this elective and this course has reaffirmed my choice.

I was not considering a career in radiology prior to this course but I am now considering it.

I was not considering a career in radiology prior to this course and that has not changed.

Q5. Please comment on how engaged you felt throughout this virtual course?

Answer Choices:

5- Completely engaged

4- Very engaged

3- Neutral

2- Somewhat engaged

1- Not engaged at all

Q6. What was your impression of the overall organization of the course in terms of the daily schedule and delivery of the material?

Answer Choices:

5- Extremely effective

4- Very effective

3- Neutral

2- Somewhat effective

1- Not at all effective

Additional comments? (open text)

Q7. What was your impression of the overall organization of the course in terms of the daily schedule and delivery of the material?

Answer Choices:

5- Extremely effective

4- Very effective

3- Neutral

2- Somewhat effective

1- Not at all effective

Additional comments? (open text)

Q8. Please rate your response to the following statement:

I feel the amount of radiology exposure during this virtual elective has helped to prepare me well for my clinical rotations and for my future career.

Answer Choices:

5- Strongly agree

4- Agree

3- Neutral

2- Disagree

1- Strongly disagree

Q9. Please rate your response to the following statement:

I feel much more comfortable looking at imaging studies after taking this course.

Answer Choices:

5- Strongly agree

4- Agree

3- Neutral

2- Disagree

1- Strongly disagree

Q10. How many of the assigned CREATE online learning modules did you complete during this course?

Answer Choices:

0–25%

25–50%

50–75%

75–100%

All

Q11. Please rate your response to the following statements about the CREATE modules using the 5-point scale provided. (This includes the RadCasts, RadTorials, RadGames, iRad series, and ICARUS.)

Answer choices: 5- Strongly Agree, 4- Agree, 3- Neutral, 2- Disagree, 1- Strongly Disagree

CREATE modules were interactive

CREATE modules were informative

CREATE modules were enjoyable

CREATE modules encouraged self-directed learning

CREATE modules fostered critical thinking

The information provided seemed relevant to my future practice

The information provided met course goals and objectives

The information provided had practical value

The CREATE modules were effective in delivery of the material

I would like to view additional CREATE modules on other topics in Radiology

Q12. What did you enjoy most about the CREATE online learning modules? (open text response)

Q13. How would you describe the overall quality of the daily virtual radiology resident lectures?

Answer Choices:

5- Very high quality

4- High quality

3- Neither high nor low quality

2- Low quality

1- Very low quality

Q14. How would you describe the overall quality of the daily 3 pm medical student radiology lectures?

Answer Choices:

5- Very high quality

4- High quality

3- Neither high nor low quality

2- Low quality

1- Very low quality

Q15. How would you describe the value of the virtual PACS cases using AMBRA links?

Answer Choices:

5- Extremely valuable

4- Very valuable

3- Neutral

2- Somewhat valuable

1- Not valuable at all

Please elaborate on your response: (open text response)

Q16. Please describe the quality of the virtual readout sessions:

Answer choices: 5- Excellent, 4- Very Good, 3- Neutral, 2- Good, 1- Poor

Abdominal Cases Virtual Readout

Chest Cases Virtual Readout

Pediatrics Cases Virtual Readout

MSK Cases Virtual Readout

Neuroradiology Cases Virtual Readout

Q17. Please describe the quality of the virtual interdisciplinary conferences:

Answer Choices: 5- Excellent, 4- Very Good, 3- Neutral, 2- Good, 1- Poor

GU tumor board

Neuro tumor board

Chest tumor board

Gyn/Onc tumor board

Any comments regarding the interdisciplinary conferences? (open text response)

Q18. What has been the best or most valuable part of the Virtual Elective? (open text response)

Q19. Has your perception of the field of radiology change after the Virtual Elective, if yes, how so? (open text response)

Q20. Do you have any comments to the course directors or suggestions for improvement for the Virtual Elective? (open text response)