Radiology Medical Student Education 2020: Surveys of the Alliance of Medical Student Educators in Radiology and Medical Students

Abstract

Rationale and Objectives

The coronavirus disease 2019 (COVID-19) pandemic has resulted in significant changes to medical student education by disrupting clinical rotations, licensing exams, and residency applications. To evaluate the pandemic's impact and required modifications of radiology medical student courses, the authors developed and administered surveys to Alliance of Medical Student Educators in Radiology (AMSER) faculty and enrolled medical students. The surveys requested feedback and insight about respondents’ experiences and innovations.

Materials and Methods

Anonymous twenty-question and seventeen-question surveys about the pandemic's impact on medical student education were distributed via email to AMSER members and medical students. The surveys consisted of multiple choice, ranking, Likert scale, and open-ended questions. Differences in the Likert score agreement was performed using one-sided Wilcoxon-Mann-Whitney tests. Survey data were collected using SurveyMonkey (San Mateo, California). This study was IRB exempt.

Results

The AMSER survey indicated 96% of institutions cancelled medical student courses and 92% resumed with virtual courses, typically general radiology. A total of 64% of faculty enjoyed online teaching, although 82% preferred on-site courses. A total of 62% of students felt an online radiology course was an excellent alternative to an on-site rotation, although 27% disagreed. A total of 69% of students who completed both on-site and online courses preferred the on-site format. Survey-reported innovations and free response comments have been collated as educational resources.

Conclusion

Faculty were able to adapt radiology courses to the online environment utilizing interactive lectures, self-directed learning, flipped classroom sessions, and virtual readouts, which were effective for student respondents. Hybrid rotations with on-site and online elements may offer the best of both worlds.

INTRODUCTION

Since March 2020, the coronavirus disease 2019 (COVID-19) pandemic has upended all facets of health care delivery and education in the United States (U.S.), with lasting long-term impacts of the pandemic likely underappreciated as of yet (1,2). What has been clearly established is the disruption to U.S. medical schools’ curricula and to their students’ education and lives (3,4).  Acting swiftly in response to the national climate, the Association of American Medical Colleges (AAMC) declared in March 2020 that “unless there is a critical healthcare workforce need locally,” students’ clinical rotations should be suspended. The AAMC stated that “The primary goals of this guidance are bending the curve for the public health of North America, conserving limited PPE supply to keep health care workers and patients safe, and maintaining public and health care workers safety given current testing availability” (5). Beyond clinical rotations, there was temporary suspension of administrations of United States Medical Licensing Examination testing (6), delay of examinations required by medical schools for students to advance, and postponement  of the timeline for residency application submission (7). In addition, at several institutions, fourth year medical students were given the opportunity to graduate several months early, thereby facilitating early entry into residencies in an effort to combat hospital staffing shortages (8).

The March 2020 regulatory actions by the AAMC and the Coalition for Physician Accountability's Work Group (a cross-organizational group composed of U.S. medical education regulatory agencies) served to sharply curtail medical student educational options (5). Because on-site clinical rotation options were precipitously reduced to limited numbers and away rotations were cancelled except under specific circumstances, radiology educators everywhere encountered a “once in a lifetime” need to reexamine pedagogy and reconfigure medical student course work (1). Members of the Alliance of Medical Student Educators in Radiology (AMSER), an affinity group of the Association of University Radiologists and the sole organization of academic radiologists devoted to medical student education, were also significantly impacted by the March 2020 regulatory actions (9).

It is well-reported that radiology residents experienced a pandemic-related decrease in imaging interpretation and procedure volume and adoption of virtual lectures and hotseat conferences at residents’ institutions, often via national radiological societies (10,11,12,13). Tumor boards, journal clubs, and Morbidity and Mortality conferences were also adapted to a virtual format (14).

This study was undertaken to determine the extent of analogous impacts on medical student education. The authors’ expectations of survey results included the following: faculty inexperience with online teaching prepandemic, faculty increased reliance on publicly available and/or proprietary teaching materials in online courses, faculty preference for on-site teaching and belief it is a more realistic representation of a radiologist's role, medical students’ lack of confidence in their ability to order and interpret critical findings on imaging studies and medical students’ preference for on-site learning.

To test these hypotheses, the authors created and distributed surveys to both faculty and medical students. The intent of the faculty survey was to determine the changes made to adapt to this pandemic and faculty perceptions of these changes. The medical student survey was intended to determine how these modifications impacted the medical students’ experiences in radiology courses.

The pandemic's impact on medical student education will be prolonged into 2021, with lasting long-term impacts unknown. Identifying preferences, challenges, and resources is important in curriculum planning to meet this imminent need. Adjustments are certain to be required for the remainder of this academic year, and lessons learned are likely to impact postpandemic medical student radiology education. This manuscript intends to offer valuable resources to radiology educators in its collation of medical student education practices reported in the surveys and very recent radiology literature.

MATERIALS AND METHODS

The multi-institutional author group developed two surveys to evaluate the educational methods and online teaching methods implemented as a result of the COVID-19 pandemic. These surveys were intended for faculty who taught and the medical students who completed radiology courses during the 2020 impact.

Questions in the faculty and student surveys focused on the following: general information about radiology course offerings and structure resulting from the COVID-19 pandemic, resources assigned by faculty, student and faculty opinions about online radiology courses, and basic demographic information. Many of the curriculum resources listed in the surveys were derived from the Educator Resources page of the AMSER website (15). Although as many of the survey questions as possible were analogous, some questions were specific to each cohort. Faculty-specific questions surveyed differences in online versus on-site radiology course enrollment and resources used in online curriculum development. Student-specific questions surveyed interest in future radiology courses and radiology as a career. The survey consisted of multiple choice, ranking, Likert scale, and open-ended questions.

In addition to the consent, nineteen questions were included in the faculty survey, six of which sought details on radiology courses offered at their institutions, three on educational materials and resources, two each on teaching responsibilities, time commitment, and prior experience with online teaching, and one each about of online radiology courses, future radiology course offerings, and demographics. An optional open-ended question asking for additional comments concluded this survey.

In addition to the consent, the student survey included sixteen questions, seven of which asked general information on the radiology course(s) they completed; three on their opinions and perceptions of the online course(s); two each on prior experience with on-site radiology courses and assigned resources. There was also a single demographics questions and an optional open-ended question asking for additional comments, which concluded this survey.

Both faculty and student respondents were encouraged to provide free response comments to several of the questions. To remove barriers to participation, survey respondents were not required to respond to every question to contribute. The full survey instruments can be viewed as appendices to this article.

The faculty survey was reviewed and vetted by AMSER leadership establishing face validity and a pilot test was undertaken before survey distribution. On September 3, 2020, an email containing a link to the approved anonymous online survey was distributed to the 90 current members of AMSER. A single reminder was sent on September 24, 2020. Survey results were collected for a total of 5 weeks.

Prior to the distribution of the medical student survey, a pilot test was performed. For distribution, the authors accessed the AAMC database (8) and identified 152 allopathic U.S. medical schools. Contact information for departmental medical student course coordinators or program coordinators was identified on 102 of the medical schools’ Department of Radiology websites. The authors contacted these coordinators via email, requesting the survey be emailed to any medical students enrolled in an institutional radiology course between February and September 2020. Surveys were first distributed via email to coordinators on August 5, 2020 with reminders sent to the coordinators at 2-week intervals. Student survey results were collected for a total of 8 weeks.

Both surveys were self-administered and voluntary. All responses were anonymous. There were no exclusion criteria applied to either group. Data for both surveys were collected using SurveyMonkey (San Mateo, California).

Statistical Analysis

We evaluated differences in the Likert score agreement using one-sided Wilcoxon-Mann-Whitney tests.  This test has been demonstrated to have similar power and Type 1 error rates for analyzing Likert data, compared to the two-sample t-test (16,17). One-sided p-values <0.05 were considered evidence of a difference in agreement between groups. One-sided p-values >0.05 were considered inconclusive.

This study was exempted by the lead author's Institutional Review Board.

RESULTS

AMSER Survey

A total of 33 faculty (37%; n = 33/90) accessed the AMSER member survey and responded to the first item, which was an informed consent attestation, in accordance with IRB requirements. A variable number of faculty answered the remaining questions, as specified for each item.

Enrollment and Course Details

Nearly all faculty respondents (96%; n = 25/26) indicated on-site rotations were cancelled at their institution as a result of the COVID-19 pandemic. 92% (n = 24/26) reported that their institutions offered at least one online radiology course, with general diagnostic radiology being the most frequently offered. A smaller number of subspecialty courses was offered. Online courses ranged in duration from 1 to 4 weeks, with 4-week rotations being the most common. Online course offerings and duration are presented in Figures 1 and 2 . Compared to on-site course enrollment, 35% (n = 8/23) reported that online enrollment had increased, while 30% (n=7/23) reported it had decreased or remained similar (35%, n = 8/23).

Figure 1.

Online Radiology Courses. General Diagnostic Radiology was the course most frequently offered and completed by both faculty and survey respondents.

Figure 2.

Course Duration - AMSER and Student Surveys. The majority of the faculty and student respondents reported teaching or completing a 4-week radiology course.

Faculty Opinions and Experience

Faculty opinions regarding online versus on-site courses were assessed through agreement to several statements using a 5-point Likert scale, with 5 indicating strong agreement. Most faculty (63.6%; n = 14/22) enjoy online teaching, although the majority (81.8%; n = 18/22) prefer teaching on-site and feel on-site courses are more interactive and engaging (77.3%; n = 17/22), and give students a more realistic experience. Nevertheless, 40% (n = 8/20) of the represented institutions plan to continue online radiology courses even after on-site radiology rotations resume, while 35% percent (n = 7/20) are undecided. These results are summarized in Table 1 .

Table 1.

Online Versus On-site Course Comparison - AMSER Survey. Faculty respondents expressed an overall preference for on-site medical student teaching, with faculty overall believing that on-site courses are more interactive and engaging and offer a more realistic representation of a radiologist's role compared to an online course

EVALUATION	STRONGLY, DISAGREE (1)	DISAGREE (2)	NEUTRAL (3)	AGREE (4)	STRONGLY AGREE (5)	WEIGHTED, AVERAGE
I enjoy online medical student teaching.	4.6% (1/22)	4.6% (1/22)	27.3% (6/22)	50.0% (11/22)	13.6% (3/22)	3.64
I prefer on-site medical student teaching.	0%	0%	18.2% (4/22)	9.1% (2/22)	72.7% (16/22)	4.55
On-site courses are more interactive and engaging than online courses.	0%	0%	22.7% (5/22)	4.6% (1/22)	72.7% (16/22)	4.50
On-site courses give students a more realistic representation of a radiologist's role when compared to online courses.	0%	0%	4.8% (1/21)	14.3% (3/21)	81.0% (17/21)	4.76
Preparation and time spent teaching an online course is greater than that of an on-site course.	0%	15.0% (3/20)	45.0% (9/20)	20.0% (4/20)	20.0% (4/20)	3.45

Although a few faculty members had taught online previously (18%; n = 4/22), none of the respondents’ institutions offered online radiology rotations prior to the pandemic. 80% (n = 16/20) of faculty reported regularly interacting with their students. The amount of time spent in online course development varied considerably, from less than 5 to more than 40 hours, with most faculty reporting 11-25 hours (61%; n = 11/18). The amount of time that faculty spent interacting with students ranged from 3 to 35 hours per week.

Educational Resources

Faculty were queried regarding the educational resources utilized in online versus on-site rotations, presented in Figure 3 . Real-time didactic lectures by faculty remained the most utilized resource in both formats, while real-time teaching at the workstation, the second most common resource on-site, was eliminated from most online courses. Pre-recorded lectures, teaching cases and web-based material saw increased utilization in online courses.

Figure 3.

Online versus On-site Educational Resource Utilization - AMSER Survey. The most frequently used resource in online courses was real-time didactic lectures, although this was used with slightly less frequency than the traditional on-site course. There was also an increase in the use of pre-recorded didactic lectures, use of both publicly available and proprietary didactic websites, apps, and PACS cases, and an increased use of ACR Radiology - TEACHES. Workstation readouts decreased precipitously.

Table 2 serves as a collated list of resources based on faculty respondents. Additionally, many of the published articles cited in the References section serve as valuable educator resources.

Table 2.

Suggested Resources for Medical Student Educational Content

Diagnostic Radiology
AMSER Curriculum https://www.aur.org/en/affinity-groups/amser/curriculum
Aquifer www.aquifer.org
Game of Unknowns http://www.create-rad.com/radgames
Icarus http://www.icarus-rad.com/
Learning Radiology www.learningradiology.com
UBC Radiology http://undergrad.ubcradiology.ca/
UVA Radiology https://www.med-ed.virginia.edu/Courses/rad/
Emergency Radiology
A Night in the ED http://radiology.cornfeld.org/EDindex.php
Neuroradiology
Learning Neuroradiology www.learningneuroradiology.com
Pediatric Radiology
Children's Hospital Cleveland Clinic Pediatric Radiology https://www.cchs.net/onlinelearning/cometvs10/pedrad/
Society of Pediatric Radiology www.pedrad.org
Thoracic Radiology
Society of Thoracic Radiology https://thoracicrad.org/
Society for Advanced Body Imaging (SABI) On-Demand Webinar: “Covid-19: Thoracic Imaging Findings and Recommendations” https://www.advancedbodyimaging.org/Education/SABI-Webinars/2020-Webinar-Coronavirus
Comprehensive Resources from the ACR https://www.acr.org/Member-Resources/Medical-Student/Medical-Educator-Hub/Curriculum , 2 week virtual medical student course, ACR-TEACHES, Radiation Safety Cases, Standardized Tool for the Assessment of Radiology Students (STARS), Educational COVID-19 Cases, Educator's Toolkit

Respondents’ Roles

63.6% of respondents were clerkship directors (n = 14/22), 9.1% (n = 2/22) were program directors, 22.7% (n=5/22) were diagnostic radiology faculty, and one respondent was a Vice Chair. There were no interventional radiology faculty respondents, aside from those serving in these leadership roles.

Subgroup Analysis

Opinions of non-course director and course director faculty are summarized in Table 3 . Noncourse director faculty indicated a preference for on-site medical student teaching, believing that on-site courses are more engaging than their course director counterparts (p-value = 0.020).

Table 3.

Faculty and Student Subgroup Analysis. Among faculty, course directors enjoyed online medical teaching more than non-course directors. Compared to course directors, non-course directors felt that on-site courses were more interactive and engaging than online courses. There is insufficient evidence that agreement is different between the remaining subgroups analyzed

Faculty	Course Director	Non-course Director	P-value
On-site courses are more interactive and engaging than online courses	4.21	5	0.020*
On-site courses give students a more realistic representation of a radiologist's role when compared to online courses	4.64	5	0.070
I enjoy online medical student teaching	3.93	3.13	0.025*
Students	Pre-Course	Post-Course
I felt confident in my ability to order appropriate imaging studies	2.45	3.76	<0.0001*
I felt confident in my ability to recognize critical imaging findings	2.52	3.94	<0.0001*
I plan to specialize in diagnostic radiology	2.24	2.45	0.28
Students	2 Week Course	4 Week Course	P-value
After completing my online course, I felt confident in my ability to order appropriate imaging studies	4	3.58	0.071
After completing my online course, I felt confident in my ability to recognize critical imaging findings	4.08	3.85	0.14
After completing my online course, I plan to specialize in diagnostic radiology	2.5	2.37	0.42
Students	General	Subspecialty	P-value
After completing my online course, I felt confident in my ability to order appropriate imaging studies	3.96	3.57	0.19
After completing my online course, I felt confident in my ability to recognize critical imaging findings	3.33	4	0.46
After completing my online course, I plan to specialize in diagnostic radiology	2.41	2.64	0.31
Students	On-site Preference	Online Preference	P-value
After completing my online course, I felt confident in my ability to order appropriate imaging studies	2.55	2	0.35
After completing my online course, I felt confident in my ability to recognize critical imaging findings	3.55	3.82	0.12
Students	Likely to Pursue Radiology	Unlikely to Pursue Radiology	P-value
Prior to my online course, I felt confident in my ability to order appropriate imaging studies	2.33	2.5	0.26
Prior to my online course, I felt confident in my ability to recognize critical imaging findings	2.56	2.5	0.41
After completing my online course, I felt confident in my ability to order appropriate imaging studies	3.89	3.71	0.31
After completing my online course, I felt confident in my ability to recognize critical imaging findings	4	3.92	0.49

^⁎indicates difference is significant (p-value <0.05).

When asked if they agreed with the statement “I enjoy online medical student teaching,” 78.5% of clerkship directors agreed or strongly agreed (n = 11/14). In contrast, of noncourse director radiology faculty, 37.5% agreed with this statement. This agreement is significantly greater for course directors, compared to non-course directors (p-value = 0.025).

There was insufficient evidence that the agreement to the statement “On-site courses give students a more realistic representation of a radiologist's role when compared to the online course” was different when comparing course directors to noncourse directors.

Comments and Recommendations

Faculty were asked what changes might be made to future on-site rotations as a result of the online course experience. The most frequent planned change was the addition or expansion of online resource utilization during on-site rotations. Many also described plans for increased use of virtual participation for both faculty and students, as well as recording and storing lectures for students to review in the future. One faculty respondent hoped to offer online material to students at other schools who were unable to enroll in a traditional on-site rotation. Another suggested that it would be helpful to incorporate the residents into the online medical student courses.

Comments comparing on-site and online radiology rotations were insightful. One respondent stated that “Students got more undivided attending attention during the online course, but at the cost of interacting with residents, as well as seeing what clinical radiology practice looks and feels like.” Another wrote that “...having a student passively watch read-outs would turn away any but the most dedicated student from pursuing radiology as a career. It has to be interactive…”. This idea was echoed by another respondent who indicated they would continue to use the “look ahead” technique (described further in the Discussion).

Student Survey

A total of 80 students accessed the medical student survey and entered at least the informed consent attestation. Because the total number of students receiving the survey was unknown, a response rate could not be determined. A variable number of students responded to the other questions, as specified for each item.

Enrollment and Course Details

45.2% (n = 14/31) of the survey respondents were fourth year medical students, 51.6% (n = 16/31) were third year students, and 3.2% (n = 1/31) were second year students. Most students (86%; 48/56) indicated that on-site rotations were cancelled at their institution as a result of the COVID-19 pandemic, and a majority completed an online radiology course (75%; n = 41/55); all but one of which was offered by the student's home institution (98%; n = 48/49). Rotation duration varied between two and four weeks. General diagnostic radiology was the most commonly taken course (Figs 1 and 2). The vast majority of students (91%; n = 38/42) completed a single online radiology course.

Online Course Quality and Student Satisfaction

Student assessments of online course quality and impact were measured through agreement to several statements using a 5-point Likert scale, with 5 indicating strong agreement, presented in Table 4 . Most students agreed online education was clinically relevant, accessible and engaging. Students reported an increased confidence in their ability to order and interpret imaging studies (p-value <0.0001). Most students (62%; 21/34) felt an online radiology course was an excellent alternative to an on-site rotation.

Table 4.

Online Course Quality Ratings - Student Survey. Among all students, there was an increase in pre- and post-course confidence in imaging ordering and interpretation

EVALUATION	STRONGLY, DISAGREE (1)	DISAGREE (2)	NEUTRAL (3)	AGREE (4)	STRONGLY AGREE(5)	WEIGHTED, AVERAGE
During the rotation, I learned clinically relevant material that was presented in an accessible and engaging way.	3.0% (1/33)	0%	12.1% (4/33)	63.6% (21/33)	21.2% (7/33)	4.00
Prior to the rotation, I felt confident in my ability to order appropriate imaging studies for common indications.	12.1% (4/33)	51.5% (17/33)	18.2% (6/33)	15.2% (5/33)	3.0% (1/33)	2.45
After the rotation, I feel more confident in my ability to order appropriate imaging studies for common indications.	0%	6.1% (2/33)	21.2% (7/33)	63.6% (21/33)	9.1% (3/33)	3.76*
Prior to the rotation, I felt confident in my ability to recognize critical imaging findings.	12.1% (4/33)	42.4% (14/33)	30.3% (10/33)	12.1% (4/33)	3.0% (1/33)	2.52
After the rotation, I feel more confident in my ability to recognize critical imaging findings.	0%	6.1% (2/33)	12.1% (4/33)	63.6% (21/33)	18.2% (6/33)	3.94*

^⁎Indicates statistical significance (p-value <0.0001).

Most students who completed online radiology rotations had already planned to take a radiology elective pre-pandemic (79%; n = 27/34). In addition to completing an online radiology course, 38% (n = 12/32) intended to take an on-site radiology course. Sixteen students completed both online and on-site radiology courses, the majority of whom (69%; n = 11/16) preferred the on-site rotation (Fig 4 ).

Figure 4.

Prior medical student radiology course experience (n = 16). A slight majority of students reported never having taken an onsite radiology course. Of the 43% of students who had taken both online and onsite courses, 69% of those students preferred their on-site experience.

Student interest in radiology as a career showed a small increase after completing an online course, although this was not significant (pre-course/post-course average = 2.24/5 and 2.45/5, respectively, p-value=0.28, n = 33).

Educational Resources

Students were asked to rank the value of various educational resources used for online teaching (Fig 5 ). Live lectures presented virtually were the highest-rated educational resource overall, ranked first for 29% of respondents (n = 10/34), and in the top three for 77% of respondents (n = 26/34). While online courses included a variety of pre-recorded content, most students (81%; n = 26/32) reported having regular interaction with faculty during the online rotations.

Figure 5.

Online Course Resource Rankings – Student Survey. Resources are presented in descending rank order from top to bottom, with 14 being the maximum achievable score. Didactic lectures were ranked most highly by student respondents, unknown case conference the lowest.

Subgroup Analysis

The following student subgroup analyses were performed: those completing 2 versus 4-week courses, opinions of those who took general radiology versus subspecialty courses, and those planning to pursue radiology versus those unlikely to pursue radiology. Additionally, responses from students who had completed both online and on-site radiology courses were compared. Each of these subgroup analyses showed there was insufficient evidence that agreement differed between the statements evaluated. Details of this analysis are summarized in Table 4.

Qualitative analysis of resources shows that students likely to specialize in radiology preferred didactic lectures, teaching at the workstation and/or readouts, and publicly available didactic apps and websites. The most useful resources reported by students unlikely to specialize in radiology were real-time didactic lectures, prerecorded faculty lectures, and ACR Radiology-TEACHES.

Comments and Recommendations

Many students appreciated the flexibility of an online course and felt that they were able to learn effectively. However, for those who had experience with both course formats, being on-site was favored, providing exposure to the radiology reading room environment, increased interaction with faculty and residents, and more real-time home institution cases. Similarly, independent case review was better received with image-viewing interfaces that simulate a PACS, with scrollable studies. Several students also mentioned a preference for being shown normal studies and atypical imaging presentations, in addition to classic presentations of common diseases.

Two respondents reported taking a hybrid course, with online and on-site components. One student wrote “The hybrid course that I am completing this survey about was very effective. I think it had distinct advantages over a traditional course.”

DISCUSSION

As the sole AMSER member survey and the largest published student survey on radiology online medical student education, the data presented here offer needed insight and guidance to radiology educators. This is of particular topical interest in the face of ongoing pandemic challenges and restrictions that will persist into 2021.

The faculty survey results indicate faculty teaching online radiology courses incorporated pre-existing online teaching materials, increased use of pre-recorded didactic lectures, and decreased workstations readouts. Survey results confirmed that while faculty enjoyed online teaching, they had a strong preference for on-site teaching. The students surveyed reported their online course instruction was effective and, for those who had experience with both online and on-site formats, their preference for on-site education.

Student and faculty survey responses also reinforce important pedagogy and teaching methods:

1. Engage students with interactive lectures.

2. Record lectures and make them available online for students to access at a later time.

3. Actively reinforce material that students have studied independently through self-assessments, image interpretation, and discussion.

4. Provide opportunities for students to independently view images, make observations, and present their findings to a faculty member, for feedback and teaching points.

While these ideas are not new, the survey results highlight the importance of adaptation of these concepts to online radiology courses.

Lectures

Despite research suggesting that traditional teaching and learning resources (textbooks, study groups, and in-person lecture attendance) are falling out of favor with contemporary medical students, our survey results indicate didactic lectures remain an important resource for medical students enrolled in radiology courses (18). Interactive lectures are preferred to increase student engagement and maintain interest. Gamified presentations and lectures that incorporate audience participation can be particularly effective (19,20). Limiting lecture length to no more than 1 hour is also important; after this point, attention begins to wane (20). For courses with large student enrollment, Durfee et al describe the incorporation of moderators to monitor chat functions in virtual meeting platforms and answer students’ questions as they arise (20). This practical solution mitigates the challenges of simultaneously teaching and managing chat conversation in real-time.

Based on survey comments and reports in the literature (18), recording lectures and providing students with access to them after, or even prior to, the formal didactic session is also helpful. It seems surprising that pre-recorded lectures were not used more frequently by faculty respondents, and the authors postulate this may be due to insufficient time for faculty to create or identify such resources. One student suggested that overall lecture time be somewhat limited to allow adequate time to independently review material.

Case Based Learning and Flipped Classrooms

Survey results showed increased incorporation of self-directed learning and case-based learning compared to on-site courses, frequently taking the form of websites, apps, and other case-based activities. Students perceived these vetted online resources as helpful, ranked fourth overall (and the highest-ranked resource from outside a student's institution). Self-directed learning and case-based learning can be combined in a synergistic manner. While independent self-study can be useful as students learn new material, it should be reinforced with knowledge application through self-assessments, image interpretation, and discussion. Unknown case conferences, while not rated highly by the student respondents in this survey (for unknown reasons), have been a successful component of other online radiology courses (19,20). The authors did not specifically seek details about the websites and apps that survey respondents used for case-based learning, although more information about these could be useful.

Developing and collating these resources and materials is a time consuming process, as evidenced by the amount of time course directors spent creating online courses. Much of this time was likely spent prior to the first offering of the online course. While for some the hours spent were considerable, it is reasonable to infer that the time invested by faculty may have decreased with subsequent offerings of the same course. Studies have shown that the adoption of digital tools can ultimately be more efficient for medical educators (21).

Independent Case Review and Virtual Readouts

Virtual readouts were rated as the third most helpful resource among all students, with subgroup analysis demonstrating that students likely to pursue radiology preferred this over students who were not. Students unlikely to specialize in radiology had a stronger preference for teaching resources such as ACR Radiology-TEACHES and Aquifer.

Independent review of home-institution cases has been reported in the literature as highly effective. Alpert et al describes the process of creating a faculty-curated subspecialty case list with scrollable studies that students can review and discuss. Results from that study suggest the process is more engaging than a passive observational student role in a traditional on-site rotation (22). Incorporating the “look ahead” technique, in which students preview a case, present the findings, and receive feedback about their interpretation, is another effective method to engage students(19,23). When adapting this technique to the remote setting, pre-review of virtual platform anonymized scrollable cases from one's home institution is followed by case discussion including teaching points (19).

Student survey comments serve as reminders to faculty that screen-sharing must be activated during virtual readouts and that review of imaging findings should be at a level appropriate for students in those sessions that are co-mingled with resident attendees.

Use of Technology

Only a few of the faculty respondents had previous experience with online teaching and none of the respondents' departments had previously offered an online radiology course. While many faculty enjoyed online teaching, the vast majority preferred teaching on-site. The reasons for this are numerous and varied, but for many likely include a lack of prior online teaching experience. O'Doherty et al report that staff-focused e-learning and training workshops can be used to overcome this inexperience (21). The authors postulate faculty may have lacked sufficient training in these new technologies given the abrupt and unanticipated nature of the pandemic impact.

The important roles institutions and departments can and/or should play in faculty development and technology resource support cannot be overstated. Additionally, a helpful suggestion offered by one student was that there should be “a central hub for communication about lecture schedules and announcements.”

Limitations

This study is limited by both faculty and student survey response rates. Although the 37% faculty response rate is similar to APDR surveys (24,25), the total number of responses was low, limiting generalizability. This also brings to light that AMSER has significantly fewer members than APDR (90 vs. 312) (25) during a time that a unified widely shared and collaborative approach on the part of AMSER could be highly advantageous.

The number of student responses was low due to the challenges posed by survey distribution itself, and likely by decreased student enrollment at some institutions. Survey administration relied on a program coordinator distributing the surveys electronically to enrolled students. Other survey methodologies entertained (Alliance of Medical Student Coordinators in Radiology, social media, online forums) were dismissed as overly problematic. Another limitation of the student survey was the high rate of survey noncompletion, potentially attributable to survey length.

While some faculty and student subgroup analysis were performed, this was limited by the small numbers of respondents across several categories. The non-significant p-values may be the results of an underpowered study, rather than an indication that these differences were not significant.

Finally, as the surveys did not request the respondents’ institutional affiliation, it remains unknown as to whether results may overrepresent certain institutions, adding potential bias.

Looking to 2021 and Beyond

The long-term implications of COVID-19 on the educational landscape remain uncertain. Online radiology courses are a viable option to on-site rotations. Our surveys suggest, however, that hybrid rotations combining elements of on-site and online courses may be the best answer. Such a rotation might include 2-3 days per week on-site, directly observing and participating in readouts and/or procedures. This achieves course goals for students to gain a better understanding of, and participate in, radiologists’ daily interaction with patients and physicians, daily workflow, and patient procedures. Online activities 2-3 days per week could include didactic lectures, self-study assignments, and virtual readouts. Much as residents are frequently an integral part of on-site medical student education, interested residents should also be invited to participate in or lead online teaching sessions (26).

Arguably, this hybrid may offer the best of both worlds. Students frequently have a passive role when on-site during a radiology rotation, but do benefit from being able to ask questions, see a variety of cases, and understand the day-to-day activities of a radiologist; these are best accomplished in person. Virtually, students may participate in active learning activities which are self-directed, case-based, and medical student-centric, and have all been shown to achieve desired results (19,20,22,23). A 2-week virtual curriculum hosted on the American College of Radiology (ACR) website (27) may also be used within an institution's radiology elective or as an independent study resource for medical students.

In the manner of a call to action, the authors proffer that a collaborative and widely shared set of updated AMSER resources and interactive sessions would be extremely helpful.

CONCLUSION

Despite the initial inexperience with online teaching and preference of on-site teaching expressed by our faculty respondents, faculty were able to successfully adapt their courses to the online environment utilizing interactive lectures, self-directed learning, flipped classroom sessions, and virtual readouts. Our data also support earlier research demonstrating that online radiology courses can be an effective way for students to learn important concepts and gain experience interpreting studies. Hybrid rotations, such as those outlined previously, may represent the best of both worlds.

DECLARATION OF COMPETING INTEREST

All authors confirm there are no known conflicts of interest associated with this publication. There has been no financial support for this work.

Appendix A

Online (Virtual) Radiology Course Survey for Faculty

Q1. I agree to participate in this research study. I understand the purpose and nature of this study and I am participating voluntarily. I understand that I can withdraw from the study at any time, without any penalty or consequences.

A.        Yes

B.        No

Q2. Were on-site medical student rotations canceled at your institution as a result of the COVID-19 pandemic?

A.        Yes

B.        No

Q3. Does/did your institution offer at least one online radiology course as a result of the pandemic?

A.        Yes

B.        No

Q4. Please indicate the duration of each online radiology course and the number of blocks offered.

(Free Text)

Q5. Which statement best describes student enrollment in online radiology courses at your institution?

A.   On-site student enrollment is greater than that of comparable online course offerings

B.  On-site and online student enrollment is similar for comparable course offerings

C.       Student enrollment for online courses is/was greater than what is typical for comparable on-site offerings

Comments

Q6. Which of the following online radiology course(s) has your department offered during the pandemic? (Select all that apply)

A.        Abdominal/Body Imaging

B.        Breast Imaging

C.       Cardiothoracic Radiology

D.       Emergency Radiology

E.        General Diagnostic Radiology

F.        Interventional Radiology

G.       Musculoskeletal Radiology

H.       Neuroradiology

I.         Nuclear Medicine

J.        Pediatric Radiology

K.        Ultrasound

L.        Other (please specify)

Q7. Which radiology course(s) have you personally taught? (Select all that apply)

A.        Abdominal/Body Imaging

B.        Breast Imaging

C.       Cardiothoracic Radiology

D.       Emergency Radiology

E.        General Diagnostic Radiology

F.        Interventional Radiology

G.       Musculoskeletal Radiology

H.       Neuroradiology

I.         Nuclear Medicine

J.        Pediatric Radiology

K.        Ultrasound

L.        Other (please specify)

Q8. Which of the following are students expected to participate in or complete during an on-site course at your institution? (Select all that apply)

A.        ACR Radiology-TEACHES

B.        Aquifer

C.       Didactic lectures, given by faculty in real time

D.       Didactic lectures given by faculty,  pre-recorded and reviewed at leisure

E.        Didactic lectures, publicly available

F.        Didactic websites/apps created by home institution

G.       Didactic websites/apps, publicly available

H.     Faculty or resident teaching at the workstation/readouts, in real time

I.     Independent review of home institution imaging studies on a real or simulated PACS workstation

J.    Independent review of publicly available imaging studies on a real or simulated PACS workstation

K.        Student presentations

L.        Textbooks

M.       Unknown case conferences, facilitated by faculty

N.       Other (please specify)

Q9. Which of the following best describes faculty teaching responsibilities during an on-site medical student course at your institution? (Select all that apply)

A.        Didactic lectures

B.        Instruction at the PACS workstation

C.       Review of assigned materials with students

D.       Unknown case conference facilitation

E.        Other (please specify)

Q10. Which of the following resources are students expected to participate in or utilize during online radiology courses at your institution? (Select all that apply)

A.        ACR Radiology-TEACHES

B.        Aquifer

C.       Didactic lectures, given by faculty in real time

D.       Didactic lectures given by faculty,  prerecorded and reviewed at leisure

E.        Didactic lectures, publicly available

F.        Didactic websites/apps created by home institution

G.       Didactic websites/apps, publicly available

H.      Faculty or resident teaching at the workstation/readouts, in real time (virtually)

I.     Independent review of home institution imaging studies on a real or simulated PACS workstation

J.       Independent review of publicly available imaging studies on a real or simulated PACS workstation

K.        Student presentations

L.        Textbooks

M.       Unknown case conferences, facilitated by faculty

N.       Other (please specify)

Q11. Does/did your department's online courses include regular faculty interaction with students?

A.        Yes

B.        No

If yes: How many hours per week did you meet? What was the primary method used to interact with students? (Free text)

Q12. Approximately how many hours did you spend creating a curriculum and educational activities for online medical student radiology course(s).

A.        0-5

B.        6-10

C.       11-15

D.       16-20

E.        21-25

F.        26-30

G.       31-35

H.       36-40

I.         More than 40

Q13. Did you use any resources to help guide you in the creation of an online medical student course?

A.        Yes

B.        No

Please describe any resources you used. (Free text)

Q14. Please indicate your level of agreement with the following statements about onsite and online radiology courses.

Answer Choices: Strongly disagree, Disagree, Neither agree nor disagree, Agree, Strongly Agree

I enjoy online medical student teaching.

I prefer on-site medical student teaching.

On-site courses are more interactive and engaging than online courses.

On-site courses give students a more realistic representation of a radiologist's role when compared to online courses.

Preparation and time spent teaching an online course is greater than that of an on-site course.

Additional comments

Q15. What changes (if any) will you or your department make to your on-site radiology rotations as a result of your online rotation experience?

(Free text)

Q16. My department will continue to offer online radiology courses for medical students after on-site rotations resume.

A.        Yes

B.        No

C.       Unsure

Q17. Prior to the COVID-19 pandemic, did your department offer online medical student radiology courses?

A.        Yes

B.        No

Comments

Q18. Did you personally have any prior experience teaching an online radiology course prior to the pandemic?

A.        Yes

B.        No

C.       Please describe any previous online radiology medical student teaching experiences.

Q19. What best describes your role?

A.        Clerkship/Course Director

B.        Program Director

C.    Diagnostic Radiology Faculty, not Clerkship/Course or Program Director

D.  Interventional Radiology Faculty, not Clerkship/Course or Program Director

E.     Other (please specify)

Q20. Additional comments:

(Free text)

Appendix B

Online (Virtual) Radiology Course Survey for Medical Students

Q1. I agree to participate in this research study. I understand the purpose and nature of this study and I am participating voluntarily. I understand that I can withdraw from the study at any time, without any penalty or consequences.

A.        Yes

B.        No

Q2. Were on-site rotations at your institution canceled as a result of the COVID-19 pandemic?

A.        Yes

B.        No

Q3. Did you enroll in and complete an online radiology course as a result of the COVID-19 pandemic?

A.        Yes

B.        No

Q4. Was the online course offered at your home medical school or at another (away) medical school?

A.        Home

B.        Away

Q5. What was the duration (in weeks) of your online course?

Q6. Which radiology course(s) did you complete? (Select all that apply)

A.        Abdominal/Body Imaging

B.        Breast Imaging

C.       Cardiothoracic Radiology

D.       Emergency Radiology

E.        General Diagnostic Radiology

F.        Interventional Radiology

G.       Musculoskeletal Radiology

H.       Neuroradiology

I.         Nuclear Medicine

J.        Pediatric Radiology

K.        Ultrasound

L.        Other (please specify)

Q7. How many online radiology courses have you completed?

(Free text)

Q8. Please rank in order (from most to least helpful) the resources utilized in your online radiology course(s).

A.        ACR Radiology – TEACHES

B.        Aquifer

C.       Didactic lectures, given by faculty in real time

D.       Didactic lectures given by faculty,  prerecorded and reviewed at leisure

E.        Didactic lectures, publicly available

F.        Didactic websites/apps created by home institution

G.       Didactic websites/apps, publicly available

H.     Faculty or resident teaching at the workstation/readouts, in real time

I.     Independent review of home institution imaging studies on a real or simulated PACS workstation

J.    Independent review of publicly available imaging studies on a real or simulated PACS workstation

K.        Student presentations

L.        Textbooks

M.       Unknown case conferences, facilitated by faculty

N.       Other

Q9. Please list any resources not mentioned previously that were used in your rotation.

(Free text)

Q10. Please rate your level of agreement with the following statements as it pertains to your online (virtual) radiology course experience. (28)

Answer Choices: Strongly disagree, Disagree, Neither agree nor disagree, Agree, Strongly Agree

A.        Prior to the cancellation of on-site rotations, I had planned to complete a radiology course.

B.        Prior to my online course, I felt confident in my ability to order appropriate imaging studies.

C.       Prior to my online course, I felt confident in my ability to recognize critical imaging findings.

D.       After completing my online course, I felt confident in my ability to order appropriate imaging studies.

E.        After completing my online course, I felt confident in my ability to recognize critical imaging findings.

F.        A virtual radiology course is an excellent alternative to an on-site rotation.

G.       I learned clinically relevant material presented in an accessible and engaging way.

H.       I plan to take an on-site radiology course.

I.       Before completing my online course, I planned to specialize in diagnostic radiology.

J.        After completing my online course, I plan to specialize in diagnostic radiology.

Q11. During your online rotation, did you interact regularly with at least one diagnostic radiology faculty member?

C.       Yes

D.       No

Q12. Have you completed an on-site radiology rotation?

A.        Yes - prior to academic changes related to COVID-19

B.        Yes - after resuming normal educational activities

C.       No

Q13. If you have completed both on-site and online radiology rotations, which format do you prefer? (28)

A.        Online

B.        On-site

Comments

Q14. Which of the following best describes your level of training at the time of your online radiology rotation?

A.        MS IV

B.        MS III

C.       MS II

D.       MS I

E.        Other (please specify)

Q15. What did you like most about your online radiology rotation?

(Free text)

Q16. How could your online radiology rotation have been improved?

(Free text)

Q17. Additional comments:

(Free text)