The Radiation Oncology Education Collaborative Study Group 2020 Spring Symposium: Is Virtual the New Reality?

Abstract

Purpose:

Due to the COVID-19 pandemic, the Radiation Oncology Education Collaborative Study Group (ROECSG) hosted its annual international symposium using a virtual format in May 2020. This report details the experience hosting a virtual meeting and presents attendee feedback on the platform.

Methods and Materials:

The ROECSG Symposium was hosted virtually on May 15, 2020. A post-symposium survey was distributed electronically to assess attendee demographics, participation, and experience. Attendee preference and experience were queried using 3-point and 5-point Likert-type scales, respectively. Symplur LLC was used to generate analytics for the conference hashtag (#ROECSG).

Results:

The survey was distributed to all 286 registrants with a response rate of 67% (n=191). Seventeen non-attendee responses were omitted from this analysis, for a total of 174 respondents included. Forty-seven percent (n=82) of attendees were present for the entire symposium. A preference for a virtual symposium was expressed by 45% (n=78) of respondents, while 25% (n=44) had no preference and 30% (n=52) preferred an in-person meeting; 86% (n=150) of respondents rated the symposium as “extremely” well-organized. Respondents who had not attended a prior in-person ROECSG symposium were more likely to prefer the virtual format (p=0.03). Respondents reported a preference for the virtual platform for reviewing scholarly work (n=78, 45%) and an in-person platform for networking (n=103, 59%). On the day of the symposium, #ROECSG had 408 tweets and 432,504 impressions.

Conclusions:

The 2020 ROECSG Symposium was well-received and can serve as a framework for future virtual meetings. While the virtual setting may facilitate sharing research, networking aspects are more limited. Effort is needed to develop hybrid virtual/in-person meetings that meet the needs of participants in both settings. Social media is a significant avenue for dissemination and discussion of information, and may be valuable in the virtual setting.

Introduction

Scientific meetings allow participants to disseminate and learn about current research while facilitating networking and future research collaboration. While a virtual meeting platform is not novel, in response to the COVID-19 pandemic, many in-person scientific meetings adapted by converting to remote platforms (1). In the scientific community, the virtual format is relatively nascent and presents several challenges, such as optimal planning and platform selection, content delivery, and participant engagement.

The Radiation Oncology Education Collaborative Study Group (ROECSG), established in 2014, is an organization that aims to improve radiation oncology education by developing and disseminating education innovations. The organization identifies areas of curriculum development in radiation oncology. Since 2018, ROECSG has hosted a small (<500 participants) annual one-day symposium.

Due to the COVID-19 pandemic, the 2020 ROECSG Symposium was hosted virtually. We hypothesize that there are advantages and disadvantages to the virtual format. Herein, we summarize post-symposium evaluations to identify successful aspects of the meeting and areas of improvement, thus suggesting a framework for future virtual scientific meetings. Lessons learned from this study have broad implications for all scientific meetings.

Methods

The study was determined exempt by the The University of Chicago Institutional Review Board. The symposium was comprised of multiple sessions with breakouts (Table 1) held over Zoom (Zoom Video Communications Inc, San Jose, CA, USA). A post-symposium survey was developed in Research Electronic Data Capture (REDCap) and distributed electronically to all registrants (2,3). The survey assessed attendee demographics, participation, and experience/preferences. The survey used a 5-point Likert-type scale (“not at all” to “extremely”) to assess attendee experience and a 3-point Likert-type scale (“virtual symposium is better”, “neutral”, “in-person symposium is better”) to assess platform preference for future symposia. Analysis of attendee preferences for in-person, virtual, or no preference was modified from the Massachusetts General Hospital telehealth survey (4). Full symposium details and the survey instrument are provided in the Supplemental Material 1 and 2, respectively. All survey questions were optional for respondents and responses were analyzed when present, with denominators adjusted accordingly.

Table 1.

Outline of 2020 ROECSG virtual symposium

Session	Description
Coffee and Asynchronous Oral Presentations	- One-hour of dedicated networking and time to view asynchronous oral presentations
Orientation	- Five-minute presentation - Brief description of Zoom meeting logistics and agenda
Welcome Remarks and Introductions	- Five-minute presentation
Report on the First Two ROECSG Annual Symposia	- Five-minute presentation
Breakout rooms	- Small group sessions comprised of up to six randomized participants - These rooms did not include a moderator or noted points of discussion - Six breakout rooms were scattered throughout the oral sessions
Oral session 1	- “Tele-Education- COVID and Beyond” - Seven oral presentations and two breakout sessions - Duration: 90 minutes
Lunch breakout session	- One-hour scheduled lunch break - Optional 15-minute breakout sessions
Updates from professional organizations	- Reports from the Association of Residents in Radiation Oncology (ARRO) and the Association of Directors of Radiation Oncology Programs (ADROP) - Two, five-minute presentations
Oral session 2	- “Live Education Post-COVID” - Seven oral presentations and two breakout sessions - Duration: 90 minutes
Keynote address	- “Professional Identity Formation: The Case for a Shift in Focus from the Professional to the Profession“ - 30-minute presentation
Keynote Discussion	- Ten-minute breakout room to discuss Keynote address
Oral session 3	- “The Profession” - Six oral presentations and two breakout sessions - Duration: 90 minutes
Closing remarks	- Brief conclusion of meeting

Symplur LLC (Pasadena, CA) was used to generate analytics for the ROECSG Twitter (Twitter Inc., San Francisco, CA) account as part of the Healthcare Hashtag Project (5), with data collected for the week of the symposium (May 13–18, 2020). Data pertaining to the hashtag ROECSG (#ROECSG) were collected and assessed. Participant time spent in the meeting was collected from Zoom, with unique participants identified by name and email address. YouTube analytics for the asynchronous presentations were collected for May 12, 2020 to May 16, 2020.

Statistical Analysis

For variables assessed using 5-point Likert-type scales, the median and interquartile range (IQR) are reported. Mann-Whitney U and Fisher’s exact tests were used to compare the length of the symposium attended (entire versus partial) and ordinal data collected from attendee assessments and categorical attendee characteristics, respectively. A univariate logistic regression was performed to determine associations between attendee characteristics and preference for a virtual symposium compared to no preference/in-person preference combined. Statistical calculations were conducted in R (The R Foundation, Vienna, Austria) using a p-value <0.05 for significance.

Thematic qualitative analysis was done to identify common themes within the free-response survey data. This was done in multiple analytical stages by two coders (XX, XX) (6). During the first stage, coders familiarized themselves with the data. In the second stage, each free-response comment was openly coded and then grouped inductively based on identified repeated concepts. Repeated concepts were then condensed into relevant themes. Themes were discussed among the coders and further condensed. Preliminary themes were categorized as subthemes while the condensed themes were categorized as overarching themes with relevant subthemes.

Results

Overall attendee characteristics

Overall, based on Zoom data, 238 unique participants attended the symposium. The post-symposium survey was sent to all 286 registrants, yielding a 67% (n=191) registrant response rate and a 73% (n=174) attendee response rate. Seventeen respondents (9%) reported that they did not attend the symposium due to other obligations or time zone differences. These respondents were excluded from subsequent analysis. Attendee characteristics are reported in Table 2. Respondents reported being involved in graduate medical education (76%, n=132) and/or undergraduate medical education (54%, n=94). Only 26% (n=45) of respondents had attended a previous ROECSG symposium.

Table 2.

Overall attendee characteristics

	Overall attendees (n=174)
Characteristic
Gender, n (%)
Female	89 (51%)
Male	77 (44%)
Other	2 (1%)
No answer	6 (3%)
Race, n (%)
White	88 (51%)
Asian	38 (22%)
Hispanic or Latino	16 (9%)
Black or African American	7 (4%)
Other	12 (7%)
No answer	13 (7%)
Professional status, n (%)
Resident	72 (41%)
PGY1	8
PGY2	16
PGY3	14
PGY4	19
PGY5	15
Attending	59 (34%)
Years out of residency
<5 years out of residency	24
5–10 years out of residency	12
11–15 years out of residency	3
16–20 years out of residency	20
Leadership positions
Clerkship director†	13
Associate program direct†	12
Program director†	22
Department chair†	1
Medical student	19 (11%)
M1	1
M2	4
M3	3
M4	11
Medical physicist	8 (5%)
Program coordinator	7 (4%)
Other	9 (5%)
Prior formal education training, n (%)
No	122 (70%)
Yes	52 (30%)
Location, n (%)
US	150 (86%)
Outside the US	24 (14%)

^†These roles are not mutually exclusive. Therefore, cumulative percentages add up to more than 100%.

Attendee participation

Overall, participants attended a median of 4 hours and 16 minutes (IQR 1 hour 38 minutes, 6 hour 48 minutes) of the 8 hour and 30 minute symposium. Eighty-two (47%) respondents self-reported that they were present for the entire symposium. The most commonly cited reasons for not attending the entire symposium were clinical (n=57, 62%) and personal (n=38, 41%) obligations. Professional status and prior formal training in medical education were not associated with length of attendance (Table 3).

Table 3.

Factors associated with amount of attendance

	Entire Attendance (n=82)	Partial attendance (n=92)	p-value
Factor
Professional status†, n (%)			0.18
Attending	23 (39%)	36 (61%)
Resident	34 (47%)	38 (53%)
Medical student	12 (63%)	7 (37%)
Prior formal education training, n (%)			0.18
No	53 (43%)	69 (57%)
Yes	29 (56%)	23 (44%)
Level of distraction, median (IQR)	2 (2–3)	3 (2–3)	<0.01
Equivalent to in-person*, median (IQR)	4 (3–5)	3 (3–4)	0.01
Opportunity to network, median (IQR)	5 (4–5)	5 (4–5)	0.06

^†Other professional statuses were excluded from analysis.

^*Question asked “Was the virtual symposium equivalent to an-in person symposium?” on a 1–5 scale (“not at all” to “extremely”)

Attendee experience and preferences

Overall, the symposium was rated as “extremely” well-organized by 86% (150/174) of respondents. Respondents rated the virtual symposium as “quite” equivalent to an in-person symposium (median 4 [IQR 3–4]). Respondents who attended the entire symposium were more likely to rate it equivalent to an in-person symposium than those who attended portions of the symposium (p=0.01) and they rated themselves as significantly less distracted (p<0.01; Table 3).

In the future, 45% (n=78) of respondents reported that they would prefer a virtual ROECSG symposium, while 30% percent (n=52) would prefer an in-person symposium, and 25% (n=44) had no preference. The most commonly cited reasons for preferring a virtual meeting were accessibility (n=20, 26%), cost (n=15, 19%), and convenience (n=13, 17%). Respondents who had not attended a prior ROECSG symposium were more likely to indicate a preference for a virtual symposium (p=0.03; Table 4).

Table 4.

Factors associated with preference for a virtual symposium (vs. in-person/no preference)

Factor	Univariate analysis OR (95% CI)	p-value
Gender
Female
Male	0.69 (0.37–1.27)	0.24
Amount of symposium logged on for
Entire
Partial	1.29 (0.71–2.37)	0.40
Live within the United States
No
Yes	0.43 (0.17–1.04)	0.07
Professional level
Attending
Resident	0.80 (0.40–1.62)	0.54
Medical student	0.59 (0.21–1.57)	0.30
Medical physicist	3.32 (0.70–23.96)	0.16
Program coordinator	1.48 (0.30–8.04)	0.63
Prior symposium attendance
No
Yes	0.46 (0.22–0.93)	0.03

On evaluation of preferred format by activity, respondents reported a preference for a virtual format to review medical education scholarship (n=78, 45%) and an in-person format for networking (n=103, 59%), collaborating (n=88, 51%) and reducing level of distraction (n=109, 63%; Figure 1).

Figure 1.

Preferred symposium platform (in-person vs. virtual) according to symposium component

Qualitative findings

Four themes were identified upon qualitative analysis. Themes and number of comments identified within each theme were as follows: 1) Advantages to the virtual format (n=141), 2) Disadvantages to the virtual format (n=105), 3) Suggestions for improvement (n=82), and 4) ROECSG-specific feedback (n=105). Themes and subthemes are outlined in Figure 2.

Figure 2.

Themes and subthemes identified from qualitative survey responses

Advantages of the virtual format included accessibility (n=17) and convenience (n=14), more avenues for discussion such as the chat function (n=10), pre-recorded presentations that could be viewed any time and covered a range of topics (n=24), and networking (n=59) with an expanded ability to network due to the random assignment of breakout session attendees (n=17). Disadvantages of the virtual format included distraction due to the chat function, which could take attention away from the individual presenting (n=7) and other technical issues (n=29) which prevented participation in breakout sessions and viewing of oral presentations, distraction by personal and clinical obligations (n=11) as well as fragmented attendance as a result of competing obligations (n=19), and awkward conversations in breakout sessions (n=12). Respondents also noted that virtual communication did not match in-person networking (n=20). Suggestions such as longer breakout sessions (n=24) and questions for discussion with moderators present during sessions (n=25) were made to enhance productivity of the virtual format. Attendees would have also liked more break time (n=9) and autonomy with regards to who they spoke to in breakout sessions (n=14). Responses that fell under the theme of ROECSG-specific feedback noted the organization of the symposium (n=16), the exchange of scholarly ideas (n=62) or educational resources (n=18), and camaraderie among attendees (n=9).

Social media analytics

Nearly one-third (29%) of respondents learned about the 2020 ROECSG Symposium via Twitter. On the day of the symposium, the ROECSG hashtag (#ROECSG) had 408 tweets, garnering 432,504 impressions. From May 13, 2020 to May 18, 2020, #ROECSG appeared in 619 tweets from 135 unique Twitter users, with a total of 781,763 impressions. From May 12, 2020 to May 16, 2020, 17 asynchronous presentations uploaded to YouTube received a total of 1,071 views. Views per video by day online ranged from 8 to 62, with a median 16 views per video per day.

Discussion

These data demonstrate successful implementation of a virtual format for the ROECSG Annual Symposium and provide a framework whereby similar small (<500 participants) scientific meetings might be conducted effectively. Advantages of the virtual platform included convenience and accessibility, while disadvantages included limited quality networking opportunities.

In addition to respondents reporting accessibility as a reason for the virtual platform preference, attendance increased dramatically from 76 in 2019 to 238 in 2020, with attendees representing 11 countries up from five the year prior (7). By reducing the time away from personal obligations and monetary resources needed to attend, these findings may reflect the potential for virtual meetings to promote inclusivity (8).

Conversely, virtual conferences come with the challenges of attendee distractibility and multi-tasking. Evidence suggests people multi-task far less effectively than they realize (9), which may reduce what is gained from meetings held remotely. Specifically, the chat function was identified as potentially problematic, as messages pertaining to things other than the current presentation would distract participant. In the future, techniques to reduce questions and comments in the online chat function may be warranted, such as chat moderators or encouraging the private chat function. Moreover, limiting competing obligations, as one would for an in-person meeting, may reduce distraction and result in a more positive experience. A similar survey found that employers were willing to grant time off to attend virtual meetings (10). However, the virtual platform allows participants to attend who may not be able to officially take time off work, even if attendance is fragmented.

While overall the in-person format was preferred for purposes of networking, there were noted advantages and disadvantages with regards to networking. Some respondents felt that the virtual format actually expanded their ability to network due to the random breakout room assignments, while others felt the breakout room format limited their ability to approach specific individuals and that the virtual experience did not match in-person networking. Similar challenges have been previously noted with the proposed solution of offering attendees both random assignment breakout sessions and “meet-the-expert” sessions (11).

Social media is a significant avenue for dissemination and discussion of information and may be more valuable in the virtual setting. Twitter was successfully used to promote the ROECSG symposium, as evidenced by the nearly 30% of respondents who learned about the symposium from Twitter. Evidence supports the ability of social media, especially Twitter, to enhance research exposure and subsequent publication citations (12) and should be recognized for its increasing potential to propagate scholarship and organizational efforts such as the ROECSG Annual Symposium.

This study has several limitations. The findings likely reflect the ideals of the ROECSG symposium and attendees, which is small in size and focused on education (7) within a specialized field of medicine. Therefore, findings may not be generalizable to other academic disciplines. Additionally, although the attendee response rate was acceptable (73%), any survey non-response introduces potential for response bias. Our survey was also not conducted by a third-party, potentially resulting in conformity bias. Additionally, participants may have overestimated the amount of time they participated in the symposium due to social desirability bias. We did not collect information on participant time zones, which may have affected their participation, and we did not query the format of a one-day symposium compared to fewer live presentations spread over 2 days. While collection of Twitter analytics allowed for an assessment of engagement with the material and other participants, future virtual conferences might consider also tracking YouTube views by day to better understand asynchronous participation, and even compare likelihood of social media engagement based on the virtual setting (live oral versus asynchronous video). Lastly, this meeting did not use participant leaderboards or awards to be able to evaluate the appropriateness of these potential elements of a virtual conference.

Conclusions

The 2020 ROECSG Symposium was successfully held virtually and suggests a framework for future scientific meetings to facilitate both presentation of scholarship and professional networking. Once travel restrictions are lifted, meeting organizers could consider using a hybrid meeting format to combine the accessibility of a virtual symposium while still providing in-person networking opportunities. However, a hybrid in-person/virtual format may lessen quality experiences by reducing attendee interactions between formats. Combining the best features of both formats would be challenging. Social media can be leveraged to advertise scientific meetings, promote and disseminate scholarship, and circulate resources.