Impact of transitioning to an online course – A report from the ESTRO gyn teaching course

Highlights

• •Converting the ESTRO gyn teaching course to an online version did not affect participant numbers.
• •Participant satisfaction and engagement remained high despite less direct contact time with faculty.
• •The number of participants submitting contouring and dose planning homework was higher than for live courses.
• •Viewing of pre-course lecture content was variable which may reflect self-perceived learning needs based on different levels of experience.

Abstract

Introduction

In 2020, the ESTRO course on image-guided radiotherapy and chemotherapy in gynaecological cancer was converted into an online version due to the COVID-19 pandemic. This paper describes the change process and evaluates the impact on participants compared with previous live courses.

Methods and materials

The 2019 live course contained 41 h of educational content, comprising 33 h of synchronous activities (lectures, interactive activities, videos) and 8 h of homework (contouring, dose planning). For the online course, the lectures were provided as pre-course material (11 mandatory, 22 optional). Contouring/dose planning homework was unchanged. The synchronous sessions were reconfigured as six 2-hour webinars (total educational content ~38 h).

Participant numbers/characteristics, engagement and satisfaction for six live courses and the online course were compared.

Results

Participant numbers for the online and live courses were similar (90 vs. mean 96). There were more participants from outside Europe (28% vs. mean 18%) and more non-doctors (47% vs. mean 33%). Proportion of participants responding to the pre-course questionnaire was similar (77% vs. mean 78%) but post-course questionnaire response was lower (62% vs. mean 92%).

43% participants viewed ≥75% of mandatory lectures before the webinars. 86% viewed the optional lectures. Submissions of contouring and dose planning homework was higher (contouring 77%–90% vs. 56%–69%, dose planning 74%–89% vs. 29%–57%).

96% (47/49) participants rated the online course as Excellent (43%) or Good (53%). Overall satisfaction was similar (4.4 vs. mean 4.6).

Conclusion

Participant satisfaction and engagement with the online course remained high despite less contact time with faculty.

Introduction

Over the past two decades, radiotherapy for cervix cancer has undergone tremendous technological change as chronicled in a publication in 2018 [1]. Magnetic resonance imaging (MRI)-based image-guided adaptive brachytherapy (IGABT) is now widely accepted as the new gold standard for cervical cancer brachytherapy. The excellent outcome of MRI-based IGABT in the prospective EMBRACE study was presented at ESTRO 2020 and has been accepted for publication in Lancet Oncology [2]. At 5 years, the local control was >90% across all stages (IB1 98%, IB2 92%, IIB 91%, IIIB 92%, IVA 91%) while severe morbidity was limited (~6%).

While the impact of IGABT on local control is unprecedented, there appears to be room for improvement of nodal control in high-risk node-positive patients through the use of advanced external beam radiotherapy (EBRT) techniques in conjunction with concurrent chemotherapy [1], [3]. For low-risk patients, there appears to be opportunity to de-escalate treatment safely to reduce morbidity [1]. These strategies are currently being investigated in the EMBRACE-II study of intensity-modulated chemo-radiotherapy (IMRT) and MRI-based IGABT for cervix cancer [1].

The implementation of IGABT and advanced EBRT for cervix cancer necessitates several conceptual and technical innovations which imply new skills and significant change in practice for staff [4]. For stage III/IVA disease, local control rates with IGABT appear to vary between centres (73–86%) in retrospective reports [5], [6], [7], [8], [9], [10], [11], [12], [13] which suggests that effective education and training are critical for optimum results. The challenges of education and training for IGABT and advanced EBRT for cervix cancer have been described in recent publications [14], [15], [16].

In 2004, ESTRO held its first teaching course on MRI-based IGABT for cervix cancer in Vienna, Austria to support the multi-disciplinary dissemination of the emerging technique. The course was subsequently expanded in 2011 to include education and training on advanced EBRT techniques. Widely regarded as one of the most successful courses on the ESTRO school calendar, by 2019 there had been 22 editions of the ESTRO gyn course (including 7 international editions) attended by >2400 participants.

Much of the success of the course can be attributed to a dynamic faculty determined to create a continually evolving programme incorporating improved educational strategies to better address the learning needs of the participants (Fig. 1). This philosophy was laid down at the outset by the founding course directors, Prof Richard Pötter from Medical University of Vienna, Austria and Dr Christine Haie-Meder from Institute Gustave Roussy, France, and has been continued by current course directors, Prof Kari Tanderup from Aarhus University Hospital, Denmark and Prof Remi Nout from Erasmus MC Cancer Institute, Rotterdam, the Netherlands.

Fig. 1.

Timeline of changes to the ESTRO gyn teaching course.

The 2020 edition of the course was due to be held in Lisbon, Portugal in September 2020. Due to the COVID-19 pandemic, the course faculty took the decision in April 2020 to convert this course into an online version. The entire course programme was reviewed and restructured into a format deemed more suitable for online teaching. This paper describes the change process and evaluates the impact on participants compared with previous live courses. The challenges encountered and the lessons learnt are discussed.

Methods and materials

In the beginning, the ESTRO gyn teaching course on IGABT was conducted over 3 consecutive days - this was increased to 5 days in 2011. The programme of the last live course held in Cluj, Romania in October 2019 was estimated to contain 41 h of educational content, comprising 33 h (80%) of synchronous activities (where participants and faculty are present together and can engage in real-time interaction) and 8 h (20%) of homework (Table 1). The homework for radiation/clinical oncologists (RO/CO) involved contouring on one EBRT case and one IGABT case using the ESTRO FALCON/EduCase platform [17]. For medical physicists (MP), the homework involved downloading DICOM datasets for one EBRT case and one IGABT case for dose planning in their own clinical software and submitting their results in an Excel spreadsheet, a process that was successfully used for accreditation in the EMBRACE-II study [14].

Table 1.

Summary of programme content (hours) for 2019 Cluj and 2020 online courses.

	Cluj course 2019	Online course 2020
Homework
Contouring/dose planning	8	8
Mandatory lectures		8
Optional lectures		8
Clinical videos		2
Total	8	26
Synchronous sessions
Lectures	21	0
Kick-off webinar		2
Interactive sessions	11	10
Clinical videos	1
Total	33	12
Course Total	41	38
Parallel sessions	8	2
Experienced track	5
Non-cervix	3

Approximately 64% of the synchronous sessions (21/33 h) were dedicated to lectures while 33% (11/33 h) were interactive activities including hands-on practical workshops. The remaining one hour was allocated for viewing of clinical videos of IGABT implant procedures. The majority of the interactive activities (8/11 h) were conducted as parallel sessions with different activities for RO/CO and MP/radiation therapists (RTT). There was also a separate “experienced track” (5 h) where multi-disciplinary teams from centres with more experience of IGABT could present their workflow and clinical cases for peer discussion. Approximately 9% of the synchronous sessions (3/33 h) were devoted to discussion of other gynaecological (non-cervix) cancers. The fee for the entire course was €600 for ESTRO members.

For the online course, the live course programme was reviewed by the faculty comprising 7 RO/CO and 4 MP. Three of the faculty members were based outside Europe (two India, one Canada). A summary of the programme content is shown in Table 1. All lectures were provided as pre-course material to be viewed by participants in their own time. Of the 35 lectures presented in the Cluj course, 11 were identified as mandatory pre-requisites for all participants while another 2 were mandatory for RO/CO and MP/RTTs respectively - these lectures were specially recorded by the faculty for the online course with an estimated viewing time of 8 h. The remaining 22 lectures (which had previously been recorded in Cluj) and one video were provided as optional material; it was estimated that participants would spend 8 h on this content. The contouring homework for RO/CO and dose planning homework for MP were unchanged in format and duration (8 h each).

The synchronous sessions were reconfigured as six 2-hour webinars held at 1500 CET from 24 September 2020 to 5 November 2020 using the Zoom videoconferencing platform. Participants were required to register for each webinar individually as well as for the course as a whole. The first kick-off webinar was used to introduce the course objectives and the interactive tools that would be used in the subsequent webinars. The format of the subsequent weekly webinars was entirely interactive and included feedback on homework, case-based discussions, quizzes using online polling software (TurningPoint), and rapid contouring exercises using a low-fidelity contouring tool (Mini-Contour) [18]. Participants were encouraged to ask questions using the Zoom chat function. Five of the webinars were designed for a multi-disciplinary audience while one was organised as parallel sessions. Two of the webinars focussed on contouring for EBRT and IGABT, respectively, while two focussed on dose planning. There was no separate experienced track. A copy of the online course programme is included in the Appendix. The total estimated educational content was 38 h. The fee for the entire course was €400 for ESTRO members.

For all the courses, participants were asked to complete a pre-course questionnaire about their experience (since 2015) and a post-course feedback questionnaire (since 2012). For the online course, participants were also asked to provide feedback (comprising one rating score and a comment box) after each webinar via a SurveyMonkey link or QR code. From 2018, all course content has been hosted on a Moodle open-source learning management system accessed through the ESTRO website. (The Moodle course content for the live courses was limited to instructional information only.)

Participant data for the six live European courses held between 2013 and 2019 were analysed and compared to the 2020 online course using descriptive statistics (percentage, mean, range). (Data for the earlier courses were no longer accessible.) The following factors were evaluated:

• •Participant numbers and characteristics (country and profession)
• •Participant engagement (pre- and post-course questionnaires, homework submission, Moodle activity logs)
• •Participant satisfaction (post-course questionnaire)

Results

Fig. 2a, Fig. 2b, Fig. 2c shows the participant numbers and characteristics for the various courses. The number of participants registered for the online course (90) was similar to the previous live courses (mean = 96, range: 82–128). The proportion of participants from outside Europe for the online course (28%) was higher than for the live courses (mean = 18%, range: 12%–24%). The proportion of participants who were not RO/CO also appeared higher (47% vs. mean of 33%, range: 23%–40%).

Fig. 2a.

Number of participants.

Fig. 2b.

Participants by country of origin.

Fig. 2c.

Participants by profession.

Fig. 3a, Fig. 3b, Fig. 3c shows participant engagement with various activities in the courses. The proportion of participants completing the contouring (EBRT 90%, IGABT 77%) and dose planning (EBRT 74%, IGABT 89%) homework in the online course was considerably higher than in the previous 3 live courses in 2017–2019 (contouring: EBRT 56%–60%, IGABT 65%–69%, Fig. 3a; dose planning: EBRT 29%–44%, IGABT 38%–57%%; Fig. 3b). The proportion responding to the pre-course questionnaire was similar (online: 77%, live: mean = 78%, range: 66% −87%, Fig. 3c). However, the response to the post-course questionnaire was lower (online: 62% vs. mean of 92% for 4 live courses, range: 90%–96%, data for 2 courses were not available, Fig. 3c). The feedback response after each webinar was also limited (mean 20%, range; 0–53%).

Fig. 3a.

Proportion of radiation/clinical oncologists submitting contouring homework.

Fig. 3b.

Proportion of physicists/dosimetrists submitting dose planning homework.

Fig. 3c.

Response to pre- and post-course questionnaires.

The attendance at each webinar of the online course was analysed separately. Overall, 37 of the 90 (41%) registered participants attended all 6 webinars while a further 25 (28%) attended 5 sessions. Ten (11%) registrants did not appear to attend any webinars and a further 4 (4%) attended only one session. Six of these were RTT while 3 were from New Zealand and one from the USA. Table 2 shows the attendance at each online webinar by profession. In general, there was no difference in multi-disciplinary attendance at the joint sessions apart from the EBRT treatment planning session which was attended by 89% of MP and 75% of RO/CO. One MP chose to attend the parallel session for RO/CO while 4 participants (one RO/CO, one MP and 2 RTT) attended both parallel sessions simultaneously.

Table 2.

Attendance at webinar by profession.

Webinar		Total (n = 90)		RO/CO (n = 48)						MP (n = 27)						RTT (n = 12)						Other (n = 3)
		Attended		Yes		No		DNR		Yes		No		DNR		Yes		No		DNR		Yes
1	Kick-off	71	79%	40	83%	3	6%	5	10%	22	81%	3	11%	2	7%	6	50%	3	25%	3	25%	3	100%
2	EBRT Contouring and Image Guidance	69	77%	39	81%	4	8%	5	10%	23	85%	1	4%	3	11%	4	33%	4	33%	4	33%	3	100%
3	EBRT Treatment Planning	68	76%	36	75%	6	13%	6	13%	24	89%	1	4%	2	7%	5	42%	4	33%	3	25%	3	100%
4	Brachytherapy Contouring	72	80%	41	85%	2	4%	5	10%	22	81%	3	11%	2	7%	6	50%	2	17%	4	33%	3	100%
5a	Brachytherapy clinical aspects (RO/CO)	40	83%	36	75%	2	4%	10	21%	2*	7%					3**	25%					2	67%
5b	EBRT and brachytherapy technical aspects (MP + RTT)	25	64%	1*	2%					20	74%	4	15%	3	11%	4	33%	6	50%	2	17%	1	33%
6	Brachytherapy Treatment Planning	54	60%	31	65%	9	19%	8	17%	17	63%	6	22%	4	15%	4	33%	4	33%	4	33%	2	67%

DNR = did not register.

Percentages are expressed as a proportion of total relevant participants.

* One participant attended both parallel sessions.

** Two participants attended both parallel sessions.

The Moodle activity logs were interrogated to explore participant interaction with the online course lecture content. Overall, 19 of the 90 (21%) participants viewed all the mandatory lectures before the relevant webinar while a further 20 (22%) viewed at least 75% (≥9) of the lectures. 31% (191/610) of first views of lectures before webinars occurred in the preceding 2 days with a further 23% (139) within 7 days. 39 (43%) participants viewed the mandatory lectures they had missed after the webinars while 44 (49%) revisited lectures that they had previously viewed. Five (6%) participants only viewed any mandatory lectures after the webinars while 10 (11%) did not view any lectures at all. 6 of the 10 participants who did not attend any webinars viewed nearly all of the mandatory lectures. 77 (86%) participants viewed the optional lectures; the mean number of lectures viewed was 7 (range: 1–22).

96% (47/49) participants rated the online course as Excellent (43%) or Good (53%). The overall satisfaction was 4.4 out of 5 which is similar to the live courses (mean = 4.6, range: 4.5–4.6 for 4 live courses). Two participants (one of whom had attended the live course in 2018) were asked to provide their reflections on the course [19] – both mentioned initial scepticism/apprehension about the online format but their expectations were met/exceeded.

Participants were asked what they liked most and disliked about the course – word clouds of their responses are shown in Fig. 4a, Fig. 4b. Participants particularly liked the pre-recorded lectures, the interactive nature of the webinars, the practical exercises and individualised feedback, and the use of the polling software during webinars. Negative comments mostly centred around internet connection issues and sound quality with some of the live sessions and pre-recorded lectures. 37 participants answered the question on whether they would register for another online course - of these, 33 (89%) stated yes while 4 (11%) stated they would prefer a live course.

Fig. 4a.

Word cloud of “liked most” comments in post-course questionnaire.

Fig. 4b.

Word cloud of “dislike” comments in post-course questionnaire.

Discussion

This analysis of the first ESTRO online gyn course has revealed some encouraging findings. Participant satisfaction remained high despite the reduction in direct contact time with the faculty. The number of participants from outside Europe increased despite the different time zones as did the proportion of participants who are not RO/CO even though the fee difference between the online and live courses was modest - this is probably a reflection of the considerable ancillary costs often associated with live courses such as travel and accommodation, a comment made by several participants in the post-course questionnaire.

While the pros and cons of online teaching are well-documented [20], there remains concern that it is less effective than face-to-face teaching. In 2009, the US Department of Education conducted a meta-analysis of random-assignment or controlled quasi-experimental studies which compared online with face-to-face learning across all age groups and subjects, including healthcare [21]. The report concluded that students who took all or part of their class online performed better, on average, than those taking the same course through traditional face-to-face instruction.

The shift of lectures to a pre-course activity is typical of a “flipped classroom”, a pedagogical approach which premises that acquisition of information can be effectively undertaken by individuals outside the classroom allowing in-class activities to focus on learning which requires high-level cognition such as application of concepts. A meta-analysis by Hew and Lo of 28 studies [22] suggested that the flipped classroom approach in health professions education yields a significant improvement in student learning compared with traditional teaching methods. In this course, several participants commented on the value of being able to study pre-recorded lectures at their own pace and use the webinars to discuss common errors and “cement learning”.

Although <50% of our participants viewed >75% of the mandatory lectures before the webinars, this may reflect self-perceived learning needs based on their different levels of experience. In the pre-course questionnaire, 61% (23/38) of RO/CO had treated >10 cervix patients with IMRT and 39% (13/33) with IGABT. The corresponding figures for MP were 31% (10/32) and 42% (13/31) respectively. Encouragingly, 43% of participants viewed the mandatory lectures they had missed after the webinars while 49% revisited lectures that they had already viewed, suggesting that the webinars may have helped to highlight gaps in knowledge.

>50% of first views of lectures occurred in the preceding 7 days before each webinar. This may be an opportunity to implement “Just-in-time teaching” (JiTT), a pedagogical strategy which uses pre-class assignments to increase the effectiveness of flipped classrooms [23]. The pre-class assignments usually took the form of short quizzes on the content to be covered in the class and were usually completed 1–24 h beforehand. In a report from the EMBRACE-II study [15], quizzes were identified as a potentially quick way of highlighting key aspects of the protocol and identifying areas of difficulty. The use of pre-class quizzes as part of a JiTT strategy for our course may therefore motivate more participants to view the mandatory lectures before the webinars if gaps in their knowledge are highlighted.

In contrast to the lectures, 90% and 77% of RO/CO completed the contouring homework, and 74% and 89% of MP the dose planning homework, for EBRT and IGABT respectively. This is perhaps not surprising as these skills are widely recognised as key competencies for optimal IMRT and IGABT. Moreover, these exercises provide a rare opportunity for benchmarking and feedback - this was much appreciated by participants in the post-course questionnaire. The reasons why submissions for both RO/CO and MP were higher than for previous live courses are unknown; one plausible explanation is that more participants were able to meet the deadlines because the assignments were spaced out.

Overall, the multi-disciplinary nature of the audience was maintained in the online course even though some webinars were more targeted at RO/CO (i.e. contouring) while others were more targeted at MP (i.e. dose planning). 14 registrants (15%) only attended 1 or 0 webinar. While the exact reasons have not been ascertained, 3 of them were from New Zealand and one from USA and time zone differences may be a factor. Possible solutions include scheduling the webinars on a Friday or at the weekend or holding duplicate webinars for different time zones although the latter would increase the workload for the faculty. It was also noteworthy from the post-course questionnaire that some participants were attempting to fit the course around their clinical duties. This highlights the importance of obtaining formal study leave for the online course as for the live course particularly as the number of educational hours for both courses is similar.

The response to the post-course questionnaire was lower than for previous courses. Questionnaire fatigue could be a contributory factor as feedback was also requested after each webinar. Incentives to encourage participation may be required such as mandatory completion of the post-course questionnaire prior to issue of course attendance certificates.

Our analysis has some limitations. Data for some of the live courses were incomplete. For the online course, access of educational content and attendance at webinars were calculated from the number of logins which did not account for simultaneous viewing by participants from a single computer. It was also not possible to establish how much time was spent viewing the pre-recorded lectures, merely the number of views. The lower response rate to the post-course questionnaire may not have captured all the issues encountered by participants. For all the courses, it was not possible to correlate participant satisfaction and/or engagement with learning outcomes.

Nevertheless, useful insights have been gained. Due to the ongoing pandemic situation, the 2021 edition of ESTRO gyn teaching course will also be held online. A number of changes will be implemented in response to participant comments - the webinars will be increased to 2.5 h to allow more time for discussion, verbal discussion will be enabled in addition to the Zoom chat function, and the parallel sessions will be held on different days to allow more participants to attend both sessions if desired. Other strategies to increase participant engagement and improve the learning quality will be considered. Discussions are also underway to explore how to integrate the best strategies from the online course into future live courses. It is likely that both live and online versions of the course will be maintained in order to better meet the diverse needs of healthcare professionals requiring education and training in advanced radiotherapy for cervical cancer.

Declaration of Competing Interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Kari Tanderup reports grants from Danish Cancer Society, grants from Varian Medical Systems, grants from Elekta, outside the submitted work. Remi Nout reports grants from Dutch Cancer Society, grants from Dutch Research Council, grants from Elekta, grants from Varian, grants from Accuray, outside the submitted work.

Appendix A. Supplementary data

The following are the Supplementary data to this article: