Abstract
BACKGROUND
The Anatomy and Radiology Contouring (ARC) Bootcamp was a face-to-face (F2F) course designed to ensure radiation oncology residents were equipped with the knowledge and skillset to use radiation therapy techniques properly. The ARC Bootcamp was proven to be a useful educational intervention for improving learners’ knowledge of anatomy and radiology and contouring ability. An online version of the course was created to increase accessibility to the ARC Bootcamp and provide a flexible, self-paced learning environment. This study aimed to describe the instructional design model used to create the online offering and report participants’ motivation to enroll in the course and the online ARC Bootcamp's strengths and improvement areas.
METHODS
The creation of the online course followed the analysis, design, development, implementation, and evaluation (ADDIE) framework. The course was structured in a linear progression of locked modules consisting of radiology and contouring lectures, anatomy labs, and integrated evaluations.
RESULTS
The online course launched on the platform Teachable in November 2019, and by January 2021, 140 participants had enrolled in the course, with 27 participants completing all course components. The course had broad geographic participation with learners from 19 different countries. Of the participants enrolled, 34% were female, and most were radiation oncology residents (56%), followed by other programs (24%), such as medical physics residents or medical students. The primary motivator for participants to enroll was to improve their subject knowledge/skill (44%). The most common strength identified by participants was the course's quality (41%), and the most common improvement area was to incorporate more course content (41%).
CONCLUSIONS
The creation of the online ARC Bootcamp using the ADDIE framework was feasible. The course is accessible to diverse geographic regions and programs and provides a flexible learning environment; however, the course completion rate was low. Participants’ feedback regarding their experiences will inform future offerings of the online course.
Keywords: online learning, radiation oncology, development, implementation, evaluation, anatomy education, students’ feedback
Background and Need for the ARC Bootcamp
Overview
This paper describes the creation of an online Anatomy and Radiology Contouring (ARC) Bootcamp designed after a face-to-face (F2F) version of the course. An online version was created to increase accessibility to the ARC Bootcamp and provide a flexible learning environment. The initial F2F ARC Bootcamp was created in 2013 to supplement radiation oncology (RO) education and ensure radiation oncologists were equipped with the knowledge and skillsets to use appropriately modern radiation therapy techniques.1 The F2F ARC Bootcamp was proven to be a useful educational intervention that allowed instructor–learner interactions and provided a kinesthetic learning environment1; however, it was costly and not easily accessible for learners located outside of Ontario, Canada. The online model offers flexible self-paced and independent learning, encourages self-assessment using evaluations that provide instant formative feedback, and is a more affordable option for learners.2 The purpose of this study was to investigate participant motivation to enroll in the online ARC Bootcamp and their feedback regarding their experience to identify the course's strengths and areas for improvement to inform future offerings. Future studies will evaluate the impact of the online Bootcamp.
RO education
RO has been impacted by significant technological advancements leading to a new standard of patient care.3 Techniques have shifted from 2-dimensional radiographs to methods that rely on 3-dimensional, multimodality imaging for target and organs-at-risk (OAR) delineation.4,6 For instance, the highly conformal technique, intensity-modulated radiation therapy, delivers an adequate dose to the target and minimizes toxicity to normal tissue and critical organs.4 This technique's success is based on accurate contouring of the target volume and OAR. Inaccurate delineation of these structures can lead to significant adverse clinical consequences and patient harm.4,6 Therefore, radiation oncologists must have foundational and comprehensive knowledge in anatomy7 and radiology to outline structures with precision and accuracy; however, instruction in these subjects tends to be minimal during RO training.8
F2F ARC bootcamp description
To address the lack of adaptations in RO postgraduate medical education curricula,8 a multidisciplinary team of physicians and educators designed and developed the ARC Bootcamp, a 3-day F2F educational intervention.1 It was designed for RO residents and medical physicists to gain additional integrated instruction in anatomy, radiology, and contouring for 5 major anatomic sites: head and neck, thorax, abdomen, male pelvis, and female pelvis. In 2011 to 2012, the ARC Bootcamp was piloted at a single institution, Western University,8,9 and the first iteration was launched in 2013 allowing participants from any institution to enroll and attend. Each iteration has incorporated modifications from participants’ feedback, which has ensured that the content is useful and relevant to learners.1
The ARC Bootcamp consisted of didactic radiology lectures led by diagnostic radiologists and radiation oncologists. The RO sessions were primarily case-based, and participants were encouraged to practice contouring in real time using the software EduCase on their personal devices. Small groups of participants interacted with anatomists to review anatomy concepts using cadaveric prosections tailored to oncological clinical correlations. The F2F ARC Bootcamp was proven to be a useful educational intervention for improving RO residents’ knowledge of anatomy and radiology and enhancing their confidence and accuracy in contouring.1
Goal of the online arc bootcamp
Acknowledging the importance of increasing access to the ARC Bootcamp and providing flexible and self-paced learning, an online version of the course was developed to offer similar content to the in-person offering. This study aims to describe the instructional design model used to create the ARC Bootcamp's online version and report learners’ feedback of their experience in the course. The study objectives are to identify the participants’ motivation to enroll in the course and identify the ARC Bootcamp's strengths and areas for improvement.
Creation of the Online ARC Bootcamp Using the ADDIE Model
Participants
All learners who enrolled in the online ARC Bootcamp and provided consent between November 2019 and January 2021 were included in the study. The research protocol was approved by the University of Western Ontario Non-Medical Research Ethics Board (Project ID: 114207).
Overview of the addie design model
The design of the online ARC Bootcamp followed the framework of the analysis, design, development, implementation, and evaluation (ADDIE) model, which includes 5 clear and defined phases: analysis, design, development, implementation, and evaluation. The analysis phase is when educators identify learners’ needs and determine what needs to be taught to accomplish the educational goals. In the design phase, the educators decide how to deliver the material to meet the learning needs effectively. The development phase includes the creation and assembly of the course content. In the implementation phase, the educators provide the instruction of material. Lastly, during the evaluation phase, the educators obtain feedback regarding the course and make any necessary modifications. Although the ADDIE model is described as consecutive phases, movement from one phase to the next does not have to be exclusively linear.10 Figure 1 provides an overview of how the ADDIE instructional design model was used to create the online ARC Bootcamp.
Figure 1.
Overview of how the ADDIE instructional design model was used as a framework to create the online ARC Bootcamp to provide a similar learning experience to the F2F offering and to continuously monitor the online course to inform course modifications. Abbreviations: ADDIE, analysis, design, development, implementation, evaluation; ARC, anatomy and radiology contouring; F2F, face-to-face; LOs, learning objectives.
Online ARC bootcamp analysis
Learners’ needs were identified by the multidisciplinary team based on F2F participants’ feedback from past iterations. This information was used to design and develop the online course's lectures and labs. Analysis of all anatomical structures discussed during the F2F anatomy sessions occurred to ensure similar content was taught during the online course. For instance, the F2F head and neck anatomy sessions taught the sites, nasal cavity and oral cavity, despite the anatomy of these sites not being included in the course manual. The anatomical structures of these sites were taught since several of these structures were identified on annotated radiologic images within the course manual. The online course formally developed anatomy lab videos for these sites. To maintain consistent course learning expectations for F2F and online participants, the F2F course learning objectives were used for the online version.
Online ARC bootcamp design
The multidisciplinary team transformed the F2F Bootcamp into an online version using the model previously described. The online course was organized into 11 locked modules, which were structured linearly. Module components unlocked after a period of 7 days regardless of whether prior modules were fully complete. The first module introduced learners to the course, describing the reason for the ARC Bootcamp, the course's goals, and what learners can expect to gain. Two modules were dedicated to preintervention and postintervention evaluations. The remaining 8 modules consisted of content related to the 5 major anatomic sites covered in the F2F offering and previously mentioned in the introduction. The number of modules per anatomic site correlated to the amount of content for each site (head and neck: 4 modules; thorax: 1 module; abdomen: 1 module; male/female pelvis: 2 modules). Each of the 8 content modules included radiology and contouring lectures, taught by a diagnostic radiologist or radiation oncologist, and anatomy labs led by an anatomist. A few of the modules included interactive activities such as a discussion forum, a quiz, and contouring exercises using the contouring software EduCase (RadOnc eLearning Center Inc., Jackson, WY, USA) (Supplemental Table 1). The learning activities, learning objectives, and evaluation methods were intentionally aligned using the course design approach, constructive alignment, which is used for formulating learning activities and evaluation tasks that directly address the intended learning objectives.11 Aligned courses lead to greater student learning, and it is more likely learners will achieve the course's goals and objectives.12
Online ARC bootcamp content development
An online course manual was developed based on the F2F ARC Bootcamp manual. The manual included 5 learning modules for each anatomic site. The modules addressed the learning objectives created by the multidisciplinary team.
The online radiology and contouring lectures were developed based on the F2F ARC Bootcamp lectures. The radiologists’ and radiation oncologists’ PowerPoint (PPT) slides used during the F2F Bootcamp were edited to focus on the essential concepts, which decreased the lecture length from 60 min to a more manageable 20 to 40 min. The screen capture software Camtasia (TechSmith Camtasia, Okemos, MI, USA) was used to record the lecturer's content and edit the videos. The online ARC Bootcamp consisted of 15 lectures, each focused on one of the significant anatomic sites (Supplemental Table 2). All 8 content modules included 1 lecture, with 4 of those modules including multiple lectures.
The anatomy labs were newly created to replace the in-person kinesthetic gross anatomy sessions. Instead of using an existing computer-based anatomy resource, such as 3-dimensional anatomy simulation, we created anatomy lab videos using the same cadaveric specimens used during the F2F course. The anatomy lab videos ranged from 10 to 15 min in length. Anatomists were recorded while they highlighted anatomical structures using cadaveric prosections and discussed spatial relationships. The anatomists annotated PPT slides that included cadaveric atlas images for structures that could not be visualized well with a cadaveric prosection. Using Camtasia's screen recorder function, the anatomist identified the necessary anatomical structures. The anatomy labs were edited using the software Camtasia. The ARC Bootcamp consisted of 12 anatomy labs, each focused on 1 of 5 anatomic sites (Supplemental Table 3). Six of the content modules included at least 1 anatomy lab, with 4 of those modules including multiple labs.
Three discussion forums were embedded in the course equally spaced apart, allowing participants to discuss concepts with their peers. A discussion forum was inserted following the head and neck content, thorax and abdomen content, and pelvis content. Two multiple-choice quizzes were embedded into the course to test the learner's knowledge of the content and provide instant feedback. Quiz 1 evaluated head and neck material, and quiz 2 evaluated thorax, abdomen, and pelvis material.
Online ARC bootcamp implementation
The online ARC Bootcamp was implemented using the course building platform Teachable (Teachable Inc.). Teachable was chosen because it allowed for multiple team members to develop sections of the course, granted an unlimited number of participants to register for the course, accepted international payments in different currencies, included built-in features, such as quizzes that could be integrated to provide instant feedback to learners, and was a secure platform that provided 24/7 monitoring. The online Bootcamp launched in November 2019. The online course cost $199 for 1 year of unlimited access.
Online ARC bootcamp evaluation
A preintervention and postintervention evaluation was embedded at the beginning and end of the ARC Bootcamp, respectively, and administered using Qualtrics (Qualtrics). The preintervention evaluation consisted of demographic questions, a self-confidence survey, an anatomy and radiology knowledge quiz, and a contouring test. The demographic questions identified learners’ characteristics and motivation to register for the ARC Bootcamp (Supplemental Table 4). The self-confidence survey asked learners to rate their self-confidence for concepts related to radiology, anatomy, and contouring using a 5-point Likert scale (1 = Not confident at all to 5 = Extremely confident) (Supplemental Table 5). The anatomy and radiology quiz consisted of 40 multiple choice questions to examine the course's impact on the learner's knowledge of anatomy and radiology (Supplemental Table 6). Learners were provided with 30 min to complete the quiz. The contouring test examined the ARC Bootcamp's impact on the learner's contouring accuracy. Using the software EduCase, learners had 30 min to contour 15 predetermined structures (Supplemental Table 7). Each knowledge quiz and contouring test question was mapped to a course learning objective and to the learning activity (ie, lecture or anatomy lab) that taught the concept.
The postintervention evaluation consisted of the same self-confidence survey, anatomy and radiology knowledge quiz, and contouring test and included a course satisfaction survey. The course satisfaction survey asked learners to provide feedback regarding their experience in the online ARC Bootcamp (Supplemental Table 8). Course feedback will be used to make modifications to future course offerings.
Statistical analysis
Qualitative preintervention and postintervention evaluation open-ended questions were coded independently by 2 researchers using NVivo software (QSR International Version 12; Melbourne, Australia). The first coder developed the codes, themes, and codebooks. Following the initial coding of the responses, the researchers redefined certain codes and reorganized themes. Using the new codebooks, researchers coded all participant responses. Using the coding data output from NVivo, the number of participants that reported each of the themes for participants’ enrollment motivation, the course's strengths, and the course's improvement areas were calculated based on the average between the 2 coders. Intraclass correlation coefficients (ICCs) were used to assess the interrater reliability of codes. The following ICC intervals were used to define the magnitude of reliability: poor (<0.40); fair (0.40-0.59); good (0.60-0.74); excellent (>0.74).13 Assessment of the relationship between independent groups and enrollment motivation was assessed with the Kruskal–Wallis test with an alpha () level of .05. This was followed by Mann–Whitney U tests with an alpha () level of .0167 (Bonferroni correction) to determine which relationships were significant, and P values subjected to the Bonferroni correction will be denoted as pbonf. Analysis was performed using SPSS (IBM SPSS Version 26).
Table 2.
Motivation to Enroll in the Course Reported as Average Number of Participants and Grouped Into Themes.
| Enrollment motivation themes | Number of participants n = 140 |
|---|---|
| Development—n(%) | 19 (14%) |
| Improve patient care | 5 (4%) |
| Personal improvement | 2 (1%) |
| Professional improvement | 13 (9%) |
| Helpful learning tool—n(%) | 15 (11%) |
| Helpful course in general | 5 (4%) |
| Exam preparation | 6 (4%) |
| Radiation oncology career preparation | 4 (3%) |
| Improve subject knowledge or skill—n(%) | 62 (44%) |
| Anatomy | 36 (26%) |
| Confidence | 2 (1%) |
| Contouring | 37 (26%) |
| Improve in general | 3 (2%) |
| Improve in general knowledge or skill | 4 (3%) |
| Radiology | 13 (9%) |
| Teaching | 1 (1%) |
| Learn subject knowledge or skill—n(%) | 35 (25%) |
| Anatomy | 12 (9%) |
| Anatomy-radiology correlations | 6 (4%) |
| Contouring | 11 (8%) |
| Learn in general | 12 (9%) |
| Radiology | 5 (4%) |
| Radiation oncology interest—n(%) | 3 (2%) |
| Interested in career | 3 (2%) |
| Refresh subject knowledge—n(%) | 9 (6%) |
| Anatomy | 4 (3%) |
| Contouring | 1 (1%) |
| Update knowledge in general | 4 (3%) |
| Radiology | 1 (1%) |
| Requirement—n(%) | 13 (9%) |
| Curriculum requirement | 13 (9%) |
The participants were organized into 3 independent groups based on educational program: RO resident, professional (radiation oncologist and medical physicist), or other program (ie, medical student or medical physics resident). The quantitative analysis was based on group to determine similarities and differences between the groups’ motivation to enroll in the ARC Bootcamp, and the qualitative analysis was not based on group.
Online ARC Bootcamp Participants’ Initial Feedback
Demographic characteristics
From November 2019 to January 2021, 140 individuals registered for the ARC Bootcamp and completed the preintervention evaluation. Of the 140 participants, 27 (19%) completed all Bootcamp components and were therefore included in this study. The demographics of participants who registered for the Bootcamp versus completed the Bootcamp are identified in Table 1. Of participants who registered, 34% were female, and of participants who completed the course, 37% were female. Participants were located globally, reaching 6 continents and spanning 19 different countries. Most learners were located in North America (registered participants: 47%). The majority of registered learners were RO residents (56%), followed by other programs (24%), such as medical physics students or medical students. Participants who completed the course were mainly in the other program category (52%), followed by RO residents (41%). The majority of RO residents that enrolled in and completed the Bootcamp were junior residents (postgraduate year [PGY] 1-3) (registered: 60%; completed: 82%) compared to senior residents (PGY 4-5).
Table 3.
ARC Bootcamp Strengths Reported as Average Number of Participants and Grouped Into Themes.
| ARC Bootcamp strength themes | Number of participants n = 27 |
|---|---|
| Environment—n(%) | 1 (4%) |
| Positive learning environment | 1 (4%) |
| Interactive components—n(%) | 4 (15%) |
| Cadaveric prosections | 2 (7%) |
| Real-time contouring | 3 (11%) |
| Learning—n(%) | 8 (30%) |
| Integrative learning | 4 (15%) |
| Multidisciplinary approach | 4 (15%) |
| Pace of learning—n(%) | 5 (19%) |
| Pace | 3 (11%) |
| Review of course material | 2 (7%) |
| Quality of course—n(%) | 11 (41%) |
| Course design | 3 (11%) |
| Course material | 4 (15%) |
| Instructors | 3 (11%) |
| Lectures | 1 (4%) |
| Overall good course | 2 (7%) |
Participants motivation to enroll in the online ARC bootcamp
Enrollment motivation themes were identified from participants’ answers to the open-ended questions in the preintervention evaluation. Table 2 highlights the participants’ motivation to enroll in the course. The primary motivator for participants to enroll in the course was to improve their knowledge or skill in a subject area. Specifically, learners wanted to improve their anatomy knowledge and their contouring abilities. The second most common motivator among participants was to learn subject knowledge or a skill, specifically in anatomy and contouring. The ICC indicated excellent reliability between the 2 coders of the coding data for participants’ enrollment motivation (ICC value: 0.88).
Table 1.
Preintervention Versus Postintervention Demographic Characteristics.
| Characteristic | Registered for Bootcamp (n = 140) |
Completed Bootcamp (n = 27) |
|---|---|---|
| Gender—n(%) | ||
| Female | 48 (34) | 10 (37) |
| Male | 92 (66) | 17 (63) |
| Location—n(%) | ||
| North America | 66 (47) | 15 (56) |
| South America | 7 (5) | 2 (7) |
| Europe | 15 (11) | 1 (4) |
| Asia | 7 (5) | 0 (0) |
| Africa | 8 (6) | 2 (7) |
| Australia | 7 (5) | 0 (0) |
| N/A | 30 (21) | 7 (26) |
| Position—n(%) | ||
| Radiation oncology resident | 79 (56) | 11 (41) |
| Radiation oncologist | 22 (16) | 2 (7) |
| Medical physicist | 3 (2) | 0 (0) |
| Other program | 34 (24) | 14 (52) |
| N/A | 2 (2) | 0 (0) |
| Postgraduate year—n(%) | (n = 79) | (n = 11) |
| 1 | 9 (11) | 0 (0) |
| 2 | 17 (22) | 4 (36) |
| 3 | 21 (27) | 5 (46) |
| 4 | 14 (18) | 1 (9) |
| 5 | 16 (20) | 1 (9) |
| N/A | 2 (2) | 0 (0) |
Abbreviation: N/A, not answered.
Significant differences between the independent groups’ motivation to enroll in the Online ARC Bootcamp were identified for 4 of the 7 motivation themes: improve knowledge or skill in a subject area (X2(2) = 10.452, P = .005), interest in RO (X2(2) = 8.870, P = .012), refresh subject knowledge (X2(2) = 8.852, P = .012), and requirement (X2(2) = 29.671, P < .001). RO residents more frequently stated that their motivation to enroll in the ARC Bootcamp was to improve their knowledge or skill in a subject area (pbonf = 0.005) or refresh their subject knowledge (pbonf = 0.009) compared to the other program group. Participants from the other program group more frequently stated that their motivation to enroll in the ARC Bootcamp was because they were interested in the field of RO compared to the RO resident group (pbonf = 0.013) and more frequently stated that the online course was a curriculum requirement compared to the RO resident and professional groups (pbonf < 0.001).
Online ARC bootcamp strengths and areas for improvement
Major themes were identified from the answers to the open-ended questions from the postintervention evaluation. Tables 3 and 4 highlights the ARC Bootcamp's strengths and areas of improvement identified by participants. The course's quality, specifically related to the design, material, and instructors, were major strengths. Additionally, the integration of anatomy, radiology, and contouring concepts was well received by the participants. Lastly, participants appreciated learning the material at their own pace as it provided a flexible learning environment. Regarding areas of improvement, participants stated increasing the amount of course content, such as adding a breast section to the course or including more contouring exercises would be beneficial. The ICC indicated excellent reliability between the 2 coders of the coding data for the course's strengths and improvement areas (ICC values: strengths: 0.96; areas of improvement: 0.99).
Table 4.
ARC Bootcamp Areas of Improvement Reported as Average Number of Participants and Grouped Into Themes.
| ARC Bootcamp areas of improvement themes | Number of participants n = 27 |
|---|---|
| Contour instruction and feedback—n(%) | 2 (7%) |
| Increase contouring feedback | 1 (4%) |
| Visualization of instructor contouring | 1 (4%) |
| Decrease amount of content—n(%) | 1 (4%) |
| Anatomy lectures | 1 (4%) |
| Increase amount of content—n(%) | 11 (41%) |
| Breast content | 2 (7%) |
| Content areas in general | 2 (7%) |
| Course manual | 1 (4%) |
| Contouring exercises | 2 (7%) |
| Head and neck | 1 (4%) |
| Neuroanatomy | 2 (7%) |
| Quizzes | 1 (4%) |
| Radiology | 1 (4%) |
| Relevant pathology | 2 (7%) |
| Increase content time—n(%) | 3 (11%) |
| Abdomen and pelvis sections | 2 (7%) |
| Nodal regions | 2 (7%) |
| Pace of learning—n(%) | 1 (4%) |
| Slow pace | 1 (4%) |
| Restructure course—n(%) | 2 (7%) |
| Discipline-specific module | 2 (7%) |
Discussion
The benefits and disadvantages of creating an online ARC bootcamp
We demonstrated the feasibility of creating an online version of the ARC Bootcamp using the ADDIE model. The model provided 5 defined phases that the multidisciplinary team followed to ensure the learning experiences developed met the learner's needs and that the course was continuously monitored to make modifications.
The main drivers for the online course development were to increase access to the ARC Bootcamp and provide a flexible, self-paced learning environment for participants. The online course was accessible, with 140 enrolled learners located in 19 different countries and from various programs such as RO and medical physics, with a handful of medical student participants interested in learning about a RO career. The online ARC Bootcamp is available to an unlimited number of learners compared to the F2F Bootcamp, which was generally limited to 40 participants to maintain opportunities for hands-on learning in small groups.1 The online offering was also more affordable than the F2F course, especially for international learners.
In addition to increased accessibility, the online Bootcamp provided participants with a self-paced and flexible learning environment, which was identified as a strength of the course. Online participants were provided with 1 year to complete the course from the date of registration. Although the F2F Bootcamp was scheduled yearly to avoid competing clinical duties and vacations,1 the online participants could complete the course on their schedule and could review course material. Since the labs and lectures were asynchronous, participants could control their learning pace by pausing, rewinding, and fast-forwarding the recordings.
Online learners were able to test their knowledge acquisition with the integration of formative knowledge quizzes. Participants were provided with immediate corrective feedback allowing learners to revisit concepts before continuing. Formative assessments were incorporated because they are intended to promote participants’ learning14,15 and could result in improved learning outcomes.15 Although the 2 quizzes tested all anatomic sites, it may have been more beneficial to incorporate a quiz directly following the content of each anatomic site. This would have ensured knowledge was tested after each section, and corrective feedback was provided at each stage of the learning process. Contouring exercises were also integrated into the course; however, immediate feedback was not provided. Increasing the contouring feedback was an area identified by the participants for improving the course.
Although there were benefits to creating the online version, we did identify some disadvantages of an online ARC Bootcamp during the design and development phase of the course. The creation of the online radiology and contouring lectures was relatively simple; however, difficulties arose when developing the anatomy component of the course. The F2F anatomy sessions included kinesthetic learning experiences with cadaveric prosections and bony skulls; therefore, the online anatomy labs had to be newly created. Rather than incorporating active learning strategies through computer-based interactive resources, which is the common trend in online anatomy education,16,17 we developed the online anatomy labs to focus on identifying anatomical structures using cadaveric prosections and the discussion of spatial relationships. The reasoning was to ensure that F2F and online participants received a similar learning experience to determine the online Bootcamp's effectiveness on the learner's knowledge acquisition by comparing to the F2F Bootcamp.
In addition to the lack of hands-on learning, the online Bootcamp also lacked immediate interactions. Discussion forums were integrated to allow learners to discuss concepts with other participants and the instructors. Although the discussion forums were meant to promote student–student and student–instructor communication, participation tended to be minimal. It remains unknown why the discussion forums lacked participation; however, factors such as instructor participation and discussion forum structure could have influenced the participation rates.18 It has been stated that instructors can encourage the learner's discussion forum participation by responding to student's questions/posts promptly.19 Also, learners are more likely to partake in discussions with a clear purpose20 rather than discussions with no guidance or structure. By developing structured discussion forums with consistent instructor feedback, learner participation rates may increase.
Ultimately, the initial experiences with the online ARC Bootcamp have been similar to the experiences of other online courses designed for different medical residency specialties, such as obstetrics and gynecology (OB/GYN)21 and family medicine.22 Similarities were noted with the course design, specifically with both courses consisting of asynchronous learning modules designed by a multidisciplinary team and pretests and posttests to assess knowledge/understanding.21,22 Lastly, one of the courses also included discussion forums monitored consistently by an instructor.21 Similar to the online ARC Bootcamp findings, the online course for OB/GYN residents reported a high drop-out rate after enrollment (35%), stating that the majority of these participants were older or in their final year of residency.21 The online ARC Bootcamp experienced a similar trend with senior residents (PGY 4-5) less likely to complete the course compared to junior residents (PGY 1-3). Lastly, the OB/GYN residents reported high satisfaction with the online course, specifically with the multidisciplinary instruction and the opportunity for a flexible learning environment,21 which were also components of the online ARC Bootcamp that were identified as strengths of the online course.
Motivation to enroll in the online arc bootcamp
The ARC Bootcamp was designed to provide high-yield knowledge of anatomy and radiology, with clinical correlates and contouring instruction. Therefore, this course was well suited for the participants’ enrolled to improve/refresh their subject knowledge in anatomy, radiology, and contouring and to learn anatomy-radiology correlates. The individuals enrolled to learn knowledge in these subject areas may have struggled with the course material. For instance, some learners recommended discipline-specific or beginner modules to be developed and added to the course. Therefore, it is highly recommended that learners have background knowledge in these 3 subject areas to gain the most from the online ARC Bootcamp.
To enhance professional development and patient care were also enrollment motivators identified by learners. Although better contouring and adherence to guidelines have been shown to lead to improved quality care,23,24 it is unknown whether knowledge gained from the ARC Bootcamp results in measurable benefits for patients, which warrants the need for additional research. A handful of participants also used the ARC Bootcamp to prepare for their role as a radiation oncologist or upcoming examinations. The use of bootcamps to supplement both undergraduate and postgraduate medical education curricula has become common among medical trainees.25 Lastly, some learners stated that the online course was a mandatory requirement of their program, such as the Commission on Accreditation of Medical Physics Education Programs (CAMPEP). In the past, CAMPEP students have attended the F2F ARC Bootcamp. With the online course development, the program made the ARC Bootcamp a curriculum requirement since the course was accessible to all learners and could be incorporated into the curriculum with ease.
Modifications to the online ARC bootcamp based on participants’ feedback
Although several participants acknowledged the course's quality as a strength, course modifications can be made to ensure the content provided is helpful to learners. The majority of participants’ feedback regarding course improvement areas focused on increasing the amount of course content. Most notably, learners expressed interest in incorporating a breast section and neuroanatomy section, which currently do not exist. Based on this feedback, new modules are currently under development. In addition to increasing the amount of course content, participants requested more time to be spent on specific topics, such as the abdomen and pelvis sections and nodal concepts. Since this is an asynchronous online course, participants can control their learning pace and revisit the sections and concepts they wish to spend more time. It is better to use our resources to create new sections that currently do not exist rather than recreate existing sections.
Limitations
Completion of the online ARC Bootcamp was measured by finishing all course content, including the postintervention evaluation. Based on this metric, a low number of participants (19%) completed all the course components compared to the number of participants who registered for the course. The average percentage of the course completed by participants was unknown; therefore, participants may have learned all of the course content but did not submit the postintervention evaluation. From the date of registration, participants were provided with 1 year to complete the course, which may have contributed to the low completion rates. A study including 129 online courses reported a negative correlation between course length and course completion rates stating that longer courses resulted in a lower course completion rate among students.28 Therefore, decreasing the availability length of the online ARC Bootcamp could impact the course completion rates.
Future directions
Future studies of the ARC Bootcamp will assess the impact the online version has on learners’ anatomy and radiology knowledge, contouring ability, and anatomy, radiology, and contouring confidence by comparing to the F2F version of the Bootcamp. Additionally, predictors of participants’ success in the ARC Bootcamp will be determined. This will include identifying correlations between the predictor, such as gender, visuospatial ability, and current career position, to participants’ anatomy and radiology knowledge, contouring ability, and confidence. Data from the preintervention and postintervention evaluations will be used to conduct future studies.
Conclusion
This study demonstrated the feasibility of creating an online course in RO using the instructional design model ADDIE. The online ARC Bootcamp offers similar content to the F2F Bootcamp by providing additional integrated anatomy, radiology, and contouring instruction. The online offering is more accessible to learners resulting in enrollments from a diverse cohort of participants, provides a self-paced, flexible learning environment, and allows learners to assess their knowledge using integrated assessments. Several participants recognized the course's exceptional quality; however, the course will be continuously monitored using the integrated evaluations and participants’ feedback to make any necessary course modifications to ensure learners’ needs are met.
Supplementary Material
Acknowledgements
The authors would like to acknowledge all individuals who volunteered their time and contributed to creating the online ARC Bootcamp. The authors also recognize the participants who completed the evaluations and provided their valuable feedback. Finally, they thank Haley Linklater, Body Bequeathal Program Supervisor, and Kevin Walker, Body Bequeathal Program Technician, at the University of Western Ontario.
Footnotes
DECLARATION OF CONFLICTING INTERESTS: The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
FUNDING: The author received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material: Supplemental material for this article is available online.
ORCID iD: Paige Eansor https://orcid.org/0000-0001-9746-5426
References
- 1.Jaswal J, D’Souza L, Johnson M, et al. Evaluating the impact of a Canadian national anatomy and radiology contouring boot camp for radiation oncology residents. Int J Radiat Oncol Biology Phys. 2015;91:701-707. doi: 10.1016/j.ijrobp.2014.11.009 [DOI] [PubMed] [Google Scholar]
- 2.Panigrahi R, Srivastava PR, Sharma D. Online learning: adoption, continuance, and learning outcome—A review of literature. Int J Inform Manage. 2018;43:1-14. doi: 10.1016/j.ijinfomgt.2018.05.005 [DOI] [Google Scholar]
- 3.Debenham B, Banerjee R, Fairchild A, Dundas G, Trotter T, Yee D. Canadian Radiation oncology resident survey. Int J Radiat Oncol Biology Phys. 2012;82:1326-1331. doi: 10.1016/j.ijrobp.2011.04.030 [DOI] [PubMed] [Google Scholar]
- 4.Bucci MK, Bevan A, Roach M. Advances in radiation therapy: conventional to 3D, to IMRT, to 4D, and beyond. Ca Cancer J Clin. 2005;55:117-134. doi: 10.3322/canjclin.55.2.117 [DOI] [PubMed] [Google Scholar]
- 5.Garibaldi C, Jereczek-Fossa BA, Marvaso G, et al. Recent advances in radiation oncology. Ecancermedicalscience. 2017;11:785. doi: 10.3332/ecancer.2017.785 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Nakamura K, Sasaki T, Ohga S, et al. Recent advances in radiation oncology: intensity-modulated radiotherapy, a clinical perspective. Int J Clin Oncol. 2014;19:564-569. doi: 10.1007/s10147-014-0718-y [DOI] [PubMed] [Google Scholar]
- 7.Chino J, Doyle S, Marks LB. The anatomy of radiation oncology residency training. Int J Radiat Oncol Biology Phys. 2014;88:3-4. doi: 10.1016/j.ijrobp.2013.09.039 [DOI] [PubMed] [Google Scholar]
- 8.D’Souza L, Jaswal J, Chan F, et al. Evaluating the impact of an integrated multidisciplinary head & neck competency-based anatomy & radiology teaching approach in radiation oncology: a prospective cohort study. Bmc Med Educ. 2014;14:124. doi: 10.1186/1472-6920-14-124 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Labranche L, Johnson M, Palma D, D’Souza L, Jaswal J. Integrating anatomy training into radiation oncology residency: considerations for developing a multidisciplinary, interactive learning module for adult learners. Anat Sci Educ. 2015;8:158-165. doi: 10.1002/ase.1472 [DOI] [PubMed] [Google Scholar]
- 10.Cheung L. Using the ADDIE model of instructional design to teach chest radiograph interpretation. J Biomed Educ. 2016;2016:1-6. doi: 10.1155/2016/9502572 [DOI] [Google Scholar]
- 11.Biggs J. Enhancing teaching through constructive alignment. High Educ. 1996;32:347-364. doi: 10.1007/bf00138871 [DOI] [Google Scholar]
- 12.Blumberg P. Maximizing learning through course alignment and experience with different types of knowledge. Innovative High Educ. 2009;34:93-103. doi: 10.1007/s10755-009-9095-2 [DOI] [Google Scholar]
- 13.Fleiss JL, Levin B, Paik MC. Statistical methods for rates and proportions. Published online 2018. John Wiley & Sons. doi: 10.1002/0471445428 [DOI]
- 14.Rolfe I, McPherson J. Formative assessment: how am I doing? The Lancet. 1995;345:837-839. [DOI] [PubMed] [Google Scholar]
- 15.Velan GM, Jones P, McNeil HP, Kumar RK. Integrated online formative assessments in the biomedical sciences for medical students: benefits for learning. Bmc Med Educ. 2008;8:52. doi: 10.1186/1472-6920-8-52 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Trelease RB. From chalkboard, slides, and paper to e-learning: how computing technologies have transformed anatomical sciences education. Anat Sci Educ. 2016;9:583-602. doi: 10.1002/ase.1620 [DOI] [PubMed] [Google Scholar]
- 17.Clunie L, Morris NP, Joynes VCT, Pickering JD. How comprehensive are research studies investigating the efficacy of technology-enhanced learning resources in anatomy education? A systematic review. Anat Sci Educ. 2018;11:303-319. doi: 10.1002/ase.1762 [DOI] [PubMed] [Google Scholar]
- 18.Fehrman S, Watson SL. A systematic review of asynchronous online discussions in online higher education. Am J Distance Educ. 2020;34:1-14. doi: 10.1080/08923647.2020.1858705 [DOI] [Google Scholar]
- 19.Dailey-Hebert A. Maximizing interactivity In online learning: moving beyond discussion boards. Journal of Educators Online. 2018;3:65-90. doi: 10.9743/jeo.2018.15.3.8 [DOI] [Google Scholar]
- 20.Delahunty J. Connecting to learn, learning to connect: thinking together in asynchronous forum discussion. Linguist Educ. 2018;46:12-22. doi: 10.1016/j.linged.2018.05.003 [DOI] [Google Scholar]
- 21.Vieira TCSB, Nakamura MU, Silva I da, et al. Experience of an online course on sexuality during pregnancy for residents. Sex Reprod Healthc. 2017;12:76-81. doi: 10.1016/j.srhc.2017.03.004 [DOI] [PubMed] [Google Scholar]
- 22.Skye EP, Wimsatt LA, Master-Hunter TA, Locke AB. Developing online learning modules in a family medicine residency.pdf. Fam Med. 2011;43:185–192. [PubMed] [Google Scholar]
- 23.Kong F, Ying H, Du C, et al. Patterns of local-regional failure after primary intensity modulated radiotherapy for nasopharyngeal carcinoma. Radiat Oncol. 2014;9:60. doi: 10.1186/1748-717x-9-60 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Furman MJ, Whalen GF, Shah SA, Kadish SP. Gastric perforation following stereotactic body radiation therapy of hepatic metastasis from colon cancer. Pract Radiat Oncol. 2013;3:40-44. doi: 10.1016/j.prro.2012.03.005 [DOI] [PubMed] [Google Scholar]
- 25.Davidson EH, Barker JC, Egro FM, Krajewski A, Janis JE, Nguyen VT. A national curriculum of fundamental skills for plastic surgery residency. Ann Plas Surg. 2017;78:121-126. doi: 10.1097/sap.0000000000000977 [DOI] [PubMed] [Google Scholar]
- 26.Duffy CC, Nawoor-Quinn Z, Burlacu CL. “Rapid sequence induction”—an anaesthesiology boot camp. Ir J Medical Sci 1971. 2020;189:1047-1051. doi: 10.1007/s11845-019-02146-w [DOI] [PubMed] [Google Scholar]
- 27.Khan U, Bakhiet A. Implementing a unique immersive near-peer-led clinical skills educational bootcamp for early-year clinical medical students. Postgrad Med J. 2021;0:1-5. doi: 10.1136/postgradmedj-2020-139233 [DOI] [PubMed] [Google Scholar]
- 28.Jordan K. Massive open online course completion rates revisited: assessment, length and attrition. Int Rev Res Open Distributed Learn. 2015;16:341–358. 10.19173/irrodl.v16i3.2112 [DOI] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.