Protocol for a randomised trial evaluating the effect of applying gamification to simulation-based endoscopy training

Abstract

Background

Simulation-based training (SBT) provides a safe environment and effective means to enhance skills development. Simulation-based curricula have been developed for a number of procedures, including gastrointestinal endoscopy. Gamification, which is the application of game-design principles to non-game contexts, is an instructional strategy with potential to enhance learning. No studies have investigated the effects of a comprehensive gamification curriculum on the acquisition of endoscopic skills among novice endoscopists.

Methods and analysis

Thirty-six novice endoscopists will be randomised to one of two endoscopy SBT curricula: (1) the Conventional Curriculum Group, in which participants will receive 6 hours of one-on-one simulation training augmented with expert feedback and interlaced with 4 hours of small group teaching on the theory of colonoscopy or (2) the Gamified Curriculum Group, in which participants will receive the same curriculum with integration of the following game-design elements: a leaderboard summarising participants’ performance, game narrative, achievement badges and rewards for top performance. In line with a progressive learning approach, simulation training for participants will progress from low to high complexity simulators, starting with a bench-top model and then moving to the EndoVR virtual reality simulator. Performance will be assessed at three points: pretraining, immediately post-training and 4–6 weeks after training. Assessments will take place on the simulator at all three time points and transfer of skills will be assessed during two clinical colonoscopies 4–6 weeks post-training. Mixed factorial ANOVAs will be used to determine if there is a performance difference between the two groups during simulated and clinical assessments.

Ethics and dissemination

Ethical approval was obtained at St. Michael’s Hospital. Results of this trial will be submitted for presentation at academic meetings and for publication in a peer-reviewed journal.

Trial registration number

NCT03176251.

Strengths and limitations of this study.

• The intervention in this randomised trial is a comprehensive gamified simulation-based curriculum in gastrointestinal endoscopy that includes a game narrative, performance tracking measures and rewards. These game-design elements are grounded in educational theory.

• The primary outcome is clinical performance of live colonoscopies on real patients, which will be assessed by two blinded independent expert endoscopists using an assessment tool with strong validity evidence.

• Participants will be assessed immediately after training for skill acquisition, and 4– 6 weeks after training to evaluate skill retention and transfer of skills to the clinical environment.

• There are significant human resources required for implementation with respect to tracking participants’ game metrics and adjusting leaderboards.

Introduction

Simulation-based training (SBT) provides a safe and effective means to enhance skills development in gastrointestinal endoscopy.^{1 2} SBT is more effective when embedded within a curriculum that is grounded in educational theory.^3–6 While previous studies have demonstrated the effectiveness of comprehensive structured curricula and curricula based on a progressive learning approach,^{4 7} other instructional strategies may further enhance procedural skills training.

One such enhancement may lie in gamification. Gamification refers to the application of game-design elements (conceptual building blocks central to creating successful games) to traditionally non-game contexts.^8–10 The overall purpose of gamification is to ‘encourage behavioral change and promote desired attitudes’.¹¹ Gamification has previously been applied in health-related settings such as health promotion and e-health.^12–14 More recently, it has been gaining traction in the medical education setting, as gamification has the potential to improve learning and learner attention, engagement, motivation and behaviour change.^{8 15} In a recent randomised trial, participants were ranked on a leaderboard as they completed cardiopulmonary resuscitation (CPR) training.¹⁶ After training, participants who were ranked on a leaderboard had significantly better technical skills acquisition on the CPR training device. Another recent trial evaluated the effect of competition on novices’ ability to learn simulated laparoscopic cholecystectomy.¹⁷ The authors found that participants who engaged in competition demonstrated fewer movements and a shorter path length, suggesting increased efficiency.

While these reports highlight the potential benefits of gamification in educational contexts, the use of leaderboards and competition represent narrow applications of gamification. To date, there are no studies that have investigated the application of a comprehensive gamified curriculum that integrates multiple game-design elements for procedural learning in medicine. Additionally, no studies have reported clinical outcomes on real patients. To bridge these gaps, we aim to determine the impact of a gamified simulation-based curriculum in endoscopy on clinical performance compared with an identical curriculum that does not incorporate game- design elements.

Methods & analysis

Study design

This single-blinded, parallel group, randomised controlled trial (RCT) is currently being conducted at St. Michael’s Hospital in Toronto, Canada. Recruitment started in June 2017. The methodology was adapted from previous studies by our group.^{3 4 18} We used the SPIRIT checklist when writing our report.¹⁹ The study design is summarised below in figure 1.

Figure 1.

Study design. VR, virtual reality.

Participants

Thirty-six novice endoscopists (performed <25 previous real and/or simulated colonoscopies) will be recruited by one author (MAS). Participants will be included if they are from the general surgery or gastroenterology residency programmes at the University of Toronto. Participants will be excluded if they have performed greater than 25 previous real and/or simulated colonoscopies.

Simulators

Bench-top simulator

The bench-top colonoscopy simulator is comprised a series of vertical wooden barriers with numbered holes conforming to 27 different sequences of varying complexity. Participants use a real video colonoscope, which provides visual output, to navigate through each sequence. This bench-top endoscopy simulator helps develop general endoscopic skills and has shown good validity evidence for training novices.²⁰

Virtual reality simulator

The EndoVR virtual reality (VR) endoscopy simulator (CAE Healthcare, Montreal) is used for the VR training and all simulator tests. It models navigation through a colon, using a specialised endoscope that is inserted into a computer-based module with a screen showing the colonic lumen of a virtual patient. It provides visual and haptic feedback related to the procedure. The simulator has several standardised case-based scenarios of varying complexity for colonoscopy and has robust validity evidence in the context of novices.^{21 22}

Experimental design

Baseline questionnaires

Participants will complete a questionnaire to collect baseline demographical information, including age, sex, level of training and previous endoscopic experience. Questions regarding experience with team sports and video games will also be included, as these may correlate with baseline endoscopic skills²³ (online supplementary file 1). Additionally, scales assessing the following variables will be administered: (1) competitiveness (Revised Competitiveness Index, online supplementary file 2); (2) self-efficacy (adapted General Self-Efficacy Scale, online supplementary file 3) and (3) game-type personality (Gamified User Personality Hexad, online supplementary file 4). All the included scales have good validity evidence.^24–26

Pretest

Participants will complete a series of assessments prior to training to assess (1) their baseline knowledge of colonoscopy (knowledge test); (2) technical skills (VR simulation test) and (3) non-technical skills (VR simulation ‘integrated scenario’ test). No feedback will be provided at any point during these assessments.

1. Knowledge Test: A 30 min, 17-item multiple choice question (MCQ) test designed to assess core concepts related to colonoscopy, including indications, pathology and theory underpinning non-technical skills (online supplementary file 5).

2. VR Simulation Test: A colonoscopy procedure on the VR simulator with a time limit of 30 min. Baseline technical proficiency will be assessed by an expert endoscopist. The procedure will be video-recorded, with identifying features hidden, to allow for a blinded assessment at a later time.²⁷

3. VR Simulation ‘Integrated Scenario’ Test: A test in which participants will complete a colonoscopy procedure on the VR simulator in a naturalistic setting (ie, endoscopy suite) while interacting with a standardised nurse and standardised patient.²⁸ Trainees will be expected to take a brief history of patients and obtain informed consent. The trainee will then carry out the procedure (EndoVR Module 3—Polypectomy) as described above while responding to the patient and interacting with the nurse as appropriate. As in the technical test, performance will be assessed in real time and videotaped, ensuring anonymity is preserved.

Training intervention

Following the pretests, participants will be randomised to one of two training groups, following a 1:1 allocation distribution with no stratification. One author (RK) used an online sequence generator (https://www.random.org/sequences/) to generate a random sequence of numbers and placed labels with these numbers into sealed envelopes. Another author (MP), not involved in sequence generation, distributed the sealed envelopes to participants as they arrived for the course. The first author (RK) was not present during envelope distribution. Investigators were blinded to group allocation.

Conventional Curriculum (controls)

The control group will receive a total of four, 1-hour, small-group teaching sessions covering the theory of colonoscopy, including pathology, anatomy and therapeutic technique. One session is dedicated to non-technical skills relevant to endoscopy (situation awareness, decision making, communication, teamwork and leadership) and how they relate to clinical performance. In this session, participants will watch a video demonstrating ideal endoscopic non-technical skills and learn about the Endoscopic Non-Technical Skills (E-NTS) checklist that will be provided for them to use during the integrated scenario training (online supplementary file 6). This checklist was developed in accordance with evidence-based recommendations, and it outlines key endoscopic non-technical skills.²⁹ Following each teaching session, a short MCQ test on the topics covered in that session will be administered, in keeping with the ‘test-enhanced learning’ literature.³⁰ In addition to teaching sessions, the control group will be given a total of 6 hours of expert-assisted instruction on both the bench-top simulator (1 hour) and the VR simulator (5 hours). Six modules of increasing difficulty in colonoscopy will be taught with one-on-one feedback from an expert academic endoscopist. The instructor will demonstrate techniques, answer questions and provide individualised performance feedback with a focus on non-technical skills. The last 2 hours of training on the VR simulator will consist of integrated scenarios that feature a standardised patient and nurse. Following each scenario, the instructor will debrief the trainees on their performance, using the ‘E-NTS checklist’ as a framework for discussing their non-technical skills.

Gamified Curriculum (GC)

This group will receive the same 4 hours of small-group teaching and 6 hours of hands-on simulator training. Within the context of the teaching sessions and simulator training, the gamified curriculum will incorporate the following game-design elements: a game narrative, performance tracking measures and rewards. First, a game narrative will underlie the delivery of the gamified curriculum. Participants will be assigned an avatar and will be tasked with completing a journey of the avatar around a game-board shaped like the colon (online supplementary file 7) with the goal of reaching the final destination, the terminal ileum. Second, performance tracking measures will be used to allow participants to gauge their performance over time. These measures will be summarised on a leaderboard, which will include four components: a non-technical skills score, a technical skills score, a cognitive skills score and an overall ranking, which will be determined through an algorithm that accounts for non-technical, technical and cognitive scores. Scoring of the non-technical and technical skills will be based on assessed performances during practice sessions on the VR simulator using the Modified Objective Structured Assessment of Non-Technical Skills (MOSANTS) (online supplementary file 8) and the Joint Advisory Group for Gastrointestinal Endoscopy’s Direct Observation of Procedural Skills (JAG DOPS) tool (online supplementary file 9), respectively. Scoring of cognitive skills will be based on MCQ test scores from the teaching sessions. Scores will be aggregated on the leaderboard for participants under training on the same days. The leaderboard will be presented to participants after they finish each hour of practice. Finally, participants will engage in a system of both short-term and long-term rewards. One short-term reward will involve badges to recognise achievements of procedural benchmarks (eg, caecal intubation) (online supplementary file 10). Another short-term reward will be the assignment of a wearable medallion, which will be given to the participant with the highest overall ranking at the end of each hour of practice. The long-term reward will be a low-cost prize (ie, less than $25 CAD) given to the participant with the highest overall ranking throughout practice. All three game-design elements (game narrative, performance tracking measures and reward system) will be introduced to participants in the gamified curriculum group prior to training with a brief tutorial video. After watching the video, participants will receive an anonymised ID to allow for self-tracking on the leaderboard while keeping individual scores private.

All three game-design elements are consistent with recommendations from the gamification and educational literature. In line with self-determination theory, leaderboards are purported to increase users’ sense of relatedness, engagement and competence through social comparison, feedback provision and documentation of achievement.³¹ The rationale for achievement badges and other rewards is that they serve as visual symbol of attained goals, thus supporting participants’ sense of competence and serving to foster external motivation and engagement.^{31 32} Finally, game narratives are thought to enhance engagement through the integration of meaning and interaction.⁹ These elements must be carefully calibrated to challenge and engage learners appropriately and to ensure maintenance of learners’ intrinsic motivation.^{8 15}

Post-test

Participants will complete a series of assessments immediately after training (immediate post-test). These assessments are: (1) knowledge acquisition, (2) technical skills acquisition and (3) non-technical skills acquisition. They will include the same Knowledge Test, VR Simulation Test and VR Simulation ‘Integrated Scenario’ Test that participants will complete during the pretest.

Delayed testing (retention and transfer)

Participants will complete a series of assessments 4–6 weeks after training to assess their retention and transfer of skills. These assessments include the following: (1) knowledge retention, (2) technical skills retention, (3) non-technical skills retention and (4) transfer of skills to the clinical environment. They will include the same Knowledge Test, VR Simulation Test and VR Simulation ‘Integrated Scenario’ Test that participants will complete during the pretest and the post-test. To assess for transfer of skills to the clinical environment, participants will also complete two live colonoscopies on real patients. These two procedures occurred simultaneously on a single day between 4 and 6 weeks after completion of training. These procedures will be videotaped in a manner that anonymises the identity of the participant and the patient. Procedures on patients with a history of colonic or pelvic surgery or difficult colonoscopy will be excluded. Sedation and monitoring will be carried out according to standard practices on the endoscopy unit. An experienced attending endoscopist (completed >500 previous colonoscopies) will provide verbal and/or hands-on assistance as necessary and take over if the participant cannot complete the procedure or if any concerns regarding patients’ safety arise. All patients were consented for the use of their procedure in this study.

Patients and public involvement

We based our approach to patients’ involvement on previously published studies focusing on clinical outcomes for endoscopic training.^{3 4} Specifically, patients’ involvement will be limited to their participation in the primary outcome, which involves assessment of clinical colonoscopies by study participants. Patients will not be required to evaluate the impact of the intervention. There will be no public involvement.

Primary outcome measure

The primary outcome measure is clinical performance during two live colonoscopies 4–6 weeks after training, as assessed by the JAG DOPS.³³ Each clinical colonoscopy will be independently assessed by two experienced endoscopists who will be blinded to group assignment. One rater will be present during the procedure and the other rater will assess the participants’ performance using the video-recorded procedure. Video-based assessment of endoscopic performances has been shown to have good validity evidence, compared with live assessment.²⁷

Secondary outcome measures

1. Knowledge acquisition, as assessed by the MCQ Knowledge Tests.

2. Technical skills acquisition during the VR Simulation Tests, as assessed by the JAG DOPS (online supplementary file 9).

3. Non-technical skills acquisition during the Integrated Scenario Test, as assessed by the Modified Objective Structured Assessment of Non-Technical Skills (M-OSANTS) for colonoscopy, which has good validity evidence for surgery and was modified for endoscopy⁵ (online supplementary file 8).

4. Patients’ comfort during the clinical colonoscopies, as assessed by the endoscopy nurses using the Nurse-Assessed Patient Comfort Score (NAPCOMS)³⁴ (online supplementary file 11).

Exploratory outcome measures

1. Participants’ self-efficacy after each simulated and clinical colonoscopy testing procedure, as measured by an adapted General Self-Efficacy Scale²⁵ (online supplementary file 3).

2. Cognitive load after each simulated and clinical colonoscopy testing procedure, as measured by the Cognitive Load Scale for Colonoscopy³⁵ (online supplementary file 12).

3. Participants’ competitiveness after each simulated and clinical colonoscopy testing procedure, measured using the Revised Competitiveness Index²⁴ (online supplementary file 2).

Experienced endoscopists will assess participants’ technical skills and non-technical skills during the pretraining, immediate and delayed post-training simulation-based assessments.

Data management

Data will be collected through paper forms directly from assessors. Data from the forms will be extracted and input into a database on a password-protected computer. There is no requirement for a data monitoring committee as this is neither a trial addressing the efficacy of a treatment nor is patients’ safety at risk. Details with respect to protection of confidentiality of participants’ data are outlined in the participants’ and patients’ consent forms (online supplementary files 13,14).

Analysis plan

Statistical analyses will be performed using SPSS V.20 (SPSS, Inc, Chicago, IL). Alpha for all statistical tests will be set at 0.05. For all primary and subgroup analysis, appropriate measures will be taken to minimise an inflated Type I error due to multiple comparisons.

Baseline questionnaire

Participants’ baseline variables will be characterised with descriptive statistics, using mean with SD for continuous variables and number frequency for categorical variables, respectively.

Clinical performance

Performance during the live colonoscopies will be compared between the two groups using the JAG DOPS, NAPCOMS and MOSANTS scores. A mixed factor 2 (Gamified vs Conventional Curriculum) × 2 (procedure 1 vs procedure 2) analysis of variance (ANOVA) will be used. ANOVA differences significant at p<0.05 will be further analysed using Tukey’s honestly significant difference (HSD) post-hoc tests. In addition, sensitivity analyses of the mixed factor ANOVA will be performed with gender and residency programme (ie, gastroenterology, general surgery) as covariates, as previous literature has identified gender differences in the acquisition of surgical skills.³⁶

Technical performance

Differences in technical skills acquisition on the simulator will be determined by comparing JAG DOPS scores between groups. Specifically, a mixed factor 2 (Gamified vs Conventional Curriculum) × 3 (pretest, post-test, retention test) ANOVA with Tukey’s HSD post-hoc tests will be conducted.

Non-Technical skill performance

Differences in non-technical skills acquisition on the simulator will be determined by comparing MOSANTS scores. A mixed factor 2 (Gamified vs Conventional Curriculum) × 3 (pretest, post-test, retention test) ANOVA with Tukey’s HSD post-hoc tests will be conducted.

Competitiveness

Baseline competitiveness, as measured by the Revised Competitiveness Index and the Gamification User Types Hexad, will be compared between groups using an independent t-test for each index.

Self-efficacy

Differences in self-efficacy between groups will be determined by comparing General Self-Efficacy Scale scores. A mixed factor 2 (Gamified vs Conventional Curriculum) × 2 (precourse vs postcourse) ANOVA with Tukey’s HSD post-hoc tests will be conducted.

Cognitive load

Differences in cognitive load between groups will be determined by comparing Cognitive Load Index of Colonoscopy scores. A mixed factor 2 (Gamified vs Conventional Curriculum) × 2 (precourse vs postcourse) ANOVA with Tukey’s HSD post-hoc tests will be conducted.

Sample size estimation

Since there are no prior studies investigating a gamified curriculum for procedural learning, we conducted the power analysis based on the effect size from a previous study that evaluated an SBT curriculum for endoscopy.³ Based on an effect size of 1.0 (Cohen’s d), an alpha of 0.05 (two-tailed), a beta of 0.10, two groups and two measurements, a minimum of 17 participants will be required to achieve a power of greater than 0.90 using repeated measures ANOVA (between factors). To accommodate for a potential 5% dropout and/or non-response, we will recruit a total of 36 participants.

Ethics and dissemination

Protocol version 1.0, dated 23 March 2017, was approved. If any protocol modifications are needed, they will be made after communication with the research ethics board and will be detailed in any subsequent publications. Informed consent will be obtained from endoscopist participants and patients by one author (MAS). No personal health data on patients will be collected. All authors will have access to trial data. We will disseminate the results of the study through peer-reviewed publication in journals and at scientific meetings. We do not plan to make participant-level data publicly available.

Feasibility

To date, 21 participants have been recruited, randomised and have completed the study. Data collection is ongoing and is intended to reach completion by August 2018. Subsequent data analysis, manuscript writing and submission for publication are anticipated to reach completion by July 2019.

Discussion

The use of SBT for procedural skills training is widespread. In the report commissioned by the Future of Medical Education in Canada Postgraduate Project, the authors conclude that ‘simulation … needs to be integrated more thoughtfully into postgraduate curricula’.³⁷ We aim to respond to this call through the development of an SBT curriculum grounded in educational theory. The strengths of this study lie in its randomised design and incorporation of various game-design elements into the curriculum. Additionally, the primary outcome is measured in the clinical setting by two blinded expert assessors using an assessment tool with strong validity evidence. Finally, participants will be assessed both immediately after training for skill acquisition and 4–6 weeks after training to evaluate skill retention and skill transfer to the clinical environment. There are several limitations to this study. First, this methodology requires substantial human resources to track participants’ game metrics and adjust the leaderboards accordingly. Second, participants who are wearing the medallion that signifies high-ranking performance are potentially identifiable as being in the intervention arm. We do not, however, anticipate that this will impact outcome measures because the medallion is not visible on the video-recordings of the procedures. Furthermore, it is not worn during live colonoscopies. Finally, participants’ frustration, which may impact performance, is not included as an outcome measure.