Simulation-based training for flexible cystoscopy – A randomized trial comparing two approaches

Abstract

Background

Simulation-based training allows trainees to experiment during training and end-of-training tests could increase motivation and retention. The aim of this trial was to determine if a simulation-based training program including directed self-regulated learning and post-testing improved clinical outcomes compared to a traditional simulation-based training program.

Methods

A randomized trial was conducted involving 32 participants without prior experience in endoscopic procedures. The intervention group practiced independently in a simulation centre and got a post-test whereas the control group received traditional instructions and demonstrations before being allowed to practice. Three weeks after the intervention the participants performed cystoscopies on two consecutive patients. Clinical performance was assessed using a global rating scale (GRS) with established evidence of validity. Independent samples t-test, Cronbach's α, Pearson's r, and paired samples t-test were used for statistical analysis.

Results

Twenty-five participants performed two cystoscopies on patients. There was no significant difference between the two study groups with respect to mean GRS of performance (p = 0.63, 95 % CI; -2.4–3.9). The internal consistency of the global rating scale was high, Cronbach's α = 0.91. Participants from both study groups demonstrated significant improvement between the first and second clinical procedures (p = 0.004, 95 % CI, 0.8–3.5). Eight (32%) and 15 (60%) participants demonstrated adequate clinical skills in their first and second procedure, respectively.

Conclusions

No significant differences were found on the clinical transfer when comparing the two programs. Neither of our training programs was able to ensure consistent, competent performance on patients and this finding could serve as an important argument for simulation-based mastery learning where all training continues until a pre-defined level of proficiency is met.

Trial registrations

The trial was submitted before enrolment of participants to the Regional Scientific Ethics Committee of the Capital Region which established that ethical approval was not necessary (H-4-2014-122). The trial was registered at Clinicaltrials.gov (NCT02411747).

Health profession; Medicine; Education; Directed self-regulated learning; Mastery learning; Transfer; Testing effect; Flexible cystoscopy; Virtual reality simulators

1. Introduction

Flexible cystoscopy is a common procedure in urology and urology trainees are required to master it early in their career [1]. During the initial part of the learning curve patients are exposed to prolonged procedure times, discomfort, and greater risk for complications [2, 3]. Thus, the classic surgical apprenticeship method (“see one, do one, teach one”) is not optimal. Simulation-based training of surgical procedures gives new trainees a possibility to practice before performing procedures on patients and is becoming an essential component of modern surgical education [4, 5]. Simulators in flexible cystoscopy have been available for more than a decade and several studies have demonstrated the impact of training [6, 7, 8]. However, acquiring the simulation equipment and implementing a simulation-based curriculum is a resource-demanding process that must be based on solid evidence [9, 10]. Optimal training programs must be efficient and planned in a manner that ensures retention of the trainees’ newly acquired skills and allows for optimal transfer of skills to consistent performance on patients.

New trainees practising on patients must be supervised at all times to improve patient safety but simulation-based training allows them to experiment and make mistakes during practice. Directed self-regulated learning (DSRL) is a learning approach where the student regulates his/her progress through a training protocol without guidance from an instructor. The theory is that this approach provides the student with the opportunity to develop "own" strategies and to learn from mistakes while also increasing the availability of training independent of supervision from a busy faculty [11]. Active, motivated learners are a prerequisite for DSRL and a test at the end of training might help ensure this. The use of post-testing as a tool to increase motivation and improve retention (testing effect) is well-described in memory science [12]. A study exploring testing effect in simulation-based resuscitation training indicated an impact on transfer but this finding needs to be confirmed [13].

The aim of this study was to compare a traditional training program consisting of expert instruction followed by simulator training to an alternative training program where trainees directed their own training and were motivated by a simulator test at the end. We were especially interested in the retention of skills, transfer of skills to patient performance, and acquiring the ability to perform in a consistent manner. Hence, three weeks after training all the trainees were tested in two consecutive procedures on real patients.

2. Methods

2.1. Design

We performed a randomized controlled superiority trial comparing two different instructional designs for simulation-based training in cystoscopy (see flowchart, Figure 2).

Figure 2.

Flowchart of the trial following the CONSORT statement for randomized trials.

2.2. Participants and setting

Thirty-two participants were enrolled in the trial complying with the CONSORT statement [14]. They were recruited through a student magazine and participation was voluntary. The participants were medical students in their last two years of medical school at the University of Copenhagen, Denmark and were included if they had no previous experience with simulation-based endoscopic training or procedures. All enrolled participants received a written theoretical introduction to the procedure prior to simulator training. Informed verbal and written consent was obtained from all participants upon their arrival at the Simulation Centre at Rigshospitalet [15]. At the time of enrolment, participants were randomized and blinded to their study group. The randomization was performed using a web-based randomization program allocated 1:1, stratified for sex (men/women) [16] (Urbaniak, G. C., & Plous, S. (2013). Research Randomizer Version 4.0. Retrieved on February 1, 2015, from http://www.randomizer.org/).

2.3. Intervention group

In the intervention group participants were asked by an instructor (SB) to perform simulator training and informed that they would be tested immediately after the simulator training. The participants had to direct their own training using:

• -A guide to a systematic examination of the bladder and detailed anatomical sheets
• -A virtual reality simulator (UroMentor™ simulator, Simbionix, Cleveland, Ohio, USA)
• -A simple rubber phantom (Uro-scopic Trainer™ simulator, Limbs and Things, Bristol, United Kingdom) with a flexible video cystoscope (Olympus, Japan)

A technical assistant (JD, OM) was available for support regarding scope handling and simulator-related problems. The technical assistants were not allowed to help with other aspects of training. After the simulator training, the participants were tested according to a predefined scenario on the virtual reality simulator by the instructor (SB) who also provided direct feedback. The test lasted 15 min and participants were not informed about the content of the test beforehand.

2.4. Control group

In the control group participants were introduced by the instructor (SB) to the procedure with a traditional presentation of 15 min. After the lecture and demonstration, the participants completed simulator training under the same conditions as the intervention group however the control group was not tested at the completion of their training (Participant training on the VR simulator, Figure 1). A time limit was introduced for both groups to ensure that one group did not train excessively compared with the other. In both groups, participants were given the option of ending training before the time limit if they felt competent with the procedure. The maximum time limit was defined based on our institution's experiences from courses in simulator training in flexible cystoscopy.

Figure 1.

Participant training on the VR simulator.

2.5. Clinical performance

Two to four weeks after simulator training the participants in both groups were assessed. Each participant performed two patient procedures directly after each other. A urology specialist (MR) supervised, directly observed, and assessed all 50 procedures performed by the remaining 25 participants with the GRS (see Appendix 1). The urology specialist was blinded to participant allocation. The supervision by the urology specialist included taking over the scope and finishing the procedure when deemed necessary to ensure patient-safety. The patients were already booked for follow-up at the Department of Urology at Rigshospitalet and had provided informed written consent prior to participation. Patients with previous reconstructive surgery of the urethra, bladder or ureter were excluded. All patient procedures were performed at the outpatient clinic of the Department of Urology at Rigshospitalet.

2.6. Outcome

An assessment tool with established evidence of validity was used to assess the clinical cystoscopy procedures [7]. The tool is based on a global rating scale (GRS) with five different parameters: respect for tissue, time and motion, handling of endoscope, flow of procedure, forward planning and knowledge of procedure. Each parameter was assessed based on a five-point Likert scale with a minimum of 1 and a maximum of 5, giving a total GRS score range of 5–25 (see Appendix 1). At our institution, we have defined a GRS score of 3 in each parameter (minimum total GRS of 15) as a minimum passing standard. Past research by Shout et al. [8] on GRS and simulation training on the UroMentor™ simulator indicated that a GRS score of 3 in each parameter in patient-related performance in flexible cystoscopy could be expected of a trained group compared to a group with no prior simulator training [8].

2.7. Statistical analysis

A power calculation was performed and power of 0.80 was chosen. All tests were two-sided and p < 0.05 was considered statistically significant. From a similar study [8], the standard deviation (SD) was assumed to be five and the means of the two groups 10 and 15 in GRS score. A total of 16 participants in each group were needed. Mean total GRS score of first and second procedure in the intervention group and the control group were compared using an independent samples t-test.

Internal consistency reliability for the GRS was determined by calculating Cronbach's α for the five parameters for all 50 procedures. Pearson's r was used to estimate the test-retest reliability. Finally, a paired t-test was performed to judge if the performances of the participants were consistent.

SPSS statistical package version 22.0 (IBM Corp. Released 2013. IBM SPSS Statistics for Windows, Version 22.0. Armonk, NY: IBM Corp) was used for the analysis of data.

2.8. Ethics

All patients and participants were given oral and written information about the trial by the principal investigator. All patients and participants gave informed written consent to participate prior to participation.

The trial was submitted to the Regional Scientific Ethics Committee, which established that ethical approval was not necessary before the enrolment of participants (H-4-2014-122). The trial was registered at Clinicaltrials.gov (NCT02411747).

3. Results

The first thirty-two novices who responded to the recruitment letter and were found eligible were enroled in the trial in the period of February to June 2015. Seven participants did not complete the two procedures on a patient; two chose to end their participation for personal reasons and five had their procedures cancelled due to patient-specific issues (no show, bladder infection on arrival, neobladder, haematuria, and severe comorbidities). Figure 2 shows a flowchart of the study and Table 1 shows baseline characteristics of the participants.

Table 1.

Participant characteristics.

	Sex		Mean age (min-max) (years)	Time spent in medical school, years	Total time of simulator training (minutes), mean (Standard Deviation, SD)	Time used on rubber phantom training (minutes), mean (SD)	Time used on virtual simulator training (minutes), mean (SD)
	Female	Male
Testing group (n = 13)	10	3	26 (25–30)	5.3	72 (7)	12 (6)	60 (7)
Lecture group (n = 12)	7	5	26 (24–31)	5.1	78 (24)	10 (5)	68 (24)
Total (n = 25)	17	8	26		p = 0.39*	p = 0.40*	p = 0.33*

The female: male participants reflected the ratio in Danish Medical students. There was no significant difference between the control and the intervention group in simulator training time or training pattern (see Table 1). The retention intervals were twenty-two days and twenty-three days in the intervention group and the control group, respectively.

There was no significant difference between study groups when comparing the means of total GRS scores for the combined results of both procedures in the intervention and control groups (13.6 compared to 14.3, respectively, p = 0.63, 95% CI, -2.4 - +3.9).

The internal consistency of GRS was high as reflected by Cronbach's α = 0.91. Participants in both study groups did not perform consistently across the two clinical procedures and the correlation between total GRS scores in first and second procedure across all study participants was only moderate with a Pearson's r = 0.68 (p < 0.001). The mean total GRS scores were 12.9 and 15.0 for the first and the second clinical cystoscopy procedure, respectively (p = 0.004, 95 % CI, 0.8–3.5).

There was no difference between intervention and control group in numbers of procedures reaching the minimum passing standard twelve versus eleven, respectively. The passing standard was achieved by eight (32%) and 15 (60%) participants for the first and the second procedure, respectively.

4. Discussion

4.1. Effect of testing

We found no significant difference with regard to GRS score when we compared the intervention group with the control group which is inconsistent with existing theories regarding the impact of the testing effect on skill retention. Based on these theories one would have expected that the intervention group would have experienced enhanced motivation and improved goal setting, after being informed that they would receive a post-test following their DSRL [17]. The testing effect in education can be defined “as the greater positive effect on future retention of a given material than the effect of re-studying in an equal amount of time on the material” [12]. The testing effect has mainly been described for verbal test material (e.g. recalling words, multiple choice tests) and a few studies have evaluated the testing effect for other materials that address spatial relationships (e.g. remembering maps [18] or objects in a three-dimensional space) [19]. The testing effect has previously been found to enhance knowledge retention compared to an equal amount of time spent on group practice of scenario-based training in resuscitation [13]. In our study we limited the amount of possible training time and this could have reduced or removed a possible positive effect on motivation since even trainees that wanted to ace the test were not allowed to practice for a prolonged period of time. Based on our results we recommend that trainees are allowed the possibility to continue training until they believe they are ready to be tested.

4.2. Retention interval

A meta-analysis from 2014 found that the retention interval (the time from the training to the testing) was a moderator of the testing effect [20]. In our trial the interval from training to testing was three weeks and it is not possible to assess whether a different interval could have changed the results. However, a randomized study on mannequin skill training for urethral catheterization found no change in participant performance when comparing results one and six weeks after skills training [21]. Furthermore, a study exploring retention of ECG interpretation skills found that participants’ skill level decreased during the first two weeks and then remained stable between two and twenty weeks after training [22]. Hence, the three-week interval in our trial seems appropriate for the detection of a possible testing effect.

4.3. Directed self-regulated learning

The intervention group used DSRL without initial instruction. Self-regulated learning (SRL) has been defined as “self-generated thoughts, feelings and actions that are planned and cyclically adapted to the attainments of personal goals” [23]. The processes of DSRL have been suggested to be composed of four components: Feedback loop, motivation, goal setting, and self-monitoring [24]. The learner is invited and challenged to learn the skill by experimenting and creating own experiences with the skill. In turn, this may increase their retention of the achieved knowledge [25, 26]. A subjective impression of the intervention group was that the participants experienced more anxiety. This could be explained by the instructional approach itself. The format was unfamiliar to the group, and this seemed to cause insecurity and anxiety, thus decreasing emotional motivation [17].

4.4. Score scale reliability

Previous studies have found the GRS to be a reliable assessment tool with regard to internal consistency reliability for cystoscopy and ureteroscopy performed on a virtual simulator [27]. The GRS could be used in our setting (i.e. is generalizable) and we found good internal consistency and reasonable test-retest reliability for GRS in the assessment of flexible cystoscopy performed on patients. These findings add validity evidence to the assessment tool.

4.5. Reaching a learning plateau

The participants improved significantly from the first to the second procedure on patients. This finding indicates that participants had not reached a level of consistent performance after the simulator training, i.e. that the amount and type of training offered in both training programs was insufficient to induce the required improvement in the medical students. All trainees learn at different paces and it is not possible to define the optimal duration of training. The concept of competency-based education as proposed in the Simulation-Based Mastery Learning (SBML) seeks “excellence for all” [28]. The principles of SBML are deliberate skills training with clearly defined objectives, formative assessment with feedback, and cessation of training only after a minimum passing standard has been achieved [10]. The time used by each learner during SBML varies, but the final skill level of each participant will ultimately be the same. A meta-analysis from 2011 found positive effects of SBML compared to the Halstedian apprenticeship [29]. In addition, emerging evidence regarding the impact on patient-related outcomes supports simulation-based skills training in medical education [30].

4.6. Acceptable clinical performance

After simulation-based training, 32% of participants were able to perform an acceptable cystoscopy for the first procedure and 60% for the second procedure. Though simulator training teaches important skills, it should be followed by a traditional apprenticeship with supervised clinical procedures [31].

The advantages of simulator training compared to traditional apprenticeship are the availability of hands-on training, the relatively stress-free learning environment, and perhaps most importantly the absence of patient risk [31]. By practising on the simulator the student advances up the initial part of the learning curve hereby surpassing the initial obstacles before facing a patient [30]. In our study we did not include an untrained control group so we cannot demonstrate an effect of simulation. However, we believe that a passing rate of more than 30% in an inexperienced group of medical students is indeed higher than it would be in a group of untrained and inexperienced medical students. However, it is clear that not all participants performed to a competent clinical level after simulator training. These participants may have benefitted from SBML which would have ensured that all participants had reached a minimum level before transferring their skills to patients.

4.7. Limitations

The sample size was limited and further reduced due to 7 participants not being able to finish the study. A systematic review from 2013 found only five randomized controlled trials (RCT) in simulation-based medical education with patient outcomes, which compared different simulation-based instructional designs [32]. The five RCTs included data from 30, 60, 26, 28 and 30 participants, respectively. Thus, there is a general need for well-powered simulation training designs with translational patient outcomes in the future. However, performances of both groups in our study were very similar and we believe the risk of a type II error is low.

The clinical procedures were only rated by a single rater, which is a limitation. A setting with several raters with direct or video-based assessment would have strengthened our trial.

Participants were volunteer medical students, who may have had a pre-existing interest in the skill and, therefore, may have outperformed non-responding students, i.e. a selection bias may be present [33].

We used a minimum passing standard, which was arbitrarily defined by our faculty.

The simulator used in this study was not fully immersive, i.e. the virtual reality was not shown on head-mounted display but on a two-dimensional monitor. A highly advanced immersive VR device might have improved the efficacy of training but could also have increased the cognitive load of the trainees [34].

To our knowledge, no prior research on a minimum standard score exists and such a score for GRS and simulator training in cystoscopy has yet to be defined. Future research is needed to establish a pass/fail standard based on accepted standard setting methods [35].

5. Conclusion

The two proposed simulation-based training programs in this study produced similar results regarding the transfer of skills to performance on patients. Sufficient competence was not acquired in the simulation centre and based on our results we recommend that all training programs should allow for and demand continued to practice until pre-defined criteria are met (i.e. mastery learning).

Declarations

Author contribution statement

S. Bube: conceived and designed the experiments; performed the experiments; analyzed and interpreted the data; contributed reagents, materials, analysis tools or data; wrote the paper.

J. Dagnaes-Hansen, O. Mahmood, M. Rohrsted, L. Salling:conceived and designed the experiments; performed the experiments; contributed reagents, materials, analysis tools or data.

F. Bjerrum, R. Hansen: conceived and designed the experiments; contributed reagents, materials, analysis tools or data; wrote the paper.

L. Konge: conceived and designed the experiments; analyzed and interpreted the data; wrote the paper.

Funding statement

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Competing interest statement

The authors declare no conflict of interest.

Additional information

The clinical trial described in this paper was registered at Clinicaltrials.gov under the registration number NCT02411747.

Appendix 1. Global Rating Scale Global Rating Scale (GRS) for cystoscopy. The GRS 3 has been modified as the participants did not handle the scope in urethra.

GRS 1 Respect for tissue	Scope frequently pushed into urothelial wall			Scope occasionally pushed into urothelial wall		No trauma to urothelial wall with scope
	1		2	3	4	5
GRS 2 Time and motion	Many unnecessary moves			Made some unnecessary moves but time more efficient		No unnecessary moves and time is maximized
	1		2	3	4	5
GRS 3 Handling of endoscope	Scope poorly aligned during procedure			Better use of scope angle during procedure		Scope always set in good angle throughout the procedure
	1		2	3	4	5
GRS 4 Flow of procedure and forward planning	Frequently stopped and needed advice			Demonstrated the ability to think forward with relatively steady progression of the procedure		Obviously planned procedure from beginning to end with fluid motion
	1		2	3	4	5
GRS 5 Knowledge of procedure	Deficient knowledge. Needed specific instruction at most procedural steps.			Knew all important aspects of procedure.		Demonstrated familiarity with all aspects of procedure
	1	2		3	4	5