Abstract
Objective
There is an increasing trend to incorporate Simulator-based training in urology residency programs. The study was designed to determine the construct validity of UroSim® for that we compared the performance of transurethral resection of bladder tumor (TURBT) between experts and novices.
Material and methods
We conducted a cross sectional study at a university hospital to determine the construct validity of UroSim® for TURBT. We compared the outcome measures between experts (urology consultants) and novices (residents) to determine relationship between clinical experience and performance on simulator. Primary outcome measure was resection time and secondary outcome measures were safety, bleeding, and visualization during TURBT on UroSim. We requested participant to resect three tumors to assess the test content of the simulator. Comparison of continuous variables such as resection time, resection, bleeding control, and visualization and blood loss using student t test. Comparison of categorical variable, i.e. perforation of bladder, using Fischer exact test.
Results
We included 30 experts and 30 novices. There was a statistically significant difference in the mean resection time between the groups (196±67.4 sec versus 374.6±179.7 sec; p=0.01), suggesting a positive relationship between clinical experience and performance on simulator. Additionally, safety parameters, namely, bleeding control, inadvertent cuts into bladder wall, ureteric orifices, and bladder perforations varied between the two groups.
Conclusion
We observed significant differences of parameters in performance between experts and novices. Simulator is a useful tool for teaching TURBT as it demonstrates good construct validity and recommended in urology training for teaching psychomotor skills.
Keywords: Simulation training, transurethral resection, urinary bladder neoplasms, validation studies
Introduction
Transurethral resection of the bladder tumor (TURBT) is the gold standard for diagnosis and initial staging of bladder cancer.[1] All urology trainees need to learn this basic skill; therefore, a standard way of learning TURBT is essential in urology training programs. The master apprentice model,[2] animal models,[3] and simulator-based training are some of the ways of learning surgical skills.[4] Simulators are now widely available and included as a part of urology training in numerous countries.[5] Many studies have shown that simulators are realistic, easy to use, and good educational tools.[6] Simulators not only assist in learning a procedure but also in maintaining skills and developing the ability to deal with complications.[7] The major advantage of virtual reality (VR) surgical simulators is that it provides repetitive practice and feedback on performance in a safe environment and in absence of direct mentor supervision.
Simulators have the potential to shorten the learning curve for complex surgical procedures. They can help create skills that can be transferred to the operating room and may shorten the learning curve. The concept of simulator-based training in urology training is well-established now. However, access to this facility is currently limited.[8] Nevertheless, before incorporating simulator-based training in urology residency program, proper validation is needed, especially in terms of face and construct validity. The construct validity is the degree to which a test measures what it claims or purports to be measuring.[9] Construct validity is usually verified by comparing the test with other tests that measure similar qualities to see the correlation between the two measures. An updated model of construct validity was described by Messick based on five components, namely, test content, response process, internal structure, relationship with other variables, and consequences. In this study, we attempted to establish the construct validity of training TURBT on a VR simulator by comparing defined outcomes on the simulator between novices and experts to establish test content and relationship with experience of UroSim.
In urology, many simulators have been validated for transurethral resection of prostate, but there is little data available on the use of simulators for learning TURBT. The Simbla® and UroSim are two of the most popular simulators in urological training; Simbla has been evaluated for TURBT [10], whereas UroSim has been validated for transuretheral resection of prostate (TURP) training. In a study by de Vries et al.[10], construct validity of Simbla for TURBT was evaluated. The median resection time of experts was significantly lower as compared with novices (64 versus 92.5 s, p value 0.04). Construct validity is established if experts and novices perform at different levels on the given task. We sought to evaluate construct validity of UroSim for TURBT by comparing performance of TURBT on UroSim between urology residents and consultants so that its role as a training tool can be assessed.
Material and methods
This prospective cross-sectional study was conducted at a center for innovation in medical education of a university hospital. Our objective was to determine construct validity of UroSim for TURBT (Figure 1). Primary outcome measure was resection time and secondary outcome measures were safety/bleeding/visualization during TURBT on UroSim. Outcome measures were compared between experts (urology consultants) and novices (residents) to determine relationship between clinical experience and performance on simulator. Resection time was defined as the time taken for resection of preselected case of bladder tumor on UroSim. Novices were postgraduate trainees in urology or rotating in urology and experts were urologists with experience of at least 50 independent TURBTs. Postgraduate residents with no previous experience of endoscopy were excluded.
Figure 1.
Training performing TURBT on UroSim simulator
A sample size of 30 participants in each group was calculated considering confidence interval of 95% and power 80% with mean difference of 43.85 in resection time between experts and novices as in study by de Vries for Simbla simulator.[10] After approval from the Ethical review committee, instructional video showing TURBT technique on UroSim was shown to each participant and 15 min were given to familiarize with UroSim. Performance of TURBT (case 2 on preinstalled software) was recorded on proforma. Each participant was instructed to resect three tumors (Figure 2): left postero-lateral wall (tumor 1), posterior wall (tumor 2), and the most challenging on dome (tumor 3, see Figure 1) to assess test content of the simulator. Resection time and other parameters (complications/bleeding/visualization) were noted from UroSim at the end of procedure for each participant (Annexure 1: UroSim end of procedure parameters). Data were analyzed on Statistical Package for the Social Sciences version 19 (IBM SPSS Corp.; Armonk, NY, USA). Continuous variables such as resection time, resection percentage, bleeding control, and visualization percentage, and blood loss were compared using student t test. Categorical variable, i.e., perforation of bladder was compared using Fischer exact test.
Figure 2.
Transurethral resection simulator view of the resectoscope and tumors in the bladder
Results
We included 30 experts and 30 novices. Most of the experts (24/30) were in academic practice and were associated with teaching institutes at the time of study. All the novices had some experience of TURBT at least to level of observation (30/30) and few had performed it under direct supervision (12/30); there was statistically significant difference in mean resection time between novice and experts suggesting positive relationship between clinical experience and performance on simulator. Most of the novices were unable to complete resection of tumor three on simulator, which was located at dome and was technically difficult. In addition, safety parameters, namely, bleeding control, cuts into bladder wall, cuts into ureteric orifices, and bladder perforations were significantly different between two groups demonstrating test content of the simulator (Table 1).
Table 1.
Comparison of experts and novices for primary and secondary outcome parameters
| Experts | Novices | p | |||
|---|---|---|---|---|---|
|
|
|
||||
| Mean | SD | Mean | SD | ||
| Resection Time | 196.10 | 67.39 | 374.60 | 179.65 | 0.00 |
|
| |||||
| Resection % | |||||
| Tumor | 1 98.77 | 2.01 | 92.77 | 25.24 | 0.20 |
| Tumor | 2 97.10 | 2.35 | 93.07 | 21.16 | 0.31 |
| Tumor 3 | 97.47 | 3.70 | 82.70 | 37.64 | 0.04 |
|
| |||||
| Bleeding control | |||||
| Visibility % | 99.77 | 0.43 | 96.43 | 6.21 | 0.01 |
| Blood loss (mL) | 0.00 | 0.00 | 7.43 | 23.67 | 0.10 |
| Remaining bleeders | 0.37 | 0.67 | 0.33 | 0.66 | 0.85 |
|
| |||||
| Safety | |||||
| Cutting while obscured (sec) | 0.00 | 0.00 | 0.07 | 0.25 | 0.16 |
| Tool active without contact (sec) | 0.17 | 0.38 | 0.10 | 0.55 | 0.59 |
| Cuts into bladder wall | 1.00 | 1.44 | 3.97 | 2.39 | 0.00 |
| Cuts into ureteric orifices | 0.10 | 0.31 | 0.73 | 1.60 | 0.04 |
| Perforation | 0.00 | 5.00 | 0.05 | ||
|
| |||||
| Visualization | |||||
| Bladder surface % | 76.90 | 20.45 | 61.27 | 22.64 | 0.01 |
| Right U.O | 3.03 | 4.05 | 5.97 | 6.20 | 0.04 |
| Left U.O | 4.87 | 3.74 | 8.13 | 11.02 | 0.13 |
SD: standard deviation; U.O: ureteric orifice
Discussion
Simulator training is supposed to amplify or replace real-life experience in order to train young surgeons in a safe, fully interactive setup. Neither all procedures nor all individuals can be reliably trained on simulators alone. However, simulators are likely to shorten the learning curve and minimize patient harm from novice surgeons. How far it is able to achieve these goals is still a matter of debate. Indeed simulation is not a substitute of real-life training; in fact, it can only complement the operating room teaching of urologic surgery. Despite significant developments in the field of simulators, there is still dearth of quality evidence to suggest that simulator-trained surgeons have a shorter learning curve. In particular, there is lack of evidence that non-technical skills training improve complication rates in the operating room.[11]
VirtaMed UroSim module on TURBT (Figure 1) has four patients with different multiple papillary and solid tumors that offer the trainee the opportunity to perform a complete TURBT procedure in a safe environment without involving patient-related risk. The complexity is added by changing location (from easy to access, to more difficult); the simulators assess complications such as bleeding and perforations. Structured residency programs all over the world are developing ways to train without compromising patient safety especially during the learning curve. Incorporating simulators into surgical training is one way of doing it. Modern urological simulators, e.g., UroSim, Urotrainer, and Simbla have excellent face and content validity. Construct validity for transurethral resection of prostate has been established.[12] However, TURBT is technically a more demanding procedure and requires training of psychomotor skills to a greater degree. Simulator-based training for TURBT is less well-studied, although considering the technically demanding nature of the procedure and risk of complications; simulator training is more suited for TURBT. De Vries et al.[10] previously established validity of Simbla simulator for TURBT. They found similar results in terms of resection time and complications; both of these parameters were significantly more for novices as compared with experts. Neumann et al.[13] have similarly shown that it is possible to assess endoscopic skills within a virtual environment. They also observed that short training improved efficacy and safety of VR-TURBT. However, like us, they also felt that translational value of improved VR performance into ability to perform better in real-life set up needs further work. Our results show that simulator-based training is a valid method for teaching TURBT. Residents are likely to face same problem as in real-life procedures, for example, a few of the residents encountered bladder perforation while resecting tumor at dome, whereas none of the experts had this complication. Additionally, resection time was almost double for novices, which demonstrates construct validity of UroSim for TURBT. Difference in safety parameters show that simulator can challenge the novice with comparable issues during learning curve. Moreover, residents can be gradually trained and evaluated for their ability to avoid complications by showing UroSim end of procedure parameters to them and pointing out the key problems. We have already incorporated simulator-based training in our residency program. We started by creating a module for TURP with the help of medical educationist at our institute. As this study has shown good construct validity for TURBT as well, we are planning to introduce TURBT module in our training program. In future, we can develop similar training module for Holmium Laser Enucleation of prostate. In addition, residents’ performance on simulator can be compared with real-life performance and we can see whether simulator-based training improves psychomotor skills and avoidance of complications in Operation Theater.
There are wide variety and types of simulators available to learn surgery of the bladder and the prostate. These include high- and low-fidelity synthetic models, biological tissue models (both animal and cadaveric human models), and VR computer-based models. In a review of 22 studies in which validation studies were conducted, Khan et al.[14] noted that these models provide supplementary training facility; these models should be compared for cost effectiveness, ability to transfer skills that can be translated into a shorter learning curve.
In a recent review by Goldenberg and Lee [15], they noted that validity of simulation should be based on gathering construct validity evidence according to Mesick’s framework consisting of five components, namely, test content, response process, internal structure, relationship to other variables, and consequences. We found that assessment scores generated by simulator objectively reflect the performance observed in real time by experts suggesting good response process. Additionally, novices consistently had higher resection time and complications compared with experts suggesting reliable internal structure of simulator for TURBT. However, in this cross-sectional study, real-life improvement in skills (Consequences) could not be evaluated. Moreover, residents were not stratified according to their experience owing to limited sample size so relationship to other variables, e.g., experience could only be partially evaluated. As there was little difference in terms of their exposure to performing TURBT, further stratification would probably provide similar results. Moreover, residents from our own institute were more familiar with simulator itself as compared with others, which may have contributed to somewhat better overall group average. The authors consider Mesick’s framework as an essential tool for the program directors and educators in assessing the validity of learning tool.
In conclusion, endoscopic simulator is a useful tool to mimic real-life difficulties encountered during TURBT. Construct validity of UroSim is well-demonstrated by this study so it can be incorporated in urology training for teaching psychomotor skills required for TURBT. Further work is required to provide quality evidence that simulator-based trainees have shorter learning curve and are safer than conventionally trained urologists.
Main Points:
Simulator-based training for TURBT provides a near life-like situation.
It has potential for training residents in stress-free environment.
UroSim® shows good construct validity for training for TURBT.
Supplementary Data
Supplementary Description
The legend for supplemental file is “Scoring system on the UroSim for grading performance.”
Annexure 1.
UroSim® end of procedure parameters example
| Resection: | Measured Goal |
|---|---|
| Amount of prostate resected | >80% |
| Amount of capsule resected | <25% |
| Procedure time | 12:30 |
| Bleeding Control: | |
| Average visibility | >80% |
| Lost blood | <100ml |
| Remaining bleeders | 0 |
| Safety | |
| Cutting while view obscured | 0.20 |
| Tool active while pushing | 0 |
| Cuts into sphincter | 0 |
| Cuts into verumontanum | 0 |
| Under mining bladder neck | 0 |
| Cuts into ureteral orifices | 0 |
| Visualization | |
| Bladder surface | >85% |
| Left ureteral orifice | 0.02 |
| Right ureteral orifice | 0.02 |
Footnotes
You can reach the questionnaire of this article at https://doi.org/10.5152/tud.2020.20099
Ethics Committee Approval: Ethics committee approval was received for this study from the ethics committee of Aga Khan University Hospital (4563-Sur-ERC-2017).
Informed Consent: We obtained written informed consent from residents and consultants who participated in this study.
Peer-review: Externally peer-reviewed.
Author Contributions: Concept – W.A.; Design – A.R., W.A.; Supervision – H.A.; Resources – A.R., W.A.; Materials – A.R., W.A., H.A.; Data Collection and/or Processing – A.R., W.A.; Analysis and/or Interpretation – A.R., W.A., H.A.; Literature Search – AR,WA,HA; Writing Manuscript – A.R., W.A., H.A.; Critical Review – H.A.
Conflict of Interest: The authors have no conflicts of interest to declare.
Financial Disclosure: The authors declared that this study has received no financial support.
References
- 1.Ather MH, Zaidi M. Predicting recurrence and progression in non-muscle-invasive bladder cancer using European organization of research and treatment of cancer risk tables. UrolJ. 2009;6:189–93. [PubMed] [Google Scholar]
- 2.Dawson SL, Cotin S, Meglan D, Shaffer DW, Ferrell MA. Designing a computer-based simulator for interventional cardiology training. Catheter Cardiovasc Inter. 2000;51:522–7. doi: 10.1002/1522-726x(200012)51:4<522::aid-ccd30>3.0.co;2-7. [DOI] [PubMed] [Google Scholar]
- 3.Schout B, Dolmans V, Bemelmans B, Schoot D, Scherpbier A, Hendrikx A. Teaching diagnostic and therapeutic procedures of bladder pathology using a newly developed pig bladder model. J Endourol. 2008;22:2547–54. doi: 10.1089/end.2008.0316. [DOI] [PubMed] [Google Scholar]
- 4.Jha AK, Duncan BW, Bates DW. Simulator-Based Training and Patient Safety. In: Shojania KG, Duncan BW, McDonald KM, Wachter RM, editors. Making health care safer: A critical analysis of patient safety practices, Evidence report/technology assessment no: 43. AHRQ publication 01-E058; Rockville, Md: 2001. 2001, agency for Healtcare Research and Quality. [PMC free article] [PubMed] [Google Scholar]
- 5.Le CQ, Lightner DJ, VanderLei L, Segura JW, Gettman MT. The current role of medical simulation in American urological residency training programs: an assessment by program directors. J Urol. 2007;177:288–91. doi: 10.1016/j.juro.2006.08.106. [DOI] [PubMed] [Google Scholar]
- 6.Buck GH. Development of simulators in medical education. Gesnerus. 1991;48:7–28. [PubMed] [Google Scholar]
- 7.Rassweiler J, Klein J, Teber D, Schulze M, Frede T. Mechanicalsimulators for training for laparoscopic surgery in urology. J Endourol. 2007;21:252–62. doi: 10.1089/end.2007.9983. [DOI] [PubMed] [Google Scholar]
- 8.Knudsen BE, Matsumoto ED, Chew BH, Johnson B, Margulis V, Cadeddu JA, et al. A randomized, controlled, prospective studyvalidating the acquisition of percutaneous renal collecting systemaccess skills using a computer based hybrid virtual reality surgicalsimulator: phase I. J Urol. 2006;176:2173–8. doi: 10.1016/j.juro.2006.07.011. [DOI] [PubMed] [Google Scholar]
- 9.Cronbach LJ, Meehl PE. Construct Validity in Psychological Tests. Psychol Bull. 1955;52:281–302. doi: 10.1037/h0040957. [DOI] [PubMed] [Google Scholar]
- 10.de Vries AH, van Genugten HG, Hendrikx AJ, Koldewijn EL, Schout BM, Tjiam IM, et al. The Simbla TURBT simulator inurological residency training: from needs analysis to validation. JEndourol. 2016;30:580–7. doi: 10.1089/end.2015.0723. [DOI] [PubMed] [Google Scholar]
- 11.Leuschner S, Leuschner M, Kropf S, Niederbichler AD. Non-technical skills training in the operating theatre: a meta-analysis ofpatient outcomes. Surgeon. 2019;17:233–43. doi: 10.1016/j.surge.2018.07.001. [DOI] [PubMed] [Google Scholar]
- 12.Rashid HH, Kowalewski T, Oppenheimer P, Ooms A, Krieger JN, Sweet RM. The virtual reality transurethral prostatic resectiontrainer: evaluation of discriminate validity. J Urol. 2007;177:2283–6. doi: 10.1016/j.juro.2007.01.120. [DOI] [PubMed] [Google Scholar]
- 13.Neumann E, Mayer J, Russo GI, Amend B, Rausch S, Deininger S, et al. Transurethral Resection of Bladder Tumors: Next-generation Virtual Reality Training for Surgeons. Eur Urol Focus. 2019;5:906–11. doi: 10.1016/j.euf.2018.04.011. [DOI] [PubMed] [Google Scholar]
- 14.Khan R, Aydin A, Khan MS, Dasgupta P, Ahmed K. Simulation-based training for prostate surgery. BJU Int. 2015;116:665–74. doi: 10.1111/bju.12721. [DOI] [PubMed] [Google Scholar]
- 15.Goldenberg M, Lee JY. Surgical education, simulation, and simulators-updating the concept of validity. Curr Urol Rep. 2018;19:52. doi: 10.1007/s11934-018-0799-7. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Annexure 1.
UroSim® end of procedure parameters example
| Resection: | Measured Goal |
|---|---|
| Amount of prostate resected | >80% |
| Amount of capsule resected | <25% |
| Procedure time | 12:30 |
| Bleeding Control: | |
| Average visibility | >80% |
| Lost blood | <100ml |
| Remaining bleeders | 0 |
| Safety | |
| Cutting while view obscured | 0.20 |
| Tool active while pushing | 0 |
| Cuts into sphincter | 0 |
| Cuts into verumontanum | 0 |
| Under mining bladder neck | 0 |
| Cuts into ureteral orifices | 0 |
| Visualization | |
| Bladder surface | >85% |
| Left ureteral orifice | 0.02 |
| Right ureteral orifice | 0.02 |