Simulation-based training in urology residency programmes in the USA: Results of a nationwide survey

Abstract

Objective

To evaluate the current usage of simulation in urological education in the USA and the barriers to incorporating a simulation-based educational curriculum, as the shift towards competency-based medical education has necessitated the introduction of simulation for training and assessing both non-technical and technical skills.

Materials and methods

Residency programme directors at Accreditation Council for Graduate Medical Education (ACGME)-accredited urology training programmes in the USA were invited to respond to an anonymous electronic survey. The study evaluated the programme directors’ experiences and opinions for the current usage of existing urology simulators. The survey also elicited receptiveness and the barriers for incorporating simulation-based training curricula within urology training programmes.

Results

In all, 43 completed surveys were received (35% response rate). Amongst responders, 97% (42/43) reported having access to a simulation education centre, and 60% (25/42) have incorporated simulation into their curriculum. A total of 87% (37/43) agreed that there is a role for a standardised simulator training curriculum, and 75% (30/40) agreed that simulators would improve operating room performance. A total of 64% (27/42) agreed that cost was a limiting factor, 12% (5/42) agreed on the cost-effectiveness of simulators, 35% (17/41) agreed there was an increased need for simulator education within work-hour limitations, and 38% (16/42) agreed a simulation programme would reduce patient risks and complications.

Conclusions

The majority of urology programme directors consider that there is a role for incorporating a simulation-based curriculum into urology training. Barriers to implementation include cost burden, need for constant technology updates, need for advanced planning, and willingness of faculty to participate in administration.

Introduction

The 80 h/week work restriction by the Accreditation Council for Graduate Medical Education (ACGME) has posed a challenge to urology resident training. This raised concerns about the shortage of surgical training time, which may affect patient safety [1]. The use of simulation can offer an alternative to obtaining the early part of the learning curve of a surgical skill outside the operating room (OR). Simulation has been used in urological education with success, not only for training but also for assessment during objective structured clinical examinations [1], [2], [3], [4], [5], [6]. A recent study reported the potential for using simulators even for selecting medical students for urology training programmes [7].

A variety of new simulators are continuously being developed to keep pace with both improvements in simulation technology and new advances in surgical practice. However, there is still a lack of a unified approach for validation studies in urology. The most widely used approach is considering validity as multiple types (e.g. construct validity, content validity, concurrent validity, and predictive validity) [8]. Unfortunately, this approach is currently outdated as it relies on the publication of McDougall and colleagues which translates the Standards for Educational and Psychological Testing (SEPT) prior to 1985. Nevertheless, the concept of validity has been changed and the latest SEPT 2014 considered validity as ‘unitary’ and validity evidences are required to either support or refute the ‘hypothesis’ that the data/scores from a certain simulator are ‘valid’ for teaching or assessing certain skill(s) in a target population [9]. Therefore, ‘a call for a shift in validation studies in urologic education’ has recently been published [10]. According to the new taxonomy, five evidences (‘evidence of content, evidence of internal structure, evidence of response processes, evidence of relations with other variables, and evidence of consequences’) should be investigated to either accept or refute the hypothesis that a simulator is valid for training or assessing a desired skill [10].

Several reports have recognised the need for a standardised way to train the next generation of urological surgeons. Pilot studies have shown a centralised simulation programme would improve surgical performance, communication skills, and patient outcomes [11], [12]. Although simulation has become increasingly accessible, urology residency training in the USA has continued to teach technical skills through hands-on experience without a nationally-adopted curriculum for teaching and testing these skills.

The goals of the present study were to evaluate the current usage of simulation, the receptiveness to incorporating a simulation curriculum if one were made available, and the barriers to implementing it into the residency training programmes, through a well-structured survey targeting the opinions of the residency programme directors nationwide. We hypothesised that many training programmes are currently using simulation without consistent direction and with training greatly varying amongst programmes. Furthermore, we hypothesised that programme directors would be receptive to implementing a simulation curriculum if one were developed, although several barriers could impede this process.

Materials and methods

After receiving Institutional Review Board waiver, a two-part questionnaire pertaining to the role of a simulation curriculum in Urology Residency training programmes was generated. The programme directors at each of 122 ACGME acknowledged urology training programmes in the USA were invited to complete the anonymous electronic survey. The programme directors were allotted 5 weeks to complete the questionnaire. Using a modified Dillman’s method, reminder emails, which included the survey link, were sent at weeks 2 and 4 [13].

The first portion of the study included nine questions evaluating the programme director experiences with, current usage of, and opinions of existing urology simulators. The opening questions surveyed the programme directors’ experience with simulators, computer-based task trainers (mannequins), and paid patients (actors, standardised patients) during both medical school and residency. Next, inquiries were made to assess the availability and current usage of simulators during urology residency. This included questions assessing the current usage of endourological, laparoscopic/robotic, as well as patient simulation. Next, opinions were elicited from the programme directors about each of these simulator categories on their usefulness as an educational tool, the realistic representation of technical competence required for each procedure, and the ease of incorporating each simulator type into the urology residency curriculum.

Part two of the questionnaire used a Likert type scale to assess the opinions of the programme directors on a series of statements with the response options of: ‘strongly disagree’, ‘disagree’, ‘neutral’, ‘agree’, and ‘strongly agree’ [14]. The items were designed to elicit responder opinions on the role for standardised simulator training as part of residency, the usefulness of simulators as teaching correct technique prior to entering the OR, the ability of simulator training to shorten the learning curve for surgical techniques, the need for simulator use to learn urology with the work-hour limitations, and the ability of a standardised simulator education programme to improve performance in the OR and to reduce patient risks and complications. There was also a statement assessing whether incorporation of non-technical skills training, such as decision making, communication, and teamwork, into a urology simulation curriculum will improve patient outcomes. The subsequent series of statements were designed to assess the barriers for implementing a standardised urology simulation curriculum for residents.

Supplementary data associated with this article can be found, in the online version, at https://doi.org/10.1016/j.aju.2018.06.003.

Questions in this study were formulated based on barriers defined in the Pentiak et al. [15] study published in 2013. These included barriers of finding time for simulation within the 80-h work week, willingness of faculty to participate, the validity of simulators as an educational tool, cost burden, the constant need for technology upgrades, and the need for advanced planning to obtain supplies required to perform simulations [15]. One item also elicited the receptiveness of incorporating a standardised skill curriculum composed of simulation-based modules into each programme’s training curriculum if one were made available. A final section of the survey had unlimited character space in which the programme directors were able to leave comments (Appendix 1 for survey items).

Supplementary data 1.

mmc1.docx ^{(20.1KB, docx)}

Results

In all, 43 completed surveys were received (35% response rate). All AUA sections were represented in the results. The majority of respondents did not use simulators or paid actors during medical school or residency to learn either basic sciences (>70%) or for testing purposes (>67%). However, 54% of programme directors had experience with using paid actors to learn clinical skill during medical school. In all, 40% of responders used simulators as part of their clinical skills training during residency, whilst mannequin trainers were used by 12 (27.91%) respondents during medical school and 11 (25.58%) respondents during residency.

Amongst responders, 97% reported having access to a simulation education centre for their urology residents. Of those, 60% had incorporated simulation-based education into their residency training programmes, whilst 40% had access to trainers, but did not have any required time spent on simulation. The most commonly used simulators were laparoscopic/robotic and TURP trainers (95% and 26%, respectively). Laparoscopic/robotic and percutaneous renal access simulators were thought to be the most useful, realistic and easily incorporated into residency training (Table 1, Table 2).

Table 1.

Programme directors’ opinions about usage and availability of simulators.

Usage/availability of simulators	% programme directors who agreed
Residents have access to simulation education centre	97
Incorporated required simulation into training programme	58
Have access to trainers, but no required simulation training	39

Table 2.

Programme directors’ opinions on varying types of simulators.

	Laparoscopic/robotic	Cystoscopy	TURP	URS	PCA	Patient-simulation training
% responders training with this tool	95	12	26	14	12	12
% responders agree useful tool	100	49	70	68	90	59
% responders agree realistic	78	47	41	53	63	48
% responders agree easily incorporated into training	93	69	76	73	83	73

PCA, percutaneous renal access; URS, ureteroscopy.

In all, 88% of responders agreed that there is a role for a standardised simulation-based training curriculum, and that simulators are a useful tool for teaching surgical techniques to improve performance in the OR. A simulation-based educational curriculum is thought to potentially shorten the resident learning curve and improve surgical performance by 61% and 75% of respondents, respectively. In all, 57% agreed non-technical skills simulation would improve patient outcomes. There were mixed views on the prospect of whether a simulation programme would lead to decreased patient risk and complications with 38% agreeing, 14% disagreeing, and 48% remaining neutral (Table 3).

Table 3.

Programme directors’ opinions about benefits of simulators.

Benefits of simulators	% of programme directors who agreed
Useful for teaching surgical techniques prior to OR	88
Improve surgical performance in the OR	75
Shorten the resident learning curve for surgical skills	62
Non-technical skills simulation will improve patient outcomes	57
Work-hour restrictions increase the need for simulators	41
Reduce patient risks and complications	38
Cost-effective method of learning	12

In all, 65% of the programme directors reported cost as a limiting factor for using simulators in the training programme, whilst only 12% thought simulators were a cost-effective means of training. In all, 50% agreed that simulators have been validated as an educational tool, 55% agreed faculty that would be willing to participate in administrating the simulation curriculum, and 35% agreed that there is an increased need for simulator education with the 80-h/week work limitations. In all, 68% reported the need for constant upgrades as a limiting factor and 60% thought the need for advanced planning was a limiting factor for incorporating a simulation-based educational curriculum into urology residency training. Of the responding programme directors, 81% agreed that their residency programme would be in favour of integrating a simulation-based educational curriculum and 57% agreed that this could be easily incorporated into residency training (Table 4).

Table 4.

Programme directors’ opinions about barriers to implementing simulator training.

Barriers to implementing simulator training	% of programme directors who agreed
Requirement for constant technology upgrade	68
Cost	65
Need for advanced planning to obtain the necessary supplies	60
Difficult to find time within work hour restrictions	35
Can NOT be easily incorporated into residency training	21
Faculty unwilling to participate	21
Not validated as an educational tool	19

Discussion

With the shift of urological interventions toward minimally invasive techniques, simulation-based training is thought to be an ideal way to develop the unique skill set required to perform minimally invasive procedures in a controlled arena. Simulator training is a great tool for surgeons to develop technical skills in a practice environment such that there is no patient risk, the procedure can be repeated until that particular skill is mastered, and it provides a basis to measure performance level and improvement. It has been shown that skills acquired in simulation-based training might translate to improved performance in the OR [16]. Thus, the development of a standardised simulation-based urology training curriculum would provide consistency in quality of training across the board.

The present results suggest that most residency training urology programmes have access to a simulation centre. All responders with access to simulation have been using it in some way to train urology residents. When compared to a similar study published by Le et al. [17] in 2007, the availability of simulators has increased 20% for laparoscopy and 18% for TURP. Furthermore, the majority of responders agreed that simulators are a useful educational tool, particularly the laparoscopic and robotic trainers. They also reported that the benefits of simulation-based training include usefulness for teaching surgical techniques prior to the OR, improvement in surgical performance in the OR, and shortening of resident learning curve for surgical skills. Nevertheless, fewer programme directors now agree that simulation-based education reduces patient risk and complications as a benefit compared to Le et al. [17] study (38% vs 96%). Unfortunately, there is paucity of studies investigating the outcome (e.g. evidence of consequences) of simulation-based training in terms of improving performance in the OR and enhancing patient safety. A recent study applied the new validity concept for revising the current literature providing validity evidences for simulators used for assessment in urology [18]. Out of the five validity evidences that are required for validation [10], the authors found that only two (the content evidence and the relations with other variables evidence) are widely used. Nevertheless, the other three evidences (internal structure evidence, the response processes evidence, and consequences evidence) were largely underrepresented [18]. The consequences evidence is particularly important, as it provides a proof for the transfer of skills obtained from training in the simulation laboratory to the OR and it is concerned with patient outcomes, hence patient and community safety. In addition, few studies were found providing part of the consequences evidence in endourological and robotic urological surgery [19], [20], [21]. Furthermore, superiority of one simulator over another has not been recognised due to an absence of randomised controlled trials [22].

Whilst a simulation-based curriculum has the potential to improve residency training, there are multiple barriers to implementing a simulation-based training programme. In the present study, programme directors agreed that the need for constant upgrades, cost, and the need for advanced planning are barriers to implementing a simulation-training curriculum at their institution. However, the use of low-fidelity simulators, such as box trainers for laparoscopy, does not need advanced planning, constant upgrading, or high cost. Nevertheless, some high-fidelity and virtual-reality simulators are very expensive. Therefore, it depends on how far a programme director needs to go during preparation of his/her simulation centre. Comparing with the Le et al. [17] study, fewer responders now agree that the urology faculty would participate/support this type of training (52% vs 95%), that simulators are a cost-effective method of learning (12% vs 63%), and that there is a need for simulation with the 80-h work week restrictions (41% vs 77%). There is a difference of opinions as to whether residents will have time to participate in simulation within the 80-h work week restriction. However, working hours should not limit incorporating simulators or other tools to residency training because simulation laboratories enjoy the advantage of providing training before or after work hours or even during vacations. These are different from animal laboratories, which need certain set-up prior to starting training and there is no flexibility regarding the time and the place of training. The bulk of responders agreed that simulation has been validated as an educational tool and simulation-based training could be easily incorporated into the training curricula. When compared to the Le et al. [17] survey, more responders agreed simulation being validated as an educational tool (80% vs 60%) and less agree that simulators can NOT be easily incorporated into the curriculum (21% vs 26%). Thus, these are not considered large barriers to instituting a urology simulation-education programme. A comparison between some shared parameters in the 2007 survey and the present survey is shown in Fig. 1. It is worth noting that there was a major decrease in the percentage of programme directors who believed that simulation would reduce patient risk and complications between the Le et al. [17] study I 2007 (96%) and the present study (38%). This might be due to the paucity of randomised studies on consequences of simulation training in terms of reducing the patient risk and complications.

Fig. 1.

Comparison between some shared parameters in the 2007 survey and the present survey.

Simulation-based training is a great option for teaching surgical skills without harming patients. There is evidence supporting its use during the initial portion of the surgical learning curve, and randomised trials have shown that simulation can work. We have shown that program directors believe that there is a role for simulator education in urology training. Although there are barriers to implementing simulation training at each institution, most responders thought a simulation-based curriculum could easily be incorporated into residency training and were receptive to adopting such a curriculum if one was created. There is now a need for optimising simulation training in order to create an effective standardised simulation-based training programme that can be integrated into urology residency training programmes across the nation.

One limitation of the present study is the overall response rate of 35% to the questionnaire; thus, the responses may not be reflective of all programme directors’ opinions. However, the response rate in the present study was higher than the response rate in Le et al. [17] study (34.5%). Additionally, response bias is possible, as responders could be more likely to be interested in and supportive of simulation than non-responders. Another limitation is that questions were written in such a way as to evaluate programme directors’ opinions of simulator trainers as a whole, and there was no investigation neither into specific simulation models such as virtual reality, low-fidelity, or high-fidelity simulators or the impact of each of these models on the quality of training. With this, responses echo opinions on a large variety of simulators available at different institutions, as each programme director will have had a different simulation experience and availability prior to completing the present survey.

Conclusions

The majority of programme directors believe that there is a role for incorporating a simulation-based curriculum into urology training and are receptive to adopting this at their institutions, if such a training programme was created. Barriers to implementing a simulation curriculum include: cost burden, the need for constant technology updates, the need for advanced planning, and the willingness of the faculty to participate in administration. Future work in this area could be accelerated by the development of a standardised simulation-based training curriculum to train the next generation of urological surgeons. We encourage urology training programmes in the Arab world to start incorporating simulation into their curricula.

Education/Training in Urology