Establishing milestones in urology training: A survey of the Canadian Academy of Urological Surgeons

Abstract

Background:

At the current time, technical skills are not directly evaluated by the Royal College of Physicians and Surgeons of Canada (RCPSC) as part of the certification process in urology. Rather, the RCPSC relies on the evaluation of Program Directors to ensure that trainees have acquired the necessary surgical skills.

Methods:

An electronic survey was sent out to the members of the Canadian Academy of Urological Surgeons (CAUS), including the 13 Canadian urology program directors, to assess the teaching and evaluation of technical skills of urology trainees.

Results:

The response rate was 37% (33/89), including 8 of the 13 (62%) Program Directors from across Canada. For the teaching of technical skills, most programs had access to live animal laboratories (69%), dedicated teaching time in simulation (59%) and physical training models (59%). Most relied on voluntary faculty. There was a wide variety of structured evaluations for technical skills used across programs, while 36% of respondents did not use structured evaluations. For trainees with deficiencies in technical skills, 67% of programs offered extra operative time with designated faculty, 26% offered additional simulation focused on the deficiency and 19% offered faculty tutorial sessions.

Conclusion:

Among Canadian urology residency programs, there is considerable variability in the assessment of technical skills of trainees. Standardized objective assessment tools would help ensure that all trainees have acquired adequate surgical proficiency to operate independently.

Introduction

The Royal College of Physicians and Surgeons of Canada (RCPSC) is the governing body responsible for setting the standards for all Canadian urology residency training programs and the requirements trainees must fulfill to receive certification in urology.^1,2 Currently, one of the requirements of certification is that trainees successfully complete the examination set out by the RCPSC.³ This examination provides a good assessment of a trainee’s knowledge base and level of medical expertise in urology.⁴ However, assessing whether trainees have acquired the necessary level of surgical technical proficiency during residency to go into independent practice is not part of the certification process. Rather, the RCPSC relies on Program Directors (PDs) to ensure that the “resident has acquired the competencies of the specialty/subspecialty as prescribed in the Objectives of Training and is competent to practice as a specialist.”⁵

Recent studies have demonstrated that many graduating urology trainees have inadequate experience in some procedures, such as retroperitoneal lymphadenectomy and urinary diversions.^6,7 Furthermore, the introduction of working-hour restrictions for trainees may decrease the opportunity for operative exposure of some of the less common urologic procedures.⁸ Finally, with the increasing use of robotic-assisted laparoscopic surgery, trainees may not be receiving adequate training to allow them to independently perform Class A procedures as described by RCPSC.⁹ The RCPSC relies on the summative evaluation of the PDs to determine whether a trainee has acquired the necessary surgical skills during residency. The aim of the present study was to assess the methods used to teach technical skills, the tools used in the evaluation of technical skills of trainees, and the remedial work for trainees not meeting their training requirements.

Methods

An electronic survey was designed to assess how trainees are evaluated in Canadian urology training programs. The survey (Appendix 1) was sent to all 89 members of the Canadian Academy of Urological Surgeons (CAUS). The survey questions included whether:

• the respondent was a PD;

• a dedicated training curriculum was in place and the number of hours trainees spent in this curriculum;

• a regular curriculum evaluation with trainees was performed;

• the American Urological Association (AUA) In-Service Examination was part of the yearly evaluation processs;

• a passing score was set for the AUA In-Service Examination and the remediation requirements for trainees who failed to meet the passing score;

• there were established methods of teaching technical skills;

• there were objective structured evaluations and remediation requirements for trainees who have an identified deficiency in surgical/operative skills; and

• technical skills should be evaluated as part of the RCPSC certification process.

To ensure confidentiality, responses were submitted anonymously. Fisher’s exact test was used to compare the responses between PDs and non-PDs. A two-tailed p-value less than 0.05 was considered significant.

Results

The response rate was 37% (33/89), including 8 of the 13 (62%) Program Directors from across Canada. Most respondents (33%) had 15 or more trainees in their program, while 27% of respondents had 11 to 14, 18% had 7 to 10, 18% had 4 to 6, and 3% had 3 or less trainees. Average duty hours for trainees ranged from less than 50 hours in 7% of respondents, 51 to 60 hours in 37%, 61 to 70 hours in 33%, 71 to 80 hours in 15%, and over 80 hours in 7%. Almost all of the respondents (96%) reported having a dedicated urology training curriculum in place. The number of hours per week that residents spent in the curriculum varied; 8% of respondents reported 10 hours or more, 21% reported 6 to 9 hours, 63% reported 3 to 5 hours, 4% reported 2 hours, and 4% reported 1 hour. Most respondents (88%) had regular curriculum evaluation by the residents. There were no significant differences in the responses from PDs and non-PDs (p > 0.05) (Table 1).

Table 1.

Reponses to the survey questions by program directors and non-program directors

Question		PDs (n=8)	Non-PDs (n=25)	Total (n=33)	p value
Average duty hours	Less than 50	0/7 (0%)	2/20 (10%)	2/27 (7%)	0.955
	51–60	3/7 (43%)	7/20 (35%)	10/27 (37%)
	61–70	2/7 (29%)	7/20 (35%)	9/27 (33%)
	71–80	1/7 (14%)	3/20 (15%)	4/27 (15%)
	More than 80	1/7 (14%)	1/20 (5%)	2/27 (7%)
Hours per week curriculum	1	1/7 (14%)	0/17 (0%)	1/24 (4%)	0.32
	2	1/7 (14%)	0/17 (0%)	1/24 (4%)
	3–5	4/7 (57%)	11/17 (65%)	15/24 (63%)
	6–9	1/7 (14%)	4/17 (24%)	5/24 (21%)
	10 or more	0/7 (0%)	2/17 (12%)	2/24 (8%)
Passing score on AUA ISE	>30th percentile	0/8 (0%)	1/23 (4%)	1/31 (3%)	0.066
	>40th percentile	2/8 (25%)	1/23 (4%)	3/31 (10%)
	>50th percentile	3/8 (37.5%)	11/23 (47%)	14/31 (45%)
	>60th percentile	0/8 (0%)	9/23 (39%)	9/31 (29%)
	No passing score	1/8 (12.5%)	0/23 (0%)	1/31 (3%)
Remediation for not passing AUA ISE	Required reading	5/8 (62%)	18/25 (72%)	23/33 (70%)	0.461
	Additional exams	0/8 (0%)	10/25 (40%)	10/33 (30%)	0.035
	Faculty tutorials	0/8 (0%)	15/25 (60%)	15/33 (45%)	0.003
	None	2/8 (25%)	0/25 (0%)	2/33 (6%)	0.053
	No answer	1/8 (12.5%)	2/25 (8%)	3/33 (9%)	0.44
Remediation for not achieving technical skills	Extra operative time	5/7 (71%)	13/20 (65%)	18/27 (67%)	0.571
	Additional simulation training	2/7 (29%)	5/20 (25%)	7/27 (26%)	0.367
	Faculty tutorial	0/7 (0%)	5/20 (25%)	5/27 (19%)	0.192
	No answer	1/7 (14%)	2/20 (10%)	3/27 (11%)	0.455
Skills assessment by RCPSC	Yes	2/7 (29%)	11/19 (58%)	13/26 (50%)	0.378

PD: program director; AUA ISE: AUA In-Service Examination; RCPSC: Royal College of Physicians and Surgeons of Canada.

All of the respondents reported using the AUA In-Service Examination (AUA ISE) as part of the yearly trainee evaluation process. However, the score that was considered a pass differed across respondents; 3% of the respondents used above the 30^th percentile as a pass, 10% used above the 40^th percentile, 45% used above the 50^th percentile, 29% above the 60^th percentile, 3% used no passing score (Table 1). Three respondents did not indicate an answer. There were no significant differences among the answers from PDs and non-PDs (Table 1). Remediation requirements for trainees not meeting their respective pass requirement differed among respondents. Five of the eight (62%) PDs used required reading, two did not have any remediation requirement, and one did not select an answer. On the other hand, non-PDs responded with the use of required reading (72%), faculty tutorials (60%) and additional examinations (40%), while 8% did not select an answer (Table 1).

With regards to the teaching of technical skills, 75% of the PDs used live animals, 50% used physical training models and 25% used dedicated teaching time in the simulation centre. Only one PD reported using virtual reality. On the other hand, most non-PDs responded with dedicated teaching time in a simulation centre (71%), live animal laboratory sessions (67%) and physical training models (62%). Other modalities less frequently used were virtual reality (29%), ex-vivo animal preparations (21%) and cadavers (8%), while 17% of non-PDs did not answer (Fig. 1). Only 3% of all respondents reported using paid teaching assistants, while the remaining respondents relied on faculty members, of which 73% were not compensated for their teaching. Although most respondents reported using various methods for the structured evaluation of technical skills, 36% of all respondents did not have any such system in place (Fig. 2). Intra-operative surgical performance devices were only used in 27% of trainee evaluations. Finally, remediation requirements differed across respondents; most PDs employed extra operative time with designated faculty to improve a trainee’s technical skills (Table 1). Half of all respondents felt that it would be important to have technical skills evaluated in the RCPSC certification process.

Fig. 1.

Teaching of technical skills.

Fig. 2.

Structured evaluation of technical skills.

Discussion

RCPSC certification in urology requires the successful completion of five requirements (Table 2).³ Completion of these requirements ensures that graduating trainees have attained the necessary knowledge required to be a practicing urological surgeon. However, a formal evaluation of technical skills is not currently part of the RCPSC certification process. Rather, the RCPSC uses the Final In-Training Evaluation Report (FITER), an evaluation form based on the CanMEDS roles that is completed by PDs, to assess whether a graduating trainee has acquired the necessary procedural skills proficiency during their training.⁵ As part of the FITER, PDs evaluate trainees on a five-point scale (rarely meets, inconsistently meets, generally meets, sometimes meets and consistently meets) on their ability to perform various endoscopic, open and laparoscopic procedures. PDs have a crucial role to ensure that graduating trainees have achieved adequate proficiency in technical skills to qualify for certification. Considering this, a survey was sent to CAUS members to determine what methods were used to teach technical skills, how trainees were evaluated, and what the remediation requirements were for trainees who had not met the predetermined requirements for their level of training.

Table 2.

Royal College of Physicians and Surgeons of Canada certification requirements in urology

Successful completion of a two year Royal College Surgical Foundations curriculum Successful completion of the Royal College Principles of Surgery examination Successful completion of a five year Royal College accredited program in urology Successful completion of at least one scholarly project related to urology, as attested by the Program Director Successful completion of the Royal College examination in urology

Adapted from the RCPSC.³

Although almost all (96%) of the respondents had a dedicated urology curriculum in place, the number of hours that trainees spent per week in the curriculum differed significantly across programs. AUA In-Service examinations were used by all respondents and these scores have been shown to be a good predictor of performance on the RCPSC examination.¹⁰ However, the “passing score” and remediation requirements for those that did not pass differed across programs. The variability between programs is not surprising considering that the RCPSC does not have specific guidelines with regards to in-training assessments; therefore, PDs have developed their own curriculum within the requirements of the RCPSC (Table 2). Although the curricula may differ from program to program, they are all generally adequate to prepare residents for the RCPSC examination, considering that most Canadian graduating trainees have successfully completed this certifying requirement over the past years.

In addition to teaching trainees the knowledge required for clinical practice, the development of technical skills is one of the basic tenets of any surgical training program. The traditional method of learning to perform procedures in the operating room has transitioned for the most part to learning and practicing procedures on various simulators and training models before applying the skills clinically. This allows trainees to learn in a non-stressful environment, practice skills repeatedly and focus on particular performance deficiencies to acquire acceptable, predetermined proficiency, away from the patient thereby reducing the risk of potential complications in clinical practice.¹¹ The results of our survey demonstrate that most trainees are taught procedural skills through simulation centres, live animals and physical training models. It is reassuring that most respondents report the use of training models, as studies have consistently shown that skills training significantly improves trainee operative time and reduces complication rates.^12–14 A year-round surgical skills curriculum was recently developed at the Southern Illinois University’s Division of Urology in response to the introduction of the Accreditation Council on Graduate Medical Education’s competency curriculum, which requires the implementation of measurable teaching methods in technical skills training.¹⁵ A survey of the participants in the training sessions showed a high degree of satisfaction; moreover, an incremental progression in proficiency was observed by the instructors and the participants. It is important to note that when teaching a new technical skill, it has been shown that trainees have significantly better retention and transfer of skills when they are taught consistently over time, rather than longer but less frequent sessions.¹⁶ On the other hand, Stefanidis and colleagues described some challenges that came up during the implementation of a laparoscopic skills curriculum in a general surgery training program.¹⁷ They found that while there was initial enthusiasm about the curriculum from residents, the attendance rates decreased quickly. This was thought to have been mostly due to the lack of supervising personnel and protected training time. With the hiring of a dedicated skills laboratory coordinator and assigning dedicated time for skills training, attendance increased dramatically. The present survey revealed that only 3% of respondents reported the use of paid teaching assistants, while the remaining respondents consistently relied on voluntary faculty members who were not compensated for their teaching. It would be interesting to see what proportion of residents participated regularly in skills training considering that their supervision may not be optimal without a paid dedicated skills laboratory coordinator and that they may not have adequate dedicated time for skills training, with 55% of respondents reporting at least 60 hours of duty per week. This was not assessed in this survey.

The practice of technical skills on a training model will help a trainee improve their technical skills. However, ultimately, their ability to operate on live patients will determine if they are capable of operating without supervision. Since studies have correlated patient outcomes with surgeon volume,^18,19 it is reasonable to assess if trainees are receiving adequate surgical exposure during residency. A study by Chang and colleagues demonstrated that while in some training programs trainees have adequate experience with urinary diversions, over 50% of graduating trainees performed less than 6 continent diversions.⁷ The same group also found that half of the graduating trainees had performed less than 3 retroperitoneal lymph node dissections as the primary surgeon or the first assistant. Although it is certainly difficult to define the number of procedures required for a resident to be considered competent, the consequences of inadequate training may negatively affect patient outcome. Since the RCPSC does not formally evaluate technical skills, it is the responsibility of PDs to determine if a graduating trainee is capable of operating independently. The present survey reveals that while most programs have some method of evaluating technical skills, 36% of respondents report no such structured evaluation. The common evaluation methods were simulation centre performance assessment in surgical skills, checklists, intra-operative assessments and global rating scales. In comparison, a survey of otolaryngology training PDs in the United States found that most respondents used the Objective Structured Assessment of Technical Skills (OSATS) and virtual reality to evaluate surgical skills competency.²⁰ That survey, however, did not report the number of respondents that did not use any objective assessment tool. With objective evaluations, instructors are able to aid in learning through directed constructive feedback, determine the proficiency level that the trainee has reached, and ensure that progress is being appropriately achieved for each trainee. However, it is not known if the evaluation methods being used currently by survey respondents have been validated for their designated use. For trainees not reaching the appropriate predetermined skills milestone, the present survey reveals that extra operative time with designated faculty will be used by respondents to help bring a trainee’s surgical skills up to an acceptable level of proficiency.

Surgical education has shifted towards being more competence-based. This paradigm shift is illustrated by that fact that certification by the American Board of Surgery (ABS) now requires successful completion of the Fundamentals of Laparoscopic Surgery (FLS). FLS, which is based on the McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS), has been extensively validated and has been shown to correlate highly with laparoscopic skills as measured in the operating room. As such, FLS could potentially be used by urology training programs to objectively evaluate basic laparoscopic skills.^21,22 While the RCPSC does not currently evaluate technical skills as part of the certification process, it is possible that they may do so in the future and half of the respondents of the present survey would support this change. If this were to occur, training programs would be much better prepared to ensure the success of their graduating residents if they were able to demonstrate a trainee’s technical proficiency through objective assessment tools already in place. At the present time, validated assessment tools in urology only exist for endourological skills.^23,24 Improving urology training and ensuring that all trainees acquire adequate surgical proficiency will require the further development of validated objective methods of evaluation, as has been done in other surgical fields,^25–29 and implementing these in training programs across Canada.

Although the present study is limited by a low response rate of 37%, most PDs were included. Furthermore, this response rate is comparable to that of other surveys of PDs, where the response rates varied between 27% and 42%.^30,31 In addition, the small sample size may not have allowed differences between PDs and non-PDs to be detected (Type II error). Furthermore, it may be possible that some training programs may be over- or under-represented. However, to ensure anonymity of responses, there were no identifying features in the survey thereby making it impossible to differentiate training programs.

Conclusion

The current certification process from the RCPSC does not include a formal evaluation of technical skills. Instead, formal evaluation is the responsibility of PDs. They must ensure that graduating trainees have reached a level of technical proficiency that is adequate to operate without supervision. A survey of CAUS members demonstrated that there is considerable variability between the different training programs, with 36% of respondents reporting no structured evaluation method in place. The development and implementation of validated objective methods of evaluation will ensure that all graduating trainees in urology receive adequate surgical training for independent practice.

Appendix 1.

Survey

Part 1	Part 2
1. I am a program director Yes No	1. What remediation do you provide for a resident who is identified as having a deficiency in surgical/operative skills? Extra operative time with designated faculty Additional simulation training specifically focused on the technique or procedure of identified deficiency Faculty tutorial sessions related to operative deficiency
2. How many residents do you have in your training program? 1–3 4–6 7–10 11–14 15 or more	2. What are the average duty hours for the residents in your urology training program? Less than 50h 51–60h 61–70h 71–80h More than 80h
3. Do you utilize the In Service Examination as part of your yearly resident evaluation process? Yes No	3. Do you have a dedicated urology training curriculum in place at your institution? Yes No
4. What do you use as the “passing score” you expect the residents in your program to achieve on the ISE? >30th percentile >40th percentile >50th percentile >60th percentile	4. How many hours per week do the residents spend in this curriculum? 1h 2h 3–5h 6–9h 10h or more
5. What remediation do you enlist for residents who do not “pass” the ISE? Required reading Additional examinations Faculty tutorials in the area of deficiency	5. Do you do regular curriculum evaluation with your residents? Yes No
6. Do you have a structured approach to evaluating residents’ technical skills? Yes, but unspecified In simulation centre By OSCE exam stations Using ex vivo animal preparations Using live animals Using structured intraoperative assessment Global rating scale Checklist No	6. Do you think that technical skills evaluation should be part of the Royal College certification process? Yes No
7. How do you teach technical skills? Dedicated teaching time in simulation centre Virtual reality Live animals Physical training models Ex-vivo animal preparations Cadavers
8. Who participates in skills training? Voluntary faculty members Paid teaching assistant
9. Faculty are compensated for teaching? Yes No
10. Do you use an intra-operative assessment device for resident evaluation? Yes No