Testing and Validation of a Low Cost Cystoscopy Teaching Model

Abstract

Objective

The objective of this study was to determine whether the use of a low cost cystoscopy model effectively trains residents in cystourethroscopy and to validate the model as a teaching tool.

Study Design

A randomized, controlled, and evaluator-blinded study was performed. Baseline skills in 29 OB/GYN residents were assessed, using the validated Objective Structured Assessment of Technical Skills (OSATS) checklists for cystourethroscopy, on fresh-frozen cadavers. Residents were randomized to one of two arms, a study arm using the cystoscopy model and a control arm. Repeat OSATS testing was performed.

Results

The study group demonstrated statistically significant decreases in cystoscope assembly time (p=0.004), and increases in task specific checklist and global rating scale scores (p values <0.0001) compared to the controls.

Conclusions

Use of the bladder model exhibited validity in enhancing performance and knowledge of cystourethroscopy among OB/GYN residents.

Introduction

Cystourethroscopy has long been used by gynecologists for diagnostic and operative indications. [1] However, a unified system for training residents in cystourethroscopy and documenting competence has not been developed. Many residents in obstetrics and gynecology are not exposed to enough intra-operative cystoscopy to confidently identify lower urinary tract injury and differentiate normal from abnormal findings. Recently, the American College of Obstetrics and Gynecology released a Committee Opinion which stated that postgraduate education in obstetrics and gynecology should include education in the instrumentation, technique, and evaluation of findings of cystourethroscopy, as well as in the pathophysiology of diseases of the lower urinary tract.[1]

The use of bench models and surgical simulators have gained popularity as efficient methods of providing training outside the operating room as well as providing a means of testing competence; however, cystoscopy models used in resident training can approach a cost of $60,000.00. [2, 3] Using easily obtainable materials, we designed a previously described a low cost cystoscopy model using a balloon, which resembles a normal bladder and urethra complete with ureteral orifices, vessels and different pathologies (Figure 1). [4] The model was originally developed to provide an inexpensive and available method for surgeons in training to learn, perform and practice cystourethroscopy in an effective environment prior to performing this technique in the operating room.

Figure 1.

A video still image showing a quad-screen comparison of cystourethroscopy on a human subject (images 1 & 3) with cystourethroscopy of the balloon model (images 2 & 4).

The objective of this study was to validate that model’s ability to effectively train Obstetrics & Gynecology residents in performing cystourethroscopy.

Materials & Methods

This was a randomized, controlled and evaluator-blinded study, to validate the bladder model as an effective teaching tool. After IRB approval was obtained, twenty-nine Obstetric & Gynecology residents from all training levels (PGY-1 through PGY-4) were recruited from The University of Alabama at Birmingham (UAB). With the assistance of professors from the UAB Department of Anatomy, all resident participants had access to fresh-frozen cadavers with intact urethra, bladder and ureteral orifices on which to perform a pre-determined series of cystoscopy skills. The skills included proper assembly of the cystoscope, proper set-up of additional cystoscopy equipment (distension medium, tubing, camera and light source), as well as performing a series of predetermined bladder and urethra survey tasks, including identification of abnormalities. A single attending physician, with significant experience in cystourethroscopy (RLH), was the blinded examiner, scoring the resident physician’s baseline and follow-up cystoscopic evaluations. Gowns, gloves, and a series of drapes were used to ensure that the examiner remained blinded to the residents’ identities. The examiner was also acoustically blinded using earbuds and an MP3 player, and further visually blinded by performing the evaluation in a space restricting the evaluator’s vision only to the table with the cystoscope, cadaver and camera screen.

Individual scores were assigned using the validated Objective Structured Assessment of Technical Skills (OSATS) checklist and global rating scale (GRS) as originally described by Reznick and colleagues. [5, 6] The Objective Structured Assessment of Technical Skills (OSATS) is made up of a series of elements scoring a participant’s level of operative performance. Several OSATS scoring checklists have been created and validated over the years for basic surgical skills such as knot-tying and laceration repair to more specialized skills such as control of hemorrhage and cystoscopy. [2, 7] The cystoscopy-specific OSATS scoring elements used in this study have been previously evaluated and determined to be reliable and valid for scoring cystoscopy technique. [2]

The first scored evaluation is a task-specific checklist (TSC) (Appendix 1) used to assess the surgeon’s ability to perform a pre-determined series of surgical skills. It consists of a series of yes/no choices and likert scales to assign a specific score. The TSC assesses cystoscope assembly, bladder survey tasks, overall technique and also provides a space for examiner comments and assembly and survey times. Some of the skills the resident needed to demonstrate were: properly assembly of the scope and using the correct lens and sheath size, choosing the correct distension media, observing the ureteral orifices, changing out the lens to properly survey the urethra, and observing abnormalities. Prior to the laboratory session, the authors placed a small amount of mid-urethral sling mesh through the dome of the bladder to create an abnormality.

Since fresh-frozen cadavers were used, certain skills of the validated scoring elements could not be directly assessed, such as observing ureteral efflux. A small quiz (Appendix 2) was incorporated at the end of each skills lab to assess the resident’s knowledge of that portion of the process, and their answers were used to complete the OSATS checklist scoring.

The second OSATS scored evaluation is a 5 point global rating scale (GRS) (Appendix 3) which assigns a number, from “1” to “5” to the surgeon’s performance of the tested skills. The residents’ proficiency in skills ranging from “respect for tissue” and “instrument handling” to “flow of operation” and “knowledge of procedure” were evaluated and scored, with a “1” being assigned to poor performance and a “5” assigned to excellent performance. Numbers “2”, “3”, and “4” are used as intermediate scores. In addition, the evaluator using the global checklist was asked to determine if “Overall, on this task, should this candidate pass or fail?” Overall scores for both the task-specific checklist and the global rating scale were converted to a 0–100 scale.

After the initial cadaver lab and baseline scoring, the residents were randomized to one of two study arms. The first arm (study arm) consisted of residents who were given a 2-hour didactic session on set-up, instrumentation, and proper use of cystourethroscopy using both the described balloon teaching model and Olympus^© and ACMI^© cystoscopes (study arm). Skills taught using the cystoscopy model and didactic session included proper cystoscope assembly and proper set-up of additional equipment (sterile water as correct fluid, fluid source set at correct height, correct set-up of tubing, camera and light source). Bladder surveillance was taught by having the trainee identify a random number of permanent “dots” drawn on the inside surface of the balloon model. The ability to find the correct number of “dots” within the bladder model would be seen as “satisfactory” for completing the didactic session. Identification of simulated ureteral orifices and/or simulated bladder pathologies was also performed. The technique of simulating ureteral orifices, the urethra, as well as other model features has been previously described. [4] The second arm was made up of resident “controls” that did not undergo the 2-hour didactic session utilizing the model.

All residents then repeated the cystoscopy evaluation using the cadavers within 1–2 weeks. The residents were asked to perform the same series of cystoscopic skills as performed at baseline. They were again scored using the validated OSATS scoring elements by the same blinded examiner. Additionally, the general quiz was repeated to assess any increase in general cystoscopy knowledge.

Scores were reported as mean ± standard deviation (SD). Based on prior studies [2] we estimated the within-group standard deviation to be 5 points, and concluded that a sample size of 28 (14 in each group) would provide over 90% power to detect a 10-point improvement (equivalent to 1 letter grade) in test scores. In order to validate the balloon model didactic as an effective teaching tool, pre- and post-scores from each study arm were individually compared using independent t-tests. Paired t-tests assessed score improvements independently for both the study and control arms. The effect of the teaching model on the change in scores was evaluated with analysis of covariance (ANCOVA) models, controlling for baseline scores by year. In addition, construct validity, an assessment of the skills of each trainee and whether scores correlate with the trainee’s PGY level, was evaluated with a 1-way analysis of variance with the year of clinical training serving as the independent variable.

Results

A total of 28/29 residents completed the study, N=14 in each study arm. There was no significant difference in gender composition (p=0.169) or PGY year (p=0.941) in either arm. No difference was noted in assembly time, TSC scores or GRS scores between the groups at baseline; specifically, TSC scores were 59.3 ± 24.5 and 55.7 ± 17.0 for the study arm and control arm respectively, p=0.65; GRS scores were 61.0 ± 20.7 and 64.4 ± 17.8 for the study arm and control arm respectively, p=0.64 (Table 1). There was a trend for difference (p=0.047) in cystourethroscopy survey time at baseline.

Table 1.

Summary of pre and post times and scores in each study arm

Item	Arm	Pre Mean (SD)	p1	Post Mean (SD)	p2	p3	p4
Cystoscope Assembly Time (sec), Mean (SD)	Model	128.8 (56.1)	0.74	54.9 (44.7)	0.001	0.001	0.004
	Control	135.3 (48.0)		126.4 (59.4)		0.534
Cystourethroscopy Survey Time (sec), Mean (SD)	Model	210.4 (43.8)	0.047	200.0 (60.3)	0.030	0.669	0.104
	Control	164.7 (70.3)		142.0 (73.1)		0.182
Task Specific Checklist Score (0–100% scale), Mean (SD)	Model	59.3 (24.5)	0.65	92.9 (8.0)	<0.0001	0.0002	<0.0001
	Control	55.7 (17.0)		52.5 (15.3)		0.537
Global Rating Scale Score (0–100% scale), Mean (SD)	Model	61.0 (20.7)	0.64	87.8 (11.2)	<0.0001	0.0004	<0.0001
	Control	64.4 (17.8)		57.6 (19.2)		0.196
Knowledge Quiz (0–100% scale), Mean (SD)	Model	60.2 (16.8)	0.49	81.8 (12.2)	0.002	0.0004	0.0004
	Control	56.1 (14.2)		62.8 (17.0)		0.067

p¹ - p value reflecting differences in scores between arms at baseline

p² - p value reflecting differences in scores between arms post intervention

p³ –p-value for pre- to post- change (within groups)

p⁴ –p-value reflecting pre- post- change between groups (p delta) controlling for baseline measures and PGY level

After the study arm underwent didactic training using the bladder model and both groups repeated the cadaver skills assessment, a statistically significant decrease in cystoscope assembly time was seen between arms (p=0.004), as well as within the study arm (p=0.001). There was no significant difference in cystoscope assembly time among controls at the time of repeat testing (p=0.534). There was a significant difference in cystourethroscopy survey time (p=0.03) between the 2 arms post didactics, however no difference in survey time within the study arm (p=0.669) or the control arm (0.182) post didactics (Table 1).

The study arm demonstrated significant improvements in the TSC scores (p=0.0002), GRS scores (p=0.0004), as well as in the Knowledge Quiz scores (p=0.0004) after the didactic session using the model. There were no differences in any outcome measure in the control group between the first and second evaluations (Table 1).

An assessment of whether scores correlated with the trainee’s PGY level (construct validity), demonstrated that 3^rd and 4^th year residents scored higher than 1^st and 2^nd year residents, in both arms, on both the task specific checklist scores (p<0.0001) and the global rating scale scores at baseline (p<0.0001). However, after the didactic session using the model, participants in the study arm showed no difference in TSC (p=0.605) or GRS (p=0.473) scores regardless of their PGY level of training. The control arm participants continued to demonstrate higher scores correlating with higher PGY level.

Comment

Our study showed that residents utilizing the cystoscopy teaching model combined with a didactic session, resulted in a significant improvement in overall cystoscopic abilities compared to their baseline evaluations and compared to the control group. In addition, we demonstrated that after the didactic session using the model, all participants in the study arm showed no difference in TSC or GRS scores regardless of their PGY level of training. That is, after the didactic session using the model, study arm PGY4’s and PGY3’s scores were not significantly different than study arm PGY2’s or PGY1’s on cystoscopic skills or cystoscopic knowledge. This demonstrates the model’s ability to rapidly provide lower-level residents with the same cystoscopic skills as their senior counter-parts.

Decreased operative time does not always translate to superior surgical skill; however, the cystoscopy specific OSATS scoring elements evaluate the participant’s cystoscope assembly time as well as cystourethroscopy survey time. Statistically significant decreases in cystoscope assembly time were noted in the study arm after the didactic, while no significant difference was noted in the control group. This would suggest an increased knowledge and ease associated with working with the instruments.

There was no significant difference in cystourethroscopy survey time in the study group after training; however, the study arm did have a higher cystourethroscopy survey time compared to the control arm. In this particular case, this finding appears to reflect essentially no change in the study arm survey time after training, with a trend toward a slight decrease in surveillance time in the control arm. Increased survey time after training could theoretically be due to a more thorough bladder survey among the study arm participants, as well as reflecting for example, an increased time in switching lenses for proper survey of the urethra. However, the study group’s survey time was increased at baseline compared to the controls despite randomization, which most likely happened by chance. We do not suggest that these findings necessarily correlate with increased cystoscopic skill. In fact, cystoscopic skill may have more of a direct correlation with cystoscope assembly time, rather than survey time, as our findings suggest.

Certification of cystoscopic abilities and a more thorough cystoscopic evaluation by gynecologists is important, as the ability to identify lower urinary tract injuries, complications and pathology within the bladder will lead to earlier diagnosis, improved treatment modalities, and hopefully a better understanding of pelvic floor anatomy. In addition, with the increasing placement of mid-urethral slings for stress incontinence and increasing surgical intervention for pelvic organ prolapse repair, the need for cystoscopy will continue to be an important component of residency training in the coming years.

All OB/GYN residency programs do not formally teach cystoscopy to their residents, despite ACOG’s recommendation for training all OB/GYN residents in cystoscopy. In addition to the new ACOG recommendations, there is a growing body of evidence suggesting that not only should cystoscopy be performed at the time of routine hysterectomy [8, 9], but that it is also cost effective [10] if ureteral injury rates exceed 1.5% for abdominal approaches and 2.0% for vaginal or laparoscopic approaches. Because of new recommendations and an increasing call for routine cystoscopy in our field, the teaching of cystoscopy basics needs to be robustly addressed.

One way to address this need is with the use of surgical simulators and bench models, however this is not always feasible because of their cost. We showed that a low-cost alternative can be used effectively. In our study, the residents’ abilities to assemble the cystoscope and perform cystourethroscopy significantly improved after undergoing the bladder model didactic. It should be stressed that the cost to build each of these low-cost bladder models is less than one dollar, [4] thus allowing it to be an alternative for any training program.

The strengths of this study include its randomized, controlled design and it was powered to answer an important question regarding cystoscopic training for OG/GYN residents. Another strength of the study is that the study arm residents and control residents were from the same training site; this is different from previous controlled, OSATS projects where branching out to other sites was performed but seen as a limitation [2]. Additionally, we find strength in having a single, visually and acoustically blinded examiner. This allowed a fair and unbiased scoring system and further strengthens our results and conclusions. The study also lays the groundwork for follow-up studies, which may be used to determine the long-term effect of this training approach.

The main limitation of this study lies in its single-study nature with no long-term re-evaluation. The validity of an instrument shows the extent to which a test or series of tests can measure what the instrument was intended to do. Since validity is normally measured in part by reproducibility, it requires more than a single study. Since this was a single study, construct validity was substituted to demonstrate its validity, which is a practice we see in other studies assessing surgical skills. [2] Despite this limitation, we still feel the balloon model is a valid instrument in resident training.

In conclusion, we showed a statistically significant increase in skill level of resident participants utilizing the balloon model as compared to those that did not randomize to a didactic session using the model. This low-cost cystoscopy model is a valid and effective teaching tool and may improve clinical performance and knowledge of cystourethroscopy among other OB/GYN residency programs.

Appendix 1 Cystoscopy Testing – Task Specific Checklist

Resident Name__________							Date_____
Examiner__________
1. Assembly
○ Able to assemble scope correctly								No		Yes (3)
○ Chooses correct sheath size (17f)								No		Yes (3)
○ Chooses correct Lens (70 deg pref. 30 deg OK)								No		Yes (3)
○ Overall Technique (Efficiency)
poor				1	2	3	4		5	excellent
Time__________
Comments____________________
2. Bladder Survey Task
○ Chooses correct fluid (Sterile Water)								No		Yes (3)
○ Places sheath with obturator first or with H20 running								No		Yes (3)
○ Observes all regions of bladder (Dome, Trigone, L&R sides)								No		Yes (3)
○ Identifies ureteral orifices (U.O.)								No		Yes (3)
○ Attempts to observe urine flow from each UO								No		Yes (3)
a. Identifies lack of flow from one ureter								No		Yes (3)
○ Indicates use of Indigo to clarify urine flow								No		Yes (3)
○ Identifies abnormality and its location								No		Yes (3)
○ Surveys urethra on completion								No		Yes (3)
a. Uses proper lens (0 deg)								No		Yes (3)
3. Overall Technique
poor	1	2	3		4	5	excellent
4. Overall assembly and survey time__________
Comments____________________
Overall Score (out of 49) __________________

Appendix 2 Cystoscopy Accessory Quiz (Completion of OSATS Data Sets)

1. Please choose the preferred sheath size for diagnostic cystoscopy.

   1. 13 fr

   2. 17 fr

   3. 21 fr

   4. 22fr

2. What is an acceptable degree lens to use for a full bladder survey?

   • ___________________________________

3. What is an acceptable degree lens to use for urethroscopy (evaluation of the urethra)?

   • ___________________________________

4. What is the correct fluid media for distending the bladder?

   • ___________________________________

5. Where would you look to identify urine flow into the bladder?

   • ___________________________________

6. How could you identify a ureteral obstruction on one side?

   • ___________________________________

7. If flow is unclear, what agent can you use to increase the visibility of urine flow?

   • ___________________________________

8. Please identify the abnormality in the bladder (you must specifically name the abnormality)

   • ___________________________________

   • Where was the abnormality located?

     1. Dome

     2. Trigone

     3. Patient’s Left

     4. Patient’s Right

9. Have you previously, as a medical student or resident, rotated on a urology or urogynecology service? __________ Yes __________No

Appendix 3 CYSTOSCOPY – GLOBAL RATING SCALE

Date:__________

Resident name:______

R1 R2 R3 R4

	1	2	3	4	5
Respect for tissue	Frequently used unnecessary force on tissue or caused damage by inappropriate use of instruments.		Careful handling of tissue, but occasionally caused inadvertent damage.		Consistently handled tissues appropriately with minimal damage.
Time and motion	Many unnecessary moves.		Efficient time/motion, but some unnecessary moves.		Economy of movement and maximum efficiency.
Instrument handling	Repeatedly makes tentative or awkward moves with instruments.		Competent use of instruments although occasionally appeared stiff or awkward.		Fluid moves with instruments and no awkwardness.
Knowledge of instruments	Frequently asked for the wrong instrument or used an inappropriate instrument.		Knew the names of most instruments and used appropriate instrument for the task.		Obviously familiar with the instruments required and their names.
Use of assistants	Consistently placed assistants poorly or failed to use assistants.		Good use of assistants most of the time.		Strategically used assistant to the best advantage at all times.
Flow of operation and forward planning	Frequently stopped operating or needed to discuss next move.		Demonstrated ability for forward planning with steady progression of operative procedure.		Obviously planned course of operation with effortless flow from one move to the other.
Knowledge of specific procedure	Deficient knowledge. Needed specific instruction at most operative steps.		Knew all important aspects of the operation.		Demonstrated familiarity with all aspects of the operation.

Overall, on this task, should this candidate:

Pass_____

Fail_____

Overall score (out of 35)_______________