A Simulation-Based, Cognitive Assessment of Resident Decision Making During Complex Urinary Catheterization Scenarios

Abstract

Background

This study explores general surgery residents’ decision making skills in uncommon, complex urinary catheter scenarios.

Methods

40 residents were presented with two scenarios. Scenario A was a male with traumatic urethral injury and scenario B was a male with complete urinary blockage. Residents verbalized whether they would catheterize the patient and described the workup and management of suspected pathologies. Residents’ decision paths were documented and analyzed.

Results

In scenario A, 45% of participants chose to immediately consult Urology. 47.5% named five diagnostic tests to decide if catheterization was safe. In scenario B, 27% chose to catheterize with a 16 French Coude. When faced with catheterization failure, participants randomly upsized or downsized catheters. Chi-square analysis revealed no measurable consensus amongst participants.

Conclusions

Residents need more training in complex decision making for urinary catheterization. The decision trees generated in this study provide a useful blueprint of residents’ learning needs.

Introduction

This study explores how simulation can be used to objectively assess decision making abilities of general surgery residents. While urinary catheterization responsibilities have shifted predominantly to nursing staff, house staff are typically responsible for complex scenarios. Competency in such scenarios, however, requires excellent psychomotor and decision-making skills. Recently, surgical educators have placed greater emphasis on evaluating clinical decision making throughout the continuum of training (1). Decision-making is considered an aspect of the hidden curriculum and could benefit greatly from explicit training and assessment (2). Prior work related to clinical decision making focused on analyzing procedural outcomes rather than examining the factors that may affect decisions before or during a procedure (3, 4, 5). In addition, feedback on decision making commonly occurs in an unstructured environment that is known to be limited in breadth and subject to instructor bias (6).

An example of formative assessment of clinical decision making and judgment is the mock oral examinations. These assessments may be administered to senior level residents multiple times a year (7). Experts recognize, however, that mock oral exams cannot capture all of the facets of real-life clinical decision making and may fail to capture whether residents can competently avoid risks based on complex clinical presentations (7). Additionally, performance on traditional verbal assessments (on rounds, clinic, etc) is not an adequate representation of actual clinical decision making skills as it may be affected by format familiarity and topic predictability (7).

The introduction of simulation significantly changes the way that assessment occurs in surgical training. Many training programs have adopted simulation and have curricula that support structured, objective assessment of trainee performance (8, 9, 10, 11, 12, 13, 14). However, many of the assessment methods that employ hands-on simulation focus on technical skills performance and include procedure-specific and general rating scales, final product analysis, or critical procedure errors (15, 16). As such, there is still a need to better understand and incorporate assessments of surgical and procedural decision making in simulation based evaluations. In our prior work, we have developed and evaluated the utility of combining decision focused assessments with technical skills assessments and providing feedback (6, 17). We have found that simulation can improve upon the process of assessment by incorporating decision making into a comprehensive, structured, and objective evaluation and feedback. While competency in urinary catheterization is expected, it remains unclear whether residents have the skill and knowledge to address complex scenarios (18). The first aim of our study is to see how simulation can be used to objectively assess decision making abilities of general surgery residents. Our second aim is to assess the surgical trainee’s ability to navigate the work up and subsequent-decision making in complex clinical scenarios involving the urinary tract. We hypothesize that residents will make inconsistent decisions for clinical scenarios where they suspect pathological conditions.

Materials and Methods

Setting and Participants

This study was a part of a longitudinal skills decay study primarily interested in skill attrition secondary to nonuse of medical and surgical knowledge. As such, surgical residents in their first year of dedicated research were the primary focus for recruitment for this study. Residents in their second year of research and clinical residents who showed interest were allowed to participate. Recruitment took place primarily by telephone or electronic correspondence to individuals in charge of resident education. All programs that were within reasonable driving distance from the home institution were contacted. The final group included seven Midwest general surgery training programs (19). Data was collected in the summer of 2014 and participation was completed voluntary. The University of Wisconsin Hospitals and Clinics (UWHC) Institutional Review Board gave approval for this study.

Research Protocol

Study participants completed a demographic survey. Following the survey, participants were directed to the urinary catheter station. Residents were first asked to perform sterile, urinary catheterization on three, anatomically different simulation models. After completion of the catheterization task, a researcher presented individual participants with cognitive scenarios regarding complex urinary pathologies. Their verbal responses were audio-recorded. Residents were allowed a total of fifteen minutes to complete this station. Those who did not complete the station were asked to stop and move on to other portions of the longitudinal study.

Participants were given two cognitive scenarios. Scenario A is a 40 year old male presenting as a trauma with blood at the urethral meatus and a non-palpable prostate. Scenario B is a 75 year old male with an enlarged prostate presenting with septic shock secondary to pneumonia, requiring urinary catheterization for monitoring of resuscitation. Participants were asked to verbalize potential difficulties associated with the pathologies and whether or not they would place a catheter in the patient based on those difficulties. The participants who vocalized that they would catheterize were asked to further elaborate on catheter sizes and types. The participants that chose to not place a catheter were asked what steps they would take to address the suspected pathology. Participant’s responses to the cognitive scenarios were recorded and transcribed using Transana (20).

Data Analysis

Using techniques described originally by decision analysis research, decision trees were developed from participant responses after they were coded and graphed (21). Serial and sequential decisions were arranged vertically along branches. Red stop signs indicate decisions to consult Urology. Chi-Square analyses tested the probability of equal frequency of decision type for the first two levels of the decision tree. Only the first two levels of the decision tree were analyzed as the majority of residents were unable to provide third or fourth level decisions on the tree. Comparative analysis was performed at the group level and the post-graduate year (PGY) level to explore potential difference in response. A power analysis was performed to ensure that the study had an adequate number of participants for the given analyses. All analyses were performed using SPSS 23 (22).

Results

Forty general surgery residents (55% female) between their second and fourth post-graduate year (M=2.74, SD=0.92) participated in the decision making exercise. Residents were at various stages in their training: 14.3% were in their first research year, 65.7% in their second research year, 2.9% were in their third year of research and 12.5% were in their clinical year. Per self-report, none of the residents had recent rotations on a Urologic service within one year prior to participating in this study. 100% of residents (N=40) completed the cognitive scenario of the male trauma and 75% (N=30) completed the geriatric male scenario. With power set at 0.8 and alpha 0.05, power analysis determined that our sample size was adequate to detect statistical significance at p<0.05.

There was no group or PGY level consensus amongst participants for the male trauma scenario. Analysis revealed that 55% (N=22) chose to proceed with catheterization, whereas 45% of participants (N=18) chose to place a Urology consult immediately (χ²=0.40, p=0.527). For the group that consulted Urology, 4 participants vocalized that they would also place a suprapubic catheter if the bladder was full (χ²=5.56, p=0.018). For those who chose to catheterize, 47.5% (N=19) vocalized the decision to gather more information by running a diagnostic test. The majority of those who chose to run diagnostic tests vocalized the need for a retrograde urethrogram (32.5%, N=13, χ²=36.55, p<0.001). The remaining tests included a cystogram (N=2.5%), a pyelogram (N=1, 2.5%), a retrograde urethrocystogram (N=2, 5%) and a bladder scan (N=1, 2.5%). Only 7.5% (N=3) of participants chose to place a catheter without first ordering an ancillary test.

If an injury was indicated by a retrograde urethrogram, participants provided very different next steps including, ordering a pyelogram (N=1, 2.5%), placing a suprapubic catheter (N=1, 2.5%), placing a urology consult (N=1, 2.5%) or placing both a urology consult and then a suprapubic catheter. A complete decision tree including all participant responses is represented in Figure 1.

Figure 1.

Decision tree showing the variety of responses received from participants for the male trauma scenario. Red stop signs indicate decisions to consult Urology. * represents a decision that was made in a statistically significant fashion (p < .05).

When evaluating the results for Scenario B (geriatric male), again, there was no consensus amongst the residents at the group or PGY level after the first decision. After reading the clinical scenario 86.7% chose to catheterize the patient. The other 13.3% (N=4) chose to place a Urology consult (χ²=17.07, p<.001). For those who chose to catheterize, the 16 French (Fr) Coude was chosen most often (36.7%, N=11). Additional catheter choices included: 12 or 14 Fr Foley catheter (30%, N=9), 12 or 14 Fr Coude (13.3%, N=4), or 16 Fr Foley (6.7%, N=2) (χ²=14.30, p=.006). One participant reported that they would first order a cystogram followed by placing a 12/14 Fr Coude catheter.

In the case of initial catheter failure, participants randomly chose to upsize or downsize catheters or consult Urology. No particular pattern of catheter choices or consultation with Urology occurred in a statistically significant fashion. A complete decision tree including all participant responses is represented in Figure 2.

Figure 2.

Decision tree showing the variety of responses received from participants for the geriatric septic male. Red stop signs indicate decisions to consult Urology. * represents a decision that was made in a statistically significant fashion (p < .05).

Discussion

The aim of this study was two-fold: 1) to explore decision making using cognitive, simulation-based scenarios and 2) determine if residents could navigate the work up and subsequent-decision making in complex clinical scenarios that may be relatively uncommon yet, still encountered as a general surgeon. Urinary catheterization was chosen as one of the study scenarios as it is regarded as an essential skill listed by the ACS/APDS Resident skills curriculum (18). While it appears, in modern hospital workflow, that nurses are primarily responsible for catheterizations, it is unclear what knowledge a resident has when performing this task. Our motivation for choosing a urethral injury is supported by the fact that general surgery residents may be responsible for working up this injury when presented in a trauma scenario. While a relatively rare event in the trauma setting, it is necessary knowledge for the trauma resident to have (18). We hypothesized that residents would make inconsistent decisions for clinical scenarios where they suspect pathological conditions. Chi-square analyses identified no significant decision tree for either scenario, suggesting that residents are poorly prepared to make decisions on difficult urinary pathology scenarios that they may encounter in practice. Our findings may be due to the limited exposure to these rare clinical events while in practice.

Using Simulation to Assess Decision Making

Currently, mock oral exams appear to be the most utilized method for assessing decision making (7). However, mock orals are limited in providing guided feedback because of the semi unstructured format and administration as a summative evaluation (7). Simulation may provide an alternative method for assessing decision making in a structured environment that could also provide feedback. Prior to doing so, a methodology that can follow decision making should be developed. Our methodology shows how residents’ decision making strategies can be elicited and accurately documented. Transcribing residents’ verbal responses and arranging the responses in a decision tree allows for objective assessment of decision making skills.

The next step would be development of acceptable consensus answers to decision making scenarios. Consensus answers could then be utilized to provide feedback in a guided, unbiased fashion in a simulated environment. We attempt the first steps of this process below in our own assessment. Specifically, we performed a literature search to identify consensus data to develop ideal decision making trees. Alternatively, cognitive task analysis could be used to question experts on their opinion of the correct decision making process (7). Development of the ideal decision making process could then utilized to identify performance weaknesses and guide feedback to improve upon the resident decision making process.

While there are limited studies in assessment and teaching of decision making in a simulated environment, simulation has proven effective at teaching procedural skills, improving performance, and patient safety (23, 24, 25). Future work should expand upon this knowledge to show how teaching in a simulated environment could lead to improved decision making. With establishment of such validity, decision making could then be integrated into teaching curriculums. Doing so would allow for continual assessment and teaching of the hidden curriculum. Ultimately, development of such scenarios and responses in simulation could be used to establish competency standards, a goal long sought after in the field of surgery (6, 17).

Evaluating Resident Performance

There was no consensus decision tree for either presented scenario. Training for urinary and suprapubic placement is listed in the American College of Surgeons and Association for Program Directors in Surgery (ACS/APDS) Resident Skills Curriculum as a Phase 1 Skill (18). As such, our expectation was that junior residents would have an understanding of the indications and workup for placement of both catheter types.

For the male trauma scenario, our inclusion of blood at the urethral meatus and a high riding prostate was to develop high concern for urethral injury (26). The algorithm that we expected from our participants was first a retrograde urethrogram (RUG), followed by a standard catheterization if no injury was found (26, 27, 28). If RUG did show a urethral injury, however, we expected participants to place a suprapubic catheter (27, 28). Nearly half of residents who were faced with the scenario immediately chose to consult Urology, suggesting little confidence to manage the suspected pathology. The inability to navigate this situation may reflect limited experience as our participants were primarily research residents. Furthermore, the inconsistency in responses may reflect complex scenarios residents would not be exposed to until they are senior level.

For the geriatric male, most participants were willing to attempt catheterization in a statistically significant fashion. Most participants chose either a 16 Fr Coude or 12/14 Fr Foley for initial catheterization in a statistically significant fashion, suggesting intention behind this decision. Decisions beyond this level though did not show any particular significance. The statistical significance suggests that perhaps residents do understand where to start with catheterization, however, with failure, there may be little knowledge on proper navigation of this pathology. The expected algorithm for this scenario was that of a standard male catheterization. We expected most participants would first start with either a 16 Fr Foley or Coude catheter. For those who failed with the Foley, we expected they would choose either a 16 Fr Coude or upsize to an 18 Fr Foley or Coude catheter. For those that failed initially with the 16 Fr Coude we similarly expected to upsize either to an 18 Fr Foley or Coude (29, 30). As residents were unable to maintain a decision tree for a standard male catheterization, we believe residents are no longer exposed to this procedure in a clinical setting. Rather, the culture has changed and nurses are more frequently assigned catheterization responsibilities, leaving residents unequipped to handle even simple cases. The limited exposures could then lead to limited interest in learning or practicing even standard urinary catheterization algorithms.

Surgical education continues to evolve to establish novel methods of assessment. The need to measure competency is well recognized (31). In recent years, studies emphasize the importance of measuring competency not only in technical skills, but also decision making skills (6, 17). This study is one of the first that attempts to objectively measure decision making in a way that could be compared to a standardized expert opinion. Doing so in a simulated environment improves upon traditional methods by providing comprehensive assessment, structure, and interventions for immediate and long-term improvement.

While the goals of the study were met, limitations still exist. This study tested its participants solely on decision making. While an appropriate start, it fails to appreciate that surgical performance is often a combination of decision making and technical performance in a simultaneous fashion. A simultaneous assessment would allow observers to see if verbally expressed decisions are appropriately performed. Additionally, simultaneous assessment of both decision making and technical abilities would be a better reflection of in vivo practice. Thus, future work in simulation should aim to develop a more comprehensive assessment.

Conclusion

This study emphasizes two important points. First, simulation provides optimal means to evaluate clinical decision making. Second, residents appear to have little experience and training in catheterization scenarios for which they may be responsible. This information enables opportunities for change in surgical education that would emphasize important clinical topics, continual assessment, and methods for feedback. Doing so will improve trainee performance and ensure competency for independent practice.