The Use of Error Analysis to Assess Resident Performance

Abstract

Background

The aim of this study is to assess validity of a human factors error assessment method for evaluating resident performance during a simulated operative procedure.

Methods

Seven PGY4-5 residents had 30 minutes to complete a simulated laparoscopic ventral hernia (LVH) repair on Day 1 of a national, advanced laparoscopic course. Faculty provided immediate feedback on operative errors and residents participated in a final product analysis of their repairs. Residents then received didactic and hands-on training regarding several advanced laparoscopic procedures during a lecture session and animate lab. On Day 2, residents performed a nonequivalent LVH repair using a simulator. Three investigators reviewed and coded videos of the repairs using previously developed human error classification systems.

Results

Residents committed 121 total errors on Day 1 compared to 146 on Day 2. One of seven residents successfully completed the LVH repair on Day 1 compared to all seven residents on Day 2 (p=.001). The majority of errors (85%) committed on Day 2 were technical and occurred during the last two steps of the procedure. There were significant differences in error type (p=<.001) and level (p=.019) from Day 1 to Day 2. The proportion of omission errors decreased from Day 1 (33%) to Day 2 (14%). In addition, there were more technical and commission errors on Day 2.

Conclusion

The error assessment tool was successful in categorizing performance errors, supporting known-groups validity evidence. Evaluating resident performance through error classification has great potential in facilitating our understanding of operative readiness.

Background

Recent literature has called into question whether graduating general surgery residents are ready for operative independence.^1,2 This question continues to fuel the debate regarding the restructuring of general surgery curricula.³ However, the majority of this data is based on anecdotal evidence and opinion surveys that can be subject to bias.^1–2,4–6 Before we can make crucial decisions regarding changes to the surgical residency curricula, we must develop appropriate assessment tools to explicitly and objectively evaluate residents’ ability to operate independently and define knowledge gaps and skill deficiencies.

Current assessment tools include task-specific and global rating scales for evaluating technical⁷; operative⁸ and non-technical^9,10 skills. The majority of assessment tools for surgical performance are based on procedure time, and subjective and objective assessments of technical skill.¹¹ Although rating scales provide a framework for assessing operative ability, they do not provide specific information regarding the causes and characteristics of failures in performance. Operative errors committed by residents may result from deficits in knowledge and skill; however, errors can also result from lapses in attention and difficulty with perception.¹² Further confounding the issue, the operating room is a complex environment thus, contributing to the difficulty in determining and evaluating the causes of intra-operative errors. An assessment tool that incorporates the evaluation of error characteristics may better help to identify specific deficits and complexities in operative training.

The field of human factors engineering (HFE) has a long history in developing error-based assessments to improve human performance in high-risk and complex industries including aviation¹³ and nuclear energy.¹⁴ HFE is the scientific discipline concerned with understanding interactions among humans and other elements of a system in order to optimize human well-being and overall system performance.¹⁵ Work in this field led to the development of classification frameworks designed to analyze the causes of errors and define generic classifications for human error mechanisms.^12–13,16 The Cognitive Error Taxonomy is a framework that allows for in depth analyses identifying which decision-making steps were made in error or simply bypassed out of habit.^13,16 Using these error based frameworks in a high-risk and complex environment, such as the operating room, allows us to identify how failures are related to cognitive processes.¹²

Simulation allows for error analysis without the cost of morbidity and mortality to patients. Our prior work involves design and fabrication of decision-based operative simulators to analyze intraoperative performance and decision making.¹⁷ This study implements HFE-based error classification methods to investigate the causes and types of errors that trainees commit while also evaluating how they recover from these errors. The study aim is to assess validity of a human factors error assessment method for evaluating resident performance during a simulated operative procedure. Our hypothesis is that the error assessment method will detect changes in performance based on procedure step and participant knowledge thus, supporting known groups validity.

Methods

Setting and Participants

This study was a post-hoc comparative analysis of resident performance during a 2008 Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) course on Advanced Laparoscopic Hernia Surgery. The course occurred over a two-day period and seven senior residents (PGY4-5) participated in the study. The University of Wisconsin Institutional Review Board granted this study exempt status.

Research Protocol

During the course, residents had 30 minutes to complete a simulated laparoscopic ventral hernia (LVH) repair using a previously developed box-trainer style simulator.¹⁷ On Day 1 of the course, the simulator contained a midline 10×10cm ventral hernia located 5 cm above the umbilicus. Surgical faculty acted as surgical assistants during the procedure and provided feedback upon completion. The feedback was not standardized, but catered to individual resident performance during the procedure. As part of the feedback, residents participated in a final product analysis of their repairs with faculty. Residents then received didactic and hands-on training through lectures and an animate lab. On Day 2, residents performed a nonequivalent LVH repair on a similar trainer. The simulator contained a 10×10cm ventral hernia located in the right upper quadrant. Residents also received individualized feedback for this repair. Information on resident demographics and laparoscopic experience were collected. In addition, all simulated procedures were video-recorded using both external and laparoscopic views. Months after the course, a blinded faculty member analyzed and individually graded resident’s final hernia repair for completeness.

Laparoscopic Ventral Hernia (LVH) Simulator

The simulator is a box-style trainer with a plastic base covered by a fabric skin that contains a ventral hernia defect. It was created with the intent to induce decision making while performing a LVH repair. The simulator was designed to look like the abdominal cavity, specifically from the diaphragm to the upper plane of the pelvic cavity. The simulator is fabricated to represent the layers of the abdominal wall including: skin, subcuticular tissue, and peritoneum. Inside, simulated organs including bowel and omentum are layered within the cavity to provide the appearance of gross anatomy. The simulator allows for the use of both laparoscopic and open surgical instruments, except cautery, during each step of the procedure. Examples at each procedure step are shown below in Figure 1.

Figure 1.

Examples of procedure steps performed during the simulated LVH with corresponding images

Post-hoc analysis

Three investigators reviewed the previously recorded LVH repairs from the SAGES course in order to identify, classify and categorize residents’ errors. According to the Bellagio Conference on Human Error, errors were defined as “…something that has been done which was: (i) not intended by the actor, (ii) not desired by a set of rules or an external observer, (iii) that led the task or system outside acceptable limits”.^{18, p.25}

Using previously developed human error classifications, errors were categorized based on type (omission versus commission) and level (cognitive versus technical).¹³ Omission errors were defined as failure to perform a step entirely. Commission errors represented failure to perform a step correctly. For example, failure to measure the hernia defect was categorized as an omission error, whereas measuring the hernia defect with an inaccurate method was categorized as a commission error. Errors in information, diagnosis, and strategy were categorized as cognitive and errors in action, procedure, or mechanics were classified as technical (Table 1). Day 1 and Day 2 comparisons were performed for error type and level as well as hernia repair completion rates. It was expected that many of the rule-based errors, which effect both level and type, would be corrected by day two. For this procedure, rule based errors include those relating to procedure steps including memory and execution.

Table 1.

Error classification definitions and examples adapted from Rasmussen’s Cognitive Error Taxonomy.¹⁶

Error Level		Definition	Example
Cognitive
	Information	Inability to detect cues arising from the change in system state.	Resident continues to use tacker multiple times despite the fact that it is out of tacks. Does not realize that no tacks are going into mesh.
	Diagnostic	Inaccurate diagnosis of the system state on the basis of the information available.	Resident recognizes mesh is slipping and incorrectly associates the issue with the size of the mesh when it is actually due to the suture slipping through the abdomen. Later correctly realizes the extent of the problem and how far it had slipped.
	Strategy	Selection of a strategy that fails to achieve the intended goal.	Resident does not put anchoring sutures in mesh.
Technical
	Mechanical	Occurrence of structural or mechanical failures that do not provide an opportunity for resident intervention.	Suture slips through Endo Close™ device.
	Procedure	Execution of a procedure inconsistent with the strategy selected.	Resident puts an inappropriate number of anchoring sutures into mesh.
	Action	Failure to properly execute the procedure.	Resident makes multiple attempts at snaring the suture with the Endo Close™ device.

Multimedia Video Task Analysis™

Investigators used commercially available software, Multimedia Video Task Analysis™ (MVTA™) (University of Wisconsin-Madison) to code and quantify errors during the repair. MVTA™ is a user interface designed to allow simultaneous video review and data annotation and categorization. Using this program, the investigators captured time of error occurrence, duration or error, and temporal relationship of errors to procedure step (Figure 2).

Figure 2.

Multimedia Video Task Analysis™ (MVTA™) user interface

Data Analysis

Paired samples t-tests were used to compare the number of errors per participant from Day 1 and Day 2. Chi-square tests were used to compare Day 1 and Day 2 for dichotomous data: (1) error types, (2) error levels, and (3) completion rate. All analyses were performed using IBM SPSS Statistics Version 21.0 (IBM Corp, Armonk, NY) and a p-value < .05 was considered significant.

Results

Seven surgery residents completed the full two-day laparoscopic hernia course. Three residents were PGY-4s and four residents were PGY-5s. Six out of seven residents were male. On average, residents reported performing 6.86 (SD=5.01) LVH procedures prior to the course.

Residents committed 121 total errors on Day 1 compared to 146 on Day 2. Errors ranged from 11 to 23 errors per resident on Day 1 compared to 14 to 32 errors per resident on Day 2. One of seven residents successfully completed the LVH repair on Day 1. The six residents that did not complete the LVH repair on Day 1 only reached the mesh orientation step. On Day 2, all seven residents successfully completed the procedure (p=.001).The proportion of omission errors decreased from Day 1 (33%) to Day 2 (14%). There were significant differences in error type (p=<.001) and level (p=.019) from Day 1 to Day 2.

Residents committed fewer cognitive, but more technical errors on Day 2. Details of intraoperative errors on Day 1 and Day 2 are illustrated in Table 2. For example, common cognitive errors on Day 1 included failure to place anchoring sutures into the mesh, failure to properly visualize the hernia defect, and ineffective strategies to roll and insert the mesh. On Day 2, common technical errors included multiple attempts to pass suture into the Endo Close™ device and tacking without visualization. These errors occurred during steps that the majority of residents did not reach on Day 1.

Table 2.

Details of intraoperative errors on Day 1 and Day 2.

	Day 1	Day 2	p-value
Total LVH Completion
No. (%) of residents with complete repairs	1/7 (14%)	7/7 (100%)	.001
Total number of errors	121	146
Mean (SD) Participant Errors	17.3 (4.3)	20.9 (5.8 )	.26
Error Type
No. (%) of omission errors	40 (33%)	20 (14%)	<.001
No. (%) of commission errors	81 (67%)	126 (86%)
Error Level
No. (%) of cognitive errors	45 (37%)	35 (24%)	.019
No. (%) of technical errors	76 (63%)	111 (76%)

Figure 3 demonstrates the proportion of cognitive versus technical errors during each step of the procedure. Residents committed fewer technical errors on Day 2 during the steps previously reached on Day 1. Technical errors increased on Day 2 during the steps that were not previously reached on Day 1. The majority of errors (85%) committed on Day 2 were technical.

Figure 3.

Proportion of cognitive versus technical errors during each step of the procedure

Discussion

Existing tools for assessing surgery resident performance largely focus on assessment of manual techniques and procedure time.¹¹ Most of these assessment tools fail to capture the causes of resident’s performance failures, which may provide information regarding knowledge gaps and skill deficits. HFE error-based assessment methods are designed to evaluate the causes for performance failures in complex environments, such as the operating room. The aim of this study was to assess validity of a human factors error assessment method for evaluating resident performance during a simulated operative procedure, the LVH repair.

In this study, residents performed a simulated LVH repair on Day 1 and Day 2 of a SAGES course. A key finding was that the number of errors increased from Day 1 (N=121) to Day 2 (N=146 total errors), despite the fact that more residents were successful in completing the hernia repair. Further analysis revealed that 85% of the Day 2 errors happened during steps that the majority of residents never reached on Day 1. This finding underscores that operative success does not necessarily equate to less errors and that final product analysis alone may miss opportunities for improvement. This finding also supports our ability to use the error assessment method as a valid measure of performance as there is a strong correlation between error type and procedure step as predicted.

Additional analysis demonstrated that the type of errors committed also changed from Day 1 to Day 2. On Day 2, there were fewer omission errors (failure to perform a step entirely), indicating that residents remembered to perform more steps of the procedure. However, despite improvements in memory, there were more commission errors (failure to perform a step correctly). This pattern, decreased omission errors and increased commission errors, represents a logical progression in knowledge acquisition. On Day 1, residents may be unaware of particular steps of the procedure resulting in omission errors and poorer performance. After feedback and instruction, residents may be cognizant of the procedure steps required, but may not have had the opportunity to actually perform the technical aspects. Resident may have additional commission errors on Day 2 as they are attempting to complete the previously unperformed procedure steps. This progression of events supports our hypothesis that the error assessment method will detect changes in performance based on procedure step and participant knowledge (including memory). These findings support content and known groups validity for the HFE error assessment method.

Lastly, there was also a shift in the level of errors performed by residents. On Day 2, residents committed fewer cognitive errors. While we are unsure of the exact scientific and theoretical basis accounting for this observation it is likely to be multi-factorial. The improvement may reflect that residents were able to incorporate faculty feedback and self-assessment into their subsequent performance. Improved memory, better strategy and incorporation of new rules are all possibilities. It is also possible that the residents were more comfortable with the simulated setting and the simulator. However, it is unlikely that this accounted for all of the improvement as some of the most disruptive cognitive errors related to port-placement strategy (distance from hernia) and placement of anchoring sutures on the mesh. Rasmussen’s skills, rules, knowledge-based error framework also helps to explain some of our observations regarding cognitive errors.¹⁶ A number of the strategy type of cognitive errors resulted from forgetting or lack of prior knowledge of general principles and rules. Rule based errors are easier to correct than some skill and knowledge-based errors. As such, many rule based errors do not require long hours of repetitive practice for improvement. Having an understanding of which deficits require high volume repetition versus short focused interventions may allow us to better design resident training curricula.

Overall, the HFE error-based assessment method provided information about resident performance that was not previously quantifiable with checklists and surveys that focused on technical skill. This study demonstrates the feasibility of applying an error-based assessment method to evaluations of resident performance in simulated environments. In addition, we were able to detect changes in the nature of resident performance that would not have been evident through final product analysis. Moreover, our results show that it is necessary to take a broader look at performance beyond operative case logs, checklists, and surveys when assessing resident performance and evaluating deficits in intra-operative knowledge and skill.

Study limitations include those inherent in qualitative research methods. Deductive reasoning and categorization based on observations is prone to observer bias.¹⁹ In addition, there are limitations in the use of simulation to assess operative performance. Despite the limitations, this study shows promising results and provides a basis for additional research categorizing errors committed during surgical procedures. The focus on error type (commission vs. omission) and error level (cognitive vs technical), helped to support our hypothesis; however, there are a number of additional classifications that may further define the nature of the errors committed. In addition, future work may help us to better understand the underlying reasons for residents’ intraoperative errors. A closer analysis of cognitive errors and how this relates to performance over time is also warranted. Moreover, additional work is needed to evaluate error detection and management. Prior literature has demonstrated that it is not only the number of errors committed that correlates with poor patient outcomes, but also how surgeons detect and manage errors.²⁰

As the debate about the length and structure of surgical residency curricula continues, we must design assessment tools that provide quantifiable data on resident performance. To assess resident readiness, we need to provide trainees with the opportunity to operate independently and commit errors thus, facilitating a better understanding of deficits in knowledge and skills. Our application of HFE-based error assessment in a simulated environment has demonstrated measurable changes in resident performance not previously quantified.