The effect of video review of resident laparoscopic surgical skills measured by self- and external assessment

Abstract

Background

Video review of surgical skills is an educational modality that allows trainees to reflect on self-performance. The purpose of this study was to determine whether resident and attending assessments of a resident’s laparoscopic performance differ and whether video review changes assessments.

Methods

Third-year surgery residents were invited to participate. Elective laparoscopic procedures were video-recorded. The Global Operative Assessment of Laparoscopic Skills evaluation was completed immediately after the procedure and again 7–10 days later by both resident and attending. Scores were compared using t- tests.

Results

Nine residents participated and 76 video reviews were completed. Residents scored themselves significantly lower than the faculty scores both prior to and after video review. Resident scores did not change significantly after video review.

Conclusions

Attending and resident self-assessment of laparoscopic skills differ and subsequent video review doesn’t significantly affect GOALS scores. Further studies should evaluate the impact of video review combined with verbal feedback on skill acquisition and assessment.

Introduction

Given the duty hour restrictions and the increasing reliance on simulation-based and self-directed learning (1), self-regulation becomes an increasingly important skill (2,3). Surgical trainees tend to overestimate their operative skills; confidence is inversely related to competence (2,4). This incongruence may have deleterious effects on patient care in the setting of surgical residency training. In the absence of feedback and debriefing as external sources of performance validation, intraoperative resident teaching may fail to convey important learning points or to promote self-regulation.

Video review has been shown to enhance performance in a number of disciplines, including law enforcement, aviation, and athletics (5–7). Prior studies have evaluated the utility of video recordings for subsequent evaluation and debriefing (8–11) and for improving self-assessment (12–14), but the vast majority studied simulated surgical scenarios or physical examination skills rather than operative videos from patient procedures. Video-based coaching can be a useful tool for providing individualized, targeted feedback for trainees (15), and reinforces intraoperative teaching points (16) in a less stressful environment and has also been shown to improve skill acquisition (17). Nevertheless, video review of unedited surgical procedures is labor-intensive and time-consuming. Despite the educational value of reviewing and critiquing videos of one’s operative performance, only 5% of residents reported that this educational modality was offered at their institution (18).

Video review and debriefing of resident intraoperative performance has been a fundamental component of the Minimally Invasive Bariatric and General Surgery rotation for third-year general surgery residents at the University of Pittsburgh Medical Center since 2004. In our study of third year general surgery residents performing a laparoscopic jejunojejunostomy, half of the residents underwent video debriefing. While there was no significant difference between the debriefed and non-debriefed group in terms of knot tying time, anastomosis time, and minor errors, there was a significant reduction in adverse operative events from technical errors in the debriefed group (10).

The present study is part of an ongoing educational intervention implemented for residents on their Minimally Invasive Bariatric and General Surgery rotation since 2011 at our institution. In response to resident feedback that objective evaluation of surgical skills was needed, we implemented postoperative assessments by both resident and faculty using Global Operative Assessment of Laparoscopic Skills (GOALS) before and after video review in 2011. In 2012, the American Board of Surgery (ABS) approved a requirement for structured assessment of operative skills of the graduating chief residents and the ABS evaluation form was used beginning in 2013.

The purpose of this study is to determine the effect of video review on resident and attending assessments of a resident’s laparoscopic surgical performance. Based on our prior experience, we have observed that video review helps the residents recall and learn from operative teaching points or events that otherwise would have been rapidly forgotten. We hypothesized that the resident and faculty assessments of the resident’s performance would differ immediately after the procedure and that subsequent video review would alter the self- and external assessment of the procedure.

Material and Methods

In this Institutional Board Review-approved study, nine third-year general surgical residents from the University of Pittsburgh Medical Center participated in the study during their 7-week Minimally Invasive Surgery rotation operating with a single surgical attending between February 2011 and August 2012.

Their performance of elective laparoscopic surgical cases, including gastric bypass, adjustable gastric banding, cholecystectomy, gastric plication, and ventral hernia repair were routinely video-recorded. Immediately following the procedure, both the resident and supervising attending evaluated the procedures using a modification of the previously validated GOALS tool (19) based on their recollection of the surgical performance. This tool rates seven different domains on a 5-point Likert scale, including depth perception, bimanual dexterity, efficiency, tissue handling, autonomy, suturing and level of difficulty. It has been shown to have construct validity and a high level of interrater reliability for laparoscopic cholecystectomy (19). Both residents and faculty received an orientation briefing them on the GOALS tool prior to the first surgical procedure performed on the Minimally Invasive Surgery service. Residents subsequently reviewed the video of the procedure with the supervising surgical attending without verbal feedback about the residents’ performance between 7 and 10 days after the initial procedure and both completed the GOALS form again based on the video review. The video was fast forwarded through segments in which there was inactivity of the surgical instruments. Each resident and attending completed the GOALS assessment simultaneously. In order to avoid influencing each other’s response, the resident and attending did not discuss their assessment of the resident’s operative performance of the case prior to completing the GOALS form. Neither attending nor residents had access to their prior evaluation or to each other’s assessment forms until after completion of the second evaluation.

The mean and standard deviations of the residents’ and faculty’s pre- and post-video review scores for each domain were computed and compared using a t-test. In addition, the mean and standard deviations of the pre-video and post-video scores for faculty and residents were computed and compared using a t-test. The p-values for individual procedures were calculated using the Wilcoxon signed rank test. A p-value of 0.05 was deemed significant.

Results

A total of 118 laparoscopic procedures were performed during the study period. Complete pre- and post-operative assessment data and videos were available for analysis in 76 cases. Procedures included gastric pouch creation (n=17) and jejunojejunostomy (n=18) as part of a laparoscopic Roux-en-Y gastric bypass, cholecystectomy (n=18), adjustable gastric band placement (n=10), gastric plication (n=7) and ventral hernia repair (n=6). Average length of review for each video was 1 hour.

In all domains evaluated, resident scores did not change significantly after video review when comparing pre-video and post-video scores (Table 1). After video review, faculty assessment of bimanual dexterity, efficiency and autonomy decreased, while scores for tissue handling increased (Table 2). In all categories, residents scored themselves lower than the faculty scores both prior to and after video review (Tables 3 and 4).

Table 1.

Resident scores before and after video review

All Procedures	No. cases	Pre-video review	Post- video review	p-value
DEPTH PERCEPTION	76	3.04±0.86	3.08±0.92	0.6341
BIMANUAL DEXTERITY	76	2.75±0.66	2.75±0.83	>0.9999
EFFICIENCY	76	2.93±0.77	2.99±0.84	0.4533
TISSUE HANDLING	76	3.01±0.81	3.07±0.79	0.52
AUTONOMY	76	2.92±0.74	2.93±0.82	0.8541
LEVEL OF DIFFICULTY	76	2.57±0.96	2.67±1.00	0.1716
SUTURING	25	3.16±0.85	3.00±0.76	0.1615

^*p < 0.05

^**p < 0.001

Table 2.

Attending scores before and after video review

All Procedures	No. cases	Pre-video review	Post- video review	p-value
DEPTH PERCEPTION	76	3.72±0.81	3.63±0.67	0.2887
BIMANUAL DEXTERITY	76	3.53±0.58	3.38±0.56	0.0268*
EFFICIENCY	76	3.36±0.60	3.18±0.45	0.0114*
TISSUE HANDLING	76	3.86±0.83	4.05±0.80	0.0046**
AUTONOMY	76	3.53±0.70	3.34±0.60	0.0123*
LEVEL OF DIFFICULTY	76	3.59±1.14	3.51±1.19	0.4264
SUTURING	25	3.52±0.71	3.56±0.65	0.3273

^*p < 0.05

^**p < 0.001;

Table 3.

Resident vs. attending scores before video review

		Resident	Attending
All Procedures	No. cases	Pre-video review	Pre-video review	p-value
DEPTH PERCEPTION	76	3.04±0.86	3.72±0.81	<0.0001**
BIMANUAL DEXTERITY	76	2.75±0.66	3.53±0.58	<0.0001**
EFFICIENCY	76	2.93±0.77	3.36±0.60	<0.0001**
TISSUE HANDLING	76	3.01±0.81	3.86±0.83	<0.0001**
AUTONOMY	76	2.92±0.74	3.53±0.70	<0.0001**
LEVEL OF DIFFICULTY	76	2.57±0.96	3.59±1.14	<0.0001**
SUTURING	25	3.16±0.85	3.52±0.71	0.0471*

^*p < 0.05

^**p < 0.001

Table 4.

Resident vs. attending scores after video review

		Resident	Attending
All Procedures	No. cases	Post-video review	Post-video review	p-value
		(b)	(d)
DEPTH PERCEPTION	76	3.08±0.92	3.63±0.67	<0.0001**
BIMANUAL DEXTERITY	76	2.75±0.83	3.38±0.56	<0.0001**
EFFICIENCY	76	2.99±0.84	3.18±0.45	0.0462*
TISSUE HANDLING	76	3.07±0.79	4.05±0.80	<0.0001**
AUTONOMY	76	2.93±0.82	3.34±0.60	<0.0001**
LEVEL OF DIFFICULTY	76	2.67±1.00	3.51±1.19	<0.0001**
SUTURING	25	3.00±0.76	3.56±0.65	0.0006**

^*p < 0.05

^**p < 0.001

Discussion

Intraoperative teaching is a fundamental component of surgical residency education for skill acquisition, communication abilities and decision-making. Several changes in residency training have reduced resident access to these educational opportunities. The emergence of complex laparoscopic and robotic procedures has shifted the operative experience from junior residents to higher level trainees (20). Duty hour restrictions have raised concerns about a decrease in resident operative case volume (21,22). Intraoperative resident learning is further limited by the increasing focus on patient safety and the potential for prolongation of operating times because of resident teaching (23). These challenges indicate the need for innovative educational strategies to maximize learning of operative skills.

Despite the importance of perioperative teaching, residents and faculty do not necessarily agree upon the key educational objectives. A survey of surgery residents and faculty demonstrated a significant difference in the perceived resident learning needs (24). Jensen showed that resident’s and attending’s perception of quality and timing of feedback were significantly different (25). There is clearly a need to provide residents with data and objective feedback about their operative skills, and residents need to learn to reflect on their prior performance in order to identify areas needing improvement.

Video review is an educational intervention that reinforces intraoperative teaching points in a less stressful setting than the operating room. The role of video review on surgical resident laparoscopic skill acquisition has been previously studied using animal models or laboratory skills. In a study of 26 surgical residents performing a porcine laparoscopic Nissen fundoplication that inter-rater reliability significantly increased between the residents’ and expert’s self-assessment after residents watched a video of their procedure (11). Jamshidi et al evaluated 14 general surgery residents who were video-recorded performing laparoscopic suturing (14). Knot-tying scores and times for task completion improved significantly in the group that watched the videos of their own performance. Bonrath evaluated individualized coaching through feedback, debriefing and behavioral modeling and showed enhanced skill acquisition of surgical skills (17).

Prior studies evaluating video review have focused on testing physical examination skills or trainees’ technical skills performing discrete tasks rather than complex procedural skills. In contrast, the present study utilized full-length surgical videos of intraoperative performance. To our knowledge, there have not been prior studies in the literature evaluating the effect of video review on resident self-assessment of surgical skill performance in the operating room through full-length operative videos.

Given the findings from prior studies demonstrating the disparity between resident and faculty perceptions of intraoperative teaching, we expected to find a discordance between resident and faculty in their assessments of operative performance. The self-evaluation scores were lower than attending scores in all categories examined and video review did not modify these scores significantly. These results suggest that evaluation of surgical performance is complex. Furthermore, because residents in our program rarely operate with peers in the same postgraduate level, they may be comparing their own performance to a faculty performance. In contrast, the faculty was evaluating residents in comparison to their own peer group. This may explain the consistently incongruent scores between residents and faculty.

Level of experience may have contributed to the disparity in scores as well. Third year residents at our institution have minimal experience with video review of their own performance and are likely not focusing on the same aspects of the performance as the faculty. Residents may benefit from concurrent verbal feedback from expert faculty during video review in order to enhance their assessment skills and intraoperative decision-making and to recognize strengths and deficiencies in surgical skills. Hamad et al. demonstrated that video review combined with verbal debriefing significantly reduced adverse events in residents’ performance of a laparoscopic jejunojejunostomy (10). Farquharson studied 48 students in a prospective randomized trial comparing verbal feedback and video review plus verbal feedback for suturing in the laboratory setting (26). The students who received the video review and verbal feedback performed better than those who received only verbal feedback. The authors suggested that the combination of video review and verbal feedback might improve surgical skill acquisition.

Self-monitoring is critically important for self-directed learning. In contrast to prior research, residents in our study consistently evaluated themselves lower than faculty. The difference in level of experience may play a role in self-monitoring. Hu et al. video-recorded 23 medical students and interns performing suturing and knot tying (27). Videos were evaluated by the trainees and by blinded faculty. The participants’ self-assessment scores were significantly higher than the faculty assessments, suggesting that trainees tend to overestimate their skill performance. In the realm of communication skills, video review of self performance and peer feedback may be useful to implement reflective practice activities and improve self-evaluation (28). Future research in video review of technical skills may befit from this approach.

The feedback from the surgical residents participating in video review has been uniformly positive. They have described the value of video review in identifying technical skills needing improvement, promoting instrument and tissue handling, improving operative efficiency, and developing operative strategies and intraoperative decision making, most of which were cited as deficiencies in Rose’s study of resident and faculty surgeons’ perceptions of intraoperative teaching (29). Faculty development to train others to engage in video review and resident evaluation of procedural skills would increase the frequency of objective feedback.

The study has several limitations, including the heterogeneity in the types of cases that were performed and the number of specific cases of each type performed by the participating residents. This is due to the inherent variability in caseloads among surgical residents on a given rotation. Although there is variability in the case complexity the GOALS tool measures general aspects of technical skills and the variables that it evaluates (such as depth perception, bimanual dexterity, etc.) are independent of specific cases and have been validated previously. The time interval between operative case and video review also varied; this may have had an impact on the second assessment. Average review time was approximately 1 hour, which could influence assessment because of evaluation fatigue. Future studies should determine how to incorporate intraoperative surgical videos into resident education in a time-efficient manner. Furthermore, use of a second blinded evaluator might have reduced bias.

Given the disparity between residents’ perception of feedback received and the faculty’s perception of quality of feedback given to trainees, the optimal methods and type of feedback remain to be defined. Further research is needed to clarify the effect of simultaneous discussion of the procedure between resident and faculty at the time of video review on resident self-assessment skills. Faculty development for assessment of resident skills would be valuable for the surgical faculty who spend most of their time educating residents in the operating room. The role of self-assessment on surgical skill acquisition is an important question and warrants further investigation.

Conclusions

With the recent ABS requirements for structured assessment of surgical skills in residency training, evaluations of residents’ surgical technical proficiency are now routinely being implemented. Given the stability of resident self-evaluation scores after video review, future studies are needed to determine the optimal timing and method of feedback administration for promotion of self-monitoring skills.