Effectiveness of repeated video feedback in the acquisition of a surgical technical skill

Abstract

Background

Previously we evaluated the effectiveness of videotaped feedback as a teaching tool for surgical residents using 3 different core skills in the field of orthopedics. Our results revealed no significant differences in technical skill among surgical residents who were exposed to videotaped feedback and those who were not. Several limitations in the study were identified, including the presence of only a single exposure to videotaped feedback. In the present study we included repeated exposures to determine if skill acquisition was enhanced.

Methods

Thirty-three surgical residents were randomly assigned to receive either no videotaped feedback (control group) or repeated video feedback over 3 consecutive weekly practice sessions on a vascular anastomosis bench model. On the fourth week, vascular surgeons blinded to the 2 groups evaluated the residents on the same vascular anastomosis task using a global rating scale and technical checklist forms.

Results

Twenty-six (79%) of the residents participated. Independent t tests comparing the scores of each of the 2 different measures between the control group and repeated videotaped feedback group revealed no statistically significant differences.

Conclusions

There was no significant difference in the performance of a vascular anastomosis in a bench-training model between residents who were exposed to video feedback over several practice sessions and those who received no video feedback.

The benefits of a bench-training model as a teaching modality compared with the operating room environment include the ability to repeatedly practise procedures and reuse materials, lower costs owing to the elimination of live patients or animals and the less stressful environment of the surgical skills labarotory.^{1,2,3,4,5,6,7,8,9} To date, our review of the literature revealed 1 study in which videotaped feedback was used with surgical residents.¹⁰ To further understand the potential of videotaped feedback in the bench-training model, we recently conducted a study in which the effectiveness of videotaped feedback was evaluated over 3 core surgical tasks in the field of orthopedics. We found no significant differences between the control group and the videotaped feedback group.¹¹ We speculated that 2 main design flaws in our study may have influenced our findings. First, we included a very heterogeneous group of residents from all 5 academic levels (PGY1–5). With approximately 35 residents in the entire orthopedic training program, we believed it was best to include all academic levels in our experiment in order to obtain maximal study power. However, the more senior residents may have had less to gain from seeing themselves on videotape because of their higher level of proficiency in the selected tasks. Second, the residents received only 1 exposure to the video feedback for each of the technical tasks. We postulated that the degree of intervention may have been insufficient to result in a measurable difference between the control and experimental groups.

A natural progression from that study was to increase the video feedback from a single tape review to several viewings in succession. This would offer the advantage of increasing the opportunity to practise a surgical task. With any surgical task, measurable changes are more readily detected if the skill can be practised more than once. By seeing oneself perform a task repeatedly, any improvement in skill attributable to video feedback would become more evident.

The purpose of this study, therefore, was to evaluate the benefits of repeated videotaped feedback for PGY1 surgery residents when performing a simulated vascular anastomosis procedure (Fig. 1). A demonstration that videotaped feedback results in enhanced speed or quality of technical skill acquisition could provide evidence to support the incorporation of this modality into the routine training of surgical residents.

FIG. 1. Set-up for the bench-training vascular anastomosis procedure.

Methods

Thirty-three first-year surgical residents from the subspecialties of orthopedics, cardiac surgery, plastic surgery, neurology, otolaryngology and general surgery were invited to participate. Individual written consent was obtained from all participants. Ethics approval was obtained from the Research Ethics Board at Mount Sinai Hospital, Toronto.

Vascular anastomosis was selected as the skill to be taught for several reasons. Vascular anastomosis is a complex task that has multiple requisite skills.¹² They include manual dexterity and visual-spatial ability to: (1) plan the sequence of events required to complete the task, (2) prepare the blood vessels and the operative field, (3) move one's body and arms as necessary to facilitate suture placement, (4) grasp the needle at the appropriate place and angle for each placement of suture, (5) adjust the patient's body position or room lighting to make the anastomosis technically possible and, finally, (6) place the sutures at appropriate intervals and distances from the edges of the blood vessels with minimal tissue damage.

Through experience and practise, an expert surgeon is able to simultaneously integrate all of these skills to successfully complete an anastomosis. When learning how to perform an anastomosis, trainees often focus on only 1 or 2 of these skills.¹³ The process of learning requires practise and multiple attempts in order to integrate the multiple skills required. We therefore hypothesized that repeated videotaped feedback with the assistance of an expert had the potential to increase the efficiency with which a trainee could master the complex task of vascular anastomosis.

The surgical skill of vascular anastomosis was taught within the vascular control module of the principles of surgery curriculum of the surgery residency program at the University of Toronto. Skills taught in these weekly lectures are practised in the laboratory sessions (Fig. 2) using human, animal and inanimate models. It is within the laboratory sessions devoted to vascular anastomosis that we conducted our study. The additional costs requiring research funds for the video feedback group included the rental of video recording equipment, videotapes and 2 video-technicians over 4 half-days. The experts volunteered their time.

FIG. 2. Set-up for the practice sessions in the surgical skills laboratory.

Verbal feedback

Before this study began, all residents were randomly assigned to 1 of 2 groups: control (expert feedback alone) or experimental (video feedback with expert review). During the 3 laboratory practice sessions, experts in the field of vascular surgery provided individualized verbal feedback to residents in both the control and experimental groups. Before the practice sessions, the experts were provided with standardized instructions on how to give verbal feedback to both groups, so that all experts were providing feedback in a similar manner. The elements of verbal feedback were based on the 24 items identified in the checklist on vascular anastomosis (Fig. 3). Residents were free to ask questions of the experts and the experts were free to provide verbal feedback as they circulated through the work stations over the 2-hour laboratory session.

FIG. 3. Task-specific checklist: vascular anastomosis.

Control group

In the control group, the experts would provide feedback as they circulated through the laboratory. During these practice sessions the residents repeatedly practised the task of vascular anastomosis.

Experimental group

The residents in the experimental group practised the skill of vascular anastomosis in the same area as residents in the control group. Videotaped feedback was given only to those in the experimental group and was provided only once during each weekly practice session from week 1 to week 3. Six stations were set up in a segregated area of the laboratory for the experimental group. Each station had a digital video camera on a tripod stand and all of the necessary surgical equipment needed to perform the vascular anastomosis. Mobile dividers separated the 6 stations to prevent “cross-contamination” of feedback from the other stations. Each resident had a maximum of 15 minutes to complete the vascular anastomosis while being videotaped. During the taping, a vascular surgeon was available to provide feedback. After the taping, each resident in the videotaped group reviewed their entire videotape up to a maximum of 15 minutes with an expert. The expert provided verbal feedback in the standardized method described above. Residents had the opportunity to fast-forward and rewind the tape and ask questions of the expert. During the videotaped review there was no contact with other residents or any other form of feedback.

After completing the videotaping, the residents returned to practise the surgical skill of vascular anastomosis with the other residents in the laboratory.

In both groups, verbal feedback was available throughout the 2-hour period simulating a typical surgical skills laboratory session. The only difference between the 2 groups was the single session of videotaped feedback with an expert (on a one-to-one basis) over 3 consecutive weeks.

During week 4 of the vascular control module, the residents participating in the study completed a mini objective structured assessment of technical skills (MOSAT) examination that assessed their ability to perform a vascular anastomosis on the same type of simulation model of human anatomy they practised on from week 1 to week 3. The evaluators were vascular surgeons.

Outcome measures

Performance was assessed using a checklist (Fig. 3) and a global rating scoring system (Fig. 4).^4,5 The detailed checklist is operation-specific. The items were selected by a panel of vascular surgeons who identified these separate actions as necessary in performing a vascular anastomosis effectively. The global rating scale is a 7-item form that uses behavioural descriptors and focuses on the overall performance of the resident, not the specifics of the surgical task.^4,5 Both the global rating scale and checklist are reliable, with interstation reliability reported at 0.843 for the global rating scale and 0.781 for the checklist.⁵ Construct validity was demonstrated for both scales where analysis of variance revealed a significant effect of training for both the checklist and the global rating score (p < 0.001).⁵

FIG. 4. Global rating scoring system.

Analysis

Each resident had 2 scores consisting of a checklist and a global rating scale for MOSAT evaluation on vascular anastomosis. We analyzed the data using descriptive analyses and independent t tests to compare the 2 scores of the residents in the experimental group with those in the control group. For all tests, a 2-tailed p value of less than 0.05 was considered significant.

Results

Twenty-six of the 33 PGY1 surgical residents participated in this study. The remaining 7 residents were unable to attend at least 2 of the 3 practice sessions due to conflicts in scheduling and were therefore eliminated from the study. Of the 26 residents participating, 7 were from orthopedics, 1 from neurosurgery, 1 from cardiac surgery, 3 from otolaryngology, 3 from plastic surgery and 11 from general surgery. There were 12 residents in the control group and 14 in the experimental group. The number of times residents had seen the task performed in the operating room was an average of 4.5 (standard deviation [SD] 2) for the control group and 2.8 (SD 2) for the experimental group. The number of times residents had actually performed a vascular anastomosis in the operating room was on average less than 1 for both groups. We did not find a significant association between the number of times the vascular anastomosis procedure was seen or done in the operating room and the checklist and global rating scores (p > 0.05). Mean scores with standard deviations between the 2 groups using the 2 outcome measures are presented in Table 1. Although the average scores for both the checklist and global rating scales are higher in the experimental group, independent t tests revealed no statistically significant differences (Table 1).

Table 1

Discussion

Our study is a follow-up to previous work that evaluated the potential benefits of videotaped feedback among orthopedic surgical residents.¹¹ In that study, residents were evaluated on 3 tasks (femoral plate application, elbow fracture wiring and z-plasty) both before and after a single intervention with videotaped feedback. No significant differences were found between the video feedback and no video feedback groups in terms of improved time to complete the task, checklist or global rating scores. We speculated that our lack of findings may have been related to inadequate exposure to the videotaped feedback. As a result, in the present study we included more videotaped feedback sessions to identify whether repeated videotaped feedback would better enhance residents' acquisition of a core surgical skill, vascular anastomosis. However, our findings revealed no statistically significant differences attributable to the use of repeated video feedback.

Although we were unable to find a statistically significant difference using the checklist and global rating scale, residents from the experimental group described some benefits of videotaped feedback in a qualitative questionnaire administered after the completion of the study. The benefits included the opportunity for immediate feedback and the opportunity to actually see mistakes instead of only being told what needed correction. Residents described a sense that the use of videotaped feedback allowed them to “fine-tune” a basic skill. One resident believed that videotaped feedback would be helpful once a resident had acquired the basic skill level for the task, suggesting that video feedback would be useful in developing an expert level of surgical proficiency.

The predominant reason why some residents did not find video feedback helpful was the perception that greater benefit could be achieved with individualized feedback from a vascular surgeon. Previous research has demonstrated the advantages of individualized feedback with an expert.^14,15,16 Yet this remains a costly and time-consuming method.

In the qualitative questionnaire, 2 residents reported inconsistencies in the expert's feedback during review of the videotapes from one week to the next. We provided a standardized set of criteria for evaluation based on the checklist form to the vascular surgeons providing feedback. It is therefore interesting that residents commented on the expert feedback as inconsistent. This finding suggests that the experts themselves may have a learning curve when using videotape to instruct residents.

In 2 cases, a single videotaped session per week was described as insufficient. This comment requires further investigation. Perhaps, videotaped feedback would be of greater benefit if the residents could use this technology on an ongoing basis to continually monitor their progress as they develop their surgical skills.

Our study may have been limited by the outcome measures used. Although global rating scale and the checklist have been shown to have validity and reliability among surgical residents,^4,5,6 both are based on rank scales of measurement that may lack the sensitivity needed to measure the subtle improvements in surgical skill. In our previous study,¹¹ we speculated that the inclusion of PGY4 and 5 residents may have been a factor in not finding any differences between feedback groups. In comparison with previous studies that predominantly involved medical students or junior residents, our investigation had groups with higher skill levels at the outset of our study. The inclusion of a wide range of trainees may have lead to a situation in which subtle improvements in surgical proficiency were less likely to achieve statistical significance. However, in the present study, despite including only PGY1 level residents we were again unable to find statistically significant differences between the groups.

The lack of a difference leads us to believe that videotaped feedback either as a short intervention or in a repeated format may have minimal benefits to surgical residents. Although the potential benefits of videotaped feedback appear to be more intuitive than quantitative, they may still have a place in surgical training programs as an adjunctive teaching tool. Perhaps, as suggested in the qualitative questionnaire, videotaped feedback may be better used after a basic skill has been acquired in order to hone skills to a higher level. Video review either by the surgical resident alone or with an expert may be most useful to senior level residents who would identify subtle areas for improvement with a more critical eye.

Conclusions

It is possible that we need to use more sensitive outcome measures to pick up subtle improvements in surgical skill. Improvements in surgical proficiency often vary and may require months or years to be noticeable. Measuring these improvements has been inherently challenging within the field of surgical education, particularly with the limited availability to use the operating room as a teaching venue. The opportunity to develop surgical proficiency using bench models and effective adjunctive feedback methods is becoming ever more critical as operating room experience and faculty hours become ever more costly and precious.

Presented at the annual meeting of The Royal College of Physicians and Surgeons of Canada, Halifax, NS, Sept. 12, 2003

Competing interests: None declared.

Correspondence to: Dr. David Backstein, Division of Orthopaedic Surgery, Mount Sinai Hospital, 600 University Ave., Ste. 476D, Toronto ON M5G 1X5; fax 416 586-8678; david.backstein@utoronto.ca

Accepted for publication June 2, 2004