QUANTIFYING TECHNICAL SKILLS DURING OPEN OPERATIONS USING VIDEO-BASED MOTION ANALYSIS

Abstract

INTRODUCTION

Objective quantification of technical surgical skill in surgery remains poorly defined although delivery of and training in these skills is essential to the surgical profession. Attempts to measure hand kinematics to quantify surgical performance primarily rely on electromagnetic sensors attached to the surgeon’s hand or instrument. We sought to determine whether similar motion analysis could be performed using a marker-less video-based review, allowing for a scalable approach to performance evaluation.

METHODS

We recorded six reduction mammoplasty operations – a plastic surgery procedure in which the attending and resident surgeons operate in parallel. Segments representative of surgical tasks were identified using Multimedia Video Task Analysis software. Video digital processing was used to extract and analyze the spatio-temporal characteristics of hand movement.

RESULTS

Attending plastic surgeons appear to use their non-dominant hand more than residents when cutting with the scalpel, suggesting more use of counter-traction. While suturing, attendings were more ambidextrous with smaller differences in movement between their dominant and non-dominant hands than residents. Attendings also seem to have more conservation of movement when performing instrument tying than residents, as demonstrated by less non-dominant hand displacement. These observations were consistent within procedures and between the different attending plastic surgeons evaluated in this fashion.

CONCLUSIONS

Video motion analysis can be used to provide objective measurement of technical skills without the need for sensors or markers. Such data should be valuable in better understanding the acquisition and degradation of surgical skill, providing enhanced feedback to shorten the learning curve.

INTRODUCTION

Despite the essential role that technical skill plays in surgery, few objective, evidence-based methods exist to help surgeons assess their technical performance during an operation. The limitations in performance-based metrics have been described both within and outside the discipline of surgery^3–7. The need for a scalable approach to objectively evaluating individual technical surgical skill therefore remains unmet.

Currently, two types of surgical skills assessment are in use: rating scales and motion analysis. Rating scales produce subjectively assigned scores; usage of these tools often requires training and recalibration to achieve test-retest consistency and/or inter-rater reliability. Motion analysis is an emerging field in skills assessment, aimed at gaining a more objective assessment of technical skill. By tracking metrics such as time, path length, number of movements, velocities, and trajectories, motion analysis addresses the objectivity and reproducibility issues in technical skill assessment. Motion analysis is able to reliably differentiate the level of dexterity between expert and novice surgeons performing bench tasks^{8, 9}, laparoscopic cholecystectomies¹⁰, and vasectomies and vasectomy reversals¹¹. Such data can be used to better understand the acquisition and degradation of surgical skill, as well as provide enhanced (i.e. specific and precise) feedback to shorten the learning curve. However, the aforementioned motion-analysis techniques rely primarily on electromagnetic sensors attached to surgeons’ hands or instruments, interrupting the natural flow of the operation and potentially impeding performance. Moreover, the additional equipment in the operative field may pose a risk to patient safety. Finally, the settings in which this analysis can be performed are limited to institutions that have this advanced technology available.

With the increasing availability of recording devices in operating rooms, video-based analysis could fill this gap and provide a generalizable and scalable approach to the assessment of surgical skills. This approach enables the capture of performance without requiring external evaluators to be present in the operating room or surgeons to be encumbered by sensors. We tested a novel tool for marker-less video based motion analysis that was developed within the field of industrial engineering^{12, 13}. Developed at the University of Wisconsin by two of our coauthors (RR and C-HC)¹², this system uses a conventional digital video. An analyst identifies a selected region of interest (ROI), and the system follows that ROI and calculates the kinematic characteristics, including displacement, velocity, and acceleration. In a study looking at a paced-load transfer task in the laboratory setting, where subjects were asked to transfer weighted bottles from a tray to a turntable at a specified pace, the system has produced similar results to manual frame-by-frame and infrared 3D motion tracking system analyses^{12, 13}. We performed a feasibility study to determine whether motion analysis using this marker-less video-based approach could be adapted for the analysis of technical skill performance during open operative procedures.

METHODS

Case and Patient Selection

In reduction mammoplasties, attending and trainee surgeons often operate simultaneously and in parallel, with one on the left breast and the other on the right breast. In addition, benign breast tissue provides a symmetrical operative field for each of the two surgeons. By contrasting the techniques of the attending and the trainee, the specific components that comprise technical expertise can be identified and studied.

Competent, adult women undergoing bilateral reduction mammoplasty were identified by weekly review of the Pre-Admission Testing Center (PATC) and Operating Room (OR) schedules. After the surgeon’s initial approval to record their cases, patients were sent a recruitment letter from the primary surgeon and the study team. Patients that did not opt out were approached pre-operatively for written informed consent using the Partners IRB Certificate of Confidentiality consent form (protocol 2011p000655).

Data Collection

As part of this feasibility study, we needed to develop the optimal approach to data collection by comparing several ways to capture video images, including in-light camera and head-mounted cameras. Between February and June 2012, 44 potential cases were identified for our study. Permission to record in the OR was obtained from 6 surgeons. Nineteen of their patients consented to the study. Eleven cases were able to be video recorded. Nine of these cases had in-light camera view, which was found to provide the only acceptable view for motion analysis. Of these 9 cases, 6 cases yielded usable video for analysis. Issues that led the remaining videos to be unusable included poor views of the hands, additional surgeons assisting in the operation, and excessive motion artifacts. The 6 cases included 3 surgeons with 8–30 years of experience and 3 residents who were in their 4^th–6^th year of training. All surgeons and residents were right-hand dominant. Completely de-identified videos were then coded and analyzed.

Analysis

Segments representative of surgical tasks were identified by a surgical resident using Multimedia Video Task Analysis^™ (MVTA^™) software (http://mvta.engr.wisc.edu/), which allows for coding of video data. The following tasks were coded: cutting with electro-cautery (bovie), cutting with the scalpel, suturing, and instrument tying. A screenshot, showing an example of the coding using this software is shown in figure 1. Using the ROI software developed by the co-investigators¹², the corresponding coded video segments were subsequently analyzed for their kinematic properties of displacement, velocity and acceleration. A rectangular region of interests (ROI) was marked manually to identify the focal area where hand activities are to be tracked, such as a point on the hand or arm. Figure 2 provides a screenshot example of an ROI. A template matching tracking algorithm was implemented to track the ROI motion trajectory over successive video frames. The video data raw location values were low-pass filtered by a Butterworth filter and scaled. The velocity and acceleration at each video frame was calculated using the method of differences.

Figure 1.

Example of the coding using the MVTA^™ software.

Figure 2.

Example of the ROI analysis software, with the right and left hands of the surgeon marked.

Exported data on displacement, velocity, and acceleration were analyzed for each task and compared between attending and resident surgeons. Descriptive statistics were generated for attending and resident surgeons’ dominant and non-dominant hand movements. As this was a feasibility study, we were not powered to achieve statistical significance. Rather, our aim was to determine whether there were variations in these metrics and to identify patterns that could generate hypotheses to be studied in future work. Analyses were performed using Statistical Analysis Software (SAS), MVTA^™, and the motion analysis software.

RESULTS

Mean displacement, velocity, and acceleration values for attending and resident surgeons’ dominant and non-dominant hands are shown for cutting with the bovie and with the scalpel in Tables 1 and 2, respectively. Tables 3 and 4 show the data for suturing and instrument tying, respectively.

Table 1.

Cutting with bovie. Mean values and standard deviation (SD) for dominant hand (D), non-dominant hand (ND), and difference between dominant and non-dominant hand (D-ND) displacement, velocity, and acceleration.

Table 2.

Cutting with scalpel. Mean values and SD for dominant hand (D), non-dominant hand (ND), and difference between dominant and non-dominant hand (D-ND) displacement, velocity, and acceleration.

Table 3.

Suturing. Mean values and SD for dominant hand (D), non-dominant hand (ND), and difference between dominant and non-dominant hand (D-ND) displacement, velocity, and acceleration.

Table 4.

Instrument tying. Mean values and SD for dominant hand (D), non-dominant hand (ND), and difference between dominant and non-dominant hand (D-ND) displacement, velocity, and acceleration.

Attending surgeons appear to use their non-dominant hand more than residents when cutting with the scalpel (1425 mm/s² vs. 991 mm/s²), suggesting more use of counter-traction in the hand that was not holding the scalpel. Attendings also seem to be more ambidextrous while suturing, with smaller differences in movement between their dominant and non-dominant hands, compared to residents, in terms of displacement (11 mm and 10 mm vs. 10 mm and 6 mm), velocity (276 mm/s and 248 mm/s vs. 241 mm/s and 167 mm/s), and acceleration (3345 mm/s² and 3246 mm/s² vs. 2867 mm/s² and 2010 mm/s²). Lastly, analysis of attendings suggests more conservation of movement when performing instrument tying, as demonstrated by less non-dominant hand displacement (14 mm vs. 23 mm), velocity (379 m/s vs. 576 mm/s), and acceleration (5273 mm/s² vs. 3877 mm/s²).

DISCUSSION

Although central to a surgeon’s profession, technical skill remains a poorly defined construct. Previous studies on motion analysis in open surgical tasks have primarily focused on the time to complete a task, number of movements, and total path length^{8, 11, 14–16}. However, using a motion-video analysis tool, we were able to deconstruct larger tasks into subcomponents that compose those tasks, including the movements involved in tying and suturing. This study suggests that surgeons may use their non-dominant hand more to assist themselves than do residents, provide themselves with more counter-traction when cutting with the bovie and scalpel and actively use the forceps to assist themselves when suturing. When performing instrument tying, surgeons appear to have more conservation of movement in their non-dominant hands than residents, suggesting more efficient motion.

This study was performed to assess feasibility. As such, results come from a single institution, with a limited number of participants. During our recorded cases, the attending surgeons operated on the right, while the resident operated on the left, as is typical during this procedure. Additionally, the surgeon supervised the resident’s performance. It is possible that this positioning and the fact that they were teaching cases may have affected the results. Because the camera was housed in an operative light that was positioned by the surgical team, and because our system is currently only able to capture 2-dimensional data, the different angles of the cameras may prevent direct comparisons between cases. However, we chose the reduction mammoplasty procedure precisely because it minimizes variation in operative context. As this was an exploratory study, and not powered for statistical significance, most of our results did not reach significance. Now that we have studied the feasibility of this method of analysis, we plan to address these limitations by analyzing technical performance across a larger range of surgical abilities and different operations within the OR and in simulation settings.

Video motion analysis holds great promise as a feasible way to collect data on technical performance, and provide an objective measurement of technical skill without the need for the surgeons to wear markers or other types of sensors. Previous studies on motion analysis in surgery have primarily focused on laparoscopic cases performed in simulated settings¹⁷. The few studies that have been performed on open procedures have necessitated the surgeons to wear sensors on their hands or instruments^{8, 11, 14–16}. In this study, we were able to extract data on surgeons’ hand movements from open surgical procedures performed in the operating room directly from digital video, without the need for any physical sensors. This non-invasive approach to data capture markedly enhances the generalizability and scalability of technical skills assessment in the operating room. Coupled with the increased availability of recording devices, implementation of this assessment modality should be feasible in a wide variety of settings.