An Evaluation of the Feasibility, Validity, and Reliability of Laparoscopic Skills Assessment in the Operating Room

Abstract

Objective:

To assess the use of a synchronized video-based motion tracking device for objective, instant, and automated assessment of laparoscopic skill in the operating room.

Summary Background Data:

The assessment of technical skills is fundamental to recognition of proficient surgical practice. It is necessary to demonstrate the validity, reliability, and feasibility of any tool to be applied for objective measurement of performance.

Methods:

Nineteen subjects, divided into 13 experienced (performed >100 laparoscopic cholecystectomies) and 6 inexperienced (performed <10 LCs) surgeons completed LCs on 53 patients who all had a diagnosis of biliary colic. Each procedure was recorded with the ROVIMAS video-based motion tracking device to provide an objective measure of the surgeon's dexterity. Each video was also rated by 2 experienced observers on a previously validated operative assessment scale.

Results:

There were significant differences for motion tracking parameters between the 2 groups of surgeons for the Calot triangle dissection part of procedure for time taken (P = 0.002), total path length (P = 0.026), and number of movements (P = 0.005). Both motion tracking and video-based assessment displayed intertest reliability, and there were good correlations between the 2 modes of assessment (r = 0.4 to 0.7, P < 0.01).

Conclusions:

An instant, objective, valid, and reliable mode of assessment of laparoscopic performance in the operating room has been defined. This may serve to reduce the time taken for technical skills assessment, and subsequently lead to accurate and efficient audit and credentialing of surgeons for independent practice.

The assessment of technical skills in the operating room is fundamental to the credentialing of surgeons for independent practice. This paper investigates the use of motion-tracking of a surgeon's hands for assessment of laparoscopic skill, in synchrony with video-based assessment of performance on a structured rating scale.

Recent publications of the rates of medical errors and adverse events within health care, and particularly during surgery, have drawn the spotlight toward the methods of credentialing surgeons to perform procedures independently.^1–4 Training boards and certifying bodies are coming under increasing pressure to ensure individuals demonstrate the necessary skills to perform operations safely.^5–9 This is not only important for patient safety but also underpins the development of a proficiency-based training curriculum.

It is somewhat surprising then that there are no tools in widespread use that are feasible, valid, and reliable for assessment of technical surgical skill (Table 1).¹⁰ Current training outcomes are assessed by live evaluations of the trainee by the master surgeon, a process that is known to be biased and subjective.¹¹ More objective data are available from morbidity and mortality data, although this is rarely a sole function of operative skill and thus does not truly reflect an individual's surgical competence.¹² The majority of trainees also maintain a log of the procedures performed, but these are indicative merely of procedural performance rather than a measure of technical ability.¹³

TABLE 1. Qualities of the Ideal Surgical Assessment Tool

Although a number of new tools have been developed to assess surgical technical performance, their use remains within the confines of surgical skills laboratories.¹⁰ These include virtual reality simulators and psychomotor training devices, which are designed primarily to assess performance during critical parts of a procedure, rather than a complete operation.¹⁴ The realism (or face validity) of such simulations is not perfect and the situations lack context, leading to a failure of operators to treat the models like real patients.¹⁵

The ideal device for objective assessment of real surgical procedures would be one that can automatically, instantly, and objectively provide feasible, valid, and reliable data regarding performance within the operating room.¹⁶ It is with this approach that our Department has developed the ROVIMAS motion tracking software, which enables surgical dexterity to be quantified and thus reported instantly by a computer program.¹⁷ Although automatic, objective, and instant, the data do not provide any information regarding the quality of the procedure performed. The system does, however, incorporate the ability to synchronously record video of the operative procedure, which can then be evaluated according to a valid and reliable rating scale. This can enable a definition of dexterity not only for whole procedures, but also for critical steps of a particular procedure.

A preliminary publication has confirmed the feasibility of using the device within the operating room to assess laparoscopic skills.¹⁷ The primary aim of this study was to determine the validity and reliability of a new concept for technical skills assessment in the operating room, a combination of motion analysis and video assessment. Both approaches have been individually validated in the literature, although the introduction of a hybrid between the 2 modes of assessment has not been previously attempted.

METHODS

Subjects

Nineteen surgeons were recruited to the study, and subdivided into 6 novice (<10 laparoscopic cholecystectomies, LCs) and 13 experienced (>100 LCs) practitioners. The aim was for each surgeon to perform a minimum of 2 procedures, with consecutive cases recorded over a period of 6 months.

Operative Procedure

LC was chosen as the operative procedure of choice as it is a common operation, performed in a fairly standardized manner and amenable to both motion tracking and video-based analysis. Furthermore, LC is an index procedure for commencement of training and ongoing assessment of laparoscopic skills.^5,9

Patients

Ethical approval was obtained from the local research ethics committee to record video data of each operation. Patients were recruited from 2 surgical departments, and all were consented prior to entry into the study. To reduce the effect of disease and patient variability, all patients recruited to the study were deemed to have a diagnosis of biliary colic. To enable objectification of this approach, inclusion and exclusion criteria were classified according to patient and disease state, and upon posthoc review of the video tape according to operative state (Table 2).¹⁸

TABLE 2. Inclusion and Exclusion Criteria for Patient Entry to the Study

Motion Analysis Device

All procedures were recorded with the ROVIMAS software. The parameters used for this study were those that have been previously validated on bench-top assessments of laparoscopic skill on a porcine model, ie, time taken, path length, and number of movements for each hand.¹⁹

The purpose of the study was explained to all surgeons prior to the patient consent process. Once scrubbed, surgeons wore one pair of sterile gloves over their hands. Sensors were then placed onto the dorsum of each hand, followed by the donning of surgical gown and a further pair of sterile gloves. This avoided the need to sterilize the electromagnetic sensors, and friction between the gloves maintained the sensors in the correct position. Once the patient had been anesthetized, the electromagnetic emitting device was placed onto their sternum, fixed firmly by Micropore tape (3M Corporation, St. Paul, MN).

Video-Based Assessment

The video feed from the laparoscopic stack was recorded onto the ROVIMAS software through a digital video link (I-link, IEEE-1394) to a laptop computer. Recording commenced upon entry of the endoscopic camera into the peritoneal cavity and was complete upon removal of the camera from the abdomen for the final time. The open parts of the procedure were not recorded, the aim being to solely assess the laparoscopic skills of the subjects. Complete, unedited videos of each procedure were recorded with the software into Microsoft Windows .avi format (Microsoft Corporation, Redmond, WA). All data files were coded by an alphanumeric code to ensure the identity of the operating surgeon and patient were blinded to the reviewers.

The objective structured assessment of technical skill (OSATS) proposes a generic evaluation of surgical performance through use of a global rating scale.²⁰ The scale was initially validated through live-marking of bench-top tasks and is said to “boast high reliability and show evidence of validity.”²¹ The aim was to determine the validity, inter-rater, and intertest reliability of this scale for assessment of laparoscopic technical skills. Rating on the scale was performed by 2 experienced laparoscopic surgeons (T.G. and K.M.) who were blinded as to the identities of the operating surgeons.

Data Collection and Analysis

Three-dimensional coordinate data from the Isotrak II motion tracking device (Polhemus Inc, Colchester, VT) were translated into useful parameters of time taken, path length, and number of movements of each hand by the ROVIMAS software (see Dosis et al for a detailed description¹⁷). With the aid of the synchronization feature of ROVIMAS, data were derived for the entire procedure, and also for predefined parts of the procedure, classified as in Table 3. It must be noted that the values for the whole procedure are not a sum of all the parts identified, ie, insertion of accessory ports, division of adhesions, removal of gallbladder, etc.

TABLE 3. Definitions of the Tasks of a Laparoscopic Cholecystectomy

Power analysis was based upon results from a previous study on motion analysis of porcine LCs, and revealed a sample size of 20 cases per group.¹⁹ Statistical analysis used nonparametric tests of significance. Construct validity was determined by comparison of performance between novice and experienced surgical groups for dexterity parameters and scores from video-rating scales with the Mann-Whitney U test. Cronbach's alpha test statistic was used to ascertain the inter-rater reliability of the video-based scoring system. Intertest reliability was assessed by comparison of the first and second consecutive procedure performed by each surgeon, once again with Cronbach's alpha test statistic.

To investigate the existence of a relationship between dexterity analysis and the video-based rating scale for assessment of surgical performance, correlations between the 2 methods were calculated with the nonparametric Spearman's rank correlation coefficient. For all tests, P < 0.05 was considered statistically significant.

RESULTS

Procedures Performed

A total of 53 procedures were performed by the 19 surgeons recruited to the study. Six cases were excluded in an independent manner by both reviewers upon the basis of intraoperative characteristics (Table 2). Of the remaining 47 cases, 14 were performed by 6 novice surgeons and 33 by the 13 experienced surgeons. The median number of cases carried out by each surgeon was 2 (range, 1–5 cases).

Motion Tracking Data

A comparison between LCs performed by novice and experienced surgeons revealed significant differences in time taken for the whole procedure (median 2175 vs. 1979 seconds, P = 0.036), although not for total path length or number of movements (Table 4). This result was replicated for “clip and cut duct” (55 vs. 33 seconds, P = 0.013) and “clip and cut artery” (37 vs. 21 seconds, P = 0.004). Only dissection of Calot triangle produced significant differences between the performance of novice and experienced surgeons for all 3 motion tracking parameters (Figs. 1–3): time taken (854 vs. 393 seconds, P = 0.002), total path length (138 vs. 73 m, P = 0.026) and total number of movements (640 vs. 367, P = 0.005).

TABLE 4. Results From Motion Tracking Parameters, Divided Into Individual Tasks

FIGURE 1. Time taken for experienced and novice surgeons to dissect Calot triangle. There was a significant difference between experienced and inexperienced groups (P = 0.002).

FIGURE 2. Total path length for experienced and novice surgeons to dissect Calot triangle. There was a significant difference between experienced and inexperienced groups (P = 0.048).

FIGURE 3. Total number of movements for experienced and novice surgeons to dissect Calot triangle. There was a significant difference between experienced and inexperienced groups (P = 0.007).

Fifteen of the 19 surgeons performed 2 or more procedures each. The intertest reliability between their first 2 consecutive cases for time taken to perform the whole procedure was α = 0.502. With regard to dissection of Calot triangle, intertest reliability was calculated for time taken (α = 0.623), total path length (α = 0.229), and total number of movements (α = 0.522).

Video-Based Data

The generic OSATS global rating scale demonstrated a significant difference in scores between the novice and experienced surgeons (median 24 vs. 27, P = 0.031), with an inter-rater reliability coefficient of α = 0.72 (Fig. 4). The intertest reliabilities of the 15 surgeons who performed 2 or more procedures for the OSATS was α = 0.72.

FIGURE 4. Inter-rater reliability of OSATS global rating score between observer 1 and 2 (Cronbach's α = 0.72).

Comparison of Motion Tracking and Video-Rating Scales

The correlations between scores obtained from the OSATS global rating scale and validated motion tracking parameters are shown in Table 5. All r values were statistically significant and ranged from between 0.4 to 0.7, indicating that there were good correlations between the 2 modes of assessment.

TABLE 5. Correlation Between OSATS Global Rating Score and Motion Tracking Data, Calculated Using Spearman Rank Correlation Test

DISCUSSION

Although surgical competence is a multimodal function, proficiency in technical skills to perform an operative procedure is fundamental to a successful outcome.^22–24 Assessment within the operating theater is not only a mode of credentialing individual surgeons but also enables collective audit of surgical units and residency training programs.²⁵ Despite the development of a number of tools for assessment of technical skills, none has been incorporated into standard practice. This is either due to their complexity, poor validity, or the lack of experienced personnel to administer them. The only way to ensure data are collected for every single operation performed within a hospital is to develop a system that automatically records and analyzes the required information, without causing any delay or difficulty to the operating room procedure.

This was our intention with the development of the ROVIMAS motion tracking system. With this study, we have reiterated the feasibility and confirmed the validity and intertest reliability of this device for assessment of laparoscopic technical skills in the operating theater. In terms of validity, time taken was the only marker to differentiate procedural performance between groups of experienced and novice laparoscopic surgeons. However, the synchronization feature of the system enabled motion tracking parameters to reveal significant differences in dexterity between the 2 groups of surgeons during dissection of Calot triangle. The novice surgeons on average were twice as slow and half as dexterous as the experienced group. This may be because it is the most difficult part of the operation, and indeed the most likely to lead to a catastrophic error.

Standardization of the procedures was performed on the basis of the patient history, preoperative investigations, and intraoperative findings. A closer inspection of this process reveals that standardization is primarily based upon the degree of inflammation at Calot triangle. It is thus not surprising that this part of the procedure gleaned significant differences in dexterity between the 2 groups of surgeons. None of the other parts of the operation demonstrated construct validity for assessment of dexterity parameters. Reasons for this are either due to simplicity of the task, eg, clip and cut, or anatomic variations such as length of the gallbladder attached to the liver bed. It may also be possible that the failure to achieve significance in dexterity parameters for the whole procedure and other parts of the operation between the 2 groups is due to an underpowered study. Although the intended 20 cases per group were recruited, 6 were excluded upon the basis of intraoperative characteristics.

Nonetheless, it is of concern to note that there remained similar degrees of variability within the novice and experienced groups in terms of dexterity assessments. It would be expected, and indeed has been shown in the literature, that experienced surgeons display a greater degree of consistency when compared with their junior counterparts.²⁶ The conflicting factor may be that surgeons performed the procedure with their “usual technique,” perhaps explaining the variability within the experienced group. It would be necessary to confirmed this by dexterity analysis of different techniques to perform laparoscopic cholecystectomy, eg, blunt/sharp versus blunt/teasing dissection methods.²⁷

A significant limitation of dexterity-based assessment using motion analysis is a failure to capture the qualitative and procedural aspects of an operation. Although number of movements can be used as a measure of operative dexterity, performance is more readily measured by number of faulty or inappropriate movements. This was the reason for integration of a video-based analysis of technical skill using the OSATS global rating scale.²⁰ Although other rating scales exist, divided broadly into checklists and error-scoring systems, the OSATS global rating scale has repeatedly been validated for skills assessment. In our study, the generic OSATS scale displayed construct validity. The aim of a global rating scale is to assess general surgical principles, whereas checklist-based assessments are by definition specific to the operation. Checklists enable detailed evaluations by specifying individual steps and substeps of the operative procedure.²⁸ This is time-consuming in terms of assessment, and awards the surgeon only if they perform the procedure in the predefined sequence of steps. However, surgery is not a mechanical process and thus it is difficult to justify its evaluation in such a rigid manner. The criticism is that such a scale can only ensure whether something was done or not, but not whether it was done well or poorly.

A number of studies have made use of the generic OSATS scale within the operating theater,^29–32 although the architects of the tool themselves are concerned that “critical aspects of technical skill are not assessed.”²¹ Furthermore, this scale was developed for use in live rather than video-based assessment. The Toronto group have subsequently developed and validated a video-based, procedure-specific objective component rating scale for Nissen fundoplication.²¹ In the same vein, researchers have recently sought to develop procedure-specific global rating scales, although their efficacy is yet to be tested.^33,34

Joice et al developed an error-based approach to surgical skills assessment that uses human reliability analysis (HRA), which is the systematic assessment of human-machine systems and their potential to be affected by human error.²⁷ Tang et al have applied an Observational Clinical Human Reliability Assessment (OCHRA) tool to laparoscopic cholecystectomy and pyloroplasty.^35–38 Observational videotape data are subjected to a detailed step-by-step analysis of surgical operative errors, which are divided into consequential or inconsequential. This is a highly specialized and labor-intensive task, although it has been suggested that OCHRA provides a comprehensive objective assessment of the quality of surgical operative performance by documentation of errors, stage of the operation when they are most frequent, and when they are consequential. In a not dissimilar manner, the use of dexterity analysis has enabled identification of Calot triangle dissection as part of the operation in which there are significant differences between novice and experienced surgeons. However, OCHRA benefits assessment in a more formative manner, enabling error modes to be studied and corrective actions to be pursued. Part of our further work is to define the relative roles of motion analysis, rating scales and the OCHRA tool for surgical skills assessment.

A further question that has rarely been discussed in terms of surgical skills research is the intertest reliability of an instrument for assessment of technical skill.³⁹ The reliability of the assessment was good for motion tracking parameters and excellent for the video-based global rating scale. This adds further weight to the use of these modes of assessment to assess improvements in performance of a trainee surgeon, or the consistency of an experienced surgeon.

It seems that reliable and valid assessment of laparoscopic skills within the operating theater can be performed with ROVIMAS motion tracking software, though not exclusively. The global rating scales were valid and demonstrated higher intertest reliabilities. It is not surprising then that a comparison of motion tracking data with video-based rating scales revealed significant correlations with the global rating scales. An assumption is that the automated motion tracking device can do the work of 2 experienced observers assessing an operation on a global rating scale. Although not true at present, this is certainly a notion we are working toward. With regard to dissection of Calot triangle, significant differences were noted for both path length and number of movements. Although both parameters are related, it is of course possible to perform a task with very fine movements, leading to a large number of movements and shorter path length. Current work seeks to define the relationship between these 2 parameters, ie, average path length per movement; a low ratio would suggest fine movements. This information may be useful to highlight accuracy, or perhaps uncertainty, during video-based assessment of the surgical procedure.

However, observer-based assessment of technical skill is time-consuming, and relies upon the availability of experienced surgeons to rate performance.²¹ The dexterity parameters from the motion tracking device may be useful as a first-pass filter, avoiding the need to view the entire procedure to obtain information regarding technical proficiency. In this manner, surgeons could be automatically assessed on motion analysis each time they performed a procedure, and parts of the video rated either by global or error-based scoring systems only if their dexterity parameters fell outside a predetermined range of values. This could reduce the time taken to assess proficiency and would lead to the development of an accurate record of operative skill. Furthermore, many surgeons already make a video record of laparoscopic procedures that they perform; the association of motion tracking data could be stored in a similar operative library.

Extending this concept further leads onto the notion of an operating room black box whereby all aspects of performance are recorded onto a single platform.⁴⁰ As well as a documentation of the surgeon's technical skills, the record would include information regarding the rest of the operative team, the patient, equipment used, etc. As in the airline black box, all of this information could be recorded in an automated manner and available for review at a later stage.

CONCLUSION

The field of surgical skills assessment has progressed significantly over the past decade. To transfer this knowledge from research laboratories and onto real cases, there is not only a need for the further development of these tools but also to define the structure of training programs.^41–43 Although the concept of competency-based training is regarded as essential to the future of surgical practice, details regarding its implementation have failed to be considered. The primary aim should be to integrate objective assessment of surgical skill into training programs and ensure substandard performance results in remediation or repetition of part of the program. This system can only be warranted with significant financial resources, enabling not only quality assurance of surgical practice, but also additional research into further developing the present tools for comprehensive surgical skills assessment.

ACKNOWLEDGMENTS

The authors thank all surgeons and patients who agreed to participate in the study, and the operating room staff who were involved.