Technical and Technological Skills Assessment in Laparoscopic Surgery

Abstract

Objectives:

Surgical appraisal and revalidation are key components of good surgical practice and training. Assessing technical skills in a structured manner is still not widely used. Laparoscopic surgery also requires the surgeon to be competent in technological aspects of the operation.

Methods:

Checklists for generic, specific technical, and technological skills for laparoscopic cholecystectomies were constructed. Two surgeons with >12 years postgraduate surgical experience assessed each operation blindly and independently on DVD. The technological skills were assessed in the operating room.

Results:

One hundred operations were analyzed. Eight trainees and 10 consultant surgeons were recruited. No adverse events occurred due to technical or technological skills. Mean interrater reliability was kappa=0.88, P=<0.05. Construct validity for both technical and technological skills between trainee and consultant surgeons were significant, Mann-Whitney P=<0.05.

Conclusions:

Our study demonstrates that technical and technological skills can be measured to assess performance of laparoscopic surgeons. This technical and technological assessment tool for laparoscopic surgery seems to have face, content, concurrent, and construct validities and could be modified and applied to any laparoscopic operation. The tool has the possibility of being used in surgical training and appraisal. We aim to modify and apply this tool to advanced laparoscopic operations.

INTRODUCTION

Surgical technical skill is an integral part of surgical competency. A recent review¹ however highlighted the lack of structured assessment of technical skills in surgeons in live operations. Most of the research on technical skills in laparoscopic surgery has been on bench models or simulators² and has assessed generic technical skills.^3–5 Some attempts have been made to develop scales to assess generic and specific technical skills, but these have not been widely adopted.^6–11

In recent years, many high-profile cases have questioned the surgical skill competency of various surgeons. With the imminent introduction of a compulsory appraisal and assessment of doctors by the General Medical Council (GMC),¹² a structured assessment tool of surgical technical skills could be beneficial. Also, a laparoscopic surgeon must be competent in the technological aspects of the operation. This includes equipment set up, instruments, and materials available. To date, this aspect has not been assessed in a structured manner on laparoscopic surgeons.

In the present study, we aimed to develop a structured assessment tool to assess technical and technological skills for live laparoscopic operations that may or may not differentiate the technical skills of a surgeon based on experience or supervision, or both. The principles of this assessment tool could be applied to basic laparoscopic training, either simulated or live cases.

METHODS

Technical Skills

To develop the assessment tool for generic and specific technical skills in laparoscopic surgery, we chose an elective operation that was routinely performed in large numbers by both consultant/attending and trainee/resident surgeons in the United Kingdom and worldwide. Laparoscopic cholecystectomy conformed to our selection criteria. In the first instance, we did a Medline search using the key words “surgery,” “skill,” “assessment,” “technological,” and “technical” to review all the literature. We then read various surgical textbooks and atlases as well as surgical training course manuals.

Generic Technical Skills

For the generic technical skills assessment, we used a checklist (Figure 1) that we have previously described.¹³ Generic technical skills errors were scored 1 point if correctly done and 0 point if incorrectly done. This was done because generic technical skills errors tend to be inconsequential errors.

Figure 1.

Generic technical skills checklist.

Specific Technical Skills

A hierarchical task analysis was constructed for laparoscopic cholecystectomy.¹⁴ From this, key zones were identified, and a checklist scale developed for specific technical skills (Figure 2). Expert panel discussions were done to outline various specific objectives of the assessment tool and confirm face and content validities of the specific technical skills scale. Minor, major, and significant major error scores were constructed and weighted after expert discussion. Minor errors were awarded 2 points if correctly done, 4 points if done incorrectly but then rectified, and 6 points if incorrectly done and not rectified. Major errors received 4 points for being correctly being done, 8 points if incorrectly being done but rectified, and 12 points if incorrectly being done and not rectified. Significant major errors received 8 points for correctly being done, 16 points if incorrectly being done but rectified, and 24 points if incorrectly being done and not rectified.

Figure 2.

Specific technical skills checklist.

These above weighted scores reflected the severity of the technical skills error on the quality of the operation and possible consequence to patient outcome. This was done because specific technical skills errors tend to lead to consequential errors.

Technological Skills

To develop the assessment tool for technological skills in laparoscopic surgery, we read various surgical textbooks as well as surgical training course manuals. After this, expert panel discussions were done to outline various specific objectives of the assessment tool and confirm face and content validities of the specific technological skills checklist scale (Figure 3). Technological skills were scored 1 point if correctly done and 0 point if incorrectly done.

Figure 3.

Technological skills checklist.

Gallbladder Grading

The grades of the gallbladder can affect the technical skills aspects of the operation.

Therefore, gallbladders were graded by a standard clinical-pathological scale:

• Grade 1 (Thin walled, no adhesions)

• Grade 2 (Thin walled with adhesion)

• Grade 3 (Thick walled)

• Grade 4 (Thick walled, chronically inflamed),

• Grade 5 (Thick walled, acutely inflamed)

Assessment of Technical Skills

Full-length versions of the operations were digitally recorded and converted to DVD. Each operation was viewed anonymously on DVD by 2 experienced general surgeons with more than 12 years postgraduate surgical experience each. Each operation DVD was viewed on a large-screen television and DVD player blindly and independently by each assessor. All of the operations were viewed in their entirety with only the noncritical aspects fast forwarded.

Assessment of Technological Skills

An independent observer who was an experienced general surgeon with more than 12 years postgraduate surgical experience assessed each technological checklist. This was then checked with the operating surgeon after the operation was completed to confirm and assess reliability.

Ethical Approval

Local ethics committee approval was achieved, each patient gave written consent, and the surgeons gave verbal consent. Either the patient or the surgeon had the opportunity to decline to be included in the study.

Patients and Setting

The study was conducted prospectively between January 2003 and August 2004. All operations were done either by consultant or trainee surgeons and were performed at a district general hospital, Central Middlesex Hospital, Park Royal, London NW10, or at a teaching hospital, St. Mary's Hospital, Paddington, London W2. All patients were between ASA 1 to 3, BMI<32, >18 years or <85 years old.

Subjects

The surgeons were divided into 2 groups: consultants (had performed >150 laparoscopic cholecystectomies) and trainees (had performed <150 laparoscopic cholecystectomies). They were all informed about the study and verbally consented to participate in it, and had the option not to participate. The video camera operator did not give advice on technical or technological aspects of the operation to the surgeons before, during, or after the operation. All the trainee operations were supervised. However, for the preoperative technological skills assessment the trainees were not supervised.

Statistical Analysis

Data were collated in an Excel database (Microsoft, Redmond, Washington, USA). Statistical analysis was carried out with SPSS statistical software (SPSS, Chicago, Illinois, USA). P<0.05 was regarded as statistically significant. For reliability analysis between the 2 observers, Kappa coefficients (k) were used; k>0.61 was deemed statistically significant, P<0.05. Confidence intervals were set at 95%. For validity analysis of the checklists, the data were nonparametric; therefore, the Mann-Whitney test was used for comparison between the 2 groups of surgeons.

RESULTS

Patients

Patients included 28 males and 72 females. Mean age was 56 years (range, 21 to 82). Mean BMI was 29 (range, 21 to 32). No major postoperative complications occurred.

Gallbladder Grade

The mean operation grade and level of surgeon are depicted in Figure 4. There were 93 gallbladders graded 1 to 3 and 7 graded 4 to 5. Using nonparametric analysis, Mann-Whitney, no significant difference was noted between the gallbladder grade and level of surgeon (consultant or trainee), P=0.135.

Figure 4.

Gallbladder grades and grade of surgeon.

Surgeons

A total of 18 surgeons participated in the study (consultants=10, trainees=8, 17 were right-handed and 1 was left-handed. Seventeen were male surgeons and one was female). Five consultants had been less than 5 years in their present grade and 5 consultants had been a consulting surgeon for more than 5 years. There were 2 registrar trainees in years 5 to 6 and 6 trainees between years 1 and 4. There were 86 consulting episodes, mean of 8 (range, 25 to 1), 14 trainees episodes, mean of 2 (range, 4 to 1).

Reliability—Technical and Technological Skills

To assess reliability between the observers for the checklist technical and technological skills assessment, we used the weighted kappa coefficient (k). Table 1 summarizes the kappa coefficients for all the checklists used. It can be seen that all the checklists have a high reliability between the observers.

Table 1.

Reliability of Technical and Technological Skills Checklists

Skill Being Assessed	Kappa Coefficients (k)	P Value	Conclusions
Generic technical skills	0.881	0.0000	Highly reliable between 2 observers
Specific technical skills—Minor events	0.924	0.0000	Highly reliable between 2 observers
Specific technical skills—Major and significant major events	0.859	0.0000	Highly reliable between 2 observers
Technological skills	0.87	0.0000	Highly reliable between observers

Validity—Technical Skills

We tested for differences between consultants and trainees through a nonparametric test for 2 independent samples, ie, Mann-Whitney using the average of the scores. Error bar charts with the central number being the mean checklist score with a 95% confidence interval are demonstrated in Figures 5, 6, and 7. Higher scores relate to fewer errors in the generic technical skills checklist scale. In the minor, major, and significant major events checklist (specific technical skills), the higher scores relate to more errors being executed. Construct validity analysis for the checklist generic technical skills was significant. Consultants as a group performed the operation better compared with trainees regarding generic technical skills. However, for the specific technical skills (minor, major, and significant major) checklist construct validity was not significant between the 2 groups. This latter fact is probably because all trainees were supervised.

Figure 5.

Generic technical skills checklist scores and grade of surgeon.

Figure 6.

Specific minor technical skills checklist scores and grade of surgeon.

Figure 7.

Specific major and significant major technical skills checklist scores and grade of surgeon.

Validity—Technological Skills

To test differences between consultants and trainees through a nonparametric test, Mann-Whitney was applied, using the average of the scores (Figure 8). Construct validity analysis for the checklist technological skills was significant, P=<0.05. Consultants as a group performed the technological skills better compared with the trainees.

Figure 8.

Technological skills checklist scores and grade of surgeon.

DISCUSSION

Self-appraisal and revalidation in surgery are topical issues. As these aspects may/will become compulsory in surgical practice, recent attempts have been made to develop tools to assess the skills of a surgeon.^15–17 In the present study, our aim was to develop a structured assessment tool to assess technical and technological skills for live laparoscopic operations.

Construct validity analysis of the checklist generic technical skills was significant. Consultants as a group performed the operation better than the trainees did, regarding generic technical skills. Generic technical skills improved with more operating experience. This explains the difference between the 2 groups of surgeons. However, for the specific technical skills (minor, major, and significant major) checklist construct validity was not significant between the 2 groups. This latter fact is probably because all trainees were supervised. These specific technical skills categories would have an affect on patient outcome and the quality of the final operation. A possible explanation of these results is that the supervised trainees were prevented from making errors because consultants supervised them.

The technological skills assessment checklist has construct validity with the consultants performing better than the trainees. This is because the consultant surgeons had performed more laparoscopic operations than had the trainees. So they knew which technological equipment and instruments were necessary to perform the operation.

This assessment tool of generic technical skills and technological skills in live laparoscopic surgery seems to have face, content, concurrent, predictive, and construct validities. We aim to develop this assessment tool and apply it to advanced laparoscopic operations.