Association Between Surgeon Technical Skills and Patient Outcomes

Key Points

Question

Is the technical skill of a surgeon associated with patient outcomes?

Findings

In this quality improvement study including 17 practicing surgeons and outcome data from the American College of Surgeons National Surgical Quality Improvement Program as a reference group, better technical skill scores, assessed via intraoperative video, were statistically significantly associated with lower rates of any complication, unplanned reoperation, and death or serious morbidity. Overall, technical skill scores were associated with approximately 26% of the variation in risk-adjusted complication rates.

Meaning

Quality improvement efforts aimed at improving patient outcomes should include a focus on improving surgeon-level technical skill.

Abstract

Importance

Postoperative complications remain common after surgery, but little is known about the extent of variation in operative technical skill and whether variation is associated with patient outcomes.

Objectives

To examine the (1) variation in technical skill scores of practicing surgeons, (2) association between technical skills and patient outcomes, and (3) amount of variation in patient outcomes explained by a surgeon’s technical skill.

Design, Setting, and Participants

In this quality improvement study, 17 practicing surgeons submitted a video of a laparoscopic right hemicolectomy that was then rated by at least 10 blinded peer surgeons and 2 expert raters. The association between surgeon technical skill scores and risk-adjusted outcomes was examined using data from the American College of Surgeons National Surgical Quality Improvement Program. The association between technical skill scores and outcomes was examined for colorectal procedures and noncolorectal procedures (ie, assessed on whether technical skills demonstrated during colectomy were associated with patient outcomes across other cases). In addition, the proportion of patient outcomes explained by technical skill scores was examined using robust regression techniques. The study was conducted from September 23, 2016, to February 10, 2018; data analysis was performed from November 2018 to January 2019.

Exposures

Colorectal and noncolorectal procedures.

Main Outcomes and Measures

Any complication, mortality, unplanned hospital readmission, unplanned reoperation related to principal procedure, surgical site infection, and death or serious morbidity.

Results

Of the 17 surgeons included in the study, 13 were men (76%). The participants had a range from 1 to 28 years in surgical practice (median, 11 years). Based on 10 or more reviewers per video and with a maximum quality score of 5, overall technical skill scores ranged from 2.8 to 4.6. From 2014 to 2016, study participants performed a total of 3063 procedures (1120 colectomies). Higher technical skill scores were significantly associated with lower rates of any complication (15.5% vs 20.6%, P = .03; Spearman rank-order correlation coefficient r = −0.54, P = .03), unplanned reoperation (4.7% vs 7.2%, P = .02; r = −0.60, P = .01), and a composite measure of death or serious morbidity (15.9% vs 21.4%, P = .02; r = −0.60, P = .01) following colectomy. Similar associations were found between colectomy technical skill scores and patient outcomes for all types of procedures performed by a surgeon. Overall, technical skill scores appeared to account for 25.8% of the variation in postcolectomy complication rates and 27.5% of the variation when including noncolectomy complication rates.

Conclusions and Relevance

The findings of this study suggest that there is wide variation in technical skill among practicing surgeons, accounting for more than 25% of the variation in patient outcomes. Higher colectomy technical skill scores appear to be associated with lower complication rates for colectomy and for all other procedures performed by a surgeon. Efforts to improve surgeon technical skills may result in better patient outcomes.

This quality improvement study examines the technical skills of surgeons performing colectomies.

Introduction

Several studies suggest that patient outcomes vary considerably across surgeons.^1,2,3 Most efforts to improve patient outcomes, however, have focused on the systems of care that surround the surgical episode rather than the conduct of the operation. Colectomy is a common surgical procedure but has one of the highest morbidity rates. The risk of any complication after a colectomy remains at approximately 20%, and surgeon-specific complication rates vary considerably.^3,4

A common assumption is that the technical skill of a surgeon is associated with patient outcomes.⁵ However, with the exception of one study,⁵ to our knowledge, no studies have demonstrated an association between technical skills and patient outcomes, particularly for a procedure such as colectomy, which has relatively high patient morbidity. Moreover, it is unknown whether the technical skill of a surgeon demonstrated for one type of procedure reliably estimates outcomes of other procedures performed by the surgeon. In addition, the proportion of patient outcomes explained by technical skill scores is unknown.

In this prospective quality improvement study, we evaluated the use of video-based peer scoring as a valid approach to objectively assess surgeon technical skill among attending surgeons. The study objectives were to examine (1) variation in technical skill scores of practicing surgeons, (2) the association between technical skills and patient outcomes for colectomy and noncolectomy procedures, and (3) the proportion of variation in patient outcomes explained by technical skills. We hypothesized that technical skill would be associated with major postoperative complications, particularly those specifically related to technical issues (eg, bleeding, reoperation, and anastomosis leak).

Methods

Study Design

Surgeons were recruited from the network of 56 hospitals participating in the Illinois Surgical Quality Improvement Collaborative (ISQIC) from March to September 2016. The study was conducted from September 23, 2016, to February 10, 2018; data analysis was performed from November 2018 to January 2019. The ISQIC participating hospitals collect detailed data on surgical patient demographics, comorbidities, operative details, and 30-day patient outcomes through the validated American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) clinical registry. Further details are presented in the eMethods in the Supplement.

Each participating surgeon was asked to submit a single representative video of a laparoscopic right hemicolectomy and participate in peer review of those videos. Each submitted video recording was scored by at least 10 randomly assigned peer study participants and 2 colorectal surgeons (B.J.C. and S.E.S.) who are experts in video-based assessment. To ensure consistency in scoring, all study participants watched an anchor video before scoring the videos, which demonstrated the various levels of technical skills and suggested scores. All scoring was double-blinded between the surgeon and peer study participant raters through an ISQIC-developed website. The study was deemed exempt by the Northwestern University Institutional Review Board. No informed consent was deemed necessary, and participants were not financially compensated.

Based on well-established instruments,^6,7,8 we constructed a 16-item scoring sheet to assess technical skills using 2 methods: a procedural, step-specific method customized for laparoscopic right hemicolectomy procedures and a global skills performance method based on the Objective Structured Assessment of Technical Skill method.⁷ The step-specific section consisted of 8 items in the following areas: (1) port insertion, (2) isolation and division of the ileocolic, (3) mobilization of the colon (lesser sac dissection, inferior-lateral terminal ileum dissection, lateral mobilization, and hepatic flexure mobilization), (4) division of the right branch of ileocolic, and (5) anastomosis. The global operative performance questions followed the anchors used in the American Society of Colon and Rectal Surgeon’s video assessment tool.^6,9 This tool was developed and pilot tested, and the anchors were validated and found to be reliable with good internal consistency, making it feasible for use to score laparoscopic right hemicolectomies. Results of interrater reliability and correlations of the surgeon technical skill score with other alternative scores can be found in eTable 1 and eTable 2 in the Supplement.

Surgeon Procedures and Patient Outcomes

Clinical data from the ACS NSQIP were gathered from January 1, 2014, to June 30, 2016. The surgery records of all colectomies performed (procedure-targeted data) and a hospital-level random sample of other noncolectomy procedures performed on patients aged 18 years or older were obtained. Our colectomy-specific outcomes included all colectomies (eg, right and left) and excluded proctectomy. Six NSQIP patient outcome measures were gathered for these procedures: any complication, mortality, unplanned hospital readmission, unplanned reoperation related to principal procedure, surgical site infection (SSI), and death or serious morbidity.¹⁰ Among these outcomes, any complication indicates the occurrence of any of the 17 common postoperative complications designated by the ACS NSQIP (eg, cardiac arrest requiring cardiopulmonary resuscitation, myocardial infarction, cerebrovascular accident/stroke with neurologic deficit, deep vein thrombosis/thrombophlebitis, sepsis, septic shock, acute renal failure, progressive renal insufficiency, urinary tract infection, on ventilator more than 48 hours after the principal procedure, pneumonia, pulmonary embolism, unplanned intubation, superficial incision SSI, deep incisional SSI, organ/space SSI, and wound dehiscence).^5,11 Death or serious morbidity is one of the key outcomes in the ACS NSQIP, indicated by the National Quality Forum, and includes the occurrence of mortality, unplanned reoperation, or any of the serious morbidity events (eTable 3 in the Supplement).¹²

In addition, we constructed 2 composite outcomes as validation tests for associations in the analysis. The composite outcomes were created a priori to reflect outcomes that conceptually should or should not be related to a surgeon’s technical skill. The first of these outcomes is a skill-related composite measure consisting of bleeding intraoperatively or within 72 hours of surgery requiring transfusion, unplanned readmission likely related to the surgical procedure, unplanned reoperation, wound dehiscence, and organ/space SSI. The second outcome is a skill-unrelated composite measure using the components of the any complication outcome but excluding skill-related complications, such as wound dehiscence and organ/space SSI.

We calculated adjusted rates for the 8 surgical outcomes within 30 days of the principal procedure, following the standard ACS NSQIP risk adjustment method (eTable 3 in the Supplement).⁵ For each outcome, a 3-level generalized linear mixed model was fitted to account for the clustering of patients within surgeons and surgeons within hospitals. We conducted separate analyses on risk-adjusted rates for colectomy only and for all noncolectomy procedures. For the analysis of all procedures (shown) and noncolectomy procedures only (analysis not shown), we also included procedure type risk adjustment.¹¹ Given that some study participants had small procedure volumes, we modeled the observed-to-expected ratios using all 43 841 procedures performed by all surgeons in the ISQIC (which included the 7512 colectomy procedures), followed by a shrinkage adjustment.¹³ After surgeon-level outcome rates were estimated, our analysis was restricted to the 17 participating surgeons who had their technical skill scored.

Statistical Analysis

Our primary analysis was conducted at the surgeon level. We divided surgeon participants into 3 groups based on the quartiles of their technical skill scores: low-skilled (quartile 1), medium-skilled (quartile 2 or 3), and high-skilled (quartile 4). Overall group differences of the scoring items were tested with 1-way analysis of variance.

We further investigated the association between a surgeon’s technical skill score and the patients’ outcomes in 2 ways. First, we compared the differences in risk-adjusted outcome rates based on the 3 skill groups (low, medium, and high), as defined above. We then used robust regression analysis to estimate the proportion of variation in a surgical outcome associated with technical skill score when used as a continuous variable. Robust regression has the advantage of minimizing potential interference from influential outliers on coefficient estimates in a small sample, such as this study. We used the Tukey studentized range test when making multiple comparisons by technical skill level on all skill assessment items and surgical outcomes for both colectomy and other types of procedures. Data on demographic and diagnostic information for patients being recorded for technical skill assessment were not collected to ensure patient confidentiality. We specifically asked surgeons to submit videos that were representative of their usual clinical practice. In an academic setting, that instruction meant surgeons were performing the operation with the assistance of residents and fellows. All analyses were performed using SAS, version 9.4 (SAS Institute Inc), and statistical significance was considered at a 2-tailed P value <.05.

Results

Surgeon Characteristics

Seventeen surgeons from 11 ISQIC hospitals voluntarily participated in the study, and each submitted a representative video of their performance of a laparoscopic right hemicolectomy. Of the 17 surgeons included in the study, 13 were men (76%) and 4 were women (24%). The participants had a range from 1 to 28 years in surgical practice (median, 11 years) (Table 1) since completion of fellowship or residency training. Nearly half (8 [47.1%]) of the participants were board certified by the American Board of Colorectal Surgery. Eight participants (47.1%) had formal laparoscopic training through a minimally invasive surgery fellowship, a colorectal fellowship, or with minimally invasive surgery training during residency, and 2 participants (11.8%) reported other formal training in minimally invasive surgery. The remaining surgeons (7 [41.2%]) had no formal laparoscopic training. Overall, participants had little prior experience recording their laparoscopic procedures (9 [52.9%] never, 7 [41.2%] occasionally).

Table 1. Characteristics of Participating Surgeons.

Characteristic	No. (%)
Individual characteristics
No. of surgeons participating	17
Surgeon technical skill score (1-5)
Mean (SD)	3.7 (0.6)
Range	2.8-4.6
IQR	3.3-4.1
Years in practice, median (range)	11 (1-28)
Surgery specialty
General	9 (52.9)
Colorectal	8 (47.1)
ABCRS certified	8 (47.1)
Formal training in MIS
MIS fellowship, colorectal fellowship, or MIS training during residency	8 (47.1)
Other formal training	2 (11.8)
No formal training	7 (41.2)
Experience with laparoscopic recording
Never	9 (52.9)
Occasionally	7 (41.2)
Often	1 (5.9)
Always	0 (0.0)
Involves residents/fellows in colectomies
Never	3 (17.6)
Rarely	2 (11.8)
Sometimes	2 (11.8)
Most of the time	2 (11.8)
Always	8 (47.1)
Primary operating surgeon in recorded procedure	12 (70.6)
Surgical volumesa
All procedures (including colectomy)
Total, No.	3063
Mean (SD)	180 (98)
IQR	133-216
Annual volume, mean (IQR)	72 (53-86)
Colectomy procedures
Total, No.	1120
Mean (SD)	66 (53)
IQR	10-87
Annual volume, mean (IQR)	26 (4-35)
% Laparoscopic colectomies (percentage point)
Mean (SD)	65.6 (18.6)
IQR	56.0-73.3

Abbreviations: ABCRS, American Board of Colon and Rectal Surgery; IQR, interquartile range; MIS, minimally invasive surgery.

^aBased on Illinois Surgical Quality Improvement Collaborative semiannual record files, January 2014 to June 2016.

Study participants varied in their volume of colorectal procedures. The ACS NSQIP had data on a mean (SD) of 66 (53) colectomies for each participating surgeon, with 65.6% (18.6%) of the colectomies being performed laparoscopically. Using the ACS NSQIP standard sampling method of noncolectomy procedures, the ACS NSQIP had data on 180 (98) other procedures for each participating surgeon. In total, the 17 participants in our study performed 3063 procedures between January 1, 2014, and June 30, 2016, for which data were available within the ACS NSQIP, including a total of 1120 colectomies (36.6%).

Surgeon Technical Skill

With a score of 5 indicating expert skill, the mean (SD) technical skill score among participants was 3.7 (0.6) (Table 1). Even in this relatively small sample, the technical skill scores revealed adequate variation to allow for differentiation among surgeons, as the range (2.8-4.6, Δ = 1.8) was wide relative to the SD (3.0 times SD). Participants were divided into 3 groups based on the quartiles of technical skill scores: low (lowest quartile; n = 4), medium (middle 2 quartiles; n = 8), and high (highest quartile; n = 5) technical skill scores (Table 2). Technical skill scores, as defined above, differed significantly on the binary port insertion item and 12 of the13 five-point individual skill items (Table 2). The mean score on the 5-point items among highly skilled surgeons (quartile 4) were 1.2 to 1.5 points (out of 5) higher than those in quartile 1. However, scores did not vary significantly by levels of skills related to the measure of assessing isolation and division of the ileocolic (low: 3.3; 95% CI, 2.6-4.1; medium: 3.7; 95% CI, 3.2-4.2; and high: 4.4; 95% CI, 3.9-4.9; P = .05) (Table 2).

Table 2. Technical Skill Items by Quartile of Surgeon Summary Skill Score.

Expert and peer rating of technical skillsa	Quartile of surgeon technical skill score, mean (95% CI)			Overall difference, P valueb	Difference: high-lowc
	1 (Low) (n = 4)	2, 3 (Medium) (n = 8)	4 (High) (n = 5)		Mean (adjusted 95% CI)	P value
Procedural step-specific questions
Port insertion (% safe)d	62.9 (46.3 to 76.9)	96.7 (88.0 to 99.8)	94.1 (79.9 to 99.4)	<.001	31.2 (13.6 to 48.9)	<.001
Isolation and division of the ileocolic	3.3 (2.6 to 4.1)	3.7 (3.2 to 4.2)	4.4 (3.9 to 4.9)	.05	1.1 (−0.1 to 2.2)	.06
Mobilization of the colon
Lesser sac dissection	2.8 (2.4 to 3.3)	3.5 (3.1 to 3.9)	4.0 (3.7 to 4.4)	.01	1.2 (0.4 to 1.9)	<.001
Inferior-lateral TI dissection	2.7 (2.4 to 3.0)	3.2 (3.0 to 3.4)	4.2 (3.9 to 4.5)	<.001	1.5 (1.0 to 2.0)	<.001
Lateral mobilization	2.8 (2.5 to 3.1)	3.5 (3.3 to 3.7)	4.4 (4.1 to 4.6)	<.001	1.5 (1.0 to 2.0)	<.001
Hepatic flexure mobilization	2.7 (2.4 to 3.0)	3.3 (3.1 to 3.6)	4.1 (3.8 to 4.4)	<.001	1.4 (0.9 to 2.0)	<.001
Global operative performance questions
Respect for tissue	3.1 (2.7 to 3.5)	3.6 (3.4 to 3.9)	4.2 (3.9 to 4.6)	.001	1.2 (0.5 to 1.8)	<.001
Time and motion	2.8 (2.5 to 3.1)	3.4 (3.2 to 3.7)	4.2 (3.9 to 4.5)	<.001	1.4 (0.8 to 1.9)	<.001
Instrument handling	2.9 (2.7 to 3.2)	3.7 (3.5 to 3.9)	4.4 (4.2 to 4.6)	<.001	1.5 (1.1 to 1.9)	<.001
Flow of operation	2.9 (2.6 to 3.2)	3.6 (3.4 to 3.7)	4.3 (4.1 to 4.6)	<.001	1.5 (1.0 to 1.9)	<.001
Exposure	2.8 (2.4 to 3.2)	3.5 (3.2 to 3.7)	4.3 (4.0 to 4.6)	<.001	1.5 (0.9 to 2.1)	<.001
Tissue planes	2.8 (2.6 to 3.1)	3.7 (3.5 to 3.8)	4.3 (4.1 to 4.6)	<.001	1.5 (1.1 to 1.9)	<.001
Completeness of dissection	3.2 (2.8 to 3.5)	3.8 (3.6 to 4.1)	4.4 (4.1 to 4.8)	<.001	1.3 (0.7 to 1.8)	<.001
Overall technical skill	2.9 (2.6 to 3.1)	3.6 (3.4 to 3.8)	4.3 (4.1 to 4.5)	<.001	1.4 (1.0 to 1.8)	<.001

Abbreviation: T1, terminal ileum.

^aPooled ratings by peers and experts. Two step-specific questions (division of the right branch of ileocolic and anastomosis) were excluded owing to insufficient data as ratings were given to only 1 video on each of those items.

^bF test in analysis of variance for overall group differences. When the Levene test for homogeneity of variance was rejected, the result of the Welch variance-weighted 1-way analysis of variance was reported.

^cDifference between the mean score of surgeon technical skill in quartile 4 (high) and the mean score in quartile 1 (low). Adjusted 95% CIs were based on Tukey studentized range test on multiple pairwise comparison.

^dAgresti-Coull 95% CI for binomial proportion for percent safe port insertion by technical skill quartile.

When comparing surgeons by the quartile-based levels of technical skill from low to high, a descending staircase pattern of risk-adjusted postcolectomy patient outcome rates appeared (Table 3). For example, low skill–level surgeons had a mean any complication rate following colectomy of 20.6% (95% CI, 17.7%-23.4%) compared with 15.5% (95% CI, 12.9%-18.1%) for high skill–level surgeons (pairwise difference high vs low, P = .03). The reoperation rate following colectomy was 4.7% (95% CI, 3.6%-5.8%) among high skill–level surgeons but 7.2% (95% CI, 6.0%-8.5%) among the low skill–level surgeons, representing a 90% relative increase (P = .02). The rate of death or serious morbidity following colectomy was 21.4% (95% CI, 18.6%-24.3%) among low skill–level surgeons and decreased to 15.9% (95% CI, 13.4%-18.4%) for high skill–level surgeons (P = .02).

Table 3. Risk-Adjusted Surgical Outcome Rates by Quartile of Surgeon Technical Skill^a.

Risk-adjusted 30-d outcomesb	Quartile of surgeon technical skill score, mean (95% CI)			Overall difference, P valuec	Difference: low-highd
	1 (Low) (n = 4)	2, 3 (Medium) (n = 8)	4 (High) (n = 5)		Mean (adjusted 95% CI)	P value
Colectomy procedurese
Any complicationf	20.6 (17.7 to 23.4)	18.3 (16.3 to 20.3)	15.5 (12.9 to 18.1)	.04	5.1 (0.4 to 9.8)	.03
Mortality	3.3 (2.9 to 3.7)	3.4 (3.1 to 3.7)	3.0 (2.7 to 3.4)	.35	0.3 (−0.4 to 0.9)	.59
Unplanned readmission	11.3 (9.8 to 12.9)	9.8 (8.7 to 10.9)	9.8 (8.4 to 11.2)	.21	1.5 (−0.9 to 4.0)	.27
Unplanned reoperation	7.2 (6.0 to 8.5)	5.3 (4.4 to 6.2)	4.7 (3.6 to 5.8)	.02	2.5 (0.5 to 4.6)	.02
SSI	11.1 (7.9 to 14.2)	9.7 (7.5 to 12.0)	7.9 (5.1 to 10.8)	.31	3.1 (−2.1 to 8.3)	.29
Death or serious morbidity	21.4 (18.6 to 24.3)	17.8 (15.8 to 19.7)	15.9 (13.4 to 18.4)	.02	5.6 (0.9 to 10.2)	.02
Skill-related compositeg	29.5 (25.0 to 33.9)	26.3 (23.1 to 29.4)	22.9 (19.0 to 26.9)	.10	6.5 (−0.8 to 13.8)	.08
Skill-unrelated compositeh	24.3 (20.0 to 28.6)	24.6 (21.5 to 27.6)	21.5 (17.6 to 25.3)	.39	2.9 (−4.2 to 9.9)	.55
All procedures (including colectomy)i
Any complicationf	9.4 (8.0 to 10.7)	8.2 (7.2 to 9.2)	6.9 (5.7 to 8.1)	.04	2.5 (0.2 to 4.7)	.03
Mortality	1.5 (1.3 to 1.6)	1.5 (1.3 to 1.6)	1.3 (1.2 to 1.5)	.37	0.1 (−0.2 to 0.4)	.54
Unplanned readmission	6.7 (5.8 to 7.6)	5.8 (5.26.4)	5.8 (5.0 to 6.6)	.19	0.9 (−0.5 to 2.4)	.25
Unplanned reoperation	3.4 (2.8 to 3.9)	2.5 (2.1 to 2.9)	2.2 (1.7 to 2.7)	.02	1.2 (0.2 to 2.1)	.01
SSI	4.7 (3.3 to 6.1)	4.1 (3.1 to 5.0)	3.3 (2.1 to 4.6)	.30	1.4 (−0.9 to 3.6)	.27
Death or serious morbidity	10.6 (9.1 to 12.0)	8.6 (7.6 to 9.6)	7.7 (6.4 to 9.0)	.02	2.9 (0.6 to 5.3)	.02
Skill-related compositeg	13.5 (11.4 to 15.6)	11.7 (10.3 to 13.2)	10.2 (8.4 to 12.1)	.07	3.3 (−0.1 to 6.7)	.06
Skill-unrelated compositeh	11.0 (8.9 to 13.1)	11.1 (9.7 to 12.6)	9.7 (7.8 to 11.6)	.42	1.3 (−2.1 to 4.7)	.59

Abbreviation: SSI, surgical site infection.

^aGeneralized linear mixed models for risk adjustment were done using all eligible surgery cases for the 391 surgeons in Illinois Surgical Quality Improvement Collaborative registry data. Multivariable analysis on adjusted outcome rates and surgical skill were at the clinician level and restricted for study participants (n = 17).

^bIn percentage points.

^cF test in analysis of variance for overall group differences. When the Levene test for homogeneity of variance was rejected, the result of the Welch variance-weighted 1-way analysis of variance was reported.

^dDifference between the mean score of surgeon technical skill in quartile 4 (high) and the mean score in quartile 1 (low). Adjusted 95% CIs were based on Tukey studentized range test on multiple pairwise comparison.

^eTotal of 7512 cases used in generalized linear mixed model; of these, 26 were performed by participants in quartile 1 of surgeon technical skill, 452 participants in quartiles 2 and 3, and 642 participants in quartile 4.

^fThe occurrence of any of the 17 postoperative common complications designated by the American College of Surgeons National Surgical Quality Improvement Program, such as cardiac arrest requiring cardiopulmonary resuscitation, myocardial infarction, cerebrovascular accident/stroke with neurologic deficit, deep vein thrombosis/thrombophlebitis, sepsis, septic shock, acute renal failure, progressive renal insufficiency, urinary tract infection, on ventilator more than 48 hours after the principal procedure, pneumonia, pulmonary embolism, unplanned intubation, superficial incision SSI, deep incisional SSI, organ/space SSI, and wound dehiscence.

^gA composite indicator for any of the following postoperative outcomes that were believed clinically likely to be related to surgeon technical skill: transfusions related to intraoperative bleeding, unplanned readmission likely related to the principal procedure, unplanned reoperation, wound dehiscence, or organ/space SSI; for colectomy, this indicator also included occurrence of postoperative anastomotic leak.

^hIndicator for presence of any of the following postoperative outcomes that were unlikely related to surgeon technical skill, including 15 of the 17 components of the any complication outcome except wound dehiscence and organ/space SSI.

ⁱTotal of 43 841 cases used in generalized linear mixed model; of these, 505 were performed by participants in quartile 1 of surgeon technical skill, 1538 participants in quartiles 2 and 3, and 1020 participants in quartile 4.

Surgeons’ technical skill scores were found to be significantly correlated with 5 of the 8 colectomy-specific patient outcomes: any complication (Spearman rank-order correlation coefficient r = −0.54, P = .03), unplanned readmission (r = −0.49, P = .04), unplanned reoperation (r = −0.60, P = .01), death or serious morbidity (r = −0.60, P = .01), and the skill-related composite (r = −0.51, P = .04) but not with mortality (r = −0.35, P = .17), SSI (r = −0.28, P = .27), or the skill-unrelated composite (r = −0.20, P = .44) (Table 4; Figure).

Table 4. Robust Regression Models Examining Risk-Adjusted Surgical Outcome Rates by Surgeon Technical Skill^a.

Variable	Risk-adjusted outcome rate, mean (SD)b	Correlation coefficientc	P valuec	Robust regression parameter estimates
				Coefficient (95% CI)	P valued	Model R2
Colectomies
Any complicatione	18.0 (3.2)	−0.54	.03	−3.1 (−5.6 to −0.7)	.01	0.258
Mortality	3.3 (0.4)	−0.35	.17	−0.2 (−0.5 to 0.1)	.24	0.086
Unplanned readmission	10.2 (1.5)	−0.49	.04	−1.4 (−2.7 to −0.1)	.04	0.231
Unplanned reoperation	5.6 (1.5)	−0.60	.01	−1.9 (−2.9 to −0.8)	<.001	0.348
SSI	9.5 (3.0)	−0.28	.27	−0.6 (−2.6 to 1.4)	.55	0.010
Death or serious morbidity	18.1 (3.2)	−0.60	.01	−2.7 (−5.4 to 0.1)	.06	0.156
Skill-related compositef	26.0 (4.6)	−0.51	.04	−3.9 (−7.5 to −0.3)	.03	0.201
Skill-unrelated compositeg	23.6 (4.0)	−0.20	.44	−1.4 (−5.4 to 2.6)	.49	0.034
All procedures (including colectomy)
Any complicatione	8.1 (1.5)	−0.54	.02	−1.5 (−2.7 to −0.4)	.01	0.275
Mortality	1.4 (0.2)	−0.38	.14	−0.1 (−0.2 to 0.1)	.22	0.092
Unplanned readmission	6.0 (0.9)	−0.52	.03	−0.8 (−1.6 to −0.1)	.03	0.239
Unplanned reoperation	2.6 (0.7)	−0.60	.01	−0.9 (−1.4 to −0.4)	<.001	0.351
SSI	4.0 (1.3)	−0.28	.27	−0.3 (−1.1 to 0.6)	.54	0.010
Death or serious morbidity	8.8 (1.7)	−0.60	.01	−1.2 (−2.5 to 0.1)	.07	0.134
Skill-related compositef	11.7 (2.2)	−0.51	.03	−2.0 (−3.7 to −0.2)	.03	0.216
Skill-unrelated compositeg	10.7 (1.9)	−0.21	.43	−0.6 (−2.5 to 1.3)	.52	0.029

Abbreviation: SSI, surgical site infection.

^aGeneralized linear mixed models for risk adjustment were done using all eligible surgery cases for the 391 surgeons in Illinois Surgical Quality Improvement Collaborative registry data. Multivariable analysis on adjusted outcome rates and surgical skill were at the clinician level and restricted for study participants (n = 17).

^bIn percentage points.

^cSpearman rank-order correlation coefficient and P value of statistical test for Prob>|r| under H0: Rho = 0 (in parenthesis).

^dχ² Test based on robust regression parameter estimation.

^eThe occurrence of any of the 17 postoperative common complications designated by the American College of Surgeons National Surgical Quality Improvement Program, such as cardiac arrest requiring cardiopulmonary resuscitation, myocardial infarction, cerebrovascular accident/stroke with neurologic deficit, deep vein thrombosis/thrombophlebitis, sepsis, septic shock, acute renal failure, progressive renal insufficiency, urinary tract infection, on ventilator more than 48 hours after the principal procedure, pneumonia, pulmonary embolism, unplanned intubation, superficial incision SSI, deep incisional SSI, organ/space SSI, and wound dehiscence.

^fA composite indicator for any of the following postoperative outcomes that were believed clinically likely to be related to surgeon technical skill: transfusions related to intraoperative bleeding, unplanned readmission likely related to the principal procedure, unplanned reoperation, wound dehiscence, or organ/space SSI; for colectomy, this indicator also included occurrence of postoperative anastomotic leak.

^gIndicator for presence of any of the following postoperative outcomes that were unlikely related to surgeon technical skill, including 15 of the 17 components of any complication outcome except wound dehiscence and organ/space SSI.

Figure. Association Between Surgeon Technical Skill Score and Risk-Adjusted Postoperative Colectomy Outcomes.

Risk-adjusted outcome rate by technical skill score with robust regression fitted line.

When combining colectomy and noncolectomy procedures, the same 5 patient outcomes (any complication: r = −0.54, P = .02; unplanned readmission: r = −0.52, P = .03; unplanned reoperation: r = −0.60, P = .01; death or serious morbidity: r = −0.60, P = .01; and the skill-related composite: r = −0.51, P = .03) were statistically significantly associated with a surgeon’s technical skill score (Table 4; Figure).

When adjusting for outliers by using robust regression despite the large pairwise correlation (r = −0.60), technical skill score was no longer statistically significantly associated with death or serious morbidity at the 0.05 level (colectomy specific P = .06). However, associations of technical skill score with any complication, unplanned readmission, unplanned reoperation, and the skill-related composite (eg, surgical bleeding, reoperation, and related readmission) remained for colectomy-specific outcomes and for outcomes from all procedures (Table 4).

For colectomy procedures, a 1-unit increase (eg, from 2 to 3 or from 3 to 4) in technical skill score was associated with a decrease in the any complication rate by 3.1 percentage points (β = −3.1; 95% CI, −5.6 to −0.7; P = .01), or nearly 1 SD (3.2 percentage points). Similarly, a 1-unit increase in technical skill score was related to a 1.9 percentage point (β = −1.9; 95% CI, −2.9 to −0.8; P < .001) decrease in unplanned reoperation rate; a 1.4 percentage point (β = −1.4; 95% CI, −2.7 to −0.1; P = .04) decrease in readmission rate; and a 3.9 percentage point (β = −3.9; 95% CI, −7.5 to −0.3; P = .03) decrease in the skill-related composite outcome. We found no evidence for surgeon technical skill being associated with the skill-unrelated composite outcome (β = −1.4; 95% CI, −5.4 to 2.6; P = .49).

Model R² was used in a robust regression model to estimate the proportion of variation in a patient outcome associated with the surgeon technical skill score (Table 4). Overall, technical skill scores were associated with 25.8% of the variation in colectomy complication rates and 27.5% of the variation when including noncolectomy procedures. When examining colectomy and other procedures, technical skill scores were associated with 21.6% of the variation in the skill-related composite outcome (R² = 0.216) but only 2.9% of the variation in the skill-unrelated composite (R² = 0.029) (Table 4; eFigure in the Supplement). When skill-related and skill-unrelated outcomes were grouped, low skill–level surgeons compared with high-skill surgeons had worse skill-related patient outcomes (13.5% vs 10.2%, P = .07), but there was no significant difference in skill-unrelated patient outcomes (11.0% vs 9.7%, P = .42) (Table 3).

Discussion

We found that better surgeon technical skills, measured by video-based peer review of laparoscopic colectomy, appeared to be significantly associated with better patient outcomes following colectomy. Moreover, we noted that a surgeon’s technical skill, scored from a single video of a colectomy, estimated the probability of patient outcomes for the surgeon’s noncolectomy procedures. Variation in surgeon technical skill score alone appeared to explain approximately 26% of the variation in patient complications. These findings suggest that focusing quality improvement efforts on the technical skill in a surgical procedure is an important, yet overlooked, opportunity to improve perioperative patient outcomes.

Little previous research focused on understanding the association between surgeon technical skill and patient outcomes despite the widely held and long-standing belief that a surgeon’s skills are important determinants in the outcomes of surgery. Our study found significant surgeon-specific variation in technical skill score (2.8-4.6) of practicing surgeons performing laparoscopic colectomy and reported that the variation was significantly associated with colectomy-specific patient outcomes, with low-skill surgeons compared with high-skill surgeons having higher complication rates (20.6% vs 15.5%, P = .04).⁵

Previous studies examined the association between video-assessed technical skills and patient outcomes for laparoscopic bariatric procedures and found mixed results.¹⁴ For Roux-en-Y gastric bypass, technical skill was associated with patient outcomes, whereas for sleeve gastrectomy, technical skill scores were not associated with outcomes. However, there are several limitations to studying this issue in bariatric surgery. Complication rates overall are low for Roux-en-Y gastric bypass and sleeve gastrectomy, the procedures are typically done by fellowship-trained surgeons, all surgeons perform the procedures at bariatric centers of excellence, many of the surgeons almost exclusively perform bariatric procedures in their practice, and there is no association with outcomes of their nonbariatric procedures. In contrast, colectomy may be a more generalizable procedure to study to understand the association of surgeon technical skills and patient outcomes. Colectomies have higher complication rates, are performed primarily by general surgeons and by fewer colorectal fellowship–trained surgeons, with many surgeons performing laparoscopic colectomy never having received any formal laparoscopic training, and laparoscopic colectomy is typically a small percentage of a general surgeon’s overall procedure volume. In this study, we enrolled both fellowship- and non–fellowship-trained surgeons to reflect the true practice mix within the US, resulting in more generalizable results.

In a systematic review by Fecso et al,¹⁵ 24 studies that investigated patient outcomes associated with technical skill were analyzed. A positive association between superior technical skill and patient outcomes was found in 21 of the 24 studies. However, with the exception of one study,⁵ none of the other studies used objective methods to assess either surgeons’ technical skills or patient outcomes. In our study, the videos were reviewed using a structured, objective scoring tool, raters and participants were blinded, and the outcomes were not collected by the research team but through the ACS NSQIP for ongoing quality improvement. The double-blind nature of both the technical skills scores and patient outcomes removes the bias affecting most earlier studies. In this study, each video was reviewed by multiple surgeons who routinely perform laparoscopic colectomy and by 2 surgeons with expertise in video-based skill assessment. The number of reviews per video is comparable to that of previous studies⁵ but many fewer than in one study that used crowd-sourced scoring of a video of a surgical procedure by dozens of lay reviewers.¹⁶ However, the use of peers and experts in our study also provided guided, formative feedback in the form of written comments, furthering the ultimate goal of video scoring to promote quality improvement rather than just a score. Use of peer surgeons also provides improved face validity for the surgeon who receives a blinded score.

To validate the conceptual link between patient outcomes and surgeon technical skill, we created a composite of outcomes conceptually associated with technical skill and found that outcomes, such as postoperative bleeding, return to the operating room, and wound dehiscence, were significantly related to a surgeon’s technical skill score. When skill-related and -unrelated outcomes were grouped, low-skill surgeons had worse skill-related patient outcomes than high-skill surgeons, but there was no significant difference in skill-unrelated patient outcomes.

To our knowledge, this study is the first to note an associated spillover association between a surgeon’s skill in one procedure to other procedures performed by the surgeon. The study by Birkmeyer et al⁵ examining skill in Roux-en-Y gastric bypass did not include other types of procedures. Varban et al¹⁴ studied technical skill scores used for Roux-en-Y gastric bypass to estimate outcomes from sleeve gastrectomy and found no association. In this study, we were able to collect patient outcomes for all types of procedures performed by each surgeon and found a significant association between technical skill score assessed for colectomy and patient outcomes for noncolectomy procedures. Moreover, there was a 26.6% relative reduction in the overall morbidity of patients operated on by high-skill surgeons compared with those operated on by low-skill surgeons.

This analysis was part of a quality improvement effort to improve patient outcomes through a video-based coaching initiative. We are in the process of collecting updated technical skill scores following patient outcomes before and after intervention, and investigating technical and nontechnical markers of improvement over time. As we strive to improve the quality and safety of surgical care and patient outcomes, these findings may have significant implications, as they suggest that quality improvement efforts need to be expanded by improvements in the systems and processes of care to include a focus on improving surgeons’ technical skills. The opportunity for practicing surgeons to receive an assessment of their technical skills with guided feedback of opportunities for improvement is currently not routinely available, but these results suggest that creating such opportunities could improve surgical patient outcomes. Furthermore, by focusing on improving the technical skills of a specific procedure, there are likely to be additional benefits or spillover effects to all procedures performed by the surgeon.

Limitations

There are several limitations to our study. First, the approach to a right hemicolectomy was intentionally not standardized (eg, lateral to medial vs medial to lateral dissection; hand-assisted surgery vs straight laparoscopic surgery). While doing so might have introduced bias toward one approach compared with another, the lack of standardization does not detract from the robustness of the findings. Furthermore, the lack of standardization likely revealed the variation in technical skill, as it reflects variation in decision-making and mimics actual surgical practice. The lack of standardization of the surgical approach is important as future studies seek to differentiate the outcome of approach vs technical skill. The same can be said about the decision to allow videos in which trainees participated in the recorded procedure. Any elements of the score attributed to the trainee would reflect skill-unrelated variation, and the association between surgeon technical skill and patient outcomes was robust enough to persist despite this decision.

Second, this study was part of the ISQIC, and other quality improvement initiatives were also being implemented during the study period. Specifically, the ISQIC was focusing on decreasing SSIs in patients after colectomy, which may have affected our SSI outcome. Third, the study did not gather any patient characteristics (eg, demographics, comorbidities, and diagnosis) for the patients undergoing the video-taped colectomy. Although this information would have provided more nuance to the research findings, it was not collected to ensure patient privacy. In addition, right hemicolectomy is primarily performed for resection of cancer, and we therefore asked all reviewers to assume the procedure was being performed for resection of a colon cancer. Another potential limitation to our study is the volume of videos recorded and submitted for review. The number of participants was dictated by the quality improvement collaboration around video coaching and not by a power analysis. A lack of association could have been from inadequate power to detect a difference. However, the fact that a statistically significant association was seen suggests the robustness of these findings. In addition, surgeons self-selected to participate and submit their videos; therefore, the findings may not be representative of all surgeons but rather a group with higher than average skills and willing to participate or a group with lower than average skills seeking feedback. This limitation can be addressed by further study in a larger sample.

Conclusions

In this study, the technical skill scores of surgeons performing laparoscopic colectomy varied widely. Higher technical skill scores were associated with fewer postcolectomy complications and fewer complications for all of a surgeon’s procedures. Variation in surgeon’s technical skill score appears to explain approximately 26% of the variation in postoperative complications. Each 1-point improvement in technical skill score was associated with a 3.1% absolute decrease in postcolectomy complication rates. These results suggest that efforts to improve surgeon technical skill scores could improve patient outcomes, and work is needed to develop feasible and sustainable ways to assess and improve surgeons’ technical skills as part of broader surgical quality improvement initiatives.

Supplement.

eMethods. Detailed Methods

eReferences

eTable 1. Inter-Rater Reliability Statistics by Technical Skill Question

eTable 2. Pairwise Pearson Correlation Coefficients Among Constructed Surgeon Skill Scores

eTable 3. Patient Factors Accounted for in Risk Adjustment and Generalized Linear Mixed Models

eFigure. Association Between Surgical Technical Skill Score and Risk-Adjusted Postoperative Outcomes for Procedures of All Types