Video Ratings of Surgical Skill and Late Outcomes of Bariatric Surgery

Abstract

Importance

Measures of surgeons’ skill have been associated with variations in short-term outcomes after laparoscopic gastric bypass. However, the impact of surgical skill on long-term outcomes after bariatric surgery is unknown.

Objective

To study the association between surgical skill and long-term outcomes of bariatric surgery

Design

Surgeons were ranked on their skill level through blinded peer video review, and sorted into quartiles of skill. Outcomes of bariatric surgery were then examined at the patient level across skill levels.

Setting

The Michigan Bariatric Surgical Collaborative, a prospective clinical registry of 40 hospitals performing bariatric surgery in the state of Michigan

Participants

20 surgeons performing bariatric surgery who submitted videos for anonymous peer ratings; patients undergoing surgery with these surgeons for whom one year follow-up data postoperatively was available.

Exposure

Surgeon skill level.

Main Outcome Measures

Excess body weight loss at one year; resolution of medical comorbidities (hypertension, sleep apnea, diabetes, hyperlipidemia), functional status, patient satisfaction.

Results

Peer ratings of surgical skill varied from 2.6 to 4.8 on a 5-point scale. There was no difference between the best (top 25%) and worst (bottom 25%) performance quartiles when comparing excess body weight loss (67.2% excess body weight loss vs 68.5%, p=.89) at one year. There were no differences in resolution of sleep apnea (62.6% vs 62.0%, p=.77), hypertension (47.1% vs 45.4%, p=.73), or hyperlipidemia (52.3% vs 63.4%, p=.45). Surgeons with the lowest skill rating had patients with higher rates of diabetes resolution (78.8%) when compared to the high-skill group (72.8%, p=0.01).

Conclusions and Relevance

In contrast to its impact on early complications, surgical skill did not impact postoperative weight loss or resolution of medical comorbidities at one year after laparoscopic gastric bypass. These findings suggest that long-term outcomes after bariatric surgery may be less dependent on a surgeon’s operative skill and instead be driven by other factors. Operative technique was not assessed in this analysis and should be considered in future studies.

INTRODUCTION

After a decade of focus on optimizing perioperative care, there has been a recent paradigm shift towards optimizing the quality of the operation itself. Until recently, surgical quality improvement has largely focused on improving systems of care, such as improving compliance with perioperative checklists or adherence to evidence based guidelines for antibiotic administration.^1–3 Efforts to improve perioperative care have contributed to significant gains in patient safety, but have failed to provide insight on the impact of the surgeon on the operation itself. Studies from the surgical education literature have demonstrated that it is possible to directly assess surgical skill with high levels of reliability and reproducibility.^4–6 Recently, these ratings of surgical skill were shown to be strongly associated with variation in postoperative outcomes.⁷

However, understanding how surgical skill relates to outcomes is in its nascent stages. While existing research has shown a link between surgeon skill level and short-term outcomes (e.g. 30-day complication rates),⁷ there has been no data evaluating the link between longer-term outcomes and surgeon performance. Long-term outcomes are important to consider in a number of operations, particularly after bariatric surgery when weight loss and comorbidity resolution occur typically beyond the first month after surgery. There are multiple ways that long-term outcomes could be impacted by surgical skill. For example, short-term outcomes (e.g., complications) could mediate suboptimal longer-term outcomes. For instance, a postoperative complication such as a leak or a stricture that resulted from an intraoperative technical error may result in a prolonged hospitalization and slow recovery. In turn, this may result in a patient’s limited ability to participate in postoperative nutritional counseling and exercise programs long-term. To date, there are no studies that evaluate the relationship between surgeon skill and these late operative outcomes. In addition, the current metrics of skill level have largely been validated in the educational realm and used to rate novices such as surgical residents. These current skill measures may have limitations in their applicability which merit further investigation.

In this context, we conducted a study evaluating the relationship between surgeon skill and late surgical outcomes. To do this we focused on laparoscopic roux-en-y gastric bypass (LRYGB) since these patients have high rates of postoperative follow-up and multiple quantitative metrics on which to measure their late outcomes of surgery. We used existing surgical skill ratings of 20 practicing bariatric surgeons in the state of Michigan, as judged anonymously by their peers who evaluated the video of a typical case using a validated instrument. We then assessed the relationship between these skill ratings and long-term outcomes of bypass surgery, chiefly weight loss and resolution of medical comorbidities.

METHODS

Data Source and Study Population

This study used data collected from the Michigan Bariatric Surgery Collaborative (MBSC), a prospectively collected clinical registry that includes 40 hospitals performing bariatric surgery within the state of Michigan.^8,9 The collaborative collects data for both primary and revisional bariatric surgery that includes approximately 6000 patients annually.

The MBSC registry includes basic patient demographics and comorbidities, short-term outcomes and 30 day complication rates; it additionally captures long-term weight loss and health status. All variable definitions are standardized across the 40 centers, and data collection is conducted by centrally trained data abstractors.¹⁰ The registry also has a high degree of intra-hospital communication and collaboration, with quarterly meetings between surgeons as well as data abstractors. Each hospital is visited annually and audited to ensure accurate and complete data collection.

Our patient population included all patients who underwent primary LRYGB by one of the 20 surgeons who participated in the video surgical skills study. Patient data was obtained from cases that occurred between 2006 (the start of the MBSC registry) and 2012 (when the video skill ratings were performed, for which one-year patient follow-up data was available‥

Participating Surgeons and Rating of Surgical Skill

In a previous study within our research group, 20 surgeons from the state were recruited to participate in this analysis. Participation was entirely voluntary. Each surgeon submitted a video of himself or herself performing a laparoscopic gastric bypass. Surgeons were instructed to submit a representative case (i.e. a “typical” case, but the choice of what video to submit was left to the individuals); all patient identifiers were removed from these videos.

Each video was professionally edited down to three critical portions of the operation: 1. Creation of the gastric pouch, 2. Gastrojejunostomy, 3. Jejunojejunostomy. Afterwards the videos were distributed electronically for peer review. The specifics of this video peer review process have been published in detail elsewhere.⁷ Each video had at least 10 peer reviews, and the rating instrument used was a modified version of the Objective Structured Assessment of Technical Skills (OSATS). The OSATS instrument is the most commonly used instrument for assessing surgical skill and has been externally validated demonstrating construct and content validity for use in surgical trainees and practicing bariatric surgeons.^4–6 Specifically for laparoscopic gastric bypass surgery, the procedure in this analysis, the OSATS was demonstrated to show wide variability among practicing bariatric surgeons. Moreover, the OSATS ratings were strongly linked to short-term outcomes, once again demonstrating construct validity.

Surgeons were rated using a 1-to-5 anchored Likert-type scale on first on overall skill, and then on the five OSATS domains of skill – 1. ease of tissue handling, 2. tissue exposure, 3. instrument handling, 4. efficiency of time and motion, and 5. flow of the operation. Anchors were used, with a skill rating of 1 indicated the level of a general surgery chief resident, a 3 that of an average bariatric surgeon, and a 5 a “master” surgeon. No training was given to the peer raters and no attempt was made to establish rating norms.

Outcome Measures

The primary outcome of interest was excess body weight loss (EBWL) at one year postoperatively. Secondary outcome measures were resolution of medical comorbidities at one year (diabetes, sleep apnea, hypertension, hypercholesterolemia). We also assessed overall health and function using the Health and Activity Limitation Index (HALex),¹¹ which has been validated for use in evaluating the quality of life in patients with obesity,¹² and patient satisfaction.

Statistical Analysis

In our conceptual model, skill measured at the surgeon level served as the exposure variable against which each outcome measure was assessed. Since each surgeon treated multiple patients, all analyses were adjusted for surgeon-level clustering.

As a measure of surgeon skill, the average peer rating across all six metrics of the rating instrument was used (the global rating and the 5 specific domains). The ratings across these domains correlated very closely with one another (correlation coefficients 0.85–0.97). This close correlation was the rationale for using a simple average, as discussed in our prior work. Surgeons were categorized into three groups, sorted by quartiles of skill level (the bottom group representing the lowest quartile; the middle group the second and third quartiles; the top group the fourth quartile). Outcome measures were assessed across these three groups using multivariate logistic regression models for each of our outcome measures. These models were risk-adjusted for age, male sex, previous venous thromboembolism, mobility limitations, coronary artery disease, and pulmonary disease; the methodology and justification for this risk adjustment has been described by our group in previous works.¹³ A post-hoc power calculation demonstrated that our analysis had >99% power to detect a 5% difference for all outcomes metrics (alpha=0.05 two sided).

All analyses were conducted in SAS 9.4 (SAS Institute, inc). This study was approved by the University of Michigan Institutional Review Board.

RESULTS

Surgeon Skill Ratings, Patient and Surgeon Characteristics

As previously described by our group, aggregate skill ratings for the 20 surgeons ranged from 2.6 to 4.8.⁷ The surgeons within the lowest skill quartile had mean skill ratings of 2.9; the highest quartile had mean ratings of 4.4 by comparison. The five individual domains of surgical skill correlated closely to the global rating, therefore we used an overall average of all six ratings for the skill measures.

Surgeons’ skill level did not correlate with years in practice, fellowship completion, or practice location (i.e. teaching vs non-teaching hospital). Surgeons in the top and bottom quartiles had each been practicing for a mean of 11 years (p=0.44); 20% of surgeons in each of the top and bottom quartiles respectively had completed fellowship training in minimally invasive or bariatric surgery (p=0.56). We did find that surgical skill level was positively associated with annual procedural volume. Our group has previously reported additional details on the demographics of the surgeons themselves.⁷

The total patient cohort included 3631 patients for whom follow-up data at one year for body weight, HALEX survey, and patient satisfaction were available. Table 1 demonstrates the characteristics of these patients across the quartiles of surgical skill. Overall the cohorts were similar in age, gender insurance status, and preoperative body mass index (BMI). There were several differences in patient comorbidities that were statistically significant (e.g. cardiovascular disease, musculoskeletal disease). However there was no clear relationship between these differences and the surgeon skill levels, and they are likely attributable to the large sample size rather than clinical significance.

Table 1.

Characteristics of patients undergoing laparoscopic roux-en-y gastric bypass, according to peer rating of surgical skill.

Patient characteristic		Level of surgical skill
		Quartile 1 5 surgeons 426 patients	Quartile 2 or 3 10 surgeons 1604 patients	Quartile 4 5 surgeons 1601 patients	p-value
Demographics
Mean age (yr)		46.8	48.5	47.2	0.002
Male sex (%)		16.7	20.1	19.3	0.272
Private insurance (%)		64.3	67.5	67.0	0.458
Clinical
Mean body mass index		49.0	48.4	48.1	0.149
Medical history (%)
Diabetes		38.0	40.0	37.7	0.370
Sleep apnea		56.6	45.9	45.3	<0.001
Hypertension		66.9	56.8	54.7	<0.001
Hyperlipidemia		58.9	57.1	54.0	0.084
Cardiovascular disease		67.6	58.4	56.7	<0.001
Musculoskeletal disorder		79.1	78.2	88.7	<0.001
Gastroesophageal reflux disease		58.0	47.8	47.7	<0.001
Psychological conditions		53.1	47.0	46.4	0.043
Smoking		40.9	45.3	43.4	0.226
Cholelithiasis		28.4	32.2	27.6	0.013
Lung disease		35.0	25.4	28.5	<0.001
Urinary incontinence		28.2	22.7	24.2	0.061
Mobility problems		11.7	5.1	4.4	<0.001
Liver disorder		3.8	6.3	4.4	0.018
Venous thromboembolism		4.7	4.7	3.4	0.125
Peptic ulcer disease		6.1	2.9	2.8	0.001

Excess body-weight loss

Body weight at one year was available for 3631 patients undergoing LRYGB among the 20 participating surgeons. There were no significant differences in EBWL at one year between the groups of surgical skill. Figure 1 shows the EBWL subdivided by quartile of surgeon skill. Patients undergoing operations by surgeons in the top quartile of skill had a 67.2% EBWL compared to 68.5% in the bottom quartile (p=0.86). Average preoperative BMI was similar for each quartile as seen in Table 1.

Figure 1.

Risk-adjusted excess body weight loss (%) at one year, according quartile of surgical skill

Resolution of medical comorbidities

One year follow-up data was available for 976 patients with diabetes. Preoperatively, the incidence of diabetes was similar across quartiles of surgical skill. Of these patients, in the highest skill quartile 72.8% had resolution of diabetes (as defined by cessation of oral medication and/or insulin therapy), compared to 78.8% in the lowest skill quartile (p=0.01). For patients with hypertension, follow-up data was available for 1530 patients. 47.1% of patients in the highest skill quartile had resolution of hypertension, compared to 45.4% in the low skill quartile (p=0.73) For the 1151 patients with sleep apnea, there was no relationship between skill and resolution of sleep apnea (62.6% vs 62.0%, p=0.77), and similarly for the 1040 patients with hyperlipidemia, no relationship was seen (52.3% vs 63.4%, p=.45). Figure 2 shows the relative rate of comorbidity resolution across surgical skill levels.

Figure 2.

Risk-adjusted rate of resolution of medical comorbidities, according to quartile of surgical skill

For each comorbidity, the number of patients is smaller compared to our overall patient cohort, since the incidence rate of each comorbidity varies. In addition one year follow-up data was not available for all comorbidity measures in some patients.

Patient function and satisfaction

The HALex survey contains questions regarding perceived health and activity limitation; scoring of the survey ranges from 0.1 (most dysfunctional) to 1.0 (healthiest state). Among the 3631 patients who completed the HALex survey at one year, no relationship between surgeon skill and patient quality of life was seen, as patients in the lowest skill quartile had a mean HALex score of 0.83, compared to 0.84 in the highest skill group (p=0.57).

When patients were asked how satisfied they were with their surgeon and surgery at one-year, we did observe a positive relationship between surgeon skill and patient satisfaction. 90.3% of patients in the highest skill quartile responded with “very satisfied” at one year; compared to 87.1% in the lowest skill quartile (p=0.05). However, while this small difference achieved statistical significance, it likely lacks clinical significance as all quartiles had high overall satisfaction. Table 2 summarizes all outcome measures according to surgeon skill level.

Table 2.

Risk-adjusted rates of long-term outcomes after laparoscopic roux-en-y gastric bypass, according to peer ratings of surgical skill

Variable		Level of surgical skill			p-value
		Quartile 1	Quartile 2 or 3	Quartile 4
Excess Body Weight Loss (%)		68.5	67.6	67.2	0.86
Comorbidity Resolution(%)
Diabetes		78.8	75.8	72.8	0.01
Sleep apnea		62.0	65.7	62.6	0.77
Hypertension		45.4	45.1	47.1	0.73
Hyperlipidemia		63.4	52.5	52.3	0.45
HALex (0–1)		0.83	0.84	0.84	0.57
Patient Satisfaction (% Very Satisfied)		87.1	89.4	90.3	0.05

Discussion

This is the first study to examine the relationship between operative skill and long-term outcomes after surgery among practicing surgeons. Across multiple patient-level measures, we found no association between levels of surgical skill and long-term outcomes after laparoscopic gastric bypass. At one year, there was no difference in excess body weight loss when comparing patients who underwent surgery by a surgeon with a lower skill rating and patient who underwent surgery by a surgeon with a higher skill rating. Patients also experienced resolution of diabetes, hypertension, and hyperlipidemia at similar rates clinically, regardless of surgeon skill. Finally, there were no differences in patient reported quality of life associated with surgical skill ratings. These findings are somewhat counterintuitive, since there is a well-known relationship between other proxies of surgical skill (e.g., surgeon volume) and most measured outcomes. In our present study, our higher skilled surgeons were, on average, higher volume but we found no relationship between OSATS peer ratings, a more direct measure of surgical skill, and long-term outcomes.

Prior studies that examine surgical skill were targeted mainly to novices and trainees and had no means of determining the long-term effect of their skill on outcomes.^4,14 For bariatric surgery, variation in long-term outcomes have largely been assessed by comparing numerous patient factors, clinical and social support structure, and variables such as physical activity level, nutrition, and counseling.^15–17 It stands to reason that a surgeon’s operative abilities also have an effect on outcomes but prior metrics have been limited. Proxies for a surgeon’s skill, such as hospital volume and advanced fellowship training have been found to be correlated with outcomes.¹⁸ However, these metrics fail to address many attributes associated with a surgeons skill, such as dexterity, precision, tissue handling, flow of operation, tissue exposure and intraoperative decision-making skills. By analyzing peer-reviewed ratings of surgical videos, we had a unique opportunity to investigate the impact of these metrics on surgical outcomes.

Our study indicates that surgical skill does not have a profound effect on long-term outcomes after LRYGB, which stands in contrast to our prior study demonstrated a strong positive association between surgeon skill level and 30-day complication rates. Why would one expect surgical skill to be associated with long-term outcomes? There are multiple ways in which surgical skill could lead to poor long term outcomes. First, skill could impact long-term outcomes as delayed sequelae of the initial complication (e.g., a perioperative sleeve leak turning into a stricture, or reflux). Second, intraoperative skill deficiencies could lead directly to long-term problems. For example, “cork-screwing” or kinking of the sleeve not cause immediate problems but could ultimately manifest as stricture, or gastroesophageal reflux. Finally, both of these short- and long-term complications may lead to additional hospitalizations or contribute to the patient’s inability to adhere to the multidisciplinary weight loss or exercise programs.

One possible reason for our finding is that the videos were rated for surgical skill alone and did not take into account technique-specific differences between each operation. That is to say, we measured how well each surgeon executed the various steps of the procedure (“surgical skill”) but did not analyze the steps themselves (“operative technique”). Given the likely variation in surgeons’ judgment about operative technique (i.e pouch size, length of roux limb, length of biliopancreatic limb, type of anastomosis and management of hiatal hernias), there may be specific technical measures of quality affecting long-term outcomes that were not measured in this study. Another plausible explanation for the difference in short- and long-term outcomes between surgeons with high and low skill ratings is that long-term outcomes may be more dependent on program-specific measures, such as dietary education, patient engagement and resources that promote healthy life-style choices. Since surgeons within the MBSC meet quarterly to discuss and participate in a wide variety of quality improvement projects, it is possible that similar practice patterns have led to similar outcomes among all patients at 1-year, regardless of surgeon’s operative ability. Finally, despite performing risk adjusted statistical models to compare patients across various hospitals, there remains numerous genetic markers that may have a significant effect on weight loss and comorbidity resolution after bariatric surgery.¹⁹ Since these types of genetic markers have not been measured and many have yet to be determined, this study lacks the ability to identify them as potential significant predictors of weight loss.

Our study does have several limitations. First, this was a retrospective observational study; as such a number of factors outside of our measured variables may influence outcomes. However, we controlled for patient level factors using the detailed patient variables from a clinical registry and the best available methods of risk-adjustment. Second, there may be selection bias among participating surgeons. Specifically, it is conceivable that surgeons with the best skill might be the only ones willing to submit videos. As a result, surgeons participating in our study may not be representative of the full range of skill. However, we did have a wide range of surgical skill ratings based on the OSATS instrument. Moreover, in a prior study these variations in skill were tightly correlated with short-term outcomes. Thus, it is very likely that the surgeon skill levels included in this study span a clinically meaningful range. Furthermore, skill ratings were obtained from a single, self-selected video that was deemed to be ‘typical’. It is possible that the videos submitted demonstrated their very best single performance and not their overall practice. However, we previously performed a sensitivity analysis using a second video sample for each of the high and low performing surgeons, and there was a high level of consistency between videos.⁷

Third, there was no attempt to standardize or train the video raters, and due to the large number of reviewers, no two surgeons were evaluated by the same set of raters. Nevertheless, there remained considerable differences among the skill ratings between each video and we noted high inter-rater reliability (0.85 to 0.97 across the skill rating domains). Fourth, our skill ratings were obtained at a single point in time, however our patient outcomes were measured over a several year period. Surgeon skill may not have been static over this period as practicing surgeons refined their technique; an improved level of skill over time would potentially bias our findings toward a null result. However, all surgeons in our cohort had been in practice for similar durations (mean 11 years in both the top and bottom quartiles), so a significant change in performance over time would have been expected in either group

Finally, our ratings only measured surgeons’ skill for discrete portions of the operation - pouch creation, gastrojejunostomy and jejunojejunostomy. These three videos were selected because we believed that they represented the most technically challenging portions of the operation, but they may not reflect all of the events occurred in the operating room. Another aspect of surgical quality that was not measured in this study was the judgment and variation in surgical technique from surgeon to surgeon. Although surgical skill may account for differences in short term complication rates, it is possible that operative technique (as discussed above) may account for long-term outcomes.

Our study has several important implications for practice. Recently, attention has turned to better trying to evaluate and improve the conduct of an operation as a whole, through video and live coaching initiatives.²⁰ Laparoscopic surgery is a prime target of such work, since videos are simple to obtain using existing equipment as part of routine practice and therefore represent an untapped data source for studying both operative technique and surgical skill. These efforts have identified the importance of measuring both operative technique and surgical skill as well as better understanding the complex interplay between the two. We believe this present study is an important negative result, since it informs this ongoing work by providing information on which outcomes we should expect to be impacted by improving surgeon skill, and which will not be impacted. Further research is needed to understand the impact of surgical “technique” (how the operation is done) on long-term outcomes after bariatric surgery. Thus far, there has been great enthusiasm for improving surgical skill directly, through initiatives such as these as well as through regional collaboratives that allow dissemination of best practices.⁹ Our findings highlight the importance of ongoing scientific work analyzing surgical videos for a broader set of procedures and a diverse set of outcomes so we can better understand which outcomes are most dependent on surgical skill.

Conclusions

Peer-review ratings of surgical skill did not impact postoperative weight loss or resolution of medical comorbidities at one year after laparoscopic gastric bypass. Although surgical skill may influence short-term complication rates and patient satisfaction ratings, these findings suggest that long-term outcomes after bariatric surgery may be more dependent on other factors not yet measured among patients, hospitals or surgeons. Future studies should take advantage of video analysis by measuring both operative technique and surgical skill as a means of understanding a surgeon’s impact on surgical quality.