Machine Learning to Delineate Surgeon and Clinical Factors That Anticipate Positive Surgical Margins After Robot-Assisted Radical Prostatectomy

Abstract

Purpose:

Automated performance metrics (APMs), derived from instrument kinematic and systems events data during robotic surgery, are validated objective measures of surgeon performance. Our previous studies showed that APMs are strong outcome predictors of urinary continence after robot-assisted radical prostatectomy (RARP). We now use machine learning to investigate how surgeon performance (i.e., APMs) and clinical factors can predict positive surgical margins (PSMs) after RARP.

Methods:

We prospectively collected data of patients undergoing RARP at our institution from 2016 to 2019. Random Forest model predicted PSMs based on 15 clinical factors and 38 APMs from 11 standardized RARP steps. Out-of-bag Gini impurity index determined the top 10 variables of importance (VOI). APMs in the top 10 VOI were assessed for confounding effects by extracapsular extension (ECE) and pathologic T (pT) through Poisson regression with Generalized Estimating Equation.

Results:

55/236 (23.3%) cases had PSMs. Of the 55 cases with PSMs, 9 (16.4%) were pT2 and 46 (83.6%), pT3. The full model, including clinical factors and APMs, achieved area under the curve (AUC) 0.74. When assessing clinical factors or APMs alone, the model achieved AUC 0.72 and 0.64, respectively. The strongest PSM predictors were ECE and pT stage, followed by APMs in specific steps. After adjusting for ECE and pT stage, most APMs remained as independent predictors of PSM.

Conclusion:

Using machine learning methods, we found that the strongest predictors of PSMs after RARP are nonmodifiable, disease-driven factors (ECE and pT). While APMs provide minimal additional insight into when PSMs may occur, they are nonetheless capable of independently predicting PSMs based on objective measures of surgeon performance.

Introduction

Mounting evidence suggesting that surgical performance is directly associated with patient outcomes has led to much interest in assessing surgical performance.^1,2 Assessment of surgical performance previously relied on subjective peer and expert review; however, the recently rising interest in surgical performance has led to more objective measures of performance, including automated performance metrics (APMs).^2–5 APMs, derived from instrument kinematic and robot systems events data during robotic surgery, are a validated measure of objective surgeon performance and previous work from our group demonstrated that APMs are strong outcome predictors of urinary continence recovery after robot-assisted radical prostatectomy (RARP).^4–6 We now extend our work into analyzing the optimal oncologic outcome of the prostatectomy trifecta: freedom from cancer recurrence.⁷

One important risk factor for prostate cancer recurrence is positive surgical margins (PSMs) as they can affect postoperative management (i.e., recommendation for adjuvant therapy) and increase fear of recurrence, thereby affecting quality of life.^8–14 Surgeon experience, as measured by prior caseload or volume, has also shown significant associations with PSMs, suggesting that surgical performance may impact PSMs.^15,16 However, scarce data exist on the association between objective measures of surgeon performance and PSMs. Herein, we utilize machine learning methods to assess how patient clinical factors and surgeon performance as measured by APMs can be utilized to predict PSMs after RARP.

Methods

We conducted a retrospective review of patients undergoing RARP at our institution from 2016 to 2019. Under Institutional Review Board approval, clinical and surgical performance (i.e., APMs) data were prospectively collected. Each RARP was segmented into 11 standardized steps, and 38 APMs were measured for each step. APMs were collected using a systems events data recorder (Intuitive Surgical, Inc.) provided by Intuitive Surgical at a sampling rate of 50 Hz. APMs included kinematic data (e.g., instrument motion time, path length, wrist angulation velocity) and systems events (e.g., clutch use, camera movement) as previously described.⁵ All cases with complete clinical and surgical APM data were included. Patients receiving neoadjuvant radiotherapy were excluded.

The primary outcome of interest was PSM on final pathologic specimen as read by our institutional pathologists. PSM was defined as “tumor that extends to the inked surface of the prostatic specimen wherein the surgeon has cut across the tissue plane” and includes “failure to excise extraprostatic extension of prostate carcinoma, as well as intraprostatic (‘capsular’) incision into otherwise organ-confined tumor.”¹⁷

Patients were stratified into two groups based on margin status: PSMs and no PSMs (i.e., negative margins). Baseline differences in clinical and disease characteristics were assessed using Fisher's exact test for categorical variables and Mann–Whitney U for continuous variables as appropriate.

Random Forest machine learning algorithm predicted PSMs using 15 patient factors and disease characteristics and 38 APMs from 11 standardized RARP steps as candidate predictors. The candidate predictors are described in Table 1. Ten-fold cross-validation evaluated model performance. The full dataset was equally divided into 10-folds. We reiterated the learning process 10 times, applying the classifiers to each testing sample. Each study sample served as an independent testing case once. Receiver operating characteristic curve was constructed using the predicted probability from the 10 testing datasets combined. The area under the curve (AUC) with 95% confidence interval was used to assess prediction accuracy.

Table 1.

Automated Performance Metrics and Clinical Factors Used to Predict Positive Margins After Robot-Assisted Radical Prostatectomy

Automated performance metrics
Time-related metrics	Endowrist articulation metrics
Time to complete the task	Total radians of right instrument shaft rotation during task
Moving time of the right instrument	Total radians of left instrument shaft rotation during task
Moving time of the left instrument	Total radians of third instrument shaft rotation during task
Moving time of the third instrument	Total radians of right instrument wrist movement during task
Time of right instrument not moving during task	Total radians of left instrument wrist movement during task
Time of left instrument not moving during task	Total radians of third instrument wrist movement during task
Time of third instrument not moving during task	Total radians of right instrument jaw opening during task
Instrument kinematic metrics	Total radians of left instrument jaw opening during task
Path length of right instrument	Total radians of third instrument jaw opening during task
Path length of left instrument	Total radians of right instrument wrist articulation during task
Path length of third instrument	Total radians of left instrument wrist articulation during task
Moving velocity of right instrument	Angular velocity of right instrument articulation
Moving velocity of left instrument	Angular velocity of left instrument articulation
Path length of all three instruments	Clinicopathologic features
Camera movement metrics	Age
Path length of camera	Body mass index
Moving time of camera	Preoperative PSA level
Time of camera not moving during task	Preoperative Gleason Score
No. of camera adjustments during task	Final Gleason Score
Frequency of camera adjustments	ASA score
Mean velocity of camera movement during task	pT stage
Mean velocity of camera movement during each camera movement	Attending surgeon
System event metrics	Neoadjuvant hormone therapy
Master clutch usage during task	Surgery time
Energy usage during task	Left nerve sparing (none, partial, full)
Frequency of master clutch usage during task	Right nerve sparing (none, partial, full)
Frequency of energy application during task	Bladder neck reconstruction
No. of times surgeon's head out of console	Prostate weight
	ECE

ECE = extracapsular extension; PSA = prostate-specific antigen; pT = pathologic T.

Within each iteration, we applied a fivefold cross-validation to each learning process to determine the final prediction model before scoring through the 10% independent testing sample. For Random Forest, we used 800 trees with maximal depth of 50 and leaf size of 16. The variable to try was the square root of variable number. Gini impurity index was used as the loss function. For imbalanced outcomes, prior correction as previously described by King and Zeng was used.¹⁸

We assessed model performance from three different sets of predictors: (1) APMs only, (2) patient clinical factors only, and (3) all data (APMs and patient clinical factors). Out-of-bag Gini impurity index (GiniOOB) selected the top 10 variables of importance (VOI) based on average Gini index score ranked in descending order. APMs in the top 10 VOI were further assessed for confounding effects from extracapsular extension (ECE) and pathologic T (pT) through Poisson regression with Generalized Estimating Equation. Although ECE and pT are highly related clinically, they were entered into the model separately to avoid collinearity.

All statistics were performed using SAS 9.4. Two-sided p-value ≤0.05 was considered statistically significant.

Results

Patient demographics and clinical data

A total of 236 patients were included with median [interquartile range] age 66 [61–71] years and prostate-specific antigen (PSA) 7.0 [5.2–10.3] ng/mL. A total of 11 surgeons were included in this study. Of the 236 patients, 20 (8.5%) underwent neoadjuvant hormone therapy before RARP. The majority (87.7%) underwent nerve-sparing procedure. The pT stages of the overall cohort were: pT2 103 (43.6%), pT3 132 (55.9%), and pT4 1 (0.4%).

After stratification by margin status, we found that the PSM group had higher rates of nonorgan confined disease (pT ≥3) (83.6% vs 48.1%, p < 0.001) and observed significant differences in final Gleason score (p = 0.001) (Table 2). No other significant baseline differences were observed. Non-nerve-sparing cases had a higher PSM rate compared to nerve-sparing cases (34.4% vs 21.7%); however, this difference was not statistically significant (p = 0.13). A detailed description of patient demographic and clinical data is presented in Table 2.

Table 2.

Patient Clinical and Surgical Data of the Overall Cohort and After Stratification by Margin Status

	Overall	No PSM	PSM	p
N	236	181	55
Age, median [IQR] years	66 [61–71]	65 [61–70]	67 [62–72]	0.47
BMI, median [IQR] kg/m2	28.1 [25.2–31.4]	28.1 [25.2–31.1]	29.3 [25.5–31.8]	0.61
ASA, n (%)
1	4 (1.7)	3 (1.7)	1 (1.8)	0.67
2	104 (44.1)	82 (45.3)	22 (40.0)
3	125 (53.0)	93 (51.4)	32 (58.2)
4	3 (1.3)	3 (1.7)	0
PSA, median [IQR], ng/mL	7.0 [5.2–10.3]	6.8 [5.3–9.7]	8.3 [5.2–12.3]	0.11
Preoperative Gleason Score, n (%)
≤6	47 (19.9)	40 (22.1)	7 (12.7)	0.10
7	134 (56.8)	105 (58.0)	29 (52.7)
8	41 (17.4)	26 (14.4)	15 (27.3)
≥9	14 (5.9)	10 (5.5)	4 (7.3)
Preoperative ADT, n (%)
Yes	20 (8.5)	15 (8.3)	5 (9.1)	0.85
No	216 (91.5)	166 (91.7)	50 (90.9)
pT Stage, n (%)
≤2	103 (43.6)	94 (51.9)	9 (16.4)	<0.001
≥3	133 (56.3)	87 (48.1)	46 (83.6)
Prostate volume, g [IQR]	49 [38–67]	50 [39–67]	46 [37–64]	0.21
Median lobe, n (%)
Yes	28 (11.9)	20 (11.0)	8 (14.5)	0.48
No	208 (88.1)	161 (89.0)	47 (85.5)
Postoperative Gleason Score, n (%)
≤6	20 (8.5)	18 (9.9)	2 (3.6)	<0.001
7	173 (73.3)	139 (76.8)	34 (61.8)
8	11 (4.7)	9 (5.0)	2 (3.6)
≥9	32 (13.6)	15 (8.3)	17 (30.9)
Nerve-sparing procedure, n (%)
Yes	207 (87.7)	162 (89.5)	45 (81.8)	0.13
No	29 (12.3)	19 (10.5)	10 (18.2)

ADT = androgen deprivation therapy; BMI = body mass index; CCI = Charlson comorbidity index; IQR = interquartile range; PSM = positive surgical margins.

Positive surgical margins

The PSM rate of our cohort was 23.3% (55/236). Of the 55 cases with PSMs, 9 (16.4%) were pT2 and 46 (83.6%) were pT3. PSM rates for pT2 and pT3 disease were 8.7% (9/103) and 33.8% (45/133), respectively. PSM locations were as follows: 11 (20.0%) at bladder neck/prostatic base only, 4 (7.3%) at prostatic apex only, 17 (30.1%) at posterolateral only, and 14 (25.5%) multifocal.

Random forest model predicting PSMs

The full model, including patient clinical factors and APMs, achieved AUC of 0.74 in predicting PSMs (Fig. 1). When assessing patient clinical factors alone or APMs alone, the model achieved AUC of 0.72 and 0.64, respectively. Of the top 10 VOI based on the full model, the strongest predictors of PSMs were ECE and pT stage followed by APMs from specific steps, attending surgeon, and postoperative Gleason score (Fig. 2). Of the APMs listed in the top 10 VOI, all were associated with either the bladder neck dissection (4/6; 66.7%) or lymph node dissection (2/6; 33.3%) steps. Of the 6 APMs in the top 10 VOI, 2 (33.3%) were related to camera usage and 3 (50%) were related to instrument usage.

FIG. 1.

ROC curves of random forest models predictive for positive surgical margins. ROC = receiver operating characteristic.

FIG. 2.

Top 10 variables of importance predictive of positive surgical margins ranked by average Gini impurity index. APM = automated performance metric; pT = pathologic T.

Assessment of adjusted APMs for confounding effect

We found that all six APMs in the VOI were significantly associated with PSMs before being adjusted for ECE and pT stage (Table 3). After adjusting for ECE and pT, only one APM (path length of camera during left lymph node dissection) was no longer statistically significantly associated with PSMs (p = 0.07). Otherwise, we did not observe any considerable change in PSM risk for all other APMs in the top 10 VOI. The detailed results are listed in Table 3. Of note, the current state of APMs makes it difficult to determine exactly how specific APMs impact performance and outcomes.

Table 3.

Assessment of Adjusted Automated Performance Metrics for Confounding Effects by Extracapsular Extension and pT Stage

Automated performance metrics	Unadjusted RR (95% CI)	p	RR adjusted for ECE	p	RR adjusted for PT	p
No. of times surgeon's head is out of console during anterior bladder neck dissection	0.12 (0.02–0.84)	0.03	0.11 (0.01–0.88)	0.04	0.12 (0.02–0.93)	0.04
Moving time of camera during left lymph node dissection	1.09 (1.05–1.13)	<0.01	1.07 (1.03–1.11)	<0.01	1.07 (1.04–1.11)	<0.01
Path length of camera during left lymph node dissection	1.41 (1.08–1.84)	0.01	1.27 (1.0–1.63)	0.05	1.27 (0.98–1.65)	0.07
Time of left instrument not moving during anterior bladder neck dissection	0.8 (0.7–0.91)	<0.01	0.82 (0.73–0.93)	<0.01	0.82 (0.73–0.93)	<0.01
Moving velocity of right instrument during anterior bladder neck dissection	2.14 (1.31–3.5)	<0.01	2.15 (1.33–3.48)	<0.01	2.23 (1.38–3.61)	<0.01
Angular velocity of right instrument during posterior bladder neck dissection	1.13 (1.02–1.25)	0.02	1.12 (1.02.–1.24)	0.02	1.13 (1.02–1.24)	0.02

CI = confidence interval; ECE = extracapsular extension; RR = risk ratio.

Discussion

Accurate prediction of patient outcomes requires a comprehensive understanding of both patient and surgeon factors. The recent rise in technology, such as genomics and MRI, and machine learning methods have increased our understanding of patient factors in the screening and diagnosis of prostate cancer.^19,20 However, there is a paucity of data on how surgeon factors and performance impacts patient outcomes after RARP, specifically PSMs. APMs are objective measures of surgeon performance capable of predicting clinical outcomes after RARP with reasonable performance.^2,4–6 A single robotic surgery produces a tremendous amount of data, allowing for detailed analyses of surgeon performance (i.e., APMs) using artificial intelligence and machine learning methods.

In contrast to our previous work demonstrating that APMs are the strongest predictors of continence recovery after RARP, we found that nonmodifiable, disease-driven factors (ECE and pT) are the strongest predictors of PSMs. Although APMs do not provide much additional insight into predicting PSMs, they are nonetheless capable of predicting PSMs independently through objective measures of surgeon performance.

Previous studies have found that nonmodifiable, disease characteristics (e.g., ECE, pT, Gleason score), rather than patient factors (i.e., age, comorbidities) are the strongest predictors of PSMs after RARP.^9,10,21 Many of these studies, however, did not include measures of surgeon performance. The present study sought to expand upon previous studies by including surgeon performance (i.e., APMs). In a multi-institutional study of 8418 patients, Patel et al. reported an overall PSM rate of 15.7% (1272/8095), and PSM rates of 9.5% and 37.2% for pT2 and pT3 disease, respectively.¹⁰ We report comparable rates, with an overall PSM rate of 23.3% (55/236), and 8.7% and 34.6% for pT2 and pT3 disease, respectively. In their analysis, pT stage and preoperative PSA levels were the most important risk factors for PSMs. A separate prospective study by Ficarra et al. reported that ECE was the most relevant predictor of PSMs after RARP.²¹

Although the specific variables predicting PSMs may vary, nonmodifiable, disease-driven factors, specifically those representing more aggressive disease, generally seem to be the strongest predictors of PSMs. Similarly, we found that disease-driven factors are the strongest predictors of PSMs as 3/10 predictors in our VOI were related to disease characteristics, and the top 2 VOI were ECE and pT. APMs provided some additional insight into predicting when PSMs occur. When assessed independently, however, we found that APMs were capable of predicting PSMs, suggesting that surgeon performance may still impact PSM risk. Thus, APMs may provide an alternative method of predicting PSM risk based on surgeon performance and should be analyzed further in the future.

Another potential risk factor for PSMs is nerve sparing.²² In our study, nerve sparing was not identified as a significant predictor of PSMs in our top 10 VOI. Although non-nerve-sparing cases had a higher PSM rate compared to nerve-sparing cases (34.5% vs 21.7%), this difference was not significant (p = 0.13). While nerve sparing may increase the risk of PSMs as the neurovascular bundles overlay the prostate, patients undergoing non-nerve-sparing cases likely have more aggressive diseases and may be at a higher risk for PSMs or ECE to begin with.

Various studies have evaluated patient and surgeon-specific variations. Increased prior caseload and more years of experience have been associated with lower risk of PSMs and better outcomes, suggesting that surgeon performance may affect PSM.^23,24 Auffenberg and colleagues reported that top-performing surgeons achieved higher rates of 3-month continence recovery in all patients and higher negative margin rates in patients with pT2 disease after adjusting for patient-specific factors, suggesting that surgeon-specific variations may drive superior outcomes.²⁵ Similarly, we found that the attending surgeon is an important factor in predicting PSMs, further suggesting that surgeon-specific variations may affect PSM risk.

On analysis of a single surgeon's cases over time, Villamil and coworkers found significant reductions in PSM rates for pT2 disease; however, no reduction was observed in pT3 disease, suggesting that a surgeon's experience may not necessarily overcome the difficulties presented in nonorgan-confined disease such as microextensions visible only on histology.²⁴ Our findings were consistent with these observations, as APMs were predictive of PSMs when assessed independently, but added little additional predictive ability once clinical factors, specifically those representing more aggressive disease, were considered. The majority (66.7%) of APMs in the VOI predicting PSMs were associated with the bladder neck dissection, a common location for PSMs after RARP.¹⁰ Even after controlling for the confounding effects of ECE and pT, we found that 5/6 (83.3%) APMs significantly affected the risk of PSMs.

These results suggest that surgeon performance may affect PSMs independently, as the influence of surgical skill on PSMs was consistent regardless of disease aggressiveness. While our findings suggest that improving surgeon performance as quantified by APMs can potentially affect PSMs regardless of ECE status and pT stage, APMs in their current form are largely measures of efficiency and the exact significance of specific APMs, and their impact on performance have yet to be elucidated and will be the aim of future studies.

Our findings should be viewed considering some limitations. First, our study was conducted at a single tertiary center and the results may not be generalizable across other institutions. Although our findings were consistent with previously published results, our results should be externally validated. Second, although our PSM rates were comparable to previously published rates, our relatively small sample size may have limited the power of our statistical analyses. Finally, while our model's predictive performance was moderate (AUC 0.74) and our findings support previously reported literature, we acknowledge that there is still room for improvement.

Conclusion

Using machine learning methods, we found the strongest predictors of PSMs after RARP are nonmodifiable, disease-driven factors (ECE and pT). While APMs do not provide additional insight into when PSMs may occur, they are objective measures of surgeon performance that are capable of independently predicting PSMs. As our data accrue, we will transition our focus onto biochemical recurrence as the ultimate oncologic outcome after RARP.

Abbreviations Used

APMs

automated performance metrics

AUC

area under the curve

CI

confidence interval

ECE

extracapsular extension

MRI

magnetic resonance imaging

PSA

prostate-specific antigen

PSM

positive surgical margin

RARP

robot-assisted radical prostatectomy

ROC

receiver operating characteristic

RR

risk ratio

VOI

variables of importance

Authors' Contributions

R.S.L.: Protocol/project development, data collection or management, data analysis, and article writing/editing. R.M.: Protocol/project development, data collection or management, data analysis, and article writing/editing. S.P.: Protocol/project development and data collection or management. J.M.-S.: Protocol/project development and data collection or management. J.H.N.: Protocol/project development, data collection or management, data analysis, and article writing/editing. M.A.: Protocol/project development and article writing/editing. S.C.: Protocol/project development, data collection or management, data analysis, and article writing/editing. S.D.: Protocol/project development, manuscript writing/editing. A.J.H.: Protocol/project development, data collection or management, data analysis, and article writing/editing.

Ethics Approval

Institutional Review Board (IRB) HS-16-00318.

Consent to Participate

Informed consent was obtained from all individual participants included in the study.

Author Disclosure Statement

A.J.H. has financial disclosures with Quantgene, Inc., (consultant), Mimic Technologies, Inc., (consultant), and Johnson & Johnson (consultant). No competing financial interests exist for the other authors.

Funding Information

This study is supported, in part, by the National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health under Award Number K23EB026493 and an Intuitive Surgical Clinical Research Grant.