Objective performance indicators provide a novel, quantitative method to evaluate surgeon proficiency in robotic lobectomy training

Abstract

Objective

Current evaluation of robotic surgeon proficiency relies on subjective assessment. The robotic platform collects highly granular kinematic data on surgeon activity, known as objective performance indicators (OPIs). We sought to compare surgeon proficiency during lobectomies across training levels using OPIs.

Methods

Under institutional review board approval, we analyzed robotic lobectomies between November 2022 and February 2023 performed by 2 expert robotic thoracic surgeons (>200 robotic lobectomies) and their trainees using OPI recorders. A professional annotator segmented each case into standardized steps, and an operating surgeon (trainee or attending) was assigned to that step on the basis of the active console. Kinematic data were compared between surgeon groups. A subgroup analysis was performed dividing the trainee group into junior (postgraduate year 3-5) and senior residents (postgraduate year 6-8).

Results

In total, 26 lobectomies with 410 discrete tasks performed by attending surgeons and 344 by trainees were included. In the attending group, there were significantly greater rates of camera clutching per minute compared with trainees (2.94 vs 2.52, respectively; P = .0005). The ratio of right to left hand use was significantly greater in the trainee group (1.52 vs 1.48, P = .0047). Average instrument speed was faster in the attending group (1.24 vs 1.13 meters/min, P = .0061). Differences in clutching and speed, but not hand dexterity, remained significant when the trainee group was subdivided into beginner and intermediate robotic surgeons.

Conclusions

There are significant differences in objective performance indicators between expert and beginner robotic surgeons. These results demonstrate the feasibility of incorporating kinematic performance data into thoracic surgeon assessment in a clinical setting.

OPIs provide a method to measure trainees' performance against expert surgeons.

Central Message.

Objective performance indicators can be used to measure trainee performance in robotic lobectomy and compare that performance against highly experienced surgeons.

Perspective.

This demonstrates a novel use of objective performance data to measure surgeons learning robotic lobectomy in a clinical setting. These findings provide a foundation for using objective data to standardize robotic training, track learning curves, and identify milestones consistent with expert-level performance. This platform could ultimately replace current standard subjective assessment strategies.

Robotic thoracic surgery volume has increased dramatically over the last 10 years.1, 2, 3, 4 In fact, a recent Society of Thoracic Surgeons database analysis showed a 502% increase in the use of robotic lobectomy from 2013 to 2022.⁵ Despite its increased use, there is still no standardized approach to training or skill assessment. For the majority of surgeons, the current training model is determined on the basis of apprenticeship with an experienced surgeon.⁶ Furthermore, the standard methods of performance assessment rely on subjective evaluation, such as the Global Evaluative Assessment of Robotic Skills or Zwisch scale,^7,8 which is time-consuming, subjective, and has the potential for bias.⁹ There is a need for more accessible and more objective assessment for surgeons learning robotic lobectomy.

A unique and evolving capability of the robotic surgical platform is the ability to provide highly granular kinematic data, called objective performance indicators (OPIs), on surgeon movement. These OPIs provide an opportunity to capture, analyze, and discover critical aspects of surgeon performance during a case using objective metrics. Other surgical fields, particularly Hung and colleagues10, 11, 12 in the University of Southern California urology group, have described using OPIs to evaluate trainees and identify behaviors associated with expert-level proficiency. In thoracic surgery, the use of OPIs to date has been limited to preclinical evaluation, such as resident performance on an ex vivo porcine lobectomy simulation.13, 14, 15 To our knowledge, there are no previous reports of using OPIs to measure thoracic surgeon performance in a clinical setting. In the present study, we combined OPIs from robotic lobectomy cases performed by both trainee and expert robotic surgeons to identify critical differences in technique between these groups.

Methods

The institutional review board approved the study protocol and publication of the data (institutional review board 2020P002613, February 2020). We retrospectively identified lobectomies with Da Vinci Data Logger recordings (Intuitive Surgical) performed at our institution between November 2022 and February 2023. All cases were performed on a dual-console system by 2 highly experienced thoracic surgeons and a trainee, and all surgeons consented to collection of performance data. All trainees completed our institutional robotic curriculum, which consists of online training and basic exercises on a simulator console. A professional annotator segmented each case into critical steps, including division of the inferior pulmonary ligament, arterial, venous, and bronchial dissection; arterial, venous, and bronchial division; mediastinal lymph node dissection; and fissure dissection. The kinematic data (OPIs) for each case were retrieved from an online database maintained by Intuitive. Attending and trainee case and step designations were assigned on the basis of the console accounting for the majority of movement activity for a case, using similar methodology to previous studies.^16,17 Trainees were listed as the primary surgeon if their console was active for >50% of the case, but all cases had an attending surgeon present from start to finish. In almost all trainee cases, the attending surgeon performed all or part of some critical steps, such as dissection of the artery. Attending surgeons were defined as experts, having performed more than 200 robotic lobectomies each. Several OPIs were chosen for the comparison: rates of camera and finger clutching, instrument travel distance (meters), ratio of right hand movement to left hand movement, total wrist angulation (radians), and average instrument speed (meters per minute). OPIs were then compared between expert and trainee groups. Continuous variables were analyzed with Mann-Whitney U test. Categorical variables were analyzed with the χ² test. All statistical analysis was performed using STATA (StataCorp). For a subgroup analysis, trainees were subdivided into beginner (<10 robotic case experience) and intermediate groups (11-50 robotic case experience). One-way analysis of variance was used for the subgroup analysis.

Results

In total, 26 lobectomies were included, 13 of which were performed by an attending and 13 of which were performed by a trainee. Both groups performed a similar number of upper and lower lobectomies, on both sides. Cases were assigned to a trainee or attending on the basis of trainee experience (senior fellow). In rare cases, such as a large central tumor abutting hilar structures, the attending surgeon would be assigned the case as the result of anticipated difficulty. In most cases, the trainees start with basic tasks like taking down the pulmonary ligament or lysing adhesions. Attendings take over when appropriate on the basis of experience level, and, in almost all cases, the attendings perform the arterial dissection and bronchial dissection if the artery has not been divided. Two expert surgeons and 10 trainees participated in the cases. Within these 26 cases, 410 discrete operative steps were performed by an attending and 344 steps were performed by a trainee. There were no serious adverse events. There were no significant differences in the patient characteristics between the 2 groups (Table 1).

Table 1.

Patient characteristics for the attending and trainee group

Patient variables	Attending (n = 13)	Fellow (n = 13)	P value
Age, y, mean	68	69	.8
BMI, mean	27.7	26.7	.63
Gender			1
Male	8	8
Female	5	5
ASA score			.33
2	1	3
3	10	10
4	2
Smoking status			.29
Current	3	4
Former	6	3
Nonsmoker	4	6
Lobe
RUL	4	3	.49
RML	1	4
RLL	1	2
LUL	5	2
LLL	2	2

There were no significant differences across patient demographics or the operative lobe. BMI, Body mass index; ASA, American Society of Anesthesiologists; RUL, right upper lobectomy; RML, right middle lobectomy; RLL, right lower lobectomy; LUL, left upper lobectomy; LLL, left lower lobectomy.

First, an aggregate analysis, inclusive of all steps from each case, was performed comparing the OPIs between expert and trainee surgeons. There were significant differences in expert measurements compared with trainee measurements for multiple OPIs (Table 2). Camera-clutching was more frequent in the expert group (2.94 clutches/min) compared with the trainee group (2.52 clutches/min, P = .0005). There was significantly farther instrument travel (5.42 vs 5.06 meters, P = .0036) and lower instrument speed (1.13 vs 1.24 meters/min, P = .0096) in the trainee group. Although there was a trend toward greater wrist angulation and finger clutching in the attending group, these were not significant.

Table 2.

Aggregate analysis of each OPI across all steps and all cases in the study, comparing attending and trainee performance

OPIs	Expert (n = 410)	Trainee (n = 344)	P value
Finger clutch per minute	1.89	1.60	.0637
Camera clutch per minute	2.94	2.52	.0005
Total distance of both arms, m	5.06	5.42	.0036
Ratio R:L hand movement	1.48	1.52	.0047
Total wrist angulation, radians/min	48.26	41.61	.0585
Average instrument speed, m/min	1.24	1.13	.0096

OPI, Objective performance indicator; R:L, right to left.

Next, OPI measurements for each step of lobectomy were compared between experts and trainees. Finger clutching per minute was not significantly different between experts and trainees for all steps of lobectomy (Figure 1). However, camera clutching per minute was greater in the expert group for the venous dissection (4.24 vs 2.79, P = .0164) and hilar dissection (4.28 vs 3.44, P = .0433) but not for the remaining steps of lobectomy (Figure 2). Similarly, instrument travel was significantly shorter for the expert group compared with trainees during the arterial dissection (3.83 vs 7.17 meters, P = .0494), hilar dissection (3.84 vs 4.65 meters, P = .0166), and mediastinal lymph node dissection (5.04 vs 8.09 meters, P = .0190) (Figure 3). Average instrument speed was only significantly different during arterial dissection (Figure 4), where experts moved faster than trainees (1.29 vs 0.99 meters/min, P = .0003).

Figure 1.

Average rates of finger clutching during each step of lobectomy between experts (blue) and trainees (red). Although the expert group tended to clutch more frequently, the differences by step were not significant compared with trainees.

Figure 2.

Average rates of camera clutching per minute during each step of robotic lobectomy by experts (blue) and trainees (red). Experts tend to clutch at greater rates, especially during the venous and hilar dissection. Steps without listed P values were not statistically different.

Figure 3.

Average instrument travel for all arms (in meters) during each step of pulmonary lobectomy, by experts (blue) and trainees (red). Experts use less motion to complete a task, particularly during the arterial dissection, hilar dissection, and lymph node dissection. Steps without listed P values were not statistically different.

Figure 4.

Average instrument speed for each step of lobectomy between experts (blue) and trainees (red). There was a trend toward faster movement for experts in all steps, but those differences were only significant during the arterial dissection.

In the subgroup analysis, the trainee group was further divided into a beginner group, consisting of 4 general surgery residents (<10 robotic case experience), and an intermediate group, consisting of 6 cardiothoracic fellows (11-50 robotic case experience). OPIs across all steps were then compared between the expert, intermediate, and beginner groups. In this analysis, the ratio of camera and finger clutching was significantly different between all 3 groups (Figure 5). Similarly, the total distance traveled was significantly different between the 3 groups (Figure 6). There was a steady decrease in distance traveled in the beginner group (34.63 meters) compared with the intermediate group (28.32 meters) and the expert group (23.26 meters) (P = .0008). Instrument speed also increased with experience, from an average speed of 1.03 m/min in the beginner group to 1.25 m/min in the expert group (P = .0002).

Figure 5.

Average rates of camera (blue) and finger (red) clutching per minute between beginner, intermediate, and expert robotic surgeons. There is a trend towards increased frequency of clutching with increasing experience.

Figure 6.

Average distance traveled per step between beginner, intermediate, and expert robotic surgeons. There is a trend toward less total movement with increasing level of experience, which was statistically significant.

Discussion

In this study, we used objective performance indicators to measure performance by surgeons of varying levels of experience during robotic lobectomy. This study provides a foundation for incorporating OPIs into surgical training and surgeon evaluation for robotic lobectomy. There are several limitations, including the small case number, the heterogeneity in experience for the trainee group, and the differences in patient selection for attendings versus trainees that could not be completely controlled for in this retrospective analysis.

The analysis of each OPI for the steps of lobectomy revealed trends in performance between the 2 groups. Vascular dissection, particularly the pulmonary artery, showed significantly better performance in the expert group for multiple OPIs. Conversely, steps that we would consider easier to master, like the pulmonary ligament division, showed more similar performance between the groups. This seems to suggest that trainees appropriately take more caution when performing the riskier parts of the operation, compared with experts who continue to operate more comfortably. Indeed, expert surgeons move more efficiently in robotic nephrectomy in similar studies in the urology and general surgery literature.^10,18,19 Some results were interesting and initially might seem counterintuitive, such as attendings using greater total motion (not significant) to divide the artery and complete the fissure. There are some possible explanations for this finding, such as attendings performing more difficult arterial divisions with a harder angle or tighter window. The attendings are also more likely to take over for a totally undeveloped fissure, whereas the trainees are often allowed to divide a near-complete fissure that involves less dissection and stapling. We would expect more motion and longer operative times in these scenarios, and indeed a similar variation in OPI performance has been shown with difficult robotic cholecystectomy, proctectomy, and prostatectomy depending on a patient's anatomy.20, 21, 22 For similar reasons, we did not compare total operative time between experts and trainees. Most operations involved some activity from both groups, so the operative times were not fully reflective of expected skin-to-skin times for experts and trainees.²³ Comparing operative times and OPIs between expert surgeons and practicing surgeons on their robotic learning curve could be a valuable study in the future that eliminates the variable involvement of a trainee.

Finally, the subgroup analysis breaking trainees into general surgery residents and cardiothoracic fellows demonstrates findings consistent with what we would expect regarding surgeon performance over time. We saw steady increases in the clutching frequency and instrument speed, as well as decreased total distance traveled, with increasing level of experience. These results are similar to findings in the general surgery and urology literature.^11,18,19,24 This opens the door to future applications of OPIs in tracking learning curves and assessing trainee readiness to perform more steps of the lobectomy.

Over the last few years, there has been increasing interest in using OPIs in thoracic surgery. For example, in a perfused porcine lobectomy model, several OPIs were found to correlate with vascular injury and bleeding.¹⁴ In addition, Lazar and colleagues²⁵ noted the value of OPIs in providing individualized, granular surgeon assessment. They also outlined the feasibility of using OPIs in clinical outcomes assessment, and called for further studies in clinical models. We believe the results of this study will further establish a foundation to use OPIs in clinical thoracic surgery. OPIs could be used to track learning curves, assess surgeon skill, and enhance, or even standardize, training curricula. Particularly in simulation, OPIs can be used to identify specific areas of weakness, and inform targeted simulation training to improve those specific OPIs. In fact, we plan to incorporate individual residents’ OPIs into a system that defines readiness to progress into increasing autonomy for lobectomies. As we collect more data, aggregate performance averages will be used as a benchmark for residents that can be reviewed quarterly. This provides a method to ensure trainees are progressing to a point that they will be comfortable independently in practice.

Future work could also investigate the associations between OPIs and clinical outcomes, offering a new avenue to explore surgical behaviors that might increase the risk of intraoperative events or postoperative complications, as done in the urology field.^10,12 Finally, OPIs could offer a basis for assessing entrustable professional activities among trainee robotic surgeons. In summary, OPIs offer valuable new insight into the safer training and practice of robotic lobectomy, and this study offers a proof-of-concept in using OPIs to identify objective differences in experienced versus novice surgeon behavior.

Conclusions

In this study, we show that expert surgeons are more efficient in some, but not all, objective performance indicators during a robotic lobectomy. These metrics provide granular, objective measurement of surgeon proficiency on the robotic platform and can be used to track progress throughout training. This is particularly valuable for residents in several ways: as a basis for focused simulation, as a tracking mechanism during their learning curves, and supporting decisions to allow increasing autonomy in performing critical steps in a robotic lobectomy.

Webcast

You can watch a Webcast of this AATS meeting presentation by going to: https://www.aats.org/resources/objective-performance-indicato-7373.

Conflict of Interest Statement

H.A. serves as a proctor for Intuitive Surgical, Inc. L.S. serves as a proctor and advisor for Intuitive Surgical, Inc. M.S. is employed by Intuitive Surgical, Inc, in the Data Analytics department. All other authors reported no conflicts of interest.

The Journal policy requires editors and reviewers to disclose conflicts of interest and to decline handling or reviewing manuscripts for which they may have a conflict of interest. The editors and reviewers of this article have no conflicts of interest.