Global Evaluative Assessment of Robotic Skills in Endoscopy (GEARS-E): objective assessment tool for master and slave transluminal endoscopic robot

Abstract

Background and study aims

 The Master and Slave Transluminal Endoscopic Robot (MASTER) is a novel robotic endosurgical system with two operating arms that offer multiple degrees of freedom. We developed a new assessment tool, the Global Evaluative Assessment of Robotic Skills in Endoscopy (GEARS-E), derived from existing tools in laparoscopic and robotic surgery, and evaluated its utility in the performance of procedures using MASTER.

Methods 

This was a pilot study conducted in vivo and ex vivo on animals. Three operators (Novice-1, Novice-2 and Expert) performed a total of five endoscopic submucosal dissections (ESD) using MASTER. Novice operators had no MASTER experience and the expert had previously performed eight MASTER ESDs. Operator performance was assessed by four independent evaluators using GEARS-E, which has a maximum score of 25 for five domains representing various skill-related variables (depth perception, bimanual dexterity, efficiency, tissue handling and autonomy).

Results 

The mean global rating scores for Novice-1 first attempt, Novice-1 second attempt, Novice-2 first attempt, Novice-2 second attempt and Expertʼs cases were 13.0, 16.0, 13.3, 15.5, and 21.5, respectively. The mean scores of each of the five domains were statistically higher for the second attempts compared to the first attempts for both Novice-1 and Novice-2. The mean scores of each of the five domains for the Expertʼs case were consistently higher than those for the two novice operators in both their first and second attempts.

Conclusion 

Results using GEARS-E showed correlations between surgical experience and MASTER ESD. As an assessment tool for evaluation of surgical skills, GEARS-E has great potential for application in MASTER procedures.

Introduction

Endoscopic submucosal dissection (ESD) has become standard care for early gastrointestinal cancers ^{1 2 3} . Master and Slave Transluminal Endoscopic Robot (MASTER) is a novel robotic endosurgical system that has two operating arms equipped with multiple degrees of freedom, enabling complex maneuvers ⁴ . The feasibility of performing MASTER ESD has been shown in previous studies ^{5 6 7} . Going forward, the value of MASTER will need to be evaluated in terms of the user’s learning curve and training efficacy. Therefore, it is essential to develop an appropriate training program and a suitable skill assessment tool.

To provide an objective assessment tool in minimally invasive surgery, the Global Operative Assessment of Laparoscopic Skills (GOALS) was developed ⁸ . More recently, the Global Evaluative Assessment of Robotic Skills (GEARS), which was modeled after GOALS, was also developed ⁹ . There has not been an objective assessment tool developed for endoluminal surgical platforms such as MASTER. In this study, we explored use of a new scoring method, the Global Evaluative Assessment of Robotic Skills in Endoscopy (GEARS-E), which was modified from both GOALS and GEARS in evaluation of the learning curve of novice operators using MASTER.

Methods

Procedures

This study involved in vivo and ex vivo experiments. Participating operators performed ESD using MASTER on two adult female pigs, as well as on isolated pig tissues obtained from a commercial source. All experiments were conducted at Innoheart Pte Ltd with prior approval from the Institutional Animal Care and Use Committee. As for the in vivo experiments, the endoscope was introduced under general anesthesia and the operator telemanipulated the two robotic arms, one holding a cautery knife and the other a grasper, via cables that passed through the two channels of the endoscope. For the ex vivo experiments, the isolated colorectal segment was mounted on a stand to simulate its normal orientation. Then, an imaginary lesion was marked on the mucosa and MASTER ESD was performed. The pigs were euthanized after the procedures.

GEARS-E

In creating GEARS-E, we expanded the domains in GOALS and GEARS to encompass features unique to MASTER procedures ( Table 1 ). Modeled after GOALS and GEARS, GEARS-E comprises six domains representing various skill-related variables (depth perception, bimanual dexterity, efficiency, tissue handling, autonomy and endoscope control). Assessment of each domain is done by scoring on a 5-point Likert scale with performance anchors at 1, 3 and 5. A rating of 1 denotes the lowest proficiency while 5 is the highest. As the focus of this study was on the novice’s training on robotic manipulators, we left the endoscope manipulation to a skilled endoscopist independently. To minimize bias from this, we excluded the sixth domain, endoscopic control, from the assessment. Therefore, the global rating score for this study was the sum of the scores for each of the remaining five domains, the maximum total score of which is 25.

Table 1. Six domains of GEARS-E.

Depth perception
1	2	3	4	5
Constantly overshoots target, wide swings, slow to correct		Some overshooting or missing of target, but quick to correct		Accurately directs instruments in the correct plane to target
Bimanual dexterity
1	2	3	4	5
Uses only one robotic arm, poor coordination		Uses both arms, but does not optimize interactions between arms		Expertly uses both arms in a complementary way to provide best exposure
Efficiency
1	2	3	4	5
Inefficient efforts, many uncertain movement, constantly changing focus or persisting without progress		Slow, but planned movements are reasonably organized		Confident, efficient and safe conduct, maintains focus on task, fluid progression
Tissue handling
1	2	3	4	5
Rough movements, injures tissues, poor grasper control, lack of tension		Handles tissues reasonably well, minor trauma to tissues, not optimal tension		Handles tissues well, applies appropriate tension, negligible injury to tissue
Autonomy
1	2	3	4	5
Unable to complete entire task, even with verbal guidance		Able to complete task safety with moderate guidance		Able to complete task independently without prompting
Endoscope control
1	2	3	4	5
Consistently does not optimize view and scope position, extremely frequent and inefficient adjustment		View and scope position is sometimes not optimal, needs to adjust scope position frequently		Controls view and scope position optimally and independently

Study design

Five procedures were performed by three operators (Novice-1, Novice-2 and Expert) ( Fig. 1 ). Novice operators were overseas gastroenterologists who had no MASTER experience, but had performed conventional ESD, with Novice-1 having accumulated experience with approximately 500 ESD cases and Novice-2, more than 1000 such cases. Expert had previously performed eight animal and human MASTER ESD. To begin with, novice operators watched a video on the performance of a previous MASTER ESD and then underwent 15 minutes of practice in manipulating the robotic arms under virtual reality simulation, as well as in a dry laboratory setting. Then, each novice operator performed one ex vivo MASTER ESD on the rectum. Subsequently, each novice operator performed one in vivo MASTER ESD (a gastric ESD by Novice-1, and a sigmoid colon ESD by Novice-2). Expert separately performed an in vivo MASTER ESD on the sigmoid colon. Every procedure was self-evaluated using GEARS-E by all three participants, and by an independent evaluator, who had previously performed three MASTER ESD but was not involved in the procedures in this study. Statistically significant differences were evaluated by paired two-sided t -test followed by multiple testing correction using Holm method ( P  < alpha = 0.05 /n).

Fig. 1 .

Study design of in vivo and ex vivo animal experiments. Five procedures in all were performed by three operators (Novice-1, Novice-2 and Expert) and assessed by four evaluators (including self-evaluations) using GEARS-E.

Results

A total of five procedures were successfully completed, of which two were done each by Novice-1 and Novice-2, and one by Expert. Every procedure was assessed by the four evaluators including the Novice-1, Novice-2, Expert and the independent evaluator. The proficiency and performances of the operators as reflected by their mean scores from 4 evaluators in the 5 domains is summarized in Table 2 . The mean global rating scores for Novice-1 first attempt, Novice-1 second attempt, Novice-2 first attempt, Novice-2 second attempt and Expert’s case were 13.0 (6 – 21), 16.0 (12 – 19), 13.3 (6 – 23), 15.5 (13 – 20) and 21.5 (14 – 25), respectively. The mean scores for each of the five domains were statistically higher in the second attempts compared to the first attempts for both Novice-1 ( P  = 0.0007 < alpha/4) and Novice-2 ( P  = 0.0486 < alpha/1) ( Fig. 2 ). The results suggested that incremental changes in proficiency following repeated procedure. On the other hand, the mean scores for each of the five domains for Expert’s case were consistently higher than those for the two novice operators in both the first and second attempts ( P  = 0.0012 < alpha/2, P  = 0.0008 < alpha/3, respectively).

Table 2. Means and ranges of scores in the five domains and global rating scores.

GEARS-E	Novice-1 1st	Novice-1 2nd	Novice-2 1st	Novice-2 2nd	Expert
Depth perception	2.5 (1 – 5)	3 (2 – 4)	2.5 (1 – 5)	3.3 (3 – 4)	4.3 (3 – 5)
Bimanual dexterity	2.8 (1 – 4)	3.3 (3 – 4)	2.5 (1 – 4)	3.3 (3 – 4)	4.3 (3 – 5)
Efficiency	2.8 (1 – 4)	3.5 (3 – 4)	3 (1 – 5)	3.3 (3 – 4)	4.3 (3 – 5)
Tissue handling	2.5 (1 – 4)	3.3 (2 – 4)	2.8 (1 – 5)	2.8 (2 – 4)	4.3 (2 – 5)
Autonomy	2.5 (2 – 4)	3 (2 – 4)	2.5 (1 – 4)	3 (2 – 4)	4.5 (3 – 5)
Global rating score	13 (6 – 21)	16 (12 – 19)	13.3 (6 – 23)	15.5 (13 – 20)	21.5 (14 – 25)

Fig. 2.

 Mean scores for each of five domains were statistically higher in the second attempts compared to the first attempts for both Novice-1 ( P  = 0.0007 < alpha/4) and Novice-2 ( P  = 0.0486 < alpha/1). Mean scores for each of five domains for Expert’s case were higher than those for the two novice operators in both the first and second attempts ( P  = 0.0012 < alpha/2, P  = 0.0008 < alpha/3, respectively).

Discussion

While operation time, completeness of resection and adverse event rates are thought to be good indices for objective assessment of ESD procedures ^{6 10 11 12} , in reality, using them to compare operator skill or the effects of training across various clinical conditions can be challenging. For tackling this, a number of assessment tools have been generated ^{13 14} . Because MASTER is a novel and unique endosurgical platform, an objective and reliable method of measuring training efficacy and outcome is needed.

In the current study, the means of the global rating scores and of each domain assessed increased alongside the increases in surgical experiences of MASTER. Especially when we put all domains together, statistically significant differences in parallel with experiences were observed. We deduced, therefore, that there is substantial possibility for GEARS-E to be an objective and consistent scale for assessment of proficiency in performing MASTER procedure.

On the other hand, we noted the wide-ranging scores of individual evaluators. For standardization, the anchoring method we applied probably didn’t work well. During evaluator training, the evaluators in the GOALS study were encouraged to use the full range of scores ⁸ . Another possible mean to help evaluators in standardizing their assessment of skills based on different skill levels is to let them watch video recordings of typical procedures performed by novice, intermediate and expert operators.

This study had some limitations. First, the evaluators were not blinded, and on the contrary, we included self-evaluations besides evaluations by other independent evaluators. Second, because this was a pilot study, sample size was small. Furthermore, due to the nature of ESD, it is challenging to equalize difficulty factors such as lesion location and endoscopic conditions. Finally, to focus on novice training, the sixth domain was excluded from the assessment.

Conclusion

In conclusion, results using GEARS-E showed correlations with surgical experience in MASTER ESD. As an assessment tool for evaluation of surgical skills, GEARS-E has a great potential for application in MASTER procedures. For the next step, we will design a training program that integrates objective GEARS-E skill assessment so that we can promote MASTER safely and assess its effectiveness across centers around the world.