Teleoperation and Contact Detection of a Waterjet-Actuated Soft Continuum Manipulator for Low-Cost Gastroscopy

Abstract

Gastric cancer is the third leading cause of cancer deaths worldwide, with most new cases occurring in low and middle income countries, where access to screening programs is hindered by the high cost of conventional endoscopy. The waterjet-actuated HydroJet endoscopic platform was developed as a low-cost, disposable alternative for inspection of the gastric cavity in low-resource settings. In this work, we present a teleoperation scheme and contact detection algorithm that work together to enable intuitive teleoperation of the HydroJet within the confined space of the stomach. Using a geometrically accurate stomach model and realistic anatomical inspection targets, we demonstrate that, using these methods, a novice user can complete a gastroscopy in approximately the same amount of time with the HydroJet as with a conventional endoscope.

I. Introduction

Gastric cancer (GC) is the the third leading cause of cancer-related deaths in both women and men, with around one million new GC cases arising annually worldwide [1]. Low and middle income countries (LMICs) in regions such as East Asia, Eastern Europe, and Central and South America, are disproportionately impacted by gastric cancer [2]. Early detection of cancer and related premalignant lesions has been shown to greatly reduce morbidity and mortality rates [3]. The standard of care for screening is inspection of the upper gastrointestinal tract (UGI) with flexible endoscopes (FEs), but despite the effectiveness of this procedure, screening rates in LMICs remain low [4].

The primary obstacle to mass gastroscopies in LMICs is the cost associated with the use of traditional FEs. These procedures require a dedicated endoscopy suite, due to the need for sedation, monitoring equipment and highly trained personnel, and the necessary equipment constitutes a large upfront cost. In addition, currently used FEs are reusable devices which require specialized reprocessing and sterilization facilities. Such economic and logistical challenges greatly limit the availability of FE-based inspections in many areas with high incidence of gastric cancer.

Soft continuum robots represent a promising potential alternative to traditional FEs that can meet the need for a cheap, disposable UGI inspection device. A soft continuum robot is a flexible elastomeric device, typically actuated by either tendons [5] or pressurized fluid-filled chambers [6] distributed along the body of the device. Due to their low cost, intrinsic safety and tunable flexibility, these robots have been proposed for a variety of applications, including several within the medical field [7]. Recently, however, a new class of soft continuum robots has emerged in which the device is actuated by a wrench applied only at the tip of the device, by magnetic manipulation [8] or water-jet propulsion [9], for example. Removing the actuators from the body of the device has the advantage of greatly simplifying manufacturing and assembly of the robot, but it also introduces some challenges in control of these new devices.

In prior work, we have presented a disposable, waterjet-actuated soft robot called the HydroJet for low-cost UGI endoscopy. In addition to reducing overall procedure cost, the device is designed to be highly compliant to prevent incidental injury to the stomach. An early prototype of the device is described in [9], and kinematic modeling, calibration, and disturbance estimation are presented in [10]. In this paper, we seek to bridge several of the gaps in the control methodology that remain between these prior experiments and the use of the HydroJet as a practical UGI inspection device for LMICs.

The first of these is the integration of a human operator into the control loop. While autonomous path following is useful for validating kinematic modeling, clinical viability requires an operator to be able to steer the device in real time. This enables medical professionals to make decisions on how to perform the procedure on a specific patient based on their findings once the endoscope has been deployed. In addition, autonomous navigation would require a method for registering to patient anatomy, and the extra equipment and clinical imaging this would require would be prohibitively expensive for low-resource settings. Teleoperation methods have been presented in literature for various continuum manipulator morphologies [11], [12] and applied to endoluminal procedures [13]. Many of the presented methods utilize the resolved motion rate approach to send Cartesian space commands to the manipulator’s end effector [14], which is the approach we take in this work.

The other key feature required to operate the device within the confined space of the stomach is a contact detection method to alert the operator when the device is experiencing significant contact with the anatomy. This feature addresses two important challenges. First, contact with the anatomy can occur behind the field of view of the camera, making it impossible for the user to visually observe the contact. Second, due to the high compliance of the manipulator, external loads due to contact can greatly affect the accuracy of the kinematic model and Jacobian matrix, compromising the ability of the teleoperation scheme to track desired input velocities. As contact loads increase and this effect becomes more significant, it can ultimately result in unexpected behavior that prevents intuitive teleoperation of the device. To combat these problems, the contact detection method is integrated within the teleoperation loop and prevents integral windup of the PID controller. In addition the operator is visually alerted to the presence and direction of contact forces, so that he/she can intuitively steer away from these obstacles. The force deviation method presented in [15] is used to detect contact and is combined with the closed-loop control scheme based on the Cosserat rod modeling framework. In this paper, due to device cost constraints, the method is adapted to utilize orientation sensors alone and not full pose sensors. Related methods for contact detection based on sensor feedback have been presented and successfully applied to other types of continuum manipulator designs, including multi-backbone robots [15] and pneumatic chamber robots [16].

The contributions of this paper are as follows: (1) We demonstrate for the first time closed-loop teleoperation of a waterjet-actuated soft continuum manipulator, (2) we validate the efficacy of the force deviation method in detecting contact within rigid environment for a tip-actuated soft continuum device and integrate it within the teleoperation scheme, and (3) we provide experimental results demonstrating intuitive operation of the HydroJet within an anatomical realistic stomach model.

II. The Hydrojet Endoscopic Platform

The HydroJet Endoscopic Device (HJ) is a soft robotic endoscope which was first introduced in [9]. The device uses three miniature waterjet actuators, distributed evenly around the tip and directed radially outward, to maneuver an endoscope camera. The overall device consists of four main parts: the distal tip, the soft sleeve, the base connector and the multilumen catheter (Fig. 2.(a)). With the exception of the catheter and sleeve, all parts are manufactured using photosensitive resin (Dental RG biocampatible resin, FormLabs, Sommerville, MA, USA) through SLA rapid prototyping.

Fig. 2.

a) Exploded view diagram of the HydroJet device. b) Photo of the prototype used in experiments. c) Head-on view diagram of the capsule tip, showing the direction of the jet locations and coordinate frame definition. d) Kinematic variable definitions used for the Cosserat rod model.

The distal tip contains a camera (AD-3915, Aidevision, China) with illumination and an inertial measurement unit (IMU) (BNO055, Bosch Sensortech, USA). It is cylindrical, with a diameter of 11.7 mm and length of 28 mm. A soft elastomer sleeve (Ecoflex 00–30, Smooth-On, USA) connects the distal end of the multilumen catheter to the tip, and encases three flexible tubes which carry pressurized water from the multilumen catheter to the jets. This structure is designed to be significantly softer and more flexible than the multilumen catheter. As a result, the forces generated by the actuators produce bending almost entirely within the soft sleeve portion of the device rather than in the multilumen catheter, enabling high bending angles within a relatively small workspace. The base connector serves as an interface between the multilumen catheter, which is connected directly to three solenoid valves, and to the individual tubes within the bending section. It also contains a second inertial sensor, which provides a reference frame in which to describe the orientation of the tip and aids in kinematic modeling by providing knowledge of the direction of external constant forces acting on the device (i.e. gravity, buoyant forces, etc.).

With each jet’s actuation force controlled individually, the net force acting on the robot’s tip produces bending of the soft sleeve, resulting in two degrees of freedom of motion. The net applied wrench in the body frame acting on the tip of the manipulator due to the water jet forces is:

$$\begin{gathered} {\left.{\mathbf{F}}_{\mathbf{a}}\right|}_{xy}=A\left[\begin{array}{l}{q}_1\\ q_2\\ q_3\end{array}\right] \\ A=\left[\begin{array}{ccc}\mathrm{s}\mathrm{i}\mathrm{n}\left(\frac{\pi }{3}\right)& 0& -\mathrm{s}\mathrm{i}\mathrm{n}\left(\frac{\pi }{3}\right)\\ \mathrm{c}\mathrm{o}\mathrm{s}\left(\frac{\pi }{3}\right)& -1& \mathrm{c}\mathrm{o}\mathrm{s}\left(\frac{\pi }{3}\right)\end{array}\right] \end{gathered}$$ (1)

where $q_1,q_2$, and $q_3$ are the three applied forces due to the water jets and ${\left.{\mathbf{F}}_a\right|}_{xy}$ represents the $x$ and $y$ components of the tip force in tip frame, as defined in Fig. 2.(c). A represents a geometric mapping according to the locations of the jets. Since the three actuators are all coplanar with the tip of the endoscope, the $z$ component of the force is zero.

III. Review of Kinematic Model

In this section, we briefly review the kinematic model used to control the HJ device in our experiments, which is discussed in greater detail in [10]. The model is derived from Cosserat rod theory, which has previously been applied to other types of soft continuum robots [17], concentric tube robots [18], tendon-driven robots [19], and magnetically steered rods [20].

The kinematic states are parameterized by the arc length variable $s$, from $s=0$ at the base to $s=L$ at the tip. Using the notation of [18], the Cosserat rod equations that govern the kinematics, expressed in local frame coordinates, are:

$$\begin{gathered} {\mathbf{p}}^{\prime }(s)=\mathit{\xi }(s)\circ {\mathbf{e}}_z\circ {\mathit{\xi }}^{-1}(s) \\ {\mathbf{q}}^{\prime }(s)=\frac{1}{2}{\mathit{\xi }}^{-1}(s)\circ \mathbf{u}(s) \\ {\mathbf{n}}^{\prime }(s)=-\stackrel{\mathbf{ˆ}}{\mathbf{u}}(s)\mathbf{n}(s)-\stackrel{-}{\mathbf{w}}(s) \\ {\mathbf{u}}^{\prime }(s)={\tilde{\mathbf{u}}}^{\prime }(s)-K^{-1}\left(\left(\stackrel{\mathbf{ˆ}}{\mathbf{u}}(s)K+K^{\prime }\right)(\mathbf{u}(s)-\tilde{\mathbf{u}}(s))+{\stackrel{\mathbf{ˆ}}{\mathbf{e}}}_z\mathbf{n}(s)\right) \end{gathered}$$ (2)

where $\mathbf{p}$ represents the position and $\mathit{\xi }$ represents the unit quaternion describing the local frame. The operator $\prime$ denotes derivation with respect to $s$. The internal force vector is $\mathbf{n}$, and the distributed external force is $\stackrel{-}{\mathbf{w}}.\mathbf{u}$ represents the local curvature vector, ${\mathbf{e}}_z$ represents the unit vector in the z-direction (tangent to the curve), and $\stackrel{\mathbf{ˆ}}{\mathbf{u}}$ and ${\stackrel{\mathbf{ˆ}}{\mathbf{e}}}_z$ represent the skew-symmetric matrix versions of each. $\tilde{\mathbf{u}}$ is the precurvature vector corresponding to the unactuated shape of the device, and $K$ represents a diagonal stiffness matrix. For the HJ, the stiffness in each of the primary directions is allowed to vary linearly from base to tip and calibrated accordingly. The boundary conditions for (2) are:

$$\begin{gathered} {\mathbf{n}}_b(L)={\mathbf{F}}_a+{\left.{\mathbf{w}}_d\right|}_f+{\mathbf{F}}_g+{\mathbf{F}}_b \\ \mathbf{u}(L)={\left.K^{-1}{\mathbf{w}}_d\right|}_m+\tilde{\mathbf{u}}(L) \\ \mathbf{p}(0)={\left[\begin{array}{lll}0& 0& 0\end{array}\right]}^T \\ R(0)=I \end{gathered}$$ (3)

where ${\mathbf{F}}_a$ is the force vector provided by the actuators, ${\mathbf{w}}_d={\left[\begin{array}{ll}{\left.{\mathbf{w}}_d\right|}_f& {\left.{\mathbf{w}}_d\right|}_m\end{array}\right]}^T$ represents a disturbance wrench that can be used during calibration to compensate for unmodeled effects, and ${\mathbf{F}}_g$ and ${\mathbf{F}}_b$ represent tip weight and tip buoyancy force, respectively. Due to the arrangement of the jets, the actuator wrench consists only of lateral forces; consequently, no axial moments are modeled resulting in zero torsional stress. The solution of the resulting boundary value problem provides the full pose of the manipulators as a function of arc length.

IV. Telerobotic Operation and Contact Detection

The model described in Section III allows for the shape of the robot and the manipulator Jacobian to be computed for any set of actuation inputs, thereby enabling trajectory control of the manipulator tip via a resolved rates approach. However, two key additions in the control method are required to enable stomach inspection with the HJ device. First, an intuitive method for telerobotic operation that maps user inputs in camera frame to changes in actuator forces is required. Second, a method for coping with environmental interactions is required in order to maintain intuitive control of the device within a confined space. During a complete stomach inspection, contact with the stomach walls occurs frequently. The contact detection method proposed in this section serves the purpose of alerting the operator of contact while preventing commanded actuator forces from growing exponentially due to the high kinematic error. It should be noted, however, that the contact detection method is not used to estimate contact forces; in the case of the HydroJet, the high compliance of the device guarantees by design that contact forces remain below the safety threshold for interaction with stomach tissue [21].

A. Teleoperation Scheme

As stated in Section II, the HJ can achieve motions in 2-DOF via the water jet actuators located at its tip. During stomach inspection, the operator can also control a third DOF of motion by manually translating the catheter within the esophagus. To provide the user with intuitive control of the two robotic degrees of freedom, we use a thumb-controlled joystick interface, as shown in Fig. 3. Deflection of the joystick in 2-DOF space provides desired directions of motion with respect to the camera frame whenever the user depresses a trigger-style “clutch” button on the controller. A second trigger-style button can be used to turn the water jets on and off, enabling the user to “reset” the device to its unpowered position if desired.

Fig. 3.

The rotation of the tip frame is commanded by summing the desired tip velocity obtained from the joystick to the orientation error obtained through inertial measurement units.

The teleoperation scheme is presented in Fig. 4. The desired twist in the tip frame consists of velocities in the roll and pitch angle of the camera frame: ${\dot{\psi }}_d={\left[\begin{array}{ll}{\dot{\theta }}_d& {\dot{\varphi }}_d\end{array}\right]}^T$. The control signal ${\psi }_{cntr}$ is obtained by summing the desired tip velocity to a proportional-derivative feedback term:

$${\mathit{\psi }}_{cntr}=\dot{\mathit{\psi }}+{\mathbf{K}}_p{\mathit{\psi }}_e+{\mathbf{K}}_d\frac{d{\mathit{\psi }}_e}{dt}$$ (4)

where ${\mathbf{K}}_p$ and ${\mathbf{K}}_d$ are the proportional and derivative feedback gains and ${\mathit{\psi }}_e$ is the pose error defined as ${\mathit{\psi }}_e={\mathit{\psi }}_d-{\mathit{\psi }}_{obs}$. The vector ${\mathit{\psi }}_{obs}$ is the observed orientation of the tip frame with respect to the base frame and ${\mathit{\psi }}_d$ is computed by integrating the desired tip velocity ${\dot{\mathit{\psi }}}_d$ over time. The control signal is converted into change in tip wrench by:

$${\dot{\mathbf{w}}}_b=\left({\mathbf{C}}^T{\left({\mathbf{C}}_b{\mathbf{C}}_b^T-\mu \mathbf{I}\right)}^{-1}\right){\mathit{\psi }}_{cntr}$$ (5)

where ${\mathbf{C}}_b$ is the body Jacobian matrix, $\mu$ is a damping factor and $\mathbf{I}$ is the identity matrix. Quadratic programming is used to find the configuration parameters that minimize the norm squared error between the desired actuator wrench, ${\mathbf{w}}_{a,des}(\mathbf{q})$, and the current applied wrench as detected by the sensors, ${\mathbf{w}}_a$:

$$\begin{array}{ll}\underset{\mathbf{q}}{\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}}& {∥A\mathbf{q}-\left({\mathbf{w}}_a+{\dot{\mathbf{w}}}^b\right)∥}^2+{\parallel \mathbf{q}\parallel }^2\\ \text{subject}\text{to}& {\mathbf{q}}_{min}\le \mathbf{q}<{\mathbf{q}}_{max}\end{array}$$ (6)

Here, ${\mathbf{q}}_{min}$ and ${\mathbf{q}}_{max}$ represent respectively the lower and upper limits of the actuator value.

Fig. 4.

Block diagram representation of the proposed teleoperation scheme.

B. Contact Detection

The teleoperation method described in Section IV.A relies on the ability to accurately compute the tip pose and Jacobian matrix associated with the device for the current set of actuator values. While the accuracy of the kinematic model is sufficient to achieve good control during free space operation, the presence of unknown external loads can lead to integral windup of the PID controller. This is especially true for very low stiffness devices, such as the HJ, in which even small external loads can produce large changes in the manipulator pose. To address this challenge, we utilize the contact detection method described in Algorithm 1, which enables the system to detect when the tip comes into contact with its environment based on commanded tip forces and orientation feedback provided by the two IMUs. It is intended to work in combination with the telerobotic scheme presented in Section IV.A. At each time step of the teleoperation loop, the inputs to the contact detection algorithm are the commanded angular vector ${\dot{\mathit{\psi }}}_d$, the kinematic error ${\mathit{\psi }}_e$ and the Jacobian matrix ${\mathbf{C}}_b$. The algorithm can be thought of as a state machine which moves between three different states: (1) no contact, (2) possible contact, and (3) confirmed contact.

The algorithm is assumed to start from a contact-free state when the user enables the jets to begin teleoperation. As the device is driven by the user, the system remains in a contact-free state as long as the kinematic error remains below a preselected threshold. Possible contact arises if

$$∥{\mathit{\psi }}_{\mathbf{e}}∥-{\mathit{\zeta }}_{\psi }>0$$ (7)

where $∥{\mathit{\psi }}_e∥$ is the norm of the kinematic error vector, and ${\mathit{\zeta }}_{\psi }$ is the threshold value on kinematic error.

Algorithm 1.

Contact Detection

1: procedure ContactDetection $\left({\mathit{\psi }}_{cntrl},{\mathit{\psi }}_e,{\mathbf{C}}_b\right)$

2:  ${\mathbf{w}}^b\leftarrow {\stackrel{-}{\mathbf{w}}}_b$

3:  repeat

4:   ${\mathbf{w}}^b\leftarrow {\mathbf{w}}^b+{\mathbf{C}}_b^{†}{\mathit{\psi }}_{cntr}$

5:   if $∥{\mathit{\psi }}_e∥>{\mathit{\zeta }}_{\psi }$ then

6:    Possible Contact ← true

7:    ${\mathbf{w}}_c={\mathbf{w}}^b$

8:   else

9:    Possible Contact ← false

10:    ${\mathbf{w}}_c=\mathbf{0}$

11:   end if

12:   if Possible Contact == true then

13:    if $∥{\left.{\mathbf{w}}^b\right|}_x-{\left.{\mathbf{w}}_c\right|}_x∥>{\mathit{\zeta }}_{\sigma }$ then

14:     if ${\left.{\mathit{\psi }}_{cntrl}\right|}_x>0$ then

15:      Contact Triggered X+ ← true

16:     else

17:      Contact Triggered X- ← true

18:     end if

19:    end if

20:    if $∥{\left.{\mathbf{w}}^b\right|}_y-{\left.{\mathbf{w}}_c\right|}_y∥>{\mathit{\zeta }}_{\sigma }$ then

21:     if ${\left.{\mathit{\psi }}_{cntr}\right|}_y>0$ then

22:      Contact Triggered Y+ ← true

23:     else

24:      Contact Triggered Y- ← true

25:     end if

26:    end if

27:   end if

28:  until Reset == true

29:  return true

30: end procedure

During possible contact, the PD controller continues to act to close the error in commanded pose by adjusting the applied wrench. If the error falls back below the threshold (consistent with transient unmodeled effects), the system returns to the contact-free state; if the actuation wrench continues to change in a certain direction without reducing the error in that direction to below the threshold, this confirms contact detection in that direction. This condition is tested by monitoring the difference between the current commanded wrench, ${\mathbf{w}}^b$, and the commanded wrench at the onset of possible contact, ${\mathbf{w}}_c$. Because we wish to display contact to the user as a combination of red bars located around the four sides of the camera view, the $x$ - and $y$-components of this difference are computed separately. The conditions for contact in the $x$ and $y$ directions can then be written as:

$$\begin{array}{rr}∥{\left.{\mathbf{w}}^b\right|}_x-{\left.{\mathbf{w}}_c\right|}_x∥& >{\mathit{\zeta }}_{\sigma }\\ ∥{\left.{\mathbf{w}}^b\right|}_y-{\left.{\mathbf{w}}_c\right|}_y∥& >{\mathit{\zeta }}_{\sigma }\end{array}$$ (8)

where the two equations describe contact conditions along the $x$-axis and $y$-axis, respectively, and ${\mathit{\zeta }}_{\sigma }$ is another pre-selected threshold on the change in commanded wrench during possible contact. The direction of contact on either axis (i.e. if contact rod modeling framework. In is occuring on the positive side of the axis or the negative) can then be deduced according to the desired direction of motion ${\mathit{\psi }}_{cntr}$.

V. Experimental Validation

To validate our method for UGI inspection with the HydroJet, we first experimentally assessed the effectiveness of our contact detection method, then performed several simulated stomach inspection trials. All experiments were performed in water to increase motion damping. Water can be utilized during UGI inspection as it aids with stomach expansion and device insertion [22].

A. Contact Detection Algorithm Validation

The first set of experiments was designed to investigate the performance of the contact detection algorithm in the idealized case in which the base does not move during teleoperation, to facilitate repeated testing under consistent conditions. Using the testbed shown in Fig. 5, the base of the HJ was clamped to a rigid, fixed frame, with the device pointing downward in an initially straight configuration. A glass container was placed around the robot, creating an obstacle in every direction for the HJ to contact with. The tip of the device was commanded in four different directions with respect to the camera frame (+X, −X, +Y, −Y) by deflecting the joystick left, right, up and down. The glass container was approached five times from each of the four directions. The time of contact according to visual inspection was recorded, along with the times at which the algorithm identified as possible contact and confirmed contact. The kinematic error threshold, ${\mathit{\zeta }}_{\sigma }$, was chosen to be the maximum kinematic error observed during calibration. The force threshold, ${\mathit{\zeta }}_{\sigma }$, was selected to be 10% of the maximum force that can be generated by a single waterjet.

Fig. 5.

a) User interface communicating left side contact detection via the red bar on the left side of the screen and the red ”Contact Triggered” status. b) Experimental setup for the repeated contact detection trials, including the contact which generated the messages on the GUI in (a).

In all 20 of these trials, contact and direction of contact were successfully detected. Representative results for one example trial in each direction are shown in Fig. 6, illustrating the algorithm’s ability to identify possible contact conditions based on a threshold on rotational error (first row on Fig. 6), and to confirm contact based on subsequent change in actuation force (second row of Fig. 6). The direction of contact can be inferred from the desired direction of motion after contact was detected (third row of Fig. 6). Five additional experiments were performed with the constraint on the base frame removed. The tether was held by the operator as shown in Fig. 7.(a), more closely simulating the conditions of teleoperation during UGI inspection. Contact was successfully detected for each of these five trials, and the results of one example contact detection are shown in Fig. 7.(b). On average across all 25 trials, visual contact was achieved 6.34 seconds before contact was confirmed by the algorithm; however, this time depends strongly on the selected force threshold, which can be tuned to achieve more or less sensitive behavior of the algorithm.

Fig. 6.

Results of contact detection trials in four directions (left, down, right, up) with respect to the camera frame. The first row shows the kinematic error measurement over time, with the horizontal line representing the threshold for possible contact. The vertical lines represent the time of visual contact, possible contact detected, and contact confirmed by the algorithm (in order from left to right). Circled locations on these plots correspond to times when transient error resulted in the algorithm identifying possible contact temporarily. The second row shows the change in force after the detection of possible contact. In each case, this measure increases in the direction of contact without bringing the error back below the possible contact threshold. The third row shows the commanded motions, which determine the direction of contact along each axis.

Fig. 7.

a) Starting configuration with user holding the HydroJet. b) Configuration after visual contact is triggered. c) Experimental plots showing the norm of the orientation error and the difference between the current commanded wrench and the commanded wrench at the onset of possible contact.

B. Stomach Inspection

To assess the ability for an operator to use the HJ to complete UGI inspection, we performed a simulated stomach inspection in a phantom model. The clinical standard for a successful completion of an UGI inspection is the visualization of six key landmarks (gastro-esophageal junction (GEJ)/cardia, antrum greater and lesser curvature (AGC and ALC), body greater and lesser curvature (BGC and BLC), and fundus). The experimental setup is presented in Fig. 8. An UGI tract phantom was constructed, consisting of a flexible esophagus and a geometrically accurate stomach with internal capacity of 1.5 L. The stomach was pressure molded using transparent plastic sheets. Six different colored pieces of tape were placed at the key landmark positions in the stomach by an experienced gastroenterologist. A novice user was asked to manually insert the device into the esophagus, then use the joystick interface to perform an inspection. The phantom model was hidden to the user, such that the only visual feedback was from the HJ’s camera view. All six landmarks were successfully inspected during the procedure, as shown in Fig. 9, with an overall procedure time of 3 minutes and 5 seconds. Contact with the stomach walls was detected in average four times during the inspection. Each time, the user was able to reposition the flexible catheter, thus adjusting the base frame of the device, and approach the target in a different way.

Fig. 8.

Teleoperation experiment with an expert gastroenterologist using the HJ device. The stomach phantom is shown in the image for illustrative purposes, but was covered during the experiments.

Fig. 9.

Endoscope configurations when visualizing various markers representing key GI landmarks.

C. Repeated Trials

To further explore the efficacy of the HJ, a total of 30 trials (15 using the HJ and 15 using a conventional endoscope as a benchmark device) were performed in the phantom model. One expert gastroenterologist (having performed >3,000 lifetime UGI inspections) and two non-expert users with minimal or no previous experience with FEs were asked to perform visualization of the six key landmarks using either the HJ or a conventional FE (Karl Storz-Tuttlingen, Germany). In all experiments, direct view of the phantom was blocked and the users were asked to rely only on camera feedback to complete the inspection. The field of view of the FE was cropped to match that of the HJ, in order to compare the effectiveness of each device without considering differences in the camera views. Each user performed the trial 5 times with each device, and for each trial, the time required to visualize all the landmarks and thus complete the procedure was recorded. The time required for each user group to complete the inspection with each device (including mean, standard deviation (STD), first quartile (Q1) and third quartile (Q3)) are shown in Table I.

TABLE I.

Repeated trials results

User Group	Inspection Time	Device
		HydroJet	Conventional Scope
Expert	Mean [min : sec]	2:32	1:05
	STD [min : sec]	1:05	0:43
	Q1 [min : sec]	1:14	0:40
	Q3 [min : sec]	2:48	1:11
Novice	Mean [min : sec]	3:30	3:21
	STD [min : sec]	1:34	1:60
	Q1 [min : sec]	2:27	2:10
	Q3 [min : sec]	4:02	4:30

The expert user took less time to complete landmark visualization with the conventional endoscope compared to the HJ (mean 1 minute and 5 seconds vs. 2 minutes and 32 seconds, respectively), which is largely due the user’s prior expertise in using traditional endoscopes. With novice users, the HJ took on average comparable time to the flexible endoscope to complete a procedure. These results, when compared to [9], show how the teleoperation and contact detection methods improve the overall controllability of the device. The fact that non-expert users completed the procedure with the two devices in comparable time provides preliminary evidence that the device will be usable by operators with a range of experience levels, which is promising for its deployment in LMICs. The presented methods allowed both user groups to operate the device in confined space. This, due to the device’s nature, would otherwise be impossible with teleoperation alone.

VI. Discussion

The work in this paper represents several key steps toward clinical use for the HJ system. The integration of the contact detection algorithm within the closed-loop controller addresses one of the major challenges of eye-in-hand teleoperation of highly compliant manipulators, enabling the use of the HJ in a confined environment. One of the remaining challenges associated with this approach, however, is a protocol for selecting the appropriate thresholds. In future work, it would be valuable to implement an automated method for tuning these thresholds in the same way that the kinematic parameters for the device are tuned automatically. In addition, future work will be necessary to explore the capabilities and limitations of the contact detection method when the device interacts with other types of surfaces, such as elastic materials. While the contact detection method can in principle be applied to other soft continuum robots, it is especially well suited for highly compliant manipulators where minimizing cost and design complexity is desirable.

In addition to validating the efficacy of the force deviation algorithm, we have demonstrated for the first time the integration of a human operator into the control loop for a waterjet-actuated soft continuum robot. The intuitive user interface we utilize enabled even novice users to complete full gastric cancer inspections in around 4 minutes, which is a small fraction of the total hospitalization time of 60–120 minutes typically required for conventional endoscopy. While these experiments provide a highly promising first investigation of the HJ’s capabilities as a telerobotic upper GI inspection device, more studies are needed to further explore its effectiveness for the proposed clinical scenario. In the future, trials in phantom models whose mechanical properties better simulate real tissue and with larger groups of participants, as well as trials in animal models will be valuable.

VII. Conclusion

This paper details teleoperation and contact detection methods for performing UGI inspection with a soft, waterjet-actuated robotic device called the HydroJet. Experimental validation of the system in phantom models demonstrates the feasibility of gastric cavity inspection with the HJ for a small fraction of the cost of conventional gastroscopy. These results are highly promising for the HJ’s potential as a first line of screening against one of the deadliest cancers worldwide.

Fig. 1.

Schematic representation of UGI screening procedure using the HydroJet Endoscopic Platform. The user controls movement of the camera via the joystick, along with manual insertion of the catheter.