A Model-Based Recurrent Neural Network With Randomness for Efficient Control With Applications

Abstract

Recently, Recurrent Neural Network (RNN) control schemes for redundant manipulators have been extensively studied. These control schemes demonstrate superior computational efficiency, control precision, and control robustness. However, they lack planning completeness. This paper explains why RNN control schemes suffer from the problem. Based on the analysis, this work presents a new random RNN control scheme, which 1) introduces randomness into RNN to address the planning completeness problem, 2) improves control precision with a new optimization target, 3) improves planning efficiency through learning from exploration. Theoretical analyses are used to prove the global stability, the planning completeness, and the computational complexity of the proposed method. Software simulation is provided to demonstrate the improved robustness against noise, the planning completeness and the improved planning efficiency of the proposed method over benchmark RNN control schemes. Real-world experiments are presented to demonstrate the application of the proposed method.

I. INTRODUCTION

Redundant manipulators demonstrated superior dexterity and are widely applied to intelligent robots. However, the redundant manipulator motion planning problem remains challenging. Actually, it has been proven that this problem is PSPACE-hard, when obstacles exist [1]. The redundant manipulator motion planning problem is to find the optimal path in the manipulator configuration space that delivers the end-effector to the desired target without breaking constraints [2]. The configuration space consists of all feasible arm joint configurations, $\mathit{q}(t)\in {ℝ}^m,$ where m denotes the Degrees of Freedom (DoF) of the arm. Given the manipulator model, each joint configuration corresponds to a uniquely defined end-effector pose as $\mathit{r}(t)=f(\mathit{q}(t)),$ where $f(\cdot )$ is the kinematic model. Therefore, the configuration space is mapped to the task space [3]. Correspondingly, the n-dimensional task space contains all feasible end-effector poses, $\mathit{r}(t)\in {ℝ}^n.$

Searching for the solution in the configuration space given a task and the kinematic model is known as the kinematic control problem [1]. The problem is often solved in the velocity space because the partial differentiation of the kinematic model linearizes and simplifies the problem to:

$${\dot{r}}_t=J{\dot{q}}_t,$$ (1)

where $J=\partial f/\partial \mathit{q}$ is the n × m Jacobian matrix [2].

For redundant manipulators, an infinite number of solutions satisfy Eqn. 1, because the redundancy m – n > 0. The redundancy corresponds to the self motion, which is useful in obstacle avoidance [3]. Existing algorithms utilizing the redundancy for obstacle avoidance can be generally divided into two categories. One utilizes “Gradient Projection” Methods to determine the joint velocity vector ${\dot{\mathit{q}}}_0$ that corresponds the self motion of avoiding obstacles [3]. The components of ${\dot{\mathit{q}}}_0$ that are in the null space of J can be selected by (I − J^† J). By adding the selected components to the motion that moves the end-effector $\left(J^{†}\stackrel{.}{\mathit{r}}\right),$ the optimal joint velocity is uniquely defined as: $\stackrel{.}{\mathit{q}}=J^{†}\stackrel{.}{\mathit{r}}+\left(I-J^{†}J\right){\stackrel{.}{\mathit{q}}}_0,$ where J^† is the Moore-Penrose pseudo-inverse defined as $J^{†}=J^{\text{T}}{\left(J{J}^{\text{T}}\right)}^{-1}$ [2], [4]. The second family of algorithms treats the obstacle avoidance as constraints and converts them into tasks, through which the task space is augmented to uniquely define the solution [2], [4]. Mathematically, $f_y(\mathit{q}(t))\in {ℝ}^{\text{s}}$ denotes the constraint of obstacle avoidance, where s is the dimension of the constraints, then the task is augmented as: ${\mathit{r}}_A(t)={\left[f{(\mathit{q}(t))}^{\text{T}},f_y{(\mathit{q}(t))}^{\text{T}}\right]}^{\text{T}},$ and the solution can be uniquely determined when s = m − n, as:

$${\stackrel{.}{\mathit{r}}}_{At}=J_A{\stackrel{.}{\mathit{q}}}_t,$$ (2a)

$$J_A={\left[J^{\text{T}},J_y^{\text{T}}\right]}^{\text{T}},$$ (2b)

where $J_y=\partial f_y/\partial \mathit{q}.$

From the previous paragraph we know the problem is mathematically well-defined. However, it is still challenging to solve the problem because the configuration spaces are often concave while obstacles exist. Neural network based methods attract attention recently, and many reported significant improvement of robot performance [5]–[18].

Most of these recent studies approach the problem from the control theory perspective, and focus on improving control stability [19], [20] or system adaptiveness [21], [22]. In real-world robotic applications, the solutions from the motion planning perspective are preferable because of the requirements on applicability, effectiveness, and efficiency. Among the works that approach the problem from the motion planning perspective, some of them assume obstacle-free environments. For example, Jin et al. proposed a Recurrent Neural Network (RNN) based solution to optimize motions for maximizing the manipulability of redundant manipulators [23]. Li et al. proposed a RNN control scheme to address the cooperative control problem for distributed redundant manipulators [24]. For environments with obstacles, Zhang et al. treats the condition of collision avoidance as an additional constraint and solved the motion planning in the velocity space [25]. Guo et al. extended the work into the acceleration space, and the proposed scheme guarantees minimum-acceleration-norm [26]. More discussion and comparison on neural network based motion planning can be found in [27]. Although these RNN control schemes have demonstrated improved control precision and efficiency, they model the problem as the constrained optimization, therefore, they suffer from the local minimum problem and lack the planning completeness.

This work aims to address the planning incompleteness problem of these neural network-based control schemes in environments with obstacles. Being different from these existing works, this work proposed a novel random RNN control scheme, inspired by the recent finding that neural network randomness correlates with superior learning abilities [28]. Therefore, the proposed method inherits the robustness, the computational efficiency, and the effectiveness of neural network-based control schemes, while it also achieves the probabilistic planning completeness. Furthermore, through learning in the process of exploration, the proposed method balances the random exploration of environments with the heuristic search and improves the planning effectiveness. In summary, the main contributions of this work are:

• We propose a novel Recurrent Neural Network (RNN) control scheme to address the planning incomplete problem. The proposed method inherits the advantages of classical RNN control schemes, including high precision, the high efficiency and the high robustness from RNN. Meanwhile, it addressed the local minimum problem and achieved guaranteed probabilistic planning completeness.

• We introduce Short Term Memory (STM) model into the proposed method to learn environmental complexity from exploration. The proposed scheme balances the random exploration and the heuristic search to improve planning efficiency.

• We prove the global stability, the planning completeness and show the computational complexity of the proposed method.

• We study the control precision, the robustness against noise and the planning completeness and the planning efficiency of the proposed method through comparing it with other three algorithms.

• We demonstrate the application of the proposed method in both software simulation and real-world experiments.

The rest of this paper is organized as follows: Section II presents the proposed RNN control schemes in detail. Section III presents the theoretical analyses of the proposed method. Section IV compares the proposed method with other three schemes in simulation experiment, and further verifies it in real-world experiments. The paper concluded with the discussion in Section V.

II. RANDOM RECURRENT NEURAL NETWORK FOR REDUNDANT MANIPULATOR MOTION PLANNING

A. Solving Kinematic Control in Dual Space with Recurrent Neural Networks

Recurrent Neural Networks (RNNs) refer to networks that have inter-layer connections. RNNs are intrinsically parallel and capable of processing sequential or time-varying data [29]. RNNs showed superior robustness and efficiency on solving the Quadratic Programming problem (QP) [30], [31], thus have been used for redundant manipulator control. The kinematic control of redundant manipulators can be express in the form of QP as:

$$\underset{\stackrel{.}{\mathit{q}}}{\min }{\stackrel{.}{\mathit{q}}}^{\text{T}}W\stackrel{.}{\mathit{q}}+{\mathit{c}}^{\text{T}}\stackrel{.}{\mathit{q}},$$ (3a)

$$\text{s.t.}{\dot{\mathit{r}}}_d=J\stackrel{.}{\mathit{q}},$$ (3b)

$$\mathit{q}\in \mathrm{\Omega },$$ (3c)

where W and c are two weighting factors that are a m × m real symmetric matrix and a real valued m-dimensional vector, respectively, $J=\partial f/\partial \mathit{q}\in {ℝ}^{n\times m},{\dot{\mathit{r}}}_d=\partial \mathit{r}(t)/\partial t\in {ℝ}^n,$ and $\stackrel{.}{\mathit{q}}=\partial \mathit{q}/\partial t,\mathit{q}\in \mathrm{\Omega },\mathrm{\Omega }\subset {ℝ}^m$ denotes the configuration space of the redundant manipulator.

To simplify the QP problem, we project it into its dual space, through designing the Lagrange multiplier, $\lambda \in {ℝ}^n,$ to correspond to the constraint (Eqn. 3b). The problem described in Eqn. 3 is simplified to:

$$L(\stackrel{.}{\mathit{q}},\lambda )={\stackrel{.}{\mathit{q}}}^{\text{T}}W\stackrel{.}{\mathit{q}}+{\mathit{c}}^{\text{T}}\stackrel{.}{\mathit{q}}+{\lambda }^{\text{T}}\left({\stackrel{.}{\mathit{r}}}_d-J\stackrel{.}{\mathit{q}}\right).$$ (4)

The Karush-Kuhn-Tucker condition (Chapter 5.5.3 in [32]) ensures that the solution to Eqn. 4 equals to the solution to the following equation:

$$\begin{gathered} \stackrel{.}{\mathit{q}}=P_{\mathrm{\Omega }}\left(\stackrel{.}{\mathit{q}}-\frac{\partial L}{\partial \stackrel{.}{\mathit{q}}}\right), \\ {\dot{\mathit{r}}}_d=J\dot{\mathit{q}}, \end{gathered}$$ (5)

where $P_{\mathrm{\Omega }}(x)=\arg {\min }_{y\in \mathrm{\Omega }}\Vert y-x\Vert$ is a projection function from domain ${\mathrm{\Omega }}^{\prime }$ to Ω, where $x\in {\mathrm{\Omega }}^{\prime }$ and $y\in \mathrm{\Omega }.$

Eqn. 5 naturally matches the neural dynamics of a projected Recurrent Neural network (Eqn. 6), and it can be proved that the equilibrium of the network equals to the optimal solution of the system described in Eqn. 3 [33], [34].

$$ϵ\stackrel{\mathrm{..}}{\mathit{q}}=-\stackrel{.}{\mathit{q}}+P_{\mathrm{\Omega }}\left(\stackrel{.}{\mathit{q}}-\frac{\partial L}{\partial \stackrel{.}{\mathit{q}}}\right),$$ (6a)

$$ϵ\dot{\lambda }={\dot{\mathit{r}}}_d-J\dot{\mathit{q}},$$ (6b)

where ϵ > 0 is a scaling factor. These types of projected RNN has simple architeture as they are a single layer neural network, in which neurons are fully connected. The output of the neural network goes to a projection function as explained in Eqn. 9.

B. Improve Control Precision and Robustness by Closing the Loop of RNN Control Scheme

The control precision and the robustness against noise are critical in motion planning, because: 1) the control precision defines the minimum safety distance to obstacles, which shrinks the configuration space; 2) high control precision and robustness against noise ensure the successful execution of planning results. Classical RNN control schemes have demonstrated improvement on the control precision, the robustness and the efficiency of redundant manipulator control, compared with their Jacobian inversion based equivalents [35], we identified the problem of error accumulations in these control schemes and design a new RNN control scheme to overcome the error accumulation problem through introducing a new optimization target. We first introduced the new design and the proof of the effectiveness will be presented in Section III-A.

Intuitively, feeding the tracking errors, $\mathit{e}={\mathit{r}}_d-\mathit{r},$ back into the optimization target forces RNN to minimize the errors. Correspondingly, the new optimization target can be designed as:

$$\underset{{}_{\stackrel{.}{\mathit{q}}}}{\min }\left({\stackrel{.}{\mathit{q}}}^{\text{T}}\stackrel{.}{\mathit{q}}+k{\mathit{e}}^{\text{T}}\mathit{e}\right),$$ (7)

where k > 0 is a weighting factor and r is the end effector position [36]. Eqn. 7 will direct RNNs to minimize the magnitude of e, because ${\stackrel{.}{\mathit{q}}}^{\text{T}}\stackrel{.}{\mathit{q}}\ge 0$ and ${\mathit{e}}^{\text{T}}\mathit{e}\ge 0.$ Control precision and robustness of RNNs based on Eqn. 7 are analyzed in theory in Subsection III-A, and empirically compared in Subsection IV-A.

With the new defined optimization function (Eqn. 7), Eqn. 4 becomes:

$$L(\stackrel{.}{\mathit{q}},\lambda )={\stackrel{.}{\mathit{q}}}^{\text{T}}\stackrel{.}{\mathit{q}}+k{\mathit{e}}^{\text{T}}\mathit{e}+{\lambda }^{\text{T}}\left({\stackrel{.}{r}}_d-J\stackrel{.}{\mathit{q}}\right).$$ (8)

The constraints of joint physical limits can be fulfilled by designing a proper projection function. A projection function has the form of:

$$P_{\mathrm{\Omega }}(\mathit{x})=\{\begin{array}{ccc}{\mathit{d}}^{-}& \text{for}& x\le {\mathit{d}}^{-},\\ \mathit{x}& \text{for}& {\mathit{d}}^{-}<\mathit{x}<{\mathit{d}}^{+},\\ {\mathit{d}}^{+}& \text{for}& {\mathit{d}}^{+}\le \mathit{x},\end{array}$$ (9)

where we design the boundary conditions as:

$$\{\begin{array}{l}{\mathit{d}}^{-}=\max \left(-c_1\left(\mathit{q}-{\mathit{q}}^{-}\right),{\mathit{w}}^{-}\right),\hfill \\ {\mathit{d}}^{+}=\min \left(-c_2\left(\mathit{q}-{\mathit{q}}^{+}\right),{\mathit{w}}^{+}\right).\hfill \end{array}$$ (10)

Eqn. 10 ensures the joint limits are met. This is because when joints approach the upper (q⁺) and the lower (q⁻) bounds of joint limits, the magnitudes of the two terms, $-c_1\left(\mathit{q}-{\mathit{q}}^{-}\right)$ and $-c_2\left(\mathit{q}-{\mathit{q}}^{+}\right)$ decrease to zeros, as the speed regulated by two positive scaling factors, c₁ and c₂. These two scaling factors can be empirically tuned to clamp the joint accelerations, which improves the control precision in mechanics with non-negligible inertial, for example, cable-driven manipulators. w⁺ and w⁻ are the upper and the lower bound of the joint speed. They can be used to clamp movement speeds, which improve manipulator performance through regulating motion patterns [36].

By substituting Eqn. 8, 9 and 10 into Eqn. 6, we have a new RNN control scheme that meets the constraints of joint limits and joint speeds in obstacle free environments.

C. Obstacle Avoidance and Complete Motion Planning with RNN

Because of the flexibility of RNN architecture, the obstacle avoidance problem can be addressed by both augmenting the task space, and converting the obstacle avoidance constraints into bounding conditions [25], [26]. The former adopts the scheme explained in Subsection II-A. To be more specific, the Jacobian and the task are augmented by the obstacle avoidance constraints, as explained in Eqn. 2. These methods require the dimension of the extra tasks, s, to meet the condition s ≤ m − n. The latter does not have such limitation. In order to explain it, we first introduce the concept of “critical points”, as indicated by green points in Fig. 2. The critical points are the points on the manipulator, whose distances to obstacles equal or are smaller than the safety threshold. Let’s denote the critical points as r_o,i, and denote the nearest points on the obstacle by o_o,i, where i indicates the i-th critical point. Then the algorithms in the latter category avoid a collision through exerting “escaping velocity” on the critical points.

Fig. 2:

Explanation to the Reason that Classical RNN Control Schemes Suffer from the Local Minimum Problem and Lack of Planning Completeness. In the simple 2-dimensional environment, classical RNN control schemes fail to find a valid pathway to move from the pose indicated by the solid line to the pose indicated by the dashed line because the configuration space is concave due to the existence of the obstacle.

The escaping velocity denotes the velocity that moves the critical points away from the obstacles:

$${\stackrel{.}{\mathit{r}}}_{\text{o}}=a\left({\mathit{r}}_o-{\mathit{o}}_o\right),$$ (11)

where a is a semi-positive scaling factor. Any critical point velocity between zero and the escaping velocity will avoid a collision. It is obvious that this method converts the constraints of obstacle avoidance, ${\stackrel{.}{\mathit{r}}}_{o,i}=a\left({\mathit{r}}_{o,i}-{\mathit{o}}_{o,i}\right),$ into a closed set in the configuration space, and in the RNN control scheme, this set bounds neuron activities:

$$\{\begin{array}{ll}\left[{\mathit{d}}_i^{-},\mathbf{0}\right],\hfill & \text{for}{\stackrel{.}{\mathit{r}}}_{o,i}<0,\hfill \\ \left[\mathbf{0},{\mathit{d}}_i^{+}\right],\hfill & \text{otherwise}.\hfill \end{array}$$ (12)

Because in this scheme, the obstacle avoidance does not explicitly “consumed” the manipulator redundancy, algorithms based on this scheme can deal with the infinite number of constraints and are more flexible.

From Eqn. 12 we know that in classical RNN control schemes, the target globally “drags” the end effector to move toward it, while obstacles “push” the critical points to move away. Intuitively, this causes the local minimum problem and the planning is guaranteed to be complete. This problem is visualized in Fig. 2. In the figure, RNN schemes failed to drive the manipulator from the position indicated by the solid red circle, to the target indicated by the dashed red circle, because the obstacle made the configuration concave and trapped the manipulator. The mathematical explanation of this local minimum problem is in Section III-B.

In order to address this problem, the manipulator needs the ability to “jump” out of local minimum, and the neural network randomness can be utilized because recent study demonstrated that the randomness correlated with superior learning abilities and has been used to demonstrate human-level concept learning ability [28].

In order to grant RNNs the randomness, we replace the global attraction from the target with random attractions. The random attractions, ${\mathit{r}}_{random}\in {ℝ}^{\text{n}},$ is designed as:

$${\stackrel{.}{\mathit{r}}}_{\text{d}}=g\cdot \left({\mathit{r}}_{\text{random}}-\mathit{r}\right),$$ (13)

where g is a non-zero weighting factor that regulates the attractions.

D. Balancing Random Exploration and Heuristic Search with Short Term Memory Model

The local minimum problem was due to the concave configuration space. In sparse environments, the heuristic search has a high probability of success and is efficient, therefore, is preferable. In environments with complex obstacles, the heuristic search is easy to fail and the random exploration is the key to success. However, the full knowledge of the environments is often not available, and it is known, calculating the configuration space for the environment is computational expensive [2]. In this work, we balance the heuristic search and the random exploration, through learning the environmental complexity online.

This environmental complexity can be learned by probabilistic methods, such as the classical theory of probability or Bayesian methods. The classical theory of probability is simple in concept as it counts the percentage of failed exploration, but these algorithms prone to bias [37]. For example, in a scenario that the target is inside a corridor, when the manipulator’s initial position is unluckily located, the manipulator reaches obstacles before it reaches the targets. Therefore, the algorithm degenerates to pure random exploration because collisions are frequent. However, we know the configuration space is wide and nicely connected. Bayesian methods do not suffer from this problem, because of the existence of the prior. However, the prior depends on the configuration space and is difficult to achieve [38], [39].

Shunting Short Term Memory model (STM) is a solution to this online learning problem. STM is a robust and powerful tool that describes how living creatures adapt to environmental changes and has been widely used in addressing the system adaptiveness problem [40]. STM was derived from the additive Short Term Memory model, which is mathematically described as [40], [41]:

$$\frac{d{x}_i}{dt}=-A_i{x}_i+\sum _{j=1}^N{f}_j\left(x_j\right)B_{ji}{z}_{ji}^{(+)}-\sum _{j=1}^N{g}_j\left(x_j\right)C_{ji}{z}_{ji}^{(-)}+I_i,$$

where −A_ix_i is a passive decay; $\sum _{j=1}^N{f}_j\left(x_j\right)B_{ji}{z}_{ji}^{(+)}$ is the positive feedback and $\sum _{j=1}^N{g}_j\left(x_j\right)C_{ji}{z}_{ji}^{(-)}$ is the negative feedback; I_i is the input inspiration (see detailed explanation in [40]).

Algorithm 1.

Proposed Random RNN Control Scheme

Input: Target (rT), Manipulator Start Position (r0)
Output: Sequence of Joint States (C)
Init : rs=r0, rd=r0, Ts = {[rs, rd, ∅]}, RRNN†, re‡
1:	if (Neural activity of RRNNSTM > rand (0,1)) then
2:	rd = random point in the task space
3:	else
4:	rd = rT
5:	end if
6:	Select the reached goals( rd in Ts), which is closes to rd, use it as rs
7:	RRNNRNN plan motion for given rs and rd, and produce a sequence of commands cs,d
8:	while re>0 do
9:	rr = end effector position
10:	if (rd = rr) then
11:	if (rd == rT) then
12:	Back tracing control sequence $C=\Sigma c_{s,d}$
13:	Break
14:	end if
15:	else
16:	rd = rr
17:	end if
18:	Feed collision into RRNNSTM
19:	Insert [rs, rd, cs,d] into Ts
20:	re = re − 1
21:	end while
22:	return C

^†The architecture of RRNN is explained in Fig. 1, which consists RRNN_RNN and RRNN_STM.

^‡A big integer to control the maximum allowed exploration time.

The intuition is easy to understand through an analogy of “pain” from the collision. If a collision was fed into a STM, the output of the STM can be seen as the memory of “pain”. In the beginning, the STM has no pain and lean to the heuristic search. While a collision happens, the neural activity of the STM increases and it tends to random exploration. The “pain” decays with time, and it tends to perform the heuristic search again. In the extreme case that collisions happen very time, STM is still able to perform the heuristic search as the decay is continuous and the neuron activities are guaranteed to be unsaturated. From the description we know that unlike to some “windowed” methods, STM does not explicitly have a fixed term of memory; instead, it nonlinear decays memory and the old ones will be “flashed” by new ones.

A single cell STM is the simplest model that meets our needs, which can be mathematically described as:

$$\frac{dx}{dt}=-Ax+(1-x)\left(I+\sum _{j=1}w{\left[x_j\right]}^{+}\right),$$ (14)

where parameters A denotes the passive decay rate; variable x is the neural activity and x ∈ [0,1] is guaranteed; the excitatory inputs (inspirations) to neurons are $I+\sum _{j=1}w{\left[x_j\right]}^{+},$ I is the excitatory input from the exploration, it can simply be 0 for no collision and 1 for collision; w_j denotes weights of the self-excitatory. The neural activities of STM, RRNN_STM, will be used to balance the random search with the heuristic search. The proposed RNN control scheme is explained in Fig. 1 and in Algorithm 1.

Fig. 1:

The Architecture of the Proposed Random Recurrent Neural Network. The proposed method consists a single layer Recurrent Neural Network(RNN) and a single cell Short Term Memory (STM). The STM learns from exploration and controls the RNN to generate precise, efficient and robust motion planning results for redundant manipulators.

In Algorithm 1, RRNN_RNN denotes the control scheme described in Eqn. 8, 9, 10 and 12, and RRNN_STM is mathematically explained in Eqn. 14.

III. THEORETICAL ANALYSES

A. Precision and Stability

Subsection II-B states that the proposed RNN control scheme overcomes the error accumulation and improves the control precision. The classical RNN control scheme is:

$$\begin{gathered} \lambda ={\lambda }_0+\frac{1}{ϵ}\int \left({\dot{\mathit{r}}}_{\text{d}}-J\dot{\mathit{q}}\right)dt, \\ ={\lambda }_0+\frac{1}{ϵ}\left({\mathit{r}}_{\text{d}}-{\mathit{r}}_{d0}-\int J\dot{\mathit{q}}dt\right), \end{gathered}$$ (15)

where λ₀ and ${\dot{\mathit{r}}}_{d0}$ denotes the value of λ and ${\dot{\mathit{r}}}_d$ at t = 0, respectively.

By replacing λ in Eqn. 6 with Eqn. 15 we have:

$$\begin{gathered} ϵ\stackrel{\mathrm{..}}{\mathit{q}}=-\stackrel{.}{\mathit{q}}+P_{\mathrm{\Omega }}\left(J^{\text{T}}({\lambda }_0+\frac{1}{ϵ}({\mathit{r}}_{\text{d}}-{\mathit{r}}_{d0}-\int J\stackrel{.}{\mathit{q}}dt)\right) \\ =-\stackrel{.}{\mathit{q}}+P_{\mathrm{\Omega }}\left(J^{\text{T}}({\lambda }_0+\frac{1}{ϵ}\left({\mathit{r}}_{\text{d}}-{\mathit{r}}_{\text{d}0}-\left(\mathit{r}-{\mathit{r}}_0\right)\right)\right) \end{gathered}$$ (16)

From Eqn. 16 we know that given arbitrary time point t = 0, the errors accumulates with time as long as errors ${\mathit{e}}_0={\mathit{r}}_{d0}-{\mathit{r}}_0\ne 0.$ With the proposed scheme (Eqn. 7), for any time point t, its initial error accumulation, ${\mathit{e}}_0={\mathit{r}}_{d0}-{\mathit{r}}_0\ne 0,$ is fed back into the controller as:

$$ϵ\stackrel{\mathrm{..}}{\mathit{q}}=-\stackrel{.}{\mathit{q}}+P_{\mathrm{\Omega }}\left(J^{\text{T}}(\frac{1}{ϵ}\left({\mathit{r}}_{\text{d}}-\mathit{r})\right)\right).$$ (17)

We can see the accumulated errors are canceled out in Eqn. 17, thus the proposed scheme does not suffer from the problem anymore.

In order to prove the global stability of the proposed method, we define the Lyapunov function as $V={\mathit{e}}^{\text{T}}\mathit{e}/2,$ where $\mathit{e}={\mathit{r}}_{\text{d}}-\mathit{r}$ denotes the tracking errors (used as feedback in Eqn. 7). Then from Eqn. 3a and 5 we know $\stackrel{.}{\mathit{r}}=J{P}_{\mathrm{\Omega }}\left(\frac{1}{ϵ}{J}^{\text{T}}\left({\mathit{r}}_d-\mathit{r}\right)\right).$ Because r_d is the goal, which is a constant, we have:

$$\dot{\mathit{e}}=-J{P}_{\mathrm{\Omega }}\left(\frac{1}{ϵ}{J}^{\text{T}}\mathit{e}\right)$$ (18)

By substituting Eqn. 18 into the defined Lyapunov function, we have:

$$\dot{V}={\mathit{e}}^{\text{T}}\dot{\mathit{e}}=-{\mathit{e}}^{\text{T}}J{P}_{\mathrm{\Omega }}\left(\frac{1}{ϵ}{J}^{\text{T}}\mathit{e}\right)$$ (19)

Because the defined projection function is a saturation function as $P_{\mathrm{\Omega }}(x)=\arg {\min }_{y\in \mathrm{\Omega }}\Vert y-x\Vert ,$ we have:

$${\Vert P_{\mathrm{\Omega }}(x)-x\Vert }^2\le \Vert y-x\Vert .$$ (20)

Therefore, by projection $\frac{1}{ϵ}{J}^{\text{T}}\mathit{e}$ to zero we have:

$$\begin{gathered} {\Vert -\frac{1}{ϵ}{J}^{\text{T}}\mathit{e}\Vert }^2\ge {\Vert P_{\mathrm{\Omega }}\left(\frac{1}{ϵ}{J}^{\text{T}}\mathit{e}\right)-\frac{1}{ϵ}{J}^{\text{T}}\mathit{e}\Vert }^2 \\ ={\Vert P_{\mathrm{\Omega }}\left(\frac{1}{ϵ}{J}^{\text{T}}\mathit{e}\right)\Vert }^2+{\Vert -\frac{1}{ϵ}{J}^{\text{T}}\mathit{e}\Vert }^2 \\ +2{\left(-\frac{1}{ϵ}{J}^{\text{T}}\mathit{e}\right)}^{\text{T}}{P}_{\mathrm{\Omega }}\left(\frac{1}{ϵ}{J}^{\text{T}}\mathit{e}\right) \end{gathered}$$

The above equation is simplified as:

$$2{\left(\frac{1}{ϵ}{J}^{\text{T}}\mathit{e}\right)}^{\text{T}}{P}_{\mathrm{\Omega }}\left(\frac{1}{ϵ}{J}^{\text{T}}\mathit{e}\right)\ge {\Vert P_{\mathrm{\Omega }}\left(\frac{1}{ϵ}{J}^{\text{T}}\mathit{e}\right)\Vert }^2.$$

From the definition of the norm, we have:

$$-{\mathit{e}}^{\text{T}}J{P}_{\mathrm{\Omega }}\left(\frac{1}{ϵ}{J}^{\text{T}}\mathit{e}\right)\le 0.$$ (21)

Eqn. 21 can be used in LaSalle’s invariant set principle to prove that e = 0 is the only solution to $\dot{V}=0,$ and the proposed control scheme globally converges to zero [42].

B. Probability Completeness

We first introduce two lemmas and a definition to facilitate the analysis.

Remark 3.1:

The task space of a manipulator is a bounded connected open set, as $X\subset {ℝ}^n$, because of the existence of obstacles, there is an obstacle space as: $X_{\text{obs}}\subset X$, and the subspace that is reachable by a manipulator becomes $X_{\text{reach}}\subset X_{\text{free}}$, where X_free is the free space as: X_free = X\X_obs.

Lemma 3.1:

For all points in the reachable space $(\forall x_{\text{d}}\in X_{\text{reach}}),$ there exists a space, B_d for x_d, and all points in the space $\left(\forall x_{\text{j}}\in B_{\text{d}}\right)$, a valid path can be found by the control scheme described in Eqn. 7.

Proof:

Because the point x_d is reachable by the manipulator, for any points $\left(\forall x_{\text{s}}\in X_{\text{reach}}\right)$, there exists at least one valid path $\delta :[0,t]\to x_{\delta },x_{\delta }\in X_{\text{reach}}$, and $x_{\delta 0}=x_{\text{s}}$, and $x_{\delta 0}=x_{\text{d}}$. However, because the control scheme defined in Eqn. 7 is consistently attracted by the target r_d, a subset of the path $\delta :[t-n,t]\to x_{\delta }$ that corresponds to the non-concave configuration subspace is directly reachable by the control scheme in Eqn. 7. If we define B_d as $\forall x_{\text{j}}\in B_{\text{d}}$ and $x_{\text{j}}\in \delta :[t-n,t]\to x_{\delta },$, Lemma 3.1 holds truth. □

Intuitively, B_d can be imagined as a basin to x_d. We know that, for most of the manipulators, if there are no joint limits, self collisions and obstacles, B_d = X_reach holds truth for any $x_{\text{d}}\in X_{\text{reach}}$. Those constrains break X_reach into basins, and here we have Lemma 3.2.

Lemma 3.2:

For a given manipulator and a environment, $\cup B_{\text{d}}=X_{\text{reach}}$.

Proof:

Lemma 3.1 proves that $\forall x_{\text{d}}\in X_{\text{reach}},B_{\text{d}}$, exists, so $\cup B_{\text{d}}\ge X_{\text{reach}}$ holds truth. Let’s define $x\in \left(\cup B_{\text{d}}\right)\backslash X_{\text{reach}}$, because x locates on at least one of the valid path, as proved in Lemma 3.1, we have $x\in X_{\text{reach}}$, therefore, $\left(\cup B_{\text{d}}\right)\backslash X_{\text{reach}}=\mathrm{Ø}$, therefore $\cup B_{\text{d}}=X_{\text{reach}}$. □

Now it is ready to prove the proposed method has the planning probabilistic completeness.

Definition 3.1:

For a given target $x_{\text{d}}\in X_{\text{reach}}$, if the set of valid paths, ${\Sigma }_{\delta }=\{\delta :[0,t]\to x_{\delta }$ equals to $\mathrm{Ø}$, it is reported in finite time. If ${\Sigma }_{\delta }\ne \mathrm{Ø},P\left({\Sigma }_{\delta }\cap T_{\text{s}}=\mathrm{Ø}\right)=0.$

Theorem 3.1:

For the proposed method, for ${\Sigma }_{\delta }\ne \mathrm{Ø}$, in finite time, $P\left({\Sigma }_{\delta }\cap T_{\text{s}}=\mathrm{Ø}\right)=0.$

Proof:

Lemma 3.1 and 3.2 show that B_d can be reached within finite time and Theorem 3.1 is immediate.

In real applications, as described in Algorithm 1, the maximum allowed exploration time needs be set to ensure the algorithm will not take arbitrarily long to explore. After exceeding the time, the algorithm will report the non-existence of a valid trajectory.

C. Efficiency

If we denote a valid path by the control points as: $\left\{x_{\text{s}},x_{\text{d}-\text{k}},\cdots ,x_{\text{d}-1},x_{\text{d}}\right\},$ where $x_{\text{s}}\in B_{\text{d}-\text{k}}$ and $\{x_{\text{d}-\text{i}}\in B_{\text{d}-i+1},\forall i\in [1,\cdots ,k]\}.$ Then we have following theorem.

Theorem 3.2:

For the given path, $\left\{x_{\text{s}},x_{\text{d}-\text{k}},\cdots ,x_{\text{d}-1},x_{\text{d}}\right\},$ the possibility of finding a path grows exponentially in n iterations as: $1-\mathit{exp}\left(\frac{2k-np}{2}\right),$ where $p=\min \left\{p_{\text{i}}\right\}$ and p_i denotes the probability of sampling a point in B_i.

Proof:

Because of the uniform random sampling is adopted in the proposed method, given the volume of B, the probability of sampling a point in B is known as the ratio of the volume of B with respect to the volume of the reachable space. However, precisely calculating the volume B is more computationally expensive than motion planning. Here we estimate the upper bound of the efficiency instead.

If we approximate p_i with p, then the probability of sampling B are independent and identical, and follow Bernoulli distribution with parameter p. Let L denotes the event of achieving the given path with n samplings. With the approximation of p, the event L follows a binomial distribution with parameters k and p, as:

$$L~\frac{n!}{k!(n-k)!}{p}^k{(1-p)}^{n-k}$$ (22a)

$$\mu =\text{E}[L]=np.$$ (22b)

$${\sigma }^2=\text{Var}(L)=np(1-p).$$ (22c)

Therefore, $P\left(L\le \frac{k\mu }{np}\right)<\mathit{exp}\left(\frac{\mu }{2}{\left(1-\frac{k}{np}\right)}^2\right)$ hold truth [45]. We also have:

$$\begin{gathered} \frac{\mu }{2}{\left(1-\frac{k}{np}\right)}^2=\frac{\mu }{2}\left(1-\frac{2k}{np}+\frac{k^2}{n^2{p}^2}\right) \\ =k-\frac{np}{2}-\frac{k^2}{2np}. \end{gathered}$$ (23a)

Because $\mathit{exp}\left(-\frac{k^2}{2np}\right)\ge 0,$ the probability of missing the path is less than $exp\left(\frac{2k-np}{2}\right)$, and the probability of finding the path is bigger than $1-exp\left(\frac{2k-np}{2}\right).$ □

IV. ILLUSTRATIVE EXAMPLES AND DISCUSSION

Table. I compares RNN control schemes for redundant manipulator motion planning. To our best knowledge, the proposed method is the first RNN control scheme that achieved planning completeness. Among all those algorithms, three were selected to compare with the proposed method, based on similarity. The first algorithm [25] solves the redundant manipulator obstacle avoidance problem in the velocity space. For conciseness, we refer the work as “Method1” in this paper.The second algorithm [26] addresses the obstacle avoidance problem in the acceleration space, and we refer to it as “Method2”. The third method [44] is not capable of avoiding obstacles, but it demonstrated the superior control precision and the outstanding robustness in obstacle-free environments. In order to provide a benchmark for the proposed method, we included that work in the comparison and refer to it as “Method3”.

TABLE I:

Comparisons among Recurrent Neural Network based Control Scheme for Redundant Manipulators^†.

	Global Convergence	Theoretical Error	Free of Error Accumulation	Free of Matrix Inversion	Physical Limits Avoidance	Smothness Optimization	Obstacle Avoidance	Planning Completeness
Model in [43]	Yes	Nonzero	No	No	No	No	No	No
Model in [44]	Yes	Zero	Yes	Yes	Yes	No	No	No
Model in [36]	Yes	Zero	Yes	Yes	Yes	Yes	No	No
Model in [35]	Yes	Zero	Yes	Yes	Yes	Yes	No	No
Model in [25]	Yes	Zero	No	Yes	Yes	No	Yes	No
Model in [26]	Yes	Zero	No	Yes	Yes	Yes	Yes	No
Proposed	Yes	Zero	Yes	Yes	Yes	Yes	Yes	Yes‡

^†Algorithms closely related to the proposed method are compared here

^‡Probabilistic completeness.

The three representative algorithms were compared with the proposed method on the Mitsubishi PA10–7C based simulation. The PA10 redundant manipulator was chosen because it has 7 DoF and its mechanics are similar to human arms [46].

In the simulation experiments, the parameters of the proposed method were empirically chosen as: k = 100, c₁ = c₂ = 0.5, A = 0.9 and w = 0.95. For the other algorithms, we followed the references to set up the parameters [25], [26], [44].

A. Control Precision and Robustness against Noise

Because “Method3” is not capable of obstacle avoidance, the four algorithms were compared in obstacle-free environments. Obstacle-free environments can also validate that the proposed method balances the random exploration and the heuristic search, and succeed in planning with one heuristic search attempt.

In the simulation experiments, the manipulator starts from a known position and tracks a circular trajectory. White Gaussian process noise was injected in order to study the robustness against noise. The white noise has three different levels of standard deviation: σ = 0.01, σ = 0.05 and σ = 0.25. End effector trajectory errors are compared in Fig. 3. From the results we can see that with the increase of noise, “Method3” and the proposed method showed stronger robustness against noise, due to their ability against error accumulation. Although the proposed method and “Method3” have similar performance, we know the proposed method is better from Table. II.

Fig. 3:

Tracking Precision Comparison. Different levels of process noise have been injected to verify the robustness of the RNN control schemes.

TABLE II:

RMS Position Tracking Error With Respect to Various Noise Level.

	σ = 0.01	σ = 0.05	σ = 0.25
Proposed	0.007	0.010	0.010
Schemel [25]	0.027	0.092	0.246
Scheme2 [26]	0.025	0.069	0.199
Scheme3 [44]	0.005	0.010	0.011

B. Planning Completeness and Efficiency

Because “Method3” is not capable of obstacle avoidance, only “Method1” and “Method2” were compared with the proposed method,

Two representative scenarios in redundant manipulator planning are adopted in the experiments. Scenario 1 is the environment with a plane-like obstacle and Scenario 2 has a window-shaped obstacle. In both scenarios, the three algorithms command the PA10 manipulator to reach a target behind the obstacles. The manipulator randomly starts from the known initial positions because both “Method1” and “Method2” suffer from the error accumulation problem. Table III compares the planning success rates, which is defined as: v_s/v_e, where v_s denotes the total number of successes, and v_e = 50 denotes the total number of experiments.

TABLE III:

Planning Success Rate Comparison.

	Method 1	Method2	Proposed
Scenario 1	92%	92%	100%
Scenario 2	72%	74%	100%

Example planning results in the scenario 1 and 2 are shown in Fig. 4(a)~(c) and (d)~(f), respectively. In the figure, the semi-transparent blue plane denotes the obstacle; the thick colored lines indicate manipulator initial configurations and the thin colored lines denote the trajectory. The goal is denoted by a red sphere, and the curved red line segments denote the end effector trajectories. From Fig. 4 and Table III it is clear that the proposed method addresses the local minimum problem and achieves the planning completeness.

Fig. 4:

Example Planning Results in Environments with a Plane-shaped Obstacle or a Window-shaped Obstacle. The semitransparentplanes denote the obstacle, the red globe denotes the target and the colored lines indicate the manipulator trajectories.

Planning efficiency of the proposed method was demonstrated by comparing the proposed method with a control scheme described in [47]. This control scheme is not capable of learning from exploration. Table IV compares the efficiency and shows how many random explorations and heuristic searches have been conducted in the planning. Fig. 5 shows how RRNN_STM neuron activities change during the exploration and adapt to different environments. From Table IV and Fig. 5 it is clear that the proposed method optimizes the exploration per environment and dramatically increases the planning efficiency.

TABLE IV:

Planning Efficiency Comparison. The averaged total numbers of explorations in 50 experiments are compared in the table.

	Without Learning			Proposed
	RE	HS	Total	RE	HS	Total
Scenario 1	20.80	21.46	42.26	19.88	2.92	22.80
Scenario 2	56.66	56.60	113.26	52.76	5.42	58.18

RE: Random Exploration, HS: Heuristic Search, Total=RE+HS

Fig. 5:

Example RRNN_STM Neuron Activity Changes in Exploration. The neural network learns from exploration. In simpler environments, it tends to perform the heuristic search; in complex environments, it leans to the random exploration.

C. Real World Experiment

The proposed method was applied to a Raven II surgical robot. Raven II surgical robots are popular in the robotic surgery community and are deployed at 18 sites worldwide. The challenges of controlling Raven II, as well as the kinematic model, can be found in [8]. The experiment simulates the popular endoscopic robotic surgeries, in which the robot reaches a target through a small orifice. Compared to exiting teleoperated robotic surgeries, the robot starts from an unknown position and autonomously reaches the goal without collision. Fig. 6a shows the Raven II surgical robot and 6b explains the experimental setup. A box with a 4 cm square opening represents the surgical environmental obstacle and the target is denoted by a pin with a red head. The robot trajectory is recorded by a stereo camera. Fig. 6c visualizes the robot trajectory, in which thick colored lines indicate the manipulator initial configuration and the thin lines indicate the end effector trajectory. The obstacle position is indicated by the semi-transparent blue plane and the goal is denoted by a red dot. From the experimental result, it is clear that the proposed method achieved success in such a complex task in robotic surgeries.

Fig. 6:

Applying the Proposed Method to Simulate Autonomous Robotic Endoscopic Surgery. The Raven II robot automatically reaches the surgical target under the control of the proposed method. (a): the Raven II surgical robot. (b): the experimental setup and the initial manipulator position. The zoomed-in area shows the manipulator reaches the goal. (c): the surgical robot trajectory.

V. CONCLUSION

This paper presents a new RNN control scheme for complete and efficient redundant manipulator motion planning. The proposed scheme addresses the local minimum problem in RNN control scheme, through introducing randomness. Moreover, it avoids the low-efficiency of the random explorations, through online learning from exploration. Rigorous theoretical analyses show the precision, the stability, and the planning completeness and the planning efficiency of the proposed method. Simulation experiments demonstrate the proposed method has better precision, robustness than other three representative RNN schemes, and more importantly, it achieves the planning completeness and improves the planning efficiency. Real-world experiments demonstrate the application of the proposed method in the endoscopic robotic surgery scenario. The proposed method shares the efficiency and the robustness with other RNN control schemes, meanwhile, it achieves higher control precision. More importantly, the proposed method is the first RNN control scheme that achieves the planning completeness in environments with obstacles and shows the much broader applicability of Recurrent Neural Network.