The general critical analysis for continuous-time UPPAM recurrent neural networks

Abstract

The uniformly pseudo-projection-anti-monotone (UPPAM) neural network model, which can be considered as the unified continuous-time neural networks (CNNs), includes almost all of the known CNNs individuals. Recently, studies on the critical dynamics behaviors of CNNs have drawn special attentions due to its importance in both theory and applications. In this paper, we will present the analysis of the UPPAM network under the general critical conditions. It is shown that the UPPAM network possesses the global convergence and asymptotical stability under the general critical conditions if the network satisfies one quasi-symmetric requirement on the connective matrices, which is easy to be verified and applied. The general critical dynamics have rarely been studied before, and this work is an attempt to gain an meaningful assurance of general critical convergence and stability of CNNs. Since UPPAM network is the unified model for CNNs, the results obtained here can generalize and extend the existing critical conclusions for CNNs individuals, let alone those non-critical cases. Moreover, the easily verified conditions for general critical convergence and stability can further promote the applications of CNNs.

1 Introduction

The two basic elements of a recurrent neural network (RNN) are: the synaptic connections among the neurons and the nonlinear activation functions deduced from the input-output properties of the involved neurons. For applications such as associative memory, synaptic connections among the neurons are designed to encode the memories we hope to recover. The activation functions are assumed to capture the complex, nonlinear response of neurons of the brain. For different purpose of simulations and applications, both of them are preassigned before use. So understanding their properties are very important, and especially exploring the characteristics of the activation functions are quite crucial to determine the performance of the RNNs. For the commonly used RNN individuals, the activation functions are monotonically nondecreasing and saturated. To study and apply RNNs only based on such two features is far from enough. To overcome the non-thorough descriptions of activation functions, many special cases of activation functions have been brought forward, resulting in many different RNNs individuals. Furthermore, in order to obtain more useful results of RNNs, e.g., the convergence and stability of those individuals, additional strict requirements are unavoidable to impose on the networks for the lack of in-depth descriptions on the activation functions. Obviously, since those individuals are studied separately, it’s inevitable that there exist large numbers of redundancy of analysis for those individual models. In order to reduce the superabundance, establishing a harmonization methodology is a challenging work.

In [16], Xu and Qiao put forward two novel concepts: uniformly anti-monotone as well as the pseudo-projection properties of the activation functions, which discover more essential characteristics other than the nondecreasing and bounded properties of the commonly used activation functions. It is shown that the proposed uniformly pseudo-projection anti-monotone (UPPAM) operator can embody most of activation operators (the precise definition of uniformly pseudo-projection-anti-monotone operator will be given in Section II), e.g., nearest-point projection, linear saturating operator, signum operator, symmetric multi-valued step operator, multi-threshold operator, winner-take-all operator, etc. Thus, the UPPAM operator can be considered as a framework of formalizing most of the activation operators of RNNs.

In this paper, we use the concept of UPPAM operators to establish a unified model for continuous-time RNNs. Let’s consider the following continuous-time UPPAM RNNs mdoel:

$$\tau \frac{dx(t)}{dt}=-x(t)+AG(Wx(t)+q)+b,x_0\in {ℝ}^N$$ (1)

where x(t) = (x₁(t), x₂(t), ···, x_N(t))^T is the neural network state, G = (g₁, g₂, ···, g_N)^T is the nonlinear activation operator deduced from all the activation functions g_i, and G owns the uniformly pseudo-projection-anti-monotone property. Both A and W are the connective weight matrices, b, q are two fixed external bias vectors and τ is the state feedback coefficient. The form of model (1) includes two basic kinds of continuous-time RNNs [17], i.e., the static RNNs and the local field RNNs. Furthermore, as proved in [16], most activation operators are special cases of the UPPAM operator. So, model (1) can be considered as a unified model of continuous-time RNNs and can include almost all of the existing continuous-time RNNs specials [4], e.g., Hopfield-type neural networks, Brain-State-in-a-Box neural networks, Recurrent Back-propagation neural networks, Mean-field neural networks, Bound-constraints Optimization Solvers, Convex Optimization Solvers, Recurrent Correlation Associative Memories neural networks, Cellular neural networks, etc. In addition, since model (1) owns the essential characteristics of the activation functions, i.e., the uniformly anti-monotone as well as the pseudo-projection properties, it can be expected that the analysis of model (1), especially the dynamics analysis can give more in-depth results and provide the unified and concise characterization of the continuous-time RNNs models. The main purpose of this paper will focus on discovering some essential global convergence and stability for the unified model (1), i.e., the critical convergence and stability.

For RNNs, one difficult problem of dynamics analysis lies in the critical analysis. Define a discriminant matrix

$$S(\mathrm{\Gamma },P)=\mathrm{\Gamma }P-\frac{\mathrm{\Gamma }AW+{(\mathrm{\Gamma }AW)}^T}{2},$$

where Γ is a positive definite diagonal matrix, P is a diagonal matrix defined by the network, and W and A are the weight matrices. If there exists a positive definite diagonal matrix Γ, such that S(Γ, 2Λ − B) > 0 (i.e., S(Γ, 2Λ − B) is positive definite), where Λ and B are the anti-monotone and pseudo-projection constant matrices of the network (the definitions of them are given in Section II), then RNNs have exponential stability [4]. Many stability results have been achieved for RNNs individuals under various specifications of S(Γ, 2Λ − B) > 0 (typically, when S(Γ, 2Λ − B) > 0 is an M-matrix), and they are called as the non-critical dynamical analysis [1]. On the other hand, if there exists a positive definite diagonal matrix Γ such that S(Γ, V) is negative definite, here V = diag{r₁, r₂, ···, r_N} with each r_i > 0 being the maximum inversely Lipschitz constant of g_i (i.e., for all s, t ∈ ℛ^N, |g_i(t) − g_i(s)| ≥ r_i|t − s|), then RNNs are globally exponentially unstable [1, 7]. Since S(Γ, 2Λ − B) > 0 is the sufficient condition on the globally exponential stability of RNNs, and S(Γ, V) ≥ 0 is the necessary condition for RNNs to be globally stable, it is quite natural to explore the gap between S(Γ, 2Λ − B) ≤ 0 (i.e., S(Γ, 2Λ − B) is negative semi-definite) and S(Γ, V) ≥ 0 (i.e., S(Γ, V) is positive definite). Such a gap is called the general critical condition, and the dynamics analysis of RNNs under such condition is referred as the general critical dynamics analysis.

For any application and practical design of RNNs, such as pattern recognition, associative memories, or as optimization solvers, the convergence and stability of RNNs are both prerequisite. For instance, when an RNN is used in associative memory or pattern recognition, any pattern we hope to store has to be an equilibrium point of the RNN. In addition, to ensure that each stored pattern can be retrieved even with noises, each equilibrium point must possess the stability. When the RNN is employed as an optimization solver, the possible optimal solutions correspond to the equilibrium of the RNN, and the convergence of the RNN is a guarantee of finding the optimal solutions. Since the general critical conditions can be considered essentially as the distinct region of stability and non-stability of RNNs, studying the general critical dynamics behaviors of an RNN can find broad applications.

Recently, due to the difficulty in the dynamical analysis of RNNs for general critical conditions, most of the studies on critical analysis have been focused on the special critical conditions, i.e., considering the asymptotic behaviors of RNNs under the condition that S(Γ, 2Λ − B) ≥ 0 (this is because S(Γ, 2Λ − B) > 0 is already known to be globally exponential stable and S(Γ, 2Λ − B) = 0 is a special case of the general critical condition). Even for this special critical condition, there only exists a few results since the analysis is much more difficult than the dynamics analysis under the non-critical condition that S(Λ, L) > 0. In [15], the globally exponential stability of a static neural network with projection operator (a special kind of UPPAM operator) has been proven under the condition that I − W is nonnegative (which is a special case of S(Γ, 2Λ − B) ≥ 0). The special critical convergence of a static neural network model with nearest point projection activation operator (special case of projection operator) on a region defined by the network has been achieved in [1] when W is quasi-symmetric. Some general critical stability conclusions for the static and the local field continuous-time RNNs with projection activation operators have been achieved in [2], but they require the network to satisfy one bounded matrix norm. In [4], for the presented unified continuous-time RNNs, namely, UPPAM RNNs, the special critical global convergence is obtained with some bound requirements on the defined nonlinear norm, but such requirements can not be verified easily in applications. In [5], some improvements on dynamics analysis of the UPPAM networks have been obtained, while they are still under the special critical conditions.

In the present paper, we give some solutions on how to assure the convergence and stability under the general critical conditions. By applying the energy function method and Lasalle invariance principle to the unified continuous-time RNNs model (1), we obtain the global convergence and asymptotical stability under some general critical conditions, that is, S(Γ, 2Λ − B)+ Ψ is positive definite for one diagonal matrix Ψ. The results only require the network to satisfy some quasi-symmetric conditions on the connection matrices. Since the conclusions obtained here are for the unified RNNs model under the general critical conditions, they can sharpen and generalize, to a large extent, the latest critical results given by [1, 2, 4, 5, 15], and they can further be extended to those non-critical conclusions (see, e.g., [6–14,18–25] and the references quoted there). Furthermore, they can can be applied directly to many individual RNN models mentioned above. They can be widely applied to solve the linear variational inequality and many other optimization problems, etc. Therefore, the study here provides an insight on the unified continuous-time RNNs model with critical analysis.

2 Preliminaries

For the activation operator G, the domain, range and fixed-point set of G are respectively defined by D(G), R(G) and F(G), and D(G) = R(G) ⊆ ℝ^N. Assume that ℝ^N is embedded with Euclidean norm ||·|| and inner product 〈 ·, · 〉.

For any x = (x₁, x₂, ···, x_N)^T ∈ D(G), write

$$G(x)={(g_1(x),g_2(x),\cdots ,g_N(x))}^T,\forall x\in \mathbf{D}(G)$$

G is said to be diagonal if g_i(x) = g_i(x_i) holds for each i = 1, 2, ···, N.

Definition 2.1

[16] (i) An operator G is said to be a pseudo-projection if there exists a positive definite diagonal matrix B = diag{β₁, β₂, ···, β_N}, such that BR(G) ⊆ D(G) and G = GBG (i.e., G(x)=G(BG(x)), ∀x ∈ D(G)). In this case, we say that G is a B-projection.

(ii) An operator G is said to be λ-uniformly anti-monotone (λ-UAM) if there is a positive constant λ such that for any x ∈ D(G) and y ∈ BR(G),

$$\langle G(x)-G(y),x-y\rangle \ge \lambda {\Vert G(x)-G(y)\Vert }_2^2$$ (2)

(iii) An operator G is uniformly pseudo-projection-anti-monotone (UPPAM) if it is pseudo-projection and uniformly anti-monotone; specially, we say it is (B, λ)-UPPAM whenever it is B-projection and λ-UAM.

Definition 2.2

Let Λ = diag{λ₁, λ₂, ···, λ_N} and B = diag{β₁, β₂, ···, β_N}. G is said to be diagonally (B, Λ)-UPPAM if each component g_i of G is a β_i-projection and λ_i-UAM.

In [16], it is shown that most of the activation operators of RNNs in the literature are special cases of UPPAM operators. Thus, the RNNs with UPPAM operators, i.e., the uniformly pseudo-projection-anti-monotone neural networks provide an appropriate and unified framework, within which most of the known RNN models can be embedded and uniformly studied.

Throughout the paper, the identity matrix is denoted by I. For a positive semi-definite diagonal matrix Δ = diag{δ₁, δ₂, …, δ_N}, let ${\mathrm{\Delta }}^{\frac{1}{2}}=\mathit{diag}\{{\delta }_1^{\frac{1}{2}},{\delta }_2^{\frac{1}{2}},\dots ,{\delta }_N^{\frac{1}{2}}\}$.

3 General Critical dynamics Results

In this section, under the general critical conditions, results on global convergence and asymptotic stability for the unified continuous-time RNN model are established, which are quite easy to be verified in applications. In the following, we denote the equilibrium state set of (1) by $F_e^{-1}(0)$, and the range of nonlinear activation operator, i.e., R(G), by Θ. Throughout this paper, we suppose that Θ is bounded, closed and convex.

Lemma 3.1

For any x₀ ∈ A(Θ) + b, x(t, x₀), the solution of (1), satisfies x(t, x₀) ∈ A(Θ) + b(t ≥ 0).

Proof

By the differential equation theory, we have

$$\begin{array}{l}x(t,x_0)=e^{-\frac{t}{\tau }}{x}_0+\frac{1}{\tau }{e}^{-\frac{t}{\tau }}{\int }_0^t{e}^{\frac{s}{\tau }}(AG(Wx(s)+q)+b)ds\\ =e^{-\frac{t}{\tau }}{x}_0+(1-e^{-\frac{t}{\tau }})\frac{{\int }_0^{\frac{t}{\tau }}{e}^{-r}(AG(Wx(t-\tau r)+q)+b)dr}{1-e^{-\frac{t}{\tau }}}\end{array}$$ (3)

where $r=\frac{t-s}{\tau }$. Since $1-e^{-\frac{t}{\tau }}={\int }_0^{\frac{t}{\tau }}{e}^{-r}dr={lim}_{n\to +\infty }{\sum }_{i=1}^n\frac{t}{\tau n}{e}^{-\frac{it}{\tau n}}$, and A(Θ)+ b is a bounded, closed and convex subset, then $P(t):={\int }_0^{\frac{t}{\tau }}{e}^{-r}(AG(Wx(t-\tau r)+q)+b)dr$, the limit of the sum ${\sum }_{i=1}^n\frac{t}{\tau n}{e}^{-\frac{it}{\tau n}}(AG(Wx(t-\frac{it}{n})+q)+b)$, should satisfy $\frac{P(t)}{1-e^{-\frac{t}{\tau }}}\in A(\mathrm{\Theta })+b(\forall t\ge 0)$. Further, by (3), we know x(t, x₀) ∈ A(Θ) + b when x₀ ∈ A(Θ) + b.

For any v ∈ Θ, we define T(v) = AG(Wv + q) + b. Since that Θ is bounded, closed and convex, then by Brouwer’s fixed point theorem, T has at least one fixed point v^*, so namely, $F_e^{-1}(0)$ is not empty.

Theorem 3.1

Assume that G is diagonally (B, Λ)-UPPAM with Θ being a bounded, closed and convex subset of ℝ^N, and A is a nonzero diagonal matrix. If there exists a positive definite diagonal matrix Γ and a diagonal matrix Ψ, such that (2Λ − B) Γ − ΓAW + Ψ is positive definite, QAW is symmetric (here Q = ((2Λ − B) Γ + Ψ)Λ⁻¹) and Q is a positive definite diagonal matrix, then RNN model (1) is globally convergent on A(Θ) + b when $F_e^{-1}(0)$ is disconnected. Moreover, when x^* is the unique equilibrium point of (1), then x^* is globally asymptotically stable on A(Θ) + b.

Proof

Denote A = diag{a₁, a₂, ···, a_N}, Γ = diag{ξ₁, ξ₂, …, ξ_N}, D = diag{d₁, d₂, ·· ·, d_N} and Ψ = diag{ϕ₁, ϕ₂, ···, ϕ_N}. For any trajectory x(t) of (1) starting from x₀ ∈ A(Θ) + b, it follows from Lemma 3.1 that x(t) ∈ A(Θ) + b. Let y₀ = Wx₀ + q, y(t) = Wx(t) + q, z(t) = AG(y(t)) + b and u(t) = z(t) − x(t).

Let

$$E(x(t))=\frac{\tau }{2}{x}^T(t)(QB-(Q+\mathrm{\Gamma })AW)x(t)-\tau {(\mathit{QAq})}^{T}x(t)-\tau {(\mathit{QBb})}^{T}x(t)+\tau \sum _{i=1}^N{\xi }_i{a}_i{\int }_{{(W{x}_0+q)}_i}^{{(Wx(t)+q)}_i}(a_i{g}_i(s)+b_i)ds$$

Since x(t) ∈ A(Θ) + b, there exists p(t) ∈ Θ, such that x(t) = Ap(t) + b. Then, Ap(t) = x(t) − b. Meanwhile, noting that QB, QAW and ΓAW are all symmetric, and thus W^T(ΓA) = (ΓAW)^T = ΓAW for the case that both A and Γ are diagonal. Then, a direct calculation shows

$$\begin{array}{l}\frac{dE(x(t))}{dt}=\langle (QB-(Q+\mathrm{\Gamma })AW)x(t),u(t)\rangle -\langle \mathit{QAq},u(t)\rangle -\langle \mathit{QBb},u(t)\rangle +\langle \mathrm{\Gamma }Az(t),Wu(t)\rangle \\ =-\langle \mathit{QAWx}(t),u(t)\rangle -\langle \mathit{QAq},u(t)\rangle +\langle \mathit{QBx}(t),u(t)\rangle -\langle \mathit{QBb},u(t)\rangle +\langle \mathrm{\Gamma }\mathit{AWz}(t),u(t)\rangle -\langle \mathrm{\Gamma }\mathit{AWx}(t),u(t)\rangle \\ =-\langle QA(Wx(t)+q),u(t)\rangle +\langle QB(x(t)-b),u(t)\rangle +\langle \mathrm{\Gamma }\mathit{AWz}(t),u(t)\rangle -\langle \mathrm{\Gamma }\mathit{AWx}(t),u(t)\rangle \\ =-\langle \mathit{QAy}(t),u(t)\rangle +\langle \mathit{QBAp}(t),u(t)\rangle +\langle \mathrm{\Gamma }\mathit{AWz}(t),u(t)\rangle -\langle \mathrm{\Gamma }\mathit{AWx}(t),u(t)\rangle \\ =-\langle \mathit{QAy}(t),u(t)\rangle +\langle \mathit{QABp}(t),u(t)\rangle +\langle \mathrm{\Gamma }AW(z(t)-x(t)),u(t)\rangle \\ =-\langle QA(y(t)-Bp(t)),u(t)\rangle +\langle \mathrm{\Gamma }\mathit{AWu}(t),u(t)\rangle \\ =-\langle QA(y(t)-Bp(t)),u(t)\rangle -\langle ((2\mathrm{\Lambda }-B)\mathrm{\Gamma }+\mathrm{\Psi }-\mathrm{\Gamma }AW)u(t),u(t)\rangle +\langle ((2\mathrm{\Lambda }-B)\mathrm{\Gamma }+\psi )u(t),u(t)\rangle \end{array}$$ (4)

Since G is a B-projection and p(t) ∈ Θ, it is clear that G(Bp(t)) = p(t). Denote the diagonal matrix Q = diag{q₁, q₂, ···, q_N}. By the diagonal nonlinear property of G, one can get that

$$\begin{array}{l}-\langle QA(y(t)-Bp(t)),u(t)\rangle =-\sum _{i=1}^N{a}_i{q}_i({(Wx(t)+q)}_i-{\beta }_i{p}_i(t))·((a_i{g}_i({(Wx(t)+q)}_i)+b_i)-(a_i{p}_i(t)+b_i))\\ =-\sum _{i=1}^N{a}_i^2{q}_i({(Wx(t)+q)}_i-{\beta }_i{p}_i(t))·(g_i({(Wx(t)+q)}_i)-g_i({\beta }_i{p}_i(t)))\end{array}$$

For each component g_i of G being a β_i-projection and λ_i-UAM, we have

$$({(Wx(t)+q)}_i-{\beta }_i{p}_i(t))·(g_i({(Wx(t)+q)}_i)-g_i({\beta }_i{p}_i(t)))\ge {\lambda }_i{(g_i({(Wx(t)+q)}_i)-g_i({\beta }_i{p}_i(t)))}^2$$

Further, since both Q and are Λ positive definite diagonal matrices, we have

$$\begin{array}{l}-\langle QA(y(t)-Bp(t)),u(t)\rangle \le -\sum _{i=1}^N{a}_i^2{q}_i{\lambda }_i{(g_i({(Wx(t)+q)}_i)-g_i({\beta }_i{p}_i(t)))}^2\\ =-\sum _{i=1}^N{q}_i{\lambda }_i{(a_i(g_i({(Wx(t)+q)}_i)-p_i(t)))}^2\\ =-{(A(G(Wx+q)-p(t)))}^T\mathrm{\Lambda }Q(A(G(Wx+q)-p(t)))\\ =-u^T(t)Q\mathrm{\Lambda }u(t)\\ =-\langle ((2\mathrm{\Lambda }-B)\mathrm{\Gamma }+\mathrm{\Psi })u(t),u(t)\rangle \end{array}$$ (5)

Then from (4), we directly have

$$\frac{dE(x(t))}{dt}\le -\langle ((2\mathrm{\Lambda }-B)\mathrm{\Gamma }+\mathrm{\Psi }-\mathrm{\Gamma }AW)u(t),u(t)\rangle$$ (6)

By S = (2Λ − B) Γ − ΓAW + Ψ being positive definite, we get that λ_min(S) > 0. From (6), we know

$$\frac{dE(x(t))}{dt}\le -{\lambda }_{\mathit{min}}(S){\Vert z(t)-x(t)\Vert }_2^2$$ (7)

Obviously, it can be deduced that $\frac{dE(x(t))}{dt}\le 0$, and the equal sign holds if and only if z(t) = x(t), i.e., $x(t)\in F_e^{-1}(0)$. Moreover, since x(t) ∈ A(Θ) + b is bounded and $F_e^{-1}(0)$ is disconnected, then by LaSalle invariance principle [26], we know that RNN model (1) is globally convergent on Θ. Furthermore, when $F_e^{-1}(0)=\{x^{\ast }\}$, it is easy to deduce that x^* is both attractive and stable on A(Θ) + b since A(Θ) + b is bounded, i.e., x^* is globally asymptotically stable on A(Θ) + b. Thus, Theorem 3.1 is proved.

Corollary 3.1

Assume that G is diagonally (B, Λ)-UPPAM with Θ being a bounded, closed and convex subset of ℝ^N, and A is a nonzero diagonal matrix. If there exists a positive definite diagonal matrix Γ such that Γ (2Λ − AW) is positive definite, then RNN model (1) is globally convergent on A(Θ) + b when $F_e^{-1}(0)$ is disconnected. Moreover, when x^* is the unique equilibrium point of (1), then x^* is globally asymptotically stable on A(Θ) + b.

Proof

Let Ψ = BΓ in Theorem 3.1, then we have (2Λ − B) Γ − ΓAW + Ψ = Γ(2Λ − AW). Since both Λ and Γ are diagonal and positive definite matrices, we have Q = 2ΛΓΛ⁻¹ = 2Γ is positive definite diagonal matrix, and QAW = 2ΓAW is obviously symmetric by the positive definite requirement of Γ(2Λ − AW). So the corollary follows directly from Theorem 3.1.

Remark 3.1

Studying the dynamic behaviors of unified model (1) can provide uniform results for RNNs and thus can deduce the numerous redundancy existing in the RNNs individuals. Further, since model (1) owns the pseudo projection as well as the anti-monotone property, then by utilizing these two properties, we can obtain some meaningful conclusions.

Recently, the dynamic studies of RNNs have attracted great interest in the critical analysis. It should be pointed out that due to the difficulty in analysis, most of them are based on the special critical conditions, i.e., discriminant matrix S(Γ, 2Λ − B) is positive semi-definite. In addition, in order to assure the stability, some other restrictions are required on the networks [1,2,4,5,15]. Obviously, just studying the special critical dynamics is far from enough in both theory and applications, and additional requirements on the networks are quite hard for applications.

Theorem 3.1 and Corollary 3.1 exploit new methods to assure the global asymptotical stability and global convergence for the unified continuous-time RNN model (1). The results obtained here are under the general critical conditions, and do not need difficultly verified requirements. For Theorem 3.1, in addition to the general critical conditions, it only requires the UPPAM network to meet a quasi-symmetric conditions. That is because, in the sense of positive definite, one can easily choose a diagonal matrix Ψ in Theorem 3.1 satisfying Ψ > (B Ψ 2Λ)Γ, where B, Λ and Γ all are positive definite matrices. Thus, Q is positive definite. Then by Theorem 3.1, in order to assure the global stability and convergence, we only need to verify that QAW is symmetric, where both A and W are connection matrices of the network. Corollary 3.1 shows that existing critical results are the special cases of the results obtained in this paper. The critical condition that Γ((2Λ − B) − AW) ≥0 in [1, 4, 5, 15] is a special case of (2Λ − B) Γ − ΓAW + Ψ ≥ 0. Further, since both Γ and B are positive definite diagonal matrices, thus Γ((2Λ − B) − AW) ≥ 0 is a particular case of Γ (2Λ − AW) > 0. The latter is just the only requirement in Corollary 3.1 to guarantee the global stability and convergence for model (1). The critical dynamic conclusions of Theorem 3.1 and Corollary 3.1 not only summarize, but also deepen to a large extent most of the existing results for the RNNs individuals.

4 Illustrative Examples

In this section, we provide two illustrative examples to demonstrate the validity of the critical convergence and stability results formulated in the previous section. It should be noticed that the known stability and convergence results developed in literature can not be applied here.

Example 4.1

Consider the following UPPAM RNN:

$$\tau \frac{dx(t)}{dt}=-x(t)+AG(Wx(t)+q)+b,x_0\in {ℝ}^6$$ (8)

here each g_i (i = 1, 2, ···, 6) is defined as follows:

$$\begin{array}{c}{g}_i(s)=\{\begin{array}{ll}1,\hfill & s>1/i\hfill \\ i\ast s,\hfill & s\in [-1/i,1/i]\hfill \\ -1,\hfill & s<-1/i\hfill \end{array}\\ In\mathit{this}\mathit{example},W=\left(\begin{array}{cccccc}3.8640& 2.5440& 4.7200& 4.7480& 2.4920& 3.3960\\ -1.7989& -1.9307& -1.9516& -2.0704& -1.3237& -1.7225\\ 2.7251& 1.5935& 3.5839& 3.0946& 1.4318& 2.4942\\ -2.3740& -1.4640& -2.6800& -2.7505& -1.0970& -1.9410\\ 1.1145& 0.8372& 1.1091& 0.9812& 0.7587& 0.8130\\ -1.3864& -0.9945& -1.7636& -1.5848& -0.7422& -1.2680\end{array}\right)\end{array}$$

and A = diag{−1, 2, −3, 4, −5, 6}, q = [−1.2290, −0.9133, 0.1524, 2.8258, −1.5383, 0.9961]^T, b = [−1.782, 1.4427, −0.1067, 1.9619, 3.0046, −2.7749]^T.

In addition, Λ = B = diag{1, 1/2, 1/3, 1/4, 1/5, 1/6}, and the equilibrium state set, $F_e^{-1}(0)$, is a single state set, which contains only one state (−1, −2.5, −3, 4, 5, 0)^T.

For this UPPAM network, almost all of the existing stability conclusions can not be used here. That is because for any positive definite diagonal matrix Γ, (2Λ − B)Γ − ΓAW is neither positive definite nor positive semi-definite, so all the non-critical and the special critical dynamical results (see, the reference mentioned in the Introduction section) are not suitable here. In addition, since this example is for the unified RNNs model, it is totally hard to calculate the nonlinear norm of the network, and the conclusions in [2] can not be used here.

In what follows, we will show that Theorem 3.1 established in this paper can be successfully applied here. By setting $\mathrm{\Gamma }=\mathit{diag}\{1,1/\sqrt{2},1/\sqrt{3},1/\sqrt{4},1/\sqrt{5},1/\sqrt{6}\}$ and Ψ = ΛΓ, we have (2Λ − B) Γ − ΓAW + Ψ ≥ 0, Q = ((2Λ − B)Γ + Ψ)⁻¹) is a positive definite diagonal matrix and QAW is symmetric. Then by Theorem 3.1, it is quite easy to achieve the global convergence of network (8) on A(Θ) + b with Θ = [−1, 1]⁶. FIG. 1 depicts the time responses of neural state variables of the system starting randomly from A(Θ) + b.

FIG. 1.

Transient behaviors of RNN in system (8) with random initial points x₀ ∈ R(G)

Example 4.2

Consider another UPPAM RNN:

$$\tau \frac{dx(t)}{dt}=-x(t)+AG(Wx(t)+q)+b,x_0\in {ℝ}^N$$ (9)

where each g_i(i = 1, 2, ···, N) is defined as follows:

$$g_i(s)=\mathit{min}(\mathit{max}(-1,s),1)$$

W = (W_ij)_N×N with each W_ij is

$$W_{ij}=\{\begin{array}{ll}{(-1)}^{i+1}·(\frac{1}{N(N-1)}+\frac{1}{iN(N-1)}),\hfill & i\ne j\hfill \\ {(-1)}^{i+1}·(\frac{1}{i}-\frac{1}{N}+\frac{1}{iN(N-1)}),\hfill & i=j\hfill \end{array}$$ (10)

and A = diag{a₁, a₂, ···, a_N} with each a_i = (−1)ⁱ⁻¹·i. q and b are two N-dim vectors, $q_i={(-1)}^{i+1}[1+(\frac{1}{i}-\frac{1}{N}+\frac{a}{i})+(N-i)·(a+\frac{a}{i})]$ and b_i = (−1)ⁱi.

In this case, Λ = B = I. For any positive diagonal matrix Γ, Γ (2Λ − B − AW) is not positive semi-definite, i.e., the latest critical results in [5] and other recent results for the special critical analysis, e.g., [2–4] all can not be used for this example. When Γ = diag{1, 1/2, ···, 1/N}, it is easy to verify that Γ(2Λ − AW) is positive definite. Then by Corollary 3.1, network (9) is global convergent to the unique equilibrium, i.e., the origin. The following FIG. 2 depicts the time responses of neural state variables of network (9) with N = 3 starting randomly from A(Θ) + b, where Θ = [−1, 1]³.

FIG. 2.

Transient behaviors of RNN in system (9) with random initial points x₀ ∈ R(G)

5 Conclusion

In the present paper, based on the unified RNN model, i.e., the uniformly pseudo-projection-anti-monotone RNNs model, the corresponding global convergence and the global asymptotic stability under the general critical conditions are given. In addition to the general critical conditions, our conclusions only require that the synaptic connective matrices defined by the network are quasi-symmetric. Comparing to the existing dynamical analysis results for RNNs, the conclusions obtained here demonstrate several advantages. Firstly, they are given under the general critical conditions, which have scarcely been studied before. Secondly, they are for the unified RNN model, so they can be applied to almost all of the existing individuals of RNNs and can be used directly in applications. Finally, with our results there is no need to verify additional intricate requirements on the network, so they can be easily used. In summary, the results achieved here are a significant step towards establishing a unified theory for the dynamics of recurrent neural networks.