Identification of the ARX Model with Random Impulse Noise Based on Forgetting Factor Multi-error Information Entropy

Abstract

Entropy has been widely applied in system identification in the last decade. In this paper, a novel stochastic gradient algorithm based on minimum Shannon entropy is proposed. Though needing less computation than the mean square error algorithm, the traditional stochastic gradient algorithm converges relatively slowly. To make the convergence faster, a multi-error method and a forgetting factor are integrated into the algorithm. The scalar error is replaced by a vector error with stacked errors. Further, a simple step size method is proposed and a forgetting factor is adopted to adjust the step size. The proposed algorithm is utilized to estimate the parameters of an ARX model with random impulse noise. Several numerical solutions and case study indicate that the proposed algorithm can obtain more accurate estimates than the traditional gradient algorithm and has a faster convergence speed.

Introduction

ARX model is an AutoRegressive model with eXogenous terms [31]. Because of its simplicity and easy parameterization, the ARX model has been widely used to model a lot of real systems, such as micro-turbines, data improving, fault detection, biomedical signals, COVID-19 case forecasting and communication systems  [1, 3, 7, 28, 34, 44].

Much research has been performed to identify ARX models in the last five decades. A piecewise auto-regressive exogenous structure was adopted to forecast the river floods [18]. A novel automated framework based on generalized spectral decomposition was proposed to estimate the parameters of an ARX model [33]. A new method based on the expectation–maximization (EM) algorithm was utilized for the identification of ARX models subject to missing data [24]. A recursive EM algorithm based on Student’s t-distribution was used for robust identification of ARX models [8]. A modified momentum gradient descent algorithm was investigated to identify ARX models [50]. A three-stage algorithm was studied for the identification of fractional differencing ARX with errors in variables [25].

However, most of the noises considered in the aforementioned papers are white noises or Gaussian noises. Random impulse signals can often be found in industrial signals, such as image signals, audio signals and communication signals  [2, 9, 46].

Identification criteria play an important role in system identification. The classical identification criteria include the least square criterion, maximum likelihood criterion and so on. These criteria have the advantages of simple calculation and easy theoretical analysis. However, the performance of the least-squares algorithm is poor for the non-Gaussian case, and the maximum likelihood algorithm needs to know the conditional probability density of the sample. Because of these problems, many researchers have put forward many other criteria, such as $p-$norm error criterion and mixed-norm error criterion [39, 53]. In recent years, information criteria have become more widespread in signal processing and system identification [14, 23, 32]. Compared with mean square error (MSE) criterion, which focuses on second-order statistics, the information-theoretic criterion (e.g., minimum error entropy (MEE) [4], Renyi’s entropy [15, 41], fixed-point maximum correntropy [21]) is related to various statistical behaviors of the probability density function (pdf) of the error. Algorithms based on information-theoretic criterion may have better performance than MSE-based algorithms [6, 15, 16].

In the last decade, entropy has found significant applications in system identification. A maximum correntropy criterion (MCC) algorithm was proposed for sparse system identification based on normalized adaptive filter theory [30]. An extended version of correntropy, whose center can be located at any position, and a new optimization criterion based on MCC with a variable center, were proposed [5]. A blocked proportionate normalized maximum correntropy algorithm and a separable maximum correntropy adaptive algorithm were presented to identify dynamic systems [29, 45].

To decrease the entropy estimators’ complexity, a stochastic information gradient (SIG) algorithm was proposed and its performance was investigated [14]. To improve the estimates, a joint stochastic gradient algorithm based on MSE and MEE was proposed and applied to identify an RBF network [4]. Though having less complexity, the SIG converges very slowly. To speed up the SIG, like the multi-innovation used in [12], a multi-error strategy is adopted and a feasible equation for calculation of the step size is introduced.

Since its introduction in 2003, the SIG algorithm has been widely used in system identification and machine learning. For example, a kernel-based gradient descent algorithms based on MEE was proposed to find nonlinear structures in the data, and its convergence rate was deduced [22]. A kernel adaptive filter for quaternion data was developed, and a new algorithm based on the SIG approach was applied to this filter [37]. To avoid unstable training or poor performance in deep learning, a strategy of directly estimating the gradients of information measures with respect to model parameters was explored, and a general gradient estimation method for information measures was proposed [52]. To avoid potentially sub-optimal solutions with respect to class separability, a dimensionality reduction network training procedure based on the stochastic estimate of the mutual information gradient was presented [38].

For several decades, data-driven techniques have been used in modeling and fault detection. For example, a surrogate model was developed based on a data-driven approach, and it can facilitate the design and optimization processes of permanent magnet systems [36]. A Matlab toolbox for data-based fault detection was developed in a unified data-driven framework. [27]. A new recursive total principle component regression-based design and implementation approach was proposed for efficient data-driven fault detection and applied to vehicular cyber-physical systems [26].

In this paper, the problem of parameter identification of the ARX model disturbed by random impulse noise is studied. The premise is that the structure of the model is known, the type of noise is known, the identification data are normal measurement data, and there are no outliers except impulse noise. The possible outliers, modeling errors, and other uncertainties in practice are not considered. Interested readers can refer to the recent literature [13, 47–49]. The main contributions of this work are as follows:

1. For the SIG algorithm, a simple step size method is proposed.

2. To make the algorithm faster, a multi-error method that uses stack error instead of instantaneous scalar error is applied.

3. Since the stack length can only be taken as an integer, a forgetting factor is used to further accelerate the algorithm.

4. The proposed algorithm is utilized to estimate the parameters of an ARX model with random impulse noise. Several numerical simulations and a case study show the effectiveness of the algorithm.

The rest of this work is organized as follows. In the next section, we describe the ARX model to be estimated. Based on an SIG algorithm in Sect. 3, a multi-error SIG with a forgetting factor is presented in Sect. 4. The convergence and the computational cost are analyzed in Sect. 5. Then, parameter estimation of an ARX model with a random impulse noise and a gas furnace dataset from the literature [42] is used to validate the proposed algorithm in Sect. 6. Finally, conclusions are presented in Sect. 7.

Problem Description

Consider an ARX model depicted in Fig. 1, where u(k) is the input and y(k) is the output. $A(z^{-1})$ and $B(z^{-1})$ are two polynomials with respect to $z^{-1}$, and their degrees are $n_a$ and $n_b$, respectively. The model is polluted by a random impulse noise v(k).

Fig. 1.

Block diagram of an ARX model

It is easy to find that

$$\begin{array}{c}\hfill y(k)=\frac{B(z^{-1})}{A(z^{-1})}u(k)+\frac{1}{A(z^{-1})}v(k).\end{array}$$ 1

Multiplying both sides of Eq. 1 by $A(z^{-1})$ gives

$$\begin{array}{c}\hfill A(z^{-1})y(k)=B(z^{-1})u(k)+v(k).\end{array}$$ 2

Suppose $A(z^{-1})=1+a_1{z}^{-1}+a_2{z}^{-2}+\cdots +a_{n_a}{z}^{-n_a}$ and $B(z^{-1})=b_1{z}^{-1}+b_2{z}^{-2}+\cdots +b_{n_b}{z}^{-n_b}$, then we can directly parameterize the model as follows,

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill y(k)& =\left(1,-,A,(,z^{-1},)\right)y(k)+B(z^{-1})u(k)+v(k),\hfill \\ \hfill & ={\phi }^T(k)\theta +v(k),\hfill \end{array}\end{array}$$ 3

with

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\theta ={\left[a_1,,,\cdots ,,,a_{n_a},,,b_1,,,\cdots ,,,b_{n_b}\right]}^T\in {\mathbb{R}}^{n\times 1},\hfill \\ \phi (k)={\left[-,y,(k-1),,,\cdots ,,,-,y,(k-n_a),,,u,(k-1),,,\cdots ,,,u,(k-n_b)\right]}^T\in \mathbb{R}{}^{n\times 1},\hfill \\ n=n_a+n_b.\hfill \end{array}\right)\end{array}$$ 4

Then, the identification of the ARX model shown in Fig. 1 can be transformed into the estimation of the parameters $\theta$ based on the observations ${\left\{u(k),y(k)\right\}}_{k=1}^N$ , where N is the data length.

However, traditional identification algorithms, such as the least-square algorithm and the mean square error algorithm, only consider the second moment of the error, and in some cases (such as the presence of random impulse noise) identification results deteriorate further. The information criterion algorithm based on a probability density function (pdf) considers the statistical information of each order of the error and is expected to achieve better estimates.

Next, we introduce the SIG algorithm and then describe our algorithm based on information gradient, which is integrated with the multi-error and a forgetting factor.

SIG of Shannon’s Error Entropy

Consider the parameterized system in Eq. 3, and denote a random error e(k) as

$$\begin{array}{c}\hfill e(k)=y^{\ast }(k)-{\phi }^T(k)\theta ,\end{array}$$ 5

where $y^{\ast }(k)$ is the system output without noise.

Shannon’s entropy for e with pdf f(e) is [43]

$$\begin{array}{c}\hfill H(e)=-{\int }_{-\infty }^{\infty }f(e)logf(e)de=E\left[-,log,f,(,e,)\right].\end{array}$$ 6

In practice, the pdf of e, i.e., f(e), is unknown. Thus, Eq. 6 cannot be used to calculate the entropy of e directly. One way is to utilize a Parzen window to approximate the unknown pdf underlying the N observations by [41]

$$\begin{array}{c}\hfill \widehat{f}(e)=\frac{1}{N}\sum _{i=1}^N{\kappa }_{\sigma }\left(e,-,e,(,i,)\right),\end{array}$$ 7

where ${\kappa }_{\sigma }(·)$ is the kernel function with size $\sigma$ [40].

At time k, a Parzen window estimate of e with window length L is

$$\begin{array}{c}\hfill \widehat{f}\left(e,(,k,)\right)=\frac{1}{L}\sum _{i=k-L}^{k-1}{\kappa }_{\sigma }\left({\Delta }_{\mathit{ki}}\right),\end{array}$$ 8

where ${\Delta }_{\mathit{ki}}\text{=}e(k)-e(i)$, and e(k), e(i) denote the error at time k, i, respectively.

Thus, the stochastic entropy estimate at time k becomes

$$\begin{array}{c}\hfill \widehat{H}(e(k))=E\left[-,log,\left(\frac{1}{L},\sum _{i=k-L}^{k-1},{\kappa }_{\sigma },\left({\Delta }_{\mathit{ki}}\right)\right)\right].\end{array}$$ 9

Dropping the expectation in Eq. 9 [14], we obtain

$$\begin{array}{c}\hfill \widehat{H}(e(k))\approx -log\left(\frac{1}{L},\sum _{i=k-L}^{k-1},{\kappa }_{\sigma },\left({\Delta }_{\mathit{ki}}\right)\right).\end{array}$$ 10

The stochastic gradient of Shannon entropy concerning $\theta$ at time k, $\mathbf{g}$, is

$$\begin{array}{c}\hfill \mathbf{g}=-\frac{\sum _{i=k-L}^{k-1}{\kappa }_{\sigma }^{\prime }\left({\Delta }_{\mathit{ki}}\right)\left(\frac{\partial e(k)}{\partial \theta },-,\frac{\partial e(i)}{\partial \theta }\right)}{\sum _{i=k-L}^{k-1}{\kappa }_{\sigma }\left({\Delta }_{\mathit{ki}}\right)},\end{array}$$ 11

where ${\kappa }_{\sigma }^{\prime }(·)$ is the derivative of the kernel function.

Using the following Gaussian kernel with variance ${\sigma }^2$, i.e.,

$$\begin{array}{c}\hfill {\kappa }_{\sigma }\left({\Delta }_{\mathit{ki}}\right)=exp\left(-,\frac{{∥{\Delta }_{\mathit{ki}}∥}^2}{2{\sigma }^2}\right),\end{array}$$ 12

Equation 11 becomes

$$\begin{array}{c}\hfill \mathbf{g}=\frac{\sum _{i=k-L}^{k-1}{\kappa }_{\sigma }\left({\Delta }_{\mathit{ki}}\right){ϵ}_{\mathit{ki}}{\Delta }_{\mathit{ki}}}{{\sigma }^2\sum _{i=k-L}^{k-1}{\kappa }_{\sigma }\left({\Delta }_{\mathit{ki}}\right)},\end{array}$$ 13

with

$$\begin{array}{c}\hfill {ϵ}_{\mathit{ki}}=\phi (k)-\phi (i).\end{array}$$ 14

The SIG for estimating the parameter vector $\theta$ is obtained as follows:

$$\begin{array}{c}\hfill \widehat{\theta }(k)=\widehat{\theta }(k-1)+\eta (k)\mathbf{g},\end{array}$$ 15

where $\eta (k)$ is the step size and is critical for convergence speed. However, equations to calculate the step size in [4] and [14] are too complicated to operate online. Here, we utilize the equation in stochastic gradient [10]:

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}& r(k)=r(k-1)+{∥\phi ,(,k,)∥}^2,r(0)=1,\hfill \\ \hfill & \eta (k)=\frac{1}{r(k)}.\hfill \end{array}\right)\end{array}$$ 16

In practice, the $\theta$ and the output without noise $y^{\ast }(k)$ in Eq. 5 are unknown. A feasible way is to replace them by $\widehat{\theta }(k-1)$ and y(k), respectively. Thus, Eq. 5 becomes

$$\begin{array}{c}\hfill e(k)=y(k)-{\phi }^T(k)\widehat{\theta }(k-1).\end{array}$$ 17

Forgetting Factor Multi-error SIG Algorithm

One drawback of the SIG algorithm is its slow convergence. To enable the algorithm converge faster, a multi-error strategy is adopted and Eq. 13 is rewritten as follows:

$$\begin{array}{c}\hfill \mathbf{g}=\frac{\sum _{i=k-L}^{k-1}{\kappa }_{\sigma }\left({\Delta }_{\mathit{ki}}\right)\Delta \mathbf{\Phi }(p;k,i)\Delta \mathbf{E}(p;k,i)}{{\sigma }^2\sum _{i=k-L}^{k-1}{\kappa }_{\sigma }\left({\Delta }_{\mathit{ki}}\right)}\end{array}$$ 18

with

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}& \Delta \mathbf{\Phi }(p;k,i)\text{=}\mathbf{\Phi }(p,k)-\mathbf{\Phi }(p,i),\hfill \\ \hfill & \Delta \mathbf{E}(p;k,i)\text{=}\mathbf{E}(p,k)-\mathbf{E}(p,i),\hfill \\ \hfill \end{array}\right)\end{array}$$ 19

where p is the stack length and $\mathbf{E}(p,k)$ and $\mathbf{\Phi }(p,k)$ are the stacked error vector and stacked information matrix, respectively,

$$\begin{array}{c}\hfill \mathbf{E}(p,k)=\left[\begin{array}{c}e(k)\\ e(k-1)\\ ⋮\\ e(k-p+1)\end{array}\right]\in {\mathbb{R}}^{p\times 1},\end{array}$$ 20

and

$$\begin{array}{c}\hfill \mathbf{\Phi }(p,k)=[\phi (k),\phi (k-1),\cdots ,\phi (k-p+1)]\in {\mathbb{R}}^{n\times p}.\end{array}$$ 21

Note that the scalar error ${\Delta }_{\mathit{ki}}$ in Eq. 13 is replaced by the vector error $\Delta \mathbf{E}(k,i)$ in Eq. 18. In other words, multi-error takes the place of a single error. Thus, the algorithm is named a multi-error SIG algorithm (ME-SIG).

The stack length p can only be a positive integer. To make the ME-SIG faster, a forgetting factor (FF) $\lambda$ is introduced. The first equation of Eq. 16 becomes

$$\begin{array}{c}\hfill r(k)=\lambda r(k-1)+{∥\phi (k)∥}^2,r(0)=1.\end{array}$$ 22

Equations 15 and 17–22 construct the FF-ME-SIG algorithm.

Performance Analysis

Convergence Analysis

The approximate linearization approach [17] is used to analyze the convergence of the proposed ME-SIG algorithm in Eq. (15) with Eqs. (18) and (19).

Subtracting ${\theta }_0$ from both sides of Eq. (15), we obtain

$$\begin{array}{c}\hfill \tilde{\theta }(k)=\tilde{\theta }(k-1)+\eta (k)\mathbf{g},\end{array}$$ 23

where $\tilde{\theta }(k)=\widehat{\theta }(k)-{\theta }_0$ is the estimation error vector of the parameter.

Approximating the gradient $\mathbf{g}$ in Eq. (18) using the first-order Taylor expansion:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathbf{g}& \approx \mathbf{g}({\theta }_0)+H_1\left(\widehat{\theta },(k-1),-,{\theta }_0\right)\hfill \\ \hfill & =H_1\left(\widehat{\theta },(k-1),-,{\theta }_0\right)\hfill \\ \hfill & =H_1\tilde{\theta }(k-1),\hfill \end{array}\end{array}$$ 24

where $H_1=\frac{\partial {\mathbf{g}}^T({\theta }_0)}{\partial \theta }$ is the Hessian matrix and is expressed as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill H_1=& \frac{\sum _{i=k-L}^{k-1}{\kappa }_{\sigma }\left({\Delta }_{\mathit{ki}}\right){\left(\Delta ,{\mathbf{E}}^T,(p;k,i)\right)}^{\prime }\Delta {\mathbf{\Phi }}^T(k,i)}{{\sigma }^2\sum _{i=k-L}^{k-1}{\kappa }_{\sigma }\left({\Delta }_{\mathit{ki}}\right)}\hfill \\ \hfill & +\frac{\sum _{i=k-L}^{k-1}{\kappa }_{\sigma }\left({\Delta }_{\mathit{ki}}\right){\Delta {\mathbf{E}}^T(p;k,i)\left(\Delta ,{\mathbf{\Phi }}^T,(k,i)\right)}^{\prime }}{{\sigma }^2\sum _{i=k-L}^{k-1}{\kappa }_{\sigma }\left({\Delta }_{\mathit{ki}}\right)}\hfill \\ \hfill & -\frac{\sum _{i=k-L}^{k-1}{\kappa }_{\sigma }\left({\Delta }_{\mathit{ki}}\right)\Delta {\mathbf{E}}^T(p;k,i)\Delta {\mathbf{\Phi }}^T(k,i)\sum _{i=k-L}^{k-1}{\kappa }_{\sigma }^{\prime }\left({\Delta }_{\mathit{ki}}\right)}{{\sigma }^2\sum _{i=k-L}^{k-1}{\kappa }_{\sigma }^2\left({\Delta }_{\mathit{ki}}\right)}.\hfill \\ \hfill \end{array}\end{array}$$ 25

Substituting Eq. (24) into Eq. (23), we obtains

$$\begin{array}{c}\hfill \tilde{\theta }(k)=\tilde{\theta }(k-1)+\eta (k)H_1\tilde{\theta }(k-1),\end{array}$$ 26

We analyze the convergence of Eq. 26 by borrowing the results from the LMS convergence theory [19, 20]. Assuming that the Hessian matrix $H_1$ is a normal matrix and can be decomposed into the following normal form:

$$\begin{array}{c}\hfill H_1=Q_1{\Lambda }_1{Q}_1^{-1},\end{array}$$ 27

where $Q_1$ is an $m\times m$ orthogonal matrix, ${\Lambda }_1=diag[{\gamma }_1,{\gamma }_2,\cdots ,{\gamma }_m],{\gamma }_i$ is the eigenvalue of $H_1$. Then, the recursive Eq. 26 can be expressed as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \tilde{\theta }(k)& =Q_1\left[I,+,\eta ,(k),{\Lambda }_1\right]Q_1^{-1}\tilde{\theta }(k-1)\hfill \\ \hfill & =Q_1\left[\prod _{i=1}^k,\left(I,+,\eta ,(i),{\Lambda }_1\right)\right]Q_1^{-1}\tilde{\theta }(0).\hfill \\ \hfill \end{array}\end{array}$$ 28

Clearly, if the following conditions are satisfied, $\tilde{\theta }(k)\to \mathbf{0}$ , i.e., $\widehat{\theta }(k)\to {\theta }_0$:

$$\begin{array}{c}\hfill \left|1,+,\eta ,(i),{\gamma }_j\right|<1,i=1,2,\cdots ,k,j=1,2,\dots ,m.\end{array}$$ 29

Thus, a sufficient condition that ensures the convergence of the algorithm is as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}& {\gamma }_j<0,j=1,2,\cdots ,m,\hfill \\ \hfill & 0<\eta (i)<\frac{2}{\underset{j}{max}|{\gamma }_j|}.\hfill \end{array}\right)\end{array}$$ 30

Computational Analysis

According to the calculation method of [11], the calculation amount of each iteration of the three algorithms is shown in Table 1, where $e^x$ is calculated by its Taylor expansion, and the first three terms are used. ‘Time’ denotes the time that the numerical example consumes.

Table 1.

Computational cost of SIG, ME-SIG and FF-ME-SIG

Algorithm	Computation cost (flops)	$n=4,L=3,p=5$	Time (s)
SIG	$3nL+17L+n+2$	93	0.1788
ME-SIG	$npL+pL+16L+2$	125	0.1860
FF-ME-SIG	$npL+pL+16L+3$	126	0.1862

From the calculation of complexity and running time, we can see that the former algorithm has lower values than the others, while the latter two algorithms have little difference. The computational complexity of the two algorithms proposed in this paper is larger than that of the first algorithm, because the latter two algorithms need the calculation of multi-error. In terms of running time, the latter two algorithms take approximately 4% more time than the previous SIG algorithm, which is not a significant difference.

Experimental Results

Consider the ARX model depicted in Fig. 1 with

$$\begin{array}{c}\hfill \left\{\begin{array}{c}A(z^{-1})=1.0-1.5z^{-1}+0.7z^{-2},\hfill \\ B(z^{-1})=1.0z^{-1}+0.5z^{-2},\hfill \end{array}\right)\end{array}$$ 31

where input data u(k) are an M-sequence and v(k) is a random impulse noise. 5% of the output data (30 outputs) are randomly selected, and 30 noises with random amplitude between 0 and 1 are added, respectively. The curves of input u(k) and output y(k) are shown in Fig. 2. All simulation experiments use this model.

Fig. 2.

Curves of input–output data

Numerical Simulation

(1) Results using SIG algorithm

The parameter estimates using the SIG with window length $L=3$ are shown in Table 2, where the estimation error $\delta$ is defined as $\delta \text{=}\frac{∥\widehat{\theta },(k),\text{-},{\theta }_0∥}{∥{\theta }_0∥}\times 100$. It is easy to determine that estimation error decreases as data length k increases (for a given L). However, the errors are very large (38.9319%) at the end of the estimation.

Table 2.

Results using SIG algorithm ($L=5$)

k	25	50	100	200	400	600	True value
$a_1$	$-$ 0.8723	$-$ 0.8514	$-$ 0.8443	$-$ 0.8453	$-$ 0.8461	$-$ 0.8436	$-$ 1.5000
$a_2$	$-$ 0.1863	$-$ 0.0802	$-$ 0.0595	$-$ 0.0454	$-$ 0.0264	$-$ 0.0135	0.7000
$b_1$	0.1902	0.2164	0.2192	0.2238	0.2288	0.2321	1.0000
$b_2$	$-$ 0.0186	0.0397	0.0456	0.0526	0.0612	0.0666	0.5000
$\delta (\%)$	69.6451	68.1897	67.5927	66.9201	66.0873	65.6092

(2) Results using ME-SIG algorithm

The parameter estimates using the proposed ME-SIG with stack length $p=5$ and $L=3$ are shown in Table 3. The estimation errors with different p are depicted in Fig. 3 ($L=3$). It can be seen that:

1. Estimation error of a given p decreases when data length k increases;

2. With stack length p increasing, the estimation error decreases quickly.

Table 3.

Results using ME-SIG algorithm ($p=5,L=3$)

k	25	50	100	200	400	600	True value
$a_1$	$-$ 1.3218	$-$ 1.3024	$-$ 1.3212	$-$ 1.3428	$-$ 1.3725	$-$ 1.3874	$-$ 1.5000
$a_2$	$-$ 0.2220	0.4978	0.5452	0.5706	0.5970	0.6133	0.7000
$b_1$	0.3139	0.8141	0.8219	0.8260	0.8375	0.8454	1.0000
$b_2$	$-$ 0.0422	0.2836	0.3057	0.3321	0.3684	0.3854	0.5000
$\delta (\%)$	29.3258	20.1085	17.7285	15.8231	13.3009	11.9745

Fig. 3.

Estimation errors using ME-SIG with different stack length p

(3) Results using FF-ME-SIG algorithm

The parameter estimates using proposed FF-ME-SIG with forgetting factor $\lambda =0.99$ are shown in Table 4, where $L=3$ and $p=5$. It can be seen that:

1. Estimation error decreases when data length k increases;

2. Compared with Table 3, the estimation error of the FF-ME-SIG is smaller at the same data length k.

Table 4.

Results using FF-ME-SIG algorithm ($p=5,L=3,\lambda =0.99$)

k	25	50	100	200	400	600	True value
$a_1$	$-$ 1.3236	$-$ 1.3184	$-$ 1.3425	$-$ 1.3746	$-$ 1.4372	$-$ 1.4641	$-$ 1.5000
$a_2$	$-$ 0.2209	0.5130	0.5627	0.6075	0.6513	0.6741	0.7000
$b_1$	0.3152	0.8302	0.8386	0.8463	0.8864	0.9188	1.0000
$b_2$	$-$ 0.0425	0.3015	0.3283	0.3724	0.4446	0.4723	0.5000
$\delta (\%)$	29.1555	18.4745	15.7657	12.6848	7.4742	4.8337

(4) Comparison of the results of SIG, ME-SIG and FF-ME-SIG algorithms

The estimation errors using SIG, ME-SIG and FF-ME-SIG are depicted in Fig. 4. It can be seen that:

Fig. 4.

Estimation errors using SIG, ME-SIG, FF-ME-SIG

1. All curves decrease when data length k increases;

2. The estimation error of the SIG algorithm is larger than that of ME-SIG, which means that multi-error can improve the accuracy of the SIG’s estimate;

3. The estimate of the FF-ME-SIG is the most accurate one of the three. In other words, the introduction of the forgetting factor improves the accuracy of the estimation.

(5) Results using FF-ME-SIG algorithm under different noise additions

To test the performance of the algorithm under different noise levels, we add 5%, 10%, 30%, and 50% of noises to the samples. The mean of the noise is 0.5, and the amplitude is between 0 and 1. The estimation results when $k=600$ are shown in Table 5, where $p=5,L=3,\lambda =0.99$. The estimation error curve is shown in Fig. 5. It can be seen from Table 5 and Fig. 5 that with the increase in the added noise, the estimation error tends to increase, but the change is small, which indicates that the proposed algorithm has strong adaptability to noise.

Table 5.

Results using FF-ME-SIG algorithm under different noise additions ($p=5,L=3,\lambda =0.99$)

Estimates	5%	10%	30%	50%	True value
$a_1$	$-$ 1.4641	$-$ 1.4322	$-$ 1.4807	$-$ 1.3607	1.5000
$a_2$	0.6741	0.6419	0.6910	0.5651	0.7000
$b_1$	0.9188	0.9084	0.9329	0.9424	1.0000
$b_2$	0.4723	0.5026	0.4043	0.4841	0.5000
$\delta (\%)$	4.8337	6.4037	5.9477	10.1580

Fig. 5.

Estimates of FF-ME-SIG using samples with different noise additions

(6) Results using the FF-ME-SIG for the identification of the Narendra difference equations

To support this paper’s argument, we add further simulation examples involving synthetic input–output relations such as the Narendra difference equations proposed in [35, 51]:

$$\begin{array}{c}\hfill \mathit{y}(n+1)=0.3\mathit{y}(n)+0.6\mathit{y}(n-1)+f[\mathit{e}(n)],\end{array}$$ 32

where

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}& f(e)=0.6sin(\pi e)+0.3sin(3\pi e)+0.1sin(5\pi \mathit{e})\hfill \\ \hfill & \mathit{e}(n)=sin\left[(1+\mathit{a}),{\omega }_0,n\right].\hfill \end{array}\right)\end{array}$$ 33

Using the following structure to the model above equation:

$$\begin{array}{c}\hfill y(k)=a_{1}y(k-1)+a_{2}y(k-2)+a_{3}y(k-3)+a_4{y}^2(k-1)+a_5{y}^2(k-2)\end{array}$$ 34

Let the data length be 240. The estimate using the proposed algorithm is $\left[0.3084,,,0.3255,,,0.3641,,,-,0.0264,,,0.0372\right]$ (when $k=240$). The predicted y(k) and observed y(k) are depicted in Fig. 6.

Fig. 6.

Predicted and observed output using FF-ME-SIG for Eqs. 32–33

(7) Results using FF-ME-SIG algorithm and RLS, SG algorithm

To prove the superiority of the proposed algorithm, the identification results of the stochastic gradient (SG) algorithm and recursive least squares (RLS) algorithm are compared with that of the proposed algorithm. Figure 7 shows the estimation error curves of the three algorithms. When $k=600$, the estimation error of the SG, RLS and FF-ME-SIG is 55.5395%, 5.2045% and 4.8337%, respectively. It can be seen that the estimation error of SG is very large, and the estimation error of the RLS algorithm is slightly larger than that of the proposed algorithm. However, due to the impulse noise, the estimation error of the RLS algorithm changes dramatically, which indicates that the estimates given by the RLS algorithm change significantly.

Fig. 7.

Estimate errors using SG, RLS and FF-ME-SIG

Case Study

The data set of a gas furnace from the literature [42] is used to validate the proposed algorithm. These data were continuously collected from a gas furnace and then read every 9 s. The air feed of the furnace was kept constant, but the methane feed rate was varied and the resulting ${\text{CO}}_2$ concentration in the off gases was measured. There are 296 input–output data in this set. The first 200 data are adopted to estimate the parameters. The curves of the input and output are shown in Fig. 8. The estimation results and the prediction errors using SIG, ME-SIG and FF-ME-SIG are listed in Table 6. The outputs y(k) and the prediction errors pe(k) using the proposed algorithm are shown in Fig. 9.

1. It can be seen from Table 6 that among three algorithms, the proposed FF-ME-SIG algorithm has the smallest RMSE, which means that the proposed algorithm can give the most accurate estimate.

2. As shown in Fig. 9, the outputs of the obtained model using the proposed algorithm can predict the observations well.

Fig. 8.

Curves of the input and output data of a gas furnace

Table 6.

Results using SIG, ME-SIG and FF-ME-SIG algorithm

k	SIG	ME-SIG	FF-ME-SIG
$a_1$	$-$ 0.4295	$-$ 0.6126	$-$ 0.6215
$a_2$	$-$ 0.3003	$-$ 0.0561	0.0029
$b_1$	$-$ 0.0868	$-$ 0.0904	0.0614
$b_2$	$-$ 0.1350	$-$ 0.2137	$-$ 0.1414
$b_3$	$-$ 0.1891	$-$ 0.4097	$-$ 0.4190
$b_4$	$-$ 0.1994	$-$ 0.4148	$-$ 0.4577
$b_5$	$-$ 0.1686	$-$ 0.2513	$-$ 0.2869
RMSE	0.2782	0.1183	0.0594

Fig. 9.

Curves of the outputs and prediction errors of a gas furnace

Conclusions

In this paper, a novel SIG algorithm based on minimum error entropy is presented. The traditional SIG algorithm needs less computation than the MSE algorithm. However, it converges slowly. A multi-error strategy and a forgetting factor are introduced to speed up the SIG. We compared the results of SIG, ME-SIG and FF-ME-SIG by estimating the parameters of an ARX model with random impulse noise and through a case study. It is found that SIG with ME and FF can obtain accurate estimates, and it has a quick convergence rate.

Symbols and abbreviations

u(k)

System input at time k

y(k)

System output at time k

$y^{\ast }(k)$

System output without noise at time k

n(k)

System noise at time k

$\theta$

Parameter vector

${\theta }_0$

True value of parameter vector

$\widehat{\theta }(k)$

Parameter estimate at time k

$\tilde{\theta }(k)$

$\tilde{\theta }(k)=\widehat{\theta }(k)-{\theta }_0$

n

Dimension of parameter vector

N

Data length

L

Parzen window length

$\sigma$

Kernel width of Gaussian kernel

e

Error variable

e(k)

Error at time k

pdf

Probability density function

f(e)

Pdf of error

$\widehat{f}(e(k))$

Estimate of f(e) at time k

H(e)

Shannon entropy for e

$\widehat{H}(e(k))$

Estimate of H(e) at time k

$E(\cdot )$

Mathematical expectation

${\Delta }_{\mathit{ki}}$

${\Delta }_{\mathit{ki}}=e(k)-e(i)$

${ϵ}_{\mathit{ki}}$

${ϵ}_{\mathit{ki}}=\phi (k)-\phi (i)$

$\mathbf{g}(k)$

Stochastic gradient of Shannon entropy

${\kappa }_{\sigma }(\cdot )$

Gaussian kernel with variance ${\sigma }^2$

${\kappa }_{\sigma }^{\prime }(·)$

Derivative of ${\kappa }_{\sigma }(\cdot )$

SG

Stochastic gradient

SIG

Stochastic information gradient

$\eta (k)$

Step size of SIG

ME

Multi-error

p

Stack length of ME

$\mathbf{E}(p,k)$

Stacked error vector

$\mathbf{\Phi }(p,k)$

Stacked information matrix

$\Delta \mathbf{\Phi }(p;k,i)$

$\Delta \mathbf{\Phi }(p;k,i)\text{=}\mathbf{\Phi }(p,k)-\mathbf{\Phi }(p,i)$

$\Delta E(p;k,i)$

$\Delta E(p;k,i)\text{=}E(p,k)-E(p,i)$

FF

Forgetting factor

$\lambda$

Forgetting factor

$H_1$

$H_1=\frac{\partial {\mathbf{g}}^T}{\partial \theta }$

I

Identity matrix

$\gamma$

Eigenvalue

Data Availability Statement

All data generated or analyzed during this study are included in the attached file “data used in the paper.xlsx”.