UKF-based closed loop iterative learning control of epileptiform wave in a neural mass model

Abstract

A novel closed loop control framework is proposed to inhibit epileptiform wave in a neural mass model by external electric field, where the unscented Kalman filter method is used to reconstruct dynamics and estimate unmeasurable parameters of the model. Specifically speaking, the iterative learning control algorithm is introduced into the framework to optimize the control signal. In the proposed method, the control effect can be significantly improved based on the observation of the past attempts. Accordingly, the proposed method can effectively suppress the epileptiform wave as well as showing robustness to noises and uncertainties. Lastly, the simulation is carried out to illustrate the feasibility of the proposed method. Besides, this work shows potential value to design model-based feedback controllers for epilepsy treatment.

Introduction

Electroencephalogram (EEG) is the most widely used approach for diagnosis, treatment and control neuropsychiatric disorders (Wang et al. 2010; Han et al. 2013), such as epilepsy. Epilepsy is so common that affects at least 50 million people around the world (Iasemidis et al. 2009; Rummel et al. 2013). The dynamic evolution during epileptic seizure is characterized by epileptiform wave (spikes or spike-wave complexes or sharp waves) and excessive high-frequency EEG waves (gamma band, 20–60 Hz and beyond) (Touboul et al. 2011; Wendling et al. 2002). There are two modeling approaches regarding the EEG phenomenon. One model approach lies at cellular and network level. In this microscopic way, the structural component and functional property of the neuron are modelled accurately and the network is built by a large number of neural cells. EEG is the summated postsynaptic membrane potential of the interconnected individual neurons. The other model technique lies at population level. In this macroscopic way, EEG is the ensemble effect of the interacted pyramidal neurons and interneurons. The neural mass model is such a macroscopic neurophysiological model, and is a capable generator of epileptic activities. The dynamic of a cortical column is represented by a series of nonlinear differential equations, regardless of the complicated network connections and computation complexity of the detailed modelling approach. This modeling approach is initially used in the works of Lopes da Silva et al. (1974, 1976), Van Rotterdam et al. (1982), Freeman (1977, 1987) and Eeckman and Freeman (1991) about 40 years ago, and then developed by Jansen et al. (1993), Wendling et al. (2000, 2002) and David et al. (2003). Their models have been generally used to study EEG rhythms (Bhattacharya et al. 2011), characterize the dynamics of EEG waveforms (Nevado-Holgado et al. 2012; Kiebel et al. 2008), and especially simulate the dynamic pattern of EEG activity during epilepsy seizures (Chakravarthy et al. 2009a; Wendling et al. 2000; Chong et al. 2012).

It is still difficult to explain why and how seizures occur with current knowledge of neurophysiology and experimental studies (Chakravarthy et al. 2009b). Nevertheless, most studies suggest that epilepsy can often be linked with hyperexcitability and hypersynchronization (Touboul et al. 2011). A highly valuable although not well-understood treatment modality for epilepsy is deep brain stimulation (DBS) (Schütt and Claussen 2012), in which strong high-frequency electrical stimulation is applied continuously through deeply implanted electrodes to the sub-cortical target areas like the anterior nucleus of the thalamus (Hodaie et al. 2002; Kerrigan et al. 2004), centromedian nucleus of the thalamus, caudate nucleus, cerebellum, or subthalamic nucleus (Halpern et al. 2008). With DBS as currently used, simulation parameters are empirically chosen by using a trial and error approach. Stimulation is always on during the application regardless of the pathological activity of neural circuits. For instance, seizure events recur unpredictably and are often separated by long interictal intervals. It is clear that battery energy cost and side effects are unavoidable. Unfortunately, there are few guidelines available to guide the selection of appropriate control parameters (Santaniello et al. 2011).

Little and Brown (2012) summarized the ways to improve DBS, one crucial way is to deliver stimulation in a closed loop mode. A number of computational and biological studies have already proved that closed loop stimulation can be highly effective in the treatment of epilepsy (Berényi et al. 2012; Morrell 2011; Fisher et al. 2010; Jobst et al. 2010). In such circumstance, it is extremely desirable to develop a closed loop controller, which automatically adjusts the stimulation signal according to the actual clinical state and duly activates the control scheme. A smart stimulator may improve the efficiency of treatment and weaken the effect on physiological processing, thus reducing energy consumption and fewer side effects (Little and Brown 2012).

In order to inhibit the epileptiform waves of epilepsy and automatically adjust the stimulation parameters according to various physiological states, this paper, introduces a novel unscented Kalman filtering (UKF)-based closed loop iterative learning control (ILC) strategy, treating DBS as an external electric field. ILC is a methodology for reducing control errors from trial-to-trial that operates repetitively (Moore 2001). The main point of ILC is captured by the saying “practice makes perfect” (Tan et al. 2012). When a task is of a repetitive nature, the control effects have the opportunity be improved in the next iteration based on the observation of the past attempts (Moore et al. 1992). ILC takes full advantages of the repetitiveness in the control process (Bristow et al. 2006). As the number of iterations increase, the control signal change through a simple self-tuning process and the tracking error reduce unremittingly. In general, the learning schemes have different types, such as P-type, PI-type, PID-type, and hybrid form combining with forgetting factor (Moore et al. 1992; Ahn et al. 2007; Chien and Liu 1996; Arimoto et al. 1990). Different from plain PID, which only takes use of the control errors to update the control signal, ILC update the control input according to both the input signals and control errors obtained from previous iteration. As compared with others advanced control strategies, ILC implementation is quite simple in the sense that accurate system model and model parameters are not needed.

For successful clinical application of closed-loop control on neurological and psychiatric disorders, the fundamental requirements are record and identify the feedback signals. Such signals should be sensitive and specific to the clinical state and be reliable over time. But some physiologically relevant states and parameters are unmeasurable by conventional means.

In the control theory, the algorithm to estimate the states or the parameters from measured variables is known as ‘observer’, ‘filter’ or ‘estimator’ (Chong et al. 2012). The widely employed stochastic filters for nonlinear systems is the UKF, proposed by Julier et al. (2000). The UKF based on unscented transform (UT) is a powerful tool for estimating states and parameters of nonlinear systems. It has been shown that UKF can approximate the posterior mean and covariance of the Gaussian random variable with a second order accuracy (Julier et al. 2000; Xiong et al. 2006). UKF generalizes elegantly to nonlinear systems without the linearization steps, eliminating the derivation and evaluation of Jacobian matrices required by the extended Kalman filter (EKF) (Julier et al. 2000). UKF has recently been extended used in the neural system to reconstruct the unobserved component of the state vector (Li et al. 2009; Nguyen et al. 2009; Ullah and Schiff 2009, 2010; Galka et al. 2008) or to estimate the parameters of the neural models (Schiff and Sauer 2008; Liu and Gao 2013). But when the system is strong nonlinearity and complexity, the state covariance matrix may be negative semi-definition, and the estimation of states and outputs are unstable. A modified UKF, based on singular value decomposition, is used in our work to ensure the positive semi-definition of the state covariance matrix (Ma et al. 2010). The UKF implemented in our framework is used to estimate the unmeasurable parameters and reconstruct the dynamics of the neural mass model from contaminated EEG.

In summary, we propose a closed loop DBS strategy to restore the epileptiform waves to the expected voltage traces in this paper. For this specified task, we use the UKF method to estimate the unmeasurable feedback signals, and we introduce the iterative learning control algorithm to reduce the PI control error unremittingly during the iterations. In the second section of the paper, we describe the neural mass model, UKF method and ILC algorithm in detail. The third section is the numerical simulation.

Model and methods

Neural mass model

Wendling’s model version includes four neuronal populations (Wendling et al. 2002). They are the pyramidal neurons, the excitatory population, the slow and the fast inhibitory population, as shown in Fig. 1. Interactions between pyramidal neurons and interneurons are represented by connectivity constant C₁ to C₇, which are the average number of synaptic contacts. The model’s excitatory input is average firing rate from neighboring areas. The model output, reflects an EEG signal, is the summated averaged postsynaptic potentials on pyramidal cells (Wendling et al. 2002).

Fig. 1.

Block diagram of neural mass model by Wendling et al. in (2002). The model includes four neuronal populations. They are the pyramidal neurons, the excitatory population, the slow and the fast inhibitory population. The model’s excitatory input is average firing rate from neighboring areas and output is the postsynaptic activity of activated pyramidal cells. Interactions between pyramidal neurons and interneurons are represented by connectivity constant, which are the average number of synaptic contacts. “+” means excitatory input, “−” represents inhibitory input. The control signal is added on the pyramidal cells

There are two operators in model, the first one is a second order differential linear transformation, that transforms the average firing rate m(t) to average postsynaptic membrane potential v(t). $v=h\otimes m$, where $\otimes$ denotes the convolution operator in the time domain and h is the impulse response. For t ≥ 0,

$$\text{the pyramidal cells and the excitatory subsets}{h}_e\left(t\right)=Aatexp\left(-at\right),$$ 1

$$\text{the slow inhibitory subsets}{h}_i\left(t\right)=Bbtexp\left(-bt\right),$$ 2

$$\text{the fast inhibitory subsets}{h}_g\left(t\right)=Ggtexp\left(-gt\right),$$ 3

A, B and G represent the average synaptic gains, their units are millivolts. $1/a$, $1/b$ and $1/g$ are the constant lumping together characteristic delay of the synaptic/somatic transmission, their unites are s^-1.

The second operator transforms the average membrane potential of the population into the average firing rate and is described by a sigmoid function $S\left(v\right)=2e_0/\phantom{2e_0\left(1+exp\left(r\left(v_0-v\right)\right)\right)}\left(1+exp\left(r\left(v_0-v\right)\right)\right)$, where e₀ is half of the maximum firing rate of the population, v₀ is the value of the potential for which a 50 % firing rate is achieved, and r is the slope of the sigmoid at v₀.

Above all, the model in Fig. 1 can be described as a set of 14 ordinary differential equations:

$$\begin{array}{cc}\hfill {\dot{x}}_{11}\left(t\right)& =x_{12}\left(t\right)+V_{ctrl}\hfill \\ \hfill {\dot{x}}_{12}\left(t\right)& =Aa\left\{p\left(t\right)+C_{2}S\left[x_{41}\left(t\right)\right]\right\}-2a{x}_{12}\left(t\right)-a^2{x}_{11}\left(t\right)\hfill \\ \hfill {\dot{x}}_{21}\left(t\right)& =x_{22}\left(t\right)\hfill \\ \hfill {\dot{x}}_{22}\left(t\right)& =Bb\left\{C_{4}S\left[x_{51}\left(t\right)\right]\right\}-2b{x}_{22}\left(t\right)-b^2{x}_{21}\left(t\right)\hfill \\ \hfill {\dot{x}}_{31}\left(t\right)& =x_{32}\left(t\right)\hfill \\ \hfill {\dot{x}}_{32}\left(t\right)& =Gg\left\{C_{7}S\left[x_{61}\left(t\right)-x_{71}\left(t\right)\right]\right\}-2g{x}_{32}\left(t\right)-g^2{x}_{31}\left(t\right)\hfill \\ \hfill {\dot{x}}_{41}\left(t\right)& =x_{42}\left(t\right)\hfill \\ \hfill {\dot{x}}_{42}\left(t\right)& =Aa\left\{C_{1}S\left[x_{11}\left(t\right)-x_{21}\left(t\right)-x_{31}\left(t\right)\right]\right\}-2a{x}_{42}\left(t\right)-a^2{x}_{41}\left(t\right)\hfill \\ \hfill {\dot{x}}_{51}\left(t\right)& =x_{52}\left(t\right)\hfill \\ \hfill {\dot{x}}_{52}\left(t\right)& =Aa\left\{C_{3}S\left[x_{11}\left(t\right)-x_{21}\left(t\right)-x_{31}\left(t\right)\right]\right\}-2a{x}_{52}\left(t\right)-a^2{x}_{51}\left(t\right)\hfill \\ \hfill {\dot{x}}_{61}\left(t\right)& =x_{62}\left(t\right)\hfill \\ \hfill {\dot{x}}_{62}\left(t\right)& =Aa\left\{C_{5}S\left[x_{11}\left(t\right)-x_{21}\left(t\right)-x_{31}\left(t\right)\right]\right\}-2a{x}_{62}\left(t\right)-a^2{x}_{61}\left(t\right)\hfill \\ \hfill {\dot{x}}_{71}\left(t\right)& =x_{72}\left(t\right)\hfill \\ \hfill {\dot{x}}_{72}\left(t\right)& =Bb\left\{C_{6}S\left[x_{51}\left(t\right)\right]\right\}-2b{x}_{72}\left(t\right)-b^2{x}_{71}\left(t\right)\hfill \\ \hfill \end{array}$$ 4

The population’s output is $y\left(t\right)=x_{11}\left(t\right)-x_{21}\left(t\right)-x_{31}\left(t\right)$.

The input $p\left(t\right)$ is modeled by a Gaussian white noise with mean value of 90 and standard deviation of 30. The effect of DBS is regarded as an external electric field V_ctrl added to the function of the pyramidal cells.

The standard values of parameters are set as

$$\begin{array}{cc}\hfill A& =3.25\text{mV,}B=22\text{mV,}G=10\text{mV,}a=100{\text{s}}^{-1},b=50{\text{s}}^{-1},g=500{\text{s}}^{-1}\hfill \\ \hfill C_1& =C,C_2=0.8C,C_3=0.25C,C_4=0.25C,C_5=0.3C,C_6=0.1C,C_7=0.8C,C=135\hfill \\ \hfill v_0& =6\text{mV,}{e}_0=2.5{\text{s}}^{-1},r=0.56{\text{mV}}^{-1}\hfill \\ \hfill \end{array}$$

The standard values were established in (Wendling et al. 2002; Jansen et al. 1993). The model produces well-defined alpha rhythms in EEG with the standard values. This EEG type is treated as the desired EEG dynamics. However, by turning some parameters, the model will produce epileptiform spikes, which is the most characteristic electrophysiological pattern at the beginning stage of epileptic seizure. This is presented in “Numerical result” section.

Unscented Kalman filter

The discrete-time filtering form of the neural mass model is described as

$$\left\{\begin{array}{cc}\hfill x_i& =F_i\left(x_{i-1},u_{i-1}\right)+q_{i-1}\hfill \\ \hfill y_i& =A_i\left(x_i\right)+r_i\hfill \\ \hfill \end{array}\right.$$ 5

x_i is the system state vector at time step i, y_i is the measurement vector, F(x) is the system transfer function, A_i is the measurement matrix. The process noise q and the measurement noise r are conform to a Gaussian distribution.

The UKF equations using 2n sigma points are summarized as the following recursive equations (Ma et al. 2010; Voss et al. 2004; Schiff 2012):

Predict step: Propagate the sigma points through the system dynamics.

$$\begin{array}{cc}\hfill \left[U,D,V\right]& =f_{SVD}\left(P_{xx}\right)\hfill \\ \hfill C& =U\left(\sqrt{nD}\right)U^{\text{T}}\hfill \\ \hfill X_j& ={\widehat{x}}_i^{+}\pm C_j,j=1,2,\dots ,2n\hfill \\ \hfill {\tilde{X}}_j& =F\left(X_j\right)\hfill \\ \hfill {\tilde{Y}}_j& =A\left({\tilde{X}}_j\right)\hfill \\ \hfill {\widehat{x}}_{i+1}^{-}& =\frac{1}{2n}\sum _{j=1}^{2n}{\tilde{X}}_j\hfill \\ \hfill {\tilde{y}}_{i+1}^{-}& =\frac{1}{2n}\sum _{j=1}^{2n}{\tilde{Y}}_j\hfill \\ \hfill P_{xx}^{-}& =\frac{1}{2n}\sum _{j=1}^{2n}\left({\tilde{X}}_j-{\widehat{x}}_{i+1}^{-}\right){\left({\tilde{X}}_j-{\widehat{x}}_{i+1}^{-}\right)}^{\text{T}}+Q_i\hfill \\ \hfill \end{array}$$ 6

Update step: Estimate covariance matrices with the predicted sigma points.

$$\begin{array}{cc}\hfill P_{xy}& =\frac{1}{2n}\sum _{j=1}^{2n}\left({\tilde{X}}_j-{\widehat{x}}_{i+1}^{-}\right){\left({\tilde{Y}}_j-{\widehat{y}}_{i+1}^{-}\right)}^{\text{T}}\hfill \\ \hfill P_{yy}& =\frac{1}{2n}\sum _{j=1}^{2n}\left({\tilde{Y}}_j-{\widehat{y}}_{i+1}^{-}\right){\left({\tilde{Y}}_j-{\widehat{y}}_{i+1}^{-}\right)}^{\text{T}}+R_{i+1}\hfill \\ \hfill K& =P_{xy}{P}_{yy}^{-1}\hfill \\ \hfill P_{xx}^{+}& =P_{xx}^{-}-K{P}_{xy}^{\text{T}}\hfill \\ \hfill {\widehat{x}}_{i+1}^{+}& ={\widehat{x}}_{i+1}^{-}+K\left(y_{i+1}-{\widehat{y}}_{i+1}^{-}\right)\hfill \\ \hfill \end{array}$$ 7

n reflects the dimension, which is equal to the number of elements in the state vector x. There are 2n sigma points and the sigma points X and Y is indexed by j. Index i indicates time. The process noise Q is added to the estimation of P_xx, and the observation noise R is added to the estimation of P_yy.

Iterative learning control algorithm

A P-type iterative learning control algorithm with the concept of the forgetting factor is used in this paper to control the epileptiform EEG series. Saab has studied the robustness of P-type learning algorithms for nonlinear systems with disturbances and noise (Saab 1994). He also figured out the convergence of the learning law. The convergence means that as the increase of iteration times, the output of controlled system will converges to desired value instead.

No matter which learning scheme type is used, the learning controller’s goal is to create an optimal control signal $V_{ctrl}^{\ast }\left(t\right)$ by evaluating the tracking error $e_k\left(t\right)=A_d\left(t\right)-{\widehat{A}}_k\left(t\right)$. A_d(t) is the desired value, ${\widehat{A}}_k\left(t\right)$ is current estimate value of the parameter, and k is the iterative index. In next iteration, the controller adjusts the control signal from current trial V_ctrl(k)(t) to new signal V_ctrl(k+1)(t).

The adjustment is done according to an appropriate algorithm. The P-type iterative learning algorithm with forgetting factor can be described as follows:

$$V_{ctrl\left(k+1\right)}\left(t\right)=\mathit{\alpha }{V}_{ctrl\left(k\right)}\left(t\right)+\left(1-\mathit{\alpha }\right)V_{ctrl\left(k-1\right)}\left(t\right)+K{e}_k\left(t\right)$$ 8

where K is a constant learning gain, and 0 < α < 1 is the forgetting factor. V_ctrl(k−1), V_ctrl(k) and V_ctrl(k+1) are the control signals of the (k − 1)th, kth, and (k + 1)th iterative processes respectively.

Numerical result

Figure 2 shows the block diagram of the UKF-based closed loop ILC system. The neural mass model describes the dynamics of the controlled object. With parameter turning, the model can produce both desired normal background EEG and epileptiform wave. The observer UKF is used to estimate the unobservable feedback signal. The controller is designed based on P-type iterative learning algorithm with forgetting factor. We illustrate the efficiency of the closed loop control system to stop epileptiform activity. We also consider about the robustness of the closed loop system. The noises and disturbance including parameter uncertainty ɛ_θ(t), model structure uncertainty ɛ_sys(t), input disturbance ɛ_p(t) and measurement noise ɛ_y(t).

Fig. 2.

The block diagram of the UKF-based closed loop iterative learning control system. The object is the neural mass model. The observer is the UKF, and the physiologically relevant parameter A is used as feedback variable. The controller is designed based on P-type iterative learning algorithm with forgetting factor. The parameter uncertainty, model structure uncertainty, input disturbance and measurement noise are also taken into account

Dynamic analysis of the neural mass model

Conversion between excitatory and inhibitory can probability predict the transitions in model dynamics. The three main parameters of the model, namely A, B and G, determine the shape of output signal. By altering the value of these parameters, the EEG signal goes from one type to another. The parameters for different types are not fixed, but can be selected from a region. More detail about the dynamic analysis can be found in (Wendling et al. 2002). Turning A is the simplest way to alter the model output EEG between desired normal background activity and epileptiform wave. That is why we mainly consider this parameter. When the parameter is set as A = 3.25 mV, the output of the model is the normal background activities. If the value of A is increases to 3.75 mV, the model produces high amplitude epileptiform wave, which is often precede the fast onset activity of epilepsy. Therefore, we use the parameter A as the feedback signal and apply the UKF to the system as the observer.

As in Fig. 3a1–a4, we reproduce the activities of a seizure. The first segment (a1) is normal back ground activity, and is treat as the expected voltage traces to track in our work. The onset of seizures is characterize by epileptiform wave (a2). Treating this waveform as controlled sequence may be beneficial to inhibit epileptic seizure at the beginning stage of epilepsy. By increasing the value of the average excitatory synaptic gain A from 3.25 to 3.75, the output will change from normal back ground to the epileptiform wave. The segment (a3) is the fast onset activity. (a4) is rhythmic spiking. Figure 3b1–b4 is the normalized power spectral density (PSD). (b3) shows that part of the activities (b3) contain gamma rhythm (20–60 Hz).

Fig. 3.

a1–a4 are the simulated EEG signals generated through the neural mass model. a1 is normal background activities, parameters setting: A = 3.25 mV, B = 22 mV, G = 10 mV, a2 is the high amplitude epileptiform wave, parameters setting: A = 3.75 mV, B = 22 mV, G = 10 mV, a3 is fast onset activity, parameters setting: A = 5 mV, B = 8 mV, G = 30 mV, a4 is rhythmic spiking, parameters setting: A = 5 mV, B = 15 mV, G = 5 mV. b1–b4 are the power spectral density of each segments of the simulated EEG signal

From the above, we assume the average excitatory synaptic gain parameter A as the cause of hyperexcitability and hypersynchronization in epilepsy, and treat A as the physiological information contained feedback signal. Accordingly, the control signal from the controller is added to the excitatory pyramidal neuron subsets to suppress excessive excitatory (Cona et al. 2011; Liu and Gao 2013).

Reconstruction and estimation of the UKF

In this part, we verify the effectiveness of UKF for estimating the feedback signal. First, the model is known and the parameter A is set at 3.25 (Fig. 4a2 green line), the output of model is measureable EEG but contained with measurement noise (Fig. 4a1 red dotted line). And then, assume all the model parameters are known except A. UKF is used to reconstruct the output (Fig. 4a1 black line) and estimate the unobservable parameter A (Fig. 4a2 black line) from the noisy measurement. Result shows that UKF does well in reconstructing the output of the model and in estimating the unobservable parameter. Similar simulation results can be found in Fig. 4b1, b2. The estimation is under epileptic states with parameter A is set at 3.75. The estimations of A are around the true value, and the distinction between normal and epileptiform EEG is apparent. The waveforms of EEG in the two states are also differentiated. The amplitude of epileptiform EEG is much larger than normal EEG, and is contain with epileptiform spikes.

Fig. 4.

States and parameter estimate for the neural mass model with UKF. Part a is under normal condition. a1 shows EEG measurement with measure noise (the red dotted line) and EEG estimate (the black line) with the application of UKF. a2 shows estimation of parameter A (the black line), and the true value of A (the green line). Part b is under epileptic condition. b1 presents the EEG measurement with measure noise (the red dotted line) and EEG estimate (the black line) with the application of UKF. b2 presents the estimation of parameter A (the black line), and the true value of A (the green line). (Color figure online)

In summary, the simulations demonstrate that the UKF is effective in reconstructing the output of the model and in estimating the parameter of the neural mass model. From the reconstruction and estimation of UKF, we can distinguish the normal states and epileptic states obviously, this discrimination provides a potential way to diagnose epilepsy and provide feedback signals. Especially by estimating the unobservable parameter, quantitative distinction can be achieved. The differences between the expected A value and estimated A value are input to the controller to generate corresponding control signals.

Control of the neural mass model

In order to find the suitable controller parameters and further measure the performance of the controller, two quantitative indicators are defined as:

1. The mean value of absolute errors in the iterative process is recorded as follows:

$$e_{mean}=\frac{1}{T}{\int }_0^T\left|e\left(\mathit{\tau }\right)\right|d\mathit{\tau }$$ 9

where e_mean describes the performance of the closed loop control system and e is the difference between the desired and real-time evolution of EEG.

• 2.The accumulated cost of the controller is measured based on the following rule:

$$J={\int }_0^T\left(e{\left(\mathit{\tau }\right)}^2+V_{ctrl}{\left(\mathit{\tau }\right)}^2\right)d\mathit{\tau }$$ 10

where V_ctrl is the control signals after iterations and T is the duration of the control.

As shown in Fig. 5, the value of forgetting factor change α varies from 0.1 to 0.9, and the range of learning gain K varies from 5 to 45 with step size 5. Figure 5a shows the mean value of absolute control error as a function of α and K. High value of forgetting factor with small learning gain and high value of learning gain with small α both lead to large errors. Figure 5b shows the value of accumulated cost. There is a regions in (K, α) space (lower right corner) in which the cost is high. To comprehensively consider the effect of the two variables, the region of (K, α) parameter space that results in small error and little cost can be identified. In fact, the optional combination situates below the counter-diagonal of the parameter space. The controller parameter is selected as (K, α) = (20, 0.1) in the presented work.

Fig. 5.

The mean value of absolute errors and the accumulated cost with the change of forgetting factor and learning gain

We design the controller with the selected parameters and examine the control effectiveness. The expected waveform (Fig. 6a green dotted line) is obtained from a noise-free neural mass model with the same input p(t) and standard parameters. As the black line in Fig. 6a, at the beginning of the simulation, the excitatory average synaptic gains A is set as 3.75 and the output of the neural mass model is epileptiform wave. The control signal is injected into the system after 4 s with the control target to suppress the high amplitude epileptiform wave. The control parameters in iterative learning controller are chosen as K = 20, α = 0.1, the initial value of the controller is set to zero and 20 iterations are conducted in total. Every 8 s is taken as one ‘iteration’. After the last iteration, the real output of model (Fig. 6a black line) has been much closer to the expected value. Figure 6b1 shows the error between desired output and real output in the first and last iteration. In Fig. 6b2, the blue line is the mean value of absolute error and the red line is the accumulated cost of controller during the iterations. After iterations, the errors have converged to a neighborhood of zero.

Fig. 6.

The effect of the closed-loop iterative learning control on the pyramidal neurons of the neural mass model in the state of epilepsy. a The expected time evolution of EEG (green dotted line) and its actual track (black line) with control signals after 4 s. b1 The tracking error for EEG at the first iteration (blue line) and the last iteration (red line) with control signals after 4 s. b2 The mean value of absolute control error (blue line) and the accumulated cost (red line) with the iterations. c1 The estimate of parameter A, the output of the UKF. c2 The mean estimates of A with the iterations. (Color figure online)

Figure 6c1 is the feedback signals in the first and last iteration. It can be seen as a quantitative indicator of epileptic state. After 20 iterations, the parameter $\widehat{A}$ (red line) has decreased to standard value which is around 3.25. Figure 6c2 shows the mean value of $\widehat{A}$ during the iterative process. With the increase of iterations, the mean value of $\widehat{A}$ obviously decreases to the expected value.

As aforementioned, it demonstrates that the UKF-based closed loop iterative learning control framework has the effect of suppressing epileptiform wave into desired waveform in a neural mass model. Simulation results indicated that the model output EEG can easily track the expected values after iteration with appropriate feedback control law and the absolute control error almost close to zero.

Noise, such as ionic channel noise, external environmental noise and unmodeled dynamics are inevitable in the complexity of neural system. We consider about the uncertainty of parameter at first. As presented in Fig. 7a the parameter A (green line) is no more fixed during the control process. From 4 to 6 s, A increases to 4 and then recovers to 3.75. Subsequently, A decreases to 3.5 and then recovers after 2 s. As shown as magenta line in Fig. 7b, model output change to different types with the change of A. The control signal is injected into the system at the beginning of the simulation. The control parameters in the iterative learning controller are chosen as K = 20, α = 0.1, the initial value of the controller is set to zero, and total 20 iterations are conducted. After the application of control effect, the model output is the black line in Fig. 7b. The result shows that the controller can still be effective when the parameter is mutated. Figure 7c is the control signals after iterations.

Fig. 7.

The control effect with parameter variation. a During 4–6 s, A increases to 4 and the then recovers to 3.75. Following, A decreases to 3.5 and then recovers after 2 s. b The abnormal output (magenta line) and the model output (black line) after controlled. The control signal is injected into the system at the beginning of the simulation. c The control signals after iterations. (Color figure online)

As shown in Fig. 2, more noises and uncertainties are taken into account, including the parameter uncertainty ɛ_θ(t), model structure uncertainty ɛ_sys(t), input disturbance ɛ_p(t) and measurement noise ɛ_y(t). These noises are represented by the white noise with the mean value of 0 and the standard deviation as σ. The simulation parameter is set as ɛ_θ ∼ N(0, 0.1), ɛ_sys ∼ N(0, 0.01²), ɛ_p ∼ N(0, 0.1²), and ɛ_y ∼ N(0, 0.1²). The control signal is added after 4 s. The control parameters in iterative learning controller are chosen as K = 20, α = 0.1, the initial value of the controller is set to zero and total 20 iterations are conducted. Simulation results are presented in Fig. 8.

Fig. 8.

The effect of the closed-loop iterative learning control on the pyramidal neurons of the neural mass model in the state of epilepsy. This time, the structure and parameter of the model are inaccurate as well as the system is with input uncertainty and measurement noise. a The expected time evolution of EEG (green dotted line) and its actual track (black line) with control signals after 4 s. b1 The control error of EEG at the first iteration (blue line) and the last iteration (red line). b2 The mean value of absolute error (blue line) and the accumulated cost (red line) with the iterations. c1 The estimate of parameter A, the output of the UKF. c2 The mean estimates of A with the iterations. (Color figure online)

Noises do have effect on the simulation results but not obvious. Fluctuations of the estimate of parameter A (Fig. 8c1 red line and blue line) are more obvious and the accumulated cost (Fig. 8b2 red line) are larger. However, it is still do well in suppressing the epileptiform wave. Along with the iterations, the estimation of the parameter y A gradually reduced (Fig. 8c2 blue line) and is finally stable around the true value of 3.25. Accordingly, the absolute error (Fig. 8b2 blue line) continues dropping to zero (Fig. 8b1).

It illustrates that the robustness of the UKF-based closed loop iterative learning control framework. It can still effectively suppress the epileptiform wave with the effect of noise. The noise immunity makes the framework more suitable for future clinical applications.

Conclusion

In this study, the P-type ILC algorithm with forgetting factor and the UKF method are utilized to provide a closed loop stimulation framework for inhibiting excessive neural excitability in epilepsy disease. Additionally, we demonstrate the treatment efficiency of the strategy on a neural mass model. First, the neural mass model is discussed, which models a population of neurons in a macroscopic way. This macroscopic model raises the feasibility of future application than the design based on microscopic model. Secondly, the effectiveness of UKF is verified for estimating the feedback signal. On one hand, since the measurements or observations are generally contaminated with noise caused by the electronic and mechanical components of the measuring devices, EEG or ECoG is not the best feedback signal. On the other hand, the physiological information contained states and parameters are often not measurable or are too costly to measure. Taken together, using the UKF to estimate the physiology related feedback signal from easily measured EEG series will balance the conflict. At last, the iterative learning control algorithm is applied to design the controller. The optimal controller parameters are selected from numerous combinations. The control signal is gradually revised during the iteration using the error between system output and the desired trajectory, and finally realizes the perfect tracking. The well-trained control signals are robust against the noises and uncertainties among the system especially the parameter fluctuations. The proposed closed loop control strategy can automatically adjust the stimulation signal according to the actual clinical state and deliver the control signal only when needed. This iterative learning control framework improves the efficient of treatment, saves energy, and reduces the side effect on patients. More details about the iterative learning control algorithm can be considered in further studies. More advanced type of the control algorithm and more accurate parameters selection will lead to much better control effect. Our works and results in this paper could have potential value for further research on epilepsy treatment implementation.