Iterative learning control with applications in energy generation, lasers and health care

Abstract

Many physical systems make repeated executions of the same finite time duration task. One example is a robot in a factory or warehouse whose task is to collect an object in sequence from a location, transfer it over a finite duration, place it at a specified location or on a moving conveyor and then return for the next one and so on. Iterative learning control was especially developed for systems with this mode of operation and this paper gives an overview of this control design method using relatively recent relevant applications in wind turbines, free-electron lasers and health care, as exemplars to demonstrate its applicability.

1. Introduction

Gantry robots are used in many industries to undertake a pick and place operation where the sequence of operations is (i) collect an object from a specified location, (ii) transfer it over a finite duration, (iii) place it at a fixed location or on a moving conveyor, (iv) return to the starting location, (v) repeat steps (i)–(iv) for as many times as required or until a halt for maintenance or other reasons is required. In ideal operation the robot would follow exactly the same user supplied path, or trajectory, on each execution but for applications a control law will be required to enforce tracking of the reference to within a specified tolerance. Iterative learning control (ILC) has been especially developed for such applications and is based on the premise that the performance of a system that executes the same task multiple times can be improved by learning from previous executions.

Given a supplied reference trajectory, the error on each execution can be constructed and used in the computation of the input to be applied on the next execution. Since its introduction, widely credited to Arimoto et al. [1], ILC has and continues to be an active area of control systems theory, design and implementation. In the literature, each execution is known as a trial or a pass and the finite-time duration the trial or pass length. For consistency in terminology, this paper will use trial and trial length and the notation for variables is of the form y_k(t), 0≤t≤α, k≥0, where y is the scalar or vector variable under consideration, $\alpha <\mathrm{\infty }$ denotes the trial length and the non-negative integer k denotes the trial number.

A particular feature of ILC research is that many designs have been experimentally verified in the research laboratory and applied in industrial and other applications. An overview of developments up to their dates of publication can be found in, e.g., [2–4], building on a small number of early research texts [5]. One distinguishing feature of the survey paper [2] is a classification of the design algorithms and their applications.

Applications areas include industrial robotics, e.g. [6], where robot operations common in many mass manufacturing processes is an immediate fit to ILC, and wafer stage motion systems, e.g. [7]. More recent applications areas include flexible valve actuation for non-throttled engine load control [8] and high-resolution electrodynamic jet printing [9]. Also there has been a transfer from engineering to health care for robot-assisted upper limb stroke rehabilitation with supporting clinical trials [10,11], which is considered in this paper.

In general, ILC laws that have found application fall into two classes—simple structure laws whose parameters are adjusted or tuned without the use of a model of the dynamics to be controlled or model-based design where a representation of the dynamics to be controlled is used. The former category of laws is similar in nature to the three-term control laws that have found extensive use in many applications areas, particularly, in the chemical process industries and there is a wealth of literature on how to do the tuning, starting from the Ziegler–Nichols rules. An overview of auto-tuning since the first work can be found in, e.g., [12] and it is estimated that 80% of industrially implemented controllers are based on this structure.

Auto-tuning algorithms do not extend to ILC design but similar structure laws have been widely used. This paper will review developments in this general area and illustrate their use in several new and diverse application areas, including wind turbines, free-electron lasers and rehabilitation. We first address smart rotor control for wind turbines. The advent of higher quality sensors and actuators and digital signal processing for implementation has meant that advanced control laws, i.e. model-based designs, can now be routinely and reliably implemented. These advances have had a dramatic effect on ILC by advancing the possibilities of model-based design. One general model-based design class is based on optimal control, i.e. the control law is selected to minimize a suitably constructed cost function. This paper then focuses on linear norm-optimal, e.g. [13] class of ILC laws and its application in free-electron lasers as one of many. The paper concludes with Newton ILC [14] as an applicable form of nonlinear model-based ILC with again an application in health care and, in particular, robot-assisted upper limb stroke rehabilitation. These applications serve as exemplars for the wide applicability of ILC.

2. Representations and design setting for iterative learning control dynamics

(a). Models for the dynamics

The widely recognized starting point for ILC is [1], which considered a simple first-order linear servomechanism system for a voltage-controlled DC servo motor. As in other areas, there is debate on the origins of ILC, for which the survey papers [2,3] and, in particular, [2] give coverage and relevant references. In the opening paragraphs of Arimoto et al. [1], the analogy between ILC and human learning is drawn in the text: ‘It is human to make mistakes, but it also human to learn from such experience. Is it possible to think of a way to implement such a learning ability in the automatic operation of dynamic systems?’.

The analysis in [1] developed, using the servomotor as a particular example, a control law applicable to systems required to track a desired reference trajectory of a fixed trial length α and specified a priori. On completion of each trial, the system states reset and during time taken to complete this task the measured output is used in the construction of the next control output. The system dynamics were assumed to be trial-invariant and invertible. These distinguishing features led to the establishment of ILC as a major and ongoing area of control systems research and applications. Several of these assumptions, e.g. trial-invariant dynamics, have been relaxed in recent years but the concept of learning from experience gained over repeated trials of a task is retained.

The notation for a scalar or vector-valued variable when ILC is applied to discrete dynamics in this paper is y_k(t), t=0,1,…,α, where the non-negative integer k is the trial number and α denotes the number of samples on each trial for a constant sampling period. Suppose also that the dynamics of the system or process considered can be adequately modelled as linear and time-invariant. Then the system dynamics can be expressed in the form

$$y_k(t)=G(q)u_k(t)+d(t),$$ 2.1

where y_k(t) is the output and u_k(t) is the input and d(t) is an exogenous signal that repeats for each trial (i.e. with k). The input and output can be scalars, termed single-input single-output (SISO), or vectors, termed multiple-input multiple-output (MIMO) in the control literature where in the latter case if the input and output vectors have the same dimensions the system is termed square. For each presentation, the dimensions of the state, input and output vectors will only be specified where actually required. In (2.1), q denotes the forward time-shift operator acting on a vector, say ω(t), as qω(t):=ω(t+1) and the linear operator G(q), a proper rational function of q (in the SISO case with an obvious generalization to MIMO) is termed the process, or plant, in the control literature and has a delay or relative degree m. The system considered is assumed to be asymptotically stable and if not a stabilizing control law must first be designed and ILC applied to the resulting controlled dynamics.

Although in applications the trial length is always finite, in analysis it is required to consider an infinite trial length, which can be undertaken mathematically by letting $\alpha \to \mathrm{\infty }$. The trial index k is usually treated as infinite, i.e. k∈{0,1,2,…}. Discrete-time is a natural setting for ILC analysis as the storage of data from previous trials is required and very often this is sampled. As detailed in, e.g. [3], (2.1) is considered in the research community to be a sufficiently general representation to include particular structures for G(q) that arise when this operator is constructed from input–output data by system identification and/or repeating disturbances and non-zero initial conditions, etc. must be modelled. If required, differential dynamics can be converted to the discrete domain by sampling.

If the dynamics of the system to be controlled cannot be adequately modelled as linear, it is necessary to base analysis and design on a nonlinear state-space model of the form

$$\left.\begin{array}{rl}\dot{x}(t)& =\tilde{f}(x(t),u(t))\\ \text{and}y(t)& =g(x(t))\end{array}\right\}$$ 2.2

or when affine (or linear) in the control

$$\left.\begin{array}{rl}\dot{x}(t)& =f(x(t))+B(x(t))u(t)\\ \text{and}y(t)& =g(x(t)),\end{array}\right\}$$ 2.3

where x is the state vector, u is the input and y is the output. The application of ILC to affine nonlinear systems uses a wide variety of laws but a critical common assumption is that the nonlinear system is smooth. This requirement is often expressed as a global Lipschitz assumption on each of the functions in (2.3) of the form, e.g. | f(x₁)−f(x₂)|≤f₀|x₁−x₂|, where f₀ is a constant that, together with the others, can be used in a contraction mapping setting to obtain (sufficient) conditions for trial-to-trial error convergence and ILC law design. Again if required differential dynamics can be converted to the discrete domain by sampling but this task is much more involved than for linear dynamics.

(b). Objectives and design settings

Let r(t) denote the supplied reference trajectory or vector in the MIMO case. Then the error on trial k is

$$e_k(t)=r(t)-y_k(t)$$ 2.4

and the core requirement in ILC is to construct a sequence of input functions u_k+1(t), k≥0, such that the performance achieved is gradually improved with each successive trial and after a ‘sufficient’ number of these the current trial error is zero or within an acceptable tolerance. Mathematically, this can be stated as a convergence condition on the input and error of the form

$$\underset{k\to \mathrm{\infty }}{lim}\parallel e_k\parallel =0\mathrm{and}\underset{k\to \mathrm{\infty }}{lim}\parallel u_k-u_{\mathrm{\infty }}\parallel =0,$$ 2.5

where $u_{\mathrm{\infty }}$ is termed the learned control and ∥⋅∥ denotes an appropriate norm on the underlying function space. As one possibility, let ∥⋅∥₂ denote the Euclidean norm of its argument. Then one choice is $\parallel e\parallel =\underset{t\in [0,\alpha ]}{max}\parallel e(t){\parallel }_2$. The reason for including the requirement on the control vector in (2.5) is to ensure that strong emphasis on reducing the trial-to-trial error does not come at the expense of unacceptable changes in the control signal from one trial to the next. In application, only a finite number of trials will ever be completed but mathematically letting $k\to \mathrm{\infty }$ is required in analysis of, e.g., trial-to-trial error convergence.

The standard form of ILC law computes the current trial input as the sum of the input used on the previous trial plus a correction term, i.e.

$$u_{k+1}=u_k+\mathrm{\Delta }(u_k,e_k),$$ 2.6

where Δ(u_k,e_k) is the correction term and is a function of the error and input recorded over the previous trial. A large number of variations exist for computing the correction term, including laws that make use of information generated on a finite number (greater than unity) of previous trials. Explicit use of information from more than one previous trial is termed higher order ILC and there is continued debate in the literature as to what benefits such a control law actually delivers. Such control laws are not considered further in this paper.

In ILC, once trial k is complete the following information is available for the computation of the control u_k+1: (i) information from the entire time duration of any previous trial and (ii) information up to the current sample on trial k+1. For standard linear systems at sample instant p, the use of information at future samples p+1,p+2,… is non-causal and, therefore, any resulting control law cannot be implemented. The ability to use non-causal temporal information in an ILC law is arguably its most important feature. A common form of non-causal ILC law is termed phase-lead and has the form

$$u_{k+1}(t)=u_k(t)+K_t{e}_k(t+\mathrm{\lambda }),$$ 2.7

where λ>0 is the phase-lead and K_t is a scalar gain to be selected (matrix in the MIMO case).

In applications terms, the design of an ILC law can be viewed as the generation of an open-loop, i.e. no feedback, signal that approximately inverts the system dynamics, track the supplied reference trajectory and reject repeating disturbances. Also in an ideal setting the law learns only repeating disturbances and ignores noise and disturbances that are not repeatable. ILC is an open-loop control and has no feedback action to respond to unanticipated, non-repeating disturbances. Consequently including feedback control action in combination with ILC can be beneficial in many applications. For a detailed treatment of this structural aspect of ILC, see [3] and the relevant cited references.

The methods for the design of ILC laws can, in general, terms be classified under four classes: (i) tuning, (ii) system inversion, (iii) robust, and (iv) optimal. Tuning designs can be applied to an application without extensive modelling or analysis. They are the ILC equivalent of auto-tuning designs for proportional plus integral plus derivative (PID) designs that still form a very high percentage of implemented control laws, where the study [12] gives a comprehensive treatment of the basics of this method in the non-ILC case. Systems inversion involves learning the inverse of the system dynamics and can converge quickly in terms of the trial error but relies heavily on the quality of the model and sensitivity to modelling errors may arise. Robust design, e.g. in an ${\mathcal{H}}_{\mathrm{\infty }}$ setting, can be used to design a law that enforces monotonic convergence in the trial number (k) of the sequence formed from a suitable measure of the error on each trial but at the possible expense of performance. Optimal design is based on designing the control law to minimize a suitable cost function and parallels the standard linear systems case. In this paper, designs under the general headings of (i) and (iv) are considered in detail.

3. Aerodynamic load control on a wind turbine blade

(a). Background

The aerodynamic forces on wind turbine blades fluctuate with both deterministic and stochastic elements. The most obvious stochastic component arises from the variable nature of the wind, which varies in both frequency and magnitude and produces a variation in the aerodynamic load that passes through the turbine system. Deterministic components include the effects of the atmospheric boundary layer, stator–rotor interaction and yaw misalignment. These disturbance result in loads that require management, especially because blades are relatively flexible structures.

In essence, wind turbine load control involves modifying the lift on the blades and can be achieved in a number of ways [15]. Here the aim is to damp fluctuations in the lift using circulation control as a model of a blade equipped with a smart rotor (a smart rotor has an actively controlled device embedded in the blade which adjusts according to the flow conditions). A reduction in load fluctuation would produce an increase in component lifespan and reduce maintenance.

Reducing maintenance costs is especially important given the ongoing trend to operate offshore, where remote environments and difficult access means a dramatic increase in the cost of maintenance operations. The main aim is to reduce fatigue loads but smart devices can also be used to attenuate ultimate or extreme loads and regulate power. As shown below, the latter can be addressed within the ILC framework by the addition of a proportional term to the basic ILC law. Examples of smart actuators are trailing edge flaps [16–18], microtabs [19,20] and active vortex generators.

Wind turbines operate in the atmospheric boundary layer where there is a wind shear with a non-uniform mean velocity profile and a regular variation in wind speed past the blade throughout a cycle, even in reasonably steady non-gusting conditions. Hence the flow past the blade will contain an oscillatory component that will be more pronounced towards the tip of the blade, where the greatest speed differential arises. This variation in the flow will cause cyclic loads on the blade, in particular, and thereby affect the lift. However, there will also be non-cyclic load from effects such as gusts, hence there is also a desire to reduce the load due to peak or extreme events, particularly those that threaten the structural integrity of the turbine. Hence, the objective for control system design should include reducing both the underlying level of disturbance and the peak load.

As the flow conditions vary along the blade, the actuation should also vary spanwise along the blade. In this work a computational fluid dynamics (CFD) model, based on a panel method, is used to simulate flow past an aerofoil, representing a section of the aerofoil. The incoming flow is assumed to be unsteady, and also has added disturbances in the form of vortices that are convected with the flow. Two types of vortex are present in this updated model. The upstream vortices represent nonlinear disturbances to the incoming flow, while vortices are released from the trailing edge every time step generate a wake downstream of the aerofoil. The vortices interact with each other and with the aerofoil, resulting in a nonlinear system. The motion of the vortices is found by solving the Euler equations. Hence, this model introduces both non-periodicity and nonlinearity into the problem in a realistic manner as it is based on a solution to the governing equations for inviscid flow with rotational disturbances convecting in a physically compatible manner.

Devices, such as trailing-edge flaps and microtabs, operate by generating circulation (vorticity) in the region of the trailing edge of the blade, thereby directly affecting the lift on the blade. In the model, the flow at the trailing edge of the aerofoil is manipulated to represent these devices in a generic manner and thereby provide the actuation for the control system. The lift on the aerofoil is used as the output of the system and the error between the lift and a fixed target value used to activate the control. Calculating the lift is easily achieved in the numerical model, but in practice would need to be estimated from measurements, e.g. from pressure sensors distributed on the surface of the blade.

The effectiveness of the control system is assessed with a two-norm measure of the disturbance, which relates to the fatigue load, and an $\mathrm{\infty }$-norm, which measures the reduction in the peak disturbance. The aerofoil used is an NREL S825 [21] (figure 1). This aerofoil is specifically designed for wind turbines and has a similar profile to the blade sections used in practice.

Figure 1.

NREL S825 aerofoil.

(b). Flow model

Flow past a wind turbine blade has two characteristic time scales, the period of rotation, approximately 5 s, and typical time for the flow to pass the blade section. The former of these remains relatively constant but the latter varies along the blade, due to both the change in the chord (length) and velocity of the blade section. The chord decreases along the blade, e.g. from 4 m near the hub to around 1 m near the tip for a Vestas V112-3.0 MW turbine which has a blade length of 54.6 m. The velocity increases along the blade, giving a significant variation in the characteristic time for the flow to pass the blade.

The problem is considered in non-dimensional form using the mean free stream velocity $V_{\mathrm{\infty }}$ on a blade section and the chord (length) of the aerofoil H as reference values. This is standard aerodynamics, for the background see, e.g., [22], and ${\mathbf{\text{v}}}^{\ast }=V_{\mathrm{\infty }}\mathbf{\text{v}}=V_{\mathrm{\infty }}(v_x,v_y)$ are the velocity components in x*=Hx=H(x,y), where the asterisk denotes a dimensional quantity. Time is considered as a non-dimensional quantity using $H/V_{\mathrm{\infty }}.$ Hence $t^{\ast }=H/V_{\mathrm{\infty }}t$ and the variation of time scales along the blade is now represented by a change in the non-dimensional period of the rotation, increasing with radius.

The lift of an aerofoil/wing section comes primarily from the pressure exerted by the fluid on the surface of the aerofoil. In normal operating conditions, the angle of attack (AoA) is not high enough to provoke separation, hence the flow remains attached, and the pressure distribution on the surface of the blade can be calculated by assuming the flow is inviscid. The model employed here is the same as used in [23] but is substantially different from that in [24] due to the inclusion of a wake generated by shedding vorticity in the form of discrete vortices from the trailing edge of the aerofoil into the flow. This vorticity then convects downstream as a solution of the Euler equations that govern inviscid flow. Thus the flow has a representation of the full inviscid dynamics, unlike the simpler model used in [24] that assumed a quasi-steady response of the lift generated on the aerofoil to changes in the incoming flow.

Relatively simple inviscid models of this kind do exclude extreme cases, such as rapid changes of direction or shear when separation is provoked, but are an acceptable alternative to full Navier–Stokes simulations, which give a complete characterization of the flow, but are too expensive computationally to allow detailed investigation of the potential of suitably designed control laws. This model allows investigation of feasible control schemes that could eventually be applied to full-scale simulations or experiments.

The boundary conditions on the surface of the body are satisfied using a panel method (details can be found in [23,24]). The base flow consists of the free-stream velocity V₀(t)=(V _0x(t),0) and the velocity field generated by the vortex panels, the vortices shed into the wake from the trailing edge and also disturbances are introduced into the flow upstream of the aerofoil, also in the form of discrete vortices. The Euler equations governing two-dimensional inviscid incompressible flow can be written in vorticity form as

$$\frac{D\omega }{Dt}=\frac{\mathrm{\partial }\omega }{\mathrm{\partial }t}+v_x\frac{\mathrm{\partial }\omega }{\mathrm{\partial }x}+v_y\frac{\mathrm{\partial }\omega }{\mathrm{\partial }y}=0,$$ 3.1

where ω=∂v_y/∂x−∂v_x/∂v_y is the vorticity and D/Dt=∂/∂t+v_x∂/∂x+v_y∂/∂y is the material derivative, i.e. the rate of change with time of a material quantity convected with the flow.

The partial differential equation (3.1) is a statement of the property that in two-dimensional inviscid flow, vorticity is convected with the flow at the local fluid velocity [25]. Consequently, the motion of an individual discrete vortex can be tracked by solving

$$\frac{\mathrm{d}{\mathbf{\text{x}}}_v}{\mathrm{d}t}=\mathbf{\text{v}}({\mathbf{\text{x}}}_v,t),$$ 3.2

where x_v is the position of the centre of the vortex. The complete velocity field v is obtained by summing the three components, i.e. the free-stream velocity, the velocity generated by the panels representing the body, and that from the vortices in the flow field. The position of the vortices can be updated by applying any standard time-stepping method to (3.2) for each of the discrete vortices. As in [26], a second order Runge–Kutta method was used to move the vortices.

Trailing-edge devices used for lift control act by modifying the flow near the trailing-edge, generating vorticity, which is shed into the wake, and thereby alters the circulation on the body and hence the lift. For example, a trailing edge flap redirects the flow. This can be modelled in a simple manner in the current framework by altering the strength of the new vortex generated at each time step, which will directly change the way the flow leaves the aerofoil at the trailing edge, i.e. by assuming the new vortex has strength

$${\mathit{\Gamma }}_n=-u,$$ 3.3

when control is applied, where u is the control input of the system, with the lift as output. The lift is calculated in standard fashion from the pressure distribution on the surface of the aerofoil (e.g. [22]).

In summary, the model given above is a relatively simple (compared with a full Navier–Stokes simulation) but realistic model of the flow and actuation which can be used to investigate control schemes aimed at damping fluctuations in the lift using trailing-edge devices for load control. To validate the model, the basic panel code was tested against results from standard sources. In particular, the lift coefficients and pressure distributions for inviscid flow past the aerofoil obtained from the method presented above were compared with those found using XFOIL [27], a well-validated code commonly used for aerofoil calculations.

A time step of Δt=0.005 was used, giving 200 steps per unit time, in the results given in the remainder of this section. This CFD model is used in the remainder of this section to represent the flow dynamics for simulation evaluation of the auto-tuning ILC designs. An open research question is how to undertake model-based design for this and other application area, see also §6.

(c). Flow with vortical disturbances

Simple-structure ILC for wind turbine blades has been investigated in [23,24], where [15] gives further background on the implementation of active load control for wind turbines. An oscillatory inflow was assumed and a range of flow conditions (amplitude and period of oscillation, AoA, flow with and without upstream disturbances) for both the simple quasi-steady flow model [24] and for the more advanced model with a dynamic wake as outlined above [23]. Also [23] considered the case with a simple first-order model of the actuator dynamics. Pure oscillatory flow produces a periodic fluctuation in the lift which can be successfully damped using simple-structure ILC, as in [23,24].

Introducing upstream vortices generates spikes in the lift as the vortices are convected past the aerofoil, interacting with the aerofoil and the other vortices in the flow in a nonlinear manner. This presents a more challenging scenario for the control scheme, but the fluctuations can still be successfully attenuated with an appropriately designed ILC law. Results will be presented here for the case with vortical disturbances introduced into the flow upstream of the aerofoil as this represents the most realistic case considered so far. The aim of the control is to damp out fluctuations in the lift generated by the oscillation to the flow and the effects of the vortices. The error at any step k is defined as

$$E^k=L^k-L_r,$$ 3.4

where L^k is the lift at step k and L_r is the target value for the lift.

The flow far upstream of the aerofoil is assumed to be periodic with velocity

$$V_{0x}=1+A\sin \left(\frac{2\pi t}{T}\right),$$ 3.5

where A is the amplitude of the oscillation and T its period (the aerofoil rotates one revolution in time T, corresponding physically to a rotation time of $HT/V_{\mathrm{\infty }}$). The initial vorticity on the aerofoil is preset to the vorticity expected for the target lift under steady conditions, i.e. the lift obtained using (3.5) with A=0. This is an assumption that the starting vortex has passed far enough downstream to assume that the mean bound vorticity on the aerofoil is constant and equal to that expected at the target lift. Hence the Wagner effect is not present at the start of the simulations, which is suitable as this work focuses on a wind turbine during continuous operation. Although in this section the lift corresponds to that for steady flow with a uniform free stream velocity, it would also be possible to use a different value, e.g. a higher value if the aim was to increase the load on the aerofoil. This was achieved using the quasi-steady flow model in [24].

The control scheme is a two-term ILC, incorporating both proportional or, P-type, and phase-lead ILC. The incoming flow operates over a cycle of N_c steps where N_c=T/Δt. Label the cycles as j, j=0,1,…, and the step within a cycle as k_c, k_c=0,1,…,N_c−1, such that k=jN_c+k_c. The control law has the form

$$\begin{array}{rl}{\hat{u}}_j^{k_{\mathrm{c}}}& =u_{j-1}^{k_{\mathrm{c}}}+{\mu }_0\mathrm{\Delta }t{E}_{j-1}^{k_{\mathrm{c}}+\mathrm{\Delta }},\end{array}$$ 3.6

$$\begin{array}{rl}{\overline{u}}^k& ={\mu }_1\mathrm{\Delta }t{E}^{k-1}\end{array}$$ 3.7

$$\begin{array}{rl}\text{and}{u}^k& =u_j^{k_{\mathrm{c}}}={\hat{u}}_j^{k_{\mathrm{c}}}+{\overline{u}}^k,\end{array}$$ 3.8

where the shift caused by Δ is allowed as the complete signal involved is already known. A factor of Δt is included in (3.6) and (3.7) to avoid scaling the gain if a different time step is used. The first step in this control scheme (3.6) has the form of an ILC phase-lead law of the form (2.7).

Consider a flow with an oscillatory free stream with A=0.1 and T=2.5 and two vortices introduced into the flow upstream of the aerofoil, one with strength Γ₁=1/10 placed at x_v1=(−30,0.25) and the other also with strength Γ₂=1/10 but at x_v2=(−20,−0.35) at the start of the simulation (t=0), and with the aerofoil at 0° AoA as shown in figure 1. With these starting values, vortex 1 will pass above the aerofoil and vortex 2 below it, and will generate a significant disturbance in the lift in addition to that from the oscillation in the free stream velocity. Figure 2 shows the error for this flow with no control for the time that the vortices are passing the aerofoil (approx. t=20 and t=30). In addition to the oscillation in lift arising from the free stream, large disturbances are generated by the vortices. Figure 2 also shows the effect of the control (3.6)–(3.8) with μ₀=1, μ₁=30 and Δ=0. Different values of the gains and the shift have been investigated [23]. The values given produced a suitable combination of attenuation of the fluctuations in the error and robustness to changes in the dynamics.

Figure 2.

ErrorE^k for case 1 without control (red). Error E^k for controller (3.6)–(3.8) with μ₀=1, μ₁=30 and Δ=0 (green). (Online version in colour.)

Calculations have been performed using the two-term ILC with μ₀=1, μ₁=30 and Δ=0 for a variety of parameter settings. The basic parameters for some representative results are listed in table 1. In this section, figure 3 is for case 4 and figure 4 is for case 6.

Table 1.

Parameters for selected cases using the two term ILC. M is the number of vortices placed upstream at the start of the simulation.

case	M	AoA	A	T	Lr	T0	T1
1	2	0°	0.1	2.5	0.379	10	50
2	2	7°	0.1	2.5	0.802	10	50
3	2	7°	0.1	2.5	0.802	10	50
4	2	0°	0.1	10	0.379	50	250
5	2	7°	0.1	10	0.802	50	250
6	12	0°	0.05	20	0.379	100	250

Figure 3.

ErrorE^k for case 4 without control (red). Error E^k for controller (3.6)–(3.8) with μ₀=1, μ₁=30 and Δ=0 (green) (error scaled by 100, magenta) and control input u^k (blue). (Online version in colour.)

Figure 4.

ErrorE^k for case 6 without control (red). Error E^k for controller (3.6)–(3.8) with μ₀=1, μ₁=30 and Δ=0 applied to case 6 (green). (Online version in colour.)

Two measures have been used to estimate the degree of damping, a 2-norm with

$${\mathcal{L}}_2={\left[\frac{1}{T_1-T_0}{\int }_{T_0}^{T_1}{(L(t)-L_r)}^2\hspace{0.17em}\mathrm{d}t\right]}^{1/2}$$ 3.9

and an $\mathrm{\infty }$-norm with

$${\mathcal{L}}_{\mathrm{\infty }}=\underset{k}{max}|L^k-L_r|.$$ 3.10

In general terms, ${\mathcal{L}}_2$ can be interpreted as measuring the fatigue load and ${\mathcal{L}}_{\mathrm{\infty }}$, the peak load on the blade. The integration for ${\mathcal{L}}_2$ was performed over the time the vortices pass the aerofoil, with an initial value (T₀) that allows the controller to settle (approx. 5T). Values of the norms for various cases listed in table 1 are given in [24] together with the ratio of the measures for controlled versus uncontrolled flow. For the case considered above (case 1), the two-term ILC produces a reduction of around two orders of magnitude in ${\mathcal{L}}_2$ and one in ${\mathcal{L}}_{\mathrm{\infty }}$. Proportionally, these were the smallest reductions in error for any of the cases listed in table 1.

The aerofoil shown in figure 1 is at 0° AoA. Pitch control, i.e. adjusting the AoA, can be used to maintain a near constant loading on the turbine as the mean flow rate varies. For 7° AoA, with the same flow parameters as above and two vortices starting in the same positions as above (case 2: table 1), the values of the ${\mathcal{L}}_2$ and ${\mathcal{L}}_{\mathrm{\infty }}$ ratios are less than a third of the magnitude of the 0° AoA case (case 1). In this case the target value was L_r=0.802, the lift for unperturbed flow with V _0x=1 and 7° AoA.

A more extreme case is obtained by changing the strength of vortex 2 to Γ₂=−3/10 (case 3). As this vortex contains (in magnitude) over a third of the circulation of that bound to the aerofoil in the reference condition, this case can be regarded as a severe test of the control scheme. There is now a larger deviation from the reference value of the lift, reflected in a larger value of ${\mathcal{L}}_{\mathrm{\infty }}$, but, again, given the extreme nature of this test, the control performs well. However, both ${\mathcal{L}}_2$ and ${\mathcal{L}}_{\mathrm{\infty }}$, the values with the control relative to those for the uncontrolled case, are larger than for the original two vortex case (i.e. that with Γ₂=1/10; case 2).

Both the chord length H and the reference velocity $V_{\mathrm{\infty }}$ will vary along a turbine blade, which implies an increase in the non-dimensional period of the oscillation T, moving towards the tip (as $t^{\ast }=(H/V_{\mathrm{\infty }})t$). All the calculations referred to so far have used T=2.5, which represents flow near the base of the blade. In practice, over much of the blade T would be much larger. Figure 3 shows the error for a run with two vortices, no control, 0° AoA and T=10 over the time that the vortices pass the aerofoil (case 4). The vortices now start much further upstream, at (−125,0.25) and (−119,0.25), with strengths Γ₁=0.1 and Γ₂=−0.1. Again there are relatively large fluctuations as the vortices pass. Figure 3 shows the error for the two-term ILC controlled case with μ₀=1 and μ₁=30. The disturbances are almost completely attenuated. Figure 3 also shows 100E^k to better illustrate the effect of the effect of the vortices when the control is active. There is a two- to three order of magnitude reduction in ${\mathcal{L}}_2$ and ${\mathcal{L}}_{\mathrm{\infty }}$. Figure 3 also shows the control input for this case. Essentially, it tracks the error but with a phase difference due to the nonlinear response of the wake to the shed vorticity. The aerofoil was then pitched to 7° AoA and the control works very well with a three order of magnitude reduction in the errors (case 5).

For the cases discussed above, only a few strong vortices have been used, generating large disturbances in the flow. Figure 4 shows the fluctuation in the lift for a run with 12 relatively weak vortices (Γ_j=±0.01) as the vortices pass the aerofoil when there is no actuation (u^k=0), a period of T=20 and a 5% amplitude in the free stream velocity oscillation (A=0.05). Here (case 6) the vortices generate relatively weak changes in the lift, superimposed on the oscillation from the unsteady nature of the free stream. Figure 4 also shows the error in the lift with the two-term ILC. Again there is a two order of magnitude reduction in ${\mathcal{L}}_2$ and ${\mathcal{L}}_{\mathrm{\infty }}$.

(d). Variable period of rotation

All the calculations so far in this application study have a fixed period of rotation. In practice, some variation could be expected. With the current control scheme, a variable rotation rate can be allowed for by updating the control assuming a fixed change in the angle of the blade between successive steps, and using a variable time step. In application, the angular position of a blade throughout a cycle would be known and it is assumed that the rate of rotation is given by

$$\frac{\mathrm{d}\theta (t)}{\mathrm{d}t}={\mathit{\Omega }}_0+{\mathit{\Omega }}_1\sin (\alpha t),$$ 3.11

where θ(t) is the current angular position of the blade, Ω₀=2π/T, Ω₁ is the amplitude of the fluctuation in the rotation rate and α its frequency. This gives

$$\theta (t)={\mathit{\Omega }}_{0}t+\frac{{\mathit{\Omega }}_1}{\alpha }[1-\cos (\alpha t)].$$ 3.12

We now assume a fixed value of Δθ between step k and k+1 so that θ_k+1 is known, and obtain the corresponding t_k+1 from (3.12). Next Δt=t_k+1−t_k is used to move the vortices and update the control (3.6)–(3.8).

A set of calculations matching those listed for cases 1 to 4 in table 1 was performed assuming a 10% variation in the rotation rate (Ω₁=1/10Ω₀) at a frequency half that of the base rate (α=1/2Ω₀). The same values were used for the control law gains as in the previous case. In general, there is an order of magnitude reduction in both measures when the control is applied. The reduction in the measures is similar to that found with a constant rotation rate in all cases.

As discussed previously in this section, the CFD model is used as a model of the dynamics to be controlled and the ILC designs are based on auto-tuning. Further discussion of this issue is given in the last section of this paper. Next, model-based ILC design is considered.

4. Norm-optimal iterative learning control

Optimal control is a long-standing topic in many areas of control systems design and application and this is also the case in ILC. Optimal control is based on designing the control law to minimize a cost function of interest. One option here is to assume that the cost function is the weighted sum of quadratic terms in the current trial error and the input on each trial. A variation on this approach is treated in this section for systems described by a discrete linear time-invariant model, but it is also possible to treat this problem in an abstract Hilbert space setting. The basis of this approach is to develop a general solution to the problem and then specialize as appropriate to a given application. One source of such an approach is given in [13,28] and termed norm-optimal ILC.

Consider the case of a plant that can be adequately modelled, at least for initial control related studies, by the following state-space model in the ILC setting

$$\left.\begin{array}{rl}{x}_k(t+1)& =A{x}_k(t)+B{u}_k(t)\\ \text{and}{y}_k(t)& =C{x}_k(t),x_k(0)=x_0,\end{array}\right\}$$ 4.1

where on trial k x_k(t) is the state vector, u_k(t) is the input vector and y_k(t) the output vector. Alternatively, the dynamics can be described by

$$y_k(t)=C{(qI-A)}^{-1}B{u}_k(t)+C{A}^t{x}_0,$$ 4.2

where I denotes the identity matrix with compatible dimensions, which is a particular case of (2.1).

In standard linear systems theory, linear quadratic optimal control [29] is a long standing design method that has seen many applications. For such a system with state vector x(t) and input u(t) the quadratic cost function is the sum of the terms x^T(t)Qx(t) and u^T(t)Ru(t), where Q and R are compatibly dimensioned symmetric definite weighting matrices, where this property is denoted from this point onwards by ≻0. By choice of Q and R relative penalties on the state vector entries and the control inputs can be imposed.

Returning to ILC design for systems described (4.1), the basic norm-optimal ILC cost function has the form

$$J_{k+1}(u_{k+1})=\frac{1}{2}\sum _{t=0}^{\alpha }\{e_{k+1}^{\mathrm{T}}(t)Q{e}_{k+1}(t)+{(u_{k+1}(t)-u_k(t))}^{\mathrm{T}}R(u_{k+1}(t)-u_k(t))\},$$ 4.3

where Q≻0 and R≻0. In this cost function, it is the difference between the inputs on two successive trials that is considered. The motivation for this is to prevent an excessive change in the control from one trial to the next.

Following [13], which derives the solution in causal form, Ratcliffe et al. [30] developed a computationally efficient implementation and the results of subsequent experimental testing on a gantry robot given and discussed. This implementation consists of the following tasks.

• — Matrix gain (Riccati) equation

  $$\begin{array}{rl}K(t)& =A^{\mathrm{T}}K(t+1)A+C^{\mathrm{T}}Q(t+1)C-[A^{\mathrm{T}}K(t+1)B\\ & \times {\{B^{\mathrm{T}}K(t+1)B+R)\}}^{-1}{B}^{\mathrm{T}}K(t+1)A],\end{array}$$ 4.4

  where K(t) is a matrix gain with terminal condition K(α)=0.
• — Predictive component equation

  $${\xi }_{k+1}(t)={\{I+K(t)B{R}^{-1}{B}^{\mathrm{T}}\}}^{-1}\{A^{\mathrm{T}}{\xi }_{k+1}(t+1)+C^{\mathrm{T}}Q{e}_k(t+1)\},$$ 4.5

  where ξ_k+1(α)=0.
• — Input update equation

  $$u_{k+1}(t)=u_k(t)-[{\{B^{\mathrm{T}}K(t)B+R\}}^{-1}{B}^{\mathrm{T}}K(t)A\{x_{k+1}(t)-x_k(t)\}]+R^{-1}{B}^{\mathrm{T}}{\xi }_{k+1}(t).$$ 4.6

The steps given above are a causal implementation of a non-causal ILC law. Also, a simplification occurs if the Riccati equation is assumed to be independent of p. It is also possible to formulate norm-optimal ILC for discrete dynamics using the alternative of a lifted model setting for design, e.g. [3], but the associated numerical costs are high, scaling as $\mathcal{O}(T^3)$ for the first two tasks above and $\mathcal{O}(T^2)$ for the input update. In §2, the benefits of ILC combined with feedback control was briefly discussed and this structure is automatically present in (4.6), i.e. state feedback in terms of the error between the state vector on the current and previous trial and feed-forward action from the term in ξ_k+1(t)). Next, the results from an actual implementation of this form of ILC is given, drawing on material in [31].

(a). Free-electron lasers

Free-electron lasers (FELs) use linear particle accelerators to increase the energy of the electrons by interaction with electromagnetic radio frequency (RF) fields; for further details of their operation and details of applications, see, e.g., [32]. These lasers operate in pulsed mode, e.g. every second there is a pulse for approximately 1 ms. This pulsed system has the following properties:

• — the characteristic disturbances and uncertainties only show small changes from pulse to pulse,

• — between pulses, several hundred milliseconds could be used to compute optimal parameters and driving signals for the next pulse,

• — the Field Programme Gate Array (FPGA) structure of the digital intra pulse controller allows arbitrary input signals at a frequency of 1 MHz,

• — appropriate models could be identified by standard methods from measurement data.

The second of these, in particular, makes this an application to which ILC can be applied.

Figure 5 shows the amplitude of the desired envelope of an RF field in this area, i.e. the reference trajectory for an ILC design. The reason why ILC can be applied is also illustrated in this figure, i.e. the time gap between one envelope ending and the start of the next. This is also the core difference between ILC and repetitive control [33] where in the latter case the reference signal is periodic and there is no time gap between the end of one evolution of this trajectory and the start of the next.

Figure 5.

One RF pulse in superconducting cavities. (Online version in colour.)

Once a desired operating point is set, the pulse trajectory remains unchanged for a large number of pulses. Therefore, repetitive disturbances can be suppressed by finding an optimal feed-forward control signal to minimize the deflection from the reference. Again consult, e.g., [32] for details of the disturbances that arise in this application and their causes.

The actuator system receives a precise RF signal of 1.3 GHz from the master oscillator. This low power sinusoidal signal can be changed by the vector modulator in amplitude and phase. The output signal of the vector modulator is amplified by a klystron, which is a RF amplifier and the amplified RF waves are transferred from the klystron to the cavities inside the cryomodules via a waveguide transmission and distribution system. For economic reasons, one high-power klystron supplies all 8–32 cavities of an RF station, thus RF fields cannot be influenced in each cavity individually and the system is therefore underactuated, i.e. fewer actuators than degrees of freedom.

A block diagram of a low-level radio frequency (LLRF) control system for this application is shown in figure 6, where the bottom part shows the digital FPGA controller. The LLRF control system has the task of keeping the pulsed RF fields in the superconducting cavities of the RF station at the reference value during the flat top phase of one RF pulse shown in figure 5.

Figure 6.

Structure of the RF system with master oscillator, vector modulator, klystron, cryomodule, measurement and calibration system and the FPGA-implemented control system. (Online version in colour.)

After measuring the actual RF field by pick-up antennas, the signals are down-converted to an intermediate frequency of 250 kHz. The real (I) and imaginary (Q) field components are digitalized in analogue–digital converters at a sampling frequency of 1 MHz. The signals shown in figure 6 are represented in terms of I and Q as follows:

• — the input signals u_I, u_Q are produced by actuator system and act directly on the vector modulator;

• — the output signals y_I, y_Q are the real and imaginary parts, respectively, of the sum of the RF field voltage vectors of eight cavities;

• — the reference signals r_I, r_Q are the real and imaginary parts, respectively, of the vector sum of the RF field’s voltage vectors given by look-up tables for the specified field gradient;

• — the feed-forward signals f_I, f_Q are among the control signals determined by open-loop control;

• — the control signals u_c,I, u_c,Q are the ILC control law output signals; and

• — the control error signals e_I, e_Q are the deviations in real and imaginary parts, respectively, of the output signals from the reference trajectories.

In control systems terms, the problem is one of MIMO ILC design, i.e. two inputs and two outputs and requires an adequate model of the underlying system dynamics. For some physical systems, e.g. mechanical or electrical systems, the laws of physics can be used but for others the route is to construct a model from measured input–output data. This last approach is used in this application area and is known as system identification. For the background on this method of constructing models of dynamics systems, see, e.g., [34]. One particular use in this application is subspace identification that directly supplies the state-space model matrices.

For the application of the ILC law, the error on trial k is e_k=[e_I e_Q]^T and the computational requirements for this application of norm-optimal ILC are also detailed in [31], where this aspect is heavily influenced by developments in implementation hardware and software. To prevent damage to system components in implementation, constraints were imposed on the maximum and minimum values of the input signals during the filling and the decay phase. See also the last section of this paper where constrained ILC is discussed.

The weighting matrices Q=100×I, R=I in the norm-optimal ILC cost function gave results suitable for implementation on the real system, where in this case the identity matrix I has dimensions 2×2. The state variables required for the state feedback component in the algorithm are obtained by simulation using the plant model. Including input and output disturbances in the simulation results for the flat top phase are shown in figure 7, where only the signals of the first input and output are given. The shape of the disturbances can be computed as the deviation from the smooth trajectory of the first trial. As the number of trials increases, the output signals approach the desired setpoint, or reference, trajectory. Rejection of the input and output disturbances can also be observed and as the input signal reaches the specified limit at the beginning of the phase, the output signal approaches the setpoint slowly in the first 100 μs.

Figure 7.

Simulation generated output and input signals after 20 trials of norm-optimal ILC. (Online version in colour.)

(b). Experimental results

The norm-optimal ILC design was successfully implemented on the real plant at the DESY test facility, resulting in the measured data shown in figures 8 and 9 for the cost function weighting matrices Q and R given above. The number of trials was initially set to 10.

Figure 8.

Measured output signals with the norm-optimal ILC applied. (Online version in colour.)

Figure 9.

Measured input signals required with the norm-optimal ILC. (Online version in colour.)

Figure 8 shows an increasing and decreasing trend of the first and the second output signal during the flat top phase. Increasing the number of trials beyond the initial 10, both the output signals approach their desired setpoints. After 10 trials, the output signals show only small deviations from their setpoints. Only the signals during the flat top phase are controlled and the input signals of the filling and the decay phases are kept constant as illustrated in figure 9.

To emphasize the improvements in the field regulation gained by application of norm-optimal ILC, [31] gives a comparison with the decentralized proportional controller previously used for field regulation and a MIMO feedback controller. One major issue with the proportional controller arises from the results. The first is the high overshoot at the beginning of the flattop, resulting from the step given in the feed-forward tables. Applying the norm-optimal ILC significantly reduces the control error to a level which is required to give an appropriate beam energy gain.

The influence on the electron beam must also be considered. This aspect is detailed in [31]. Overall, this application demonstrates that norm-optimal ILC has transferred in a very short, in comparative terms, time span from basic theory to actual application with significant benefits over alternatives. Next, an application that demonstrates that ILC can also be applied outside physical systems. This application to robot-assisted stroke rehabilitation exploits auto-tuning and linear and nonlinear model-based ILC.

5. Iterative learning control in health care

(a). Planar tasks with phase-lead iterative learning control

Stroke is a leading cause of disability worldwide and is usually caused when a blood clot blocks a vessel in the brain, stopping the blood reaching the regions downstream. As a result, some of the connecting nerve cells die and the person commonly suffers partial paralysis on one side of the body, termed hemiplegia. Stroke is an age-related disease and, as the number of people aged 60 and above is predicted to increase from year to year, incidence is likely to rise. Prevalence is also likely to rise due to better survival rates and long-term care. Associated with these figures, stroke burden is projected to increase from around 38 million disability-adjusted life years (DALYs) lost globally in 1990 to 61 million DALYs in 2020 [35].

The consequences of age-related conditions, such as stroke, thus have a growing impact on the health and economic prosperity of countries. Neurorehabilitation, i.e. therapy to assist recovery of functions lost as a result of a neurological disorder, across the world is fragmented and the current prognosis for upper limb recovery following stroke remains poor; for 30–60% of stroke patients the hemiplegic arm remains without function [36]. Hence there is an obvious need to improve the efficiency and effectiveness of treatment under the constraint that resources for health care workers and rehabilitation costs are limited. Technologies have the potential to provide intensive practice of a task, variety and feedback, which have been shown by conventional therapy and motor learning theory to be important.

Brain cells killed after a stroke cannot re-grow, but the brain has some spare capacity and hence new connections can be made. The brain is continually and rapidly changing as new skills are learned, new connections are formed and redundant ones disappear. A person who re-learns skills after a stroke goes through the same process as someone learning to play tennis, requiring sensory feedback during repeated practice of a task. Unfortunately, they can hardly move and, therefore, do not receive feedback on their performance.

Research on how to help a stroke patient re-learn a task of daily living involves regaining motor control. The application of conventional therapy and motor learning theory provides evidence that intensity of practice of a task and feedback are important, e.g. [37]. In turn, this is motivating the development of novel treatments, such as those based on robotic therapy, which provide the opportunity for repetitive movement practice.

Clinical evidence exists to support the therapeutic use of functional electrical stimulation (FES) to improve motor control [38]. This form of stimulation makes muscles work by causing electrical impulses to travel along nerves in much the same way as electrical impulses from the brain, and if carefully controlled a useful movement can be made. Theoretical results from neurophysiology [39] and motor learning research support clinical research with the conclusion that the therapeutic benefit of stimulation is maximized when applied coincidently with a patient’s own voluntary intention to move [40].

Repeating the same finite duration task over and over again is a core part of ILC and the research question is: can ILC be used to effect in stroke rehabilitation? In particular, can ILC be used to control the level of FES applied and is it possible in clinical trials to detect the critical property for rehabilitation—if the patient is improving with successive attempts, or trials, does this mean that if the patients voluntary effort is increasing with each successive attempt then the level of FES applied decreases? A related issue is that FES is higher frequency and can induce muscle fatigue and if this arises during a session with a patient then activity must cease. This, in turn, can be demotivating for patients.

This section describes the application of ILC in stroke rehabilitation, including clinical trials that constitute the first major stage towards eventual transfer into practice. In contrast with the other approaches employed to control FES, ILC exploits the repeating nature of the patient’s tasks in order to improve performance by learning from past experience. By updating the control input using data collected over previous attempts at the task, ILC is able to respond to physiological changes in the system, such as spasticity and the presence of the patient’s voluntary effort, which would otherwise erode performance. Use of ILC can also closely regulate the amount of stimulation supplied, ensuring that minimum assistance is provided, thereby promoting the patient’s maximum voluntary contribution to the task completion. As the treatment progresses this control action encourages patients to exert increasing voluntary effort with each trial, leading to a corresponding decrease in the level of FES applied.

As a starting point, attention was initially restricted to planar tasks, such as reaching out over a desktop to a cup. Figure 10 shows a stroke patient using the system designed for this purpose. The patient in this figure is seated with her impaired arm supported by the robot, and elliptical trajectories, the reference trajectory in ILC, are projected onto a target above the hand. Also FES is applied to her triceps, using the surface electrodes, in order to assist tracking of a point that moves along the reference trajectory. At the end of the task, the arm is returned to the starting position in preparation for the next trial. During the reset time, plus a rest time to assist in countering muscle fatigue and allowing transients to decay, an ILC law is used to calculate the stimulation to be applied on the next trial. The stimulation applied to the triceps muscle produces a torque about the elbow and the control problem is equivalent to controlling the angle ϑ_f in this figure. The shoulder strapping is to prevent forward movement by the patient’s trunk during the trials, which would conflict with the desired objective of reaching out with the arm.

Figure 10.

A patient using the robot-based stroke rehabilitation system: (1) is the strapping used to prevent voluntary trunk movement in reaching out with the impaired arm, (2) denotes the reference trajectory for ILC and (3) denotes the patches used to apply the assistive stimulation, ϑ_f denotes the angle to be controlled. (Online version in colour.)

Figure 11 shows a plan view of the patient’s movement in the planar case where the analogy with the pick and place operation for an industrial robot is clear, i.e. repeated executions/trials of a pre-specified trajectory under control action, where after each trial the arm is reset to the starting location. During the arm resetting time, plus a rest time to prevent muscle fatigue and allow transients to decay, at the end of trial k, the ILC law uses a biomechanical model of the arm and muscle system, along with the previous tracking error, to produce the control signal, i.e. the FES, for application on the next trial. The stimulation applied to the triceps muscle produces a torque about the elbow and the control problem is equivalent to controlling the angle ϑ_f in figure 10 and is the trial output.

Figure 11.

Plan view of the patient’s movement in the planar case.

Figure 12 shows a schematic or block diagram of the ILC scheme developed for the planar case, for which the various blocks are discussed next with the technical details in [10,11]. This scheme led on to clinical trials, also described below, where the critical feature discussed above was detected.

Figure 12.

Block diagram of the control system for the planar task.

The signal u, a processed form of the FES signal is applied to the relevant muscle, the triceps, for this planar task. This produces a torque about the elbow that is used to generate the path followed by the arm attached to the robot, which can be related to the angle ϑ_f as a function of time. The block ‘Arm and Muscle System’ represents the dynamic coupling between u and ϑ_f where the relationship between the torque generated and ϑ_f can be modelled by a nonlinear second order differential equation with coefficients that vary from person to person. The relationship between the input u and the torque about the elbow requires the modelling of the response of human muscle to electrical stimulation.

This latter modelling problem has seen considerable research effort and one source for the control oriented literature are the references in [41]. In this work, the model used is a static nonlinearity, representing the isometric recruitment curve, in cascade with a linear second order critically damped dynamics. The isometric recruitment curve is defined as the static gain relation between the stimulus activation level and output force when the muscle is held at full length and the structure of the model is given in (5.3) where it is also used in rehabilitation for three-dimensional tasks.

The dynamics of the ‘arm and muscle system’ are nonlinear and one obvious approach to control systems design in such cases is to first linearize the dynamics, which is the role of the ‘linearizing controller’ block. Many methods exist for this task, e.g. assume small angles or design a linearizing control law, but in this initial work a ‘physically based’ approach was used involving, e.g., numerical inversion of the isometric recruitment curve. In many ILC applications and indeed advanced control designs, a preliminary feedback control loop is employed and the ‘PID controller block’ denotes the use of a proportional plus derivative controller. One of the main reasons for using this loop is to compensate for unmodelled effects, such as involuntary hand movement, which will become a more pressing issue when the hand is not firmly strapped to the robot.

To design the ILC law, a reference trajectory, ${\vartheta }_f^{\ast }(t)$ in figure 12, is required. In this work, the reference signal was determined by an assessment of each patient by a health care professional. In the initial research, the ILC controller was phase-lead of the form (2.7), where the previous trial error is ${\vartheta }_f^{\ast }(t)$ minus ϑ_f(t) measured during completion of this trial. An open research question is to provide tools to assist the choice of the reference trajectory in this application for ILC. The scheme of figure 12 was tested on unimpaired volunteers as a necessary step in obtaining ethical approval for patient trials. Results from these tests are given in [10] with supporting discussion.

Having obtained the required ethical approval, five patients participated in a clinical trial. As one set of results from this trial, figure 13 shows typical changes in the angle of the shoulder and elbow during over the duration of a reference trajectory. The solid line shows the ideal movements that would be required to complete the trajectory successfully; the dotted-dashed line represents unassisted movement, and the dash-dotted line shows movement assisted by FES. Figure 13c shows the FES pulse-width that is applied using ILC in order to produce these assisted movements.

Figure 13.

Tracking performance plots.

In [42,43], a clinical assessment of the results obtained with the five patients is given. This includes evidence of the presence of the required property that as a patient improved with successive trials there was an increase in voluntary effort and a decrease in applied stimulation. The results in [43] deal with the patient’s interaction with robot-assisted stroke rehabilitation where other evidence confirms that patients can fail to engage for many reasons and without a sufficient level of engagement the technology advances are of very limited/no use.

In this case, auto-tuning ILC gives acceptable performance, as supported by the clinical trials. Model-based ILC has also been applied to this case, giving results that do not give significantly improved performance over the phase-lead design. More complicated movements, such as lifting and reaching with the affected arm, with more than one muscle involved, will require more powerful ILC action and also the numerical inversion of the nonlinearities is very unlikely to be successful.

Before proceeding, a discussion of what makes ILC particularly suitable in an application area is given. In the vast majority of applications, there is more than one possible control system and a choice must be made. The reasoning behind such a choice is often not totally transparent, but for this application health care-specific reasons can be given. In particular, repeated executions of the same finite duration task is a well-recognized rehabilitation method and there is the requirement to control the level of additional stimulation applied. This is exactly the premise of ILC, i.e. design the current trial input using knowledge of the previous trial output and input. Also there is the requirement to encourage/reward voluntary effort. These requirements cannot be achieved with feedback control although such action may be applied as a pre-stabilizer in the control loop.

(b). Three-dimensional tasks with model-based iterative learning control

In the previous research, the patient’s forearm is constrained to lie in a horizontal plane and the next stage, using the clinical trial reports for the planar task as motivation, is to consider a wider range of more functional movements, which more closely resemble the tasks necessary for daily living and are aligned with the activity-based measures used by health professionals. This section describes the development of the previous model of the arm to remove the planar forearm constraint, permitting unconstrained movement, and applies model-based ILC. First, the development of the robotic systems used is described in this section, taken, in the main, from [44] and the cited references in this paper.

When providing assistive FES during upper limb reaching movements, stimulation must be applied using a controlled environment in order to ensure safety and comfort across a broad spectrum of patient ability. Therefore, a commercially available mechanical exoskeleton support structure has been used to provide anti-gravitational assistance to the patient during treatment. The Armeo Spring was selected as a purely passive gravity-compensation device that provides the necessary support to overcome gravity via two springs incorporated in the mechanism, thereby allowing patients to focus practice on the impaired muscles rather than those acting against gravity. This device is supplied with its own diverse range of virtual reality tasks, but these are not suitable for control law evaluation in this application and a custom task display system was developed. The mechanical exoskeleton employed is shown in figure 14, where the caption summarizes how each part contributes to the overall system.

Figure 14.

Three-dimensional ILC system components: (1) gravity-compensating robotic device, (2) surface electrodes on triceps and anterior deltoid, (3) FES module, (4) real-time processor and interface module, (5) PC, (6) monitor displaying task and (7) operator monitor. (Online version in colour.)

Figure 15a shows the biomechanical system comprising the human arm and exoskeleton mechanical support system; figure 15b shows the kinematic structure of the exoskeleton support, where the joint variables Θ=[θ₁,θ₂,θ₃,θ₄,θ₅]^T correspond to the measured joint angles. Note that the parallelogram structure of the upperarm section results in θ₃=−θ₃′.

Figure 15.

Kinematic system relationships, (a) combined system, (b) ArmeoSpring support and (c) human arm. (Online version in colour.)

The human arm is shown in figure 15c, and as it is strapped to the support, its position can also be described using the same variable set. However, to simplify the FES control scheme, it is desirable that those axes about which electrical stimulation produces a moment correspond with joint variables. Spasticity in stroke patients typically produces a resistance to elbow extension during reaching tasks, associated with overactivity of the biceps, and a loss in activity of the triceps and anterior deltoid (this difficulty in performing full elbow extension has been verified in trials with stroke patients attempting to complete reaching tasks). These muscles have, therefore, been selected for stimulation. It is assumed that application of stimulation to the triceps produces a moment about an axis orthogonal to both the forearm and upper arm and that FES to the anterior deltoid produced a moment about an axis which is fixed with respect to the shoulder. These anthropomorphically motivated variables are given by Φ=[ϕ₁,ϕ₂,ϕ₃,ϕ₄,ϕ₅]^T and are shown in figure 15c. It is assumed that the anterior deltoid axis is fixed with respect to the shoulder and determined by two constant rotation transformations that are introduced into the human arm kinematic chain.

Within the necessary joint ranges, there exists a unique bijective transformation between these coordinate sets, given by Φ=k(Θ). Through application of Lagrangian analysis, a dynamic model of the Armeo Spring system is given by

$${\mathit{\text{B}}}_a(\mathit{\Theta })\ddot{\mathit{\Theta }}+{\mathit{\text{C}}}_a(\mathit{\Theta },\dot{\mathit{\Theta }})\dot{\mathit{\Theta }}+{\mathit{\text{F}}}_a(\mathit{\Theta },\dot{\mathit{\Theta }})+{\mathit{\text{G}}}_a(\mathit{\Theta })+{\mathit{\text{K}}}_a(\mathit{\Theta })=-{\mathit{\text{J}}}_a^{\mathrm{T}}(\mathit{\Theta }){\mathit{\text{h}}}_a,$$ 5.1

where h_a is a vector of externally applied force and torque, B_a(⋅) and ${\mathit{\text{ C}}}_a(\mathit{\Theta },\dot{\mathit{\Theta }})$ are 5×5 inertial and Corelis matrices. In addition, J(⋅) is the system Jacobian matrix, and F_a(⋅) and G_a(⋅) are friction and gravitational vectors, respectively. The vector K_a(⋅) comprises the moments produced through gravity compensation given by each spring, which are functions of θ₃ and θ₅, respectively, so that K_a(⋅) takes the form [0,0,k₃(θ₃),0,k₅(θ₅)]^T.

Similarly, the dynamic model of the human arm can be represented by

$${\mathit{\text{B}}}_h(\mathit{\Phi })\ddot{\mathit{\Phi }}+{\mathit{\text{C}}}_h(\mathit{\Phi },\dot{\mathit{\Phi }})\dot{\mathit{\Phi }}+{\mathit{\text{F}}}_h(\mathit{\Phi },\dot{\mathit{\Phi }})+{\mathit{\text{G}}}_h(\mathit{\Theta })={\mathit{\tau }}_h.$$ 5.2

In this model, τ_h comprises the moments produced through application of FES to muscles, which are of the form $\mathit{\text{ g}}(\mathit{\text{ u}},\mathit{\Phi },\dot{\mathit{\Phi }})={[0,g_2({\varphi }_2,{\dot{\varphi }}_2,u_2),0,0,g_5({\varphi }_5,{\dot{\varphi }}_5,u_5)]}^{\mathrm{T}}$. Also u₂(t) and u₅(t) are the electrical stimulation applied to the triceps and anterior deltoid muscles, respectively. From Le et al. [41], each is taken, as in the planar case above, to be of the form

$$g_i({\varphi }_i,{\dot{\varphi }}_i,u(t))=h_i(u_i,t)\times F_{m,i}({\varphi }_i,{\dot{\varphi }}_i),i=\{2,5\}.$$ 5.3

The term, h_i(u_i,t) is a Hammerstein structure incorporating a static nonlinearity, h_IRC,i(u_i), representing the isometric recruitment curve, cascaded with linear activation dynamics, h_LAD,i(t), and $F_{m,i}({\varphi }_i,{\dot{\varphi }}_i)$ models the multiplicative effect of the joint angle and joint angular velocity on the active torque developed by the muscle. Let the state-space system corresponding to linear activation dynamics, h_LAD,i(t), have states x_i, and state-transition and output matrices M_t(⋅) and M_o(⋅), respectively.

The rigid connection between structures means the combined model assumes the form

$$\mathit{\text{B}}(\mathit{\Phi })\ddot{\mathit{\Phi }}+\mathit{\text{C}}(\mathit{\Phi },\dot{\mathit{\Phi }})\dot{\mathit{\Phi }}+\mathit{\text{F}}(\mathit{\Phi },\dot{\mathit{\Phi }})+\mathit{\text{G}}(\mathit{\Phi })+\mathit{\text{K}}(\mathit{\Phi })=\mathit{\text{g}}(\mathit{\text{u}},\mathit{\Phi },\dot{\mathit{\Phi }})-{\mathit{\text{J}}}^{\mathrm{T}}(\mathit{\Phi }){\mathit{\text{h}}}_a.$$ 5.4

Introducing the state vector x such that $\mathit{\text{ x}}={[\mathit{\Phi },\dot{\mathit{\Phi }},{\mathit{\text{ x}}}_2,{\mathit{\text{ x}}}_5]}^{\mathrm{T}}$ gives the combined model

$$\begin{array}{rl}\dot{\mathit{\text{x}}}(t)& =\left[\begin{array}{c}[0I00]\hspace{0.17em}\mathit{\text{x}}(t)\\ {\mathit{\text{B}}}^{-1}(\mathit{\text{x}}(t))(\mathit{\text{C}}(\mathit{\text{x}}(t))\dot{\mathit{\Phi }}-\mathit{\text{F}}(\mathit{\text{x}}(t))-\mathit{\text{G}}(\mathit{\text{x}}(t))-\mathit{\text{K}}(\mathit{\Phi })+{\mathit{\text{M}}}_o(\mathit{\text{x}}(t),\mathit{\text{u}}(t)))\\ {\mathit{\text{M}}}_t(\mathit{\text{x}}(t),\mathit{\text{u}}(t))\end{array}\right]\\ & =f(\mathit{\text{x}}(t),\mathit{\text{u}}(t))\\ \mathit{\Phi }(t)& =[I\hspace{0.17em}0\hspace{0.17em}0\hspace{0.17em}0]\hspace{0.17em}\mathit{\text{x}}(t)=h(\mathit{\text{x}}(t)).\end{array}$$

This model is used by the FES control system to produce an input signal that results in accurate tracking of a reference trajectory. As assistive torque is applied about the ϕ₂ and ϕ₅ axes only, the system is underactuated. When applied during the treatment of patients, the ILC law assists tracking about ϕ₂ and ϕ₅ alone, and, in response to clinical guidelines, it is assumed that the patient has sufficient control over the remaining axes to adequately perform the task.

The goal of the FES controller is to assist a patient’s tracking of a functional task in three-dimensional space. Unlike the two-dimensional case, which used a ‘real-world’ task consisting of tracking a moving light spot in the horizontal plane, here a virtual task is displayed to the patient. Details of how this was achieved and the roles of the features marked as (6) and (7) in fig. 14 are given in [44].

The tasks presented to patients during treatment consist of repeated tracking movements for their affected arm, with a rest period in between during which their arm is returned to the starting position. As in the planar case, a simple structure ILC law, such as phase-lead, could be applied. However, the dynamics in this case are much more complex and hence it is to be expected that the simple structure law will struggle to deliver the required level of performance. Hence the Newton method-based ILC was used, starting from the system model (5.4). Figure 16 shows the full structure of control scheme, where, as in the previous study, an ILC law and a feedback controller are combined in a parallel layout. In this framework, voluntary effort of the patient can be treated as a trial-invariant disturbance and hence compensation for its effects can be applied. Many possible approaches to nonlinear-based ILC design exist, such as Newton method-based ILC [14].

Figure 16.

Block diagram representation of the ILC control scheme for the three-dimensional rehabilitation system.

Suppose that the controlled dynamics are described by a discrete-time state-space model of the form

$$\left.\begin{array}{rl}{x}_k(t+1)& =f(x_k(t),v_k(t))\\ \text{and}{q}_{u,k}(t)& =h(x_k(t)),\end{array}\right\}$$ 5.5

where t∈[0,1,2,…,α−1], x_k(t) is the state vector, and α=T/T_s+1 with T_s the sampling frequency. Introducing the vectors

$$\left.\begin{array}{rl}{v}_k& ={[v_k{(0)}^{\mathrm{T}},v_k{(1)}^{\mathrm{T}},\dots ,v_k{(\alpha -1)}^{\mathrm{T}}]}^{\mathrm{T}}\\ \text{and}{\mathit{\Phi }}_{u,k}& ={[{\mathit{\Phi }}_{u,k}{(0)}^{\mathrm{T}},{\mathit{\Phi }}_{u,k}{(1)}^{\mathrm{T}},\dots ,{\mathit{\Phi }}_{u,k}{(\alpha -1)}^{\mathrm{T}}]}^{\mathrm{T}}\end{array}\right\}$$ 5.6

and the reference vector

$${\mathit{\Phi }}_u^{\ast }={[{\mathit{\Phi }}_u^{\ast }{(0)}^{\mathrm{T}},{\mathit{\Phi }}_u^{\ast }{(1)}^{\mathrm{T}},\dots ,{\mathit{\Phi }}_u^{\ast }{(\alpha -1)}^{\mathrm{T}}]}^{\mathrm{T}},$$ 5.7

the Newton method-based ILC update takes the form

$$v_{k+1}=v_k+g^{\mathrm{\prime }}{(v_k)}^{-1}{e}_k,$$ 5.8

where $e_k={\mathit{\Phi }}_u^{\ast }-{\mathit{\Phi }}_{u,k}$ is the tracking error. The term g^′(v_k) is equivalent to the system linearization around v_k, with the system ${\tilde{q}}_u=g^{\mathrm{\prime }}(v_k)\tilde{v}$ corresponding to the linear time-varying, denoted LTV, system

$$\left.\begin{array}{rl}\tilde{x}(t+1)& =A(t)\tilde{x}(t)+B(t)\tilde{v}(t)\\ \text{and}{\tilde{q}}_u(t)& =C(t)\tilde{x}(t)+D(t)\tilde{v}(t)\hspace{0.17em}t=0,1,\dots ,\alpha -1\end{array}\right\}$$ 5.9

with

$$\begin{array}{rl}A(t)& ={\left(\frac{\mathrm{\partial }f}{\mathrm{\partial }x}\right)}_{v_k(t),x_k(t)},B(t)={\left(\frac{\mathrm{\partial }f}{\mathrm{\partial }{v}_k}\right)}_{v_k(t),x_k(t)}\\ C(t)& ={\left(\frac{\mathrm{\partial }h}{\mathrm{\partial }x}\right)}_{v_k(t),x_k(t)},D(t)={\left(\frac{\mathrm{\partial }h}{\mathrm{\partial }{v}_k}\right)}_{v_k(t),x_k(t)}.\end{array}$$

The term g^′(v_k)⁻¹ in (5.8) is computationally expensive and may be singular or contain excessive amplitudes and high frequencies. To overcome this difficulty, introduce

$$e_k=g^{\mathrm{\prime }}(v_k)\mathrm{\Delta }{v}_{k+1}$$ 5.10

and then Δv_k+1=v_k+1−v_k equals the input that forces the LTV system (5.9) to track the error e_k. This is itself an ILC problem and can be solved in between experimental trials using any ILC algorithm that converges globally. In this case norm optimal ILC is used, with the input and output on trial j denoted by e_k,j and Δv_k+1,j, respectively. On trial j+1, the trade-off between minimizing the tracking error, e_k−e_k,j, and the change in control input, Δv_k+1,j+1−Δv_k+1,j, is represented by a cost function of the form (4.3). The ILC computation is stopped after 100 trials or after the error reaches a preset threshold. The input obtained, Δv_k+1,j, is then used to approximate Δv_k+1 in (5.8) to generate the control input for the next trial.

Following the same route as the planar task considered previously, the design of this section was also used in a small-scale clinical trial. In this case, a wider variety of reference trajectories is possible, e.g. short, medium and long, again determined by an assessment of the patient by a qualified health professional. This aspect is detailed in [44]

Figure 17 shows representative input, output and error signals recorded on trial 8 for one patient and confirm that a high level of tracking performance is achieved with an input signal that is not excessive. The theoretical feature of rapid monotonic convergence to zero tracking error is degraded due to inaccuracies in the human arm model which deteriorate performance, motivating development of more accurate identification procedures, and future use of on-line and recursive techniques. Such identifications routines, however, must be suitable for application within the restrictive conditions of clinical trials, where there is limited set-up time, little opportunity to repeat measurements, and satisfactory results for a wide range of patients and changing physiological conditions are required.

Figure 17.

Tracking performance of the Newton method-based ILC design. (Online version in colour.)

The eventual aim in the development of new methods for stroke rehabilitation is to have hardware/software that the patient can use in their home, based on the established fact that intensity of practice is a critical feature of progress in rehabilitation following a stroke. This feature would be further enhanced if a health professional could remotely monitor the progress of more than one patient outside the hospital environment. Clearly, the three-dimensional robotic system described in this section is not suitable for this purpose but there has been research undertaken using a Kinect motion capture device that could with sufficient onward development produce equipment that can be taken home by the patient. Progress to-date is given in [11] and supporting clinical trial data in [45].

6. Conclusion and open research problems

The three application areas covered in this paper demonstrate that ILC, both tuning based, i.e. no model of the dynamics is used (the wind turbine), or model based (the free-electron laser and stroke rehabilitation) can be applied with success to physical examples. As the planar stroke rehabilitation demonstrates, there will be cases where tuning and model-based designs have similar performance and in such cases the tuning design should be employed due to less implementation costs. The free-electron laser and three-dimensional stroke rehabilitation are cases where model-based design is required.

Much further research is required before a full evaluation of ILC for stroke rehabilitation can be made, starting with removing the presentation to the patient of a prescribed path to follow from start to finish. In reality, the patient should decide and progress towards this objective [11], starting from point-to-point ILC [46]. Application of FES to muscles is at higher frequency than the dynamics and this can induce muscle fatigue. A clinical trial will specify a total time any patient can spend in one session and if fatigue occurs then a recovery time must occur and this will in most cases mean the end of the session. Preliminary results on introducing compensation to counter fatigue into the control system can again be found in [11], where evaluation is based on a model constructed from patient data. Also there has been productive research reported on ILC for FES-based rehabilitation for other forms of disability [47], opening up the possibility of wider use in health care.

The vast majority of the successful implementations reported in the literature have been designed on the assumption that the control signals required can be generated and applied without damaging the equipment involved. In some cases, such as systems designed to execute precisely defined motions, the mechanical design can result in kinematic constraints, such as maximum velocities and accelerations, which cannot be exceeded. A practical approach in such cases would be off-line trajectory planning subject to these constraints but this may often not be desirable or possible.

An alternative is to consider design with constraints imposed on the allowable control signals and this area has been the subject of some previous research, such as [48] where a constrained convex optimization setting and an interior-point-type method was used to solve the constrained ILC problem for linear systems with saturation and rate constraints. Also, an ILC design problem with general convex input constraints has been considered, e.g. [49], where it was established that the constrained ILC problem can be formulated in a setting that allows the development of a method for constraint handling together with solution algorithms that have well-defined convergence properties.

In this latter work, it is assumed that the input is constrained to be in a set Ω, taken to be a closed convex set in some Hilbert space H. In practice, the set Ω will often be of simple structure. Recently, as one example, Chu et al. [50] have reported experimental verification on a rack feeder system and there is much productive research to be undertaken in this general area, including the links with model predictive control and in the context of networked systems. Another area is stochastic dynamics for which the first theoretical results include [51]. Nonlinear ILC is another relatively open area in terms of designs that have seen experimental verification despite the wealth of papers in the literature that deal with theoretical issues such as proofs of convergence. New designs from this theory-based research will be available for any application where ILC is a candidate.

Authors' contributions

The writing of this paper was shared equally among the two authors.

Competing interests

We declare we have no competing interests.

Funding

We received no funding for this study.