Adaptive Optimal Control of Hybrid Electric Vehicle Power Battery via Policy Learning

Abstract

An online policy learning algorithm is used to solve the optimal control problem of the power battery state of charge (SOC) observer for the first time. The design of adaptive neural network (NN) optimal control is studied for the nonlinear power battery system based on a second-order (RC) equivalent circuit model. First, the unknown uncertainties of the system are approximated by NN, and a time-varying gain nonlinear state observer is designed to address the problem that the resistance capacitance voltage and SOC of the battery cannot be measured. Then, to realize the optimal control, a policy learning-based online algorithm is designed, where only the critic NN is required and the actor NN widely used in most design of the optimal control methods is removed. Finally, the effectiveness of the optimal control theory is verified by simulation.

1. Introduction

Nowadays, electric vehicles are developing at a high speed [1]. The power battery provides the required high power for vehicle start stop, acceleration and deceleration, and other instabilities and greatly improves the service life of fuel cells by controlling the charging and discharging power of the power battery [1, 2]. As an important energy storage part of fuel-cell hybrid vehicles, it has far-reaching significance for the research of power cells. The state of charge (SOC) in the battery is one of the important parameters of the battery management system (BMS), but SOC cannot be directly measured by the on-board sensors. Therefore, SOC estimation is a very important problem in the theory and application. Moreover, the power battery is a highly complex nonlinear system in its working state, which greatly increases the difficulty of estimation [3].

In order to meet the requirements of accurate, fast, and real-time estimation of power battery SOC under different conditions, scholars have carried out a lot of advanced achievements. In [4], the authors proposed an observer-based unilateral Lipschitz conditional nonlinear system control method for a class of nonlinear systems with time-varying parameter uncertainties and norm bounded disturbances. For the state-space equation of the equivalent circuit model, a power battery SOC estimation method based on nonlinear observer is proposed in [5]. The authors in [6] introduced the second-order resistance capacitance (RC) model of the battery pack. Under the unilateral Lipschitz condition, a nonlinear observer based on the H∞ method is designed, but whether the optimal performance of the observer can be guaranteed remains to be verified. For the problem of optimal control design of the observers, the authors proposed an adaptive neural network backstepping recursive optimal control method for nonlinear strict feedback systems with state constraints [7]. The neural network (NN) state identification is used to approximate the unknown nonlinear dynamics, and under the actor-critic structure, the virtual and actual optimal controllers are constructed through the backstepping recursive control algorithm. Because actor-critic structure-based adaptive laws are generated on the basis of the square of Behrman residual error obtained by the gradient descent method, these methods are too complex and difficult to implement. In this regard, the authors in [8] proposed an optimal control method based on reinforcement learning (RL) for a class of nonlinear strict feedback systems with unknown dynamic functions. This method eliminates the persistent excitation assumption necessary for most RL-based adaptive optimal control. On this basis, the adaptive NN output-feedback optimal control problem for a class of strict feedback nonlinear systems with unknown internal dynamics, input saturation, and state constraints is studied in [9]. In [10, 11], the authors proposed the novel optimal control algorithm based on advanced AI techniques, which further promotes the development of the optimal control theory.

Inspired by the abovementioned research results, a nonlinear observer with time-varying gain is designed in this paper. Based on the unilateral Lipschitz condition, the nonlinear dynamic problem contained in the system output is solved. The internal unknown dynamic function is approximated by NN to estimate the SOC and the resistance capacitance voltage of the dynamic battery in the power system. Then, based on estimated system states, we develop a policy learning-based optimal control and the estimated weight error is convergence to zero. Finally, the simulation results show the effectiveness of the proposed method.

The innovations of this paper are summarized as follows:

1. The optimal control method based on critic NN is used to solve the optimal control problem of the power battery SOC observer for the first time.

2. Only one critic NN is used to ensure the convergence of the NN weights; thus, the actor NN widely used in most design of optimal control methods [12–14] is removed.

3. Unlike the existing optimal control with known state, the battery state in this paper is unknown. This leads to a complex optimal control problem.

2. System Modeling

In this paper, we consider the second-order RC equivalent circuit model as shown in Figure 1 [15], where U_oc is the open-circuit voltage (OCV) respected to SOC, I_T represents the current, U_T denotes the terminal voltage, R₀ is the ohmic resistance, R₁ and R₂ are the electrochemical polarization resistance and the concentration polarization resistance, respectively, and C₁ and C₂ are the capacitances. U₁ and U₂ show the voltage of the electrochemical capacitor C₁ and concentration polarization capacitor C₂, respectively.

Figure 1.

The schematic diagram of the second-order RC model.

Then, based on the Kirchhoff voltage laws, the state equation of Figure 1 can be given as

$$\begin{array}{c}\left\{\begin{array}{c}{\dot{U}}_1=-\frac{1}{R_1{C}_1}{U}_1+\frac{1}{C_1}{I}_T,\\ {\dot{U}}_2=-\frac{1}{R_2{C}_2}{U}_2+\frac{1}{C_2}{I}_T,\\ S\dot{O}C=-\frac{1}{Q_n}{I}_T\text{,}\end{array}\right.\end{array}$$ (1)

where Q_n is the nominal capacity of the battery.

Then, its output equation can be defined as

$$\begin{array}{c}{U}_T=U_{\mathrm{o}\mathrm{c}}\left(\mathrm{S}\mathrm{O}\mathrm{C}\right)-R_0{I}_T-U_1-U_2,\end{array}$$ (2)

where 0 ≤ SOC ≤ 1, and U_oc(SOC) is the nonlinear monotone increasing function.

Based on (1) and (2), we can obtain state space equation as follows:

$$\begin{array}{c}\left\{\begin{array}{c}\dot{x}=Ax+Bu,x\left(0\right)=x_0,\\ y=g\left(x\right)+Cx-R_{0}u,\end{array}\right.\end{array}$$ (3)

where $x={\left[\begin{array}{ccc}{U}_1& U_2& \mathrm{S}\mathrm{O}\mathrm{C}\end{array}\right]}^T\in {\mathbb{ℝ}}^3$, y=U_T ∈ ℝ, u=I_T ∈ ℝ, g(x)=U_oc(SOC) ∈ ℝ, and x₀ is the initial state.

$$\begin{array}{c}A=\left[\begin{array}{ccc}-\frac{1}{R_1{C}_1}& 0& 0\\ 0& -\frac{1}{R_2{C}_2}& 0\\ 0& 0& 0\end{array}\right]\in {\mathbb{R}}^{3\times 3},B={\left[\begin{array}{ccc}\frac{1}{C_1}& \frac{1}{C_2}& -\frac{1}{Q_n}\end{array}\right]}^T\in {\mathbb{R}}^3,\\ C=\left[\begin{array}{ccc}-1& -1& 0\end{array}\right]\in {\mathbb{R}}^{1\times 3}\text{.}\end{array}$$ (4)

As the power battery is a highly complex nonlinear system in its working state, there are many unknown uncertainties such as ambient temperature, battery self-discharge, battery life, and cycle interval. Therefore, the state space expression (3) can be expressed as follows:

$$\begin{array}{c}\left\{\begin{array}{c}\dot{x}=\mathrm{A}\mathrm{x}+\mathrm{B}\mathrm{u}+d\left(x\right),x\left(0\right)=x_0,\\ y=g\left(x\right)+Cx-R_{0}u,\end{array}\right.\end{array}$$ (5)

where d(x) represents nonlinear characteristics.

Assumption 1 . —

In this paper, we assume that (A, B) is stabilizable and (A, C) is detectable. The nonlinear term d(x) is continuous and bounded.

Control objective: for the second-order RC equivalent model of power battery, based on an adaptive observer a policy learning algorithm-based optimal controller is designed to guarantee all signals of the closed-loop system uniformly ultimately bounded (UUB).

According to the second-order RC model of the power battery, we can derive its state space (3) or (5); then, we should design the control law u for the derived state space equation. Thus, we will use the NN observer and the policy learning algorithm to design the control law u.

3. Optimal Control of Power Battery

3.1. Observer Design via NN

This section will design an observer to estimate the battery voltage and SOC. Thus, we assume

$$\begin{array}{c}d\left(x\right)=W_1^T\sigma \left(x\right)+\epsilon \left(x\right)\text{,}\end{array}$$ (6)

where W₁ ∈ ℝ^N is the ideal NN weights, σ(x) ∈ ℝⁿ⟶ℝ^N is the activation function, and ε(x) ∈ ℝ denotes the NN error.

In this paper, the function d(x) is unknown continuous; hence, the estimated function is

$$\begin{array}{c}\widehat{d}\left(x\right)={\widehat{W}}_1^T\sigma \left(x\right)\text{,}\end{array}$$ (7)

where ${\widehat{W}}_1$ is the estimation of W₁.

Then, based on (5) and (7), the observer can be designed as

$$\begin{array}{c}\left\{\begin{array}{c}\dot{\widehat{x}}=A\widehat{x}+Bu+{\widehat{W}}_1^T\sigma \left(\widehat{x}\right)+L{\left[\frac{\partial g}{\partial x}\right]}_{x=\widehat{x}}^T\left(y-\widehat{y}\right),\\ \widehat{y}=C\widehat{x}+g\left(\widehat{x}\right)-R_{0}u\text{,}\end{array}\right.\end{array}$$ (8)

where $\widehat{x}$ is the estimation of x, L=P⁻¹ ∈ ℝ^3×3 is the observation matrix, P is the positive matrix, and $\widehat{y}$ is the estimation of y.

We define the observation error

$$\begin{array}{c}\tilde{x}=x-\widehat{x}\text{.}\end{array}$$ (9)

Then, from (5) and (8), we can obtain the observation error dynamic equation as

$$\begin{array}{c}\dot{\tilde{x}}=\left[A-L{\left(\frac{\partial g}{\partial x}\right)}^{T}C\right]\tilde{x}-L{\left(\frac{\partial g}{\partial x}\right)}_{x=\widehat{x}}^T\tilde{g}+W_1^T\left(\sigma \left(x\right)-\sigma \left(\widehat{x}\right)\right)+{\tilde{W}}_1\sigma \left(\widehat{x}\right)+\epsilon \text{,}\end{array}$$ (10)

where $\tilde{g}=g\left(x\right)-g\left(\widehat{x}\right)=\partial g/\partial x\left|x=\xi \left(x-\widehat{x}\right)\right.,{\tilde{W}}_1={\widehat{W}}_1-W_1$ is the NN weight error.

Lemma 2 . —

For system (5), if it adopts designed observer (8), the NN weights ${\widehat{W}}_1$ satisfy the adaptive law

$$\begin{array}{c}{\dot{\widehat{W}}}_1=-\sigma \left(\widehat{x}\right){\tilde{x}}^{T}P\text{.}\end{array}$$ (11)

This can guarantee that errors $\tilde{x}$ and ${\tilde{W}}_1$ are UUB.

Proof —

Consider a Lyapunov function

$$\begin{array}{c}{V}_1=\frac{1}{2}{\tilde{x}}^{T}P\tilde{x}+\frac{1}{2}tr\left({\tilde{W}}_1^T{\tilde{W}}_1\right)\text{.}\end{array}$$ (12)

From [15], we have ${\left[\partial g/\partial x\right]}_{x=\widehat{x}}^T=\left[\mathrm{0,0},{\dot{U}}_{oc}\left(S\widehat{O}C\right)\right]$ with ${\alpha }_{\min }\le {\dot{U}}_{oc}\left(S\widehat{O}C\right)\le {\alpha }_{\max }$, where α_min and α_max are the minimum and maximum values of the change rate of the ${\dot{U}}_{\mathrm{o}\mathrm{c}}$ function, respectively. Then, the derivation of (12) gives

$$\begin{array}{c}{\dot{V}}_1\le \left.\frac{1}{2}{\dot{\tilde{x}}}^T\left[PA+A^{T}P-RMC-C^T{\left(RM\right)}^T\right]-2Q\right]\tilde{x}\\ +{\tilde{x}}^{T}P{W}_1^T\left(\sigma \left(x\right)-\sigma \left(\widehat{x}\right)\right)+{\tilde{x}}^{T}P·{\tilde{W}}_1\sigma \left(\widehat{x}\right)+{\tilde{x}}^{T}P\epsilon +\frac{1}{2}tr\left({\dot{\tilde{W}}}_1^T{\tilde{W}}_1+{\tilde{W}}_1^T{\dot{\tilde{W}}}_1\right)\text{,}\end{array}$$ (13)

where $M={\left[\begin{array}{ccc}{m}_1,& m_2,& m_3\end{array}\right]}^T\in {\mathbb{ℝ}}^3$.

According to the unilateral Lipschitz condition [9], the following inequalities can be obtained:

$$\begin{array}{c}{\tilde{x}}^{T}P\epsilon \le \frac{1}{2}{‖\tilde{x}‖}^2+\frac{1}{2}{‖P‖}^2{\displaystyle \sum }_{i=1}^3{\epsilon }_i^{\ast 2}\text{,}\end{array}$$ (14)

$$\begin{array}{c}{\tilde{x}}^{T}P{W}_1^{\ast T}\left(\sigma \left(x\right)-\sigma \left(\widehat{x}\right)\right)\le {‖\tilde{x}‖}^2+{‖P‖}^2{‖W_1‖}^2\text{.}\end{array}$$ (15)

Taking (14) and (15) into (13), and considering tr(ab^T)=tr(b^Ta)=b^Ta, we have

$$\begin{array}{c}{\dot{V}}_1\le \frac{1}{2}{\tilde{x}}^T\left[PA+A^{T}P-RMC-C^T{\left(RM\right)}^T-2Q\right]\tilde{x}+{‖\tilde{x}‖}^2+{‖P‖}^2{‖W_1‖}^2+\frac{1}{2}{‖\tilde{x}‖}^2\\ +\frac{1}{2}{‖P‖}^2{\displaystyle \sum }_{i=1}^3{\epsilon }_i^{\ast 2}+tr\left({\tilde{W}}_1^T\sigma \left(\widehat{x}\right){\tilde{x}}^{T}P+{\tilde{W}}_1^T{\dot{\tilde{W}}}_1\right)\text{.}\end{array}$$ (16)

Based on [8], let PA+A^TP − RMC − C^T(RM)^T − 2Q=−Ψ, where $Q=\left[\begin{array}{ccc}0& 0& 0\\ 0& 0& 0\\ 0& 0& {\alpha }_{\min }^2\end{array}\right]$; thus, (16) can be further written as

$$\begin{array}{c}{\dot{V}}_1\le -a_0{‖\tilde{x}‖}^2+\frac{1}{2}{‖P‖}^2{‖{\tilde{W}}_1‖}^2+D_0\text{,}\end{array}$$ (17)

where a₀=λ_min(ψ) − 3/2 and D₀=‖P‖²‖W₁‖²+1/2‖P‖²∑_i=1³ε_i².

If $\widehat{d}\left(x\right)⟶d\left(x\right)$, then the term $1/2{‖P‖}^2{‖{\tilde{W}}_1‖}^2+D_0$ in (17) can converge to zero. Moreover, by selecting the appropriate matrix ψ, λ_min(ψ) can be relatively large. According to (17), the observation error can converge to a small neighborhood containing the origin.

3.2. Optimal Control Design Based on the Observer

3.2.1. Online Policy Learning Algorithm

In this section, based on critic NN, we construct the policy learning law. Thus, system (8) can be rewritten as

$$\begin{array}{c}\dot{\widehat{x}}=F\left(\widehat{x}\right)+\mathrm{B}\mathrm{u}\text{,}\end{array}$$ (18)

where $F\left(x\right)=Ax+{\widehat{W}}_1^T\sigma \left(x\right)+L{\left[\partial g/\partial x\right]}_{x=\widehat{x}}^T\left(y-\widehat{y}\right)$, and L is the Lyapunov function.

To realize the optimal control, we first define the cost function as\

$$\begin{array}{c}V\left(\widehat{x},u\right)={\int }_0^{\infty }r\left(\widehat{x},u\right)\mathrm{d}s\text{.}\end{array}$$ (19)

With $r\left(\widehat{x},u\right)={\widehat{x}}^T{Q}_s\widehat{x}+u^T{R}_{s}u$ being the utility function, Q_s ∈ ℝ^3×3 and R_s ∈ ℝ are the weight matrices of proper dimension.

We define the Hamiltonian function of the optimal control problem and the optimal cost function as

$$\begin{array}{c}H\left(\widehat{x},u,\nabla V\left(\widehat{x}\right)\right)=r\left(\widehat{x},u\right)+{\left(\nabla V\left(\widehat{x}\right)\right)}^T\left(F\left(\widehat{x}\right)+\mathrm{B}\mathrm{u}\right)\text{.}\end{array}$$ (20)

$$\begin{array}{c}{V}^{\ast }\left(\widehat{x}\right)=\underset{u}{\min }{\int }_0^{\infty }r\left(\widehat{x},u\right)\mathrm{d}s\text{.}\end{array}$$ (21)

The optimal cost function $V^{\ast }\left(\widehat{x}\right)$ is the solution of the following HJB equation:

$$\begin{array}{c}0=\underset{u}{\min }H\left(\widehat{x},u,\nabla V^{\ast }\left(\widehat{x}\right)\right)\text{.}\end{array}$$ (22)

With ∇V^∗(x)=∂V^∗(x)/∂x, we can obtain this optimal control action as

$$\begin{array}{c}{u}^{\ast }=-\frac{1}{2}{R}_s^{-1}{B}^T\nabla V^{\ast }\left(\widehat{x}\right),\end{array}$$ (23)

and the HIB equation in terms of ∇V^∗(x) as

$$\begin{array}{c}0={\widehat{x}}^T{Q}_s\widehat{x}+{\left(\nabla V^{\ast }\left(\widehat{x}\right)\right)}^{T}F\left(\widehat{x}\right)-\frac{1}{4}{\left(\nabla V^{\ast }\left(\widehat{x}\right)\right)}^{T}B{R}_s^{-1}{B}^T\nabla V^{\ast }\left(\widehat{x}\right),\end{array}$$ (24)

with V^∗(0)=0.

To realize the policy learning, some iteration procedure can be given as follows:

• (1)Select the small positive number τ. Set i=0 and V⁽⁰⁾=0, and then give an initial admissible control u⁽⁰⁾.
• (2)
  Using the control u⁽ⁱ⁾, resolve

  $$\begin{array}{c}0=r\left(\widehat{x},u\right)+{\left({\nabla }^{i+1}V\left(\widehat{x}\right)\right)}^T\left(F\left(\widehat{x}\right)+B{u}^i\right),\end{array}$$ (25)

• with V⁽ⁱ⁺¹⁾(0)=0.

• (3)
  Update the control action using

  $$\begin{array}{c}{u}^{\left(i+1\right)}=\frac{1}{2}{R}_s^{-1}{B}^T\nabla V^{\left(i+1\right)}\left(\widehat{x}\right)\text{.}\end{array}$$ (26)
• (4)If $‖V^{\left(i+1\right)}\left(\widehat{x}\right)-V^{\left(i\right)}\left(\widehat{x}\right)‖\le \tau$, stop, then apply the optimal control; else, let i=i+1 and go back to (2).

This algorithm will be convergence to the optimal control and optimal cost function when i⟶∞. The convergence of this algorithm can be referred to [16, 17].

3.2.2. NN Implementation

We assume the cost function $V\left(\widehat{x}\right)$ is continuously differentiable. Then, we can use the NN reconstruct the $V\left(\widehat{x}\right)$ as

$$\begin{array}{c}V\left(\widehat{x}\right)=W_2^T{\sigma }_c\left(\widehat{x}\right)+{\epsilon }_c\left(\widehat{x}\right)\text{,}\end{array}$$ (27)

where W₂ ∈ ℝ^N is the ideal NN weights, σ_c(x) ∈ ℝⁿ is the activation function, and ${\epsilon }_c\left(\widehat{x}\right)\in \mathbb{ℝ}$ denotes the NN error. Then,

$$\begin{array}{c}\nabla V\left(\widehat{x}\right)={\left(\nabla {\sigma }_c\left(\widehat{x}\right)\right)}^T{W}_2+\nabla {\epsilon }_c\left(\widehat{x}\right)\text{,}\end{array}$$ (28)

where $\nabla \sigma \left(\widehat{x}\right)=\partial {\sigma }_c\left(\widehat{x}\right)/\partial \widehat{x}$ and $\nabla {\epsilon }_c\left(\widehat{x}\right)=\partial {\epsilon }_c\left(\widehat{x}\right)/\partial \widehat{x}$ are the gradient of the activation function and NN error, respectively. According to (28), we can obtain the Lyapunov function as

$$\begin{array}{c}0=r\left(\widehat{x},u\right)+\left(W_2^T\nabla {\sigma }_c\left(\widehat{x}\right)+{\left(\nabla {\epsilon }_c\left(\widehat{x}\right)\right)}^T\right)\dot{\widehat{x}}\text{.}\end{array}$$ (29)

Assumption 3 . —

(see [12–14, 18]). If the NN weight W₂, the NN error ε_c, the gradient ∇σ_c, and derivative ∇ε_c are bounded, then we can have ε_c⟶0 and ∇ε_c⟶0.

We define the estimation of (27) as

$$\begin{array}{c}\widehat{V}\left(\widehat{x}\right)={\widehat{W}}_2^T{\sigma }_c\left(\widehat{x}\right)\text{.}\end{array}$$ (30)

Then, we have

$$\begin{array}{c}\nabla \widehat{V}\left(\widehat{x}\right)={\left(\nabla {\sigma }_c\left(\widehat{x}\right)\right)}^T{\widehat{W}}_c.\end{array}$$ (31)

with $\nabla \widehat{V}\left(\widehat{x}\right)=\partial \widehat{V}\left(\widehat{x}\right)/\partial \widehat{x}$. Thus, the estimated Hamiltonian function can be given as

$$\begin{array}{c}H\left(\widehat{x},u,{\widehat{W}}_2\right)=r\left(\widehat{x},u\right)+{\widehat{W}}_2^T\nabla {\sigma }_c\left(\dot{x}\right)\dot{\widehat{x}}=e_c\text{.}\end{array}$$ (32)

To minimize error (32), we construct the objective function J=(1/2)e_c^Te_c, and then the descent algorithm can be designed as

$$\begin{array}{c}{\dot{\widehat{W}}}_2=-{\alpha }_1\left[\frac{\partial J}{\partial W}\right]=-{\alpha }_1\left[\frac{\partial e_c}{\partial W}\right],\end{array}$$ (33)

with α₁ > 0 being the learning gain of the NN.

Based on (29), the Hamiltonian function can be rewritten as

$$\begin{array}{c}H\left(\widehat{x},u,W_2\right)=r\left(\widehat{x},u\right)+W_2^T\nabla {\sigma }_c\left(\widehat{x}\right)\dot{\widehat{x}}=e_h\text{,}\end{array}$$ (34)

where $e_h=-{\left(\nabla {\epsilon }_c\left(\widehat{x}\right)\right)}^T\dot{\widehat{x}}$ is the residual error.

Define $\varphi =\nabla {\sigma }_c\left(\widehat{x}\right)\dot{\widehat{x}}$, if there is a positive constant ϕ_M such that ‖ϕ‖ ≤ ϕ_M, and denote the weight estimation error ${\tilde{W}}_2=W_2-{\widehat{W}}_2$, and then based on (32) and (34), we have $e_h-e_c={\tilde{W}}_2^T\varphi$; thus, we have the dynamic of the weight estimation error as

$$\begin{array}{c}{\dot{\tilde{W}}}_2=-{\dot{\widehat{W}}}_2={\alpha }_1\left(e_h-{\tilde{W}}_2^T\varphi \right)\varphi \text{.}\end{array}$$ (35)

The persistent excitation (PE) condition is required to tune the NN, guaranteeing ‖ϕ‖ ≥ ϕ_m with ϕ_m being the positive constant. To this end, a probing noise is inserted into the system to meet the PE.

In this case, the optimal control action can be given as

$$\begin{array}{c}{u}^{\ast }=-\frac{1}{2}{R}_s^{-1}{B}^T\left({\left(\nabla \sigma \left(\widehat{x}\right)\right)}^T{W}_2+\nabla {\epsilon }_c\left(\widehat{x}\right)\right)\text{,}\end{array}$$ (36)

and its estimation is

$$\begin{array}{c}\widehat{u}=-\frac{1}{2}{R}_s^{-1}{B}^T{\left(\nabla \sigma \left(\widehat{x}\right)\right)}^T{\widehat{W}}_2\text{.}\end{array}$$ (37)

Equation (37) shows that using the trained critic network, the control policy can be derived directly; thus, the actor NN is removed in this paper. The structural diagram of the algorithm is given in Figure 2.

Figure 2.

The structural diagram of the algorithm.

Lemma 4 . —

For system (18), the adaptive law for the NN is provided by (33), and then the weight estimation error of NN is UUB.

Proof —

Choose the Lyapunov function as $K\left(t\right)=\left(1/{\alpha }_1\right)\mathrm{t}\mathrm{r}\left({\tilde{W}}_2^T{\tilde{W}}_2\right)$. The time derivative of the Lyapunov function along the trajectory of error dynamics (35) is

$$\begin{array}{c}\dot{K}\left(t\right)=\frac{2}{{\alpha }_1}\mathrm{t}\mathrm{r}\left({\tilde{W}}_2^T{\dot{\tilde{W}}}_2\right)=\frac{2}{{\alpha }_1}tr\left({\tilde{W}}_2^T{\alpha }_1\left(e_h-{\tilde{W}}_2^T\varphi \right)\varphi \right)\text{.}\end{array}$$ (38)

After doing some basic manipulations, we have

$$\begin{array}{c}\dot{K}\left(t\right)\le -\left(2-{\alpha }_1\right){‖{\tilde{W}}_2^T\varphi ‖}^2+\frac{1}{{\alpha }_2}{e}_h^2\text{.}\end{array}$$ (39)

Considering the Cauchy–Schwarz inequality and noticing the assumption ‖ϕ‖ ≤ ϕ_M, we can conclude that $\dot{K}\left(t\right)<0$ as long as 1 < α₁ < 2 and

$$\begin{array}{c}‖{\tilde{W}}_2‖>\sqrt{\frac{e_h^2}{{\alpha }_1\left(2-{\alpha }_1\right){\varphi }_M^2}}\text{.}\end{array}$$ (40)

According to the Lyapunov theory, we obtain that the dynamics of the weight estimation error is UUB. The norm of the weight estimation error is bounded as well.

It is noted that the estimated weight ${\widehat{W}}_2$ is optimal to W₂, and this indicates that the solution $\widehat{V}$ can be extracted from the estimated vector ${\widehat{W}}_2$ given in (30). Thus, one can derive the actual control $\widehat{u}=-1/2R_s^{-1}{B}^T{\left(\nabla \sigma \left(\widehat{x}\right)\right)}^T{\widehat{W}}_2$ for system (18) based on ${\widehat{W}}_2$. As a consequence of Lemma 4, we can conclude that $\widehat{u}$ will converge to the optimal control u^∗, i.e., $‖\widehat{u}-u^{\ast }‖⟶0$ such that the control system stability can be retained based on Lemma 4.

Remark 5 . —

In this paper, an observer is designed using NN to online estimate the unknown state (SOC); then, based on the estimated state, we develop a policy learning algorithm to online resolve the optimal control of the battery. The proposed methods are different from our previous work, such as [18], where the system states are assumed to be known, and this limits the application of the optimal control algorithm in practice.

Remark 6 . —

To realize the output-feedback control using the policy learning, the PE condition is required in this paper. As shown in [14, 17], to guarantee the PE condition, an alternative way is to insert an exploration noise into the system for the first two seconds [17].

4. Simulation Results

For the second-order RC equivalent model of power battery, the effectiveness of the optimal control theory in this paper is verified by simulation based on Matlab. The values of resistance, capacitance, and battery capacity in the second-order RC equivalent model (5) are as follows: R₀=10.822mΩ, R₁=3.103mΩ, R₂=2.611mΩ, C₁=8.4379kF, C₂=91.401kF, and Q_n=45A · h.

Let M=I, then we can obtain P and L as

$$\begin{array}{c}P=\left[\begin{array}{ccc}14.1250& 0& 19.6371\\ 0& 128.7451& 178.9860\\ 19.6371& 178.9860& 0\end{array}\right]\text{,}\\ L=\left[\begin{array}{ccc}0.0638& -0.007& 0.005\\ -0.007& 0.0008& 0.005\\ 0.005& 0.005& -0.0036\end{array}\right]\text{.}\end{array}$$ (41)

Given the design parameters in learning law (33) as α₁=0.1 and the initial values as x₁(0)=0.1, x₂(0)=0.2, x₃(0)=1, ${\widehat{x}}_1\left(0\right)=0.01,{\widehat{x}}_2\left(0\right)=0,{\widehat{x}}_3\left(0\right)=0.99$, and ${\widehat{W}}_2=\left[0.39090.58121.05760.10.21\right]$, we design the regressor of the critic NN as σ(x)=[x₁², x₁x₂, x₁x₃, x₂², x₂x₃, x₃²]^T.

We aim at obtaining an optimal control policy that can stabilize system (18). For system (18), we need to find a feedback control policy that minimizes the cost function.

$$\begin{array}{c}V\left(\widehat{x},u\right)={\int }_0^{\infty }\left({\widehat{x}}^T{Q}_s\widehat{x}+u^T{R}_{s}u\right)\mathrm{d}s,\end{array}$$ (42)

with Q_s=I and R_s=2I. We adopt the online policy iteration algorithm to tackle the optimal control problem, where a critic network is constructed to approximate the cost function. During the implementation process of the policy learning algorithm, we introduce the noise to meet the PE condition. The exponentially decreasing probing noise and sinusoidal signals with different frequencies are used. They are introduced into the control input and thus affect the system states.

The evolution of the state trajectory is depicted in Figure 3, and this can be used to further design the optimal controller for the proposed system. Figure 4 gives the good estimated weights, where we have that the convergence of the weight has occurred after 1000 s. Then, the probing signal is turned off. This good convergence of the NN weights can ensure the stability of the controlled system, which can be found in Figure 5. Figure 5 is the controller system trajectory with the designed optimal controller. We see that the state converge to zero after the probing noise is turned off. Figure 6 shows the cost of the system under which is smooth, and this indicates that the designed controller is effective. The control action is given in Figure 7, which is bounded. This further shows Lemma 4 is true.

Figure 3.

Estimated system states.

Figure 4.

NN weights.

Figure 5.

System trajectory.

Figure 6.

Cost function.

Figure 7.

Control action u.

To show the improved performance of the proposed single critic NN-based ADP for solving the derived optimal control problem, a critic-actor NN-based online learning method [19] is also used for comparison. Moreover, in this comparison, we add the robustness verification of the proposed method. To this end, we set the nonlinear term d(x)=0.5 sin(x₁). The profiles of the critic NN and actor NN weights can be found in Figure 8 and the corresponding control performances are given in Figure 9. Compared with Figures 9(a) and 9(b), it is clear that the proposed single critic NN-based can achieve faster transient state convergence even if there is a nonlinear term.

Figure 8.

NN weights.

Figure 9.

System state x (a) using the proposed method and (b) the method proposed in [19, 20].

Generally, the modeling accuracy and control structure will influence the control performance of the closed-loop control systems. In this paper, the main factors affecting the control performance are the modeling uncertainties of the system and the convergence performance of critic NN weights. Moreover, better convergence of critic NN weights, i.e., faster convergence speed can help to achieve better control performance. In this respect, different choices of critic NN parameters and structure will affect the convergence of critic NN weights and the control performance. Hence, proper selection of NN parameters and structure, such as the initial value of weights, learning gain, and regressor structure, is helpful to further improve the control response.

5. Conclusion

For the second-order RC equivalent nonlinear system of power battery, the unknown uncertainty of the system is approximated by NN, and a time-varying gain nonlinear state observer is designed to solve the problem that the resistance capacitance voltage and charge (SOC) of the battery cannot be measured. Then, to realize the optimal control, a policy learning-based online algorithm is designed, where only the critic NN is required, and the actor NN widely used in most design of the optimal control methods is removed. Finally, the effectiveness of the optimal control theory is verified by simulation.

Data Availability

The data used to support the findings of this study are available upon request from the corresponding author.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Authors' Contributions

Qinglin Zhu and Jun Zhao conceptualized the study; Huanli Sun and Ziliang Zhao were responsible for methodology; Ziliang Zhao and Yixin Liu performed formal analysis; Qinglin Zhu wrote the original draft; Qinglin Zhu and Yixin Liu reviewed and edited the manuscript; and Huangli Sun and Ziliang Zhao were responsible for funding acquisition. All authors have read and agreed to the published version of the manuscript.