Reinforcement Learning Based Optimal Tracking Control Under Unmeasurable Disturbances With Application to HVAC Systems

Abstract

This paper presents the design of an optimal controller for solving tracking problems subject to unmeasurable disturbances and unknown system dynamics using reinforcement learning (RL). Many existing RL control methods take disturbance into account by directly measuring it and manipulating it for exploration during the learning process, thereby preventing any disturbance induced bias in the control estimates. However, in most practical scenarios, disturbance is neither measurable nor manipulable. The main contribution of this article is the introduction of a combination of a bias compensation mechanism and the integral action in the Q-learning framework to remove the need to measure or manipulate the disturbance, while preventing disturbance induced bias in the optimal control estimates. A bias compensated Q-learning scheme is presented that learns the disturbance induced bias terms separately from the optimal control parameters and ensures the convergence of the control parameters to the optimal solution even in the presence of unmeasurable disturbances. Both state feedback and output feedback algorithms are developed based on policy iteration (PI) and value iteration (VI) that guarantee the convergence of the tracking error to zero. The feasibility of the design is validated on a practical optimal control application of a heating, ventilating, and air conditioning (HVAC) zone controller.

I. Introduction

REINFORCEMENT learning (RL) is a class of artificial intelligence algorithms, which has gained significant attention in the control community for its potential use in designing intelligent controllers that learn the optimal actions without needing prior knowledge of the system model. This model-free design is desirable because system models are generally hard to obtain and modeling uncertainties can significantly affect the closed-loop stability and the optimality of the controller. one of the important control applications of RL is in solving the optimal tracking problem, which involves designing a controller that forces the system to follow a prescribed reference signal. Tracking control finds application in diverse areas such as robotics, autonomous vehicles, aerospace, building controls, and multiagent systems [1]–[5]. The presence of external disturbances, however, makes the tracking problem more challenging.

One of the pioneering developments in RL-based optimal tracking control involves the idea of state augmentation [6]–[9]. Disturbance rejection capabilities have recently been incorporated in RL by adapting ideas from game theory [10]. The presence of parametric and nonparametric uncertainties [11] and extensions to nonlinear systems [12] have also been considered. Disturbance rejection controllers based on the H_∞ design have been presented to solve the optimal tracking problem [13]–[15]. Extensions of these learning approaches employing policy iteration methods have also been presented recently [16]. In all the works discussed so far, the disturbance is treated as a decision maker with the disturbance signal being a measurable signal whose L₂ norm is bounded. However, in a practical setting a disturbance is not an intelligent decision maker, and it cannot be measured or influenced by the controller. Consequently, ignoring the disturbance in the learning equation leads to control estimates that may become biased because the learning equation does not hold true as a result of the missing disturbance terms. An analysis of the bias terms arising in the closed-form value function as a result of non-disturbance sources has been carried out in [17], [18]. Different from the state augmentation approach, output regulation based on the internal model principle [19] has also recently been considered in the learning control literature [20], [21]. In these approaches the reference and the disturbance are assumed to be generated by an internal model.

To address the above difficulties, instead of directly measuring the disturbance, we introduce bias terms in the Q-function. The Q-learning algorithm is then designed to learn this modified Q-function, which includes the estimates of the bias incurred by the disturbance. Explicitly including the estimates of the bias terms prevents the crucial control parameters from being affected. To achieve disturbance rejection while tracking, we augment the dynamics with the integral of the tracking error. The Q-learning scheme learns the control parameters for this augmented system while also countering the disturbance induced bias to prevent the estimates from drifting away during the learning phase. As will be shown later in the article, although the integral action helps in rejecting the disturbance to ensure asymptotic tracking, it alone is insufficient to prevent the biasing effect of the disturbance. To relax the exploration condition, we employ off-policy learning in a way similar to [22], in which the behavioral policy that is used to generate system data does not follow the intermediate policies being learned.

The contributions of this present work are summarized as follows. In recent work [13], where the disturbance was assumed to be measurable during the data collection and learning phase, an off-policy RL technique was proposed to solve the optimal tracking problem subject to an L₂ disturbance. This approach also required a discounted cost function. As acknowledged in [13], discounted cost functions may not guarantee closed-loop stability and a system dependent bound on the discounting factor needs to be satisfied (see [23] for discrete-time problems). This work solves the tracking problem in the presence of external disturbances using an integral augmentation approach that does not require a discounted cost function. More importantly, compared to the robust off-policy techniques [13], [15], [24], [25], this work does not require the measurement of the disturbance, which does not need to be an L₂ signal. In another recent work [26], the internal model principle is employed to generate a disturbance, therefore, the disturbance is implicitly measurable. A separate identification process is needed in that approach to solve a set of regulator equations. This work attempts to address the above mentioned difficulties in solving the optimal tracking problem in the presence of unmeasurable disturbances.

The proposed scheme is demonstrated through zone control in a heating, ventilating, and air conditioning (HVAC) application. In this case study, the zone is a room in a commercial office building and the goal is to maintain the zone temperature at the desired set point. The zone temperature is affected by the weather, the airflow rate, the supply air temperature, the thermal mass of the building materials, and the internally generated thermal loads (from equipment, people, etc.). In this scenario the air-handling unit (AHU), which produces the supply air at a given temperature and airflow rate, is the actuator that manipulates the zone temperature. The optimal operation of the system is based on the balance between maintaining the zone at a specified temperature and the cost of the energy required to meet that need.

The remainder of this article is organized as follows: Section II provides a description of the problem. Section III presents the main theoretical development of this paper, where we introduce a bias compensation mechanism and integral action to create a modified Q-function. Then, the design of a Q-learning scheme is presented that learns this Q-function to solve the optimal tracking problem. In particular, we present four Q-learning algorithms based on policy iteration (PI) and value iteration (VI) using state feedback and output feedback. Section IV includes the application of the proposed scheme to the design of an HVAC zone controller. Some concluding remarks are made in Section V.

II. Problem Description

Consider a discrete-time linear time-invariant system in the state space form

$$\begin{array}{l}{x}_{k+1}=A{x}_k+B{u}_k+D{d}_k,\\ y_k=C{x}_k,\end{array}$$ (1)

where $x_k\in {ℝ}^n$ is the system state, $u_k\in {ℝ}^m$ is the control input, $d_k\in {ℝ}^q$ is the external disturbance, and $y_k\in {ℝ}^p$ is the system output. We define the tracking error as

$$e_k=y_k-r_k,$$

where $r_k\in {ℝ}^p$ is the reference trajectory. We assume that m ≥ p. The control problem is to find the optimal control sequence $u_k^{\ast }$ with feedback gain K* that guarantees asymptotic output tracking, i.e., lim_k→∞ e_k = 0, while minimizing a quadratic cost function of the form

$$J=\sum _{i=0}^{\infty }\left(e_i^{\text{T}}{Q}_e{e}_i+{\tilde{u}}_i^{\text{T}}R{\tilde{u}}_i\right),$$ (2)

where (A, (Q_e)^1/2C) is observable, and Q_e ≥ 0 and R > 0 are the cost weighting matrices that penalize the performance and control energy in terms of the tracking error and the relative control ${\tilde{u}}_k=u_k-u_{\text{ss}}$, respectively, with the subscript _ss indicating the steady-state values.

III. Design Methodology

A. State Augmentation With Integral Action

In this section, we introduce the integral action to compensate for external disturbances and to guarantee tracking error convergence. To this end, we introduce a new state w_k that accumulates the tracking error, which is the discrete-time equivalent of the integral action in the continuous-time setting. Based on this new state, we form the following augmented system:

$$\begin{array}{l}{x}_{k+1}=A{x}_k+B{u}_k+D{d}_k,\\ w_{k+1}=w_k+e_k,\end{array}$$

which can be represented compactly in terms of the augmented state vector $X_k={\left[\begin{array}{c}{x}_k^{\text{T}}{w}_k^{\text{T}}\end{array}\right]}^{\text{T}}$ as

$$\begin{array}{l}{X}_{k+1}=\left[\begin{array}{cc}A& 0\\ C& I_p\end{array}\right]X_k+\left[\begin{array}{c}B\\ 0\end{array}\right]u_k+\left[\begin{array}{c}D\\ 0\end{array}\right]d_k-\left[\begin{array}{c}0\\ I_p\end{array}\right]r_k\\ \triangleq \overline{A}{X}_k+\overline{B}{u}_k+\overline{D}{d}_k+\overline{R}{r}_k,\\ Y_k=\left[\begin{array}{cc}C& 0\\ 0& I_p\end{array}\right]X_k\triangleq \overline{C}{X}_k.\end{array}$$ (3)

Assuming that the tracking problem is solvable with a steady-state state X_ss, and a steady-state control u_ss that balances the effect of the disturbance and the reference by means of the integral action, we can obtain the following error dynamics:

$$\begin{array}{l}{\tilde{X}}_{k+1}=\overline{A}{\tilde{X}}_k+\overline{B}{\tilde{u}}_k,\\ {\tilde{Y}}_k=\overline{C}{\tilde{X}}_k,\end{array}$$ (4)

where ${\tilde{X}}_k=X_k-X_{\text{ss}}$. For this augmented system, we define the augmented cost function in terms of (2) as

$$\begin{array}{l}J=\sum _{i=0}^{\infty }\left({\left[\begin{array}{c}{\tilde{x}}_i\\ {\tilde{w}}_i\end{array}\right]}^{\text{T}}\left[\begin{array}{cc}{C}^{\text{T}}{Q}_{e}C& 0\\ 0& Q_w\end{array}\right]\left[\begin{array}{c}{\tilde{x}}_i\\ {\tilde{w}}_i\end{array}\right]+{\tilde{u}}_i^{\text{T}}R{\tilde{u}}_i\right)\\ \triangleq \sum _{i=0}^{\infty }\left({\tilde{X}}_{i}Q{\tilde{X}}_i+{\tilde{u}}_i^{\text{T}}R{\tilde{u}}_i\right).\end{array}$$ (5)

To design an optimal controller we first establish the controllability conditions for the augmented dynamics.

Lemma 1:

The augmented system (3) is controllable if the original system (1) is controllable and has no invariant zeros at z = 1, where z is the z-transform variable.

Proof:

By the Popov-Belevitch-Hautus (PBH) test, the augmented system is controllable if and only if $\left[\begin{array}{c}\overline{A}-\lambda I_{n+p}\overline{B}\end{array}\right]$ has full row rank of n + p. In view of the definition of $\overline{A}$ and $\overline{B}$ in (3), the rank of $\left[\begin{array}{c}\overline{A}-\lambda I_{n+p}\overline{B}\end{array}\right]$ is evaluated as

$$\rho \left[\begin{array}{c}\overline{A}-\lambda I_{n+p}\overline{B}\end{array}\right]=\rho \left[\begin{array}{ccc}A-\lambda I_n& 0& B\\ C& (1-\lambda )I_p& 0\end{array}\right].$$

For λ ≠ 1, we can cancel out the entries of C using the columns of I_p to result in

$$\begin{array}{c}\rho \left[\begin{array}{c}\overline{A}-\lambda I_{n+p}\overline{B}\end{array}\right]=\rho \left[\begin{array}{ccc}A-\lambda I_n& B& 0\\ 0& 0& (1-\lambda )I_p\end{array}\right]\\ =\rho \left[\begin{array}{c}A-\lambda I_{n}B\end{array}\right]+p.\end{array}$$

Because the original system (1) is controllable, we have $\rho \left[\overline{A}-\lambda I_{n}B\right]=n$, and therefore,

$$\rho \left[\overline{A}-\lambda I_{n+p}\overline{B}\right]=n+p.$$

For λ = 1, we have

$$\rho \left[\begin{array}{cc}\overline{A}-\lambda I_{n+p}& \overline{B}\end{array}\right]=\rho \left[\begin{array}{cc}A-\lambda I_n& B\\ C& 0\end{array}\right].$$

Recall that the system (A, B, C) has a zero at z = 1 if and only if

$$\rho \left[\begin{array}{cc}A-\lambda I_n& B\\ C& 0\end{array}\right]<n+\min \{p,m\}=n+p$$

and, as a result, $\rho \left[\overline{A}-\lambda I_{n+p}\overline{B}\right]=n+p$ if the system (A, B, C) has no invariant zeros at z = 1. This completes the proof. ■

Under the conditions of controllability of $(\overline{A},\overline{B})$ and observability of $\left(\overline{A},{(Q)}^{1/2}\right)$, where (Q^1/2)^T(Q)^1/2 = Q, there exists a unique optimal control given by

$${\tilde{u}}_k^{\ast }=-{(R+{\overline{B}}^{\text{T}}{P}^{\ast }\overline{B})}^{-1}{\overline{B}}^{\text{T}}{P}^{\ast }\overline{A}{\tilde{X}}_k=-K^{\ast }{\tilde{X}}_k,$$ (6)

where P* is the unique positive definite solution to the algebraic Riccati equation (ARE) [27]

$${\overline{A}}^{\text{T}}\overline{P}\overline{A}-P+Q-{\overline{A}}^{\text{T}}P\overline{B}{(R+{\overline{B}}^{\text{T}}P\overline{B})}^{-1}{\overline{B}}^{\text{T}}P\overline{A}=0.$$ (7)

Moreover, (6) and (7) suggest that the optimal feedback gain K* for the error dynamics (4) is identical to that of the augmented dynamics (3). As such, K* can be obtained independent of the disturbance, reference, and steady-state offsets, which are handled by the integral action, as will be seen.

The design procedure discussed above is an offline approach that assumes the availability of a perfectly known model of the system. That is, the system dynamics matrices are available so that K* can be obtained by solving the ARE (7). In this work, we are interested in learning K* by employing the framework of RL. In particular, we present the design of a completely model-free Q-learning method that enables us to learn K* online. The existing RL control literature identifies a difficulty in applying RL control to the system dynamics (3) that stems from the presence of the extra term corresponding to the external disturbances [28], which are generally not available for measurement in an online setting. The disturbance, if not accounted for, results in bias in the Q-learning estimates, causing them to be suboptimal and, more importantly, may render the closed-loop system unstable. Therefore, in the following, we present a Q-learning scheme that accounts for the biasing effect of the disturbances.

B. Bias Compensated Q-function

In this section, we first seek to derive a Q-function for the augmented system dynamics while accounting for the bias effect of the disturbances. For the design of an online algorithm, we consider the dynamics (3) rather than the error dynamics (4), which is for analysis only and involves the steady-state values that are not available a priori. Nevertheless, the resulting optimal feedback control matrix K* is the same for both (3) and (4), as mentioned in Section III-A. For a stabilizing control u_k = −K X_k with policy K, the total cost incurred when starting from any state X_k is quadratic in the state as given by

$$V_K\left(X_k\right)=X_k^{\text{T}}P{X}_k,P=P^{\text{T}}>0.$$ (8)

The Q-function associated with K is [29]

$$Q_K=X_k^{\text{T}}Q{X}_k+u_k^{\text{T}}R{u}_k+V_K\left(X_{k+1}\right),$$ (9)

which is the sum of the one-step cost of taking an arbitrary action u_k from the state at time k, X_k, plus the total cost of using policy K from time k + 1 onward. The reference r_k and the disturbance d_k are neither the state nor the decision makers and are, therefore, considered external signals that influence the dynamics. Substituting the dynamics (3) in (9), we have

$$\begin{array}{l}\begin{array}{l}{Q}_K={\left[\begin{array}{c}{X}_k\\ u_k\\ r_k\end{array}\right]}^{\text{T}}\left[\begin{array}{ccc}Q+{\overline{A}}^{\text{T}}P\overline{A}& {\overline{A}}^{\text{T}}P\overline{B}& {\overline{A}}^{\text{T}}P\overline{R}\\ {\overline{B}}^{\text{T}}P\overline{A}& R+{\overline{B}}^{\text{T}}P\overline{B}& {\overline{B}}^{\text{T}}P\overline{R}\\ {\overline{R}}^{\text{T}}P\overline{A}& {\overline{R}}^{\text{T}}P\overline{B}& {\overline{R}}^{\text{T}}P\overline{R}\end{array}\right]\left[\begin{array}{c}{X}_k\\ u_k\\ r_k\end{array}\right]\\ +2X_k^{\text{T}}{\overline{A}}^{\text{T}}\overline{P}\overline{D}{d}_k+2u_k^{\text{T}}{\overline{B}}^{\text{T}}\overline{P}\overline{D}{d}_k+2r_k^{\text{T}}{\overline{B}}^{\text{T}}\overline{P}\overline{D}{d}_k\\ +d_k^{\text{T}}{\overline{D}}^{\text{T}}\overline{P}\overline{D}{d}_k\end{array}\\ \begin{array}{l}\triangleq {\left(z_k^{\prime }\right)}^{\text{T}}{H}^{\prime }{z}_k^{\prime }+2X_k^{\text{T}}{\overline{A}}^{\text{T}}\overline{P}\overline{D}{d}_k+2u_k^{\text{T}}{\overline{B}}^{\text{T}}\overline{P}\overline{D}{d}_k\\ +2r_k^{\text{T}}{\overline{B}}^{\text{T}}\overline{P}\overline{D}{d}_k+d_k^{\text{T}}{\overline{D}}^{\text{T}}\overline{P}\overline{D}{d}_k,\end{array}\end{array}$$ (10)

where the last four terms involving the unmeasurable signal d_k result in an estimation bias. As d_k is not known, we can lump it together with the unknown system dynamics matrices to write the Q-function more compactly as

$$\begin{array}{l}{Q}_K={\left[\begin{array}{c}{X}_k\\ u_k\\ r_k\\ c\end{array}\right]}^{\text{T}}\left[\begin{array}{cccc}Q+{\overline{A}}^{\text{T}}P\overline{A}& {\overline{A}}^{\text{T}}\overline{P}\overline{B}& {\overline{A}}^{\text{T}}P\overline{R}& b_1\\ {\overline{B}}^{\text{T}}P\overline{A}& R+{\overline{B}}^{\text{T}}P\overline{B}& {\overline{B}}^{\text{T}}P\overline{R}& b_2\\ {\overline{R}}^{\text{T}}P\overline{A}& {\overline{R}}^{\text{T}}P\overline{B}& {\overline{R}}^{\text{T}}P\overline{R}& b_3\\ b_1^{\text{T}}& b_2^{\text{T}}& b_3^{\text{T}}& b_4\end{array}\right]\left[\begin{array}{c}{X}_k\\ u_k\\ r_k\\ c\end{array}\right]\\ \begin{array}{l}\triangleq {\left[\begin{array}{c}{X}_k\\ u_k\\ r_k\\ c\end{array}\right]}^{\text{T}}\left[\begin{array}{cccc}{H}_{XX}& H_{Xu}& H_{Xr}& b_1\\ H_{uX}& H_{uu}& H_{ur}& b_2\\ H_{rX}& H_{ru}& H_{rr}& b_3\\ b_1^{\text{T}}& b_2^{\text{T}}& b_3^{\text{T}}& b_4\end{array}\right]\left[\begin{array}{c}{X}_k\\ u_k\\ r_k\\ c\end{array}\right]\\ \triangleq z_k^{\text{T}}H{z}_k,\end{array}\end{array}$$ (11)

where c is an arbitrary bias scaling factor. It is worth pointing out that (11) is an extension of the LQR Q-function that incorporates the biasing effect of the disturbance and the b_i ’s and c are dependent on the disturbance. The optimal Q-function $Q_K^{\ast }$ and its corresponding matrix H* are obtained using the above expression by substituting P = P*. The optimal feedback term can then be obtained as

$$K^{\ast }={\left(H_{uu}^{\ast }\right)}^{-1}\left(H_{uX}^{\ast }\right).$$

The result is the same as the feedback gain K* defined in (6). This suggests that learning the optimal Q-function amounts to learning the optimal feedback controller. In the Section III-C, we will present iterative Q-learning algorithms that provide estimates of this optimal Q-function.

C. Full State Feedback Q-learning Algorithms

In this section, we will present a state feedback Q-learning scheme incorporating the integral action toward solving the optimal tracking problem. Before introducing the bias compensated algorithms, we will present an uncompensated Q-learning algorithm for the augmented system (3). Let $Q_K^{\prime }={\left(z_k^{\prime }\right)}^{\text{T}}{H}^{\prime }{z}_k^{\prime }$ be the uncompensated Q-function that does not fully take into account the effect of the disturbance. The Q-learning Bellman equation corresponding to this Q-function is obtained as [30]

$$Q_K^{\prime }\left(X_k,u_k\right)=X_k^{\text{T}}Q{X}_k+u_k^{\text{T}}R{u}_k+Q_K^{\prime }\left(X_{k+1},-K{X}_{k+1}\right),$$

or equivalently

$${\left(z_k^{\prime }\right)}^{\text{T}}{H}^{\prime }{z}_k^{\prime }=X_k^{\text{T}}Q{X}_k+u_k^{\text{T}}R{u}_k+{\left(z_{k+1}^{\prime }\right)}^{\text{T}}{H}^{\prime }{z}_{k+1}^{\prime }.$$ (12)

Algorithm 0.

State Feedback Q-learning Policy Iteration Algorithm for Tracking Control

input: input-state data

output: H*

1: initialize. Select a stabilizing initial policy $u_k^0=-K^0{X}_k+v_k$ with vk being an exploration signal. Set j ← 0.

2: acquire data. Apply input $u_k^0$ to collect L ≥ l(l + 1)/2 datasets of (Xk, uk, rk).

3: repeat

4:  policy evaluation. Determine the least-squares solution of $${\left(z_k^{\prime }\right)}^{\text{T}}{H}^{\prime j}{z}_k^{\prime }=X_k^{\text{T}}Q{X}_k+u_k^{\text{T}}R{u}_k+{\left(z_{k+1}^{\prime }\right)}^{\text{T}}{H}^{\prime j}{z}_{k+1}^{\prime }.$$

5:  policy improvement. Determine an improved policy as $$K^{j+1}={\left(H_{uu}^{\prime j}\right)}^{-1}\left(H_{uX}^{\prime j}\right).$$

6:  j ← j + 1.

7: until $\Vert K^j-K^{j-1}\Vert <\epsilon$ for some small ε > 0.

Algorithm 0, the uncompensated Q-learning algorithm, is based on this Q-learning equation and includes an integral feedback term to compensate for the steady-state tracking error resulting from the disturbance. However, as will be shown, the integral action alone will not prevent the disturbance from incurring bias in the Q-learning estimates during learning.

We now proceed to present the bias compensated Q-learning algorithms. For the compensated Q-function in (11), we have the following Q-learning Bellman equation

$$z_k^{\text{T}}H{z}_k=X_k^{\text{T}}Q{X}_k+u_k^{\text{T}}R{u}_k+z_{k+1}^{\text{T}}H{z}_{k+1},$$ (13)

or equivalently

$${\overline{H}}^{\text{T}}{\overline{z}}_k=X_k^{\text{T}}Q{X}_k+u_k^{\text{T}}R{u}_k+{\overline{H}}^{\text{T}}{\overline{z}}_{k+1},$$ (14)

where

$$\begin{array}{l}\overline{H}=\mathrm{vec}(H)\\ \triangleq [h_{11}2h_{12}\dots 2h_{1l}{h}_{22}2h_{23}\dots \\ {2h_{2l}\dots h_{ll}]}^{\text{T}}\in {ℝ}^{l(l+1)/2},l=n+m+2p+1,\\ {\overline{z}}_k={\left[z_{k1}^2{z}_{k1}{z}_{k2}\dots z_{k1}{z}_{kl}{z}_{k2}^2{z}_{k2}{z}_{k3}\dots z_{k2}{z}_{kl}\dots z_{kl}^2\right]}^{\text{T}}\end{array}$$

with z_ki being the components of z_k. Based on (14), both PI and VI algorithms are considered next to learn the Q-function and the optimal feedback controller. Algorithm 1 presents a PI Q-learning algorithm for the linear quadratic tracking problem. This is essentially a two-step procedure. In the policy evaluation step, we use the key equation (14) to solve for the unknown vector $\overline{H}$ in the least-squares sense by collecting L ≥ l(l + 1)/2 data samples of (X_k, u_k, r_k) to form the data matrices $\text{Φ}\in {ℝ}^{l(l+1)/2\times L}$ and $\text{Υ}\in {ℝ}^{L\times 1}$, defined by

$$\begin{array}{l}\text{Φ}=\left[\begin{array}{cccc}{\overline{z}}_{k-L+1}-{\overline{z}}_{k-L+2}& {\overline{z}}_{k-L+2}-{\overline{z}}_{k-L+3}& \dots & {\overline{z}}_k-{\overline{z}}_{k+1}\end{array}\right],\\ \text{Υ}=\left[\begin{array}{c}{X}_{k-L+1}^{\text{T}}Q{X}_{k-L+1}+u_{k-L+1}^{\text{T}}R{u}_{k-L+1}\\ X_{k-L+2}^{\text{T}}Q{X}_{k-L+2}+u_{k-L+2}^{\text{T}}R{u}_{k-L+2}\\ ⋮\\ X_k^{\text{T}}Q{X}_k+u_k^{\text{T}}R{u}_k\end{array}\right].\\ \end{array}$$

Algorithm 1.

Bias Compensated State Feedback Q-learning Policy Iteration Algorithm for Tracking Control

input: input-state data

output: H*

1: initialize. Select a stabilizing initial policy $u_k^0=-K^0{X}_k+v_k$ with vk being an exploration signal. Set j ← 0.

2: acquire data. Apply input $u_k^0$ to collect L ≥ l(l + 1)/2 datasets of (Xk, uk, rk).

3: repeat

4:  policy evaluation. Determine the least-squares solution of $${\left({\overline{H}}^j\right)}^{\text{T}}\left({\overline{z}}_k-{\overline{z}}_{k+1}\right)=X_k^{\text{T}}Q{X}_k+u_k^{\text{T}}R{u}_k.$$

5:  policy improvement. Determine an improved policy as $$K^{j+1}={\left(H_{uu}^j\right)}^{-1}\left(H_{uX}^j\right).$$

6:  j ← j + 1

7: until $\Vert K^j-K^{j-1}\Vert <\epsilon$ for some small ε > 0.

Then, the least-squares solution of (14) is given by

$${\overline{H}}^j={\left({\text{ΦΦ}}^{\text{T}}\right)}^{-1}\text{ΦΥ},$$ (15)

where ${\overline{H}}^j$ is the jth estimate of the unknown vector $\overline{H}$. Because u_k = −K X_k, which is linearly dependent on X_k, (15) will not have a unique solution, which is needed for convergence to the optimal parameters. To overcome this issue, an excitation noise is added in u_k to guarantee a unique solution to (15). Note, however, that the exploration noise is unable to excite the bias term c in the vector z_k, and therefore, there will be a zero entry corresponding to the quadratic term in c in the regression vector ${\overline{z}}_k-{\overline{z}}_{k+1}$. Nevertheless, because the bias scaling factor itself is arbitrarily selected, this issue can be tackled by separately adding an arbitrary offset in the corresponding entry of ${\overline{z}}_{k+1}$. The above measures ensure that the following condition can be satisfied:

$$\rho (\text{Φ})=l(l+1)/2.$$ (16)

Clearly, the rank condition (16) is necessary to obtain the optimal solution, which is a unique solution to the least-squares problem (15). This rank condition is crucial to exploration in off-policy RL control algorithms [22]. Interested readers can refer to [31] for the stochastic version of this condition. Algorithm 1 requires a stabilizing (not necessarily optimal) policy at initialization. However, this requirement could be restrictive in certain applications and as such a policy may be difficult to obtain because the system dynamics are not known in advance or the dynamics are nonlinear [32]. To address this difficulty, we can refer to a slightly different iterative technique, VI, which does not impose this restriction. A bias compensated Q-learning VI algorithm is presented in Algorithm 2.

Algorithm 2.

Bias Compensated State Feedback Q-learning Value Iteration Algorithm for Tracking Control

input: input-state data

output: H*

1: initialize. Select an arbitrary policy $u_k^0=-K^0{X}_k+v_k$ with vk being an exploration signal. Set j ← 0 and H0 ≥ 0.

2: acquire data. Apply input $u_k^0$ to collect L ≥ l(l + 1)/2 datasets of (Xk, uk, rk).

3: repeat

4:  value update. Determine the least-squares solution of $${\left({\overline{H}}^{j+1}\right)}^{\text{T}}\left({\overline{z}}_k\right)=X_k^{\text{T}}Q{X}_k+u_k^{\text{T}}R{u}_k+{\overline{H}}^j{\overline{z}}_{k+1}.$$

5:  policy improvement. Determine an improved policy as $$K^{j+1}={\left(H_{uu}^{j+1}\right)}^{-1}\left(H_{uX}^{j+1}\right)$$

6:  j ← j + 1.

7: until $\Vert K^j-K^{j-1}\Vert <\epsilon$ for some small ε > 0.

The data matrices $\text{Φ}\in {ℝ}^{l(l+1)/2\times L}$ and $\text{Υ}\in {ℝ}^{L\times 1}$ for the case of VI are defined by

$$\begin{array}{l}\text{Φ}=\left[\begin{array}{cccc}{\overline{z}}_k^1& {\overline{z}}_k^2& \dots & {\overline{z}}_k^L\end{array}\right],\\ \text{Υ}={\left[r^1\left(y_k,u_k\right)+{\overline{H}}_{j-1}^{\text{T}}{\overline{z}}_{k+1}^1\dots r^L\left(y_k,u_k\right)+{\overline{H}}_{j-1}^{\text{T}}{\overline{z}}_{k+1}^L\right]}^{\text{T}}.\\ \end{array}$$

Remark 1:

Algorithms 1 and 2 are the extension of the standard LQR Q-learning algorithms found in the literature [33]. They learn the disturbance induced bias terms without measuring the disturbance directly, which enables these algorithms to prevent the biasing components from affecting the optimal control parameters.

D. Output Feedback Q-learning Algorithms

Section III-C presented designs that were based on the feedback of the state x_k. However, in practice, only a subset of the state is measurable through the system output. Classical state estimation techniques that estimate x_k through input-output measurements do not apply as the system dynamics are unknown. In our previous work [34], we designed a Q-learning scheme based on a parameterization of the system state in terms of a sequence of delayed measurements of input, output, and disturbance, as follows

$$x_k=M_y{\overline{y}}_{k-1,k-N}+M_u{\overline{u}}_{k-1,k-N}+M_d{\overline{d}}_{k-1,k-N},$$ (17)

where N ≤ n is an upper bound on the system’s observability index. Interested readers can refer to [34] for the details of this parameterization.

We use state parameterization (17) to describe the Q-function in (9). Based on (17), we obtain the parameterization of the augmented state X_k as follows:

$$\begin{array}{l}{X}_k=\left[\begin{array}{cccc}{M}_u& M_y& M_d& 0\\ 0& 0& 0& I_p\end{array}\right]{[\begin{array}{cccc}{\overline{u}}_k^{\text{T}}& {\overline{y}}_k^{\text{T}}& {\overline{d}}_k^{\text{T}}& w_k^{\text{T}}\end{array}]}^{\text{T}}\\ \triangleq \left[\begin{array}{cccc}{\overline{M}}_u& {\overline{M}}_y& {\overline{M}}_d& {\overline{M}}_w\end{array}\right]{[\begin{array}{cccc}{\overline{u}}_k^{\text{T}}& {\overline{y}}_k^{\text{T}}& {\overline{d}}_k^{\text{T}}& w_k^{\text{T}}\end{array}]}^{\text{T}}.\end{array}$$ (18)

Next, we derive the expressions for the output feedback Q-function $Q_{𝒦}$ and the associated output feedback policy $𝒦$.

Substitution of the augmented state vector with its parameterized form (18) into the state feedback Q-function (11) results in

$$\begin{gathered} Q_{𝒦}={\left[\begin{array}{c}{\overline{u}}_k\\ {\overline{y}}_k\\ w_k\\ u_k\\ r_k\\ c\end{array}\right]}^{\text{T}}\left[\begin{array}{cccccc}{\mathscr{H}}_{\overline{u}\overline{u}}& {\mathscr{H}}_{\overline{u}\overline{y}}& {\mathscr{H}}_{\overline{u}w}& {\mathscr{H}}_{\overline{u}u}& {\mathscr{H}}_{\overline{u}\overline{r}}& b_1\\ {\mathscr{H}}_{\overline{y}\overline{u}}& {\mathscr{H}}_{\overline{y}\overline{y}}& {\mathscr{H}}_{\overline{y}w}& {\mathscr{H}}_{\overline{y}u}& {\mathscr{H}}_{\overline{y}r}& b_2\\ {\mathscr{H}}_{w\overline{u}}& {\mathscr{H}}_{w\overline{y}}& {\mathscr{H}}_{ww}& {\mathscr{H}}_{wu}& {\mathscr{H}}_{wr}& b_3\\ {\mathscr{H}}_{u\overline{u}}& {\mathscr{H}}_{u\overline{y}}& {\mathscr{H}}_{uw}& {\mathscr{H}}_{uu}& {\mathscr{H}}_{ur}& b_4\\ {\mathscr{H}}_{r\overline{u}}& {\mathscr{H}}_{r\overline{y}}& {\mathscr{H}}_{rw}& {\mathscr{H}}_{wu}& {\mathscr{H}}_{rr}& b_5\\ b_1^{\text{T}}& b_2^{\text{T}}& b_3^{\text{T}}& b_4^{\text{T}}& b_5^{\text{T}}& b_6^{\text{T}}\end{array}\right]\left[\begin{array}{c}{\overline{u}}_k\\ {\overline{y}}_k\\ w_k\\ u_k\\ r_k\\ c\end{array}\right] \\ \triangleq {\zeta }_k^T\mathscr{H}{\zeta }_k \end{gathered}$$ (19)

where $\mathscr{H}={\mathscr{H}}^T\in {ℝ}^{l\times l}$ with l = mN + pN + m + 2p + 1 and the submatrices are defined in an obvious way as in [34]. Note that $b_i^{\prime }s$ and c are again the disturbance dependent terms as seen in the case of the state feedback Q-function (11). Notice that the delayed disturbance dependent term ${\overline{d}}_k$ introduced as a result of the state parameterization has been lumped together with the biasing term c.

The optimal output feedback policy ${𝒦}^{\ast }$ can be obtained when the optimal output feedback function $Q_{𝒦}^{\ast }$ is minimized with respect to u_k. This results in

$${𝒦}^{\ast }={\left({\mathscr{H}}_{uu}^{\ast }\right)}^{-1}[\begin{array}{ccc}{\mathscr{H}}_{u\overline{u}}^{\ast }& {\mathscr{H}}_{u\overline{y}}^{\ast }& {\mathscr{H}}_{uw}^{\ast }\end{array}].$$

Finally, we have the following feedback control law:

$$\begin{array}{l}{u}_k^{\ast }=-{\left({\mathscr{H}}_{uu}^{\ast }\right)}^{-1}\left({\mathscr{H}}_{u\overline{u}}^{\ast }{\overline{u}}_{k-1,k-N}+{\mathscr{H}}_{u\overline{y}}^{\ast }{\overline{y}}_{k-1,k-N}+{\mathscr{H}}_{uw}^{\ast }{w}_k\right)\\ \triangleq -{𝒦}^{\ast }{\left[{\overline{u}}_{k-1,k-N}^{\text{T}}{\overline{y}}_{k-1,k-N}^{\text{T}}{w}_k^{\text{T}}\right]}^{\text{T}}.\end{array}$$ (20)

It will be shown in the proof of Theorem 1 that the integral action w_k is able to compensate for the unmeasurable disturbances ${\overline{d}}_k$ and d_k in a way similar to the state feedback case.

Having formulated the output feedback Q-function, we need to develop a Q-learning scheme that can learn this function. In view of the output feedback Q-function (19), the output feedback Q-learning Bellman equation follows from (13) as

$${\overline{\mathscr{H}}}^{\text{T}}{\overline{\zeta }}_k=Y_k^{\text{T}}{Q}_y{Y}_k+u_k^{\text{T}}R{u}_k+{\overline{\mathscr{H}}}^{\text{T}}{\overline{\zeta }}_{k+1}$$ (21)

where

$$\overline{\mathscr{H}}=\mathrm{vec}(\mathscr{H})\in {ℝ}^{l(l+1)/2},l=mN+pN+m+2p+1.$$

The regression vector ${\overline{\zeta }}_k\in {ℝ}^{l(l+1)/2}$ is defined as

$$\overline{\zeta }={\left[\begin{array}{c}{\zeta }_{k1}^2{\zeta }_{k1}{\zeta }_{k2}\dots {\zeta }_{k1}{\zeta }_{kl}{\zeta }_{k2}^2{\zeta }_{k2}{\zeta }_{k3}\dots {\zeta }_{k2}{\zeta }_l\dots {\zeta }_{kl}^2\end{array}\right]}^{\text{T}}$$

where ${\zeta }_k=\left[\begin{array}{c}{\zeta }_{k1}{\zeta }_{k2}\dots {\zeta }_{kl}\end{array}\right]$.

In comparison with (14), the output feedback learning equation (21) involves more parameters, as the internal state information is not readily available, but rather is embedded in the sufficiently long sequence of input–output data ${\overline{u}}_{k-1,k-N}$ and ${\overline{y}}_{k-1,k-N}$.

Equation (21) is utilized in the output feedback PI and VI algorithms. The policy iteration Algorithm 3 is the output feedback counterpart of the policy iteration algorithm, Algorithm 1. The two key differences between these two algorithms can be seen in the policy evaluation and the policy update steps, where we observe that the learning and the control update equations do not involve the state information. Algorithm 4 operates in the same way as the state feedback VI Algorithm 2, but without requiring the measurement of the internal state. Furthermore, it also relaxes the condition of a stabilizing initial gain ${𝒦}^0$. It is worth noting at this point that both Algorithms 3 and 4 must also satisfy the rank condition (16), with more exploration compared to the state feedback algorithms because the number of unknown parameters is larger.

Algorithm 3.

Bias Compensated Output Feedback Q-learning Policy Iteration Algorithm for Tracking Control

input: input-output data

output: ${\mathscr{H}}^{\ast }$

1: initialize. Select a stabilizing initial policy $u_k^0=-{𝒦}^0\left[\begin{array}{c}{\overline{u}}_{k-1,k-N}{\overline{y}}_{k-1,k-N}{w}_k\end{array}\right]+v_k$ with vk being an exploration signal. Set j ← 0.

2: acquire data. Apply input $u_k^0$ to collect L ≥ l(l + 1)/2 datasets of $({\overline{u}}_{k-1,k-N},{\overline{y}}_{k-1,k-N},w_k,u_k,r_k)$.

3: repeat

4:  policy evaluation. Determine the least-squares solution of $${\left({\overline{\mathscr{H}}}^j\right)}^{\text{T}}\left({\overline{\zeta }}_k-{\overline{\zeta }}_{k+1}\right)=Y_k^{\text{T}}{Q}_y{Y}_k+u_k^{\text{T}}R{u}_k.$$

5:  policy improvement. Determine an improved policy as $${𝒦}^{j+1}={\left({\mathscr{H}}_{uu}^j\right)}^{-1}\left[{\mathscr{H}}_{u\overline{u}}^j{\mathscr{H}}_{u\overline{y}}^j{\mathscr{H}}_{uw}^j\right].$$

6:  j ← j + 1.

7: until $\Vert {\overline{𝒦}}^j-{\overline{𝒦}}^{j-1}\Vert <\epsilon$ for some small ε > 0.

Algorithm 4.

Bias Compensated Output Feedback Q-learning Value Iteration Algorithm for Tracking Control

input: input-output data

output: H*

1: initialize. Select an arbitrary policy $u_k^0=-{𝒦}^0\left[\begin{array}{c}{\overline{u}}_{k-1,k-N}{\overline{y}}_{k-1,k-N}{w}_k\end{array}\right]+v_k$ with vk being an exploration signal. Set j ← 0 and ${\mathscr{H}}^0\ge 0$.

2: acquire data. Apply input $u_k^0$ to collect L ≥ l(l + 1)/2 datasets of $({\overline{u}}_{k-1,k-N},{\overline{y}}_{k-1,k-N},w_k,u_k,r_k)$.

3: repeat

4:  value update. Determine the least-squares solution of $${\left({\overline{\mathscr{H}}}^{j+1}\right)}^{\text{T}}\left({\overline{\zeta }}_k\right)=Y_k^{\text{T}}{Q}_y{Y}_k+u_k^{\text{T}}R{u}_k+{\overline{\mathscr{H}}}^j{\overline{\zeta }}_{k+1}.$$

5:  policy improvement. Determine an improved policy as $${𝒦}^{j+1}={\left({\mathscr{H}}_{uu}^{j+1}\right)}^{-1}\left[{\mathscr{H}}_{u\overline{u}}^{j+1}{\mathscr{H}}_{u\overline{y}}^{j+1}{\mathscr{H}}_{uw}^{j+1}\right].$$

6:  j ← j + 1.

7: until $\Vert {\overline{𝒦}}^j-{\overline{𝒦}}^{j-1}\Vert <\epsilon$ for some small ε > 0.

We now establish the convergence of the proposed scheme toward achieving optimal tracking in Theorem 1:

Theorem 1:

Assume that the controllability conditions in Lemma 1 hold and $(\overline{A},{(Q)}^{1/2})$ (for state feedback) or $(\overline{A},{(Q_y)}^{1/2}\overline{C})$ (for output feedback) is observable. Then the proposed scheme generates a sequence of controls {$u_k^j$, j = 1, 2, 3, …} that converges to the optimal feedback controller under the rank condition (16), and the tracking error e_k converges to zero if the disturbance and reference vary infrequently relative to the control dynamics.

Proof:

Consider first the state feedback algorithms, Algorithms 1 and 2. The bias compensated Q-function (11) satisfies the state feedback Q-learning equation (13), which forms the basis of Algorithms 1 and 2. This equation has a unique solution if the rank condition (16) holds. Given the controllability and observability assumptions on the pairs $(\overline{A},\overline{B})$, and $(\overline{A},{(Q)}^{1/2})$, respectively, the PI and VI Q-learning algorithms, Algorithms 1 and 2, converge to the optimal feedback matrix K* as shown in [30], [35]. Under K*, the closed-loop dynamics are given by

$$X_{k+1}=\left(\overline{A}-\overline{B}{K}^{\ast }\right)X_k+\left[\begin{array}{c}\begin{array}{cc}\overline{D}& \overline{R}\end{array}\end{array}\right]{\left[\begin{array}{c}\begin{array}{cc}{d}_k^{\text{T}}& r_k^{\text{T}}\end{array}\end{array}\right]}^{\text{T}}.$$

The disturbance and reference can be considered in a steady state if they vary infrequently relative to the dynamics. Because $\overline{A}-\overline{B}{K}^{\ast }$ is Schur stable, under these external steady-state inputs, X_k reaches its steady-state X_ss and, therefore, w_k+1 = w_k. Then, because w_k+1 = w_k + e_k, it follows that the tracking error e_k converges to zero asymptotically.

Consider next the output feedback algorithms, Algorithms 3 and 4. The bias compensated output feedback Q-function satisfies the output feedback Q-learning equation (21), which is employed in Algorithms 3 and 4. Because $(\overline{A},\overline{B})$ and $(\overline{A},{(Q_y)}^{1/2}\overline{C})$ are controllable and observable, respectively, the output feedback policy iteration and VI algorithms converge to the optimal output feedback gain ${𝒦}^{\ast }$, as shown in our previous work [36], under the rank condition (16). The closed-loop dynamics under the output feedback control are

$$\begin{gathered} X_{k+1}=\overline{A}{X}_k-\overline{B}{K}^{\ast }{\left[{\left(M_u{\overline{u}}_{k-1,k-N}+M_y{\overline{y}}_{k-1,k-N}\right)}^{\text{T}}{w}_k^{\text{T}}\right]}^{\text{T}} \\ +\left[\begin{array}{cc}\overline{D}& \overline{R}\end{array}\right]{[\begin{array}{cc}{\overline{d}}_k^{\text{T}}& r_k^{\text{T}}\end{array}]}^{\text{T}}. \end{gathered}$$

Adding and subtracting the missing disturbance sequence $M_d{\overline{d}}_{k-1,k-N}$ to obtain the state feedback form using the parameterization (17) results in

$$\begin{gathered} X_{k+1}=\left(\overline{A}-\overline{B}{K}^{\ast }\right)X_k+\left[\begin{array}{cc}\overline{D}& \overline{R}\end{array}\right]{\left[\begin{array}{cc}{d}_k^{\text{T}}& r_k^{\text{T}}\end{array}\right]}^{\text{T}} \\ +\overline{B}{K}_{1:n}^{\ast }{M}_d{\overline{d}}_{k-1,k-N}. \end{gathered}$$

Then, similar to the state feedback case, we also have d_k−1,k−N in the steady state, which, in view of the fact that $\overline{A}-\overline{B}{K}^{\ast }$ is Schur stable, implies that X_k will reach the steady-state X_ss and w_k+1 = w_k. Therefore, the tracking error e_k converges to zero asymptotically. ■

IV. Application to HVAC Zone Control

In this section, we apply the proposed scheme to design an HVAC controller for a zone in a commercial building. This is a multi-objective optimal control problem that requires accounting for both the zone comfort and the energy consumption. To formulate this problem into the presented mathematical framework, the zone comfort is associated with obtaining the desired thermal state (i.e., set point temperature) and the energy cost corresponds to the control energy utilized by the actuators. We consider the AHU as the actuator that supplies the zone with air of an appropriate temperature (i.e., supply air) to manipulate the zone temperature. The dynamic model of an HVAC zone used in this case study is adapted from [37]. The thermal dynamics of a building zone are given by the following set of differential equations:

$$\begin{array}{l}\frac{d{T}_z}{dt}=\frac{f_{sa}{\rho }_a{c}_{pa}}{C_z}\left(T_{sa}-T_z\right)+2\frac{U_{w_{ew}}{A}_{w_{ew}}}{C_z}\left(T_{w_{ew}}-T_z\right)\\ +2\frac{U_{w_{ns}}{A}_{w_{ns}}}{C_z}\left(T_{w_{ns}}-T_z\right)+\frac{K_o}{C_z}\left(T_o-T_z\right)+\frac{q}{C_z},\\ \frac{d{T}_{w_{ew}}}{dt}=\frac{U_{w_{ew}}{A}_{w_{ew}}}{C_{w_{ew}}}\left(T_z-T_{w_{ew}}\right)+\frac{U_{w_{ew}}{A}_{w_{ew}}}{C_{w_{ew}}}\left(T_o-T_{w_{ew}}\right),\\ \frac{d{T}_{w_{ns}}}{dt}=\frac{U_{w_{ns}}{A}_{w_{ns}}}{C_{w_{ns}}}\left(T_z-T_{w_{ns}}\right)+\frac{U_{w_{ns}}{A}_{w_{ns}}}{C_{w_{ns}}}\left(T_o-T_{w_{ns}}\right),\end{array}$$

which is discretized with a sampling period of one minute to obtain a state space model of the form (1). The description of the quantities is given in the Nomenclature. The nominal parameters given in the Nomenclature are only used to compute the true values of optimal parameters in order to compare our estimates. In other words, the proposed control scheme itself does not require any knowledge of these parameters and the optimal control parameters are learned online.

In this model the outside temperature and the heat gains from the occupants, lights, etc., are the disturbances, which are all assumed to be unmeasurable. Let the user-defined performance index be specified as Q_e = 300, Q_w = 60, and R = 100. Note that Q_w ≠ 0 is needed for the observability of the pair $(\overline{A},{(Q)}^{1/2})$ of the augmented system. The optimal feedback control gain for the augmented dynamics (3) can be found by solving the Riccati (7) and is given by

$$K^{\ast }=\left[\begin{array}{c}\begin{array}{cccc}1.6864& 0.1413& 0.1829& 0.5906\end{array}\end{array}\right].$$

Before presenting the results of the bias compensated Q-learning algorithms, we will first test the uncompensated Q-learning algorithm, Algorithm 0, to analyze the effect of the unmeasurable disturbances. The parameter estimates experience bias as a result of the external disturbances. The final estimate of the control matrix is

$$\widehat{K}=\left[\begin{array}{c}\begin{array}{cccc}-0.6861& 25.4644& -32.9402& -1.0840\end{array}\end{array}\right].$$

In this particular case, even though the algorithm was initialized with a stabilizing control matrix, the final control gain estimate is not only biased but also destabilizing. This can be seen from the eigenvalues of the resulting closed-loop dynamics matrix $\overline{A}-\overline{B}\widehat{K}$, which are 0.3235, 1.5391, 0.9695 ± j0.0114 with λ = 1.5391 being the unstable eigenvalue. For comparison with existing works, recall the robust off-policy algorithms in [24] and [13]. The off-policy method in [24] employs the knowledge of the input and disturbance matrices, whereas the method in [13] removes this requirement. However, both of these methods require access to the disturbance data during the learning phase (see Step 1 of Algorithm 2 in [13], [24]). The model-free algorithm in [13] is applied to our problem for comparison with our model-free method, first with the disturbance data during the learning phase and then without. A discounting factor of α = 0.1 and the disturbance attenuation level of γ = 5 were selected for the H_∞ cost function described in [13]. The optimal control gain obtained by this method is

$$K^{\ast }=\left[\begin{array}{c}\begin{array}{cccc}44.7092& 0.1066& 0.1331& -45.6562\end{array}\end{array}\right],$$

where the last element corresponds to the reference trajectory, T_zr. The learning algorithm in [13] was then applied, with disturbance signals T_o and q known during the learning phase. The final estimate of the control gain is

$$\widehat{K}=\left[\begin{array}{c}\begin{array}{cccc}44.7095& 0.1067& 0.1329& -45.6565\end{array}\end{array}\right]$$

which shows convergence to the optimal solution, consistent with the results in [13]. Next, we apply the learning algorithm from [13] without the measurement of the disturbance, although the same disturbances act on the system. In this case, the final estimate of the control gain is

$$\widehat{K}=\left[\begin{array}{c}\begin{array}{cccc}-72.1094& -0.6641& -2.7734& 52.1094\end{array}\end{array}\right].$$

The lack of measurement of the disturbances has resulted in a bias in the estimates. Our algorithms address this limitation.

We will now focus on our proposed bias compensated Q-learning algorithms. Let us consider first the policy iteration algorithm, Algorithm 1. We start the algorithm with a stabilizing initial control, which, in this example, is a simple proportional-integral controller. The proportional gain is 0.8432 and the integral gain 0.2953. This corresponds to an initial policy K⁰ = [0.8432 0 0 0.2953]. To satisfy the rank condition (16), we add sinusoids of different frequencies and magnitudes to the feedback control policy for the supply air temperature T_sa. This initial control is applied during the first 30 minutes to collect online system data. Fig. 1 shows the evaluation of the closed-loop response under Algorithm 1.

Fig. 1.

Evolution of the closed-loop system under Algorithm 1.

During the first 30 minutes we see an exploratory response while the output is still trying to track the reference signal. This is a result of applying an already stabilizing policy that could provide suboptimal tracking in the presence of added exploratory signals. These 30 minutes of online data are then utilized to solve the Bellman equation in the policy iteration step and to update the control parameters in the subsequent iterations j = 1, 2, ….

At the beginning of hour 2, a disturbance is introduced as a result of a decrease in the outside temperature, which is almost seamlessly compensated by the controller with an expected increase in the supply air temperature to compensate for the outside temperature drop. At the same time, the desired temperature set point increases. As can be seen, the zone temperature responds to this change and converges to the new set point. Similarly, when the heat gain load changes because of occupancy, lighting, or other sources, the proposed scheme is able to track the desired reference trajectory in the remainder of the period. The proposed Q-learning scheme learns the optimal control parameters for the augmented system while compensating for the external disturbances that would otherwise cause Q-learning to diverge. The final estimates of the optimal control gain are

$$\widehat{K}=\left[\begin{array}{c}\begin{array}{cccc}1.6865& 0.1414& 0.1831& 0.5907\end{array}\end{array}\right],$$

which is close to the optimal value despite the presence of the unmeasurable disturbances. This is a result of the bias compensation mechanism introduced in the Q-function and is an advantage of the proposed scheme.

We now proceed to validate the proposed VI algorithm, Algorithm 2. We test this algorithm under the same conditions as for Algorithm 1. Different from the PI algorithm, we initialize the VI algorithm with zero feedback gain, that is, K⁰ = [0 0 0 0]. Clearly, this gain is nonstabilizing and we cannot expect tracking during the first 30 minutes of learning, as can be seen in the zone temperature response in Fig. 2. It is interesting to note that the post learning trajectories, after the first 30 minutes of learning, are the same for both the PI and VI algorithms. This is because both the algorithms eventually converge to the optimal control parameters. The final estimate of the optimal control gain is

$$\widehat{K}=\left[\begin{array}{c}\begin{array}{cccc}1.6862& 0.1410& 0.1822& 0.5905\end{array}\end{array}\right].$$

Fig. 2.

Evolution of the closed-loop system under Algorithm 2.

Note that more iterations are required for Algorithm 2 to converge to the optimal parameters because the search space, which is not limited by a stabilizing initial controller, is larger.

The results presented so far dealt with the full state feedback case, that is, the measurement of the internal state was required for both Algorithms 1 and 2 to learn the optimal control parameters. In the following, we present results for Algorithms 3 and 4 that do not impose this requirement. These algorithms are driven completely by the input-output data instead of requiring the internal state information. For the HVAC application, this means that we no longer need to install sensors on the walls to measure the wall temperature. Instead, only zone temperature measurements are required. This reflects a more realistic HVAC control system.

For the purpose of comparison, the user-defined cost matrices and the rest of the conditions for the output feedback algorithms are kept the same as with the state feedback algorithms. We first validate the output feedback policy iteration algorithm, Algorithm 3. We utilize a proportional-integral controller for initial tracking. The nominal optimal output feedback control parameters for the augmented dynamics (3) can be found by solving the Riccati (7) and the state parameterization (17) and are given by

$$\begin{array}{l}{𝒦}^{\ast }=[\begin{array}{cccc}0.4230& -4.5286& 3.1156& 19.2045\end{array}\\ \begin{array}{ccc}-24.2879& 8.1313& 0.5906\end{array}].\end{array}$$

Algorithm 3 involves a longer learning phase because there are more unknown parameters to be determined. Specifically, we collected 50 datasets of the input-output data as compared to the 30 datasets for the state feedback algorithms. It can be seen in Fig. 3 that the output feedback algorithm regulates the zone temperature well, similar to the state feedback algorithm but without requiring wall temperature measurements. The final estimate of the output feedback optimal control gain using Algorithm 3 is

$$\begin{array}{l}\widehat{𝒦}=[\begin{array}{cccc}0.4230& -4.5232& 3.1105& 19.1830\end{array}\\ \begin{array}{ccc}-24.2534& 8.1178& 0.5906\end{array}]\end{array}$$

which is close to the optimal output feedback parameters ${𝒦}^{\ast }$.

Fig. 3.

Evolution of the closed-loop system under Algorithm 3.

If stabilizing initial output feedback parameters are not known, then Algorithm 4 can be applied. In this case the output feedback control parameters are initialized to zero. The system response is shown in Fig. 4. In the absence of a stabilizing initial feedback law, the zone temperature is unable to track the desired reference temperature. In the post learning response, we see that it is able to track the reference signal quite well even in the presence of the unmeasurable disturbances arising from heat gain and outside climate variations. The final estimate of the output feedback optimal control gain using Algorithm 4 is

$$\begin{array}{l}\widehat{𝒦}=[0.4230-4.53503.122219.2303\\ -24.33128.14840.5906]\end{array}$$

which is also close to the optimal output feedback parameters ${𝒦}^{\ast }$.

Fig. 4.

Evolution of the closed-loop system under Algorithm 4.

V. Conclusions

This article presented a model-free solution to the optimal tracking problem involving unmeasurable disturbances based on the framework of reinforcement learning. A new Q-learning based scheme was proposed with a bias compensation mechanism to account for the effect of the disturbance on the learning estimates. An extended Q-function was employed that includes bias compensation terms to prevent the control parameters from drifting away in the presence of the disturbance. Both PI and VI algorithms based on state feedback and output feedback were presented to learn the optimal parameters and to guarantee convergence of the tracking error to zero. Finally, the proposed scheme was validated by designing an optimal set point tracking controller for a practical HVAC zone system in the presence of the unknown disturbances related to outside climate variations and the internal heat gains. In our future work, we will consider extending the design to develop a distributed control scheme for a more complex HVAC system.

Nomenclature

A_wew

Area of East/West walls = 9 m²

A_wns

Area of North/South walls = 12 m²

C_pa

Specific heat of air = 1.005 kJ/kg-C

C_wew

Thermal capacitance of East/West walls = 70 kJ/C

C_wns

Thermal capacitance of North/South walls = 60 kJ/C

C_z

Thermal capacitance of the zone = 60 kJ/C

f_sa

Volume flow rate of the supply air = 0.192 m³

q

Heat gain from occupants, lights, doors in Watts (W)

T_o

Outside temperature in degrees Celsius (C)

T_sa

Supply air temperature in degrees Celsius (C)

T_wew

Temperature of the East/West walls in degrees Celsius (C)

T_wns

Temperature of the North/West walls in degrees Celsius (C)

T_z

Temperature of the zone in degrees Celsius (C)

U_wew

Heat transfer coefficient of East/West walls = 2 W/m²-C

U_wns

Heat transfer coefficient of North/West walls = 2 W/m²-C

K_o

Thermal transfer coefficient between the zone and the outside = 9 W/C

ρ_a

Density of air = 1.25 kg/m³

Biographies

Syed Ali Asad Rizvi received the B.E. degree in industrial electronics from the Institute of Industrial Electronics Engineering, NED University of Engineering and Technology, Karachi, Pakistan, in 2012, the M.S. degree in electrical engineering from the National University of Sciences and Technology, Islamabad, Pakistan, in 2014, and the Ph.D. degree in electrical engineering from the University of Virginia, Charlottesville, VA, USA, in 2020.

He is currently a Post-Doctoral Fellow with the National Institute of Standards and Technology (NIST), Gaithersburg, MD, USA. His current research interests include artificial intelligence, reinforcement learning control, robust control, distributed learning and optimization, and their application in cyber-physical systems.

Amanda J. Pertzborn received the Ph.D. degree from the University of Wisconsin-Madison, Madison, WI, USA.

She is currently the PI of the Intelligent Building Agents Project with the Building Energy and Environment Division, National Institute of Standards and Technology (NIST), Gaithersburg, MD, USA. Her research focuses on intelligent control of heating, ventilation, and air conditioning (HVAC) systems.

Zongli Lin (Fellow, IEEE) received the B.S. degree in mathematics and computer science from Xiamen University, Xiamen, China, in 1983, the Master of Engineering degree in automatic control from the Chinese Academy of Space Technology, Beijing, China, in 1989, and the Ph.D. degree in electrical and computer engineering from Washington State University, Pullman, WA, USA, in 1994.

He is currently the Ferman W. Perry Professor with the School of Engineering and Applied Science and a Professor of electrical and computer engineering with the University of Virginia, Charlottesville, VA, USA. His current research interests include nonlinear control, robust control, and control applications.

Dr. Lin is also a fellow of IFAC and the American Association for the Advancement of Science (AAAS). He was the Program Chair of the 2018 American Control Conference and the General Chair of the 13th and 16th International Symposium on Magnetic Bearings, held in 2012 and 2018, respectively. He was an Associate Editor of IEEE Transactions on Automatic Control from 2001 to 2003, IEEE/ASME Transactions on Mechatronics from 2006 to 2009, and IEEE Control Systems Magazine from 2005 to 2012. He was elected as a member of the Board of Governors of the IEEE Control Systems Society from 2008 to 2010 and 2019 to 2021 and chaired the IEEE Control Systems Society Technical Committee on Nonlinear Systems and Control from 2013 to 2015. He has served on the operating committees of several conferences. He also serves on the editorial boards of several journals and book series, including Automatica, Systems & Control Letters, and Birkhauser book series, Control Engineering.