Approximate Hamilton-Jacobi Reachability Analysis for a Class of Two-Timescale Systems, with Application to Biological Models

Abstract

Hamilton-Jacobi reachability (HJR) is an exciting framework used for control of safety-critical systems with nonlinear and possibly uncertain dynamics. However, HJR suffers from the curse of dimensionality, with computation times growing exponentially in the dimension of the system state. Many autonomous and controlled systems involve dynamics that evolve on multiple timescales, and for these systems, singular perturbation methods can be used for model reduction. However, such methods are more challenging to apply in HJR due to the presence of an underlying differential game. In this work, we leverage prior work on singularly perturbed differential games to identify a class of systems which can be readily reduced, and we relate these results to the quantities of interest in HJR. We demonstrate the utility of our results on two examples involving biological systems, where dynamics fitting the identified class are frequently encountered.

I. INTRODUCTION

As computers have become faster, Hamilton-Jacobi reachability (HJR) is an increasingly useful tool for analyzing and controlling safety-critical systems using the theory of differential games [1]. In particular, HJR determines the states, known as the backward reachable set (BRS), from which a system can be guaranteed able to achieve some goal, such as reaching a target or avoiding an obstacle, despite uncertain model dynamics or presence of an adversary [2]–[4]. In the process, one also obtains a control law by which to achieve the goal. Substantial work has been performed to extend and apply HJR to problems in domains such as aerospace, robotics, and reinforcement learning [5]–[7].

The key factor limiting the more widespread use of HJR is the curse of dimensionality. Even with modern processors, the algorithm typically becomes computationally intractable for systems of dimension 6 or higher. This limitation produces a need for model reduction techniques that can be applied in the context of differential games.

In the control theory community, singular perturbation methods are mainstays of reduced modeling of autonomous and controlled systems, specifically those that evolve on multiple timescales [8], [9]. Although results on model reduction for singularly perturbed (SP) differential games also exist, they have yet to see widespread use, particularly in the context of reachability analyses [10]. This gap is likely in part due to the complexity of analytically finding the proper reduced model for a given SP differential game, if the game admits such a reduced model in the first place.

A. Background and prior work

Some of the earliest work studying differential games with both fast and slow state dynamics was by Gardner and Cruz in the late 1970s, in which the authors demonstrated that naively performing model reduction via singular perturbation prior to evaluating the payoff at the Nash equilibrium gives a result different from evaluating the true payoff in the limit of increasingly separated timescales [11]. In their study, the fast and slow systems were linear and the payoff was quadratic. These authors refer to such a singularly perturbed differential game as ill-posed. Indeed, the discovery of an ill-posed game in such a simple setting motivated the careful study of singular perturbation methods for differential games. Research along these lines was continued by Khalil and Kokotovic soon after, demonstrating the link between well-posedness and the information structure of the game [12].

Later on, the advent of viscosity solutions of Hamilton-Jacobi equations [13], [14] and the study of their relationship to differential games [15], [16] ignited renewed interest in the study of differential games with non-anticipative information structures, nonlinear dynamics, and non-quadratic payoff functions. Leveraging these results, guarantees of well-posedness for these general differential games under the added complication of singularly perturbed dynamics was undertaken by Gaitsgory in [10]. Here, Gaitsgory demonstrated that such a game could be decomposed into an “associated fast” differential game involving only the fast dynamics and a reduced differential game involving only the slow variables. However, certifying well-posedness using the theory outlined in this work for a specific system is typically challenging, involving a number of assumptions that may be difficult to check. Moreover, in practice, using the theory often requires one to explicitly obtain the value function for the associated fast game, which may be impractical. These results have also been adapted for discrete-time settings [17].

Extensive further work has been pursued on linear-quadratic differential games, with and without noise, on finite and infinite intervals, and for zero-sum and non-zero-sum settings [18]–[21]. Additional works have proceeded for the nonlinear case by approaches rooted in weak solutions to Hamilton-Jacobi Bellman equations, but each typically involves conditions that may be challenging to verify for general systems [22]–[24].

Thus, in this work, we instead choose to focus on a class of nonlinear systems for which well-posedness can be certified via Gaitsgory’s results, and perhaps more importantly, we demonstrate how one can use this convergence result to provide bounds on quantities of interest in HJR, such as the BRS, for such systems.

B. Contributions

In this work, we provide three main contributions. First, we identify a class of differential games for which a reduced model can be analytically derived and rigorously justified via the results in [10]. Second, we derive inner and outer approximations of the backward reachable sets for these systems. Finally, we provide examples of how to apply these results in the context of models of biomolecular system models, where two-timescale dynamics arise frequently [25].

II. SYSTEM DESCRIPTION

Let $𝒰\subset {ℝ}^{n_{\text{u}}}$ and $𝒟\subset {ℝ}^{n_{\text{d}}}$ be non-empty and compact. For each $t<0$, let ${\mathbb{U}}_{[t,0)}$ and ${\mathbb{D}}_{[t,0)}$ be the set of measurable functions from $[t,0)$ to $𝒰$ and $[t,0)$ to $𝒟$, respectively.

A. Singularly perturbed system

We consider the following SP system, parameterized by a “small” parameter $\epsilon >0$:

$$\dot{\mathbf{\text{z}}}=f(\mathbf{\text{z}},\mathbf{\text{u}},\mathbf{\text{d}})+M(\mathbf{\text{z}})A(\mathbf{\text{z}},\mathbf{\text{u}},\mathbf{\text{d}})\mathbf{\text{y}}$$ (1)

$$\epsilon \dot{\mathbf{\text{y}}}=g(\mathbf{\text{z}},\mathbf{\text{u}},\mathbf{\text{d}})+A(\mathbf{\text{z}},\mathbf{\text{u}},\mathbf{\text{d}})\mathbf{\text{y}},$$ (2)

where $f:{ℝ}^{n_{\text{z}}}\times 𝒰\times 𝒟\to {ℝ}^{n_{\text{z}}}$, $g:{ℝ}^{n_{\text{z}}}\times 𝒰\times 𝒟\to {ℝ}^{n_{\text{y}}}$, $M:{ℝ}^{n_{\text{z}}}\to {ℝ}^{n_{\text{z}}\times n_{\text{y}}}$, and $A:{ℝ}^{n_{\text{z}}}\times 𝒰\times 𝒟\to {ℝ}^{n_{\text{y}}\times n_{\text{y}}}$. In the above, $\mathbf{\text{z}}$ and $\mathbf{\text{y}}$ represent the “slow” and “fast” state variables, respectively, and $\mathbf{\text{u}}$ and $\mathbf{\text{d}}$ represent control and disturbance signals, respectively.

This formulation arises commonly in models of biological systems, where simple chemical reactions often have dynamics that are affine (or nearly affine) in the species’ concentrations (i.e. $\mathbf{\text{y}}$) and occur on much faster timescales than more complex processes. We make the following assumptions:

Assumption 1 (Regularity):

There is some $K>0$ such that $\parallel M(z)\parallel +\parallel A(z,u,d)\parallel \le K$ and $\parallel f(z,u,d)\parallel +\parallel g(z,u,d)\parallel \le K(1+\parallel z\parallel )$ for all $z\in {ℝ}^{n_{\text{z}}}$, $u\in 𝒰$, and $d\in 𝒟$. Moreover, $f$, $g$, $M$, and $A$ are all locally Lipschitz in $z$ and continuous on their domains.

Assumption 2 (Stability of fast dynamics):

There is a symmetric, positive definite matrix $P\in {ℝ}^{n_{\text{y}}\times n_{\text{y}}}$ such that

$$A{(z,u,d)}^{\top }P+PA(z,u,d)\prec 0$$

for each $z\in {ℝ}^{n_{\text{z}}}$, $u\in 𝒰$, and $d\in 𝒟$.

Assumption 3 (Isaacs’ condition for reduced model):

For each $\lambda$, $z\in {ℝ}^{n_{\text{z}}}$,

$$\begin{gathered} \underset{u\in 𝒰}{\min }\underset{d\in 𝒟}{\text{max}}{\lambda }^{\top }[f(z,u,d)-M(z)g(z,u,d)] \\ =\underset{d\in 𝒟}{\text{max}}\underset{u\in 𝒰}{\min }{\lambda }^{\top }[f(z,u,d)-M(z)g(z,u,d)]. \end{gathered}$$

The reduced model to which Assumption 3 is related will be introduced shortly. In practice, these assumptions are often satisfied. Indeed, Assumption 2 is an analog of the stability requirement of standard singular perturbation methods, and Isaacs’ condition is assumed commonly in differential games.

For each $\epsilon >0$, $t<0$, $z\in {ℝ}^{n_{\text{z}}}$, $y\in {ℝ}^{n_{\text{y}}}$, $\mathbf{\text{u}}\in {\mathbb{U}}_{[t,0)}$, and $\mathbf{\text{d}}\in {\mathbb{D}}_{[t,0)}$, we let $\left({\mathbf{\text{z}}}_{t,z,y}^{\epsilon ,\mathbf{\text{u}},\mathbf{\text{d}}},{\mathbf{\text{y}}}_{t,z,y}^{\epsilon ,\mathbf{\text{u}},\mathbf{\text{d}}}\right):[t,0]\to {ℝ}^{n_{\text{z}}}\times {ℝ}^{n_{\text{y}}}$ be the Carathéodory solution of (1)–(2) for which ${\mathbf{\text{z}}}_{t,z,y}^{\epsilon ,\mathbf{\text{u}},\mathbf{\text{d}}}(t)=z$ and ${\mathbf{\text{y}}}_{t,z,y}^{\epsilon ,\mathbf{\text{u}},\mathbf{\text{d}}}(t)=y$ (existence and uniqueness of this solution follows from Assumption 1; see Theorem 1.2.1 in [26]).

B. Description of the differential game

For each value of $\epsilon >0$, we consider a differential game in which the control “player” $\mathbf{\text{u}}$ wishes to drive the slow variable $\mathbf{\text{z}}$ to be in some open set $𝒮\subseteq {ℝ}^{n_{\text{z}}}$ at time 0, and the disturbance “player” $\mathbf{\text{d}}$ wishes to prevent this outcome.

When formulating the game mathematically, we wish to ensure the disturbance player only has information regarding the control player’s decisions up to the current time. This restriction is codified via the following definition.

Definition 1:

Given $t<0$, a map $\gamma :{\mathbb{U}}_{[t,0)}\to {\mathbb{D}}_{[t,0)}$ is a non-anticipative disturbance strategy on $[t,0)$ if for each $s\in [t,0)$ and $u_1$, $u_2\in {\mathbb{U}}_{[t,0)}$, we have that $\gamma \left[u_1\right](\tau )=\gamma \left[u_2\right](\tau )$ for a.e. $\tau \in [t,s)$ whenever $u_1(\tau )=u_2(\tau )$ for a.e. $\tau \in [t,s)$.

We denote the set of non-anticipative disturbance strategies on $[t,0)$ by ${\text{Γ}}_{[t,0)}$.

The game proceeds as follows. Denote by $t<0$ the initial time and by ($z$, $y$) the initial system state. The disturbance player selects a $\gamma \in {\text{Γ}}_{[t,0)}$, and the control player the selects a $\mathbf{\text{u}}\in {\mathbb{U}}_{[t,0)}$. We let $\mathbf{\text{d}}=\gamma [\mathbf{\text{u}}]$ be the resulting disturbance signal. If ${\mathbf{\text{z}}}_{t,z,y}^{\epsilon ,\mathbf{\text{u}},\mathbf{\text{d}}}(0)\in 𝒮$, the control player wins the game; otherwise the disturbance player wins.

C. Backward reachable set and the value function

The first quantity of interest in HJR is the backward reachable set (BRS). For each $\epsilon >0$ and $t<0$, we let

$$\begin{gathered} \text{BR}{\text{S}}_{\text{ε}}(t)=\{(z,y)\in {ℝ}^{n_{\text{z}}}\times {ℝ}^{n_{\text{y}}}\mid \\ \text{∀}\gamma \in {\text{Γ}}_{[t,0)}\exists \mathbf{\text{u}}\in {\mathbb{U}}_{[t,0)}{\mathbf{\text{z}}}_{t,z,y}^{\epsilon ,\mathbf{\text{u}},\gamma [\mathbf{\text{u}}]}(0)\in 𝒮\}. \end{gathered}$$

Note that $\text{BR}{\text{S}}_{\text{ε}}(t)$ represents the set of initial states from which the control player will win the game corresponding to parameter $\epsilon$ and inital time $t$ if both players act rationally. For convenience, we also set $\text{BR}{\text{S}}_{\text{ε}}(0)=𝒮$.

In the HJR framework, we implicitly encode the target set $𝒮$ via a function $\ell$ and then define a value function for the game, where $\ell$ serves as the terminal payoff function. More explicitly, we choose $\ell :{ℝ}^{n_{\text{z}}}\to ℝ$ to be a bounded, Lipschitz function for which

$$𝒮=\left\{z\in {ℝ}^{n_{\text{z}}}:\ell (z)<0\right\}.$$

For each $\epsilon >0$, we define the value function $V_{\epsilon }:(-\infty ,0]\times {ℝ}^{n_{\text{z}}}\times {ℝ}^{n_{\text{y}}}\to ℝ$ by

$$V_{\epsilon }(t,z,y)=\{\begin{array}{cc}{\sup }_{\gamma \in {\text{Γ}}_{[t,0)}}{\text{inf}}_{\mathbf{\text{u}}\in {\mathbb{U}}_{[t,0)}}\ell \left({\mathbf{\text{z}}}_{t,z,y}^{\epsilon ,\mathbf{\text{u}},\gamma [\mathbf{\text{u}}]}(0)\right)& t<0,\\ \ell (z)& t=0.\end{array}$$

It can be seen from these definitions that for each $t\le 0$,

$$\begin{gathered} \left\{(z,y)\in {ℝ}^{n_{\text{z}}}\times {ℝ}^{n_{\text{y}}}\mid V_{\epsilon }(t,z,y)<0\right\}\subseteq \text{BR}{\text{S}}_{\text{ε}}(t) \\ \subseteq \left\{(z,y)\in {ℝ}^{n_{\text{z}}}\times {ℝ}^{n_{\text{y}}}\mid V_{\epsilon }(t,z,y)\le 0\right\}. \end{gathered}$$ (3)

D. Backward reachable tubes and backward avoid tubes

The BRS is the set of states from which the controller can ensure the system will be in the target set at the final time. We may also be interested in the set of states from which the controller can ensure the system will be in the target set by or until the final time, known as the Backward Reachable Tube (BRT) and the Backward Stay Tube (BST), respectively.

For each $\epsilon >0$ and $t<0$, we let

$$\begin{gathered} \text{BR}{\text{T}}_{\text{ε}}(t)=\{(z,y)\in {ℝ}^{n_{\text{z}}}\times {ℝ}^{n_{\text{y}}}\mid \\ \text{∀}\gamma \in {\text{Γ}}_{[t,0)}\exists \mathbf{\text{u}}\in {\mathbb{U}}_{[t,0)}\exists s\in [t,0]{\mathbf{\text{z}}}_{t,z,y}^{\epsilon ,\mathbf{\text{u}},\gamma [\mathbf{\text{u}}]}(s)\in 𝒮\}, \end{gathered}$$

$$\begin{gathered} \text{BS}{\text{T}}_{\text{ε}}(t)=\{(z,y)\in {ℝ}^{n_{\text{z}}}\times {ℝ}^{n_{\text{y}}}\mid \\ \text{∀}\gamma \in {\text{Γ}}_{[t,0)}\exists \mathbf{\text{u}}\in {\mathbb{U}}_{[t,0)}\text{∀}s\in [t,0]{\mathbf{\text{z}}}_{t,z,y}^{\epsilon ,\mathbf{\text{u}},\gamma [\mathbf{\text{u}}]}(s)\in 𝒮\}. \end{gathered}$$

For convenience, we also set $\text{BR}{\text{T}}_{\text{ε}}(0)=\text{BS}{\text{T}}_{\text{ε}}(0)=𝒮$. Note that for each $t\le 0$ and $s\in [t,0]$

$$\text{BR}{\text{S}}_{\text{ε}}(s)\subseteq \text{BR}{\text{T}}_{\epsilon }(t),\text{and}\text{BS}{\text{T}}_{\text{ε}}(t)\subseteq \text{BR}{\text{S}}_{\text{ε}}(s).$$ (4)

E. Reduced model

We consider as a candidate for approximating the SP system the following “reduced” model:

$$\dot{\mathbf{\text{z}}}=f(\mathbf{\text{z}},\mathbf{\text{u}},\mathbf{\text{d}})-M(\mathbf{\text{z}})g(\mathbf{\text{z}},\mathbf{\text{u}},\mathbf{\text{d}}).$$ (5)

For each $t<0$, $z\in {ℝ}^{n_{\text{z}}}$, $\mathbf{\text{u}}\in {\mathbb{U}}_{[t,0)}$, and $\mathbf{\text{d}}\in {\mathbb{D}}_{[t,0)}$ we let ${\overline{\mathbf{\text{z}}}}_{t,z}^{\mathbf{\text{u}},\text{d}}$ be the unique Carathéodory solution of (5) for which ${\overline{\mathbf{\text{z}}}}_{t,z}^{\mathbf{\text{u}},\text{d}}(t)=z$ (existence and uniqueness of this solution follows from Assumption 1; see Theorem 1.2.1 in [26]). We then define the “reduced” value function $\overline{V}:(-\infty ,0]\times {ℝ}^{n_{\text{z}}}\to ℝ$ by

$$\overline{V}(t,z)=\{\begin{array}{cc}{\text{sup}}_{\gamma \in {\text{Γ}}_{[t,0)}}{\text{inf}}_{\mathbf{\text{u}}\in {\mathbb{U}}_{[t,0)}}\ell \left({\overline{\mathbf{\text{z}}}}_{t,z}^{\mathbf{\text{u}},\gamma [\mathbf{\text{u}}]}(0)\right)\hfill & t<0,\hfill \\ \ell (z)\hfill & t=0.\hfill \end{array}$$

We now explain how to use the reduced value function to obtain bounds on the BRS, BRT, and BST of the SP system.

III. RESULTS

The main result is given by Theorem 1, which uses level sets of the reduced value function $\overline{V}$ to provide inner and outer approximations of $\text{BR}{\text{S}}_{\text{ε}}(t)$ when $\epsilon$ is sufficiently small. First, a lemma (see Appendix for its proof):

Lemma 1:

Suppose Assumptions 1–3 hold. Then for any $𝒯\subset (-\infty ,0]$, $𝒵\subset {ℝ}^{n_{\text{z}}}$, and $𝒴\subset {ℝ}^{n_{\text{y}}}$ all non-empty and compact,

$$\underset{(t,z,y)\in 𝒯\times 𝒵\times 𝒴}{\sup }\left|V_{\epsilon }(t,z,y)-\overline{V}(t,z)\right|\stackrel{\epsilon \to 0^{+}}{\to }0.$$

Theorem 1:

Suppose Assumptions 1–3 hold. Let $\eta >0$, and let $𝒯\subset (-\infty ,0]$, $𝒵\subset {ℝ}^{n_{\text{z}}}$, and $𝒴\subset {ℝ}^{n_{\text{y}}}$ each be compact. There exists an ${\epsilon }_0>0$ such that

$$\begin{gathered} \{z\in 𝒵\mid \overline{V}(t,z)<-\eta \}\times 𝒴\subseteq \text{BR}{\text{S}}_{\epsilon }(t)\cap (𝒵\times 𝒴) \\ \subseteq \{z\in 𝒵\mid \overline{V}(t,z)<+\eta \}\times 𝒴 \end{gathered}$$ (6)

for all $\epsilon \in \left(0,{\epsilon }_0\right)$ and $t\in 𝒯$.

Proof: By Lemma 1, we can choose ${\epsilon }_0>0$ such that for all $\epsilon \in \left(0,{\epsilon }_0\right)$, $t\in 𝒯$, $z\in 𝒵$, and $y\in 𝒴$,

$$\left|V_{\epsilon }(t,z,y)-\overline{V}(t,z)\right|<\eta .$$

It follows that for each such $\epsilon$, $t$, $z$, and $y$,

$$\begin{gathered} \{z\in 𝒵\mid \overline{V}(t,z)<-\eta \}\times 𝒴 \\ =\{(z,y)\in 𝒵\times 𝒴\mid \overline{V}(t,z)<-\eta \} \\ \subseteq \left\{(z,y)\in 𝒵\times 𝒴\mid V_{\epsilon }(t,z,y)<0\right\} \\ \subseteq \left\{(z,y)\in 𝒵\times 𝒴\mid V_{\epsilon }(t,z,y)\le 0\right\} \\ \subseteq \{(z,y)\in 𝒵\times 𝒴\mid \overline{V}(t,z)<+\eta \} \\ =\{z\in 𝒵\mid \overline{V}(t,z)<+\eta \}\times 𝒴. \end{gathered}$$

The result then follows from (3). ◼

By setting $𝒯=[t,0]$, we can also use Theorem 1 and (4) to get bounds on the BRT and BST of the SP system:

Corollary 1:

Suppose Assumptions 1–3 hold. Let $\eta >0$ and $t\le 0$, and let $𝒵\subset {ℝ}^{n_{\text{z}}}$, and $𝒴\subset {ℝ}^{n_{\text{y}}}$ both be compact. There exists an ${\epsilon }_0>0$ such that for all $\epsilon \in \left(0,{\epsilon }_0\right)$

$$\left\{z\in 𝒵\mid \underset{s\in [t,0]}{\min }\overline{V}(s,z)<-\eta \right\}\times 𝒴\subseteq \text{BR}{\text{T}}_{\epsilon }(t)\cap (𝒵\times 𝒴),$$

and

$$\text{BS}{\text{T}}_{\text{ε}}(t)\cap (𝒵\times 𝒴)\subseteq \left\{z\in 𝒵\mid \underset{s\in [t,0]}{\text{max}}\overline{V}(s,z)<+\eta \right\}\times 𝒴.$$

IV. EXAMPLES

A. Genetic circuit with negative feedback

We first consider a model of a simple genetic circuit engineered in a bacterium (Figure 1). This circuit contains a gene G, which produces a transcription factor T. A kinase K expressed naturally in the cell chemically modifies T into its active form ${\text{T}}^{\ast }$ via a reaction known as phosphorylation. ${\text{T}}^{\ast }$ can then bind to the promoter region of G to down-regulate production of T, creating a negative feedback loop. Additionally, an inhibitor molecule I can be added by the user to the cellular environment to up-regulate the production rate of T by reducing the ability of ${\text{T}}^{\ast }$ to bind to the promoter of G. Because the foreign circuit pulls away machinery (e.g. RNAP, ribosomes, energy) used in other functions of the cell, quantities such as the growth rate of the cell, concentration of K, and production rate of T from G are known to fluctuate in a manner that is generally difficult to predict. We assume the transcription factor is fluorescently tagged so that its concentration can be observed by the user.

Fig. 1.

Depiction of a bacterium, engineered with the circuit of interest. Solid arrow represents chemical conversion of T to ${\text{T}}^{\ast }$ via K; dashed arrow represents down-regulation of T production by ${\text{T}}^{\ast }$ via binding to the promoter of G; dotted arrows represent activity modulation by control and disturbance signals. In particular, $\mathbf{\text{u}}$ (representing the inducer I) modulates the ability of ${\text{T}}^{\ast }$ to bind to the promoter of G, ${\mathbf{\text{d}}}_1$ modulates kinase activity, ${\mathbf{\text{d}}}_2$ modulates the cell growth rate, and ${\mathbf{\text{d}}}_3$ modulates the activity of G.

When the rate of phosphorylation is much faster than the rate of cell growth, we can model this circuit as follows:

$$\dot{\mathbf{\text{z}}}=\alpha {\mathbf{\text{d}}}_1\mathbf{\text{y}}-{\mathbf{\text{d}}}_2\mathbf{\text{z}},$$ (7)

$$\epsilon \dot{\mathbf{\text{y}}}={\mathbf{\text{d}}}_3\frac{{\mathbf{\text{u}}}^2}{{\mathbf{\text{u}}}^2+{\mathbf{\text{z}}}^2}-{\mathbf{\text{d}}}_1\mathbf{\text{y}}.$$ (8)

In the above, $\mathbf{\text{z}}$ represents the (normalized) concentration of ${\text{T}}^{\ast }$ and $\mathbf{\text{y}}$ represents the (normalized) concentration of T, respectively. The control $\mathbf{\text{u}}$ represents the (normalized) concentration of I, and the disturbances ${\mathbf{\text{d}}}_i$ capture the effects of the circuit on the natural cellular processes. The parameter $\epsilon \ll 1$ quantifies the separation between the timescale at which the phosphorylation reaction occurs and the timescale on which cell growth, and the parameter $\alpha$ is a positive constant related to both the basal kinetic rates of the various processes and the concentrations of K and G in the cell.

The corresponding reduced model for this system is

$$\dot{\mathbf{\text{z}}}=\alpha {\mathbf{\text{d}}}_3\frac{{\mathbf{\text{u}}}^2}{{\mathbf{\text{u}}}^2+{\mathbf{\text{z}}}^2}-{\mathbf{\text{d}}}_2\mathbf{\text{z}}.$$ (9)

The BRS of the SP model (7)–(8), along with the inner and outer bounds from Theorem 1 are shown in Figure 2 for large and small values of $\epsilon$. Note that when $\epsilon$ is not sufficiently small, the bounds do not necessarily hold. However, when $\epsilon$ is sufficiently small, the bounds hold and the value function appears almost independent of $y$.

Fig. 2.

Contour plots at time $t=-0.5$ of the value function $V_{\epsilon }(t,z,y)$ of the SP system (7)–(8). The target set is $𝒮=(0.25,0.75)$, and the terminal payoff function is taken to be $\ell (z)=\min \{10(|z-0.5|-0.25),3\}$. The intersection of this system’s BRS with the compact set $𝒵\times 𝒴={[0,1]}^2$ is the region inside the black curve. The inner and outer bounds from (6), with $\eta =0.1$, are the regions inside the yellow and magenta curves, respectively. Parameters were $𝒰=[0.1,1]$, $𝒟={[0.5,2]}^3$, $\alpha =1$. (Top) Results for $\epsilon =1$. Note that the bounds do not hold because $\epsilon$ is not sufficiently small. (Bottom) Results for $\epsilon =0.01$. Note that the bounds now do hold as $\epsilon$ is sufficiently small, so the system is approximately one-dimensional.

B. Metabolic reaction network in a growing population

We next consider a model of a metabolic reaction network (MRN) in a growing population of cells. In particular, a user supplies the population with a molecule m₁, which is rapidly metabolized by the cells into other intermediate metabolites ${\text{m}}_i(i=2\dots N)$. These intermediates are processed into a molecule p, which is required for cell growth. Such an MRN is shown in Figure 3 (Top) with $N=20$.

Fig. 3.

(Top) Example MRN, here with $N=20$. Nodes in the network represent metabolites, and edges represent reactions. The upstream molecule is m₁ (coral), which is rapidly converted in metabolites that are themselves ultimately metabolized into the downstream molecule p (cyan). Edge thickness is proportional to the reaction’s rate coefficient. If $T$ is the adjacency matrix of the graph, with $T_{ij}$ being the weight of the edge from node $j$ to $i$, one obtains the corresponding matrix $A_{\text{MRN}}$ in (13) by taking the submatrix of $T$ corresponding to all nodes other than p and subtracting from each diagonal element the corresponding column sum. One obtains the corresponding matrix $C_{\text{MRN}}$ in (10) by taking the row of $T$ corresponding to node p and subsequently eliminating the element of this row corresponding to node p. (Bottom) Contour plot of the value function $\overline{V}$ for the reduced model (14)–(16), in which the metabolic network is as shown above. Here $t=-3$, $𝒰={[0,1]}^2$, $𝒟=[0.9,1.1]$, and the weights of the edges in the metabolic network were sampled from a unit uniform distribution. Only the $z_1=0$ slice of the value function is shown. The target set $𝒮$ is the interior of the solid grey line, while the sub-level sets of the reduced value function for levels $\pm \eta =\pm 0.5$ lie below the dashed magenta and dotted yellow lines, respectively. Trajectories of the SP dynamics (10)–(12) with $\epsilon =0.01$ are shown as a white lines, with the initial state marked with an ◯, and the final with an $\times$. The initial states are ${\mathbf{\text{z}}}_1(t)=0$, ${\mathbf{\text{z}}}_2(t)=0.025$, ${\mathbf{\text{z}}}_3(t)=0.1$, $\mathbf{\text{y}}(t)=(0,\dots ,0)$ and ${\mathbf{\text{z}}}_1(t)=0$, ${\mathbf{\text{z}}}_2(t)=0.15$, ${\mathbf{\text{z}}}_3(t)=0.1$, $\mathbf{\text{y}}(t)=(0,\dots ,0)$. The optimal control law was chosen based on the value function computed for the reduced model, and the disturbance was chosen as a random signal of points uniformly sampled from $𝒟$. The target is $𝒮=ℝ\times (.4,.6)\times (.4,.6)$ and the payoff function is $\ell (z)=\min \{\text{max}\{10\left(z_2-0.5\right)-1$, $10\left(z_3-0.5\right)-1\},4\}$.

Each cell additionally produces a separate molecule q, whose production rate can be also controlled (e.g. by optogenetics). Ultimately, the user is interested in ensuring both the number of cells in the population and the amount of q produced are between some desired bounds at the final time.

Under the assumption that the rate of the metabolic reactions are much faster than the maximal rate of cell growth, we can consider the following model for this system:

$${\dot{\mathbf{\text{z}}}}_1=C_{\text{MRN}}{\mathbf{\text{d}}}_1\mathbf{\text{y}}-\frac{{\mathbf{\text{z}}}_1}{{\mathbf{\text{z}}}_1+1}{\mathbf{\text{z}}}_1$$ (10)

$${\dot{\mathbf{\text{z}}}}_2=\left(\frac{{\mathbf{\text{z}}}_1}{{\mathbf{\text{z}}}_1+1}-{\mathbf{\text{z}}}_2\right){\mathbf{\text{z}}}_2$$ (11)

$${\dot{\mathbf{\text{z}}}}_3={\mathbf{\text{z}}}_2{\mathbf{\text{u}}}_2$$ (12)

$$\epsilon \dot{\mathbf{\text{y}}}=A_{\text{MRN}}{\mathbf{\text{d}}}_1\mathbf{\text{y}}+{\widehat{e}}_1{\mathbf{\text{u}}}_1,$$ (13)

where $A_{\text{MRN}}\in {ℝ}^{n_{\text{y}}\times n_{\text{y}}}$, $C_{\text{MRN}}\in {ℝ}^{1\times n_{\text{y}}}$ and ${\widehat{e}}_1={\left(10\dots 0\right)}^{\top }\in {ℝ}^{n_{\text{y}}\times 1}$.

In this model, the $i$-th entry of $\mathbf{\text{y}}$ is the concentration of molecule ${\text{m}}_i(i=1,\dots ,N)$, ${\mathbf{\text{z}}}_1$ is the concentration of molecule p, ${\mathbf{\text{z}}}_2$ is the population size, ${\mathbf{\text{z}}}_3$ is the total number of copies of molecule q in the population, ${\mathbf{\text{u}}}_1$ is the rate at which the user supplies ${\text{m}}_1$, ${\mathbf{\text{u}}}_2$ is the signal used to control production of q, and ${\mathbf{\text{d}}}_1$ is a possible disturbance to the population’s global metabolic efficiency (with all quantities normalized). Here, $A_{\text{MRN}}$ and $C_{\text{MRN}}$ are derived from the (weighted) adjacency matrix of the MRN (see caption of Figure 3 for details), and $\epsilon$ is a parameter which is small when the metabolic reactions rates are much faster than the maximal cell growth rate. Importantly, the matrix $A_{\text{MRN}}$ is invertible because all metabolites ${\text{m}}_1,\dots ,{\text{m}}_N$ are eventually converted into molecule p.

The reduced model for the above SP system is

$${\dot{\mathbf{\text{z}}}}_1=C_{\text{MRN}}{A}_{\text{MRN}}^{-1}{\widehat{e}}_1{\mathbf{\text{u}}}_1-\frac{{\mathbf{\text{z}}}_1}{{\mathbf{\text{z}}}_1+1}{\mathbf{\text{z}}}_1$$ (14)

$${\dot{\mathbf{\text{z}}}}_2=\left(\frac{{\mathbf{\text{z}}}_1}{{\mathbf{\text{z}}}_1+1}-{\mathbf{\text{z}}}_2\right){\mathbf{\text{z}}}_2$$ (15)

$${\dot{\mathbf{\text{z}}}}_3={\mathbf{\text{z}}}_2{\mathbf{\text{u}}}_2$$ (16)

Remark 1:

The natural domain for (10)–(13) is $\left({\mathbf{\text{z}}}_1(\tau ),{\mathbf{\text{z}}}_2(\tau ){\mathbf{\text{z}}}_3(\tau ),\mathbf{\text{y}}(\tau )\right)\in [0,\infty )\times [0,1]\times [0,\infty )\times {ℝ}^N$ However, the results in this paper are for systems defined on all of ${ℝ}^{n_{\text{z}}+n_{\text{y}}}$. Such a discrepancy is of no concern because these dynamics can be extended to all of ${ℝ}^{n_{\text{z}}+n_{\text{y}}}$ in such a way that Assumption 1 is satisfied.

Level sets of the reduced value function are shown in Figure 3 (Bottom). While it is generally intractable to compute the value function for a model with 23 states (as in the SP model) for ground-truth comparison, we can demonstrate the utility of our method by using the reduced value function to provide a “best guess” for the optimal control: $u^{\ast }(z,y,t)=\arg {\min }_{u\in 𝒰}{\text{max}}_{d\in 𝒟}{\text{∇}}_z\overline{V}{(z,t)}^{\top }F(z,u,d)$, where $F$ is the dynamics function of (14)–(16) [note that the above control is $y$-independent]. Indeed, by using this optimal control law estimate, we observe that two simulated trajectories of the SP model (10)–(13), do and do not reach the target in accordance with the bounds in Theorem 1.

V. CONCLUSION

In this work, we identified a class of SP differential games that can be readily reduced by leveraging the results in [10], and we related the BRS, BRT, and BST of the SP differential game to the reduced value function. We then demonstrated the particular applicability of these results in analyzing biological systems, which often have dynamics that fit the identified class of systems. In the second example, we empirically found that using the reduced value function to obtain an optimal control law allowed one to properly control the state of the SP model. Future work may include studying when we can guarantee this procedure will well approximate the optimal control law of the SP differential game when the parameter $\epsilon$ is sufficiently small.

Appendix

For convenience throughout this appendix, we will let $F:{ℝ}^{n_{\text{z}}}\times 𝒰\times 𝒟\to {ℝ}^{n_{\text{z}}}$ and $H:{ℝ}^{n_{\text{z}}}\times {ℝ}^{n_{\text{z}}}\to ℝ$ be given by

$$F(z,u,d)=f(z,u,d)-M(z)g(z,u,d),$$

$$H(z,\lambda )=\underset{u\in 𝒰}{\min }\underset{d\in 𝒟}{\text{max}}{\lambda }^{\top }F(z,u,d)=\underset{d\in 𝒟}{\text{max}}\underset{u\in 𝒰}{\min }{\lambda }^{\top }F(z,u,d).$$

Given $t_0,\dots ,t_n\in ℝ$ such that $t_0<\cdots <t_n$, and given functions ${\mathbf{\text{h}}}_1:\left[t_0,t_1\right)\to {ℝ}^m,\dots ,{\mathbf{\text{h}}}_n:\left[t_{n-1},t_n\right)\to {ℝ}^m$, we define the function $\left[{\mathbf{\text{h}}}_1,\dots ,{\mathbf{\text{h}}}_n\right):\left[t_0,t_n\right)\to {ℝ}^m$ such that $\left[{\mathbf{\text{h}}}_1,\dots ,{\mathbf{\text{h}}}_n\right)(s)={\mathbf{\text{h}}}_j(s)$ whenever $s\in \left[t_{j-1},t_j\right)$. For each $[t,s)\subset ℝ$, we also let ${\mathbb{U}}_{[t,s)}\left({\mathbb{D}}_{[t,s)}\right)$ be the set of measurable functions from $[t,s)$ to $𝒰([t,s)$ to $𝒟$).

A. Boundary Layer Model

For each $z\in {ℝ}^{n_{\text{z}}}$, we consider the following “boundary-layer” model:

$$\dot{\mathbf{\text{y}}}=f(z,\mathbf{\text{u}},\mathbf{\text{d}})+A(z,\mathbf{\text{u}},\mathbf{\text{d}})\mathbf{\text{y}}.$$ (17)

For each $t<0$, $y\in {ℝ}^{n_{\text{y}}}$, $z\in {ℝ}^{n_{\text{z}}}$, $\mathbf{\text{u}}\in {\mathbb{U}}_{[t,0)}$, and $\mathbf{\text{d}}\in {\mathbb{D}}_{[t,0)}$, we let ${\widetilde{\mathbf{\text{y}}}}_{t,y}^{z,\mathbf{\text{u}},\mathbf{\text{d}}}:[t,0]\to {ℝ}^{n_{\text{y}}}$ be the unique Carathéodory solution of (17) satisfying ${\widetilde{\mathbf{\text{y}}}}_{t,y}^{z,\mathbf{\text{u}},\mathbf{\text{d}}}(t)=y$.

For each $t<0$, $y\in {ℝ}^{n_{\text{y}}}$, $z\in {ℝ}^{n_{\text{z}}}$, and $\lambda \in {ℝ}^{n_{\text{z}}}$, we define the payoff functional ${\tilde{J}}_{t,y}^{z,\lambda }:{\mathbb{U}}_{[t,0)}\times {\mathbb{D}}_{[t,0)}\to ℝ$ by

$$\begin{gathered} {\tilde{J}}_{t,y}^{z,\lambda }(\mathbf{\text{u}},\mathbf{\text{d}})=\frac{1}{\left|t\right|}{\int }_t^0{\lambda }^{\top }[f(z,\mathbf{\text{u}}(\tau ),\mathbf{\text{d}}(\tau ))+ \\ M(z)A(z,\mathbf{\text{u}}(\tau ),\mathbf{\text{d}}(\tau )){\widetilde{\mathbf{\text{y}}}}_{t,y}^{z,\mathbf{\text{u}},\mathbf{\text{d}}}(\tau )]\text{d}\tau . \end{gathered}$$

Moreover, for each $t<0$, $y\in {ℝ}^{n_{\text{y}}}$, $z\in {ℝ}^{n_{\text{z}}}$, and $\lambda \in {ℝ}^{n_{\text{z}}}$, and for each partition $P$ of the interval $[t,0)$ into the subintervals $\left[t_0,t_1\right),\dots ,\left[t_{r-1},t_r\right)$ (where $t_0=t$ and $t_r=0$), we define the following upper and lower value functions for the boundary-layer model:

$$\begin{gathered} {\tilde{V}}_{z,\lambda }^{P,+}(t,y)=\underset{{\mathbf{\text{u}}}_1\in {\mathbb{U}}_1}{\inf }\underset{\begin{array}{c}{\mathbf{\text{d}}}_1\in {\mathbb{D}}_1\end{array}}{\sup }\dots \underset{{\mathbf{\text{u}}}_r\in {\mathbb{U}}_r}{\inf }\underset{{\mathbf{\text{d}}}_r\in {\mathbb{D}}_r}{\sup } \\ {\tilde{J}}_{t,y}^{z,\lambda }\left(\left[{\mathbf{\text{u}}}_1,\dots ,{\mathbf{\text{u}}}_r\right),\left[{\mathbf{\text{d}}}_1,\dots ,{\mathbf{\text{d}}}_r\right)\right), \end{gathered}$$ (18)

$$\begin{gathered} {\tilde{V}}_{z,\lambda }^{P,-}(t,y)=\underset{{\mathbf{\text{d}}}_1\in {\mathbb{D}}_1}{\sup }\underset{{\mathbf{\text{u}}}_1\in {\mathbb{U}}_1}{\inf }\dots \underset{{\mathbf{\text{d}}}_r\in {\mathbb{D}}_r}{\sup }\underset{{\mathbf{\text{u}}}_r\in {\mathbb{U}}_r}{\inf } \\ {\tilde{J}}_{t,y}^{z,\lambda }\left(\left[{\mathbf{\text{u}}}_1,\dots ,{\mathbf{\text{u}}}_r\right),\left[{\mathbf{\text{d}}}_1,\dots ,{\mathbf{\text{d}}}_r\right)\right), \end{gathered}$$ (19)

where ${\mathbb{U}}_j={\mathbb{U}}_{\left[t_{j-1},t_j\right)}$ and ${\mathbb{D}}_j={\mathbb{D}}_{\left[t_{j-1},t_j\right)}$ for each $j$.

B. Intermediate Lemmas

Lemma 2:

Let $z\in {ℝ}^{n_{\text{z}}}$. There are constants $\alpha$, $\kappa >0$ such that for each $t<0$, $\mathbf{\text{u}}\in {\mathbb{U}}_{[t,0)}$, and $\mathbf{\text{d}}\in {\mathbb{D}}_{[t,0)}$,

$$\Vert e^{{\int }_t^{0}A(z,\mathbf{\text{u}}(r),\text{d}(r))\text{d}r}\Vert \le \alpha e^{\kappa t}.$$ (20)

Proof: Let $\text{Λ}:𝒰\times 𝒟\to ℝ$ be such that $\text{Λ}\left(u,d\right)$ is the largest eigenvalue of $A{(z,u,d)}^{\top }P+PA(z,u,d)$, where $P$ is as in Assumption 2. Then $\text{Λ}$ is continuous, so by Assumption 2, there is some $\nu >0$ such that $\text{Λ}(u,d)\le -\nu$ for each $u\in 𝒰$ and $d\in 𝒟$.

Fix some $t<0$, $\mathbf{\text{u}}\in {\mathbb{U}}_{[t,0)}$, and $\mathbf{\text{d}}\in {\mathbb{D}}_{[t,0)}$. Let $\mathbf{\text{w}}:[t,0]\to {ℝ}^{n_{\text{y}}}$ be the Carathéodory solution to

$$\dot{\mathbf{\text{w}}}=A(z,\mathbf{\text{u}},\mathbf{\text{d}})\mathbf{\text{w}},\mathbf{\text{w}}(t)=w.$$

A standard argument (see e.g. the proof of Theorem 4.10 in [27]) using the Lyapunov function $V(w)=w^{\top }Pw$ shows

$$\frac{\text{d}}{\text{d}\tau }[V(\mathbf{\text{w}}(\tau ))]=-2\kappa V(\mathbf{\text{w}}(\tau ))-\mu (\tau )$$

for a.e. $\tau \in (t,0)$, where $\kappa :=\frac{\nu }{2{\lambda }_{\text{max}}(P)}$ and $\mu (\cdot )\ge 0$.

Since $\mathbf{\text{w}}$ is absolutely continuous with a compact domain, and since $V$ is locally Lipschitz, then $V\circ \mathbf{\text{w}}$ is absolutely continuous, so that

$$V(\mathbf{\text{w}}(0))=V(w)e^{2\kappa t}-{\int }_t^0\mu (s)e^{2\kappa s}\text{d}s\le V(w)e^{2\kappa t},$$

and thus $\parallel \mathbf{\text{w}}(t)\parallel \le \parallel w\parallel \alpha e^{\kappa t}$, where $\alpha =\sqrt{\frac{{\lambda }_{\text{max}}(P)}{{\lambda }_{\min }(P)}}$ (again see the proof of Theorem 4.10 in [27] for details). But since $w\in {ℝ}^{n_{\text{y}}}$ was arbitrary and $\mathbf{\text{w}}(t)=e^{{\int }_t^{0}A(z,\mathbf{\text{u}}(r),\mathbf{\text{d}}(r))}w$, the result follows.

Lemma 3:

Let $y\in {ℝ}^{n_{\text{y}}}$, $z\in {ℝ}^{n_{\text{z}}}$, and $\lambda \in {ℝ}^{n_{\text{z}}}$. There is a $\beta >0$ such that for all $t<0$, $\mathbf{\text{u}}\in {\mathbb{U}}_{[t,0)}$, and $\mathbf{\text{d}}\in {\mathbb{D}}_{[t,0)}$,

$$\left|{\tilde{J}}_{t,y}^{z,\lambda }(\mathbf{\text{u}},\mathbf{\text{d}})-\frac{1}{\left|t\right|}{\int }_t^0{\lambda }^{\top }F(z,\mathbf{\text{u}}(\tau ),\mathbf{\text{d}}(\tau ))\text{d}\tau \right|\le \frac{\beta }{\left|t\right|}.$$

Proof: Fix some $t<0$ and some $\mathbf{\text{u}}\in {\mathbb{U}}_{[t,0)}$ and $\mathbf{\text{d}}\in {\mathbb{D}}_{[t,0)}$. Then ${\tilde{J}}_{t,y}^{z,\lambda }(\mathbf{\text{u}},\mathbf{\text{d}})=I_1+I_2+I_3$, where

$$I_1=\frac{1}{\left|t\right|}{\int }_t^0{\lambda }^{\top }f(z,\mathbf{\text{u}}(\tau ),\mathbf{\text{d}}(\tau ))\text{d}\tau ,$$ (21)

$$\begin{gathered} I_2=\frac{1}{\left|t\right|}{\int }_t^0{\int }_t^{\tau }{\lambda }^{\top }M(z)A(z,\mathbf{\text{u}}(\tau ),\mathbf{\text{d}}(\tau ))e^{{\int }_{\tau }^{0}A(z,\mathbf{\text{u}}(r),\mathbf{\text{d}}(r))\text{d}r} \\ g(z,\mathbf{\text{u}}(s),\mathbf{\text{d}}(s))\text{d}s\text{d}\tau , \end{gathered}$$ (22)

$$I_3=\frac{1}{\left|t\right|}{\int }_t^0{\lambda }^{\top }{e}^{{\int }_{\tau }^{0}A(z,\mathbf{\text{u}}(r),\mathbf{\text{d}}(r))\text{d}r}y\text{d}\tau .$$ (23)

Choose $\alpha >0$ and $\kappa >0$ as in Lemma 2. Then

$$\left|I_3\right|\le \frac{\alpha \parallel \lambda \parallel \parallel y\parallel }{\left|t\right|}{\int }_t^0{e}^{\kappa \tau }\text{d}\tau =\frac{\alpha \parallel \lambda \parallel \parallel y\parallel }{\kappa \left|t\right|}\left(1-e^{\kappa t}\right).$$

By The Fundamental Theorem of Calculus for Lebesgue Integrals (see Theorem 3.35 in [28]), for a.e. $\tau \in (t,0)$,

$$\frac{\text{d}}{\text{d}\tau }{\int }_{\tau }^{0}A(z,\mathbf{\text{u}}(r),\mathbf{\text{d}}(r))\text{d}r=-A(z,\mathbf{\text{u}}(\tau ),\mathbf{\text{d}}(\tau )),$$

so that

$$\frac{\text{d}}{\text{d}\tau }{e}^{{\int }_{\tau }^{0}A(z,\mathbf{\text{u}}(r),\mathbf{\text{d}}(r))\text{d}r}=-A(z,\mathbf{\text{u}}(\tau ),\mathbf{\text{d}}(\tau ))e^{{\int }_{\tau }^{0}A(z,\mathbf{\text{u}}(r),\mathbf{\text{d}}(r))}.$$

Since ${\int }_{\tau }^{0}A(z,\mathbf{\text{u}}(r),\mathbf{\text{d}}(r))\text{d}r$ is absolutely continuous as a function of $\tau$ on $[t,0]$, and since the exponential of an absolutely continuous function on a compact interval is absolutely continuous, then $e^{{\int }_{\tau }^{0}A(z,\mathbf{\text{u}}(r),\mathbf{\text{d}}(r))}$ is also absolutely continuous as a function of $\tau$ on the interval $[t,0]$. Again by the The Fundamental Theorem of Calculus for Lebesgue Integrals, for each $s\in [t,0]$

$$\begin{gathered} e^{{\int }_s^{0}A(z,\mathbf{\text{u}}(r),\mathbf{\text{d}}(r))\text{d}r}-I= \\ {\int }_s^{0}A(z,\mathbf{\text{u}}(\tau ),\mathbf{\text{d}}(\tau ))e^{{\int }_{\tau }^{0}A(z,\mathbf{\text{u}}(r),\mathbf{\text{d}}(r))\text{d}r}\text{d}\tau , \end{gathered}$$

where $I$ is the identity matrix in ${ℝ}^{n_{\text{y}}\times n_{\text{y}}}$. By changing the order of integration in (22),

$$\begin{gathered} I_2=\frac{1}{\left|t\right|}{\int }_t^0{\int }_s^0{\lambda }^{\top }M(z)A(z,\mathbf{\text{u}}(\tau ),\mathbf{\text{d}}(\tau ))e^{{\int }_{\tau }^{0}A(z,\mathbf{\text{u}}(r),\mathbf{\text{d}}(r))\text{d}r} \\ g(z,\mathbf{\text{u}}(\tau ),\mathbf{\text{d}}(\tau ))\text{d}\tau \text{d}s \\ =\frac{1}{\left|t\right|}{\int }_t^0{\lambda }^{\top }M(z)\left[e^{{\int }_s^{0}A(z,\mathbf{\text{u}}(r),\mathbf{\text{d}}(r))\text{d}r}-I\right] \\ g(z,\mathbf{\text{u}}(s),\mathbf{\text{d}}(s))\text{d}s. \end{gathered}$$

Note that

$$\begin{gathered} \left|\frac{1}{\left|t\right|}{\int }_t^0{\lambda }^{\top }M(z)e^{{\int }_s^{0}A(z,\mathbf{\text{u}}(r),\mathbf{\text{d}}(r))\text{d}r}g(z,\mathbf{\text{u}}(s),\mathbf{\text{d}}(s))\text{d}s\right|\le \\ \frac{\alpha \parallel \lambda \parallel \parallel M(z)\parallel B}{\left|t\right|}{\int }_t^0{e}^{\kappa s}\text{d}s=\frac{\alpha \parallel \lambda \parallel \parallel M(z)\parallel B}{\kappa \left|t\right|}\left(1-e^{\kappa t}\right), \end{gathered}$$

where $B:={\text{max}}_{u\in 𝒰}{\text{max}}_{d\in 𝒟}\parallel g(z,u,d)\parallel$. The lemma then follows with $\beta :=\frac{\alpha \parallel \lambda \parallel }{\kappa }(\parallel y\parallel +B\parallel M(z)\parallel )$.

Lemma 4:

Given $\lambda$, $z\in {ℝ}^{n_{\text{z}}}$ and $t<0$,

$$\begin{gathered} \underset{\mathbf{\text{u}}\in {\mathbb{U}}_{[t,0)}}{\inf }\underset{\mathbf{\text{d}}\in {\mathbb{D}}_{[t,0)}}{\sup }\frac{1}{\left|t\right|}{\int }_t^0{\lambda }^{\top }F(z,\mathbf{\text{u}}(\tau ),\mathbf{\text{d}}(\tau ))\text{d}\tau \\ =\underset{\mathbf{\text{d}}\in {\mathbb{D}}_{[t,0)}}{\sup }\underset{\mathbf{\text{u}}\in {\mathbb{U}}_{[t,0)}}{\inf }\frac{1}{\left|t\right|}{\int }_t^0{\lambda }^{\top }F(z,\mathbf{\text{u}}(\tau ),\mathbf{\text{d}}(\tau ))\text{d}\tau =H(z,\lambda ). \end{gathered}$$

Proof: Let $u^{\ast }\in \arg \underset{u\in 𝒰}{\min }\underset{d\in 𝒟}{\text{max}}{\lambda }^{\top }F(z,u,d)$, and let $d^{\ast }\in \arg \underset{d\in 𝒟}{\text{max}}\underset{u\in 𝒰}{\min }{\lambda }^{\top }F(z,u,d)$. Then

$$H(z,\lambda )=\underset{d\in 𝒟}{\text{max}}{\lambda }^{\top }F\left(z,u^{\ast },d\right)=\underset{u\in 𝒰}{\min }{\lambda }^{\top }F\left(z,u,d^{\ast }\right)$$ (24)

by Assumption 3. Observe that

$$\begin{gathered} \underset{\mathbf{\text{u}}\in {\mathbb{U}}_{[t,0)}}{\inf }\underset{\mathbf{\text{d}}\in {\mathbb{D}}_{[t,0)}}{\sup }\frac{1}{\left|t\right|}{\int }_t^0{\lambda }^{\top }F(z,\mathbf{\text{u}}(\tau ),\mathbf{\text{d}}(\tau ))\text{d}\tau \\ \le \underset{\mathbf{\text{d}}\in {\mathbb{D}}_{[t,0)}}{\sup }\frac{1}{\left|t\right|}{\int }_t^0{\lambda }^{\top }F\left(z,u^{\ast },\mathbf{\text{d}}(\tau )\right)\text{d}\tau \\ =H(z,\lambda )=\underset{\mathbf{\text{u}}\in {\mathbb{U}}_{[t,0)}}{\inf }\frac{1}{\left|t\right|}{\int }_t^0{\lambda }^{\top }F\left(z,\mathbf{\text{u}}(\tau ),d^{\ast }\right)\text{d}\tau \\ \le \underset{\mathbf{\text{d}}\in {\mathbb{D}}_{[t,0)}}{\sup }\underset{\mathbf{\text{u}}\in {\mathbb{U}}_{[t,0)}}{\inf }\frac{1}{\left|t\right|}{\int }_t^0{\lambda }^{\top }F(z,\mathbf{\text{u}}(\tau ),\mathbf{\text{d}}(\tau ))\text{d}\tau \\ \le \underset{\mathbf{\text{u}}\in {\mathbb{U}}_{[t,0)}}{\inf }\underset{\mathbf{\text{d}}\in {\mathbb{D}}_{[t,0)}}{\sup }\frac{1}{\left|t\right|}{\int }_t^0{\lambda }^{\top }F(z,\mathbf{\text{u}}(\tau ),\mathbf{\text{d}}(\tau ))\text{d}\tau \end{gathered}$$

where both equalities are justified by (24), and where the final inequality is justified by the standard minimax inequality.

Lemma 5:

Let $y\in {ℝ}^{n_{\text{y}}}$, $z\in {ℝ}^{n_{\mathrm{z}}}$, and $\lambda \in {ℝ}^{n_{\text{z}}}$. There is some $\beta >0$ such that for all $t<0$ and for each partition $P$ of the interval $[t,0)$,

$$\left|{\tilde{V}}_{z,\lambda }^{P,\pm }(t,y)-H(z,\lambda )\right|\le \beta /\left|t\right|.$$

Proof: Let ${\mathbb{U}}_1,\dots ,{\mathbb{U}}_r$ and ${\mathbb{D}}_1,\dots ,{\mathbb{D}}_r$ be as in the definitions of ${\tilde{V}}_{z,\lambda }^{P,\pm }$. First note that it follows from repeated use of the standard minimax inequality that

$$\begin{gathered} \underset{\mathbf{\text{d}}\in {\mathbb{D}}_{[t,0)}}{\sup }\underset{\mathbf{\text{u}}\in {\mathbb{U}}_{[t,0)}}{\inf }\frac{1}{\left|t\right|}{\int }_t^0{\lambda }^{\top }F(z,\mathbf{\text{u}}(\tau ),\mathbf{\text{d}}(\tau ))\text{d}\tau \\ =\underset{{\mathbf{\text{d}}}_1\in {\mathbb{D}}_1}{\sup }\dots \underset{{\mathbf{\text{d}}}_r\in {\mathbb{D}}_r}{\sup }\underset{{\mathbf{\text{u}}}_1\in {\mathbb{U}}_1}{\inf }\cdots \underset{{\mathbf{\text{u}}}_r\in {\mathbb{U}}_r}{\inf } \\ \frac{1}{\left|t\right|}{\int }_t^0{\lambda }^{\top }F\left(z,\left[{\mathbf{\text{u}}}_1,\dots ,{\mathbf{\text{u}}}_r\right)(\tau ),\left[{\mathbf{\text{d}}}_1,\dots ,{\mathbf{\text{d}}}_r\right)(\tau )\right)\text{d}\tau \\ \le \underset{{\mathbf{\text{u}}}_1\in {\mathbb{U}}_1}{\inf }\underset{{\mathbf{\text{d}}}_1\in {\mathbb{D}}_1}{\sup }\cdots \underset{{\mathbf{\text{u}}}_r\in {\mathbb{U}}_r}{\inf }\underset{{\mathbf{\text{d}}}_r\in {\mathbb{D}}_r}{\sup } \\ \frac{1}{\left|t\right|}{\int }_t^0{\lambda }^{\top }F\left(z,\left[{\mathbf{\text{u}}}_1,\dots ,{\mathbf{\text{u}}}_r\right)(\tau ),\left[{\mathbf{\text{d}}}_1,\dots ,{\mathbf{\text{d}}}_r\right)(\tau )\right)\text{d}\tau \\ \le \underset{{\mathbf{\text{d}}}_1\in {\mathbb{D}}_1}{\sup }\cdots \underset{{\mathbf{\text{d}}}_r\in {\mathbb{D}}_r}{\sup }\underset{{\mathbf{\text{u}}}_1\in {\mathbb{U}}_1}{\inf }\cdots \underset{{\mathbf{\text{u}}}_r\in {\mathbb{U}}_r}{\inf } \\ \frac{1}{\left|t\right|}{\int }_t^0{\lambda }^{\top }F\left(z,\left[{\mathbf{\text{u}}}_1,\dots ,{\mathbf{\text{u}}}_r\right)(\tau ),\left[{\mathbf{\text{d}}}_1,\dots ,{\mathbf{\text{d}}}_r\right)(\tau )\right)\text{d}\tau \\ =\underset{\mathbf{\text{u}}\in {\mathbb{U}}_{[t,0)}}{\inf }\underset{\mathbf{\text{d}}\in {\mathbb{D}}_{[t,0)}}{\sup }\frac{1}{\left|t\right|}{\int }_t^0{\lambda }^{\top }F(z,\mathbf{\text{u}}(\tau ),\mathbf{\text{d}}(\tau ))\text{d}\tau . \end{gathered}$$

It then follows from Lemma 4 that

$$\begin{gathered} H(z,\lambda )= \\ \underset{{\mathbf{\text{u}}}_1\in {\mathbb{U}}_1}{\inf }\underset{{\mathbf{\text{d}}}_1\in {\mathbb{D}}_1}{\sup }\dots \underset{{\mathbf{\text{u}}}_r\in {\mathbb{U}}_r}{\inf }\underset{{\mathbf{\text{d}}}_r\in {\mathbb{D}}_r}{\sup }\frac{1}{\left|t\right|}{\int }_t^0{\lambda }^{\top }F(z,\mathbf{\text{u}}(\tau ),\mathbf{\text{d}}(\tau ))\text{d}\tau . \end{gathered}$$ (25)

Then by (18) and (25) together with repeated use of the standard inequality for the absolute difference of suprema/infima

$$\begin{gathered} \left|{\tilde{V}}_{z,\lambda }^{P,\pm }(t,y)-H(z,\lambda )\right|\le \\ \underset{{\mathbf{\text{u}}}_1\in {\mathbb{U}}_1}{\sup }\underset{{\mathbf{\text{d}}}_1\in {\mathbb{D}}_1}{\sup }\dots \underset{{\mathbf{\text{u}}}_r\in {\mathbb{U}}_r}{\sup }\underset{{\mathbf{\text{d}}}_r\in {\mathbb{D}}_r}{\sup }\mid {\tilde{J}}_{t,y}^{z,\lambda }\left(\left[{\mathbf{\text{u}}}_1,\dots {\mathbf{\text{u}}}_r\right),\left[{\mathbf{\text{d}}}_1,\dots ,{\mathbf{\text{d}}}_r\right)\right) \\ -\frac{1}{\left|t\right|}{\int }_t^0{\lambda }^{\top }F\left(z,\left[{\mathbf{\text{u}}}_1,\dots {\mathbf{\text{u}}}_r\right)(\tau ),\left[{\mathbf{\text{d}}}_1,\dots ,{\mathbf{\text{d}}}_r\right)(\tau )\right)\text{d}\tau | \end{gathered}$$

The result follows from Lemma 3 (the result for ${\tilde{V}}_{z,\lambda }^{P,-}$ follows similarly).

C. Proof of Lemma 1

Having shown Lemma 5, Lemma 1 is now a mostly straightforward consequence of the main result in [10]:

Proof: [Proof of Lemma 1] Applying the main result (Theorem 4.2) in [10] to our setting requires verifying a number of assumptions, which we do now (unfortunately the assumptions are too extensive to reproduce in this text). We note that Assumption 1 in [10] follows from Lemma 5 in this text; Assumptions (A1)-(A3) in [10] follow from Assumption 1 in this text; Assumptions (B1)-(B3) in [10] follow from Theorem 6.2, 7.1, and 7.2 in [10] together with Assumption 2 in this text (as this assumption justifies Assumptions (C1) and (C2) in [10], which are needed in Theorems 6.1, 7.1, and 7.2). [Also note that by Lemma 5, the quantities $R_{\delta }^{up}$, $R_{\delta }^{lo}$, and $R$ in the hypotheses of Theorem 4.2 are all equal to the quantity $-H$ defined in this text.]

By [15] and [29], $\overline{V}$ is the unique continuous viscosity solution (in the sense described in Remark 4.1 of [10]) of the Hamilton-Jacobi PDE

$$\{\begin{array}{c}-{\partial }_{t}v(t,z)+H\left(z,{\nabla }_{z}v(t,z)\right)=0,\hfill \\ v(t,z)=\ell (z),\hfill \end{array}\begin{array}{l}(t,z)\in \text{Ω},\hfill \\ (t,z)\in \partial \text{Ω},\hfill \end{array}$$

where $\text{Ω}=(-\infty ,0)\times {ℝ}^{n_{\text{z}}}$ and $\partial \text{Ω}=\left\{0\right\}\times {ℝ}^{n_{\mathbf{\text{z}}}}$. Having verified the above assumptions, Theorem 4.2 in [10] implies $V_{\epsilon }(t,z,y)\to \overline{V}(t,z)$ as $\epsilon \to 0^{+}$ uniformly on compact sets in $(-\infty ,0]\times {ℝ}^{n_{\text{z}}}\times {ℝ}^{n_{\text{y}}}$, which is our desired result.

Remark 2:

We note two nuances in this proof which may catch the attention of the careful reader. First, the theory in [10] uses the Friedman definition of the value function (see [26]), whereas we use the Elliot-Kalton definitions for $V_{\epsilon }$ and $\overline{V}$. These definitions are indeed equivalent in our setting, because they are both the unique viscosity solutions to a corresponding Hamilton-Jacobi-Isaacs PDE by [15], [16], [29]. Second, the assumptions made in [15] and [16] on boundedness of the system dynamics virtually never hold for systems (1)–(2) and (5). Moreover, the definition of a viscosity solution used in these two early works postulates boundedness and uniform continuity of the solution, whereas the definition in [10] merely postulates continuity. It is known (and in fact even stated in [15]) that such strong assumptions are not required to characterize the value function as the unique viscosity solution of a Hamilton-Jacobi PDE. Indeed the “existence” proofs from [15] and [16] (i.e. that the value functions of a differential game is a [possibly unbounded and continuous but not uniformly so] viscosity solution of the corresponding Hamilton-Jacobi-Isaacs PDE) follow similarly under only Assumption 1 and continuity of the terminal payoff function $\ell$. On the other hand, the uniqueness result follows under these weaker assumptions and less restrictive definition of a viscosity solution from Theorem 2.5 in [29]. Both these observations are not new (c.f. Sections III.3.2, VIII.1, and the end of VIII in the classic text [30]).