Optimal Control in Combination Therapy for Heterogeneous Cell Populations with Drug Synergies

Abstract

Cell heterogeneity plays an important role in patient responses to drug treatments. In many cancers, it is associated with poor treatment outcomes. Many modern drug combination therapies aim to exploit cell heterogeneity, but determining how to optimise responses from heterogeneous cell populations while accounting for multi-drug synergies remains a challenge. In this work, we introduce and analyse a general optimal control framework that can be used to model the treatment response of multiple cell populations that are treated with multiple drugs that mutually interact. In this framework, we model the effect of multiple drugs on the cell populations using a system of coupled semi-linear ordinary differential equations and derive general results for the optimal solutions. We then apply this framework to three canonical examples and discuss the wider question of how to relate mathematical optimality to clinically observable outcomes, introducing a systematic approach to propose qualitatively different classes of drug dosing inspired by optimal control.

Introduction

Combination drug therapies are a mainstay of modern treatments for a wide range of diseases with cancer perhaps being the most notable example (Lieftink and Beijersbergen 2021; Marjanovic et al. 2013; Altschuler and Lani 2010; Hausser and Alon 2020; Crucitta et al. 2022; Proietto et al. 2023). The success of combination therapies relative to single-drug therapies stems from their ability to exploit drug synergies (Lieftink and Beijersbergen 2021), target a diverse cell population within a target tissue (Marjanovic et al. 2013; Altschuler and Lani 2010; Hausser and Alon 2020; Crucitta et al. 2022), and avoid the toxicity associated with high doses of a single drug (Lieftink and Beijersbergen 2021). However, designing combination treatments is not straightforward, owing to the difficulties in predicting not only the responses of different cells present in the target tissues to any individual drug, but also responses to all possible drug combinations. Making such predictions is all the more important in settings where the presence of multiple cell types – or the presence of cells with different features, such as where they are in the cell cycle – is clinically significant. For example, cell type heterogeneity in cancer is associated with poor prognosis, which may be due to sub-populations of cancer cells that are more proliferative, or less sensitive to drug treatment (Hausser and Alon 2020; Crucitta et al. 2022; Proietto et al. 2023; Dagogo-Jack and Shaw 2018; Lawrence et al. 2022). In other diseases, such as type I diabetes (Richard Benninger and Vira 2022) and hepatic fibrosis (Ramachandran et al. 2020), diversity of the target cell populations is also known to be clinically important. Therefore, to design combination treatments, one needs to carefully describe how each cell type will respond over time to the drug combinations present in a putative treatment. At the same time, one needs to take into account practical considerations, such as drug toxicity and cost. Optimal control theory provides an established framework to combine differential equation modelling of the tissue response to treatment with a cost function that models treatment effect, as well as additional penalties for factors such as cost and toxicity (Lenhart and Workman 2007). Optimal control has been utilised in many mathematical models of tumour progression to predict optimal dosing regimes and responses. Early work in this area considered the evolution of a tumour cell population under the continuous administration of a single anti-cancer drug (Swan George 1979; Swan 1988; Swierniak 1988) and aimed to minimise the total drug administered while keeping the tumour cell population at a desired level. These works demonstrated the utility of control theory frameworks in the study of treatment dosing and have, therefore, been built upon extensively and expanded over recent decades (for a review of previous work, see the review by Schättler and Ledzewicz (2015)). Current oncological control studies provide dosing insights into cutting-edge treatments such as oncolytic therapy (Ratajczyk et al. 2018), hyperthermia treatment (Sérgio and Eduardo 2024), and combination therapy (Ledzewicz et al. 2008). In the context of combination therapy, each control framework is largely model-specific and a general ordinary differential equation (ODE) control framework for multi-drug actions on different discrete cell populations has not yet been developed. In this paper, we propose such a framework, and explicitly compute optimal control solutions that correspond to various treatment scenarios.

The benefit of developing a general model for multi-drug, multi-cell population interactions lies in its applicability to a wide range of problems. Previously, ODE models have been used to predict the effect of drug treatments on heterogeneous cell populations (Lenhart and Workman 2007; Maalavika et al. 2023; Clairambault 2023; Moradi Kashkooli et al. 2020; Kirschner and Webb 1996; Wang and Schättler 2016). However, the large amount of (phenomenological) functional forms employed in such models reduces their transferability between problems and hinders the ability to systematically analyse the resulting optimal controls. In this work we show how a general model can capture important cell-drug interactions while easily identifying optimal pharmacodynamic regimes. This can, in turn, guide novel insights into designing therapies. The framework introduced in this paper captures three key phenomena to first order. First, we model cell proliferation and death, assuming a linear growth rate in the absence of drugs, which is appropriate at small population numbers and few environmental factors limiting growth. More complicated growth functions, like polynomials or Hill functions, are commonly employed in ODE models of tumour growth. While biologically necessary in some contexts, they result in intractable non-linear terms that make analytical treatment of the general optimal control problem infeasible. For that reason, this paper is limited to approximate linear growth rates, and use cases where this is appropriate. Tumour growth can be mediated by drugs: for example, paclitaxel is a chemotherapeutic agent (Maurie and MekhailTarek 2002) that prevents mitosis. Secondly, we allow cells to spontaneously convert to another cell type at fixed rates (Marjanovic et al. 2013; Yuan et al. 2016), which may also be mediated by drug treatment. For instance, in neuroblastoma, undifferentiated sympathoblast cells can convert to neuroblasts in the presence of retinoic acid (RA) (Lawrence et al. 2022; Zeineldin et al. 2022; Jansky et al. 2021). Finally, several drugs can interact with one another to produce synergistic effects (Lieftink and Beijersbergen 2021). Hence, drug-mediated cell differentiation and proliferation pathways should be made to depend on combinations of drugs in the model.

Moreover, the model presented in this work describes the direct influence of the effective drug concentrations present in the relevant target tissues, i.e. we consider a control problem for the optimal pharmacodynamics. In so doing, we explicitly ignore the pharmacokinetics, which describe how the administered drugs are absorbed, metabolised, and cleared from the system. This is a necessary choice for a generalised framework, since the pharmacokinetics of different drugs can vary vastly, for example, depending on the drugs’ metabolic properties, or route of administration (such as oral, intramuscular or intravenous). Hence, pharmacokinetics are an inherently application-specific problem. While understanding the specific dosing that needs to be administered in time to target tissues is key to implement therapies downstream, this work will limit itself with modelling what effective drug concentrations should be present in the target tissues, regardless of how they are administered. Subsequent efforts can then be made to couple a tailored, application-specific model corresponding to the specifics of individual drugs and the minutiae of administration.

Before proceeding, we note that in some clinical settings the apparent advantages of combination regimens derive not from true drug–drug synergies but from inter-patient heterogeneity in drug responses. In particular, Palmer Adam and Sorger Peter (2017) have argued that different patients may each benefit from one of multiple administered drugs, such that a combination of drugs simply raises the chance that any given patient receives an effective monotherapy. By contrast, our focus here is on intra-patient heterogeneity – multiple subpopulations coexisting within the same tumour and the genuine pharmacodynamic interactions between drugs acting on those subclones.

The structure of this work is as follows. In Section 2 we introduce the general modelling framework. In Section 3 we compute the optimality conditions and the optimal control policy arising from our framework. We then apply this framework to three canonical examples for treating heterogeneous (cancer) cell populations. In Section 4 we provide a first example to control a two-population model, based on ovarian cancer cells treated with a synergistic combination of chemotherapy drugs (Lieftink and Beijersbergen 2021). In Section 5 we explore an optimal control formulation for controlling cell populations in neuroblastoma through drugs targeting the tropomyosin A and B signalling pathways. Finally in Section 6 we introduce a way to extend our framework to control the relative abundances of cell types within a population. Together, our work provides a template model that is widely applicable for the control of heterogeneous cell populations and their responses to treatment with several, interacting drugs. Example code has been made available for all computations at our GitHub repository: https://github.com/SWSJChCh/multiplicativeControl.

Modelling Treatment Response of a Heterogeneous Cell Population with Multiplicative Control and Drug-Drug Interactions

In this section, we introduce a general ODE model for the treatment response of a heterogeneous cell population with drug synergies. In Section 2.1 we formulate the general model and in Section 2.2 we propose a simple functional form for the model.

ODE Model: General Formulation

In this work, we are interested in the dynamics of cell proliferation and differentiation in a heterogeneous cell population in the presence of different drugs which interact with each other. We represent cell counts in a vector $\mathbf{x}\in {\mathbb{R}}^n$ and the effective action of each of the drugs in a vector $\mathbf{u}\in {\mathbb{R}}^m$. At this point, we emphasise that the quantities $\mathbf{u}$ represent the pharmacodynamics of the different drugs, rather than their absolute concentration in the cell microenvironment. The advantage of this formulation is that we bypass the complicated functional forms expressing the pharmacokinetics, which are often highly problem-specific. For this reason, we assume that $0\le {\mathbf{u}}_k\le 1$ for all $1\le k\le m$. We describe the direct effect of a single drug on the differentiation and cell proliferation of cell type j as a linear combination of all possible monomials in the form ${\mathbf{u}}_k{\mathbf{x}}_i$ for $1\le k\le m$ and $1\le i\le n$. We describe the effect of interactions between drugs using a minimal model whereby the governing equation for the j-th cell type contains a linear combination of all possible polynomials in the form ${\mathbf{x}}_i{\mathbf{u}}_k{\mathbf{u}}_{\ell }$. Here, we exclude terms containing ${\mathbf{u}}_k^2$ for every $1\le k\le m$, as this is not a drug synergy. Denoting the terms containing products of $\mathbf{u}$ and $\mathbf{x}$ that are linear in $\mathbf{u}$ as $L(\mathbf{u},\mathbf{x})$, and the terms that are nonlinear in $\mathbf{u}$ as $N(\mathbf{u},\mathbf{x})$, we obtain that the dynamics of the vector $\mathbf{x}$ are described by an ordinary differential equation in the form

$$\begin{array}{c}\hfill \dot{\mathbf{x}}=A\mathbf{x}+B\mathbf{u}+L(\mathbf{u},\mathbf{x})+N(\mathbf{u},\mathbf{x}),\end{array}$$ 1

where A is a matrix representing standard linear population growth terms, and B is a matrix representing linear terms relating to drug breakdown. Note that care must be taken with the sign of all polynomial terms to ensure that no population can become negative and the problem is well-defined. This equation will be coupled to a cost functional to yield an optimal control problem.

Functional form for the ODE Model

Here we explain how the terms containing products of the entries of $\mathbf{u}$ and $\mathbf{x}$ in Equation (1) can be described with elementary matrix operations. For the terms that are linear in $\mathbf{u}$, we note that the model must account for monomials of the form ${\mathbf{u}}_k{\mathbf{x}}_i$ for $1\le k\le m$ and $1\le i\le n$ in the ODE for each of the ${\mathbf{x}}_j$ for $1\le j\le n$. Hence, we introduce matrices $C_i$ for $1\le i\le n$ such that

$$\begin{array}{c}\hfill L(\mathbf{u},\mathbf{x})=\sum _{i=1}^n{C}_i\odot ({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_i)\mathbf{u},\end{array}$$ 2

represents all the terms linear in $\mathbf{u}$ in the right hand side of Equation (1), and $\odot$ represents the usual Hadamard product. Here we introduced the notation

$$\begin{array}{c}\hfill {\mathcal{E}}_i={\mathbf{e}}_i^n{\mathbf{1}}_m^T,\end{array}$$

with ${\mathbf{e}}_i^n$ being the i-th standard basis vector in ${\mathbb{R}}^n$ and ${\mathbf{1}}_m\in {\mathbb{R}}^m$ a column vector containing only ones, i.e. ${\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_i$ is the $m\times n$ matrix containing only the i-th entry in $\mathbf{x}$. In other words, the entry ${(C_i)}_{k\ell }$ of the matrix $C_i$ contains the coefficient of the monomial ${\mathbf{u}}_{\ell }{\mathbf{x}}_i$ in the ODE for ${\mathbf{x}}_k$. For the terms containing interaction terms, i.e., products of the entries of $\mathbf{u}$, we again seek a functional form ensuring that the ODE for ${\mathbf{x}}_j$ for every $0\le j\le n$ contains terms in the form ${\mathbf{x}}_i{\mathbf{u}}_k{\mathbf{u}}_{\ell }$. To this end, we introduce lower triangular matrices $D^{\mathit{ij}}$ and define

$$\begin{array}{c}\hfill N(\mathbf{u},\mathbf{x})=\sum _{i=1}^n\sum _{j=1}^n({\mathbf{e}}_j^n{\mathbf{x}}^T{\mathcal{E}}_i)(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m^T)\mathbf{u},\end{array}$$

where ${\mathcal{E}}_i$ is as above, and analogously to $L(\mathbf{u},\mathbf{x})$ in Equation (2), the entry ${(D^{\mathit{ij}})}_{k\ell }$ of the matrix $D^{\mathit{ij}}$ represents the coefficient of the term ${\mathbf{x}}_i{\mathbf{u}}_k{\mathbf{u}}_{\ell }$ in the ODE for ${\mathbf{x}}_j$. We introduce the convention that

$$\begin{array}{c}\hfill {(D^{\mathit{ij}})}_{\mathit{kk}}=0,\text{for all}1\le i,j\le n,1\le k\le m,\end{array}$$

that is, we exclude quadratic terms in the action of any of the drugs in any of the governing equations, since we assume that the pharmacokinetics of any one drug are fully captured by the terms linear in $\mathbf{u}$. Finally, we note that $D^{\mathit{ij}}$ is lower triangular to represent symmetry in pairwise interactions. Put together, we can rewrite Equation (1) as

$$\begin{array}{c}\hfill \dot{\mathbf{x}}=A\mathbf{x}+B\mathbf{u}+\sum _{i=1}^n{C}_i\odot ({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_i)\mathbf{u}+\sum _{i=1}^n\sum _{j=1}^n({\mathbf{e}}_j^n{\mathbf{x}}^T{\mathcal{E}}_i)(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m^T)\mathbf{u},\end{array}$$ 3

We finish the exposition of the functional form of our model by comparing it to another well-studied problem in optimal control, namely that of the linear quadratic regulator (LQR) (Safaei et al. 2012). The equation for a LQR, which considers the effect of the control, $\mathbf{u}$, to be additive in the governing ODE of the state vector, $\mathbf{x}$, is given by

$$\begin{array}{c}\hfill \dot{\mathbf{x}}=A\mathbf{x}+B\mathbf{u}.\end{array}$$ 4

It can be readily seen that our framework provides additional terms that include various nonlinear ways in which the control interacts with the governing equations. Given that an important appeal of the LQR is the ease with which the optimal control can be expressed as feedback control (Lenhart and Workman 2007), we will focus on demonstrating that the model extension considered in this problem lends itself to similar amenable properties for computation.

Optimal Control of an ODE with Multiplicative Controls

Having formulated the governing equations for the model, we now turn to formulating an optimal control problem related to treatment response. In this continuous control problem, we consider a problem given in the form of Equation (3), and seek controls such that the corresponding solution of the model is an extremal of the functional

$$\begin{array}{c}\hfill J(u)=\frac{1}{2}\left[{\mathbf{x}}^T,(T),M,\mathbf{x},(T),+,{\int }_0^T,\left\{{\mathbf{x}}^T,(t),Q,(t),\mathbf{x},(t),+,{\mathbf{u}}^T,(t),R,(t),\mathbf{u},(t)\right\},\text{d},t\right].\end{array}$$ 5

As is standard in optimal control theory, M and Q and R are symmetric matrices, where M, Q are positive semi-definite, and R is positive definite for all $t\in [0,T]$ (Lenhart and Workman 2007). The interpretation of the different terms of the cost function in Equation (5) is as follows. The matrix M associates a cost to the terminal entries of the state. In mathematical oncology models, for example, this could represent the final tumour population. The matrix Q describes the cost associated with the state across the whole interval [0, T]. The matrix R describes costs associated with the drug $\mathbf{u}$. In models describing treatment, this term is often taken to represent treatment toxicity or cost (Lenhart and Workman 2007).

Having introduced the governing equation in Equation (3) together with its cost functional in Equation 5, it is possible to follow standard arguments in optimal control theory to formulate the conditions for the control to be optimal. We first formulate a Hamiltonian, H, associated with the problem,

$$\begin{array}{c}\hfill H=\frac{1}{2}\left[{\mathbf{x}}^T,(t),Q,(T),\mathbf{x},(t),+,{\mathbf{u}}^T,(t),R,(t),\mathbf{u},(t)\right]+\mathit{\lambda }·\dot{\mathbf{x}}.\end{array}$$ 6

Here, $\mathit{\lambda }$ is the vector of adjoints, or co-states, and it satisfies the system of equations

$$\begin{array}{c}\hfill {\dot{\mathit{\lambda }}}_k=-\frac{\partial H}{\partial {\mathbf{x}}_k},\end{array}$$ 7

for all $k\in \{1,\cdots ,n\}$. To close the system of equations, we impose the optimality condition ensuring that the control is in fact a minimiser of the cost functional. This optimality condition is given by

$$\begin{array}{c}\hfill \frac{\partial H}{\partial {\mathbf{u}}_k}=0,\end{array}$$ 8

for all $k\in \{1,\cdots ,n\}$ (Lenhart and Workman 2007). We remark here that in most well-posed problems, the optimality condition provides a means to algebraically express $\mathbf{u}$ in terms of $\mathit{\lambda }$ and $\mathbf{x}$. In some cases, which we will not consider in this paper, however, $\partial H/\partial {\mathbf{u}}_k$ does not depend on $\mathbf{u}$. In this case, several techniques exist to address the problem (Lenhart and Workman 2007).

Adjoint Equations

The adjoint equation in Equation (7) can be expressed in terms of the model coefficients defined in Equation (3). The derivation in Appendix A shows that the adjoint equation in this problem is given by

$$\begin{array}{c}\hfill \dot{\mathit{\lambda }}=-Q\mathbf{x}-A^T\mathit{\lambda }-\sum _{i=1}^n{\mathbf{e}}_i^n{(C_i\mathbf{u})}^T\mathit{\lambda }-\sum _{i=1}^n{\mathbf{e}}_i{\left(\sum _{j=1}^n,{\mathbf{e}}_j^n,{\mathbf{1}}_m^T,(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m^T),\mathbf{u}\right)}^T\mathit{\lambda }.\end{array}$$ 9

Optimality Conditions

The optimality condition in Equation (8) (see Appendix B) gives rise to

$$\begin{array}{c}\hfill {\mathbf{u}}^{\star }=-{\left(R,+,\sum _{i=1}^n,\sum _{j=1}^n,{({\mathbf{e}}_i^n)}^T,\mathbf{x},{\mathit{\lambda }}^T,{\mathbf{e}}_j^n,D^{\mathit{ij}}\right)}^{-1}{\left(B,+,\sum _{i=1}^n,C_i,\odot ,({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_i)\right)}^T\mathit{\lambda },\end{array}$$ 10

whenever the inverse in Equation (10) exists. If we compare this solution to the optimal control in the case of the LQR, we find that

$$\begin{array}{c}\hfill R\mathbf{u}+B^T\mathit{\lambda }=\mathbf{0}.\end{array}$$ 11

Hence, we observe that both the multiplicative control and the drug-drug interaction terms contribute additional terms compared to the optimal control of the LQR. Moreover, these corrections depend directly on both the state and the co-state variables, $\mathbf{x}$ and $\mathit{\lambda }$, respectively. Note that the matrix in the first term of the optimal control as defined in Equation (10) cannot always be inverted, as opposed to that of the control in the LQR in Equation (11), for which an inverse always exists. In such cases, one needs to resort to Pontryagin’s optimality criterion, or the solution of singular optimal control problems, which can be avoided in specific examples (Lenhart and Workman 2007). Having established a framework to perform optimal control in a system consisting of a heterogeneous cell population with multiple interacting drugs, we focus on showcasing the utility and applicability of the method on three illustrative problems.

Example 1: Controlling Growth in a Two-Population Model

In cancer treatment, drug efficacy is known to vary with the phase of the cell cycle that a cell occupies (Shah Manish and Schwartz Gary 2001; Lieftink and Beijersbergen 2021). Drug efficacy dependence on the cell-cycle offers a potential means of maximising the efficacy of therapies whilst minimising drug dosing, and hence the adverse effects on healthy tissue (Lieftink and Beijersbergen 2021). In this example, we explore the possible synergies of two drugs in a simplified system consisting of cervical cancer cells. We distinguish between cells in the G1 phase (denoted by A in Figure 1) and those in the S/G2 phase (denoted by B in Figure 1). The cells are treated with a combination of paclitaxel (with pharmacodynamics denoted by $u_p$) and cisplatin (with pharmacodynamics denoted by $u_c$). Paclitaxel disrupts the normal microtubule dynamics required for cell division, and hence, arrests cell division in the S/G2 phase (Maurie and MekhailTarek 2002). Owing to the arrest in the G2/S phase, cell death occurs in a process known as mitotic catastrophe (Maurie and MekhailTarek 2002). Cisplatin, on the other hand, prevents DNA replication, leading to synchronisation of the cell cycle in the G1 phase (Romani 2022). Cells that fail to properly replicate their DNA undergo apoptosis as a result (Romani 2022). The combination therapy of cisplatin and paclitaxel has been shown to be more favourable in the treatment of stage III and stage IV cervical cancer than single-drug cisplatin or paclitaxel treatments (Muggia Franco et al. 2000). However, both drugs are associated with significant side effects: fever and alopecia commonly occur with paclitaxel, and neurotoxicity and anemia commonly occur with cisplatin. As such, it is desirable to minimise the dose of both drugs given to a patient whilst also minimising the population of cancer cells.

Fig. 1.

Schematic of the two-population ODE model. Cervical cancer cells are divided into two populations, A and B, according to their phase of the cell cycle. G1 phase (non-proliferative) cells are denoted A, whilst S/G2 phase (proliferative) cells are denoted B. Cells enter the cell cycle at rate k. Cisplatin (concentration denoted by $u_c$), reduces the rate of transfer of population A to population B and kills cells in the G1 phase at rate ${\delta }_A$. Paclitaxel (concentration denoted by $u_p$), reduces the rate of proliferation in population B and leads to cell death at rate ${\delta }_B$

ODE Model

Assuming exponential growth of each population, and writing the number of cells in population A by $n_A$ and the number of cells in population B as $n_B$, the dynamics between drugs and cells represented in Figure 1 can be expressed with the following system of coupled ODEs

$$\begin{array}{cc}\hfill \frac{\text{d}{n}_A}{\text{d}t}& =2{\lambda }_B(1-u_p)n_B-k{n}_A(1-u_c)-{\delta }_A{u}_c{n}_A,\hfill \\ \hfill \frac{\text{d}{n}_B}{\text{d}t}& =-{\lambda }_B(1-u_p)n_B-{\delta }_B{u}_p{n}_B+k{n}_A(1-u_c).\hfill \end{array}$$

Here, ${\lambda }_B$ is the rate of division for cells in the G2/S phase, k is the rate at which cells transition from the G1 phase to the G2 phase, ${\delta }_A$ is the rate of cell death induced by cisplatin for cells in the G1 phase, and ${\delta }_B$ is the rate of cell death induced by paclitaxel for cells in the G2/S phase. The dimensionless quantities $u_{\bullet }\in [0,1]$ represent the concentration of the different drugs in the system. Since cervical cancer cells spend roughly the same time in the G1 phase (11 hours, and 1 hour in M phase) and the G2/S phase (8 and 4 hours, respectively) (Cooper and Hausman 2007), we set ${\lambda }_B=k$. Non-dimensionalising the system by taking $\tau =kt$, $N_A=k{n}_A$, and $N_B={\lambda }_B{n}_B$, one obtains the dynamics

$$\begin{array}{cc}\hfill \frac{\text{d}{N}_A}{\text{d}\tau }& =2N_B(1-u_p)-N_A(1-u_c)-\alpha u_c{N}_A,\hfill \end{array}$$ 12

$$\begin{array}{cc}\hfill \frac{\text{d}{N}_B}{\text{d}\tau }& =-N_B(1-u_p)+N_A(1-u_c)-\beta u_p{N}_B,\hfill \end{array}$$ 13

with $\alpha ={\delta }_A/k$ and $\beta ={\delta }_B/k$.

Having expressed the model in non-dimensionalised form, the relative importance of terms in Equations (12) and (13) is readily determined by the values of the dimensionless parameter groupings, $\alpha$ and $\beta$. Furthermore, the dynamics of the system are now described on the timescale for the rate of differentiation of cells in G1 (type A) into cells in G2/S phase (type B).

Optimal Control Formulation

To express the cost function associated with the treatment, we set the penalty for the cell counts equal to the identity, i.e., $Q=\mathbb{I}$, and the penalty for the drug toxicity equal to $R={10}^{-1}\mathbb{I}$. This is a baseline choice to balance the competing objectives of minimising total cell population and limiting drug toxicity. Both objectives are non-dimensionalised to be of order one, and the chosen weights reflect a slight prioritisation of reducing cell population while still penalising excessive drug use. This weighting serves as an initial reference point for exploring the behaviour of the system under optimal control. Note that the limiting case where there is no drug toxicity, i.e. $R=0$, results in the RHS of the optimality condition in Appendix B becoming independent of the control variable, $\mathit{u}$. In such a case, standard results from optimal control theory indicate that the solution is of a bang-bang type (Lenhart and Workman 2007). Specifically, the optimal control is given by $\mathit{u}=1$ whenever $-\partial H/\partial \mathit{u}<0$, leading to maximally applied drug concentrations whenever they are effective. This result confirms that, in the absence of toxicity constraints, the optimal strategy is to apply drugs at full strength to rapidly reduce the population. Despite the evolving population structure, the absence of a toxicity term removes any incentive to modulate the treatment dynamically. This highlights how the inclusion of toxicity plays a central role in generating time-varying optimal solutions under more realistic conditions.

Equations (12) and (13) can be represented in the matrix form of Equation (3) by setting

$$\begin{array}{c}\hfill A=\left(\begin{array}{cc}-1& 2\\ 1& -1\end{array}\right),C_1=\left(\begin{array}{cc}1-\alpha & 0\\ -1& 0\end{array}\right),C_2=\left(\begin{array}{cc}0& -2\\ 0& 1-\beta \end{array}\right).\end{array}$$

Since there are no drug-drug interactions in this example, it follows that $D^{\mathit{ij}}=0$, for all i, j. In the general case presented in the previous section, the optimal control defined in Equation (10) is coupled with the adjoint and state equations. In this example, however, the fact that there are no drug-drug interactions, i.e., $D^{\mathit{ij}}=0$, means that the optimal control as defined in Equation (10) always exists, with

$$\begin{array}{c}\hfill {\mathbf{u}}^{\star }=-R^{-1}{\left(\sum _{i=1}^n,C_i,\odot ,({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_i)\right)}^T\mathit{\lambda },\end{array}$$ 14

subject to the condition that $0\le {\mathbf{u}}_k(t)\le 1$, for all k and all $t\ge 0$. The functional form of the control in Equation (14) resembles that encountered in the LQR problem. One might think that in this problem, therefore, it is possible to cast the optimal solution ${\mathbf{u}}^{\ast }$ as a feedback control. Such a control would be expressed in the state vector alone by eliminating the co-state variable from the control equation through the use of a Ricatti equation (Lenhart and Workman 2007). This, however, is not possible due to the coupling of the control, $\mathbf{u}$, and its bounds, with the governing equation for the co-state, $\mathit{\lambda }$. For the benefit of the reader we demonstrate this in Appendix D.

Numerical Implementation

We numerically solve the boundary value problem arising from the initial condition for the cell populations and the transversality condition, given by ${\lambda }_{\bullet }(T)=0$, where $\bullet =B$ in the above example. To this end, in each simulation we employ the bvp4c scheme of Kierzenka et al. implemented in the |SciPy| Python package (Virtanen et al. 2020). This scheme implements a fourth-order collocation algorithm with control of the residuals and uses a damped Newton method with an affine-invariant criterion function. For the implementation, we set a maximum relative tolerance of ${10}^{-3}$, an absolute boundary value tolerance of ${10}^{-8}$ and specify a maximum of $5\times {10}^5$ nodes for the collocation algorithm. Example code has been made available for all computations at our GitHub repository: https://github.com/SWSJChCh/multiplicativeControl.

Results

Having two parameters in the model given by Equations (12) and (13), we can explore the effect of varying system parameters on the resulting optimal controls. For all simulations, we start with an equal population of cells in G1 and S/G2: $N_A=N_B=1$, and simulate a standard one week course of chemotherapy, which in non-dimensional units is roughly equal to $T=7$ (since time has been re-scaled by the typical cell cycle length, which is roughly 24 hours). The first question we investigate is how effective the optimal control is in reducing the cell count at the end of the treatment interval, which is a clinically relevant question when it comes to improving treatment outcomes. Naively, one might expect the optimal control to always outperform a solution with constant pharmacodynamics with the same area under the curve. A priori, there is no reason to assume that this is guaranteed: the optimal control is defined to minimise the cost functional in Equation (5), which considers the dynamical trajectory of both the cell counts and pharmacodynamics. As such, the following results illustrate the utility of optimal control in identifying pharmacodynamics that outperform the naive constant dosing treatment. This demonstrates that one could in fact use optimal control to find solutions that are not just mathematically optimal according to Equation (5), but are also practically useful.

To make a systematic and fair comparison between an optimal control treatment and that with constant pharmacodynamics, we simulate different model trajectories for $\alpha ,\beta \in [0.05,0.5]$ and compare each of the solutions arising from the optimal control to treatment with constant pharmacodynamics with the same average concentration defined by

$$\begin{array}{c}\hfill {\overline{\mathbf{u}}}_{\bullet }=\frac{1}{T}{\int }_0^T{\mathbf{u}}_{\bullet }(\tau )\text{d}\tau ,\end{array}$$ 15

for each individual drug, ${\mathbf{u}}_{\bullet }$. This constant administration regime yields a corresponding solution, $\overline{\mathbf{x}}$. We then measure efficacy of the combination treatment by using the ratio

$$\begin{array}{c}\hfill \eta =\frac{{\sum }_{i=1}^n{\overline{\mathbf{x}}}_i(T)}{{\sum }_{i=1}^n{\mathbf{x}}_{\mathbf{i}}(T)}.\end{array}$$ 16

The heat map in Figure 2 (panel A) shows that there is a complicated and non-linear relationship between system parameters and the efficacy of the drug. For higher values of $\beta$, i.e., for higher paclitaxel drug kill, the optimal solution outperforms the constant control significantly. On the other hand, when cisplatin drug kill is high and that of paclitaxel is low, the constant drug regime performs better. Therefore, the optimal control framework leads to practical improvements in treatment outcome by taking into account temporal variations in pharmacodynamics in the high paclitaxel drug kill regime, whereas the benefit of doing so in the high cisplatin drug kill regime is not present. To avoid confusion, at this point we stress that the optimal control as defined by the cost functional J in Equation (5) produces a trade-off between drug toxicity and cell counts during the experiment. For this reason, it cannot be immediately expected that the optimal control will always outperform a constant drug dose when comparing only final cell counts between different administration regimes.

Fig. 2.

Efficacy of optimal control treatment in two-population model. A: Heat map of relative treatment efficacy, $\eta$ (as defined in Equation (16)) for different system parameters, $\alpha$, and $\beta$. The horizontal colour-bar above panel A indicates the relative efficacy of the optimal control, $\eta$: values $\eta >1$ (red) correspond to an improvement over the baseline schedule, whereas $\eta <1$ (blue) correspond to a loss in efficacy. B: Total integrated cost of the drug administered. C: model solutions using optimal control at $\alpha =0.05$, $\beta =0.5$ (dashed line) and $\alpha =0.5$, $\beta =0.05$ (solid line). D: Optimal controls in the same parameter regimes as panel C. E-G: Optimal controls for $\alpha =\beta =0.5$ and varying $R_0$ (the entry along the leading diagonal of the cost matrix R)

In practice, matching the optimal dosing profiles exactly is infeasible due to imprecise drug administration. To assess the robustness of the optimal dosing trajectories predicted by our analysis, we performed a Monte Carlo sensitivity analysis in which we added temporally-correlated Gaussian noise of various amplitudes to the optimal controls, re-simulated the ODEs, and recorded the resulting final cell counts and cost differences (see Supplementary Information). In general, we find that for the range of dosing errors considered, final total cell counts remain within 10% of the optimal outcome, indicating that our predicted dosing schedules retain near-optimal efficacy even in the presence of clinically-relevant administration uncertainties (Supplementary Information, Figure 1).

To tease apart the possible mechanisms that could explain why in some parameter regimes the constant treatment is more effective than the optimal control, or vice versa, we turn to the question of how much drug is administered when the system parameters are varied. The heat map in Figure 2 (panel B) shows the integrated drug cost, i.e. ${\int }_0^T{\mathbf{u}}^{T}R\mathbf{u}\text{d}\tau$, for different parameter values. This figure shows that for higher values of paclitaxel drug kill, $\beta$, the drug cost decreases. The cost also decreases for lower values of cisplatin drug kill, $\alpha$. Put together with the findings in panel A of Figure 2 we find that, generally, for small circulating amounts of the drug, varying the treatment with time according to the optimal control prediction is more effective that simply applying a constant, averaged treatment. Here, this might suggest that, when only a small amount of drug is administered, the temporal distribution of the drug significantly impacts treatment response. Finally, we remark that the optimal control terminal condition, ${\lambda }_{\bullet }(T)=0$, was used to ensure optimality of the control as it is the transversality condition required by standard optimal control theory results (Lenhart and Workman 2007). In our model it translates to the drugs being withheld at the end of the treatment window. For this reason, we can expect the corresponding model solutions to show a modest increase in the drug effect, $\mathit{u}$, in the last part of the treatment window, whereas this is not the case for the constant control, which maintains non-zero treatment across the entire window.

Finally, in the preceding analysis we have noted that the optimal control formulation using the cost function J in Equation (5) makes a trade-off between reducing the total cell count and reducing the toxicity associated with drug treatment. For this reason, we now explore how the optimised treatment protocols are influenced when model parameters are kept the same, but the matrix R controlling drug toxicity is varied. To do this, we take $\alpha =\beta =0.5$ and let the drug toxicity vary by considering multiples of the identity matrix, i.e. $R=R_0·\mathbf{I}$, where $R_0$ is a constant. We see in Figure 2 (panels E-G) that increasing $R_0$ from $R_0={10}^{-2}$ to $R_0={10}^2$ has a dramatic impact on the dynamics of the optimal control. For low drug toxicity, $R_0={10}^{-2}$, the cost functional J allows for possibly large amounts of drug administration, leading to a near-constant drug profile (panel E). When $R_0$ is increased to $R_0=1$, the cost formulation leads to a trade-off where one of the two drugs is favoured. Finally, when $R_0$ is further increased to $R_0={10}^2$, the amount of drug is drastically reduced for both drugs. We conclude that calibrating the drug penalty is therefore crucial for possible biomedical applications of optimal dosing, although it remains an open question as to how this should be done, and how the parameter $R_0$ can be measured experimentally.

In the preceding analysis, we have used a cost functional that penalises both drugs, cisplatin and paclitaxel, equally. However, in many clinical settings it might be more desirable to avoid the use of one drug rather than another. This might be due to the fact that one of them is more toxic, but also it is possible that specific patients tolerate one drug better than the other. To investigate the role of varying the relative toxicity of the drugs, we introduce a model extension to vary the relative penalty of one drug versus another. We do this by letting the cost matrix R be given by

$$\begin{array}{c}\hfill R=\left(\begin{array}{cc}{R}_{\alpha }& 0\\ 0& R_{\beta }\end{array}\right),R_{\alpha }+R_{\beta }=R_0,R_{\alpha },R_{\beta }>0,\end{array}$$

where $R_0$ is given by $R_0=2\times {10}^{-1}$, i.e., as in the standard case in the preceding analysis. We then investigate which regions of parameter space incur the highest cost due to either cisplatin or paclitaxel administration (Figure 3, panels A and B).

Fig. 3.

Efficacy of treatment upon varying drug toxicity. A: Heat map showing the cost incurred by paclitaxel administration with constant cost matrix as in Figure 2. B: Heat map showing cost incurred by cisplatin administration. Black circle indicates the endpoint of the trajectory in parameter space visualised in D. C: Parameterised trajectory through parameter space along curve with significant cisplatin cost variation. D: Heat map showing treatment efficacy along the path in panel C as relative cisplatin toxicity is varied

In the heat maps in Figure 3 (panels A and B) we note that there are different regions of parameter space in which the cost arising from administration of either of the two drugs varies. To investigate how this balance changes when we penalise administration of one drug more than the other, we set a path through parameter space which shows high variability in cisplatin cost, which for ease we parameterise as

$$\begin{array}{c}\hfill \alpha ={\gamma }_0+\gamma sin(\phi ),\beta ={\gamma }_0+\gamma cos(\phi ),\phi \in \left[0,,,\frac{2\pi }{9}\right],\end{array}$$ 17

with ${\gamma }_0=0.05$ and $\gamma =0.4$, where the maximum value of $\phi$ corresponds to the approximate point of intersection with the slender white region in Figure 3B. By varying, $R_{\alpha }$ and $\phi$, we have a means to investigate the role of penalising the different drugs as we move through regions of parameter space that favour one drug more than the other. The heat map in Figure 3 (panel D) shows that treatment efficacy increases when toxicity for cisplatin is reduced compared to that of paclitaxel. This is more pronounced for larger values of $\phi$, i.e., closer to the $\alpha$ axis in the schematic of Figure 3 (panel C). That is, when the tolerance for cisplatin is increased in a parameter region that favours the administration of this drug, it can be expected that the optimal drug dosing regime will allow for more of it to be administered, with a favourable effect on treatment outcome.

Put together, we have shown that optimal control can be used to formulate a time-dependent dosing regime that can outperform naive treatment with constant pharmacodynamics. This is not intuitive a priori, but is useful because it allowed to obtain better treatment outcomes in the simulations of Figure 2 when only small amounts of drug were administered. Therefore, the wider utility of optimal control in drug treatment may include finding bespoke temporally varying dosing regimes of different drugs when only very little drug can be administered, for example in contexts of high drug toxicity, where one might otherwise consider a constant administration of very little drug. In addition, a careful calibration of the relative toxicity of the drugs can have a dramatic impact on the relative amounts of drug administered, as well as the overall efficacy of the drug treatment. Together, we have presented a mathematical case study in how to explore the utility and applicability of optimal control to a treatment problem.

Example 2: Modelling Neuroblastoma Differentiation and Regression

In this example, we study a control problem arising from the treatment of neuroblastoma – a pediatric cancer in which cellular heterogeneity plays a large role in disease outcomes (Zeineldin et al. 2022; Cohn et al. 2009), and for which several different drug treatments exist (Zeineldin et al. 2022; Zhou et al. 2023). We demonstrate in this section that, while a realistic ODE model is highly complicated and contains many different terms, optimal control theory can be used to tease apart the many different responses of the cell population. In the context of using optimal control for treatment design, we use this as a case study to address the general point of exploring possible treatment strategies when the number of drugs is very large, and the possible temporal dynamics of drug administration are mathematically intractable.

Neuroblastoma: Cellular Heterogeneity and the Extracellular Environment

Neuroblastoma is an extra-cranial solid cancer which is thought to originate from a neural-crest derived progenitor cell population in children (Lawrence et al. 2022; Zeineldin et al. 2022; Jansky et al. 2021; Körber et al. 2023). Neuroblastoma tumours originate early in fetal and infant development (Körber et al. 2023) and are thought to arise due to a differentiation arrest of the early sympathoadrenal lineage (Zeineldin et al. 2022). The result of this differentiation arrest is the presence of tumours along the sympathetic nervous system containing cells in an immature cell differentiation state (Lawrence et al. 2022; Zeineldin et al. 2022). This heterogeneity has important consequences for drug treatment (Lawrence et al. 2022; Zeineldin et al. 2022). In the rest of this section, we will provide a thorough overview of cell types, drugs, and their interactions, to thoroughly motivate an ODE model for optimal control.

A common framework of the cell types most frequently found within neuroblastoma tumours classifies the cells in a neuroblastoma tumour as sympathoblasts (I-type), adrenergic (N-type), or mesenchymal (S-type) (Zeineldin et al. 2022). I-type cells are proliferative cells that originate from the neural crest. Due to their multi-potency, they can produce both N-type and S-type cells (Zeineldin et al. 2022; Jansky et al. 2021). Immature I-type cells are thought to be a potential reservoir for malignant cells to expand and form neuroblastoma tumours (Jansky et al. 2021; Körber et al. 2023). N-type cells are further differentiated into cells that appear neuroblastic, and can differentiate into neuronal tissue in the presence of retinoic acid (RA) (Zeineldin et al. 2022). Terminally differentiated neuronal cells are minimally proliferative (Zeineldin et al. 2022). S-type cells are substrate-adherent, express vimentin, and produce collagen as well as fibronectin (Zeineldin et al. 2022; Jansky et al. 2021). S-type and N-type neuroblastoma cells can inter-convert between cell types (Zeineldin et al. 2022; Van Groningen et al. 2017).

While the exact role of each cell type in the development of neuroblastoma is still largely unclear (Lawrence et al. 2022; Zeineldin et al. 2022; Maris John 2010; Karin et al. 2021), there are some general clinical themes. All three cell types have been found in neuroblastoma tumours, but their relative abundance is highly variable between patients (Lawrence et al. 2022). Patients with a higher abundance of differentiated N-type cells have overall better prognoses (Zeineldin et al. 2022). Tumour heterogeneity, therefore, has important consequences for drug treatments. Various drug treatments exist for neuroblastoma whereby drugs are used on their own, or in addition to surgical resection (Cohn et al. 2009; Maris John 2010; Karin et al. 2021; Brodeur and Bagatell 2014). A first treatment is the induction of neuronal differentiation as a therapeutic. Treatment with RA has been shown to lead to irreversible differentiation of I-type cells into N-type cells in vitro and in vivo (Matthay Katherine et al. 1999). For this reason, the RA derivative 13-cis RA is now part of the standard of care for neuroblastoma (Zeineldin et al. 2022). We note, however, that S-type cells are not sensitive to treatment with RA (Lawrence et al. 2022). A second treatment uses a cytotoxic chemotherapeutic agent and is given in several cycles before or after surgery (Matthay et al. 2016). Being less proliferative, N-type cells are significantly less sensitive to treatment with a chemotherapeutic agent, and S-type cells are widely understood to be resistant to chemotherapy (Lawrence et al. 2022; Zeineldin et al. 2022; Boeva et al. 2017).

A feature of neuroblastoma is that cells in the sympathoadrenal lineage respond to environmental signals that drive differentiation and development, which is in large part mediated by the tropomyosin receptor kinase family (trk) (Zeineldin et al. 2022; Kasemeier-Kulesa Jennifer et al. 2018; Kasemeier-Kulesa et al. 2015). The first member of the trk family is trkA, which has an important role in cellular differentiation. It is believed that I-type cells that express trkA and are in an environment with the extracellular factor NGF differentiate more readily into N type cells (Kasemeier-Kulesa et al. 2018; Brodeur and Bagatell 2014). When N-type cells, in contrast, express trkA but are not in contact with NGF, they undergo apoptosis (Brodeur and Bagatell 2014). The second member of the trk family is trkB, which is known to have an important role in disease progression (Kasemeier-Kulesa et al. 2015). In brief, N-type cells in the presence of trkB and the extracellular factor BDNF can de-differentiate into I-type or S-type cells (Brodeur and Bagatell 2014; Kasemeier-Kulesa et al. 2015; Kasemeier-Kulesa Jennifer et al. 2018; Kasemeier-Kulesa et al. 2018). Currently, treatments exist that inhibit receptors of the trk family using pan-trk inhibitors (i.e. they inhibit both trkA and trkB simultaneously) (Lawrence et al. 2022; Brodeur and Bagatell 2014). An example of a pan-trk inhibitor used in neuroblastoma is entrectinib (Radhika et al. 2016). The trk signalling pathways are complicated to model since they interact with other factors present – in this case NGF and BDNF – the concentration of which can vary according to the age of the patient and the anatomical location of the tumour (Kasemeier-Kulesa et al. 2015; Kasemeier-Kulesa Jennifer et al. 2018). Therefore, the effect of any one drug on trk inhibition will also depend on other chemical species present, leading to nonlinear effects in Equation (3), which provides an important extension to previous work in our optimality condition.

We remark that there are many other treatments for neuroblastoma that selectively target specific subpopulations, such as anti-GD2 immunotherapy, or treatment with metaiodobenzyl- guanidine (MIBG) to name two examples (Zeineldin et al. 2022). These treatments can be added to the model if desired. For the ease of exposition, we have opted to limit the discussion to one drug per effect. We summarise this system containing three cell populations and drug interactions in Figure 4. In the remainder of this example, we formulate a system of ODEs describing these multi-drug interactions in Section 5.2, describe the corresponding optimal control problem in Section 5.3, and provide insights into drug synergies in neuroblastoma drug treatment in Section 5.4.

Fig. 4.

Schematic of the three-population neuroblastoma ODE model. Cells are divided into I-type sympathoblasts, N-type adrenergic cells, and S-type mesenchymal cells. The model includes spontaneous interconversion between all cell types, treatment with RA acting on the I-to-N differentiation pathway, chemotherapy affecting I-type cells, combined trkA-NGF signalling affecting I and N-type cells, and trkB-BDNF signalling affecting N and S-type cells

A Multi-Drug, Multi-Population Model of Cell Proliferation, Differentiation, and Death in Neuroblastoma

In this section, we formulate a system of ODEs describing the multi-drug and multi-cell population dynamics described in Section 5.1. Denoting the cell numbers of the I-type, N-type and S-type cell populations as $n_{\text{I}},n_{\text{N}},n_{\text{S}}$, respectively, we introduce first the governing equation for the number of I-type cells, $n_{\text{I}}$:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \frac{\text{d}{n}_{\text{I}}}{\text{d}t}=\stackrel{\text{proliferation}}{\overbrace{{\lambda }_{\text{I}}{n}_{\text{I}}}}& -\stackrel{\text{spontaneous interconversion}}{\overbrace{(k_{\text{IN}}+k_{\text{IS}})n_{\text{I}}+k_{\text{NI}}{n}_{\text{N}}+k_{\text{SI}}{n}_{\text{S}}}}-\stackrel{\text{RA-induced diff.}}{\overbrace{{\delta }_{\text{RA}}{u}_{\text{RA}}{n}_{\text{I}}}}-\stackrel{\text{chemo-induced apoptosis}}{\overbrace{{\delta }_{\text{chemo}}{u}_{\text{chemo}}{n}_{\text{I}}}}\hfill \\ \hfill & -\underset{\text{trkA+NGF induced diff.}}{\underbrace{{\delta }_{\text{diff}}(1-u_{\text{trk}})u_{\text{NGF}}{n}_{\text{I}}}}+\underset{\text{trkB+BDNF induced de-diff.}}{\underbrace{{\delta }_{\text{BDNF}}(1-u_{\text{trk}})n_{\text{N}}.}}\hfill \end{array}\\ \hfill \end{array}$$ 18

Here, ${\lambda }_{\bullet }$ represents the proliferation rate of $\bullet$-type cells, the constants $k_{\bullet †}\ge 0$ represent the rates of spontaneous interconversion between cell type $\bullet$ and $†$. The constants ${\delta }_{\text{RA}}$, ${\delta }_{\text{chemo}}$, ${\delta }_{\text{diff}}$, and ${\delta }_{\text{BDNF}}$ correspond to the effects of retinoic acid on differentiation, chemotherapy on cell death, trkA on cell differentiation, and BDNF on de-differentiation, respectively. For the governing equation for the number of N-type cells, $n_{\text{N}}$, we write

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \frac{\text{d}{n}_{\text{N}}}{\text{d}t}=\stackrel{\text{proliferation}}{\overbrace{{\lambda }_{\text{N}}{n}_{\text{N}}}}& -\stackrel{\text{spontaneous interconversion}}{\overbrace{(k_{\text{NS}}+k_{\text{NI}})n_{\text{N}}+k_{\text{IN}}{n}_{\text{I}}+k_{\text{SN}}{n}_{\text{S}}}}+\stackrel{\text{RA-induced diff.}}{\overbrace{{\delta }_{\text{RA}}{u}_{\text{RA}}{n}_{\text{I}}}}\hfill \\ \hfill & -\stackrel{\text{trkA-NGF induced apoptosis}}{\overbrace{{\delta }_{\text{trk,NGF}}(1-u_{\text{NGF}})(1-u_{\text{trk}})n_{\text{N}}}}\hfill \\ \hfill & +\underset{\text{trkA+NGF induced diff.}}{\underbrace{{\delta }_{\text{diff}}(1-u_{\text{trk}})u_{\text{NGF}}{n}_{\text{I}}}}-\underset{\text{trkB+BDNF induced de-diff.}}{\underbrace{2{\delta }_{\text{BDNF}}(1-u_{\text{trk}})n_{\text{N}}.}}\hfill \end{array}\end{array}$$ 19

Here, ${\delta }_{\text{trk,NGF}}$ is a parameter that describes the rate of apoptosis induced by trkA signalling in the absence of NGF. For the governing equation of the number of S-type cells, $n_{\text{S}}$, one obtains

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \frac{\text{d}{n}_{\text{S}}}{\text{d}t}& =\stackrel{\text{proliferation}}{\overbrace{{\lambda }_{\text{S}}{n}_{\text{S}}}}-\stackrel{\text{spontaneous interconversion}}{\overbrace{(k_{\text{SN}}+k_{\text{SI}})n_{\text{S}}+k_{\text{IS}}{n}_{\text{I}}+k_{\text{NS}}{n}_{\text{N}}}}+\stackrel{\text{trkB+BDNF induced de-diff.}}{\overbrace{{\delta }_{\text{BDNF}}(1-u_{\text{trk}})n_{\text{N}}.}}\hfill \end{array}\end{array}$$ 20

Finally, we remark that $u_{\text{trk}}$ is an inhibitor of trk, whereas $u_{\text{RA}},u_{\text{chemo}}$ and $u_{\text{NGF}}$ are all drugs that are added to the cell population. Finally, the standard bounds, $0\le {\mathbf{u}}_k\le 1$, for all k, are imposed.

Optimal Control Formulation

The model in Equations (18)-(20) contains a large number of free parameters. Each of these parameters is clinically relevant, and perhaps impossible to determine empirically. However, this complexity can be reduced in a biologically informed way, such that optimal control theory can be applied to gain insight into the effects of different parameters on treatment outcomes. This, as we will show, is inherently valuable in designing and understanding possible treatment strategies, as it allows us to identify a number of different motifs in the shape of the pharmacodynamics profiles of the different drugs. These in turn can be used to devise novel experimental and clinical strategies.

As a first simplification, and in the absence of knowledge of the rates of spontaneous interconversion, we choose all spontaneous interconversion constants to be equal, i.e. $k_{\bullet †}=k\ge 0$. As done for the example in Section 4, we rescale time, in this case by making the substitution $\tau =kt$ and rescaling all quantities by k. We furthermore assume that the effects of all the drugs on the interconversion rates can be described using a single quantity, $\delta$, i.e., we set ${\delta }_{\text{RA}}={\delta }_{\text{diff}}={\delta }_{\text{BDNF}}=\delta$. Finally, since the cytotoxic chemotherapeutic drug leads to cell death as a result of the cell failing to pass through the checkpoint at the end of the G2/S phase, we assume that the cell death rate is equal to $\lambda$ whenever the chemotherapeutic drug is maximally effective, i.e., when $u_{\text{chemo}}=1$. To achieve this, we set ${\delta }_{\text{chemo}}=2\lambda$. This results in the reduced system,

$$\begin{array}{cc}\hfill \frac{\text{d}{n}_{\text{I}}}{\text{d}\tau }& =(\lambda -2)n_{\text{I}}+n_{\text{N}}+n_{\text{S}}-(\delta u_{\text{RA}}+2\lambda u_{\text{chemo}}+\delta (1-u_{\text{trk}})u_{\text{NGF}})n_{\text{I}}\hfill \\ \hfill & +\delta (1-u_{\text{trk}})n_{\text{N}},\hfill \end{array}$$ 21

$$\begin{array}{cc}\hfill \frac{\text{d}{n}_{\text{N}}}{\text{d}\tau }& =-2n_{\text{N}}+n_{\text{I}}+n_{\text{S}}+\delta (u_{\text{RA}}+(1-u_{\text{trk}})u_{\text{NGF}})n_{\text{I}}\hfill \\ \hfill & -({\delta }_{\text{apop}}(1-u_{\text{NGF}})(1-u_{\text{trk}})+2\delta (1-u_{\text{trk}}))n_{\text{N}},\hfill \end{array}$$ 22

$$\begin{array}{cc}\hfill \frac{\text{d}{n}_{\text{S}}}{\text{d}\tau }& =-2n_{\text{S}}+n_{\text{I}}+n_{\text{N}}+\delta (1-u_{\text{trk}})n_{\text{N}},\hfill \end{array}$$ 23

which only depends on three free parameters, $\delta$, ${\delta }_{\text{apop}}$, and $\lambda$. The model in Equations (21)-(23) can be cast in the form of Equation 3, with $\mathbf{n}={(n_{\text{I}},n_{\text{N}},n_{\text{S}})}^T$, $\mathbf{u}={(u_{\text{RA}},u_{\text{chemo}},u_{\text{trk}},u_{\text{NGF}})}^T$, and A given by

$$\begin{array}{c}\hfill A=\left(\begin{array}{ccc}\lambda -2& 1+\delta & 1\\ 1& -2-2\delta -{\delta }_{\text{apop}}& 1\\ 1& 1+\delta & -2\end{array}\right).\end{array}$$

Since there is no additive control in this model, $B=0$. Furthermore, the terms linear in $\mathbf{u}$ are given by

$$\begin{array}{c}\hfill L(\mathbf{u},\mathbf{n})=\left(\begin{array}{c}-\delta u_{\text{RA}}{n}_{\text{I}}-2\lambda u_{\text{chemo}}{n}_{\text{I}}-\delta u_{\text{NGF}}{n}_{\text{I}}-\delta u_{\text{trk}}{n}_{\text{N}}\\ \delta u_{\text{RA}}{n}_{\text{I}}+\delta u_{\text{NGF}}{n}_{\text{I}}+{\delta }_{\text{apop}}{u}_{\text{NGF}}{n}_{\text{N}}+{\delta }_{\text{apop}}{u}_{\text{trk}}{n}_{\text{N}}+2\delta u_{\text{trk}}{n}_{\text{N}}\\ -\delta u_{\text{trk}}{n}_{\text{N}}.\end{array}\right).\end{array}$$

Therefore, the coefficient matrices, $C_{\bullet }$, are given by the following

$$\begin{array}{c}\hfill C_{\text{I}}=\left(\begin{array}{cccc}-\delta & -2\lambda & 0& -\delta \\ \delta & 0& 0& \delta \\ 0& 0& 0& 0\end{array}\right),C_{\text{N}}=\left(\begin{array}{cccc}0& 0& -\delta & 0\\ 0& 0& {\delta }_{\text{apop}}+2\delta & {\delta }_{\text{apop}}\\ 0& 0& -\delta & 0\end{array}\right),\end{array}$$

and $C_{\text{S}}=\mathbf{0}$.

For the terms containing multi-drug interactions,

$$\begin{array}{c}\hfill Q(\mathbf{u},\mathbf{n})=\left(\begin{array}{c}\delta u_{\text{trk}}{u}_{\text{NGF}}{n}_{\text{I}}\\ -\delta u_{\text{trk}}{u}_{\text{NGF}}{n}_{\text{I}}-{\delta }_{\text{apop}}{u}_{\text{trk}}{u}_{\text{NGF}}{n}_{\text{N}}\\ 0\end{array}\right).\end{array}$$

Therefore, the coefficient matrices $D^{\mathit{ij}}$ are given by

$$\begin{array}{c}\hfill D^{\text{IN}}=\left(\begin{array}{cccc}0& 0& 0& 0\\ 0& 0& 0& 0\\ 0& 0& 0& -\delta \\ 0& 0& 0& 0\end{array}\right),D^{\text{NN}}=\left(\begin{array}{cccc}0& 0& 0& 0\\ 0& 0& 0& 0\\ 0& 0& 0& -{\delta }_{\text{apop}}\\ 0& 0& 0& 0\end{array}\right),\end{array}$$

with $D^{\text{II}}=-D^{\text{IN}}$, and all other interaction coefficient matrices satisfying $D^{\mathit{ij}}=\mathbf{0}$. Finally, as in Example 1, we set $R={10}^{-1}\mathbb{I}$.

Results

By focusing on the reduced model in Equations (18)-(20), we can investigate the impact of varying the model parameters on the resulting drug application regimes. For all simulations we solve the model in Equations (18)-(20) using the numerical scheme described in Section 4.3. At this point, we expect there to be a delicate interplay between different model parameters and the resulting optimal controls. For this reason, the first question we are interested in is what parameters have the most impact on the total dose given of each of the different drugs. To this end, we introduce the notion of maximum marginal drug cost, c. Given two parameters, $\xi$ and $\zeta$, we define this cost as

$$\begin{array}{c}\hfill c(\zeta ,\xi )=\underset{\rho }{max}\tilde{c}(\zeta ,\xi ,\rho ),\end{array}$$

where $\tilde{c}$ is the optimal control cost associated with the parameter set $(\zeta ,\xi ,\rho )$, and $\rho$ is the parameter that is being marginalised out. The maximum marginal cost provides a measure of which parameter regimes can incur potentially the highest drug cost. For all parameters we use the ranges $\lambda ,{\delta }_{\text{apop}},\delta \in [0.05,0.5]$.

The plots of marginal maximum drug cost in Figure 5 (panel A) show that there are clear regions of parameter space that correspond to high drug cost. The one-dimensional traces on the diagonal show that drug cost increases as the spontaneous rate of cell inter-conversions increases (as the possibly transient I-type cell population needs to be controlled constantly), decreases as the effectiveness of the trkA-NGF in inducing apoptosis increases (as then less RA is needed to induce drug kill with little NGF), and increases as the proliferation rate increases (as more chemotherapeutic agent is required to kill the cell population). As there is a clear monotonic relationship between the rate of spontaneous cell type interconversion, $\delta$, and total drug cost, we explored which drugs are most sensitive to the magnitude of this parameter, and how this is affected by the proliferation rate and the apoptosis rate. To do this, we compute a similar metric to that of maximum marginal drug cost, now intended to capture how much the concentration of each drug varies when $\delta$ is increased. We do this by computing the variance of the drug ${\mathbf{u}}_{\bullet }$, denoted $v_{\bullet }(\lambda ,{\delta }_{\text{apop}})$, which we define as

$$\begin{array}{c}\hfill v_{\bullet }(\tilde{\lambda },{\tilde{\delta }}_{\text{apop}})={\text{Var}}_{\delta }\left(\frac{1}{T},{\int }_0^T,{\mathbf{u}}_{\bullet },\text{d}\tau |\lambda =,\tilde{\lambda },,,{\delta }_{\text{apop}},=,{\tilde{\delta }}_{\text{apop}}\right),\end{array}$$

that is, we capture the variability of the mean amount of drug ${\mathbf{u}}_{\bullet }$ for fixed $\lambda$ and ${\delta }_{\text{apop}}$ as $\delta$ is varied. We then compute for which drug this quantity is highest to produce a a partition of the phase space between $\lambda$ and ${\delta }_{\text{apop}}$. The phase space in Figure 5 shows that there is a clear distinction in parameter space, with higher sensitivity for RA for smaller proliferation rates, $\lambda$, and higher sensitivity for chemotherapy for higher proliferation rates. This first investigation therefore showed that by carefully inspecting how optimal controls change across parameter space, one can learn how different drugs respond to changes in parameters, which directly correspond to the biological context. This allows one to understand the expected change in dosing necessary to respond to patient variability. For example, the findings in Figure 5 suggest that in patients with rapidly dividing cancer cells (large values of $\lambda$), the amount of chemotherapy given needs to be carefully titrated according to an estimate of cell type interconversion, whereas patients with genetic features that make neuronal cells highly responsive to trkA/NGF deprivation need careful consideration of their RA dosing as RA treatment is most sensitive in that corresponding parameter regime (high ${\delta }_{\text{apop}}$ in Figure 5).

Fig. 5.

Effect of varying key system parameters on drug cost. A: Diagonal plots showing maximum marginal cost of drug treatment as system parameters are varied. Off-diagonal plots showing pairwise maximum marginal costs as system parameters are varied. B: Phase plane showing regions of parameter space where variation in $\delta$ leads to most change in drug response. The two sensitive drugs are chemotherapy and RA, and they form two distinct regions of the (${\delta }_{\text{apop}}$, $\lambda$) space. Colour indicates which drug’s marginal cost is most sensitive to changes in the death-rate parameter $\delta$. Teal marks combinations of $({\delta }_{\text{apop}},\lambda )$ for which chemotherapy has the larger $\partial \{\text{drug cost}\}/\partial \delta$, whereas pink marks combinations where the retinoic-acid (RA) therapy is more sensitive

Having understood that varying one key parameter, $\delta$, in the model in Equations (18)-(20) has an unequal impact on the total amount of RA and chemotherapy that will be given according to the optimal control in different parameter regimes, we are interested in learning whether the solutions in those parameter regimes exhibit qualitative similarities in the shape of the pharmacodynamics profiles, which we call motifs. To this end, we choose representative values in either regime and vary $\delta$ to investigate the behaviours of the model. For the RA sensitive regime, we choose ${\delta }_{\text{apop}}=0.3$ and $r=0.2$. For the chemotherapy sensitive regime we choose ${\delta }_{\text{apop}}=0.3$ and $r=0.4$. In both regimes we vary the value of the cell type interconversion rate, $\delta =0.05,0.25,0.45$.

Figure 6 shows that the pharmacodynamics profiles in the two different parameter regimes are qualitatively similar, with variations resulting from changes in the parameter $\delta$. In the chemo-sensitive regime (teal in Figures 5 and 6), there is only a small amount of RA given at the end of the treatment window, whereas in the RA-sensitive regime, all solutions start with a higher value for the pharmacodynamics at the start of the treatment window, which then decrease. For that reason, the two different parameter regimes can be interpreted as corresponding to different motifs in pharmacodynamics. These different motifs can be used to suggest possible distinct treatment mechanisms across different contexts. Quantitatively, varying $\delta$ has a noticeable effect on the RA dynamics in the different regimes. Put together, the analysis of this model has shown that different parameter values, corresponding to the biological characteristics of the cell population, may have a profound effect on the temporal administration of the different drugs. In particular, the complicated parameter set of this model allows to express the parameter space in regions in which the drugs are more sensitive to changes of one parameter than in other regions.

Fig. 6.

Optimal control of the three-population model in the chemotherapy-sensitive parameter regime and RA-sensitive regime. A: model solutions in the chemotherapy-sensitive regime, ${\delta }_{\text{apop}}=0.3$ and $r=0.4$ (teal boxes) with different values of $\delta$. B: model solutions in the RA-sensitive regime, ${\delta }_a=0.3$ and $r=0.2$. C-D: corresponding optimal controls. Please note the difference in the scale of the vertical axes across the drug plots

Example 3: Controlling Proportions of Cell Populations

Having understood how an optimal control framework can be applied to control the numbers of cells in different sub-populations, we show in this example how the framework introduced can be modified to control not the absolute cell counts of the sub-populations, but their proportions of the total cell population. Such problems might arise, for example, in bio-engineering, when one would like to produce many clonal copies of several cell types for high-throughput experiments. To extend our framework from cell counts to cell proportions, we consider the total cell population at time t,

$$\begin{array}{c}\hfill N(t)={\mathbf{1}}_n^T\mathbf{x}(t),\end{array}$$

and let $\mathbf{r}(t)=\mathbf{x}(t)/N(t)$, so that $\mathbf{r}(t)$ represents the proportion of different cell types at time t. Now, we have that

$$\begin{array}{c}\hfill \dot{\mathbf{r}}=\frac{\dot{N}\mathbf{x}}{N{(t)}^2}-\frac{N\dot{\mathbf{x}}}{N(t)}=\frac{\dot{N}\mathbf{r}-\dot{\mathbf{x}}}{N(t)},\end{array}$$

and

$$\begin{array}{c}\hfill \frac{\dot{N}}{N(t)}={\mathbf{1}}_n^T\frac{\dot{\mathbf{x}}}{N(t)},\end{array}$$

such that

$$\begin{array}{c}\hfill \dot{\mathbf{r}}=\left({\mathbf{1}}_n^T,\frac{\dot{\mathbf{x}}}{N(t)}\right)\mathbf{r}-\frac{\dot{\mathbf{x}}}{N(t)}.\end{array}$$ 24

Now, we will assume that $B=0$ in Equation (3), which corresponds to negligible drug breakdown and note that

$$\begin{array}{c}\hfill \frac{\dot{\mathbf{x}}}{N(t)}=A\mathbf{r}+\sum _{i=1}^n{C}_i\odot ({\mathbf{1}}_n{\mathbf{r}}^T{\mathcal{E}}_i)\mathbf{u}+\sum _{i=1}^n\sum _{j=1}^n({\mathbf{e}}_j^n{\mathbf{r}}^T{\mathcal{E}}_i)(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m^T)\mathbf{u}=f(\mathbf{r},\mathbf{u}),\end{array}$$ 25

i.e., $\dot{\mathbf{r}}$ in Equation (24) only depends on $\mathbf{r}$ and $\mathbf{u}$. This means that the optimal control framework introduced above can be used to control the proportions of cells in a population, when one is interested in this rather than their absolute cell counts. Now, one may define a target proportion, $\tilde{\mathbf{r}}$, and modify the cost function so that we penalise deviations from this target, i.e.,

$$\begin{array}{c}\hfill J={\int }_0^T\left[{(\mathbf{r}-\tilde{\mathbf{r}})}^T,R,(\mathbf{r}-\tilde{\mathbf{r}}),+,{\mathbf{u}}^T,Q,\mathbf{u}\right]\text{d}t.\end{array}$$

In this case, one can compute the adjoint and optimality conditions associated with this optimal control problem in a similar fashion to the previous problems. The adjoint equation (see Appendix (E.1)) is given by

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \dot{\mathit{\lambda }}& =-\frac{\partial H}{\partial \mathbf{r}}=-Q\mathbf{r}+(Q^T+Q)\tilde{\mathbf{r}}-({\mathbf{1}}_N^{T}f(\mathbf{u},\mathbf{r}))\mathit{\lambda }\hfill \\ \hfill & -\left(A^T,{\mathbf{1}}_n,+,\sum _{i=1}^n,({\mathbf{1}}_n^T{C}_i\mathbf{u}),{\mathbf{e}}_i^n,+,\sum _{i=1}^n,{\mathbf{e}}_i^n,{\mathbf{1}}_n^T,\sum _{j=1}^n,{\mathbf{e}}_j^n,{\mathbf{1}}_m^T,(D_{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m),\mathbf{u}\right)\odot \mathit{\lambda }\odot \mathbf{r}\hfill \\ \hfill & -\left(A^T,\mathit{\lambda },+,\sum _{i=1}^n,{\mathbf{e}}_i^n,{(C_i\mathbf{u})}^T,\mathit{\lambda },+,\sum _{i=1}^n,{\mathbf{e}}_i,{\left(\sum _{j=1}^n,{\mathbf{e}}_j^n,{\mathbf{1}}_m^T,(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m),\mathbf{u}\right)}^T,\mathit{\lambda }\right).\hfill \end{array}\end{array}$$

For the optimality condition, one finds (see Appendix E.2) that

$$\begin{array}{cc}\hfill \frac{\partial H}{\partial \mathbf{u}}=& R\mathbf{u}+({\mathit{\lambda }}^T\mathbf{r}-1)\left(\sum _{i=1}^n,\sum _{j=1}^n,{({\mathbf{e}}_i^n)}^T,\mathbf{r},{\mathit{\lambda }}^T,{\mathbf{e}}_j^n,D^{\mathit{ij}}\right)\mathbf{u}\hfill \\ \hfill & +{\left(\sum _{i=1}^n,C_i,\odot ,({\mathbf{1}}_n{\mathbf{r}}^T{\mathcal{E}}_i)\right)}^T\left(({\mathit{\lambda }}^T\mathbf{r}),{\mathbf{1}}_n,-,\mathit{\lambda }\right).\hfill \end{array}$$

While the resulting equations are significantly less tractable than the optimal control conditions derived in examples 1 and 2, we note that the governing equations for the control are still given in the form of a coupled system of ODEs between $\mathit{\lambda }$ and $\mathbf{r}$, meaning that they can be solved with a straightforward boundary value numerical solver. Put together, we have shown how the framework used in the previous two examples can be easily modified to account for controlling cell proportions in a population whose size can be arbitrary.

Discussion

In this work, we have presented a framework for optimally controlling the treatment response of heterogeneous populations to combination treatment. This framework describes cells of different cell types in terms of different well-mixed compartments, which interact with the administered drugs according an ODE model based on the law of mass action. As such, the framework presents a straightforward way to model an arbitrary number of cell types together with an arbitrary number of drugs. By expressing a general optimal control formulation, our framework is widely applicable, and simple to implement numerically using a boundary value problem solver. We have applied our work to three illustrative examples in mathematical biology: modelling the population of cervical cancer cells in different stages of the cell cycle when treated with two different drugs; modelling cells in different stages of development in neuroblastoma; and a general problem where one is interested in controlling the proportions of different cells in a well-mixed population.

In the system comprised of cervical cancer cells in different stages of the cell cycle, we carefully considered the question of whether optimal control approaches can identify treatment schedules with varying pharmacodynamics that can out-perform constant dosing. We found that optimal control of two antagonistic drugs, cisplatin and paclitaxel, provides model predictions that would lead to more effective treatment responses than treatment with a constant action of the drug. This example is illustrative of how to one might link the notions of mathematical optimality to practical outcomes. Our approach shows that, with respect to clinically observable parameters, dosing regimes that are predicted by optimal control theory do not always share a one-to-one correspondence with dosing regimes that maximise or minimise such observables. However, our analysis shows that in certain biological contexts, optimal control theory can be used to predict regimes of administration that outperform constant dosing with respect to clinically observable parameters. The general insight from this is that one must be careful in interpreting what it means for solutions to be optimal, on the one hand, and for performing a thorough analysis of their practical utility, on the other. Of most practical interest, we found that when the total cost incurred by the drugs was low, i.e., when otherwise the total administered constant dose would have been small, the (time-varying) optimal control pharmacodynamics traces outperformed constant dosing. This is a promising insight therapeutically, since optimal control can provide ways to optimally design treatments at low dosage, which can help reduce the burden of drug toxicity, while improving treatment response. We also found that varying the penalty for total drug administration has a dramatic effect on the treatment profile given. While this offers tremendous opportunities to implement bespoke treatments based on patient toleration of the different drugs and differences in drug toxicity more general, it remains an open challenge to interpret and calibrate the parameters to penalise the different drug regimes.

In the example studying neuroblastoma heterogeneity, we similarly found that there is significant variability in optimal treatments depending on patient-specific parameters, such as the cell proliferation rate and the sensitivity to differentiation-inducing agents such as RA. Our approach showed that the parameter space can be studied systematically to reveal that there exist specific regions of parameter space that have characteristic shapes of the optimal control pharmacodynamics traces, which can be used to inspire dosing regimes in practice. In addition, the shapes of such pharmacodynamics traces can be linked to specific regions of parameter space, such that they might be used to probe different general themes of treatment strategies depending on the individual characteristics of the cell population under investigation. For example, we found that in some regimes, application of RA toward the end of the treatment window was most effective, whereas in others one should start with a high dose of RA. Models such as these, when calibrated to data, can be used to inform optimal treatments in a personalised manner, since individual treatment responses to individual drugs can be assessed more carefully. This, down the line, can inform treatment choices regarding which drugs to administer and how to titrate them during treatment.

There are several ways in which our framework can be extended and improved. First, our model was designed to incorporate only the pharmacodynamics of the administered drugs. In applications of optimal control to drug dosing, the question of interest becomes how the optimal solutions found through our framework might be achieved by optimal drug dosing. This would entail that the optimal control framework in this paper be coupled to a model describing the relevant pharmacokinetics of the system under investigation. We expect this problem to be challenging computationally, due to feedback effects, e.g. metabolism of the drug by the targeted cells. Second, the framework developed in this paper assumes that cell populations are well mixed and that cell heterogeneity is not spatial. In some contexts, however, the spatial location within a tissue or tumour is known to have an important influence on gene expression and cell identity (Hausser and Alon 2020). For this reason, ideas about optimal control like those explored in this paper can be extended to PDE models which directly incorporate spatial location within the tumour and the relevant anatomy to make interactions between cells and their environment more realistic. This would allow for formulating treatment plans dependent on the population environment, in addition to the population-intrinsic properties considered in this work. Third, the framework presented in this work considers a fixed time horizon when, in situations such as cancer therapy, treatment is delivered over a period which is often initially undetermined, and continues until either the patient has been cured or tumour recurrence occurs. In these cases, model predictive control should be implemented in tandem to design the best treatment option. Furthermore, it would be of interest to couple the framework presented in this work to existing inference methods in order to seek patient-specific parameters, for which the treatment could then be personalised using optimal control.

As a final note, we highlight that to make advances the models presented in this work can be calibrated to experimental data. To make advances in the analysis of the models, we made assumptions about the magnitude about certain parameters, and made a careful parameter sweep to understand the behavior of the system as the parameters were varied. However, experimental methods exist to calibrate the key parameters of interest in the models to data, which can then be used to compute optimal more biologically feasible solutions for the system under investigation.

A Computation of the Adjoint Conditions

We compute the adjoint equations defined by

$$\begin{array}{c}\hfill {\dot{\lambda }}_k=-\frac{\partial H}{\partial {\mathbf{x}}_k}.\end{array}$$

Note first

$$\begin{array}{c}\hfill \frac{\partial }{\partial {\mathbf{x}}_k}{\mathit{\lambda }}^T\sum _{i=1}^n{C}_i\odot ({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_i)\mathbf{u}=\frac{\partial }{\partial {\mathbf{x}}_k}{\mathit{\lambda }}^T{C}_k\odot ({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_k)\mathbf{u}=\frac{\partial }{\partial {\mathbf{x}}_k}{\mathbf{x}}_k{\mathit{\lambda }}^T{C}_k\odot ({\mathbf{1}}_n{\mathbf{1}}_n^T{\mathcal{E}}_k)\mathbf{u},\end{array}$$

such that

$$\begin{array}{c}\hfill \frac{\partial }{\partial {\mathbf{x}}_k}{\mathit{\lambda }}^T\sum _{i=1}^n{C}_i\odot ({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_i)\mathbf{u}={\mathit{\lambda }}^T{C}_k\mathbf{u},\end{array}$$

where the last equality is due to the fact that ${\mathbf{1}}_n{\mathbf{1}}_n^T{\mathcal{E}}_k$ is an $n\times n$ matrix filled with ones. Hence

$$\begin{array}{c}\hfill \frac{\partial }{\partial \mathbf{x}}{\mathit{\lambda }}^T\sum _{i=1}^n{C}_i\odot ({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_i)\mathbf{u}=\sum _{i=1}^n{\mathbf{e}}_i^n{\mathit{\lambda }}^T{C}_i\mathbf{u}=\left(\sum _{i=1}^n,{\mathbf{e}}_i^n,{(C_i\mathbf{u})}^T\right)\mathit{\lambda }.\end{array}$$

Similarly, we obtain

$$\begin{array}{c}\hfill \frac{\partial }{\partial {\mathbf{x}}_k}{\mathit{\lambda }}^T\sum _{i=1}^n\sum _{j=1}^n({\mathbf{e}}_j^n{\mathbf{x}}^T{\mathcal{E}}_i)(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m^T)\mathbf{u}={\mathit{\lambda }}^T\sum _{j=1}^n{\mathbf{e}}_j^n{\mathbf{1}}_n{\mathcal{E}}_k(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m^T)\mathbf{u}.\end{array}$$

Now, noticing

$$\begin{array}{c}\hfill {\mathbf{e}}_j^n{\mathbf{1}}_n^T{\mathcal{E}}_k={\mathbf{e}}_j^n{\mathbf{1}}_n^T{\mathbf{e}}_k^n{\mathbf{1}}_m^T={\mathbf{e}}_j^n{\mathbf{1}}_m^T,\end{array}$$

we find

$$\begin{array}{c}\hfill \frac{\partial }{\partial {\mathbf{x}}_k}{\mathit{\lambda }}^T\sum _{i=1}^n\sum _{j=1}^n({\mathbf{e}}_j^n{\mathbf{x}}^T{\mathcal{E}}_i)(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m^T)\mathbf{u}={\mathit{\lambda }}^T\sum _{j=1}^n{\mathbf{e}}_j^n{\mathbf{1}}_m^T(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m^T)\mathbf{u},\end{array}$$

such that

$$\begin{array}{c}\hfill \frac{\partial }{\partial \mathbf{x}}{\mathit{\lambda }}^T\sum _{i=1}^n\sum _{j=1}^n({\mathbf{e}}_j^n{\mathbf{x}}^T{\mathcal{E}}_i)(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m^T)\mathbf{u}={\left(\sum _{i=1}^n,{\mathbf{e}}_i,\sum _{j=1}^n,{\mathbf{e}}_j^n,{\mathbf{1}}_m^T,(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m^T),\mathbf{u}\right)}^T\mathit{\lambda }.\end{array}$$

Put together,

$$\begin{array}{c}\hfill \frac{\partial }{\partial \mathbf{x}}{\mathit{\lambda }}^T\dot{\mathbf{x}}=A^T\mathit{\lambda }-\sum _{i=1}^n{\mathbf{e}}_i^n{(C_i\mathbf{u})}^T\mathit{\lambda }-\sum _{i=1}^n{\mathbf{e}}_i{\left(\sum _{j=1}^n,{\mathbf{e}}_j^n,{\mathbf{1}}_m^T,(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m^T),\mathbf{u}\right)}^T\mathit{\lambda }.\end{array}$$ A1

B Computation of the Optimality Conditions

Notice first

$$\begin{array}{c}\hfill \frac{\partial }{\partial \mathbf{u}}{\mathit{\lambda }}^T\sum _{i=1}^n{C}_i\odot ({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_i)\mathbf{u}={\left(\sum _{i=1}^n,C_i,\odot ,({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_i)\right)}^T\mathit{\lambda }.\end{array}$$ B2

Now we compute

$$\begin{array}{c}\hfill \frac{\partial }{\partial {\mathbf{u}}_{\mathbf{k}}}{\mathit{\lambda }}^T({\mathbf{e}}_j^n{\mathbf{x}}^T{\mathcal{E}}_i)(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m)\mathbf{u}=\frac{\partial }{\partial {\mathbf{u}}_{\mathbf{k}}}{\mathit{\lambda }}^T{\xi }^{\mathbf{ij}},\end{array}$$

where ${\xi }^{\mathbf{ij}}=({\mathbf{e}}_j^n{\mathbf{x}}^T{\mathcal{E}}_i)(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m)\mathbf{u}$ is an $n\times 1$ vector with a non-zero element ${\alpha }_{\mathit{ij}}$ only in the $j^{\text{th}}$ entry:

$$\begin{array}{c}\hfill {\alpha }_{\mathit{ij}}=\sum _{p=1}^m{u}_p\sum _{q=1}^m{x}_i{D}_{\mathit{qp}}^{\mathit{ij}}{u}_q,\end{array}$$ B3

and $D^{\mathit{ij}}$ is a matrix as defined previously. In differentiating ${\xi }^{\mathbf{ij}}$ with respect to ${\mathbf{u}}_k$, we apply simple product rule differentiation to (B3) to obtain

$$\begin{array}{cc}\hfill \frac{\partial }{\partial {\mathbf{u}}_{\mathbf{k}}}{\alpha }_{\mathit{ij}}& =\sum _{p\ne k}{\mathbf{u}}_p{\mathbf{x}}_i{D}_{\mathit{qp}}^{\mathit{ij}}{\mathbf{u}}_q+{\mathbf{u}}_k{D}_{\mathit{qp}}^{\mathit{ij}}{\mathbf{u}}_q+{\mathbf{u}}_k{D}_{\mathit{kk}}^{\mathit{ij}}{\mathbf{u}}_i\hfill \\ \hfill & =\sum _{p=1}^n{\mathbf{u}}_p{\mathbf{x}}_i{D}_{\mathit{qp}}^{\mathit{ij}}+{\mathbf{u}}_k{D}_{\mathit{kk}}^{\mathit{ij}}{\mathbf{x}}_i,\hfill \end{array}$$

where we have used the lower triangularity of $D^{\mathit{ij}}$. Now

$$\begin{array}{c}\hfill \frac{\partial }{\partial {\mathbf{u}}_k}{\mathit{\lambda }}^T({\mathbf{e}}_j^n{\mathbf{x}}^T{\mathcal{E}}_i)(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m)\mathbf{u}=\mathit{\lambda }\left[\sum _{p=1}^n,{\mathbf{u}}_p,{\mathbf{x}}_i,{(D^{\mathit{ij}})}_{\mathit{kp}},+,{\mathbf{u}}_{\mathbf{k}},{(D^{\mathit{ij}})}_{\mathit{kk}},{\mathbf{x}}_i\right].\end{array}$$

Defining now the matrix ${\mathcal{D}}^k$ by

$$\begin{array}{c}\hfill {({\mathcal{D}}^k)}_{\mathit{ij}}={(D^{\mathit{ij}})}_{\mathit{kk}},\end{array}$$

we find

$$\begin{array}{c}\hfill \sum _{i=1}^n\sum _{j=1}^n{\mathbf{u}}_k{(D^{\mathit{ij}})}_{\mathit{kk}}{\mathbf{x}}_i{\mathit{\lambda }}_j={\mathbf{u}}_k{\mathbf{1}}^T{\mathcal{D}}^k\odot (\mathbf{x}{\mathit{\lambda }}^T)\mathbf{1},\end{array}$$

and also

$$\begin{array}{c}\hfill \sum _{i=1}^n\sum _{j=1}^n{\mathbf{x}}_i{\mathit{\lambda }}_j(D^{\mathit{ij}})\mathbf{u}·{\mathbf{e}}_k=\sum _{i=1}^n\sum _{j=1}^n{({\mathbf{e}}_i^n)}^T\mathbf{x}{\mathit{\lambda }}^T{\mathbf{e}}_j^n(D^{\mathit{ij}}\mathbf{u})·{\mathbf{e}}_k^m,\end{array}$$

and hence

$$\begin{array}{cc}& \frac{\partial }{\partial {\mathbf{u}}_k}{\mathit{\lambda }}^T\sum _{i=1}^n\sum _{j=1}^n({\mathbf{e}}_j^n{\mathbf{x}}^T{\mathcal{E}}_i)(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m)\mathbf{u}\hfill \\ \hfill & ={\mathbf{u}}_k{\mathbf{1}}_m^T{\mathcal{D}}^k\odot (\mathbf{x}{\mathit{\lambda }}^T){\mathbf{1}}_m+\sum _{i=1}^n\sum _{j=1}^n{({\mathbf{e}}_i^n)}^T\mathbf{x}{\mathit{\lambda }}^T{\mathbf{e}}_j^n(D^{\mathit{ij}}\mathbf{u})·{\mathbf{e}}_k^m,\hfill \end{array}$$

which gives

$$\begin{array}{cc}& \frac{\partial }{\partial \mathbf{u}}{\mathit{\lambda }}^T\sum _{i=1}^n\sum _{j=1}^n({\mathbf{e}}_j^n{\mathbf{x}}^T{\mathcal{E}}_i)(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m)\mathbf{u}\hfill \\ \hfill & =\left(\sum _{i=1}^m,{\mathbf{e}}_i,{\mathbf{1}}^T,{\mathcal{D}}^i,\odot ,(\mathbf{x}{\mathit{\lambda }}^T),\mathbf{1}\right)\odot \mathbf{u}+\left(\sum _{i=1}^n,\sum _{j=1}^n,{({\mathbf{e}}_i^n)}^T,\mathbf{x},{\mathit{\lambda }}^T,{\mathbf{e}}_j^n,D^{\mathit{ij}}\right)\mathbf{u}.\hfill \end{array}$$ B4

Finally, note that for the model we have assumed that ${\mathcal{D}}^i=0$ for all i since there are no terms in the form ${\mathbf{u}}_k{\mathbf{u}}_k$ for any k in any of the equations. For that reason,

$$\begin{array}{c}\hfill \frac{\partial }{\partial \mathbf{u}}{\mathit{\lambda }}^T\sum _{i=1}^n\sum _{j=1}^n({\mathbf{e}}_j^n{\mathbf{x}}^T{\mathcal{E}}_i)(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m)\mathbf{u}=\left(\sum _{i=1}^n,\sum _{j=1}^n,{({\mathbf{e}}_i^n)}^T,\mathbf{x},{\mathit{\lambda }}^T,{\mathbf{e}}_j^n,D^{\mathit{ij}}\right)\mathbf{u}.\end{array}$$

Now putting everything together, we obtain an expression for $\partial H/\partial \mathbf{u}$ given by

$$\begin{array}{c}\hfill \frac{\partial H}{\partial \mathbf{u}}=R\mathbf{u}+B^T\mathit{\lambda }+{\left(\sum _{i=1}^n,C_i,\odot ,({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_i)\right)}^T\mathit{\lambda }+\left(\sum _{i=1}^n,\sum _{j=1}^n,{\mathbf{e}}_i^n,\mathbf{x},{\mathit{\lambda }}^T,{\mathbf{e}}_j^n,D^{\mathit{ij}}\right)\mathbf{u}.\end{array}$$

C Solution of the Optimality Condition

Imposing the restriction ${\mathbf{u}}_k\in [0,1]$ for all k implies via the standard theory of optimal control that

$$\begin{array}{c}\hfill {\mathbf{u}}_k^{\ast }=\left\{\begin{array}{cc}0\hfill & \frac{\partial H}{\partial {\mathbf{u}}_k}<0,\hfill \\ 0\le {\mathbf{u}}_k\le 1\hfill & \frac{\partial H}{\partial {\mathbf{u}}_k}=0,\hfill \\ 1\hfill & \frac{\partial H}{\partial {\mathbf{u}}_k}>0.\hfill \end{array}\right)\end{array}$$

Concretely, the middle case corresponds to solutions of the equation

$$\begin{array}{c}\hfill \mathbf{0}=R\mathbf{u}+B^T\mathit{\lambda }+{\left(\sum _{i=1}^n,C_i,\odot ,({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_i)\right)}^T\mathit{\lambda }+\left(\sum _{i=1}^n,\sum _{j=1}^n,{\mathbf{e}}_i^n,\mathbf{x},{\mathit{\lambda }}^T,{\mathbf{e}}_j^n,D^{\mathit{ij}}\right)\mathbf{u},\end{array}$$

if they exist. In that case, the optimality condition reads

$$\begin{array}{c}\hfill \left(R,+,\sum _{i=1}^n,\sum _{j=1}^n,{\mathbf{e}}_i^n,\mathbf{x},{\mathit{\lambda }}^T,{\mathbf{e}}_j^n,D^{\mathit{ij}}\right)\mathbf{u}=-{\left(B,+,\sum _{i=1}^n,C_i,\odot ,({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_i)\right)}^T\mathit{\lambda },\end{array}$$

and so, if the inverse exists,

$$\begin{array}{c}\hfill {\mathbf{u}}^{\star }=-{\left(R,+,\sum _{i=1}^n,\sum _{j=1}^n,{\mathbf{e}}_i^n,\mathbf{x},{\mathit{\lambda }}^T,{\mathbf{e}}_j^n,D^{\mathit{ij}}\right)}^{-1}{\left(B,+,\sum _{i=1}^n,C_i,\odot ,({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_i)\right)}^T\mathit{\lambda }.\end{array}$$

D On why There is no Closed-form Expression for a Feedback Control in Example 1

In the case of the LQR problem, which has dynamics given by Equation (4) and optimises the cost function given in Equation (5), it can be shown that the optimal control is given by

$$\begin{array}{c}\hfill {\mathbf{u}}^{\ast }=-R^{-1}{B}^{T}S\mathbf{x},\end{array}$$

where S is a matrix that satisfies a quadratic first-order ODE called the Ricatti equation (Lenhart and Workman 2007). In optimal control terminology, the matrix $R^{-1}{B}^{T}S$ is called the gain. The matrix S arises from making the ansatz $\lambda (t)=S(t)\mathbf{x}(t)$. Making this substitution, it follows that

$$\begin{array}{c}\hfill \dot{\mathit{\lambda }}=\dot{S}(t)\mathbf{x}+S(t)\dot{\mathbf{x}}(t),\end{array}$$

so that, remembering that ${(D^{\mathit{ij}})}_{\mathit{qp}}=0$ for all $p\ge q$,

$$\begin{array}{c}\hfill \dot{S}(t)\mathbf{x}+S(t)\dot{\mathbf{x}}(t)=\dot{\mathit{\lambda }}=-Q\mathbf{x}-A^T\mathit{\lambda }-\sum _{i=1}^n{\mathbf{e}}_i^n{(C_i\mathbf{u})}^T\mathit{\lambda },\end{array}$$

from which we obtain that

$$\begin{array}{cc}\hfill \dot{S}(t)\mathbf{x}& =-S(t)\dot{\mathbf{x}}(t)-Q\mathbf{x}-A^T\mathit{\lambda }-\sum _{i=1}^n{\mathbf{e}}_i^n{(C_i\mathbf{u})}^T\mathit{\lambda }\hfill \\ \hfill & =-S(t)\left(A,\mathbf{x},+,\sum _{i=1}^n,C_i,\odot ,({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_i),\mathbf{u}\right)-Q\mathbf{x}-A^T\mathit{\lambda }-\sum _{i=1}^n{\mathbf{e}}_i^n{(C_i\mathbf{u})}^T\mathit{\lambda }.\hfill \end{array}$$

At this point, if $\mathbf{u}$ were unbounded, one would substitute the control, Equation (14), to compute

$$\begin{array}{cc}\hfill \sum _{i=1}^n{C}_i\odot ({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_i)\mathbf{u}& =\sum _{i=1}^n{C}_i\odot ({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_i)\left(-,R^{-1},{\left(\sum _{j=1}^n,C_j,\odot ,({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_j)\right)}^T,\mathit{\lambda }\right)\hfill \\ \hfill & =-\left(\sum _{i=1}^n,\sum _{j=1}^n,[C_i\odot ({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_i)],R^{-1},{[C_j\odot ({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_j)]}^T\right)\mathit{\lambda }\hfill \\ \hfill & =-\left(\sum _{i=1}^n,\sum _{j=1}^n,[C_i\odot ({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_i)],R^{-1},{[C_j\odot ({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_j)]}^T\right)S\mathbf{x}\hfill \\ \hfill & \equiv {\mathcal{R}}_1\mathbf{x}.\hfill \end{array}$$

As for the second term involving $\mathbf{u}$, we compute

$$\begin{array}{cc}\hfill \sum _{i=1}^n{\mathbf{e}}_i^n{(C_i\mathbf{u})}^T\mathit{\lambda }& =-\sum _{i=1}^n{\mathbf{e}}_i^n{\left(C_i,R^{-1},{\left(\sum _{j=1}^n,C_j,\odot ,({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_j)\right)}^T,\mathit{\lambda }\right)}^T\mathit{\lambda }\hfill \\ \hfill & =-\sum _{i=1}^n{\mathbf{e}}_i^n\left({\mathit{\lambda }}^T,\left(\sum _{j=1}^n,C_j,\odot ,({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_j)\right),R^{-T},C_i^T\right)\mathit{\lambda }\hfill \\ \hfill & =-\sum _{i=1}^n{\mathbf{e}}_i^n\left({\mathbf{x}}^T,S,\left(\sum _{j=1}^n,C_j,\odot ,({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_j)\right),R^{-T},C_i^T\right)S\mathbf{x}\hfill \\ \hfill \&\equiv {\mathcal{R}}_2\mathbf{x}.\end{array}$$

Put together, we would obtain

$$\begin{array}{c}\hfill \dot{S}\mathbf{x}=\left[-,S,A,+,{\mathcal{R}}_1,-,Q,-,A^T,S,-,{\mathcal{R}}_2\right]\mathbf{x},\end{array}$$

so that

$$\begin{array}{c}\hfill \dot{S}=\left[-,S,A,+,{\mathcal{R}}_1,-,Q,-,A^T,S,-,{\mathcal{R}}_2\right]=M.\end{array}$$ D5

Notice that Equation (D5) is strictly speaking not a Ricatti equation as it is linear in S. Proceeding one would substitute back to find that the control is only a type of feedback control,

$$\begin{array}{c}\hfill {\mathbf{u}}^{\star }=-R^{-1}{\left(\sum _{i=1}^n,C_i,\odot ,({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_i)\right)}^{T}S\mathbf{x}.\end{array}$$

The complication in this procedure arises as a result of the bounds imposed on $\mathbf{u}$. In the substitution of the control given by Equation (14), no bounds were assumed. Therefore, the closed form solution for the equation for $\dot{S}$ that arises takes no restrictions on $\mathbf{u}$ into account. We note that in numerical implementations, one can easily incorporate these bounds when computing the derivative of S.

E Computation of Optimality and Adjoint Equations for Example 3 in the Main Text

E.1 Adjoint Conditions

We first compute the quantity $\partial H/\partial \mathbf{r}$. Since

$$\begin{array}{c}\hfill {(\mathbf{r}-\tilde{\mathbf{r}})}^{T}Q(\mathbf{r}-\tilde{\mathbf{r}})={\mathbf{r}}^{T}Q\mathbf{r}+{\tilde{\mathbf{r}}}^{T}Q\tilde{\mathbf{r}}-(Q^T\tilde{\mathbf{r}})·\mathbf{r}-(Q\tilde{\mathbf{r}})·\mathbf{r},\end{array}$$

we find that

$$\begin{array}{c}\hfill \frac{\partial H}{\partial \mathbf{r}}=Q\mathbf{r}-(Q^T+Q)\tilde{\mathbf{r}}+\frac{\partial }{\partial \mathbf{r}}{\mathit{\lambda }}^T\dot{\mathbf{r}}.\end{array}$$

The third term is given by

$$\begin{array}{cc}\hfill \frac{\partial }{\partial {\mathbf{r}}_k}{\mathit{\lambda }}^T\dot{\mathbf{r}}& =\frac{\partial }{\partial {\mathbf{r}}_k}{\mathit{\lambda }}^T\left(\left({\mathbf{1}}_n^T,\frac{\dot{\mathbf{x}}}{N}\right),\mathbf{r},-,\frac{\dot{\mathbf{x}}}{N}\right)\hfill \\ \hfill & ={\mathit{\lambda }}_k\left({\mathbf{1}}_n^T,\frac{\dot{\mathbf{x}}}{N}\right)+{\mathit{\lambda }}_k{\mathbf{r}}_k\left(\frac{\partial }{\partial {\mathbf{r}}_k},{\mathbf{1}}_n^T,\frac{\dot{\mathbf{x}}}{N}\right)-\frac{\partial }{\partial {\mathbf{r}}_k}\left({\mathit{\lambda }}^T,\frac{\dot{\mathbf{x}}}{N}\right).\hfill \end{array}$$

Observe first that the final term in the last expression is equal to the $k^{\text{th}}$ component of the expression in Equation (A1) with $\mathbf{x}$ replaced by $\mathbf{r}$. Now, the following computation is essentially the same as that for the adjoint condition that we did previously, but we include it for completeness here

$$\begin{array}{cc}& \frac{\partial }{\partial {\mathbf{r}}_k}\left({\mathbf{1}}_n^T,\frac{\dot{\mathbf{x}}}{N}\right)\hfill \\ \hfill & =\frac{\partial }{\partial {\mathbf{r}}_k}{\mathbf{1}}_n^T\left(A,\mathbf{r},+,\sum _{i=1}^n,C_i,\odot ,({\mathbf{1}}_n{\mathbf{r}}^T{\mathcal{E}}_i),\mathbf{u},+,\sum _{i=1}^n,\sum _{j=1}^n,({\mathbf{e}}_j^n{\mathbf{r}}^T{\mathcal{E}}_i),(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m^T),\mathbf{u}\right)\hfill \\ \hfill & ={\mathbf{1}}_n^{T}A{\mathbf{e}}_k^n+\frac{\partial }{\partial {\mathbf{r}}_k}\left[{\mathbf{1}}_n^T,\left(C_k,\odot ,({\mathbf{1}}_n{\mathbf{r}}^T{\mathcal{E}}_i),\mathbf{u}\right),+,{\mathbf{1}}_n^T,\sum _{j=1}^n,({\mathbf{e}}_j{\mathbf{r}}^T{\mathcal{E}}_k),(D_{\mathit{kj}}\odot \mathbf{u}{\mathbf{1}}_m),\mathbf{u}\right],\hfill \end{array}$$

where the second equality is due to the fact that the matrix $C_i\odot ({\mathbf{1}}_n^T{\mathbf{r}}^T{\mathcal{E}}_i)$ contains only terms in ${\mathbf{r}}_i$ for each i, and the matrix $({\mathbf{e}}_j^n{\mathbf{r}}^T{\mathcal{E}}_i)$ contains only ${\mathbf{r}}_i$ on the j-th row for each $0\le i,j\le n$. Proceeding, and using the fact that the aforementioned matrices are constant in ${\mathbf{r}}_i$ for each i,

$$\begin{array}{cc}& \frac{\partial }{\partial {\mathbf{r}}_k}\left({\mathbf{1}}_n^T,\frac{\dot{\mathbf{x}}}{N}\right)\hfill \\ \hfill & ={\mathbf{1}}_n^{T}A{\mathbf{e}}_k^n+\frac{\partial }{\partial {\mathbf{r}}_k}\left[{\mathbf{r}}_k,{\mathbf{1}}_n^T,\left(C_k,\odot ,({\mathbf{1}}_n{\mathbf{1}}_n^T{\mathcal{E}}_k),\mathbf{u}\right),+,{\mathbf{r}}_k,{\mathbf{1}}_n^T,\sum _{j=1}^n,({\mathbf{e}}_j{\mathbf{1}}_n^T{\mathcal{E}}_k),(D_{\mathit{kj}}\odot \mathbf{u}{\mathbf{1}}_m),\mathbf{u}\right]\hfill \\ \hfill & ={\mathbf{1}}_n^{T}A{\mathbf{e}}_k^n+{\mathbf{1}}_n^T\left(C_k,\odot ,({\mathbf{1}}_n{\mathbf{1}}_n^T{\mathcal{E}}_k),\mathbf{u}\right)+{\mathbf{1}}_n^T\sum _{j=1}^n({\mathbf{e}}_j{\mathbf{1}}_n^T{\mathcal{E}}_k)(D_{\mathit{kj}}\odot \mathbf{u}{\mathbf{1}}_m)\mathbf{u},\hfill \end{array}$$

and so we conclude that

$$\begin{array}{cc}& \frac{\partial }{\partial \mathbf{r}}\left({\mathbf{1}}_n^T,\frac{\dot{\mathbf{x}}}{N}\right)\hfill \\ \hfill & =A{\mathbf{1}}_n+\sum _{i=1}^n{\mathbf{e}}_i^n{\mathbf{1}}_n^T\left(C_i,\odot ,({\mathbf{1}}_n{\mathbf{1}}_n^T{\mathcal{E}}_i),\mathbf{u}\right)+\sum _{i=1}^n{\mathbf{e}}_i^n{\mathbf{1}}_n^T\sum _{j=1}^n({\mathbf{e}}_j{\mathbf{1}}_n^T{\mathcal{E}}_k)(D_{\mathit{kj}}\odot \mathbf{u}{\mathbf{1}}_m)\mathbf{u}\hfill \\ \hfill & =A{\mathbf{1}}_n+\sum _{i=1}^n({\mathbf{1}}_n^T{C}_i\mathbf{u}){\mathbf{e}}_i^n+\sum _{i=1}^n{\mathbf{e}}_i^n{\mathbf{1}}_n^T\sum _{j=1}^n{\mathbf{e}}_j^n{\mathbf{1}}_m^T(D_{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m)\mathbf{u}.\hfill \end{array}$$

Letting $f(\mathbf{u},\mathbf{r})$ be as in Equation (25), we obtain

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \frac{\partial H}{\partial \mathbf{r}}=Q\mathbf{r}& -(Q^T+Q)\tilde{\mathbf{r}}+({\mathbf{1}}_N^{T}f(\mathbf{u},\mathbf{r}))\mathit{\lambda }\hfill \\ \hfill & +\left(A,{\mathbf{1}}_n,+,\sum _{i=1}^n,({\mathbf{1}}_n^T{C}_i\mathbf{u}),{\mathbf{e}}_i^n,+,\sum _{i=1}^n,{\mathbf{e}}_i^n,{\mathbf{1}}_n^T,\sum _{j=1}^n,{\mathbf{e}}_j^n,{\mathbf{1}}_m^T,(D_{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m),\mathbf{u}\right)\odot \mathit{\lambda }\odot \mathbf{r}\hfill \\ \hfill & +\left(A^T,\mathit{\lambda },+,\sum _{i=1}^n,{\mathbf{e}}_i^n,{(C_i\mathbf{u})}^T,\mathit{\lambda },+,\sum _{i=1}^n,{\mathbf{e}}_i,{\left(\sum _{j=1}^n,{\mathbf{e}}_j^n,{\mathbf{1}}_m^T,(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m),\mathbf{u}\right)}^T,\mathit{\lambda }\right).\hfill \end{array}\end{array}$$

E.2 Optimality Conditions

For the optimality condition, we compute

$$\begin{array}{c}\hfill \frac{\partial H}{\partial \mathbf{u}}=R\mathbf{u}+\frac{\partial }{\partial \mathbf{u}}{\mathit{\lambda }}^T\dot{\mathbf{r}}=R\mathbf{u}+\frac{\partial }{\partial \mathbf{u}}{\mathit{\lambda }}^T\left(\left({\mathbf{1}}_n^T,\frac{\dot{\mathbf{x}}}{N}\right),\mathbf{r},-,\frac{\dot{\mathbf{x}}}{N}\right).\end{array}$$

Notice that, as in Appendix E.1, the term given by $(\partial /\partial \mathbf{u})\left({\mathit{\lambda }}^T,\frac{\dot{\mathbf{x}}}{N}\right)$ is equal to that given in the optimality condition for $\mathbf{x}$ with $\mathbf{x}$ replaced by $\mathbf{r}$. Therefore, we only need to compute the first term. Notice that

$$\begin{array}{c}\hfill \frac{\partial }{\partial {\mathbf{u}}_k}{\mathit{\lambda }}^T\left({\mathbf{1}}_n^T,\frac{\dot{\mathbf{x}}}{N}\right)\mathbf{r}=\frac{\partial }{\partial {\mathbf{u}}_k}\left({\mathbf{1}}_n^T,\frac{\dot{\mathbf{x}}}{N}\right){\mathit{\lambda }}^T\mathbf{r},\end{array}$$

and

$$\begin{array}{cc}& \frac{\partial }{\partial {\mathbf{u}}_k}\left({\mathbf{1}}_n^T,\frac{\dot{\mathbf{x}}}{N}\right)\hfill \\ \hfill & =\frac{\partial }{\partial {\mathbf{u}}_k}{\mathbf{1}}_n^T\left(A,\mathbf{r},+,\sum _{i=1}^n,C_i,\odot ,({\mathbf{1}}_n{\mathbf{r}}^T{\mathcal{E}}_i),\mathbf{u},+,\sum _{i=1}^n,\sum _{j=1}^n,({\mathbf{e}}_j^n{\mathbf{r}}^T{\mathcal{E}}_i),(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m^T),\mathbf{u}\right)\hfill \\ \hfill & =\frac{\partial }{\partial {\mathbf{u}}_k}{\mathbf{1}}_n^T\left(\sum _{i=1}^n,C_i,\odot ,({\mathbf{1}}_n{\mathbf{r}}^T{\mathcal{E}}_i),\mathbf{u},+,\sum _{i=1}^n,\sum _{j=1}^n,({\mathbf{e}}_j^n{\mathbf{r}}^T{\mathcal{E}}_i),(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m^T),\mathbf{u}\right).\hfill \\ \hfill \end{array}$$

From Equation (B2), one can deduce that

$$\begin{array}{c}\hfill \frac{\partial }{\partial \mathbf{u}}{\mathbf{1}}_n^T\left(\sum _{i=1}^n,C_i,\odot ,({\mathbf{1}}_n{\mathbf{r}}^T{\mathcal{E}}_i),\mathbf{u}\right)={\left(\sum _{i=1}^n,C_i,\odot ,({\mathbf{1}}_n{\mathbf{r}}^T{\mathcal{E}}_i)\right)}^T{\mathbf{1}}_n.\end{array}$$

Likewise, from Equation (B4), one obtains that

$$\begin{array}{c}\hfill \frac{\partial }{\partial \mathbf{u}}{\mathbf{1}}_n^T\left(\sum _{i=1}^n,\sum _{j=1}^n,({\mathbf{e}}_j^n{\mathbf{r}}^T{\mathcal{E}}_i),(D^{\mathit{ij}}\odot \mathbf{u}{\mathbf{1}}_m^T),\mathbf{u}\right)=\left(\sum _{i=1}^n,\sum _{j=1}^n,{({\mathbf{e}}_i^n)}^T,\mathbf{r},{\mathbf{1}}_n^T,{\mathbf{e}}_j^n,D^{\mathit{ij}}\right)\mathbf{u}.\end{array}$$

Put together,

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \frac{\partial H}{\partial \mathbf{u}}=R\mathbf{u}& +({\mathit{\lambda }}^T\mathbf{r})\left({\left(\sum _{i=1}^n,C_i,\odot ,({\mathbf{1}}_n{\mathbf{r}}^T{\mathcal{E}}_i)\right)}^T,{\mathbf{1}}_n,+,\left(\sum _{i=1}^n,\sum _{j=1}^n,{({\mathbf{e}}_i^n)}^T,\mathbf{r},{\mathbf{1}}_n^T,{\mathbf{e}}_j^n,D^{\mathit{ij}}\right),\mathbf{u}\right)\hfill \\ \hfill & -{\left(\sum _{i=1}^n,C_i,\odot ,({\mathbf{1}}_n{\mathbf{x}}^T{\mathcal{E}}_i)\right)}^T\mathit{\lambda }-\left(\sum _{i=1}^n,\sum _{j=1}^n,{({\mathbf{e}}_i^n)}^T,\mathbf{r},{\mathit{\lambda }}^T,{\mathbf{e}}_j^n,D^{\mathit{ij}}\right)\mathbf{u}.\hfill \end{array}\end{array}$$

Re-writing this expression now yields

$$\begin{array}{cc}\hfill \frac{\partial H}{\partial \mathbf{u}}& =R\mathbf{u}+({\mathit{\lambda }}^T\mathbf{r}-1)\left(\sum _{i=1}^n,\sum _{j=1}^n,{({\mathbf{e}}_i^n)}^T,\mathbf{r},{\mathit{\lambda }}^T,{\mathbf{e}}_j^n,D^{\mathit{ij}}\right)\mathbf{u}\hfill \\ \hfill & +{\left(\sum _{i=1}^n,C_i,\odot ,({\mathbf{1}}_n{\mathbf{r}}^T{\mathcal{E}}_i)\right)}^T\left(({\mathit{\lambda }}^T\mathbf{r}),{\mathbf{1}}_n,-,\mathit{\lambda }\right).\hfill \end{array}$$

Author Contributions

S.M.-P. conceived the project. S.M.-P., R.M.C., and S.W.S.J. developed the model in example 1. S.M.-P., R.E.B., J.K., and P.C.K. developed the model in example 2. S.M-P. and S.W.S.J. performed the analysis for the optimal control in examples 1 and 2. S.W.S.J. wrote the computational scripts used for data analysis and carried out the main analyses. S.M.-P. produced the figures and wrote the article, on which all other authors commented. All authors agreed to the publication of this manuscript.

Funding

S.M.P. would like to thank the Foulkes Foundation for funding. S.W.S.J. receives funding from the Biotechnology and Biological Sciences Research Council (BBSRC) (BB/T008784/1). R.M.C. would like to thank the Engineering and Physical Sciences Research Council (EP/T517811/1) and the Oxford-Wolfson-Marriott scholarship at Wolfson College, University of Oxford for funding. R.E.B. is supported by a grant from the Simons Foundation (MP-SIP-00001828).

Code Availability

Code to perform all numerical computations in this paper is accessible at our GitHub repository: https://github.com/SWSJChCh/multiplicativeControl

Declarations

Competing interests

We declare that we have no competing interests.

In Memoriam: Paul Kulesa

Paul gained his doctorate in mathematical biology under the supervision of Professor J.D. Murray. As a theoretician, Paul felt frustrated that model predictions were rarely tested so he trained as an experimentalist under the guidance of Professor Scott Fraser. This gave him the powerful set of combined skills in being able to determine which biological questions were amenable to mathematical modelling and which were not. Thus began a fruitful collaboration between us and the Kulesa laboratory that spanned over a decade. During this time, we combined theory with experiment to unearth a number of key findings in neural crest biology. Paul was an excellent scientist, always full of new ideas and very passionate about combining mathematics with biology to lead to insights that could not be achieved by either discipline alone. He was also very warm, kind and supportive, with a cheeky sense of humour that always made us laugh. He leaves behind his wife, Jennifer Kasemeier-Kulesa, who is also a Senior Researcher in his lab, and five children. Paul will be sorely missed by all who knew him, but he leaves behind a rich legacy of scientific work that will live long into the future.