An Energy-Optimal Approach for Entrainment of Uncertain Circadian Oscillators

Abstract

We develop an approach to find an energy-optimal stimulus that entrains an ensemble of uncertain, uncoupled limit cycle oscillators. Furthermore, when entrainment occurs, the phase shift between oscillators is constrained to be less than a predetermined amount. This approach is illustrated for a model of Drosophila circadian activity, for which it performs better than a standard 24-h light-dark cycle. Because this method explicitly accounts for uncertainty in a given system and only requires information that is experimentally obtainable, it is well suited for experimental implementation and could ultimately represent what is believed to be a novel treatment for patients suffering from advanced/delayed sleep-phase syndrome.

Introduction

Synchronization and entrainment are important topics of study with biological relevance including insulin secretion in pancreatic cells (1,2), synchronization of neural spiking in Parkinson’s disease (3–5), and circadian entrainment for maintenance of rhythmic physiological functions (6–8). Researchers studying circadian oscillations have found entrainment particularly important, inasmuch as irregularities in circadian regulation have been shown to be associated with a variety of maladies such as cancer, psychiatric disorders, and cardiovascular disease (9–14).

Here, we focus our attention on a condition known as advanced/delayed sleep-phase disorder in which individual circadian rhythms display free-running periods that are too different from normal. In the absence of external stimuli such as light, most people display internal circadian oscillations with natural periods that are close to 24 h. This similarity to the earth’s rotational period allows one’s circadian oscillations to entrain to a 24-h cycle with external perturbations from natural sunlight, or even controlled candlelight in an experimental setting (15). However, few human circadian clocks have an intrinsic period of exactly 24 h in the absence of environmental cues. Indeed, the individual free-running period of one’s circadian clock has been linked to his or her self-identification as a “morning person” or “night owl” (16,17). It has also been suggested that abnormally long free-running periods of oscillation may contribute to delayed sleep-phase syndrome in adolescents (18), and far shorter than usual free-running periods have been connected to familial advanced sleep-phase syndrome (19,20). Individuals affected by these sleep disorders have a difficult time falling asleep at a conventional time, leading to insomnia and excessive daytime sleepiness, disrupting work and family life.

In this article, we develop a mathematical procedure to find an efficient, exogenous perturbation to entrain an ensemble of uncertain circadian oscillators. Here, we take the term “uncertain” to mean that the system parameters are not fully known, as might be the case during in vivo experimentation. We further require that when entrainment is achieved, the phase difference is small when compared to some reference oscillator. An advantage of this method is that it does not need the full model dynamics, but actually only requires approximate knowledge of the system’s infinitesimal phase-response curve (PRC) (6,21–25), which is experimentally measurable by perturbing an oscillatory cycle at different phases, and determining the change in timing of an event (26).

Uncertainty in these models can lead to variation in both the shape of the PRC and the free-running period of oscillation, and unlike Zlotnik and Li (27), this work explicitly allows for uncertainty in the system with the additional guarantee that the phase dynamics will be close to that of a desired reference cycle. Also, unlike Bagheri et al. (28,29), this method does not require feedback from the system and can be administered in an open-loop manner.

The organization of this article is as follows: in Energy-Optimal Entrainment with a Limited Phase Shift between Oscillators, we develop a method for entraining an uncertain oscillator to a desired reference oscillator. The Model describes the specific model we will used to illustrate this control strategy, with Results and Discussion presented next, followed by Conclusion.

Energy-Optimal Entrainment with a Limited Phase-Shift Between Oscillators

Suppose we have two oscillators defined by

$$\begin{array}{l}{\dot{\theta }}_1={\omega }_o+Z\left({\theta }_1\right)u\left(t\right),\hfill \\ {\dot{\theta }}_2={\omega }_o+\Delta \omega +\left(Z\left({\theta }_2\right)+\Delta Z\left({\theta }_2\right)\right)u\left(t\right),\hfill \end{array}$$ (1)

where θ₁ ∈[0,2π) is the phase of a nominal reference oscillator with angular frequency ω_o, PRC Z(⋅), and external control u(t) = ϵu₁(t), where u₁(t) is of order 1 and ϵ is a small, positive constant; and θ₂ ∈[0,2π) is the phase of another oscillator with natural frequency that may differ from nominal by Δω ∈[−Δω₋, Δω₊], and PRC that may differ from nominal by –E₋(θ) ≤ ΔZ(θ) ≤ E₊(θ). Here, the functions E_(θ) and E₊(θ) define an envelope in which the second oscillator’s PRC might exist (see Fig. 1). In Eq. 1, θ₁ captures the nominal properties of the system and θ₂ is necessary to account for uncertainty. We note that θ₂ could also represent an expected range of properties in a heterogeneous population of oscillators. In this sense, the following analysis is valid for an ensemble of heterogeneous oscillators, each within the expected range; note that we assume that Δω is an $\mathcal{O}$(ϵ) term. Clearly if Δω ≠ 0, in the absence of external stimuli these two oscillators will not achieve frequency synchronization (i.e., θ₁ ≠ θ₂), inasmuch as their natural frequencies differ. Suppose we want to design a T-periodic, energy-optimal stimulus (in the sense that $\int u{\left(t\right)}^{2}dt$ is minimized), which will stably entrain θ₁ and any θ₂ such that when the oscillators are entrained, θ₂ − θ₁ $\mathcal{O}$(ϵ)∈$\mathcal{D}$, where $\mathcal{D}$ ≡ [Δθ₋, Δθ₊]. In other words, we want θ₂ − θ₁ to be at most within $\mathcal{O}$(ϵ) of $\mathcal{D}$ when the oscillators are entrained. Note that in this article, we will work exclusively with infinitesimal PRCs, as opposed to finite PRCs. Finite PRCs describe the phase shift, Δθ, that results when a given (usually pulsed) input is applied at θ. Infinitesimal PRCs are related to the gradient of the phase near the periodic orbit and are generally more useful than finite PRCs because they are independent of the magnitude of the control input, provided the control input is small enough. For a more complete discussion on the distinction between infinitesimal and finite PRCs, we refer the reader to Izhikevich (25) and Sacré and Sepulchre (30).

Figure 1.

(Solid line) Nominal PRC; (dot-dashed lines) boundaries of the envelope within which the PRC of θ₂ exists.

To begin, we define a new variable, ϕ = θ₂ − θ₁, so that

$$\dot{\varphi }=\Delta \omega +\left[Z\left({\theta }_1+\varphi \right)-Z\left({\theta }_1\right)+\Delta Z\left({\theta }_1+\varphi \right)\right]u\left(t\right),$$ (2)

and we asymptotically expand θ₁ as

$${\theta }_1={\theta }_1^{\left(0\right)}\left(t\right)+ϵ{\theta }_1^{\left(1\right)}\left(t\right)+{ϵ}^2{\theta }_1^{\left(2\right)}\left(t\right)+\dots .$$ (3)

Recall that u(t) is $\mathcal{O}$(ϵ), so that Eq. 1 implies that θ₁⁽⁰⁾(t) = θ₁(0) + ω_ot and that θ₁(t) = θ₁(0) + ω_ot + $\mathcal{O}$(ϵ). For simplicity, we take θ₁(0) = 0, but note that the following analysis can still be performed for θ₁(0) ≠ 0. Substituting Eq. 3 into Eq. 2, we Taylor-expand terms of the form Z(⋅) in powers of ϵ to yield

$$\dot{\varphi }=\Delta \omega +\left[Z\left({\omega }_{o}t+\varphi \right)-Z\left({\omega }_{o}t\right)+\Delta Z\left({\omega }_{o}t+\varphi \right)\right]u\left(t\right)+\mathcal{O}\left({ϵ}^2\right).$$ (4)

Using averaging theory from Guckenheimer and Holmes (31) and Sanders et al. (32) to approximate Eq. 4, we obtain

$$\begin{array}{c}\dot{\phi }=\Delta \omega +\underset{f\left(\phi \right)}{\underbrace{\frac{1}{T}{\int }_0^T\left[Z\left({\omega }_{o}t+\phi \right)-Z\left({\omega }_{o}t\right)\right]u\left(t\right)dt}}+\underset{e\left(\phi \right)}{\underbrace{\frac{1}{T}{\int }_0^T\left[\Delta Z\left({\omega }_{o}t+\phi \right)\right]u\left(t\right)dt}}+\frac{1}{T}{\int }_0^T\mathcal{O}\left({ϵ}^2\right)dt,\\ =\Delta \omega +f\left(\phi \right)+e\left(\phi \right)+\frac{1}{T}{\int }_0^T\mathcal{O}\left({ϵ}^2\right)dt.\end{array}$$ (5)

If Eq. 5 has a stable fixed point, then for small enough ϵ, Eq. 4 has a corresponding stable periodic orbit which remains close to $\mathcal{O}$(ϵ) (32). Therefore, we can achieve our control objective by requiring that

$$\exists {\phi }_0\in \mathcal{D}\text{such}\text{that}0=\Delta \omega +f\left({\phi }_0\right)+e\left({\phi }_0\right)\text{and}\frac{d}{d\phi }{|}_{\phi ={\phi }_o}\left(f\left(\phi \right)+e\left(\phi \right)\right)<0.$$ (6)

Note that the second condition in Eq. 6 is a stability requirement.

To proceed, we use a sufficient condition to achieve the required control:

$$f\left(\Delta {\theta }_{+}\right)+e\left(\Delta {\theta }_{+}\right)<-\Delta {\omega }_{+}$$ (7)

$$f\left(-\Delta {\theta }_{-}\right)+e\left(-\Delta {\theta }_{-}\right)>\Delta {\omega }_{-}.$$ (8)

If these conditions hold, then for any choice of θ₂,

$$\dot{\phi }\left(-\Delta {\theta }_{-}\right)=f\left(-\Delta {\theta }_{-}\right)+e\left(-\Delta {\theta }_{-}\right)+\Delta \omega >0$$

and

$$\dot{\phi }\left(\Delta {\theta }_{+}\right)=f\left(\Delta {\theta }_{+}\right)+e\left(\Delta {\theta }_{+}\right)+\Delta \omega <0.$$

The condition in Eq. 6 follows as a consequence of the intermediate value theorem. An illustration of requirements from Eqs. 7 and 8 is given in Fig. 2. Using Eq. 5, we rewrite Eq. 7 as

$$\begin{array}{l}f\left(\Delta {\theta }_{+}\right)+e\left(\Delta {\theta }_{+}\right)<-\Delta {\omega }_{+},\hfill \\ f\left(0\right)+{\int }_0^{\Delta {\theta }_{+}}{f}^{\prime }\left(s\right)ds+e\left(0\right)+{\int }_0^{\Delta {\theta }_{+}}{e}^{\prime }\left(s\right)ds<-\Delta {\omega }_{+},\hfill \\ \frac{1}{T}{\int }_0^T\left[\left({\int }_0^{\Delta {\theta }_{+}}{Z}^{\prime }\left({\omega }_{o}t+s\right)ds\right)u\left(t\right)+\Delta Z\left({\omega }_{o}t\right)u\left(t\right)+\left({\int }_0^{\Delta {\theta }_{+}}\Delta Z^{\prime }\left({\omega }_{o}t+s\right)ds\right)u\left(t\right)\right]dt<-\Delta {\omega }_{+},\hfill \end{array}$$ (9)

where ′ = d/dθ, and in the last line we use f(0) = 0. Next, we make use of the equality

$${\int }_0^R\Delta Z^{\prime }\left(\theta +s\right)ds=\Delta Z\left(\theta +R\right)-\Delta Z\left(\theta \right)$$

to yield

$$-E_{-}\left(\theta +R\right)-\Delta Z\left(\theta \right)\le {\int }_0^R\Delta Z^{\prime }\left(\theta +s\right)ds\le E_{+}\left(\theta +R\right)-\Delta Z\left(\theta \right).$$ (10)

Letting ${\int }_0^{\Delta {\theta }_{+}}{Z}^{\prime }\left({\omega }_{o}t+s\right)ds\equiv g_{+}\left(t\right)$ and using Eq. 10, we conclude that if

$$\frac{1}{T}{\int }_0^T\left[\left(g_{+}\left(t\right)+E_p\left(u\left(t\right),{\omega }_{o}t+\Delta {\theta }_{+}\right)\right)u\left(t\right)\right]dt<-\Delta {\omega }_{+},$$ (11)

where

$$E_p\left(u,\theta \right)=\{\begin{array}{ll}{E}_{+}\left(\theta \right),\hfill & \text{if}u\ge 0,\hfill \\ -E_{-}\left(\theta \right),\hfill & \text{if}u<0,\hfill \end{array}$$

then Eq. 9, and hence Eq. 7, must hold. Similarly defining

$$g_{-}\left(t\right)\equiv {\int }_0^{-\Delta {\theta }_{-}}{Z}^{\prime }\left({\omega }_{o}t+s\right)ds,$$

we can conclude that

$$\frac{1}{T}{\int }_0^T\left[\left(g_{-}\left(t\right)-E_m\left(u\left(t\right),{\omega }_{o}t-\Delta {\theta }_{-}\right)\right)u\left(t\right)\right]dt>\Delta {\omega }_{-},$$ (12)

where

$$E_m\left(u,\theta \right)=\{\begin{array}{cc}{E}_{-}\left(\theta \right),& \text{if}u\ge 0,\\ -E_{+}\left(\theta \right),& \text{if }u<0\end{array}$$

is a sufficient condition for Eq. 8 to be true.

Figure 2.

Illustration of a sufficient condition for the control objective (24) to hold. The function f(φ) + e(φ) must cross both horizontal dotted lines with a negative slope at some φ ∈[−Δθ₋, Δθ₊]. To ensure this happens, it is sufficient to require f(Δθ₊) + e(Δθ₊) < Δω₊ and f(−Δθ₋) + e(−Δθ₋) < Δω₋.

Thus, our control objective is now to design an input such that Eqs. 11 and 12 are satisfied. Furthermore, if the absolute positions of the oscillators, and not just their relative phases, are important, it is useful to include the constraint

$$\begin{array}{l}{\theta }_1\left(0\right)=\beta ,\hfill \\ {\theta }_1\left(T\right)=2\pi +\beta ,\hfill \end{array}$$ (13)

where β represents the desired absolute position of θ₁ during entrainment. We solve for the optimal control using a Hamilton-Jacobi-Bellman (HJB) approach (33), by first defining a new state variable, z so that

$$\dot{z}=\left[\begin{array}{c}\dot{a}\\ \dot{b}\\ \dot{\theta }\end{array}\right]=\left[\begin{array}{c}\frac{1}{T}\left(g_{+}\left(t\right)+E_p\left(u\left(t\right),{\omega }_{o}t+\Delta {\theta }_{+}\right)\right)u\left(t\right)\\ \frac{1}{T}\left(g_{-}\left(t\right)-E_m\left(u\left(t\right),{\omega }_{o}t-\Delta {\theta }_{-}\right)\right)u\left(t\right)\\ {\omega }_o+Z\left(\theta \right)u\left(t\right)\end{array}\right],$$ (14)

where a and b are auxiliary variables that come from constraints in Eqs. 11 and 12, respectively. For a given initial state, z, the energy-optimal stimulus will minimize

$$J\left(z,u\left(t\right)\right)={\int }_0^T{u}^{2}dt+\gamma q\left(z\left(T\right)\right),$$ (15)

where ${\int }_0^T{u}^{2}dt$ represents the power consumed by the stimulus; q(z(T)) is an end-point cost function, which will be described in further detail in the Results and Discussion section; and γ is a penalizing scalar that determines the relative importance of the terms. We note that Eq. 15 could be modified to include terms other than energy (see Wilson and Moehlis (34) for an example). We consider bounds on the inputs, which might occur during practical implementation, given by u_min ≤ u ≤ u_max, where u_min ≤ 0 and u_max ≥ 0. We define the cost-to-go function, also known as the value function,

$$\mathcal{V}\left(z,\tau \right)=\underset{\begin{array}{c}{u}_{\text{min}}\le u\left(t\right)\le u_{\text{max}}\\ \forall t\in \left[\tau ,T\right]\end{array}}{\text{inf}}J\left(z,u\left(t\right)\right).$$ (16)

We can find the optimal stimulus for Eq. 15 by solving the HJB equation (33),

$$0=\frac{\partial \mathcal{V}}{\partial t}\left(z,\tau \right)+\underset{u_{\text{min}}\le u\left(t\right)\le u_{\text{max}}}{\text{min}}\mathcal{H}\left(z,\nabla \mathcal{V},u\right),$$ (17)

where

$$\mathcal{H}\left(z,\nabla \mathcal{V},u\right)={\left[u\left(t\right)\right]}^2+{\left[\nabla \mathcal{V}\left(z\left(t\right),t\right)\right]}^T\dot{z},$$ (18)

and with end-point boundary condition

$$\mathcal{V}\left(z\left(T\right),t_{\text{end}}\right)=\gamma q\left(z\left(T\right)\right).$$ (19)

Here ▿$\mathcal{V}$ is the gradient of the value function with respect to the state z. The resulting optimal control can be calculated by taking the derivative of the Hamiltonian equation (Eq. 18) with respect to u and setting the result equal to zero, yielding two possibilities:

$$u^{\ast }\left(z,t\right)\in \left\{\text{min}\left(u_{\text{max}},\text{max}\left(-\frac{1}{2}{\left[\nabla \mathcal{V}\left(z\left(t\right),t\right)\right]}^T\left[\begin{array}{c}{g}_{+}\left(t\right)+E_{+}\left({\omega }_{o}t+\Delta {\theta }_{+}\right)\\ g_{-}\left(t\right)-E_{-}\left({\omega }_{o}t-\Delta {\theta }_{-}\right)\\ Z\left(\theta \right)\end{array}\right],0\right)\right),\text{max}\left(u_{\text{min}},\text{min}\left(-\frac{1}{2}{\left[\nabla \mathcal{V}\left(z\left(t\right),t\right)\right]}^T\left[\begin{array}{c}{g}_{+}\left(t\right)-E_{-}\left({\omega }_{o}t+\Delta {\theta }_{+}\right)\\ g_{-}\left(t\right)+E_{+}\left({\omega }_{o}t-\Delta {\theta }_{-}\right)\\ Z\left(\theta \right)\end{array}\right],0\right)\right)\right\}.$$ (20)

The first possibility corresponds to the minimum of the Hamiltonian (Eq. 18) when the stimulus is positive, and the second corresponds to the minimum when the stimulus is negative. The optimal control is the option that gives a lower value of the Hamiltonian. Once Eq. 17 has been solved for $\mathcal{V}$(z(T),t), with appropriate end-point boundary condition (Eq. 19), the optimal control can be obtained using the initial condition z(0) = [0, 0, β]^T.

In summary, the steps required to compute the optimal control are as follows:

• 1.Evaluate the range of PRCs and natural periods in the system.
• 2.In reference to a nominal (i.e., expected) oscillator, determine the range of Δω_o from Eq. 1, as well as E₊(θ), and E₋(θ) (shown in Fig. 1).
• 3.Determine appropriate values of Δθ₊ and Δθ₋, and compute g₊(t) and g₋(t). Note that choosing a smaller value of Δθ₊ and Δθ₋ will result in a smaller phase difference between the entrained oscillators, at the expense of using more energy.
• 4.Compute the cost-to-go function, $\mathcal{V}$(z,τ), by solving Eq. 17 using an end-point cost based on required conditions Eqs. 11–13.
• 5.Using $\mathcal{V}$(z,τ), compute the optimal control according to Eq. 20 by solving Eq. 14 with initial condition z(0) = [0, 0, β]^T.

We caution that if Δθ₊ and Δθ₋ values are chosen to be too small (i.e., the required phase difference between the entrained oscillators is too small), the resulting stimulus u(t) may become large enough that the small input approximation is invalidated. If this is the case, one must either increase Δθ₊ and Δθ₋, or reduce the uncertainty in the system.

Model

To illustrate the control strategies presented above, we use a 10-dimensional model that captures the circadian oscillations of Period (PER) and Timeless (TIM) proteins in Drosophila (35) (commonly known as the fruit fly). For the model’s dynamic equations, we refer the reader to the Appendix. In this model, the presence of light influences the system by increasing the value of the parameter v_d, the doubly phosphorylated TIM protein degradation rate. In total darkness, this model exhibits a stable, nonlinear limit cycle, which arises from negative feedback between the PER and TIM proteins. The equations can be written as a set of differential equations of the form

$$\dot{\mathbf{x}}=F\left(\mathbf{x},\mathbf{p}\left(t\right)\right),$$ (21)

where x is the state of the system, and p(t) is a vector of parameters of the system. When p(t) is set to its nominal, constant value, the model possesses a stable limit cycle with oscillations of selected variables shown in the top panel of Fig. 3.

Figure 3.

(Top panel) Limit cycle oscillations of the concentration of PER mRNA (M_p) and total amount of PER proteins (P_t) for the nominal parameter set. (Bottom-left panel) Nominal period plotted against the value of the parameter k₃ (when all other parameters remain at their nominal values). (Bottom-right panel) Similar plot when k_dN is varied. (Horizontal dotted line) Period for the nominal parameter set.

To apply the methodology presented in the previous section, we need equations of the form of Eq. 1. To this end, we let p(t) = p_o + Δp(t), where p_o represents the nominal parameter set, and Δp(t) represents a small time-varying perturbation. Equation 21 can be rewritten as

$$\frac{d\mathbf{x}}{dt}=F\left(\mathbf{x},{\mathbf{p}}_{\mathbf{o}}+\Delta \mathbf{p}\left(t\right)\right)=F\left(\mathbf{x},{\mathbf{p}}_{\mathbf{o}}\right)+\sum _i\frac{\partial F}{\partial p_i}\Delta p_i\left(t\right).$$ (22)

Performing a phase reduction around the periodic orbit yields (compare to Taylor et al. (36))

$$\begin{array}{c}\frac{d\theta }{dt}=\frac{\partial \theta }{\partial \mathbf{x}}\cdot \frac{dx}{dt},\\ =\frac{\partial \theta }{\partial \mathbf{x}}\cdot \left[F\left(\mathbf{x},{\mathbf{p}}_{\text{o}}\right)+\sum _i\frac{\partial F}{\partial p_i}\Delta p_i\left(t\right)\right],\\ =\omega +\sum _i\sum _i\frac{\partial \theta }{\partial x_j}\cdot \frac{\partial F}{\partial p_i}\Delta p_i\left(t\right),\\ =\omega +\mathbf{p}\mathbf{P}\mathbf{R}\mathbf{C}\left(\theta \right)\cdot \Delta \mathbf{p}\left(t\right),\end{array}$$ (23)

where

$$\mathbf{p}\mathbf{P}\mathbf{R}\mathbf{C}i\left(\theta \right)=\frac{\partial \theta }{\partial p_i}=\sum _j\frac{\partial \theta }{\partial x_j}\cdot \frac{\partial F_j}{\partial p_i}$$

is the ith component of the parameter phase response curve (pPRC); θ ∈[0,2π] is the phase of oscillation; ω = 2π/T, with T being the period of oscillation; and ∂θ/∂x is a vector of PRC corresponding to each state variable in the model. We note that Eq. 23 is of an appropriate form to use the methods from Energy-Optimal Entrainment with a Limited Phase Shift between Oscillators.

In this study, we are interested in entraining an ensemble of uncertain oscillators. To include heterogeneity in the natural periods in this model, we allow for variation in either k₃ or k_dN, which correspond to the rate constant for the formation of the PER-TIM complex, and the rate constant for the degradation of the nuclear PER-TIM complex, respectively. The bottom panels of Fig. 3 show the effect of modifying these parameters on the natural period of oscillation. Increasing k_dN shortens the natural period whereas decreasing k₃ lengthens the natural period. A more complete discussion of this model is given in Leloup and Goldbeter (35).

Results and Discussion

We induce long- and short-period phenotypes in the model of Drosophila circadian rhythms by decreasing the nominal value of the parameter k₃ and increasing the nominal value of k_dN, respectively (See Tables 1 and 2 for parameter sets). For values of k₃ ∈[0.3, 1.2] nM⁻¹ h⁻¹ and all other parameters at nominal, the free-running circadian period takes values T_l ∈[24.14, 29.72] h, with the period increasing monotonically with decreasing k₃. For values of k_dN ∈[0.01, 0.45] h⁻¹ and all other parameters at nominal, the free-running circadian period takes values T_s ∈[18.03, 24.14] h, with the period decreasing monotonically with increasing k_dN.

Table 1.

Relevant parameters for entrainment using v_s

Parameter set	kdN ∈[0.01, 0.1] h−1, kdN = 0.01 h−1, k3 = 1.2 nM−1 h−1, k3 ∈[0.5, 1.2] nM−1 h−1
Natural period	T ∈[21, 27.6] h
ωo	0.260 h−1
Δω+	0.0389 h−1
Δω−	0.0326 h−1
Δθ+	0.52 (2 h)
Δθ−	0.52 (2 h)

Table 2.

Relevant parameters for entrainment using v_d

Parameter set	Short-period phenotypes	Long-period phenotypes
	kdN ∈[0.05, 0.1] h−1, k3 = 1.2 nM−1 h−1	kdN 0.01 h−1, k3 ∈[0.7, 0.9] nM−1 h−1
Natural period	T ∈[21, 22.5] h	T ∈[25.2, 26.2] h
ωo	0.290 h−1	0.245 h−1
Δω+	0.010 h−1	0.005 h−1
Δω−	0.010 h−1	0.004 h−1
Δθ+	0.26 (1 h)	0.13 (0.5 h)
Δθ−	0.26 (1 h)	0.13 (0.5 h)

To design a stimulus that can better entrain these circadian oscillators using procedures from the Methods, we first calculate PRCs using XPPAUT (37), and then calculate pPRC of the form of Eq. 23. For this study, we take the time at which the total amount of the PER protein is at its minimum value to correspond to θ = 0. A few representative pPRCs are shown in Fig. 4. We see from the plot that pPRCs for short phenotypes (red curves) have much more variability than those with long-period phenotypes (blue curves). For our control strategy, we would intuitively like a phase response with little variability and large magnitude, inasmuch as increased variability will degrade the overall control, and a large magnitude will allow for stronger effect of control. In terms of our control objective, we would like to minimize and maximize the left-hand sides of Eqs. 11 and 12, respectively, using the smallest possible deviation from nominal parameters. With this in mind, we define two new values for each parameter p_i,

$${\chi }_{p_i}=\frac{1}{10}{p}_i^{\text{nom}}{\int }_0^{2\pi }\left[S\left(E_{+}\left(\theta \right),E_{-}\left(\theta \right),g_{+}\left(\theta \right),\Delta {\theta }_{+}\right)\right]d\theta ,$$ (24)

$${\kappa }_{p_i}=\frac{1}{10}{p}_i^{\text{nom}}{\int }_0^{2\pi }\left[-S\left(E_{+}\left(\theta \right),E_{-}\left(\theta \right),g_{-}\left(\theta \right),-\Delta {\theta }_{-}\right)\right]d\theta ,$$ (25)

where S(A(θ), B(θ), C(θ), D) ≡ (C(θ) + A(θ + D), −C(θ) + B(θ + D)), g₊(θ), E₊(θ), E₋(θ), Δθ₊, and −Δθ₋ are functions and parameters defined in Methods using the pPRCs for p_i, and p_i^nom is the nominal value of that parameter. Intuitively, Eq. 24 (respectively, Eq. 25) gives the minimum (respectively, maximum) value of the left-hand side of Eq. 11 (respectively, Eq. 12) when the magnitude of the perturbation is constrained to be exactly 10% of the value of p_i. For a given set of PRCs, these values give a quick numerical estimate of the ability to entrain a reference oscillator without solving the more costly HJB equation (Eq. 17). Of the parameters in this model, v_s, the rate of synthesis of PER and TIM mRNA is found to be the best candidate for an entraining stimulus, with

$$\left({\chi }_{v_s},{\kappa }_{v_s}\right)=\left(-0.113,0.195\right).$$

The associated pPRCs are shown in the top-left panel of Fig. 4. For the parameter that is affected by external light, with pPRCs shown in the bottom-left panel of Fig. 4,

$$\left({\chi }_{v_d},{\kappa }_{v_d}\right)=\left(-0.042,0.162\right).$$

Also, corresponding to the parameter in the bottom-right corner of Fig. 4,

$$\left({\chi }_{k_1},{\kappa }_{k_1}\right)=\left(0.021,0.028\right),$$

meaning that there is so much uncertainty in the pPRC that we do not expect to be able to find a stimulus corresponding to this parameter with good entrainment properties.

Figure 4.

Representative parameter PRCs for long- and short-period mutant phenotypes of the Drosophila circadian rhythm. (Black, blue, and red lines) The pPRCs for nominal, long, and short phenotypes. To see this figure in color, go online.

Because v_s was found to be the best candidate for an entraining stimulus, we use this parameter to implement the control strategy detailed in Energy-Optimal Entrainment with a Limited Phase Shift between Oscillators. To determine the envelope E_m and E_p, we assume that the oscillator can have both long and short free-running periods. For this illustration, we use a subset of the possible long- and short-period phenotypes. The nominal parameters for the short phenotypes include k_dN ∈[0.01, 0.1] h⁻¹ with k₃ = 1.2 nM⁻¹ h⁻¹, with the shortest free-running period being 21 h (see Tables 1 and 2 for parameter sets). The nominal parameters for the long-period phenotypes include k₃∈[0.5, 1.2] nM⁻¹ h⁻¹ with k_dN = 0.01 h⁻¹, with the longest free-running being 27.6 h. The nominal free-running period is taken to be 24.14 h. A summary of important values for solving the optimal control problem is given in Table 1. For this example, because there is a relatively large amount of uncertainty, we allow for a relatively large Δθ₋ = Δθ₊ = 0.52, which corresponds to a trapping region of ±2 h from nominal. The HJB (Eq. 17) is solved using u_min = −0.25 nM/h, u_max = 0.25 nM/h, γ = 1.5, and end-point cost,

$$q\left(a\left(T\right),b\left(T\right),\theta \left(T\right)\right)=P_{\theta }+P_{a,b},$$ (26)

where

$$\begin{array}{c}{P}_{\theta }=1/\left(1-\text{exp}\left(-5{\left(\theta \left(T\right)-2\pi \right)}^2\right)\right),\\ P_{a,b}=1/\left(1+\text{exp}\left(-5\left(\mathcal{D}\left(a\left(T\right),b\left(T\right)\right)-0.7\right)\right)\right),\end{array}$$

and

$$\mathcal{D}\left(a,b\right)=\{\begin{array}{ll}-d\left(\left(a,b\right),{\mathcal{T}}^c\right)\hfill & \text{if}\left(a,b\right)\in \mathcal{T}\hfill \\ d\left(\left(a,b\right),\mathcal{T}\right)\hfill & \text{if}\left(a,b\right)\in {\mathcal{T}}^c.\hfill \end{array}$$

Here d(y,Ω) = inf_w∈Ωd(y,w), where d is taken to be the 2-norm, and Ω is a set of points, $\mathcal{T}$ = {(a,b)|a < −Δω₊ and b > Δω₋}, and $\mathcal{T}$^c is its complement. Note that $\mathcal{D}$(a,b) is the signed distance between (a,b) and the target set. Using this end-point cost gives one penalty for not reaching the desired final value of θ and an independent penalty for not reaching the desired values of a and b mandated by Eqs. 11 and 12. Note that in this example, we have taken β from Eq. 13 to be zero. In the results that follow, we use Δp_i to represent the deviation of the parameter p_i from its nominal value.

Results are shown in Fig. 5. The top-left panel shows the envelope for which the pPRCs for the parameter v_s can exist as a shaded region, with the nominal PRC plotted as a solid line. The top-right panel shows the resulting optimal control. The middle panel plots the time-averaged, periodic function

$$f\left(\frac{24\phi }{2\pi }\right)+e\left(\frac{24\phi }{2\pi }\right)$$

for various oscillators with allowed long- and short-period mutants. Note that 24φ/2π gives the phase difference measured in hours. For this problem, Δω₊ and Δω₋ correspond to 0.0389 h⁻¹, and 0.0326 h⁻¹, respectively, and are represented as horizontal dashed lines. Note that all of the averaged curves cross these horizontal thresholds within a ±2 h phase difference. The bottom-right panel shows the phase difference between various mutant phenotypes and nominal using Eq. 2 with control applied repeatedly. Our control strategy guarantees the existence of a stable periodic orbit, approximately satisfying

$$-2<\frac{24\phi }{2\pi }<2\text{h,}$$

and we see that this is the case for all of these oscillators.

Figure 5.

Optimal control for a family of long- and short-period mutants. Long- and short-period mutants are obtained from the parameter set in Table 1. (Top-left panel) Envelope for which the pPRCs for the parameter v_s can exist (shaded region), with the nominal PRC plotted (solid line). (Top-right panel) Resulting optimal control, where Δv_s is the size of the perturbation from the nominal value of v_s. Note that the x axis of the top-left panel has been scaled to facilitate comparison with the top-right panel. (Middle panel) Time-averaged, periodic function f(24φ/2π) + e(24φ/2π) for various oscillators with long- and short-period mutants. (Horizontal, dashed lines) Targets that the averaged function must reach with 24φ/2π between ±2 to achieve the desired control. (Bottom-left panel) Phase of various mutant phenotypes (θ₂ from Eq. 1) simulated with control applied repeatedly; (bottom-right panel) plots of φ from Eq. 2 for the phenotypes with the fastest and slowest natural period.

Next, we apply the optimal control to the full system of equations, and compare to natural entrainment by sunlight. Results are shown in Fig. 6. We find that when sunlight is used to modulate v_d, there is an ∼7 h difference between the phase of the fastest oscillator and the slowest oscillator. When the optimal control of v_s is used in the absence of sunlight, there is an ∼3 h difference between the fastest and slowest oscillators. We note that the free-running periods vary by a large amount (between 21 and 27.6 h), and that even tighter control could be obtained if there was less variability in either the natural frequencies or the PRCs.

Figure 6.

Control of long and short phenotypes using v_s. (Top two panels) Time evolution of P_t for long (blue curves), short (red curves), and normal (black curve) phenotypes when they are entrained by sunlight through modulation of v_d. (Bottom two panels) Time evolution of P_t for long (blue curves), short (red curves), and normal (black curve) phenotypes when they are entrained through periodic, optimal modulation of the parameter v_s from nominal, without sunlight modulation. (Gray-shaded regions) Distribution of when the phase, θ, reaches zero on one cycle. All plots are shown after oscillations have reached steady state. To see this figure in color, go online.

We then use control of v_d, as might be implemented with light pulses, to determine an optimal control for a set of short-period phenotypes. We note that v_d is a more difficult parameter to use, because light pulses can only increase the value of v_d so that u_min = 0. Also, as can be seen in the lower-left panel of Fig. 4, the pPRC varies by a large amount as the parameters in the model change. Nevertheless, light is a convenient and noninvasive way to influence a circadian oscillator.

For these examples, we choose two separate families of long- and short-period mutant phenotypes and assume that we do not know the exact free-running period or the exact pPRC. We take short-period oscillators to have k_dN ∈ [0.05, 0.1] h⁻¹. The resulting free-running periods are between 21.0 and 22.5 h, and we take the nominal period to be 21.7 h so that ω_o = 0.290 h⁻¹. We also take Δθ₋ = Δθ₊ = 0.26, corresponding to a trapping region of ±1 h from nominal. For the family of long-period oscillators, we take k₃ ∈[0.7, 0.9] nM⁻¹ h⁻¹. The resulting free-running periods are between 25.2 and 26.2 h, and we take the nominal period be 25.7 h so that ω_o = 0.245 h⁻¹. We also take Δθ₋ = Δθ₊ = 0.13, corresponding to a trapping region of ±0.5 h from nominal. A summary of important values for solving the optimal control problem are listed in Table 2. The HJB equation (Eq. 17) is solved using u_min = 0 nM/h, u_max = 1 nM/h, and γ = 15, and u_min = 0 nM/h, u_max = 0.7 nM/h, and γ = 15 for the short- and long-period phenotype end-point costs (Eq. 26), respectively. Results are shown in Fig. 7, with the left and right sides corresponding to the control problem for the short-and long-period phenotypes, respectively. The top-left panels of each side give the envelopes for which the pPRCs for the parameter v_d can exist as a shaded region, with the nominal PRC plotted as a solid line.

Figure 7.

(Left) Optimal control for a family of short-period mutants. (Right) Optimal control for a family of long-period mutants. The structure of the left and right sides is identical to that of Fig. 5. Recall that the control guarantees a stable periodic orbit approximately satisfying −1 < 24φ/2π < 1 h and −0.5 < 24φ/2π < 0.5 h for the short-and long-period phenotypes, respectively.

Note that compared to the previous example, the target v_d is not as ideal for entraining these oscillators, and the variability in the family of oscillators that can be entrained has been limited accordingly. The top-right panels of each side show the resulting optimal stimulus. Note that the optimal control for the long and short phenotypes is qualitatively similar, with the short-period phenotypes receiving a larger relative stimulus during subjective dusk to slow down the oscillations, and the long-period phenotypes receiving a larger relative stimulus in the subjective early morning to speed up the oscillations. Both pulses are important, however, to account for uncertainty, for which we will give an intuitive explanation later. The middle panel of both sides shows the time-averaged periodic function

$$f\left(\frac{24\phi }{2\pi }\right)+e\left(\frac{24\phi }{2\pi }\right)$$

for three different oscillators, with horizontal lines representing Δω₊ and Δω₋ for the short-period phenotypes, and within a ±0.5 h phase difference for the long-period phenotypes. The bottom-left panel of each side shows the phase of various mutant phenotypes (θ₂ from Eq. 1) simulated with control applied repeatedly, and the bottom-right panel of each side shows plots of ϕ from Eq. 2 for the fastest and slowest phenotype from each simulation.

Finally, we apply the optimal control to the full system of equations, and compare to the entrainment caused by a 24-h light-dark cycle, with results shown in Fig. 8. When the oscillators are entrained to a 24-h light-dark cycle, the short (respectively, long) period oscillators lead (respectively, lag) the normal period oscillator by as much as 4 h (respectively, 1 h) at θ = 0. When the short (respectively, long) period oscillators are entrained to their respective optimal control, the phase differs from the normal period oscillator by only 1.5 h (respectively, 0.2 h) at θ = 0.

Figure 8.

Control of long and short phenotypes using v_d. (Left) Optimal control for a family of short-period mutants. (Right) Optimal control for a family of long-period mutants. The structure of the left and right sides is identical to that of Fig. 6 (red, blue, and thick black lines correspond to short, long, and nominal phenotypes, respectively). (Gray-shaded regions) Distribution of when the phase, θ, reaches zero in one cycle; (vertical dotted line) time at which the phase of the nominal oscillator reaches zero. Note that the nominal value of v_d is 2 nM/h, and all plots are shown after oscillations have reached steady state. We find that the optimal control for the long and short phenotypes is qualitatively similar, with the short-period phenotypes receiving a larger relative stimulus during subjective dusk and the long-period phenotypes receiving a larger relative stimulus in the subjective early morning. To see this figure in color, go online.

Using the same model of Drosophila circadian rhythms, Bagheri et al. (29) report that short- and long-period mutant phenotypes are best entrained to a 24-h cycle with bright light after subjective dusk and early subjective morning, respectively. These external stimuli are effective because they are appropriately timed to speed up the oscillation of the long-period phenotype and slow down the oscillation of the short-period phenotype. These strategies are appropriate for a closed-loop control strategy where small errors can be fed back to the controller and corrected by the control algorithm. In this study, it is somewhat unexpected to find that for an open loop control strategy, it is important to administer light at subjective dusk and subjective morning for both short- and long-period phenotypes. Furthermore, for a short (respectively, long) phenotype, it is important for the pulse at subjective dusk (respectively, early morning) to be more intense than the pulse at subjective early morning (respectively, dusk). The more intense pulse has the effect of correctly advancing or delaying the oscillation, but both pulses are important to allow for uncertainty in the system’s natural period. The intuition behind this apparently new control strategy is best explained by considering a heterogeneous group of oscillators. This control strategy works by applying a positive stimulus at phases θ where Z′(θ) is expected to be negative for all oscillators in the ensemble. Doing so will yield a negative Lyapunov exponent (compare to Eq. 5) causing the trajectories of each oscillator to exponentially converge. Upon entrainment, the exponential convergence caused by the stimulus balances the linear divergence caused by the Δω term.

It is interesting to note the similarities between our findings and those of Pfeuty et al. (38), which reports circadian oscillators can be robustly entrained to varying light pulses when their PRCs have dead zones (i.e., zones in which Z(θ) = 0), punctuated by zones in which the derivative of the PRC changes rapidly. The method of robust entrainment described in this article requires that the derivatives of the PRCs are large at various locations relative to the uncertainty in the PRC envelope. If the derivatives are too small compared to the uncertainty, ${\chi }_{p_i}$ from Eq. 24 could be too large or ${\kappa }_{p_i}$ from Eq. 25 could be too small to find an entraining stimulus (we found this to be the case with the pPRC for k₁).

We note that the optimal light stimuli for light perturbations are similar to skeleton photoperiods commonly used in experiments on circadian entrainment. Skeleton photoperiods have been used for half a century (39,40) in experimental studies to maintain entrainment while limiting the direct effects of light on an organism. Although this study was performed using a Drosophila model of circadian activity, the PRCs to light of many organisms are qualitatively similar to the PRCs of Drosophila, and would yield similar optimal entraining controls. We postulate that the optimal qualities of skeleton photoperiods for entraining uncertain circadian oscillators have helped it remain a useful experimental methodology decades after its inception.

Conclusion

We have presented a methodology for entraining an uncertain circadian oscillator to an external stimulus with a guarantee that the phase difference between oscillators is close to a predetermined amount. Using phase reduction (25) and averaging techniques (31,32), we are able to determine conditions for which stable entrainment is guaranteed in a given region (Eqs. 11 and 12). Then, using HJB techniques, we can solve for the optimal stimulus which entrains the required group of oscillators. Using this method, we are also able to find stimuli which entrain groups of uncoupled long- and short-period mutant oscillators and perform better than natural sunlight in keeping the phase of oscillation close to a nominal oscillator.

This method is relatively flexible and can be adapted to handle other important considerations by adding extra terms to the optimization. For instance, the stimuli calculated in this article may be difficult to implement during in vivo experimentation. If it is desired to use a continuous waveform as the control, Eq. 1 could be modified to include a Δu(t) term to allow for uncertainty in the administration of the stimulus. Doing so would add a second error term to the optimization, which could be handled with a simple modification to Eq. 4. Also, this method could be adapted to include predetermined time-dependent perturbations, such as a sinusoidal natural light-dark cycle. We note that if these time-dependent perturbations are of large amplitude, it may be helpful to use methods detailed in Kurebayashi et al. (41), which define PRCs not only for small amplitude perturbations, but also for large-amplitude, slowly varying perturbations. Furthermore, if it is desired to implement a control strategy with a series of discrete pulses instead of a continuous u(t), it would be appropriate to use dynamic programming (33), the discrete analog to the Hamilton-Jacobi-Bellman approach described in this article, to minimize the cost function (Eq. 15).

Although circadian entrainment is an important biological concern, it can also be important to correct circadian misalignment, commonly referred to as “jet lag”, which can be caused by travel through multiple time zones. It has been shown that the application of appropriately timed light pulses can help correct this misalignment, which has led researchers to search for optimal time schedules to quickly reset a misaligned rhythm. For instance, Zhang et al. (42) proposes a control algorithm to correct circadian misalignment in the minimum possible time, and Serkh and Forger (43) developed a separate minimum-time control algorithm that is found to be robust to noise and interindividual variation. Although the problems of designing stimuli to mitigate circadian misalignment and to entrain oscillators to a desired cycle are different from a control perspective, both biological maladies manifest in short-term tiredness and sleep disturbance, and can lead to more serious health consequences in the long term.

This method was illustrated in a model of Drosophila circadian dynamics and has potential applications to the treatment of advanced/delayed sleep-phase syndrome in humans. By designing stimuli to allow long- and short-period phenotypes to maintain closer circadian synchrony to their normal counterparts, it is conceivable that such treatments could alleviate the insomnia and daytime sleepiness associated with this syndrome. There are many different stimuli other than light that have been shown to promote circadian entrainment in animal and human studies, including temperature fluctuations (44,45), melatonin treatments (46,47), and scheduled feeding (48,49). Using a combination of these could be useful for implementing this control strategy in vivo. Furthermore, this methodology could have further applications to other biological problems where entrainment is important, such as in the synchronization of pancreatic cells (1,2) for insulin secretion. This methodology could also be adapted to desynchronize a population of pathologically entrained oscillators, which is thought to be important in the treatment of Parkinson’s disease (4,34,50).

A major benefit of the method detailed in this article is that it explicitly accounts for uncertainties in the PRC and natural period of the oscillator. In an experimental setting, there will always be uncertainty in these measurements, but this method still guarantees an adequate stimulus provided the measurements can be taken with sufficient precision. Furthermore, because this method only requires knowledge of the system that has been shown to be experimentally measurable for control purposes (51), we expect it could be successfully tested in a laboratory setting.

Appendix: Drosophila Circadian Rhythm Model Equations

The dynamic equations for the Drosophila model for circadian oscillations are

$$\dot{M_p}=v_{sP}\frac{K_{IP}^n}{K_{IP}^n+C_N^n}-v_{mP}\frac{M_P}{K_{mP}+M_P}-k_d{M}_P,$$

$$\dot{P_0}=k_{sP}{M}_P-V_{IP}\frac{P_0}{K_{IP}+P_0}+V_{2P}\frac{P_1}{K_{2P}+P_1}-k_d{P}_0,$$

$$\begin{array}{l}\dot{P_1}=V_{IP}\frac{P_0}{K_{IP}+P_0}-V_{2P}\frac{P_1}{K_{2P}+P_1}-V_{3P}\frac{P_1}{K_{3P}+P_1}+V_{4P}\frac{P_2}{K_{4P}+P_2}-k_d{P}_1,\\ \dot{P_2}=V_{3P}\frac{P_1}{K_{3P}+P_1}-V_{4P}\frac{P_2}{K_{4P}+P_2}-k_3{P}_2{T}_2+k_{4}C-v_{dP}\frac{P_2}{K_{dP}+P_2}-k_d{P}_2,\\ \dot{M_T}=v_{sT}\frac{K_{IT}^n}{K_{IT}^n+C_N^n}-v_{mT}\frac{M_T}{K_{mT}+M_T}-k_d{M}_T,\\ \dot{T_0}=k_{sT}{M}_T-V_{1T}\frac{T_0}{K_{1T}+T_0}+V_{2T}\frac{T_1}{K_{2T}+T_1}-k_d{T}_0,\\ \dot{T_1}=V_{1T}\frac{T_0}{K_{1T}+T_0}-V_{2T}\frac{T_1}{K_{2T}+T_1}-V_{3T}\frac{T_1}{K_{3T}+T_1}+V_{4T}\frac{T_2}{K_{4T}+T_2}-k_d{T}_1,\\ \dot{T_2}=V_{3T}\frac{T_1}{K_{3T}+T_1}-V_{4T}\frac{T_2}{K_{4T}+T_2}-k_3{P}_2{T}_2+k_{4}C-v_{dT}\frac{T_2}{K_{dT}+T_2}-k_d{T}_2,\\ \dot{C}=k_3{P}_2{T}_2-k_{4}C-k_{1}C+k_2{C}_N-k_{dC}C,\\ \dot{C_N}=k_{1}C-k_2{C}_N-k_{dN}{C}_N,\end{array}$$ (27)

with total amounts of Period (PER) and Timeless (TIM) proteins, P_t and T_t, given by

$$\begin{array}{l}{P}_t=P_0+P_1+P_2+C+C_N,\hfill \\ T_t=T_0+T_1+T_2+C+C_N.\hfill \end{array}$$

Unless otherwise stated, parameters are symmetrical for the steps involving PER and TIM, with individual values given in Fig. 2 of Leloup and Goldbeter (35).