Designing optimal stimuli to control neuronal spike timing

Abstract

Recent advances in experimental stimulation methods have raised the following important computational question: how can we choose a stimulus that will drive a neuron to output a target spike train with optimal precision, given physiological constraints? Here we adopt an approach based on models that describe how a stimulating agent (such as an injected electrical current or a laser light interacting with caged neurotransmitters or photosensitive ion channels) affects the spiking activity of neurons. Based on these models, we solve the reverse problem of finding the best time-dependent modulation of the input, subject to hardware limitations as well as physiologically inspired safety measures, that causes the neuron to emit a spike train that with highest probability will be close to a target spike train. We adopt fast convex constrained optimization methods to solve this problem. Our methods can potentially be implemented in real time and may also be generalized to the case of many cells, suitable for neural prosthesis applications. With the use of biologically sensible parameters and constraints, our method finds stimulation patterns that generate very precise spike trains in simulated experiments. We also tested the intracellular current injection method on pyramidal cells in mouse cortical slices, quantifying the dependence of spiking reliability and timing precision on constraints imposed on the applied currents.

controlling the activity of neurons is a valuable tool in both the study of neural circuits and in medical applications. Direct electrical stimulation of intact brains, for example, has provided important insight into the function of various neural circuits in information processing and learning (Salzman et al. 1990; Ditterich et al. 2003). More recently, optical methods have been developed for highly precise and noninvasive control of spike timing in neuronal populations via optical uncaging of caged neurotransmitters (Callaway and Yuste 2002; Nikolenko et al. 2007, 2008; Matsuzaki et al. 2008) or via photo-stimulation of light-sensitive ion channels (Boyden et al. 2005; Han and Boyden 2007; Mohanty et al. 2008; Gunaydin et al. 2010; Gradinaru et al. 2010). In all of these methods, however, the computational task of designing the stimulation pattern so as to elicit a desired spike train in a single cell or a group of cells, as faithfully as the hardware allows, given safety constraints, can be complicated (Histed et al. 2009).

In this study, we address the following computational problem: how can we artificially stimulate a neuron (e.g., electrically or optically) so that it will output a target spike train r_target with optimal precision? As an example, consider the case of photo-stimulation. Suppose we have a good understanding of how a certain laser light interacts with the ion channels in a neuron and thus affects its membrane potential and spiking activity. Ours is the reverse question: how to find the best photo-stimulation pattern (best modulation of laser intensity with time) that according to our model will result in the neuron producing a spike train that will likely be close to the desired r_target, while minimizing laser power. The problem is formally similar in the case of electrical stimulation.

This general method of optimal spike-train control can have several applications in neuroscience as well as in neuroprosthetics. As an example, we can use this framework to produce a set of desired spike trains (in vivo or in vitro) in a number of selected neurons out of some network and observe the effect of the produced spikes on the subsequent activity of the neurons in the network or on behavior (Salzman et al. 1990). In particular, this can be very useful in studying connectivity patterns in the network (Petreanu et al. 2007; Wang et al. 2007; Arenkiel et al. 2007; Matsuzaki et al. 2008). Another application can be in studying the role of different possible factors (inhibition, etc.) in producing precisely timed spikes, given biophysical constraints (Mainen and Sejnowski 1995). Model-based optimization methods have also been used in controlling firing rates in the subthalamic nucleus as a therapy for advanced Parkinson's disease (Feng et al. 2007a,b).

Another important application of this general problem is found in sensory neural prosthetics, such as cochlear (Loizou 1998) or retinal implants (Weiland et al. 2005). In such applications, one needs to solve the following problem: given an auditory or visual sensory signal, how can we transduce this information by stimulating sensory neurons (e.g., in the cochlear or optic nerve) to fire in such a way that some percept of the sensory environment can be restored to the patient? More precisely, any sensory neural prosthetic must at least implicitly solve the following sequence of signal-processing problems in real time. First, sense the stimulus (e.g., via a microphone or a video camera). Then, simulate the transduction of the sensory signal into a multineuronal spike train response r (e.g., via a model of cochlear nerve or retinal ganglion cell response properties). Finally, stimulate real nerve cells to fire with the desired output pattern r_target = r. The final step corresponds to the problem that is the focus of the current study.

Of course, the optimal control problem as stated above is ill-posed: for example, if we can inject any arbitrary current into a cell, we can simply make the cell fire with any desired pattern for a short period of time. However, this may damage the cell or radically alter its biophysics. In addition, injecting too high a current from one extracellular electrode targeted at a particular cell will activate neighboring cells. Instead, we need to solve a constrained optimization problem, since there are limitations on the stimuli we can safely apply in any physiological preparation. Thus, as we will see, our task becomes a constrained optimization problem, and as we will show below, we can use fast and efficient optimization methods to solve it. We note that real-time convex optimization methods and their numerous applications, e.g., in control problems, constitute a major new theme in the engineering literature and have been reviewed recently in Mattingley and Boyd (2010).

In methods, we will formalize the general problem, and in results and discussion, we will consider the more concrete cases of electrical, optical, and sensory stimulation.

METHODS

General Formulation

Let us return to the general problem. We assume that we have a model that accurately describes how an artificial stimulus x (such as a time-varying injected current or laser that opens certain ion channels or pumps in the neuron) affects the output spike train of a neuron. Mathematically, this can be formalized by a conditional probability distribution p(r|x), which tells us how the neuron's response (spike train), r, varies as the applied stimulus, x, changes. We also denote the constrained set of allowed stimuli by S (a concrete example of S, which we will use in Optimal Control of Electrical Stimulation for the case of electrical stimulation, is the set of all current traces that never get larger in absolute value than some fixed maximum allowed current). As discussed in the Introduction, such constraints have to be imposed to guard against unwanted physiological damage and other undesirable effects. Suppose we are given a measure of tolerance for how far the actually emitted spike train, r, can be from the target spike train, r_target. We can quantify this by a cost function, C(r, r_target), which becomes larger as r deviates further from r_target. Then, we formalize our problem as finding the x (subject to the constraint space S) such that the average cost, given that we applied x to the system, is minimized

$${\text{x}}_{*}=\underset{\text{x}\in S}{\arg \min }{\langle C\left(r,r_{\text{target}}\right)\rangle }_{p\left(r|\text{x}\right)}=\underset{\text{x}\in S}{\arg \min }{\displaystyle \sum _{r}C\left(r,r_{\text{target}}\right)p\left(r|\text{x}\right)},$$ (1)

where the sum is over all possible spike trains, r. Here and below, “arg min” means the value of the optimized variable, x, which yields the minimum value for the expression. Unfortunately, depending on the precise form of the cost function C(·,·) and the details of the conditional distribution p(r|x), it may be quite challenging to solve this constrained optimization problem to choose the best stimulus x fast enough, especially for online applications, although the general problem can in principle be dealt with if computational time is not a crucial factor. One approach for making the problem more tractable is to use the sharp cost function, C(r, r_target) = −δ(r, r_target). The “delta function,” δ(r, r_target), is defined so that it equals one when the two arguments are exactly the same and is zero otherwise (i.e., for any deviation of r from r_target). Using this cost function, the sum in Eq. 1 collapses and we obtain

$${\text{x}}_{*}=\underset{\text{x}\in S}{\arg \min }-p\left(r_{\text{target}}|\text{x}\right)=\underset{\text{x}\in S}{\arg \min }-\log p\left(r_{\text{target}}|\text{x}\right),$$ (2)

i.e., we need to optimize the likelihood function instead (in writing the last form we used the monotonicity of the logarithm). This will be tractable because we do not need to evaluate an integral involving the likelihood function as in Eq. 1 in each step of the minimization algorithm. Furthermore, in most of the models we consider the log-likelihood log p(r_target|x) is concave (i.e., the cost function minimized in Eq. 2 is convex so that fast convex optimization methods can be used for solving Eq. 2; c.f. appendix b). We mention that even though our convenient choice of the delta function as cost is singular (putting a relatively large cost on very small deviations from the target spike train), this will not be problematic in applications where the optimal stimulus, x_*, does indeed elicit a spike train close to r_target with high accuracy (see, e.g., results). However, when due to the shortcomings of the stimulation method or the circuit physiology we are unable to precisely control all spike times, it will be preferable to devise a more practical cost function that is smoother and is less sensitive to small deviations from r_target; e.g., one that does not significantly vary as long as the spikes in r are close to the spikes in r_target, up to some finite time scale.¹

We will also generalize Eq. 1 to include soft constraints. By “soft constraints,” we mean constraints that do not force the optimal x to be strictly in a certain subset of all possibilities (the subset S) but rather penalize some x's more than others in a gradual way. This can be formalized by adding a penalty term, R(x), to the cost function in Eq. 2,

$${\text{x}}_{*}=\underset{\text{x}\in S}{\arg \min }\left[-\log p\left(r_{\text{target}}|\text{x}\right)+R\left(\text{x}\right)\right].$$ (3)

Equation 3 summarizes our general approach to optimal spike train control. We will discuss a few concrete examples of this approach in the following sections. However, first it is worth noting that the optimization problem (3) is formally identical to the problem of maximum a posteriori (MAP) based decoding of the stimulus x given the observed spike train r, as discussed, e.g., in Pillow et al. (2011). In a decoding problem the task is to estimate the stimulus, x, that gave rise to an observed spike train, r. According to Bayes' rule, the posterior distribution of the stimulus given the observed spike train, p(x|r) is proportional to the product of the likelihood p(r|x) and the prior distribution p(x). Taking the logarithm, we therefore have log p(x|r) = log p(r|x) + log p(x) + const.. In MAP-based decoding, we take the MAP stimulus as the posterior estimate, i.e., the x which maximizes p(x|r), or equivalently

$${\text{x}}_{\text{MAP}}=\underset{\text{x}}{\arg \min }\left[-\log p\left(r|\text{x}\right)-\log p\left(\text{x}\right)\right].$$ (4)

If we take the prior p(x) to be zero for x ∉ S [so that −log p(x) = ∞], and let p(x) ∝ exp [−R(x)] for x∈ S, where R(x) is smooth, then Eq. 4 becomes identical to Eq. 3. Thus the penalty term may be interpreted in this context as a log-prior. We note that the relationship between optimal control and inference and estimation problems has been noted before in the literature, in other settings [see, e.g., Todorov (2008)].

Optimal Control of Electrical Stimulation

Now we will discuss the implementation of the optimization problem Eq. 3 in more specific details for concrete examples. First, let us consider direct electrical stimulation of the circuit via a single stimulating electrode. Our goal is to devise a tractable model of the responses p(r|x), which we now write as p(r|I) where I_t is the injected current at time t (we use I to denote the whole history of the injected current at all times). The model, which we will explicate below, describes the influence of I on the response of the neuron, r, in two stages. The first stage describes how I influences the neuron's membrane potential, and the second stage describes how subsequently the membrane potential affects the neuron's spiking probability. We will start by describing the latter mechanism, as this stage of the model is the same as in the other models we will analyze in the following sections in other settings for spike train control.

We decompose the total membrane potential of the neuron as a sum of two contributions, V_t + h_t, where V_t is the contribution of the injected current (we will model the relationship between V_t and I_t below; see Eq. 7), and h_t accounts for refractoriness and depends only on the history of the spiking activity, r; given this history, h_t is independent of the injected current. We model the spiking activity as a point process with an instantaneous firing rate, λ_t, which depends on V_t + h_t according to the simple model

$${\lambda }_t=f\left(V_t+h_t\right).$$ (5)

In other words, the spiking probability in a small enough time bin of size dt is given by λ_tdt. The nonlinearity f(·) (which we take to be monotonically increasing and approaching zero for large negative values of its argument) can be thought of as representing a soft spiking threshold as we will discuss further below. Using the standard point-process likelihood formula (Snyder and Miller 1991), we thus have

$$\begin{array}{cc}-\log p\left(r|V\right)={\displaystyle \sum _t\left({\lambda }_{t}dt-r_t\log {\lambda }_t\right)}+\text{const}.,& {\lambda }_t=f\left(V_t+h_t\right),\end{array}$$ (6)

where r_t is a binary vector representing the spike train (r_t is 1 if we have a spike in bin t, and 0 otherwise), and we use V to denote the whole history of V_t at all times. The constant term in Eq. 6 does not depend on V. We mention that this model is an instance of a generalized linear model (GLM; Brillinger 1988; McCullagh and Nelder 1989; Paninski 2004; Truccolo et al. 2005; Pillow et al. 2008) and spike-response model (Gerstner and Kistler 2002; Jolivet et al. 2003). Since we will be using the framework of Eq. 2, we will use the fixed target spike train, r_target, in Eq. 6. The spike history contribution, h_t, is then formed by convolving a refractory kernel with the fixed, known target spike train r_target(t);² thus we take h_t to be known and fixed [in particular, we will not need to average over history-dependent variability in h_t here; for a treatment of the latter problem, see Toyoizumi et al. (2009)].

We now turn to the first stage of the model that describes the influence of current on the membrane potential. For a type I neuron (Rinzel and Ermentrout 1989; Tateno et al. 2004), we model V_t as a simple noisy, leaky integrator

$$\frac{dV\left(t\right)}{dt}=-\frac{1}{{\tau }_V}V\left(t\right)+\frac{1}{C}I\left(t\right)+\sigma \epsilon \left(t\right),$$ (7)

where τ_V is the membrane time constant, ε(t) is standard Gaussian white noise, σ sets the strength of this noise, and C is the membrane capacitance divided by a gain factor, which depends on the electrode and is the fraction of the measured current that actually enters the cells. Ideally, all these model parameters should be fit to data for each neuron.

We pause here to note an important relationship between the full model described by Eq. 7 together with Eq. 5 and the widely used integrate and fire model of neurons. In the integrate and fire model, spikes are activated by a sharp threshold mechanism: when the membrane potential reaches the spiking threshold, the neuron spikes, and the potential is immediately set to some reset value well below the threshold. In cases where the differential equation obeyed by the subthreshold membrane voltage is linear (as is the case with Eq. 7), it is possible to write the full solution (i.e., with reset) for the membrane voltage in such a model as a sum of two parts: a part that obeys that differential equation without any reset mechanism, and a part that is given by a sum of negative reset kernels contributed at past spike times.³ This corresponds precisely to the manner in which we have decomposed the membrane potential in Eq. 5, with V_t integrating the current according to Eq. 7 without any reset mechanism; in this analogy, h_t corresponds to the voltage reset mechanism. To describe spiking, however, we have adopted the somewhat different model, Eq. 5, which in the light of this analogy can be thought of as a soft and stochastic threshold mechanism. The advantage of using a soft threshold is that this model is more robust to small changes in parameter settings; in addition, these soft-threshold models are easier to estimate from data (McCullagh and Nelder 1989; Paninski 2004; Truccolo et al. 2005).

Going back to Eq. 7, we discretize time in steps of dt, and using the fact that the noise in Eq. 7 is Gaussian, we obtain for the likelihood

$$-\log p\left(V|I\right)=\frac{1}{2{\sigma }^{2}dt}{\displaystyle \sum _t{\left(V_{t+dt}-V_t+\frac{dt}{{\tau }_V}{V}_t-\frac{dt}{C}{I}_t\right)}^2}+\text{const}.,$$ (8)

where the constant term is independent of I and V (see appendix a for a detailed derivation of Eq. 8). Now to choose the best stimulus current, I*_t, we optimize the penalized log-likelihood of r_target given I_t, under some physiologically imposed constraints:

$$\underset{\left(V.I\right):\left|I_t\right|<I_{\max }}{\min }-\log p\left(r_{t\arg et}|V\right)-\log p\left(V|I\right)+R\left(I\right)$$ (9)

where −log p(r|V) is given by Eq. 6, and the soft constraints are imposed by

$$R\left(I\right)=c{\displaystyle \sum _t{J}_t^{2}dt}.$$ (10)

Here, J_t is a low-pass filtered version of the injected current I

$${\tau }_J\frac{d{J}_t}{dt}+J_t-I_t,$$ (11)

and c is a constant that sets the relative importance of the penalty term R(I). J_t is proportional to the charge built up on an electrode with an RC time τ_J. The current is thus constrained to remain within an acceptable safety range |I_t|< I_max, while the quadratic cost R(I) ensures that no significant charge builds up on the electrode over a time scale τ_J [since high charge densities lead to cell damage; see, e.g., Sekirnjak et al. (2006) for further discussion]. We note that it is also possible to impose a similar hard constraint on J_t as well, in addition to or instead of the quadratic cost Eq. 10. Figure 1 shows an example of joint optimization of V_t (top row) and I_t (middle row) according to Eq. 9. The main features of the optimized current are the sharp rises preceding the spike times followed by sharp inhibitory drops to the minimum allowed current (in Fig. 1; −12 nA) immediately afterwards. Had there been no cost associated to the smoothed energy J_t² (see Eq. 10), the optimal current would stay at the minimum bound throughout most of the silent region, but due to the nonzero cost, Eq. 10, the current rises up to a value closer to zero to minimize that cost.

Fig. 1.

Example of an optimized control current. Top row: predicted voltage trace created by the optimized current. Middle row: optimal current. Bottom row, top: target spike train in gray. Here, the dashed lines show upper and lower current bounds, ±I_max, where I_max = 12 nA. Value of the other model and cost function parameters (see Eqs. 7, 10, and 11) used in this example were C = 20 nF, τ_V = 20 ms, σ = 0.0007 mV/ms, c = 7 × 10⁻⁵ ms/nA², and τ_J = 15 ms. In this simulation, and in all other simulations and tests in this study, we used an exponential function for the threshold nonlinearity of Eq. 5. Bottom row, bottom: spike rasters (in black) at show 20 trial responses to the optimal current injection to test the reliability of the resulting spike trains. This simulation shows that with physiologically reasonable parameters and constraints the algorithm is able to elicit relatively precise spike trains close to the target.

The constraint parameters (I_max, c) should be set according to physiological safety margins and determine, along with the 1/C (which determines the strength of the impact of I_t on V_t), the trade-off between the size of the applied current and the reliability of the output spike train: making I_max larger, for example, leads to larger allowed currents and therefore more reliable spike outputs. Figure 2 shows the effect of constraints (in this case the bounds set by I_max) on the performance of the optimal current. The spike rasters in Fig. 2 show the simulated performance of the neuron after being injected with the optimized current subject to three different values of I_max. When I_max is too low, the current cannot become large enough such that one could induce a spike by a brief, large current pulse. Instead, the optimal current stays at the maximum allowed value for a relatively long time to compensate for the small magnitude. This in turn leads to a reduced temporal resolution in the timing of the induced spikes.

Fig. 2.

Dependence of optimal control on maximum current bound, I_max. First, third, and fifth rows: optimized current injections for I_max = 12, 8, and 4 nA, respectively (all other model and cost parameters and attributes were fixed in all three cases and were the same as those used in Fig. 1). Dashed lines show upper and lower current bounds, ±I_max. Second, fourth, and sixth rows: target spike trains at top (in gray) and rasters of 20 trial responses to test the reliability of the resulting spike train given the optimal currents for those values of I_max. As expected, imposing more restrictive constraints on the current reduces the accuracy of the spike trains elicited by injecting the optimal current.

The astute reader may have realized that in writing Eq. 9 we have apparently deviated from the proposed framework of Eq. 3; since in the present case the artificial stimulus, x, over which we have control is the current, I, and not the pair (V, I), according to Eq. 3 we should have minimized −log p(r|I) + R(I), instead of Eq. 9. This would require us to evaluate the marginal probability

$$p\left(r|I\right)={\displaystyle \int p\left(r|V\right)p\left(V|I\right)dV.}$$ (12)

However, if the voltage noise parameter, σ, is small enough, the integrand in Eq. 12 (considered as a function of V) will be sharply peaked around its maximum, allowing us to approximate the integral using the Laplace method [see, e.g., Kass et al. (1991) and Berger (1993)]. To a first approximation, this will result in

$$\log p\left(r|I\right)\approx \underset{V}{\max }\left[\log p\left(r|V\right)+\log p\left(V|I\right)\right],$$ (13)

up to corrections that are systematically smaller (Paninski et al. 2010). However, using this approximation in Eq. 3 (with I replacing x), one arrives at Eq. 9, and the use of the latter optimization is justified. For truly noisy neurons with large σ (e.g., neurons receiving unknown inputs from many separate sources so that their total input other than I_t can be assumed to be Gaussian), however, the approximation Eq. 13 fails and using Eq. 9 may lead to suboptimal results. In that case, one could potentially use other methods (such as particle filtering) to evaluate the integral in Eq. 12. We will not consider such cases in this paper.

Finally note that, from Eqs. 6, 8, 10, and 11, the cost function of Eq. 9 is jointly convex in (V, I) when f(·) is convex and log-concave (Paninski 2004). This means that we can carry out the minimization in Eq. 9 using efficient gradient-based approaches such as the well-known Newton-Raphson algorithm (Press et al. 1992). Even so, this problem is still in general computationally expensive, as the computational cost of the matrix inversion involved in the Newton-Raphson step generically scales cubically with the temporal duration of the spike train. However, it turns out that if the Hessian matrix of the minimized cost function is banded, we can improve this scaling down to linear computational time. This is true of the Hessian of the cost function (9) (it is in fact block tridiagonal with 2 × 2 blocks), and therefore, we can carry out the corresponding minimization in computational time that scales only linearly with the duration of the spike train. For more details about the form of the Hessian and the fast optimization, method see appendix b.

Generalization to oscillatory neural dynamics.

The framework of Eqs. 7–9 can also be extended to cover neurons that cannot be effectively modeled as leaky integrators. As an example, certain neurons show oscillations and subthreshold resonance in their membrane potential. Such behavior has been observed in thalamic neurons (Jahnsen and Karnup 1994; Puil et al. 1994), neocortical neurons (Gutfreund et al. 1995; Hutcheon et al. 1996; Dickson et al. 2000), and hippocampal neurons (Leung and Yu 1998; Pike et al. 2000). In the simplest case, the state variable of an oscillatory neuron can be modeled as a two dimensional vector W, which obeys a linear dynamics that gives rise to damped oscillations in its two components (Izhikevich 2001; Badel et al. 2008). Linearizations of the Hodgkin-Huxley model in certain parameter regimes also give rise to such dynamics (Koch 1999). In this model, it is only the first component of W that contributes to the membrane voltage and hence the firing rate. If we denote this component by V(t), Eq. 5 is unchanged for this model. The only modification is that the dynamical equation Eq. 7 changes to

$$\frac{d\text{W}\left(t\right)}{dt}=A\text{W}\left(t\right)+\text{B}I\left(t\right)+\epsilon \left(t\right),$$ (14)

where

$$\left(\begin{array}{c}V\left(t\right)\\ Y\left(t\right)\end{array}\right)$$

is the two-dimensional state vector [Y(t) is the second state variable]; B is a two-dimensional vector representing the coupling strength of the current, I(t), to the components of W(t); and A is some 2 × 2 dynamics matrix, which could give rise to damped oscillating modes. Also, normally we can assume that the injected current will only directly enter the equation for V(t) and not Y(t), and therefore we take

$$\left(\begin{array}{c}B\\ \text{o}\end{array}\right)$$

.

Thus the cost function, Eq. 10, and Eq. 6 remain unchanged, but Eq. 8 is modified

$$-\log p\left(\text{W}|I\right)=\frac{1}{2dt}{\displaystyle \sum _t{\left[{\text{W}}_{t+dt}-{\text{W}}_t-dt\left(A{\text{W}}_t+\text{B}{I}_t\right)\right]}^T{C}_{\epsilon }^{-1}\left[{\text{W}}_{t+dt}-{\text{W}}_t-dt\left(A{\text{W}}_t+\text{B}{I}_t\right)\right]+\text{const}.},$$ (15)

where C_ε is the 2 × 2 covariance of the temporally white noise ε

$$C_{\epsilon }=\left[\begin{array}{cc}{\sigma }_V^2& \rho \sqrt{\sigma V\sigma W}\\ \rho \sqrt{\sigma V\sigma W}& {\sigma }_W^2\end{array}\right],$$ (16)

and ρ is the correlation coefficient between the two components. The optimization is formally the same as in Optimal Control of Electrical Stimulation except we have three variables, (W_t, I_t), per time step to optimize over instead of the former two (V_t, I_t). The optimization problem is

$$\underset{\left(\text{W},I\right):\left|I_t\right|<I_{\max }}{\min }-\log p\left(r|V\right)-\log p\left(\text{W}|I\right)+R\left(I\right).$$ (17)

In particular, the fast decoding method referred to in Optimal Control of Electrical Stimulation and discussed in detail in appendix b, can be applied in this case as well. Figure 3 shows a simulated example of optimizing the current injection for this model. In this example, the matrix A had the form

$$A=\left[\begin{array}{cc}-\frac{1}{{\tau }_V}& -\omega \\ \omega & -\frac{1}{{\tau }_V}\end{array}\right].$$ (18)

With this form, and in the absence of the input I(t) and the noise, according to Eq. 14 the components of W will have damped harmonic oscillations with a decay time of τ_V, and a period of 2π/ω. Notice the marked difference between the optimized current in this case and those shown in Figs. 1–2, demonstrating the major difference between oscillatory and leaky dynamics. As expected, the optimized current itself oscillates at the natural (resonance) frequency of the state vector, at which the latter has the highest susceptibility. The exact form of the optimal solution, however, is quite complicated in this case, and it is unlikely that simple, ad hoc approaches for designing optimal stimuli will work acceptably here. One interesting aspect of the result is that in the spike rasters for test trials (testing the performance of the optimized current), the unwanted spikes that are fired are not entirely random. The cell's state vector has a natural oscillation period of 20 ms, which leads to the same periodic tendency in its spike firing. This is the case even if the target spike train does not have a similar periodic pattern; the cell is more likely to fire (unwanted) spikes after about a period since its last firing. This is visible in Fig. 3, bottom row, in the timing of the unwanted spikes.

Fig. 3.

Example of an optimized control current for a neuron with oscillatory dynamics modeled by Eq. 14. In this example, we took the matrix A in Eq. 14 to have the form Eq. 18, with a decay time τ_V = 20 ms and the oscillation period T = 2π/ω = 20 ms. Noise covariance Cε was taken to be proportional to the identity, with σ_V² = 10⁻⁸ mV²/ms. Electrode time constant, τ_J, and the penalty strength, c, were 15 ms and 0.01 ms/nA², respectively. Top row: predicted trace of V(t) [first component of W(t) created by the optimized current]. Middle row: optimized current. Dashed lines show upper and lower current bounds, ±I_max, where I_max = 10 nA (taking B⁻¹ = 20 nF). Bottom row, top: target spike train (in gray) and the raster shows 20 trial responses to the optimal current injection to test the reliability of the resulting spike trains.

We mention that the problem of optimizing currents for eliciting spikes in various neural oscillator models have also been studied using analytical methods in Moehlis et al. (2006); we will discuss this point further in the discussion.

Generalization to many cells and electrodes.

The formalism introduced above easily generalizes to the case of many cells stimulated by many extracellular electrodes. For simplicity, we will focus here on the case of the leaky integrator neurons described by Eq. 7, although the generalization to oscillatory neurons is straightforward. We introduce a N × M matrix of gain factors, R_ij, describing the influence of electrode j on neuron i (N is the number of cells and M is the number of electrodes). In this setting, Eqs. 5 and 7 generalize to

$${\lambda }_i\left(t\right)=f\left(V_i\left(t\right)+h_i\left(t\right)\right),$$ (19)

$$\frac{d{V}_i\left(t\right)}{dt}=-\frac{1}{{\tau }_i}{V}_i\left(t\right)+{\displaystyle \sum _{j=1}^M{R}_{ij}{I}_j\left(t\right)}+{\sigma }_i\epsilon \left(t\right),$$ (20)

with i ranging over all the cell numbers. We have allowed for cells to have different membrane time constants, τ_i, and noise input strengths, σ_i. The objective function of Eq. 9 is also modified accordingly; −log p(r_target|V) and −log p(V|I) are each replaced by a sum over the contributions of different cells, with each cell contributing a term similar to Eqs. 6 and 8, respectively, and R(I) is replaced by a sum over all electrodes of terms similar to Eq. 10. In the case where there is a one-to-one correspondence between electrodes and cells (M = N), and furthermore the electrodes designated for each cell do not affect other cells (diagonal R_ij matrix), the computational cost of the optimization clearly scales linearly with the number of cells. This is because, in this case, the problem of finding the optimal current in each electrode is independent of the others, and we only need to repeat the single-electrode process for all electrodes. However, if the influence of the control agent designated for one cell on the membrane voltage of other cells cannot be completely avoided [e.g., as in extracellular current stimulation (Histed et al. 2009)], the computational time of a naive joint optimization will scale cubically with the number of cells (see appendix b for a discussion of the scaling of the Newton-Raphson optimization), which can make the procedure unfeasible for a large number of cells. It may be possible to improve this undesirable scaling by exploiting the sparse structure of these cross-talk terms (normally, each electrode will affect only a few cells out of all cells of interest); this remains an important direction for future work.⁴ However, our simulations with groups of cells containing up to ∼30 cells show that, even without exploiting sparsity and despite the adverse scaling, the joint optimization algorithm is still tractable for such cases, even for a general coupling matrix R_ij.

Figure 4, A–C and D–F, shows two examples of current optimization in such a setting, for N = M = 6 and N = M = 26, respectively. In these two examples, the matrix R_ij was composed of positive elements only and had the following form. All the diagonal elements were taken to be equal with R_ii = R, while the off-diagonal elements were sampled independently from a uniform distribution on the interval [0,R/5] (resulting in an average of R/10). Figure 4, C and F, shows the result of an approximate solution, obtained by simply ignoring the off-diagonal elements of the R_ij matrix; when optimizing for the current from electrode i, the optimizer assumes this current will only affect cell i and is blind to its effect on the firing of other cells. However, such effects do in fact exist, leading to the suboptimality of the result. In particular, if in the vicinity of the spike times of some cells, some of the currents intended for other cells become negative, the cell's true current input (which normally peaks immediately before the spike times) can become considerably smaller than what the approximate optimizer assumes, leading to a poor success rate in inducing spikes. The effect is obviously more severe when there are more electrodes, as is visible in Fig. 4F where M = N = 26; here the approximate solution is useless. The full method based on joint optimization of all electrode currents, however, has no such problem; it takes into account all the cross-couplings and balances the electrode currents accordingly, achieving a high performance even in this complicated setting. Here, the optimized currents have no simple relationship with the spike trains, and simple, by hand approaches for constructing control currents cannot compete with the optimal control method presented here.

Fig. 4.

Examples of current optimization in the case of many electrodes stimulating many cells. A–F: simulation with 6 (26) cells and 6 (26) electrodes. A and D: target spike trains, with different rows depicting the target spike train of a different cell. B, C, E, and F: top is plot the optimized currents, while bottom is rasters of spike train response of one of the cells in the group (whose target spike train is shown in gray on top of the raster, as well as in the top on each side) to the optimized current, shown together, in 60 test trials. Optimized currents for the electrode corresponding to the cell whose test spike trains are depicted in the raster are plotted with a thick black curve, while the currents for other electrodes are plotted in gray and are thinner and paler to improve visibility (in E and F, the currents in other electrodes are not plotted to avoid confusion). B and E: show these for the case of joint optimization of the current in all electrodes without any approximation. C and F: optimized currents and the resulting rasters for the approximate optimization where only the diagonal of the cell-electrode coupling matrix is retained. For the exact form of this matrix used in these examples, see the main text. Parameters used in these simulations were R = 20 nF, τ_i = 20 ms, σ_i = 0, c = 7 × 10⁻⁵ ms/nA², and τ_J = 20 ms. Examples show that for a large enough number of electrodes, each coupled to more than one cell, the uncoupled approximation fails completely, yet the full optimization according to Eq. 19, by optimally adjusting the input into each cell from all electrodes, can achieve accurate spike trains.

Optimal Control of Photo-Stimulation

Another setting of interest involves photo-stimulation of the circuit instead of direct electrical stimulation. Optical methods for perturbing neural circuits with high spatiotemporal precision have developed dramatically over the last decade (Callaway and Yuste 2002): for example, neurons can be excited (inhibited) locally via optical uncaging of caged glutamate (GABA) compounds (Nikolenko et al. 2007; Matsuzaki et al. 2008; Nikolenko et al. 2008) or via photostimulation following membrane expression of channel rhodopsin-2 (ChR2) or halorhodopsin pumps (NpHR; Boyden et al. 2005; Han and Boyden 2007; Mohanty et al. 2008; Gunaydin et al. 2010).

To develop optimal stimulation protocols in this setting, we again need to start with a model of the neural response p(r|x), where now x corresponds to various light intensity traces L_t^a, where a is an index distinguishing different light colors. Typically each color of light interacts mainly with one kind of ion channel or pump, although there is some nonzero cross-coupling. For example, ChR2 and NpHR are activated by a blue and yellow light, respectively. In the model below, we therefore use the superscript E (I) for both the ChR2 conductance (NpHR pump current) and the blue light (yellow light) that activates it. Again, we use a two-stage model of the response, with the instantaneous firing rate depending instantaneously on the membrane voltage, which in turn is described by a conductance-based leaky integrator model:

$${\lambda }_t=f\left(V_t+h_t\right),$$ (21)

$$\frac{d{V}_t}{dt}=-\frac{1}{{\tau }_t}{V}_t+g_t^{\text{E}}\left(V^{\text{E}}-V_t\right)-I_t^{\text{I}}+\sigma {\epsilon }_t,$$ (22)

where ε_t is Gaussian white noise. Here g_t^E models the conductance of the excitatory ChR2 channels with reversal potentials V^E, and −I_t^I is the inhibitory NpHR pump current. To model the interaction of g_t^E and I_t^I with light, we begin here with a simple first-order model for the dynamics of conductances and pumps driven by photo-stimulation (although, again, more complicated models maybe handled using similar techniques):

$$\frac{d{g}_t^{\text{E}}}{dt}=-\frac{g_t^{\text{E}}}{{\tau }_{\text{E}}}+w_{\text{EI}}{L}_t^{\text{I}}+w_{\text{EE}}{L}_t^{\text{E}},$$ (23)

$$\frac{d{I}_t^{\text{I}}}{dt}=-\frac{I_t^{\text{I}}}{{\tau }_{\text{I}}}+w_{\text{II}}{L}_t^{\text{I}}+w_{\text{IE}}{L}_t^{\text{E}},$$ (24)

where L_t^E and L_t^I are the applied light intensities in the excitatory and inhibitory channels, respectively, and the weights w_ab (where a, b ∈ {I, E}), represent a 2 × 2 gain matrix summarizing how strongly the optical pulse L_t^b influences the conductance g_t^a; this is a function of the optical qualities of the stimulating laser, and the overlap in the absorption spectra of the optical sensors, as well as the concentration of the caged compound in the neuropil or opsin channel expression in the membrane. Depending on the application, if saturation effects turn out to be important, then we should modify Eqs. 23 and 24 into nonlinear equations in which the conductances cannot grow without bound if a large light intensity is used. We will, however, ignore this complication in this study.

Now we may write down our optimization problem. Our major constraint in the electrical stimulation setting was that the applied current I_t was bounded in absolute value; here, the applied light intensities L_t^E and L_t^I are constrained to be nonnegative and bounded above by L_max^E and L_max^I, respectively. Therefore, we want to solve the problem

$$\underset{\left(V,g^{\text{E}},I^{\text{I}}\right):0\le L_{\text{t}}^{\text{I}}\le L_{\max ;}^{\text{I}}0\le L_{\text{t}}^{\text{E}}\le L_{\max }^{\text{E}}}{\min }-\log p\left(r|V\right)-\log p\left(V|g^{\text{E}},I^{\text{I}}\right)+R\left(L^{\text{E}},L^{\text{I}}\right),$$ (25)

where log p(r|V) is again given by Eq. 6, and

$$-\log p\left(V|g^{\text{E}},I^{\text{I}}\right)=\frac{1}{2{\sigma }^{2}dt}{\displaystyle \sum _t{\left\{V_{t+dt}-V_t-dt\left[-\frac{1}{{\tau }_V}{V}_t-I_t^{\text{I}}+g_t^{\text{E}}\left(V^{\text{E}}-V_t\right)\right]\right\}}^2,}$$ (26)

$$R\left(L^{\text{E}},L^{\text{I}}\right)=c_{\text{E}}{\displaystyle \sum _t{\left(L_t^{\text{E}}\right)}^{2}dt+c_{\text{I}}{\displaystyle \sum _t{\left(L_t^{\text{I}}\right)}^{2}dt}},$$ (27)

where we obtained Eq. 26 by discretization of Eq. 22 (similar to Eq. 38), using the fact that t is Gaussian (cf. the derivation of Eq. 8). In writing Eq. 25 as an instance of Eq. 3, we used −log p(r|L^E, L^I) ≈ min_V [−log p(r|V) − log p(V|g^E, I^I)], similar to the approximation Eq. 13 that we used in writing Eq. 9 (cf. the discussion leading to Eq. 13). The soft constraint R(L^E, L^I) controls the total energy of the light pulses, which we need to limit to avoid photodamage. It is implicit in Eq. 25 that when evaluating R(L^E, L^I) (and checking the hard constraints on L^E, L^I) for some value of the optimized vector (V, g^E, I^I), we are using Eqs. 23 and 24 to calculate (L^E, L^I) in terms of (g^E, I^I); the relationship between (L^E, L^I) and (g^E, I^I) is one-to-one and deterministically fixed via Eqs. 23 and 24. For the same reason, one need not add (L^E, L^I) to the list of optimized variables (V, g^E, I^I) in Eq. 25 (see appendix b for more details on the choice of the set of variables with respect to which we optimize the objective). Once the optimization is finished, we can use Eqs. 23 and 24 again to compute the optimal (L^E, L^I), which is what we set out to compute.

The dynamics described by Eq. 21 are nonlinear in the state variables (V, L^E, L^I) due to the multiplication of the conductance, g_t^E, and V_t, and therefore the resulting likelihood term, Eq. 26, and hence the optimization problem, Eq. 25, may be nonconvex. This prevents us from readily using efficient convex optimization methods for solving Eq. 25. However, we can obtain a good initializer by replacing Eq. 22, with the following approximate current-based system:

$$\frac{d{V}_t}{dt}=-\frac{1}{{\tau }_V}{V}_t+g_t^{\text{E}}\left(V^{\text{E}}-\overline{V}\right)-I_t^{\text{I}}+\sigma {\epsilon }_t,$$ (28)

where V̄ is an averaged, nontime-varying voltage (e.g., we can take it to be the membrane's rest potential), and the conditional intensity function λ_t and conductances g_t^a follow the same dynamics defined above. The optimization Eq. 25 is modified to

$$\underset{\begin{array}{c}\left(V,g^{\text{E}},I^{\text{I}}\right):0\le L_t^{\text{I}}\le L_{\max ;}^{\text{I}}0\le L_t^{\text{E}}\le L_{\max }^{\text{E}};\\ V_t\le V^{\text{E}}\end{array}}{\min }\left[-\log p\left(r|V\right)-\log p\left(V|g^{\text{E}},I^{\text{I}}\right)+R\left(L^{\text{E}},L^{\text{I}}\right)\right],$$ (29)

where now the second term

$$-\log p\left(V|g^{\text{E}},I^{\text{I}}\right)=\frac{1}{2{\sigma }^{2}dt}{\displaystyle \sum _t{\left\{V_{t+dt}-V_t-dt\left[-\frac{1}{{\tau }_V}{V}_t-I_t^{\text{I}}+g_t^{\text{E}}\left(V^{\text{E}}-\overline{V}\right)\right]\right\}}^2},$$ (30)

is convex. We are only using Eq. 30 as an auxiliary optimization for obtaining a good initialization for our nonconvex problem (25). However, given that the reversal potential of the excitatory ChR2 channels are well above the subthreshold values of the membrane potential, we can ignore the time variations of the difference V^E − V_t in Eq. 22 in the first approximation; this corresponds to using Eq. 28. Thus in fact the solution of Eq. 29 gives a good approximation to the final solution of Eq. 25. Note that in Eq. 29 we have now also included an upper bound on V_t, since in the original conductance-based model, in the deterministic case (corresponding to σ = 0 in Eq. 22), V_t will never rise above the reversal potential of the excitatory channels V^E, whereas this bound is not respected automatically in the current-based model.

The convex optimization problem, Eq. 29, can be solved with our usual fast barrier techniques once we write the Hessian in block-tridiagonal form, with each block of size 3 × 3 [one dimension each for V_t, g_t^I and g_t^E]. Again, generalizations to the multineuronal case are straightforward. Let us denote the solution to the auxiliary optimization problem (Eq. 29) by [V₍₀₎, g₍₀₎^E, I₍₀₎^I]. Once [V₍₀₎, g₍₀₎^E, I₍₀₎^I] is found, we can use it as an initialization for the original nonconvex problem Eq. 25, which would now be effectively convex if [V₍₀₎, g₍₀₎^E, I₍₀₎^I] were close enough to the true minimum of Eq. 25. However, we saw that depending on the model and constraint parameters sometimes [V₍₀₎, g₍₀₎^E, I₍₀₎^I] was not already in such a locally convex region close enough to the minimum of Eq. 25. On the other hand, our simulations indicate that for a wide range of parameters, this problem can be solved if we carry out the optimization of the full model in two steps. First, initializing [V₍₀₎, g₍₀₎^E, I₍₀₎^I], we optimize the objective in Eq. 25 with respect to (V, I^I) but keep g^E fixed and equal to g₍₀₎^E throughout the optimization; note that even though the objective Eq. 25 is not jointly convex in (V, g^E, I^I) it is convex in (V, I^I). We denote the resulting solution by [V₍₀₎, g₍₀₎^E, I₍₀₎^I]. Finally, we use [V₍₀₎, g₍₀₎^E, I₍₀₎^I] to initialize the full, original optimization, Eq. 25, optimizing all three vectors to obtain the true optimal solution (V_*, g_*^E, I_*^I). As we noted above, we then use Eqs. 23 and 24 to solve for (L_*^E, L_*^I) in terms of the optimized (g_*^E, I_*^I).

Figure 5, A–C, shows the final result for an example of such a three-stage optimization process. The main aspect of the optimized light traces is that for eliciting a spike, an excitatory light pulse is followed by a sharp inhibitory light pulse. The shapes of these pulses are quite stereotypical, except that for spikes that follow other spikes within ≤20 ms, the optimal pulse shapes can depend in a nontrivial way on the duration of the corresponding interspike interval. One aspect of the result worth noting is that controlling inhibition is crucial for eliciting precisely timed spikes; because the intrinsic time scale of the membrane potential, τ_V, is ∼10–20 ms, without the sharp inhibitory light pulses, the voltage peaks generated by excitatory pulses alone would be too wide, staying near threshold on a time scale set by τ_V. This would lead to a (poor) spike timing precision of the same order. On the other hand, with inhibition included, spikes can be elicited with the temporal accuracy of 1–3 ms.

Fig. 5.

Spike train control by light intensity traces, optimized according to the scheme described after Eq. 30 (A–C), and optimized only according to the auxiliary model, Eqs. 28–30 (D–F). C and F, bottom: target spike train (gray spikes). A and D: membrane potential voltage (the sum of the optimized V_t and the history contribution h_t from the target spike train). B and E: optimized light intensity traces. Yellow light, L_t^I, opens inhibitory pumps, and the blue light, L_t^E, opens excitatory channels. Spike rasters (black) in C and F each show 20 trials of simulated responses generated by the full model, described by Eqs. 21–24, after it was exposed to the optimized light traces of B and E, respectively. In A and D, the smooth thin curve shows the voltage trace as optimized according to Eq. 25 (Eqs. 29 and 30), while the slightly deviating thicker and more jagged curve is the actual voltage trace generated stochastically (since σ ≠ 0) according to Eq. 22 in response to the optimized light intensities of B and E. Parameters used in this simulation were: V^E = 20 mV [in Eqs. 21–24, all voltages are measured from the reversal potential of the leak conductance that was taken to be −65 mV in this simulation], σ = 0.36 mV/ms, τ_V = 10 ms, τ^E = 5 ms, τ^I = 10 ms, L_*^E = L_*^I = 100, c^E = c^I = 3 × 10⁻⁵, w_EE = 0.2, w_II = 5, and w_IE = w_EI = 0. We set the average voltage, V, used in the auxiliary model, Eq. 28, equal to −70 mV.

Figure 5, D–F, shows the same setting, but with light intensity traces optimized only according to the auxiliary model in Eqs. 28–30. As opposed to the case of full optimization (Fig. 5), here the actual voltage deviates significantly from the optimized voltage trace, being systematically lower in a window of ∼20 ms after a spike has been fired. As a result, the reliability of induced spikes which trail an initial spike within such a window are significantly degraded. This shows that when the model of the interaction of the stimulating agent and the membrane voltage used in the control optimization (in this case the auxiliary model in Eq. 28) miss important biophysical aspects (in this case the different effects of conductance vs. current input), the results can be highly suboptimal.

Before closing this section, we note that, in the first approximation, we can also use the above model with minimal modification to model how optical stimulation using glutamate and GABA uncaging works. Since GABA activates inhibitory channels, as opposed to pumps, we have to model it similarly to the excitatory conductance in Eqs. 22 and 23. So we rewrite Eqs. 22 and 24 as

$$\frac{d{V}_t}{dt}=-\frac{1}{{\tau }_V}{V}_t+g_t^{\text{E}}\left(V^{\text{E}}-V_t\right)+g_t^{\text{I}}\left(V^{\text{I}}-V_t\right)+\sigma {\epsilon }_t,$$ (31)

$$\frac{d{g}_t^{\text{I}}}{dt}=-\frac{g_t^{\text{I}}}{{\tau }_{\text{I}}}+w_{\text{II}}{L}_t^{\text{I}}+w_{\text{IE}}{L}_t^{\text{E}},$$ (32)

and replace I_t^I in Eq. 26 and the auxiliary Eq. 30 with g_t^I(V^I − V_t) and g_t^I(V^I − V̄), respectively. Now we also need to impose the constraint V_t ≥ V^I when solving the modified auxiliary problem Eq. 29, for reasons similar to those given after Eq. 30. We have to mention, however, that the spatial diffusion of such uncaged neurotransmitters is potentially a significant factor which is ignored in the simplest treatment here, and will need to be considered in a more detailed study.

Optimal Control of a Sensory Stimulus

Another application of the optimal control method, Eq. 3, can be found in cases where one aims to design or control a natural sensory stimulus, x, for inducing a target spike train r_target. In this case, the likelihood function p(r|x) can be interpreted as an encoding model that effectively describes the sensory processing in the neural circuit starting from the periphery to the neuron in which we seek to induce r_target. The GLM framework provides an accurate and commonly used model for this purpose, especially for neurons in early sensory areas (Brillinger 1988; McCullagh and Nelder 1989; Paninski 2004; Truccolo et al. 2005; Pillow et al. 2008).

We need only to minimally modify the formalism to cover this case. In particular, Eq. 6 remains valid. The modification enters in the relationship between V_t (which enters Eq. 5 for instantaneous firing rate) and the stimulus x, which now replaces the current, I_t, in the last section. In the GLM framework, the relationship is provided by a linear filter k_t, representing the cell's “receptive field,”

$$V_t={\text{k}}_t\cdot \text{x}.$$ (33)

Since in this case, the relationship between V_t and the stimulus is deterministic (no noise enters Eq. 33), the second term vanishes when modifying Eq. 9, and we instead should solve the optimization problem

$$\underset{\text{x}:\left|\text{x}\right|\le x_{\max }}{\min }-\log p\left(r_{t\arg et}|V_t={\text{k}}_t\cdot \text{x}\right)+R\left(\text{x}\right),$$ (34)

where the first term in the cost function is given by Eq. 6 with the proper substitutions. As before, R(x) is a cost function similar to that in Optimal Control of Electrical Stimulation, and depending on the problem at hand, can include various different terms that, e.g., may put soft constraints on the power of x or penalize sharp variations. Recall, as discussed above, that there is a close relationship between the optimal control setting in Eq. 3 and MAP decoding in the GLM framework, as studied at more length in Pillow et al. (2011).

Unlike that of Eq. 9, the Hessian of the likelihood term in Eq. 34, is not block tridiagonal. However, due to the finite temporal length of the filter k, the Hessian of this term is still banded; the second derivative of the likelihood term in Eq. 34 with respect to x_t1 and x_t2 is zero when |t₁ − t₂| is larger than twice the temporal length of k. The set of linear equations at each Newton-Raphson iteration, Eq. 41, can be solved in O(T) computational time, even when the equation matrix is not tridiagonal, as long as it is banded [see, e.g., Paninski et al. (2010) and Pillow et al. (2011) for further details]. Thus we retain the O(T) scaling even in this case, as long as the cost function R(x) also has a banded Hessian. This is the case for many commonly used cost functions, including the ones mentioned above.

As an example, in the case of controlling the spiking activity of a retinal ganglion cell, the filter k is the spatiotemporal visual receptive field of the cell, and the stimulus x can be a spatiotemporally varying contrast pattern on a monitor that is optically projected onto the retina. In this case, the sharp bound x_max can represent the bound on the contrast that the monitor can produce. Figure 6 shows such an example of optimizing a visual stimulus for two simulated retinal neurons simultaneously. In this example, we modeled the cells using the GLM parameters fit to experimental recordings, as reported in Pillow et al. (2008). The stimulus had nine pixels (arranged on a 3 × 3 square), covering the cells' receptive fields center and most of the surround. We took the smooth cost function R(I) in Eq. 34 to be zero, but we imposed the rigid bound |x| ≤ x_max, for x_max corresponding to a bound on the contrast of the stimulus. The optimized stimulus for the pixel covering the overlapping receptive field centers of both cells (Fig 6, marked by a gray dot at top left) shows a noticeable difference to that of the other pixels. Namely, unlike the optimized stimulus for the other pixels, it spends more time at intermediate values, away from the rigid bounds ± x_max. This is because spiking probability depends more sensitively on the value of the contrast in the center pixel compared with other pixels, as the receptive field filter is stronger in the center. Hence, the likelihood term in Eq. 34 is much more sharply concentrated as a function of the stimulus values for this pixel, compared with the other ones, and is thus able to confine the optimized x away from ± x_max for this dimension. By contrast, the log-likelihood is relatively flat along the dimensions corresponding to the other pixels, and thus the cost function of Eq. 34 tends to acquire its minimum on the boundaries ± x_max for those dimensions [see Pillow et al. (2011) for further discussion of this effect].

Fig. 6.

Optimizing a visual stimulus (a black and white checkered stimulus updated at 120 Hz) to induce target spike trains in a pair of ON and OFF retinal ganglion cells with more or less coinciding receptive fields. Top row: stimulus, where the 3 × 3 square covered by the stimulus is shown by a 9 pixel column for illustration purposes. Pixel covering the receptive field centers (of both cells) is marked by a gray dot at left. Middle and bottom rows: target spike trains (top, middle, and bottom rows in grey at top) for the ON and the OFF cells and rasters of 20 trial responses (black) of each cell to the optimized stimulus, respectively.

Online Implementation

For many practical applications, the above setup has to be implemented in an online manner. For example, in a sensory neuroprosthesis application where we need to artificially stimulate some sensory neurons to emit the appropriate spike trains in response to the sensory stimulus, the following sequence has to be performed repeatedly. As incoming sensory information keeps arriving, the putative response of the sensory neurons to that stimulus is simulated according to some encoding model for some time ahead; this constitutes the target spike train for the optimal control algorithm. Then, the portion of the optimal artificial stimulus, x, corresponding to this near future time span has to be computed and “injected” to cause the sensory neurons to actually emit a spike train as close to the predicted one as possible. Another conceivable application is suppressing spike firing in neural circuits that exhibit rhythmic (i.e., predictable) spontaneous activity; if a rhythmic series of spikes is detected, one could potentially in real-time compute and deliver the best stimulus to disrupt the unwanted rhythm. In both applications, the difference with the approach of the previous sections is that the earlier portions of the artificial stimulus are optimized earlier, and only with information about the target spike train a finite time into the future, rather than with full knowledge of the spike train for the entire span of the application.

As a first solution, we will carry this out as follows. Suppose we are at real time t = t₀. We have sensed the sensory stimulus up to that time, and based on that observation (possibly taking into account correlations in the stimulus itself, which allow for its probabilistic prediction further into the future), we are able to estimate a prediction for the spike train up to some later time t₀ + τ_rp. Thus at this point, we have access to r_target(t) for t ∈ (−∞, t₀ + τ_rp). We can also assume that there is a feedback mechanism that has measured the actually emitted spike times before t₀ − τ_ro (for some fixed τ_ro > 0, which is a characteristic of the observation mechanism), allowing us to update r_target(t) for t ∈ (−∞, t₀ − τ_ro) such that it agrees with the real (as opposed to predicted) spikes for those times.

At this point, we will run the optimization algorithm to find the artificial stimulus x(t) for t in some subinterval of [t₀, t₀ + τ_rp]. Let us denote this subinterval by I₁ = [t₀ + τ_i, t₀ + τ_f ], where

$$0<{\tau }_i<{\tau }_f<{\tau }_{rp}.$$ (35)

Suppose that the optimization program takes τ_opt1 to run, and at time t₀ + τ_opt1 we have obtained the optimal solution x₁(t) on I₁. At this point we should start the optimization process anew, with the updated r_target (which is now predicted up to t₀ + τ_opt1 + τ_rp) and for the interval I₂ = [t₀ + τ_opt1 + τ_i, t₀ + τ_opt1 + τ_f ], and simultaneously “inject” the optimized x₁(t) during the time that the second optimizaton is running (taking τ_opt2), i.e., on [t₀ + τ_opt1, t₀ + τ_opt1 + τ_opt2]. Clearly, the latter part is only possible if I₁ covers [t₀ + τ_opt1, t₀ + τ_opt1 + τ_opt2]; hence, we must have

$${\tau }_i<{\tau }_{\text{opt}1,}$$ (36)

$${\tau }_{\text{opt}1,}+{\tau }_{\text{opt}2,}<{\tau }_f<{\tau }_{rp.}$$ (37)

These are important bounds on τ_i and τ_f. Finally, at time t₀ + τ_opt1 + τ_opt2 we will start injecting the new solution x₂(t) (found on I₂), instead of the (possible) remainder of x₁(t). This cycle is continued for the entire span of the application. The optimization times τ_opti depend on the computer and amplifier hardware and details of software implementation and can vary from case to case. Figure 7 shows these various time intervals.

Fig. 7.

Sketch of different timings in the online implementation of optimal control. First row: sensory stimulus for all times. Second and third rows: show the predicted and emitted spike trains for all times. Fifth row: actually injected stimulus for all times. Dashed (solid) vertical line crossing the first to fifth rows shows the real time before the first optimization has started (ended). At the start of the first optimization, i.e., at t = t₀, the putative spike train response is predicted up to t₀ + τ_rp (show in second row). Also, the feedback observation mechanism has updated the past spike times to set them to those of the actually emitted spikes before t − τ_ro (shown in the third row). In the first optimization step, the optimization algorithm is carried out with this data for the interval I₁. Fourth row: this interval and the obtained optimal solution x₁(t) on it. When the algorithm finishes, we reach the vertical line (in the first to fifth rows), at which point we inject the obtained solution. The actually injected part of this solution is shown as the gray part of the curve in the fifth panel. At the same time, we start the second optimization process, with updated data, for the interval I₂ (bottom). By the time the second optimal solution, x₂(t) is ready, we stop injecting the first solution x₁(t) (therefore the gray part of the curve in the fifth row does not include the whole solution shown in the fourth row), and use x₂(t) instead (sixth row).

RESULTS

We tested the offline optimal current injection method described in methods in one simple setting using intracellular injections into patch-clamped pyramidal neurons in mouse cortical slices. All animal handling and experimentation was according to the National Institutes of Health guidelines and approved by the Columbia University Institutional Animal Care and Use Committee. Coronal slices 350-μm thick were prepared from C57BL/6 mice at age P14 using a Leica VT1000-S vibratome. Pipette solution contained 130 mM K-methylsulfate 2 mM MgCl₂, 0.6 mM EGTA 10 mM HEPES, 4 mM ATP-Mg, and 0.3 mM GTP-Tris, pH 7.2 (295 mosmol/kgH₂O). All recordings were made at 33–36°C from layer 5 pyramidal neurons in the primary somatosensory cortex using the Multiclamp 700B amplifier (Molecular Devices, Sunnyvale, CA) digitized with a National Instruments 6259 multichannel card and recorded using custom software written using the LabView platform (National Instruments, Austin, TX).

To apply the method, we first need to estimate a model for p(r|I). To do this, we injected 5 min of white noise current (with a sampling rate of 1 kHz) into the cell in current clamp mode, and recorded the emitted spike times (extracted from the recorded voltage trace). Based on this data, we fit the model parameters. In fitting the model to recorded data, we took a slightly different approach from that in methods. The model of Eq. 7, in the absence of noise (i.e., for σ = 0), can be solved explicitly for V_t. The result can be written as the convolution of an exponential filter with the current: V_t = K * I_t, where K (t) = 1 / C exp( − t/τ_V). This means that for σ = 0 this model can be rewritten in the GLM form of Eq. 33, where now the filter has the particular exponential form and is parametrized by τ_V and R. However, instead of fitting these two parameters to the recorded data, for computational purposes we parametrized the kernel as a weighted sum of exponential filters with different, yet known, time constants. I.e., we wrote K(t) = ∑_ik_i exp (−t/τ_i) where we fixed the τ_i beforehand, choosing them to be in the range of 1–100 ms, and we maximized the model likelihood in terms of k_i to find the best fit.⁵ The advantage of this approach is that the parametrization of the log-likelihood in terms of k_i is concave, whereas it need not be so in terms of τ_V, and therefore the likelihood optimization for fitting could be done more rapidly and reliably. With this parametrization, and limiting ourselves to the σ = 0 case (justified because intracellular noise is small), we used the GLM framework of Eq. 34 (with x replaced by I_t and k replaced by the estimated filter) to carry out the optimal control as well. We then ran our optimization program for six different target spike train patterns and for nine different values of the current magnitude bound I_max (see Eq. 9) to study the effect of this constraint on the achieved spike timing precisions and reliabilities. Then, we injected these optimized currents in a series of 40 trials (separated by 0.5-s long silent intervals) to measure the performance. Figure 8, top, shows the optimized current for the highest value of I_max we used. The curve is qualitatively similar to Fig. 1, and differences correspond to differences in model parameters and the penalty strengths. The dependence on I_max (not shown in Fig. 1) is also similar to Fig. 2; as I_max decreases, the peaks of the optimal current broaden, leading to lower spike timing precision.

Fig. 8.

Experimental test of the current optimization method using in vitro intracellular injection into a mouse pyramidal neuron in layer 5 of the primary somatosensory cortex (in a cortical slice). Left: 5 sets of rasters (each based on 40 trials) for 5 different values of the maximum current magnitude I_max (see Eq. 9). Top of each raster shows the (same) target spike train (red spikes). Timing precision and reliability of the elicited spikes is visibly reduced as we constrain the range of current more (i.e., decrease I_max). Right: plots show this more quantitatively based on averages over all the trials of all the tested target spike trains. Right, top and bottom: plots show the timing precision (SD of the trial to trial timing jitter of each spike), and the reliability (percentage of trials in which the target spike was indeed elicited), respectively, as a function of the maximum allowed current. The method is able to achieve submillisecond precision and very high reliability with a biologically safe range of currents.

Because the timing precision was high, association of trial spikes to those in the target spike train was unambiguous. We then defined reliability (success rate for eliciting a desired spike) as the percentage of the trials in which the corresponding spike was indeed elicited within a 3-ms window around the target spike train. The timing precision was defined as the standard deviation, among the trials in which the corresponding trial spike was emitted, of the timing difference of the corresponding trial spike to that in the target spike train. The resulting rasters for one of the spike patterns are shown in Fig. 8. A noteworthy aspect of the results is the fact that spikes trailing other spikes within a short time window are the first that fail to be elicited as we lower I_max. Due to refractory and short-term adaptation effects, the cell is less excitable for a few tens of milliseconds after firing a few spikes, requiring a high current to be driven to spiking, which may not be feasible if I_max is too low. Figure 8, right, shows the dependence of the obtained timing precision and reliability, averaged over all spikes in all target patterns, on the bound I_max. For this simplest of settings the method is capable of achieving sub-millisecond timing precision and very high reliability with biologically safe input levels. The data collection for fitting the model, and the likelihood optimizations for the fit and for optimal control overall do not take >2–3 min, thus allowing the method to be used as a tool in electrophysiology experiments.

DISCUSSION

In this study, we provide a general formulation for the problem of optimal control of neural spiking. We tailored this general framework for a few specific cases of interest involving electric, optical, or sensory stimulation of neurons using relatively simple models that describe the input-output mechanism of the neuron. Importantly, we showed that the method based on these models is amenable to fast convex optimization methods. This property is key for online implementation of these methods (discussed in Online Implementation), which is necessary in most applications. In addition, convexity guarantees that the optimizer will not be caught in a nonglobal, local optimum; it will always find the globally optimal solution. We also noted the close relationship between our optimal control framework and the problem of MAP decoding of spike trains. In many cases of interest, the optimal stimuli we obtain using our methods are quite nonintuitive (see, e.g., the examples in Figs. 3, 4, and 6). Even when the optimal stimuli have an intuitive form, the dependence of important details on the model choice and parameters could be complicated (cf. the discussion of Fig. 5). It is unlikely that in such cases, simple, by-hand approaches for designing stimuli can perform as satisfactorily as a systematic method taking into account the detailed properties of the neurons and the stimulating agents. Furthermore, the example of Fig. 5 shows that when the employed neuronal models miss important qualitative aspects of the biophysics, the stimulus optimized according to them can in fact be considerably suboptimal. At the same time, our investigations show that the optimal stimulus found according to the models used in this study are reasonably robust with respect to reasonable errors in model parameters; as long as the parameters used for optimizing the stimulus do not deviate by a very large amount from the true parameters, the performance of the resulting stimuli is not noticeably compromised. The conclusion we can draw is that as long as the employed model captures the qualitative biophysics correctly, the performance of the resulting optimized stimuli is in most cases quite robust with respect to reasonable quantitative errors in model parameters.

There are several directions for improvement and extension of our framework and the specific models it relies on, which we intend to address in future work. The model considered in Optimal Control of Photo-Stimulation for the interaction of the laser with cellular pumps and channel conductances may prove to be too simplistic in certain applications, as it ignores saturation effects. These can be accounted for by adding appropriate nonlinear corrections to Eqs. 23 and 24. The drawback, however, is that these add new sources of nonconvexity to the cost function and hence reduce the efficiency and accuracy of the optimization algorithm [for preliminary work on enhancing the modeling of photo-sensitive channels along these lines without severely compromising the efficiency of the optimization, see Grosenick et al. (2011)]. Also, in the case of uncaging of caged neurotransmitters (discussed in Optimal Control of Photo-Stimulation) a more accurate model should account for relevant spatial effects in the diffusion of uncaged compounds, ignored by the simple, nonspatial models described by Eqs. 21 and 23 and Eqs. 31 and 32. We also plan to improve the online implementation discussed in Online Implementation so as to minimize redundancies. We noted in results that due to refractory and short-term adaptation effects spikes that trail other spikes by ≥50 ms are the least robustly reproducible ones (this is also visible, in a different setting, in the comparison made in Fig. 5, albeit for shorter interspike intervals). Another direction for future research is to investigate whether improving the employed neuronal models such that they account better for short-term adaptation could make the control of such spikes more robust.

In addition to the numerical methods that we considered here, the stimulus optimization problem can also be studied by adopting an analytical, continuous time approach. In this approach, the optimal solution is found by solving the Euler-Lagrange equations corresponding to the minimized cost function. As we mentioned in Optimal Control of Electrical Stimulation, in Moehlis et al. (2006), the authors adopted such an approach to find analytical optimal solutions for input currents to a certain class of neural oscillator models [similar methods have been applied to other related problems in theoretical neuroscience; see, e.g., Badel et al. (2005, 2008); Paninski (2006); Cocco et al. (2009)]. Such approaches to stimulus optimization are quite fast by virtue of being analytic, but they only apply to rather specific problem settings. Our numerical approach, on the other hand, is much more general. However, there could be ways of combining the two methods to gain from the advantages of both. In our future work, we intend to use the mentioned analytic method to study the stimulus optimization in simple settings, such as the case of a target spike train containing a single, isolated spike. It may then be possible, in certain cases, to obtain a satisfactory, approximately optimal solution for a general target spike train, by a simple superposition of the analytically obtained solution for single spikes. Even if not very close to the optimal solution, this can provide a very good initialization for the numerical optimization algorithm, and hence improve its speed.

Many interesting applications of optimal spiking control could be carried out in vivo. This creates a host of complications. For example, in the case of in vivo extracellular current injections, the relative position of the electrodes to various controlled neurons, as well as the intrinsic biophysics of those cells, are all subject to time-dependent variations. This means that model parameters (electrode couplings to various cells, membrane time constants, etc.) change with time, and therefore a single estimation of these before the main application (as we did in our experimental test in results) will not be sufficient. Furthermore, in many situations we may not be able to observe the spiking output our stimuli are creating (or we might only have crude information, such as behavioral feedback in a sensory prosthetic setting), so we might have to use a crude encoding model. It will thus be crucial to explore ways in which we can make our methods more robust to model mis-specification in the future. As a first step towards this goal, we plan to test a simple approach that is based on a small modification of the framework of Eq. 3, used in this study. The spirit of this modification is similar to the method described in footnote 1, for generalizing beyond the sharp delta function cost function (see the discussion after Eq. 1) without losing computational tractability. Let us show the model parameters collectively by θ. It is implicit in Eq. 3 that the distribution p(r|x) is conditioned on a single set of model parameters, θ̂, estimated from the relevant observed data; more explicitly, the distribution used in Eq. 3 is p(r|x,θ̂). Suppose we represent our uncertainty about the model parameters by a posterior distribution conditional on the observed data, and we have n samples, θ_i (i = 1, …, n), from this distribution. Now, to take into account the model mis-specification, we modify the minimized objective in Eq. 3 replace the overly confident log logp(r|x,θ̂) in Eq. 3 by its average, 1/n ∑ⁿ_i=1 log p(r|x, θ_i), over the samples from the parameter posterior. The important point is that with this modification of Eq. 3, the minimized objective (as a function of the stimulating agent, x) is still convex, and therefore we can still use fast convex optimization methods for minimizing it.

Finally, we note that one could test our algorithms in other experimental settings including extracellular current injection and optical uncaging of caged glutamate and GABA compounds, to control the spiking activity of several neurons. The experimental test presented in this study is only a simple first step; spike control using the above methods is more complicated than the intracellular current injection presented in results and is of more interest from the point of possible applications.

GRANTS

Y. Ahmadian is supported by the Robert Leet and Clara Guthrie Patterson Trust Postdoctoral Fellowship, Bank of America, Trustee. The laboratory of R. Yuste is supported by National Eye Institute Grant EY-11787 and the Kavli Institute for Brain Studies. L. Paninski and R. Yuste share a Collaborative Research in Computational Neuroscience Award NSF-IIS-0904353. L. Paninski is in addition supported by a National Science Foundation CAREER award and a McKnight Scholar award.

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the author(s).

APPENDIX A: DERIVATION OF THE LIKELIHOOD EXPRESSIONS

In appendix a, we derive the formula Eq. 8 for the log-likelihood log p(V|I). First, we rewrite Eq. 7 in discrete time (with step size dt) as

$$\begin{array}{cc}{V}_{t+dt}=V_t-\frac{dt}{{\tau }_V}{V}_t+\frac{dt}{C}{I}_t+\sqrt{dt}\sigma {\epsilon }_t,& {\epsilon }_t\sim N\left(0,1\right)\end{array}$$ (38)

where N(0, 1) denotes the distribution e−1 / 2 ε_t²/

$$\sqrt{2\pi }$$

, i.e., the standard Gaussian distribution with mean 0 and variance 1. The Gaussian noise ε_t is scaled by dt to ensure a proper continuous time limit. The ε_t at different time steps is sampled independently from N(0, 1). Since the voltage is determined by the current, up to the Gaussian noise ε_t, which is integrated linearly, the joint conditional probability distribution p(V|I) takes the Gaussian form. In fact, the joint probability distribution of V for all times is, up to normalization, given by the joint distribution of ε_t at all times. Since the latter are independent at different times, their joint distribution is simply the product of their distributions at each time step. We thus have

$$p\left(V|I\right)\propto {\displaystyle \prod _t{e}^{-\frac{1}{2}{\epsilon }_t^2,}}$$ (39)

where ∝ means equality up to a constant multiplicative factor. Using Eq. 38 to solve for t in terms of V and I, we obtain

$$p\left(V|I\right)\propto {\displaystyle \prod _t{e}^{-\frac{1}{2{\sigma }^{2}dt}{\left(V_{t+dt}-V_t+\frac{dt}{{\tau }_{\text{V}}}{V}_t-\frac{dt}{C}{I}_t\right)}^2}}$$ (40)

Taking the logarithm of both sides we obtain Eq. 8.

APPENDIX B: O(T) OPTIMIZATION

Here we discuss in detail the fast optimization methods referred to in Optimal Control of Electrical Stimulation. As mentioned at the end of that section, we use the approximation (Eq. 13) in carrying out the optimization in (Eq. 3; with I replacing x). We thus need to minimize the convex cost function of Eq. 9 in the joint state vector (V, I). Let us schematically denote the optimized variables (V, I) by the vector x and denote the function minimized in Eq. 9 by F(x). At every step of the Newton-Raphson algorithm, when the optimized variable has the current value x, one uses a second order Taylor expansion of F(x) around x to approximate this function by a quadratic function. Then one moves the optimized variable to the point, x + y, that minimizes this quadratic function. Minimizing the latter yields [see, e.g., Press et al. (1992)] the following set of linear equations for the displacement, y:

$${H|}_{\text{X}}\text{y}=-\nabla {F|}_{\text{x}},$$ (41)

where H|_x and ▿F|_x are, respectively, the Hessian matrix of F(x) [the matrix of its second order partial derivatives with respect to the variables (V_t, I_t)] and its gradient at the current point, x. To summarize, at each step of the Newton-Raphson algorithm one moves the current value x, by an amount y which is found by solving Eq. 41 [in practice, to guarantee convergence, we move the state vector x by αy, 0 < α ≤ 1, where α is found by a simple, stable backtracking procedure (Press et al. 1992)]. This step is repeated until we converge to some local minimum. Since the objective is convex, this minimum is guaranteed to be the global minimum.

In general, solving a set of d linear equations such as Eq. 41 requires O(d³) elementary equations, and hence the computational time spent in every Newton-Raphson step should scale cubically with the duration, T, of the interval in which we intend to control the spike train- since in our case the dimension of x is proportional to the number of time steps. However, in our case the Hessian matrix of the first two log-likelihood terms in the objective Eq. 9 is block tridiagonal with blocks of size 2 × 2 (one dimension each for V_t and I_t).⁶ This is because due to locality in time (the fact that each summand of the temporal sum in Eqs. 6 and 8 only depends on state variables at the time step t or adjacent time steps), the partial second derivate of each log-likelihood term with respect to a variable (V or I) at time t₁ and a second variable at time t₂ is zero, unless t₂ = t₁, or t₂ = t₁ ± 1. When the matrix of the linear set of equations is block-tridiagonal (or, more generally, banded with a fixed band-width), the scaling of the computational time is improved to O(d). In our case, this means that the computational time scales as O(T), i.e., only linearly with the duration of the control interval [for a review of the application of fast optimization methods to problems in neuroscience, see Paninski et al. (2010)].

The only complication for using the Newton-Raphson method is that we have imposed a hard constraint on the current. This is because the Newton-Raphson method, Eq. 41, assumes a smooth objective function. To overcome this, we use the barrier or the interior-point method (Boyd and Vandenberghe 2004; Paninski et al. 2010) in which instead of directly enforcing the hard constraint, |I_t| < I_max, we add a soft logarithmic barrier function of the form η

$$\underset{t}{\Sigma }$$

[log(I_max − I_t)+ log(I_max + I_t)] to the objective, and optimize the latter in this augmented form, but without the explicit bound |I_t| < I_max enforced. By iterating this procedure for progressively sharper barrier functions (corresponding to smaller η), we approach the solution to the hard constraint problem with any desired accuracy. Since the Hessian of the barrier function turns out to be diagonal, the total Hessian of the objective with respect to (V, I) remains block-tridiagonal, and we can still apply fast inversion algorithms in each iteration and hence the computational cost still scales as O(T).

Another small complication arises because the Hessian of the cost function Eq. 10 with respect to I_t is not banded. However, from Eq. 11 we can write I = AJ, where A is a bidiagonal Toeplitz matrix with zero entries everywhere except on the main diagonal (where they are all equal to 1/τ_J − 1/dt) and the first diagonal above it (where they are all equal to 1/dt). Given that J and I are related by a deterministic linear transformation, we can change variables and carry out the minimization in Eq. 9 in terms of (V, J) instead of (V, J). Once the optimal (V, J) are found, we can multiply the optimized J by A to obtain the desired optimal solution for I. However, the Hessians of all three terms in the objective function in Eq. 9 with respect to (V, J) are banded, and thus the minimization in terms of the new variables can be done in O(T) fashion. To see the bandedness of the corresponding Hessians note that the Hessian of the cost function term Eq. 10 with respect to J is proportional to the identity matrix. The bandedness of the Hessians of the first two terms in Eq. 9 is also preserved after the variable change [note that they were block-tridiagonal with respect to (V, I)]. This is because, by the chain rule for derivatives, these Hessians are only modified by multiplications by A^T on the left side or by A on the right side or both (depending on the number and order of derivatives with respect to I in the original variables), and multiplication by banded matrices preserves bandedness.

The situation is quite similar in the optimization for optical stimulation, Eq. 25. Here, the natural choice for the optimized variables would have been (V, L^E, L^I), and not (V, g^E, I^I). But the Hessian of the second term in the cost function is not banded with respect to (L^E, L^I), whereas it is clearly block tridiagonal with respect to (g^E, I^I). Furthermore, if we define L̃^E = w_EIL^I + w_EEL^E, and L̄^I = w_IIL^I + w_IEL^E, then Eqs. 23 and 24 take the same form as Eq. 11 (after discetization). Therefore, we have L̃^E = A^Eq^E and L̃^I = A^II^I, where the bidiagonal Toeplitz matrices A^E and A^I have the same form as the matrix A defined in the last paragraph, with τ_E and τ_I replacing τ_J, respectively. Using the chain rule for differentiation, we see that by the same argument as in the last paragraph, the Hessian of the third term R(L^E, L^I) in the cost function Eq. 25 is also banded with respect to (g^E, I^I). Thus the optimization in Eq. 25 can be carried out using the O(T) method.

Finally, we found that in many cases where the Hessian matrix, H|_x, is very ill-conditioned, we could improve the convergence of the Newton-Raphson algorithm by regularizing the Hessian matrix, as follows. Instead of solving Eq. 41 at each step of the algorithm, we solved

$$\left({H|}_{\text{x}}+{\eta }^{2}I\right)\text{y}=-\nabla {F|}_{\text{x}},$$ (42)

for a small η, where I denotes the identity matrix. Mathematically, convergence according to this rule (i.e., reaching a point x where the displacement found by solving Eq. 42 once more is too small) still requires reaching a point where ▿F|_x vanishes (or is very small). But convexity and smoothness guarantee that a stationary point with zero gradient is indeed the global minimum. As usual, Eq. 42 can be solved efficiently, since if H is banded then H + η²I is also banded.