Spike-induced ordering: Stochastic neural spikes provide immediate adaptability to the sensorimotor system

Significance

The functional advantages of using a stochastically spiking neural network (sSNN) instead of a nonspiking neural network (NS-NN) have remained largely unknown. We developed an architecture which enabled the parametric adjustment of the spikiness (i.e., impulsive dynamics and stochasticity) of the sSNN output and observed that stochastic spikes instantaneously induced the ordered motion of a dynamical system. We demonstrated the benefits of sSNNs using a musculoskeletal bipedal walker and, moreover, showed that the decrease in the spikiness of motor neuron output leads to a reduction in adaptability. Stochastic spikes may aid the adaptation of a biological system to sudden perturbations or environmental changes. Our architecture can easily be connected to the conventional NS-NN and may superimpose the on-site adaptability.

Abstract

Most biological neurons exhibit stochastic and spiking action potentials. However, the benefits of stochastic spikes versus continuous signals other than noise tolerance and energy efficiency remain largely unknown. In this study, we provide an insight into the potential roles of stochastic spikes, which may be beneficial for producing on-site adaptability in biological sensorimotor agents. We developed a platform that enables parametric modulation of the stochastic and discontinuous output of a stochastically spiking neural network (sSNN) to the rate-coded smooth output. This platform was applied to a complex musculoskeletal–neural system of a bipedal walker, and we demonstrated how stochastic spikes may help improve on-site adaptability of a bipedal walker to slippery surfaces or perturbation of random external forces. We further applied our sSNN platform to more general and simple sensorimotor agents and demonstrated four basic functions provided by an sSNN: 1) synchronization to a natural frequency, 2) amplification of the resonant motion in a natural frequency, 3) basin enlargement of the behavioral goal state, and 4) rapid complexity reduction and regular motion pattern formation. We propose that the benefits of sSNNs are not limited to musculoskeletal dynamics. Indeed, a wide range of the stability and adaptability of biological systems may arise from stochastic spiking dynamics.

Stochastic spiking neurons fire transient action potentials to generate irregular spike trains. The stochastic spike train of a presynaptic neuron is propagated toward the synaptic terminal and induces irregular and discontinuous time series of the postsynaptic potential (PSP) in a connected neuron or a muscle cell (1). Conventionally, neural spiking is considered to underlie stable and efficient digitalized signal transmission (2). In addition, recent studies have demonstrated that the transient characteristics of stochastic spikes enable the exotic state transition of a neural network, allowing for spike annealing. Such studies have indicated that stochastically spiking neural networks (sSNNs) enable significantly more rapid and efficient optimization than do networks involving rate-based model neurons (3–5). Nevertheless, other merits of irregular/impulsive PSP time series remain largely unclear.

Theoretically, it is well known that noise is beneficial in a nonlinear excitable system that generates all-or-none responses (i.e., neurons or cells): Noise induces ordering of the nonlinear system and aids in the synchronization and formation of spatiotemporal patterns in the excitable media (6–10). The occurrence of noise-induced ordering (11, 12) is not limited to an excitable system and is observed in other systems such as granular particles perturbed by vibration (12). Thus, it is highly feasible that similar noise-induced ordering effects occur in a dynamical system exposed to stochastic spikes.

The computational results of the present study suggest that an sSNN may be the core prerequisite for realizing the instant adaptability of a biological sensorimotor system. In the following sections, we first briefly describe the sSNN model, following which we discuss the architecture that enables us to modulate the spikiness of PSPs generated by the sSNN, where we define the term spikiness as a root-mean square of the second derivative of the PSPs to represent the impulsive and irregular dynamics of the PSPs. We demonstrate how stochastic spikes help improve the adaptability of a musculoskeletal bipedal walker; we demonstrate that decreases in the spikiness of PSPs lead to a reduction in the adaptability. To further analyze how stochastic spikes help improve the adaptability of a bipedal walker, and to understand how sSNNs may help other more-general agents, we study several simple sensorimotor models driven by sSNNs and present four sSNN functions. Furthermore, we propose that spike-induced ordering is manifested as the spatiotemporally coordinated motion of a bipedal walker, such as the smooth and stable gait (re)initialization and robust balancing.

Results

Model of a Spike-Based Sensorimotor Agent.

Because the parameters of sSNNs and conventional nonspiking neural networks (NS-NNs) are substantially different (e.g., time constants, stochasticity, and the nonlinearity of the activation function), simple comparisons of benchmark scores of sSNNs and NS-NNs would not provide sufficient understanding of the functions of stochastic neural spikes; it would be challenging to distinguish the effect of stochastic spikes from that of other parameters. To investigate the functions of stochastic spikes, we considered a sensorimotor agent driven by sSNNs which convert the smooth signal generated by a base controller into the correspondent spike trains, as shown in Fig. 1A. Employing this architecture enables the modulation from stochastic and discontinuous sSNN behavior to conventional rate-coded NS-NN behavior by altering PSP timescale and sSNN size (altering these parameters is roughly correspondent to changing the parameters of a lowpass and an averaging filter, respectively) as described later (Fig. 1 C and D).

Fig. 1.

(A) Schematic model of a spike-based sensorimotor agent for seamless modulation from spiking signals to nonspiking, rate-coded signals by altering PSP duration and sSNN size. The neurons in an sSNN receive common (smooth) input and convert the smooth signal to a stochastic spike train. An agent is controlled based on the PSP generated by the sSNNs. (B) Spikiness with respect to $N$ and ${\tau }_s$. The spikiness decreases and approaches the smooth, rate-coding signal as $N$ and ${\tau }_s$ increase. (C1 and C2) The PSP signals are induced by the periodic input $I\left(t\right)$ of 0.1 Hz with $N=10$ and ${\tau }_s=5$ ms and $N=200$ and ${\tau }_s=30$ ms, respectively. (D) The histogram of $y$ in response to $I=5$. The average values are the same: 0.66606 with $N=5$ and 0.66600 with $N=100$.

The $i$th sSNN $S_i$ in Fig. 1 consists of $N$ stochastic leaky integrate-and-fire (LIF) neurons, and each neuron in $S_i$ receives a common input signal $I_i\left(t\right)$ from a base controller (such as a conventional feedback controller). Furthermore, the $j$th neuron (in the $i$th sSNN) receives independent bias input $b_{ij}$ and Gaussian noise $D_i{\xi }_{ij}\left(t\right)$, where $D_i$ indicates the intensity.

As a synapse connecting an sSNN neuron and an actuator, we assumed the simplest linear synapse model (6, 13–15). Then, the activation level $A_i\left(t\right)$ of the $i$th actuator, which is coupled with the $i$th sSNN, is determined by the total PSP $y_i\left(t\right)$ as

$${\tau }_s{\stackrel{\cdot }{y}}_i=-y_i+\frac{1}{N}\sum _{j=1}^N{\sigma }_{ij}\left(t\right),$$ [1a]

$$A_i\left(t\right)=g_i^A{y}_i\left(t\right)-A_i^0,$$ [1b]

where ${\sigma }_{ij}\left(t\right)$ is the spike train of the $j$th LIF neuron, ${\tau }_s$ is the PSP timescale, and $g_i^A$ is the amplification gain to convert excitatory PSP (EPSP) to the activity of an actuator. $A_i^0$ is the offset inhibition, and $A_i^0=0$, unless otherwise stated. Note that $A_i$ is basically positive (although $A_i$ may be offset by $A_i^0$); therefore, the actuator driven by an sSNN can generate only unidirectional output.

It is important to note that $N$ does not influence $⟨y\left(t\right)⟩$ (where the operator $⟨\cdot ⟩$ denotes the averaging over simulation trials) or $\overline{y}=\int y\left(t\prime \right)dt\prime$, but influences only the variance of $y$ if each spike train of different neurons is independent. [Note that some input may break the independency of each neuron activity as a strong common input inducing the neural synchronization (16, 17)]. Fig. 1D shows that $N$ does not change $\overline{y}$ and changes only the variance of $y$ (we used $I=5$, $b=4$, $D=10$, and ${\tau }_s=5$ ms). Moreover, the stochasticity and discontinuity of the PSP signal are clearly scaled by the PSP timescale ${\tau }_s$. Hereafter, we discuss how $N$ and ${\tau }_s$ scale the smoothness and stochasticity of $y$ and approximate the sSNN output to the rate-coded signal. For further discussions, we introduce the variable $Z$, representing the mean magnitude of spikiness during a certain period $t=\left[t_0,t_0+T\right]$ as follows:

$$Z={\left(\frac{1}{T}{\int }_{t\prime =t_0}^{t\prime =t_0+T}{\left(ÿ\left(t\prime \right)\right)}^{2}dt\prime \right)}^{1/2}.$$ [2]

Of note, $Z$ can represent the stochasticity and discontinuity and, therefore, can be a good index for distinguishing an sSNN and a conventional rate-coded neuron network. Using Gaussian approximation of the spike trains of a neuron ensemble (18, 19) and linear-response techniques (6, 20, 21), we can show that $Z\propto 1/\left(\sqrt{N}{\tau }_s^2\right)$. In fact, $Z$ can be modulated by both the sSNN neuron size, $\sqrt{N}$, and the PSP timescale, ${\tau }_s$, as shown in Fig. 1C. See Materials and Methods for details.

Hereafter, we use $N$ and ${\tau }_s$ to parametrically morph the sSNN output to a conventional rate-coded neuron model. Besides the modulation of spikiness, the PSP timescale ${\tau }_s$ plays a special role, described in detail in later sections of the text. Furthermore, it is important to note that modulating ${\tau }_s$ does not influence the total amount of actuator activation $⟨{\int }_0^{\infty }{y}_{i}dt⟩$ (SI Appendix, section S1).

An sSNN-Driven Musculoskeletal Bipedal Walker.

We demonstrate how stochastic spikes generated by sSNNs are integrated into the adaptability of a bipedal musculoskeletal system. The configurations for muscle–tendon units (MTUs), a bipedal model, and a reflexive control system are based on ref. 22, as shown in Fig. 2A. The smooth nonspiking input to the $i$th sSNN, $I_i\left(t\right)$, is computed using the reflex rules proposed in refs. 22 and 23 and then passed to the corresponding $i$th sSNN. The MTU activity $A_i$ is innervated by the EPSP induced by the corresponding neuron ensemble (Eq. 1), and the generative force of an MTU is computed based on the dynamics proposed in ref. 23.

Fig. 2.

An sSNN-driven musculoskeletal system of a bipedal walker. (A) The bipedal walker consists of 16 muscle–tendon units (MTUs) in total, and the activity of an MTU is determined by the corresponding sSNN, which converts the smooth control signal of a reflex system to an impulsive PSP. We considered the hip flexor muscle group (HFL), the gluteus muscle group (GLU), a biarticular hamstring muscle group (HAM), the femoris muscle group (RF), the vasti muscle group (VAS), the soleus muscle (SOL), the gastrocnemius muscles (GAS), and the tibialis anterior muscle (TA). (B1) The sSNN-controlled walker can realize the instant walking rhythm reformation in an extremely slippery environment $\mu =0.04$. Results are shown for $v^g$ in the environment where the friction coefficient is shifted from $\mu =10$ to $\mu =0.04$ during $t=\left[\mathrm{10,40}\right]$ (blue line) and in the default environment where the friction coefficient is constant and $\mu =10$ (red line). The frequency of walking rhythm changes in response to slipping, as shown in B2. The numerical simulation was obtained using an sSNN with $N=125$ and ${\tau }_s=5$ ms. (B2) Enlarged view of $v^g$ during slip and nonslip walking. The sSNN-driven agent can slow down and reform the walking rhythm in response to slipping, as shown in B3. (C) Successful gait-initiation ratio $p_g$ of reflex, sigmoid, and sSNN controllers (with $\tau =5$ ms and several different $N$ values) in a slippery environment for $x=\left[1.25,5\right]$.

A conventional sigmoid-type controller and a simple reflex controller are prepared to compare the experimental results. We use the real-coded genetic algorithms (GAs) using a blend crossover $\alpha$ (BLX-$\alpha$) method (24) to optimize the controllers.

Dynamic Walking Rhythm Formation during Slipping.

Bipedal locomotion in a slippery environment requires an agent to decrease the stride frequency. In fact, it is reported that the stride rate in submaximal skating is less than 1 Hz (25) whereas the stride rate during normal walking is usually greater than 1 Hz (26). Because the reflex-based controller involves passive-walking dynamics during the swing phase (23), the original reflex-based controller has the potential ability to adaptively alter the locomotion rhythm in response to environmental changes, such as changes in the slope angle. The sSNN controller inherits this potential ability to ensure reformation of the walking rhythm.

Fig. 2 B1 and B2 demonstrates that the sSNN can change the walking rhythm in response to the sudden changes in the friction coefficient to $\mu =0.04$ during the period $t=\left[\mathrm{10,40}\right]$ s and $\mu =10$ otherwise ($\mu =0.04$ is smaller than that of a banana skin on the floor and mostly similar to a ski on wet snow) (27). Movie S1 demonstrates bipedal walking using an sSNN controller that successfully slows down the rhythm but fails to find a new stable gait pattern. Successful identification of a new stable gait pattern and instantaneous transition to the stable walking pattern are demonstrated in Movie S2. Note that Movies S1 and S2 are obtained using the same parameters. The difference between Movies S1 and S2 can be accounted for by whether the sSNN instantaneously found and transitioned to the new stable gait.

The sSNN ability for the instant rhythm formation becomes most pronounced in an experiment where a walker is required to initiate a stable gait in a slippery environment; a walker experiences slipping within the first few steps and is required to realize the drastic state transition from a static standing state to a dynamic and stable locomotion gait while preventing falling by slip. Movies S3 and S4 demonstrate that the sSNN controller can slow down the timing of the leg swing in response to the slip of the stance leg and prevent falling, while the reflex-based controller fails to change the walking rhythm, resulting in falling. We used $\mu =0.04$ for $x=\left[1.25,5\right]$. The successful gait-initiation rate $p_g$ in this environment is shown in Fig. 2C. Note that $p_g=C_{suc.}/C_{tot.}$, where $C_{suc.}$ is the number of trials the walker successfully initiated the gait and kept walking during 20 s, and $C_{tot.}=500$ is the number of total trials. The reflex and sigmoid controllers exhibited a threshold-like behavior in response to $\mu$; the reflex and sigmoid controllers required $\mu \ge 0.05$ and $\mu \ge 0.09$, respectively, to realize gait. The sSNN controller largely relaxed this threshold-like constraint and realized a locomotion with a smaller friction coefficient $\mu <0.04$. It is also shown that there was an optimal neuron ensemble size $N$ to maximize the adaptability to slipping; $N=125$ has greater adaptability than $N=50$ or $N=500$. To obtain Fig. 2C, we first collected the sigmoid and reflex controllers satisfying the optimization cost $J<0.2$ and then selected the controllers that had the highest successful gait-initiation rate.

Emergent Well-Coordinated Motions.

The sSNN-driven walker sometimes exhibits athletic and well-coordinated motion, depending on the initial simulated states. Movie S5 and SI Appendix, Fig. S1A demonstrate that the sSNN-driven walker tolerates the extremely long-term and long-distance slipping of approximately 0.5 s and 0.5 m slipping. Furthermore, the sSNN-driven walker can recover from the near-falling state if the walker reaches the end of the slippery surface and can have a contact with the nonslippery ground, as shown in Movie S6 and SI Appendix, Fig. S1B. Skating-like motion, that is obtained using the different parameters of the reflex system and sSNN from other movies, on slippery ground with $\mu =0.1$, is demonstrated in Movie S7.

Adaptability to External Random Force Perturbation.

To further investigate the adaptability of the system, we examined the falling-prevention ratio $p_F$ when a random perturbing external force was applied: Every 0.25 s, the agent upper body receives an external force $F_{ext}=F_0+F_D{\xi }_F$ with a probability of $1/4$, where $F_D$ is the noise intensity and ${\xi }_F$ is a value sampled every 0.25 s from a Gaussian distribution (i.e., with a probability of $3/4$, $F_{ext}=0$; note that $F_{ext}$ is constant for 0.25 s and is updated to a new value every 0.25 s). We compute $p_F$ using 100 trials of 30-s numerical simulations.

Adaptability Provided by Spikiness.

The results shown in Fig. 3A demonstrate that the short PSP duration ${\tau }_s=2$ ms with a limited neuron ensemble size $N<100$ is associated with very high adaptability. It is worth noting that the graph for fall-prevention ability with respect to $N$ and ${\tau }_s$ is highly similar to the graph of spikiness shown in Fig. 1C. This result clearly implicates that spikiness plays an important role in the ability for instant adaptation.

Fig. 3.

The falling-prevention ratio $p_F$ in response to an external perturbing force. We found that adaptability manifested in two different ways: Adaptability was significantly improved by spikiness, as shown in A. In addition, adaptability was improved by an optimal PSP timescale range, i.e., ${\tau }_s:\left[\mathrm{15,30}\right]$, although these effects were only weakly related to the neuron ensemble size $N$ (B). (A) Adaptability in response to external perturbation is maximized with $N<100$ and ${\tau }_s<5$ ms, exhibiting strong similarity to the graph of spikiness shown in Fig. 1B. We used $F_0=50$ N and $F_D=100$ N. (B) We found another parameter set for which adaptability occurred critically dependent on ${\tau }_s$ and almost insensitive to $N$. We used $F_0=30$ N and $F_D=75$ N. (C) $p_F$ of sSNN, sigmoidal, and reflex controllers in response to the stationary force input $F_0$ to a hip. The sSNN controller used in A outperformed the other controllers.

Furthermore, an sSNN controller exploiting spikiness outperformed a reflexive controller and a sigmoidal controller in the falling-prevention task, as shown in Fig. 3C; the admissible force load of an sSNN controller (with $N=30$ and ${\tau }_s=2$ ms) was fivefold and twofold larger than that of a sigmoidal and a reflex controller, respectively (i.e., the maximum force load for an sSNN, a sigmoid, and a reflex controller was 110, 18, and 50 N, respectively). The procedure to select the sigmoidal/reflex controller used in Fig. 3C is the same as that to obtain Fig. 2C. Note that the reflex controller has a higher falling-prevention ratio than the sigmoidal controller. This might be because the sigmoid controller could not generate the large MTU activation necessary to prevent falling due to the saturation dynamics.

Adaptability Provided by Optimal PSP Timescale.

The fall-prevention probability shown in Fig. 3B is only weakly dependent on the neuron ensemble size $N$, while the PSP timescale ${\tau }_s$ is of critical importance for the ability to stabilize. In Fig. 3B, ${\tau }_s:\left[\mathrm{15,30}\right]$ ms maximizes stabilization during walking, and the rate of fall prevention is almost insensitive to the size of $N$. In fact, stability is not reduced or enhanced in the rate-coding regime, such as when $N=\mathrm{1,000}$. This result implies that an sSNN can produce adaptability within the musculoskeletal system of a bipedal walker based on principles other than spikiness. Furthermore, Fig. 3B implies the existence of an optimal PSP timescale ${\tau }_s$ to maximize adaptability.

Analyses Using Simple Sensorimotor Agents.

Balancing of a standing biped system is often modeled using an inverted pendulum on a ground (28, 29) supported by passive torque generated by the ankle joint stiffness. Such a system has a bistable potential function as $V\left(x\right)=\cos \left(x\right)+k{x}^2$, where $x$ is the scaled pendulum angle and $k$ is a spring coefficient (30, 31), and increasing the muscle coactivation at the ankle or hip joint increases the joint stiffness $k$ and improves the posture stability (32, 33). Gait (re)initiation from the initial posture or the near-falling state requires a transition from the such-stabilized (and rather rigidized) posture to a dynamical and periodic alternation of joint posture such as forward–backward swinging of the upper body (34, 35). Furthermore, to improve energy efficiency and locomotion stability, the periodic motions of an upper body or a swinging leg must match the natural frequency of a passive mechanical system (23, 36, 37). Bipedal walking and recovery from the near-falling state require solving these issues. Hereafter, we discuss how sSNNs help an agent achieve these subtasks by using several simplified abstract models.

Synchronization to the Natural Frequency.

We start our analysis for how sSNNs may help a tight synchronization to the natural frequency (SNF) of a target system. We consider generating the oscillatory motion of a particle connected to a spring, for simplicity, because the mass–spring system has an explicit form of a natural frequency. The motion equation of the spring–mass system is described as $\stackrel{‥}{x}=-\gamma \stackrel{\cdot }{x}-kx+\left(A_0-A_1\right)+K_p\left(x_g-x\right)$, where $k$ is the spring coefficient, $\gamma$ is the damping coefficient, and $A_{0/1}$ is the force generated by sSNNs. Note that if $K_p$ is as small as $K_p<1/4{\gamma }^2$, the system has convergent solution and $x\to x^g$ (with the assumption $A_i=0$). Likewise, if $K_p=1/4{\gamma }^2$, the system exhibits stable periodic behaviors, and for $K_p>1/4{\gamma }^2$, the system exhibits periodic and divergent behaviors.

The input and output of sSNNs are designed so that sSNNs generate a periodic oscillatory motion of a spring–mass system; sSNNs form a positional feedback controller where the gains $g_s$ and $g^A$ are adjusted sufficiently large to generate the oscillatory motion of a particle. The input to sSNNs is described as $I_0\left(t\right)=g_s\left(x-x^g\right)$ and $I_1\left(t\right)=g_s\left(x^g-x\right)$. Furthermore, we used the bias input $b=0.5$ (SI Appendix, section S3).

The natural frequency of the spring–mass system is $f_0^{\gamma }=\sqrt{k+K_p-1/2{\gamma }^2}/\left(2\pi \right)$ Hz, and the oscillation amplitude of $x\left(t\right)$ diverges if the frequency of any external forces matches to $f_0^{\gamma }$. It is clear that introducing the positional feedback gain $K_p$ is equivalent to modulating the spring coefficient $k$ as $k\to \left(k+K_p\right)$. Therefore, theoretically, all smooth rate-based feedback controllers inevitably distort the original intrinsic dynamics of a subject system. This is clearly shown in Fig. 4A, where we define $R_f$ as the ratio of $f$, the actual frequency of the oscillatory motion of $x\left(t\right)$, and the natural frequency $f_0^{\gamma }$ as $R_f=f/f_0^{\gamma }$. Evidently, perfect synchrony to the intrinsic dynamics gives $R_f=1$. Fig. 4B demonstrates that the sSNN can achieve $⟨R_f⟩\sim 1$ with a wide range of $f_0^{\gamma }$ and $g^A$. Note that, however, for very low natural frequencies such as $f_0^{\gamma }\le 0.1$, the neuron time constant $\tau$ or PSP time constant ${\tau }_s$ must be adjusted to realize $⟨R_f⟩\sim 1$. See also SI Appendix, Fig. S2 for further analyses for the $⟨R_f⟩$ dependency on the parameters such as $\tau$ and ${\tau }_s$.

Fig. 4.

The comparison $R_f=f/f_0^{\gamma }$ using the conventional feedback controllers versus sSNNs. (A) $⟨R_f⟩=f/f_0^{\gamma }$ obtained using a conventional feedback controller with $x_g=0$. Note that increasing controller gain $K_p$ distorts the motion frequency apart from the natural frequency $f_0^{\gamma }$. (B) $⟨R_f⟩$ obtained using an sSNN controller. $⟨R_f⟩$ is only weakly sensitive to the changes in the controller gain $g^A$, unlike the conventional feedback controller. The $R_f$ was computed using the data obtained during $t=200\text{to}500$ s, and $⟨R_f⟩$ was obtained by averaging the 1,000 simulation trials with $\gamma =1$. We used $\tau =5$ s, ${\tau }_s=10$ s, $g_s=50$, and $D=\mathrm{3,000}$ for $f_0^{\gamma }=0.1$ Hz; $\tau =500$ ms, ${\tau }_s=500$ ms, $g_s=5$, and $D=\mathrm{1,500}$ for $f_0^{\gamma }=0.5$ Hz; and $\tau =10$ ms, ${\tau }_s=100$ ms, $g_s=1$, and $D=15$ for $f_0^{\gamma }=5$ Hz. Note that $⟨R_f⟩$ for $f_0^{\gamma }=0.1$ Hz corresponds to the upper x axis.

As a subsequent measure to observe the sSNN synchrony to the intrinsic dynamics, we used the signal-to-noise ratio (SNR) around the natural frequency $f_0^{\gamma }$ (with $K_p=0$). The SNR was computed as $\mathrm{S}\mathrm{N}\mathrm{R}=10{\log }_{10}\left(A_s^2/A_n^2\right)$, where the signal power, $A_s$, is the spectrum amplitude around the $f_0^{\gamma }$, and the noise power, $A_n$, is computed by subtracting $A_s$ from the total power spectrum. (The SNR was computed using the data obtained during $t=200\text{to}500$ s.) The resonant motion of a mass–spring system is generally evaluated from the maximum amplitude of the induced oscillatory motion. However, the maximum amplitude of the motion cannot discriminate resonances in the natural frequency and in the frequency distorted by a smooth controller. Therefore, we used SNR such that only the resonant motions in the natural frequency will result in high SNR.

PSP Resonance: Resonance Amplitude in Natural Frequency Is Modulated by PSP Time Constant.

Fig. 5 shows the graphs of SNR for several natural frequencies $f_0^{\gamma }$ and neuron time constants $\tau$ with respect to the PSP timescale ${\tau }_s$. These graphs indicate that sSNNs can induce resonant motion (i.e., SNR $>0$) with a significantly wide sSNN parameter range. Furthermore, Fig. 5 A and B indicates an interesting characteristic of sSNNs: SNR drastically changes by a small difference of ${\tau }_s$. It is shown that $W_s$, i.e., the width of ${\tau }_s$ for the SNR peak (see Fig. 5B for the schematic explanation of what $W_s$ indicates), increases corresponding to the range of ${\tau }_s$. For example, the $W_s$ (for $f_0^{\gamma }$ = 3) was $<0.2$ s for the peak around ${\tau }_s\approx 1.2$ s, and for the peak around ${\tau }_s\approx 6$ s, $W_s$ increases to $\sim 2$ s. Furthermore, Fig. 5B shows that the resonance peak positions are different for the different values of $f_0^{\gamma }$; e.g., the peaks for $f_0^{\gamma }=3$ are given by ${\tau }_s=\left\{\mathrm{22,31},\dots ,\mathrm{400,600,1,200,6,000}\right\}$ ms, and the peaks for $f_0^{\gamma }=5$ are given by ${\tau }_s=\left\{8.5,9.5,11,\dots ,\mathrm{520,2,100}\right\}$ ms (SI Appendix, Fig. S3).

Fig. 5.

(A and B) SNR with respect to ${\tau }_s$, $f_0^{\gamma }$, and $\tau$. A and B show that a stochastic spiking controller can generate resonance in the natural frequency of a system using a wide variety of parameter regions (i.e., SNR $>0$ for a wide range of ${\tau }_s$). Additionally, it should be noted that the value of SNR is further amplified by optimal values of ${\tau }_s$. Here, we used $\gamma =1$, $g_s=1$, $b=0.5$, $D=15$, $g^A=5\times 10^5$, and $N=10$. These results are only weakly sensitive to the changes in $N$ (SI Appendix, Figs. S2 and S3).

To summarize, sSNNs do not distort the intrinsic dynamics of a system unlike the conventional smooth rate-based controllers, and therefore an sSNN-based motion controller would be more suitable for exploiting profoundly more mechanical intrinsic dynamics. For example, the locomotion rhythm change in slipping shown in Fig. 2 would be accounted for by SNF; in slipping, the potential of a mechanical bipedal system is reduced from $V\left(x\right)=\cos \left(x\right)+k{x}^2$ to $\cos \left(x\right)$ because the ankle support drops so that $k\sim 0$, and therefore the natural frequency of the system is modulated by slipping. Furthermore, we observed a locomotion gait where the agent swings the legs and the upper body largely as shown in Movie S8, which is very similar to passive dynamic walking (38). This gait is used to obtain the result in Fig. 3B, and therefore this gait stability is mostly dependent on the PSP timescale ${\tau }_s$ and only weakly sensitive to the sSNN size $N$. This gait pattern might be realized using the SNF and PSP resonance.

Enlarging the Basin of a Goal State.

Here, we analyze how sSNNs help bipedal balancing in a near-falling state. Because the bipedal system is analogously represented by a spring-supported inverted pendulum (30, 31) which has a double-well potential, we consider the particle agent in a double-well potential as shown in Fig. 6A. The motion equation is described as $\stackrel{‥}{x}=-\gamma \stackrel{\cdot }{x}-{\partial }_{x}V\left(x\right)+\left(A_0-A_1\right)+H\left(x,t\right)$; $\gamma$ is a friction coefficient, and ${\partial }_x$ denotes differentiation with respect to $x$. Note that we consider a more mathematically generalized double-well potential $V\left(x\right)=1/4x^4-1/2x^2$.

Fig. 6.

Stabilization in the goal state by the sSNN controller in a double-well potential. (A) Potential landscape and controller setup used in the experiment. See Enlarging the Basin of a Goal State for details. (B) Time-averaged PSP in response to $x$. We fixed the particle position and observed the stationary value of $⟨\overline{y}⟩$. (C1) Superimposed particle trajectories obtained with the initial position $x_0=-1$ with several $v_0$ values such that $-0.5\le v_0\le 1$. We used $K_p=0.75$. The other parameters are listed within the panels. Note that linear feedback controllers cannot stabilize the particle around $x_g=0$ even if noise intensity is $D_f>0$. (C2) Typical particle trajectory obtained with $\gamma =0.5$ (Top) and with $\gamma =0.1$ (Bottom). In the low-friction environment $\gamma =0.1$, the particle often overshoots the goal position; the starting agent $x_0=-1$ overshot the goal and reached $x=1$, and then it rebounded to reach the goal state. For $\gamma =0.5$, we used $v_0=0.35$, $b=3$, $D=10$, $g_s=10$, and $g^A=0.3$. Likewise, for $\gamma =0.1$, we used $v_0=0.4$, $b=4$, $D=15$, $g_s=15$, and $g^A=0.21$.

The sSNNs are configured to compose a feedback controller to reach the potential top $x_g=0$. The inputs to $S_0$ and $S_1$ are $I_0\left(t\right)=g_s\left(x^g-x\right)$ and $I_1\left(t\right)=g_s\left(x-x^g\right)$, respectively. The controller gains $g_s$ and $g^A$ were designed small so that the agent which started from the potential bottom $x=\pm 1$ could not reach $x_g$. The output of an sSNN controller is approximately sigmoidal with respect to the agent position $x$, as shown in Fig. 6B. We prepare a stochastic feedback controller for comparison; $H\left(x,t\right)=-K_{p}x+D_f{\xi }_f\left(t\right)$, where $K_p$ is the control gain and $D_f{\xi }_f\left(t\right)$ is the Gaussian random perturbation with the intensity $D_f$. Note that $H\left(x,t\right)=0$ when we use sSNN controllers. For the detailed description of the controller setup, see SI Appendix, section S4).

Fig. 6 C1 demonstrates that $H\left(x,t\right)$ cannot stabilize the agent around $x=0$ even with the external help that is given as $v_0$ or with the presence of noise $D_f{\xi }_f$, if $K_p$ is not large enough to lift the agent from $x_{\pm }=1$ up to $x_g=0$. In contrast, Fig. 6 C2 demonstrates that the sSNN controller induces a metastable state around the goal state and exploits the external help (i.e., $v_0$) and force noise generated by the stochastic spikes to overcome the potential barrier to reach the goal state. This characteristic helps the agent start from the wider range of the initial state $\left(x_0,v_0\right)$ to reach and stay in the goal state. Therefore, we investigate the basin size, which reflects the size of the set of the initial states $\left(x_0,v_0\right)$ that enables the agent to reach $x_g=0$ and stay within $\left[-0.1,0.1\right]$ for a sufficiently long duration, i.e., 30 s.

The basin size is computed as follows: For the $m$th pair of initial position and velocity $X_m={\left(x_0,v_0\right)}_m$, we set $B\left(X_m\right)=1$ if the agent succeeds in staying on the ridge $\left[-0.1,0.1\right]$. Otherwise, $B\left(X_m\right)=0$. We define the basin rate $S_b$ as $S_b=\frac{1}{M}{\sum }_{m=1}^{M}B\left(X_m\right)$, where $M$ is the total number of $X_m$. Fig. 7 A1 and A2 represents the basin $B\left(X_m\right)$ obtained with $\gamma =0.5$, $\gamma =0.1$, respectively. (Note that we set the pixel to 1 if the corresponding initial state ${\left(x_0,v_0\right)}_m$ is a part of the basin.) Fig. 7 B1 and B2 indicates the basin area rate $S_b$ computed using several bias input values for $b$, which determines the mean-firing rate of an sSNN. We observed the tendency of the basin size to decrease with respect to $N$ for most of the $b$ and $\gamma$. However, if the friction is low, $\gamma =0.1$, and if the bias is as small as $b\le 0$ and the sSNN is in the subthreshold regime (i.e., the sSNN does not generate spikes near $x=0$), the small $N$ reduces the basin size.

Fig. 7.

Basin area rate decreases with respect to $N$. (A1 and A2) The basin plot for the initial state $X_i={\left(x_0,v_0\right)}_i$ and with $\gamma =0.5$ (A1) and 0.1 (A2). Note that the basin size decreases as $N$ increases. The several stripes in A2 reflect the overshooting behavior of the particle (in the low-friction environment, the agent cannot stop at the goal state if $v_0$ is very large; see Fig. 6 C2). (B1 and B2) The basin area rate exhibits monotonic decreases with respect to $N$ for $\gamma =0.5$. If the sSNN bias is small and in the subthreshold regime (e.g., $b\le 0$), the basin size decreases with small $N$, particularly in the low-friction environment $\gamma =0.1$ because the spike frequency of sSNN is low and therefore the agent is rather uncontrolled near $x=0$. See text for further analysis. The parameters used in the simulations are listed in the corresponding panels. For the other sSNN parameters, we used ${\tau }_s=1$ ms, $D=10$, $g_s=10$, and $g^A=0.3$ for $\gamma =0.5$ and ${\tau }_s=1$ ms, $D=15$, $g_s=15$, and $g^A=0.21$ for $\gamma =0.1$.

The basin-enlargement phenomenon implies that sSNNs can induce an inhomogeneous noise effect. We speculate that this inhomogeneous noise effect is based on the interaction of the nonlinear firing function $r\left(z\right)$ and noise of the sSNNs: In the sSNN suprathreshold regime $b>0$ (i.e., an sSNN emits spikes without any input $I$), $⟨\overline{y}⟩$ can be approximated as $⟨\overline{y}\left(x\right)⟩\approx 6x+0.3$ for $x\sim 0$, and $⟨\overline{y}\left(x\right)⟩\approx 1.2x+1.7$ for $x\gg 0$, as shown in Fig. 6B. This implies that the diffusive motion of an agent induced by noise is less suppressed if the agent is far from the goal state; $x\gg 0$ but more suppressed near the goal state $x_g=0$. In the subthreshold regime such as $b\le 0$, on the other hand, $⟨\overline{y}\left(x\right)⟩\approx 0$ for $x\sim 0$. Therefore, the diffusive motion of an agent is rather uncontrolled near the goal state in the subthreshold regime $b\le 0$. This would lead to the result where the small $N$ degrades the basin rate with $b=0$ as shown in Fig. 7 B2.

Rapid Complexity Reduction and Rhythm Formation.

In gait (re)initialization, a bipedal system must transit from a rigidized posture (that is induced by the muscle coactivations to improve the balance stability) to a dynamical, periodic motion regime. This transition is analogous to escaping from a quasi-stable state and forming a dynamic motion pattern in a multistable potential (note that in such dynamic motion, the quasi-stable state will be destabilized because of the large motion velocity). Here, we considered a particle system in a triple-well potential $V_3\left(x\right)=x^6-6x^4+8x^2$, as shown in Fig. 8A.

Fig. 8.

Rapid complexity reduction due to stochastic spikes. (A) Potential landscape and controller setups in the experiment. (B) The position trajectories obtained using a stochastic nonspiking controller $H\left(x,t\right)$. The noise intensity $D_F=50$ (Bottom) generated a more regular oscillatory motion and resulted in lower $\mathrm{A}\mathrm{p}\mathrm{E}\mathrm{n}=0.01$, compared with a lower-noise controller with $D_F=0.005$ (enlarged view in Top and Bottom in the same scale with $D_F=50$) ($\mathrm{A}\mathrm{p}\mathrm{E}\mathrm{n}=0.354$). We used $K=100$. (C) The position trajectories obtained by using the sSNN controller with $N=1$ (Bottom) and $N=500$ (enlarged graph in Top and the same scale with $N=1$ in Bottom). We used ${\tau }_s=5$ ms, $g^A=100$, $b=0.3$, $D=15$, and $g_s=4$. (D) Averaged time $⟨T_r⟩$ with respect to $N$ and actuation gain $g^A$. Note that $⟨T_r⟩=\mathrm{1,000}$ indicates that the agent could not escape the local minimum. (E) ApEn with respect to $N$ and several actuation gains. ApEn increases with $N$ for a wide variety of $g^A$ values. The ApEn values were calculated using the data obtained during $t=150$ to 1,000 s. D and E were computed by averaging the results of 50 trials of numerical simulations performed using the equations $x_0^g=-1$, $x_1^g=1$, ${\tau }_s=5$ ms, $b=0.3$, $D=15$.

The motion equation of an agent is given by $\stackrel{‥}{x}=-\gamma \stackrel{\cdot }{x}-{\partial }_x{V}_3\left(x\right)+\left(F_0+F_1\right)+H\left(x,t\right)$, where $\gamma =0.5$ is a friction coefficient, and ${\partial }_x$ denotes differentiation with respect to $x$. The sSNN controllers $S_0$ and $S_1$ were designed to mimic the function of antagonist muscles; $S_0$ and $S_1$ generate forces driving toward two different goal states $x_0^g=-1$ and $x_1^g=1$, respectively. In particular, we assumed that $S_0$ received $I_0\left(t\right)=g_s{\left[x-x_0^g\right]}_{+}$ and generated $F_0\left(t\right)=-A_0\left(t\right)$ toward $x_0^g$. Likewise, $S_1$ received $I_1\left(t\right)=g_s{\left[x_1^g-x\right]}_{+}$ and generated $F_1\left(t\right)=A_1\left(t\right)$ toward $x_1^g$. The stochastic feedback controller to mimic the function of antagonist muscles was introduced for the sake of comparison, as $H\left(x,t\right)=\left(B_1+B_0\right)+D_F{\xi }_F\left(t\right)$, where $B_i=K_p\left(x_i^g-x\right)$ (see SI Appendix, section S5 for the detailed controller design). $H\left(x,t\right)$ simply increases the stability of the quasi-stable state if $x_0^g=-x_1^g$ as we can see that $H\left(x,t\right)$ is reduced to a simple spring-like controller as $H\left(x,t\right)=-2K_{p}x+D_F{\xi }_F\left(t\right)$. Note that this would be corresponding to the increased joint rigidity by muscle coactivation (32, 33). Basically, $H\left(x,t\right)$ prevents the smooth transition from the stabilized posture regime to a periodic motion regime without the help of noise, as shown in Fig. 8B. In contrast, sSNNs with small $N$ have enhanced escaping from the quasi-stable well $x=0$ and generate a periodic motion among $x_{\pm }$ as shown in Fig. 8C. Fig. 8D indicates that the time required to obtain an oscillatory motion pattern, $T_r$ tends to increase with $N$, where $T_r$ is obtained by observing the time when the particle finishes passing both of $x_{\pm }^l$. For relatively small actuator gains such as $g^A=100$, escaping from $x=0$ is realized only with a small $N$ such as $N<50$.

Furthermore, escaping from the quasi-stable well where two controllers generate conflicting forces leads to the complexity reduction. In fact, Fig. 8 B and C demonstrates that the approximate entropy (ApEn) (39), a measure to quantify complexity, is drastically reduced by escaping from the quasi-stable well. This is because the agent outside the quasi-stable well is influenced more by the deterministic large force generated by the potential $V_3\left(x\right)$. (Intuitively, ApEn quantifies the infrequency of the same data pattern in a time series, and therefore regular and periodic time series give small ApEn. See SI Appendix, section S6 for the detailed definition of ApEn.) Fig. 8E shows a result that smaller neuron sizes $N$ (particularly $N<100$) can more effectively reduce ApEn than larger neuron size such as $N\ge 500$.

The ability of stochastic spikes to form patterns is similar to the well-established coherence resonance induced by noise (see ref. 6 for review) and noise-induced posture transition (40), and, therefore, similar to sSNNs, a linear controller can achieve a periodic motion if there is the help of noise $D_F\xi \left(t\right)$, as shown in Fig. 8E.

Discussion

Adaptability Improvement by Spikiness.

It is well known that walking requires the zero-moment point (ZMP) (41) to be within a certain range, which (in our framework) mainly corresponds to upper-body control toward and within a certain range. Note that this upper-body control is analogous with the problem formulation introduced in Enlarging the Basin of a Goal State. Furthermore, finding and transitioning to a new stable walking pattern upon perturbation must be analogous to rapid pattern formation discussed in Rapid Complexity Reduction and Rhythm Formation. Because these functions are enhanced by large spikiness (Figs. 6 and 8), it would be reasonable to say that the fall-prevention ability in response to the slipping, that is maximized by large spikiness, would be largely dependent on the basin-enlargement and complexity-reduction ability.

Adaptability Improvement by SNF and PSP Resonance.

Several studies of animal locomotion have proposed that locomotion frequency is generated near the natural frequency of the musculoskeletal system (23, 36, 37). The parameter band ${\tau }_s:\left[\mathrm{15,30}\right]$ that realizes high falling-prevention probability would correspond with the PSP resonance band shown in Fig. 5. One interpretation for the parameter band ${\tau }_s:\left[\mathrm{15,30}\right]$ would be that sSNNs exploited the musculoskeletal natural frequency by SNF and stabilized the rhythmic motion by PSP resonance.

Why Most Biological Systems Rely on Spiking Neurons.

Even the unicellular Paramecium use the action potential to generate behavior (42). Although biological organisms contain a few types of nonspiking interneurons (43–46), sensory information and motor activation are mainly regulated by spiking action potentials in biological systems. Recent developments of neuromorphic chips have revealed that learning systems using spiking neurons can solve optimization problems with remarkably higher-power efficiency (47–49). Nevertheless, the merits of biological spiking neurons remain to be fully elucidated. In fact, state-of-the-art, biologically inspired engineering frameworks such as deep learning (50), deep reinforcement learning (51), and their derivatives still employ nonspiking rate-based neurons because spiking neurons have not yet been associated with any remarkable benefits in learning and optimization ability other than the computational efficiency.

Our findings suggest that the merits of spiking action potentials are beyond the scope of the conventional framework for learning: We believe that the functions of spiking neurons must be within the scope of emergent adaptability and instantaneous behavioral optimization. Moreover, we deduce that the emergent adaptability would increase the chances of success in a sensorimotor task and therefore enable a smooth and rapid sensorimotor learning process.

An sSNN to Realize a Brownian Ratchet in Large-Scaled Systems.

Our results imply that the sSNNs can provide the inhomogeneous noise effect to a musculoskeletal system; the force noise generated by sSNNs helps an agent overcome the potential barrier to reach a goal state for a certain task, and simultaneously, a control signal to prevent the force noise from pushing the agent from the goal state is provided by the nonlinear firing function of an sSNN. Similar adaptive and autonomous switching between exploiting and blocking the noise effect is seen in a model of neural dendrosomatic interaction (52). An sSNN may be a key platform to realize a Brownian ratchet-like mechanism (53, 54) in a large-scaled system such as animal musculoskeletal systems where the effect of thermal noise is negligible. A theoretical analysis must be necessary for further understanding of this ratchet-like behavior of sSNNs.

From the Perspective of Spike-Induced Ordering.

Animal skeletal muscle groups are controlled by a limited number of motor neurons. For example, the motor axons of a human skeletal muscle group (such as the tibialis anterior muscle [TA], which consists of more than 250,000 muscle fibers) are controlled by at most 500 neurons (55). This indicates that the PSP signals induced in a muscle group are principally not smooth and are very far from the perfect rate-coding scheme. Our experimental results suggest that jerking and spiking MTU control signals lead to adequate complexity reduction and basin enlargement, as shown in Figs. 3A, 6, and 8. These results provide insight into why most animal skeletal muscles are innervated by a limited number of neurons: Motion control by a limited number of neurons may extend beyond the economical purpose of minimizing the control cost and act to exploit the effect of spike-induced ordering. Furthermore, we expect that the target function of spike-induced ordering is not limited to musculoskeletal motion control and may be applicable to the homeostatic regulation of endocrine systems, ordered peristalsis motions of digestive organs, and ordered respiratory dynamics.

From the Perspective of SNF and PSP Resonance.

In biological systems, the duration of an action potential (AP) is tightly coupled to the PSP duration and amplitude (56, 57). Therefore, we speculate that the effects induced by modulating the PSP timescale as demonstrated in this paper may be applicable to the effects induced by modulating AP duration. In particular, we hope that our results provide a theoretical basis as to why the AP duration in biological systems differs, depending on the neuron type and target function (56). Furthermore, we speculate that one of the functionalities of neuromodulators such as noradrenaline (NA), acetylcholine, and serotonin (5-HT) may be in the controlling AP waveform and duration, and PSP time constant (58–61), to synchronize with the system’s intrinsic dynamics. In fact, it is known that NA in a respiratory CPG modulates the respiration frequency (62). Likewise, 5-HT in the spinal central pattern generators (CPGs) is required to generate stable locomotion pattern (63). We further expect that the effects of PSP resonance must not be limited to the generation of rhythmic motion and may play a certain role in the formation of rhythmic brain activity, e.g., $\alpha$, $\beta$, $\gamma$, and $\delta$ oscillations.

Note that an sSNN described in this paper does not contain any synaptic connection among the neurons in an ensemble. We point out that several computational models for spinal and respiratory CPG circuits (64, 65) are based on the same assumption: A neuron population consisting of 50 to 100 spiking neurons does not involve any synaptic connection within a population and has synaptic interconnection only with other neuron populations.

Conclusion

In this study, we investigated the function of stochastic spikes with respect to the spikiness and PSP timescale. We reported four findings (F1 to F4) regarding the benefits of stochastic spikes: F1) synchronization to an intrinsic frequency, F2) amplification of the resonant motion in a natural frequency by PSP timescale, F3) basin enlargement, and F4) rapid complexity reduction and pattern formation. We demonstrated that the integration of these four functions leads to the on-site adaptability of a sensorimotor agent. We further observed that F3 and F4 are roughly correlated with the level of spikiness, while F1 and F2 are correlated with PSP timescale. Artificial intelligence systems developed using stochastic spikes may exhibit substantial differences in adaptability when compared with conventional systems engineered using nonspiking neuron models.

Data Availability.

The source codes used to generate the results in this paper are available at http://www.isi.imi.i.u-tokyo.ac.jp/public/spike_induced_order/.

Materials and Methods

A Model of Stochastic LIF Neuron Ensemble.

The membrane voltage of the $j$th neuron (in the $i$th sSNN) is represented by $v_{ij}$, and when $v_{ij}$ reaches the threshold $v^{\theta }$ ($v^{\theta }=20$ throughout this paper), the neuron emits a spike represented by the Dirac delta function $\delta$ and enters a refractory period described by ${\tau }_{ref}$. Simultaneously, the membrane voltage is reset to $v_{ij}^R$. Formally, the full dynamics of an LIF neuron can be described as (66)

$$\tau {\stackrel{\cdot }{v}}_{ij}=-\gamma v_{ij}+b_{ij}+I_i\left(t\right)+\sqrt{2D_i}{\xi }_{ij}\left(t\right),$$ [3a]

$$v_{ij}=v^{\theta }\Rightarrow t_{ijk}≔t,v_{ijk}\left(t+{\tau }_{ref}\right)\hspace{0.17em}=\hspace{0.17em}{v}_{ij}^R,$$ [3b]

$${\mathcal{T}}_{ij}=\left\{..,t_{ijk-1},t_{ijk},t_{ijk+1},..\right\},$$ [3c]

where $b_{ij}$ is the bias current, $D_i$ is the noise intensity of the $i$th sSNN, ${\tau }_{ref}$ is the refractory period, ${\tau }_{ref}=2$ ms, $\xi$ is the Gaussian noise of the unit intensity, and $\tau$ is the membrane time constant and $\tau =10$ ms unless otherwise stated. $\xi$ represents the noisy synaptic background activity (6, 66, 67) and is the source for the stochasticity of an sSNN. Note that we use the suffixes $i$, $j$, and $k$ to represent the sSNN index, the neuron index, and the spike index, respectively. The spike train of the $j$th LIF neuron is formally described as ${\sigma }_{ij}\left(t\right)={\sum }_{t_{ijk}\in {\mathcal{T}}_{ij}}\delta \left(t_{ijk}-t\right)$ (6). The $v_{ij}^R$ and $b_{ij}$ are different for different neurons and are randomly sampled from the range $V_i^R:\left[v_{min}^R,v_{max}^R\right]$ and $B_i:\left[b_{min},b_{max}\right]$, respectively. Note that the $i$th sSNN has each independent profile for $D_i$, $V_i^R$, and $B_i$, and $I_0,I_1,\dots$ are independent of each other. Furthermore, we used different $b_{ij}$ values for different neurons in an ensemble in our numerical simulations of a musculoskeletal bipedal walker, and we used the same bias signal $b_{ij}=b$ in the numerical simulations of simplified sensorimotor agents.

Main Idea: Parametric Control of the Stochasticity and Discontinuity of sSNN Output.

To simplify the theoretical analysis, we assume that the $b_{ij}$ are identical for all neurons in the same sSNN as $b_{ij}=b_i$. Based on the finite-size mean-field theory developed in refs. 18 and 19 and linear-response theory (6, 20, 21), the averaged spike train can be described using Gaussian approximation as

$$\frac{1}{N}\sum _{j=1}^N{\sigma }_{ij}\left(t\right)\approx r\left(b_i+I_i\left(t\right)\right)+\sqrt{\frac{1}{N}}{\eta }_i\left(t\right),$$ [4a]

$$\approx r_0\left(b_i\right)+\sum _{\omega }G\left(\omega \right)s\left(\omega ,t\right)+\sqrt{\frac{1}{N}}{\eta }_i\left(t\right),$$ [4b]

where, $r\left(x\right)$ is a certain firing function, $r_0$ is the stationary firing rate, and $G\left(\omega \right)$ is the linear-response function. $I_i\left(t\right)$ consists of several periodic signals such as $I_i\left(t\right)={\sum }_{\omega }s\left(\omega ,t\right)$ (21). We used a simplified approximation that the spike train of a LIF neuron is based on the Poisson process and assumed that ${\eta }_i\left(t\right)$ is Gaussian white noise of the variance $r_0\left(b_i\right)$ (6). Note that ${\eta }_i$ has a scaling factor of $\sqrt{N}$ rather than $N$ because of the independency of ${\sigma }_{ij}$ and ${\sigma }_{ij\prime }$; i.e., ${\int }_0^{\infty }{\sigma }_{ij}\left(t\right){\sigma }_{ij\prime }\left(t\right)dt=0$ ($j\hspace{0.17em}\ne \hspace{0.17em}j\prime$). Note that to be theoretically exact, however, it is shown that ${\eta }_i\left(t\right)$ is nonwhite but colored because of the LIF refractory period and membrane dynamics (18, 19). The plot of $r\left(x\right)$ and $r_0\left(x\right)$ mostly takes a sigmoidal shape with respect to the input $x$ (6, 20), providing an activation function similar to that used in conventional nonspiking neural networks.

Using the abovementioned approximation of ${\sigma }_{ij}$ in Eq. 4, the dynamics of $y_i\left(t\right)$ in Eq. 1 are approximated as follows:

$${\tau }_s{\stackrel{\cdot }{y}}_i\approx -y_i+r_0\left(b_i\right)+\sum _{\omega }G\left(\omega \right)s\left(\omega ,t\right)+\sqrt{\frac{1}{N}}{\eta }_i\left(t\right).$$ [5]

Eq. 5, which is obtained by substituting Eq. 4 for Eq. 1, implies that the PSP signal generated by the output spikes of the sSNN asymptotically approaches the rate-coded smooth behavior and that this asymptotic scaling is primarily based on $\sqrt{N}$. If the PSP signal $y\left(t\right)$ can be expressed similarly to Eq. 5 as

$$y_i\left(t\right)=Y_0\left(b_i\right)+\sum _{\omega }\mathcal{F}\left(\omega \right)s\left(\omega ,t\right)+\sqrt{\frac{1}{N}}{\zeta }_i\left(t\right),$$ [6]

where $Y_0$ is the stationary function of $y$, $\zeta$ is a certain stochastic function, and $\mathcal{F}$ is a linear-response function of $y$ against the input $s\left(\omega ,t\right)$, we can compute the analytical form of $Z$ as

$$Z=\frac{1}{{\tau }_s^2\sqrt{N}}{\left(N{\mathrm{\Phi }}_0^2+\text{Var}\left[\zeta \right]+\text{Var}\left[\eta \right]+{\tau }_s^2\text{Var}\left[\stackrel{\cdot }{\eta }\right]\right)}^{1/2},$$ [7]

where ${\mathrm{\Phi }}_0=\left(r_0\left(b_i\right)-Y_0\left(b_i\right)\right)$ corresponds to the stationary nonspiking behavior. $\stackrel{\cdot }{\eta }\left(t\right)$ is a stochastic process obtained from the derivative of $\eta \left(t\right)$ (if $\eta \left(t\right)$ is a Gaussian white noise, $\stackrel{\cdot }{\eta }\left(t\right)$ is referred to as purple noise), assuming that the time averages of the stochastic variables $\eta \left(t\right)$, ${\eta }^{\prime }\left(t\right)$, and $\zeta \left(t\right)$ are 0. Furthermore, we assumed $s\left(\omega \right)=0$ for simplicity. Eq. 7 indicates that $Z$ is modulated by both $N$ and ${\tau }_s$.

Configuration of a Musculoskeletal Bipedal Walker.

The bipedal body consists of seven rigid bodies: an upper body along with the left and right thighs, shanks, and feet. Each leg of the musculoskeletal system has eight MTUs, resulting in a total of 16 MTUs for the legged system. We used basically the same parameters as ref. 22 except that we use the maximum contract velocity of a muscle $v_{max}^c=12l_{opt}\hspace{0.17em}{\text{s}}^{-1}$, where $l_{opt}$ [$m$] is the optimal muscle length to generate the maximum force. The numerical simulation of the rigid bodies, the joint-limiting dynamics, the body-ground contacts, the ground reaction force, and the MTU forces are integrated by the Open Dynamics Engine (ODE) (68), using a step size of 1 ms.

Base Reflex System.

Each leg of a walker has four states: stance, swing initiation, swing, and stance preparation. In each phase, the nonspiking activation signals for the corresponding MTUs are determined using reflex rules proposed in ref. 22. Here, we describe the reflex rules only briefly.

The MTU activation signal is determined by three basic rules: positive force feedback, MTU length feedback, and proportional-derivative (PD) control. The MTU activation $R_m^F$ generated by the positive force feedback rule is determined using the gain $G_i^F$ as $R_i^F=G_i^F{F}_i^{MTU}\left(t\right)$, where $F_i^{MTU}\left(t\right)$ is the force generated by the $i$th MTU. The length feedback rule works to contract the MTU toward the desired length and contributes to MTU activation by $R_i^L$ as $R_i^L=G_i^L{\left[l_i-L_i^g\right]}_{\pm }$, where $L_i^g$ is the desired muscle fiber length, and ${\left[z\right]}_{+}=z$ if $z$ is positive and 0 otherwise. Likewise, ${\left[z\right]}_{-}=z$ if $z$ is negative and 0 otherwise. PD control mainly regulates the angle of the upper body and contributes via $R_i^{\theta }$ as $R_i^{\theta }=K_i^p{\left[\theta \left(t\right)-{\theta }_i^g\right]}_{\pm }+K_i^d{\left[\stackrel{\cdot }{\theta }\right]}_{\pm }$, where $\theta$ is the rigid-body posture (such as upper body) or the joint angle (e.g., knee angle), $K_i^p$ is the positional feedback gain, and $K_i^d$ is the derivative feedback gain.

A smooth nonspiking activation signal $I_i^r$ generated by a base reflex system is basically composed of a combination of these three reflex rules as $I_i^r=R_i^F+R_i^L+R_i^{\theta }+R_i^c-U_i^a$, where $R_i^c$ is the constant activation signal, and $U_i^a$ is inhibitory input from the correspondent antagonist muscle (e.g., TA receives the inhibitory input from its antagonist soleus muscle [SOL] as $U_i^a=F_{SOL}^{MTU}$ during the stance phase).

A smooth, nonspiking activation signal $I_i^r$ (although it is highly nonlinear because of the threshold function ${\left[z\right]}_{\pm }$) computed by the base reflex system is passed to the corresponding ensemble of $N$ LIF neurons as $I_i\left(t\right)=g_i^r{I}_i^r$, where $g_i^r$ is the input gain. The MTU activation level is determined by the EPSP signal induced by the corresponding neuron ensemble, and the generative force of an MTU is computed based on the dynamics proposed in ref. 23.

A simple reflex controller activates an MTU simply as $A_i\left(t\right)=I_i^r\left(t\right)$, and a sigmoidal controller activates an MTU using a sigmoid function as $A_i=g_i^s\left(1/\left(1+\exp \left(-c_i^s{I}_i^r\right)\right)-1/2\right)$, where $g_i^s$ is the actuation gain and $c_i^s$ is the input gain.

Optimization of Controllers.

The cost function $J$ for the real-coded GA with the BLX-$\alpha$ method (24) was designed so that the center-of-mass (CoM) velocity of the agent was within the range of $v^g\pm 0.1$, where we used the target velocity $v^g=1.75$ m/s that was correspondent to “very fast walking speed” (22). In particular, we used $J=\frac{1}{T}{\int }_{t_0}^{t_0+T}{J}_v\left(|v^g-v\left(t\right)|\right)dt+J_p$, where $v^g$ and $v$ were the desired and actual CoM velocities, respectively, and $J_v\left(z\right)\hspace{0.17em}=\hspace{0.17em}z$ for $z>0.1$ m/s and$\hspace{0.17em}=\hspace{0.17em}0$ otherwise. We basically used $t_0=5$ s and $T=20$ s but applied $T=140$ s for $10\%$ individuals to avoid the local minimum. $J_p$ is the penalty term for falling, jumping, toe striking, or stopping (SI Appendix, section S2).

A simple reflex controller was obtained by optimizing all parameters of the base reflex systems such as $G_i^{F/L}$, $L_i^g$, and $K_i^{p/d}$. Please see ref. 22 for the full parameters list of the base reflex system that must be optimized. An sSNN controller was obtained by optimizing all parameters for the base reflex system and the parameters $V_i^R$, $B_i$, $D_i$, $g_i^A$, and $A_i^0$ of the sSNN. As for the $\tau$, ${\tau }_s$, and $N$, we used the fixed values $\tau =10$ ms, ${\tau }_s=5$ ms, and $N=40$ during optimization. Likewise, a sigmoidal controller was obtained by optimizing the base reflex systems and the parameters of sigmoid functions $g_i^s$ and $c_i^s$. Note that the parameters of the base reflex system are different among reflex, sigmoidal, and sSNN controllers. This is because, in general, a reflexive signal that is optimal to drive an MTU via an sSNN or via a sigmoid function is different from the optimal reflexive output required for direct control of an MTU.