A spiking neural program for sensorimotor control during foraging in flying insects

Significance

Living organisms demonstrate remarkable abilities in mastering problems imposed by complex and dynamic environments, and they can generalize their experience in order to rapidly adapt behavior. This paper demonstrates the benefits of using biological spiking neural networks, sparse computations, and local learning rules. It highlights the functional roles of temporal- and population-sparse coding for rapid associative learning, precise memory recall, and transformation into navigational output. We show how memory formation generalizes to perform precise memory recall under dynamic, nonstationary conditions, giving rise to nontrivial foraging behavior in a complex natural environment. Results suggest how principles of biological computation could benefit agent-based machine learning to deal with nonstationary scenarios.

Abstract

Foraging is a vital behavioral task for living organisms. Behavioral strategies and abstract mathematical models thereof have been described in detail for various species. To explore the link between underlying neural circuits and computational principles, we present how a biologically detailed neural circuit model of the insect mushroom body implements sensory processing, learning, and motor control. We focus on cast and surge strategies employed by flying insects when foraging within turbulent odor plumes. Using a spike-based plasticity rule, the model rapidly learns to associate individual olfactory sensory cues paired with food in a classical conditioning paradigm. We show that, without retraining, the system dynamically recalls memories to detect relevant cues in complex sensory scenes. Accumulation of this sensory evidence on short time scales generates cast-and-surge motor commands. Our generic systems approach predicts that population sparseness facilitates learning, while temporal sparseness is required for dynamic memory recall and precise behavioral control. Our work successfully combines biological computational principles with spike-based machine learning. It shows how knowledge transfer from static to arbitrary complex dynamic conditions can be achieved by foraging insects and may serve as inspiration for agent-based machine learning.

Navigating toward a food source during foraging requires dynamical sensory processing, accumulation of sensory evidence, and appropriate high-level motor control. Navigation based on an animal’s olfactory sense is a challenging task due to the complex spatiotemporal landscape of odor molecules. A core aspect of foraging is the acquisition of sensory cue samples in the natural environment, where odor concentrations vary rapidly and steeply across space. Experimental access to the neural substrate is challenging in freely behaving insects. Biologically realistic models thus play a key role in investigating the relevant computational mechanisms. Consequently, recent efforts at understanding foraging behavior have focused on identifying viable computational strategies for making navigational decisions (1).

An odor plume is often considered a volume wherein odor concentration is generally above some behavioral threshold. At macroscopic scales and in a natural environment, however, plumes are turbulent (2, 3). In turbulent conditions, a plume breaks up into complex and intermittent filamentous structures that are interspersed with clean air pockets or below behavioral-threshold concentration patches (4, 5). The dispersing filaments form the cone-like shape of the macroscopic plume, where the origin of the cone yields the position of the odor source. When entering the cone, flying insects encounter odor filaments as discrete, short-lived sensory events in time.

Several features have been derived from the statistics of an odor plume that provide information regarding the location of the odor source (3, 4). The mean concentration varies smoothly in lateral and longitudinal directions of time-averaged (and laminar) plumes. However, for behavioral strategies, animals cannot afford the time it takes to obtain stable macroscopic estimates of mean concentrations (2). Hildebrand and colleagues (6) proposed the time interval between odor encounters as an informative olfactory feature, while ref. 3 suggested intermittency, the probability of the odor concentration being above some behavioral threshold, as the relevant feature. However, similarly to estimating mean concentration, acquiring a sufficient number of samples for stable estimates of these quantities exceeds the time typically used to form behavioral decisions (2). Hence, obtaining time-averaged quantities is not an optimal strategy to guide navigational decisions, as concluded by ref. 7.

Most animals perform searches at large distances from the odor source, where the intermittency of plumes poses a more severe problem, as available sensory cues become more sparse in space and time. Thus, strategies that exploit brief, localized sensory cues for navigation have been studied by several groups. One strategy for medium- and long-range navigation that has consistently been observed across species of flying insects emerges from a sequence of chained sensorimotor reflexes: casting and surging (8). Encountering a whiff of odor triggers an upwind surge behavior, during which the insect travels parallel to the wind direction. After losing track of the plume, it evokes a cross-wind cast behavior, in which a flight path perpendicular to the direction of airflow is executed. Performing repeated casts by U-turning allows the insect to reenter and locate the plume in order to trigger the next upwind surge (8–10). As the subject approaches the source, it increasingly makes use of visual cues for navigation as the plume narrows down (8).

A number of studies have proposed abstract mathematical models for optimal search algorithms that assumed different types of relevant navigational cues. The infotaxis method proposed in ref. 11 depends on extensive memory and priors regarding a plume’s structure. To the contrary, in ref. 8, only local cues are used. A standard algorithm for navigational problems in robotics is simultaneous localization and mapping (SLAM), which has been used in ref. 12 to study olfactory navigation in bumblebees. An algorithm that works without space perception has been proposed by ref. 13 using a standardized projection of the probability of source position and minimization of a free energy along the trajectory. Finally, the work of ref. 10 compares several models and shows that it is difficult to discriminate between different models based on behavioral responses. A recent work (7) using information-theoretic analysis shows that plumes contain both spatial and temporal information about the source’s position.

While all of these previous mathematical methods for olfactory search algorithms have proven to successfully solve this task based on the respective assumptions, they share the same major drawback: None of them uses the computational substrate of the brain, spiking neurons and networks thereof. Instead, all methods make heavy use of symbolic math and advanced mathematical concepts that are not available to the biological brain. It is further unclear how and to what extend these methods could be implemented or learned by the nervous system. Additionally, the problem of navigation and foraging is often considered as an isolated task, independent from sensory processing.

Our approach distills recent experimental results to formulate a biologically plausible and detailed spiking neural-network model supporting adaptive foraging behavior. We thereby take advantage of the rapidly accumulating knowledge regarding the anatomy (e.g., refs. 14–16) and neurophysiology (e.g., refs. 17–19) of insect olfaction and basic computational features (20, 21). We follow the idea of compositionality, a widely used concept in mathematics, semantics, and linguistics. According to this principle, the meaning of a complex expression is a function of the meanings of its constituent expressions [Frege principle (22)]. In the present context of foraging and navigation, this means dynamically recombining memories of individual sensory cues present within a plume.

Results

We approached the problem of foraging by decomposition into four components: first, sensory processing with temporal sparse and population sparse coding in the mushroom body (MB); second, associative learning for assigning a valence to individual odor identities; third, the time-dependent detection of valenced cues resulting from encounters of discrete odor filaments to provide an ongoing and robust estimate of sensory cue evidence; and fourth, the translation into online motor command signals to drive appropriate behavior.

For sensory processing, we used a three-layer spiking neural network model of the insect olfactory pathway (Fig. 1). The generic blueprint of the insect olfactory system is homologous across species and comprises three successive processing stages (see Materials and Methods for details): the periphery with olfactory receptor neurons (ORNs), the antennal lobe (AL), and the MB. Excitatory feed-forward connections across layers from ORNs to projection neurons (PNs), from ORNs to local interneurons (LNs), and from PNs to the MB Kenyon cells (KCs) are fixed. Lateral inhibition within the AL uses fixed synaptic weights from LNs to PNs. For neuron numbers and their connectivity patterns, we here rely on the adult Drosophila melanogaster, where anatomical knowledge is most complete (14, 23, 24). A single MB output neuron (MBON) receives input from all KCs and plasticity at the synapses between KCs, and the MBON enables associative learning (25, 26).

Fig. 1.

Spiking network model of the insect olfactory system. Olfactory receptor neurons (N = 2,080) at the antenna bind and respond to volatile odorant compounds. ORNs expressing the same (1 of 52 different) genetic receptor type converge onto the same glumerus in the AL. Each of the 52 glomeruli comprises one PN and one LN. Each LN forms lateral inhibitory connections with all PNs. PNs randomly connect to a large population of KCs (N = 2,000), where each KC receives input from, on average, 6 random PNs. All KCs project to a single MBON via plastic synapses.

Sparse Coding in Space and Time.

The olfactory system transforms a dense olfactory code in the AL into a sparse stimulus code at the MB level. In the large population of KCs, a specific odor stimulus is represented by only a small fraction of all KCs (population sparseness), and each stimulus-activated KC responds with only a single or very few action potentials (temporal sparseness). In our model, temporal sparseness is achieved through the cellular mechanisms of spike-frequency adaptation (SFA; refs. 21, 27, and 28) implemented at two levels of the system.

ORNs show clear stimulus-response adaptation that has been attributed to the spike-generating mechanism (29). Based on this experimental evidence, we introduced a slow and weak SFA conductance in our model ORNs (Materials and Methods). At the level of the MB, KCs have been shown to express strong SFA-mediating channels (18). This is matched by the SFA parameters of our model KCs (Materials and Methods; ref. 21). As an effect of cellular adaptation in ORNs and KCs, odor stimulation (Fig. 2A) results in temporally precise and adaptive responses across all layers of the network (Fig. 2B). The effect of SFA implemented in ORNs is transitive and, thus, carries over to the postsynaptic PN and LN populations in agreement with experimental observations across species (30–33). SFA has the additional advantage of inherently reducing the variability of spiking output (28, 34).

Fig. 2.

Rapid associative learning expressed in neuronal plasticity and CR behavior. (A) Sketch of differential conditioning protocol. In appetitive trials, a first sensory cue (conditioned stimulus; CS+; orange) is paired with a reward (unconditioned stimulus; US). In aversive trials, a second sensory cue (CS−; blue) is paired with a punishment. Both trial types are presented randomized within blocks (Materials and Methods). (B) Sensory input and neuronal responses across all four circuit layers (ORN, AL, MB, and MBON) in response to a CS+ odor presentation during the 10th training trial. Stimulus onset is at $t=0$ s. From top to bottom: Model input is provided through independent noise-current injection into the ORNs. The stimulus-induced input currents are clearly visible (hot colors) on top of the background noise for the subset of ORNs that are sensitive to the CS+ odor (stimulus profile). Stimulus response is clearly visible by an increase in the spiking activity across all neuron populations. For ORNs (blue) and PNs (red), relevant subsets of 60 and 35 neurons are shown. The population of 2,000 KCs (magenta) shows a temporal and spatial sparse odor response. Only $2\%$ of all KCs are activated during a brief transient response following stimulus onset (black histogram). The MBON generates a single action potential in response to cue onset, which is the correct learned response to the CS+ odor. (C) Learning performance of the MBON across $N=100$ independent models as a function of the number of training trials. In any given trial, the MBON response was correct if exactly one action potential was generated during CS+ presentation or if no action potential was generated during CS− presentation. (D) The behavioral learning curve expresses the percentage of individuals that showed a correct behavior in the respective CS+ or CS− trial. The behavioral output is binary with either response or no response. The model triggers a response if the MBON generates one or more spikes.

In the KC population, the background firing rate is very low ($\sim 0.4$ Hz). This is partially due to the outward SFA conductance and in agreement with experimental results (17). The KC population response is highly transitive where individual responding cells generate only a single or very few response spikes shortly after stimulus onset. This is in good qualitative and quantitative agreement with the temporal sparse KC spike responses measured in various species (17, 30, 35).

Population-sparse stimulus encoding at the level of KCs is supported by two major factors. First, the sparse divergent-convergent connectivity between the PNs and the 20-times-larger population of KCs is the anatomical basis for sparse odor representation (15, 20, 21, 36) (SI Appendix, Table S1). Second, lateral inhibition mediated by the LNs in the AL (37) facilitates decorrelation of odor representations (37) and contributes to population sparseness (21). The sparse code in the KC population has been shown to reduce the overlap between different odor representations (38, 39), and, consequently, population sparseness is an important property of olfactory learning and plasticity models in insects (40–44). The system response to a single odor presentation in Fig. 2B demonstrates the transformation of a dense olfactory code at the ORN and PN layers into a population-sparse representation at the KC layer, where less than $<2\%$ of the total KC population is active at any time during stimulus presentation. This is in good agreement with quantitative estimates in the fruit fly (23, 39, 45).

Few-Shot Learning Rapidly Forms an Associative Memory of Single Cues with Rewards.

Many insects exhibit a rapid learning dynamics when trained in classical olfactory conditioning tasks. They typically acquire high retention scores (test accuracy $>60\%$) for a binary conditioned response (CR) behavior within only very few trials (e.g., refs. 46–48).

We here mimic a standard experimental laboratory protocol for differential conditioning (or acuity learning) to form associative memories and to generate a binary CR behavior by training our network (Fig. 1). Across successive learning trials, we present two different odors in pseudo-random trial order (Fig. 2A). Each trial constitutes a single odor presentation for $500$ ms, followed by a reinforcing stimulus (US) occurring shortly after the stimulus presentation. The CS+ odor is paired with a reward, the CS− odor with a punishment (Materials and Methods). In order to establish a neural representation of the odor valence at the MB output (49–52), the MBON is trained (25, 26) to elicit exactly one action potential in response to the CS+ stimulus that is paired with the reward and zero action potentials when the CS− stimulus is presented (Materials and Methods). The system response to a single CS+ stimulus after nine conditioning trials is shown in Fig. 2B.

In a first step, we quantified the learning performance by considering the accuracy of the MBON response. MBON output is counted as correct if exactly one spike is generated during a CS+ trial and zero spikes during a CS− trial. The average accuracy over $N=100$ independently trained model instances across successive trials is shown in Fig. 2C. The learning dynamics shows a steep and steady increase, indicating that an accurate memory is formed rapidly reaching up to $80\%$ accuracy after 50 ($25\times$ CS+ and $25\times$ CS−) training trials.

Next, we consider the behavioral learning curve, i.e., the acquisition of a binary CR behavior across successive learning trials. In each trial, the model generates a behavioral response if the MBON produces one or more action potentials in response to the stimulus. No response is generated if the MBON remains silent. A CR is counted as correct if the MBON generates a response to the CS+ cue or no response to the CS− cue. The learning curve in Fig. 2D represents the percentage of correctly responding individuals across $N=100$ independently trained models. The untrained model, by default, does not generate any output spike; consequently, $100\%$ of the independent models correctly respond to CS− trials from the beginning (Fig. 2D, blue). The orange curve shows a rapid learning success where up to $70\%$ of individuals generated the correct, appetitive CR to the CS+ stimuli within only three to five trials. The learning curve saturates after $\sim 10$ trials with an asymptotic value of $\sim 80\%$ correct responders. This reproduces the rapid learning dynamics of insects in classical conditioning experiments and fits qualitatively and quantitatively the CR behavior in honeybees (for review, see ref. 47).

We conclude that our statically configured sensory network model with a single plastic readout neuron is capable to successfully form associative memories by few-shot learning, replicating the classical conditioning experiments in the typical laboratory situation. The computational mechanism of population sparseness implemented in our model increases discriminability of the two different stimuli supporting a rapid learning dynamics and a high accuracy of memory recall (SI Appendix, Figs. S1 and S2).

Robust Dynamic Memory Recall and Odor-Background Segregation in Complex Sensory Scenes.

We now challenge our previously trained model (Fig. 2) in a novel task asking whether the already-learned odor associations can be reactivated in a complex and dynamic olfactory scene. To this end, we mimicked the encounter of odor filaments in a turbulent odor plume during a foraging flight (Fig. 3A). For this, we presented random sequences of nonoverlapping olfactory cues within $T=10$ s (Materials and Methods). Each cue was of variable duration in the range between $1$ and $200$ ms. Odor identity of each cue was randomly assigned to either the CS+, CS− odor or one out of three additional background odors (Fig. 4A). The use of nonoverlapping cues follows the rationale that, in nature, filaments originating from different odors do not mix perfectly (53).

Fig. 3.

Recognition of valenced odor cues in complex dynamic scenes. (A) Sketch of the dynamical memory-recall task mimicking the sensory experience during a natural foraging flight. In each single trial of $10s$ duration, the model encounters multiple (on average five) cues of different odor identities, including the CS+ odor (orange), the CS− odor (blue), and background odors (gray). (B) Network response to one example input sequence made up of three CS+ cues, two CS− cues, and two distractor cues (bg odor), as indicated at the top. The ORN input sequence indicates the fluctuating duration of odor cues. Transient PN and LN response profiles faithfully represent individual odor-cue onsets in time and odor identity across neuronal space. The KC population shows clear responses to all individual odor cues, albeit with $<4\%$ of activated cells at any time. The MBON correctly produced a single action potential in response to each of the three CS+ cues, and zero output else.

Fig. 4.

Performance in the dynamic memory recall task. (A) Input activation profiles across ORNs for five different odors. The model had been previously trained in the differential conditioning protocol using the orange CS+ and the blue CS− odors. In the dynamic memory-recall task (see Robust Dynamic Memory Recall and Odor-Background Segregation in Complex Sensory Scenes), background odors (gray) were presented as distractor cues, along with CS+ and CS− cues. In the first task variant, two background odor profiles were rather distinct from CS+ (dark gray), and one was more similar (light gray). (B) Task accuracy across 200 trials in four different scenarios. A single trial consists of a random temporal sequence of sensory-cue presentations during $10$ s. Each single cue is of random duration between 1 and 200 ms. The single trial model response was correct if the MBON generated exactly as many action potentials as CS+ cues had been presented. The four task scenarios varied number (between one and three) and type (distinct vs. similar) of distractor cues, as indicated. (C) In the second task variant, the same odors were used for cue presentation in a random temporal sequence. However, the model had been trained with reversed CS+/CS− odor contingency, such that two distractor odors (light gray) were now more similar to the CS+ odor (orange). (D) As in B, but for reversed CS+/CS− odor contingency, where distractor cues were overall more similar to the CS+ odor, increasing task difficulty.

The objective in this memory-recall task is to correctly detect the occurrences of the positively valenced odor (CS+) by means of a single MBON action potential as model output, while no output should be generated for all other cues (CS− or distractor odors). Fig. 3B shows the system’s response to a single random stimulus sequence where the MBON correctly generated a single action potential in response to each of three CS+ encounters. For quantification of task accuracy, we considered the overall response to a given sequence to be correct if the number of action potentials generated by the readout neuron is equal to the number of CS+ cues.

For assessing model performance, we systematically varied task difficulty by varying the number of possible background odors (between one and three) and their similarity with the CS+ odor (Fig. 4). In a first task variant, background-odor activation profiles are rather distinct from the CS+ odor and more similar to the CS− odor (Fig. 4A). Accuracy of the model response was computed across 200 test sequences as shown in Fig. 4B. We find that our previously trained model successfully generalized to this new task with $\sim$80% accuracy for different sequence complexity in terms of identity and number of background odors. In a second task variant, we reversed the odor contingency of the CS+ and CS− odors during initial differential conditioning. Thus, the reward predicting odor CS+ is now more similar to two of the background odors, while similarity with the third background odor remains unchanged (Fig. 4C). In this more challenging case, accuracy reduces to $\sim 50\hspace{0.17em}\%$ of sequences for which the model produced the correct number of MBON output spikes. Note that the accuracy measure in Fig. 4 is based on the correct cumulative spike count during a complete trial of $10$ s. The more similar a background-odor stimulus profile is to the CS+ odor, the more likely the model will produce false-positive (FP) action potentials in response to such a similar odor and, thus, a total spike count that is higher than the number of CS+ occurrences. This is reminiscent of the effect observed in insects and other animals in odor-discrimination tasks, where perceptually similar odors are more difficult to distinguish from previously learned CS+ odors than perceptually dissimilar odors during memory-retention tests. This might be overcome if similar odors are used during the initial differential conditioning.

We conclude that our network model is able to recall previously learned neural representation of odors and signal their valence in a temporally dynamic setting, where the rewarded and punished odors appear with up to 500-times shorter durations and within an unpredictable temporal cue sequence of previously unknown background odors. The model, thus, also solves the problem of odor vs. background segmentation under quasinatural conditions (54).

Accumulation of Sensory Evidences Informs Motor Control in Foraging.

We now consider the situation of foraging within a natural environment (Fig. 5A). The objective is to locate the food source, which emits an attractive odor (CS+), by utilizing the sensory cues present in its turbulent odor plume. We show that cast-and-surge behavior can emerge by accumulation and exploitation of sensory evidence of sequentially experienced individual cues.

Fig. 5.

Dynamical sensory processing and motor control serving chemotaxis. (A) Sketch of a typical olfactory experimental setup in a wind tunnel with a pleasant odor source (orange flower) and a second distractor source (gray flower). Due to turbulence, the odor molecules emitted by a single source form dispersing, intermittent filaments within a cone-like boundary that constitutes the odor plume. The plume is modeled as Gaussian-distributed filaments. A behaving model insect (here, D. melanogaster) performs stereotypic cast-and-surge behavior to locate the food source. This constitutes alternating between scanning cross-wind and U-turning after running past the plume-cone boundary where no filaments are present. Eventually, after several casts (here, three), it surges upwind until it loses track of the plume cone and starts over. (B) Filament encounters during this behavior result in sequential brief on/off stimulations of the olfactory system. The probability of encountering filaments is $>0$ within the plume and zero outside of the plume. Sensory evidence $e\left(t\right)$ can be viewed as a likelihood function of filament encounters that increases toward the plume’s center line and is zero outside of the plume. The properties of this function can be used to generate optimal motor commands for chemotaxis. (C) Evidence $e\left(t\right)$ and derivative $\frac{de}{dt}$ over four simulated successive casting trajectories estimated from the MBON spiking activity. U-turn motor commands (purple diamonds) are generated when $e\left(t\right)$ runs below a fixed threshold (0.01), and surge motor commands (purple circles) are generated when $\frac{de}{dt}$ turns negative. The motor commands generated by the model match well with the theoretically optimal commands, as sketched in B. (D) Spiking activity of the MBON (orange) in response to 10 casting trajectories. The MBON reliably predicts the true sensory cues of positively valenced filaments (dark gray) and ignores background cues (light gray). (E) Smooth peristimulus time histogram computed over 10 casting trials recovers an accurate estimate of the true underlying sensory-cue distribution simulated as Gaussian distribution.

For this task, we assumed that thin odor filaments within a cross-wind plane of the concentric odor plume were approximately Gaussian-distributed. This is a reasonable assumption, particularly in a wind-tunnel setting with laminar flow, as typically used in experimental settings (8, 55). When the insect performs a cast through the plume, it encounters filaments as short-lived discrete, sequential events, where each encounter represents a single sensory cue (sketch in Fig. 5B). Therefore, in our simulation of casting flights, the agent encounters sequences of cues and distractors where cue onsets for the CS+ odor are drawn from a Gaussian distribution, while distractor cue onsets appear uniformly distributed over time (Materials and Methods). We further assume that the subject has already formed an association of food with the attractive odor, either through learning or through some genetically predetermined innate valence. To this end, we again used the trained model from the classical conditioning task above (Fig. 2) without any further retraining.

We simulated four consecutive casting trajectories where the agent senses odor cues of sequentially experienced filament encounters. Ongoing accumulation of sensory evidence (Fig. 5C) by low-pass filtering of the readout neuron’s output assumes positive values shortly after entering the plume cone and further increases while approaching the plume’s center line. When traveling beyond the center line, sensory evidence slowly decreases until the agent leaves the plume cone boundary. When sensory evidence drops to zero and after a fix delay, the agent initiates a U-turn motor command to perform another cross-wind cast.

Responses from our model’s readout neuron precisely followed the ground truth of CS+ odor cues, as shown by 10 random casting trajectories in Fig. 5D. Performing analysis by averaging of sensory evidence across these 10 casting trajectories yielded an average evidence (Fig. 5E) that faithfully resembled the underlying, true Gaussian profile of the simulated filaments.

We conclude that the model output provides an accurate and robust estimate of sensory evidence that can be used to reason about a plume’s spatial extend and center line. Both pieces of information are crucial to generate appropriate motor commands for U-turn and upwind surge behavior, necessary to successfully execute the cast-and-surge strategy. Apart from the existence of filaments inside a plume and absence outside a plume’s cone, our model does not make any specific assumption regarding the plume’s structure and statistics. It, thus, provides a generic mechanism implemented in a neural system to perform cast-and-surge behavior during foraging flights.

Discussion

Distinct Functional Roles for Population and Temporal Sparse Stimulus Encoding.

Population sparseness improves discriminability of different stimuli to facilitate associative learning. This has been demonstrated in theory and experiment (15, 20, 36, 39). We have shown that our neural network model implements this feature in a biologically realistic way, and our results confirm the functional role of population sparseness to support rapid and robust memory acquisition through associative learning.

Experimental (39) and theoretical (20, 56) studies in the fruit fly suggest that inhibitory feedback through the anterior paired lateral (APL) neuron improves population sparseness and learning in the KC population. In the adult fly, it likely receives input in both the calyx and the lobes of the MB, and it is thought to widely inhibit KCs and possibly PN synaptic boutons that are presynaptic to KCs in the calyx. Whether this neuron is generating sodium spikes or whether it is a nonspiking neuron is currently under debate (57). Here, we tested the performance of our model when implementing the APL neuron as spiking neuron that receives excitatory input from all KCs and inhibits all KCs via weak inhibitory synapses (SI Appendix, Fig. S2). Feedback inhibition increased the robustness of model function: In the case of more dense PN–KC connectivity and moderate to low sparseness ($\sim 5-10\%$ response-activated KCs), inhibitory feedback could partly re-establish rapid learning during training as well as high performance in the transfer test (SI Appendix, Fig. S2) and the foraging task (SI Appendix, Fig. S3). GABAergic feedback from MB output onto MB input is a recurring motif across species. In the locust, this is provided by a single nonspiking neuron (58). In the cockroach, there are exactly four spiking inhibitory feedback neurons per hemisphere that have been shown to receive input from the calyx and the lobes (59), while in the honey bee, a population of $\sim 50$ spiking and plastic neurons provides GABAergic feedback onto the synaptic boutons of PNs in the calyx (60, 61).

Our model demonstrates how temporal sparseness can be exploited to generate short-patterned signaling of cue identity. This enables perception of high temporal stimulus dynamics. In our model, this is achieved independently of the duration of individual stimulus incidents and their distribution in time and makes temporally precise and robust sensory evidence available. It allows for the ongoing computation of derived estimates such as cue distributions or changes in cue density. Maintaining temporally sparse representations mechanistically supports the principle of compositionality [or Frege principle (22)], where an atomic stimulus entity is represented and can be learned by the readout neuron before processing this output—for example, by estimation of densities or recombination with other entities to form composite perception or memory readout. The temporal stimulus dynamics remains intact throughout the system even after learning of stimulus relevance. Thus, valence is encoded with the same dynamics and faithfully captures occurrences of relevant cues. This allows compression of code to relevant stimuli, while retaining full stimulus dynamics of the external world. Compression of code along the sensory-processing pipeline is particularly relevant for small-brained animals like insects, which need to economize on their neuronal resources.

Odor-Background Segregation: A Joint Effect of Temporal and Population Sparse Cue Representation.

The task presented in Fig. 3 implicitly addresses the issue of odor-background segregation. This refers to the problem that, in nature, cues of multiple odors of different sources are present, either in terms of mixtures or stimulus-onset asynchrony due to turbulent conditions (53, 54). For behavior, it is relevant to reliably isolate and detect the relevant cues from any background or distractor cues. The results presented in Fig. 4 show that this works nicely in our system. This is achieved by exploiting the joint effect of temporal and population sparseness. Optimal discrimination of cue representation is guaranteed by population sparseness and temporal precision by means of temporal sparseness. Our plastic output neuron requires population sparseness for learning, and the plasticity rule (25, 26) allows for temporally precise memory recall. We predict that our model can solve the challenge of odor-background segregation.

Rapid Learning within Few Trials.

The ability of insects to quickly form associative memories after three to five trials has been demonstrated experimentally (47). However, in general, few-shot learning remains a difficult task for computational models, including insect-inspired models (62). We find that, when compared with learning-dynamics data of real insects (47), our model is able to show realistic learning dynamics that matches with the experimental observations. Due to frequent changes in the environment, it might be a better strategy to trade off fast and reasonable accurate learning against slow and high-precision learning. Additionally, acquisition of training samples might be costly, or they generally occur very sparsely.

In fact, few-shot learning likely is a fundamental skill for survival. We have demonstrated that our neurobiologically motivated approach using spike-based computations is capable to perform few-shot learning with similar speed as insects. We further showed that our model can transfer learned associations to novel, complex combinations that have not been part of the training data.

Innate vs. Learned Behavior.

Cast-and-surge behavior belongs to the innate behavioral repertoire of airborne insects and emerges from a set of sensorimotor reflexes (8). It can be considered as a base strategy that guarantees survival. The base system can be modulated and improved throughout an animal’s lifespan by experience-based learning. This is superior to alternative strategies that would solely rely on learning appropriate behaviors and, thus, require constant retraining, as is the case in machine-learning (ML) approaches. Here, we assumed that our readout neuron was tuned to a pleasant odor. In the present work, this tuning was learned (adaptive process) in a classical conditioning task. However, a tuning can generally be learned by other mechanisms, e.g., reinforcement or unsupervised learning. We demonstrated that the existence of such a tuned neuron allows cast-and-surge foraging behavior to emerge.

There are other ways how such a tuned neuron can come about, for example, due to genetically predetermined wiring or during development from larval to adult stage. The cast-and-surge behavior can be executed on innately valenced olfactory cues, and our suggested model for motor control during cast and surge (Fig. 5 A and B) also works for innate valenced stimuli. Learning is important to adapt behavior to changing environmental circumstances, and associative learning provides a means to learn new valences on demand in such situations. Our model learns odor valences at the MB output, and it has been shown that MBONs signal odor valence (49–52). We suggest that this valence is then used downstream to execute higher-level functions of motor control. At this processing stage, it might be integrated with innate valences and other necessary sensory modalities to form behavioral decisions.

Implications for Other Sensory Systems.

Sparse stimulus encoding has been identified as a powerful principle used by higher-order brain areas to encode and represent features of the sensory environment in invertebrate (17, 23, 45) and vertebrate (63–66) systems. Sensory systems with similar coding principles may share similar mechanisms when it comes to learning and multimodal sensory integration. The MB is a center for integration of multimodal sensory information. Thus, our model can be extended to incorporate input from different sensory modalities. It is known that olfactory search and foraging strategies do not solely rely on olfactory cues, but require additional sensory information from at least visual cues and wind direction. Extending our model to include additional sensory-processing systems for vision and wind direction can provide a comprehensive functional model to study foraging and navigation.

Potential Improvement through Multiple Readout Neurons.

Our current approach only comprises the simplest case of a single readout neuron. This model can be extended to multiple readout neurons. Different readout neurons can be tuned to different odors or groups of odorants. This would allow foraging for different types of food sources and, further, be useful for multimodal sensory integration and learning of valences of multiple odors. Another way to use multiple readout neurons is to create an ensemble learning model. Particularly, one can perform bootstrap aggregation (bagging) to decrease variance of predictions. With this technique, multiple, independent readout neurons can be trained for the same target, and their outputs are averaged to produce a single output. This approach can be useful when the level of noise increases due to different input models used to drive the network. Another possible extension is to use a single readout neuron to code for multiple odors by associating different numbers of action potentials to different odors (e.g., two or three). The choice of model for the readout neuron and the plasticity rule allows us to do this (25).

Top-Down Motor Control and Lateral Horn.

The model currently lacks a neuronal implementation of sensory-evidence integration and generation of motor commands. Integration of sensory evidence is modeled by low-pass filtering of the readout neuron’s spike train, and its derivative is numerically estimated. Drift-diffusion models (67) provide a straightforward alternative to implement integration of evidence to generate a stochastic decision variable that can be thresholded in order to take a behavioral decision for turn or surge (68, 69). Our ultimate goal is a model extension by a neuronal circuit for behavioral decision and top-down motor control. This circuit may involve basic neuronal mechanisms. For example, ref. 70 has shown that a single-compartment Hodgkin–Huxley neuron can operate in two computational regimes. One is more sensitive to input variance, and the model then acts as differentiator, while in the other regime it acts as integrator. Similarly, ref. 71 has shown that the subthreshold current of neurons can encode the integral or derivative of their inputs based on their tuning properties. This and other suggested mechanistic implementations (e.g., ref. 72) could serve as a basis for estimating the low-pass-filtered sensory evidence and its derivative solely by using neural computations. The initiation of a turning behavior based on a time-dependent evidence signal could be implemented, e.g., through disinhibition of motor-command neurons when reaching a decision threshold. The mechanism for U-turning could rely on either cell-intrinsic properties such as SFA, where a neuron initiates a fast turning movement that decays with a fixed time constant, or through state-switching dynamics in neuronal populations.

Relevance for ML and Artificial Intelligence.

Learning and building artificial intelligence (AI) agents capable of interacting with their environment are major objectives in the fields of ML and AI. Deep artificial neural networks (73) have demonstrated great success over the recent years, particularly in the domains of image recognition, natural language processing, and deep reinforcement learning (74, 75). Three challenges currently hamper further progress in the theoretical understanding of deep neural networks. These are expressivity, optimization, and generalizability (76). The latter is of interest in the context of the present results. It refers to a model’s ability to correctly classify previously unseen data samples during testing that stems from the same a priori defined sample distribution. A related, but much harder, problem is classification of unseen out-of-distribution (OOD) samples that are drawn from a different distribution. Several methods have been proposed for mastering generalization or OOD in deep-learning models, including retraining on new tasks, continual learning (77), meta-learning (78), and transfer-learning (79). However, these methods bear new challenges, such as catastrophic forgetting (80), and some progress has been made toward novel solutions (81). We trained our model on exactly two odor stimuli (CS+ and CS−; Fig. 2A). These were drawn from a distribution that is predefined in time (fix stimulus duration) and in neuronal space (receptor activation model; Fig. 4A). After training, the model performed well in the dynamical memory-recall task (Fig. 4 B and D) and in the cast-and-surge task (Fig. 5 D and E). In both test situations, the previously trained model faced novel stimuli of varying receptor-activation profiles and random stimulus durations, both drawn from different distributions, as used during training. Adding to previous studies on generalizability and learning convergence of the multispike tempotron (MST) model in ML-related problems (25, 26), our results support the idea that spiking neural networks can generalize and solve OOD problems in a biological context, which can possibly be carried over to ML-related tasks. We predict that knowledge transfer from research in neural computation to research in ML and AI will become increasingly important.

Materials and Methods

Code and data sets are available through our GitHub profile at: https://github.com/nawrotlab (82).

Spiking Network Model.

All neurons of the olfactory network are modeled as conductance-based leaky integrate-and-fire neurons with SFA. Specifically, the membrane potential follows the dynamical current balance (Eq. 1). On threshold crossing, a hard reset of the membrane potential is performed by Eq. 2. SFA is modeled as outward current by term 4 of Eq. 1. Strength of the adaptation current is modeled by a constant ($b$) decrease on each threshold crossing. Input to the model is modeled as direct, time-dependent current injection of shot noise to all ORNs by the term $I_{stim}\left(t\right)$. All simulations of the network are carried out by using the BRIAN2 (83) simulator. The membrane potential of each neuron within a population is initialized randomly $\in \left[V_{rest},V_{threshold}\right]$. To avoid any artifacts, the network is brought to equilibrium by driving the network for 2 s with background activity only before starting the actual simulation.

$$C_m\frac{\mathrm{d}v}{\mathrm{d}t}=g_l\left(E_l-v\right)$$ [1]

$$+g_e\left(E_e-v\right)-g_i\left(E_i-v\right)\hspace{0.17em}-g_{Ia}\left(E_{Ia}-v\right)\underset{\text{only}\hspace{0.17em}\text{for}\hspace{0.17em}\text{ORNs}}{\underbrace{+\hspace{0.17em}{I}_{stim}\left(t\right)}}v=V_{rest}\hspace{0.17em}\text{on}\hspace{0.17em}\text{threshold}\hspace{0.17em}\text{crossing}$$ [2]

$${\tau }_{Ia}\frac{\mathrm{d}{g}_{Ia}}{\mathrm{d}t}=-g_{Ia}$$ [3]

$$g_{Ia}=g_{Ia}-b\hspace{0.17em}\text{on}\hspace{0.17em}\text{threshold}\hspace{0.17em}\text{crossing}$$ [4]

For this work, the number of neurons within each layer and connectivity schemes are chosen to match the numbers found in the adult D. melanogaster (14, 24). Our model comprises 2,080 explicitly modeled ORNs organized in 52 different receptor types. ORNs of the same receptor type converge onto the same glomerulus (52) by feed-forward excitatory synapses. Each glomerulus is formed by a PN and a LN. LNs provide lateral inhibition to all other PNs and LNs. PNs randomly project to a large population (2,000) of KCs with excitatory synapses, such that each KC on average receives input from six random PNs (connectivity degree parameter $K$). This sparse random convergence implements population-sparse responses. The single, plastic MBON is fully connected to all KCs.

We used the cellular mechanism of SFA to achieve temporal sparseness. ORNs are configured to have slow and weak SFA in accordance with experimental findings (29, 32). For PNs and LNs SFA has been turned off and KCs are set to produce fast and strong adaptation currents (18, 84). The property of temporal sparseness can also be achieved by an alternative implementation through feedback inhibition as proposed by (56, 85).

The synaptic weights of all connections within the network have been manually determined such that an average background firing rate of 8 to 10 Hz is achieved in the LN population.

Stimulus-Response Profile of ORNs.

The stimulus-response profile of ORNs is determined by the ORN tuning curves. We follow a similar method as used in ref. 21, where cyclical tuning over receptor types is modeled as half-period sine waveforms. Our model comprises $N_{type}=52$ receptor types and supports 52 different stimuli (e.g., different odors), where $k_{type}$ refers to the receptor-type index ($\in \left[\mathrm{0,51}\right]$) and $k_{odor}$ to the stimulus index ($\in \left[\mathrm{0,51}\right]$). $N_{orn}=15$ determines the number of receptor types activated by a stimulus. The tuning strength $r$ of the ORNs can be computed as 0.5 cycle of a sine wave with peak amplitude $r_{max}=1$. In the present work, all tuning profiles are normalized to have a peak amplitude of one.

$$x=\frac{k_{type}-k_{odor}\text{mod}{N}_{type}}{N_{orn}+1}$$ [5]

$$r=r_{max}\left\{\begin{array}{cc}sin\left(x\pi \right)\hspace{0.17em}\hfill & \hspace{0.17em}\text{for}0<x<1\hfill \\ 0\hspace{0.17em}\hfill & \hspace{0.17em}\text{else}\hfill \end{array}\right.$$ [6]

Model Input.

Input to the MB model is modeled as time-dependent, direct-current injection into all ORNs. In the absence of any stimuli, ORNs exhibit spontaneous activity (29). The model input thus consists of spontaneous background activity and stimulus-related activity. To generate the background activity, a current time-series is generated for each ORN by simulating shot noise. For each ORN, background activity events are generated from a Poisson process with high rate ($\lambda =300$) (independent Poisson processes are drawn for each individual neuron). Events of the Poisson process are filtered by a low-pass filter with $\tau =0.6$ s. Using this shot-noise model is consistent with experimental findings of odor transduction at the ORNs (29). To induce stimulus-related activity to this time-series of ORN $j$, it is multiplied point-wise with a stimulation protocol time-series $s_j\left(t\right)$, which is rescaled by a constant determined by the tuning strength ($r_j\in \left[\mathrm{0,1}\right]$) to the specific odor of the ORN. This results in a current time-series, where during stimulus, the current magnitude is increased proportional to the ORN’s tuning strength and otherwise remains at the magnitude of the background activity.

We define a stimulation-protocol function $s\left(t\right)$, which is a step function taking on the value one at all time points $t$ where a stimulus or sensory cue is active. For each ORN, a rescaled instance of the stimulation protocol is defined as $s_j\left(t\right)=r_{j}s\left(t\right)$, where the scaling parameter $r_j\in \left[\mathrm{0,1}\right]$ is given by the stimulus response profile (Eq. 6) of the ORN to the specific stimulus.

$$s\left(t\right)=\left\{\begin{array}{cc}1\hspace{0.17em}\hfill & \hspace{0.17em}\text{if}\hspace{0.17em}\text{some}\hspace{0.17em}\text{stimulus}\hspace{0.17em}\text{is}\hspace{0.17em}\text{present}\hfill \\ 0\hspace{0.17em}\hfill & \hspace{0.17em}\text{else}.\hfill \end{array}\right.$$

Sequences of Sensory Cues.

Each sequence has a duration of 10 s. Sequences of sensory cues are generated by drawing the total number of cues within a single sequence from a Poisson distribution with mean $\lambda =8$. Onset times of the cues between 0 and 10 s are drawn from a random uniform distribution, and it is assured that there is no temporal overlap between cues. A stimulus relates to a single sensory cue, and its duration is drawn uniformly between $\left[\mathrm{1,200}\right]$ ms. Finally, each sensory cue is associated with a random odor drawn from a fixed set of possible odors (random sampling with replacement and equal probability). This results in sequences with a random number of sensory cues, random onset, random duration, and randomized odor and distractor combinations.

Model of Sensory Cues within (Gaussian) Plume.

The same procedure is used as above to simulate the experience of sensory cues during a single casting trajectory within a turbulent odor plume. The number of pleasant cues experienced in a casting trajectory is drawn from a Poisson distribution with mean $\lambda =14$. The cue onset times are drawn from a Gaussian distribution with $\mu =5,\sigma =1.5$. The number of distractor cues is drawn from a Poisson distribution with mean $\lambda =5$ and are distributed uniformly in time. Duration of both, pleasant and distractor cues, is drawn uniformly between $\left[\mathrm{100,500}\right]$ ms. In total, 200 different casting trajectories have been generated by using this procedure.

Readout Neuron and Learning Rule.

To fit the readout neuron to the stimuli such that it generates one spike for pleasant odor stimuli (CS+) and zero spikes for any other stimuli (CS−), we used a modified implementation of the MST (25, 26). Thus, the readout neuron is modeled as voltage-based leaky integrate-and-fire neuron with soft reset following the dynamical Eq. 7. Incoming spikes evoke exponentially decaying postsynaptic potentials. When the membrane potential reaches the spiking threshold at some time $t_0$, an output spike is generated, and the membrane potential is reset by the last term of Eq. 7.

$$\begin{array}{ll}\hfill V\left(t\right)& =\underset{≔0}{\underbrace{V_{rest}}}+\sum _{i=1}^N{\omega }_i\sum _{t_i^j<t}\stackrel{\text{exp}.\hspace{0.17em}\text{PSP}\hspace{0.17em}\text{kernel}}{\overbrace{K\left(t-t_i^j\right)}}\hfill \\ \hfill & \hspace{0.17em}-\left(\underset{≔1}{\underbrace{𝜗}}-V_{rest}\right)\sum _{t_{sp}^j}{e}^{-\frac{t-t_{sp}^j}{{\tau }_m}}\hfill \end{array}$$ [7]

The dynamical equation can be decomposed into two parts, the unreset subthreshold potential $V_0\left(t\right)$ (Eq. 8) minus the remaining terms for the soft reset. The neuron was trained to generate one spike for pleasant odor stimuli (CS+) and zero spikes for any other stimuli (CS−). To fit the desired neural code, a training step was performed after each stimulus presentation. A training step was performed only if the number of spikes generated in response to a stimulus was not correct. The training target was given by the difference between number of output spikes the model generated and the number of output spikes associated with the stimulus. We denote the desired critical threshold value, the voltage value that generates $d=1$ spike, as $𝜗*$ and the time point where this voltage value is reached by $t*$ (more generally the critical threshold value to generate $d$ spikes). We briefly sketch the idea and intuition of the MST learning rule. For detailed derivation of the rule, we refer to ref. 25 (section the $𝜗*$ gradient). The MST training algorithm works by differentiating the membrane potential of the critical threshold with respect to the synaptic weights ($\overrightarrow{\omega }$). This can be done since $𝜗*$ is a regular voltage value, that can be expressed by the neuron’s dynamical equation (Eq. 7), with the special identities shown in Eq. 10. This allows us to take the full derivative, as shown in Eq. 11.

$$V_0\left(t\right)=\sum _{i=1}^N{\omega }_i\sum _{t_i^j<t}K\left(t-t_i^j\right)\hspace{0.17em}unresetsub-thresh.potential,$$ [8]

$$V\left(t\right)=V_0\left(t\right)-𝜗\sum _{t_{sp}^j}{e}^{-\frac{t-t_{sp}^j}{{\tau }_m}},$$ [9]

$$𝜗*=V\left(t*\right)=V\left(t_{sp}^j\right)\hspace{0.17em}\text{critical}\hspace{0.17em}\text{thresh}.\hspace{0.17em}\text{that}\hspace{0.17em}\text{makes}\hspace{0.17em}\mathrm{d}\hspace{0.17em}\text{spikes},$$ [10]

$${\nabla }_{\overrightarrow{\omega }}𝜗*=\frac{\partial }{\partial \omega }V\left(t*\right)+\sum _{j=1}^m\frac{\partial }{\partial t_{sp}^j}V\left(t*\right)\frac{\mathrm{d}}{\mathrm{d}\omega }{t}_{sp}^j.$$ [11]

The gradient of the critical threshold with respect to a single synapse $i$ is given by Eq. 12:

$${\left({𝜗}_i*\right)}^{\prime }=\frac{\mathrm{d}}{\mathrm{d}{\omega }_i}𝜗*=\frac{\mathrm{d}}{\mathrm{d}{\omega }_i}V\left(t*\right)=\frac{\mathrm{d}}{\mathrm{d}{\omega }_i}V\left(t_{sp}^j\right),$$ [12]

$${\left({𝜗}_i*\right)}^{\prime }=\frac{\partial }{\partial {\omega }_i}V\left(t*\right)+\sum _{j=1}^m\frac{\partial }{\partial t_{sp}^j}V\left(t*\right)\frac{\mathrm{d}}{\mathrm{d}{\omega }_i}{t}_{sp}^j\hspace{0.17em}\text{recursive}\hspace{0.17em}\text{expr}.\hspace{0.17em}\text{exists}.$$ [13]

Data Availability.

Code and datasets are available through our GitHub profile at https://github.com/nawrotlab. Datasets and source code data have been deposited on GitHub at https://github.com/nawrotlab/SpikingNeuralProgramForagingInsect-PNAS.