An end-to-end model of active electrosensation

SUMMARY

Weakly electric fish localize and identify objects by sensing distortions in a self-generated electric field. Fish can determine the resistance and capacitance of an object, for example, even though the field distortions being sensed are small and highly-dependent on object distance and size. The neural computations underlying these remarkable behavioral capacities are poorly understood. We measured responses of electroreceptor afferents and constructed a filter-based model that accurately accounts for them. We also built models of the electric fields generated by the fish and of the distortions in these fields due to objects of different resistances and capacitances at different locations. Combining these models provides an accurate and efficient method for generating large artificial data sets simulating fish interacting with a wide variety of objects. Using these sets, we trained an artificial neural network (ANN), representing brain areas downstream of the electroreceptors, to extract the 3D location, size, and electrical properties of objects. Model performance is comparable to that of real fish in experimentally tested behavioral tasks. Performance is maximized if the ANN operates in two stages: first estimating object distance and size and then using this information to extract electrical properties. These results highlight the potential of end-to-end modeling for studies of electrosensation and suggest a specific form of modularity in electrosensory processing that can be tested experimentally.

eTOC:

Turcu et al. develop an end-to-end framework for studying active electrosensation combining models of electrical field generation, peripheral electroreceptor encoding, and central neuronal processing. The model matches behavioral performance of real fish and suggests a modular architecture for object localization and discrimination.

INTRODUCTION

Weakly electric fish sense their environment by emitting electrical fields known as electric organ discharges (EODs)¹. The EOD consists of a basal component, which is the field that would exist in empty water, plus a component induced by nearby objects, called the electric image, which appears as a distortion in the basal EOD. Nearby objects with electrical resistances higher than the surrounding water (e.g. rocks) result in less EOD-induced current flow near the object, producing a local decrease in the amplitude of the EOD. Living objects (e.g. small invertebrates that are prey for the fish), on the other hand, have lower electrical resistances than water and hence increase field amplitude. Living objects also have sizable electrical capacitances, which alters the temporal waveform of the EOD. Behavioral experiments have shown that weakly electric fish are capable of spatially localizing objects and discriminating their properties (including size, shape, and electrical resistance and capacitance) in the dark, based solely on information extracted from their EODs^2–4. The unique ability of electric fish to discriminate electrical properties is likely to be of special importance for foraging, for example by aiding the fish in finding preferred prey⁵. These behavioral capacities are remarkable given that the object-induced perturbations of the EOD that support electrosensation are typically small relative to the basal EOD (on the order of 1%) and are highly sensitive to distance (decreasing as 1/distance⁴) and to object size (increasing as radius³)^6–10. Although much is known about specific aspects of the electrosensory and electromotor systems of fish^11–13, how the various components of this system work together to support object localization and discrimination is poorly understood.

End-to-end modeling using artificial neural networks to map sensory input onto neural activity or behavior provides a powerful approach for studying neural systems (e.g.^14–16). Such models can often be trained to perform complex tasks similar to those done by animals. The solutions found by such models provide well-defined hypotheses that can be tested using recordings and manipulations of neuronal activity. The goal of the present work was to develop an end-to-end model of active electrosensation. We first constructed a model of electroreceptor responses based on neural recordings and an efficient model of the fish’s electrical field. Together, these allowed us to simulate electrosensory input due to objects with varying locations, sizes, and electrical properties. We then used these large simulated datasets to train a variety of ANN architectures to solve a challenging benchmark task: the simultaneous localization and identification of object size and electrical properties. Performance could be enhanced by solving this task in two stages: first estimating object distance and size and then using this information to extract electrical properties. Finally, we demonstrated that the model performance is similar to that of real fish on tasks designed to match those used in prior behavioral experiments^3,5. In addition to suggesting specific directions for future experiments on active electrolocation, these results provide a versatile modeling framework that may be broadly useful for studies of electrosensory and electromotor systems.

RESULTS

Measuring and modeling electroreceptor responses

For characterizing electroreceptor responses, we focused on the commonly studied African pulse-type electric fish Gnathonemus petersii, commonly known as the elephant-nose fish. The EODs of Gnathonemus are ~ 1 V in amplitude and ~ 300 μs in duration and, are sensed by approximately 1, 000 electroreceptor organs distributed across the fish’s body surface¹⁷. Each electroreceptor organ contains two types of receptor cells, known as A- and B-cells, that encode different features of the EOD and project to separate regions of the electrosensory lobe (ELL), the first stage of electrosensory processing in the fish’s brain^18–21. Anatomical tracing has shown that projections from these two regions, the medial zone (MZ) for A-cells and the dorsolateral zone (DLZ) for B-cells, converge in the midbrain and are subsequently processed within an interconnected network of brain regions including the optic tectum, thalamus, telencephalon, and cerebellum^22–24.

Our first goal was to develop a model of the sensory information transmitted by A- and B-type electroreceptors (see^25,26 for a biophysical model). Prior electrophysiological recordings have shown that A-type receptors are primarily sensitive to changes in EOD amplitude, whereas B-type receptors respond to both amplitude and waveform changes, but a precise description of the stimulus features encoded by A- and B-type receptors is lacking^18,20,21,27. To address this, we recorded responses in both the MZ and DLZ to a large set of simulated EODs designed to mimic objects with different resistances and capacitances (Figure 1A). In these experiments, fish were paralyzed, blocking the action of the electric organ, so both the basal EOD and the distortions in it were generated artificially, triggered by electrophysiological measurement of EOD motor command signals. Delivered fields were recorded to verify that they matched the desired waveforms (Figure S1). These stimuli generated prominent field potentials which we recorded with microelectrodes positioned at matched somatotopic locations in the MZ and DLZ. Based on previous results, we used the amplitude of the first negative peak of the LFP (henceforth called the LFP amplitude) as a proxy for the activity of individual electroreceptor afferent nerve fibers^28–30. Distorting the basal EOD evoked large and reliable changes in LFP amplitude in both zones (Figure 1B). We report sensory responses to distorted EODs as percentage modulations from the response to the basal EOD.

Figure 1. Responses to resistive and capacitive stimuli in the medial and dorsolateral zones of the ELL.

A Examples of delivered EODs. The basal EOD (green) is plotted behind individual examples (gray) that include distorted EODs with different amplitudes and waveform shapes, simulating the effects of objects with different electrical properties. Methods - “Electrosensory stimulation” describes how these stimuli were generated. B Example LFP responses to delivered stimuli from a single fish. The color for each trace reflects the PP amplitude modulation for MZ (left) and the P/N ratio modulation for DLZ (right). Dashed black line marks the timing of the EOD. Traces for each of the 56 different distorted stimuli delivered in this example experiment are shown. Red arrows indicate the most prominent negative peaks in the LFP traces used to quantify the responses. C Summary of LFP responses for all stimuli in the PP and P/N feature space, color-coded by the MZ response (left) and the DLZ response (right). Data from a single fish from an experiment in which 240 different distorted stimuli were delivered. Arrows indicate the directions of the gradients of MZ and DLZ responses in the feature space. See also Figure S1.

Previous studies^5,31 have characterized distortions of the EOD due to resistive and capacitive objects in terms of changes in the peak-to-peak amplitude (PP = P + N) and the positive-to-negative peak ratio (P/N) of the EOD waveform (Figure 1A). Plotting LFP amplitude as a function of the PP and P/N values of the corresponding stimuli, we found that responses in the MZ depend primarily on the PP value of the stimulus (Figure 1C, left), while those in the DLZ depend more on P/N (Figure 1C, right). However, the gradients of the measured responses are not truly aligned (see Methods - “Alignment of LFP responses with features” for alignment details) with either of these two features (arrows in Figure 1C).

To provide a better description of the measured LFP responses, we constructed a model based on convolving the distorted EOD waveforms with two filters, one for the MZ and another for the DLZ (Figure 2A). Because we characterize electrosensory responses by the amplitude at a single time point (the magnitude of the negative peak in the LFP, see Figure 1B), the convolution took the form of a projection, i.e. a product of the stimulus waveform and the filter, integrated over time. We also added an offset parameter to this sum. We determined the filter shapes and offsets by minimizing the squared difference between the prediction of the filter model and the data across EOD stimuli. Artificial stimulus noise was included as a regularizer in this minimization, with the appropriate noise value chosen by cross-validation (Figure S2A). The resulting model explains 92% of the variance across single trial responses (Figure 2C) and provides interpretable convolutional filters that are robust across experiments (Figure 2B). The filter shapes suggest that A-type receptors weigh and sum the three peaks of the EOD waveform (Figure 2B, left), while the B-type receptors are sensitive to temporal features of the EOD waveform, including the slope and timing of the zero-crossing (Figure 2B, right).

Figure 2. Convolutional filter model of MZ and DLZ responses.

A Schematic of the model. Raw EOD waveforms are convolved with two filters to produce the predicted response. Fitting minimizes the mean squared error between the filter responses and the LFP modulations. B Filters obtained from fitting the data for both filter types. Base EOD (black), average filter (solid color), and individual experiments filters (light color) are shown. Average filters are extracted from all experimental data across fish, not by averaging the individual filters. C Performance of the filter model in predicting the single-trial LFP response. True LFP modulation is the percent change between the LFP amplitude in response to distorted stimulus and to the basal EOD. D Summary of LFP responses for all stimuli in the filter feature space, color-coded by the MZ response (left) and the DLZ response (right). Data from a single fish, same experiment shown in Figure 1C. Arrows indicate the direction of gradients of the responses in the feature space. E Summary across experiments of the alignment of the LFP responses with the PP & P/N features (left) and the filter features (right) (n = 13, MZ; n = 12, DLZ). See also Figure S2.

As in Figure 1C, we plotted the experimental responses as a function of stimulus features, only now using the projections of the stimuli onto the two filters as our axes (Figure 2D). The MZ and DLZ responses are better aligned (Figure 2E) with the features extracted from our model (arrows in Figure 2D) than with the PP and P/N feature space (arrows in Figure 1C). We also compared our filters with the results of a principal component analysis (PCA) on the set of experimentally delivered stimuli. Two principal components (PCs) explain most of the variance across stimuli (Figure S2B), matching the number of sensory cell types. Moreover, the first two PCs resemble the filters extracted by our model (Figure S2C, compare to Figure 2B), with the main difference being that the PCs are required to be orthogonal by definition.

From objects to EODs

To study the neural computations underlying realistic electrosensory tasks, we need to expand beyond the experimental data to compute responses from the entire electroreceptor array for objects that vary in their electrical properties, size, and location. Detailed numerical models have been developed to compute the spatial patterns of object-induced modulations of the basal EOD^32,33. While these models have proven extremely useful for studies of electrosensory systems^32,34–38, they have two significant drawbacks for our purposes. First, they are static methods, meaning that they do not simulate objects with capacitive properties. Second, they are computationally intensive^39,40, making them poorly suited for generating the large amounts of simulated sensory input required for training ANN models. An alternative is to use approximate analytic models^6,7 or electric circuit models³¹, but these do not provide the full flexibility needed for our purposes. We therefore developed a field model framework (see Data S1) that can quickly and flexibly generate realistic EOD patterns. Our framework captures the spatial geometry of the fish and objects (Figure 3A), using the field model fitted to data in⁸. It captures the EOD distortions due to both resistive and capacitive properties of objects (Figure 3B) by solving the dipole distortion problem^6,7 in a computationally efficient way tailored to this system (Data S1). It can also reproduce the spatial pattern of an object’s electrical image on the body of the fish (Figure 3C). This framework can simulate many fish-object conditions at approximately 50 times real-time on a personal computer.

Figure 3. An efficient electric field model for simulating effects of resistive and capacitive objects.

A 3D visualization of the fish near a spherical object in the aquarium. An electric image induced by the object is shown on a fish covered in model electroreceptor organs. Equipotential surfaces around the fish during the EOD capture the funneling effect due to the shape of the fish. B Example distortions of the EOD due to objects with different electrical properties. The basal EOD is shown as the dashed-black curve. Purely resistive objects distort the amplitude of the EOD, either increasing (distortion due to metal object with small resistance in green) or decreasing (distortion due to rock object with large resistance in orange) the amplitude. Living objects with low resistance and capacitive properties distort both the amplitude and the waveform of the basal EOD (blue). Note that the live object induces a second, small positive peak after the prominent negative trough, that is not induced by either purely resistive objects. All three objects simulated were spheres of diameter 15 mm with a distance of 5 mm from the skin to the nearest point on the object. C Close-up of the electric image on the skin of the fish visible in A. Modulation is shown as percentage of the basal signal. Individual simulated receptors are visible as green circles on the skin of the simulated fish. See also Data S1.

Characterization of object electrical properties

While it has been assumed that the electrosensory system derives the resistive and capacitive properties of objects by combining input from A- and B-type receptors, little is known about where and how this occurs in the mormyrid brain^22,23. We used a modeling approach to gain initial insights into the computational challenges associated with characterizing object electrical properties. We began by training a small, feedforward ANN (Methods - “Small ANN for electric properties only” & Figure S3A) to extract resistive and capacitive properties of a 2.5 cm diameter spherical object with a fixed distance of 1 cm from the skin of fish to the closest point on the sphere. We simulated the input data using our field model and our electroreceptor model, and chose the spatial values to match typical experimental conditions^2,41. The input to the ANN was restricted to a single electroreceptor organ on the skin containing both A- and B-type receptor cells, chosen as the most modulated receptor under the geometric projection of the object onto the skin (Figure 3).

We presented the ANN with A- and B-cell inputs in response to field distortions due to a range of resistances and capacitances that would likely be encountered by a fish in its natural environment and in experiments (Figure 4A). In particular, we chose the range and the logarithmic spacing of the resistances and capacitances we simulated based on previous experiments^2,3,5. The network successfully extracts these electrical properties with good accuracy on validation data, especially for capacitance (Figure 4B,C). The ANN can also extract resistance and capacitance on held-out individual trials, when provided with the experimental LFP waveforms recorded in the MZ and DLZ as input (prediction error on held-out data was below 5% from true capacitance or resistance values).

Figure 4. Extracting electric properties for objects of fixed location and size.

A Objects with different resistances and capacitances occupy different regions of the feature space defined by the MZ and DLZ filters for a single receptor. This feature space corresponds to the feature space of the most modulated receptor feeding as input to the ANN. Lines of constant resistance (blue palette) and constant capacitance (orange palette) are shown. The origin in modulation space corresponds to no object present. Resistance and capacitance are reported as the base 10 log of these quantities in Ω or F. B Performance of ANN extracting the resistance of different objects with fixed spatial properties. C Equivalent to B but for capacitance. See also Figure S3.

Object localization and characterization

Challenges due to distance and size effects

The ANN model described in the previous section extracted the capacitance and resistance of an object of fixed size and at a fixed distance. Determining electric properties under natural conditions is more challenging, since object size and distance will also vary simultaneously. The difficulty of this problem can be illustrated by plotting MZ–DLZ feature maps (Figure 4A) as a function of the distance to and size of the object generating the EOD distortion (Figure 5A,B). To obtain these results, we simulated EOD distortions due to objects of varying resistances and capacitances with different distances between the skin of fish and the surface of the object (Figure 5A) and for objects of varying size (Figure 5B). We chose distance and size values that are typically encountered in experiments^41–45. Fish–object distance and object size have a large effect on EOD distortions. As shown in Data S1 Equation 7, the feature space scaling with inverse distance follows a polynomial of degree 4 for objects within a body length of the fish, and the feature space scaling with object radius follows a polynomial of degree 3. These scalings have a dramatic effect on the performance of the models we now consider.

Figure 5. Feature space for active electrosensation.

A Multiple feature spaces defined by modulations of the MZ and DLZ filters due to objects with varying resistance and capacitance, with each horizontal plane (different colors) corresponding to a different distance from the fish. Object size and lateral location are fixed. The feature space shrinks by a degree-4 polynomial with inverse distance. B Similar to A, except each horizontal plane corresponds to a different object radius, with the distance and lateral location fixed. The feature space increases by a degree-3 polynomial with radius. C Amplitude and waveform modulations of the stimulus for objects of fixed size, but of different electrical properties and distances from the fish. Individual points are colored by the distance to the fish and the different lines represent distinct combinations of resistance and capacitance. Points corresponding to fixed electric properties but different distances define electric color lines. The resistance of objects varied between 0.25 and 50 kΩ, and the capacitance varied between 0.25 and 50 nF. In the PP and PN feature space, the electric color lines are not straight — average R² for all 20 shown lines is 0.93 with standard deviation 0.16. D Similar to C, the same objects were simulated, but in the filter feature space. The electric color line is perfectly straight in this space — average R² for all 20 shown lines is 1.00 with standard deviation < 10⁻¹¹. See also Data S1.

Previous work defined an “electric color line” to capture distance effects^5,46. These lines, plotted in the PP and P/N feature space (as in Figure 1C), are traced out by the EOD modulations produced by an object with fixed electrical properties and fixed size located at different distances from the fish. It was noted that points corresponding to specific objects at different distances lay on approximately straight lines. Along one of these “electric color” lines, an object can be perceived as having the same identity independent of how far away it is from the fish, in analogy to visual colors with a range of physical characteristics appearing similar. Our electric field model replicates this result, showing that the ratio between the qualitative PP & P/N features fall roughly on a line for the same object when the distance to the fish is varied, but this “line” has some curvature (Figure 5C). This is because the features PP & P/N are not defined by linear operations on the stimulus. Our electroreceptor model is linear, and the electric color line in the filter feature space (as opposed to PP & P/N) is truly linear (Figure 5D). See Methods - “Electric colors” for details on how the electric color line is maintained despite variations in object size.

End-to-end ANN models

Our next goal was to construct an end-to-end model that could solve the challenging benchmark task of extracting both spatial and electric object properties, using the physics model to generate electrosensory data and the electroreceptor model to provide the sensory input. Prior work indicated that electrical properties are best encoded in the responses of the most modulated receptors, e.g. those close to the peak response across the skin surface (see electric image on skin in Figure 3A). On the other hand, spatial properties are encoded in spatial features of the electric image across the fish’s skin, such as its 2D location and overall width and height^41,47,48. Thus, for the end-to-end ANNs, we modeled receptors across the full surface of the fish, not a single receptor as in the previous section (see Methods - “CNN model” for data and model details). To accommodate this array of detectors, we used a more sophisticated architecture, a convolutional neural network (CNN), to extract object properties (Figure S3B). Like the active electrosensory system of the fish, the spatial convolutional filters of the CNN integrate information across the skin array and process this information in sequential stages of convolutional and feedforward layers. We tested performance of CNNs with varying numbers of layers and parameters to cover the model space from underparameterized to overparameterized and ensure robustness of results. Explicitly mapping the layers of our model onto processing stages in the brain is not currently possible due to lack of neural data. However, this has been done with notable success in other sensory systems and represents an important goal for future studies^14–16.

We trained end-to-end models to extract the properties (lateral position, vertical position, distance, radius, resistance, and capacitance) of simulated objects from sensory input. The models were trained to minimize the mean square error (MSE) between the simulated and model-predicted object properties. The MSE of each of the six object properties was added independently to the loss function. We normalized the values for each object property based on the whole range of simulated values such that the MSE for each property is weighted equally in the loss function.

We first trained CNNs to extract only the electrical properties of simulated objects from sensory input, with no specific information about the spatial variables provided during training, i.e. we excluded the spatial properties from the loss function, although spatial properties still affect the sensory input. The resulting networks perform poorly on extracting electrical properties (Figure 6A) compared to the previous results when distance and size were held constant (Figure 4B,C). This is not surprising given the high degree of sensitivity of the feature space to distance and size (Figure 5A,B). Interestingly, properties of the feature space also account for the structure of the errors in these networks. For resistance, the models distinguish between large values and small values, effectively performing a binary classification. This can be attributed to the fact that scaling of the feature space, when distance and size vary, preserves angles but not magnitudes. This allows models to use linear decision boundaries that pass through the origin of the feature space (e.g. Figure 4A) for classification. Based on the lines of equal resistance, splitting the predictions into large and small values results in better than chance performance. For capacitance, the same principle applies, but the equal-capacitance lines are not as favorable for linear decision boundaries passing through the origin, so performance is worse than for resistance.

Figure 6. Object localization and characterization by CNN models.

All results reported here are based on cross-validated trials that were not part of the training dataset. A Performance of spatial-unaware models on extracting the resistance (top) and capacitance (bottom) from the sensory input. This performance shows a binary classification bias of the models extracting the resistance, due to the fact that the scaling rule preserves angles but not magnitudes in the feature space. This angle-preserving effect impairs capacitance performance, explained in part by the orientation of the equi-capacitance lines in Figure 4A. Example performance for one model is shown. The average (± standard deviation) R² for each property across n = 10 models was: resistance – 0.621 ± 0.008, capacitance – 0.353 ± 0.019. B Full end-to-end CNN model performance on spatial and electrical properties of simulated objects. Example performance for one model is shown. The average (± standard deviation) R² for each property across n = 10 models was: lateral position – 0.938 ± 0.017, vertical position – 0.925 ± 0.017, distance – 0.919±0.027, radius – 0.825±0.019, resistance – 0.785±0.021, capacitance – 0.542±0.018. C Performance of a hybrid CNN-ANN model with internal spatial estimates from the CNN fed to an ANN that has learned the scaling rule of the feature space and extracts the resistance (left) and capacitance (right) from the sensory input. The average (± standard deviation) R² for each property across n = 10 models was: resistance – 0.779 ± 0.017, capacitance – 0.597 ± 0.031. D Performance of the trained ANN component of the hybrid model when it receives the true spatial values on extracting the resistance (left) and capacitance (right) from the sensory input. The average (± standard deviation) R² for each property across n = 10 models was: resistance – 0.857 ± 0.019, capacitance – 0.797 ± 0.09. See also Figure S3.

We next examined if this problem could be solved by training the end-to-end CNN models on the full task, including both spatial and electrical information during training in the loss function. The resulting models perform well on spatial localization over the extended range of simulated objects chosen to match typical experiments (Figure 6B left & center). However, they show similar structured errors for the electrical properties (Figure 6B right) as the electrical-only models, albeit with somewhat improved performance. We reasoned that, even when a CNN is able to extract spatial properties, it may not be able to fully use this information to solve the problems raised by the severe scaling of the feature space, which makes extracting electrical properties hard. In addition, given the steep dependence of the scaling, the CNN’s estimates of spatial properties may not be accurate enough to provide sufficient robustness. In fact, both of these effects contribute to network performance.

To address the problem of using spatial information to inform electrical feature extraction, we combined a CNN trained to determine the spatial features of an object with the small ANN that we used previously to extract purely electrical properties (see Results - “Characterization of object electrical properties”), with one new wrinkle. To combine the CNN with the ANN, we allowed the model to make use of appropriate multiplicative rules to scale its electrical feature space on the basis of object distance and radius values, with the parameters controlling these rules determined by learning (see Methods - “Hybrid model”). The CNN fed its extracted distance and radius values into the scaling rule, which multiplied the feature space accordingly to feed into the downstream ANN that extracted electrical properties (Figure S3C). This hybrid model slightly improved performance for capacitance extraction, but did not improve resistance performance (Figure 6C). Nevertheless, the models learned the scaling rules successfully and, importantly, accurately extracted the electrical properties of simulated objects with widely varying locations and sizes when the true values of distance and size were fed into it to drive the scaling (Figure 6D). Potential biological implementations of such a two-stage solution are discussed further below (see Discussion).

Behavioral performance of the models

Finally, we sought to compare the performance of our models directly with those of real fish. Behavioral data directly comparable to the benchmark task described above are not available. As an alternative, we tested each of our models (full end-to-end, hybrid with estimates, and hybrid with values) on additional tasks designed to be as similar as possible to published behavioral experiments on Gnathonemus ^3,5. See Methods - “ANNs performance on realistic behaviors” for details on how we implemented these tasks, how we evaluated the models, and how we estimated the electrosensory input noise to match experimental results.

Gnathonemus has been shown to discriminate between preferred prey and various non-preferred items, based on their electrical properties, in a foraging scenario in which the discriminated items varied in size and distance⁵. We simulated five different object types corresponding to those used in the experiment: preferred prey, non-preferred prey, water plants, plastic, and metal (Figure 7A), each having two different sizes (1.8 or 3 mm diameter) and placed at various distances from the skin (1, 2, 3, 4, or 5 mm). We passed the sensory input from these objects through the three model types and used the outputs of the models as input to simple classifiers (Random Forest Classifier, or RFC) to decide whether the fish would intake or reject an object. We trained the classifiers for objects placed at 1, 3, and 5 mm away and tested them on objects placed at 2 and 4 mm away (Figure 7B). Our simplest model, the full end-to-end model, performs this task relatively well, showing high intake percentage for the preferred prey object and low intake percentages for all the other objects, regardless of object size and distance from the skin (Figure 7B, left). The hybrid model that uses the internally estimated spatial values reaches a higher intake percentages for the preferred object and lower intake percentages for the other objects (Figure 7B, center), and the hybrid model that uses the exact spatial values performs even better (Figure 7B, right).

Figure 7. Comparison of model performance with behavior.

A Five different object types with various radii and at various distances were simulated, shown here in the filter feature space of the most modulated electroreceptor. Scatter circles are the objects, and dotted lines are the corresponding electric-color line for each object type. The electric properties of for each object type in these simulations were: metal – 10 kΩ; plastic – 60 kΩ; preferred prey – 10 kΩ, 1.5 nF; non-preferred prey – 10 kΩ, 7 nF; water plant – 50 kΩ, 1.5 nF. For the object size simulated, the displaced water resistance would be approximately 35 kΩ. These values were chosen to match the electric color lines of real objects from⁵. B Intake percentages of each object type for each size and tested distance, by model type. Colors correspond to object types in A. Size identifiers refers to the small 1.8 mm (1) and large 3 mm (2) objects. Distance identifiers represent the distance in mm. Error bars are across 10 different models, each with an associated RFC, and tested on 100 trials each. This panel is designed after⁵, and the light blue dashed lines indicate approximate average performance of fish from that experiment under similar conditions. C Example psychometric discrimination curves for the preferred object (S+ = 10 nF, marked by gray line, ) and the non-preferred object of capacitance value marked by the x-axis, by model type. This panel and the following are designed after³, and the light blue lines indicate approximate performance of fish from that experiment under similar conditions. D Just noticeable difference in capacitance as a function of the S+ capacitance value for each model type, when the non-preferred stimulus had smaller/larger (left/right) capacitance than S+. Error contours are across 10 different models, each with an associated RFC, and tested on 100 trials each. See also Figures S3 and S4.

Another previous experiment quantified how well Gnathonemus can distinguish between objects of different capacitances, when all other object properties were kept constant³. In the experiment, fish had to choose between two objects with different capacitances and were rewarded for choosing to swim towards the pre-defined preferred object (S+)³. We replicated this experiment using our models, similarly to above, by using the output of our models as input to simple classifiers. We simulated objects with similar spatial properties as in the experiment (2.5 cm distance between skin and object, 1.1 cm diameter) and with nearly 0 resistance. We varied their capacitance within the whole range possible in this small resistance regime such that our numerical simulations remained stable. We found U-shaped discrimination curves, similar to the psychometric functions observed experimentally for all three type of models, albeit with narrower width for the hybrid models (Figure 7C). Additionally, we computed the just noticeable difference (JND, defined in³) for each model type for various S+ capacitance values (Figure 7D). The mid-range of capacitance values in our simulations corresponds to typical live objects that fish forage (Figure 7A). For extreme capacitive values, objects induce very similar electroreceptor modulation values, making it more difficult to differentiate between objects. We observe that, in the mid-range, the hybrid models have lower JNDs than the full end-to-end model, and that better spatial estimates helps reduce the JND even more. Overall, these results show that our models perform comparably with real fish on realistic electrosensory tasks involving object localization and discrimination.

DISCUSSION

Summary

We developed an end-to-end model of active electrosensation spanning from EOD emission to object localization and discrimination. The model consists of an efficient and flexible framework for simulating realistic electrosensory input patterns that is used to train ANN models (proxies for the CNS of the fish). Using this approach, we successfully trained models to solve ecologically relevant electrosensory tasks similar to those performed by weakly electric fish. Our results suggest that the steep scaling of input features with object size and distance poses a computational challenge for electrosensory systems that could potentially be solved by a modular, two-stage architecture in which estimates of object distance and size are fed as input to a second processing stage that computes object electrical properties.

Computational challenges for active electrosensation

Electrical images are known to depend strongly on object distance and size (see^6–10 and Figure 5A,B). This steep scaling of the input feature space with distance and size made it difficult for CNN models to extract object electrical properties when their size and distance also varied (Figure 6A,B). We found that performance on such tasks could be substantially improved by training a hybrid ANN model that first estimates spatial properties and then uses these estimates to multiplicatively scale the feature space (Figure 6C,D). Successful foraging requires that fish discriminate electrical properties under conditions in which object size and distance vary⁵. However, to our knowledge, there have been no systematic tests of how behavioral discrimination of electrical properties depends on object size or distance. While modular processing and multiplicative scaling are common features of neural systems^49–53, it is currently unknown whether such strategies are implemented in electrosensory systems. While the first central stage of electrosensory processing in the hindbrain electrosensory lobe (ELL) has been extensively studied in Gnathonemus,^29,54,55, virtually nothing is known about higher stages of electrosensory processing, where representations of object size, distance, and electrical properties are presumably formed. Anatomical tracing suggests that information from the MZ and DLZ is fused into a single somatotopic map in a midbrain structure known as the torus semicircularis^22–24. Studies of the jamming avoidance response in the South American wave-type electric fish (a behavior that allows fish to avoid electric interference from conspecific EODs), recorded from individual midbrain neurons that combine input from amplitude- and phase-coding pathways (similar to the A- and B-type receptor pathways discussed here)⁵⁶. More recent studies of wave-type fish have also documented considerable diversity in the coding of electrocommunication signals in midbrain^57,58. However, little is known about the specific roles of these neuronal populations in object processing⁵⁹. Mormyrid fish have undergone a dramatic encephalization related at least in part to electrolosensation⁶⁰. Midbrain neurons project to higher electrosensory processing stages in the optic tectum, cerebellum, and thalamus and also send feedback projections to the ELL via the preeminential nucleus (PE)^22,61. This latter pathway has been shown to adaptively shape ELL responses to looming and receding objects in South American weakly electric fish⁶². Large portions of the pallium of mormyrid fish also respond to electrosensory stimuli⁶³. A goal for future studies is to combine the end-to-end modeling approach developed here with multi-area electrophysiological recordings to characterize where and how representations of object electrical properties, size, and distance are formed. The fact that input to the electrosensory system in pulse-type electric fish is strictly locked to the time of the EOD can potentially be leveraged to test the hypothesis from the present study that spatial representations should emerge prior to accurate representation of electrical properties in neuronal population responses (since in our models the former are used to refine the latter).

Limitations of the model

There are, of course, many differences between real fish and our models that complicate direct comparisons, even on tasks that are nominally similar. For example, fish utilize multiple senses to forage⁶⁴ and actively shape their motor behavior to optimize the acquisition of electrosensory information^37,45. Perhaps most pertinent to the present study, Gnathonemus emit EODs at rates up to 80 Hz during foraging or exploration of novel objects and often take several seconds to inspect objects before successfully discriminating between them in behavioral tasks^{43,45,65–68}. In contrast, our models extracted information from an electric image based on single EODs. Providing multiple consecutive EODs as input to our models and allowing them to use feedback from previous EODs to maintain internal representations over time using recurrent architectures would likely improve their performance on the tasks we have considered and is an important goal for future work. Specifically, integrating over multiple EODs (possibly in conjunction with tail and body movements) could provide the accurate estimates of object size and distance required by our models to extract electrical properties.

Additional applications for studies of electrosensory systems

Studies of electrosensory and electromotor systems have been used to address a wide range of problems in biology, from motor pattern generation and receptor biophysics to social behavior⁶⁹. The end-to-end modeling approaches developed here can easily be adapted for comparative studies of active electrolocation in other species and can also be combined with electrophysiological recordings in freely behaving fish (e.g.^54,70). Furthermore, the simulation framework developed here can be extended to study other aspects of electrosensory systems. The accuracy and relative ease with which electrosensory input patterns can be modeled makes this an appealing direction. For example, combining our electrical field model with EOD recordings and multi-animal pose tracking based on video⁷¹ will enable high-throughput, fully-automated approaches for accurately assigning EODs to individuals⁷² and modeling spatial and temporal patterns of electrosensory input in the context of freely interacting social groups of electric fish. These rich datasets are expected to be useful for future studies of both active and passive electrosensation⁷³, electrocommunication^74,75, and collective sensing based on EODs³⁸.

STAR METHODS

RESOURCE AVAILABILITY

Lead contact

Requests for further information and resources should be directed to and will be fulfilled by the lead contact, Denis Turcu (denis.turcu.11@gmail.com).

Materials availability

This study did not generate new materials.

Data and code availability

• All experimental data used for modelling in this paper is available at github.com/DenisTurcu/efi receptors-model as of the date of publication.

• All original code has been deposited for each component of this work. The electroreceptors model code is available at github.com/DenisTurcu/efish-receptors-model, the field model framework is available at github.com/DenisTurcu/efish-physics-model, and code for the localization and characterization models is available at github.com/DenisTurcu/efish-characterization.

• Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.

Experimental model and study participant details

Male and female wild-caught Mormyrid fish of the species Gnathonemus petersii were used in these experiments (fish were 7–12 cm in length, of unknown age, and sex was not specifically selected for). Fish were housed in 60 gallon tanks in groups of 5–20. Water conductivity was maintained between 70–150 μS both in the fish’s home tanks and during experiments. All experiments performed in this study adhere to the American Physiological Society’s Guiding Principles in the Care and Use of Animals and were approved by the Institutional Animal Care and Use Committee of Columbia University.

Method details

Surgical procedures

For surgery to expose the brain for recording, fish were anesthetized (MS:222, 1:25,000) and held against a foam pad. Skin on the dorsal surface of the head was removed and a long-lasting local anesthetic (0.75% Bupivacaine) was applied to the wound margins. A plastic rod was cemented to the anterior portion of the skull to secure the head. The posterior portion of the skull overlying the ELL was removed. Gallamine triethiodide (Flaxedil) was given at the end of the surgery (~ 20 μg/cm of body length) and the anesthetic was removed. Aerated water was passed over the fish’s gills for respiration. Paralysis blocks the effect of electromotoneurons on the electric organ, preventing the EOD, but the motor command signal that would normally elicit an EOD continues to be emitted at an average rate of 2 to 5 Hz.

Electrophysiology

The EOD motor command signal was recorded with a Ag-AgCl electrode placed over the electric organ. The command signal is the synchronized volley of electromotoneurons that would normally elicit an EOD in the absence of neuromuscular blockade. The command signal lasts about 3 ms and consists of a small negative wave followed by three larger biphasic waves. Onset of EOD command was defined as the negative peak of the first large biphasic wave in the command signal. Recordings of local field potentials were made with low resistance (< 5 MΩ) glass microelectrodes filled with 2M NaCl. Signals were recorded and filtered at 3–10 kHz (Axoclamp 2B amplifier, Axon Instruments) and digitized at 20–40 kHz (CED power1401 hardware and Spike2 software; Cambridge Electronics Design, Cambridge, UK). For most experiments, recordings were made simultaneously from the MZ and DLZ by placing two electrodes in somatotopically matching locations in the granular layers of the two zones. Somatotopic location and depth within the ELL was judged based on LFP responses to the EOD motor command and to local electrosensory stimuli delivered by a hand-held dipole electrode that could be positioned over various regions of the skin.

Electrosensory stimulation

Simulated EODs designed to mimic objects with different resistances and capacitances were delivered using a stimulus isolation unit (A-M systems, model 4100) in constant current mode connected to a pair of carbon rods (2 mm diameter, 4 cm length) placed lengthwise (~ 1 cm distance from the skin) on either side of the head of the fish (Figure S1). Current amplitude was adjusted for each fish such that the baseline stimulus evoked LFPs of ~ 70% maximal amplitude. A baseline stimulus consisting of an EOD waveform measured from a discharging fish with no object present was delivered for ~ 30 minutes before delivering the set of perturbed stimuli. All stimuli were triggered at a brief delay (4.5 ms) following the fish’s spontaneously emitted EOD motor commands.

Three sets of simulated EODs, containing 240, 99 and 56 stimuli, were designed to approximately cover a small rectangular grid in the PP and P/N modulation space. One set of stimuli was used for each fish. These stimuli were designed based on previous electric circuit models of weakly electric fish EODs^31,76, before our field model was fully developed. We varied the resistances (1 – 100 kΩ) and capacitances (0.1 – 10 nF) and filtered stimuli such that we obtain an approximately rectangular grid in the PP and P/N modulation space, e.g. grids formed by scatter points in Figure 1C. Stimuli were delivered in random order in 1–2 bouts containing 10–15 repetitions of a distorted EOD separated by 5–10 repetitions of the baseline stimulus.

Alignment of LFP responses with features

The alignment of electroreceptor organ LFP responses with the feature directions, summarized in Figure 2E and indicated by arrows in Figure 1C and Figure 2D, was computed as the average direction of the slope of the responses in the feature space. For each feature in a feature pair, we computed the slope of the LFP response with respect to that feature’s modulation, while maintaining the other feature approximately constant by binning its modulation values. We used 6 bins for the results reported in Figure 2E, but results were similar when using other number of bins from 4 to 8. Using more bins resulted in too few samples per bin, and using fewer bins broke the assumption that the other feature was approximately constant. We computed the gradient direction of the alignment as the average of the slopes across the bins.

Electric colors

Figure 5D shows that the ratio between the MZ and DLZ filter modulations remain constant for an object of constant electric properties regardless of the distance between the fish and the object. This observation follows from Data S1 Equation 7 and from the linearity of our filter model, since the field distortion solution, i.e. φ^s(t), does not depend on the distance, or other spatial variables.

Similarly, φ^s(t) does not depend on the object size for fixed electrical material properties of the objects, i.e. conductivity σ_obj and relative permittivity k_obj, according to Data S1 Equation 5. Thus, for objects with fixed electrical properties (material properties), by the linearity of our electroreceptor filter model, varying the object radius does not change the ratio between the MZ and DLZ filter modulations. This is experimentally observed in the electric color lines⁵ for natural objects with constant material properties.

However, experiments involving artificial objects typically do not measure the material property of objects, but rather the macroscopic properties, i.e. resistance and capacitance. Data S1 Equation 6 introduces a dependence on object radius in Data S1 Equation 5, thus breaking the electric color line for objects with varying radius when the resistance and capacitance are kept fixed. Nevertheless, the electric color lines are still approximately straight when variations of object size are small, and the “radius³” scaling principle from Data S1 Equation 7 still applies approximately.

Neural networks details

Small ANN for electric properties only

A small feed-forward ANN extracted the resistance and capacitance of the objects from a single electroreceptor’s responses (Figure S3A). The ReLU (ReLU(x) = max(0,x)) activation function was used between all layers of the network, except before the output layer, which was linear. We trained these models using stochastic gradient descent (SGD) optimizer with a learning rate of 0.001, and momentum of 0.9⁷⁷. We trained the models to convergence on training data, typically for 100 epochs with a batch size of 50, 000. We tested both ReLU and TanH (hyperbolic tangent) activation functions between layers, and both Adam⁷⁸ and SGD optimizers with learning rates ranging from 0.0001 to 0.01, but didn’t find significant differences in performance. We found that smaller architectures, i.e. fewer neurons per layer or just 2 instead of 3 layers, had slightly decreased performance. Since training time was not a bottleneck for these small models, we fixed the architecture as reported in Figure S3A.

CNN model

We generated electrosensory data using our field model framework to train the models. The electrosensory dataset contained objects with six properties (3D location, radius, resistance, capacitance) that were independently varied on a grid of values, with between 11 and 32 possible values for each property. The 3D location is represented by the lateral (rostro-caudal) and vertical (dorso-ventral) position of the object with respect to the fish, and the distance from the skin of the fish to the object. The simulated values were selected to cover the distributions of values typically used in experiments for all six properties. The individual ranges and spread for each property are marked by the true values (x–axis) of Figure 6. Each object generated it’s own electric image on the 20×30 array of receptors that we simulated on the fish body with cylindrical symmetry (Figure S3B, left). We chose the number of receptors based on the estimated number of receptors fish have on their trunk¹⁷, and distributed them on a uniform grid to ensure compatibility with CNN implementations. The training dataset contained approximately 40 · 10⁶ simulated objects around the fish, and the validation dataset contained approximately 10 · 10⁶ objects. All training was performed in Python using the PyTorch library⁷⁹ and the PyTorch Lightning library⁸⁰.

The CNN used in Results - “Object localization and characterization” has similar structure as AlexNet⁸¹, specifically shown in Figure S3B. The ReLU activation function was used between all layers of the network, except before the output layer, which was linear. We used dropout layers with a 0.5 dropout rate in the feedforward layers during training. We trained the models using SGD optimizer with a learning rate of 0.01 and momentum of 0.9⁷⁷. We trained the CNNs to convergence on training data, typically for 50 epochs with a batch size of 30, 000. Batch size was chosen to maximize the amount of data that fit in CPU and GPU memory to speed up training. We used the validation dataset to monitor the performance of the networks during training, and to select the best model (all results shown are on validation data not seen during training, except otherwise specified, e.g. in Figure S4).

We trained a few models with size variations of the base architecture (Figure S3B) to find the balance between model performance and training time, and to test the robustness of the results. We found that smaller architectures had slightly decreased performance without improving training time, and larger architectures did not improve performance during the allowed training time of 10 days per model. The models trained and presented throughout the paper maximized the number of parameters and training data per batch that fit in the available GPU memory (4× NVIDIA Titan Xp, 12 GB per node, CUDA Version 11.4). Thus, smaller models did not significantly improve runtime since parallelization was still efficient for the presented models, and larger models could not be trained in reasonable time since we needed to decrease the amount of training data to fit in memory. We grid-searched the model size by varying the number of parameters from approximately 200 · 10³ to 40 · 10⁶. We varied the number of parameters by changing the number of layers, channels, and neurons. These hyper-parameters ranged from 2–5 convolutional layers with 8–128 channels and 2–4 feedforward layers with 64–5120 neurons. We tested both ReLU and TanH activation functions between layers, and both Adam and SGD optimizers with learning rates ranging from 0.0001 to 0.02. We also varied the batch size from 2, 000–35, 000 to maximize the amount of data that fit in CPU and GPU memory for a training step, according to the network size.

Hybrid model

The hybrid model architecture (Figure S3C) is only slightly more complex than the CNN (Figure S3B). In particular, it uses two separate heads applied to the sensory input — the head receiving the electric image as input and extracting spatial properties of the object (i.e. the same CNN described above), and the head receiving the electric image along with spatial properties, provided either by the spatial head or externally, and extracting electrical properties of the object.

The electric head of the ANN has a much simpler structure than the spatial head. First, the electric head receives the electric image as input, and processes it to extract the responses of the most modulated electroreceptor organ. This is done by smoothing the electric image with a fixed spatial convolutional filter (kernel size 7 × 7), followed by extracting the signed response of the receptor with the maximum absolute value modulation. Second, the responses of the most modulated electroreceptor are multiplied by scale factors computed by learned polynomial functions of object distance and radius. Finally, a small ANN (Figure S3A) extracts the electrical properties of the object from the scaled feature space. The scaling factors are learned by the network, by learning the coefficients of two separate polynomial functions that scale the features with object distance (5 parameters of a degree–4 polynomial) and radius (4 parameters of a degree–3 polynomial).

ANNs performance on realistic behaviors

We tested our trained models on realistic behavior based on previous experiments to assess their performance. All input to our ANN models was generated from our field model using physical properties similar to those reported in the experiments. The relative position between the fish and the object was matched to the experimental setup. Since our simulated objects are limited to spherical shape, the size of our simulated objects was chosen to approximately match the volume of the objects used in the experiments, such that the amount of water displaced by the objects is the same. As described in Methods - “Electric colors”, our field model is grounded in material, rather than macroscopic, electric properties of objects. As such, we estimated the electric properties of the objects used in the experiments and tuned those properties to obtain a similar spread in the electric color lines for the simulated objects.

To evaluate our models on the discrimination tasks without altering the internal representations and computations performed by the ANNs, we used their raw output as input to simple classifiers generically used in the field, Random Forest Classifiers (RFCs). RFCs are a type of ensemble classification method that is typically used to establish a lower bound performance on non-linear classification problems. We obtained each discrimination output by coupling each of our trained and fixed ANNs with an RFC. First, we passed a noisy electric image through an ANN to obtain it’s output, i.e. the six estimated object properties referred to as the internal representation of the object. Then, we used the ANN’s internal representation to train an RFC, and validated the RFC discrimination performance on data not seen during training. RFC training was performed with the class weights flag set to “balanced” to ensure the classifiers are not biased towards either choice. All discrimination results shown are for electric images that were not seen during training of the RFC, except for Figure S4, where we show both training and validation performance.

In order to directly compare model performance with that of real fish, we added noise to the electric image input to the ANNs. Without the addition of such noise, model performance would fit the training data perfectly, preventing any comparison with the performance of real fish. As described below, we selected a noise level based on prior experimental results from three separate studies involving: (1) measurements of distance thresholds for object detection derived from recordings from ELL output neurons and (2) behavioral responses, and (3) estimates of the signal-to-noise ratios of individual electroreceptor afferent nerve fibers. We used Gaussian noise (0-mean and fitted standard deviation) independently added to each simulated electroreceptor of the electric image. Using a grid-search, we found that a noise standard deviation of 0.1% modulation matched the detection limit derived from recordings from ELL output neurons found in⁴⁵ (Figure S4). For the grid-search, we simulated small (3 mm), high-contrast (2kΩ, 1.5 nF) objects at various distances from the fish, similar to objects in⁴⁵. We trained RFCs to discriminate the presence of the object based on the equivalent early stages of electrosensory processing of our models, namely the output of our electroreceptor organ model. Importantly, we also found that using this same noise value, the object distance detection threshold of our models matched the behaviorally measured thresholds for Gnathonemus ⁸². For the detection task, we simulated large (30 mm), high-contrast (all 6 combinations of 1, 25 kΩ, and 0.7, 1.5, 7 nF) objects at various distances from the fish, similar to objects in⁸², trained RFCs to discriminate the presence of the object based on the internal representations of the ANNs, and found the detection threshold drops to chance level at distances between 50 and 70 mm depending on object contrast, consistent with the results from⁸². Additionally, we also estimated the response noise of individual afferent fibers based on published results⁶⁷ to further test our choice of noise amount to use in the simulations. The most sensitive recorded afferents (Fig 8 D in⁶⁷) had a 0.4 ms/% stimulus amplitude sensitivity. The typical response latency had approximately ±0.114 ms jitter (Fig 2 C in⁶⁷) across 200–500 discharges, suggesting an approximate 0.114/3 = 0.038 ms standard deviation in the latency of the responses (a 3–sigma event occurs every 370 samples on average). This suggests that the most sensitive afferents signal-to-noise ratio (SNR) is approximately 0.038/0.4 = 0.095% modulations in stimulus amplitude. Though this calculation is based on the most sensitive afferent fibers recorded, the noise amount we used in the simulations falls within the distribution of individual afferents’ SNR (computed based on the distribution of sensitivities from Fig 8 D in⁶⁷). The linearity of our electroreceptor organ model allows us to use this value as the noise standard deviation added to the MZ and DLZ filters modulations.

Quantification and statistical analysis

Statistical details for the experimental procedures are reported in methods and in the respective figure legends. All statistical details for the computational components are reported in the figure legends.

Supplementary Material

Supplementary figures S1-S4 and their legends in a LATEX-generated PDF

Data S1 “Electric Field Model” in a LATEX-generated PDF

Data S1

Data S1: Electric field model framework. Related to STAR Methods and Figures 3 and 5.

This supplement describes the electric field model framework used to simulate the electric images of objects on the fish’s skin. It includes the derivations of the electric field equations relevant for simulation.

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Deposited data
Experimental data used for modelling in this paper was collected in the Sawtell laboratory. Local field potential recordings were made in the medial zone and the dorsolateral zone of the electrosensory lobe.	Data is available at https://github.com/DenisTurcu/efish-receptors-model in the ./data folder as of the date of publication.
Experimental models: Organisms/strains
Male and female wild-caught Mormyrid fish of the species Gnathonemus petersii were used in these experiments (fish were 7–12 cm in length, of unknown age, and sex was not specifically selected for).	Wild-caught animals.	N/a.
Software and algorithms
Electroreceptors model software and analysis.	https://github.com/DenisTurcu/efish-receptors-model
Electric field model simulation framework.	https://github.com/DenisTurcu/efish-physics-model
Localization and characterization models, and behavioral task performance comparison software and analysis.	https://github.com/DenisTurcu/efish-characterization

Highlights.

• An end-to-end model provides insight into active electrosensation

• Neural networks perform comparably to real fish on realistic electrosensory tasks

• A two-stage computation is proposed for localizing and discriminating objects