Skip to main content
NCBI home page
Cognitive Neurodynamics logoLink to Cognitive Neurodynamics
. 2023 Sep 5;18(4):1977–1988. doi: 10.1007/s11571-023-09989-1

Predicting the temporal-dynamic trajectories of cortical neuronal responses in non-human primates based on deep spiking neural network

Jie Zhang 1,2,3,#, Liwei Huang 3,4,#, Zhengyu Ma 3,✉, Huihui Zhou 1,3,✉
PMCID: PMC11297849  PMID: 39104695

Abstract

Deep convolutional neural networks (CNNs) are commonly used as computational models for the primate ventral stream, while deep spiking neural networks (SNNs) incorporated with both the temporal and spatial spiking information still lack investigation. We compared performances of SNN and CNN in prediction of visual responses to the naturalistic stimuli in area V4, inferior temporal (IT), and orbitofrontal cortex (OFC). The accuracies based on SNN were significantly higher than that of CNN in prediction of temporal-dynamic trajectory and averaged firing rate of visual response in V4 and IT. The temporal dynamics were captured by SNN for neurons with diverse temporal profiles and category selectivities, and most sensitively captured around the time of peak responses for each brain region. Consistently, SNN activities showed significantly stronger correlations with IT, V4 and OFC responses. In SNN, correlations with neural activities were stronger for later time-step features than early time-step features. The temporal-dynamic prediction was also significantly improved by considering preceding neural activities during the prediction. Thus, our study demonstrated SNN as a powerful temporal-dynamic model for cortical responses to complex naturalistic stimuli.

Keywords: Neural prediction, Temporal dynamics, Spiking neural network, Convolutional neural network, Visual cortex

Introduction

Deep convolutional neural networks (CNNs), the biologically inspired hierarchical neural networks (Hubel and Wiesel 1959, 1962), demonstrated remarkable performance in the field of computer vision (LeCun et al. 2015). The validity of modeling the primate ventral stream using CNNs has been widely verified, and different hierarchical layers from different models successfully predicted the responses of neurons in the primate ventral stream including V1 (Cadena et al. 2019; Dapello et al. 2020; Zhuang et al. 2021), V4 (Yamins et al. 2014; Nayebi et al. 2018; Schrimpf et al. 2018; Zhuang et al. 2021), and inferior temporal (IT) (Cadieu et al. 2014; Yamins et al. 2014; Nayebi et al. 2018; Schrimpf et al. 2018; Kubilius et al. 2019; Chang et al. 2021; Higgins et al. 2021; Zhuang et al. 2021). Most of these studies have been focused on predicting the averaged firing rates in certain time window, while the temporal dynamics of the visual activities are fundamental in visual representation (Brincat and Connor 2006; Meyer et al. 2014; Issa et al. 2018) and categorization (Kar et al. 2019; Kubilius et al. 2019). A few studies tackled correlations between the model activities and neural responses across time, and found that deeper or recurrent CNNs showed stronger correlations with visual responses during the later period than that of the feedforward CNNs (Nayebi et al. 2018; Kar et al. 2019).

Different from CNNs, spiking neural networks (SNNs) are constituted by spiking neurons, which encode information with spikes in time series, inherently integrating both temporal and spatial features like actual neurons. The biologically plausible spike outputs make SNN a suitable candidate for simulating temporal trajectories of neural activities. And the latest development in directly trained deep SNNs (Fang et al. 2021; Zheng et al. 2021) provided the opportunity to explore the effect of neural temporal-dynamic trajectory predictions with SNN. A recent study showed that SNN might serve as a better model to predict the averaged firing rate of ventral visual pathway (Huang et al. 2023). There is still a lack of investigation on using SNNs to model the temporal-dynamic responses in the visual stream.

In this work, we predicted the averaged firing rates and temporal-dynamic responses of neurons in both visual and prefrontal cortex using model activities based on deep SNN and CNN with almost the same architecture. We found that SNN showed higher prediction accuracy than CNN in predicting the response trajectories in IT, V4, and orbitofrontal cortex (OFC), and in predicting the averaged firing rates in IT and V4. SNN captured the profiles in the response trajectories for neurons with different preferences and temporal dynamics, with the strongest prediction around the time of the peak firing rates in the response trajectories, about 120 ms for IT and V4 neurons, and ~ 150 ms for OFC neurons. Consistently, SNN activities showed significantly stronger correlations with IT, V4, and OFC responses. Furthermore, inspired by the method in a previous study (Oláh et al. 2022), we also developed a new method considering preceding neural activities during the prediction, which reduced the predicted error significantly. This work suggested SNN as a powerful temporal-dynamic model for cortical response to complex naturalistic stimuli.

Methods

Data collection

Single and multi-unit spikes were recorded from visual and prefrontal cortex of two awake behaving rhesus monkeys (Macaca mulatta, 12 and 15 kg) using 24 or 32-contact electrodes (V-Probe or S-Probe, Plexon Inc, Dallas, USA) in a 128 channel Cerebus System (Blackrock Microsystems, Salt Lake City, UT, USA). All animal procedures were approved by the Animal Care and Use Committees of Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences.

Monkeys were trained to perform a category-based visual search task (Zhang et al. 2021) under the control of a computer using MonkeyLogic software (University of Chicago, IL; Asaad et al. 2013). After monkeys fixated a central fixation point for 400 ms, a Cue stimulus was presented on the center of screen for a random duration between 500 and 1300 ms to indicate what category the target stimuli belonged to during the subsequent search phase. Monkeys were required to keep fixation on the central Cue during its presentation, and then the cue was replaced by the central fixation point. After monkeys fixated the central fixation point for 500 ms, a search array with multiple stimuli was presented on the screen. Monkeys freely moved eyes to search for the target stimulus and fixated it for at least 800 ms to get a reward. The cue stimulus was selected randomly from an image set consisting of 80 natural object images from 2 categories (Face 40; House 40), subtending an area of approximately 2 × 2 degrees. On average, each image was presented for about 12 times.

Recording chambers over areas V4, IT and OFC located by Magnetic Resonance Imaging (MRI) scans before surgery were implanted. Neural signals were filtered between 250 Hz and 5 kHz, amplified and digitized at 30 kHz to obtain spike data. The locations of recordings in V4, IT and LPFC were verified with MRI. Eye movements were recorded by an infrared eye tracking system (iViewX Hi-Speed, SensoMotoric Instruments (SMI), Teltow, Germany) at a sampling rate of 500 Hz.

Data analysis

Neuron selection

Neurons with visual responses and category selectivity to the Cue stimulus were selected for predicting. Visual response was detected by comparing the firing rates during a post-stimulus period (50 to 200 ms after stimulus onset) with the baseline firing rates during a pre-stimulus period ( − 150 to 0 ms before stimulus onset) using a Wilcoxon rank-sum test (Wilcoxon rank-sum, p < 0.05). Neurons shown category selectivity included Face-selective neurons and House-selective neurons. Face-selective neurons were defined as neurons whose responses to images in face category were significantly higher than that in the house category (Wilcoxon rank-sum, p < 0.05), and the averaged response to face images was at least 30% higher than the averaged response to house images. The numbers of Face-selective neurons in IT, V4, and OFC were 474, 252, and 567 respectively. House-selective neurons were defined as neurons whose responses to images in house category were significantly higher than that in the face category (Wilcoxon rank-sum, p < 0.05), and the averaged response to house images was at least 30% higher than the averaged response to face images. The numbers of House-selective neurons in IT, V4, and OFC were 210, 277, and 16, respectively.

Models

In our previous study, we used Spiking-Element-Wise (SEW) ResNet and ResNet with 18, 34, 50, 101 and 152 layers, and found consistent better performances of SEW ResNet (Huang et al. 2023). We employed SEW ResNet18 and ResNet18 in this study. The SEW ResNet was pretrained on the ImageNet Dataset using SpikingJelly (Fang et al. 2021, 2020). The well-known ResNet is from the Torchvision model zoo (Paszke et al. 2019). These two models have the same depth and almost the same structure, providing a fair comparison between SNN and CNN.

To extract features from both models, we fed the 80 images seen by the monkeys to the pre-trained models and collected features activated by Relu units for CNN, or activated by spiking neurons for SNN from layer 1 to layer 17. SNN produced time series output, in our case, the four-time-steps spikes for a static image. We averaged the activities from SNN over time for most of our analyses (T-avg). Other than that, we concatenated activities from all single-time-steps in some other analyses (Ts). Before predicting the neural responses with the model features, we first reduced the features dimensions of the 80 images to avoid overfitting, by performing truncated singular value decomposition, and collecting the top features of each layer according to the dimension number needed.

Neuron prediction

Our main purpose was predicting the neural responses in IT, V4, and OFC using SNN and CNN models mentioned above. The neural responses to be predicted were the responses to the cue stimuli, either in the form of averaged firing rates (Fig. 2), or in the form of temporal trajectories of responses (Figs. 3, 4, 5, 6 and 7). The averaged firing rates were calculated by averaging the number of spikes in the 50–250 ms window after stimulus onset, which is similar as (Chang et al. 2021; Huang et al. 2023). The temporal trajectories of responses were calculated by concatenating the averaged responses in every 50 ms bin with the last time point within the bin across the time range of 50–250 ms. The 50 ms sliding window slid for every 10 ms, resulting in 21 time steps of responses in total for each trajectory. For each image, the repeated trials were averaged to obtain reliable responses for each neuron.

Fig. 2.

Fig. 2

Open in a new tab

Neural similarity of model predictions and stimuli representations with SNN and CNN. A Correlations between actual averaged firing rates and model predicted activities averaged over IT neurons (n = 684). Neural responses were predicted either using SNN features averaged over time (orange line), or CNN features (green line) from layer 1 to layer 17, respectively. The SEMs of these averages across neurons were marked by error bars. B–C Correlations between actual and SNN/CNN predicted activities averaged over V4 neurons (n = 529) and OFC neurons (n = 583), respectively. D Correlations between stimuli representations of IT neural population and that of SNN features averaged over time (orange line) vs. CNN features (green line) in layer 1–17. E–F Correlations between stimuli representations of SNN/CNN models features and neural population in V4 and OFC

Fig. 3.

Fig. 3

Open in a new tab

Predicted and actual temporal trajectories of neuronal responses to 80 single-stimuli of an example IT neuron. The prediction was conducted using the 4-time-step model features from SNN, as well as the preceding responses (SNN (Ts) + PreFR). X-axis was from 50 to 250 ms relative to the time of stimulus onset

Fig. 4.

Fig. 4

Open in a new tab

Predictions of IT neuronal responses trajectories using SNN and CNN with and without preceding neural responses. A Averaged error percentages over IT neurons in prediction of the temporal responses trajectories by SNN features averaged over time (orange line) vs. CNN features (green line) from layer 1 to layer 17, respectively, and by SNN vs. CNN features along with preceding neural responses (red line vs. blue line). B Averaged prediction error percentages of IT neurons using SNN features from all time steps with (light orange line) and without (rosy line) preceding neural responses. C Example IT neurons plotted with predicted and actual temporal trajectories of neuronal responses average across 40 face images (red curve) and 40 house images (blue curve), respectively. The prediction was conducted using the 4-time-step model features (Ts) from SNN. Shading around average firing rates indicates the SEM (±). Prediction error percentage for each neuron was marked above

Fig. 5.

Fig. 5

Open in a new tab

Predictions of V4 and OFC neuronal responses trajectories using SNN and CNN with and without preceding neural responses. A Averaged prediction error percentages of V4 neurons using SNN features averaged over time (orange line) vs. CNN features (green line) and using SNN vs. CNN features along with preceding neural responses (red line vs. blue line). B Example V4 neurons plotted with predicted and actual temporal trajectories of neuronal responses average across 40 face images (red curve) and 40 house images (blue curve), respectively. The prediction was conducted using the 4-time-step model features (Ts) from SNN. Shading around average firing rates indicates the SEM (±). Prediction error percentage for each neuron was marked above. C Averaged prediction error percentages of OFC neurons using SNN vs. CNN with and without preceding neural responses. D Example OFC neurons with averaged predicted and actual temporal trajectories of neuronal responses

Fig. 6.

Fig. 6

Open in a new tab

Temporal dynamics of predictions in early and late layers of SNN and CNN. A Correlations between actual IT neural trajectories and trajectories predicted by SNN features averaged over time (orange line) vs. CNN features (green line) in layer 3. Shading around average indicates the SEM (±). B Correlations between actual IT neural trajectories and SNN/CNN predicted trajectories in layer 14. C Population responses across all IT neurons, which was the averaged firing rate to all images for each neuron. Shading around average firing rates indicates the SEM (±). D and E Correlations between actual V4 neural trajectories and SNN/CNN predicted trajectories in layer 3 and layer 14, respectively. G and H Correlations between actual OFC neural trajectories and SNN/CNN predicted trajectories in layer 3 and layer 14, respectively. F and I Population responses of V4 cells, and OFC cells, respectively

Fig. 7.

Fig. 7

Open in a new tab

Temporal dynamics of IT predictions using separate time steps of SNN. Correlations between actual IT neural trajectories and trajectories predicted by features in each of the 4 time steps of SNN were plotted for each layer. Shading around average correlations over neurons indicates the SEM (±)

The neural responses were not only to be predicted using model activities. Inspired by the work of Oláh et al. (2022), we also included preceding neural activities to predict the responses trajectories in some analyses (+ PreFR). These preceding responses were calculated by concatenating the averaged responses in 50 ms bins with their ending time points across the time range of − 14 to 0 ms just before the responses needed to be predicted. The 50 ms sliding window slid for every 2 ms, resulting 8 time steps of responses in total for each response needed to be predicted.

Overall, for SNN, we used five different types of feature matrixes resulting from either only model activities or both model and preceding neural activities to predict the neural responses, which were constructed following the procedures described below. (i) SNN (T-avg): the activities in each SNN layer were averaged over the four time steps and then reduced to 40 dimensions. (ii) SNN (T-avg) + PreFR: we averaged the activities in each SNN layer over the four time steps and reduced dimensions to 32, and then concatenated them and 8 time steps of preceding neural responses. (iii) SNN (Ts): the dimensions of activities in each layer were reduced independently for each time step to 10, and then concatenated across the 4 time steps. (iv) SNN (Ts) + PreFR: we reduced the dimensions of activities in each layer independently for each time step to 8, and then concatenated them from the 4 time steps and 8 time steps of preceding neural responses. (v) T1, T2, T3, and T4: we used SNN activities in each layer from every time step separatelyand reducing the dimensions to 40 to access the independent effect of features from each time step in predicting IT neurons in Fig. 7. For CNN, we used two different types of feature matrixes resulting from either model activities or model and preceding neural activities to predict the neural responses, which were constructed following the procedures described below. (i) CNN: the activities in each CNN layer were reduced to 40 dimensions. (ii) CNN + PreFR: we reduced the dimensions of activities in each CNN layer to 32, and then concatenated them and 8 time steps of preceding neural responses. After dimensionality reduction, the dimension numbers of the feature matrixes used to predict the neural responses were all the same as 40. These 40-dimension feature matrixes inputs were mapped to each neuron linearly by Ridge regression in an eightfold cross-validation paradigm for each time bin. Specifically, we split the images into 8 subsets randomly, resulting 10 images per subset; each of the subsets was used as testing set (10 images) while the rests were the training images (10 × 7 = 70 images); weights of the linear combination were determined by the activities of the model and the neuron to 70 training images, and then neural responses to the 10 testing images were predicted using the same linear transform. The linear transformation for different time bins along the response trajectories were independent. All results reported here were predicted rather than fitted responses.

Quantification of model performance

Prediction error percentage

The prediction error percentage for each neuron was calculated as follows. For each image, the absolute errors between the temporal response prediction and actual response trajectory from 50 to 250 ms after stimulus onset were obtained, and summed up across time, approximating the area between the 200 ms-long prediction and actual trajectories. For each image in each neuron’s preferred category, the actual response trajectory from 50 to 250 ms was summed up across time, approximating the area of the 200 ms-long actual response trajectory. Neurons used for prediction were either Face-selective neurons or House-selective neurons whose averaged response amplitudes were significantly different to stimuli in the face category and house category. A neuron exhibited stronger responses to stimuli in its preferred category, suggesting that these stimuli held great significance for this neuron’s representation. Conversely, responses evoked by stimuli in its non-preferred category were smaller, indicating a comparatively lower level of responsiveness or importance assigned by the neuron to stimuli outside its preferred category. Given that for a single neuron, prediction error should consider all stimuli as a unified metric, the averaged response in its preferred category was used as the denominator for normalization. Specifically, the percentage of prediction error for each neuron was computed by normalizing the average of response area errors over all images for that neuron by its actual response area average in the preferred category. To calculate the prediction error percentage for each brain region, we averaged prediction error percentages over all neurons from that brain region.

Correlation

We also assessed the prediction accuracy of the model with Pearson correlation coefficients between the predicted and measured responses across all images (Carandini et al. 2005; David and Gallant 2005; Yamins et al. 2014; Schrimpf et al. 2018; Nayebi et al. 2018; Huang et al. 2023). To obtain the correlation coefficient of each brain region, correlation coefficients across all neurons in each brain region were averaged separately.

Representational similarity analysis (RSA)

To measure representational similarity, we calculated RSA following the methodology commonly used in previous studies (Kriegeskorte et al. 2008; Cadieu et al. 2014; Huang et al. 2023). Firstly, we calculated representational dissimilarity matrices (RDM) for the neural representations and the model representations of image stimuli based on the population activities in each brain region, and based on the population activities in each layer for each model, respectively. The RDMs were calculated with each paired stimuli across the total 80 stimuli in our task, resulting two 80×80 matrixes, respectively. It was defined as

RDMϕxij=1-covϕxi,ϕxjvarϕxi·varϕxj,

where ϕx indicates the population representational activities vector for each stimulus, i and j are indexes of two stimuli, cov and var indicate covariance and variance. Then, the RSA score was obtained by calculating the Spearman rank correlation coefficient between the flattened upper triangles of these two RDMs, which measure the similarity between stimuli representation of the model layer and that of the brain region.

Results

We recorded neural activities of 684 units in IT, 529 units in area V4, and 583 units in OFC when two monkeys were presented with 80 natural object images comprising Face and House categories for at least 500 ms (Fig. 1). All units showed significant visual responses (Wilcoxon rank-sum test, p < 0.05), and category selectivity (the response to the preferred category was at least 30% higher than the response to the alternative category, see Methods). The analyses in Fig. 2 were based on averaged firing rates in the 50–250 ms window after stimulus onset. For the rest of our analyses, the temporal trajectories of responses from 50 to 250 ms were used.

Fig. 1.

Fig. 1

Open in a new tab

Schematic of the framework to predict neural activities with SNN. The activities of neurons in brain areas were predicted by the activities from SNN elicited by the same images

We employed SNN and CNN models: SEW ResNet18 and ResNet18, respectively, which have the same depth and almost the same architecture, and were both pre-trained on the ImageNet dataset. Different from CNN, the model activities of SNN are spikes with four time steps. For the averaged firing rates, the activations of units from layer 1 to layer 17 for each model were used to predict the neuron response by Ridge regression methodology. For the prediction of temporal-dynamic trajectories, apart from the model features, we also included totally 8 time steps of preceding neural responses in the input matrix of the Ridge regression (+PreFR). Before regression, the feature dimension of each model layer was reduced to 40 or 32 for each image by performing truncated singular value decomposition to avoid overfitting. The input matrixes were all 40 dimensions, only that if 8 time steps of the preceding responses were included as part of the inputs, the model features were narrowed down to 32 dimensions, while if not, the 40 dimensions were all from model features. For all conditions, SNN and CNN counterpart were fit with the same form of linear model, using the same number of parameters.

Prediction of averaged firing rates in visual and prefrontal cortex based on SNN and CNN activities

Firstly, we predicted the averaged firing rates in the 50–250 ms window using SNN (T-avg) and CNN features from each layer, and qualitatively measured the prediction performance by calculating the Pearson correlation coefficient between the predicted and actual responses. Figure 2A shows correlations based on SNN and CNN, averaged over IT neurons. The correlation trajectory across model layers of CNN was generally below the correlation trajectory of SNN, and the differences were much larger for lower and intermediate layers. For the two models, most late layers predicted IT responses increasingly better than early layers, with the correlations reaching the maximum both in layer 14. For a fair comparison, we applied Wilcoxon signed-rank test to the respective best layers of SNN and CNN, and SNN showed a significantly stronger predictive power of IT averaged firing rates than CNN (layer 14: p < 0.001), which was consistent with the previous study using other datasets (Huang et al. 2023). For V4, the best layer of SNN predicted V4 average responses significantly better than the best layer in CNN (Fig. 2B; layer 13 vs. layer 14: Wilcoxon signed-rank test, p < 0.05). The prediction of OFC responses using the best layer of SNN was not significantly different from that of CNN, however, for most corresponding layers, SNN outperformed CNN, especially for the early layers (Fig. 2C). It’s possible that SNN prediction performed better because the representation of different stimuli by its features were more similar to representation by the neural population. We performed the RSA (Kriegeskorte et al. 2008) to quantify the representation similarity. Indeed, the RSA scores were higher in SNN for most layers, compared with the corresponding CNN layers for IT neurons (Fig. 2D). The trends continued to hold in both V4 and OFC (Fig. 2E and F).

Predicting temporal-dynamic trajectories of neuronal responses with SNN and CNN

Although static images produce temporal-dynamic neural spiking series as visual responses, most of the previous studies predicted averaged firing rates with CNNs which highly simplified the computational properties of biological neurons and discard the temporal dynamics. In this study we predicted the temporal-dynamic trajectories of neuronal responses using SNN for the first time, other than predicting averaged firing rates in the primate visual and prefrontal cortex. Figure 3 shows the neural response trajectory of a single IT neuron and the predicted response trajectory to single-images, with 0.59 as the time averaged correlation between actual and predicted responses. Examining the specific predictions for individual images shows that the predicted trajectories resemble the actual responses in general by capturing the major characteristic for responses of each image, with small deviations in different phases.

To quantify how well the temporal responses can be predicted by our models more accurately, besides measuring the correlation, we took prediction error percentage as a more intuitive approach: for each individual neuron, we computed the averaged response area between the predicted and actual response trajectories from 50 to 250 ms over all images, normalized by the neuron’s actual response area averaged across images in its preferred category (see Methods). The prediction error percentage of the neuron in Fig. 3 was 16%. We compared the prediction error percentage averaged over IT neurons for each model layer, using either SNN (T-avg) features (Fig. 4A orange line) or CNN features (Fig. 4A green line). Similar as the averaged firing rates prediction, SNN predicted the temporal response trajectories significantly better than CNN even for the best layer (layer 14: Wilcoxon signed-rank test, p < 0.001).

In addition to predictions with only the features of models, we also introduced the preceding neural response to improve the prediction performance (Oláh et al. 2022). Specifically, we used the top 32 features from the model layers and 8 time steps of the preceding responses extending totally 64 ms to predict response trajectories. For both SNN (T-avg) and CNN, the prediction error percentage reduced approximately 5% with the preceding neural responses (Fig. 4A). Moreover, SNN still outperformed CNN in predicting the temporal responses of IT neurons for the best layer (layer 14 vs. layer 16: Wilcoxon signed-rank test, p < 0.001). The four-time-step spike series generated by SNN provided another way to extract features instead of averaging over time, thus we reduced the feature dimension to 10 for each time step separately and collected all reduced features together as 10*4 model features. Using these SNN (Ts) features further improved the prediction performance of IT response trajectories (Fig. 4B light orange line). The strongest prediction achieved by using SNN (Ts) + PreFR features (Fig. 4B rosy line), which consisted of the top 8 features from each time step of the models (8 × 4 in total) and 8 time steps of the preceding responses. They all outperformed their CNN counterpart (Wilcoxon signed-rank test, p < 0.001). For illustration, we showed category-averaged actual and predicted response trajectories based on SNN (Ts) features from 6 example neurons in IT with different prediction error percentages, category selectivities, and temporal dynamics (Fig. 4C).

We also predicted neural response trajectories of V4 and OFC neurons. The trends observed in IT continued to hold in both V4 and OFC predictions. SNN predicted V4 neurons’ temporal response trajectories significantly better than CNN for their respective best layers (Fig. 5A, layer 14) with (Wilcoxon signed-rank test, p < 0.001) and without preceding responses (Wilcoxon signed-rank test, p < 0.01). SNN outperformed CNN in its prediction of responses trajectories from neurons in OFC for the best layers (Fig. 5C; with preceding responses: layer 14 vs. layer 16, Wilcoxon signed-rank test, p < 0.001; without preceding responses: layer 14, Wilcoxon signed-rank test, p < 0.01). Predicted and actual category-averaged response trajectories from three example neurons in V4 and OFC were shown in Fig. 5B and D, respectively.

Temporal characteristics of prediction in early and late layers of SNN and CNN

Although in general, the prediction error percentages were systematically smaller for SNN compared with the same layer in CNN, there were larger differences for early layers, and smaller differences for later layers for all three areas (Figs. 4A, 5A and C). Thus, close inspections of the prediction performances of the response trajectories in the early and late layers were necessary. We showed the correlation between predicted and actual responses along time for layer 3 and layer 14, averaged over IT neurons (Fig. 6). The correlations using SNN (T-avg) features were higher than that of CNN features for almost every time bins for layer 3 (Fig. 6A, Wilcoxon signed-rank test, p < 0.05 for 20 out of 21 time bins). For layer 14, the stronger correlations of SNN compared to CNN mainly distributed within the time period after 90 ms (Fig. 6B, Wilcoxon signed-rank test, p < 0.05). The strongest correlations across time for both layers were around 120 ms, which was the time of the peak response for the IT population (Fig. 6C, peak of the population response for all images was at 130 ms). SNN (T-avg) features in layer 3 were also yielding stronger correlations at the time period from 100 to 250 ms in V4 (Fig. 6D, Wilcoxon signed-rank test, p < 0.05) and mostly after 120 ms in OFC (Fig. 6G, Wilcoxon signed-rank test, p < 0.05), comparing with CNN features. For layer 14, SNN yielded stronger correlations at the time period mainly from 120 to 160 ms in V4 (Fig. 6E, Wilcoxon signed-rank test, p < 0.05), however, the stronger correlations of SNN were only at the time of 190 ms in OFC (Fig. 6H, Wilcoxon signed-rank test, p < 0.05). Similarly, the strongest correlations across time were ~ 120 ms and ~ 150 ms for V4 and OFC, respectively, which were the time of peak responses for the two areas (Fig. 6F and I, peaks of the population responses were at 120 ms for V4 and at 150 ms for OFC). These results were consistent with the previous studies (Nayebi et al. 2018; Kar et al. 2019), which confirmed the relatively weak correlations between CNN outputs and the response trajectories during later period, and extended the modeling of temporal-dynamic trajectory of visual responses to SNN, which employed more biological realistic neurons and achieved better performances.

Temporal-dynamic response predictions based on single-time-step features in SNN

To determine which time step in SNN was critical for the accurate predictions of the temporal trajectories of neuronal responses, we conducted prediction and the corresponding quantification based on the model features from single time steps in SNN. Specifically, we first collected the features for one time step in SNN, and reduced the dimensions to 40 to predict the response trajectories of individual IT neurons. Then the correlation trajectories across time were calculated for all 17 layers and 4 time-steps. For all of the layers, the correlations were significantly influenced by SNN time step (Fig. 7, Kruskal–Wallis One-Way ANOVA, p < 0.001). For most layers, the IT response trajectory predictions using the last time step (red curves) were better than the first two time steps, while the predictions using the first time step (blue curves) were the worst for all of the layers. This indicated the different contributions of the four time steps for predicting the response trajectories.

Discussion

To the best of our knowledge, it is the first study to introduce deep SNN to model the temporal dynamics of neural responses in ventral visual stream, and prefrontal cortex. SNN showed higher prediction accuracy than CNN in predicting neural responses in IT, V4, and OFC. SNN captured the profiles in the response trajectories for neurons with different preferences and temporal dynamics. Consistently, SNN activities show significantly stronger correlations with IT, V4, and OFC responses, compared with CNN activities. Furthermore, considering the preceding neural activities during prediction significantly reduced the predicted errors.

Neural processing evolves through time even for static external stimulation, thus the temporal dynamics of the responses are fundamental in visual representation (Brincat and Connor 2006; Meyer et al. 2014; Issa et al. 2018) and categorization (Kar et al. 2019; Kubilius et al. 2019). Normally and abnormally arranged face images elicited different neuronal responses during late period in the higher IT region, and in intermediate IT regions, while firings in the first 100 ms were equivalent (Issa et al. 2018). These phenomena might indicate code error signals feedback from higher areas through time. A research about image familiarity found that responses of IT population differentiated preferred and non-preferred images first, and then differentiated familiar and unfamiliar images about 50 ms later (Meyer et al. 2014). Integrating representations of simple shapes into that of complex objects was also a temporal dynamic process of IT neurons, with simple parts represented by the neural responses first, and then configuration information of simple parts emerged later to explicitly represent the complex object (Brincat and Connor 2006). Thus, predictions of the temporal-dynamic trajectories are important for neural responses modeling.

In earlier research, models used to predict temporal-dynamic neural responses were usually from the field of computational neuroscience using hand-designed filters and parameters. For example, linear-nonlinear-Poisson cascade models were used to predict the peri-stimulus time histograms of ganglion cells of macaque retina in vitro to white noise stimli (Pillow et al. 2008), which captured 80–95% of the variance in 26 out of 27 cells. A simple recurrent neural network model simulated posterior IT neurons in response to contour stimuli (Brincat and Connor 2006), and captured the different temporal profiles for single-part and multi-part single/population neural responses. A three-layer neural network modeled the temporal dynamics of neural population responses to face images in posterior, central and anterior IT with each layer, respectively (Issa et al. 2018). The simulated inputs were simplified as sigmoidal ramps rather than actual images. This model predicted the difference between temporal-dynamic population responses to two types of face images, with 0.9 as the goodness of fit. Normally, neural activities were assumed as linear combinations of the stimulus features, however, these traditional models are not good at inferring features distributions of complex images (Triplett and Goodhill 2019).

Apart from neuroscientific models, recent progresses in computer vision makes deep CNN, which can extract features from complex images, the most precise model in predicting individual responses of neurons. While most of the studies modeled averaged neuronal firing rates (Cadieu et al. 2014; Yamins et al. 2014; Nayebi et al. 2018; Schrimpf et al. 2018; Cadena et al. 2019; Kubilius et al. 2019; Dapello et al. 2020; Chang et al. 2021; Higgins et al. 2021; Zhuang et al. 2021), there were a few studies concentrated on predicting temporal-dynamic responses. Mounier et al (2021) predicted the single-neuron response patterns along time under simple visual stimuli in lateral geniculate nucleus, and achieved a mean correlation of 0.7 between the predicted and actual firing rates elicited by the flashed stimuli of a pixel or checkerboard patterns. While we predicted the response trajectories elicited by complex static natural images of neurons in higher brain areas. Deep recurrent CNNs were used to predict responses of single neurons in IT and V4 to natural object images along time (Nayebi et al. 2018; Kar et al. 2019), which showed stronger correlations with neural responses than that of the feedforward CNNs during the later period. Both this deep RCNN and our model evolved along time. While in the RCNN, activities along time were used to predict corresponding temporal responses of IT neurons with the same linear transformation, in our study, fixed model activities directly predicted responses of single neurons at different time with independent linear transformations. Even without recurrent connections, we found the predicted response trajectories by SNN matched the actual trajectories with higher precision and significantly lower prediction error compared with that of CNN. Similarly, we found that SNN activities correlated better than CNN along time, especially in the late period. Further study is needed to extend to SNN and CNN with recurrent connections.

In our study, we employed SEW ResNet, the most representative deep SNN with skip connections and remarkable object recognition performance (Fang et al. 2021). Skip-layer connections have been discovered in brain architecture, suggesting the biologically plausibility of SEW ResNet’s structure (Winding et al. 2023). As brain-inspired models such as CORnet-S, which specifically designed according to the connectivity of brain areas, outperformed CNNs in modeling the primate ventral visual stream (Kubilius et al. 2019), we found that CORnet-S did not exhibit comparable representational similarity to the spiking SEW ResNet14 model with the same depth and fewer parameters (Huang et al. 2023). Therefore, we selected SNN based on ResNet architecture in this study. However, it was possible to further enhance the performance of SNN in predicting neural responses by incorporating more brain-inspired architectures.

Although the SNN showed superior performance with most layers, which might owe to its spike mechanism, the prediction performance were comparable between SNN and CNN in layer 16 and layer 17 (Figs. 2A–C, 4A, 5A and C), suggesting that the higher-order and more abstract representations in these layers of SNN and CNN might be similar under static stimuli (Fig. 2D–F). Through comparisons, we observed a negative correlation between the representational similarities of brain and network activities and the prediction errors, which might offer some insight into their prediction performances. Further study is required to further elucidate the relationship between late layers in SNN and CNN. Typically, neural responses from different brain regions do not exhibit optimal fits with a single specific layer across different models, reflecting variations in hierarchy or anatomical alignment to the brain (Yamins et al. 2014; Nayebi et al. 2018; Huang et al. 2023). Thus, the comparisons of model performances were always conducted using their respective best layers (Kubilius et al. 2019). In this study, layer 16 and 17 were rarely the best layers for fitting in SNN, with lower correlations between the predicted and observed responses in these late layers compared to the earlier ones (Fig. 2). Overall, this study demonstrated the higher performance of SNN in modeling temporal dynamics in ventral visual stream.

Acknowledgements

This work was supported by The Major Key Project of the Peng Cheng Laboratory, the National Natural Science Foundation of China (62027804), and Shenzhen Fundamental Research Program (JCYJ20200109114805984).

Data availability

The datasets generated and analyzed during the current study are available from the corresponding author on reasonable request.

Declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Footnotes

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Jie Zhang and Liwei Huang are co-first authors.

Contributor Information

Zhengyu Ma, Email: mazhy@pcl.ac.cn.

Huihui Zhou, Email: zhouhh@pcl.ac.cn.

References

  1. Brincat SL, Connor CE (2006) Dynamic shape synthesis in posterior inferotemporal cortex. Neuron 49(1):17–24. 10.1016/j.neuron.2005.11.026 10.1016/j.neuron.2005.11.026 [DOI] [PubMed] [Google Scholar]
  2. Cadena SA, Denfield GH, Walker EY, Gatys LA, Tolias AS, Bethge M, Ecker AS (2019) Deep convolutional models improve predictions of macaque V1 responses to natural images. PLoS Comput Biol 15(4):e1006897. 10.1371/journal.pcbi.1006897 10.1371/journal.pcbi.1006897 [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Cadieu CF, Hong H, Yamins DL, Pinto N, Ardila D, Solomon EA, Majaj NJ, DiCarlo JJ (2014) Deep neural networks rival the representation of primate IT cortex for core visual object recognition. PLoS Comput Biol 10(12):e1003963. 10.1371/journal.pcbi.1003963 10.1371/journal.pcbi.1003963 [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Carandini M, Demb JB, Mante V, Tolhurst DJ, Dan Y, Olshausen BA, Gallant JL, Rust NC (2005) Do we know what the early visual system does? J Neurosci 25(46):10577–10597 10.1523/JNEUROSCI.3726-05.2005 [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Chang L, Egger B, Vetter T, Tsao DY (2021) Explaining face representation in the primate brain using different computational models. Current Biology 31(13):2785–2795. 10.1016/j.cub.2021.04.014 10.1016/j.cub.2021.04.014 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Dapello J, Marques T, Schrimpf M, Geiger F, Cox D, DiCarlo JJ (2020) Simulating a primary visual cortex at the front of CNNs improves robustness to image perturbations. Adv Neural Inform Process Sys 3:13073–13087 [Google Scholar]
  7. David SV, Gallant JL (2005) Predicting neuronal responses during natural vision. Network 16(2–3):239–260 10.1080/09548980500464030 [DOI] [PubMed] [Google Scholar]
  8. Fang W, Yu Z, Chen Y, Huang T, Masquelier T, Tian Y (2021) Deep residual learning in spiking neural networks. Adv Neural Inf Process Syst 34:21056–21069 [Google Scholar]
  9. Fang W, Chen Y, Ding J, Chen D, Yu Z, Zhou H, Tian Y et al (2020) SpikingJelly. https://github.com/fangwei123456/spikingjelly. Accessed: 2022–03–04
  10. Higgins I, Chang L, Langston V, Hassabis D, Summerfield C, Tsao D, Botvinick M (2021) Unsupervised deep learning identifies semantic disentanglement in single inferotemporal face patch neurons. Nat Commun 12(1):6456. 10.1038/s41467-021-26751-5 10.1038/s41467-021-26751-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Huang L, Ma Z, Yu L, Zhou H, Tian Y (2023) Deep spiking neural networks with high representation similarity model visual pathways of macaque and mouse. arXiv preprint arXiv:2303.06060
  12. Hubel DH, Wiesel TN (1959) Receptive fields of single neurones in the cat’s striate cortex. J Physiol 148(3):574–91. 10.1113/jphysiol.1959.sp006308 10.1113/jphysiol.1959.sp006308 [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Hubel DH, Wiesel TN (1962) Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J Physiol 160(1):106–54. 10.1113/jphysiol.1962.sp006837 10.1113/jphysiol.1962.sp006837 [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Issa EB, Cadieu CF, DiCarlo JJ (2018) Neural dynamics at successive stages of the ventral visual stream are consistent with hierarchical error signals. Elife 7:e42870. 10.7554/eLife.42870 10.7554/eLife.42870 [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Kar K, Kubilius J, Schmidt K, Issa EB, DiCarlo JJ (2019) Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior. Nat Neurosci 22(6):974–983. 10.1038/s41593-019-0392-5 10.1038/s41593-019-0392-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Kriegeskorte N, Mur M, Bandettini P (2008) Representational similarity analysis - connecting the branches of systems neuroscience. Front Syst Neurosci 2:4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Kubilius J, Schrimpf M, Kar K, Rajalingham R, Hong H, Majaj N, Issa E, Bashivan P, Prescott-Roy J, Schmidt K, Nayebi A, Bear D, Yamins DL, DiCarlo JJ (2019) Brain-like object recognition with high-performing shallow recurrent ANNs. Adv Neural Inf Process Syst 2019:12805–12816 [Google Scholar]
  18. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–44. 10.1038/nature14539 10.1038/nature14539 [DOI] [PubMed] [Google Scholar]
  19. Meyer T, Walker C, Cho RY, Olson CR (2014) Image familiarization sharpens response dynamics of neurons in inferotemporal cortex. Nat Neurosci 17(10):1388–1394. 10.1038/nn.3794 10.1038/nn.3794 [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Mounier E, Abdullah B, Mahdi H, Eldawlatly S (2021) A deep convolutional visual encoding model of neuronal responses in the LGN. Brain Inform 8(1):11. 10.1186/s40708-021-00132-6 10.1186/s40708-021-00132-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Nayebi A, Bear D, Kubilius J, Kar K, Ganguli S, Sussillo D, DiCarlo JJ, Yamins DL (2018) Task-driven convolutional recurrent models of the visual system. In: Advances in Neural Information Processing Systems, 31
  22. Oláh VJ, Pedersen NP, Rowan MJM (2022) Ultrafast simulation of large-scale neocortical microcircuitry with biophysically realistic neurons. Elife 11:e79535. 10.7554/eLife.79535 10.7554/eLife.79535 [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L et al (2019) Pytorch: an imperative style, high-performance deep learning library. Adv Neural Inf Process Syst 32:8024–8035 [Google Scholar]
  24. Pillow JW, Shlens J, Paninski L, Sher A, Litke AM, Chichilnisky EJ, Simoncelli EP (2008) Spatio-temporal correlations and visual signalling in a complete neuronal population. Nature 454(7207):995–999. 10.1038/nature07140 10.1038/nature07140 [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Schrimpf M, Kubilius J, Hong H, Majaj NJ, Rajalingham R, Issa EB, Kar K, Bashivan P, Prescott-Roy J, Geiger F, Schmidt K, Yamins DLK, DiCarlo JJ (2018) Brain-score: which artificial neural network for object recognition is most brain-like? BioRxiv 407007
  26. Triplett MA, Goodhill GJ (2019) Probabilistic encoding models for multivariate neural data. Front Neural Circu 13:1. 10.3389/fncir.2019.00001 10.3389/fncir.2019.00001 [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Winding M, Pedigo BD, Barnes CL, Patsolic HG, Park Y, Kazimiers T, Fushiki A, Andrade IV, Khandelwal A, Valdes-Aleman J, Li F, Randel N, Barsotti E, Correia A, Fetter RD, Hartenstein V, Priebe CE, Vogelstein JT, Cardona A, Zlatic M (2023) The connectome of an insect brain. Science 379(6636):eadd9330 10.1126/science.add9330 [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Yamins DL, Hong H, Cadieu CF, Solomon EA, Seibert D, DiCarlo JJ (2014) Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proc Natl Acad Sci USA 111(23):8619–8624. 10.1073/pnas.1403112111 10.1073/pnas.1403112111 [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Zhang J, Zhu X, Wang S, Ma Z, Esteky H, Tian Y, Desimone R, Zhou H (2021) Visual attention in the fovea and the periphery during visual search. BioRxiv 2021–11. 10.1101/2021.11.22.469359
  30. Zheng H, Wu Y, Deng L, Hu Y, Li G (2021) Going deeper with directly-trained larger spiking neural networks. Proceed AAAI Conf Artif Intell 35(12):11062–11070. 10.1609/aaai.v35i12.17320 10.1609/aaai.v35i12.17320 [DOI] [Google Scholar]
  31. Zhuang C, Yan S, Nayebi A, Schrimpf M, Frank MC, DiCarlo JJ, Yamins DLK (2021) Unsupervised neural network models of the ventral visual stream. Proc Natl Acad Sci U S A 118(3):e2014196118. 10.1073/pnas.2014196118 10.1073/pnas.2014196118 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

The datasets generated and analyzed during the current study are available from the corresponding author on reasonable request.

Articles from Cognitive Neurodynamics are provided here courtesy of Springer Science+Business Media B.V.

RESOURCES