Low‐Latency Visuotactile Neuron Using Self‐Oscillating Memristor

ABSTRACT

Low latency perception of visual and tactile information is a prerequisite for embodied intelligence, and combined with bio‐inspired spiking neural network (SNN), it enables efficient interaction with the physical environment. However, achieving low‐latency perception and spike conversion of external stimuli at the hardware level remains challenging, while biological systems utilize the time‐to‐first‐spike (TTFS) and firing rate of neurons to perceive surrounding signals in real time. Herein, a self‐oscillating neuron based on NbO_x memristors is demonstrated, leveraging intrinsic parasitic capacitance as the sole integration element to enable simultaneous TTFS and rate encoding of visual and pressure stimuli with an ultra‐low first spike latency of 260 ns. By eliminating redundant capacitance and exploiting the adaptive timing characteristics of TTFS encoding, an intrinsically low‐latency signal transmission pathway is established. Evaluation on the CIFAR‐10 dataset demonstrates that the fusion of TTFS and rate coding achieves higher accuracy, improved noise robustness, and reduced temporal latency compared to rate coding schemes. Furthermore, multisensory integration is validated through Braille datasets recognition, demonstrating improved accuracy under visually constrained conditions. These results highlight the potential of multisensory neuromorphic perception systems for real time human‐machine interaction and embodied intelligence applications.

1. Introduction

Embodied intelligence and robotics require the acquisition of multi‐sensory information, which is used to construct multidimensional features of objects, in order to enable comprehensive perception and accurate awareness of the physical world [1, 2]. Human beings interact naturally with the physical world through tactile and visual cues that are information‐rich and dynamic (Figure 1a), providing inspiration for the development of electronic counterparts [3, 4]. Accordingly, low‐latency visual and tactile perception is a crucial prerequisite for accurate object location, distance estimation, and safe physical human‐machine interaction [5, 6]. Biological neural networks achieve rapid temporal response and high energy efficiency by integrating sensing, memory, and processing functions, with neurons transmitting external information in spike sequences [7, 8]. Under sustained stimulation, neurons can autonomously generate repetitive spiking sequences with intrinsic temporal dynamics [9]. Notably, both the firing rate and the time to the first spike in sparse spike sequences generated by neurons are modulated by the stimulus intensity from the physical world [10, 11]. For example, in the human visual system, spiking sequences with a higher rate and shorter latency are generated when photoreceptor cells receive stronger light stimuli [12]. Despite these appealing properties, faithfully replicating spike‐based multisensory processing systems analogous to biological systems at the hardware level remains a significant challenge.

FIGURE 1.

Bioinspired neuromorphic Visuotactile system. (a) Vision and tactile information are obtained by the eyes and skin, respectively, which are transmitted to the cerebral cortex in the form of sparse spikes by neurons. With the accumulation of external stimuli, Na⁺ ions flow into the membrane, causing the membrane potential rise. Once the membrane potential reaches a threshold, an action potential is triggered, accompanied by the generation of a firing spike. The time to the first spike, as well as the firing rate, is regulated by the intensity of external stimuli. (b) Bio‐inspired near‐sensor computing architecture, where photosensitive and pressure sensitive resistors mimic the function of biological sensory cells, with memristor units used for coding and processing. (c) Rate and TTFS encoding methods present in biological retinas for multimodal SNN. The rate coding process fixes the time window, resulting in high latency. In contrast, the TTFS coding process fully preserves temporal information, exhibiting low latency characteristics.

To construct a spike‐based perception system inspired by human sensory processing, a promising strategy is to employ sensor arrays to capture multisensory signal patterns and post‐processing units for information classification and recognition [13, 14]. However, sensory data are typically unstructured and redundant in the analog form, thus necessitating the conversion to digital format via analog‐to‐digital converter (ADC) and voltage‐to‐spike conversion (VSC) units when processed within conventional von Neumann architecture, which relies on physically separated storage and processing units [15]. The explosive growth of sensor nodes, coupled with a large number of ADCs and frequent data transfers, leads to area burden and power consumption dilemma [16]. Near‐sensor computing architectures and neuromorphic devices have therefore emerged as promising approaches to overcome these bottlenecks by integrating sensing and processing units tightly (Figure 1b). In this context, hardware devices capable of intrinsically generating spike signals are essential for realizing artificial neuron functionalities. As a representative Mott material, NbO₂ exhibits rapid phase transition properties in the presence of electric fields and Joule heating [17], enabling the emulation of spontaneous oscillatory behavior of biological neurons [18]. When combined with sensing elements whose resistance varies in response to pressure or light stimuli, such systems enable direct acquisition of multisensory information. Building on this principle, NbO_x‐based memristor have been reported to emulate human visual neurons and achieve low‐latency perception with response times down to 1.04 µs [19]. Multisensory perception of temperature and pressure has also been demonstrated to further improve object classification accuracy by leveraging complementary sensory information [20]. Collectively, the incorporation of memristors with sensing units eliminates redundant components, faithfully simulating the biological receptor and afferent nerve functions that transmit information with sparse spikes. This strategy not only exhibits improved biological plausibility but also facilitates high energy efficiency and future integration.

Progress has been made in neural circuits to emulate the plasticity of biological neurons and encode information from the external world through firing rates [15, 21, 22]. However, the composition of the neural circuit employs external capacitors to facilitate signal accumulation and membrane potential built‐up [20, 23], which increases the effective integration time constant, slows voltage accumulation, and consequently delaying the generation of the first spike. Consequently, rapid stimulus to spike conversion becomes difficult to realize in hardware neural system. Conventional rate based coding schemes, which typically extract features by averaging spike counts over fixed temporal windows [14, 24], fail to utilize spatiotemporal information properties and introduce additional latency [25, 26] (Figure 1c). Meanwhile, in most studies, neural devices function as spike encoders without investigating the leaky‐integrate‐and‐fire (LIF) neural model [27]. These limitations highlight the need for hardware neural systems capable of efficiently implementing spike‐based encoding schemes that jointly exploit firing rates and spike timing information, thereby enabling more biologically realistic perception and decision‐making processes.

In this work, we report a bio‐inspired multisensory perception neuron with outstanding low‐latency characteristics, which integrates TTFS and firing rate encoding strategies. Self‐oscillating NbO_x artificial neural devices are employed to enable multisensory encoding of tactile and visual stimuli. The proposed neuron device exhibits ultra‐low and controllable time delays (0.26–4 µs) together with high spiking rate characteristics (0.1–5 MHz), surpassing those of previously reported artificial neurons. Such extremely low latency and high firing rate, satisfying the demand of rapid response and precise decision‐making in dynamically realistic environments. Moreover, the combination of rate and TTFS coding demonstrated higher recognition accuracy and better noise robustness, achieving superior performance on the standard CIFAR‐10 dataset. Finally, by acquiring multisensory information through near‐sensor architecture, the proposed system enables multisensory decision‐making behavior, thereby enhancing the perceptual capability of electronic systems. These results suggest a promising pathway toward low‐latency autonomous driving and the development of multisensory embodied intelligence platforms.

2. Results

2.1. Self‐Oscillation Artificial Spiking Neuron

Amorphous NbO_x films are deposited on Indium Tin Oxide (ITO)/glass substrates by magnetron sputtering, detailed process can be found in the method sections. The conductive ITO film serves as the bottom electrode, while the top electrode consists of a 30 nm thick platinum (Pt) layer deposited via magnetron sputtering. A rectangular functional area with a length of 50 µm is fabricated (Figure 2a). X‐ray diffraction results show no characteristic peaks of NbO_x, confirming that the deposited film is amorphous (Figure S1). Figure 2b shows the cross‐sectional view and elemental distribution, revealing the layered structure of Pt/NbO_x/ITO/glass. The scanning electron microscopy (SEM) image shows a sharp interface between the NbO_x and the electrode layers. Energy dispersive spectroscopy (EDS) elemental mapping clearly shows distinct spatial distributions of platinum, niobium, indium and oxygen with well‐defined boundaries. To further determine the composition of the film at the atomic scale, X‐ray photoelectron spectroscopy (XPS) reveals mixed oxidation (Figure S2). Quantitative fitting of the relative atomic percentages revealed Nb⁴⁺ as the dominant component. Atomic force microscopy discloses a flat surface with a root‐mean‐square roughness of 2.3 nm (Figure S3).

FIGURE 2.

Low‐latency neuron with rate and TTFS coding. (a) Structural schematic of NbOx memristors for neuronal functions, with Pt as the top electrode and ITO as the bottom electrode. (b) Cross‐section SEM image and EDS mapping of Pt/NbO_x/ITO/glass memristor, featuring a clearly defined layered structure. (c) I–V characteristics of the NbO_x memristor, demonstrating stable threshold switching behavior. Voltage was swept from 0 to 3.5 V and back to 0 V over 128 cycles with a compliance current of 1 mA. (d) The experimental results demonstrate varying time delays and firing rates, with identical voltage amplitude inputs but differing load resistances (5, 6, 7, 8, and 9 kΩ respectively). (e) Statistical relationship of the exact timing of the first spike with the load resistance and the magnitude of the input voltage. (f) Comparison of the time delay with previously reported neuron devices under the TTFS encoding strategy. (g) Experimental realization of the all‐or‐nothing firing pattern. (h) Experimental implementation of the leaky, integrate and firing behavior. (i) Comparison of simulation results with experimental results revealing consistent threshold voltages.

DC scanning experiments are conducted on the fabricated neural devices at a compliance current of 1 mA. After the electroforming process, the device exhibits classical threshold switching (TS) behavior (Figure S4a). This is attributed to the formation of localized NbO₂ crystalline regions during scanning, exhibiting Mott‐type phase transition properties [18, 28]. During the voltage sweep, when the voltage bias applied to the device exceeds the threshold voltage (V_th = 2.7 V), the steep increase in current indicates that the memristor has transitioned from a high‐resistance state to a low‐resistance state. As the sweeping voltage decreases, the device remains in a low‐resistance state until the voltage falls below the holding voltage (V_h = 1.8 V), which triggers a rapid transition back to the high‐resistance state. In addition to voltage sweeps, the I–V characteristics under current sweeping are determined. With increasing scanning current, an S‐type negative differential resistance (NDR) is observed (Figure S4b). 128 cyclic tests were performed on the device in the voltage sweep mode shows excellent stability of TS characteristics (Figure 2c). Additionally, the mott phase transition endows the device with rapid switching capability, requiring about 100 ns to transition from the off state to the on state and less than 60 ns to return to the off state (Figure S5a). The endurance is assessed by repeated pulse voltage measurement using a pulse width of 0.25 µs cycled 10⁵ times (Figure S5b). When a constant current of 1 mA is applied to the circuit (Figure S6a), the voltage across the NbO_x memristor is monitored in real time using a high impedance (1 MΩ) oscilloscope channel. The voltage across the device exhibits a spontaneous oscillatory behavior, hopping between the V_th = 2.7 V and the V_h = 1.8 V. Within the 40 ms experimental window, the device cycles between high‐resistance and low‐resistance states 7 × 10⁴ times, demonstrating exceptional stability, boding well for self‐oscillating neural circuits (Figure S6b). Beyond the single device level, statistical analysis is performed on the uniformity and cycle stability between different devices, with 10 cyclic tests performed on 5 randomly selected devices (Figure S7). The result demonstrates that the devices exhibit good stability and uniformity, making it promising candidates for integration into physical networks.

2.2. TTFS and Rate Coding Based on Low Latency Artificial Neuron

Compact neural circuits are constructed using the NbO_x device to simulate spike firing behavior in the voltage input mode (Figure S8a). Real‐time monitoring of the bias voltage across the device and current spike signals within the circuit reveals a biological neuron‐like information transmission process. When the load resistance (R_L) in the neural circuit is fixed, the amplitude of the input voltage modulates the oscillation frequency of the voltage across the device and the rate of current spikes in the circuit (Figure S8b–e). The oscillation range of the voltage across the device matches the threshold switching voltage during the scanning process, with the voltage oscillation synchronized to the current firing frequency. As shown in Figure S9a, the spike firing behavior in neural circuits under different R_L is presented and statistically analyzed. The firing behavior is jointly governed by the input voltage amplitude and R_L, with higher spike firing rates occurring at higher voltage amplitudes and lower R_L, while lower spike firing rates occur conversely (Figure S9b,c). While neuronal plasticity and firing behavior are experimentally demonstrated in the above process, only the spike firing rate information of the device is used. However, more crucial temporal‐spatial relevance traits in biological neurons remain inadequately demonstrated, resulting in the spatiotemporal dynamics characteristics still being fragmentary [29]. In order to more faithfully simulate the features of biological neurons, the spatiotemporal correlation between external stimuli and spike firing is studied, while simultaneously achieving both rate and TTFS encoding for external information.

Monitoring methods are adjusted to experimentally verify that external stimuli influence the spike firing time and rate (Figure S10a). With the R_L fixed, the time delay of the first spike significantly shortens, accompanying the increase in firing rates (Figure S10b–f), when the intensity (voltage amplitude) of the external increase. As shown in Figure 2d,a fixed input voltage bias of 6 V is applied to enable active control over the frequency and temporal delay of firing spikes by adjusting the R_L in the circuit, which mimics the behavior of biological neurons encoding information with temporal and spatial features. Figure 2e summarizes that the time delay is jointly influenced by both the input voltage and load resistance, indicating that TTFS is proportional to the voltage amplitude and inversely proportional to the load resistance. Compared with relevant artificial neuron works (Figure 2f), we demonstrate not only the coexistence of frequency and TTFS encoding [30, 31], but also the lowest time delay and highest frequency encoding scheme (Table S1). In addition to experimentally realizing periodic firing characteristics, the all‐or‐nothing firing behavior is explored (Figure 2g). More firing patterns of biological neurons are demonstrated without requiring additional circuitry complexity (Figure S11). Apart from encoding external information and simulating the firing patterns of biological neurons. More importantly, the LIF neuronal model has also been validated using our device (Figure 2h). In biological neurons, the membrane potential gradually increases, with the accumulation of the stimulus, the spike firing behavior occurs once the threshold is reached (Figure S12a). In the neural circuit show in Figure S12b, we realistically simulate the evolutionary dynamics of biological cell membrane potentials during information transmission (Figure S12c). To demonstrate the stability of our neural devices in simulating LIF dynamics, we performed over 10⁴ repeated cycles (Figure S13). By integrating sensing and processing capabilities, this device can homogeneously implement both spiking encoder and LIF neuron functions (Figure S14), facilitating the hardware deployment of spiking neural networks.

The abrupt transition in memristor conductance can be attributed to insulator‐to‐metal transitions (IMT) within the NbO₂ filament region [32]. A stable TS process can be explained by the Poole‐Frankel (PF) mechanism dominated current increase combined with Newton's cooling law, where the strong electric field and Joule heating contribute to a localized temperature rise in the device, reaching the phase transition critical threshold of NbO₂ [33, 34]. During reverse voltage sweep, the progressive reduction in Joule heating allows thermal dissipation to dominate, leading to a temperature drop that destabilizes the metallic states and cause the device to revert to the high resistance state. The resistance of the filaments in the phase‐change region can be described as [28]:

$${\mathrm{R}}_1={\mathrm{R}}_0\exp \left[\frac{1}{{\mathrm{k}}_{\mathrm{B}}\mathrm{T}}\left({\mathrm{E}}_{\mathrm{a}}-q\sqrt{\frac{q\mathrm{E}}{\pi {\epsilon }_0{\epsilon }_{\mathrm{r}}}}\right)\right]$$ (1)

where R₀ is a constant, k_B is the Boltzmann's constant, T is the temperature, E_a is the electronic activation energy, E is the electric field intensity, and ε₀ and ε_r are the vacuum permittivity and relative permittivity, respectively. The detailed values for each parameter can be found in the supplementary materials. Based on this theory, simulation of the circuit system for the NbO_x based memristor is performed with the simulation program with integrated circuit emphasis (SPICE). Figure S15 illustrates the principle and schematic diagram employed in the simulation. The simulated results exhibit good agreement with the experimental results for V_th and V_h (Figure 2i).

The mechanism of the simulation process is consistent with the previously reported NbO_x memristor core‐shell model [35, 36]. Based on the model, the influence of the capacitor values for spike firing in neural circuits is simulated (Figure S16). TTFS exhibits an exponential decay as the parallel capacitance value decreases, accompanied by an increase in the firing rate. Hence, ultra‐low latency spike firing can be accomplished by reducing the capacitance in the neural circuit. Parasitic capacitance is a common phenomenon in memristors [37], enabling the design of neurons without the need for additional capacitors. When utilizing the inherent parasitic capacitance of the device, which is determined by its intrinsic properties, the neural circuit exhibits a minimum capacitance. Therefore, the NbO_x memristor can be regarded as a parallel combination of the TS device and a capacitor (Figure S17), with the value of the parasitic capacitance determined based on the theory of equivalent parallel plate capacitors (63 pF). When the voltage bias of the device is lower than the V_th, the memristor remains in a high‐resistance state, forming a charging circuit with the parasitic capacitance. However, as the voltage exceeds the phase change threshold voltage of NbO₂, the low‐resistance state yields a smaller time constant, accompanied by the generation of a current spike. Owing to the voltage division of the load resistor combined with the TS characteristics of the device, cyclic transitions between two circuit states are possible, enabling self‐oscillating neural circuits. To verify that devices with the same structural dimensions have the same parasitic capacitance, resulting in consistent delay times. We tested the nine devices in a 3 × 3 array and observed that the timing of the first firing spike was consistent when only parasitic capacitance was utilized (Figure S18a). The effect of external capacitance on the time delay between the stimulus and the firing spike was investigated, revealing that adding additional capacitance on the parasitic capacitance results in an increase in the delay time (Figure S18b). Overall, based on simulation and theoretical results, utilizing the parasitic capacitance effect promises to imitate the low latency TTFS encoding and high rate firing properties of biological neurons.

2.3. Multisensory Characteristics of Low Latency Perception Neuron System

To achieve a more comprehensive perception of the external environment, inspired by human information acquiring patterns, sensory data collected by different types of receptors is transmitted via nerve fibers in the form of electrical spikes to relevant regions of the cerebral cortex for further processing, tactile visual multisensory low‐latency neural systems are hardware implemented. In traditional perception systems, data collected by different types of sensors is unstructured and redundant, requiring preprocessing by a series of ADC and VSC units before being used in processing networks. A neuron system based on a near sensor architecture, which consists merely of sensors and memristors, eliminating the additional circuitry cost and reducing integration complexity compared to traditional architectures. Replacing the R_L with a photo‐resistor enables the neural circuit to acquire visual information without increasing complexity or component count, as this operation merely substitutes a necessary component in the voltage driven mode (Figure 3a). Biological visual systems sense the external world through the timing and frequency of spikes. When the intensity of external light stimuli increases, the firing rate increases, accompanied by a shortening of the first spike latency. Figure S19 illustrates the characteristics of the photoresistor we employed, which exhibits a broad range of resistance variability under light stimulation across the red, green, and blue spectral ranges, giving it the ability to simulate the visual perception systems of biological in the visible light spectrum.

FIGURE 3.

Low latency encoding for the multisensory information. (a) Schematic of artificial visual perception neurons. Photoresistor replaces the load resistor in the circuit, connected with a memristor to sparsely encode light information. Channel 1 of the oscilloscope monitors the input voltage in the circuit, while channel 2 displays the current oscillation. (b) Even with a constant input voltage, the oscillations and time delays in the circuit exhibit significant differences under varying light conditions. (c) The statistical relationship between the effect of blue light intensity on the frequency and time delay of spike firing. (d) Under constant input voltage (6 V, 7 µs), the experimental results of oscillation in the circuit under dark, 450, 520, and 635 nm light stimulation are presented respectively. (e,f) Real‐time encoding demonstration for the letters B, U, A, A, utilizing a 3 × 3 array of photoresistors connected to niobium oxide memristors. (g) Schematic of an artificial tactile neuron, with pressure sensors replacing load resistors. (h) Maintaining a constant pressure level, the input voltage duration of 5µs with amplitudes of 4, 5, 6, and 7 V, which are accompanied by variations in the firing rate and first spiking time within the circuit. (i) Under a pressure maintained at 0.66 N, statistical results of oscillation frequency and first spike firing time versus input voltage.

With the dynamic variations in blue light intensity, the light‐sensitive resistor value changes accordingly, dynamically modulating the rate and timing of neuronal firing activity in real time, even when the input voltage bias of the neuronal circuit remains constant (Figure 3b). Statistical results indicate that the exact timing of the first spike exhibits a nonlinear correlation, while the firing rate shows a linear relationship with light intensity (Figure 3c). Both TTFS and rate encoding approaches are realized for light information, exhibiting a maximum 4 MHz firing rate and the lowest 0.37 µs time delay for blue light stimulation. Besides the perception of blue light information, experiments verify the spatiotemporal encoding for red and green light information (Figure 3d). Figures S20–S22 respectively present the detailed spike characteristics and statistical results under 635, 520, and 450 nm light stimulation. Even when exposed to light stimuli of the same intensity, organisms exhibit different responses due to variations in pupil size. In order to simulate biological adaptation to light environments, a constant light environment is employed, and varying light sensitivities are obtained by adjusting the voltage bias of neural circuits (Figure S23). With a higher input voltage, the firing rate increases, accompanied by a shorter TTFS. Active adaptation to the environment is simulated, which allows the enhanced photosensitivity under low‐light conditions. To demonstrate the image sensing and encoding capabilities of the neural perception system, a 3 × 3 pixel array is constructed using photoresistors (Figure 3e), achieving sensing and encoding for BUAA. We use different photomasks with varying letter shapes to achieve selective illumination of the photoresistor, and use an oscilloscope to detect current spikes in the circuit. Each letter corresponds to a different number of photoresistors being exposed to light, which results in different loads in the circuit. When a 7 V input voltage is applied to the circuit, each letter pattern exhibits a different firing rate and delay time. By utilizing these different delay times, we can distinguish between the letters, which facilitates low‐latency information transmission and recognition (Figure 3f).

Simulation of the biological tactile behavior is validated using a pressure‐sensitive resistor (Figure 3g), further enhancing the information sensing capabilities of artificial neurons. The characteristics of the commercial pressure sensors are shown in Figure S24, where the resistance changes under different pressure conditions. In addition to perceiving different intensities of external stimuli, organisms can also regulate their attention mechanisms to produce varying levels of response to the same stimulus. As shown in Figure 3h, the biological attention mechanism is simulated by modulating the voltage bias amplitude to enable dynamic control of the rate and delay time while maintaining a constant pressure level. Here, stronger attention corresponds to higher bias voltage, resulting in a more robust response for the same pressure level, meaning a higher firing rate and lower latency. Figure 3i summarizes the relationship between frequency and TTFS coding with different levels of attention, demonstrating the varying response to the same stimulus in organisms. At a fixed attention intensity, which means a constant voltage bias, the firing behavior of neural circuits is verified under different pressure conditions (Figure S25a). When no pressure is applied to the resistor, no spike signals are generated in the neural circuit. With increasing pressure, the rate of the spike increases linearly, while the latency time of the first spike shortens nonlinear (Figure S25b).

2.4. Rate and TTFS Coding for Image Recognition

In the human retina, firing rate and temporal delay jointly encode external stimuli to enable rapid and precise perception [38, 39]. Inspired by this mechanism, NbO_x memristors and sensing units are combined to realize dual coding visual information processing within a spiking neural network (SNN) framework. For image classification tasks based on feed‐forward dense and convolution architectures, a hybrid strategy combining pre‐training and fine‐tuning is adopted on large‐scale datasets such as CIFAR‐10, allowing the SNN to achieve performance comparable to artificial neural networks. Specifically, the pre‐trained weights from a VGG16 ReLU model are losslessly mapped to the SNN [40], following the implementation method described in ref. [40]. Subsequently, through gradient‐based fine‐tuning to obtain a TTFS based SNN for image recognition [41]. Benefiting from the extremely low encoding latency and fast temporal processing enabled by NbO_x devices, the proposed TTFS SNN is well‐suited for image‐based target recognition in rapidly changing scenarios.

Figure 4a illustrates the overall workflow of the fine‐tuned TTFS based SNN. Inspired by biological retinal processing, both temporal latency and firing rate characteristics are extracted from external visual stimuli to jointly represent input information. The ability of our neural device to perceive and encode light information across the red, green, and blue spectral regions has been demonstrated, and the mapping relationships between light intensity with firing rate and TTFS have been established respectively. For each pixel of the input image, RGB channel intensities are independently normalized to the range [0, 1]. Based on the relationship between firing spike and light intensity, two complementary representations are generated. In the TTFS encoding strategy, higher pixel values correspond to earlier spike times according to a nonlinear intensity‐latency relationship. In parallel, under the rate coding scheme, higher pixel values correspond to higher firing rates, determined by the linear mapping relationship between intensity and rate. Instead of explicitly generating discrete spike trains at the input layer, the encoded spike timing and firing rate information are represented as continuous‐valued features, corresponding respectively to spike latency and equivalent firing activity. In the fusion encoding strategy, these two representations are normalized to a consistent numerical range and combined through a weighted summation, forming a unified input feature matrix. This fused representation simultaneously preserves temporal (latency) and frequency related information derived from neural spiking behavior. The fused input is then fed into a TTFS based SNN, whose internal neuron dynamics are governed by spike generation and propagation in the time domain. Within the network, neuron states and inter‐layer signal transmission are determined by firing time computations rather than conventional activation amplitudes. Importantly, the rate‐derived component is not explicitly converted into spike sequences. Instead, it modulates neuronal firing behavior through the unified latency‐domain representation. During training, network parameters are fine‐tuned via gradient descent, allowing the model to adaptively exploit discriminative features originating from spike timing, firing rate, or their combination.

FIGURE 4.

Demonstration of the superior performance of rate and TTFS coding fusion in SNN. (a), Schematic diagram of image classification using SNN, with different strategies employed for feature extraction. (b) Firing behavior of the output layer neurons represented as a raster plot. The horizontal axis corresponds to discretized firing time bins for visualization, while the vertical axis denotes output neurons associated with different classes. For the illustrated input image belonging to class 3, the corresponding output neuron fires first, consistent with the TTFS decision rule. (c) Recognition accuracy under different encoding strategies, demonstrating that the fusion mode is more applicable to image classification tasks. (d) The confusion matrix for classifying ten object categories demonstrates excellent feature learning capabilities. (e), Loss values of different encoding methods during SNN training. (f) Summary of the trend in the influence of image noise on recognition accuracy for different encoding methods.

Figure 4b shows the firing characteristics of the ten neurons in the output layer at different time steps, revealing a significant sparse spike distribution. In SNN, not all the neurons are active at any moment, with merely a partial neuron being activated, a trait that facilitates lower power consumption. For the final classification of input information, the label corresponds to the output layer neuron that first exhibits spike firing behavior. The recognition accuracy of the bio‐inspired visual perception system based on NbOx has been validated with different encoding schemes (Figure 4c). While high recognition accuracy can be achieved with either single firing rate coding or TTFS coding, yet the weighted fusion of these two coding methods demonstrates even higher recognition accuracy. Figure 4d shows the confusion matrix during the classification process, indicating that each category of image information was correctly classified. Hybrid encoding schemes demonstrate superior performance during classification, which is also demonstrated by the loss function during training (Figure 4e). Combining linear and nonlinear encoding methods in a hybrid coding scheme facilitates the network's learning of task features, enabling higher recognition accuracy. Furthermore, with the introduction of Gaussian noise, the recognition accuracy of networks using a single encoding method exhibits a rapid degradation. In contrast, the hybrid mode shows superior noise robustness, consistently maintaining the highest recognition accuracy without any noticeable degradation of performance (Figure 4f).

2.5. Visual‐Tactile Perception for Multisensory Recognition

Visual and tactile information play an irreplaceable role in human interactions with the physical surroundings, with the fusion of sensory information facilitating comprehensive perception of the external environment [42], even yielding unexpected enhancements in interpretation. Therefore, multisensory perception and recognition properties of our device are validated on a Braille dataset by utilizing tactile and visual sensory information. A multisensory fusion model based on an SNN is constructed, where memristors perform sparse encoding of external information and function as LIF neurons. Braille images are classified by simulating the information processing methods of biological neural systems. In physical environments, which are affected by the presence of obstacles and lighting conditions, relying solely on visual information for interaction with the external environment is not sufficiently reliable [43]. With the assistance of tactile information, the scope and depth of information perception can be expanded, thereby outperforming single channel information acquisition in terms of reliability (Figure 5a). Therefore, with the cooperation of the multisensory assistant system we propose, individuals with visual disabilities can achieve low‐latency perception when navigating complex physical environments. The fusion of multisensory information is expected to further enhance the interaction between artificial intelligence and the real world, driving the progress of embodied intelligence.

FIGURE 5.

A Multisensory assistance system based on SNN for Braille recognition. (a) Light sensors and pressure sensors assist the blind in obtaining Braille image information for high‐precision recognition. (b) Schematic diagram of the 24 standard Braille English letters for classification. (c) When the letter “h” serves as input information for the SNN, the corresponding neuron in the output layer exhibits the highest number of firing behaviors. (d) Accuracy of the recognition process for Braille datasets based solely on visual and tactile information. (e) When obstacles are exist, the visual method can only obtain partial information about the image. The left half of the image relies on visual information to read features, while the right half relies on tactile senses to obtain information. (f) Confusion matrix for classifying 24 categories of letters in the multisensory recognition process. (g) Accuracy of recognition based on visual, tactile, and multisensory information in the presence of obstacles.

We utilize a Braille dataset comprising the universal 26 English letters (Figure S26), with 55 images per letter category, to validate the performance of the SNN‐based assistive system. The training set (80%) and test set (20%) are divided using stratified sampling to ensure each letter category maintains the same proportion to avoid data imbalance. Each image contains 32 × 32 pixels (Figure 5b), mapping the grayscale value of the image through the relationship between firing rate and light intensity or pressure. The encoded sparse spike sequence serves as the input layer information for the SNN, which will be used for network inference and recognition. We first perform image recognition on the Braille dataset using visual and tactile information separately to evaluate the classification performance of the SNN. Figure 5c illustrates the firing behavior of neurons in the output layer during the recognition of the letter “h” based on visual information. Only a fraction of neurons is activated and exhibit firing behavior. The accumulated number of firings for each neuron is counted, with the label corresponding to the neuron with the highest firing numbers serving as the network's recognition result. A classification accuracy exceeding 80% can be achieved when complete image information is available, whether based on tactile or visual information. This demonstrates that the SNN constructed based on the characteristics of our device has outstanding learning and classification capabilities (Figure 5d).

However, the real world presents more complex situations. Poor lighting conditions or the presence of obstacles often hinder our ability to accurately recognize targets [42]. To simulate this challenging scenario, the task is designed such that 50% of the image information can only be gathered through visual methods, while the remaining portion can only be perceived through tactile. For example, only partial information of the Braille graphic is perceived when obstacles are present, leading to confusion in classification results (Figure 5e). Therefore, the process of recognition relying solely on information from a single sense encounters dilemmas, which are well resolved when two types of sensory information are fused. Figure 5f demonstrates the confusion matrix for the hybrid mode, revealing that each letter can be correctly classified. In contrast, when relying solely on a single sensory mode, significant errors occur in classifying part of the letters (Figure S27). The recognition accuracy also demonstrates the superiority of multisensory information fusion. For images with partially obscured features, only about 60% recognition accuracy is achieved using either visual or tactile features alone. 90% accuracy is attained when combining both sensory modalities (Figure 5g), validating the enhanced performance of multisensory recognition.

3. Conclusions

A low‐latency multisensory neuron device with a minimum first spike delay of 260 ns, in which LIF neuron dynamics and spatiotemporal encoding of visual and tactile information are integrated within a compact neuron circuit is described. Based on PF theory and Newton's cooling law, the TS behavior of the NbO_x memristor is systematically simulated at the circuit level. The simulation results reveal that neuron firing properties are critically governed by the effective capacitance, indicating that leveraging the intrinsic parasitic capacitance of the NbO_x device enables low‐latency firing. By incorporating pressure‐ and photo‐sensitive resistors into the neuron circuit, the device is capable of encoding different levels of light intensity and pressure through both TTFS and firing rates. An image classification SNN integrating TTFS and rate information at the input layer is implemented and tested on the CIFAR‐10 dataset based on the characteristics of the proposed neuron device. Fusion of TTFS and rate features improves classification accuracy, achieving a recognition accuracy close to 90%, which is higher than that obtained using single‐feature inputs. Moreover, variations in classification accuracy under different noise levels demonstrate that combining the two types of features enhances the noise robustness of the image classification process. Furthermore, the effects of multisensory information fusion on recognition accuracy are studied by constructing a bio‐inspired visuotactile assistive system that mimics the human cognitive behavior enabled by multisensory perception. The integration of tactile and visual information enriches sensory perception and supports accurate decision‐making in environments where information from a single sensory modality is limited.

4. Methods

4.1. Sample Preparation

At room temperature, a 100 nm thick NbO_x film was deposited on the ITO/Glass substrate by magnetron sputtering using polycrystalline NbO₂ as the target material. The deposition was carried out at a pressure of 0.5 Pa using an argon‐to‐oxygen flow ratio of 40:1. The top electrode (Pt) was deposited by magnetron sputtering using a hard mask with a length of 50 micrometers.

4.2. Materials Characterization

X‐ray diffraction was performed on the NbO_x films on a Bruker D8 Advance instrument with a 2θ range from 10 to 80° in steps of 0.03°. The cross‐sectional SEM images and EDS maps were acquired on the ZEISS Gemini SEM 300. Atomic force microscopy was performed on the commercial Asylum Research MFP‐3D probe microscope and XPS was conducted on the ESCA Lab 250Xi.

4.3. Device Characterization

The DC mode was measured by Keithley 4200 semiconductor parameter analyzer. The optoelectronic measurements were performed by using semiconductor lasers with various wavelengths of 450, 520, and 650 nm. A Tektronix AFG3100 arbitrary function generator was used to produce µs wide voltage pulses during testing of the neuronal firing behavior. Real‐time monitoring of dynamic variations in currents and voltage in the circuits was through the Tektronix TBS2000 digital oscilloscope. Pressure sensing was carried out using the S14‐G flexible thin film pressure sensor (CHENGTEC), while light perception is accomplished by a cadmium sulfide photoresistor (DZYJ‐5516).

4.4. SNN Processing Framework fo the CIFAR‐10

For each color channel of every pixel, the TTFS encoding value $T_{ij}^{(c)}\in [0,1]$ and the frequency encoding value $R_{ij}^{(c)}\in [0,1]$ are calculated separately, and then fused according to their respective weights:

$$V_{ij}^{(c)}=\alpha T_{ij}^{(c)}+(1-\alpha )R_{ij}^{(c)},\alpha \in [0,1]$$

The fused value is interpreted as the time of a spike being fired by the input neuron corresponding to that pixel.The SNN model used in this task adopts a TTFS encoding scheme, where each neuron fires at most once and the precise timing of the spike encodes the information. The network architecture is a feed‐forward structure. The membrane potential $V_i^{(n)}(t)$ of neuron i in layer n follows:

$${\tau }_c\frac{d{V}_i^{(n)}}{dt}=\left\{\begin{array}{c}{A}_i^{(n)}+\sum _j{W}_{ij}^{(n)}H(t-t_j^{(n-1)}),t<t_{min}^{(n)}\\ B_i^{(n)},t_{min}^{(n)}\le \mathrm{t}\le t_{max}^{(n)}\end{array}\right.$$

Here,τ _c  is a time scaling factor, $W_{ij}^{(n)}$ are the synaptic weights, $t_j^{(n-1)}$ are the spike times from the previous layer, H is the Heaviside step function.

4.5. SNN Processing Framework fo the Braille Dataset

To provide a clearer description of the SNN used, we have included the relevant details in the main text. For recognition on the Braille dataset, we constructed a network with an input layer containing 1024 neurons, three hidden layers containing 512, 256, and 128 neurons, respectively, and an output layer containing 26 neurons. The neurons in the network employ a discrete LIF model, whose parameters were extracted from experimental results on NbO_x devices. The equation describing the evolution of the membrane potential over time is:

$${\mathrm{V}}_{\mathrm{mem}}\left(\mathrm{t}+1\right)=\beta {\mathrm{V}}_{\mathrm{mem}}\left(\mathrm{t}\right)+\sum _{\mathrm{i}}{\mathrm{W}}_{\mathrm{i}}^{\mathrm{T}}{\mathrm{X}}_{\mathrm{i}}(\mathrm{t})-\mathrm{R}·{\mathrm{V}}_{\mathrm{th}}$$

Here, V_mem(t) represents the membrane potential of neuron at time step t, β is the decay rate of the membrane potential.

$$\mathrm{R}=\left\{\begin{array}{c}\mathit{1},\mathrm{if}{\beta }_{\mathrm{mem}}\left(t\right)+\sum _i{W}_i^{\mathrm{T}}{X}_i\left(i\right)>{\mathrm{V}}_{\mathrm{th}}\\ \mathit{0},\mathrm{if}{\beta }_{\mathrm{mem}}\left(t\right)+\sum _i{W}_i^{\mathrm{T}}{X}_i\left(i\right)\le {\mathrm{V}}_{\mathrm{th}}\end{array}\right.$$

Since the impulse function is not differentiable, this paper employs a surrogate gradient method to implement backpropagation. In backpropagation, the gradient of the step function is approximated as a continuous function:

$$\frac{\partial \mathrm{S}}{\partial \mathrm{V}}\approx \frac{1}{{\left(1+\mathrm{V}-{\mathrm{V}}_{\mathrm{th}}\right)}^2}$$

The loss function takes the form of labeled smooth cross‐entropy:

$$\mathrm{L}=-\sum _{\mathrm{i}=1}^{26}\stackrel{\sim }{{\mathrm{y}}_{\mathrm{i}}}\log \left({\mathrm{p}}_{\mathrm{i}}\right)$$

Here, “pi” represents the probability obtained by normalizing the pulse counts in the output layer via Softmax, and $y_i^{smoot\mathrm{h}}$ represents the smoothed label distribution.

Author Contributions

A.H. initiated the research. The sample preparation, the device fabrication, and the device measurement were done by P.L. with support from J.H., Q.Y., and Y.Z. P.L., A.H., and P.K.C. wrote the manuscript. Q.G., Z.X., and X.Z. participated in the discussion on the manuscript.

Conflicts of Interest

The authors declare no conflicts of interest.

Supporting information

Supporting File: smll73955‐sup‐0001‐SuppMat.docx.

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon request.