Biomimetic ferroelectric-semiconductor transistor enables neuronal multisensory integration

Abstract

Human brain seamlessly integrates multisensory stimuli to synthesize complementary information for enhanced perceptions, depending on neural principles of superadditivity, inverse effectiveness, and temporal congruency. Replicating multisensory integration in artificial intelligences has remained challenging due to the inefficiency of algorithmic fusions and the absence of hardware-native mechanisms. Here, we demonstrate biomimetic audiovisual integration at the device level of Bi₂O₂Se ferroelectric-semiconductor field-effect transistors (FeS-FETs) through multiphysics coupling. Our FeS-FETs simultaneously accomplish the superadditive integration factor of 2800%, dynamical reweighting inputs of inverse effectiveness, and prolonged temporal congruency beyond 10³ s. Furthermore, when configured into memristor-chip-based spiking neural networks, the resultant multisensory system is capable of executing the sensory synaptic plasticity, population-coded spiking, and Bayesian-optimal fusion, which promotes the excellent recognition accuracy of 98.2% for fuzzy objects, surpassing that identified from conventional fusion algorithms. By the exploration of multi-physical computing to mirror the biological multisensory hierarchy, we establish a physics-aware framework for neuromorphic multisensory intelligences, bridging physical dynamics with neurobiological principles for self-adaptive edge computing.

Replicating the brain’s multisensory integration is vital for artificial intelligences. Liu et al. have developed biomimetic ferroelectric-semiconductor field-effect transistors to mimick the brain’s audiovisual integration driven by hardware-native multi-physics coupling.

Introduction

Seeing a speaker’s lip while listening to his voice is known to enhance language understanding¹, which exemplifies the computational power of human brain to synthesize disparate sensory modalities into a unified percept through multisensory integration^2,3. This process generates emergent perceptual qualities unattainable from any single modality, leading to improve the perceptual accuracy, accelerate decision-making, and underlay a range of psychophysical phenomena^4–7. Among varied sensory modalities, audio and visual inputs dominate over 90% of our interactions with the external world, underpinning critical cognitive functions from spatial navigation to attentional selection^8–10. Recent advances in neurophysiology have begun to uncover how multisensory neurons in biological brains can integrate audiovisual inputs from synaptic-level plasticity to large-scale cortical interactions^11–13. At its core, multisensory neurons integrate stimuli through three key principles^4,14: superadditivity (i.e., integration responses surpassing the sum of unisensory inputs), inverse effectiveness (i.e., maximal integration for weak unimodal signals, and minimum for strong ones), and temporal congruency (i.e., temporally aligned cross-modal inputs for integration).

Harnessing these multisensory-integration mechanisms could revolutionize artificial intelligence by enabling self-adaptive perceptions for robotics, autonomous vehicles, and smart homes. This has long been a frontier in neuromorphic and electronic engineering, yet facing huge challenges^15,16. Current approaches primarily rely on combining multiple sensors with algorithmic frameworks such as weighted averaging, feature concatenation, and principal component analysis^17–20, whereas lacking the multiplicative nonlinearity and adaptive reweighting intrinsic to biological neuronal networks²¹. While artificial neural networks approximate the superadditivity via nonlinear activation functions, they struggle to autonomously implement the inverse effectiveness that dynamically reweights synaptic inputs based on real-time reliability of signals²². Since biological multisensory neurons exploit the inverse effectiveness through self-tuned thresholds, most algorithmic fusions typically emulate this inverse behavior by hand-engineered rules or complex Bayesian inference layers that impose substantial computational overhead^23,24. Moreover, maintaining the temporal congruency amid dynamically-reweighted inputs is non-trivial¹⁴, as algorithmic synchronizations often conflict with the adaptive processing. These limitations are becoming particularly pronounced in edge hardware, where stringent energy-efficiency requirements are imposed on computational resources.

Recent advances in physical computing offer promising potential to implement neural principles directly in hardware through device physics^25–28, yet to be fully explored. Here, we demonstrate biomimetic ferroelectric-semiconductor field-effect transistors (FeS-FETs) that natively emulate audiovisual integration at the device level. By exploiting the coupling dynamics of ferroelectric polarization and semiconducting photocurrent in the ferroelectric-semiconducting Bi₂O₂Se channel, our FeS-FETs simultaneously implement neural integration features of superadditivity, inverse effectiveness, and temporal congruency within a single compact architecture. In particular, an ultrahigh superadditive integration factor exceeding 2800% is achieved alongside the prolonged temporal congruency over 10³ s. Furthermore, we integrate the FeS-FET array with memristor-chip-based spiking neural networks (SNNs) to construct a hierarchical multisensory recognition system, which replicates versatile neurobiological behavior, including the sensory synaptic plasticity, population-coded spiking, and Bayesian-optimal fusion. This multisensory system reaches an excellent recognition accuracy of 98.2% for fuzzy objects, outperforming conventional unimodal and algorithmic-fusion approaches. Through adopting the intrinsic multi-physical dynamics of Bi₂O₂Se FeS-FETs, our work introduces a hardware-adaptive paradigm for edge-compatible multisensory intelligences.

Results and discussion

Ferroelectric semiconductor field effect transistors

As an emerging oxide semiconductor, Bi₂O₂Se uniquely combines the characteristics of visible-to-near-infrared bandgap, high charge-carrier mobility, and strong spin-orbit coupling^29–31, positioning it as a versatile candidate for multimodal sensing. We have pioneered the disclosure of ultra-sensitive bipolar photoresponse, broad-range temperature detection, and high-resolution bolometric sensing in Bi₂O₂Se FETs^32,33. Structurally, Bi₂O₂Se crystallizes in a tetragonal system (space group I4/mmm), with alternating [Bi₂O₂]²⁺ cationic and Se²⁻ anionic layers stabilized by van der Waals interactions and weak ionic bonding³⁴. This layered configuration allows for tunable atomic displacements that potentially induce ferroelectric polarizations. To exploit this structural flexibility, we deploy strain engineering during the chemical vapor deposition (CVD) of Bi₂O₂Se by applying a low deposition pressure (i.e., 40 Pa). Unlike conventional CVD growths (e.g., 10³−10⁴ Pa)^35–37, a low-pressure CVD extends the mean free path of Bi₂O₂Se precursors, leading to the nucleation with localized strains. And the steep thermal-stress gradient arising from the slowed layer-by-layer deposition also introduces lattice strains, thereby promoting Bi₂O₂Se atomic displacements.

The cross-section atomic distribution of Bi₂O₂Se crystal is characterized by high-angle annular dark-field scanning transmission electron microscopy (HAADF-STEM). The intrinsic Bi₂O₂Se crystal is central-inversion symmetric of point group D_4h without charge polarizations³⁸. HAADF-STEM reveals the reduced in-plane and out-plane lattice ratio a/c, which signifies the tensile out-of-plane stress induced by the low-pressure deposition (Fig. 1a, b). Notably, Se atoms vertically displace about 0.2 Å from their centrosymmetric positions between Bi-O layers. We process the density functional theory (DFT) calculation to estimate that a 6% uniaxial out-of-plane stretch induces around 0.2 Å displacement of Se atoms (Fig. 1c). This structural distortion breaks the central-inversion symmetry of Bi₂O₂Se lattice, thus driving charge redistributions with the electron accumulation near oxygen sites proximal to displaced Se atoms and the corresponding depletion at distal sites (Fig. 1d, e). Layer-resolved charge-polarization profile illustrates that the interlayer polarization extends through Bi₂O₂Se interlayers (Fig. 1f), creating a double-well potential with distinct polarization states separated by the barrier of 23 meV per unit cell (Supplementary Fig. 1). DFT calculations further indicate that the out-of-plane Se displacements yield stronger ferroelectricity (e.g, 13 μC cm⁻²) than that of in-plane Bi-O shifts (e.g., 1.2 × 10⁻³  μC cm⁻²) (Supplementary Fig. 2).

Fig. 1. Characterizations of Bi₂O₂Se ferroelectric semiconductor.

a High-angle annular dark-field scanning transmission electron microscopy (HAADF-STEM) of conventional Bi₂O₂Se crystal, and the inset illustrating the central-inversion symmetry of Se atoms, scale bar is 1 nm; b HAADF-STEM of strain-engineered Bi₂O₂Se crystal, and the inset illustrating asymmetric Se displacements, scale bar is 1 nm; c schematic comparing of conventional (left) and strain-engineered (right) Bi₂O₂Se lattices calculated by first-principles density functional theory; charge-density distributions of (d) symmetric, and (e) asymmetric Bi₂O₂Se crystals; f interlayer charge-polarization profile of strain-engineered Bi₂O₂Se crystal; (g) piezoresponse force microscopy (PFM) phase image and (h) amplitude image of strain-engineered Bi₂O₂Se crystal, scale bar is 2 μm; i bidirectional polarization switching in PFM hysteresis loops; j clockwise hysteresis for 20-nm HfO₂ dielectric thicknesses, and k counterclockwise hysteresis for 10-nm HfO₂ dielectric thicknesses of Bi₂O₂Se ferroelectric-semiconductor transistors; l ferroelectric-conductance switching during cyclic programing/erasing operations.

Piezoresponse force microscopy (PFM) reveals the ferroelectric domains with opposite polarizations in our strain-engineered Bi₂O₂Se, evidenced by clearly distinguishable 180° phase-contrast and amplitude variations (Fig. 1g, h). The phase hysteresis loop and butterfly-shaped amplitude response confirm the reversible ferroelectric-polarization switching (Fig. 1i). We further perform cryogenic electrical characterizations, wherein thermally-activated defect trapping and ionic migration are completely suppressed, and our Bi₂O₂Se FeS-FETs still exhibit distinct counterclockwise hysteresis loops (Supplementary Fig. 3), indicating that the observed hysteresis primarily originates from intrinsic ferroelectric polarization of our strain-engineered Bi₂O₂Se. It is noteworthy that the coupling between channel charges and ferroelectric polarizations of FeS-FETs leads to unique dielectric-thickness-dependent switching dynamics (Supplementary Note 1, and Supplementary Fig. 4)³⁹. Similar to α-In₂Se₃, our Bi₂O₂Se FeS-FETs exhibit the clockwise hysteresis with 20-nm HfO₂, while the counterclockwise hysteresis with 10-nm HfO₂ (Fig. 1j, k). These devices exhibit exceptional FeS-FET performances, including an on/off ratio of 10⁴, cycle endurance exceeding 10⁶ (Fig. 1l), and projected retention times over 10 years at ±5 V programming voltages (Supplementary Fig. 5). It is noteworthy that the device exhibits a systematic reduction in the hysteresis window and a deterioration of the intrinsic ferroelectric characteristics, and further optimizations on manufacturing is underway to improve the device reliability (Supplementary Fig. 6).

Audiovisual integration in Bi₂O₂Se transistors

The combination of ferroelectric and semiconducting properties in our strain-engineered Bi₂O₂Se enables direct audiovisual integration at the device level. Light stimuli generate photocarriers in the Bi₂O₂Se channel via the photoconduction effect, while audio signals are transduced into electrical pulses through a commercial LM386 module, which converts acoustic pressures into gate voltages to regulate the ferroelectric polarization of the Bi₂O₂Se channel (Fig. 2a and Supplementary Fig. 7). This dual-input configuration integrates sensory inputs through the polarization-controlled modulation of photocurrents in-situ the Bi₂O₂Se channel, bypassing the need for complicate fusion algorithms. The audio transduction exhibits an excellent sensitivity of 1.53 × 10⁵ kPa⁻¹ across the audio pressure of 80–110 dB in the frequency range of 10–4096 Hz (Fig. 2b and Supplementary Figs. 8, 9). Concurrently, the photoresponse exceeds 10⁵ A W⁻¹, accompanied by the specific detectivity of 10¹³ Jones and the external quantum efficiency of 10⁷ % in the wavelength range of 400–1200 nm (Fig. 2c and Supplementary Fig. 10a–c). An extended light-off decay indicates the interplay between ferroelectric polarizations and photocarriers within the Bi₂O₂Se channel (Supplementary Fig. 10d).

Fig. 2. Audiovisual integrations in-situ Bi₂O₂Se ferroelectric-semiconductor transistors.

a Schematic architecture of the Bi₂O₂Se transistor for in-situ audiovisual integration; b audio response as a function of sound pressure; c photoresponsivity under varied gate voltages; d audiovisual integration responsivity across varied audio intensities under 200 μW cm⁻² illumination; e audiovisual integration responsivity under varied light intensities at 85 dB sound pressure; f full mapping of multisensory integration factor MIF across stimulus intensities; g temporal congruency of integrated responses over extended durations of 10³ s; h coefficient of variation (CoV) analysis showing the improved signal reliability of audiovisual (VA) integration, as compare with weak unimodal audio (A) and visual (V) stimuli of 85 dB and 200 μW cm⁻², respectively; schematic mechanisms of ferroelectric-polarization-modulated photocurrents within Bi₂O₂Se channel for (i) no polarization, j intermediate polarization, k full channel polarization.

We quantify the integration effect using the multisensory-integration factor (MIF) defined as⁴⁰: $\mathrm{MIF}=\frac{R_{\mathrm{VA}}}{R_{\mathrm{A}}{+R}_{\mathrm{V}}}$, where R_VA, R_A and R_V are the audiovisual, audio-only, and visual-only response currents of the Fe-FETs, respectively. Under weak light (e.g., 200 μW cm⁻²), the photocurrent response is obviously amplified by audio stimuli (e.g., 80–110 dB) (Fig. 2d), corresponding to the superadditive effectiveness. It is noteworthy that the MIF decreases with an increased audio intensity, consistent with the inverse effectiveness that the maximal integration occurs for weak unimodal inputs, and diminishes for strong ones. Similarly, fixing the audio stimulus at 85 dB while varying light intensities (e.g., 67–2870 μW cm⁻²) produces the same superadditive and inverse effects (Fig. 2e). A full MIF mapping across varied audio and light intensities solidates that our FeS-FETs automatically replicate the superadditive and inverse principles of biological multisensory integration (Fig. 2f). Remarkably, audio-induced ferroelectric polarizations persist over 5 × 10³ s, providing a broad time window for integrating with light inputs (Fig. 2g), ensuring that the hardware-level fusion remains robust even if sensory inputs are slightly asynchronous. This prolonged temporal congruency, coupled with the superadditivity and inverse effectiveness, establishes our FeS-FETs as a promising single-device hardware platform for multisensory integration. We also evaluate the cycle-to-cycle stability of FeS-FETs by subjecting the optical responses to repeated auditory set and electrical reset cycles, revealing that the optical and acoustic responses exhibit highly reproducible temporal dynamics (Supplementary Fig. 11).

Biological multisensory integration also follows the Bayesian-optimal principle by weighting sensory inputs according to their reliability to minimize uncertainty in perceptual decisions⁴¹. Our FeS-FETs replicate this through intrinsic device physics to implement variance-weighted fusion (Supplementary Note 2). As the audio intensity increases from 80 dB to 110 dB, its response distribution obviously sharpens (Supplementary Fig. 12a), revealing the device capacity for adaptive noise suppressions. A parallel effect occurs in the visual pathway (Supplementary Fig. 12b). Notably, concurrent audiovisual integration yields a two-order-of-magnitude reduction in the coefficient of variation (CoV)⁴², as compared to unimodal responses under weak inputs (Fig. 2h and Supplementary Fig. 13). This dramatic reliability enhancement provides direct evidence for our device-implemented Bayesian optimality.

These fusion mechanisms arise from dynamic interactions between ferroelectric polarization and photocarrier transport in the ferroelectric-semiconducting Bi₂O₂Se channel, exhibiting three distinct operational regimes. Below the ferroelectric-switching threshold, photocarrier transport depends solely on longitudinal gate fields (Fig. 2i and Supplementary Fig. 14a). Moderate audio intensities induce partial depolarizations in the Bi₂O₂Se channel that enhance both photocarrier separation and transportboth enhance the photocarrier separation and transport (Fig. 2j), thus producing response amplitudes exceeding the sum of unimodal inputs as a clear signature of superadditive integration (Supplementary Fig. 14b). Strong audio stimuli fully polarize the Bi₂O₂Se channel (Fig. 2k), driving the device into the saturation with diminishing marginal gains, thereby tuning to the principle of inverse effectiveness (Supplementary Fig. 14c). This tripartite response profile mirrors the essential features of neural multisensory integration. Crucially, all neural computations are directly processed through the coupled ferroelectric-semiconducting dynamics. In this regard, the optical stimuli generate additional channel carriers through the photoconduction effect, allowing the device to operate at a minimal source-drain bias of V_d = 0.1 V. Under these conditions, the intrinsic dark current, representing the baseline current we must supply, is around 1.5 μA. Thus, the device-level energy consumption is as low as 15 pJ for a 100 μs operational window, providing an efficient physical platform for edge-deployable multisensory processing (Supplementary Table 1).

Multisensory integration with spiking neural networks

Based on the device-level multisensory integration, we construct a complete neuromorphic sensory system by integrating our FeS-FETs with a spiking generation circuit and a memristor-based SNN chip, wherein our FeS-FET array performs the front-edge audiovisual integration through physical computing, the fused responses are then encoded into temporal spike patterns through the circuit, followed by feeding into the memristive SNN for perceptual processing (Fig. 3a). This hardware implementation of multimodal integration, spiking encoding, and memristive computation creates an efficient multisensory hierarchy that closely mimics biological functionalities. A 3 × 3 Bi₂O₂Se FeS-FET array is fabricated as a basic neuromorphic receptive field (Fig. 3b), where each pixel response is simultaneously modulatedmanipulated by the audio-modulated ferroelectric polarizations and the photocarriers. The threshold-triggered spiking generation circuit transforms these analog responses into precisely -timed voltage spikes (Fig. 3c), which are optimized to real-time operations at 50 Hz with an excellent signal-to-noise ratio (SNR) of 60 dB through adaptive threshold controls. This spike encoding preserves the superadditive and Bayesian-optimal features at the device level while transforming them into temporally-coded formats suitable for neuromorphic computation. By employing a shallow SNN, we demonstrate that the hardware excellence of our in-sensor fusion can be directly translated into a lightweight high-performance recognition system that maintains the superior accuracy even when sensory inputs are physically compromised. We note that the implementation of relatively shallow networks verifies the inherent high quality of the pre-processed physical signals, which are expected to be effectively integrated into large-scale intelligent systems.

Fig. 3. Audiovisual integration spiking neural network (SNN).

a Schematic hardware architecture of audiovisual integration system; b scanning electron microscopy image of the 3 × 3 FeS transistor array, scale bar is 20 μm; c schematic threshold-triggered spike generation circuit; d frequency-dependent (0.2–10 Hz) audio postsynaptic currents at 90 dB; e frequency-dependent (0.5–10 Hz) optical postsynaptic under 600 nm illumination of 200 μW cm⁻²; spiking impulses of Bi₂O₂Se transistor array under varied audiovisual stimulus intensities of (f) 90 dB and 200 μW cm⁻², g 105 dB and 500 μW cm⁻², and (h) 110 dB and 2095 μW cm⁻².

Our FeS-FETs exhibit remarkable fidelity to neural multisensory processing through distinct yet complementary plasticity mechanisms in their audio and visual pathways. The auditory modality presents the spike-timing-dependent plasticity (STDP) with frequency-dependent postsynaptic currents (PSCs) (Fig. 3d). High-frequency audio stimuli increase synaptic weights through sustained ferroelectric polarizations, while lower frequencies permit partial relaxations, reproducing the characteristic STDP curves observed in biological synapses. Conversely, the visual pathway displays preferential sensitivity to low-frequency optical stimuli (Fig. 3e), stemming from the inherent timescales of photocarrier dynamics in the channel that cannot fully respond to rapid successive stimuli. The synergy between sensory synaptic plasticity and integrated superadditivity remarkably decreases the response time to reach firing thresholds, for example, 20 optical (e.g., 200 μW cm⁻²) and 50 audio (e.g., 90 dB) pulses are needed to reach the activation threshold (e.g., 10 μA), respectively, while just 3 integrated stimuli can trigger the firing (Supplementary Fig. 15). Eventually, the circuit implementation amplifies the superadditive feature with the maximum MIF of 2880%, largely enhancing the SNR to enable robust recognitions for fuzzy targets (Supplementary Fig. 16). By consolidating these complex functions within a single FeS-FET channel, we eliminate the need for the algorithm-heavy processing in hybrid sensors, providing a transformative and scalable pathway for the next generation of energy-efficient spatiotemporal trajectory tracking technologies (Supplementary Table 2).

Biological neural computation prefers coordinated population networks via the ‘like-to-like’ connectivity^43,44, wherein similarly-activated neurons synchronized to collectively process information. Our multisensory system emulates this population coding through integrating responses across the 3×3 FeS-FET array, transforming individual device responses into meaningful network-level representations. The FeS-FET array exhibits remarkable uniformity with consistent pixel-to-pixel responses to identical stimuli and stable total spiking output (Supplementary Fig. 17), establishing the foundation for reliable integration computing. When subjected to identical optical (i.e., 200 μW cm⁻²), auditory (i.e., 85 dB), and integrated multisensory stimuli, the response amplitudes are nearly identical across all 9 devices, yielding 77 ± 7 nA for optical response, 241 ± 12 nA for auditory response, and 456 ± 67 nA for the multisensory response (Supplementary Fig. 18). The population coding of our multisensory system becomes particularly evident under weak stimuli (e.g., 200 μW cm⁻² light intensity, and 90 dB sound pressure), where synchronized activations appear across all pixels (Fig. 3f). Critically, the integration responses persist beyond stimulus offset due to sustained ferroelectric-photocarrier interactions. This hysteresis effect mirrors the short-term memory retention observed in biological circuits mediated by functionally-connected neural populations^45,46, elucidating how a population network arises from coordinated device interactions rather than simple signal summation. The computational flexibility of our multisensory system further manifests through the dynamic adaptation to varying stimulus conditions. When presented with strong unimodal inputs, the system transitions to an inverse-effectiveness dominated regime, preferentially responding to the more intense single-modality stimulus (Fig. 3g, h). This adaptive gain control precisely replicates the intensity-dependent modulation observed in biological sensory cortices, where neural populations adjust their integration properties according to relative stimulus strengths¹². The seamless transition between integration and suppression modes in our hardware demonstrates how fundamental principles of biological sensory processing, from population coding to adaptive gain control, can emerge from carefully engineered device physics and network architectures.

Multisensory enhanced recognition for fuzzy objects

The recognition layer of our neuromorphic multisensory system employs a TaO_x-based one-transistor-one-resistor (1T1R) memristor chip (Fig. 4a)⁴⁷, wherein each memristor cell encodes 4-bit weight values to enable parallel multiply-accumulate (MAC) operations. The 1-megabit TaO_x memristor array exhibits reliable bipolar switching characteristics with well-defined set/reset states under optimized programming conditions (Fig. 4b and Supplementary Fig. 19a). The memristor cell performs remarkable endurances, maintaining the stable low-resistance and high-resistance switching throughout 10⁴ cycles (Supplementary Fig. 19b), while the pulse-controlled conductance enables reproducible 16-state analog programming (Supplementary Fig. 19c). This multi-level programmability supports the implementation of 4-bit precision SNNs, where the analog tuning of resistance transitions naturally emulates synaptic weight updates.

Fig. 4. Memristor-chip-based multisensory recognition system for fuzzy objects.

a Optical micrograph of the TaO_x-based memristor chip; b cross-sectional scanning electron microscopy of the 1-megabit (1 M) one-transistor-one-resistor (1T1R) memristor array; c current spectral maps of audiovisual integration response of 20 × 20-pixel vehicle images with corresponding acoustic signatures; d architecture of 4-bit spiking neural network implementing convolutional layers for dimensionality reduction and fully-connected layers for classification; e current responses for three vehicle categories of sedan, bus, and truck; f recognition accuracy comparison and (g) noise-robustness comparison among unimodal, algorithmic, and our device-level fusions.

Traditional unimodal recognition systems face inherent limitations in far-field conditions, as both audio and visual resolutions seriously degrade with environmental interferences. Our neuromorphic multisensory system addresses these by implementing the front-end in-sensor-processing integration. This system processes fuzzy 20 × 20-pixel vehicle images synchronized with their corresponding acoustic signatures through direct analog fusions in-situ FeS-FETs, generating the current spectral maps that preserve essential discriminative features (Fig. 4c). These fused spectral maps provide high-quality integration data for subsequent neural recognitions. The recognition architecture implements a 4-bit SNN organized in two computational blocks (Fig. 4d). The first block performs feature extractions through convolutional layers that reduce the dimensionality, while preserving critical spatial information. And the second one consists of fully -connected classification layers that generate probabilistic outputs to distinguish vehicle categories (i.e., sedans, buses, and trucks). Network training are performed off chip via a Python-based framework, while the optimized weights are programmed on chip into conductance states using a write-verify algorithm, achieving ≤5% conductance variations (Supplementary Fig. 20). Statistical analysis confirms the programming precision that over 90% of synaptic weights stay within 0.5 μA of target values (Supplementary Fig. 21a). The memristor array demonstrates ideal analog computing characteristics with an excellent linearity between measured and expected currents across the operational range (Supplementary Fig. 21b).

Rapid developments of stimulus selectivity with the target-channel current are observed, while non-target responses suppress within the training epochs (Fig. 4e). A complete training provides robust classifications through obvious current differentials between vehicle categories. Comparative performance analysis reveals the advantage of our device-based integrations. Conventional unimodal networks show fundamental limitations in the recognition accuracy (i.e., vision-only: 85.7%; audio-only: 88.9%) due to their inherent vulnerability to modality-specific noises (Fig. 4f). A dual-branch SNN that utilizes a late fusion strategy is employed as the algorithmic fusion method, which only provides a slight improvement (i.e., 93.1%). In contrast, our multisensory-integration hardware reaches the remarkable classification accuracy of 98.2% for fuzzy vehicles. In the algorithmic late-fusion approach, if a visual target is blurred, the visual branch independently processes noisy data, often failing to extract meaningful features. By performing this fusion at the sensing stage, our FeS-FET supplies the back-end network with pre-aligned fusion information that is inherently more robust than the disjointed data processed by independent algorithmic branches. To evaluate environmental robustness, zero-mean Gaussian noise N (0, σ²) is injected into the normalized sensory inputs. While unimodal and algorithmic fusion exhibit obvious performance degradations under increased noises (i.e., σ = 0.05), our system maintains a nearly perfect recognition accuracy of 99.8%. Even under heavy noises (i.e., σ = 0.1), where visual-only and algorithmic-fusion accuracies collapse, our device fusion sustains a high recognition accuracy of 95.1% (Fig. 4g). Through integrating these FeS-FET arrays with CMOS-compatible chips, we are actively exploring the feasibility of hierarchical multisensory processing, laying the foundation for more complex large-scale models.

In conclusion, we have demonstrated that the multi-physical coupling in ferroelectric-semiconductor transistors can intrinsically replicate core principles of neural multisensory integration at the device level. Further interconnection with a memristor-based SNN chip unlocks more neural functionalities, including the sensory synaptic plasticity, population-coded spiking, and Bayesian-optimal fusion. These synergies ultimately lead to high recognition performances for fuzzy objects. Our work highlights the promising potential of physical computing for neuromorphic multisensory processing, paving the way to multifunctional, self-adaptive, and energy-efficient edge sensory hardware.

Methods

Low pressure chemical vapor deposition of Bi₂O₂Se

Bi₂O₂Se thin films were grown on fluorophlogopite mica (f-mica) substrates using a homemade low-pressure CVD system equipped with a 1.2 m horizontal tube furnace and a 50-mm-diameter quartz tube. 0.2 g Bi₂Se₃ (Alfa Aesar, 5 N) and 0.4 g Bi₂O₃ (Alfa Aesar, 5 N) were placed at the center of the tube with a separation distance of 15–19 cm, respectively. The growth process began by heating the tube center from room temperature to 640 °C in 25 min. No gas flow was introduced during the initial heating stage. Once reaching the target temperature, 100 sccm of argon was introduced as a carrier gas to transport the vaporized precursors downstream. The mica substrates were positioned 11–19 cm from the heating center, and the system pressure was maintained at ~40 Pa throughout the deposition.

Morphology analysis was performed by atomic force microscopy (AFM, Bruker Dimension Icon with Nanoscope V controller). Transmission electron microscopy (TEM) was conducted on a Thermo Scientific Tecnai F20 operated at 200 kV, enabling bright-field high-resolution imaging and HAADF-STEM-EDS measurements. Cross-sectional TEM samples were prepared using a Thermo Scientific Helios G4 HX focused ion beam system.

Fabrication and characterizations of Bi₂O₂Se transistors

The as-grown Bi₂O₂Se on f-mica substrates was first covered by spin-coating a 200 nm poly(methyl methacrylate) (PMMA) layer, followed by baking at 170 °C for 3 min. The PMMA-covered film was then immersed in a diluted hydrofluoric (HF) acid solution (1% concentration in deionized water) for 4–12 h to etch the f-mica substrate. After thorough rinsing in deionized water, the film was transferred onto a HfO₂/SiO₂ substrate and annealed at 80 °C for 2 min to enhance adhesion. The PMMA layer was subsequently removed by soaking in acetone. For electrode fabrications, standard electron beam lithography (EBL) and electron beam evaporation (EBE) were employed to define and deposit the drain and source contacts. Lift-off was performed by immersing the device in acetone for 1 h. To manufacture the transistor array, the device was patterned again via EBL, followed by inductively coupled plasma (ICP) etching to define the channel regions. Finally, a second round of EBL and EBE was conducted to deposit Ti/Au (5/45 nm) drain and source electrodes. Electrical characterizations of as-fabricated devices were performed using a Keithley SCS-4200 semiconductor parameter analyzer.

Simulation study of Bi₂O₂Se ferroelectricity

First-principles density functional theory (DFT) calculations were performed to investigate the ferroelectric properties of Bi₂O₂Se using the Vienna ab initio simulation package (VASP). We employed the Perdew-Burke-Ernzerhof approximation for solids (PBEsol) functional, a modified Perdew-Burke-Ernzerhof approximation optimized for solids, to describe exchange-correlation effects, while core-valence interactions were treated using the projector-augmented-wave (PAW) method. The plane-wave basis set was expanded with a kinetic energy cutoff of 520 eV to ensure complete convergence of the electronic wavefunctions. For Brillouin zone integration, we adopted a 12 × 12 × 4 Monkhorst-Pack k-point mesh for both structural relaxations and self-consistent calculations. Structural optimizations were considered converged when the residual forces on all atoms fell below 0.02 eVÅ⁻¹.

Manufacture of spiking generation circuit

The spiking generation circuit comprises three functional modules: an analog front-end (AFE) for signal conditioning, a microcontroller unit (MCU) for digitization and protocol processing, and a power supply unit for stable system powering. The signal processing pipeline initiates with the transimpedance amplifier of AFE that converts the photocurrent response into voltage signals, which subsequently undergo precise filtering and amplification. The conditioned signal is then digitized by the 12-bit analog-to-digital converter (ADC) within the MCU (STMicroelectronics STM32G474CEU6), featuring a 4MSPS sampling rate that enables real-time signal acquisition. The digitized data are transmitted to a host computer via an asynchronous serial interface (UART) using the Standard SCPI (Standard Commands for Programmable Instruments) Commands, ensuring standardized communication interface compatibility. The AFE employs AD8676 rail-to-rail operational amplifiers (Analog Devices, Inc.) for the transimpedance, filtering, and voltage amplification stages. These amplifiers are selected for their low noise and high precision, thereby ensuring accurate signal conditioning. The MCU utilizes the STMicroelectronics STM32G474CEU6 microcontroller, which features a 12-bit ADC with a sampling rate of 4 MSPS, enabling real-time data acquisition and protocol conversion. The power supply unit is designed to provide stable and low-noise power to the system, which includes: 1) an ADP5070 DC-DC converter (Analog Devices, Inc.) to generate ±6 V from a 5 V USB input, 2) low-noise high-power-supply-rejection-ratio linear regulators (LT3045 and LT3094) are used to produce ultra-low-noise ±5 V supplies for the AFE, 3) a TPS74801 linear regulator (Texas Instruments) steps down the 5 V supply to 3.3 V to power MCU, and 4) a TI REF3033 bandgap reference voltage chip provides a stable 3.3 V low-drift reference for the ADC.

Architecture of the spiking neural network

The spiking neural network (SNN) is comprised of two convolutional layers (3 → 6 → 10 channels with 3 × 3 kernels), followed by three fully-connected layers (250 → 32 → 16 → 3), with all inter-layer information transfer in the form of spike sequences. The network processes information through sequential transformations, with the core operational principles illustrated by:

$$I_i\left(t\right)=f_i\left(S_{i-1}\left(t\right)\right)$$ 1

$$S_i\left(t\right),Me{m}_i\left(t\right)=LI{F}_{\beta }\left(I_i\left(t\right),Me{m}_i\left(t-1\right)\right)$$ 2

where I_i(t) is the input current to the i^th leaky integrate-and-fire (LIF) neuron layer at time step t, S_i(t) is the spike output, Mem_i(t) is the membrane potential, and f_i is the layer-specific transformation, with all LIF neurons employing a leakage factor β = 0.9.

We use a leaky integration model to update neuronal membrane potential Mem_t as the following:

$$Mem\left[t\right]=\beta Mem\left[t-1\right]+I\left[t\right]$$ 3

where β is the leak parameter controlling the decay rate of membrane potential, and I[t] is the input current at time step t. When the membrane potential exceeds the threshold, the neuron fires a spike and resets:

$$S\left(t\right)=\Theta \left(Mem\left[t\right]-V_{\mathrm{th}}\right)$$ 4

$$Mem\left[t\right]=Mem\left[t\right]\cdot \left(1-S\left(t\right)\right)+Me{m}_{reset}\cdot S\left(t\right)$$ 5

where S(t)∈{0, 1}ⁿ is the binary spike output of the neuron at time step t, with 1 indicating a spike and 0 indicating no spike. Θ is the Heaviside step function, and V_th = 1.0 is the firing threshold. When the membrane potential Mem_t exceeds V_th, the neuron generates a spike and the membrane potential is reset according to equation (1.5). The final output is derived by averaging the output spike of the final layer across all time steps:

$$Output=\frac{1}{num\_steps}{\sum }_{t=0}^{num\_steps-1}S\left(t\right)$$ 6

Multisensory fusion algorithm

To process multimodal data effectively, we employ a dual-branch convolutional neural network that independently extracts discriminative features from audio and visual inputs before fusion. The visual processing branch applies a series of convolutional transformations to extract hierarchical visual features from the input image V. Similarly, the audio processing branch extracts frequency-temporal patterns from the mel-spectrogram representation of audio input A:

$$v=SNN\left(V\right)$$ 7

$$a=SNN\left(A\right)$$ 8

where the SNN model is consistent with the above equations 1.1–1.6. After independent feature extraction, we implement a late fusion strategy by concatenating the flattened feature representations from both modalities:

$$va=concat\left(flatten\left(v\right),flatten\left(a\right)\right)$$ 9

This multimodal representation preserves the complementary information from both modalities while allowing the network to learn cross-modal relationships. The combined features are then processed through a classification layer:

$$o=argmax\left(W\cdot va+b\right)$$ 10

where W is the weight matrix and b the bias vector for the classification layer. The final prediction is determined by the maximum value in the output, corresponding to the most probable class.

Author contributions

M.H. conceived the original idea for the project. S.L., A.C., Z.D., J.L., X.G., Y.Z., Z.Z. and X.W. fabricated and characterized the devices; L.Z., L.H., F.L. and P.H. processed the simulations; Z.W. (Zhiyuan Wu) and L.S. (Linxiao Shen) fabricated the pulse conversion circuits; R.X., L.S. (Linbo Shan), Z.W. (Zongwei Wang), and Y.C. manufactured the 1T1R memristor chip. The manuscript was written by M.H. and S.L. with inputs from the other authors. All authors contributed to the scientific planning and discussion.

Peer review

Peer review information

Nature Communications thanks Sungjun Kim, Xiaodong Yan and the other anonymous reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Funding

This work was supported by the National Key R&D Program of China (2022YFB4400100), the Natural Science Foundation of China (92164205, 61927901), the Beijing Nova Program (No. 20240484743), the 111 Project (B18001), and the Peking Nanolab.

Data availability

The data supporting the findings of this study are available within the article and its Supplementary Information files. All other relevant raw data can be obtained from the corresponding author upon request.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s41467-026-74648-y.