In-sensor multispectral perception for near- and mid-infrared discrimination

Abstract

Multispectral features across visible (Vis), near-infrared (NIR) and mid-infrared (MIR) provide complementary information for intelligent vision in challenging environments. Precise extraction of multispectral features within the sensor level becomes increasingly critical, yet remains elusive for next-generation vision systems. Here we report a multispectral sensor constructed by a MoTe₂/BP heterojunction with gate-tunable band alignments between type-Ⅰ and type-Ⅱ, enabling bipolar Vis/NIR (473 ~ 1064 nm) and unipolar MIR (3.7 and 4.6 μm) responses under zero bias for in-sensor Vis/NIR convolution and MIR target recognition. Beyond single-device operation, a 9×1 linear array validates device uniformity and preliminary array-level feasibility. Furthermore, the multispectral sensor extracts crosstalk-free Vis/NIR and MIR features under co-illumination of 980-nm and 3.7-μm light, increasing recognition accuracy from 28% to 84% in reconstructed system-level demonstrations. By fully leveraging the tunable 2D-material band structure, this work demonstrates a universal sensor configuration for multispectral perception with in-sensor processing functions.

Achieving precise extraction of multispectral features is a critical bottleneck for next-generation vision systems. Fang et al. report a multispectral sensor that distinguishes input light in the visible/near-infrared ranges from that in the mid-infrared range at the sensor level, enabling in-sensor multispectral processing.

Introduction

The light spectrum in natural visual environments spans a broad range of wavelengths, including the visible (Vis), near-infrared (NIR), and mid-infrared (MIR) regions. Each spectral band carries fundamentally distinct physical information and contributes complementary modes of visual perception. Among them, Vis and NIR light primarily convey rich static spatial and dynamic spatiotemporal features, forming the basis of most state-of-the-art intelligent photodetectors for in-sensor visual perception^1–7 In addition, MIR light encodes intrinsic thermal radiation and molecular fingerprint features, which not only enable penetration through visual obstructions but also facilitate material identification and enhanced target discrimination under challenging conditions^8–15 Collectively, the complementary information carried by Vis, NIR, and MIR bands establishes multispectral perception as a critical foundation for next-generation intelligent vision systems.

To meet the demands of Vis-to-MIR multispectral sensing, conventional vision systems typically rely on bulky optical filters and physically separated photodetectors dedicated to different spectral bands, such as silicon-based cameras for Vis/NIR and thermal cameras for MIR sensing^16–18. These architectures increase system complexity and hinder device miniaturization and integration. In contrast, recent advances in narrow-bandgap materials have enabled intrinsically broadband photoresponse spanning from Vis to MIR region^19–23. However, their strong broadband sensitivity to both Vis/NIR and MIR light inevitably induces spectral crosstalk, making it difficult to extract spectral features independently at the sensor level. Despite rapid progress in intelligent vision sensors with multifunctional in-sensor operations^6,7,24,25, achieving low-crosstalk, spectrally distinguishable Vis-to-MIR multispectral intelligent sensing within a single device remains elusive.

In this work, we report a Vis/NIR- and MIR-distinguishable multispectral vision sensor based on a gate-tunable bipolar heterojunction. A heterojunction formed by bipolar molybdenum ditelluride (MoTe₂) and narrow-bandgap black phosphorus (BP) enables gate-controlled switching between type-I and type-II band alignments with opposite built-in potentials under negative and positive gate voltages, respectively. In the type-I configuration, Vis/NIR-generated carriers are separated on the MoTe₂ side, producing positive photocurrents due to the built-in electric field from BP to MoTe₂. In contrast, the MIR photocurrent is negligible because MIR-generated carriers are confined within the BP side by interfacial barriers for both electrons and holes arising from the straddling band alignment. In the type-II configuration, both Vis/NIR- and MIR-generated carriers are separated by the built-in electric field from MoTe₂ to BP enabled by the staggered band alignment, yielding negative photocurrents. This non-trivial, spectrally selective response to Vis/NIR and MIR lays the foundation for independent extraction of Vis/NIR and MIR features. Consequently, the heterojunction demonstrates bipolar photoresponses to Vis/NIR illumination (473–1064 nm) and unipolar photoresponses to 3.7 and 4.6 μm MIR light, enabling in-sensor Vis/NIR convolution and MIR target recognition. Beyond single-device operation, a 9 × 1 linear device array further verifies the device-to-device uniformity and suggests the potential of array-level implementation. Furthermore, under co-illumination with 980-nm and 3.7-μm light, the sensor performs in-sensor multispectral perception by separating representative Vis/NIR and MIR features, improving recognition accuracy from 28 to 84% in reconstructed system-level demonstrations of a complex lighting environment. This work highlights the application potential of intelligent vision sensors with in-sensor multispectral perception across the Vis-to-MIR range, with prospects for extended spectral coverage in future implementations.

Results

Design of multispectral vision sensors

Complex lighting conditions in real-world environments present major challenges for vision sensors trying to extract features across different spectral bands. When sensing is extended into the MIR range, in conditions without using bulky optical filters, shorter-wavelength Vis/NIR radiation can introduce spectral crosstalk that degrades MIR measurements. Figure 1a shows a schematic of nighttime vision in rainy weather, where conventional Vis/NIR sensors suffer severe interference due to scattering and attenuation by rain and fog²⁶. In this scenario, a multispectral sensor must recognize three representative targets, including cars, traffic signs and pedestrians, where MIR thermal signals can penetrate through rain and fog, providing an important complement to Vis/NIR features. Commercial driving assistance systems combine Vis/NIR cameras with separate thermal detectors to cover both bands, which require bulky optics and increase system complexity^17,27,28. Here, we propose to integrate multispectral sensing and processing in a single sensor, combining Vis/NIR convolution with MIR-enhanced vision functions. In the schematic, cars exemplify Vis/NIR features, whereas pedestrians exemplify MIR thermal signatures. Embedded with Vis/NIR kernel operation and tunable MIR responsivity, the multispectral vision sensor can extract the car’s edge feature and pedestrians’ clear profile (Fig. 1b). To achieve broadband vision from Vis/NIR to MIR together with spectral selectivity, we designed a gate-tunable MoTe₂/BP heterojunction with rationally designed band alignment. The bandgap of MoTe₂ (~1.0 eV) and BP ( ~ 0.3 eV) corresponds to the intrinsic absorption of Vis/NIR and MIR light, respectively. By tuning the Fermi level of the bipolar MoTe₂ to enable either p-type or n-type conduction, the heterojunction can be switched between type-Ⅰ and type-Ⅱ band alignments under negative and positive gate voltages (-V_g and +V_g), respectively (Fig. 1c). In the switch between these two band alignments, the photogenerated carriers in MoTe₂ are supposed to generate bipolar photovoltaic response due to the reversal of built-in electric field, as indicated by the upwards and downwards band bending of MoTe₂ near the interface. In contrast, photogenerated carriers in BP encounter barriers in both the conduction and valence bands under type-I alignment, which hinders charge extraction. Under type-II alignment, the staggered band offset promotes carrier separation and yields a unipolar photovoltaic response (detailed mechanisms are illustrated in Fig. 2a–c). Based on this understanding, the response characteristics can be proposed and schematically shown in Fig. 1d, demonstrating gate-tunable bipolar Vis/NIR and unipolar MIR photoresponses.

Fig. 1. Design of the multispectral vision sensors.

a Schematic of the multispectral in-sensor vision architecture, where a 2D-material-based photodetector array directly performs sensing and in-sensor computing. b Corresponding in-sensor computing process of Vis/NIR convolution and MIR-enhanced vision, where the edge of a car and the pedestrian silhouette are highlighted. c Band-alignment switching of the MoTe₂/BP junction between type-Ⅰ and type-Ⅱ modulated by gate voltage (V_g). d Response characteristics showing the bipolar Vis/NIR and tunable MIR photoresponse. Source data are provided as a Source Data file.

Fig. 2. Gate-tunable bipolar NIR and unipolar MIR photoresponses.

a Energy-band diagram of the MoTe₂/BP junction at zero bias, showing the gate-tunable Fermi level of MoTe₂. b,c Corresponding photoresponse mechanism under 633-/980-nm Vis/NIR and 3.7-μm MIR lights, respectively, demonstrating different photoresponse behaviors of 633/980 nm and 3.7 μm in the switching between type-Ⅱ and type-Ⅰ band alignment. d,g Zero-bias photovoltage measurements under d 980 nm and g 3.7-µm illumination, respectively, where the polarity of the photovoltage indicates the direction of the built-in electric field and carrier separation behaviors, consistent with the mechanisms illustrated in b and c. e,h Gate-tunable photocurrents under e 980-nm and h 3.7-μm illumination. f,i Photocurrents as a function of power density for f 980-nm and i 3.7-μm light, showing linear relationship under different gate voltages. Source data are provided as a Source Data file.

Gate-tunable visible/near-infrared and mid-infrared response

Leveraging the spectrally selective photoresponse to Vis/NIR and MIR illumination, the MoTe₂/BP heterojunction provides a platform for multispectral sensing with in-sensor spectral feature extraction. The devices are fabricated using mechanical exfoliation and dry transfer. Firstly, isolated MoTe₂ and BP transistors are fabricated to confirm the efficient contact between materials and metal contact, as well as the bipolar transfer characteristics of MoTe₂ (Supplementary Fig. 1). Then the optical microscopy image of the fabricated MoTe₂/BP heterojunction is shown in Supplementary Fig. 2a. The detailed gate-tuning working mechanism is shown in Fig. 2a–c, where 633/980 nm and 3.7 μm lasers are used as representative sources for Vis/NIR and MIR tests, respectively. Figure 2a shows the band alignment of MoTe₂ and BP before equilibrium, where the MoTe₂ with bandgap of around 1.0 eV absorbs Vis/NIR light and narrow-bandgap BP (0.3 eV) absorbs Vis/NIR/MIR light^11,29. Note that both MoTe₂ and BP exhibit above-bandgap optical absorption in the visible range, leading to response characteristics similar to those at 980 nm (verified in Supplementary Fig. 8 and Fig. 13d). Therefore, the 980-nm NIR response characteristics is hereafter used as a representative of the Vis/NIR response. The Fermi level of MoTe₂ can be tuned upwards and downwards by applying positive and negative gate voltages, respectively. After equilibrium, the intrinsic MoTe₂-BP forms a type-Ⅰ junction, exhibiting rectifying characteristics under forward bias when MoTe₂ is grounded, as shown by the I-V curve (Supplementary Fig. 2b). The transfer curve (Supplementary Fig. 2c) confirms the bipolar transport characteristics of the heterojunction, enabling well-tuned band alignment from type-Ⅰ to type-Ⅱ in the MoTe₂-BP heterojunction, and thus establishing spectral-selective photoresponse under NIR and MIR illumination. As shown in Fig. 2b, c, MoTe₂-BP forms type-Ⅱ and type-Ⅰ heterojunctions at large positive V_g (blue rectangle) and negative V_g (red rectangle), respectively. Under 980-nm NIR light, charge carriers are generated in both MoTe₂ and BP and then separated near the depletion region (Fig. 2b). In the staggered type-Ⅱ junction, photogenerated carriers in both MoTe₂ and BP can be collected and yield a negative photocurrent (Fig. 2b, top). In the straddling type-Ⅰ junction, only carriers generated in MoTe₂ can be extracted, while those in BP encounter prominent barriers and are effectively blocked (Fig. 2b, bottom). In parallel, 3.7 μm MIR illumination excites carriers only in BP, producing a negative photocurrent in the type-Ⅱ junction (Fig. 2c, top), while the photovoltaic response in the type-I junction remains near zero (Fig. 2c, bottom). Notably, under 980 nm illumination, the temperature rise is typically negligible and thus the photothermoelectric (PTE) contribution can be ignored, whereas MIR radiation is more prone to induce local heating and yield a detectable negative PTE component¹¹, details are discussed in Supplementary Note 1.

The following gate-tuning I-V tests (Supplementary Fig. 2d) confirm the band configurations illustrated in Fig. 2b, c, showing distinct rectifying behaviors of the PN and PP⁺ junctions^4,30 In addition, the open-circuit photovoltage measurements provide key insights into the electron-hole separation enabled by the built-in electric field in different junctions^31,32. As shown in Fig. 2d, the photovoltages under 980-nm NIR light exhibit a clear bipolar behavior, averaging −0.10 V at V_g = −20 V and +0.05 V at V_g = +40 V. In contrast, under 3.7-μm MIR illumination, the photovoltage at V_g = −20 V is almost negligible compared to that at V_g = +40 V (Fig. 2g), verifying that MIR-generated carriers are inefficiently separated in the type-Ⅰ junction. Gate-tunable photocurrents and a linear dependence on incident power are critical for achieving the in-sensor computing functions²⁴, while the capability of bias-free detection further ensures energy-efficient operation. Thus, all photoresponse measurements in this work are taken at zero bias. Under 980 nm illumination, the device shows pronounced gate-tunable bipolar photocurrents over a V_g range from negative to positive (Fig. 2e), which is the key device performance for in-sensor convolution operations^5,33. In contrast, the photocurrents at 3.7 μm exhibit unipolar behavior, with near-zero response at V_g = 0 V and −20 V, and a monotonically increasing negative response from V_g = +15 V to +35 V (Fig. 2h). The photocurrent results are consistent with the photovoltage measurements, confirming the distinct NIR and MIR response characteristics in the gate-tunable type-Ⅱ and type-Ⅰ heterojunctions. The in-sensor multiply-accumulate operations rely on linear photoresponse to the incident light inputs^7,24 In our device, the photocurrents for both 980-nm and 3.7-μm light exhibit a linear dependence on incident light power (Fig. 2f, i and Supplementary Fig. 3). In addition, the linear dependence of the device on a broader range of 980 nm light intensities is demonstrated, with a linear dynamic range of 62.5 dB (Supplementary Fig. 4), which is crucial for in-sensor computing in practical complex illumination scenarios³⁴. It is worth noting that the 3.7-μm responses from V_g = −20 V to 0 V are almost negligible, ensuring that the response at negative V_g originates solely from 980-nm light and enabling a high spectral rejection ratio, which is crucial for the switchable detection between NIR and MIR bands. The device also exhibits strong polarization sensitivity, with an extinction ratio of approximately 50: 1 under linearly polarized 3.7-μm illumination (Supplementary Fig. 5). Moreover, the response speed is shown in Supplementary Fig. 6, demonstrating a fast response time ranging from 40 to 60 μs under NIR and MIR illumination. The noise spectra at different gate voltages are measured, and the corresponding 980-nm detectivities are calculated as shown in Supplementary Fig. 7, suggesting that the detectivity is mainly dominated by the gate-tuned responsivity in this device. The full-spectral response is further measured at 473, 532, 633, 980 and 1064 nm for Vis/NIR detection and 3.7 and 4.6 μm for MIR detection (Supplementary Fig. 8), showing spectral-selective response between Vis/NIR and MIR.

In-sensor image convolution and letter recognition

Beyond single-device characterization, we further fabricated a 9 × 1 MoTe₂/BP linear device array to evaluate the device-to-device uniformity and array-level feasibility. As shown in Fig. 3a, the linear array consists of nine MoTe₂/BP heterojunction units contacted by metal electrodes. The I-V characteristics of the nine devices exhibit similar output behaviors (Fig. 3b), indicating reliable electrical contact and consistent junction transport across the array. In addition, the gate-dependent photocurrent under 980-nm illumination shows reproducible bipolar modulation among different devices, confirming stable gate control and good uniformity of the photoresponse (Fig. 3c). These results suggest that the MoTe₂/BP heterojunction devices exhibit reasonable uniformity and stability for preliminary array-scale validation. To implement the in-sensor convolution functions, the uniform bipolar photoresponse states of the nine devices were then programmed as positive and negative convolution weights. As shown in Figs. 3d and 3e, a representative edge-extraction convolution kernel consisting of −1 at the edges and 8 at the center is constructed by the nine devices^5,33. Based on experimentally measured response, this numerical implementation uses photocurrents of +40 nA at V_g1 = −15 V and −5 nA at V_g2 = +14 V for kernel element 8 and −1, respectively. The detailed gate-tuned responsivities of a single device under various power intensities are shown in Supplementary Fig. 9. Figure 3f shows the input and output images of the edge-extraction processing, where the outlines of three typical objects of a roundabout traffic sign, a left-turn arrow marking and a car, can be clearly discerned from the background (Fig. 3g). In addition, other common convolution functions, including Emboss and Sharpen, are further enabled by the multi-value bipolar states. As shown in Supplementary Fig. 10a, an Emboss kernel is used to enhance edges in specific directions and simulate the bright and dark surfaces under lighting through the differences in adjacent pixels. And a Sharpen kernel enhances image details and edges by increasing contrast in areas of transition, transforming a blurred image into a clearer one (Supplementary Fig. 10b). These operations provide a numerically constructed proof-of-concept demonstration of versatile in-sensor NIR image convolution functions enabled by the experimentally measured photoresponses.

Fig. 3. In-sensor NIR image convolution based on 9 × 1 MoTe₂/BP linear array and letter recognition implemented with MIR responsivities.

a Optical microscopy image of the fabricated 9 × 1 MoTe₂/BP linear array. Scale bar, 40 μm. b I–V characteristics and c gate-dependent 980-nm photocurrent of the nine devices in the array, showing good device-to-device consistency and reliable gate modulation. d Implementation of bipolar 980-nm response states for edge-extraction convolution Kernel. e Photocurrent distribution from device1 to device9 (D₁ to D₉) under the two programmed states at V_g1 and V_g2. f Numerical demonstration of in-sensor convolution of a nighttime image using the Kernel constructed by the array in d and e. g Extracted edge-enhanced features, including a Roundabout traffic sign, a Left-turn road marking, and a car. h Representative input images of three N T U letters in 9 × 9 pixels. i Training process of the in-sensor neural network, which consists of an input layer and an output layer, with 3.7-μm responsivity states employed as synaptic weights. j Normalized in-sensor outputs f₁, f₂ and f₃ of the corresponding N, T and U classifier, respectively, as functions of training epochs. A larger output indicates higher in-sensor recognition confidence of each letter. k Weight distribution after 10 epochs of training. Source data are provided as a Source Data file.

Moreover, to illustrate the potential of the device for in-sensor computing with MIR optical inputs, we build a two-layer neural network using experimentally measured reconfigurable 3.7-μm responsivity states as shown in Supplementary Fig. 9d–f. The training dataset is made of 9 × 9-pixel images including three letters of N, T and U with different noise levels (Fig. 3h and Supplementary Fig. 11). The image vectors of three letters are separately mapped in the input layer with 9 × 9 × 3 input neurons and fully connected to the three output neurons for N-, T-, U-classifiers (Fig. 3i). Each output neuron applies an activation function to its weighted sum to produce normalized outputs f₁, f₂ and f₃ ranging from 0 to 1. During training, these outputs are compared with their target values to calculate the output errors, which are then backpropagated through the network to update the weights (details see Methods). Figure 3j shows the normalized outputs of each classifier with different inputs, which simulate the total output currents of the device when conducting in-sensor classification of N T U images, and the input image can be classified by simply comparing the three output currents^1,33 In the ten epochs of training, the output f₁ which corresponds to N-classifier is gradually enhanced when inputting the N letter images while rapidly suppressed when inputting T and U images, and the T-classifier and U-classifier also exhibit similar recognition capabilities for their target letter (Fig. 3j). The average recognition accuracy of the three classifiers reaches 100% in ten epochs, with target-to-distractor output ratio gradually increasing with training epochs and finally exceeding two orders of magnitude (Supplementary Fig. 12a, b). Figure 3k shows the weight distribution of final training, displaying distinct features of the respective letters and exhibiting effective learning progress compared to initial training (Supplementary Fig. 12c, d).

Multispectral feature extraction

The above results suggest the potential for reconfigurable in-sensor image processing for both NIR and MIR under individual illuminations. However, in practical scenarios, vision sensors must process multispectral inputs simultaneously, where different bands encode complementary features essential for precise perception under diverse environmental conditions^35–37. Here we demonstrate a proof-of-concept street scenario where vision sensors in the drone receive substantial information across different spectral bands, as illustrated in Fig. 4a. The Vis/NIR features include cars, traffic signs (such as numbers on speed limit signs), and pedestrians are here employed as representative MIR targets for illustrative purposes although the MIR radiation from human body is at longer wavelengths (9-10 µm) than we experimentally tested at 3.7 μm. The detection ability of MoTe₂/BP heterojunction for 9-10 µm is limited by the weak absorption of BP, which could be improved by using materials with smaller bandgap (detailed discussion see Supplementary Note 2). To effectively detect both Vis/NIR and MIR signals, vision sensors require broadband detection capability. However, the complex spectral information carried by diverse targets can introduce spectral crosstalk when distinguishing Vis/NIR and MIR targets, thereby degrading the recognition performance of the vision system. To suppress the spectral crosstalk between Vis/NIR and MIR optical inputs, we fully exploit the distinct gate-tunable response of Vis/NIR and MIR. As shown in Fig. 4b, we tested the gate-tuning photocurrents under individual illumination of 980-nm NIR, 3.7-μm MIR and co-illumination by the two wavelengths. The photocurrents of co-illumination are almost equal to the sum of their respective responses across the gate voltage range from −20 V to +30 V, as shown by blue line and black line in Fig. 4b. The individual response of 980-nm light shows bipolar characteristics with reversal point at +5 V gate voltage while 3.7-μm response is near zero from −20 to 0 V, which provides a large spectral rejection ratio (R_{980 nm}/R_{3.7 μm}) of more than four levels of magnitude at −20 V (Supplementary Fig. 13). Benefit from this distinct response between NIR and MIR illumination, the net NIR features can be extracted with positive photocurrents by switching the gate voltage to −5 V (Frame 1), as represented by a car in blue. Then switching gate voltage to +25 V (Frame 2), both NIR and MIR features are captured with negative photocurrents, as represented by a car and a pedestrian in red. Due to that the 980-nm responses at −5 V and +25 V are equal with opposite polarity, the net MIR features can be captured by adding Frame 1 and Frame 2. The switchable photocurrents in single pixel are shown in Fig. 4c, where co-illumination photocurrent at −5 V is +36 nA, which is equal to net 980-nm photocurrent, and co-illumination photocurrent at +25 V is −58 nA. In result, adding +36 nA and −58 nA is −22 nA, which is equal to net 3.7-μm photocurrent. For visible light, it has been verified that 633-nm photoresponse varied with gate voltages is consistent with that of 980-nm NIR light within the range of −20 to +30 V (Supplementary Fig. 13d). Therefore, in this process the NIR signal can be used to represent the result of overall Vis/NIR band. Based on the experimentally measured Vis/NIR and MIR characteristics, the operation of the switchable Vis/NIR and MIR vision is further numerically reconstructed and schematically illustrated in Fig. 4d, where Frame 1 contains net Vis/NIR features with positive outputs (blue car), while Frame 2 comprises both Vis/NIR and MIR features with negative outputs (red car and pedestrian overlapped). By summing up the outputs of Framews 1 and 2, the Vis/NIR outputs are canceled by the opposite photocurrents with same intensity, yielding an isolated MIR image of the pedestrian. Subsequently, this switchable photoresponse enables selective extraction of Vis/NIR and MIR signals, thereby suppressing spectral crosstalk, which is hard to avoid in broadband photodetectors at the sensor level. In addition to the two-frame operation mode, a subpixel-array architecture could be adopted, where different subpixels within one pixel are programmed into distinct operating states¹, enabling simultaneous acquisition of Vis/NIR and MIR information within a single exposure, as discussed in Supplementary Fig. 19 and Note 3.

Fig. 4. Application demonstration of Vis/NIR and MIR distinguishable vision sensor.

a Schematic illustration of an application scenario incorporating complementary Vis/NIR (car, digit) and MIR (human) features. b Gate-tunable photocurrents under individual illumination and co-illumination of 980-nm NIR and 3.7-μm MIR light. The photocurrents under simultaneous illumination equals the sum of the individual 980-nm and 3.7-µm photocurrents, enabling extraction of pure Vis/NIR and mixed Vis/NIR + MIR features, as represented by a blue car image and a red car+human image, respectively. c Corresponding I-t tests showing the gate-switching operation for extracting Vis/NIR and MIR features. At V_g1 = −5 V, the sensor only responds to Vis/NIR features with positive photocurrent (I₁ = +36 nA). At V_g2 = +25 V, the sensor response to both NIR and MIR features, with negative photocurrent (I₂ = −58 nA). The MIR features can be isolated by adding I₁ and I₂. d Proof-of-concept illustration of multispectral vision sensor operated in two frames, showing crosstalk-free Vis/NIR (car) and MIR (human) features. The input images are numerically constructed based on the spectral noise extracted from experimentally measured device responses, as shown in Supplementary Figs. 14–17. e Three-layer artificial neural network for classifying multiple targets, including cars, humans, and digits from 0 to 9. f Comparison of the classifying accuracy among multispectral, broadband and narrowband vision sensors. Source data are provided as a Source Data file.

As illustrated in Fig. 4a, when multispectral signals enter the vision sensor, the sensor must separate target features from complex backgrounds while suppressing spectral crosstalk. Conventional approaches rely on separate Vis/NIR cameras and MIR thermal imaging cameras, along with bulky optical filters, mechanical switching components and complex backend algorithms to achieve spectral separation^16,17,38. This leads to large system volume, high power consumption, and additional computation latency. For comparison, broadband detectors such as BP-based photodetectors cover a wide spectral range from Vis/NIR to MIR^19,20,39, but they suffer from inter-band interference between Vis/NIR and MIR signals under complex illumination conditions. Narrowband photodetectors, on the other hand, provide selective sensitivity to a designated spectral window, yet inherently lack perceptual capability for other bands^40,41. Meanwhile, nonlinear upconversion has demonstrated considerable promise as a complementary system-level approach for high-speed and snapshot MIR spectral imaging by transferring MIR signals to wavelengths accessible to mature silicon cameras^14,42–44. However, further developments may be needed to enable the simultaneous acquisition of native Vis/NIR and upconverted MIR signals on a common silicon imaging platform while preserving clear spectral discrimination. In our Vis-to-MIR multispectral vision sensor, spectral crosstalk can be suppressed by gate-voltage modulation, enabling in-sensor extraction of Vis/NIR and MIR features. To demonstrate the in-sensor multispectral vision capabilities of the device at a numerically reconstructed system level, we build a customized dataset including cars and digits representing Vis/NIR features and humans representing MIR features (Supplementary Fig. 14)^35,45. The demonstration numerically implements a two-channel, three-layer artificial neural network (Fig. 4e) to classify 12 types of Vis/NIR and MIR features including handwriting digits 0 ~ 9, cars and humans.

The Vis/NIR and MIR switchable vision sensors enable clear extraction of both types of spectral features, with MIR-to-Vis/NIR spectral crosstalk suppressed to below 1% in a wide gate voltage range (Supplementary Fig. 15a). And the Vis/NIR-to-MIR spectral crosstalk arises from the residual error due to the incomplete cancellation of the positive and negative Vis/NIR responses, which can be quantitatively represented by the standard deviation of the 980-nm responsivity under varying light intensities. The resulting crosstalk can be suppressed to approximately 3% (Supplementary Fig. 15b). Due to the low spectral crosstalk between Vis/NIR and MIR, the inputs used for network testing are low-noise images of digits, cars and humans (Supplementary Fig. 15c). As a result, the Vis/NIR-MIR switchable vision sensors achieve high overall recognition accuracy of 84% (Fig. 4f, Supplementary Fig. 15d, e and Note 4).

Here broadband vision sensors are compared using the same network, which can be represented by the type-Ⅱ MoTe₂-BP junction with gate tuning to +30 V, where the device can respond to both Vis/NIR and MIR lights with relatively large responsivities (Supplementary Note 4). Similarly, we also defined the spectral crosstalk between Vis/NIR and MIR, as shown in Supplementary Fig. 16. According to the noise-to-signal ratio, 50% and 200% Gaussian noise are added to the testing images of digits/cars and humans, respectively. The resulting overall recognition accuracy for the three categories is 67% (Fig. 4f, Supplementary Fig. 16), indicating that the detection and discrimination capability to multispectral features of the broadband detector is significantly affected by spectral crosstalk. For narrowband vision sensors, either narrowband Vis/NIR or MIR sensors exhibit no responsivity to the complementary spectral band, which inevitably results in recognition failure of the associated targets. For instance, when a narrowband MIR sensor is incapable of detecting Vis/NIR signal, the corresponding Vis/NIR channel in the network is input with images made of random noise (Supplementary Fig. 17), resulting in car/digit recognition accuracy equal to zero and an overall accuracy as low as 28%. The comparison of overall recognition accuracy of the three types of devices as a function of training epochs is summarized in Fig. 4f, highlighting the precise perception of the multispectral vision sensors for both Vis/NIR and MIR features. The accurate recognition of Vis/NIR and MIR features may enable more reliable target detection in adverse environments such as fog, rain, and nighttime conditions. By adaptively capturing the most informative spectral cues, multispectral vision sensors are promising to enhance robustness against crosstalk and improve situational awareness in complex application environments, such as autonomous driving, intelligent surveillance, night-time navigation and others.

Discussion

In summary, we have demonstrated a Vis/NIR- and MIR-distinguishable multispectral vision sensor based on a gate-tunable bipolar MoTe₂/BP heterojunction, which can be reconfigurably switched between type-Ⅰ and type-Ⅱ band alignments. This spectral-switching detection enables either selective Vis/NIR sensing or broadband Vis/NIR + MIR sensing operated at the sensor level. This distinct spectral response allows the clear extraction of both Vis/NIR and MIR features, suppressing spectral crosstalk and ensuring accurate multispectral perception. Such multispectral sensors hold significant application potential for enhancing machine vision extended to the MIR region in complex environments.

Distinctively, this work underscores the unique value of passive MIR thermal imaging for advanced intelligent sensing. Beyond Vis/NIR detection, MIR encodes intrinsic thermal emission that is especially informative under low-light or low-visibility conditions^17,28,46. While MIR imaging is established in specialized fields such as defense, surveillance and advanced driver assistance systems, its integration into general-purpose vision systems remains rare due to detector material constraints, cooling requirements, and cost^17,46–48. Of course, the scalable implementation of 2D-material heterojunction detector arrays still faces key challenges in materials growth, CMOS-compatible fabrication, heterogeneous integration, and backend interfacing, as further discussed in the Supplementary Note 3. Nevertheless, our results demonstrate a sensing capability that is both distinctive and underexplored, with significant potential for broader machine-vision applications as MIR technologies advance. Building on this paradigm, similar heterojunction devices with switchable band alignment could be extended to the long-wave infrared regime (detailed discussion in Supplementary Note 2 and Note 5), paving the way toward broader multispectral sensing and advanced intelligent vision systems.

Methods

Fabrication and characterization of devices

The devices reported in this work were fabricated by stacking mechanically exfoliated molybdenum ditelluride (MoTe₂) and black phosphorus (BP) flakes to form a van der Waals heterojunction. Bulk MoTe₂ and BP crystals were mechanically exfoliated onto polydimethylsiloxane (PDMS) stamps. Flakes with suitable thicknesses were first identified by optical microscopy. The selected MoTe₂ flake ( ~ 15 nm) was first transferred onto SiO₂ (285 nm) substrate with highly doped Si, followed by the sequential transfer of the BP flake ( ~ 50 nm) using a PDMS-assisted dry transfer. After the transfer, Cr/Au (5/50 nm) metal source and drain electrodes were patterned by a direct-write lithography system and deposited by electron-beam evaporation. After lift-off, the devices were covered by a polymethyl methacrylate (PMMA) layer to protect BP from degradation for subsequent tests. For array fabrication, the MoTe₂/BP heterostructure was patterned into individual device units by reactive ion etching (RIE), with the gases of Ar, CHF₃ and O₂. The highly doped Si substrate served as a back gate to electrostatically tune the Fermi level of MoTe₂. Electrical tests were conducted with a precision source/measure unit (Keysight B2912A). Response time was measured by a digital oscilloscope (Keysight DSOX3054T). Light sources were provided by tunable mid-infrared lasers at 3.7 and 4.6 μm (Daylight Solution, MWIRCat), laser diodes at 473 nm, 633 nm, 1064 nm and fiber-coupled lasers at 532 nm and 980 nm. Optical power was monitored with a thermal power meter (Ophir, Nova display-ROHS) and an optical power meter (Thorlabs, PM100D). All experiments were conducted in ambient atmosphere at room temperature.

In-sensor image convolution and classification

In-sensor image convolution was implemented by directly encoding bipolar 980-nm responsivity states as convolution kernel weights. The output photocurrent represented the analog multiply–accumulate (MAC) operation between the optical intensity and the responsivity-encoded weight. For the in-sensor letter classification, Gaussian noise (σ  = 0.1, 0.2, and 0.3) was added to the binarized N T U image in 9 × 9 pixel, 1000 images were used for training, and 150 images were used for testing for each letter. The photocurrent outputs were normalized by f = (1 + e^-αI)⁻¹ to produce the outputs.

Author contributions

Q.W. supervised the project. S.F., F.W., and Q.W. conceptualized the idea. S.F. designed and fabricated the devices. S.F., F.C., and J.H. performed the image classification. W.C., C.H., H.M., J.Y., Z.Z., and C.W. provided testing assistance. S.F., F.W., and Q.W. analyzed the data. S.F. and F.W. drafted the manuscript, and Q.W. revised the manuscript. All authors have discussed the results and commented on the manuscript. S.F. and F.W. contributed equally to this work.

Peer review

Peer review information

Nature Communications thanks Jie Yang and the other anonymous reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Funding

This work was supported by funding from A*STAR Programmatic Funds (R23I0IR041, M23M2b0056), the National Research Foundation Competitive Research Program (NRF2023-ITC004-001, NRF-CRP29-2022-0003, and NRF-MSG-2023-0002) and Ministry of Education (MOE) funding (MOET32024-0001).

Data availability

Source data are provided with this paper.

Code availability

The codes that support the findings of this study are available from the corresponding author upon request.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at https://doi.org/10.1038/s41467-026-77260-2.