A high-resolution polarization-hyperspectral image sensor for complex intelligent sensing

Abstract

Light carries high-dimensional information such as spectrum and polarization, but conventional photodetectors measure only intensity, making snapshot acquisition of multiple optical dimensions difficult. Here, we report a multidimensional on-chip optical imaging (MOCI) architecture comprising three functional layers, including a multidimensional encoding layer to encode incident light, an image acquisition layer to collect coupled intensity measurements, and a computing layer to recover multidimensional images from a snapshot measurement. Following this architecture, we developed an on-chip polarization-hyperspectral imaging (PHI) sensor operating at 74 frames s⁻¹ with 2048  × 2448 pixels, 61 visible-to-near-infrared spectral channels, and four polarization states. The sensor acquires spectral and polarization signatures simultaneously in complex environments, enabling polarization-enhanced hyperspectral imaging through haze, suppression of reflection and glare, and hyperspectral 3D modeling with normal and height maps. This compact, on-chip, and high-throughput sensor provides a route to integrated snapshot multidimensional imaging and machine perception under challenging optical conditions.

This study reports a multidimensional on-chip optical imaging (MOCI) technique, and developed a polarization-hyperspectral imaging sensor, enabling robust joint polarization-hyperspectral imaging and intelligent sensing in complex environments.

Introduction

Traditional optical imaging techniques primarily capture light intensity, while light contains richer details beyond intensity, including but not limited to spectral and polarization information^1,2. These high-dimensional light information provides more compressive representations of objects’ physical, chemical, and morphological properties in the real world, offering enhanced detection capabilities for next-generation machine intelligence^3,4. For example, hyperspectral imaging extends spatial intensity to spatio-spectral data cubes that contain hundreds or even thousands of spectral channels, enabling precise material identification⁵. With these advantages, hyperspectral imaging has been widely applied in various fields, including remote sensing, medical diagnostics, agricultural monitoring, and machine vision^6,7. Similarly, polarization imaging enhances spatial imaging with additional polarization information that characterizes the vector distribution of structured light beams, revealing target shape, texture, stress, and so on ref. ⁸. This technique has crucial applications in fields such as biomedical imaging, remote sensing, and industrial inspection⁹.

As current photodetectors are typically limited to intensity-only acquisition due to speed-constrained photoelectric conversion, high-dimensional imaging remains challenging and requires additional optical and mechanical elements to separately capture high-dimensional information^2,10. For example, traditional spectral imaging systems commonly employ grating, prisms, or narrowband filters to acquire spectral cubes through spatial or spectral scanning¹¹. Likewise, conventional polarization imaging techniques typically rely on polarizers, polarizing beam splitters, or liquid crystal variable retarders to obtain polarization information¹². These systems usually involve complex mechanical structures and optical components, which limit the spatial, spectral, and temporal resolution, and make it challenging to achieve system integration and miniaturization¹³. To address these challenges, innovations in materials science and nanofabrication technologies have driven the miniaturization and on-chip integration of high-dimensional imaging devices, with metasurfaces^14,15, Fabry-Pérot filter arrays¹⁶, broadband spectral modulation dyes¹⁷, aluminum nanowires¹⁸, color thin films¹⁹, and liquid crystal materials being widely utilized.

Since multidimensional optical information is typically coupled rather than independent, its separate acquisition may overlook the intrinsic inter-relationships²⁰. In complex environments, such as dense haze, strong scattering, or strong glare, single-dimensional or separate measurements generally fail to resist such degradations and provide accurate scene understanding or reliable material identification. With such a motivation, we introduce simultaneous multidimensional optical acquisition on a single chip, which can decouple and reconstruct multidimensional images that are resistant to propagating interference. By advancing multidimensional optical imaging with a single sensor, the imaging and sensing capabilities can be significantly enhanced by removing propagation degradation, and thus improving the understanding of material properties and achieving robust intelligent sensing under complex conditions^21,22.

Currently, benefiting from flexible light control of metasurfaces, they have been applied in joint spectral and polarization imaging^15,21. However, such methods face several challenges, including strong dependence on input light angle, high fabrication costs, difficulties in large-scale processing, and limitations in spatial resolution, which limit their flexibility and practical applicability^23,24. With the advancement of micro-electromechanical systems (MEMS)^25,26, the integration of MEMS with two-dimensional materials or photonic crystal slabs has been reported for multidimensional sensing^22,27. In addition, optical thin-film interfaces with spatial and frequency dispersion have been applied in spectral and polarization encoding². However, these approaches require complex components for experimental imaging, making it challenging to achieve high spatial resolution on-chip in practical applications.

To address the above challenges, this work reports a multidimensional On-Chip optical Imaging (MOCI) architecture that can jointly acquire multiple high-dimensional light information at a snapshot. As shown in Fig. 1a, the MOCI architecture is functionally composed of three layers, including the multidimensional encoding layer, the image acquisition layer, and the computational reconstruction layer. The multidimensional encoding layer contains multiple modulation sub-layers, with each one modulating a specific dimension (e.g., spectrum, polarization, phase, etc.) of incident light. Through this multi-sublayer structure, multiple dimensions of light information can be simultaneously encoded at light speed, which is then coupled into the intensity dimension and captured by the subsequent image acquisition layer. Finally, the underlying multidimensional optical images are reconstructed from the coupled acquired image in the computational reconstruction layer, using compressive sensing or deep learning algorithms. Such an architecture maintains the same fundamental performance of the underlying acquisition layer, including spatial resolution and frame rate, while each frame can furthermore produce multiple high-dimensional information in a lightweight on-chip manner. Besides, the encoding-decoding principle compresses high-throughput information into a small number of acquired data, which can effectively decrease data amount under a limited bandwidth.

Fig. 1. The principle and exemplar results of the MOCI architecture.

a The MOCI architecture consists of three functional layers, including a multidimensional encoding layer, an imaging acquisition layer, and a computational reconstruction layer. The encoding layer comprises multiple sublayers that modulate the incident light along specific dimensions. For the polarization-hyperspectral imaging (PHI) sensor, the encoding layer consists of two sublayers: a broadband multispectral filter array (BMSFA) and a linear polarization filter array (LPFA), which modulate light along spectral and polarization dimensions, respectively. The BMSFA is fabricated using photolithography and consists of a cyclic 4 × 4 arrangement of broadband organic spectral modulation materials. The LPFA consists of 2 × 2 arrangement of linear polarizers at specific angles (0^∘, 45^∘, 90^∘, and 135^∘). b Polarization-hyperspectral joint imaging of birefringent phenomena using the PHI sensor. Birefringent phenomena induce spectral-polarization coupling, where polarized illumination passing through a phase retarder produces wavelength-dependent polarization states that are subsequently decoupled by the PHI sensor. In the experiment setup, incident light is homogenized by a diffusing sphere and then polarized to form a well-defined beam. This beam propagates through a birefringent phase-retarder sheet before being measured by the PHI sensor. The reconstructed results demonstrate the sensor’s simultaneous polarization-hyperspectral acquisition capability, revealing the wavelength-dependent variations in degree of polarization (DOP) and angle of polarization (AOP).

Building on this architecture, we for the first time fabricated an on-chip polarization-hyperspectral imaging (PHI) sensor. This sensor integrates micro-nano fabricated broadband multispectral filter arrays (BMSFA) and linear polarization filter arrays (LPFA) together, enabling simultaneous control of incident light in the spectral and polarization dimensions. Specifically, the incident light first passes through the BMSFA for spectral modulation, and then through the LPFA for polarization encoding. The coupled multidimensional information is subsequently captured by a CMOS image chip. Finally, the original polarization-hyperspectral images are reconstructed from the acquired single intensity image using a joint-reconstruction neural network. The spectral range of the PHI sensor is from 400 to 1000 nm, with an average spectral resolution of 2 nm. The spatial resolution of hyperspectral images (HSIs) is 2048  × 2448, with a frame rate of 74 FPS and 61 channels. In addition, each pixel can simultaneously produce signals across four linear polarization angles to retrieve the polarization information for each spectral channel, with the linear polarization extinction ratio being as high as 5 orders of magnitude. To sum, the data throughput contains 2048  × 2448 pixels  × 61 channels  × 4 angles  × 74 FPS. A detailed comparison of performance with other works is provided in Table 1.

Table 1.

Comparison of different polarization-spectral imaging techniques

Integration type	Imaging mode	Spatial resolution (pixels)	Polarization channel	Spectral range	Spectral resolution / Channels	Ref
System	Scanning	1024  × 1024	Full-Stokes	400–1000 nm	5 nm / -	Opt. Express 202151
		1	Full-Stokes	1400–1500 nm	2 nm / -	eLight 202252
		/	4 channels (0∘, 45∘, 90∘, 135∘)	8–14 μm	0.6 μm / -	Optica 202443
		/	4 channels (0∘, 45∘, 90∘, 135∘)	400–750 nm	25 nm / -	eLight 202453
		/	Full-Stokes	1520–1620 nm	2.5 nm / -	Nano Lett. 202454
		2016  × 2016	Full-Stokes	400–700 nm	- / 28 channels	Laser Photonics Rev. 202555
	Snapshot	400  × 400	Full-Stokes	400–850 nm	- / 72 channels	Adv. Photonics Nexus 202356
		68 × 40	Full-Stokes	400–900 nm	2.7 nm / -	Nature 20242
On-chip	Scanning	1	3 channels (0∘, 45∘, 90∘)	400–750 nm	16.9 nm / -	Sci. Adv. 202121
		1	Full-Stokes	1–8 μm	0.5 μm / -	Nat. Commun. 202457
		1	Full-Stokes	520–680 nm	8.7 nm / -	Light Sci. Appl. 202458
		1	Full-Stokes	1476–1599 nm	- / 10 channels	Nat. Photonics 202527
	Snapshot	42  × 50	Full-Stokes	700–1150 nm	0.23 nm / -	Sci. Adv. 202415
		2048  ×  2448	4 channels (0∘, 45∘, 90∘, 135∘)	400–1000 nm	2 nm / 61 channels	MOCI

*The “system” in the integration type column denotes the ones that consist of separate modulation and sensing elements in their experiment setups, while the “on-chip” refers to the integration of all elements. The “scanning” in the imaging mode column means that the imaging mode is sequential acquisition along one specific dimension, such as spectral, polarization, or spatial, while the “snapshot” means capturing the entire information in a single shot. The bold row indicates the performance metrics of the present work.

To demonstrate the practicality and application potential of the PHI sensor, we conducted a series of experiments in complex environments, including polarization-enhanced hyperspectral sensing against haze, hyperspectral separation of reflected and transmitted scenes, high-dynamic-range hyperspectral imaging with glare removal, and hyperspectral 3D modeling. In hazy environments, optical imaging suffers from geometric distortion, radiometric degradation, and loss of spatial and spectral details. Using the PHI sensor, we achieved polarization-enhanced hyperspectral imaging and sensing against haze, enabling high-fidelity haze suppression and physically interpretable reconstruction in practical outdoor scenes. This polarization-enhanced suppression of atmospheric scattering can not only significantly improve visual quality but also obtain accurate high-dimensional sensing information through complex media that conventional cameras or dehazing algorithms cannot access. In the reflection and transmission separation experiment, when observing objects through glass, the imaging quality is typically compromised by reflections from the surrounding scene. Using our PHI sensor, we successfully separated and reconstructed the spectra of reflected and transmitted light from a snapshot measurement. The hyperspectral separation capability can not only enhance imaging quality, but also enable precise material recognition in complex illumination conditions. In the interference of glare light, which typically degrades spectral imaging quality, the PHI sensor employed the linear polarization property and achieved high-dynamic-range hyperspectral imaging. Furthermore, due to the three-dimensional structural information embedded in polarization data, the PHI sensor concurrently captures the three-dimensional shape, surface normals, and spectral details of objects. To sum up, the experiments validate that the PHI sensor can not only tackle complex lighting conditions such as haze, reflections, and glare, but also facilitate the acquisition of multidimensional features using a single image chip. We anticipate that the reported MOCI architecture and PHI sensor can advance the next-generation intelligent image sensor beyond CMOS or CCD in the last decades, and support future advances in machine intelligence with multidimensional imaging and sensing.

Fabrication and performance of the PHI sensor

The MOCI architecture employs a hierarchical design to encode, acquire, and resolve multidimensional information. The multidimensional encoding layer contains multiple modulation sub-layers, with each modulating a specific optical dimension. The image acquisition layer uses a two-dimensional image chip to capture the coupled multidimensional optical information as intensity measurements. Finally, the computational reconstruction layer is employed to restore the underlying multidimensional images from the coupled measurements using compressive sensing or deep learning algorithms.

To demonstrate the MOCI architecture, we fabricated an on-chip Polarization-Hyperspectral Imaging (PHI) sensor that can simultaneously capture high-dimensional hyperspectral and polarization images, as shown in Fig. 1a. In this case, the multidimensional coding layer consists of a broadband multispectral filter array (BMSFA) and a linear polarization filter array (LPFA). We used broadband spectral modulation materials, together with photolithography techniques, to fabricate the BMSFA for spectral modulation¹⁷. The BMSFA utilizes 16 broadband spectral modulation materials, which combines photoresists with organic dyes for compaction and integration. The materials were arranged in a 4 × 4 tiled array and solidified onto a wafer, enabling pixelation and array formation. For polarization encoding, the LPFA consists of 2 × 2 polarization units, with each spatial pixel corresponding to a miniature unit. These units, which employ nanowire grid polarizers to generate linear polarization states of 0°, 45°, 90°, and 135°, are arranged in a periodic mosaic pattern to achieve pixel-level polarization encoding.

We conducted both qualitative and quantitative evaluations of the PHI sensor. The measurements captured by the PHI sensor contain both polarization and spectral information, encoded by BMSFA and LPFA simultaneously, with 2048 × 2448 spatial pixels. The high-dimensional hyperspectral and polarization information can be reconstructed using a neural network (PSRNet) at a video rate (see Supplementary Section 4 for more details). The reconstructed polarization-hyperspectral images have dimensions of 4 × 61 × 2048 × 2448, including hyperspectral information for four linear polarization states (0°, 45°, 90°, and 135°). For each polarization state, the reconstructed HSIs are sampled at 10 nm intervals, resulting in 61 spectral channels (see Supplementary Section 5 for more details). We further utilized the PHI sensor to reveal the intrinsic relationship between polarization and spectral information across different wavelengths. As shown in Fig. 1b, birefringent phenomena were employed as a representative case, where wavelength-dependent phase delays imposed by birefringent materials generate distinct polarization states. The reconstructed results demonstrate the sensor’s polarization-hyperspectral acquisition capability, revealing the wavelength-dependent variations in degree of polarization (DOP) and angle of polarization (AOP).

To evaluate the sensor’s performance, we employed a monochromator (Omno151) to produce a series of monochromatic light with a 10 nm full width at half maximum (FWHM), which were for calibrating the polarization-hyperspectral sensing matrix of the PHI sensor across different polarization states, as illustrated in Fig. 2a(i). Using the calibrated sensing matrix and compressive sensing algorithms, we characterized the sensor’s multidimensional encoding properties and achieved accurate reconstruction of monochromatic light. For a quantitative analysis of spectral reconstruction accuracy, we compared the FWHM and center wavelength of monochromatic light between the reconstructed results and the commercial spectrometer. The average FWHM and central wavelength errors are 0.26 and 0.22 nm, respectively, as illustrated in Fig. 2a(ii) and Fig. 2a(iii). To further validate the system’s reconstruction capability under extreme conditions, we tested it with monochromatic light at a 2 nm FWHM, with the reconstructed spectrum presented in Fig. 2a(iv).

Fig. 2. The performance of the PHI sensor.

a (i) Spectral comparison of the PHI sensor (green solid lines) and commercial spectrometer (black dashed lines, Ocean Optics USB 2000 +). The calibrated monochrome light was generated by the commercial grating monochromator (Omno151) with 10 nm FWHM. (ii-iii) The error of the FWHM and center wavelength between the reconstructed monochrome spectrum and the ground truth. (iv) The results of the extreme spectral resolution using monochrome light with 2 nm FWHM at 0.2 nm spectral interval. b Extinction ratio of PHI sensor in the 400–1000 nm spectral range. c The exemplar hyperspectral and polarization joint imaging results. The 2D measurement is captured by the PHI sensor, containing coupled hyperspectral and polarization information. The reconstructed results of polarization-hyperspectral information under four states are synthesized as RGB images for visualization. Reconstructed spectral images under 0^∘ polarization state in exemplar channels are shown on the right. d Comparison among representative prior works. This figure compares representative prior works in terms of spatial resolution, spectral resolution, and data throughput. Data throughput is a composite metric used to quantify the amount of information, which is defined as the number of spatial pixels multiplied by the number of polarization states and the spectral bandwidth, and normalized by the spectral resolution. arb. units, arbitrary units.

In addition, the polarization extinction ratio at different wavelengths is used to evaluate the PHI sensor’s polarization properties, with the linear polarization extinction ratio being as high as 5 orders of magnitude, as presented in Fig. 2b. As shown in Fig. 2c, the HSIs reconstructed from the four polarization states were synthesized into a pseudo color image for display. By applying the sensor’s quantum efficiency (QE) curve, all the reconstructed spectral images were merged into a single-channel DOP and AOP image, providing the overall polarization features. We also compared the reconstructed spectra with the ground truth obtained by a commercial spectrometer, achieving an average spectral fidelity of 99.88%. Furthermore, a comparison with representative prior works is presented in Fig. 2d, considering spatial resolution, spectral resolution, and data throughput. Data throughput is defined as a composite metric given by the number of spatial pixels multiplied by the number of polarization states and the spectral bandwidth, and normalized by the spectral resolution, where a higher value indicates superior high-dimensional information acquisition capability. These results demonstrate that this work achieves the highest data throughput among the compared approaches, outperforming the others by more than two orders of magnitude. A more detailed comparison is provided in Table 1.

Application for polarization-enhanced hyperspectral sensing against haze

Haze particles cause severe geometric distortion, radiometric degradation, and loss of spatial and spectral details in practical optical imaging²⁸. Conventional dehazing techniques generally build on computer vision algorithms, employing the atmospheric scattering model to enhance haze-degraded images²⁹. However, these model-based methods rely on idealized assumptions of atmospheric scattering, resulting in poor sensitivity to weakly scattering media, inaccurate parameter estimation, and strong dependence on illumination conditions. To mitigate these limitations, a range of approaches introduce statistical or empirical priors derived from natural images, such as dark channel prior, non-local prior, color-lines, and haze-lines, to better regularize the dehazing process^30,31. Nevertheless, such prior-based methods are typically limited to specific scenes, facing limited generalization, a lack of physical interpretability, and loss of fine structural details. Different from the above algorithmic techniques, there is a set of optical techniques exploring polarization detection for dehazing, which employ strategies such as depth-chromaticity compensation regularization³², polarization achromatism, and polarization attenuation prior³³. Such techniques rely on enhanced spatial structures for subsequent sensing, overlooking the physical dependence of atmospheric scattering on the intrinsic polarization-spectral characteristics, as well as the material-specific spectral information needed for robust object recognition³⁴.

To address the above challenges, we applied the reported PHI sensor under complex imaging conditions with strong haze, and successfully demonstrated polarization-enhanced hyperspectral sensing against haze, not only high-fidelity haze removal but also the recovery of material-specific spectral information for physical interpretation in practical outdoor scenes. Specifically, as illustrated in Fig. 3a, the sensor captures light scattered by haze particles through both transmission and scattering channels, where the degree of polarization (DOP) is directly sensitive to airlight components. In the experiment setup, we placed real green plants and artificial plastic plants side by side to evaluate the imaging performance of different cameras, including the PHI sensor, a common RGB camera (Sony, IMX264), a polarization camera (Sony, IMX250MZR), and a commercial push-broom hyperspectral camera (FigSpec 23). These cameras were enclosed within a sealed acrylic chamber, and a smoke generator (Ulanzi, FM01) was employed to produce haze with controllable concentrations. The ground-truth polarization-hyperspectral data were first recorded under haze-free conditions as a reference.

Fig. 3. Application for polarization-hyperspectral sensing against haze.

a Experiment configuration of polarization-hyperspectral sensing against haze. The PHI sensor operates based on polarized atmospheric scattering to acquire haze-degraded measurements. b Imaging comparison among different cameras, including a common RGB camera, a polarization (POL) camera, and the PHI sensor. The spectral comparison of real and plastic plants is presented on the right side. c Exemplar polarization-enhanced hyperspectral images of the PHI sensor at representative wavelengths (450, 550, 650, and 750 nm). Locally magnified regions are provided to distinguish between real and plastic green plants, revealing significant differences in their textural characteristics in the near-infrared range. d Dehazed reference of the commercial push-broom hyperspectral imaging (FigSpec 23). On the left, the HSI acquired under haze is synthesized as an RGB image for visualization. The measured spectra (Meas) are compared with the ground truth (GT) spectra obtained under haze-free conditions, revealing a significant discrepancy due to information degradation caused by haze. For further comparison, the dark channel prior algorithm and two SOTA deep-learning dehazing methods (FFA-Net and DehazeNet) were employed to enhance the hazy HSIs. From left to right, these figures display the HSIs processed by these three enhancement methods and their corresponding spectral comparison results at several locations marked on the synthesized RGB images. arb. units, arbitrary units.

The imaging results of different cameras are illustrated in Fig. 3b. Compared with conventional RGB and polarization (POL) cameras, we can see that the PHI sensor achieved a clearer separation between the target scene and haze, revealing finer spatial and spectral details otherwise lost in scattering haze. To reconstruct dehazed high-fidelity spectra, a physics-guided dehazing model is constructed by embedding polarization constraints into the spectral reconstruction process. By estimating the polarized airlight component from multi-angle polarization states, the model suppresses scattering and restores the intrinsic reflectance spectra across 61 spectral channels. In the right subfigure of Fig. 3b, the spectral comparison between real and artificial plants shows that, even under strong haze conditions, the PHI sensor can preserve the distinct spectral signatures of real plants (P1) and artificial plants (P2), achieving reconstructed spectral fidelity of 99.81% and enabling accurate material identification despite visually similar colors. This result indicates that the sensing outcome is determined by the recovered material-specific spectra rather than by image contrast enhancement alone. The quantitative error analysis also reveals that the PHI sensor broadens the pixel-intensity distribution, with the contrast increasing from 54.25 to 74.60, indicating effective suppression of haze-related attenuation. Furthermore, as shown in Fig. 3c, the HSIs produced by the PHI sensor at representative wavelengths (450, 550, 650, and 750 nm) exhibit improved visual clarity and spectral consistency, particularly in the near-infrared range, where differences between real and plastic plants become more distinct.

In Fig. 3d, we present the reference imaging results (in the synthesized RGB format) based on the measurements of a commercial push-broom hyperspectral imaging camera (FigSpec 23). We can see that the acquired spectra under haze conditions (dot curves) are seriously degraded and present a severe discrepancy compared to the ground truth captured under haze-free conditions. In such a case, the camera can not distinguish between different materials, including real plants (P1, P3), artificial plastic plants (P2), and even plaster walls of a different color (P4). Further, we applied several representative dehazing algorithms to enhance the acquired HSIs of the push-broom camera, including the dark-channel-prior algorithm³⁰ and two advanced deep learning-based dehazing methods (FFA-Net³⁵ and DehazeNet³⁶). We can see that the visual quality can be enhanced to a limited extent, and the enhanced spectra are still much different from the ground truth. The spectral fidelities of these three enhanced algorithms are 71.81, 70.03, and 72.68%, respectively, which are comparable to that of the unprocessed data (71.80%), validating that the combination of separate spectral measurement and dehazing algorithms does not work well for practical imaging conditions to obtain high-fidelity spectra. In contrast, by joint acquisition of polarization-hyperspectral images on a single sensor, the reported PHI sensor can allow polarization-dependent scattering components to be suppressed while material-specific spectra are recovered, which are not accessible using conventional cameras or computer vision algorithms. Benefiting from the intrinsic snapshot multidimensional imaging capability, the reported approach effectively extends the visual range and produces high-fidelity multidimensional images for intelligent sensing. This capability provides a potential approach to enhancing visual range in complex environments, with applications in autonomous navigation and remote sensing.

Application for hyperspectral sensing against strong reflection and glare

Conventional image sensors suffer significant degradation of imaging quality under complex light conditions, such as specular reflections and strong glare interference, reducing target object identification precision. Traditional enhancement techniques typically rely on computer vision algorithms to enhance degraded images, such as the removal of reflections or high-dynamic-range (HDR) imaging³⁷. On the other hand, there are several methods that rely on complex optical elements, such as adaptive optics and optical metamaterials, to address the complex light conditions³⁸. However, the adaptive optics technique builds on the feedback mechanism that is hard for synchronized enhancement, and requires bulky mechanical elements with extremely high cost³⁹. The optical metamaterial technique is limited to laboratory settings and demonstrations, without outdoor experiments for practical applications⁴⁰. Above all, the mentioned techniques are mostly limited to monochrome or color image enhancement and do not address high-dimensional HSIs. More importantly, polarization-only imaging lacks spectral fingerprints for material recognition, whereas conventional hyperspectral imaging records mixed or degraded spectra under reflection and glare interference, making it difficult to recover latent spectra with reliable material signatures.

We applied the PHI sensor to exploit polarization-spectral information under complex light conditions, aiming not merely at reflection or glare suppression, but at single-shot hyperspectral separation and material characterization. We used the PHI sensor to acquire images corrupted by nonuniform illumination by leveraging the inherent refractive index contrast in multilayer dielectric media (glass/air interfaces). By physics-driven data modeling, we developed a separation strategy to disentangle reflection and transmission components from measurements, as shown in Fig. 4a. Subsequently, we performed a spectral reconstruction for both decomposed scenes, visualizing the reconstructed HSI in the RGB format and validating it against commercial spectrometer measurements (Fig. 4b). As illustrated in Fig. 4c, the reconstructed spectra of real (P1) and artificial plastic (P2) green plants in the transmitted scene show accurate detection of metamerism, a phenomenon where objects appear similar in color but have different spectra.

Fig. 4. Application for hyperspectral sensing against strong reflection and glare.

a Pipeline of reflection and transmission HSI separation. The PHI sensor captures measurements through glass, where the acquired data contains aliased spatial-spectral information from both the reflected scene on the same side as the camera and the transmitted scene on the opposite side. Through HSI reconstruction and reflection & transmission separation, the hyperspectral information for both the transmitted and reflected scenes can be obtained clearly. b Separation result of Transmission and reflection HSIs. Examples of spectral images are shown at 500, 600, 700, and 800 nm. c Comparison of reconstructed spectra (RS) and ground truth (GT) in Transmission HSI. Points P1 (real green plant) and P2 (artificial plastic green plant) are marked in the central leaf areas of the transmission scene in Fig. 4a, highlighting the reconstructed accuracy and recognition ability from the differences in spectral dimension between real and artificial plastic plants with the same color. d Glare removal diagram and pipeline. The scene with strong glare is first polarized using a linear polarizer and then captured by the PHI sensor for polarization detection. A fusion model is applied to obtain high-dynamic-range HSIs with glare removal. e Glare removal and spectral reconstruction result. The first row, from left to right, shows the measurement without polarization processing under glare conditions, the original values captured by the PHI sensor, and the glare removal results after fusion processing. On the left of the second row is the reconstructed HDR HSI, which is synthesized as RGB images for visualization. On the right, the performance of the reconstructed spectra is compared with the ground truths contained by the commercial spectrometer. arb. units, arbitrary units.

Glare interference significantly degrades visual quality, hindering accurate hyperspectral imaging. To address this challenge, our PHI sensor employs an innovative polarization modulation technique that effectively suppresses glare, enabling the acquisition of HDR HSI in a single frame exposure. By combining the PHI sensor with a linear polarizer, each frame records photoelectric responses under four orthogonal polarization states (0°, 45°, 90°, and 135°), obtaining coupled measurements with varying texture and spectral details. These images under four states are then adaptively fused using our proposed HDR algorithm, which incorporates exposure suppression and linear weighted fusion to enhance dynamic range while preserving spectral fidelity and eliminating glare (Fig. 4d). As shown in Fig. 4e, glare-affected regions in the raw data exhibit significant overexposure, obscuring fine spatial and spectral details. However, the measurement of the PHI sensor contains more detailed information under four polarization states, which can improve the HDR of reconstruction HSI and achieve glare removal. A comparison of the reconstructed spectral curves with ground truth measurements obtained by a commercial spectrometer demonstrates a spectral fidelity of 99.93%. The results validate the PHI sensor’s capability for high-precision spectral sensing under complex light conditions, enabled by the single-shot, on-chip joint acquisition of polarization-hyperspectral information, which suppresses strong optical interference while preserving material-specific spectral signatures.

Application for hyperspectral 3D modeling

Polarization has been widely applied for geometric feature extraction in 3D reconstruction, where polarized LiDAR enables distance field and surface normal acquisition⁴¹, and multi-view polarization techniques recover 3D morphology through polarized light-field analysis⁴². However, the existing techniques exhibit overemphasis on uni-dimensional geometric analysis with deficient material characterization capabilities. Originating from the simultaneous polarization and hyperspectral imaging ability of the PHI sensor, we demonstrate the hyperspectral 3D modeling application, in which both spatial geometry and resolved spectra are recovered from a snapshot measurement. This enables synchronized morphology-material characterization beyond either polarization-based 3D reconstruction or hyperspectral imaging alone.

Specifically, we developed a prototype (Fig. 5a) to capture polarization and hyperspectral data coupled in a snapshot, and prepared a variety of smooth and low-texture objects with minimal surface features. We used our PHI sensor and a 3D scanner (EINSTAR VEGA) to establish a dedicated paired dataset, containing both polarized spectral data and ground-truth 3D models. Leveraging a deep learning-based approach for polarized hyperspectral 3D reconstruction (Fig. 5b), we extracted normal maps, height maps, and HSIs from the single-shot measurement. Figure 5c presents the PHI sensor’s raw measurements, reconstructed height and normal maps, and the HSIs synthesized in RGB format. The reconstructed spectra, referenced by spatial addresses in the synthesized RGB images, were compared with the ground-truth data captured by the commercial spectrometer, achieving a spectral fidelity of above 99%. In addition, we compared the reconstructed 3D maps with ground truth from the 3D scanner, producing a PSNR of 35.07 dB for the height maps and 35.32 dB for the normal maps. As shown in Fig. 5d, the reconstruction results are visualized from three different viewpoints, including height maps, normal maps, and spectral images at several example wavelengths (450, 550, 650, 750, 850 nm). Both the quantitative and qualitative experiment results demonstrate the PHI sensor’s effectiveness for real-time and high-resolution hyperspectral 3D modeling. The recovered height maps, normal maps, and HSIs are spatially synchronized because they originate from the same snapshot measurement, enabling concurrent analysis of surface morphology and material identity.

Fig. 5. Application for hyperspectral 3D modeling.

a The prototype of hyperspectral 3D modeling. The source illuminates the objects, and the PHI sensor acquires the reflected light. b Principle of polarization recovery of HSI and 3D information. The measurements captured by the PHI sensor are processed through an end-to-end restoration network based on the UNet framework to obtain the high-dimensional information, including height maps, normal maps, and hyperspectral image (HSI). c Reconstruction results of hyperspectral 3D modeling. The two sample results are displayed. For clarity visualization, height maps are shown in grayscale images, normal maps synthesize surface normal vectors along the [x, y, z] axes as three-channel RGB color images, and the reconstructed HSIs are also synthesized as RGB images. On the right is the comparison between reconstructed spectra (RS) and ground truth (GT) spectra, with the randomly selected points marked in the synthesized RGB images. d 3D hyperspectral images reconstruction from single-frame measurement. To better visualize the 3D modeling performance, we present views from three different angles of the results, including 3D height maps, 3D normal maps, and example 3D spectral images at 450, 550, 650, 750, and 850 nm.

Discussion

This work tackled the long-standing unresolved challenge of high-resolution imaging and intelligent sensing in various complex propagating conditions, such as medium scattering and strong glare, by introducing simultaneous multidimensional optical acquisition on a single chip. We report a multidimensional on-chip optical imaging (MOCI) architecture that utilizes optical encoding and computational imaging theories to simultaneously acquire multiple high-dimensional optical information. The MOCI consists of three key components, including the multidimensional encoding layer, the image acquisition layer, and the computational reconstruction layer, which together facilitate the encoding, acquisition, and decoding of multidimensional optical data. Building on this architecture, we successfully fabricated a polarization-hyperspectral imaging (PHI) sensor that integrates a broadband multispectral filter array and a linear polarization filter array, achieving joint spectral-polarization encoding using a single sensor. As summarized in Table 1, existing polarization-hyperspectral imaging techniques typically rely on system-level or mechanical-scanning complex optical architectures, resulting in bulky, low-integration systems that are unsuitable for lightweight and real-time applications⁴³. Scanning-based on-chip approaches alleviate some of these constraints but generally focus on single-point sensing, failing to obtain high-spatial-resolution data cubes²¹. System-level snapshot solutions based on spatial modulation structures, such as optical thin-film interfaces, still require external optical paths for modulation, limiting device miniaturization and yielding spatial resolution of only thousands of pixels². In addition, some approaches employ metasurfaces to develop on-chip snapshot polarization-hyperspectral imaging, but the limited imaging area and pixel count still hinder the acquisition of high-resolution, high-dimensional data¹⁵. In contrast, our PHI sensor acquires the original multidimensional optical information at video rates (74 FPS) with high spatial-spectral resolution (2048 × 2448 pixels × 61 channels × 4 angles), which is reconstructed from the snapshot coupled measurements using a deep-learning neural network. The information throughput is far higher than competing techniques.

The practical experiments demonstrate that the PHI sensor produces advanced performance under complex light conditions, enabling polarization-enhanced hyperspectral sensing against haze, hyperspectral separation of reflected and transmitted scenes, and high-dynamic-range hyperspectral imaging with glare removal. In addition, the sensor achieves hyperspectral 3D modeling, leveraging polarized structured information to obtain height maps, normal maps, and point cloud data from a single frame of coupled measurement. Compared to other snapshot computational imaging techniques that are typically designed to capture only specific optical dimensions, the MOCI technique offers advantages including multiple imaging dimensions, high integration, high resolution, and high video rate, enabling snapshot acquisition of compressive optical information and providing a more comprehensive characterization of target properties, potentially driving the advancement of next-generation intelligent imaging and sensing devices.

The present implementation still leaves room for further enhancement. First, the recovery of weak narrowband peaks and high-frequency spectral structures remains challenging under low-SNR conditions, reflecting the general trade-off among compressed optical encoding, measurement noise, and reconstruction constraints in computational imaging. Second, large-scale reconstruction of high-resolution polarization-hyperspectral data cubes still requires substantial computational resources, which could be alleviated by lightweight networks, task-oriented reconstruction, model compression, and hardware-aware deployment⁴⁴. Third, improving the generalization and robustness of PHI sensing under varying illumination and low-light conditions remains important, which may be addressed through multi-illuminant calibration, illuminant-aware physical modeling, and noise-aware reconstruction⁴⁵. Addressing these limitations will further improve the reliability and deployability of on-chip multidimensional optical sensing in complex practical environments.

Considering the dimensionality, the successful demonstration of the PHI sensor highlights the effectiveness of the MOCI architecture and points to a broader route toward advanced multidimensional optical sensing. First, in the view of compressive encoding, advanced optical encoding structures⁴⁶, optical coding materials⁴⁷, and micro-electromechanical systems (MEMS)⁴⁸ can be introduced to further acquire phase, light field, and other high-dimensional optical information within a single miniature sensor. Second, in view of computational decoding, advanced deep learning, and big model techniques can be introduced to further enhance reconstruction and unmixing precision, helping dig into the multidimensional optical correlation⁴⁹. Third, the above computation can be packed into a dedicated chip, which can further help integration without requiring additional high-power computational graphics⁵⁰.

Methods

PHI sensor fabrication and integration

The on-chip polarization–hyperspectral imaging (PHI) sensor was fabricated by vertically integrating a broadband multispectral filter array (BMSFA), a linear-polarization filter array (LPFA) and a monochrome CMOS detector. The BMSFA comprised a 4 × 4 array of 16 organic modulation materials selected for weakly correlated transmission spectra across 400–1000 nm, with each element measuring 10 μm in lateral dimension. The LPFA contained linear polarization analyzers oriented at 0°, 45°, 90° and 135°, enabling the simultaneous acquisition of four polarization-resolved measurements in a single exposure. The BMSFA was integrated to the Sony IMX250MZR LPFA-CMOS image platform using a photoresist-mediated bonding process. The directly bonded interface minimized propagation-induced spatial spreading and interlayer optical crosstalk. Following integration, the assembled sensor was calibrated at the pixel level to determine its joint spectral and polarization modulation.

Forward model of multidimensional optical imaging

The PHI sensor encodes spectral and polarization information into a single two-dimensional detector measurement. At spatial coordinate (h, w) and polarization channel p, the recorded intensity is modeled as

$$b\left(h,w,p\right)={\int }_{{\lambda }_1}^{{\lambda }_2}r\left(h,w,p,\lambda \right)d\left(h,w,p,\lambda \right)c\left(\lambda \right)\mathrm{d}\lambda +n\left(h,w,p\right),$$ 1

where b(h, w, p) is the measured intensity, r(h, w, p, λ) presents the high-dimensional hyperspectral-polarization information of the target scene, d(h, w, p, λ) displays the spectral-polarization dependent modulation imposed by BMSFA-LPFA encoding layer. The term c(λ) describes the wavelength-dependent response of the remaining optical system, including the imaging optics and monochrome CMOS detector, whereas n(h, w, p) represents measurement noise.

After discretizing the 400-1000 nm spectral range into (M = 61) channels at 10-nm intervals and vectorizing the detector measurement and polarization-resolved hyperspectral cube, the forward model is expressed as b = Hr + n, where b is the vectorized two-dimensional compressed measurement, r is the vectorized polarization-resolved hyperspectral data cube, and H is the calibrated sensing matrix. The reconstruction process can be formulated as a traditional regularized optimization problem. The high-dimensional polarization-hyperspectral cube can be recovered using either iterative model-based optimization or learned inverse mappings conditioned on the calibrated forward model.

PHI sensor calibration

The spatial–spectral-polarization sensing matrix was characterized after full sensor integration using an automated calibration system comprising a monochromator (Omno151, 200–2000 nm), an integrating sphere, optical power sensors (S130VC, Thorlabs), and a rotatable broadband linear polarizer (300–2700 nm, Edmund Optics). The monochromator generated monochromatic light from 400 to 1000 nm. At each wavelength and input polarization orientation (0°, 45°, 90° and 135°), the two-dimensional sensor response, exposure time and incident optical power were recorded synchronously. The response maps were normalized to the exposure time and optical power and assembled into the pixel-level sensing matrix H. Because calibration was performed after bonding and optical integration, H captured the combined responses of the BMSFA, LPFA, CMOS detector and optical path. The calibrated matrix was subsequently used in the forward imaging model and as a physics prior for reconstruction.

Polarization-hyperspectral image reconstruction

Polarization-hyperspectral data were reconstructed from single measurements using a Polarization-Spectral Reconstruction Network (PSRNet). The raw measurements were mapped to a tensor resolving four polarization angles (0°, 45°, 90°, 135°). PSRNet utilizes a multiscale U-Net architecture that integrates a Polarization-Spectral Attention Fusion Module (PSAFM) at each stage, mapping the coupled measurement to original hyperspectral and polarization information. Each PSAFM features dual parallel branches, in which a spectral branch applies multi-head channel-wise self-attention to capture long-range spectral dependencies, and a polarization branch utilizes self-attention to extract complementary inter-channel responses. These branches are fused via a learned sigmoid gate and refined by a convolutional feed-forward block. Crucially, the calibrated sensing matrix is resized and injected at each network scale as conditioning information. PSRNet was optimized using a composite objective that jointly constrains spectral reconstruction, spatial regularity, and polarization fidelity, as follows,

$$\mathcal{L}={\lambda }_{\mathrm{RMSE}}{\mathcal{L}}_{\mathrm{RMSE}}+{\lambda }_{\mathrm{MRAE}}{\mathcal{L}}_{\mathrm{MRAE}}+{\lambda }_{\mathrm{TV}}{\mathcal{L}}_{\mathrm{TV}}+{\lambda }_{\mathrm{pol}}{\mathcal{L}}_{\mathrm{pol}}$$ 2

While the first three terms enforce spectral reconstruction and spatial smoothness, ${\mathcal{L}}_{\mathrm{pol}}$ preserves wavelength-dependent polarization fidelity by applying Smooth-L1 penalties between the degree and angle of linear polarization (DoP and AoP) maps, calculated independently for each spectral band. This model was trained on hyperspectral and polarization images, utilizing 256 × 256 patches. The network was optimized for 200 epochs using Adam (β₁ = 0.9, β₂ = 0.999), with an initial learning rate of 4 × 10⁻⁴ and a cosine-annealing learning-rate schedule. Training was implemented in PyTorch on a single NVIDIA GeForce RTX 4090 GPU and required approximately 20 h.

Author contributions

L.B. and Z.W. conceived the idea. Z.W. and L.B. designed and fabricated the modulation layers and implemented sensor integration. Z.W. calibrated the sensor and tested its imaging performance. Z.W., P.P., Z.Z., and C.Z. designed and implemented the hyperspectral 3D reconstruction experiment. Z.W., P.P., and J. Zhou conducted experiments on hyperspectral sensing against strong reflection and glare. P.P., Z.W., and Y.F. conducted experiments on polarization-enhanced hyperspectral sensing against haze. L.B., Z.W., P.P., Z.Z., R.Y., H.X., and J. Zhang prepared the figures and wrote the manuscript with input from all the authors. L.B. supervised the project.

Peer review

Peer review information

Nature Communications thanks the anonymous reviewers for their contribution to the peer review of this work. A peer review file is available.

Funding

This work was supported by the National Natural Science Foundation of China (62322502, 624B2028, 61827901), Scientific Research Innovation Capability Support Project for Young Faculty (ZYGXQNJSKYCXNLZCXM-I4), and Beijing Natural Science Foundation (JQ25017).

Data availability

The data supporting the findings of this study are publicly available at https://drive.google.com/drive/folders/1_D6fNFY5SeuNiiDc8ej6E5L5MFeAcUhS?usp=sharing. Source data are provided in this paper.

Code availability

The code of this work is available from the public repository at the following link: https://github.com/bianlab/MOCI.

Competing interests

L.B., Z.W., and J. Zhang hold patents on technologies related to the devices developed in this work (China patent numbers ZL 2024 1 0797667.2, and ZL 2022 1 0764166.5) and submitted related patent applications. The remaining authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at https://doi.org/10.1038/s41467-026-76971-w.