Fluorescence localization and tracking imaging with a spectral-splitting perovskite single-pixel detector

Abstract

Image-guided surgery systems require precise fusion of fluorescence mapping and structural background imaging, yet current dual-camera system methods suffer from field-of-view misalignment and pixel offsets, risking millimeter-scale surgical errors. To address these challenges, we developed an fluorescence/background fusion single-pixel imaging system based on spectral-splitting perovskite photodetectors. By incorporating gradient-optimized wide-bandgap perovskite filter layers, the perovskite photodetector can achieve spectrally selective detection through efficient separation of 520 nm fluorescence signals (S₁) from 450 nm backscattered light (S₂). With a high signal suppression ratio (S₁/S₂) of 57 times of the optimized photodetector, the system achieves a low detection limit of 50 nmol/mL for sodium fluorescein aqueous solution. More importantly, the active single-pixel imaging architecture ensures perfect field-of-view and pixel-level consistency across multiple detectors’ images, eliminating the need for complex registration algorithms or sophisticated dual-optical path designs. Finally, through in murine tumor experiments, we achieved precise real-time fluorescence-labeled tumor localization and tracking imaging, demonstrating the reliability of our developed system in fluorescence/background fusion imaging. This provides a fluorescence-targeting approach for medical imaging, demonstrating the utility of single-pixel imaging in advanced diagnostics.

In this paper, a perovskite-based single-pixel imaging system is developed that simultaneously captures fluorescence and background images with perfect alignment, enabling real-time tumor localization and tracking in mice without complex registration algorithms.

Introduction

Surgical navigation represents a critical technological advancement for ensuring procedural precision and safety. Conventional systems primarily depend on spatial co-registration between preoperative imaging and intraoperative anatomical landmarks (such as MRI/CT)^1,2. However, intraoperative tissue deformation, instrument obstruction, and physiological dynamics frequently cause spatial discrepancies between preoperative data and real-time field of view (FOV), significantly compromising clinical utility. Fluorescence imaging provides functional molecular visualization through targeted labeling of critical structures (tumor margins^3–7, neural bundles^6,8–11, vascular networks^6,12–15), yet suffers from inherent limitations: absence of anatomical context and vulnerability to ambient light interference, tissue autofluorescence, and nonspecific binding. While backscatter imaging offers structural reference, it lacks sensitivity for pathological discrimination. Consequently, developing a registration-free fluorescence/background fusion imaging system capable of simultaneous molecular punctuation of tumor information and anatomical structure information has emerged as a pivotal solution for advancing surgical navigation capabilities.

Current multi-modal fluorescence/background fusion imaging approaches predominantly employ three conventional strategies, each presenting significant limitations for surgical applications:

1) Dual-camera synchronous acquisition^7,16: While offering high temporal resolution through beam-splitter optical paths, this method suffers from prohibitive hardware complexity, bulky form factors unsuitable for surgical integration, and substantial cost barriers.

2) Time-multiplexed single-camera imaging¹⁷: Utilizing filter wheels or broadband spectrometry for alternating signal capture inherently compromises temporal resolution and sacrifices dynamic synchronization, critical parameters for real-time surgical navigation.

3) Spectral-separation single-camera imaging^18–20: Although achieving theoretically synchronous imaging through on-chip filter arrays, this approach faces fundamental constraints including lower pixel resolution, complex microfabrication requirements and unacceptable channel crosstalk.

These methods have the common challenges of difficulty in achieving synchronization, spatial registration accuracy and system complexity. Particularly in dual-camera systems, parallax-induced FOV misalignment necessitates complex post-acquisition registration algorithms a process further compromised by intraoperative tissue deformation and instrument movement. While spectral-splitting techniques circumvent geometric registration, their dependence on computationally intensive unmixing algorithms and vulnerability to ambient light render them unsuitable for robust surgical implementation.

Fourier single-pixel imaging (FSPI) emerges as an evolving alternative, leveraging its unique optical architecture and computational imaging paradigm to address these limitations. In this context, FSPI technology offers an alternative solution for operative navigation due to its unique optical architecture and computational imaging algorithm. FSPI combines structured light modulation with single-pixel photodetectors (PDs) and reconstructs images based on the spatial frequency features of Fourier-based structured light patterns^21,22. The fundamental advantage lies in its spatial consistency; the imaging geometry (FOV and pixel dimensions) is solely determined by the encoded spatial light patterns, while fluorescence and background signals share the same optical path²³. This intrinsic property ensures perfect spatial registration when acquiring multi-spectral images using PDs with different spectral responses, completely eliminating alignment errors. Furthermore, single-pixel PDs offer compact size, low cost, and strong interference resistance, overcoming the limitations of conventional optical systems (relatively higher cost, complex configuration, and bulky size) while guaranteeing multi-channel synchronization.

Building on these considerations, conventional multimodal imaging systems predominantly use array-based image sensors, such as charge-coupled devices (CCDs) and complementary metal-oxide-semiconductor (CMOS) detectors, for fluorescence and background signal acquisition. Beyond the aforementioned limitations in spectral separation, these array sensors suffer from fundamental constraints: 1) the small photosensitive area of individual pixels in high-resolution arrays severely limits detection sensitivity, 2) inadequate photon capture capability for weak fluorescence signals, and 3) substantial readout noise that can obscure faint biological signals, collectively rendering them suboptimal for fluorescence detection applications.

Last decade, metal halide perovskites have demonstrated significant potential across optoelectronic applications including photovoltaics, light-emitting diodes (LEDs)²⁴, PDs^25,26, and lasers²⁷, owing to their improved optoelectronic properties (such as high light absorption coefficient, high mobility, low binding energy, adjustable band gap) coupled with cost-effective solution processability^28,29. For photodetection applications, precise composition engineering enables spectral response tuning to match target fluorescence and background bands. Furthermore, perovskite PDs integrated with single-pixel imaging have demonstrated potential across various imaging applications, including bionic vision systems³⁰, wide-field imaging³¹, and visible-near-infrared dual-modal imaging³². More importantly, unlike CMOS arrays where 30–50% of the surface area is occupied by circuitry, perovskite thin films achieve near-unity fill factors (>90%), offering reliable performance for weak fluorescence signal detection.

Here, we present an intraoperative real-time fluorescence-background fusion imaging system based on metal-halide perovskite photodetectors (PVK PDs) and active-mode single-pixel computational imaging, referred to as the perovskite single-pixel fluorescence-background fusion imaging system (PVK-SPI-FB System). The system architecture incorporates perovskite filter layers (PFLs) for spectral splitting of background and fluorescence signals, enabling synchronous dual-modal acquisition through two PDs. Experimental results demonstrate that the inherent characteristics of FSPI ensure perfect spatial congruence between fluorescence and background images obtained from different detectors, with identical FOV and pixel-level registration, eliminating the need for complex image alignment algorithms and significantly improving workflow efficiency. In vivo murine validation confirms the system’s clinical potential, establishing a paradigm for dynamic intraoperative multi-modal imaging and advancing the application of FSPI in surgical and biomedical fields.

Results and discussion

PVK-SPI-FB System

This study takes sodium fluorescein as an example to study the fusion of fluorescence/background. Figure 1a illustrates the fundamental principle of fluorescence emission: under excitation at specific wavelengths, electrons in the fluorophore absorb photons and transition to higher energy states, followed by rapid return to the ground state with release of excess energy as photons. During this process, non-radiative relaxation (e.g., internal conversion) dissipates partial energy as heat, resulting in emitted photons with lower energy (longer wavelength) than the excitation photons. This energy difference, known as the Stokes shift, enables spectral separation between excitation and emission wavelengths, allowing distinct acquisition of fluorescence and background signals through spectrally optimized PDs. As shown in Fig. 1b, conventional approaches typically employ two CCD cameras to simultaneously capture fluorescence and background images. Here, the fluorescence channel camera requires a high-performance short-wave cutoff filter (taking sodium fluorescein’s absorption/emission wavelengths as an example, market research indicates typical unit prices ranging from ¥3000 to ¥5,000 RMB) to completely block background interference from the excitation wavelength band. Moreover, due to the inherent challenge of achieving consistent viewing angles between cameras, fluorescence and background images exhibit spatial mismatches that preclude precise alignment, fundamentally limiting the accuracy of fluorescence localization imaging. In addition, due to differing sensitivity requirements between the fluorescence and background cameras, the two systems often use distinct camera models, leading to a significant pixel count mismatch between fluorescence and background images. Conventional multi-camera fluorescence-background fusion imaging systems face three inherent challenges: signal crosstalk, perspective misalignment, and pixel resolution disparity. These issues collectively lead to significant image registration errors and degraded fusion quality, as shown in Fig. 1c.

Fig. 1. Perovskite single-pixel fluorescence-background fusion imaging method.

a The luminescence principle of the fluorescent probe (sodium fluorescein). b System architecture of the conventional dual-camera setup for simultaneous fluorescence/background imaging. c Challenges in fusing fluorescence and background images acquired by dual-camera systems. d Our proposed PVK-SPI-FB System. e Schematic diagram of the structure of PVK-PD. f PVK-SPI-FB System effectively resolves the fusion challenges between fluorescence and background images. DOI: 10.5281/zenodo.3925901.

To address the inherent image fusion challenges of conventional dual-camera systems, we developed a PVK PD-based single-pixel imaging system for fluorescence-background fusion imaging (PVK-SPI-FB System) (Fig. 1d), which integrates a light source module for simultaneous fluorescence excitation and backscattered signal generation, a digital micromirror device (DMD) for generating pre-encoded Fourier-based patterns carrying spatial information, a perovskite filtered detector (PVKF-PD) specifically for fluorescence signal detection, and a perovskite broadband detector (PVK-PD) for backscattered signal acquisition. The system features p-i-n structured photovoltaic perovskite detectors (PVK PDs) (Fig. 1e) that offer fast response speed and sensitivity for weak fluorescence detection, while their self-powered operation significantly simplifies the imaging architecture. The fabrication process of PVK PDs is shown in Supplementary Fig. 1. Supplementary Fig. 2 shows the survey XRD spectra of the perovskite PD layer (Cs_0.05FA_0.79MA_0.16PbI_2.5Br_0.5). Building on our previous findings demonstrating the inherent FOV invariance and the advantages of perfect pixel registration in active-mode single-pixel imaging systems [30], this integrated solution effectively overcomes the critical limitations in fusion between fluorescence and background images. Specifically, active-mode single-pixel imaging is such that the image FOV and pixel dimensions are determined solely by the spatial illumination patterns rather than the location of PDs. This fundamental characteristic enables the two PVK PDs to simultaneously acquire perfectly registered fluorescence and background images with identical spatial coordinates, eliminating the need for post-acquisition registration algorithms. Fluorescence imaging is typically actively excited, making the integration of active-mode single-pixel imaging systems particularly promising for resolving the fusion and registration challenges between fluorescence and background images, as illustrated in Fig. 1f.

Spectral-splitting mechanism of perovskite single-pixel PDs

In addition to adopting single-pixel imaging methodology to resolve image registration challenges, we have developed spectral-splitting PVK PDs that fundamentally addresses the critical issue of fluorescence-background signal separation. Unlike conventional approaches relying on costly optical filters, our solution employs an intrinsic wavelength self-filtering strategy enabled by the unique bandgap tunability of perovskite materials. Through controlled ion-exchange processes, we engineered two functionally distinct perovskite films: The small-bandgap perovskite films can be fabricated into broadband PDs (PVK-PD) specifically optimized for detecting the dominant backscattered signals. By strategically combining the small-bandgap PVK-PD with large-bandgap perovskite films exhibiting short-wavelength cutoff characteristics, we developed a narrowband PVK PD (PVKf-PD). This design effectively filters out the shorter-wavelength excitation backscattered light while selectively transmitting the desired fluorescence signals, enabling pure fluorescence detection as demonstrated in Fig. 2d. However, the intensity of backscattered signals significantly exceeds that of fluorescence signals (approximately 17-fold greater, Fig. 2i), necessitating the use of deep-cutoff short pass filters to effectively suppress the backscattered components, a key factor contributing to the high cost of conventional solutions.

Fig. 2. Spectral-splitting PVK PDs.

a The generation process of fluorescence/backscattered light. b The spectral curves of the fluorescence signal of sodium fluorescein and the excitation light signal. c Schematic illustration of photon absorption and electron-hole separation mechanism in broadband PVK-PD for background signal detection. d Working principle of filtered PVKf-PD with additional PKL for background rejection and selective fluorescence detection, showing photon absorption and charge separation processes. e The absorption spectrum of PVK film and the transmission spectrum of PVK filter films (Film 1-5 correspond to precursor concentrations ranging from 0.75 to 1.75 M). f Experimental wavelength-dependence responsivity curves of PVK-PD and PVKf-PDs. g Transient photoresponse curves of PVK-PD. h Linear dynamic range (LDR) curve of PVK-PD. i. Measured backscattered signal intensity (Signal 1) and fluorescence intensity (Signal 2) from s sodium fluorescein aqueous solutions at varying concentrations under 11.09 mW cm⁻² excitation power, detected by a reference PD at the imaging position. j1 Fluorescence and background signal intensities detected by broadband PVK-PD across different sodium fluorescein concentrations. j2–j6 Fluorescence/background signal intensities measured by filtered PVKf-PDs with progressive optimization across different sodium fluorescein concentrations. k Complete signal inversion between background and fluorescence intensities in PVKf-PD5 compared to PVK-PD at all tested concentrations, confirming successful development of spectrally-selective PDs.

Perovskite materials offer unique advantages in this regard, as their bandgap tunability enables precise optimization of short-wavelength cutoff characteristics through simple precursor concentration modulation. To systematically investigate this property, we developed five distinct large-bandgap PFLs with varying precursor concentrations (1: 0.75 M, 2: 1 M, 3: 1.25 M, 4: 1.5 M, 5: 1.75 M) and characterized their transmission spectra to evaluate the short-wavelength blocking and long-wavelength transmission capabilities. As shown in Fig. 2e, increasing the precursor concentration leads to enhanced film thickness and improved short-wavelength cutoff performance, albeit at the cost of reduced long-wavelength transmittance. This trade-off prompted a comprehensive optimization study across seven concentration gradients of PFLs, with detailed analysis of their spectral filtering characteristics (Supplementary Fig. 3). The results reveal a non-monotonic trend in the transmission ratio between 640 nm and 450 nm as the concentration increases, exhibiting an initial rise followed by a decline (Supplementary Fig. 3b). Through comprehensive evaluation of both long-wavelength fluorescence transmission and short-wavelength excitation suppression, PFL5 demonstrates optimal spectral-splitting capability in this study. Figure 2e presents the transmission curves of five representative concentration-gradient PFLs, along with the spectrum curves of the narrowband PVK PDs, where the intersection region between transmission and absorption curves defines the spectral response window for fluorescence detection.

Through systematic optimization of PFLs, we developed six distinct PVK PDs, including broadband perovskite detectors (PVK-PD) and narrowband fluorescence detectors incorporating PFLs (PVKf-PDs). As illustrated in Fig. 2f, the spectral response curves of these six PDs demonstrate that the incorporation of PFLs significantly suppresses the device’s detection capability for short-wavelength backscattered signals. The detailed variations in detector response parameters and corresponding signal suppression ratios are summarized in Table S1, with PVKf-PD5 exhibiting optimal signal separation performance among all devices. Furthermore, we characterized the transient photoresponse properties of PVK-PD, as shown in Fig. 2g, which exhibits an exceptionally fast rise time of 44 μs. The linear dynamic range (LDR) of PVK-PD was measured to reach 178.2 dB (Fig. 2h), enabling stable operation across an irradiance range from 5.3 × 10⁻⁶mW cm⁻² to 4.36 × 10³mW cm⁻² while maintaining detection performance. Additional performance characterizations of the PVK-PD are presented in Supplementary Figs. 4–6. These results collectively demonstrate the effective photovoltaic performance of PVK-PD.

To evaluate the practical performance of PVK-PDs in separating backscattered and fluorescence signals, we conducted signal measurements under real-world fluorescence imaging conditions. Figure 2i presents the fluorescence and backscattered signal intensities obtained from sodium fluorescein aqueous solution with varying concentrations (100, 50, 10, and 1 μmol/mL) under 450 nm excitation light at 11.09 mW cm⁻² intensity, with detailed values provided in Table S2. The results clearly demonstrate that the fluorescence signal intensity is significantly weaker than the backscattered signal, highlighting the critical requirement for PVKf-PDs to possess signal discrimination capability. Figure 2j1 demonstrates the fluorescence and background signal intensities detected by the broadband-responsive PVK-PD, revealing that the PD primarily captures backscattered signals with negligible fluorescence intensity, indicating its suitability for single-pixel background imaging. Figure 2j2–6 illustrates the detection capabilities of PVKf-PDs for both fluorescence and background signals, where progressive optimization of the PFL thickness leads to a complete inversion of signal intensity ratios in PVKf-PD5. The detailed signal intensity data are presented in Supplementary Table 2. This transition demonstrates the detector’s full capability for single-pixel fluorescence detection, with the optimized PVKf-PD5 exhibiting enhanced performance in selectively capturing fluorescence signals while effectively suppressing background interference. The systematic improvement across the device series highlights the critical role of PFL engineering in achieving high-purity fluorescence detection. The quantitative relationship governing this transition is delineated in Fig. 2k.

Imaging performance of PVK-SPI-FB System

Based on the aforementioned research findings, the PVK-PD and PVKf-PD5 are selected as the background signal detector and fluorescence detector, respectively, and subsequently integrated into our perovskite-based single-pixel imaging system for systematic evaluation of its imaging performance. Figure 3 presents the performance evaluation of our fluorescence/background dual-modal imaging system based on PVK single-pixel PDs. The two-dimensional images were reconstructed using a four-step phase-shifting Fourier algorithm. Specifically, by measuring the light intensity values D_θ (θ = 0, π/2, π, 3π/2) for each structured illumination pattern, we computed the Fourier coefficients α = [D₀-D_π] + j·[D_π/2- D_3π/2], and subsequently reconstructed the spatial-domain images via inverse Fourier transform (IFT), as illustrated in Fig. 3a. Supplementary Fig. 7 presents the schematic diagram of the imaging algorithm and detailed algorithmic principles.

Fig. 3. Fluorescence and background imaging results based on PVK PDs.

a FSPI principle. b, c. Investigation of the relationship between M and reconstructed image resolution under white excitation light at 477.46 mW cm⁻² irradiance. d, e Schematic diagram of imaging device and the imaging results obtained by PDs under 450 nm excitation light at 11.09 mW cm⁻² irradiance. f 3D intensity mapping of the corresponding boxed area in Fig. (e). g Functional relationship between the concentration of sodium fluorescein aqueous solutions and the grayscale value of the image.

In Fourier spectral analysis, the low-frequency components govern the overall image profile, whereas the high-frequency components encode fine structural details. Consequently, the number of projected structured illumination patterns (M), which directly correlates with the sampling rate, determines the resolution of the reconstructed images. Figure 3b, c demonstrates the system’s imaging performance for sodium fluorescein aqueous solutions (100, 50, 10, and 1 μmol/mL) loaded in Eppendorf (EP) tubes under visible-light illumination. Under visible-light illumination, the fluorescence signals were overwhelmed by backscattered light, resulting in reconstructed images that retained only the outlines of the EP tubes. Notably, as M increased, the image resolution improved significantly, with progressively sharper contours - a finding that validates the sampling theorem of FSPI.

Figure 3d schematically illustrates the dual-modal single-pixel fluorescence/background imaging system, which employs a 450 nm excitation light source. The imaging target, identical to that shown in Fig. 3c, consisted of four EP tubes containing sodium fluorescein aqueous solutions with sequentially decreasing concentrations from left to right. As demonstrated in Fig. 3e, the PVK-PD and PVKf-PD single-pixel detectors were employed for backscattered signal and fluorescence detection, respectively. The PVK-PD, serving as the background signal PD, failed to reconstruct complete EP tube contours due to severe localized glare signals caused by the smooth tube surfaces, resulting in substantially reduced image contrast. Notably, since the glare signals fundamentally originate from backscattered light, the progressively enhanced filtering capability of PVKf-PDs successfully suppressed these interference artifacts, thereby enabling clear fluorescence imaging. Furthermore, we performed image fusion between the white-light background image obtained in Fig. 3c and the fluorescence image captured by PVKf-PD5, as demonstrated in Fig. 3e–7. These results preliminarily illustrate the unique fusion advantages of our developed PVK-SPI-FB System. Notably, in practical biological fluorescence imaging scenarios, glare artifacts are typically absent, enabling fluorescence/background localization imaging under single-excitation illumination conditions. This capability will be comprehensively demonstrated in Fig. 4. Figure 3f displays 3D grayscale intensity mappings of the selected regions (boxed areas in Fig. 3e), providing quantitative evaluation of PVKf-PDs’ glare suppression capability. Figure 3g establishes a quantitative correlation between sodium fluorescein aqueous solution concentration and image grayscale values to evaluate PVK-SPI-FB System sensitivity.

Fig. 4. The spatial consistency advantages of the PVK-SPI-FB System and its application in tumor localization imaging via fluorescence-background fusion.

a, b Schematic diagram of a multi-camera imaging system and its spatial registration results. c, d Structural illustration of a multi-detector single-pixel imaging system and corresponding spatial registration outcomes (100% full sampling). e–g Commercial camera-captured images of tumor-simulating models injected in the left hindlimb, right hindlimb, and dorsal region of mice. h–p. Background imaging results(h-j), Fluorescence imaging results(k-m), Dual-modal fused images(n-p) of our designed PVK-SPI-FB System (30% sampling rate, M = 10000). q–s 3D intensity mapping corresponding to the fluorescent regions in images of k-m.

Based on the detection limit of our PVKf-PD5-based PVK-SPI-FB system and fitted curve analysis, the system demonstrates a detection threshold of 0.05 μmol/mL, highlighting its low-light detection capability. In addition, we evaluated the imaging sensitivity of the system (Supplementary Fig. 20). The system exhibited an LDR for imaging detection of 138.5 dB, with a minimum detectable fluorescence intensity of 3.08 × 10⁻⁴ mW cm⁻². Furthermore, owing to the structured light's robust spatiotemporal encoding, the imaging system exhibits inherent interference resistance and robustness. Ambient surgical lighting has minimal impact on the system’s imaging performance, as demonstrated in Supplementary Fig. 21.

Biomedical imaging results of PVK-SPI-FB System

Conventional multi-camera imaging systems are inherently plagued by spatial registration challenges. We designed an experiment in which images of the same object were acquired at three fixed positions using a camera (Fig. 4a, b) and an SPI system (Fig. 4c, d), respectively, in order to experimentally demonstrate the unique advantage of SPI in spatial registration. The SPI imaging setup adopted the same active SPI architecture as shown in Fig. 1d, with a projector serving as the spatial light modulator (SLM), and the remaining structure was consistent with that in Fig. 1d. As illustrated in Fig. 4a, b, the inability to maintain strict co-axial alignment among different cameras introduces variations in the optical paths of reflected light received by each device. This inevitably leads to FOV misalignment and spatial information discrepancies in the acquired images, posing significant obstacles for multi-modal image fusion. In stark contrast, our developed PVK-SPI-FB System ensures complete spatial registration and pixel-level correspondence across images acquired by PDs at different positions. Specifically, their spatial information is entirely determined by the pre-defined structured light patterns, rendering them independent of PD positioning. This unique characteristic enables multiple PDs positioned at distinct spatial locations to acquire image data with perfectly matched FOV and pixels alignment (Fig. 4c, d, Supplementary Fig. 8). The experiments in Supplementary Fig. 9 demonstrate the system’s robust non-line-of-sight imaging capability and further corroborate the aforementioned conclusions. Above experimental results demonstrate that these images can be seamlessly fused through direct superposition, unequivocally validating the distinctive advantages of single-pixel imaging for multi-modal image integration.

Building upon these foundations, we designed an in murine model to validate the practical utility of this imaging methodology. The experimental setup is illustrated in Fig. 1d and Supplementary Fig. 10. Three tumor-simulating models were established through injections in the left hindlimb, right hindlimb, and dorsal region of the mice, with sodium fluorescein serving as the tumor-marker via local injection. Consequently, the luminescent areas in fluorescence images exclusively represented the tumor-simulating regions. Prior to formal experiments, this study conducted preliminary validation tests using chicken wings, with the results presented in Supplementary Fig. 11. Our system successfully acquired both diffuse reflectance background images and specific fluorescence images simultaneously (Fig. 4e–m). As demonstrated in Fig. 4n–p, the fused images enabled clear discrimination between marked regions and normal tissues. Quantitative analysis of fluorescence signal intensity distributions permitted precise determination of both geometric dimensions and spatial coordinates of the tumor-simulating models, thereby providing reliable data support for surgical navigation applications.

To meet the real-time performance requirements of surgical navigation systems, we further improved the imaging speed of the system from the original 1 frame per second (fps) to 10 fps by reducing the sampling rate and designing a hardware-software co-design architecture. The detailed optimization process is provided in the Supporting Information (Section 3). Reducing the total number of projections from 10,000 to 1000 yielded a boost in imaging speed yet incurred a significant degradation in image quality (Fig. 3c). To mitigate such image quality deterioration, we developed a dedicated image enhancement algorithm termed FSI-SwinIR. This artificial intelligence (AI) model was fine-tuned from the classic SwinIR model on a Fourier single-pixel imaging dataset. The architectural design and underlying working principles of this model are elaborated in Supplementary Fig. 13, with the source code of FSI-SwinIR attached in Supplementary Code 1. As illustrated in Fig. 5a, the ultra-low undersampled Fourier single-pixel images processed by FSI-SwinIR achieved significant enhancement, which effectively counteracted the degradation in imaging performance caused by the pursuit of high-speed single-pixel imaging. Figure 5b further demonstrates the capability of FSI-SwinIR to enhance Fourier single-pixel images under different undersampling ratios.

Fig. 5. Real-time fluorescence localization and tracking imaging.

a A transformer-based image enhancement architecture fine-tuned with a Fourier single-pixel image dataset (FSI-SwinIR). b Image restoration at different undersampling rates is performed using FSI-SwinIR. c Under blue light irradiation, SPI notes written with fluorescent (upper) and black (lower) markers were tracked. PVK PD visualized black notes and marker outlines, while PVKf PD only detected fluorescent notes. FSI-SwinIR-based restoration of fused images enabled real-time fluorescence/background fused localization imaging. d Real-time dynamic fluorescence localization/tracking imaging of tumor marker-bearing mice. The part of AI for fusion is obtained by video post‑processing.

To demonstrate the dynamic fluorescence localization and tracking capabilities of our single-pixel video imaging system in real-world scenarios, two separate experiments were conducted: fluorescent note tracking and dynamic tumor marker tracking.

Supplementary Movies 1 and 2 demonstrate the one-click imaging functionality, acquired using 10,000 projected patterns. Figure 5c presents the tracking results of the SPI notes inscribed with a fluorescent marker (upper) and a black marker (lower) under blue light irradiation. Specifically, the PVK photodetector (PD) enabled the visualization of both the black notes and the marker contours, whereas the PVKf PD only allowed the observation of fluorescent notes. We additionally applied the FSI-SwinIR algorithm to restore the fused images, which yielded a real-time imaging performance for the fused localization of fluorescent targets and background information. Notably, real-time imaging has currently been achieved only for image acquisition and reconstruction, while AI enhancement is implemented as a post-processing procedure. In the future, we will accelerate both the single-pixel image reconstruction algorithm and the AI algorithm using GPUs, enabling full-process, real-time imaging. Figure 5d displays the in vivo imaging results of live mice bearing tumor markers, with the enhanced imaging outputs by FSI-SwinIR provided simultaneously. Benefiting from the real-time imaging capability for fluorescence/background fusion of the proposed system, the proposed system enabled precise positional mapping of tumor markers on the mice with high clarity, even when the mice exhibited slight spontaneous movements. The video imaging results are presented in Supplementary Movies 3, 4. This robust dynamic fluorescence-labeled imaging technology holds clinical significance for the implementation of tumor resection surgeries in real clinical scenarios.

Discussion

This study developed a real-time fluorescence-background dual-modal video imaging system based on perovskite-based single-pixel PDs, successfully addressing the critical challenge of multi-modal image fusion in conventional surgical navigation. Our single-pixel imaging system inherently overcomes the pixel registration and spatial alignment challenges inherent in conventional dual-camera fluorescence-background fusion imaging, enabling precise tumor fluorescence localization and tracking. To address the spectral overlap between fluorescence and background signals, we developed self-filtering PVK-PDs that achieved efficient spectral splitting between fluorescence emission (520 nm) and backscattered excitation light (450 nm). The optimized PVKf-PD demonstrates enhanced performance, with a backscattered light suppression ratio of 57 times (evaluated as the responsivity ratio at 640/450 nm, see Supplementary Table 1) and a fluorescence sensitivity of 107 mA W⁻¹ at 520 nm (PVKf-PD5). Ultimately, in murine experiments successfully overcame the detection challenges posed by tissue-induced fluorescence attenuation, enabling precise localization and tracking imaging of tumor-simulating models. This work presents an imaging solution for intraoperative real-time navigation, characterized by hardware simplicity, cost-effectiveness, and improved performance, demonstrating significant potential for applications in surgical oncology and precision medicine.

Methods

PVK PDs preparation

The fabrication process of the Cs_0.05FA_0.79MA_0.16PbI_2.5Br_0.5 PVK PD was carried out as follows: First, a homogeneous precursor solution was prepared by dissolving 516 mg lead iodide(PbI₂), 171 mg formamidinium iodide (FAI), 23.2 mg methylammonium bromide (MABr), and 73.4 mg lead bromide (PbBr₂) in 1 mL of a N,N-Dimethylformamide(DMF)/dimethyl sulfoxide (DMSO) mixed solvent (3:1 v/v) with continuous stirring at 60 °C for 5 h, followed by the addition of 39 μL cesium iodide (CsI)(390 mg/mL in DMSO) and subsequent overnight stirring at 60 °C. The device was fabricated through sequential spin-coating of functional layers: A poly(triarylamine) (PTAA) hole transport layer was first deposited at 4000 rpm for 30 s, followed by thermal annealing at 100 °C for 10 min. The perovskite active layer was then formed via a two-step spin-coating process - initial coating at 500 rpm for 6 s followed by 5000 rpm for 45 s, with 200 μL chlorobenzene anti-solvent dripping during the last 10 s, and subsequent annealing at 110 °C for 60 min. Subsequently, a [6,6]-phenyl-C61-butyric acid methyl ester (PCBM) electron transport layer was spin-coated at 2000 rpm for 30 s and annealed at 65 °C for 3 min, followed by deposition of a bathocuproine (BCP) modification layer at 3000 rpm for 30 s. Finally, a 100 nm-thick Ag electrode was thermally evaporated to complete the device fabrication. By spin-coating MAPbBr_2.4Cl_0.6 precursor solutions with concentrations of 0.75, 1, 1.26, 1.5, and 1.75 mol/L, PVK filter layers with different optical filtering effects were obtained. All ingredients were purchased from Xi’an Polymer Light Technology Corp Company and accurately weighed according to a certain stoichiometric ratio.

Imaging system

The schematic diagram of the experimental setup is shown in Fig. 1d, which mainly consists of a SLM, two PVK PDs (PVK-PD and PVKf-PD), a data acquisition system, a computer, and a white light source. The excitation blue light (~435–465 nm) was obtained by filtering the white light, with an illumination intensity of about 2.37 mW/cm² at the PD position. A DMD (V-7001 VIS, ViALUX) with a projection rate of up to 22 kHz was used as the SLM. The data acquisition system comprises a VK7016-Pro (VKINGING Company) and two PDA200C amplifiers (THORLABS Company). The photocurrent signals are first filtered and amplified by the amplifier, and then fed into the data acquisition card, which samples at a rate of 250,000 Hz. The specific imaging process is as follows: First, a set of four-step phase-shifted sinusoidal patterns was generated by the computer and loaded onto the DMD for structured light modulation. The PVK-PD and PVKf-PD collected and recorded the light intensity values of each structured light pattern through a multi-channel acquisition system. The set of current intensities was referred to as the I-m curve, where m is the pattern number. Finally, the I-m curve was processed using a Fourier transform algorithm to reconstruct the fluorescence and background images. A simple superposition of the two images yielded the fused image. The principle of the Fourier transform algorithm is detailed in the Supporting Information, as shown in Supplementary Fig. 7. The source code of FSPI has been included in Supplementary Code 2. To achieve high-quality real-time imaging capability, we proposed a physically fused transformer architecture (FSI-SwinIR) for enhancing Fourier-domain images under ultra-low undersampling rates. The detailed principles and training methodology of this architecture are illustrated in Supplementary Fig. 13, and the source code is provided in Supplementary Code 1. Furthermore, to enable one-click imaging without the need for post-processing, we developed a hardware-software co-designed framework (Supplementary Fig. 17), which achieves real-time synchronized fluorescence/background localization imaging. For the detailed process, please refer to Supplementary Movies 1 and 2.

Imaging experiments

First, in vitro experiments were conducted using sodium fluorescein aqueous solutions at four concentrations (100, 50, 10, and 1 μmol/mL) as imaging targets. Subsequently, murine experiments were performed by mixing agar solution with sodium fluorescein aqueous solutions at a 1:1 volume ratio. A 150 μL aliquot of the mixture was subcutaneously injected into the right hind limb, left hind limb, and back of mice to establish tumor-mimicking models, The intensity of subcutaneous fluorescence in the mouse model was measured, with values of (~ 0.712 μW/cm²). In the pen-stroke tracking experiment, the setup involved a black marker and a blank marker filled with sodium fluorescein solution. The fluorescence intensity varied with motion, reaching a maximum of approximately 1.76 10⁻²mW/cm². The experimental animals were four-week-old female KM and BALB/c-nu mice (SPF grade) purchased from Chengdu Dossy Experimental Animals Company. The mice were housed under controlled conditions with a temperature of 22 ± 1 °C and a humidity of 50 ± 5%, in group cages (4-5 mice per cage). Prior to experiments, the mice were acclimatized for at least 7 days with ad libitum access to a standard laboratory diet and sterilized water. All animal experiments were approved by the Experimental Animal Ethics Committee of Chengdu Dossy Experimental Animals Company (Approval No. CDDEACL2024-16) and conducted in accordance with the Guidelines for the Care and Use of Laboratory Animals. Under the aforementioned experimental system and conditions, we successfully achieved simultaneous tracking of both fluorescent and black marker pen strokes, as well as dynamic fluorescence tracking in tumor-bearing mice, resulting in high-quality video imaging outcomes, as shown in Supplementary Figs. 18, 19 and Supplementary Movies 3, 4.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Author contributions

Conceptualization: Y.J.L., H.L.W., Z.J., and X.L.C. Methodology: Y.J.L., H.L.W., X.Y.Y., Z.H.Z., J.J.L. Investigation: Y.J.L., H.L.W., X.P.H., J.Y.X., H.Y.Z., J.Q.W. Visualization: Y.J.L., H.L.W., H.W., J.Y.T., X.P.H. Funding acquisition: Y.J.L., Z.J., X.L.C. Supervision: Y.J.L., H.L.W., Z.J., X.L.C. Writing - original draft: YJL, HLW Writing - review & editing: Y.J.L., H.L.W., Z.J., X.L.C.

Peer review

Peer review information

Nature Communications thanks the anonymous reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Funding

This work was supported by the National Natural Science Foundation of China (62405236, 62275210, 62375210); the National Leading Talent Program, the National Young Talent Program, the Key Research and Development Program of Shaanxi (2024SF2-GJHX-25), the Shaanxi Young Top-notch Talent Program; Xidian University Specially Funded Project for Interdisciplinary Exploration (TZJH2024059, TZJH2024060); Fundamental Research Funds for Central Universities (QTZX24079).

Data availability

The data supporting the findings of this study is a vailable in the main text and Supplementary Materials.

Code availability

The codes used in this study are available within Supplementary Files. And they has been uploaded to GitHub:https://github.com/W-HL-lab/Supplementary-Codes-1-and-2.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s41467-026-74142-5.