Video-rate gigapixel ptychography via space-time neural field representations

Abstract

Achieving gigapixel space-bandwidth products (SBP) at video rates represents a fundamental challenge in imaging science. Here we demonstrate video-rate lensless ptychography that overcomes this barrier through the co-design of optics, sensing scheme, and computation: a coded surface encodes the dynamic object, time-sequential sensing spreads its information across frames, and a space-time neural-field framework recovers the dynamics by exploiting spatiotemporal correlations. Our approach transforms SBP scaling from sequential measurements to efficient correlation extraction. We demonstrate video-rate gigapixel imaging with centimeter-scale coverage while resolving 308-nm linewidths, achieving an SBP throughput of 20.8 gigapixels per second. Experimental validations span from monitoring mesoscale dynamics of snowflakes, bacteria, stem cells, microneedles, to characterizing time-varying dynamics in extreme-ultraviolet (EUV) experiments, demonstrating versatility across wavelengths. By transforming temporal variations from a constraint into exploitable correlations, we enable single-sensor gigapixel video that extends naturally to short-wavelength and electron regimes where radiation sensitivity has traditionally precluded high-resolution dynamic imaging.

Researchers demonstrate a mesoscale lensless imaging system that records complex wavefields at video rate from a coded sensor, reaching a throughput of 20.8 gigapixels per second while resolving sub-micron detail across centimeter-scale scenes.

Introduction

The space-bandwidth product (SBP) quantifies the total number of effective pixels during acquisition, defining the intrinsic information limit of imaging systems^1,2. Achieving gigapixel-scale SBP at video rates faces two fundamental barriers: optical systems must collect and relay gigapixel-scale information, while sensors must capture and read out this information at camera framerates. Conventional lenses suffer from aberrations that grow polynomially with both aperture size and field angle. This scaling law forces a trade-off: high numerical aperture (NA) objectives minimize aberrations but cover only a small field of view, while wide-field lenses attempting large field coverage accumulate aberrations that degrade resolution. It is challenging to achieve both wide field coverage and high resolution for delivering gigapixel SBP. Similarly, image sensors face physical constraints on pixel count, size, and readout speed that prevent gigapixel acquisition at video rates. Together, these optical and sensor limitations in SBP create a seemingly insurmountable barrier to video-rate gigapixel imaging.

Current computational imaging methods overcome the static SBP limit through sequential acquisition strategies. Whole slide scanners mechanically tile high-resolution images across real space^3–6, while Fourier ptychography synthesizes apertures in reciprocal space through angle-varied illumination^7,8. Both achieve gigapixel SBP but require minutes to acquire the dataset. These approaches fundamentally treat gigapixel SBP as a multiplexing problem, where acquisition time scales linearly with real- or reciprocal-space coverage. Coded aperture temporal imaging offers an alternative temporal multiplexing strategy by mechanically translating coded masks during exposure to compress video sequences into snapshot measurements⁹. However, this approach addresses temporal compression without achieving the resolution improvement necessary for gigapixel SBP, thereby exchanging one bottleneck for another. In contrast to sequential multiplexing, multiscale optical designs employ a large objective lens that collects light from a wide angle, then relays this to an array of smaller lenses with detectors that each image a portion of the field¹⁰. This architecture demonstrates that lens speed and field of view can be scale-independent, enabling snapshot gigapixel photography^11,12. Likewise, array microscopy pursues the same parallel acquisition goal through distributed imaging systems and requires proportionally complex multi-camera hardware^6,13–16. Whether through time-multiplexed sequential acquisition or hardware-intensive parallelization, the fundamental constraint remains: achieving gigapixel SBP demands either sacrificing temporal resolution or substantially increasing system complexity, making single-sensor gigapixel video acquisition seemingly impossible.

This barrier persists only as long as the missing bandwidth must be physically acquired. Our key observation is that the missing bandwidth can be recovered from the correlations inherent in dynamic scenes. To exploit this, we build on neural field representations, which parameterize continuous space-time volumes via coordinate-based networks^17–32, efficiently encoding high-dimensional data while naturally regularizing inverse problems. By representing dynamic scenes as continuous neural fields, we can exploit the rich spatiotemporal correlations inherent in videos. Beyond neural field representations, deep image priors have been explored for dynamic Fourier ptychography in simulation³³, motivating the broader question of how to achieve gigapixel SBP at video rate.

Here, we demonstrate video-rate gigapixel ptychography through the co-design of optics, sensing scheme, and computation to address the traditional SBP scaling challenges. A coded surface encodes full-field spatial information in every snapshot, free of the aberration issues of conventional lenses; time-sequential sensing distributes this information across frames; and a space-time neural field recovers the dynamics by exploiting spatiotemporal correlations, so that each measurement simultaneously enhances spatial resolution toward gigapixel scales while maintaining video-rate temporal sampling. This fundamentally changes how SBP scales: it grows with correlation extraction rather than increasing system complexity or time.

Our system resolves 308-nm linewidths across centimeter-scale fields at 30 frames per second (fps), corresponding to an SBP throughput of ~20.8 gigapixels per second (30,480 × 22,800 pixels × 30 fps). This imaging throughput is achieved from a single coded sensor, without any lens or multi-camera hardware. Importantly, the approach generalizes across wavelengths, successfully recovering time-varying probe beams in extreme ultraviolet (EUV) experiments where wavefront instabilities challenge conventional methods³⁴. It also naturally extends to electron^35,36 and X-ray ptychography³⁷, where radiation-induced sample evolution presents analogous spatiotemporal challenges that require robust dynamic phase retrieval³⁸.

Results

Video-rate ptychography with physics-informed neural fields

Figure 1 shows the video-rate ptychography framework and its experimental validation. The architecture (Fig. 1a, top panel) factorizes the space-time volume into hash-encoded spatial features²⁰ and compact temporal features^20,22,26, compressing gigabytes of data into megabyte-sized representations without involving motion models. To efficiently represent spatial coordinates, we employ multi-resolution hash encoding with 16 levels and two features per level (Methods). At any spatiotemporal point, these hash-encoded spatial features ($w\times h\times 32$ channels) combine with temporal features ($32\times T$ channels) through Hadamard product before feeding dual multi-layer perceptrons (MLPs) that decode real and imaginary field components separately. We adopt this real-imaginary parameterization rather than conventional amplitude-phase representations to maintain continuous, smooth functions throughout the reconstruction volume, avoiding the non-differentiable 2π discontinuities at phase-wrapping boundaries that trap optimization in local minima. This continuity ensures stable gradient flow during optimization, enabling successful convergence even for samples with large phase variations where amplitude-phase methods produce artifacts and convergence failures (Supplementary Fig. S1a-d vs. e-h).

Fig. 1. Space-time neural fields for video-rate gigapixel ptychography.

a Top panel: Neural field architecture showing low-rank decomposition of space-time volume into spatial and temporal feature tensors. The temporal basis features ${\tau }_i\left(i=1\cdots T\right)$ define a compact representation of temporal variation, from which continuous temporal features τ ̃_t are interpolated for arbitrary time indices. The spatial feature tensor is multi-resolution hash-encoded from spatial coordinates into feature vectors and reshaped into a two-dimensional array. Each spatial feature vector then performs a Hadamard product with the same temporal feature τ ̃_t. Each combined feature vector is fed into dual MLPs that output the real and imaginary components of the complex field at that spatial coordinate, and all outputs together form a reconstructed frame at time t. Bottom panel: Physics-informed optimization pipeline adopts the gradient-domain loss to update the MLPs’ weights and feature tensors. b Experimental setup where a 405-nm laser diode illuminates the dynamic object. A coded sensor is translated by compact actuators for diffraction data acquisition. Inset shows the coded sensor chip with the coded surface serving a spatially extended ptychographic probe beam. c Resolution characterization of a USAF target moving at ~0.05 mm/s. d Raw diffraction image stack captured during target translation. e Time-resolved reconstruction resolving the 308-nm linewidth of the group 10 element 5, corresponding to a full pitch resolution of 616 nm. In the reconstruction, each raw pixel is up-sampled into 6 by 6 sub-pixels.

The optimization pipeline (Fig. 1a, bottom panel) employs a gradient-domain loss^39–44 operating on spatial derivatives of amplitude differences in real space: ${\mid \mid \nabla \sqrt{J_t\left(x,y\right)}-\nabla \sqrt{I_t\left(x,y\right)}\mid \mid }_2$, where $J_t\left(x,y\right)$ denotes the measured intensity at time t, $I_t\left(x,y\right)$ denotes the estimated intensity from the forward imaging model of ptychography, ∇ denotes the gradient operator, and ||·||₂ denotes the L₂ norm. This gradient-domain formulation differs from conventional absolute-amplitude losses, providing enhanced robustness for dynamic phase reconstruction. The loss back-propagates through the dual MLPs to update both network weights and the space-time feature tensors. The physics-informed approach ensures that learned features respect optical constraints while requiring no data pre-training. The synergistic combination of low-rank feature decomposition, gradient-domain loss, and the real-imaginary formulation enables robust convergence for challenging dynamic samples with rapid phase variations that exceed multiple wavelengths—conditions in which conventional iterative phase retrieval methods fail.

Our experimental configuration (Fig. 1b) employs a distinct measurement strategy compared to conventional or Fourier ptychography, where confined measurements are acquired in either real or reciprocal space. Dynamic specimens are illuminated by a laser diode without additional optics, and we acquire images by translating a coded sensor beneath the dynamic object using voice-coil actuators. The coded surface on the image sensor, featuring randomized intensity and phase scatters, serves as an effective probe function as in conventional ptychography^6,38. The system calibration and positional tracking process are detailed in Supplementary Note 1. At each coded sensor position, the system generates a unique diffraction pattern by modulating the object’s exit wave field. Each of these measurements encodes full-field spatial information across all frequencies, in contrast to sequential, confined sampling as in conventional or Fourier ptychography. The imaging model and conventional phase retrieval strategies are detailed in Supplementary Note 2.

In Fig. 1c, we test the resolution performance by translating a USAF target at ~0.05 mm/s. We acquire the raw images for a duration of 50 s, at 30 frames per second (Fig. 1d). Critically, the neural field framework leverages the entire 50-second dataset to jointly recover both spatial and temporal features. As shown in Fig. 1e, we resolved the 308-nm linewidth at different time points (detailed in Supplementary Note 3), corresponding to a full pitch resolution of 616 nm. This collective optimization strategy is essential: attempting reconstruction from a single (or a few) diffraction pattern yields an underdetermined system that cannot converge. By simultaneously processing all temporal data using space-time factorization, our framework exploits temporal redundancy to achieve robust convergence for dynamic samples.

Video-rate gigapixel imaging at the mesoscale

Snowflake imaging has fascinated microscopists since Wilson Bentley captured the earliest photograph in 1885, establishing that no two snowflakes are identical. Despite over a century of advances in microscopy, it is challenging to simultaneously achieve centimeter-scale coverage, nanometer-scale phase sensitivity, sub-micrometer spatial resolution, and video-rate temporal sampling necessary to reveal the melting dynamics of fragile ice crystals. The reported framework overcomes these fundamental limitations by factorizing the entire measurement sequence into shared spatial and temporal features and exploiting temporal redundancy to regularize the ill-posed dynamic phase retrieval problem. Figure 2a, b, Supplementary Fig. S2, and Supplementary Movies 1, 2 demonstrate this capability across snowflakes of varying sizes and morphologies. The recovered phase maps quantitatively track melting front propagation and reveal the evolution of water film thickness. Fine dendritic structures remain clearly resolved throughout the thermal transition, demonstrating the framework’s ability to maintain spatial fidelity during morphological changes. Complementary to melting dynamics, crystallization from solution presents phase accumulation challenges as material thickness rapidly increases. We demonstrate video-rate gigapixel monitoring of crystal formation across the entire sensor field of view, capturing the complete evolution from dendritic expansion to polycrystalline film formation at mesoscale (Fig. 2c, d). This reconstruction spans gigapixel-scale SBP, simultaneously resolving sub-micron crystal striations while maintaining centimeter-scale coverage across the entire sensor (Supplementary Figs. S3-S4).

Fig. 2. Video-rate, gigapixel imaging of dynamic melting and crystallization processes enabled by joint spatiotemporal optimization of neural fields.

a Recovered amplitude (left) and phase (right) of the snowflake melting process, showing preserved dendritic structure across a large field of view. Snowflake melting imaging was performed on 5 independent specimens with similar results; representative results are shown. b Time-lapse phase imaging over the 50-second duration, revealing melting front propagation at selected time points. c Gigapixel-scale crystal formation process over the entire ~40 mm² sensor field of view. Crystallization imaging was performed on 4 independent runs with similar results; representative results are shown. d Temporal sequence showing crystal formation process at selected time points. The reconstruction leverages all temporal data through space-time neural fields, enabling artifact-free high-resolution imaging not achievable with conventional frame-by-frame approaches. Supplementary Fig. S2 and Movies 1-2 show the melting processes of large and small snowflakes. Supplementary Figs. S3-S4 and Movie 3 show the recovered gigapixel-scale video of the crystal formation process.

Critical to the imaging performance of video-rate ptychography is the synergistic choice of loss function and optical field representation. We demonstrate this using the crystallizing sample, where rapid phase accumulation creates optical path differences exceeding many 2π wraps. Supplementary Fig. S5 systematically evaluates four loss functions (L1, L2, smooth L1, and gradient-domain loss) across both amplitude-phase and real-imaginary representations (Supplementary Fig. S5a–h). Only the gradient-domain loss with real-imaginary representation achieves successful convergence with artifact-free reconstruction; all other combinations fail to converge or exhibit artifacts. We also compare the reported neural field framework against common iterative phase retrieval algorithms, including ePIE^45,46, momentum-PIE (mPIE)⁴⁷, Quasi-Newton⁴⁸, and least-squares⁴⁹ methods (Supplementary Fig. S5i–l), where conventional algorithms are initialized using a static reconstruction before temporal refinement⁵⁰ (Supplementary Note 2). All conventional methods exhibit artifacts and convergence failures in some regions. Supplementary Movie4further compares the reconstructed videos between the reported framework and other common iterative phase retrieval methods. This comparison reveals that frame-by-frame reconstruction is fundamentally underdetermined for dynamic imaging. In contrast, our space-time neural field jointly optimizes across all temporal measurements, exploiting spatiotemporal correlations through shared low-rank features to transform the severely underdetermined problem into a tractable one. Supplementary Fig.  S6 further confirms these advantages using a microneedle patch, in which the reported framework demonstrates improved reconstruction quality.

Live-cell dynamics monitoring with post-measurement digital refocusing

Video-rate ptychography enables label-free monitoring of cellular processes by detecting quantitative phase variations without the phototoxicity associated with fluorescent labeling. To demonstrate this capability, we monitored bone marrow mesenchymal stem cells during wound healing assays within an incubator environment (Fig. 3a). Our mesoscale gigapixel phase reconstructions reveal detailed cellular morphology and intercellular structures across the entire sensor field of view, with the recovered height maps resolving tunneling nanotubes that form critical communication pathways between cells (Fig. 3b). The high temporal resolution captures complete cell division dynamics, revealing characteristic mitotic rounding accompanied by progressive phase shifts that precede cytokinesis (Fig. 3c). In Fig. 3d, our system can also visualize intercellular transport along tunneling nanotubes – events typically challenging to observe without fluorescent markers. In Fig. 3e, we induced necrosis through hypotonic shock, and the resulting time-lapse sequences capture rapid cellular swelling followed by catastrophic membrane rupture, enabling quantitative monitoring of cell death kinetics.

Fig. 3. Live stem cell monitoring within an incubator.

a Experimental setup showing the portable video-rate ptychographic sensor in an incubator. b Reconstructed mesoscale height map showing wound healing process of the bone marrow mesenchymal stem cells. The white arrow highlights the recovered tunneling nanotube with high spatial resolution. c Cell division sequence showing mitotic rounding, cytokinesis, and daughter cell separation. d Intercellular transport through tunneling nanotubes. e Cell necrosis showing rapid swelling followed by membrane rupture. Supplementary Movies 5-7 show the videos of cell division, intercellular transport, and necrosis. Supplementary Figs. S7-S8 and Movie 8 show the post-measurement refocusing capability.

A distinctive advantage of video-rate ptychography is its post-acquisition digital refocusing capability, which is essential for dynamic samples where focal planes vary during acquisition. Unlike conventional ptychography with translated probe beam and Fourier ptychography with angle-varied illumination, which require modeling the object-probe interaction under the thin-object approximation^51,52, our system directly reconstructs the optical field exiting the sample. This fundamental difference enables digital propagation of the recovered complex field to arbitrary axial positions post-measurement, without the thin-sample approximations that constrain traditional ptychographic approaches. Supplementary Fig. S7 validates this capability by computationally refocusing microneedles and live stem cell cultures, revealing how different structural features achieve optimal focus at distinct depths. Supplementary Fig. S8 compares fixed-plane reconstructions with dynamically refocused sequences during necrosis, demonstrating improved image quality through adaptive focusing. Supplementary Movie 8 shows the post-measurement refocusing process of a microneedle patch. By eliminating real-time autofocus requirements, the post-acquisition flexibility simplifies experimental protocols while enabling retrospective 3D analysis of archived datasets⁵³.

In Fig. 4, we also demonstrate the platform’s versatility through longitudinal tracking of bacterial growth with implications for antibiotic susceptibility testing (AST). Time-lapse phase imaging captures E. coli proliferation within a free-flow microfluidic channel, where antibiotic gentamicin treatment induces filamentation—some cells elongate while continuing to grow. This morphological response, visible as extended thread-like cells in Fig. 4c–g, represents a survival mechanism that protects bacteria from antibiotics and promotes virulence factors, including biofilm formation. Our platform tracks both this elongation phenotype and biomass accumulation from sparse inoculation to dense coverage (Fig. 4a–g and Supplementary Movie 9). The quantitative phase signal provides direct biomass measurements, with phase shifts linearly correlating to cellular dry mass density (Methods). Despite cells elongating into filaments rather than dividing normally, we observe exponential mass accumulation in Fig. 4h, revealing how bacteria maintain growth even under antibiotic stress. This label-free quantification could accelerate clinical decision-making by detecting resistance phenotypes hours before conventional culture methods.

Fig. 4. Quantitative monitoring of bacterial growth dynamics in liquid culture.

a–g Time-lapse phase imaging of E. coli proliferation within a free-flow microfluidic channel. Gentamicin added in the culture medium induces filamentation, causing some cells to elongate without dividing – a behavior associated with antibiotic resistance and biofilm formation. Magnified regions in (e-g) highlight individual cells. h Quantitative dry mass accumulation shows exponential growth. The measurement enables label-free growth kinetics quantification in flowing liquid culture with picogram sensitivity. Supplementary Movie 9 shows the real-time bacterial growth process in liquid culture. Supplementary Movie 10 shows the real-time bacterial growth on solid agar plate.

Complementing liquid culture monitoring, we extended our approach to solid-phase bacterial growth on agar plates, achieving centimeter-scale field-of-view while maintaining high temporal resolution (Supplementary Fig. S9 and Movie 10). This configuration monitors colony formation and expansion on nutrient agar surfaces with high phase sensitivity. The reported platform thus bridges multiple culture formats, from microfluidic liquid cultures to traditional agar plates, providing continuous, quantitative monitoring across different growth conditions without phototoxicity constraints.

Longitudinal 3D characterization of biomedical devices

Transdermal microneedles enable drug delivery via painless self-administration, yet their clinical translation requires standardized quality-control methods that are currently absent from pharmaceutical practice. We demonstrate how video-rate ptychography addresses these unmet needs through non-destructive 3D characterization of microneedle arrays in Fig. 5. The recovered phase maps in Fig. 5a cover the entire centimeter-scale microneedle patch containing hundreds of individual needles. Phase unwrapping converts the wrapped phase to absolute height measurements, confirming 700-μm needle heights matching design specifications in Fig. 5b. The 3D surface reconstruction also shows uniform needle geometry with sharp tips essential for skin penetration, validating manufacturing consistency across the array. Supplementary Fig. S10 shows the mesoscale 3D characterization process. In Fig. 5c and Supplementary Movie 11, we perform time-resolved imaging to capture dissolution dynamics as microneedles interact with surrogate dissolution medium. Sequential 3D reconstructions track volumetric reduction, where individual needles show slightly varying dissolution rates, potentially reflecting local flow conditions or manufacturing variations. The volumetric data enables precise pharmacokinetic modeling, with dissolution curves extracted for each needle providing statistical distributions of release rates – essential data for regulatory approval. While conventional quality control requires destructive sampling at multiple timepoints, our approach provides continuous, non-invasive monitoring at video rates, transforming pharmaceutical characterization from endpoint testing to real-time validation.

Fig. 5. Longitudinal 3D characterization of microneedle patch.

a Transdermal microneedle patch and its corresponding phase map covering a centimeter-scale field of view. The zoomed-in view shows the wrapped phase for height extraction. b 3D reconstruction validating the 700-μm design height with sharp tip geometry preserved across hundreds of needles. c Time-resolved dissolution monitoring showing progressive height reduction. d Graph quantifies dissolution kinetics through volumetric measurements, enabling pharmacokinetic modeling of drug release profiles. Supplementary Fig. S10 shows mesoscale characterization of the microneedle patch. Supplementary Movie 11 shows the dissolution dynamics.

Characterizing time-varying dynamics in extreme-ultraviolet (EUV) ptychography

A key property of the neural field framework is its agnosticism to which component of the forward model is time-varying: the same temporal feature factorization that captures dynamic objects in the visible light experiments can equivalently represent time-varying probe beams while keeping the object static. To validate this flexibility beyond visible light, we tested it on EUV ptychography, where time-varying illumination presents severe reconstruction challenges³⁴. In Fig. 6, we demonstrate that the reported space-time neural fields achieve robust reconstruction even with reduced measurements and minimal iterations – capabilities essential for EUV, X-ray and electron imaging, where source stability and radiation damage are limiting factors.

Fig. 6. Time-varying probe characterization in EUV ptychography.

a Experimental setup showing a spiral phase mask for generating the probe beam on the resolution target, with the raw image captured in reciprocal space (250 frames in total, with only 24% overlap in-between adjacent measurements). b Sample reconstruction via conventional orthogonal probe relaxation using 1500 iterations. c The recovered first three principal modes of the varying probe beam. d Sample reconstruction via the reported video-rate ptychography framework using 40 iterations. e The recovered temporal evolution of probe beam across all frames. Supplementary Note 4 shows detailed comparisons with conventional methods at 1.2%, 10.3% and 24.4% overlap ratios. Supplementary Movie 12 shows complete time evolution of the recovered probe beam.

Figure 6a shows the EUV experimental setup using a spiral mask for probe generation and a resolution target as the sample³⁴. To test robustness under challenging conditions, we used a set of 250 raw diffraction patterns with only ~24% overlap ratio in Fig. 6 – significantly below the typical 50% overlap in conventional reconstruction. This reduced overlap ratio combined with probe variations during acquisition creates a demanding test case for ptychographic algorithms. As shown in Fig. 6b-c, conventional orthogonal probe relaxation (OPR) methods⁵⁴ struggle with these conditions. Figure 6b shows the OPR-reconstructed object with 1500 iterations, exhibiting artifacts and incomplete convergence. The recovered orthogonal modes in Fig. 6c attempt to capture probe variations but produce ambiguous decompositions. In contrast, our video-rate ptychography framework achieves successful reconstruction in 40 iterations (Fig. 6d). Rather than decomposing into orthogonal modes, our framework treats the probe beam as a temporal function across all 250 frames. Figure 6e shows the temporal evolution of the probe beam across different frames. This natural parameterization through temporal features provides stronger constraints than orthogonal decomposition, enabling robust reconstruction from sparse, unstable measurements. We further tested reconstruction limits by reducing overlap ratios to extreme conditions (Supplementary Note 4). At 10.3% overlap, the framework maintains successful object recovery with probe variations. Even at 1.2% overlap, the recovered object remains recognizable despite increased artifacts, demonstrating robustness against undersampling conditions. Supplementary Note 4 provides detailed comparisons across multiple algorithms and conditions. The success with challenging EUV data demonstrates our framework’s potential for electron, X-ray, and EUV ptychography, where minimizing exposure while handling beam variations remains critical⁵⁵.

Discussion

Ptychography reconstructs complex optical fields from overlapping diffraction measurements^38,56,57, achieving imaging capabilities across scales – from atomic-resolution electron microscopy³⁶, nanoscale X-ray ptychography³⁷, to meter-scale macroscopic Fourier ptychography⁵⁸. Whether the confined measurement constraint manifests in real space⁵⁹ or reciprocal space^7,8, ptychography faces an irreconcilable conflict: the data redundancy essential for phase retrieval necessitates sequential measurements with substantial overlaps that cannot capture instantaneous dynamic states. This fundamental bottleneck has limited ptychography to quasi-static samples despite its unique ability to surpass the physical limitations of lens-based optics. In electron and X-ray ptychography, this limitation becomes critical as radiation damage leads to structural changes during data acquisition, creating an inherent race against sample degradation^38,55,56.

Our approach addresses these limitations by coupling three elements: a coded surface that encodes full-field spatial information in every snapshot, time-sequential sensing that distributes this information across frames, and a space-time neural field that recovers the dynamics by exploiting the spatiotemporal correlations inherent in dynamic scenes. The design insight is that these are not three independent improvements but rather a single bandwidth-allocation strategy: the optics and sensing distribute spatial bandwidth along the temporal axis, and the space-time neural-field framework is the reason distributed bandwidth is recoverable. The advantage of the coupled design rests on a property of natural dynamic objects: they exhibit strong correlations, with spatial structures persisting across frames while temporal dynamics follow predictable patterns. By encoding these correlations through low-rank feature decomposition, each measurement simultaneously contributes to spatial resolution enhancement and temporal evolution tracking. This fundamentally changes the scaling relationship: SBP grows with the efficiency of correlation extraction rather than the number of independent measurements.

This implicit approach also contrasts with explicit motion-model-based methods such as space-time Fourier ptychography, which incorporates optical flow priors to correct for sample movement between frames⁶⁰. Such approaches rely on the brightness constancy assumption and parametric warping functions, which are effective for rigid translations but can struggle with complex non-rigid dynamics such as cell division, membrane rupture, or crystal branching. Our low-rank neural field factorization requires no motion estimation and naturally accommodates arbitrary spatiotemporal variations through shared temporal features.

The physics-informed architecture amplifies this efficiency through complementary algorithmic choices. Gradient-domain optimization exploits spatial-gradient sparsity and enables robust phase retrieval compared to conventional amplitude-based optimization. Real-imaginary parameterization maintains smooth optimization landscapes, avoiding phase-wrapping discontinuities. Hash-encoded spatial features provide memory-efficient multi-resolution representation, while temporal factorization captures dynamics without restrictive motion models. Together, these mechanisms enable reconstruction of gigapixel video streams at camera framerates.

Our diverse demonstrations validate capabilities extending from materials science to biomedicine, several of which are difficult to achieve with existing techniques. Time-resolved phase imaging of snowflake melting requires the simultaneous combination of centimeter-scale coverage, sub-micrometer resolution, and video-rate sampling–inaccessible to whole-slide scanners (too slow), high-NA objectives (field of view too small), conventional ptychography (static), and Fourier ptychography (thin sample only). We simultaneously track dendritic growth patterns across centimeter-scale fields at sub-micron resolution and 30 fps, enabling observation of both microscopic events and macroscopic pattern formation within the same dataset. In biomedical applications, label-free monitoring of stem cell dynamics revealed subcellular structures like tunneling nanotubes typically requiring fluorescent labeling, while continuous 3D monitoring of microneedle dissolution at video rate recovered volumetric pharmacokinetic data inaccessible to conventional approaches. The post-acquisition refocusing capability proves particularly valuable for dynamic samples where focal planes shift unpredictably, as observed during cell necrosis. These results are enabled not by any single algorithmic component in isolation, but by the co-design of a lensless coded surface that provides gigapixel-scale information capacity without thin-object constraints, a sensing scheme that distributes bandwidth across time, and a neural field framework whose synergistic combination of gradient-domain loss and real-imaginary parameterization achieves convergence where individual components fail (Supplementary Figs. S5-S6).

The reported approach also addresses fundamental challenges across ptychographic modalities. In EUV ptychography, source instabilities cause probe variations between acquisitions³⁴. We demonstrated successful recovery of both sample structure and time-varying probe evolution, treating illumination fluctuations as continuous temporal functions rather than discrete orthogonal modes. The framework’s robustness proves particularly valuable for short-wavelength regimes where minimizing exposure is critical. For X-ray and electron ptychography, radiation damage progressively alters samples during measurement, forcing a trade-off between dose and resolution. Our framework can explicitly model this radiation-induced evolution, enabling full-dose imaging while compensating for structural changes. Beyond microscopy, we can also handle atmospheric turbulence in long-range ptychographic imaging^61–63 by learning spatiotemporal phase distortions directly from the measurement sequence, eliminating the need for adaptive optics. For endoscopic implementations, we can model the fiber bundle modulation profile as a time-varying coded surface for synthetic aperture imaging^64,65. Across all scales, temporal variation transforms from a fundamental limitation into useful information that enhances reconstruction quality.

Future developments could extend the reported framework in multiple directions. Integration with event-based sensors could enable microsecond temporal resolution for capturing rapid cellular dynamics. Combining our spatial-temporal factorization with multi-wavelength measurements could provide spectroscopic information alongside morphological data^66,67. The neural field representation naturally extends to three-dimensional tomographic reconstruction by incorporating multiple illumination angles^68–70. Moreover, the compressed representation facilitates machine-learning analysis directly on latent features, potentially enabling real-time classification of cellular behaviors or pharmaceutical quality metrics. As sensors approach fundamental limits while computational power continues to grow exponentially, such approaches will increasingly define the frontier where algorithm sophistication overcomes physical constraints.

Methods

Video-rate gigapixel ptychographic sensing system

The video-rate ptychographic system is a compact, handheld device with the coded surface fabricated directly on top of the image sensor (onsemi AR2020, 5120 by 3840 pixels, 1.4 μm pixel size, 30 fps). The optical scheme is shown in Supplementary Fig. S11, where a 405-nm laser diode (Thorlabs, LP405-SF10) illuminates the sample with approximately 10-mW power, providing coherent illumination across the entire centimeter-scale field of view. The coded surface consists of a disorder-engineering pattern with sub-micron intensity and phase scatterers. It is fabricated directly on the sensor’s protective glass, facing the pixels and does not contact the sample, ensuring robustness against contamination and mechanical wear across extended use, including operation within incubator environments. The coded surface pattern is pre-calibrated using a static blood smear as a reference object via joint recovery with the ePIE algorithm ⁷¹. The sample is positioned at a distance of 0.2–2 mm above this surface. For dynamic imaging, the coded sensor translates continuously beneath the sample via magnetic actuation by voice coil actuators, achieving acquisition rates up to the sensor's framerate of 30 fps. The lateral positions $\left(x_t,y_t\right)$ at each time point are tracked using a static fiducial marker placed at the corner of the field of view. Cross-correlation between the fiducial’s diffraction pattern in consecutive frames provides sub-pixel registration accuracy for distinguishing sensor motion from sample dynamics. In Supplementary Note 1, we discussed the calibration process of the coded surface and the positional tracking of sensor motion. In Supplementary Note 2, we discussed the imaging model of our ptychographic system and conventional phase retrieval algorithms. To quantify the system’s information capacity, we evaluate its space-bandwidth throughput, defined as the number of resolvable pixels recovered per unit time. With a reconstructed field of 30,480 × 22,800 pixels per frame at 30 fps (Fig. 2, Supplementary Figs. S3-S4 and Movie 3), the system achieves a throughput of approximately 20.8 gigapixels per second from a single sensor. Supplementary Table 1 compares this value with representative computational imaging systems.

Space-time neural field representations

The space-time neural field framework decomposes the complex optical field into low-rank spatial and temporal components, addressing the fundamental challenge of jointly reconstructing all time points while maintaining computational tractability. As demonstrated in Supplementary Fig. S1, we employ separate neural field generators for real and imaginary components rather than amplitude-phase representation, avoiding phase-wrap discontinuities that would otherwise cause convergence failures at crystal boundaries and regions with rapid phase variations.

The spatial feature representation leverages multi-resolution hash encoding (Supplementary Note 3), a technique that captures image details at different scales simultaneously. The encoding employs $L=16$ hierarchical resolution levels, starting from a coarse pixel grid and progressively refining by a factor of 1.3 at each level. For example, the first level covers 128×128 pixels, the second 166×166 pixels, the third 216×216 pixels, and so on up to the 16th level. This multi-scale hierarchy ensures that both broad structural patterns and fine details are captured effectively. At each resolution level, the hash encoding extracts $F=2$ feature values per spatial location. Across all 16 levels, this yields 32-dimensional feature vectors that serve as compact spatial descriptors. These 32 numbers encode everything from coarse object boundaries to finest interference patterns, providing the neural network with rich multi-scale information to reconstruct the complex field. Supplementary Fig. S12 shows the 32 feature channels with real-imaginary splitting, which enables efficient representation of dynamic processes while maintaining spatial resolution. Supplementary Fig. S13 compares reconstruction performance with different activation functions in the neural field framework.

The hash table implementation as detailed in Supplementary Note 3 provides high storage efficiency for compressing the space-time volume. Rather than storing features at every pixel location, which would require billions of parameters for gigapixel images, the hash encoding uses a lookup table with up to 2²⁶ entries that are shared across all spatial positions through a hashing function. This achieves up to 1000-fold compression while maintaining high spatial resolution in reconstruction. The system maps normalized spatial coordinates (x, y)$\mathrm{\in }$ [0,1]² to these stored features through hash table lookups at surrounding grid vertices, with bilinear interpolation ensuring smooth transitions between discrete grid points (Supplementary Note 3).

For temporal representation, we employ $T=10$ learnable feature vectors ${\tau }_i\in {\mathbb{R}}^{32}\left(i=\mathrm{1,\; 2},\dots 10\right)$ that capture time-varying patterns across the acquisition sequence. Linear interpolation between adjacent temporal features enables continuous reconstruction at arbitrary time points. These temporal features are combined with the 32-dimensional spatial features via element-wise multiplication (Hadamard product). The resulting fused features are decoded by dual MLPs, each containing two hidden layers with 32 neurons. The dual-MLP architecture separately generates the real and imaginary components of the complex field, thereby avoiding the phase discontinuities inherent in amplitude-phase representations. Each MLP output is reshaped from flattened feature vectors to 2D spatial grids matching the target reconstruction dimensions, producing the final complex-valued optical field at each time point.

Reconstruction via gradient-domain loss

The neural field parameters are optimized using a physics-informed loss operating in the gradient domain. The forward model follows the physical imaging process: the complex object field first propagates to the coded surface plane through angular spectrum propagation over distance d₁, modulates with the shifted coded surface pattern based on tracked sensor positions, and then propagates to the detector plane over distance d₂. The final intensity is down-sampled by the magnification factor to match the raw measurement dimensions. Rather than directly comparing measured and predicted intensities, the adopted gradient-domain loss operates on spatial derivatives of amplitude fields, computing gradients of the square root of intensities. This formulation provides more robust convergence, demonstrated in Supplementary Fig. S5-S6.

The reconstruction process employs the Adam optimizer with an initial learning rate of 10⁻³, typically converging within 10 epochs. In practice, the Adam optimizer adaptively adjusts the effective step size during training, which reduces sensitivity to the initial learning rate choice. The key to robust performance lies in exploiting temporal correlations across the entire image sequence – transforming what would be an underdetermined single-frame problem into a well-constrained global optimization. While conventional iterative methods (detailed in Supplementary Note 2) produce artifacts and often fail to converge entirely, our spatiotemporal approach maintains stable reconstruction even for challenging dynamic samples in Supplementary Fig. S5-S6 and Movie 4. Supplementary Figs. S14-S15 further demonstrate this advantage using bone marrow mesenchymal stem cells and Na₂CO₃ crystallization, where our framework successfully tracks dynamic morphological changes while maintaining phase continuity and structural fidelity. In both cases, conventional phase retrieval algorithms and orthogonal mode decomposition^54,64 fail to produce satisfactory reconstructions, confirming that neither frame-by-frame refinement nor post-hoc modal decomposition can capture the complex, non-stationary dynamics present in these experiments.

Operating regime and limitations

The framework demonstrates robustness to temporal sampling rates ranging from 30 to 3 fps when all acquired measurements are retained (Supplementary Fig. S16). In this multi-frame strategy, consecutive raw diffraction images share temporal coordinates: for 30 fps, each raw image has a unique temporal coordinate; for lower rates, multiple images share coordinates (every 2 images for 15 fps, every 5 for 6 fps, every 10 for 3 fps). Resolution is preserved across all sampling conditions because the total measurement redundancy is maintained.

However, when frames are actually discarded rather than temporally grouped, performance depends on the sample’s phase complexity (Supplementary Fig. S17). We systematically characterized this relationship by decimating frames from the Na₂CO₃ crystallization dataset at 30, 10, 5, and 3 fps across regions with different complexity (Supplementary Fig. S17). For regions with dense phase wraps and rapid temporal variations, the full 30 fps acquisition is required for successful convergence; reducing the frame rate leads to convergence failure as the spatiotemporal correlations become insufficient for the neural field to exploit. In contrast, regions with moderate phase variations achieve successful convergence at all frame rates down to 3 fps. These results establish that the required temporal sampling is dictated by the rate and complexity of the sample dynamics rather than the algorithm itself. Consequently, for slower imaging systems, the framework applies equally well as long as the sample dynamics are correspondingly slower. The detailed interplay between temporal variation rate and sample phase complexity in determining the failure boundary is an interesting direction that warrants further investigation.

The framework also scales favorably with sensor resolution and acquisition duration. For higher-resolution sensors, the hash table size can remain fixed while spatial resolution improves, since the multi-resolution hash encoding is designed precisely for this type of scaling. For multi-day experiments, one practical strategy is to segment the sequence into overlapping temporal windows, each with its own set of temporal basis vectors but sharing the same spatial hash encoding.

Mesoscale wound healing monitoring within incubator

Bone marrow mesenchymal stem cells (RASMX-01001, Cyagen) were cultured in a specialized growth medium (RASMX-90011, Cyagen) supplemented with 10% FBS. For wound healing assays, cells were seeded in standard culture dishes at a density of 2×10⁵ cells/cm² and grown to 90% confluence. Linear wounds were created using a sterile pipette tip by scraping through the cell monolayer. The culture dishes were then gently washed with PBS to remove floating cells and replenished with fresh medium. The entire ptychography system was positioned inside a standard incubator (37 °C, 5% CO₂), enabling continuous monitoring without sample transfer – eliminating mechanical perturbations and contamination risks that compromise traditional time-lapse microscopy of wound healing dynamics.

Dry-mass tracking of bacterial growth

We used the E. coli ATCC 25922 strain for the bacterial growth experiment. The bacteria were cultured in fresh Mueller-Hinton broth. In Fig. 4b, the dry mass density at each pixel is calculated as $\rho \left(x,y\right)=\left(\lambda /2\pi \gamma \right)\varphi \left(x,y\right)$, where λ is the center wavelength, γ is the average refractive increment of protein (0.2 mL/g)⁷², and $\varphi \left(x,y\right)$ is the phase recovered using the video-rate ptychography system.

Transdermal microneedle patch fabrication and characterization

Microneedle patches were fabricated through a multi-step molding process. Silicon master molds with pyramidal cavities were first created via two-photon polymerization (Nanoscribe), achieving sub-micron feature resolution. These masters were replicated in polydimethylsiloxane (PDMS) to produce flexible, reusable negative molds. For the biodegradable microneedles, we selected polylactic-co-glycolic acid (PLGA, 50:50, acid-terminated, 15 kDa), which degrades via hydrolysis in physiological conditions. PLGA solution (20% w/v in acetone) was cast, dried, and compression-molded into the PDMS cavities at 60 °C under vacuum. The final patches contained approximately 100 microneedles per 1 × 1 cm² area, with each needle designed to be 700 μm in height with a base diameter of 300 μm. These dimensions were specifically chosen to ensure effective penetration of the stratum corneum while maintaining sufficient mechanical strength for insertion. For 3D characterization of the fabricated device, the microneedle patches were imaged to obtain quantitative phase maps across the entire device area. The recovered phase images were then unwrapped, and the physical height was calculated using the equation $h\left(x,y\right)=\left(\lambda /2\pi △n\right)\varphi \left(x,y\right)$, where $\varphi \left(x,y\right)$ is the unwrapped phase, and $△n$ is the refractive index difference of the microneedle and the surrounding environment. The reconstructed 3D profiles provided comprehensive geometric information including needle height, tip sharpness, and array uniformity.

Validation with extreme ultraviolet ptychography

To demonstrate wavelength-agnostic performance, we validated our framework on extreme ultraviolet ptychography where probe instabilities challenge conventional reconstruction. The same low-rank temporal decomposition that captures sample dynamics in visible light was adapted to model time-varying probe beam $P_t\left(x,y\right)$ while keeping the object static (implementation details in Supplementary Note 4). We benchmarked our approach against conventional OPR methods with ePIE, mPIE, quasi-Newton, and least-squares algorithms, under progressively challenging overlap conditions of 24.4%, 10.3%, and 1.2% (Supplementary Figs. S20–S24). While OPR methods decompose probe variations into fixed orthogonal spatial modes, our framework models probe dynamics as continuous temporal functions. At the overlap of 24.4%, conventional OPR methods exhibit artifacts and distortions in the recovered Siemens star test pattern. As overlap decreases to 10.3% and especially at the extreme 1.2% overlap, OPR methods produce heavily degraded reconstructions with severe artifacts. In contrast, our space-time approach maintains recognizable feature recovery across all overlap conditions by leveraging temporal coherence in probe evolution. These results demonstrate that modeling temporal dynamics with neural fields offers a fundamentally more robust approach than static modal decomposition, extending the operational regime of ptychography to previously inaccessible low-overlap conditions that are critical for low-dose imaging and sensing applications.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Author contributions

G.Z. conceived the original concept of video-rate ptychography and supervised the project. R.W. performed the experiments and acquired ptychographic datasets. Q.Z. and Z.H. developed the space-time neural-field framework. R.W., Q.Z., Z.H., and Q.M. prepared the display items and Supplementary Movies. Q.Z. and G.Z. prepared the Supplementary Information. F.H. and T.D.N. prepared the stem cell and microneedle experiments. T.W., S.J., L.Y., L.J., L.S., D.G., M.L., C.A., A.B., and D.B. participated in the discussion and interpretation of the results. Q.Z. and G.Z. wrote the manuscript with the input from all authors.

Peer review

Peer review information

Nature Communications thanks Christopher Metzler, Haowen Zhou, and the other anonymous reviewer for their contribution to the peer review of this work. A peer review file is available.

Funding

This work was partially supported by the Department of Energy SC0025582 and the National Institutes of Health R01-EB034744. The content of the article does not necessarily reflect the position or policy of the US government, and no official endorsement should be inferred.

Data availability

The data supporting the findings of this study are available within the paper and its Supplementary Information. The experimental datasets including the snowflake melting, crystallization, bone marrow mesenchymal stem cell, and EUV time-varying-probe datasets have been deposited in the Zenodo repository under accession 10.5281/zenodo.17552294 (ref. ⁷³). Source Data for the quantitative plots in Fig. 4h and 5d are provided with this paper.

Code availability

The code implementing the space-time neural field reconstruction, including the forward imaging model, multi-resolution hash encoding, and gradient-domain optimization, is available in the Zenodo repository at 10.5281/zenodo.17552294 (ref. ⁷³). The implementation is written in Python using PyTorch and depends on the tiny-cuda-nn library for hash encoding; installation instructions and dataset-specific reconstruction notebooks are provided in the Supplementary Note 5.

Competing interests

G. Z. is a named inventor of a related patent application. T. D. N. has conflict of interest with PiezoBioMembrane Inc. and SingleTimeMicroneedles Inc. Other authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s41467-026-75092-8.