Compressed Ultrafast Tomographic Imaging by Passive Spatiotemporal Projections

Abstract

Existing streak-camera-based two-dimensional (2D) ultrafast imaging techniques are limited by long acquisition time, the trade-off between spatial and temporal resolutions, and a reduced field of view. They also require additional components, customization, or active illumination. Here, we develop compressed ultrafast tomographic imaging (CUTI), which passively records 2D transient events with a standard streak camera. By grafting the concept of computed tomography to the spatiotemporal domain, the operations of temporal shearing and spatiotemporal integration in a streak camera’s data acquisition can be equivalently expressed as the spatiotemporal projection of an (x, y, t) datacube from a certain angle. Aided by a new compressed-sensing reconstruction algorithm, the 2D transient event can be accurately recovered in a few measurements. CUTI is exhibited as a new imaging mode universally adaptable to most streak cameras. Implemented in an image-converter streak camera, CUTI captures the sequential arrival of two spatially modulated ultrashort ultraviolet laser pulses at 0.5 trillion frames-per-second. Applied to a rotating-mirror streak camera, CUTI records an amination of fast-bouncing balls at 5 thousand frames-per-second.

Ultrafast optical imaging is indispensable to numerous studies in physics, chemistry, and biology [1]. Advances in optoelectronic instrumentation propel ultrafast optical imaging with higher imaging speeds, higher sensitivity, and broader operating spectra [2]. Among existing devices, streak cameras are popularly used to passively record dynamic events [3]. By using a temporal shearing unit to convert the time of arrival to spatial deflection, commercially available streak cameras can directly measure transient optical signals with a temporal resolution of down to hundreds of femtoseconds [4]. Streak cameras have found widespread applications, including inertial confinement fusion [5], characterization of laser filaments [6], and fluorescence lifetime imaging [7].

Technical specifications of streak cameras have been greatly improved in recent years. Multiple sweep ranges with timespans from picoseconds to milliseconds are commonly implemented. Readout cameras with several millions of pixels have become standard configurations. Both advances endow streak cameras with multi-scale imaging speeds from thousands of frames per second (kfps) to trillions of frames per second (Tfps) [2]. Moreover, progress on new materials for photocathodes has expanded the spectral range of streak cameras from far-infrared regions to visible, ultraviolet (UV), even x-ray wavelengths [7–10]. Finally, to circumvent the space-charge effect in electron imaging and the Coulomb repulsive force in electron-photon conversion, innovative optical and electron imaging approaches have been implemented in designing novel temporal shearing units. The resultant new types of streak cameras have enhanced signal-to-noise ratios in the acquired images [11] and have improved temporal resolutions in recovered movies [12].

Despite this remarkable progress, in their conventional operation, streak cameras are still restricted to one-dimensional (1D) imaging Due to the time-to-space mapping in the temporal shearing operation, temporal information occupies one spatial axis on the two-dimensional (2D) readout camera. To avoid spatiotemporal ambiguity, spatial information can only be recorded on the other spatial axis. Thus, a narrow entrance slit (typically 50–100 μm wide) is added to restrict ultrafast imaging to a 1D field of view (FOV).

To overcome this limitation, many 2D streak imaging approaches have been developed. Existing methods can be generally divided into the multiple-shot and single-shot categories. For the former, the (x, y, t) information is acquired by combining the conventional operation of streak cameras with a scanning operation in the spatial dimension orthogonal to that of the entrance slit [13]. Although retaining the intrinsic contrast and resolutions of streak cameras, this approach requires a large number of measurements along the scanning direction to synthesize the (x, y, t) datacube. Alternatively, the (x, y, t) information can be obtained in a single measurement by combining streak imaging with other advanced imaging strategies. For example, by implementing a compressed sensing (CS) paradigm in a streak camera, compressed ultrafast photography (CUP) records a transient event compressively into a 2D snapshot and subsequently leverages the prior knowledge of the imaging model and the spatiotemporal sparsity of the scene to retrieve the (x, y, t) datacube [14]. In two other examples, a 2D-1D fiber array is used to maps a 2D FOV to a line [15], or a tilted lenslet-array is implemented to generate several replicas of the dynamic scene at different heights [16]. Both methods reduce the imaging dimension to accommodate the streak camera’s conventional operation so that a 2D ultrafast movie is recovered by allocating pixels in the acquired single streak image to the correct spatiotemporal positions in the (x, y, t) datacube. Despite contributing to many new studies [17–20], these methods reduce the spatial and/or temporal resolutions due to spatial encoding [14] or the FOV due to focal plane division [15,16]. Meanwhile, additional and customized components are either added in front of or inserted into the streak cameras to enable these operations. The increased system complexity may limit the application scope of these techniques [9].

Tomographic imaging is a promising method to surmount the limitations in existing 2D streak imaging techniques. Despite initially developed for recording (x, y, z) information, many tomographic imaging techniques have been implemented to record spatiotemporal (i.e., x, y, t) information [21, 22]. In a typical configuration, multiple identical ultrashort pulses [21] or a spatially chirped pulse [22] probes transient events. The transmitted light is measured by spectral interferometry to obtain the angular projections, which are fed to reconstruction algorithms to recover a movie with imaging speeds up to Tfps [22]. However, relying on active laser illumination, existing ultrafast tomographic imaging techniques are not applicable to imaging self-luminescent and color-selective dynamic events [23].

Here, we overcome these problems by developing compressed ultrafast tomographic imaging (CUTI), Grafting the principle of computed tomography to the spatiotemporal domain, CUTI uses temporal shearing and spatiotemporal integration to equivalently perform passive projections of a transient event By leveraging multiple sweep ranges readily available in a standard streak camera and a new CS-based reconstruction algorithm, the (x, y, t) datacube of the transient event can be accurately recovered using a few streak images.

The operating principle of CUTI is shown in Fig. 1. A repeatable dynamic scene, I(x, y, t), is directly imaged by a standard streak camera for a total of N times. The entrance port of the streak camera is wide-open to retain 2D spatial (i.e., x, y) information at each time point. In the i^th acquisition (i = 1, 2, … , N), a shearing velocity, denoted by v_i, is used by the temporal shearing unit. This process is denoted by the temporal shearing operator S_i. Afterward, a 2D readout camera records the data by spatially integrating over each pixel and temporally integrating over the exposure time to a snapshot E_i. This process is denoted by the spatiotemporal integration operator T. Overall, the forward model of CUTI is described as

$$E=\mathit{TS}I\left(x,y,t\right),$$ #(1)

Fig. 1.

Operating principle of compressed ultrafast tomographic imaging (CUTI). TTR, TwIST-based tomographic reconstruction. Insert in the dashed box: Illustration of the equivalent spatiotemporal projections in data acquisition.

where E = [E₁, E₂, … , E_N]^T and S = [S₁, S₂, … , S_N]^T . This forward model equivalently describes the different passive spatiotemporal projections of I(x, y, t) in the y-t domain (see the inset in the dashed box in Fig. 1). The angle of the i^th projection is θ_i = tan⁻¹(υ_i/|υ_max|), where |υ_max| is the maximum shearing speed of the streak camera. Thus, θ_i ∈ [−45°, +45°] (see detailed derivation in Supplement 1).

After data acquisition, I(x, y, t) is reconstructed by a new algorithm developed based on the framework of sparse-view computed tomography and the two-step iterative shrinkage/thresholding (TwIST) algorithm [24]. In this TwIST-based tomographic reconstruction (TTR) algorithm, with an initialization ${\widehat{I}}_0={\left(\mathit{TS}\right)}^{T}E,I\left(x,y,\mathit{t}\right)$ is recovered by solving the optimization problem of

$$\widehat{I}=\arg \underset{I}{\min }\left\{\frac{1}{2}{\Vert E-\mathit{TS}I\Vert }_2^2+\tau {\Phi }_{\mathrm{TV}}\left(I\right)\right\},$$ #(2)

where τ is the regularization parameter, and Φ_TV(·)is the three-dimensional total variation (TV) regularization function [14]. The reconstructed datacube has a sequence depth (i.e., the number of frames) of N_t = rt_s, where t_s is the sweep time, r = |υ_max|/p_c denotes the imaging speed of CUTI, and p_c is the pixel size of the readout camera. Each frame in the datacube has a (x, y) frame size of N_x ≤ N_h and N_y ≤ N_v − N_t + 1 pixels. Here, N_h and N_v are the horizontal and vertical pixel counts of the readout camera. Thus, CUTI is conceptually different from existing snapshot compressive imaging techniques [25–28] (see details in Supplement 1 and Visualization S1).

We demonstrated the feasibility of CUTI by simulating a dynamic jellyfish scene with the size of N_x × N_y × N_t = 512 × 512 × 80 pixels. Five projections (λ_i = 0, ±22.5°, and ± 45°) were applied to this scene according to the forward model [i.e. Eq. (1)]. All the projected images were input to the TTR algorithm (with τ = 0.0059) for image reconstruction. As a comparison, the algorithms of back projection (BP), recursive spatially adaptive filtering (RSAF) [29], and least squares (LSQR) [30] were also used for image reconstruction.

The results from all four algorithms are compared with the ground truth (GT) in Fig. 2 and Visualization 1. Four representative frames and local zoom-in views as well as the peak signal-noise ratio (PSNR) of each frame are shown in Figs. 2(a)–(b). These results show the superior performance of TTR In particular, TTR can recover more spatial details compared to BP and RSAF and has fewer artifacts than LSQR. Besides, to analyze the relationship between reconstructed image quality and the number of projections, we reconstructed twelve datacubes with the numbers of projections from 1 to 35, and the angles of these projections (i.e., θ_i) were uniformly distributed from −45° to +45° [Fig. 2(c)]. The normalized average correlation coefficient (NACC) between each reconstructed datacube and the GT was calculated. This result verifies that a reconstruction of good quality (i.e. NACC≥ 0.70) can be achieved in CUTI with ~5 projections.

Fig. 2.

Comparison of the ground truth (GT) with the reconstruction using four algorithms. (a) Four representative frames of the GT and the reconstructions by back projection (BP), recursive spatially adaptive filtering (RSAF), least squares (LSQR), and TTR. Last column: Zoomed-in views of a local feature in the 80^th frame (marked by the cyan dashed box). (b) Peak signal-to-noise ratio (PSNR) of the reconstructions with the reconstructed results in (a). (c) Normalized average correlation coefficient (NACC) of TTR’s reconstruction with different numbers of projections. Error bar: standard deviation.

To demonstrate CUTI with an image-converter streak camera, we imaged a dynamic UV scene [Fig. 3(a)]. In particular, a 266-nm, 100-fs laser pulse was split into two arms by a beam splitter. In each arm, the laser pulse was retro-reflected by a mirror. A manual translation stage was added into one arm to generate a 1.6-ns time delay. The mirror M₂ was slightly tilted with respect to the normal of the incident beam to generate a lateral shift to the reflected pulse. These two spatially and temporally separated UV pulses transmitting through a resolution target that contains patterns as shown in the inset in Fig. 3(a).

Fig. 3.

Demonstration of CUTI using an image-converter streak camera. (a) Experimental setup. Magenta-boxed inset: The reference image captured without using temporal shearing. (b) Representative frames of the reconstruction scenes. (c) Selected cross-sections of the resolution target in the x-direction (at y = 2.2 mm, green dash-dotted line) and in the y-direction (at x = 3.6 mm, blue dashed line) at t= 150 ps. The corresponding curves from the reference image are shown as magenta solid lines. (d) As (c), but shows the profiles in the x-direction (at y = 2.2 mm, blue dash-dotted line) and in the y-direction (at x = 5.5 mm, green dashed line) at t= 1746 ps. (e) Temporal trace of this event. FWHM, full width at half maximum.

We imaged this transient event using a standard UV-streak camera (AXIS-2DX-Pd, Axis Photonique), which has |υ_max| = 10 μm/ps, t_s = 2.8 ns, and p_c = 20 μm. The intrinsic spatial and temporal resolutions of this streak camera are 22.5 lp/mm and 6 ps, respectively. In this configuration, CUTI had an imaging speed of r = 0.5 Tfps, a sequence depth of N_t = 1400 frames, and a frame size of N_x × N_y = 1024×1024 pixels. 11 projections were acquired using θ_i ∈ [−45°, +45°] with a 9° angular step. The regularization parameter was set to τ = 0.0204. The reconstructed movie is shown in Visualization 2. Six representative frames are presented in Fig. 3(b), showing two sequentially arrived laser pulses whose spatial profiles are modulated by the resolution target. To quantitatively analyze the reconstructed image qualify, we extracted selected cross-sections in the first pulse (at 150 ps) and the second pulse (at 1746 ps), as shown in Figs. 3(c)–(d). These results were also compared with the reference image captured without introducing temporal shearing Using the 10% contrast as the criterion, at t = 150 ps, the spatial resolutions were determined as 15.6 and 14.1 lp/mm in the x- and y-directions, respectively. At t = 1746 ps, the values were 13.2 and 14.1 lp/mm. We also analyzed the reconstructed temporal trace of this event [Fig. 3(e)]. The time delay between the two pulses is 1596 ps, which is consistent with the pre-set value. The full widths at half maximum of these pulses are 8 ps and 10 ps, respectively. This temporal broadening effect results synthetically from a limited range of projection angles, a limited number of projections, and the intrinsic temporal resolution of the streak camera. It is also noted that both the spatial resolution and the temporal accuracy of the second pulse are slightly decreased, which is attributed to the stronger image distortion at the bottom of the streak image (further explained in Supplement 1).

To demonstrate CUTI with a rotating-mirror streak camera, we imaged fast-moving ball patterns at 5 kfps. This datacube was repeatedly displayed by a digital micromirror device (DMD, AJD-4500, Ajile Light Industries). A collimated continuous-wave laser beam shone onto the DMD at an incident angle of ~24°. The light diffracted by the patterns was captured by a rotating-mirror streak camera built in-house, which uses a galvanometer scanner (6220H, Cambridge Technology) for temporal shearing and an electron-multiplying CCD camera (HNü 1024, Niivu Caméras) for spatiotemporal integration [Fig. 4(a)]. The pre-set parameters were |υ_max| = 29.0 μm/ms, t_s = 50 ms, and p_c = 5.8 μm. Thus, CUTI operated at r = 5 kfps with a sequence depth of N_t = 250 frames and an image size of N_x × N_y = 512 × 512 pixels. 15 projections (θ_i ∈ [−45°, +45°] with a 6.4° angular step) were recorded in this experiment, and the regularization parameter was set to τ = 0.0463. The reconstructed movie is Visualization 3, and Fig. 4(b) presents selected frames of the GT and the TTR-reconstructed results. To evaluate CUTI’s performance, we calculated the PSNR and structural similarity index measure (SSIM) of TTR’s output [Fig. 4(c)]. Moreover, the centroids of each ball [labeled as B1 and B2 in the first panel of Fig. 4(b)] were traced [Fig. 4(d)]. The root-mean-square errors of reconstructed centroids along the x- and y- directions were calculated to be 19.18 and 19.16 μm for B1, and 18.36 and 18.03 μm for B2, respectively. These analyses confirm CUTI’s feasibility with rotating-mirror streak cameras.

Fig. 4.

Demonstration of CUTI using a rotating-mirror streak camera. (a) Experimental setup. Lens 1 (AC508-100, Thorlabs); Lenses 2 an 3 (AC508-075, Thorlabs). (b) Selected frames of the GT (top row) and TTR’s reconstruction with 15 projections (bottom row). (c) PSNR and structural similarity index measure (SSIM) of TTR’s reconstructions. (d) Tracing the centroids of B1 and B2 [marked in the first panel of (b)].

We have developed CUTI that synergizes streak imaging, tomographic imaging and CS. By implementing the concept of tomography in the spatiotemporal domain, CUTI passively records the spatiotemporal projections in an angular range from −45° to +45°. The acquired projections are processed by a newly developed TTR algorithm to accurately recover the dynamic scene. CUTI is a receive-only tomographic imaging paradigm with scalable imaging speeds. Implemented in an image-converter streak camera, CUTI captured the sequential arrival of two spatially modulated UV pulses at 0.5 Tips with a datacube of 1024×1024×1400 (x, y, t) pixels in size. Applied to a rotating-mirror streak camera, CUTI imaged fast-moving ball patterns at 5 kfps with a datacube size of 512×512×250 (x, y, t) pixels. As a universal scheme, CUTI can be readily applied to streak cameras without any hardware modification. Compared to the scanning-based multiple-shot 2D streak imaging approaches, CUTI largely reduces the data acquisition time. Compared to the singleshot methods, CUTI eliminates the trade between streak camera’s spatiotemporal resolution or FOV to temporal resolution. In the future, CUTI’s reconstruction qualify could be further boosted by applying other advanced CS algorithms [9]. Moreover, image rotators [18] could be integrated into the front optics of the streak cameras to further increase the number and coverage of projection angles. As a new imaging mode of streak cameras for 2D time-resolved imaging, CUTI will likely find new applications in time-of-flight ranging [31], laser manufacturing [32], and biomedicine [33].