Increasing the acquisition speed in oblique plane microscopy via aliasing

Abstract

Oblique plane microscopy (OPM), a variant of light-sheet fluorescence microscopy (LSFM), enables rapid volumetric imaging without mechanically scanning the sample or an objective. In an OPM, the sample space is mapped to a distortion-free image space via remote focusing, and the oblique light-sheet plane is mapped onto a camera via a tilted tertiary imaging system. As a result, the 3D point-spread function and optical transfer function (OTF) are tilted to the optical axis of the tertiary imaging system. To satisfy Nyquist sampling, small scanning steps are required to encompass the tilted 3D OTF, slowing down acquisition and increasing sample exposure. Here, we show that a judicious amount of under-sampling can lead to a form of aliasing in OPM that can be recovered without a loss of spatial resolution while minimizing artifacts. The resulting speed gains depend on the optical parameters of the system and reach 2–4-fold in our demonstrations. We leverage this method for rapid subcellular 3D imaging of mitochondria and the endoplasmic reticulum.

1. Introduction

Light-sheet fluorescence microscopy (LSFM), owing to its low sample irradiance and rapid volumetric imaging capabilities, has found numerous applications in biological and biomedical research [1]. By illuminating the sample with a sheet of light, intrinsic optical sectioning is achieved, which limits unnecessary sample irradiation outside the focal plane and allows rapid, parallelized image acquisition with modern scientific cameras. In a typical light-sheet microscope, illumination and detection are implemented on separate objective lenses, typically oriented orthogonally to each other. 3D image acquisition is either performed by scanning the sample through the light-sheet, or by optically scanning the light-sheet and mechanically translating the detection objective [1].

Oblique plane microscopy [2] (and related techniques such as Swept confocally-aligned planar excitation, SCAPE [3]) is a type of LSFM where both illumination and detection are performed by a single primary lens. This improves sample accessibility, and in recent implementations of OPM and SCAPE using rapid optical scanning technologies [4,5], much higher volumetric acquisition speed can be obtained than using traditional LSFM architectures.

In a traditional wide-field microscope, the principal axes (x, y, z) of the point spread function (PSF) are aligned with the principal axes (s_x, s_y, s_z) of sampling (i.e. along the optical axis and in the axes of the pixel array on the camera). In an OPM, because the light sheet is launched from the primary objective at an oblique angle, the image plane is also tilted by an oblique angle. As such, for an OPM, both s_x and s_z have components in the x and z directions. As a result, lateral and axial resolution are coupled, and relatively fine step sizes are needed to acquire 3D data. This in turn slows down 3D imaging and burdens the sample with enhanced radiation exposure, compared to an imaging system where the PSF and OTF is aligned with the optical axis.

Here we show that some degree of under-sampling in OPM leads to a form of frequency aliasing that can be recovered without a loss of spatial resolution while minimizing artifacts. This is because the tilted OTF leaves some voids in 3D Fourier space which can accommodate aliased information without overlap.

The amount of tolerable under-sampling, and resulting gain in imaging speed, depends on the properties of the imaging system. We demonstrate two-fold gains for a high resolution OPM and 4-fold speed gains for a mesoscopic system. We leverage the method to achieve rapid volumetric imaging of mitochondrial dynamics and re-arrangements of the endoplasmic reticulum at two-fold higher speeds than using critical Nyquist sampling.

2. Theory

Figure 1(A) shows schematically the principle of OPM. A light-sheet (blue) emerges at an oblique angle from the primary objective (O1). It excites two emitters (green dots), whose fluorescence emission is partially captured by the primary objective (green cones). A remote focusing system [6] maps the image of the emitters via a matched secondary objective (O2) into a remote space. Due to the limited collection angle, the 3D image of an emitter, i.e. its point spread function (PSF), is elongated along the optical axis (z). The 3D PSF is invariant (within a finite axial range [6]) within the remote focus volume. Thus, both emitter images share the same PSF, even though they originate from different depths. A tertiary imaging system (Objective O3), tilted to the same angle as the lights-sheet, maps the fluorescence emission onto a detector [2]. By probing the remote space at an angle, the point spread function appears tilted in the tertiary reference frame (X'-Y '-Z ', see also Fig. 1(B)). We ignore here effects of only capturing a subset of the light-cone emitted by O2 with O3, which may introduce an additional source of PSF tilt.

Fig. 1.

Schematic illustration of an OPM, and its imaging properties. A Simplified optical train in OPM: A tilted light-sheet (blue) emerges from the primary objective (O1). The fluorescence light from two emitters (green dots) is relayed via the secondary objective (O2) to the tilted tertiary objective (O3). B Volumetric image of the two emitters, as acquired in the reference frame of the tertiary objective (Z’-X’). C Fourier transform of the volumetric dataset of the emitters shown in B. The optical transfer function is represented in light-blue.

A Fourier transform of the 3D PSF yields the optical transfer function (OTF, Fig. 1(C)), which is correspondingly also tilted. For simplicity, we assume that the OTF has an ellipsoidal shape, with the magnitude of its major axes scaling proportionally to the resolving power. Typically, the lateral resolution (i.e. X and Y direction) is 3-4 fold better than in the third dimension (Z direction), and hence the OTF has a similar aspect ratio. We note that the overall OPM PSF is the product of the light-sheet intensity distribution and the detection PSF, and the OTF is the convolution of their respective Fourier transforms. As a result, an experimental OPM OTF might appear slightly skewed due to the light-sheet tilt (i.e. the light-sheet propagation axis and the Z axis are not orthogonal to each other, as in a traditional LSFM).

Figure 2(A) illustrates critical Nyquist sampling requirement for the lateral, p_x’ and axial, p_z’ sampling steps (i.e. voxel sizes). For the sake of argument, we assume a 45-degree tilt of the OTF, and an aspect ratio of K_x/K_z= 4, with K_x and K_z being the cut-off frequencies of the OTF in the lateral and axial direction, respectively. The Fourier transform of the pixel grid with p_x’ and p_z’ voxel size in real space corresponds to a grid with 1/p_x’ and 1/p_z’ spacing in reciprocal space. At every grid point, a copy of the OTF is located. In order to avoid overlap of the OTF copies, we find that p_x’=p_z’, and that their magnitude needs to exceed ${\mathrm{K}}_{\mathrm{x}}/\sqrt{2}$ . Then the borders of the numerical domain of a discrete Fourier transform (black box in Fig. 2(A)) encompass the central copy of the OTF, and no neighboring copy of an OTF overlaps with that domain.

Fig. 2.

Sampling of the tilted OPM OTF. A Critical Nyquist sampling of the OPM OTF (blue ellipses). B Axial undersampling by a factor of 2. p_x, p_z: lateral and axial pixel size. Kx,Kz: cut-off frequencies of the OTF.

If we double the axial step size, we expect to violate the Nyquist condition. Indeed, in Fig. 2(B), one can see that the grid in reciprocal space has contracted by a factor of 2 in the k_z’ direction. Because of the under-sampling, not all frequency components of the central copy of the OTF are contained within in the numerical domain of a discrete Fourier transform. Furthermore, parts of two OTF copies, located at 0,1/p_z’ and 0,-1/p_z’, have leaked into the numerical domain.

Importantly, there is no overlap between OTF copies themselves in the scenario shown in Fig. 2(B), but there is a misplacement of Fourier components. We reasoned that if we could properly re-arrange the Fourier components, the full support of the OTF could be obtained. This idea is inspired by how structured illumination microscopy (SIM) reassigns misplaced Fourier components during reconstruction [7]. However, in contrast to SIM, there are no overlapping Fourier components, thus no prior unmixing needs to be performed.

Our proposed reconstruction scheme is shown in Fig. 3. In Fig. 3(A), the actual observed information in a 2x under-sampled dataset in Fourier space is shown. We reasoned that if we concatenated three copies of the dataset in the k_z direction in reciprocal space, a re-assignment of the Fourier components for the central copy of the OTF would be achieved (see also Fig. 3(B)). Obviously, there are still additional copies of the OTF that would cause severe aliasing in real space. However, since those components are separated, they can be removed with a masking step, as illustrated in Fig. 3(C).

Fig. 3.

Reconstruction of OPM OTF after under-sampling. A Fourier transform of an axially under sampled OTF. B Reconstruction of the OTF by concatenating three copies of the OTF along the k_z direction. C Masking of the central OTF. Areas shaded in black are set to zero.

The amount of under-sampling that can be applied in our technique depends on the tilt angle and aspect ratio (i.e. K_x/K_z) of the OTF (see also Supplementary Figure S1-S2 for a numerical simulation and Supplementary Figure S3 for a geometrical construction). In practice, we empirically found the tolerable under-sampling factors for the OPM systems used: the axial resolution starts to deteriorate when the under-sampling factor becomes critical as then the aliased copies of the OTF begin to overlap (Supplementary Figure S4). Importantly, one can in principle run higher under sampling factors, at the cost of a degradation of axial resolution (Supplementary Figure S4).

3. Numerical algorithm

Numerically, we found different implementations to apply the outlined theory to OPM data. To concatenate the Fourier transforms of OPM data (Fig. 3), we multiply a Dirac comb function in real space along the z’ axis (Supplementary Figure S1-S2 for a numerical simulation, and Figure S5-S6 for a schematic illustration, and Figure S7 with experimental data), whose spacing corresponds to the under sampling factor. As an example, if OPM data is under sampled by a factor of two, we interleave a zero slice in between every adjacent slice in the z’ direction. This, in effect, is equivalent to multiplying the fully sampled stack with a Dirac delta comb function, but it is important to understand that no data is being erased; interpolation is merely being replaced by the zero-slice interleaving operation.

After interleaving the zero slices, we apply de-skewing as one would apply to a fully sampled stack. After de-skewing, a 3D Fast Fourier transform is taken, and a top-hat filter is applied to remove unwanted copies of the OTF. An inverse 3D Fast Fourier transform is taken to yield the reconstructed image data which is then rotated into the x-y-z coordinate frame of the primary objective (Fig. 4(A)).

Fig. 4.

Flow chart for the proposed reconstruction algorithm for under sampled OPM data. A Variant that applies a top hat filter after de-skewing. B Variant that applies a top hat filter after de-skewing and rotation.

Alternatively, top hat filtering can be applied after the de-skewing and rotation steps (Fig. 4(B), and Supplementary Figure S8). The two methods yield comparable results (Supplementary Figure S9), but there are differences in computation time (Supplementary Figure S10 and Supplementary Note 1). Throughout this manuscript, we used top hat filtering after de-skewing but before rotation (Fig. 4(A)).

In this manuscript we use the PetaKit 5D software [8] for de-skewing and rotation, but any software can perform these operations in our workflow, and in the same way as if the data was not under sampled in the first place. In prior work, Maioli and Davies et al [9,10] stressed the importance of performing interpolation/up-sampling of OPM data prior to rotation to avoid post-processing artifacts. This was followed both by our proposed algorithm but also when processing data conventionally for comparison in this manuscript (See also Supplementary Figure S9). To be more specific, in our reconstruction algorithm, the up-sampling to isotropic voxel size was accomplished by interleaving zero slices at the beginning of the process. Further details on the steps of our algorithm are provided in Supplementary Note 1.

When we processed OPM data conventionally, the PetaKit can perform all steps (de-skew, interpolate and rotate) in a single matrix operation, which yields the fastest processing times (Figure S10). However, to clearly separate the interpolation step, we decomposed the steps and interleaved our own interpolation routine between de-skew and rotate (for which there are separate functions in the PetaKit). In terms of the interpolation artifacts introduced, we found that the differences between the single matrix operation and the decomposed steps are minor though (Figure S9).

4. Results

4.1. Fluorescent nanospheres

We have acquired 3D datasets with two OPM systems to demonstrate our method. In Fig. 5(A), the OTF of a high resolution OPM system is shown (40× NA 1.25 Silicone oil objective primary lens, 45-degree light-sheet tilt, as detailed in [11]), where we adjusted the step size to critical Nyquist sampling (See also Supplementary Note 2). That threshold was determined visually, i.e. the OTF is just contained within the numerical domain, and any further down-sampling would lead to the appearance of aliased copies and artifacts in real space.

Fig. 5.

Experimental OPM OTFs, shown as a cross-section and represented as the logarithm of the power spectrum. A Critically sampled OTF of a high resolution OPM system. B Two-fold under-sampled OTF. The arrows point at Aliasing of OTF components. C Threefold repeat of the under sampled OPM OTF. The red lines outline the masking region: All Fourier components outside the band encompassed by the red lines were set to zero.

In Fig. 5(B), an OTF with 2× under-sampling is shown. The red arrows point at two copies of the OTF that have been mixed in via aliasing. These two copies come close to the central body of the OTF, but do not overlap with it yet. Figure 5(C) shows a three-fold repeat of the under-sampled OTF shown in Fig. 5(B). As predicted in Fig. 3, a full central OTF is reconstructed, with additional partial copies flanking it. For reconstruction, we only kept the Fourier components within the two red lines shown in Fig. 5(C) and set the rest to zero outside of that band. This represents a top hat (sinc in real space) filter, which may cause ringing in real space. Alternative windowing methods could be applied, but for this first proof of principle, we employed top hat filtering only. The full reconstruction workflow is also shown on experimental data in Supplement 1 ^{(2.8MB, pdf)} Figure S7.

In Fig. 6, we compare the real space results for fluorescent nanospheres, as imaged with a high resolution and a mesoscopic (termed meso) OPM. For the high resolution OPM, we imaged 100 nm fluorescence nanospheres on a coverslip. The data was de-skewed and rotated using the PetaKit5D software [8], and is presented in an Y-Z coordinate frame, projected along the X direction. Figure 6(A) shows the ground truth dataset acquired with critical Nyquist sampling. Figure 6(B) shows the same beads, but with 2× under-sampling. Typically, under-sampled datasets in OPM are interpolated at the rotation step to achieve the proper z-step size, but this leaves pronounced aliasing artifacts in place (red arrows). Figure 6(C) shows our reconstruction from the under-sampled data, in which such artifacts are noticeably absent.

Fig. 6.

Point Spread Functions for OPM. A Critically sampled PSFs for a high resolution OPM. B The same region but acquired with 2x under-sampling. C Reconstruction from the 2× under-sampled data. D Critically sampled PSFs for a mesoscopic OPM. E The same region but acquired with 4x under-sampling. F Reconstruction from the 4× under-sampled data. The red arrows point at artifacts in the under-sampled PSF, such as bifurcation, or concentration into a single spot. Scale bars: C: 10 microns; F 50 microns.

For the meso OPM using a previously published system [6], we imaged 500 nm fluorescent nanospheres in agarose. We empirically found that 4× under-sampling is possible (Supplement 1 ^{(2.8MB, pdf)} S4). Figure 6(D) shows a ground truth data set acquired with critical Nyquist sampling, whereas Fig. 6(E) shows the same area, but for 4× under-sampling. Red arrows point at aliasing artifacts, such as bifurcation and concentration into a single spot. Figure 6(F) shows our reconstruction of the 4× under-sampled dataset, where the artifacts seen in Fig. 6(E) have been removed. Details on the sample preparation are included in Supplement 1 ^{(2.8MB, pdf)} Note 3.

4.2. Rapid imaging of mitochondrial dynamics and endoplasmic reticulum

Next, we applied our accelerated acquisition scheme to rapid imaging of subcellular dynamics using the high resolution OPM system. We imaged U2OS cells labeled with OMP25-GFP, an outer membrane marker for mitochondria, over 50 timepoints. Details on the sample preparation are included in Supplement 1 ^{(2.8MB, pdf)} Note 3. We applied 2× under-sampling (same settings as in Fig. 6(A)-(C)) yielding a volume rate of ∼2 Hz. Figure 7(A) shows a cross-sectional view after conventional OPM post-processing (de-skewing, rotation into X-Y-Z reference frame, and interpolation of the Z-axis). The red arrows point at similar aliasing artifacts as seen in Fig. 7(B), such as bifurcation or concentration of the PSF. Figure 7(B) shows the same cross-section, but reconstructed using our method, which is free of those artifacts. We also acquired a separate stack at full sampling rate for a ground truth comparison (Supplement 1 ^{(2.8MB, pdf)} Figure S11), which was also free of artifacts. Digital down-sampling (removing slices from the stack before further processing) produced artifacts of similar appearance to the once shown in Fig. 7(A).

Fig. 7.

Volumetric imaging of mitochondrial dynamics at 2 Hz volume rate in U2OS cells labeled with OMP25-GFP. A cross-sectional view of 2× under-sampled dataset. Red arrows point at aliasing artifacts. B Same view but using our reconstruction technique. C Maximum intensity projection of the U2OS cell. D Magnified view of the boxed region in C. White arrows point at an extruding mitochondrion. Scale Bar: A, C: 10 microns; D: 2 microns

Figure 7(C) shows the first timepoint of the time lapse series, and the insets in Fig. 7(D) show three consecutive timepoints. The increased acquisition speed allowed us to observe rapid rearrangements of the mitochondria (Visualization 1 ^{(9.3MB, mp4)} ).

To further push the imaging speed, we imaged rapid re-arrangements of the endoplasmic reticulum (ER), endogenously labeled with APMAP-mNG, in a U2OS cell at 4.3 Hz volume rate using 2× under-sampling and our reconstruction scheme. Details on the sample preparation are included in Supplement 1 ^{(2.8MB, pdf)} Note 3. The under-sampling produced artifacts including striping of the nuclear membrane (Supplement 1 ^{(2.8MB, pdf)} Figure S12) using traditional post processing only (de-skewing and rotation).

Figure 8(A) shows a maximum intensity projection of the first timepoint of a time series encompassing 50 volumes. Figure 8(B)-(D) show magnified views of selected timepoints that show rapid re-arrangements of the ER network (See also Visualization 2 ^{(13.1MB, mp4)} ).

Fig. 8.

Volumetric imaging of the endoplasmic reticulum, labeled with APMAP-mNG, in an U2OS cell at 4.3 Hz volume rate using 2× under-sampling and our reconstruction method. A Maximum intensity projection of the first time point. B-D Three selected timepoints of the white boxed area in A. White arrows point at ER tubules undergoing rapid re-arrangements. Scale Bars: A 10 µm, D 5 µm

5. Discussion

We have introduced a method to accelerate the volumetric acquisition rate of OPM through under-sampling and subsequent recovery of aliased information. We exploit that the major axes of the OTF are not co-aligned with the reference frame of OPM’s tertiary imaging system. As such, the tilted OTF leaves empty spaces in the rectangular Fourier domain, which can be partially filled with aliased information stemming from under-sampling. As long as there is no overlap between the primary OTF and the aliased information, we show that we can restore the full OTF and achieve imaging performance in real space that is equivalent to Nyquist sampling.

We have demonstrated rapid volumetric imaging of subcellular dynamics encompassing an entire cell at 4 Hz rate. Conceivably, one could find a smaller cell (needing fewer slices) or employ a faster camera and may achieve faster volume rates using conventional Nyquist sampling. Importantly, however, whatever limits the acquisition time (camera framerate or sample brightness), our method can double to quadruple that limit. Or, in other words, we fundamentally extend the spatiotemporal bandwidth of OPM and SCAPE.

The degree of under-sampling depends on the aspect ratio of the OTF, and as such of the PSF lateral-axial anisotropy. For a high-resolution system, this afforded a two-fold speed increase over critical Nyquist sampling, whereas the mesoscopic OPM, with a larger degree of PSF anisotropy, could perform four-fold faster acquisitions. Indeed, there are mesoscopic OPM systems with even higher PSF anisotropy where even higher speed gains might be possible [12]. This could further accelerate the use of OPM for whole brain and whole organism functional 3D imaging [13].

In terms of computational cost, our method adds two steps, up sampling of the raw data by interleaving zero slices and a filtering step in Fourier space, to standard OPM processing. While the up sampling is negligible, the additional Fourier transforms add considerable computational time. Using GPU acceleration and the version of the algorithm that performs the Fourier filtering at the end, the top hat filter step took roughly as long as the de-skew and rotation operation of the PetaKit. As an example, to process the largest dataset in this manuscript, the beads in Agarose in Fig. 6(F) (0.1 GB with 4× under-sampling), took on average 173 ms (Supplement 1 ^{(2.8MB, pdf)} Figure S10) with our processing scheme. To process the fully Nyquist sampled stack (0.4 GB in size, Fig. 6(D)), PetaKit was still about ∼3 times faster. Thus, our scheme in its current form does not enable savings in computational time yet, despite the smaller size of the input data. Importantly, the acquisition side fully gains from the reduction in data size, in terms of increased imaging speed and reduced sample irradiance. We think that with future code optimization, and potentially other filtering methods, even real-time processing of OPM data might become possible with our method.

Importantly, our reconstruction is straightforward: three (or more, see for example Supplement 1 ^{(2.8MB, pdf)} Figure S7) copies of the Fourier transform of an under-sampled data set are concatenated in the k_z’ direction through interleaving zero slices in real space. No sub-pixel shifts, interpolation or averaging of spectral overlap is needed. The reconstructed OTF must be masked from residual other OTF copies. This masking step was in this work done with a top-hat function but could also involve other filter types in future work.

We have introduced a heuristic stopping criterion for the under-sampling factor, namely once the axial resolution starts to get lower than the one of the ground truth. We reason this happens once the OTF copies start to touch, and one must make the filter band narrower. It is important to know that our reconstruction scheme seems to tolerate some overlap. This is because the parts of the OTF that overlap have opposite phases, which annihilate the Fourier components when superimposed (see Supplement 1 ^{(2.8MB, pdf)} Figure S13 and Supplement 1 ^{(2.8MB, pdf)} Note 4). As such, the overlap leads to a shrinking of the OTF in the k_z direction. Thus, OPM could potentially be operated in an even faster volumetric mode at the expense of axial resolution.

In summary, we have shown that the OTF in OPM allows a certain degree of deliberate axial under-sampling, which leads to a form of aliasing that can be recovered in a loss-less manner while minimizing artifacts. In principle this should improve the spatiotemporal product of any OPM and SCAPE system while reducing the amount of raw data and sample irradiation. Our reconstruction scheme is simple and robust and can be directly integrated into the standard OPM post-processing workflow.

We hope that our scheme will find widespread use in OPM and SCAPE systems and will accelerate 3D imaging while reducing data size and sample irradiation.

Funding

National Institute of General Medical Sciences10.13039/100000057 (R35GM133522, R35GM119768, F32GM154450); National Institute of Biomedical Imaging and Bioengineering10.13039/100000070 (R01EB035538).

Disclosures

The authors declare no conflict of interest.

Data availability

Example data to run the reconstructions shown in this manuscript are available at [14]. Example code to run the reconstructions shown in this manuscript are deposited at [15].

Supplemental document

See Supplement 1 ^{(2.8MB, pdf)} for supporting content.