Efficient Bayesian-based multiview deconvolution

Abstract

Light-sheet fluorescence microscopy is able to image large specimens with high resolution by capturing the samples from multiple angles. Multiview deconvolution can substantially improve the resolution and contrast of the images, but its application has been limited owing to the large size of the data sets. Here we present a Bayesian-based derivation of multiview deconvolution that drastically improves the convergence time, and we provide a fast implementation using graphics hardware.

Modern light-sheet microscopes^1–3 acquire images of large, developing specimens with high temporal and spatial resolution typically by imaging them from multiple directions (Fig. 1a). Deconvolution uses knowledge about the optical system to increase spatial resolution and contrast after acquisition. An advantage unique to light-sheet microscopy, particularly the selective-plane illumination microscopy (SPIM) variant, is the ability to observe the same location in the specimen from multiple angles, which renders the ill-posed problem of deconvolution more tractable^4–10.

Fig. 1. Principles and performance.

(a) Basic layout of a light-sheet microscope capable of multiview acquisitions. (b) Illustration of ‘virtual’ views. A photon detected at a certain location in a view was emitted by a fluorophore in the sample; the PSF assigns a probability to every location in the underlying image having emitted that photon. Consecutively, the PSF of any other view assigns to each of its own locations the probability to detect a photon corresponding to the same fluorophore. (c) Example of an entire virtual view computed from observed view 1 and the knowledge of PSF 1 and PSF 2. (d) Convergence time of the different Bayesian-based methods. We used a known ground-truth image (Supplementary Fig. 5) and let all variations converge until they reached precisely the same quality. The increase in computation time for an increasing number of views of the combined methods (black) is due to the fact that with an increasing number of views, more computational effort is required to perform one update of the deconvolved image (Supplementary Fig. 4) (e) Convergence times for the same ground-truth image of our Bayesian-based methods compared to those of other optimized multiview deconvolution algorithms^5–8. The difference in computation time between Java implementations and IDL implementations, OSEM⁶ and SGP⁸, results in part from nonoptimized IDL code. (f) Corresponding number of iterations for our algorithm and other optimized multiview deconvolution algorithms.

Richardson-Lucy (RL) deconvolution^11,12 (Supplementary Note 1) is a Bayesian-based derivation resulting in an iterative expectation-maximization (EM) algorithm^5,13 that is often chosen for its simplicity and performance. Multiview deconvolution has previously been derived using the EM framework^5,9,10; however, the convergence time of the algorithm remains orders of magnitude longer than the time required to record the data. We addressed this problem by deriving an optimized formulation of Bayesian-based deconvolution for multiple-view geometry that explicitly incorporates conditional probabilities between the views (Fig. 1b,c and Supplementary Fig. 1) and combining it with ordered subsets EM (OSEM)⁶ (Fig. 1d and Supplementary Fig. 2), achieving substantially faster convergence (Fig. 1d–f).

Bayesian-based deconvolution models images and point spread functions (PSFs) as probability distributions. The goal is to estimate the most probable underlying distribution (deconvolved image) that best explains all observed distributions (views) given their conditional probabilities (PSFs). We first rederived the original 645RL deconvolution algorithm and subsequently extended it to multiple-view geometry, yielding

$$f_{\text{RL}}^v(\xi )=\underset{x_v}{\int }\frac{{\varphi }_{v(X_v)}}{{\int }_{\xi }{\psi }^r(\xi )P(x_v\mid \xi )d\xi }P(x_v\mid \xi )d{x}_v$$ (1)

$${\psi }^{r+1}(\xi )={\psi }^r(\xi ){\prod }_{v\in V}{f}_{\text{RL}}^v(\xi )$$ (2)

where ψ^r(ξ) denotes the deconvolved image at iteration r and ϕ_v(x_v) denotes the input views, both as functions of their respective pixel locations ξ and x_v, whereas P(x_v|ξ) denotes the individual PSFs (Supplementary Note 1). Equation (1) denotes a classical RL update step for one view; equation (2) illustrates the combination of all views into one update of the deconvolved image (Supplementary Video 1). In contrast to the maximum-likelihood (ML) EM^5,13 that combines RL updates by addition, equation (2) suggests a multiplicative combination. We proved that equation (2), just as the ML-EM^5,13 algorithm, converges to the ML solution (Supplementary Note 2). The ML solution is not necessarily the correct solution if disturbances such as noise or misalignments are present in the input images (Fig. 2). Importantly, previous extensions to multiple views5–10 assume individual views to be independent observations (Supplementary Fig. 2). Assuming independence between two views implies that by observing one view, nothing can be learned about the other view. We showed that this independence assumption is not required to derive equation (2) (Supplementary Note 3). Our solution represents, to our knowledge, the first complete derivation of RL multiview deconvolution based on probability theory and Bayes’ theorem.

Fig. 2. Deconvolution of simulated 3D multiview data.

(a) Left, 3D rendering of a computer-generated volume resembling a biological specimen. The red outlines mark the wedge removed from the volume to show the content inside. Right, sections through the generated volume in the lateral direction (as seen by the SPIM camera, top) and along the rotation axis (bottom). (b) Same slices as in a with illumination attenuation applied (left), convolved with a PSF of a SPIM microscope (center) and simulated using a Poisson process (right). The bottom right panel shows the unscaled simulated light-sheet sectioning data along the rotation axis. (c) Slices from views 1 and 3 of the seven views generated from a by applying processes pictured in b and rescaling to isotropic resolution. These seven volumes are the input to the fusion and deconvolution algorithms quantified in d and visualized in e. (d) Cross-correlation of deconvolved and ground-truth data as a function of the number of iterations for MAPG⁷ and our algorithm with and without regularization (reg). The inset compares the computation (comp.) time. (Both algorithms were implemented in Java to support partially overlapping data sets; Supplementary Fig. 10). (e) Slices equivalent to c after content-based fusion¹⁴ (first column), MAPG⁷ deconvolution (second column), our approach without regularization (third column) and with regularization¹⁵ (fourth column; Tikhonov¹⁵ regularization parameter λ = 0.004). (f) Areas marked by boxes in a,c,e at higher magnification. Note the increased artificial ring patterns in MAPG⁷.

As we do not need to consider views to be independent, we next asked whether the conditional probabilities describing the relationship between two views can be modeled and used to improve convergence behavior (Supplementary Figs. 1 and 3 and Supplementary Notes 3 and 4). If we assume that a single photon is observed in the first view, the PSF of this view and Bayes’ theorem can be used to assign a probability to every location in the deconvolved image having emitted this photon (Fig. 1b). On the basis of this probability distribution, the PSF of the second view directly yields the probability distribution describing where to expect a corresponding observation for the same fluorophore in the second view (Fig. 1b). Thus, we argue that it is possible to compute an approximate image (‘virtual’ view) of one view from another view provided that the PSFs of both views are known (Fig. 1c).

We used these virtual views to perform intermediate update steps at no additional computational cost, decreasing the computational effort approximately twofold (Fig. 1d and Supplementary Note 4). The multiplicative combination (equation (2)) directly suggests a sequential approach, wherein each RL update (equation (1)) is directly applied to ψ^r(ξ) (Supplementary Fig. 2 and Supplementary Note 5). This sequential scheme is equivalent to the OSEM⁶ algorithm and results in a 13-fold decrease in convergence time. This gain increases linearly with the number of views⁶ (Fig. 1d and Supplementary Fig. 4). To further reduce convergence time, we introduced ad hoc simplifications (optimizations I and II) for the estimation of conditional probabilities that achieve up to 40-fold improvement compared to deconvolution methods that assume view independence (Fig. 1d–f, Supplementary Figs. 4 and 5 and Supplementary Notes 6 and 7). The new algorithm also performs well in the presence of noise and imperfect PSFs (Supplementary Figs. 6–8). If the input views show a very low signal-to-noise ratio (SNR), atypical for SPIM, the speedup is preserved, but the quality of the deconvolved image is reduced. Our Bayesian-based derivation does not assume a specific noise model, but it is in practice robust with respect to Poisson noise, which is the dominating source of noise in light-sheet microscopy acquisitions.

We compared the performance of our method with that of previously published multiview deconvolution algorithms^5–10 in terms of convergence behavior and run time on the central processing unit (CPU) (Figs. 1e,f and 2d and Supplementary Figs. 4b and 9a,b). For typical SPIM multiview scenarios consisting of around seven views with a high SNR, our method requires sevenfold fewer iterations and is at least threefold faster than OSEM⁶, scaled gradient projection (SGP)⁸ and maximum a posteriori with Gaussian noise (MAPG)⁷. At the same time our optimization is able to improve the image quality of real and simulated data sets compared to MAPG⁷ (Fig. 2e,f and Supplementary Fig. 9c–h). A further speedup of threefold and reduced memory consumption is achieved by using our CUDA (Compute Unified Device Architecture) implementation (Supplementary Fig. 10g). Moreover, our approach is capable of dealing with partially overlapping acquisitions typical in multiview imaging (Supplementary Fig. 10 and Online Methods).

In order to evaluate our algorithm on realistic three-dimensional (3D) multiview image data, we simulated a ground-truth data set resembling a biological specimen (Fig. 2a). We next simulated image acquisition in a SPIM microscope from multiple angles by applying signal attenuation across the field of view, convolving the data with the PSF of the microscope, simulating the multiview optical sectioning and using a Poisson process to generate the final pixel intensities (Fig. 2b and Online Methods). We deconvolved the generated multiview data (Fig. 2c) using our algorithm with and without regularization (regularization adds smoothness constraints to the deconvolution process to achieve a more plausible solution for this ill-posed problem) and compared the results to the content-based fusion¹⁴ and the MAPG⁷ deconvolution (Fig. 2d–f). Our algorithm reached optimal reconstruction quality faster (Fig. 2d) and introduced fewer artifacts than MAPG⁷ (Fig. 2e,f and Supplementary Videos 2 and 3). Tikhonov regularization¹⁵ was required to converge to a reasonable result under realistic imaging conditions (Fig. 2d–f).

We applied our deconvolution approach to multiview SPIM acquisitions of Drosophila melanogaster and Caenorhabditis elegans embryos (Fig. 3a–e). We achieved a substantial increase in contrast as well as resolution with respect to the content-based fusion¹⁴ (Fig. 3b and Supplementary Fig. 11); only a few iterations were required and computation times were typically in the range of a few minutes per multiview acquisition (Supplementary Table 1). We applied the deconvolution to a four-view acquisition of a fixed C. elegans in larval stage 1 (L1) expressing GFP-tagged lamin (LMN-1–GFP) labeling the nuclear lamina and stained for DNA with Hoechst (Fig. 3f,g). Multiview deconvolution improved contrast and resolution compared to the input data and enabled unambiguous segmentation of nuclei in problematic areas of the nervous system¹⁶ (Supplementary Videos 4–7). The algorithm dramatically improved multiview data acquired with OpenSPIM¹⁷ (Supplementary Fig. 12), and its effi-ciency makes it applicable to spatially large multiview data sets (Supplementary Fig. 13) and to processing of long-term time lapses from the Zeiss Lightsheet Z.1 (Supplementary Videos 8–11 and Supplementary Table 1).

Fig. 3. Application to biological data.

(a) Comparison of reconstruction results using content-based fusion¹⁴ (top row) and multiview deconvolution (bottom row) on a four-cell–stage C. elegans embryo expressing a PH domain–GFP fusion marking the membranes. Dotted lines mark plots shown in b; white arrowheads mark PSFs of a fluorescent bead before and after deconvolution. (b) Line plot through the volume along the rotation axis (yz, contrast locally normalized). This orientation typically shows the lowest resolution of a fused data set in light-sheet acquisitions, as all input views are oriented axially (Supplementary Fig. 11). SNR is substantially enhanced; arrowheads mark points illustrating increased resolution. (c,d) Cut planes through a blastoderm-stage Drosophila embryo expressing His-YFP in all cells. (e) Magnified view on parts of the Drosophila embryo. The left panel is a view in lateral orientation of one of the input views; the right panel shows a view along the rotation axis characterized by the lowest resolution. (f,g) Comparison of deconvolution and input data of a fixed L1 C. elegans larva expressing LMN-1–GFP (green) and stained with Hoechst (magenta). (f) Single slice through the deconvolved data set; arrowheads mark four locations of transversal cuts shown below. The cuts compare two orthogonal input views (0°, 90°) with the deconvolved data. No input view offers high resolution in this orientation approximately along the rotation axis. (g) The left box in the first row shows a random slice of a view in axial orientation (worst resolution). The second row shows a view in lateral orientation (best resolution). The third row shows the corresponding deconvolved image. The right boxes each show a slice through the nervous system. The alignment of the C. elegans L1 data set was refined using nuclear positions (Online Methods). The C. elegans embryo (a,b) and the Drosophila embryo (d,e) are each one time point of a time series (none of the other time points is used in this paper). The C. elegans L1 larva (f,g) is an individual acquisition of one fixed sample.

Multiview deconvolution increases contrast in SPIM data after acquisition, complementary to hardware-based contrast enhancement achieved by digital scanned laser light-sheet microscopy (DSLM-SI)¹⁸ (Supplementary Fig. 14). Moreover, multiview deconvolution produced superior results when comparing an acquisition of the same sample with SPIM and a two-photon microscope (Supplementary Fig. 15). Finally, the benefits of the multiview deconvolution approach are not limited to SPIM, as illustrated by the deconvolved multiview spinning disc confocal microscope acquisition of a C. elegans in L1 stage¹⁴ (Supplementary Fig. 16).

A major obstacle for widespread application of deconvolution approaches to multiview light-sheet microscopy data is the lack of usable and scalable multiview deconvolution software. Therefore, we implemented our fast converging algorithm as a Fiji¹⁹ plug-in taking advantage of ImgLib2 (ref. 20) and GPU processing (http://fiji.sc/Multi-View_Deconvolution). The only free parameter of the method that must be chosen by the user is the number of iterations for the deconvolution process. We facilitate this choice by providing a debug mode allowing the user to inspect all intermediate iterations and identify optimal trade-off between quality and computation time. Our Fiji¹⁹ implementation synergizes with other related plug-ins and provides an integrated solution for the processing of multiview light-sheet microscopy data of arbitrary size.

Methods

Methods and any associated references are available in the online version of the paper.