Computational adaptive optics for live three-dimensional biological imaging

Abstract

Light microscopy of thick biological samples, such as tissues, is often limited by aberrations caused by refractive index variations within the sample itself. This problem is particularly severe for live imaging, a field of great current excitement due to the development of inherently fluorescent proteins. We describe a method of removing such aberrations computationally by mapping the refractive index of the sample using differential interference contrast microscopy, modeling the aberrations by ray tracing through this index map, and using space-variant deconvolution to remove aberrations. This approach will open possibilities to study weakly labeled molecules in difficult-to-image live specimens.

The present renaissance in microscopy has been made possible in large part by progress in imaging and analysis of three-dimensional (3D) information in whole cells and tissue (1, 2). The development of intrinsically fluorescent proteins (3–5) opened avenues for a wide range of dynamic studies in live sample microscopy. Much like kinetic analysis contributed to understanding mechanisms of chemical reactions, in vivo microscopy is providing new insight into complex, time-dependent cellular mechanisms. Because these mechanisms are often mediated by a small number of molecules, the ability to detect faint signals in vivo is critical.

Under their design conditions, modern microscope optics produce nearly ideal aberration-free imaging. However, these conditions are in general true only if the object of interest is immediately adjacent to the coverslip. When focusing into thick samples, the 3D optical characteristics of the sample itself must be considered as part of the optical system. For homogeneous specimens, such as fixed and embedded tissue, the major problem is sample-depth-dependent spherical aberration (6–8); this can be largely corrected for by a choice of appropriate refractive index immersion oil for a particular depth (9, 10), by adjustment of correction collars on special objectives (11), by adjustable optics beyond the objective lens (12), or by computational corrections (13, 14).

When imaging is attempted in live cells, one has in addition to address further problems that are less severe in fixed preparations: live cells contain a number of organelles, ranging from submicron vesicles to the many micron-sized nuclei, each with its own refractive index (15, 16). The refractive index heterogeneity is equivalent to adding optical elements that locally modify the properties of the 3D microscopic imaging, and thereby cause distortions, degrade image resolution, and reduce the signal-to-noise ratio. As the index variations are local, different regions show different distortions that cannot be corrected by global approaches. A demonstration of such distortions is shown in Fig. 1. These distortions spread out the intensity of a specific fluorescent label, causing what would have been a detectable signal to disappear below the background noise level. Although single fluorophores have been imaged on slides (17–22), even multilabeled sites are difficult to detect in vivo. In the case of homogeneous samples, computational deconvolution methods are very effective at removing out-of-focus contributions, in effect reassigning the photons back to their correct positions. However, in live sample microscopy, the spatially varying aberrations render common space-invariant deconvolution methods less effective, and can cause false sample features to appear in the reconstructions, leading to incorrect interpretations of the biological data.

Figure 1.

The problem: Sample-induced optical aberration due to heterogeneous refractive index. (A) An example of a nonaberrated 3D image of 0.1-μm bead using high magnification (100×/NA1.3 objective) microscopy. In-focus and +/− 3 μm defocus images are shown. The subpanel A' is an xz section through the center of the 3D bead image. (B) A fluorescent bead imaged three-dimensionally (as in A) under a polytene nucleus in a Drosophila salivary gland cell, showing sample-induced distortions. B' is the corresponding xz section. (C) The image of a field of view containing several beads under a polytene nucleus, showing the spatial dependence of these distortions. The image is taken slightly out of focus to emphasize the different distortions. (D) DNA-specific dye Oli Green staining of polytene chromatin in a live cell. The optical section shown was processed by deconvolution using the symmetric unaberrated PSF. The familiar chromosome bands are almost entirely obscured by distortions for both the unprocessed or processed image. (Scale bars, 2 μm.)

An analogous problem exists in astronomy. Atmospheric air turbulence, with refractive index variations that fluctuate on millisecond time scales, severely distorts stellar images. Astronomers have shown that if they map these optical distortions by using guide stars, they can correct local imaging problems dynamically with adaptive optics based on programmable deformable mirrors (23–26). Similar methodologies have been recently extended to microscopic imaging (27, 28). However, in microscopy it is not practical to embed guide stars in live samples, but the 3D refractive index variations can be estimated by using Nomarski Differential Interference Contrast (DIC) microscopy (29–33). This refractive index information can be used to model the sample-induced distortions of the image of a point source (the point spread function, PSF) and correct their effects by space-variant deconvolution methods. In this paper, we use test samples to demonstrate the feasibility of this approach.

Materials and Methods

Microscopy.

A Zeiss Axiomat microscope under computer control was modified to use fiber optics illuminators stabilizing spatial light variations for transmission (DIC) and fluorescence imaging (34). Temporal variations of the illumination were corrected by hardware and computer image-by-image processing (34). Fluorescent beads were prepared on slides as described (9). Oil droplets were sprayed on the slides from a capillary tube by using a pressurized air stream and mounted in glycerol (29).

Ray Tracing in a Medium with Continuously Varying Refractive Index.

Wavefront distortions were calculated by ray tracing through refractive index maps derived from DIC data by line integration (29) or model fitting.‖ The problem of ray tracing in an analytically known index has been treated in the literature (35). To allow rapid tracing through pixelated index data, we developed a different algorithm. Within each voxel, we approximate the refractive index by a linear gradient. Snell's equation (ref. 36, p. 84) for the ray path within the voxel can then be solved analytically for ray direction vector:

1a

for ray position:

1b

and for ray optical path

1c

where V is the unit direction vector of the gradient, v is the distance along V, r is the position of the ray with refractive index N, α is the incident angle of S with V, W is the unit vector perpendicular to V in the plane defined by V and the incident direction S,

and

This formulation allows tracing millions of rays through samples within minutes. In this work, we did not consider polarization or reflection effects.

Calculation of the PSF.

The image formed from a propagating light wavefront can be evaluated by the Fresnel–Kirchhoff interference integral (ref. 36, p.462; refs. 37 and 38), which has been applied in different levels of approximation to light microscopy (6–8). For low numerical aperture image formation, the Fraunhofer approximation (e.g., ref. 36, pp. 401–434), can be applied. In this article, we use ray tracing on the high-numerical-aperture sample side of the microscope optics, followed by a Fraunhofer diffraction integral to model the image formation on the camera side, where aperture angles are small and the Fraunhofer approximation therefore excellent. This model allows the PSFs to be calculated rapidly by using the Fast Fourier Transform.

To take into account the unknown aberrations in the microscope itself, we derived the following modified calculations that apply the aberrated wavefront to the microscope's unaberrated measured PSF without requiring exact knowledge of the complex pupil function of the microscope. The intensity PSF of the microscope under ideal conditions can be written as the squared magnitude of the 3D Fourier Transform of a 3D pupil function P(k) defined on a spherical shell cap (39):

2a

where FT is the 3D Fourier transform, k is a 3D frequency space vector, P(k) is non-zero only for vectors k whose directions lie within the aperture angle of the objective lens and whose |k| is one over the fluorescence wavelength.

An aberrating sample will distort the amplitude and phase of the wavefront, effectively multiplying the pupil function by a complex aberration function U(k). The resulting aberrated point spread function becomes

2b

We can estimate U(k) by ray tracing, but still cannot evaluate Eq. 2b directly, because the complex pupil function P(k) is unknown. An approximated relation is used instead. We define W by

2c

where ⊗ denotes a convolution; from the Fourier convolution theorem

2d

where

2e

The merit of this apparently circular argument is this: if we calculate W(k) from Eq. 2d as a function of P, we find that the ratio between the aberrated and unaberrated OTFs renders W relatively insensitive to the details of P. We can therefore substitute P = 1 in Eqs. 2a and 2b, evaluate W from Eq. 2d for every ray-traced PSF, and convolve it with the experimentally measured unaberrated PSF (Eq. 2c) to obtain a greatly improved approximation for the sample-aberrated PSF_A.

Deconvolution.

The deconvolution algorithms were used as described in ref. 40. In this demonstration, spatially variant deconvolution was simulated by processing separate subvolumes by spatially invariant deconvolution with separate PSFs and reassembling them afterward. The reassembling process adds only one to several minutes, depending on the number and size of the subvolumes.

Results and Discussion

A direct way to characterize the effect of refractive index heterogeneities on optical system distortions is to image a known test object, such as a subresolution fluorescent bead. For nondistorting optics, the image of a point source, the PSF, closely approaches a diffraction limited pattern, essentially determined by the wavelength of light and the numerical aperture of the objective (6–9). Fig. 1A shows such a measured 3D image pattern for a 0.1-μm bead under undistorted conditions, displaying the characteristic diffraction ring pattern on both sides of a sharp focus. A very different pattern is observed when fluorescent beads are imaged under a sample having a heterogeneous refractive index; in this case Drosophila salivary gland tissue containing polytene nuclei (Fig. 1B). The 3D bead image is greatly distorted, displaying poorly defined focus and significant asymmetry, radial and axial, at both sides of the focal plane. These distortions depend on position. In Fig. 1C several beads imaged in the same field of view show different distortions for different locations within the sample. These patterns of distortion are readily correlated with the fluctuations in the refractive index above each bead. Fig. 1D shows that live biological images are strongly degraded by these effects, which increase dramatically as a function of depth.

The alternative 3D microscopic imaging method, confocal scanning, is similarly sensitive to aberrations, arguably more so as aberrations accumulate on both the excitation and detection optical paths (7, 8). Samples used in this paper were also imaged on a confocal microscope to verify that the observed effects on wide-field microscopy are also problematic for confocal microscopy. The measured fluorescent bead images were indeed fainter, more extended, and distorted unlike the unaberrated images for the same confocal microscope.

In addition to refractive distortions, optically inhomogeneous samples can cause scattering. A characteristic property of scattering is that it decreases rapidly with increasing wavelength. In our observations, images of green and red fluorescent beads under the same Drosophila salivary gland tissue show very similar degrees of aberration. Therefore, we attribute the distortions seen in Fig. 1 mainly to light refraction, not scattering.

Our approach to addressing sample-induced aberration consists of three steps: (i) mapping the refractive index of the sample; (ii) aberration modeling by ray tracing; and (iii) space-variant deconvolution. Refractive index mapping has been described elsewhere (29–32). This article mainly addresses the second step and reports preliminary results as a proof of principle for the third. We use a simple test case, an oil drop in glycerol as the aberrating sample. Full restoration of degraded biological images, such as those in Fig. 1, will require additional work.

The approach is diagrammed in Fig. 2. In microscopic imaging, fluorescence emitted from a point within the sample creates a diverging wavefront that propagates through the sample and into the entrance pupil of the objective lens. It is transformed by the microscope optics into a converging spherical wavefront that forms the image on the camera. When refractive index heterogeneities in front of a point source perturb the imaging optical path (Fig. 2A), the wavefront at the entrance pupil is distorted. To calculate these distortions we first map the refractive index variations using DIC images of the same sample (Fig. 2B). DIC image intensities approximate the gradient of the refractive index along the shear direction defined by the DIC optics. Images acquired for a set of optical sections are line integrated along this shear direction, yielding (with appropriate calibration) an approximate refractive index map of the total sample volume (29–31). To define the distorted optical wavefront from a point source located at a point within the sample, a large number of rays are computationally traced from that point. The trajectories of these rays are refracted and phase delayed by the local variations in refractive index and define the distorted wavefront.

Figure 2.

The solution: A schematic outline. (A) Sample-induced aberration: The imaging situation is modified by variations in refractive index within the sample volume, resulting in highly distorted wavefront originating from the point source. For the presentation, three spheres with refractive index higher by 0.15 units above the medium were simulated in the sample volume. For viewing purposes, the in-focus image is 30 times attenuated with respect to the out-of-focus images shown (unaberrated cases would have 300 times the corresponding attenuation). (B) Evaluating the refractive index variations within the sample: The imaging conditions in A are observed with DIC optics (symbolically shown by the Wollaston prism), resulting in the 3D image consisting of a series of two dimensional gradient images of the refractive index of the sample. The three DIC images shown are at focal planes centered about the three spherical objects. They are processed by line integration (29) to yield a 3D map of the refractive index of the sampled volume. (C) Ray tracing through the sample: Light rays emerge isotropically from a point within the sample volume. Computerized ray tracing for the shown three spheres in the sample volume gives the distorted diverging wavefront at the entrance pupil of the microscope due to refraction of the ray directions and deviation of their optical paths from the ideal spherical wavefront (illustrated by extended ray ends). The Kirchhoff interference integral is applied to this wavefront to calculate the corresponding distorted point diffraction 3D image. (D) Deconvolution by using the distorted image reconstructs a point source.

To test the validity of this approach for local PSF characterization, we prepared samples consisting of oil droplets (Cargille Laser Liquid, refractive index 1.5180) dispersed in a glycerol layer over a field of 0.1-μm fluorescent beads. The difference in refractive index between glycerol and oil is comparable, or slightly higher, than the variations in a typical biological sample. Fluorescence and DIC images were collected from the same area (Fig. 3 A and B). The DIC images yield the refractive index map for the data set. For comparison between the ray-traced and the measured PSFs, a simulated refractive index map was constructed in which the oil drop was modeled as a sphere of known refractive index with dimensions based on the DIC images. Ray tracing was then used to calculate the local 3D PSF at the bead positions (Fig. 3C). The ray-traced PSFs closely reproduced the details of the measured bead images, feature by feature in all dimensions.

Figure 3.

Proof of principle: (A) Submicron beads on a slide were imaged under an oil droplet (18 μm in diameter). Distorted aberrated images were recorded. In the Inset, three optical sections for the bead 1 display a strong out-of-focus “flare” and highly asymmetric out-of-focus diffraction pattern (arcs instead of Airy rings). (B) The DIC images of this sample recorded three-dimensionally. Orthogonal section views are presented, with the superimposed projected positions of the beads. (C) PSFs computed by ray tracing through the oil droplet for the positions of the beads. The insert is three optical sections calculated for the bead 1. (Scale bars, 2 μm.)

Fig. 4 compares the measured PSF with the PSFs calculated from ray tracing through the modeled oil drop or through the measured DIC refractive index data. As seen, the two ray trace calculations yield very similar results, supporting the validity of the DIC method. The local differences between the two calculations are about 20% of the maximum intensity (Fig. 4 B, C, and F–G). To keep the evaluation of the ray tracing and PSF calculation separate from questions of the accuracy of the DIC index measurements, we use PSFs calculated from the model index map in the following comparisons. Note the difference in contrast and sharpness of interference details between the measured and calculated PSFs; this difference is further corrected by applying Eqs. 2a–2e (Fig. 4 D and H).

Figure 4.

Comparison of measured and calculated PSFs: A–D are defocused optical sections of bead 1 (as labeled in Fig. 3A) at 3.0 μm below focus and E–H are optical sections of bead 2 at 2.75 μm below focus. (A and E) A measured image of the 0.1 μm bead. (B and F) A computed 3D ray-traced PSF using a refractive index map from the line integrated oil drop DIC data. (C and G) A ray-traced PSF using simulation of an oil drop with uniform known refractive index. (D and H) A computed PSF in which the aberrated wavefront calculated by ray tracing through a simulated oil drop of uniform refractive index was applied to a measured, unaberrated PSF by using Eqs. 2a–2e. (Scale bars, 2 μm.)

The features of the calculated PSFs look remarkably similar to those measured experimentally (Fig. 4). However, some minor differences can still be discerned—in particular the contrast of out-of-focus features is higher in the calculated image than in the measured data. We studied the parameters that might be causing such differences. First, Fluorescent images contain a range of wavelengths that blur the PSF laterally and axially away from focus, as the PSF dimensions scale with the wavelength. In addition, chromatic aberrations, corrected close to the coverslip, may not be negligible for deep sample imaging. Simulation of these wavelength effects, however, cannot account for the above differences. Secondly, apodization of the pupil transfer function modifies the PSF (see discussion in ref. 36, pp. 496–497 for telescopes). Measured back aperture intensities for our microscope fits the sine rule (Abbe's condition; ref. 36, pp. 225–226), which excludes a significant intensity apodization factor. However, such intensity measurements cannot exclude polarization and phase shift effects, which are known to increase at the high numerical aperture edges of even best objectives (41). These differences are a point of concern for optimal use of the method.

We use the experimental data in Fig. 3 that contains abberrated bead images and apply to each one a separate constrained, iterative deconvolution (Fig. 5A). To quantify the improvement achieved by the ray tracing method, we compared the deconvolution results on one of the bead images using three different PSFs: the bead image itself (the “true PSF,” Fig. 5B, solid line); the ray traced PSF modified by Eqs. 2 (“ray traced PSF,” Fig. 5B, dash-dot); and a measured, unaberrated PSF (“standard PSF,” Fig. 5B, dashed). We compared two quantities: the integrated intensity in the main peak, and the integrated intensity in the undesired flare caused by aberration. Both the ray traced and true PSFs caused the peak integrated intensity to increase by a factor of 3.5 compared to the standard PSF. The flare aberration was a factor of 4.5 lower for the ray traced PSF than with the standard PSF, to be compared with a 7.7-fold reduction for the true PSF. The ratio of peak (signal) to flare (non-signal) intensity increased by a factor of 15.7 for the ray-traced PSF relative to the standard PSF. This improvement should correspond to a significantly enhanced ability to detect low signal-to-background objects.

Figure 5.

Deconvolution results: (A) The beads shown in Fig. 3A were deconvolved with the ray-traced PSFs based on the refractive index map computed from Fig. 3B. The line profiles through the bead centers plot the logarithmic intensities down to the 10⁻⁴ noise level. (Scale bar, 2 μm.) (B) A quantitative comparison of the effect of various PSFs on the deconvolution process for bead 1 of Fig. 3A is shown as a function of the number of iterations. (Top) The peak integrated intensity. (Middle) The integrated intensity of the flare. (Bottom) The ratio of peak to flare intensity, a figure of merit for deconvolution effectiveness. Each plot has been normalized to the predeconvolution integrated intensity. The PSFs used to deconvolve bead 1 are: the image itself (Fig. 4A, solid line), a bead image taken under conditions of minimal distortions (Fig. 1A, dashed line), a ray-traced computed PSF based on the simulated refractive index map (Fig. 4C, dotted line), and a PSF calculated by applying the ray-traced wavefront distortions to a measured, unaberrated PSF using Eqs. 2a–2e (Fig. 4D, dash–dot line). After 15 deconvolution iterations, the signal (peak) intensity to nonsignal (flare) intensity ratio is improved 8-fold by using the ray-traced PSF and 15-fold by using the PSF modified by Eqs. 2a–2e, when compared with those using an unaberrated PSF.

The rigorous method for correcting live sample images would be to estimate a separate PSF for each point in the sample, and then apply a point-by-point fully space-variant deconvolution. Unfortunately, this approach is computationally intractable today for 3D data. For example, when astronomers apply this approach to 2D images, the processing time is on the order of days (42). However, one could use a “piecemeal” deconvolution approach in which an image containing N voxels is subdivided into a number of subimages, each deconvolved by a different PSF. These subimages are then merged together onto the large N voxel volume, which can also be separately deconvolved by a PSF before merging. This allows use of Fast Fourier Transform methods for each of the subvolumes, thus speeding up processing time. Piecemeal deconvolution lends itself to highly parallel processing schemes, and can be spread out over a number of conventional computer workstations (42–44). By using a network of 800 MHz Pentium III workstations, we have implemented a parallel, piecemeal deconvolution and tested its performance. For 15 iterations of Gold's method on a 256 × 256 × 32 voxel image, a model piecemeal deconvolution using the root processor for the whole image volume and eight other processors for two subvolumes with image merging every five cycles required 142 seconds, not a great increase from the 68 seconds required for a space-invariant deconvolution of the large volume alone.

The piecemeal approach assumes that the PSF varies smoothly with position. It may seem that refractive index discontinuities that are close to ray position r would move fast in and out of the ray cone as a function of r, causing the PSF to vary rapidly with r. However, because the wavefront distortions are accumulative over the entire ray path, the contribution of close by (and inevitably small) objects falls is not a large fraction of the total distortion. Model simulations for many dispersed oil droplets, such as Fig. 2B, show that the distortions in the PSF vary continuously with position. Thus, we expect the piecemeal method to be useful for many biological samples.

The application of this approach to live samples is technically well within reach: Fast acquisition of multicolor (DIC and fluorescence) 3D images is possible by using computerized microscope systems. Algorithms that may further reduce processing time include new variants of the deconvolution algorithms using Toeplitz matrix inversion mathematics (45), and multiresolution hierarchical approaches (46, 47). We anticipate that correction of sample-induced distortions will be essential for studying processes in live cells that are mediated by crucial but rare components probed by fluorescent fusion proteins.

Abbreviations

PSF

point spread function

DIC

differential interference contrast

3D

three-dimensional