High dynamic range fluorescence imaging.

Abstract

Fluorescence acquisition and image display over a high dynamic range is highly desirable. However, the limited dynamic range of current photodetectors and imaging CCDs impose a limit on the fluorescence intensities that can be simultaneously captured during a single image acquisition. This is particularly troublesome when imaging biological samples, where protein expression fluctuates considerably. As a result, biological images will often contain regions with signal that is either saturated or hidden within background noise, causing information loss.

In this manuscript we summarize recent work from our group and others, to extended conventional to high dynamic range fluorescence imaging. These strategies have many biological applications, such as mapping of neural connections, vascular imaging, bio-distribution studies or pharmacologic imaging at the single cell and organ level.

I. INTRODUCTION

INNOVATIVE high-throughput and high-resolution imaging modalities along with the development of new clearing techniques [1], have provided the necessary tools to facilitate whole organ mapping at the microscopic level [2,3]. Even though terabyte-size datasets can be collected, recognizing and enumerating the elemental cellular components within whole organs requires the use of sophisticated unsupervised image processing techniques. Ultimately, the quality of the collected data depends on the acquisition parameters and the accuracy of the segmentation process, irrespectively of the specific algorithms used.

In fluorescence microscopy, the dynamic range of the photodetectors determines the maximum range of the detectable input fluorescence signal and thus the maximum and minimum intensities that can be simultaneously recorded. For biological samples, the fluorescence dynamic range of a captured image, which is related to the distribution of protein expression or fluorescently labeled antibodies staining, is typically larger compared to the dynamic range of the microscope detector which commonly spans over several orders of magnitude. This contributes in making it extremely difficult, if not impossible, to obtain accurate segmentations across different areas in both single images and whole organs. Moreover, noise and background signals contribute also negatively on the actual dynamic range, effectively reducing the detectors’ bit depths. Thus, extending the fluorescence images’ dynamic range and SNR is crucial for accurate cellular segmentation and quantification.

Multi-exposure acquisition strategies coupled with fusion algorithms have become standard in photography [4] and their principles may be extended to fluorescence microscopy. In these techniques, images with varying degrees of saturation are fused together to give rise to a single unsaturated image covering a dynamic range larger than the one of the single acquired images. Lately this strategy has also been extended into the biological, biomedical and clinical fields [5] as well as other imaging areas [6–8]. High dynamic range (HDR) X-ray radiographic imaging experiments have been shown to accurately resolve the internal features of complicated structural components at different thicknesses [9]. High dynamic range full-field optical coherence tomography based on exposure bracketing has been proposed, demonstrating reduction in the CCD frames spatial noise for improvement in fringe contrast [10]. High dynamic range three dimensional laminar optical tomography has also been demonstrated for increased imaging penetration depth and range in fluorophore detection [11].

Here, we present recent results from our and other groups, to develop fluorescence imaging acquisitions with improved dynamic range. In particular we focus on the development of a new technical approach for high dynamic range confocal and two-photon microscopy that extends the imaging dynamic range in fluorescence optical scanning microscopy [3]. With this technique we have demonstrated composite 32-bits HDR images of the brain with improved SNR and dynamic range. The technique is also suited for real-time in vivo imaging and has been demonstrated for longitudinal studies as well as for quantitative imaging of in vivo fast tracer kinetics. Alternative approaches are presented also for both microscopic and macroscopic in vivo and ex vivo fluorescence imaging.

II. Imaging And Acquisition Strategy

Although a relative high number of bits (8–16) are typically available for common fluorescence imaging detectors, only a limited portion of the detector’s dynamic range is available for proper signal quantization, with both dark noise and saturated signals near the detector’s maximum threshold disregarded. A classic method to increase the dynamic range consists in using a multi-exposure approach as delineated by Debevec et al. [12]. With this approach the dynamic range extension is obtained algorithmically through the fusion of multiple low dynamic range images with the same field of view (Fig. 1), without the need of a new type of detector. Depending on the setup configuration, images can be equivalently acquired sequentially or simultaneously from one single detector or multiple ones respectively.

Fig. 1.

Principle of high dynamic range imaging (HDR). Low dynamic range (LDR) images progressively saturated in intensity are acquired sequentially or in real time and an HDR image is reconstructed. Each image covers different areas of the total fluorescence dynamic range, with different sampling resolution. The stained vasculature of the heart presents great variability in fluorescence signal in relation to the labeling dye variation, and is better emphasized in the HDR fused image than in the associated LDR ones. Here the heart vasculature is labeled with Dil and organs are cleared. Scale bar, 125 μm. Adapted and reprinted with permission from [13].

A scheme illustrating a typical processing pipeline for obtaining HDR images is given in Fig. 2, and it can be generally described as consisting of five distinct phases. During the first one, images with a low dynamic range are acquired using different exposure times, or laser powers (see Section III). The acquisition can be performed simultaneously or sequentially, depending on the imaging setup available and the specific imaging modality. Images are then processed for noise reduction and/or motion artifacts removal, if present. Correction for any eventual detector signal response over its dynamic range is also necessary. During a fourth phase, HDR images are obtained by fusing information present in the acquired multiple images as illustrated in Section III. Finally, HDR images are remapped at a lower dynamic range, typically using a global nonlinear transformation followed by histogram equalization, to generate remapped HDR (rHDR) images for proper display visualization (see Section IV).

Fig. 2.

High dynamic range (HDR) image processing pipeline.

III. Fundamental Concepts of High Dynamic Range Acquisitions

Commonly, the first step for obtaining HDR images consists in acquiring a series of N distinct low dynamic range (LDR) images I_n with the intent to sample the entire signal range of the object under investigation, and create a final HDR image I_HDR with a dynamic range greater than that of the individual LDR ones. A detailed scheme of the image processing algorithm is given in Fig. 3. The different LDR images are typically obtained by varying a parameter Δα_k in the acquisition process (e.g. increasing the excitation power, or the acquisition integration time), so that the detector is subject to different light conditions. In practice two or three images (N=2,3) are sufficient to achieve good results when dealing with biological samples and with reasonable good quality detectors (e.g. 12 bits). Also, if the images are acquired simultaneously in real-time no inter-frame movement will be present during acquisition, and HDR images can be seamlessly obtained without motion induced artifacts.

Fig. 3.

Scheme of the HDR image processing algorithm for the generic case of N acquired LDR images. The indeces (i,j) run over all pixels of the image matrix.

Because the sensor response function is not necessarily linear over its dynamic range, a necessary step consists then in characterizing the sensor response to mathematically correct each individual LDR image. In digital photography, the sensor response is usually recovered from the acquired images on a case-by-case basis [12], while in confocal and multiphoton imaging the sensor response can be easily characterized before an experiment takes place. Besides, algorithms that exploit a sensor response known a priori, lead to more robust results, in particular against noise.

After the acquisition phase, images are then fused together and the final image I_HDR is reconstructed. The fusion of the LDR images is a highly parallelizable process. Considering an image as a matrix, a HDR pixel at position (i,j)_HDR can be calculated from the corresponding (i,j) pixels within the series of the N individual LDR images (i,j)_LDRk with k=1,2,..N (Fig. 3). In this way the process to compute a HDR image can be limited to one pixel and iterated accordingly over the entire matrix.

While for CCD based imaging all image pixels are associated with a different physical detector (represented by the collection of all pixels present within the sensor array), for confocal and multiphoton microscopy the detector is unique and associated with all the image pixels (single point detection). Therefore the fusion algorithm will not depend on the (i,j) position within the image.

A generic pixel p_k associated with the k-th LDR image, can then be transformed according to

$$T_k=f\left(r,p_k,\Delta {\alpha }_k\right)$$

where r corresponds to the detector response and Δα_k is the modulator factor. The generic pixels $p_{HD{R}_k}$ of the HDR image is then computed from the T_k values weighted by a window function w (e.g. triangular or bell-shaped functions) defined such that non informative (i.e. saturated values) and low informative (noise) pixels carry less importance in the reconstructed image:

$$p_{HDR}=\frac{{\sum }_k{T}_{k}w\left(p_k\right)}{{\sum }_{k}w\left(p_k\right)}$$

By iterating this process for all pixels within the LDR series, the HDR image I_HDR is finally generated (Fig. 3).

IV. Basic Principles for dynamic range compression

After processing, fusion algorithms allow to build images with extended dynamic range. While such images are more informative, their direct visualization might be troublesome because monitors are not designed to match the newly obtained extended dynamic ranges and alternative solution such as tone mapping are required. Amongst all possible signal transformation, the linear ones are usually not recommended. In fact small variations of signal within homogenously distributed areas are not enhanced from such a remapping, resulting in linearly flattened images. Tone mapping methods based on non-linear transformations and global or local operators are instead preferable [14]. A global non-linear function such as the logarithm is as simple as effective at compressing the range algorithmically. However, the logarithmic compression greatly enhances the weakest signal regions, which are also the ones with highest noise. Alternatively, a global non-linear adjustment based on exponentiation could be used,

$$I_{rHDR}=Z_{max}{\left(\frac{I_{HDR}}{{\left(I_{HDR}\right)}_{max}}\right)}^{\gamma }$$

where Z_max is a rescaling factor used to fill the display range, I_rHDR is the remapped HDR image, (I_HDR)_max is a normalization factor and γ is a manually selected value (see Ref. [13] for more details). Still, sub-regions of the remapped HDR image can lack sufficient contrast with hidden details present within. Histogram equalization allows for stretching the pixel intensity distribution with global contrast enhancement. As this method applies a transformation which is based on the pixel intensity distribution of the whole image, it is not designed for improving contrast locally. For this reason, algorithms exploiting local operators are normally preferred.

A particular class of equalization methods works locally so that even restricted low contrast image areas may reveal structures. However, the advantage of increasing the local contrast may lead to a very noisy signal. To overcome this issue, methods based on local histogram equalization can incorporate a control that limits noise amplification. CLAHE (contrast limited adaptive histogram equalization) [15] is a an algorithm that combines local processing with noise control and it is already found as a built-in function in common image processing programs for research (Fiji, Matlab) in biomedicine. CLAHE algorithm divides an image into tiles and then applies histogram equalization to each one. Tiles are reassembled seamlessly and noise issues are prevented through a histogram clipping procedure. CLAHE algorithm is very easy to use as just a few parameters can be manually determined to achieve good results. In addition parameters may also be automatically computed [15], which is useful if high-throughput or limited user intervention is a requirement.

V. Laser Scanning Microscopy

A schematic representation of a two-photon and a confocal imaging setup for real-time and sequential HDR imaging is given in Fig. 4. During image acquisition the imaging sample is illuminated with a laser beam focused to a diffraction-limited spot, with acquisition occurring on a sequential point-by-point excitation and the laser point moving across a raster scan path over the entire field of view [16–18]. In the two photon real time configuration system (Fig. 4a), different detectors simultaneously acquire images. The presence of the beamsplitters in the fluorescence emission, assures that the detectors cover different areas of the sample’s dynamic range.

Fig. 4.

Schematic representation of two different setups for real time two-photon and sequential confocal HDR imaging microscopy. PMT, photomultiplier. BS, beam splitter. VA, variable attenuator. FL, focusing lens. BP, bandpass filter, DM, dichroic mirror. GS, galvo scanner. Adapted and reprinted with permission from [13].

The high splitting ratio channels of the beamsplitters provide high sampling resolution at low signal values, where saturated signal areas are not resolvable. The low splitting ratio channels instead, provide sampling resolution of the saturated pixels and extend the detectable dynamic range of the PMT. The use of multiple PMTs not only speeds-up imaging acquisition, making it ideal for accelerated acquisition of large datasets, but also enables HDR imaging of samples in motion making this configuration particularly suitable for in vivo applications. In fact, simultaneously acquired images require no registration prior to fusion as images perfectly overlap, and no further processing is needed [19]. Therefore, no artifacts due to image mismatching are introduced by the merging algorithm.

In the confocal sequential imaging configuration (Fig. 4b), HDR imaging can be achieved by varying some of the microscope’s acquisition parameters such as the PMT voltage or the laser power, so that each measurement captures a subrange of the whole dynamic range to be covered. Each choice presents some limitations in terms of noise background or bleaching effects. While this option can be implemented on any imaging system without any particular setup modification, it is not ideal for in vivo imaging applications.

Another elegant solution consists in actively regulating the laser power pixel-by-pixel using negative feedback electronics [20,21] in order to avoid saturation and to keep the signal above the noise background. In a recently proposed configuration [22] laser power and signal are simultaneously detected and controlled, with the feedback loop controlled by a digital field programmable gate array (FPGA).

VI. Ex Vivo And In Vivo Applications

As potential application of our technique, remapped HDR images of a whole clear brain slice is shown in Fig. 5. The visualization of both dim and bright structures within a single image, due to the varying degree of fluorescent protein expression, is greatly enhanced with respect to the associated low dynamic range images enabling to reveal structures previously unattainable with single acquisitions. This enables accurate measurements of fluorophore concentration and better quantification via improved segmentation of cellular and dendritic structures. Quantitative results on HDR two-photon imaging brain reconstructions have been also demonstrated [13].

Fig. 5.

HDR imaging of a brain section (a) with respective LDR images (b-d). LDRs images (e,f) and remapped HDR (g) maximum intensity projection images of the boxed area α in (a). (a-d) Scale bar, 250 μm. (e–g) Scale bar, 100 μm. Adapted and reprinted with permission from [13].

HDR imaging is also particularly well suited for maximum intensity projections (MIP) as a result of the great amount of difference in GFP accumulation between nuclei and dendrites present within the imaged volumes (Fig. 6).

Fig. 6.

Two photon HDR imaging of pyramidal neurons projecting into the cerebral cortex. LDRs images (a,b). Remapped HDR single plane (c) and maximum intensity projection (d), images. Scale bar, 200 μm. Adapted and reprinted with permission from [13].

The simultaneous acquisition in real time of multiple LDR images allows us to obtain in vivo HDR images that are not degraded by the presence of physiologically induced motion artifacts, enabling accurate quantitative assessments of fluorescent drug kinetics in different tissue compartments. In Fig. 7 the kinetic of an intravascular dye within a dorsal window chamber mouse is shown for normal LDR images and for real time HDR quantitative imaging at different regions of interest. The corresponding time-intensity curves showcase the ability that HDR images offer to capture in a quantitative fashion the entire concentration range variability of fluorescent probes.

Fig. 7.

Real time HDR imaging for in vivo measurements of intravascular dye kinetics. Remapped HDR (a), LDRs images (b,c), and corresponding time-intensity curves at different region of interest (ROIs) (d-f). Injected dye, 2MDa FITC-Dextran. Scale bar, 125 μm. Adapted and reprinted with permission from [13].

VII. HDR Fluorescence Molecular Tomography

In vivo whole body visualization of fluorescent probes and proteins can offer insights into molecular targets and physiology. Among the different volumetric imaging modalities that help visualizing fluorescence signal deep in turbid tissue, fluorescence molecular tomography (FMT) offers the capability to obtain 3D tomographic mapping of fluorescence activity throughout whole small animal bodies. Despite being low resolution, the technique is non-invasive and it has been used so far for in vivo imaging of specific molecular targets, pathways and physiological effects in vivo [23].

Typically the sample under study is illuminated with a beam of light (excitation wavelength) at multiple projections and fluorescence images are acquired in transillumination mode using a CCD in combination with imaging optics. Maps of the underlying imaging fluorescence contrast can be obtained using well established mathematical models of tissue photon propagation. Due to the limited dynamic range of the imaging CCD, the quality of the projection images is severely constrained. This poses a great challenge in terms of fluorescence image acquisition particularly when fluorescent targets with large differences in concentration are present in the sample. It also affects the quality of the absorption images necessary for the Born-normalized approach that takes into account the spatially dependent component of the excitation signal [24]. Therefore a trade-off between working at the saturation level and above the noise has to be made, resulting in detrimental reconstruction artifacts.

A method to overcome these limitations has been recently proposed for CCD-based free-space fluorescence molecular tomography [25]. A series of progressively saturated fluorescence image projections are acquired to characterize the CCD response function of the instrument, and HDR fluorescence projection images at every source position are obtained to perform FMT reconstructions. Because the number of images could be extremely high, in order to obviate for the increasingly long acquisition times due to the multiple exposure acquisition scheme (typically three images per HDR reconstruction), images were reconstructed using an iterative re-weighted L1 regularization scheme limiting the number of total HDR projections. Results on phantoms and in vivo on tumor-bearing mice (Fig. 8) show that HDR-FMT provides quantitative results and improves the localization performance of fluorescent targets presenting large concentration differences, with great potential for preclinical and clinical work.

Fig. 8.

HDR-FMT of a tumor-bearing mouse intra-tumorally injected with HPPS fluorescent nanoparticles. Low dynamic range projection image (a) and its superimposition on a white light image (b). (c) XCT image of the mouse during fluorescence acquisitions. T1 and T2 indicate the location of the two seeded tumors. FTM reconstructions using low dynamic range image projections at progressively increasing acquisition times are given in (d-f). HDR-FMT reconstruction (g) and overlay between a reconstructed oblique section and XCT (h). (i) Intensity profiles along the green line in (h) for the HDR and three LDR-FMT reconstructions show that HDR-FMT recovers intensity profiles of the fluorescent targets with the best accuracy. Reprinted with permission from [25].

VIII. HDR Optical Projection Tomography

Optical projection tomography (OPT) [26] is a three-dimensional imaging technique based on transillumination measurements of both transmitted and fluorescently emitted light, with broad use in both developmental biology and gene expression studies [27] and for resolving targeted and activatable near-IR (NIR) fluorescent molecular beacons [28].

For samples to be imaged they need to present very low light diffusion properties with both low scattering and absorption coefficients. This is typically achieved when imaging very small transparent organisms in vivo or by making samples transparent using clearing solutions for ex vivo applications. Fluorescent and non-fluorescent biological samples up to a few centimeters in size can be imaged with resolutions up to 10 microns.

During typical acquisitions, multiple images are taken over 360° with 1° or less steps. In general, the imaging system consists of a lens with high telecentricity, which project photons traveling parallel to the lens optical axis onto a CCD camera [29]. Thanks to the low or relatively absent scattering contributions, the absorption reconstructions are analogous to the one performed for x-ray computed tomography and can be obtained using a common Radon backprojection algorithm. Due to the presence of varying spatially dependent absorption within the sample, fluorescent protein or fluorescence molecular probes distribution reconstructions can be obtained using a weighted method based on Born normalized approach [28,30].

HDR imaging has recently been proposed for OPT by different groups [31,32]. The methodology is similar to the one described above, and image fusion of multiexposure projections of absorption signal are taken to increase the dynamic range of single projections and better resolve contrast elements. The technique has been used by Fei et al. [31] for reconstruction of embryonic vessels in fixed stained zebrafish demonstrating its ability to resolve fine details deep inside the organs without the need for any clearing.

A similar approach based on image fusion, but less data intensive compared to the one cited above, was also used by Cheddad et al. [32] for imaging stained mice organs. In Fig. 9 a comparison between a reconstruction obtained using conventional OPT vs image fusion IF-OPT is shown, emphasizing the benefits in using this approach for better and accurate reconstructions.

Fig. 9.

Renderings of OPT reconstructions of the left lateral mouse liver lobe vessels labeled for smooth muscle alpha-actin. Conventional (a) and high dynamic range image fused (b) OPT reconstructions, obtained using image fusion of multiexposure prejections. Reprinted with permission from [32].

IX. Conclusions

The distribution and concentration of fluorescent protein expression can vary considerably over several orders of magnitude in biological specimens. Developing methods to acquire data over a larger dynamic range and displaying this data in its correct spatiotemporal coordinates is highly desirable. We here present a review of recent progresses in high dynamic range fluorescence imaging at the microscopic and macroscopic scale. We expect these technologies to gain importance as current imaging trends are aimed at capturing cellular distribution information at the whole organ level [33,34]. Other possible applications involve in vivo studies for quantification of drug pharmacokinetics.

Biographies

Claudio Vinegoni (PhD 2002) is currently Asst. Professor at Harvard Medical School, and works at the Center for Systems Biology at Massachusetts General Hospital (MGH). He has published over 90 original publications in peer reviewed journals. His research activity involves the development of novel optical imaging instruments and techniques with applications in the clinical and biomedical arena.

Paolo Fumene Feruglio (PhD 2010) is Visiting Research Fellow at the Center for Systems Biology, Harvard Medical School and Adjunct Professor at the University of Verona. He graduated in Electrical Engineering at the University of Padova and received his Ph.D in Multimodal Imaging in Biomedicine at the University of Verona. His research involves the developing of models and algorithms to process and analyze data with application to in vivo and ex vivo molecular imaging techniques at macro and microscopic level.

Ralph Weissleder (PhD 1985) is the Thrall Professor of Radiology at Harvard Medical School, Director of the Center for Systems Biology at Massachusetts General Hospital (MGH), and Attending Clinician (Interventional Radiology) at MGH. Dr. Weissleder is also Professor of System Biology and a faculty a member of the Department of Systems Biology at HMS and the Dana Farber Harvard Cancer Center. He has published over 900 publications in peer reviewed journals (h-index 175) and has authored several textbooks. He is a member of the National Academy of Medicine, the American Academy of Arts and Sciences, the National Academy of Inventors and the German Academy of Sciences (Leopoldina). Website: csb.mgh.harvard.edu.