Fast optically sectioned fluorescence HiLo endomicroscopy

Abstract.

We describe a nonscanning, fiber bundle endomicroscope that performs optically sectioned fluorescence imaging with fast frame rates and real-time processing. Our sectioning technique is based on HiLo imaging, wherein two widefield images are acquired under uniform and structured illumination and numerically processed to reject out-of-focus background. This work is an improvement upon an earlier demonstration of widefield optical sectioning through a flexible fiber bundle. The improved device features lateral and axial resolutions of 2.6 and 17 μm, respectively, a net frame rate of 9.5 Hz obtained by real-time image processing with a graphics processing unit (GPU) and significantly reduced motion artifacts obtained by the use of a double-shutter camera. We demonstrate the performance of our system with optically sectioned images and videos of a fluorescently labeled chorioallantoic membrane (CAM) in the developing $G$. gallus embryo. HiLo endomicroscopy is a candidate technique for low-cost, high-speed clinical optical biopsies.

1. Introduction

Optical sectioning endomicroscopy has been gaining interest in the medical imaging community. Current techniques to obtain optical sectioning can be broadly separated into scanning and nonscanning. Scanning techniques can be implemented by raster scanning a laser beam across the proximal face of a flexible fiber bundle,^1,2 by distal scanning obtained by vibrating the end of a single optical fiber,^3,4 or by distal scanning with a microelectromechanical (MEMS) device.^5,6 These techniques are limited by the requirement of fast, precise scanning mechanisms. The serial acquisition of pixels makes these techniques susceptible to streaking artifacts in the presence of sample or probe movement.

Nonscanning widefield techniques have been developed that do not rely on fast scanning systems. For example, a miniaturized plane illumination endomicroscope was demonstrated that limits excitation to a thin sheet in the objective focal plane.⁷ While this technique provides optical sectioning, it requires the insertion of a microprism into the tissue to generate the light sheet. Another widefield optical sectioning technique is structured illumination microscopy,⁸ where an illumination grid pattern is projected into the sample, and at least three images are acquired while laterally phase-stepping the grid. An image-processing routine then synthesizes an optically sectioned result. This technique has been implemented with a rigid Hopkins-type endoscope,⁹ as well as through a flexible fiber bundle.¹⁰ However, structured illumination microscopy is susceptible to artifacts due to imprecise grid translation or sample movement. Recently, we introduced a technique called HiLo imaging, which uses alternating uniform and structured illumination to synthesize an optically sectioned image.¹¹ HiLo has reduced mechanical complexity compared with scanning techniques and is insensitive to imperfections in the illumination structure. We previously demonstrated an endomicroscope system that performed HiLo imaging through a flexible fiber bundle,¹² but was limited by a low frame rate, fiber core artifacts, and the need for extensive post-processing. Here we present an improved HiLo endomicroscope system with reduced core and motion artifacts and with near video-rate acquisition capability with real-time image processing.

2. Methods

2.1. Setup

Our HiLo endomicroscope setup is illustrated in Fig. 1. A single-axis galvanometer mirror (Thorlabs GVS001) directs a laser beam (Omicron PhoxX 488, 80 mW) between two illumination paths. In the uniform path the beam travels through a $10\times$ expanding telescope and polarizing beam splitter and is focused onto the back aperture of a $20\times$ objective (Olympus Plan Achromat, 0.4 NA). This uniformly illuminates the proximal face of a flexible imaging fiber bundle (600 μm active dia; 30,000 cores). The distal face of the fiber bundle is equipped with a $2.5\times$ water-immersion micro-objective (Mauna Kea Technologies; 0.8 NA) to give additional magnification and working distance of 60 μm. The fiber bundle is 3 meters long with the distal tip 2.7 mm in diameter and 14 mm in length. The generated fluorescence from the sample is relayed through the fiber bundle, spectrally separated with a dichromatic mirror (Semrock FF506-Di03-25x36) and emission filter (Semrock FF01-536/40-25) and recorded with a CCD camera (PCO Pixelfly USB; binning $2\times 2$; double-shutter mode). The structured path is identical to the first with the addition of a transmission Ronchi ruling (Edmund Optics; $50\mathrm{lp}/\mathrm{mm}$) in a plane conjugate to the objective and micro-objective focal planes. The image of the Ronchi ruling is projected into the sample, modulating the sample fluorescence with a high-contrast grid pattern.

Fig. 1.

(a) HiLo endomicroscope setup. A galvanometer mirror (GM) directs a laser beam between two paths, one delivering uniform illumination and the other projecting the image of a Ronchi ruling (RR) into the sample. The paths are combined with a polarizing beam splitter (PBS). A flexible imaging fiber bundle (FB) is equipped with a micro-objective ($\mu O$) for additional magnification and working distance. Generated fluorescence is spectrally separated by a dichromatic mirror (DM) and emission filter ($\mathrm{F}$), and images are recorded with a digital camera (CCD). (b) Representative uniform and (c) structured images.

2.2. Image Preprocessing

Several difficulties are associated with the use of a fiber bundle for fluorescence imaging. The core fill factor of our fiber bundle is only about 30%, meaning our raw images are sparsely sampled. This sampling is irregular and imparts a quasi-hexagonal honeycomb pattern to the raw images with contrast typically much larger than the sample contrast of interest. Moreover, the fiber cores are heterogeneous in size and thus transmission efficiency. Finally, the fiber core material is autofluorescent, which introduces an extraneous background in our raw images. These problems were found to significantly undermine the quality of our final HiLo images and needed to be addressed.

Several groups have suppressed the appearance of fiber cores by applying Gaussian blurring filters,² disk filters,¹³ or spatial frequency domain filters.^14–17 Glass fiber bundles have perfectly periodic hexagonal sampling that concentrates the core pattern in a few localized regions in spatial frequency space, making frequency domain filtering an attractive option. Quartz fiber bundles like those used in this work are distinctly aperiodic and are less amenable to spectral domain techniques. Histogram equalization can be used in conjunction with simple spatial filtering to enhance image contrast for visualization purposes,¹⁸ but the resulting images are no longer proportional to fluorescence concentration (i.e. are no longer quantitative). The main drawback of simple blurring is loss of resolution and image contrast since relatively wide kernels are required to sufficiently attenuate the core pattern. Also, due to core sampling sparsity, many of the camera pixels do not measure intensity from the sample, but rather from autofluorescence or extraneous reflections. A method proposed by Perchant and Le Goualher involves finding all 30,000 core centroids with a segmentation process, measuring the maximum intensity from each core, then interpolating the scattered data on a uniformly sampled rectilinear grid.^1,19,20 This process is most comprehensive, but is computationally intensive and difficult to achieve in real time.

We introduce here a simple preprocessing routine that corrects for sparse sampling, core heterogeneity, and autofluorescence. This routine is based on an iterative segmentation-interpolation algorithm that removes the appearance of the cores without sacrificing spatial resolution or sample contrast. We begin our iterative procedure with the raw camera image itself, $I^{(0)}(\overrightarrow{\rho })$, which serves as the initial estimate of a core-suppressed image $I_{\mathrm{cs}}(\overrightarrow{\rho })$. For each iteration, a binary core mask, $M^{(n)}(\overrightarrow{\rho })$, is generated to distinguish core pixels from cladding pixels. This is done by setting a local, spatially varying threshold level determined by low-pass filtering the current estimate. Camera pixels above this threshold level are identified as core pixels because the cores are almost universally brighter than the cladding. Similarly, the cladding pixels are identified by the complementary mask $1-M^{(n)}(\overrightarrow{\rho })$. To obtain an improved estimate, $I_{cs}^{(n+1)}(\overrightarrow{\rho })$, the cladding pixels are then assigned the low-pass filtered values while the core pixels retain their original values. The full width at half maximum (FWHM) of the Gaussian low-pass filter kernel is approximately one inter-core distance. This simple segmentation-interpolation procedure is iterated until a desired smoothness is achieved; typically two to five iterations suffice. We emphasize here that the low-pass filter used in this procedure is highly local and averages only over nearest neighbor cores. The core pixels thus retain their original intensity throughout the process, while the cladding pixels are filled in to approach the average of their nearest neighbor cores. The algorithm features the low complexity and fast execution time of Gaussian blurring, and minimal loss of sample resolution and contrast. However, it should not be regarded as simple filtering since the procedure is neither linear nor spatially invariant. The algorithm is summarized below.

Algorithm

$I_{\mathrm{cs}}^{(0)}\leftarrow I_{\mathrm{raw}}$

for $n=0$ to $N-1$ do

$$M^{(n)}\leftarrow \{\begin{array}{cl}1& I_{\mathrm{cs}}^{(n)}>\mathrm{LP}\{I_{\mathrm{cs}}^{(n)}\}\\ 0& \text{otherwise}\end{array}$$

$$I_{\mathrm{cs}}^{(n+1)}\leftarrow M^{(n)}{I}_{\mathrm{cs}}^{(n)}+(1-M^{(n)})\mathrm{LP}\{I_{\mathrm{cs}}^{(n)}\}$$

end for

The next steps in our preprocessing routine involve subtracting the contributions from autofluorescence and correcting for heterogeneities in fiber core efficiency. We follow a modified procedure laid out by Perchant and Zhong.^20,21 Three calibration images are acquired before imaging begins. The first, $I_{\mathrm{bg}}$, is acquired with the laser turned off while imaging a nonfluorescent blank solution, such as water, in a dark compartment. This provides a measurement of camera bias and diffuse room light that passes the emission filter. The second, $I_{\mathrm{af}}$, is acquired under the same conditions but with the laser turned on, providing an estimate of the autofluorescence of the fiber core and cladding material. Since the autofluorescence is a function of laser power while the bias and room light are not, the two images need to be acquired separately. The third image, $I_{\mathrm{hf}}$, is acquired while imaging a homogeneously labeled fluorescent fluid. When camera bias, room light, and autofluorescence signals are subtracted out, this third image provides a mapping of the core transmission and collection efficiencies. In practice, each calibration image is derived from averaging many ($\sim 100$) raw images to improve signal-to-noise ratio. Calibration images are reacquired each time the fiber bundle probe is connected and aligned to the system.

Image preprocessing can be summarized as a two-step process. First, raw images are subject to iterative segmentation-interpolation to fill in the cladding pixels, yielding $I_{\mathrm{cs}}(\overrightarrow{\rho })$. Background subtraction and normalization then yield the preprocessed image

$$I_{\mathrm{pp}}(\overrightarrow{\rho })=\frac{I_{\mathrm{cs}}(\overrightarrow{\rho })-(I_{\mathrm{af}}(\overrightarrow{\rho })-I_{\mathrm{bg}}(\overrightarrow{\rho }))\frac{P}{P_{\mathrm{af}}}-I_{\mathrm{bg}}(\overrightarrow{\rho })}{I_{\mathrm{hf}}(\overrightarrow{\rho })-(I_{\mathrm{af}}(\overrightarrow{\rho })-I_{\mathrm{bg}}(\overrightarrow{\rho }))\frac{P}{P_{\mathrm{af}}}-I_{\mathrm{bg}}(\overrightarrow{\rho })},$$ (1)

where $P$ and $P_{\mathrm{af}}$ are the illumination powers used during the acquisition of $I_{\text{raw}}(\overrightarrow{\rho })$ and $I_{\mathrm{af}}(\overrightarrow{\rho })$, respectively. Note that because the reference images containing laser-induced and laser-independent backgrounds are acquired separately, the former can be scaled to the laser power in use. Our preprocessing is thus robust to changes in illumination power. In particular, this allows us to dynamically adjust the laser power to properly fill the camera dynamic range. After preprocessing the raw structured- and uniform-illumination images, these are then ready for HiLo processing.

2.3. HiLo Principle

The benefit of HiLo over standard widefield imaging is that it provides out-of-focus background rejection. As described in previous work, the uniform and structured images can be roughly decomposed into in-focus and out-of-focus components.

$$I_u(\overrightarrow{\rho })=I_{\text{in}}(\overrightarrow{\rho })+I_{\text{out}}(\overrightarrow{\rho }),$$ (2)

$$I_s(\overrightarrow{\rho })=I_{\text{in}}(\overrightarrow{\rho })[1+M\sin (2\pi {\kappa }_{g}x)]+I_{\text{out}}(\overrightarrow{\rho }),$$ (3)

where $\overrightarrow{\rho }=\{x,y\}$ is the lateral position vector, ${\kappa }_g$ is the grid spatial frequency and $M$ is the modulation depth (note: for ease of presentation, we have used a sine function to model the grid function). The imaged in-focus signal appears modulated by the grid pattern, while the out-of-focus signal does not. It follows that a local evaluation of the grid contrast scales with the in-focus signal. Here we use a rectified subtraction of normalized images to compute the local contrast,

$$C(\overrightarrow{\rho })=|\frac{I_s(\overrightarrow{\rho })}{\langle I_s(\overrightarrow{\rho })\rangle }-\frac{I_u(\overrightarrow{\rho })}{\langle I_u(\overrightarrow{\rho })\rangle }|\propto M|\sin (2\pi {\kappa }_{g}x)|,$$ (4)

where $\langle I(\overrightarrow{\rho })\rangle$ denotes a local average evaluated by Gaussian low-pass filtering with frequency cutoff much lower than ${\kappa }_g$. This not only ensures the modulated component is locally centered about zero, but also makes the evaluation insensitive to differences in global illumination profile. Since the contrast is normalized and thus scales as the proportion of in-focus signal, it is used as a weighting function to select the in-focus portion of $I_u(\overrightarrow{\rho })$. This demodulation technique is reliable for low spatial frequencies below ${\kappa }_g$. An accurate measure of in-focus high spatial frequencies above ${\kappa }_g$ is found by applying a complementary high-pass filter to $I_u(\overrightarrow{\rho })$ directly. The weighting function is not needed for these high spatial frequencies because the frequencies are already inherently axially resolved.²² The final HiLo image is the addition of the high and low spatial frequency images. A scaling factor, $\eta$, is used to level the intensities and ensure a seamless fusion. More detailed descriptions of the algorithm, including how we calculate $\eta$ and deal with noise-induced bias in the contrast measurement, are found in Refs. 11, 23, 24.

$$I_{\text{low}}(\overrightarrow{\rho })=\mathrm{LP}\{C(\overrightarrow{\rho })\times I_u(\overrightarrow{\rho })\}$$ (5)

$$I_{\text{high}}(\overrightarrow{\rho })=\mathrm{HP}\{I_u(\overrightarrow{\rho })\}$$ (6)

$$I_{\mathrm{HiLo}}(\overrightarrow{\rho })=I_{\text{high}}(\overrightarrow{\rho })+\eta I_{\text{low}}(\overrightarrow{\rho })$$ (7)

The final HiLo image is axially resolved across the entire spatial frequency spectrum, from dc to the bandwidth of our imaging optics. The lateral resolution, $\mathrm{\Delta }\rho$, is limited by the sampling of the fiber cores and is given by $2d_{\mathrm{ic}}/M_{\mu O}$, where $d_{\mathrm{ic}}$ is the average fiber inter-core separation, and $M_{\mu O}$ is the micro-objective magnification. The axial resolution of the low-frequency component, $I_{\text{low}}(\overrightarrow{\rho })$, can be approximated by considering a thin fluorescent plane and calculating the decay of the grid contrast as a function of defocus. The contrast of the grid pattern is attenuated by both the illumination and detection optical transfer functions (OTFs), and the axial resolution is defined as the FWHM of this function. Using the Stokseth approximation²⁵ for the 3D OTF, assuming similar excitation and detection wavelengths, and assuming ${\kappa }_g$ is much smaller than the imaging bandwidth, the axial resolution is given by $\mathrm{\Delta }z\approx 0.54{({\kappa }_g\mathrm{NA})}^{-1}$. The transmission of low frequencies is found to decay as ${|z|}^{-3}$ for large defocus $|z|$. In comparison, the axial resolution of the high-frequency component at the filter cutoff frequency ${\kappa }_c$ is given by $\mathrm{\Delta }z\approx 0.68{({\kappa }_c\mathrm{NA})}^{-1}$. This is slightly broader due to the action of only the detection OTF providing axial confinement and is found to decay with defocus as ${|z|}^{-3/2}$ for large $|z|$. A matching of the axial resolution at the transition between high- and low-frequency components (i.e. at ${\kappa }_c$) involves selecting the cutoff frequency appropriately. We have found that choosing ${\kappa }_c$ to be slightly lower than ${\kappa }_g$ provides a reasonably accurate match while still maintaining a reduced artifact in the low channel found at twice the grid frequency. The lateral and axial resolutions of the current system are approximately $\mathrm{\Delta }\rho \approx 2.6\mathrm{\mu m}$ (governed by the Nyquist sampling criterion associated with the fiber cores) and $\mathrm{\Delta }z\approx 17\mathrm{\mu m}$ (governed by the structured illumination grid period).

2.4. Fast Image Processing with a GPU

Our image processing routines were developed and tested in MATLAB (MathWorks). The processing speed was found to be unacceptably slow for real-time processing ($<1\mathrm{Hz}$). To enable real-time processing and display, a library of image management and manipulation functions (memory allocation and deallocation, arithmetic operations, filtering in Fourier space) were written in CUDA-C and compiled to a graphics processing unit (GPU) target (NVIDIA GeForce GTX 280). The library was compiled as a dynamic linked library for maximum utility. A LabVIEW interface (National Instruments) was created to coordinate communication among the camera, CPU memory, and GPU memory. The multicore processor architecture (Intel i7 quad core) and inherent parallelism of LabVIEW greatly improved the speed of the endomicroscope. The entire processing pipeline—preprocessing, HiLo processing, and memory transfers between the CPU and GPU—occurs in 30 ms for an image resolution $520\times 696\text{\hspace{0.17em}\hspace{0.17em}pixels}$. The endomicroscope is thus capable of real-time image processing at speeds up to $\sim 35\mathrm{Hz}$.

3. Results

Figure 2 shows a 1951 USAF resolution target (Edmund Optics) before and after preprocessing. The weak contrast of the resolution elements is overwhelmed by autofluorescence background and the high contrast of the cores. After preprocessing, the resolution elements become clearly visible. A line profile through the preprocessed image shows clear contrast of elements in group 7, including the smallest features (element 6, $288\mathrm{lp}/\mathrm{mm}$) spanning approximately two fiber cores. This result demonstrates that image resolution and contrast are not significantly degraded by our iterative segmentation-interpolation algorithm.

Fig. 2.

USAF 1951 resolution target. (a) Raw and (b) after preprocessing. The weak sample contrast is initially overwhelmed by fiber core contrast, efficiency heterogeneity, and autofluorescence background. Preprocessing improves sample contrast. (c) Line profile through raw (top trace) and preprocessed (bottom trace) images. The preprocessed image reveals the resolution elements in group 7 with discernible contrast (elements 1–6 identified with bars). The fundamental spatial frequency of the smallest feature (element 6, $228\mathrm{lp}/\mathrm{mm}$) is 55% of the Nyquist limit imposed by the average fiber core separation.

To demonstrate the suitability of our HiLo endomicroscope for in vivo imaging we utilized a chick embryo, which is commonly studied in developmental biology research.²⁶ In particular, the chorioallantoic membrane (CAM) was examined as it is a model for angiogenesis and lymphangiogenesis^27–29 and is readily accessed by producing a small hole in the egg shell. Several videos were acquired to showcase the image resolution, background-rejection capacity and speed of HiLo imaging (9.5 Hz).

3.1. Chick Embryo Preparation

Fertilized eggs of the Silkie chick (Gallus gallus) were obtained from a local farm (Golden Egg Farm; Hardwick, MA) and incubated at 38°C and 80% humidity until approximately embryonated day E5. Imaging of the embryo was performed through a small hole in the egg shell. A fluorescent solution (50 mM fluorescein in ${\mathrm{H}}_2\mathrm{O}$) was injected into the amnion and allowed to diffuse for approximately 1 hr before imaging. For some samples, the solution was injected intravenously whereupon it bound to serum albumin and diffused through the embryo. After imaging the embryos were euthanized by hypothermia by storing the eggs at $-15°\mathrm{C}$.

A video of a thin fluorescently-labeled membrane floating through focus provides a demonstration of our sectioning capacity. The raw uniform image directly acquired by the camera [Fig. 3(a)] is included to illustrate the deleterious effects of the fiber cores. The preprocessed uniform image [Fig. 3(b)] shows a marked improvement in sample contrast and features small in-focus structures with full resolution. However, this image also features considerable out-of-focus background haze. The final HiLo image [Fig. 3(c)] preserves the in-focus structures while rejecting this out-of-focus background haze.

Fig. 3.

Thin fluorescently labeled membrane floating through focus. (a) Raw, (b) preprocessed, and (c) HiLo result. The HiLo video preserves the small, bright, in-focus objects with full resolution while rejecting the out-of-focus background. (Video 1, MPEG, 7.9 MB). [URL: http://dx.doi.org/10.1117/1.JBO.17.2.021105.1]Download video file ^{(7.7MB, mpg)}

Figure 4 shows blood vessels and other features of the CAM membrane after intravenous injection of fluorescein. Figure 5 shows a torrent of red blood cells (RBCs) passing through a CAM sinus and later collecting into a venule. The RBCs do not take up the fluorescein stain and appear as shadows on a bright background. These videos illustrate the capacity of our HiLo endomicroscope to capture fast dynamics.

Fig. 4.

CAM membrane after intravenous injection of a fluorescein solution. (a) Raw, (b) preprocessed, and (c) HiLo result. The HiLo video shows enhanced contrast of blood vessels and RBCs. (Video 2, MPEG, 11.5 MB). [URL: http://dx.doi.org/10.1117/1.JBO.17.2.021105.2]Download video file ^{(11.3MB, mpg)}

Fig. 5.

Torrent of RBCs flowing through a CAM sinus and collecting in a venule. (a) Raw, (b) preprocessed, and (c) HiLo result. The HiLo video demonstrates the capacity of the endomicroscope to capture fast dynamics with minimal motion artifacts. (Video 3, MPEG, 10.0 MB). [URL: http://dx.doi.org/10.1117/1.JBO.17.2.021105.3]Download video file ^{(9.8MB, mpg)}

3.2. Imaging Speed and Motion Artifacts

As observed in Fig. 5, our HiLo endomicroscope is largely immune to artifacts arising from sample motion (e.g. respiration, blood flow) or probe motion. Part of the reason for this immunity comes from the fact that all the high-frequency information is derived from a single image ($I_u(\overrightarrow{\rho })$). On the other hand, the low-frequency information is more susceptible to motion artifacts since it is derived from a comparison of two serially acquired images. While the consequence of intra-frame motion during the camera integration time is loss of spatial resolution in the raw images, a much more pronounced motion artifact arises from inter-frame motion when significant translation occurs between the two raw image exposures. The magnitude of this artifact can be roughly estimated by considering two serially acquired images of a sharp edge, both under uniform illumination. In the absence of sample motion, both images are the same, and we have $C(\overrightarrow{\rho })=I_{\text{low}}(\overrightarrow{\rho })=0$ [see Eq. (4)]. If instead the sample translates laterally between images, an artifactual signal occurs in the neighborhood of the edge that appears to be in focus. The trapezoidal profile of this signal can be approximated as a box-top function of unity height and width $\upsilon ({\tau }_{\exp }+{\tau }_{\mathrm{if}})$, where $\upsilon$ is the translation velocity (assumed constant), ${\tau }_{\exp }$ is the exposure time and ${\tau }_{\mathrm{if}}$ is the inter-frame delay. The low-pass filtering that occurs to generate $I_{\text{low}}(\overrightarrow{\rho })$ serves to spatially distribute this artifact, reducing the maximum error at a given pixel. For small $\upsilon$, the error bound is $E_{\max }\propto {\kappa }_c\upsilon ({\tau }_{\exp }+{\tau }_{\text{if}})$, where ${\kappa }_c$ is the low-pass filter cutoff frequency. Under typical imaging conditions ${\tau }_{\mathrm{if}}\gg {\tau }_{\exp }$ and the error is governed by the camera readout time.

To minimize ${\tau }_{\mathrm{if}}$, we made use of a double-shutter camera (PCO Pixelfly USB). Such a camera acquires images pairwise with an inter-frame delay of only $\sim 5\mathrm{\mu s}$, followed by a paired readout. While the net frame rate for image pairs (i.e. the HiLo frame rate) is the same with a double-shutter camera as with a traditional camera, the key benefit is the reduction in ${\tau }_{\text{if}}$, which greatly suppresses $E_{\max }$. We note that in our case ${\tau }_{\text{if}}$ was limited to a minimum $\approx 1\mathrm{ms}$ by the need to switch illumination paths with the galvanometer mirror. In principle, two dedicated, electronically shuttered illumination sources could be used to overcome this limitation, though with diminishing returns when ${\tau }_{\mathrm{if}}\ll {\tau }_{\exp }$. In practice, motion artifacts are minimal when displacements are kept below $\sim {(2{\kappa }_g)}^{-1}$.

To illustrate the improved immunity to motion artifacts obtained with a double-shutter camera, we compare HiLo videos acquired with double-shutter versus traditional timing (see Fig. 6). The videos share the same raw data actually acquired in double-shutter mode, and traditional camera timing was simulated here by staggering the raw data (i.e. using uniform and structured raw images from neighboring frame pairs instead of the same frame pair). The effective inter-frame delay for the traditional camera [Fig. 6(a)] was 106 ms, while the double-shutter camera [Fig. 6(b)] was 1 ms. Motion artifacts arising from the misregistration of raw images are easily identified as bright low-frequency blobs in the neighborhood of moving RBCs. The reduced inter-frame delay from the double-shutter timing manifestly attenuates these artifacts and provides crisper imaging. This demonstration is somewhat exaggerated as scientific camera readout times are typically faster than 100 ms. Nevertheless, it serves as an illustration of motion artifact reduction.

Fig. 6.

Comparison of HiLo videos acquired with traditional versus double-shutter camera timing. (a) HiLo with traditional timing, (b) HiLo with double-shutter timing, representing a 100-fold reduction in inter-frame delay over traditional camera timing. Motion artifacts arising from raw image misregistration are observed in (a) as bright low-frequency blobs in the neighborhood of moving RBCs. (Video 4, MPEG, 10.0 MB). [URL: http://dx.doi.org/10.1117/1.JBO.17.2.021105.4]Download video file ^{(9.8MB, mpg)}

4. Summary

We have demonstrated some of the performance properties of our improved HiLo endomicroscope. Our key improvements are (1) the introduction of image preprocessing to mitigate the effects of fiber core sparsity, heterogeneity and autofluorescence background; (2) faster speed achieved by the operation of our preprocessing and HiLo algorithms on a GPU, enabling us to attain a net frame rate of 9.5 Hz in real time; and (3) significantly enhanced immunity to motion artifacts with the use of a double-shutter camera. Our goal with these improvements is to further lay the groundwork in establishing HiLo endomicroscopy as a viable technique for clinical applications.

HiLo endomicroscopy is not without its limitations. The main limitation comes from the fact that we use structured illumination to identify out-of-focus background. The contrast imparted by this structured illumination must be large enough to be visible. That is, it must be greater than noise-induced contrast—in particular shot-noise-induced contrast introduced by the background itself. When the background is large, this can impose limitations on the maximum grid frequency, which, in turn, limits background rejection capacity and axial resolution. Nevertheless, as we have shown here, HiLo endomicroscopy provides significant background rejection even when imaging thick in vivo samples. This, along with its remarkable simplicity, speed, and robustness, make HiLo endomicroscopy an attractive imaging technique.

Finally, to further advertise HiLo imaging in general, we have made available our HiLo algorithm as an ImageJ plug-in. This may be found on our website http://biomicroscopy.bu.edu.