Speckle illumination holographic non-scanning fluorescence endoscopy

Abstract

Optical sectioning endoscopy such as confocal endoscopy offers capabilities to obtain three-dimensional (3D) information from various biological samples by discriminating between the desired in-focus signals and out-of-focus background. However, in general confocal images are formed through point-by-point scanning and the scanning time is proportional to the 3D space-bandwidth product. Recently, structured illumination endoscopy has been utilized for optically sectioned wide-field imaging, but it still needs axial scanning to acquire images from different depths of focal plane. Here, we report wide-field, multiplane, optical sectioning endoscopic imaging, incorporating 3D active speckle-based illumination and multiplexed volume holographic gratings, to simultaneously obtain images of fluorescently labeled tissue structures from different depths, without the need of scanning. We present the design, and implementation, as well as experimental data, demonstrating this endoscopic system's ability to obtain optically sectioned multiplane fluorescent images of tissue samples, with cellular level resolution in wide-field fashion, and no need for mechanical or optical axial scanning.(A) Schematic drawing of the SIHN endoscopy to simultaneously acquire multiplane images from different depths. (B) Uniform, and (C) SIHN illuminated images of standard fluorescence beads (25 μm in diameter) for the two axial planes. (D) Intensity profile on fluorescently labeled signal (ie, in-focus) and background (ie, out-of-focus) of microspheres.

Graphical Abstract

1 ∣. INTRODUCTION

Optical fluorescence endoscopy such as wide-field optical endoscopy has been developed for a variety of medical applications. Standard wide-field endoscopy provides wide field-of-view (FoV) and three-dimensional (3D) information of tissue samples; however, wide-field endoscopy has poor optical sectioning ability to reject out-of-focus background, which limits its medical use for imaging thick biological samples [1, 2]. Many recent improvements in optical fluorescence endoscopy have development of optical sectioning capabilities, particularly centered on increasing acquisition speed, and enhancing image quality of resolution and contrast [1, 3–12].

In confocal scanning endoscopy [4–7, 13], a pinhole, located at the conjugate of the illuminated point, is utilized to reject out-of-focus background light. Although image contrast is significantly enhanced by the combination of active focused illumination, but confocal endoscopic approach requires time-consuming process for scanning in both lateral and axial dimensions. Structured illumination endoscopy [5, 14] for optical sectioning capability, using computational reconstruction algorithms, aims to speed the process to acquire wide-field images at one depth of a sample. Some recent examples include HiLo illumination endoscopy [1, 12], which can be thought as more general than structured illumination endoscopic imaging and implemented with any type of nonuniform illumination, including a grid pattern. Although HiLo endoscopic imaging could be faster and more cost effective than confocal imaging techniques, HiLo endoscopy does not eliminate axial scanning operation, and z-stacks are still obtained through scanning the sample in the axial direction. Here, we experimentally demonstrate a multidepth, optical sectioning endoscopic system, incorporating nonuniform speckle illumination, and multiplexed volume hologram gratings (MVHGs). The proposed speckle holographic illumination non-scanning (SIHN) endoscopy has significant advantage over other interferometric endoscopic imaging [8, 15, 16], and earlier described systems in that SIHN does not require scanning to view multidepth fluorescence information within a sample, or stringent interferometric requirements of forming holograms during imaging. Optically sectioned endoscopic images of tissue samples with cellular level resolution from different depths are simultaneously acquired from different planes, without axial scanning, as is required in confocal or structured illumination microscopy. Unlike our earlier study [17] for a table-top microscopic system, the main contribution for the presented work is to develop speckle illumination based endoscopy. In contrast to the system in our previous work [18] using sinusoidal gird and Talbot effect techniques, in the SIHN approach, an arbitrary arrangement of longitudinal foci can be recoded and probed, and this is because SIHN endoscopic imaging does not rely on periodic diffractive elements to generate Talbot patterns. Hence, the SIHN system greatly simplifies 3D sectioning endoscopy and especially its applications to holographic fluorescence endoscopy. In addition, the SIHN system is capable of capturing multidepth resolved images at any excitation or emission wavelength.

2 ∣. METHOD

A schematic diagram of the SIHN endoscopy is shown in Figure 1. The SIHN system acquires a pair of multidepth fluorescent images, one with nonuniform speckle-based illumination by putting a diffuser in front of incident laser to generate the speckle pattern, and the other with uniform illumination. The speckle pattern generated from diffuser will be projected on the object, where the tissue sample located. The speckle pattern, projected on fluorescently labeled samples, has highest contrast in focal plane, and the contrast is decayed away from focal plane during imaging [17, 19], so that the speckle pattern has the sectioning modulation. In our work, an Argon (Ar⁺) laser operated at λ = 488 nm is used in Figure 1 and multiplane fluorescent imaging is detected by MVHGs and displays laterally on the CCD.

FIGURE 1.

Schematic of the SIHN system setup. In this diagram, axial layers 1 and 2 correspond to different focal planes of a sample, and the separation between two axial planes is ~100 μm. The transverse separation of images displayed on the CCD is accomplished using MVHGs positioned at the Fourier plane. Imaged planes at both depths (ie, depths 1 and 2 at top-left) are displayed onto a CCD, and separated at some lateral distance, which is specified by the focal length of the collector. The system was illuminated with a diffuser to induce 3D speckle patterns onto the sample plane at multiple depths

2.1 ∣. Endoscopic probe design

There are three major parts inside the endoscope, including MVHGs, miniature objective lens and optical relays. The miniature objective lens, consisting of three lenses providing a field of view of 1 mm with a NA 0.26, has been optimized for the visible wavelength to offer moderate contrast for high spatial frequency of line pairs as shown in the modulation transfer function (MTF) plot. The diameter of the objective lens is 8 mm. The second part is a Hopkins rod lens relay and a field lens, which are configured in a symmetry for reducing odd aberrations. The rod lens relay has been proved for a better throughput compared with a conventional lens [4], while the field lens helps to bend the light into a small refractive angle to prevent vignetting. The achromatic doublet relay system is composed of two achromatic doublets to reduce spherical aberration and field curvature. The functionality of the endoscopic probe acts as an infinite corrected objective lens to provide more flexibility to add additional optical components such as MVHGs, and dichroic mirrors. For image resolution measurements, the endoscopic probe resolves 4.4 μm line pair width (ie, minimum line width of 2.2 μm) in the UFAS resolution chart, as shown in Figure 2B. To compare with MTF plot in ZEMAX in Figure 2C, the yellow dash line in 288.1 lp/mm has a contrast 0.3, which is found to agree well with measurements.

FIGURE 2.

(A) 2D drawing of an endoscope probe. (B) Image of a USAF resolution chart. The minimum line width of 2.2 μm (dash line) is resolved. (C) the modulation transfer function (MTF) plot

2.2 ∣. Principle of multiplane volume holography

A volume hologram, consisting of MVHGs, is located in the Fourier plane of SIHN endoscopic system for multiplane fluorescent imaging. The MVHGs are formed, using shift-angular multiplexing [20, 21] inside a thick phenanthrenquinone-doped polymethyl methacrylate (PQ-PMMA) holographic material, in which interference patterns by mutual coherent beams are sequentially recorded. The design principle of the shift-angular multiplexed MVHGs can be illustrated using a K-sphere diagram [22, 23]. Schematic drawings in Figure 3 describe recording process and probe process during reconstruction respectively through K-sphere diagrams for a single volume hologram, consisting of MVHGs, in 3D space.

FIGURE 3.

(A) K-sphere diagram for recording process. The reference bream $\stackrel{\rightharpoonup }{k}{}_{ri}$ (i = 1,2 in this study) is a collimated wave while signal beam is composed of a spherical wave generated by moving lens L₁ in recording process. (B) K-sphere diagram under reconstruction condition. $\stackrel{\rightharpoonup }{k}{}_{p1}$ and $\stackrel{\rightharpoonup }{k}{}_{p2}$ are incident probe beams, from two objective planes of the sample through an objective lens, under the Bragg condition to diffract two respective collimated emission beams ($\stackrel{\rightharpoonup }{k}{}_{d1}$ and $\stackrel{\rightharpoonup }{k}{}_{d2}$), which laterally generate two foci through a collector lens onto a CCD in the image space. (C) Bragg wavelength degeneracy using a blue recording laser ($\mid \stackrel{\rightharpoonup }{k}{}_{p\lambda 1}\mid =\mid \stackrel{\rightharpoonup }{k}{}_{d\lambda 1}\mid ,{\lambda }_1=488\text{nm}$) and probing with green wavelength. ($\mid \stackrel{\rightharpoonup }{k}{}_{p\lambda 2}\mid =\mid \stackrel{\rightharpoonup }{k}{}_{d\lambda 2}\mid ,{\lambda }_2\cong 530\text{nm}$)

In the recording process, as showed in Figure 3A, let $\stackrel{\rightharpoonup }{k}{}_{si}$ and $\stackrel{\rightharpoonup }{k}{}_{ri}$ represent signal and reference wave-vectors respectively. The reference wave ($\stackrel{\rightharpoonup }{k}{}_{ri}$) is a collimated beam. The signal wave ($\stackrel{\rightharpoonup }{k}{}_{si}$) is controlled by moving the first lens (L₁) along the optical axis with some shifted distance (Δz) corresponding for each depth while lens L₂ stays fixed. MVHGs are recorded by interference patterns, which are formed by signal and reference wave-vectors. By changing the incident angle Δθ of the reference beam, different MVHGs can be recorded onto the same holographic pupil. The i-grating vector $\stackrel{\rightharpoonup }{K}{}_{gi}$ is generated by reference beam $\stackrel{\rightharpoonup }{k}{}_{ri}$ and signal beam $\stackrel{\rightharpoonup }{k}{}_{si}$. According to Bragg match constraint [17, 20], the i-th MVHG grating vector, formed by signal and reference wavefronts ($\stackrel{\rightharpoonup }{k}{}_{si}$ and $\stackrel{\rightharpoonup }{k}{}_{ri}$), can be expressed as

$$\stackrel{\rightharpoonup }{K}{}_{gi}=\stackrel{\rightharpoonup }{k}{}_{si}-\stackrel{\rightharpoonup }{k}{}_{ri},\text{where}i=1,2\dots \dots ,N.$$ (1)

$\mid \stackrel{\rightharpoonup }{k}{}_{si}\mid =\mid \stackrel{\rightharpoonup }{k}{}_{ri}\mid =2\pi \mathrm{n}∕{\lambda }_r$, n denotes the refractive index of the PQ-PMMA recording material, λ_r is the recording wavelength in free space, and N is the number of MVHGs (N = 2 in this study). Figure 3B shows the Bragg-matched K-sphere diagram for the reconstruction process. Two probe beams, which propagate along the optical axis (z) from different depths, are diffracted at diffractive angles (Δθ~1°) under Bragg-matched condition to corresponding MVHGs. Since in SIHN the transverse separation of image projected on the CCD is accomplished using MVHGs, it is worth mentioning that a changed diffraction angle (Δθ) between diffraction beams ($\stackrel{\rightharpoonup }{k}{}_{d1}$, $\stackrel{\rightharpoonup }{k}{}_{d2}$) is necessary such that an overlap of displayed imaged planes on the CCD can be avoided. In addition, the diffracted ($\stackrel{\rightharpoonup }{k}{}_{di}$) and probe ($\stackrel{\rightharpoonup }{k}{}_{pi}$) beams, satisfying Bragg-matched condition, can be expressed as:

$$\stackrel{\rightharpoonup }{K}{}_{gi}=\stackrel{\rightharpoonup }{k}{}_{di}-\stackrel{\rightharpoonup }{k}{}_{pi},$$ (2)

where $\mid \stackrel{\rightharpoonup }{k}{}_{di}\mid =\mid \stackrel{\rightharpoonup }{k}{}_{pi}\mid =2\pi \mathrm{n}∕{\lambda }_p,{\lambda }_p$, λ_p represents the free-space fluorescence emission wavelength. Because of the Bragg wavelength degeneracy, a MVHG, recorded at a wavelength of λ_r, can be reconstructed at a different probe wavelength of λ_p. Therefore, as illustrated in Figure 3C, a MVHG is utilized to image using a different wavelength (λ + dλ) at corresponding angle (θ + dθ), based on Bragg degeneracy between angle and wavelength, which is given as [22].

$$d\theta ∕d\lambda =K∕4\pi n\sin (\alpha -\theta ).$$ (3)

In Figure 3C, $\mid \stackrel{\rightharpoonup }{k}{}_{d\lambda 1}\mid =\mid \stackrel{\rightharpoonup }{k}{}_{p\lambda 1}\mid =\frac{2\pi n}{\lambda 1}$, and $\mid \stackrel{\rightharpoonup }{k}{}_{d\lambda 2}\mid =\mid \stackrel{\rightharpoonup }{k}{}_{p\lambda 2}\mid =\frac{2\pi n}{\lambda +d\lambda }$. With respect to the normal to a MVHG surface, α is the angle of grating vector, and θ represents the angle of the incident beam.

2.3 ∣. HiLo computational principle for optically sectioned VHE using speckle illumination

Because of previously described Bragg degeneracy in Eq. (3), under broad wavelength emission, viewing angle of SIHN endoscopy expands along degeneracy direction (ie, x-direction in this study) such that the system's FoV is increased. However, optical sectioning capability using a MVHG becomes poor due to broad fluorescence bandwidth. For example, in Figure 4A, fluorescently labeled blurred microspheres, selected by dashed yellow circular shapes, are out-of-focus, indicating poor optical sectioning under standard uniform illumination. In addition, Figure 4A shows speckles on the in-focus microspheres with obviously clear contrast, while the speckle contrast on the out-of-focus microspheres is barely observed. Thus, this inspires the use of speckle-based HiLo computational process in SIHN to reduce haze from out-of-focus background. The HiLo principle [1, 2, 12, 14] possesses optically sectioning ability by calculating the standard deviation of speckle pattern on the in-focus image, which is enhanced while the defocus information is significantly suppressed. The reconstructed resultant image, using typical speckle-based HiLo principle, is the combination of the high frequency in-focus content (ie, Hi image) extracted from uniformly illuminated image, and the low frequency in-focus content (ie, Lo image) obtained from combination of ⇀uniform and speckle illuminated images. In our approach, the HiLo principle has been modified for our proposed SIHN system. Let $I_u\left(\stackrel{\rightharpoonup }{r},{\mathrm{z}}_i\right)$ and $I_s\left(\stackrel{\rightharpoonup }{r},{\mathrm{z}}_i\right)$ represent the uniform and structured speckle illuminated images obtained from the SIHN at the axial planes z_i, respectively. The contrast map $\left(C_{\delta s}\left(\stackrel{\rightharpoonup }{r},{\mathrm{z}}_i\right)\right)$ of speckles at object, which is used as weighting function of focus image, is written as:

$$C_{\delta s}\left(\stackrel{\rightharpoonup }{r},{\mathrm{z}}_i\right)={\sigma }_{\Lambda }\left(I_s\left(\left(\stackrel{\rightharpoonup }{r},{\mathrm{z}}_i\right)\right)-I_u\left(\left(\stackrel{\rightharpoonup }{r},{\mathrm{z}}_i\right)\right)\right)∕{\langle I_u\left(\left(\stackrel{\rightharpoonup }{r},{\mathrm{z}}_i\right)\right)\rangle }_{\Lambda },$$ (4)

where ${\sigma }_{\Lambda }\left(\stackrel{.}{\mathrm{c}}\right)$ represents the standard deviation operator of variation between standard uniform illuminated images (I_u) and speckle illuminated images (I_s). ${\langle I_u\left(\stackrel{\rightharpoonup }{r},{\mathrm{z}}_i\right)\rangle }_{\Lambda }$ denotes the mean value of the image under uniform illumination, within a resolved sampling window (Λ) at the axial plane z_i. In HiLo process, by multiplying $I_u\left(\stackrel{\rightharpoonup }{r},{\mathrm{z}}_i\right)$ by $C_{\delta s}\left(\stackrel{.}{\mathrm{c}}\right)$, in-focus low frequency information is acquired after applying low-pass filter (LP_kc,zi), while focused high frequency content is obtained after applying high-pass filter (HP_kc,zi) with cut-off frequency kc. The optically sectioned SIHN images at the axial plane z_i can then be obtained by:

$$I_{\text{SIHN}}={\mathcal{F}}^{-1}\left\{\begin{array}{c}\hfill {\text{HP}}_{kc,{\mathrm{z}}_i}\left\{\mathcal{F}\left[I_u\left(\stackrel{\rightharpoonup }{r},{\mathrm{z}}_i\right)\right]\right\}\\ \hfill \\ \hfill +{\eta }_{{\mathrm{z}}_i}\times {\text{LP}}_{kc,{\mathrm{z}}_i}\left\{\mathcal{F}\left[C_{\delta s}\times I_u\left(\stackrel{\rightharpoonup }{r},{\mathrm{z}}_i\right)\right]\right\}\end{array}\right\}.$$ (5)

FIGURE 4.

(A) Structured illuminated images of 90μm fluorescence beads for the two axial planes. The red zoom-in window in layer 2 shows an in-focus bead with obviously clear speckles, while dashed circular region in yellow color shows out-of-focus beads without speckles. The scale bar at left lower corner indicates 100 μm. (B) SIHN images at the corresponding axial planes, acquired using modified HiLo pair-wise imaging algorithm. (C) Intensity profile at the signal of in-focus microspheres and background from out-of-focus beads along dashed line at depth 2. (D) Normalized axial contrast plot of the speckle image with wide-field illumination, which represents the point spread function (FWHM ~70 μm) in depth

3 ∣. EXPERIMENTAL RESULTS

The ability of the SIHN to observe a volumetric sample was first verified by imaging fluorescence labeled microspheres (90 μm, Polysciences, Warrington, Pennsylvania), within a 1 mm thick slab of agarose gel. The fluorescence microspheres were excited using an Ar⁺ laser source (Coherent Inc., Santa Clara, California), operated at λ = 488 nm, and a dichroic mirror (Q505lp, Chroma Technology Corp., Bellows Falls, Vermont) was used to reject stray light and noise from excitation while imaging. To generate randomly distributed speckles within a volumetric sample, an optical diffuser (Luminit, KCN1-25, Torrance, California) was utilized in the illumination. Two MVHGs, located in the Fourier plane within a volume hologram with average diffraction efficiency of ~40%, acted as depth-resolved filter to simultaneously observed speckle-defined depths. The depth separation between two speckle-defined depths, displayed onto a CCD (Hamamatsu ORCA4.0 sCMOS, Hamamatsu Japan), was ~100 μm. The SIHN had a miniature objective lens (NA = 0.26), as shown in Figure 2, and a Mitutoyo MPlanAPO10X objective was used as a collector lens. In our experiment, two-depth resolved images of fluorescence beads were shown in Figure 4A,B, using speckle illumination, and speckle-based HiLo computational principle, respectively. The sampling window Λ of a 13 × 13 (pixel²) matrix was selected for speckle contrast calculation and HiLo process. Figure 4C shows comparison of local contrast between in-focus (signal) beads and defocus background rejection, with intensity profiles, plotted along a dashed line within highlighted box on the second plane, between uniform illumination and speckle-based HiLo process. Figure 4C demonstrates that out-of-focus background was significantly suppressed by utilizing the complete SIHN principle, including standard uniform illumination and HiLo post-processing. In addition, optical sectioning capability of the SIHN approach was verified by measuring the speckle contrast between the maximum and minimum intensity alone the axial direction, conducted using a step size of 2 um, as Figure 4D shows, and the experimental measurement of full width at half maximum (FWHM) is 71.6 μm. Therefore, without lateral and axial scanning, the SIHN system shows solid evidence to provide fine optical sectioning for multidepth resolved imaging.

For further experimental demonstration of imaging tissue samples through SIHN system, in vitro villi of fluorescently labeled mouse intestine samples, stained with Alexa-488, was performed. Figure 5 shows the in vitro images of villi taken by our SIHN system, and two planes with separation of ~100 μm in depth were imaged simultaneously onto different lateral regions of the CCD. Figure 5A shows a villi image of two planes, simultaneously acquired in one shot using MVHGs under uniform illumination, with no speckle pattern. However, there is obviously visible haze because of the wide-field illumination and thick nature of the sample. Figure 5B shows the haze at both planes is effectively removed by utilizing the complete SIHN principle. Therefore, the SIHN system succeeds at suppressing out-of-focus background, and low contrast fine features are clearly observed, as shown in Figure 5C-F. Figure 5C,D shows the resultant images at depth 1 with relatively good signal to background ratio, while Figure 5E,F is at depth 2 at a location with still decent signal to background ratio. Figure 5G,H compares the signal-to-background using standard uniform illumination and SIHN illumination at both planes, by plotting intensity cross sections.

FIGURE 5.

(A) in vitro images of mice intestine sample taken from two axially separated depths, under standard uniform illumination using the same endoscopic system of Figure 1. The scale bar at left lower corner indicates 100 um. (B) Corresponding SIHN images using our modified HiLo algorithm. (C, D) Zoom-in to the solid-box region of (A, B), respectively. (E, F) Zoom-in to the dashed-box region of (A, B), respectively. (G, H) Intensity cross sections along the dashed line shown in corresponding depth 1 and depth 2 of the uniformly illuminated and SIHN illuminated images (C, D) and (E, F), respectively

4 ∣. CONCLUSIONS

Wide-field, multiplane, optical sectioning endoscopic imaging, incorporating MVHGs and 3D speckle-based illumination to simultaneously capture images of tissue structures from different depths without scanning and mechanical moving parts, has been developed. The proposed SINH system is simple, and robust, as well as faster than alternative techniques since SINH only requires two shots to simultaneously acquire multiple depths, with no need for lateral and axial scanning. In this study, although optical sectioning images of tissue samples from two different planes have been demonstrated, the SINH imaging is able to be further extended to take more depths with more MVHGs [20–22]. In addition, the capability for rejection of out-of-focus background can be improved using denser speckle patterns, and HiLo algorithm process with additional wavelet filtering [2, 24] can also enhance the sectioning ability. It is worth mentioning that SINH does not rely on periodic structure to generate Talbot pattern [18, 23], and thus there is no constraint to select axial planes of interest for imaging. To enable realtime 3D video endoscopy, it is possible to use a digital micro-mirror device to switch between structured and uniform illumination, and reconstruct entire images, adapting a faster computing device with more MVHGs, in a fully automated fashion.