FIMic: design for ultimate 3D-integral microscopy of in-vivo biological samples

Abstract

In this work, Fourier integral microscope (FIMic), an ultimate design of 3D-integral microscopy, is presented. By placing a multiplexing microlens array at the aperture stop of the microscope objective of the host microscope, FIMic shows extended depth of field and enhanced lateral resolution in comparison with regular integral microscopy. As FIMic directly produces a set of orthographic views of the 3D-micrometer-sized sample, it is suitable for real-time imaging. Following regular integral-imaging reconstruction algorithms, a 2.75-fold enhanced depth of field and $\sqrt{2}$-time better spatial resolution in comparison with conventional integral microscopy is reported. Our claims are supported by theoretical analysis and experimental images of a resolution test target, cotton fibers, and in-vivo 3D-imaging of biological specimens.

1. Introduction

In 1908 G. Lippmann proposed the integral photography (IP) based on the use of a microlens array (MLA) [1]. This architecture permitted to record a collection of elemental images (EIs) of 3D scenes, each with different perspective information. This pioneering design was done with the aim of displaying auto-stereoscopic 3D images through an IP monitor. Today, this aim is still a hot topic of research, and many innovative proposals have been presented in the recent years [2–5]. The plenoptic camera was proposed for the capture of multi-perspective 2D images of 3D far objects. This camera is basically the result of inserting a MLA at the image plane of a conventional camera, and displacing the sensor to the MLA focal plane [6–10]. The camera was designed for capturing, after a single shot, the spatial-angular information of 3D far scenes. This information is used to compute orthographic views and 3D depth maps.

The fast development of opto-electronic technology, the increasing capacity and speed of software tools, and the improvement of the quality of manufactured microlenses, has led the plenoptic technology to be highly profitable for many innovative applications. Among them, some biomedical applications are remarkable, like the use of plenoptic technology in otoscopy [11], ophthalmology [12], endoscopy [13-14] or for deep-tissue inspection [15-16]. Plenoptic technology has been proposed also for wavefront sensing [17], 3D imaging using long-wave infrared light [18], or head-mounted display technology [19].

A natural application of the IP concept is therefore 3D microscopy. First, Jang and Javidi, recognized the utility of IP for displaying 3D microscopic images [20]. But it is just after the Stanford group’s contributions that integral microscopy (IMic), also named as plenoptic or lightfield microscopy, was considered as an interesting alternative technique for real 3D imaging of micrometer-sized objects [21-22]. Therefore, the most wanted feature that IMic offers to microscopists is the possibility of observing the samples, almost in real time, from many different perspectives.

The above-mentioned possibilities for IMic are unfortunately hinder by its poor spatial resolution and limited depth of field (DOF). Along the past few years many techniques have been proposed for overcoming these flaws, some based on time or space multiplexing [23–25] some other based on computational interpolation or deconvolution [26-27]. An additional approach makes use of a two-stage relay for enlarging the aperture plane of the microscope to fit the size of the imaging lens array [28]. The latter is composed by 25 CCTV lenses that are placed close to the enlarged aperture plane to record view images, which are thereafter processed. However, all these mainly computationally driven solutions have an associated trade-off of enormous data processing concealing, for instance, real time applications.

As a proposal to overcome those limitations of 3D Integral microscopy, our group has proposed recently Fourier Integral Microscopy (FIMic) [25]. In that proof-of-concept proposal a regular photographic camera scanned manually a conjugate plane of the aperture stop. The time-multiplexed recorded view-images provided the 3D information of the sample. This concept has been used very recently for proposing a new instrument for the rapid inspection of neural activities [29]. However, that instrument does not take full profit from FIMic concept as the attained resolution and the DOF are far from being optimized.

In this work the FIMic concept is boosted to produce a totally optically driven solution to the flaws of IMic, and attain the best combination of resolution and DOF. The main outcomes achieved here, as compared with the previous reports of FIMic, are the following. First, theoretical formulae that relate the lateral resolution and the DOF with experimental parameters are reported. Second, based on those formulae ad hoc configurations for optimized resolution and DOF can be easily designed. Third, now the MLA is placed at the aperture stop (AS) of the microscope objective (MO) to capture directly the plenoptic far field in a single-shot mode. Fourth, the new realization turns FIMic into a very compact instrument. In summary, the new approach reported here minimizes the time and data consumption, enlarges significantly the DOF, and optimizes the resolving power; allowing the capturing, in real time with a compact instrument, of 3D images of microscopic live specimen.

2. Parameters design

The core of the innovative solution to the main drawbacks of IMic is to place the MLA at the AS of the MO. Since in this configuration a Fourier or far-field elemental image (EI) is recorded by each microlens, we have coined this architecture as FIMic [25]. In this arrangement, the plenoptic field is directly recorded by means of the sampling that the MLA provides of the spatio-angular information at the far field located at the AS. In Fig. 1 a schematic layout of FIMic is shown. As for regular commercial MOs the AS may not be mechanically accessible, a relay system is necessary to conjugate the MLA with the AS. In our case, the relay is composed by two converging lenses, RL1 and RL2, with focal lengths $f_1$ and $f_2$, arranged to form a telecentric relay setup that conjugates $(x_{\text{F}},y_{\text{F}})$ and $(x_{\text{F}}^{\prime },y_{\text{F}}^{\prime })$ planes. The digital sensor, a CCD camera for instance, is set at the rear focal plane of the MLA where magnified images of the field stop (FS) are recorded. The FS, placed at the common focal plane for RL1 and RL2, is chosen such that the EIs are tangent at the recording plane to optimize the use of the digital sensor. According to the proposed setup, the size of the FS (${\varphi }_{\text{FS}}$) has to be ${\varphi }_{\text{FS}}={\varphi }_{\text{EI}} f_2/f_{\text{L}}$, where ${\varphi }_{\text{FS}}$ equals the size of each elemental image. Equivalently, as the FS is seen in the object space, it determines the field of view (FOV) of the FIMic, such that $FOV={\varphi }_{\text{FS}} f_{\text{MO}}/f_1$. The optimum use of the available area of the digital sensor is achieved when ${\varphi }_{\text{EI}}$ equals the pitch, $p$, of the MLA, hence the FOV of FIMic is given by

$$FOV=2\frac{f_{\mathrm{MO}}{f}_2}{f_1}N{A}_{\mathrm{MLA}},$$ (1)

where $N{A}_{\text{MLA}}=p/2f_{\text{L}}$ is the numerical aperture of the MLA.

Fig. 1.

Schematic layout of Fourier Integral Microscopy (FIMic). A collection of orthographic views (or EIs) is directly obtained. ROP is the reference object plane; RL1 and RL2 are the relay lenses, and CCD is the digital sensor.

The underlying concept in FIMic is the spatial-angular sampling at the AS of the MO of the host microscope. The latter is the optical microscope in which the MLA is inserted. As the performance of the host microscope is mainly determined by the diameter of AS (${\varphi }_{\text{AS}}$), any intervention in AS will therefore modify the overall performance of the microscope. Such sampling, defined here along one space direction but extendable to the both orthogonal, of AS can be therefore quantified in terms of the number of microlenses, $N={\varphi }_{\text{AS}}/\overline{p}$ (where $\overline{p}=p{f}_1/f_2$), that are fitted within. In other words, N is the number of samples that are taken along the diameter of the AS, i.e FIMic provides N elemental images. Hence FIMic can be understood as a new microscope with an effective aperture stop ${\varphi }_{\text{AS}}/N$. As for the trivial case N = 1, FIMic provides only a single EI with exactly the same features of the host microscope. As N grows, the number of samples increases raising the number of EIs at the cost of reducing the size of the effective AS. Therefore, this shrinkage brings along a reduction of the effective numerical aperture up to $N{A}_{\text{H}}/N$, being $N{A}_{\text{H}}$ the numerical aperture of the host microscope. This fact implies a reduction of the spatial resolution and an increase of the depth of field (DOF). This feature of FIMic invites to consider the existing trade-off between the DOF and the spatial resolution in terms of the number of EIs.

With the above ideas in mind and having set up the FIMic for optimized use of the digital sensor, one can consider that the complete width of the sensor is covered by the N elemental images. This means that each EI is recorded by T/N pixels with T being the total number of pixels of the sensor. The physical width of the sensor Tδ, with δ the pixel size, must match the size of the imaged AS, namely $T\delta ={\Phi }_{\mathrm{AS}}{f}_2/f_1$, hence the pixel size of the digital sensor must fulfil the condition

$$\delta =\frac{N}{T}p\frac{f_2}{f_1}$$ (2)

for the optimized use of the digital sensor.

The spatial resolution of FIMic is twofold determined. Firstly, according to wave-optics theory, two point objects separated a distance $\rho$ have to be resolved by each of the EIs that sample the AS, namely $\rho \ge N\lambda /2N{A}_{\text{H}}$, with $N{A}_{\text{H}}={\varphi }_{\text{AS}}/2f_{\text{MO}}$ the numerical aperture of the host MO. Secondly, according to Nyquist, the distance between the two points should be large enough to be recorded by different pixels leaving at least an empty pixel in between. Therefore, $\rho \ge 2\delta f_2{f}_{\text{MO}}/f_{\text{L}}{f}_1$. The combination of these two factors leads to an overall resolution limit for FIMic given by

$${\rho }_{\mathrm{EI}}\ge \max \left\{N\frac{\lambda }{2N{A}_{\mathrm{H}}},2\delta \frac{f_2{f}_{\mathrm{MO}}}{f_{\mathrm{L}}{f}_1}\right\}.$$ (3)

If the pixel size is selected smartly, so that the two terms have the same value, we obtain

$${\rho }_{\mathrm{EI}}\ge \frac{\lambda }{2N{A}_{\mathrm{H}}}N.$$ (4)

As for the DOF, we simply need to adapt the classical formulae to the effective NA [30] of the FIMic, that is

$$DO{F}_{\mathrm{EI}}=\lambda \frac{N^2}{N{A}_{\mathrm{H}}^2}+\delta \frac{N}{N{A}_{\mathrm{H}}}\frac{f_2{f}_{\mathrm{MO}}}{f_{\mathrm{L}}{f}_1}.$$ (5)

Assuming now the same value for the pixel size as above,

$$DO{F}_{\mathrm{EI}}=\frac{5}{4}\frac{\lambda }{N{A}_{\mathrm{H}}^2}{N}^2.$$ (6)

Before going to the next section, and aiming to provide the analytical tools for the comparison between IMic and FIMic we present the formulae for the resolution and the DOF in IMic. For the understanding of these equations, it is necessary to take into account that in IMic the resolution and the DOF are determined by the MLA pitch, which plays here the same role as the pixel size in FIMic. The lateral resolution of views computed from IMic microimages is then given by [21]

$${\rho }_{\mathrm{View}}\ge \left\{\frac{\lambda }{2N{A}_{\mathrm{H}}},2\frac{p}{M_{\mathrm{H}}}\right\}=\mu \frac{\lambda }{N{A}_{\mathrm{H}}}.$$ (7)

In this equation $M_{\mathrm{H}}$ is the magnification of the host microscope and $\mu$ is a parameter defined here as the quotient between $p$ and the wave-optics resolution limit as evaluated in the image plane. The DOF of computed views is given by

$$DO{F}_{\mathrm{View}}=\frac{\lambda }{N{A}_{\mathrm{H}}^2}+\frac{\mu p}{M_{\mathrm{H}}N{A}_{\mathrm{H}}}=\left(1+\frac{{\mu }^2}{2}\right)\frac{\lambda }{N{A}_{\mathrm{H}}^2}.$$ (8)

From Eqs. (4), (6), (7), and (8) we obtain

$${\rho }_{\mathrm{EI}}=\frac{N}{2\mu }{\rho }_{\mathrm{View}} \text{and} DO{F}_{\mathrm{EI}}=\frac{5N^2}{4+2{\mu }^2}DO{F}_{\mathrm{View}}.$$ (9)

From these equations it is apparent that given an IMic, it is always possible to design FIMic with the same resolution but much better DOF, or with the same DOF but much better resolution, or even with simultaneous improvement of resolution and DOF. This superiority of FIMic is obtained at the prize of obtaining a lower number of orthographic views. Note however that this lost is not critical, provided that the number of EIs captured with FIMic is enough for the calculation of refocused images or depth maps.

3. Comparative performance analysis

This section is devoted to contrast the more relevant features of IMic and FIMic. To this end, a conventional IMic with in-focus lateral resolution of 6.2 µm and 80 µm DOF was built initially. Then, to compare the performance of IMic to FIMic, two experimental set-ups of the FIMic were realized. The superior performance of the FIMic over the IMic has been validated under two conditions to ensure fair comparison (i) DOF performance assessment with the FIMic built to have the same spatial resolution as the IMic, (ii) Spatial resolution performance assessment with the FIMic designed to have the same DOF as the IMic. Further comments on the number of views, refocusing capabilities and data processing are also presented.

3.1 Both microscopes operating at similar spatial resolution to test their DOF

Using an MLA composed of microlenses with $f_{\text{L}}=6.5 mm$, $p=1.0 mm$, and $N{A}_{\text{MLA}}=0.077$ (APH-Q-P1000-R2.95 manufactured by AMUS), and an infinity corrected MO ($f_{\text{MO}}=9.0 mm$ and NA = 0.4) a FIMic microscope was built. Since the AS was not mechanically accessible, we determined by optical means the position of the exit pupil and also its diameter, ${\varphi }_{\text{AS}}=7.1 mm$. A proper choice of two achromatic doublet lenses with $f_1=50 mm$ and $f_2=50 mm$ were used as relay system to specifically match the MLA and the AS such that the resolution equals that of the IMic built in our lab. This configuration fits $N=7.1$ microlenses in the AS. As digital sensor we have utilized a CMOS camera (EO-5012c ½”) with 2560x1920 square pixels of $\delta =2.2 \mu m$ in side. According to Eq. (3) and (5), this setup was expected to give a lateral resolution limit of $6.1 \mu m$ and a DOF of $240 \mu m$.

In order to evaluate the DOF we used as the object an USAF resolution chart, which was placed first at the reference object plane (ROP) (see Fig. 1) and later displaced axially in regular steps up to $Z=110 \mu m$. For all these axial positions of the chart we captured directly the views with the FIMic and the microimages with the IMic. In the latter case, the views were computed from the microimages. Next, in Fig. 2 we show the central views. Note that we show only the view for distances moving away in the $Z>0$ direction, but it is seen the same for the negative direction. For both microscopes the smallest element that is correctly resolved in the ROP is the element 3 of group 7, which corresponds to a resolution limit of 6.20 µm. As assuming the DOF as twice the distance from the ROP to that plane where the resolution has decreased by a factor $1/\sqrt{2}$, the DOF extends up to where the element 6 of group 6, 8.76 µm, is resolved. From the views shown in Fig. 2 we find a DOF for IMic of 80 µm whereas it is of 220 µm for FIMic. An improvement of 2.75-times in the DOF is achieved, showing the superior performance of FIMic in terms of the DOF as both microscopes are setup for having the overall same spatial resolution.

Fig. 2.

Central view provided by the two microscopes when operating both at resolution limit of about $6.20 \mu m$. The label Z indicates the distance from the ROP. Providing both microscopes the same resolution at the ROP, the DOF extends up to $Z=+110 \mu m$ for the FIMic, but only up to $Z=+40 \mu m$ for the IMic.

3.2 Both microscopes operating at similar DOF to compare their spatial resolution

In order to implement a FIMic with the same DOF as our regular integral IMic, we have used the same MLA as in section 3.1, but used new MO ($f_{\text{MO}}=10.0 mm$, $NA=0.5$, and ${\varphi }_{\text{AS}}=10.0 mm$) an relay ($f_1=200 mm$ and $f_2=100 mm$). This setup renders to a FIMic with N = 5.0 which provides an expected resolution for the central view of 3.4 µm and the sought DOF of 77 µm. The central views for both microscopes are shown in Fig. 3.

Fig. 3.

Central view provided by both microscopes when operating with the same DOF. The FIMic provides better resolution at the edge of its DOF than the IMic at its ROP.

From Fig. 3 it is apparent that along the complete DOF the FIMic provides a spatial resolution of the order 3.9 µm, which is superior in comparison with the 6.2 µm provided by the IMic. This superior performance of FIMic can be foreseen from the fact that for achieving the DOF provided by our IMic, the FIMic has to be set up with just N = 5.0¸ what renders to a reduction of the resolution of the host microscope in just the same figure. The reader should be also aware that FIMic has quite better spatial lateral resolution at the edges of its DOF than IMic at its ROP. This feature can be added to the wish list the boosted FIMic is offering at the time of doing 3D integral microscopy.

3.3 Elemental images

The hallmark in integral microscopy is with no doubt its possibility of recording views of 3D samples to produce a-posteriori refocusing and/or 3D-rendering. In this sense, the direct recording of elemental images by FIMic is a notable difference between the IMic and FIMic.

In the case of IMic, the elemental images are computed from the recorded microimages. The n^th elemental image is obtained by composing, in an orderly manner, the n^th pixels of each microimage. Consequently, from IMic one can compute as many elemental images as pixels form each microimage. The number of the pixels in each microimage is determined by the quotient T/p. In Fig. 4, panel (a) there is a typical IMic recording of an USAF chart (see the tangent microimages in the zoomed-in area). In panel (b), the set of computed EIs is shown.

Fig. 4.

Microimages and EIs for IMic. Panel (a) shows the microimages recorded directly by the digital sensor. Panel (b) is the complete set of EIs computed from the microimages. In this experiment each micro image was formed by 17x17 pixels, hence the same number of EIs is computed. Zoomed-in areas are magnified by factor of four.

A notable feature of FIMic is its capacity of recording directly the EIs. This simplifies the use of the microscope, for instance in real-time applications as shown some paragraphs below. Additionally to real time applications, the direct recording of the elemental images allows the user to make decisions on the achieved resolution, the focusing of the recorded images, and the shot scene. In Fig. 5 is shown a direct recording of FIMic corresponding to N = 3.

Fig. 5.

Elemental images directly recorded in FIMic. The available area of the digital sensor is utilized for the recording of the $N=\delta T{f}_1/p{f}_2$ elemental images.

3.4 Refocusing capabilities and data processing

One of the most important, thrilling and desired capabilities of integral microscopy is its ability of making a-posteriori refocusing through the 3D scene. Once the EIs are captured/calculated, the major difference between IMic and FIMic, in terms of the refocusing capabilities, lays on number of planes (and the distance between them) in which the scene can be refocused. In the case of applying, for instance, the refocusing algorithm of pixel shifting and summing [31] in FIMic, the refocusing distances from the ROP are given by

$$Z_{\mathrm{R}}=n\frac{f_{\mathrm{MO}}^2}{f_{\mathrm{L}}}{\left(\frac{f_2}{f_1}\right)}^2\frac{\delta }{p},$$ (10)

with n being the (positive or negative) number of pixels that the EIs are shifted before summed. Indeed, on considering that one pixel is the smallest amount to be shifted, the step between the refocused images is $\text{Δ}{Z}_{\text{R}}=\left(f_{\text{MO}}^2{f}_2^2\delta \right)/\left(f_{\text{L}}{f}_1^{2}p\right)$. Because in FIMic the EIs have regularly a larger number of pixels than in IMic, the step between the refocusing planes is much smaller, and therefore the number of refocused planes is much bigger for a given DOF. This feature provides the FIMic with the capacity of refocusing with finer detail than IMic.

The refocusing step, $\text{Δ}{Z}_{\text{R}}$, could be made finer in IMic by upsizing, computationally, the elemental images. This strategy allows non-integer numbers of shifted pixels, but has two essential drawbacks. One is that, due to the fact that the upsizing does not add any additional information to the EIs, there is a limit beyond which raising the number of refocused planes does not produce differentiated refocused images. The other drawback is that an extreme upsizing can lead to surpass the computational capabilities of the computing system.

Another point that should be stressed is that in IMic the resolution of refocused images is not homogeneous [32]. In short, there are planes with worse resolution (the ones corresponding to an integer number of shifted pixels), and planes with better resolution (non-integer number of shifted pixels). However, as in FIMic any EI is composed by a large number of pixels, it has the desirable feature on maintaining the same spatial resolution for all the refocused images along the DOF.

In Fig. 6, some of the above-mentioned features of the refocusing capabilities of FIMic v.s. IMic are illustrated. This figure shows cotton fibers imaged with both microscopes operating at same DOF. In top row, refocused images for IMic are presented. The images for $Z_{\text{R}}=0 \mu m$ is calculated directly from the EIs, with no need of upsizing. This is a plane corresponding to an integer value of $n$, and therefore with worse resolution (see pixelation in the zoomed-in area). However, the computation of the images for $Z_{\text{R}}=11 \mu m$ and for $Z_{\text{R}}=21 \mu m$ was possible only after upsizing the EIs by factor six. Despite of the important increase of the data to be processed, there is not too much information that can be retrieved from the additional planes refocused after the upsizing of the EIs.

Fig. 6.

Refocusing capabilities FIMic v.s. IMic. FIMic provides better resolution along the entire DOF.

In the bottom row the refocused images for FIMic are shown. Here the directly-recorded EIs have 454 pixels, hence it is possible to directly refocus 454 planes. Among such number of planes we have limited to show only those for the same reconstruction distances utilized for IMic. In a clear difference with the refocused images for IMic, those for FIMic show greater detail for the different planes than in IMic, that is derived from the much better spatial resolution that FIMic has when it operates at the same DOF than IMic. Furthermore, the invariant spatial resolution of FIMic can be observed for the different refocused planes, which contrast with the strong changes of the spatial resolution of IMic as for the different focused planes.

4. In-vivo imaging of a biological sample

Beyond the experiments to compare the performance of FIMic with a regular IMic, we present further examples of use of FIMic to show some of the possible applications of this ultimate design of integral microscopy as for imaging in-vivo biological specimens.

The simplicity and robustness of FIMic design carries along no sample preparation for in-vivo imaging. Algae were collected from seashore and preserved in a regular container. We poured some few algae with sea water over a glass slide and then both were covered with a regular cover slide. The sample included not only the algae, but also a marine nematode swimming through them. The sample was placed in the sample holder of the FIMic used in Section 3.2. With this setup we recorded a $45 s$ video (4 fps) comprising seven different perspectives of the live sample. The complete video, but speeded up to 12 fps, is shown in Visualization 1 ^{(26.3MB, avi)} . A single frame of it is shown in Fig. 7. It is remarkable that the FIMic microscope holds good enough resolution to image internal details of the nematode

Fig. 7.

Frame extracted from Visualization 1 ^{(26.3MB, avi)} , a video of a marine nematode swimming through algae. Panel (a) shows the directly recorded EIs. In panel (b) there is a 3-times zoomed-in of the central EI. The red arrow points the nematode.

Using as frames the static EIs shown Fig. 7(a), we composed a video, Visualization 2 ^{(50.5MB, avi)} , to see the perspectives rendered by the FIMic. Additionally to the regular movement of the specimen, the 3D-image provided by the stack of perspectives would definitely add valuable information for those trained in the study of this particular organism.

The final feature of FIMic we want to show is its capability of a-posteriori refocusing from the directly recorded EIs. From the EIs of the frame 315 of Visualization 1 ^{(26.3MB, avi)} , we have computed 40 refocusing planes. Visualization 3 ^{(23MB, avi)} shows the different planes where different sections of the scene are clearly refocused. In this video the refocusing distance $Z_{\text{R}}$ is measured in μm.

5. Summary

We have reported on Fourier Integral Imaging Microscopy (FIMic), a single-shot, single-camera, 3D integral microscope that allows real-time imaging of the studied sample. FIMic exceeds 2.75-time the depth of field and $\sqrt{2}$-time the spatial resolution of regular integral microscopes (IMic). The enhanced performance of the proposed FIMic microscope relies on multiplexing the spatial-angular information located at the aperture stop of the microscope objective. By placing the microlenses array at the said stop, FIMic records a set of orthoscopic views, whose reconstructed images of the 3D-micrometer-sized sample exhibit enhanced depth of field and better spatial resolution than IMic. The performance of these two integral microscopes have been contrasted to support the reported claims.

The enhanced performance of FIMic, the simplicity of 3D image reconstruction along with its more compact and solid setup, suit FIMic as a very competitive tool for the production of real-time imaging of 3D samples of the micrometer-sized world. Cotton fiber and in-vivo marine nematode have been imaged with FIMic to show real world applications of its enhanced performance on 3D microscopy.

Funding

Spanish Ministry of the Economy and Competitiveness (DPI2015-66458-C2-1R); GVA, Spain (PROMETEOII/2014/072).

Disclosures

The authors declare that there are no conflicts of interest related to this article.