Multimodal image reconstruction from tomographic diffraction microscopy data

Summary

Tomographic diffraction microscopy (TDM) is a tool of choice for high‐resolution, marker‐less 3D imaging of biological samples. Based on a generalization of digital holographic microscopy with full control of the sample's illumination, TDM measures, from many illumination directions, the diffracted fields in both phase and amplitude. Photon budget associated to TDM imaging is low. Therefore, TDM is not limited by phototoxicity issues. The recorded information makes it possible to reconstruct 3D refractive index distribution (with both refraction and absorption contributions) of the object under scrutiny, without any staining. In this contribution, we show an alternate use of this information. A tutorial for multimodal image reconstruction is proposed. Both intensity contrasts and phase contrasts are proposed, from the image formation model to the final reconstruction with both 2D and 3D rendering, turning TDM into a kind of ‘universal’ digital microscope.

1. INTRODUCTION

Optical microscopy is essential in bio‐imaging. Fluorescence microscopy has become the reference, thanks to its unique chemical, therefore functional, specificity and its unsurpassed resolution, allowed by the development of fluorescence nanoscopy techniques. ¹ , ² , ³ , ⁴ However, fluorescence can also present limitations like photobleaching of fluorophores, which handicaps long‐term measurements and can induce phototoxicity, ⁵ or difficulties in labelling in some cases, or because of legal requirements, in particular in biology or in food industry, when one cannot label samples or use genetically modified organisms. Also, markers can sometimes interfere with the measurements. ⁶ , ⁷

Interferometric microscopy techniques can be considered to tackle this issue. These methods allow for quantitative phase imaging (QPI) in biomedicine. ⁸ Implementations based on short coherence interferometry, ⁹ Fourier ptychography, ¹⁰ and digital holographic microscopy (DHM) ¹¹ have been demonstrated. In the remaining, we will focus on DHM implementations of QPI. DHM, in association with light‐backpropagation algorithms, ¹² , ¹³ makes it possible for pseudo‐3D volumetric reconstruction over a large depth of field. ¹⁴ However, resolution along the light propagation axis is lost, inhibiting DHM to be considered as a full 3D imaging technique. ¹⁵ , ¹⁶

Full 3D reconstruction of biological samples have been demonstrated with tomographic diffraction microscopy (TDM). ¹⁵ , ¹⁶ , ¹⁷ , ¹⁸ , ¹⁹ TDM is a generalization of DHM experiments with a full control of the investigated object illumination angle. This angle can be modified by rotating the object within the illumination beam, ²⁰ sweeping the illumination beam on the object ²¹ or combining both approaches. ²² , ²³ , ²⁴ , ²⁵ From these acquisitions, it is possible to reconstruct, in 3D, both the real (refraction) and imaginary part (absorption) of the imaged sample's refractive index, ²⁶ , ²⁷ , ²⁸ with an improved lateral resolution, compared to conventional microscopes, ¹⁶ even down to the 100 nm resolution. ²⁹ Three‐dimensional isotropic resolution imaging has also been recently demonstrated. ²³ , ²⁵ The possibility of extracting 3D information about the refractive index of the imaged sample is an appealing feature of TDM. However, selectivity of the content is lost, and structural details can be missed. Therefore, TDM images can be difficult to interpret compared to images extracted with more common microscope modalities.

In this work, we propose to take benefit of the capabilities of TDM to accurately measure the 3D optical field coming from an object, to reconstruct images that would have been obtained with other microscopic imaging modalities. This will allow to take benefits of the TDM capabilities, together with the content discrimination brought by other techniques. Starting from TDM acquisitions, and using the image formation models of classical optical microscopes, we are able to simulate the behaviour of ‘routine’ microscopes (e.g. bright‐field or dark‐field microscopes), phase‐contrast improved microscopes (e.g. phase contrast or differential interference contrast [DIC] microscopes), as well as some less common imaging modalities (e.g. oblique illumination, or Rheinberg illumination microscopes), leading to a kind of ‘universal digital microscope’ from TDM acquisitions.

2. TDM PRINCIPLES

QPI techniques rely on the use of interferometry to extract both amplitude and phase of the field diffracted by the imaged object. As far as DHM is concerned, this extraction is either performed by spatial ³⁰ or temporal demodulation. ³¹ , ³² TDM is a generalized version of DHM with a full control of the object illumination angle. ¹⁷ , ³³ A sketch of the transmission TDM we developed is proposed in Figure 1. Light, emerging from a He–Ne laser ($\lambda =632.8$ nm), is split into a reference beam (depicted in red in Figure 1) and an object beam (in blue in Figure 1). Both paths are brought to interfere on the CMOS (Complementary Metal Oxyde Semi‐conductor) sensor within an off‐axis Mach–Zenhder configuration. The main difference between DHM and TDM consists in the use of a Tip/Tilt mirror (Newport™ FSM 300 fast steering mirror) for condenser back‐focal plane scanning. Therefore, a plane wave with a controlled angle illuminates the sample. From these acquired holograms, 3D information can be reconstructed using a synthetic aperture process. Considering the first Born approximation, the object wavevectors ${\mathbf{k}}_{\mathrm{o}}$ can be reconstructed using the elasticity condition

$${\mathbf{k}}_{\mathrm{o}}={\mathbf{k}}_{\mathrm{d}}-{\mathbf{k}}_{\mathrm{i}},$$ (1)

with ${\mathbf{k}}_{\mathrm{i}}$ the illumination wavevector (associated with the illumination angle) and ${\mathbf{k}}_{\mathrm{d}}$ the associated set of diffracted wavevectors for each hologram. ³⁴ The ${\mathbf{k}}_{\mathrm{d}}$ wavevectors map the Ewald sphere. ²⁶ However, as diffracted light is collected in transmission with a limited numerical aperture (NA), the measured ${\mathbf{k}}_{\mathrm{d}}$ vector set maps the Mc Cutchen pupil, which can be viewed as a cap of sphere, whose cord is limited by the collection objective NA. ³⁵ Construction of the 3D object potential ${\mathbf{k}}_{\mathrm{o}}$ is depicted Figure 1B. The diffracted wavectors ${\mathbf{k}}_{\mathrm{d}}$ maps the Ewald sphere (blue dashed line Figure 1). The Mc Cutchen pupil, denoted ${\mathbf{k}}_{\mathrm{d}}^{\mathrm{NA}}$ is shown in red. From Equation (1), information is reallocated by subtracting the illumination vector contribution. The case of perpendicular illumination, along the optical axis, corresponding to classical DHM, is illustrated in Figure 1(i). Case of illumination at maximal NA is proposed in Figure 1(ii,iii). Combining tens to hundreds of holograms makes it possible to build the full 3D object potential as can be seen in Figure 1(iv) in transmission ³⁶ (similar constructions holds for reflection or 4Pi TDM ¹⁷ , ³⁷ ). Taking the 3D inverse Fourier transform of this 3D object potential, one can reconstruct the 3D complex refractive index distribution of the investigated sample.

FIGURE 1.

(A) Sketch of a typical DHM‐based TDM configuration. Red beam is associated with the reference wavefront, blue beam is the illumination wavefront and green beam is the wavefront diffracted by the investigated sample. ${\mathrm{TL}}_1$ and ${\mathrm{TL}}_2$, respectively, denote the condenser and microscope objective tube lenses. The sampling doublet made of ${\mathrm{SL}}_1$ and ${\mathrm{SL}}_2$ lenses ensures adequate sampling of the interferograms. FD and AD are the Field Diaphragm and the Aperture Diaphragm, respectively. (B) Building of the 3D object potential ${\mathbf{k}}_{\mathrm{o}}$ from sequential hologram acquisitions (illustrated in 2D for convenience). Hologram spectrum registering according to Equation (1) for perpendicular illumination (i), at collection objective maximum NA (ii,iii), full 3D object potential (iv)

An example of tomographic reconstruction is proposed in Figure 2. The $(x,y)$ slices of the 3D reconstructed data cube show both the refraction (A) and absorption (B) of a Helianthus tuberosus pollen grain for a given z position (3D rendering of both refraction and absorption is provided in Media TDM.avi). The associated TDM acquisition is consisted in the sequential acquisition of 400 holograms. The illumination scanning pattern was chosen to be the 3D‐Uniform Sampled scheme proposed in Refs. 36, 37, as it has been shown to provide the best possible filling of the Fourier space. It can also be noticed that both images reveal different information about the sample. For instance, looking at the absorption image (Figure 2B), we can remark that this pollen only weakly absorbs light, except for some structures of its double‐wall envelope, ³⁸ a behaviour already observed for Betula pollen grain. ²³

FIGURE 2.

Reconstruction of a Helianthus tuberosus pollen grain acquired with our TDM configuration. (A) Refraction contribution. (B) Absorption contribution

So, to reconstruct 3D index of refraction images, TDM requires precise measurement of the diffracted field. This represents the entirety of the physical information carried out by the light‐sample interaction, contrary to conventional transmission microscopes, which record intensity‐only images. This property can be beneficially used if one is interested in obtaining an image of the sample under another imaging modality. Knowing the image formation model of a specific microscope, it is indeed possible to simulate the behaviour of classical intensity (bright‐field, dark‐field, Rheinberg, ...) and phase‐contrast (Zernike, DIC, …) microscopes, as shown in the next sections.

3. FROM TDM TO OPTICAL MICROSCOPY

As recalled in Section 2, a TDM acquisition consists in the sequential capture of holograms acquired with different illumination angles. But instead of performing a classical tomographic reconstruction, one can also directly use the optical fields extracted for each hologram, using either Born or Rytov approximations. ³⁹ Combination and proper filtering of the holograms will then allow for mimicking different microscopic imaging modalities. We split these methods in two main families: intensity techniques relying on the combination of intensity‐only images, and phase‐dependent modalities, for which a phase modulation, possibly combined with an amplitude modulation, is introduced. One can also differentiate 2D approaches, recombining information at focal plane of acquisition, from true 3D reconstructions, which make use of the angular spectrum approach described in Figure 1. A collection of the accesible 2D microscopy techniques is proposed Figure 3, and will be used for further discussions in the next section.

FIGURE 3.

Simulated multimodal 2D images of a Helianthus tuberosus pollen grain. (A) Bright‐field microscopy, (B) Dark‐field microscopy, (C,D) Oblique illumination microscopy with an oblique filter at 144° (C) and 312°, (E) Rheinberg illumination microscope, (F) positive phase‐contrast microscope, (G) negative phase‐contrast microscope, (H) DIC microscope. Scale bar is 10 $\mu \mathrm{m}$

3.1. Intensity‐only imaging

3.1.1. Bright‐field imaging

Bright‐field imaging corresponds to optimal use of the condenser to simultaneously illuminate the sample under many directions, using incoherent light (Köhler illumination): in bright‐field imaging, intensity information for each illumination angle is simply multiplexed. So, from an information point of view, bright‐field microscopy can be considered as a parallel information processing system, while TDM is a sequential one. The synoptic of the bright‐field microscope emulation is therefore straightforward, as proposed in Figure 4. Starting from the wavefront extracted from each angular hologram, an intensity image is calculated. All the intensity images are simply added to obtain the final bright‐field image. Applying this scheme makes it possible to obtain the image in Figure 3A. As it has been noticed from Figure 2, this pollen grain is weakly absorbing. This point is also noticeable on the reconstructed bright‐field image which presents a small contrast, except in the pollen wall structures.

FIGURE 4.

Synoptic of bright‐field imaging from TDM acquisition

For such weakly scattering samples, other imaging modalities, such as dark‐field, oblique, Rheinberg microscopy have been developed, which can also easily be computed from TDM data, as developed in the next paragraphs.

3.1.2. Dark‐field imaging

In conventional dark‐field operation, the investigated sample is illuminated using an annular illumination, at a higher angle that than of the maximal collecting angle of the objective. Doing so ensures that only the light diffracted by the sample is captured, so that its image appears on a black background, therefore the name of the technique, and its high sensitivity. To simulate dark‐field microscopy, we therefore only keep the holograms distributed along an annulus associated to the maximal NA of the condenser, while limiting the NA of the collection through Fourier space pupil filtering. Taking the intensity of each filtered hologram and summing the results allows to obtain the dark‐field image proposed in Figure 3B, which has been simulated according to the scheme proposed Figure 5.

FIGURE 5.

Synoptic of dark‐field imaging from TDM acquisition

Compared to the result obtained in Figure 3B, structural details about the pollen's membrane can be clearly noticed. This is an intrinsic property of dark‐field microscopy that allows to enhance high spatial frequency content in the images.

Note here the peculiar advantage of this purely numerical approach, which is that one can perform same NA, or even higher NA, at detection than at illumination by filtering out the specular illumination information in the Fourier space, while a physical dark‐field microscope set‐up imposes a lower NA at detection than at illumination. Other high‐pass filtering can also be considered to extract smaller details about cellular samples. ⁴⁰ However, we here limited our discussion to the most commonly available techniques.

3.1.3. Oblique illumination microscope

Methods based on the directional selection of the illumination angle make it possible to enhance contrast along a specific direction. This is, for instance, the case of oblique illumination microscopy, ⁴¹ for which part of the illumination is blocked, as in dark‐field microscopy, but here not only high‐angle illuminations are allowed, and detection can also be performed at high NA. To mimic this behaviour, the synoptic of Figure 6 can be considered. Here, only half of the condenser pupil is selected from the acquired holograms. The remaining holograms are then processed according to the classical bright‐field scheme, that is, the reconstructed optical field's intensities are summed for each processed hologram.

FIGURE 6.

Synoptic of oblique illumination microscope from TDM acquisition

Comparing Figure 3C,D with the bright‐field result (see Figure 3A), it can be noticed that the contrast is improved for the features whose direction is perpendicular to the illumination filter. This point is specifically pointed out when considering two different orientations of the oblique filter. Results proposed in Figure 3C for an oblique filter rotation of 144°, and 3D with an oblique filter rotated at an angle of 312°. Different features, pointed out by red arrows in Figure 3C,D can be extracted from the two oblique filter position images. Full rotation of the oblique filter is proposed in Media ObliqueStack.avi.

3.1.4. Rheinberg contrast imaging

In some cases, both low and high spatial frequency content is of interest, and in such cases, other methods, like Rheinberg contrast microscopy, also known as optical staining, can be of interest. ⁴² Rheinberg illumination is similar to dark‐field imaging, but (in its most classical form) instead of simply blocking by an opaque screen the central part of the cone of light emerging from the condenser, a coloured disk filter is used. For high angle incidence illumination rays, a different‐coloured annular filter is used (note that many different, more complex shapes of Rheinberg filters do exist). As a consequence, the observed specimen appears with the colour of the outer rays, on a background of the central filter colour, which greatly increases contrast.

Adapting the Rheinberg filter can improve the image contrast, but such modification is not really convenient/rapid on an actual microscope, and also the set of available Rheinberg filters is usually limited. On the contrary, one can easily imagine an equivalent of the coloured illumination filter in the numerical version of Rheinberg microscope, which could provide a much greater flexibility, as in Ref. 43.

A holographic implementation of Rheinberg microscope has been recently proposed, ⁴⁴ and we will here focus on the hologram processing.

The simulation scheme for Rheinberg contrast microscopy is proposed in Figure 7. As in its classical implementation, the illumination is colour‐filtered according to its angle, that is, holograms are associated to a colour filter according to their angles of incidence. The spectrum of each hologram (equivalently the collection objective back‐focal plane) is then computed and filters are applied accordingly, in a similar way as what is done with dark‐field microscopy.

FIGURE 7.

Synoptic of Rheinberg imaging from TDM acquisition

Here, we propose a triple‐colour Rheinberg filter. Three filters are applied to the holograms: the first one for the low illumination angles, in green in Figure 7), the second one in red for the mid‐illumination angles (in red in Figure 7), and the last one in blue for the highest illumination angles. Three images are computed accordingly, as in bright‐field microscopy. These three images are then fused to build an RGB‐composite intensity image, giving the Rheinberg contrast image proposed in Figure 3E. As expected, low spatial frequency content like the image background or slowly varying features appear in green, while high spatial frequency content associated with the pollen shell is red‐ and blue‐coloured.

Up to now, we described intensity‐based microscopic imaging techniques. However, techniques based on the phase information can also be considered, as presented in the next section.

3.2. Phase‐induced enhanced contrast

In the previous sections, we have drawn attention to intensity‐based microscopy techniques. But contrast can also be brought to images by acting on the phase of the light field. This can conveniently be performed from TDM images as we are able to measure both amplitude and phase of the optical field scattered by the investigated object. Classical techniques, like phase‐contrast imaging, and DIC microscopy can therefore also be emulated.

3.2.1. Zernike phase‐contrast microscopy

Phase‐contrast microscopy has been introduced by Zernike in 1935. ⁴⁵ , ⁴⁶ , ⁴⁷ The main idea of the method is, qualitatively, the following: an annular illumination is used, and noticing that at microscopic scale, a ray traversing a cell experiences an about 1/4 wavelength delay, inducing a supplemental 1/4 wavelength phase‐shift between the illumination wave and the wave scattered by the object results in an enhancement of the image contrast, as both waves interfere destructively.

To mimic this behaviour using TDM data, the processing scheme presented in Figure 8 is considered. After proper selection of the illumination angle, a back‐focal plane filter $\mathcal{P}(k_x,k_y)$, defined as

$$\mathcal{P}\left(k_x,k_y\right)=\alpha \left(k_x,k_y\right)\exp \left[i\mathrm{\Phi }\left(k_x,k_y\right)\right]$$ (2)

allows for shifting the phase of the specular illumination spot, while also slightly attenuating its amplitude. In Equation (2), α corresponds to the amount of attenuation of the specular beam (note that it also attenuates/shifts the corresponding portion of the diffracted rays).

$$\alpha \left(k_x,k_y\right)=\left\{\begin{array}{c}\alpha \mathrm{if}\sqrt{{\left(k_x-k_{i_x}\right)}^2+{\left(k_y-k_{i_y}\right)}^2}\le {\rho }_{\mathrm{pc}},\hfill \\ 1\mathrm{elsewhere},\hfill \end{array}\right.$$ (3)

with $k_{i_x}$ and $k_{i_y}$ the spectral coordinates of the specular illumination spot and ρ_pc the radius of the pupil filter. The introduced phase‐shift $\mathrm{\Phi }(k_x,k_y)$ can be similarly defined by

$$\mathrm{\Phi }\left(k_x,k_y\right)=\left\{\begin{array}{c}\mathrm{\Phi }\mathrm{if}\sqrt{{\left(k_x-k_{i_x}\right)}^2+{\left(k_y-k_{i_y}\right)}^2}\le {\rho }_{\mathrm{pc}},\hfill \\ 0\mathrm{elsewhere}.\hfill \end{array}\right.$$ (4)

Reconstruction of the pollen grain with phase‐contrast configuration is depicted in Figure 3 for both positive (F) and negative (G) phase contrasts. Here, $\alpha =0.7$ and $\mathrm{\Phi }=\pm \pi /4$ (see Equations 3 and 4) for positive and negative phase contrast, respectively. As expected, one can notice than the pollen images exhibit better contrasts than the bright‐field image reconstructed in Figure 4, which is explained by the attenuation coupled with the introduced phase‐shift, as in an actual phase‐contrast microscope.

FIGURE 8.

Synoptic of phase‐contrast imaging from TDM acquisition

3.2.2. DIC microscopy

DIC microscopy, proposed by Nomarski in 1955, is another classical technique for the characterization of weakly scattering samples. ⁴⁸ , ⁴⁹ Here, the investigated object is illuminated using two orthogonally polarized light fields (ordinary and extraordinary light fields emerging from a Nomarski prism). Parallel and slightly shifted orthogonally polarized rays passing through the observed samples are recombined, and then pass through an analyzer. This leads to two cases. If both rays experience the same dephasing, they are recombined by the Nomarski prism into a linearly polarized ray, blocked by the analyzer. If one or the other ray emerges in advance, an elliptically polarized beam emerges from the analyzer to form an image onto the detector. Properly adjusted, the imaged contrast corresponds to gradients in specimen optical path lengths, hence the name of Differential Interference Contrast.

Simulations of DIC images have been proposed under the framework of DHM acquisition. ⁵⁰ , ⁵¹ Moreover, an implementation has recently been proposed for generalization to tomographic experiment. ⁵² As our reconstruction algorithms are based on the scalar implementation of the Born formalism, polarization of the incident light is not accounted for. This issue can easily be overcome considering that two orthogonally polarized beam are in phase quadrature. Thus, for each hologram ${\mathcal{H}}_{\theta }$ corresponding to an illumination angle θ, we will estimate the DIC image I _DIC considering

$$I_{\mathrm{DIC}}={\left|{\mathcal{H}}_{\theta }\left(x,y\right)-{\mathcal{H}}_{\theta }\left(x+\mathrm{\Delta }x,y+\mathrm{\Delta }y\right)\exp \left(-i{\mathrm{\Phi }}_{\mathrm{DIC}}\right)\right|}^2,$$ (5)

with Φ_DIC and $\mathrm{\Delta }x,y$, denoting the introduced phase‐shift and the spatial shift between the two holograms, respectively.

The associated simulation scheme is proposed in Figure 9. For each hologram, a difference image is calculated according to Equation (5). Results are then summed in intensity for all illumination angles, leading to the final DIC image.

FIGURE 9.

Synoptic of DIC microscopy from TDM acquisition

The obtained image is presented in Figure 3H. The gradient effect, which is a basic signature of DIC microscopy, is here clearly noticeable, and makes it possible to better point out the sample structure.

Classical microscopy set‐ups have been successfully simulated from TDM acquisitions. Depending on hologram combination and filtering, we were able to obtain multimodal 2D information about the investigated sample. But as far as we are working with holographic data, we can also mimic the focusing/defocusing occurring when moving the sample holder along the microscope optical axis, as described in the next section.

3.3. Numerical refocusing

Illustrations proposed up to now consisted in the simulation of 2D microscopic image modalities. In this situation, determining the best focalized image of a thick sample can be challenging. ⁵³ So for our simulation framework to be complete, we can also consider taking into account the focus adjustment. This can be performed using classical angular spectrum propagation algorithms. ¹² , ¹³ Other approaches can also be envisaged. ⁵⁴ , ⁵⁵ Considering the simulated intensity images ${\mathcal{I}}_{z=0}(x,y)$ for a given imaging modality, the intensity after propagation over a distance $\mathrm{\Delta }z$ is given by

$${\mathcal{I}}_{\mathrm{\Delta }z}\left(x,y\right)={\mathcal{F}}^{-1}\left\{\mathcal{F}\left[{\mathcal{I}}_{z=0}\left(x,y\right)\right]\times H_{\mathrm{\Delta }z}\left(k_x,k_y\right)\right\},$$ (6)

with $\mathcal{F}$ and ${\mathcal{F}}^{-1}$, respectively, denoting Fourier and inverse Fourier transforms. Here, $H_{{\mathrm{\Delta }}_z}(k_x,k_y)$ is the angular spectrum propagation term expressed as

$$H_{{\mathrm{\Delta }}_z}\left(k_x,k_y\right)=\exp \left[i\frac{2\pi n_{\mathrm{imm}}}{\lambda }\sqrt{1-{\left(\lambda k_x\right)}^2-{\left(\lambda k_y\right)}^2}\right],$$ (7)

with n _imm the refractive index of the immersion medium. Illustration of numerical refocusing is available in supplementary materials (Media Zfocus.avi).

4. EXTENSION TO 3D IMAGING

Reconstruction modalities presented so far were built by acting on the reconstruction of individual holograms, leading to 2D modalities (with possible refocusing), as often done in holographic imaging. However, TDM being intrinsically a true 3D imaging approach, applying our schemes to the 3D object frequency support will allow a direct extension to the most common microscopy imaging modalities, delivering 3D images.

4.1. Dark‐field and phase‐contrast 3D microscopy

The processing scheme for 3D generalization of both dark‐field and phase‐contrast microscopy from tomographic data is proposed in Figure 10. Here, as for 2D schemes, the acquired spectra are filtered in the low spatial frequency region. ⁵⁶ In such case, the filter to apply is defined as

$${\mathcal{P}}_{3D}\left(k_x,k_y,k_z\right)={\alpha }_{3D}\left(k_x,k_y,k_z\right)\exp \left[i{\mathrm{\Phi }}_{3D}\left(k_x,k_y,k_z\right)\right],$$ (8)

with

$${\alpha }_{3D}\left(k_x,k_y,k_z\right)=\left\{\begin{array}{c}{\alpha }_{3D}\mathrm{if}\sqrt{k_x^2+k_y^2+k_z^2}\le {\rho }_{3D},\hfill \\ 1\mathrm{elsewhere},\hfill \end{array}\right.$$ (9)

and

$${\mathrm{\Phi }}_{3D}\left(k_x,k_y,k_z\right)=\left\{\begin{array}{c}{\mathrm{\Phi }}_{3D}\mathrm{if}\sqrt{k_x^2+k_y^2+k_z^2}\le {\rho }_{3D},\hfill \\ 0\mathrm{elsewhere}.\hfill \end{array}\right.$$ (10)

Here, $k_{x,y,z}$ are the 3D frequency coordinates of the object's spectrum and ${\rho }_{3D}$ is the filtering sphere radius. Attenuation and filter phase‐shift are, respectively, given by ${\alpha }_{3D}(k_x,k_y,k_z)$ and ${\mathrm{\Phi }}_{3D}(k_x,k_y,k_z)$. If one wants to simulate dark‐field images, Equations (9) and (10) are simply modified so that:

$${\alpha }_{3D}\left(k_x,k_y,k_z\right)=\left\{\begin{array}{c}0\mathrm{if}\sqrt{k_x^2+k_y^2+k_z^2}\le {\rho }_{3D},\hfill \\ 1\mathrm{elsewhere},\hfill \end{array}\right.$$ (11)

and

$${\mathrm{\Phi }}_{3D}\left(k_x,k_y,k_z\right)=0.$$ (12)

Here appears one advantage of this numerical approach compared to an actual dark‐field or phase‐contrast microscope. In these systems, an annular illumination has to be used to increase contrast, but can be detrimental to image reconstruction quality, especially in terms of resolution, as the optical transfer function is not optimally filled in Fourier space, contrary to TDM, which can use, via synthetic aperture, the full NA of the condenser, ²³ , ³³ , ⁵⁷ the necessary phase/amplitude correction factor being applied after acquisition of the data. Results obtained for both dark‐field and phase‐contrast microscopy are presented in Figure 11 for qualitative comparison (see Media 3D Modalities.avi for z‐slicing). Imaging properties of both modalities are here compared with an axial cut (A,B) and longitudinal cut (C,D) in the reconstructed volume.

FIGURE 10.

TDM data processing for 3D dark‐field and phase‐contrast imaging

FIGURE 11.

Slices of the 3D reconstruction considering (A) dark‐field and (B) phase‐contrast modalities, (C, D) associated $xz$‐plane cuts

4.2. DIC microscopy

The 3D generalization of DIC microscopy is fairly straightforward. Remembering that the contrast in DIC is directly proportional to gradients in the optical specimen, instead of shifting and phase‐shifting the holograms to build a DIC image as in Section 3.2.2 (see Figure 9), one can now apply this approach directly to the 3D reconstructed tomographic images.

Obtained results are proposed in Figure 12A,C (see Media 3D Modalities.avi for z‐slicing). Here, a 3 pixel spatial shift along the x‐axis, corresponding to a 165‐nm physical shift (accounting for imaging system magnification) and a 0.01 rad phase‐shift is applied to the tomographic volume. As for the 2D case, the gradient effect can be clearly noticed. The advantage of this numerical approach is that bias and retardation can easily be tuned to highlight different features of interest. One could even apply gradient operation in out‐of‐plane directions (i.e. not only in the x‐y plane), something which is not feasible in an actual DIC microscope. An example is given in Section 5. Such a feature remains to be fully explored, as it adds a new parameter to DIC imaging, which is promising but may also complicate interpretation of the observed contrast.

FIGURE 12.

Slices of the 3D reconstruction considering (A) DIC and (B) Rheinberg illumination contrast modalities, (C, D) associated $xz$‐plane cuts, (E) $(k_x,k_y)$ cut of the frequency support, (F) $(k_x,k_z)$ cut of the frequency support

4.3. Rheinberg illumination contrast

Rheinberg illumination contrast (see Figure 7) can also be generalized for 3D operation. To this aim, each acquired hologram is associated to a filter colour according to its corresponding illumination angle. For each colour channel, a separate TDM reconstruction is performed. Reconstructed intensities are finally merged to build a composite RGB reconstruction of the object as depicted in Figure 12B,D. Here, green channel is associated with small illumination angle holograms, red channel with ‘intermediate’ illumination angles and blue channel with the highest illumination angles. Size and position of these illumination filters can easily be tuned depending on the sample's feature one wants to highlight.

Note that mimicking the Rheinberg illumination gives a different‐coloured spectrum than simply assigning colours to low‐, intermediary‐ and high frequencies, as could also be done in Fourier filtering. The peculiarities of 2D/3D image formation in transmission microscopy/TDM ³³ , ⁵⁷ make that, for example, high illumination angles, here assigned blue colour, also contribute to the low‐frequency spectrum of the observed object, as can be seen in Figure 12E,F (see Media 3D Modalities.avi for z‐slicing).

5. DISCUSSION

This study is mainly driven by the fact that tomographic microscopes become more and more common, with commercial TDM devices being even now available. ⁵⁸ , ⁵⁹ While this image contrast presents several advantages, it can in some cases remain difficult to interpret, due to the lack of selectivity of the reconstructed data. For this reason, manufacturers are now proposing coupled fluorescence/TDM devices. Another possible way to add selectivity in data reconstruction is to reexplore conventional imaging contrasts, in order to provide experimentalists, with images they are more familiar with and used to interpret. This is made possible by the QPI nature of a TDM configuration. In other words, this article is a tutorial on how to obtain different imaging contrasts, from TDM data (holograms), when available. Considering this, some remarks can be formulated considering the obtained results.

When using a DHM configuration with coherent illumination of the sample, coupled with simple/rapid Fourier inversions, noise can be noticed in reconstructed images in Figure 3. This noise structure is coming from a constructive interference phenomenon occurring at the sensor's protection window plane. This effect can be limited if one consider a short coherence light source, or using an image sensor without protection windows. The immediate consequence is that the simulated results are of a lower overall quality than that possibly obtained using a dedicated phase contrast, DIC… configuration. However, all the simulated results share the same field of view, which is not that trivial for multimodal imaging if one has to use multiple microscopes, as such modalities can exclude each other: a physical dark‐field microscope, for example, cannot use the same high NA objectives than a DIC .

As 3D microscopy techniques are concerned (see Figures 2, 11, 12 and 14), reconstructions can also be prone to Gibbs artefact at the border of the obtained field of view. These are coming from the Fourier space processing of acquired data. Moreover, looking at the associated supplementary media files, a strong noise structure can be noticed in the central imaging plane. This plane corresponds to the coherent summation of the reference beam for all the acquired holograms. It can be reduced considering either a blank acquisition (i.e. a sample less acquisition) or using a sensor without protection window. It should be noted that as 3D techniques are concerned, there is, to the best of our knowledge no commercial equivalent to our simulation protocol.

FIGURE 14.

Simulated composite image combining Rheinberg contrast on the ‘R’ and ‘G’ channels, and DIC microscope on the ‘B’ channel. (A) $xy$‐plane cut, (B) $xz$‐plane cut

This work therefore does not aim at replacing routine microscopes. Instead, we propose a simple way to possibly add visualization modalities to existing TDM systems. As a matter of fact, superposing conventional imaging results to TDM results can bring back some of the contrast selectivity, which is lost when considering a marker‐less technique.

6. CONCLUSION AND OUTLOOKS

The sequential nature of data acquisition in TDM can be a drawback in terms of speed, but, contrary to classic optical transmission microscopes, allows for proper identification of the contribution of each illuminating ray, in both amplitude and phase. This makes it possible, considering adequate data processing, to emulate conventional 2D intensity and phase‐contrast microscopes, even taking into account possible defocus, similar to what occurs on a bench‐top microscope. We have therefore studied reconstructing conventional microscopy imaging modalities from TDM acquisitions. Furthermore, considering the 3D reconstruction possibilities of TDM, we extended this work to full 3D imaging.

Note that other imaging modalities could also be similarly emulated. For example, Hoffman modulation contrast ⁶⁰ is based on an amplitude spatial filter, inserted at the back‐focal plane of the objective, therefore performing Fourier filtering, combined with a (usually offset) slit illumination, also equivalent to properly selecting illumination angles in TDM.

This study also paves the way for imaging new microscopy modalities that are not easily feasible in practice, for example, out‐of‐focal‐plane DIC, as suggested in previous section, and as illustrated in Figure 13. Here, effect of both lateral and axial shearing in either $x,y$ or $x,z$ cuts is illustrated. If lateral shearing is routinely performed with a DIC microscope, axis shearing cannot be performed. It, however seems to improve axial detection of our 3D digital DIC microscope. Similarly, confocal transmission microscopy should be feasible, as already done in reflection TDM. ⁶¹ One of the difficulty of transmission confocal microscopy is to keep confocalized both illumination and detection, which are performed through two different optical trains, while reflection‐ and fluorescence confocal microscopy are autoconfocal. A numerical approach could help leverage this difficulty, even if transmission confocal does not provide same optical sectioning as fluorescence confocal microscopy. Imaging modalities based on Fourier filtering using phase plates or Spatial Light Modulators could also be simulated. ⁶² , ⁶³ , ⁶⁴ Residual aberrations correction has already been numerically implemented. ⁶⁵ , ⁶⁶ , ⁶⁷ Finally, some microscopy modalities combination not implementable in practice, can easily be computed, for example, the combined Rheinberg–DIC microscopy, as illustrated in Figure 14, in which Rheinberg illumination is simulated in green ($\mathrm{NA}<0.4$) and red ($0.4\le \mathrm{NA}\le 1$) and DIC is computed for $\mathrm{NA}>1$. Axial (A) and longitudinal (B) cuts are proposed. Three‐dimensional rendering is available in Media RheinbergDIC.avi.

FIGURE 13.

Influence of the shearing direction on the simulated 3D DIC results. (A) $x,y$ cut, (C) $x,z$ cut with a lateral shearing along x‐direction. (B) $x,y$ cut, (D) $x,z$ cut with an axial shearing along z‐direction. Shearing direction is figured out by the arrows in red

In the given simulated modalities, pre‐determined intensity filters have been proposed to reconstruct dark‐field, Rheinberg or oblique illumination imaging. Similarly, fixed phase changes (phase contrast) or bias and retardation (DIC) have been used. A promising approach could be to introduce dynamic variations of the Fourier filtering, in both amplitude and phase, which, coupled to an adapted image contrast optimization criteria, could lead to a dynamic, purely‐numerical contrast optimization, as performed in intensity‐only imaging in Refs. 43, 68, in order to efficiently highlight the structures of interest to be investigated.

A scalar‐approach TDM microscope delivers all the optical information, in terms of amplitude and phase, resulting from the sample under investigation with controlled illumination, and has already clearly demonstrated improved resolution compared to conventional transmission microscopy. ¹⁶ , ²¹ , ³² Note that extension towards polarized TDM has already been proposed, ¹⁷ , ⁶⁹ , ⁷⁰ , ⁷¹ as well as multispectral or hyperspectral tomography. ⁷² , ⁷³ , ⁷⁴

So, at least in theory, a combined polarized, hyperspectral TDM system could, with proper data processing, emulate all kind of existing optical transmission microscopes, even the classical polarization microscope, which works in white light. Such an approach would be tremendously challenging in terms of instrumentation (speed of acquisition ⁷⁵ , ⁷⁶ , ⁷⁷ and data processing ⁷⁸ , ⁷⁹ , ⁸⁰ ), but could pave the way towards a kind of all‐in‐one, universal system for optical transmission microscopy.

CONFLICT OF INTEREST

The authors declare that there is no conflict of interest that could be perceived as prejudicing the impartiality of the research reported.

Supporting information

Data S1

Data S2

Data S3

Data S4

Data S5

Abbessi, R. , Verrier, N. , Taddese, A. M. , Laroche, S. , Debailleul, M. , Lo, M. , Courbot, J.‐B. , & Haeberlé, O. (2022). Multimodal image reconstruction from tomographic diffraction microscopy data. Journal of Microscopy, 288, 193–206. 10.1111/jmi.13131