Dynamic spatial filtering using a digital micromirror device for high-speed optical diffraction tomography

Abstract

Optical diffraction tomography (ODT) is an emerging microscopy technique for three-dimensional (3D) refractive index (RI) mapping of transparent specimens. Recently, the digital micromirror device (DMD) based scheme for angle-controlled plane wave illumination has been proposed to improve the imaging speed and stability of ODT. However, undesired diffraction noise always exists in the reported DMD-based illumination scheme, which leads to a limited contrast ratio of the measurement fringe and hence inaccurate RI mapping. Here we present a novel spatial filtering method, based on a second DMD, to dynamically remove the diffraction noise. The reported results illustrate significantly enhanced image quality of the obtained interferograms and the subsequently derived phase maps. And moreover, with this method, we demonstrate mapping of 3D RI distribution of polystyrene beads as well as biological cells with high accuracy. Importantly, with the proper hardware configuration, our method does not compromise the 3D imaging speed advantage promised by the DMD-based illumination scheme. Specifically, we have been able to successfully obtain interferograms at over 1 kHz speed, which is critical for potential high-throughput label-free 3D image cytometry applications.

1. Introduction

Optical diffraction tomography (ODT) has recently become a popular microscopy technique for measuring three-dimensional (3D) refractive index (RI) distributions of optically transparent microscopic specimens such as biological cells [1–4]. Although first theoretically proposed by E. Wolf in 1969 [5] and then experimentally demonstrated in late 1970s [6,7], ODT has not been applied for cell imaging until 2009 [8]. Compared with other existing 3D optical microscopy techniques such as confocal microscopy, light sheet microscopy and multi-photon microscopy, ODT is distinguished by its label-free nature to obtain the 3D shape information of objects through using quantitative amplitude and phase measurements offered by quantitative phase microscopy. Over the last few years, the application of ODT has gained wide interest and rapid growth in various areas including cell biology [9–11], hematology [12,13] and pathophysiology [14].

In principle, the majority of ODT modalities rely on measuring multiple holograms of the specimen with various angles of illumination, and then reconstructing the 3D RI tomogram from these measured holograms by solving an inverse scattering problem. Therefore, it is important to precisely and stably control the incident angle of the illumination beam impinging onto the specimen. Traditionally, two orthogonal galvanometric mirrors, placed at the conjugate planes of the sample through a 4f system, have been used to control the illumination angle by mechanically tilting the mirrors through changing the applied voltage values [15]. However, this technique suffers from a relatively long mechanical settling time for the mirrors (settling time for the mirror of 1 inch size and angle change magnitude of 10% full scale is typically ~3 ms), which affects 3D imaging throughput. Furthermore, it is impossible to simultaneously conjugate each mirror surface of the two-axis galvanometric mirrors to the sample, which results in undesired distortion to the plane wave illumination beam. Other works featuring the use of spatial light modulators (SLM) possess the advantages of high stability and wavefront correction capability [16]; however, such approach is limited by the low response rate and relatively higher cost. Recently, a novel active illumination control method based on a digital micromirror device (DMD) was proposed to break the limitations [17]. The DMD contains up to several millions of individually switchable micromirrors (typical pitch size is ~10 μm) with switching speed ranging from several kHz to tens of kHz (settling time for full scale angle change is around 10 μs). The principle of Lee hologram can be employed in order to control the illumination angle [18]. Specifically, the DMD is programmed to display a series of binary hologram patterns, which act as a dynamic reflection grating to transform the incident plane wave into multiple diffraction orders. The 1st order diffraction beam is selected with a Fourier space filter and used to provide the active illumination onto the sample. Over the last three years, DMD has been successfully applied to ODT with its high-speed control and extraordinarily stable operation without mechanical movements [17,19].

Since each micromirror of DMD has only on and off states, 1-bit (binary) amplitude grating patterns can be displayed by the DMD in its high-speed operation mode, which inevitably generates multiple diffraction orders. However, only the first diffraction order is desired and as such the presence of other orders deteriorate the imaging quality of the final reconstructed RI tomogram. To solve this issue, the first adopted strategy was to use an annular aperture located at the Fourier plane of the DMD, which only lets the 1st order diffraction beam through and blocks other beams [17,19]. However, this aperture design cannot fully eliminate the unwanted diffraction beams for every angle of illumination (there are always some unwanted diffraction orders that happen to pass through the aperture), and thus cannot free the measured holograms from noise. Furthermore, the use of this aperture strictly limits the available range of illumination angle; specifically, the reachable Zenith angle of illumination beam is limited. Last but not the least, this kind of annular aperture is not commercially available and needs to be custom made, which takes efforts to determine the suitable size of the annulus. Also, the annulus aperture alignment in the Fourier space is not trivial. Most recently, a time-multiplexed structured illumination scheme was proposed, where a sinusoidal pattern instead of the binary hologram is displayed on the DMD using the high-bit-depth video mode [20]. This approach allows generating only three diffraction orders (0th and ± 1st) that are combined to form the structured illumination, which perfectly removes the diffraction noise. However, the maximum frame rate of the video mode is only 60 Hz, thus losing the high-speed operation advantage of DMD.

Here, we propose a novel dynamic spatial filtering method that perfectly eliminates the diffraction noise for DMD based high-speed angle-scan ODT systems. More specifically, we use another DMD for spatial filtering besides the original one for illumination angle control; by synchronizing the two DMDs, the 1st order diffraction beam can be accurately selected without stray light contamination for any illumination angle. In the following sections, the principle of the proposed method will be illustrated and the corresponding filtering quality will be assessed by comparing with previous methods. In addition, we demonstrate 3D RI tomography of polystyrene beads and biological cells using the proposed filtering method.

2. Experimental setup

Figure 1 shows the schematic layout of the dual DMD-based ODT system. In brief, the proposed setup is based on the classic Mach–Zehnder interferometric microscope combined with an illumination control part using two DMDs. A 532-nm laser beam is split by a 1 × 2 single-mode fiber coupler (SMFC) into two arms: the sample beam and the reference beam. The sample beam, collimated by lens L1, is reflected by the mirror M1 and the first DMD D1 (DLP LightCrafter 9000) before illuminating the sample. D1 is programmed to display binary amplitude grating patterns with the Lee hologram scheme that generates multiple diffraction orders, among which the three primary ones, 0th and ± 1st orders, are shown in Fig. 1. Afterwards, these reflected beams form a series of diffraction spots through lens L2 at the Fourier plane, where the second DMD D2 (DLP LightCrafter 3000) is placed. Next, a filter mask pattern illustrated in the inset 1 of Fig. 1 is displayed onto D2 so that only the 1st order diffraction spot falls into the open region (marked in white color) of the pattern and be reflected onto the mirror M2. Other diffraction spots that fall onto the closed region (marked in black color) of the pattern would not be reflected into the subsequent optical system and thus be eliminated. In this way, the sample beam can be totally free from diffraction noises. Intuitively, the mask pattern on DMD D2 acts like a digital pinhole that only allows the desired 1st order diffraction light passes through the rest of the optics. In terms of this pinhole shape design, we simply used an open circle whose diameter is slightly larger than that of the 1st order diffraction spot. In this way, we can avoid the leakage of the diffraction noise from other diffraction orders, thus creating a clean sample illumination beam. When the Lee hologram on DMD D1 is updated, the location of the circular filter on DMD D2 is also updated immediately through a hardware trigger. For better illustration, in the inset 2 of Fig. 1 we simultaneously show 16 circular filters that correspond to 16 angles of illumination (note that in experiments at one time only one specific circular pinhole is opened). After collimation by lens L3, the filtered sample beam angle is magnified to achieve the desired angle-scan range through a 4-f system composed of a tube lens L4 (f = 350 mm) and an objective lens OL1 (Zeiss, 40X/1.3, oil immersion) that serves as a condenser lens. This arrangement allows us to illuminate the sample with plane waves of desired illumination angle up to 48° from the optical axis (in medium with RI of 1.337). The scattered light from this sample is collected by an objective lens OL2 with same specifications as that of OL1 and a tube lens L5 (f = 200 mm). The last 4-f system composed of two lenses L6 (f = 75 mm) and L7 (f = 300 mm) is used to provide another 4 × lateral magnification to satisfy the sampling condition [21]. To retrieve both the amplitude and phase information of the sample in a single shot, an off-axis holography setup is adopted. Specifically, the sample and reference beams are brought back together through a beam splitter BS to form the interferogram, which is recorded by a complementary metal-oxide semiconductor (CMOS) camera (such as Photron Inc., Fastcam APX RS and Optronis CP90-25-M-72 used in our experiments).

Fig. 1.

Schematic system setup. L1-L8, lenses (f1, 5, 8 = 200 mm, f2 = 125 mm, f3 = 150 mm, f4 = 350 mm, f6 = 75 mm, f7 = 300 mm); M1-M2, mirrors; D1-D2, digital micromirror devices (DMDs); OL1-OL2, objective lenses (Zeiss, 40X, oil immersion, NA = 1.3); SMFC, single-mode fiber coupler; BS, beam splitter. Inset shows the mask pattern displayed by D2.

To dynamically select the 1st order diffraction beam generated by DMD D1, the two DMDs D1 and D2 must be synchronized. Each time D1 displays a different grating pattern, correspondingly the 1st order diffraction spot changes its position at the plane of D2; therefore, the mask pattern displayed by D2 must be updated accordingly so that only the 1st order diffraction beam is reflected into the subsequent optical system. Thanks to the extraordinary hardware triggering capability of the commercially available DMDs, the time lag between two DMDs’ synchronization is around 10 μs, which is sufficient for high-speed operation of the proposed dynamic filtering scheme for diffraction noise free ODT.

3. Results and discussion

Figure 2 illustrates the dynamic spatial filtering capability of our proposed method. Figures 2(a1)-2(a3) show the raw interferogram of a 10 µm diameter polystyrene bead (Duke Standards, 4210A), modulus of the Fourier transform (FT) of the interferogram and the corresponding calculated phase image (in radians), respectively, for 48° illumination angle with respect to the normal incident beam (also known as Zenith angle in medium with RI of 1.337) when no spatial filtering is applied. On the other hand, Figs. 2(b1)-2(b3) correspond to the case where an annular aperture as used in [17] is applied, whereas Figs. 2(c1)-2(c3) correspond to the case where a mask pattern as shown in the inset of Fig. 1 is utilized. By looking at the FT figures of the measured raw interferograms shown in Figs. 2(a2) and 2(b2), it can be found that a significant amount of diffraction noise (indicated by the red arrows) is introduced by DMD D1, and that the use of an annular iris cannot be sufficient to completely remove the noise, thus leaving some unwanted residual signals (indicated by the black arrows). Undoubtedly, the diffraction noise significantly deteriorates the image quality of calculated phase images as shown in Figs. 2(a3) and 2(b3). However, the proposed dynamic spatial filtering method can perfectly get rid of all the diffraction noise; see Fourier map in Fig. 2(c2) that results in a descent phase image illustrated in Fig. 2(c3). In summary, this set of comparisons clearly demonstrates the outstanding diffraction noise removal capability of our proposed dynamic spatial filtering method. To be noted, in our experiments, we found that for some azimuthal angles, the diffraction noise is enormous, but for others, the situation can be less severe. To highlight our comparison, we hereby selected one of the azimuthal angles where only our proposed dynamic spatial filtering can completely remove the diffraction noise.

Fig. 2.

A comparison of diffraction noise removal performance. (a1)-(a3) show the raw interferogram of the 10 µm polystyrene bead, modulus of the Fourier transform of the interferogram and the correspondingly calculated phase image, respectively, with respect to the Zenith illumination angle of ~48° (in medium with RI of 1.337) when no spatial filtering is applied. (b1)-(b3) correspond to the case where an annular aperture is used. (c1)-(c3) correspond to the case where a mask pattern as proposed in this study is used. The color bars in (a2), (b2) and (c2) are in logarithm scale. The color bars in (a3), (b3) and (c3) are in radians.

After obtaining multiple holograms corresponding to a set of illumination angles, a 3D RI distribution of the sample can be reconstructed by using the ODT algorithm [8]. In brief, the algorithm is based on the Fourier diffraction theorem [5] that maps multiple 2D Fourier spectra of the complex optical fields into the 3D Ewald sphere in the Fourier space, and then solves out the scattering potential of the sample by applying a 3D inverse FT of this Ewald sphere map. In addition, piecewise smoothness and positivity regularizations are applied during the reconstruction to alleviate the missing cone problem [22]. In summary, the algorithm used in this article can be formulated as the following equation:

$$\Phi (f;g)=\frac{1}{2}{\displaystyle {\sum }_n{\Vert A_{n}f-g_n\Vert }^2}+\alpha J(f),$$ (1)

where $\Phi (f;g)$ is the cost functional; $\alpha$ is the regularization coefficient; $J(f)$ is the penalty functional; $f$is the objective functional (i.e., the scattering potential function) to be reconstructed; $A_n$ is the nth diffraction projection operation on $f,$ which results in a 2D scattering field of the 3D object for the incident angle; $g_n$ is the measured scattering field. By minimizing the cost functional in Eq. (1), $f$ can be solved out, from which the RI map of the sample can be determined by using the following equation:

$$n(r)=n_0\sqrt{1-f(r)/k^2},$$ (2)

where $k=2\pi /\lambda ,$$\lambda$ is the wavelength of the incident beam in the medium; $n(r)$ is the complex RI of the sample in 3D; and $n_0$ is the RI of the medium. More details of this ODT reconstruction algorithm can be referred to previous literature [22,23].

In order to demonstrate the 3D imaging capability of the present setup, we first measured the 3-D RI distribution of a polystyrene micro-bead of 10 μm diameter (Duke Standards, 4210A, RI = 1.592 at wavelength of 532 nm) immersed in oil with RI of 1.56 (Cargille Labs). During the experiment, interferograms corresponding to a total of 57 illumination angles are collected. More specifically, one interferogram corresponds to normal incident beam, 20 of the illumination beams are at an elevation angle of ~27°, and the rest are at an elevation angle of ~48° (in medium with RI of 1.337). For each elevation angle, the azimuthal illumination angles are separated equally. The imaging frame rate of the camera is set at 1500 fps, which is also synchronized with the DMDs. Figure 3 shows the reconstructed RI tomogram of the polystyrene bead in x-y, x-z and y-z cross-section planes, respectively. It is clearly shown that both the 3D shape and the retrieved RI value of the bead are in good agreement with the manufacturer’s specification.

Fig. 3.

Cross-sectional slices of a reconstructed RI tomogram of a 10 μm diameter polystyrene bead in x–z, x–y, and y–z planes.

To further demonstrate the imaging applicability of the present method, the 3D RI distribution of a human red blood cell (RBC) was measured and shown in Fig. 4, where x-z, x-y and y-z cross–section plane images are plotted. The measurement follows the same experimental procedure and reconstruction scheme. The RBC was diluted in Alsever’s solution (Sigma Aldrich Inc., A3551, RI = 1.337), and sandwiched between two coverslips before measurement. Figure 4 clearly shows the characteristic biconcave shape of a RBC collected from a patient with sickle disease (donated from Massachusetts General Hospital, MA, USA) and the uniform RI distribution in the x-y plane cross-section slice indicates a uniform distribution of the hemoglobin (Hb) protein in RBC cytoplasm. RBCs are mainly composed of Hb solution, and it has been well established that there is a linear relationship between the concentration and the RI value of Hb solution, described by $C=(n-n_m)/\alpha$, where C is the Hb concentration, n is the RI value of Hb, n_m is the RI value of surrounding solution (n_m = 1.337 at 532 nm) and α is the RI increment that can be set as 0.148 ml/g at 532 nm for oxygenated Hb [24]. The average RI value for the RBC in Fig. 4 is 1.391, and the corresponding Hb concentration is 36.5 g/dL, which is within the normal range of the healthy physiological condition of RBCs [25].

Fig. 4.

Cross-sectional slices of a reconstructed RI tomogram of a healthy human RBC in x–y, x–z, and y–z planes.

To illustrate the capability of imaging more complex cells, HeLa cells suspended in cell culture medium (Phosphate Buffered Saline (PBS), Sigma-Aldrich) in ball shapes were also measured. The results are presented in Fig. 5, where x-z, x-y and y-z cross–section plane images are plotted. As can be seen, in the x-y cross-section plane of Fig. 5, the green and yellow colored region in the right side of the cell where the RI values are lower than the surrounding regions clearly represent the nucleus of the eukaryotic cells, which is consistent with previous findings [26].

Fig. 5.

Cross-sectional slices of a reconstructed RI tomogram of a HeLa cell in x–y, x–z, and y–z planes.

To demonstrate the capability of high-speed 3D RI imaging, we tracked the 3D trajectory of a 2 μm silicon dioxide bead in the water. Note that we used a different high-speed camera (Optronis CP90-25-M-72) for easier streaming image data to the computer via a frame grabber, so that the maximum image volume is only constraint by the RAM or hard drive size of the computer. We specially used this camera configuration for long-term imaging that lasts seconds to minutes and its maximum frame rate is around 1136 fps with 256x256 pixels resolution, which is constraint by the data transferring speed of the frame grabber that we are using. The illumination angle was scanned in a circular pattern in addition to the normal illumination with 16 steps for minimizing scanning time. We imaged the bead for ~6.4 s and reconstructed consecutive 3D RI tomograms that achieved 3D imaging at 71 frames per second. Figure 6(a) shows the 3D trajectory of the bead, from which the mean-squared displacement (MSD) of the bead can be calculated. In Fig. 6(b), the MSD of the bead was plotted as a function of the lag time marked as blue dots and an orange fitted line was plotted by doing a linear regression. The R squared error of this linear regression is about 0.9889, which indicates that the MSD has a good linear relationship with the lag time. Actually, this finding exactly matches the diffusion equation in 3D: $MSD\left(t\right)=6D\tau ,$ where $D$is the diffusion coefficient, and $\tau$ is the lag time. For a spherical object, the diffusion coefficient can be obtained from the Stokes–Einstein relation, $D=k_{B}T/6\pi \eta r,$ where $k_B$ is the Boltzmann constant, $T$ is the temperature, $\eta$ is the dynamic viscosity of immersion solution, and $r$ is the radius of the bead. Therefore, we can first obtain the diffusion coefficient by measuring the slope of the fitted line in Fig. 6(b), and then the dynamic viscosity of the immersion solution can be calculated using the Stokes-Einstein relation. Here in our experiment, we repeated 6 measurements at a temperature of ~24 °C and the dynamic viscosity of the water media was calculated as $0.858\pm 0.035\text{cp}$, which is consistent with the standard value of 0.911 cp reported elsewhere [27]. To be noted, the spatial resolution of our system in the z axis is ~0.6 μm while that in the x and y axis is ~0.3 μm, which makes it difficult for our system to accurately determine the 3D position of a bead when the motion is less than 0.6 μm. This could explain why the plotted line of MSD in Fig. 6(b) is not perfectly straight when MSD is small.

Fig. 6.

Validation of the capability of high-speed 3D RI imaging. (a) shows the 3D trajectory of a 2 μm silicon dioxide bead in the water; (b) shows the corresponding MSD plot as a function of lag time and its fitted line by doing a linear regression.

Besides accurate 3D RI mapping free of diffraction noise, the proposed DMD-based ODT system also features superior imaging speed. As mentioned earlier, most ODT systems developed over the past decade rely on scanning galvanometric mirrors for illumination angle control. The highest reported angle scan speed has been 4500 Hz for single axis angle scanning [28] and 1000 Hz for two axis angle scanning [29,30]. In general, two axis angle scanning is always preferred as it enables more complete frequency sampling of the Ewald sphere. Since galvanometric mirrors must be held stationary during the image acquisition for each illumination angle, the imaging speed is constrained by the settling time of the mirrors. Therefore, ~1000 frames/sec is the upper limit for imaging acquisition speed for galvanometric mirror based ODT system. Since higher imaging speed is desirable for observing high-speed 3D dynamics of biological samples, the DMD-based ODT systems are most suitable to meet such the high-speed imaging needs (DMD speed upper limit ranges from 4 to 50 kHz for various models) due to its extremely small settling time of ~10 μs.

For the sake of demonstration, interferograms were recorded at 1.5 kHz, which is among the highest measurement speed, to the best of our knowledge, for angle-scan based ODT so far. Since we used only 57 interferograms, corresponding to various illumination angles, to reconstruct one RI tomogram, a video rate 3D imaging speed of ~26 tomograms per second was achievable. Currently, our image frame rate is limited to 1.5 kHz by the laser power and camera frame rate. By choosing an appropriate laser source with more power and a higher speed camera, our system can possibly allow ~9 kHz image acquisition for 2D holograms, which would correspond to 3D imaging speed of ~160 tomograms per second. With time lapse measurements, we can observe fast-evolving 3D cell dynamics in many applications such as bacteria swimming, RBC oxygenation modeling, and cell response to dramatic culture environment changes. Furthermore, this speed is also critical for implementing high-throughput 3D imaging cytometry, where an automatic translational sample stage and an image stitching algorithm if developed, will allow us to image thousands of cells within a short time.

4. Conclusion

In this paper, we have presented a novel dynamic spatial filtering method based on a digital micromirror device. Current emerging ODT systems utilize DMD for fast illumination angle control but suffer from the diffraction noise that results in a deteriorated imaging quality for the reconstructed RI tomograms. Using our proposed method, we have successfully removed all the diffraction noise, while still maintaining the high-speed operation capability of the DMD. In order to validate the feasibility of the presented method, we have measured the 3-D RI distributions of colloidal and biological samples at the speed of ~26 tomograms/sec. We note that the system throughput can be easily increased by a factor of ~6-7 by simply choosing a higher power laser source and a faster camera. Finally, the reported high-speed ODT system is widely applicable to a range of biological applications involving fast 3D cell dynamics such as structural changes in sickle red blood cells during deoxygenation.

Funding

National Institutes of Health (NIH) (1R01HL121386-01A1, NIH9P41EB015871-26A1); The Chinese University of Hong Kong Direct Research Grant (Project Code 4055080); Hamamatsu Corp; Singapore-MIT Alliance for Research and Technology Centre (SMART).