Abstract
Lens-free digital in-line holography (LDIH) is a promising technology for portable, wide field-of-view imaging. Its resolution, however, is limited by the inherent pixel size of an imaging device. Here we present a new computational approach to achieve sub-pixel resolution for LDIH. The developed method is a sparsity-based reconstruction with the capability to handle the non-linear nature of LDIH. We systematically characterized the algorithm through simulation and LDIH imaging studies. The method achieved the spatial resolution down to one-third of the pixel size, while requiring only single-frame imaging without any hardware modifications. This new approach can be used as a general framework to enhance the resolution in nonlinear holographic systems.
Lens-free digital in-line holography (LDIH) is an emerging technology for point-of-care medical diagnosis1,2,3,4. The technology records holograms of objects directly placed on imagers, and digitally reconstructs the images of the objects. With no intermediate optical components between objects and imagers, LDIH is simple, compact, cost-effective, and capable of large field-of-view imaging. Recent applications of the LDIH technique has shown its potential for biomedical applications by molecularly profiling cancer cells3 and morphologically discerning a wide range of organisms (e.g., bacteria, blood cells, C. elegans, plankton)1,5,6.
The spatial resolution of LDIH is limited by the physical size of individual sensing elements (pixels) on a semiconductor imaging sensor (an imager). Spatial information smaller than a pixel is lost in the imaging process, because the recorded holograms are sub-sampled versions of the true hologram. While imagers with smaller pixels could improve resolution, the approach is challenged by practical issues such as prohibitive device cost, heating from high density pixels, and often reduced field-of-view. Alternatively, new methods for resolution enhancement have been explored. For example, a sequence of low resolution images were numerically reconstructed7,8,9; the pixel function of an imaging sensor was predetermined and a deconvolution was used to improve spatial resolution10. Although these methods achieve sub-pixel resolution, they require additional hardware components, and increase the complexity of image acquisition.
Compressed sensing (CS) is a new signal recovery method11,12 that has been applied to super-resolution imaging on a wide range of platforms. Based on the sparse nature of images, CS can improve spatial resolution from under-sampled measurements11,12. This approach is particularly appealing for hologram reconstruction, because it entails no modification to the optical setup and reduces data acquisition time. CS has been applied to conventional holographic systems, including off-axis holography, multiple view projection holography, and Fresnel holography5,13,14,15,16. Directly applying CS algorithms to LDIH, however, is a challenging task. LDIH is inherently a nonlinear system; unlike conventional holography, which preserves both the intensity and the phase of light, LDIH records the light intensity only. Conversely, most CS algorithms require linearity and random sampling to accurately reconstruct data, and thereby are incompatible with LDIH17,18.
Here, we report on a new computational framework for super-resolution on LDIH systems. This approach uses the sparse nature of images to recover high frequency components. Specifically, we devised an algorithm compatible with non-linear LDIH by adopting the L0-norm minimization. The developed algorithm achieved higher resolution (up to 300%) than conventional reconstruction methods, recovering both amplitude and phase information at sub-pixel resolution. We validated the approach through numerical simulation, and applied it to images acquired with a LDIH system. The algorithm required only a single image taken with uniform sampling, and could be readily combined with existing LDIH systems without any hardware modification. Such features could facilitate the point-of-care application of LDIH by enabling simpler optical setup and faster image acquisition. Furthermore, this method can be used as a general framework to enhance resolution in non-linear holographic systems.
Results
System setup
The LDIH imaging system (Fig. 1A) has simple optics, consisting of a light-emitting diode (LED), a pinhole, and a semiconductor imaging sensor. The pinhole is used to generate a coherent spherical wavefront. The interference between the incident light and the scattered light from the sample is the observed hologram19. The imager is positioned directly underneath the sample to record the hologram, and the object image is numerically reconstructed (see Methods for details). Without a lens between the sample and the imager, the system has a unit magnification to offer a wide field-of-view3. The imaging resolution, however, is limited by the discreteness of imager pixels; features smaller than the pixel size are under-sampled and only blurred images can be reconstructed.
Sparsity algorithm for LDIH
We reasoned that the L0-norm minimization, which is compatible with non-linear systems, could be adapted to improve the spatial resolution from LDIH. The L0-norm minimization requires a sparse input signal to reconstruct the missing information. As an input, we thus chose the object image which is generally sparser than the holographic pattern. The relationship between the observed hologram (y) and the object image (x) was modeled as y = |B·H·x|, where H denoted the hologram operator. B is a low-pass, or blur, filter that accounts for the down-sampling effect coming from the discrete size of the imager pixels. We then solved the following optimization problem:
where ||x||0 is the nonzero-counting L0-norm of x, namely the number of nonzero elements in vector x, which also represents the sparsity of the signal; ||y − y0||22 (=ε) is the difference between the measured (y0) and the estimated (y) holograms; and ε0 is the error threshold. This optimization aims at finding the sparsest possible solution of x (i.e., reconstructed image) but penalizes estimations that deviate significantly from the measured hologram by imposing ε < ε0.
To numerically solve Eq. (1), we developed a two-constraint iterative scheme (Fig. 1B), which finds the optimal minimization of the L0-norm (||x||0) while maintaining data consistency by minimizing L2-norm (||y − y0||22) at each step. As an initial input, the algorithm uses an object image (x0) reconstructed from a measured hologram (y0) via diffraction theory (see Methods for details). It then executes an iterative process. The first step, thresholding, removes a pixel in the object image (x), whose value is closest to the background. This reduces the number of non-zero elements in x and thereby results in a smaller ||x||0 (L0-norm minimization). The second step, shrinking, finds the optimal x that minimizes ε (the change in the hologram), while keeping ||x||0 constant (L2-norm minimization). The found x, which is sparser than the initial input, is then used as a new input. The algorithm exits the iteration when ε is smaller than the predetermined threshold (ε0), yielding the final object image (xε).
Algorithm optimization
To handle the large number of pixels within an image, we adopted the Limited-memory Broyden–Fletcher–Goldfarb–Shannon (L-BFGS) method, which can solve large scale nonlinear minimization problems20. L-BFGS is a quasi-Newton algorithm designed for optimization problems with large number of variables. We applied L-BFGS to find x that minimizes ε (=||y − y0||22). Note that L-BFGS enforces data consistency between the measured (y0) and the newly estimated (y) holograms; sparsity is maintained through the L0-norm constraint. We also implemented a fast-computing algorithm based on TwIST (Two-step Iterative Shrinking and Thresholding) to improve computational speed for wide field-of-view images21.
We found that a key criterion for stable computation was to establish an optimal exit condition (ε < ε0) from the iterative loop. As a quantitive measure of precision (convergence), we defined the difference (ε) from a pixel-by-pixel comparison:
where y(m, n) and y0(m, n) are the components of y and y0, respectively, at the spatial coordinate index (m, n). M (and N) is the total number of rows (and columns) in the hologram matrix. We then set the the threshold (ε0) such that the average precision for all pixels was >98%; with this condition, all test images converged to their final solution.
Figure 2 shows an example of the iterative process using a simulated object. The test pattern (Fig. 2A-i) had the smallest feature size of 1.2 μm, and we calculated an ideal hologram (no-loss). We then simulated the LDIH measurement with the imager pixel size of 2.4 μm (Fig. 2A-ii). This step produced a down-sampled hologram (y0) and its corresponding reconstructed image (x0). We next applied the sparsity algorithm using x0 and y0 as an input. With each iteration, the resolution of the spatial image progressively improved (Fig. 2A-iii). With the improvement of the spatial resolution, the difference (ε) between the simulated hologram (y) and the reconstructed hologram (y0) decreased (Fig. 2B). The algorithm finally recovered the original pattern, even though its inputs were blurred holograms (Fig. 2C). We also tested the algorithm in the presence of pixel noise. We added varying levels of Gaussian noise to all pixels in the hologram (y0). The algorithm could recover the original object even at the noise level of 5% in signal intensity (Fig. S1).
Spatial Resolution
We next determined the resolution enhancement achievable by the sparsity algorithm. As a true object image (xs), we positioned two objects at a distance smaller than the pixel size (Fig. 3A, left), and then simulated a measured hologram (y0 = |B·H·xs|) according to the discrete pixel size of the LDIH. The initial reconstruction of the object image (x0) through the Rayleigh diffraction (see Materials and Methods) could not resolve the objects (Fig. 3A, middle). The resolution of the LDIH is primarily determined by the physical pixel size, even though the holograms were up-sampled via linear interpolation. Subsequent application of the sparsity algorithm, however, could separate these objects (Fig. 3A, right). To find the resolution limit, we varied the distance between the objects as well as the simulated pixel size. The sparsity algorithm showed a sub-pixel resolving power (Fig. 3B), giving two- to three-fold increase in resolution across a range of detector pixel size (0.6–8 μm).
Simulation studies
We validated the developed algorithm through numerical studies. We first simulated binary images consisting of transparent and opaque patterns. Large or irregular patterns with the smallest feature size of 0.8 μm were used. Holograms were calculated and processed with a low pass filter (B) to simulate LDIH measurements with a 2.4-μm pixel imager. Conventional reconstruction produced blurred object images, as the high frequency signal was lost in the simulated input holograms. The sparsity-based reconstruction, on the other hand, could reliably recover the lost high frequency information, consistently achieving three-fold resolution enhancement (Fig. 4).
We further extended the algorithm to recover the object phase as well as the magnitude. The phase information is particularly relevant in biological applications, because it can enhance contrast between different objects of similar sizes (e.g., biological cells vs. labeling microspheres). To incorporate phase reconstruction, we used a complex object vector (x) whose components are
where A(m, n) and P(m, n) are components of the amplitude (A) and the phase (P) matrices, respectively, at the spatial position index (m, n). We modified the reconstruction algorithm to include both A and P matrices in its routine; the algorithm minimizes the number of nonzero elements both in A and P, and adjusts the pixel values in A and P to minimize ε. These matrices were not treated independently: A was used to determine the weakest pixel to remove, and the decision propagated to P.
We tested the extended algorithm using simulated objects with homogeneous but different phase values (Fig. 5A, left). Holograms were calculated assuming an imager with 2.2-μm pixel size. The extended algorithm accurately recovered the phase information (Fig. 5A, right). Note that the relative error decreased as the phase differences between the object and the background increased. We further simulated an object with varying phase distributions with feature size (1.1 μm) smaller than the imager pixel size (Fig. 5B). The algorithm reconstructed both amplitude and phase, while maintaining the capacity for sub-pixel resolution enhancement. The computational load was approximately twice of the amplitude-only version as the number of variables doubled.
Sub-pixel imaging with LDIH
We applied the sparsity algorithm to reconstruct actual LDIH images. We used a LDIH setup with a 2.2 μm pixel complementary metal-oxide-semiconductor (CMOS) imager to acquire holograms (see Materials and Methods). We first used a standard test pattern (USAF1951) to check the resolution enhancement. We acquired holograms of parallel line patterns, and compared reconstructed images (Fig. S2). The sparsity algorithm recovered all patterns that have varying widths and spacings. We next imaged complex patterns with sub-pixel line width. The objects were patterned on glass substrates via electron-beam lithography followed by metal deposition. Conventional, diffraction-based reconstruction (x0) failed to resolve winding geometries (e.g., ‘H’, ’S’; Fig. 6A). The sparsity-based processing (xε), however, restored the sub-pixel patterns, and achieved the resolution close to reference images acquired by a bright field microscope with a 100 × objective lens.
High resolution imaging of biological objects
We further applied the sparsity algorithm to objects with varying phases. Initial tests employed simple spherical objects: polystyrene microbeads (diameter, d = 4 μm), silica microbeads (d = 5 μm), and leukocytes (d = 10 μm). Following the hologram acquisition, we used the extended sparsity algorithm to recover both amplitude and phase information. The reconstructed images had high contrast, allowing us to distinguish these objects (Fig. 7A). Note that the large field-of-view (24 mm2) of LDIH, combined with high resolution reconstruction, enabled us to detect >104 micrometer-scale objects in a single image acquisition (Fig. S3).
We next prepared mixed-phase objects by labeling cells with polystyrene microbeads conjugated with antibodies (see Materials and Methods). The labeled cells produced non-symmetric, complex diffraction patterns (Fig. 7B, left). In the reconstructed images (Fig. 7B, middle), however, cells and beads were clearly identified, which was further validated by bright field microscopy (100 ×; Fig. 7B, right). We could also differentiate the nucleus from cytoplasm in a cell (Fig. 7C). To enhance the optical contrast, we stained cell nuclei with a chromophore (see Materials and Methods). As the chromophore reduced the optical transmittance, cell nuclei appeared darker than cytoplasm in the reconstructed image.
Discussion
We have developed a sparsity-based algorithm for LDIH resolution enhancement. This new method can improve the resolution of LDIH down to one-third of the imager pixel size, and recovers both the magnitude and the phase of objects. Importantly, the sparsity method provides sub-pixel resolution enhancement from a single-frame measurement, without requiring additional hardware or image acquisition steps. The algorithm empowers the LDIH system to achieve the imaging capacity close to high-resolution microscopy, and is effective for imaging sensors with a broad range of pixel size.
Sparsification of images generally has a sharpening effect, which, if unrestricted, could introduce artifacts. We minimize such risks by enforcing data consistency (||y0 − y||22) between the measured (y0) and the estimated holograms (y = |B · H · x|) while fulfilling a sparsity constraint (||x||0). This approach is key to achieving high resolution that conforms to true object images. In the current application, we used a sparse object image (x) as an input. The developed algorithm, however, could also be applied to less-sparse images. In those cases, the sparsity requirement can be fulfilled with a basis (Φ) transformation, such that Φx is sparse instead of x itself. Wavelet transformations, such as Daubechies Wavelets, are extensively used methods for such purposes22,23.
With the developed algorithm, LDIH could achieve a resolving power close to that of high magnification optical microscopy (100 ×) while retaining the benefit of a large field-of-view. This is a significant advance from our previous system whose resolution was comparable to that of 20 × microscopy3. The high resolution and the capacity to recover object’s optical property could lead to many biological applications. For example, LDIH could be a platform for multiplexed cellular analysis. We could simultaneously profile multiple surface markers by labeling cells with microbeads of different sizes and optical signatures. Furthermore,intracellular markers could be detected after staining with chromophores.
We anticipate that the present work will provide a framework for applying sparsity-based methods to nonlinear holographic systems. The developed method can be further improved by i) adopting other up-sampling processes (e.g., floating pixel) to prepare an improved initial input (x0) to the sparsity algorithm24; ii) expanding the algorithm to process non-sparse samples through mathematical transformations (e.g., wavelet transformation); and iii) porting the numerical routine to parallel computing (e.g., graphic processing unit) to enable real-time image reconstruction. Such developments would significantly aid in the point-of-care applications of LDIH platforms, allowing for faster and more accurate diagnostics.
Materials and Methods
Construction of a LDIH system
The LDIH system consisted of a light-emitting diode (LED), a pinhole, and an imager. The illumination central wavelength of LED was 405 nm. A pinhole (aperture diameter, 100 μm) was placed in front of the LED to generate a coherent spherical wavefront. Samples were positioned 10 cm under the pinhole and 1 mm above an imager (Aptina Imaging MT9P031). The imager had 2592 × 1944 pixels with the pixel size of 2.2 × 2.2 μm2.
Preparation of test patterns
Samples with test patterns were created on glass substrate. Patterns were transferred to the slides via electron-beam lithography (Raith-150, Raith) followed by metal deposition (Ti, 10 nm; Au, 100 nm). The size of patterned features were measured via scanning electron microscopy (Ultra 55, Zeiss).
Cell labeling with microbeads
SkBr3 human cancer cells were labeled first with biotinylated anti-EpCAM antibody followed by streptavidin-coated polystyrene particles (7 μm diameter, Spherotech), each for 10 min at room temperature. Leukocytes were prepared from 0.6 mL blood samples mixed with 12 mL BD Phosflow Lyse/Fix buffer (1 ×) for 15 min at 37 °C. The cells were resuspended in PBS.
Cell staining
5 × 105 lymphoma cells (Daudi) were placed on a glass slide. Adhered cells were fixed and permeabilized using Foxp3/Transcription Factor Staining Buffer Set (00-5523-00, Affymetrix) per manufacturer’s protocol. Subsequently, cells were labeled with anti-Ki67 antibody (556003, BD) at 4 °C, followed by 3 × wash and incubation with biotinylated anti-mouse secondary antibody (BA-2000, Vector Laboratories). Cells were washed again and incubated with ABC peroxidase (PK6100,Vector Laboratories). Finally, AEC + substrate-chromogen (K3464, Dako) was added for color development.
Diffraction-based reconstruction
The reconstruction routine that we previously developed for a smartphone-imaging system3 was adopted to the LDIH system. In brief, we recorded two holograms, one without samples (a reference image) and the other with samples (a target image). A normalized hologram was then obtained by calculating a pixel-by-pixel ratio between the target and the reference images. The normalized hologram was up-sampled four times through linear interpolation, and entered an iterative phase-retrieval routine25,26. First, the normalized hologram was numerically back-propagated for an optical distance between the object and the imager planes. The field propagation was based on the Rayleigh–Sommerfeld diffraction theory; the optical field was calculated by the inverse Fourier transform of the multiplication between the Fourier transform of field and the transfer function. Second, physical constraints (optical transmission ) were applied to the back-propagated image to correct for artificial twin image. Third, the updated image was propagated to the imager plane to produce an updated hologram with nonzero phase information. The process was then repeated using the updated hologram as a new input. The iteration stopped when retrieved phase information converged. A typical iteration number was 10~30.
Sparsity-based reconstruction
We created a custom-designed package (in MATLAB) to perform the resolution enhancement. The measured hologram and the object image obtained through the conventional reconstruction (described above) were used as an input to the sparsity-based processing. The core of the package was the sparsity-based reconstruction, an iterative program seeking to find the optimized object image. The optimized object image was determined by two constraints: it needs to be sparsest possible solution which minimizes the error between the newly estimated hologram and the measured hologram. Through the iterations, the image sparsity was gradually decreased by merging the weakest pixel of each iteration into background. After each change in image sparsity, the program estimated a new hologram, and minimized the error value between estimated and measured holograms by adjusting the intensity of non-background pixels. The computation was run on a workstation with Intel® Xeon® EP processors and 64 GB memory (Lenovo ThinkStation S30).
Additional Information
How to cite this article: Song, J. et al. Sparsity-Based Pixel Super Resolution for Lens-Free Digital In-line Holography. Sci. Rep. 6, 24681; doi: 10.1038/srep24681 (2016).
Supplementary Material
Acknowledgments
We thank R.M. Westervelt (Harvard), C. Vinegoni, and P.F. Feruglio for helpful discussions. This work was supported in part by NIH Grants R01HL113156, R01EB004626, R01EB010011, HHSN268201000044C, U54-CA119349, T32-CA79443, 1K99CA201248-01 and F32-CA192531; and Department of Defense Ovarian Cancer Research Program Award W81XWH-14-1-0279.
Footnotes
Author Contributions J.S. and H.L. conceived the research. J.S. and C.L.S. developed the sparsity algorithm. J.S., C.L.S., H.I., S.J., D.P., Y.I. and M.P. performed the experiments. J.S., C.L.S., H.I., D.P., R.W. and H.L. wrote the manuscript.
References
- Garcia-Sucerquia J. et al. Digital in-line holographic microscopy. Appl. Opt. 45, 836–850 (2006). [DOI] [PubMed] [Google Scholar]
- Greenbaum A. et al. Imaging without lenses: achievements and remaining challenges of wide-field on-chip microscopy. Nat. Methods 9, 889–895 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]
- Im H. et al. Digital diffraction analysis enables low-cost molecular diagnostics on a smartphone. Proc. Natl. Acad. Sci. USA 112, 5613–5618 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
- Xu W., Jericho M. H., Meinertzhagen I. A. & Kreuzer H. J. Digital in-line holography for biological applications. Proc. Natl. Acad. Sci. USA 98, 11301–11305 (2001). [DOI] [PMC free article] [PubMed] [Google Scholar]
- Marim M., Angelini E., Olivo-Marin J. C. & Atlan M. Off-axis compressed holographic microscopy in low-light conditions. Opt. Lett. 36, 79–81 (2011). [DOI] [PubMed] [Google Scholar]
- Seo S. et al. High-throughput lens-free blood analysis on a chip. Anal. Chem. 82, 4621–4627 (2010). [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bishara W. et al. Holographic pixel super-resolution in portable lensless on-chip microscopy using a fiber-optic array. Lab Chip 11, 1276–1279 (2011). [DOI] [PMC free article] [PubMed] [Google Scholar]
- Guo K., Dong S., Nanda P. & Zheng G. Optimization of sampling pattern and the design of Fourier ptychographic illuminator. Opt. Express 23, 6171–6180 (2015). [DOI] [PubMed] [Google Scholar]
- Zheng G., Lee S. A., Yang S. & Yang C. Sub-pixel resolving optofluidic microscope for on-chip cell imaging. Lab Chip 10, 3125–3129 (2010). [DOI] [PubMed] [Google Scholar]
- Greenbaum A. et al. Increased space-bandwidth product in pixel super-resolved lensfree on-chip microscopy. Sci. Rep. 3, 1717 (2013). [Google Scholar]
- Candès E. J. Compressive sampling. Proc. Intern. Congr. Math 3, 1433–1452 (2006). [Google Scholar]
- Donoho D. L. Compressed sensing. IEEE Trans. Inform. Theory 52, 1289–1306 (2006). [Google Scholar]
- Rivenson Y., Stern A. & Javidi B. Compressive Fresnel Holography. J. Display Technol. 6, 506–509 (2010). [Google Scholar]
- Rivenson Y., Stern A. & Rosen J. Compressive multiple view projection incoherent holography. Opt. Express 19, 6109–6118 (2011). [DOI] [PubMed] [Google Scholar]
- Liu Y. et al. Scanning-free compressive holography for object localization with subpixel accuracy. Opt. Lett. 37, 3357–3359 (2012). [DOI] [PubMed] [Google Scholar]
- Cull C. F., Wikner D. A., Mait J. N., Mattheiss M. & Brady D. J. Millimeter-wave compressive holography. Appl. Opt. 49, E67–82 (2010). [DOI] [PubMed] [Google Scholar]
- Szameit A. et al. Sparsity-based single-shot subwavelength coherent diffractive imaging. Nat. Mater. 11, 455–459 (2012). [DOI] [PubMed] [Google Scholar]
- Bruckstein A. M., Donoho D. L. & Elad M. From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images. SIAM Rev. 51, 34–81 (2009). [Google Scholar]
- Fung J. et al. Measuring translational, rotational, and vibrational dynamics in colloids with digital holographic microscopy. Opt. Express 19, 8051–8065 (2011). [DOI] [PubMed] [Google Scholar]
- Liu D. C. & Nocedal J. On the limited memory BFGS method for large scale optimization. Math. Program. 45, 503–528 (1989). [Google Scholar]
- Bioucas-Dias J. M. & Figueiredo M. A. A new twIst: two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans. Image Process 16, 2992–3004 (2007). [DOI] [PubMed] [Google Scholar]
- Daubechies I., Han B., Ron A. & Shen Z. Framelets: MRA-based constructions of wavelet frames. Appl. Comput. Harmon. Anal. 14, 1–46 (2003). [Google Scholar]
- Lustig M., Donoho D. & Pauly J. M. Sparse MRI: The application of compressed sensing for rapid MR imaging. Magn. Reson. Med. 58, 1182–1195 (2007). [DOI] [PubMed] [Google Scholar]
- Edo T. B. et al. Sampling in x-ray ptychography. Phys. Rev. A 87, 053850 (2013). [Google Scholar]
- Fienup J. R. Phase retrieval algorithms: a comparison. Appl. Opt. 21, 2758–2769 (1982). [DOI] [PubMed] [Google Scholar]
- Mudanyali O. et al. Compact, light-weight and cost-effective microscope based on lensless incoherent holography for telemedicine applications. Lab Chip 10, 1417–1428 (2010). [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.