Digital pathology with Fourier ptychography

Abstract

Fourier ptychographic microscopy (FPM) is a recently introduced method of acquiring high-resolution, wide field of view (FOV) giga-pixel histology images. The FPM procedure first acquires a sequence of low-resolution images of a sample under variable-angle illumination. It then combines these images using a novel phase retrieval algorithm to improve the employed microscope’s resolution beyond its conventional limit. Here, we first describe how FPM’s resolution improvement can enhance wide FOV histology imaging. Second, we show that FPM also records a thin sample’s optical phase, which can help pathologists digitally extract as much information as possible from a given histology slide.

1. Introduction

Conventional digital microscopes are limited in the number of resolvable pixels they can capture in a single image. Wide field-of-view (FOV) objective lenses cover a large sample area, but do so at poor resolution. Higher-resolution objective lenses capture more sample details, but only over a small region of a sample. Optical aberrations are the primary source of this fundamental tradeoff. It is only possible to optically correct for aberrations over a limited spatial region of a microscope’s image plane [1]. The limited number of resolvable spots that a lens is able to transfer the sample to image plane is often referred to as the lens space-bandwidth product (SBP). Independent of the employed digital sensor, most objective lenses are limited to a SBP of approximately 10–50 megapixels. Several recent camera architectures aim at overcoming this physical limit via jointly designed optics and digital processing [2, 3], but few have attempted a solution within a microscope setup.

The limited prior work examining microscope image resolution improvement all require significant modification to a conventional microscope platform’s optical layout, and typically operate only under highly coherent illumination [4–7]. Fourier ptychographic microscopy (FPM) instead acquires a sequence of low-resolution images under angularly varying LED illumination and combines them with a novel phase-retrieval algorithm to extend the SBP of a fixed digital microscope to upwards of 1 gigapixel. The output of a Fourier ptychographic microscope is a wide FOV (120 mm²), high-resolution (0.37 μm at the FOV’s center) image that also includes the sample’s phase content. The only physical modification needed to convert a conventional digital microscope into an FPM-capable device is the insertion of a square array of LEDs approximately 10 cm below the sample stage.

While a fully detailed account of FPM’s operation is included in prior work [8–10], we here offer just a succinct description before examining its utility specifically for digital pathology. The interested reader will find a more complete description of its optical and digital processing steps in the above cited work. First, the FPM process sequentially turns on one LED within the square array at a time and captures a set of microscope images (see diagram in Fig. 1(a)). More precisely, with n LEDs in the 2D array, each illuminating the sample with a plane wave at unique angle ( ${\theta }_j^x,{\theta }_j^y$), we capture a digital image I_j(x,y), for j ∈ [1, n]. We note that the resolution of each of these images, as with most microscope images, will be limited by the finite size of the objective lens aperture. However, each image in this set will contain different information about the sample surface, since each LED will shift a different segment of the light field emerging from the sample through the lens aperture.

Fig. 1.

The FPM setup. (a) LED array sequentially illuminates a sample from different directions, which is then imaged by a 2X microscope objective (MO) lens. (b) Actual FPM setup, showing the LED array and an inset of a single color LED. (c) Single image of resolution target with this 2X objective offers a wide FOV, but cannot resolve group 8 (~6 μm resolution). (d) FPM-reconstructed image resolves group 9 (~0.75 μm resolution).

Specifically, illuminating a thin sample with a plane wave at 2D angle ( ${\theta }_j^x,{\theta }_j^y$) shifts the narrow cone of wave-vectors emerging from the sample that can pass through the microscope objective aperture’s circular pass-band by ( $k_{0}sin{\theta }_j^x,k_{0}sin{\theta }_j^y$), where k₀ = 2π/λ. In the Fourier transform domain, each detected image may thus be represented as a set of spatial frequencies that must fall within a circular region of radius k*NA, whose center is shifted by a distance ( $k_{0}sin{\theta }_j^x,k_{0}sin{\theta }_j^y$) from spatial frequency (0,0). Here, NA denotes the microscope objective numerical aperture. After data acquisition, the goal of FPM’s second step is to computationally stitch these shifted spectra together into a final, high-resolution image.

In principle, if the microscope’s digital detector acquired the complex field (i.e., the optical field’s amplitude and phase) of each image U_j(x,y), then our high-resolution image reconstruction task would be straightforward. We would simply need to Fourier transform each acquired complex image U_j(x,y) into its spectrum 3̂_j(k_x,k_y), shift this spectrum by the unique offset caused by the j^th angled-LED, and then add it to a high-resolution spectrum guess of our sample, Ĝ(k_x,k_y). After all complex image spectrums are added to our guess Ĝ, we’d compute its inverse Fourier transform to yield a high-resolution, wide FOV complex sample image G(x,y). This task is directly connected to the concept of a synthetic aperture, where having the detected signal’s phase is vital to accurately switching between the spatial and Fourier domains.

Unfortunately, optical detector arrays measure an incoming field’s intensity, I_j(x,y) = |U_j(x,y)|², and cannot capture any phase information. While this makes our reconstruction method a bit more complex, it still follows closely in concept with what is outlined above. While details are provided in [8], our process of adding shifted spectra in the Fourier domain additionally incorporates an iterative procedure to converge upon an accurate estimate of the unknown sample phase. In short, it begins with an arbitrary high-resolution complex spectrum guess, Ĝ₀. For j ∈ [1, n], we then select band-pass region j from Ĝ₀, Fourier transform it to form image guess V_j, and replace the amplitude of image guess V_j with $\sqrt{I_j(x,y)}$, the known amplitude corresponding to band-pass region j. Leaving the phase unchanged, we then inverse Fourier transform back to create an improved complex spectrum guess Ĝ_j. This iterative process closely follows other phase retrieval procedures, and consistently converges to an accurate gigapixel complex image solution G_n when overlapping spectra are used.

As a baseline example of the above reconstruction procedure, Fig. 2 displays a single image of an Air Force resolution target captured by our microscope under illumination from the central LED using a 2X microscope objective. Here, the 2X objective lens’s small aperture limits the image resolution to approximately 6 μm. In Fig. 1(d), we show the result of an FPM reconstruction of the same resolution target after acquiring 137 different images, each under unique illumination from a single LED in the array. Resolution improvement over the single image in Fig. 1(c) is clear (from 6 μm to 0.7 μm). Across the entire resolution target FOV, the FPM reconstruction has increased the single image’s total number of resolvable pixels (i.e., SBP) from 23 megapixels to 900 megapixels. This enhancement in information throughput can come of great aid during acquisition of digital pathology images, as we discuss next.

Fig. 2.

Example grayscale FPM image of a small central area (0.6mm²) of histology slide. (a) Single image viewed in detail (bottom inset) exhibits a low resolution. (b) FPM recovers enhances resolution, and reconstructs the phase of the optical field exiting the sample as shown in (c).

2. Digital refocusing and aberration correction in pathology

As a by-product of digital resolution enhancement, the FPM recovery algorithm also solves for the phase of the optical field at microscope’s sample plane. In this section, we discuss three direct benefits of this acquired phase that broadly apply to most microscope samples. In the subsequent section, we turn our attention to a fourth application of the optical field’s phase that is specific to digital pathology – to learn about a sample’s scattering properties.

First, the phase of the optical field emerging from an imaged sample offers additional information not available in an intensity-only image. As is well known with other phase imaging microscope setups (e.g., differential interference and phase contrast microscopy), optical phase may be manipulated to improve the visibility of primarily transparent samples, and provides a direct indication of the optical path length difference between adjacent sample regions. As demonstrated in [9], FPM’s quantitative phase measurements similarly offer a direct indication of sample thickness. As shown in Fig. 2, this quantitative phase helps FPM reveal otherwise invisible structures contained within a pathology sample (e.g., as the black arrow indicates). Such a primarily transparent sample region exhibits a varying thickness that would be quite challenging to detect via direct intensity-only observation.

Second, FPM’s acquired phase can be used to digitally refocus images into sharp focus. As detailed in [8], we achieve digital refocusing by propagating the complex field solution towards or away from the image plane via a propagation simulator. Since digital refocusing can be applied to FPM data at any point after image acquisition, we anticipate that it may help to improve the efficiency of a pathologist’s diagnosis pipeline. For example, a slide that was originally slightly tilted or a histology sample that was not perfectly flattened between its slide and cover glass will not have to be re-imaged. Instead, a technician can computationally correct minor slide positioning errors at any later date using the acquired sample phase information. What’s more, by digitally refocusing any sample region not in sharp focus to its correct focal plane, FPM can significantly extend a microscope’s effective depth-of-field, e.g. from 80 μm to 300 μm for the setup demonstrated in [8, 9]. An example of using digital refocusing to bring a resolution target axially offset from the focal plane by +/− 150 μm is in Fig. 3. One may either use an a-priori estimate of the required defocus distance, as in Fig. 3’s example, or rely upon an image sharpness metric to establish an unknown defocus distance.

Fig. 3.

FPM enables digital refocusing. (Top) AF resolution target axially offset from the microscope’s focal plane captures raw images that exhibit defocus artifacts with respect to an in-focus raw image, shown in center. (Bottom) FPM reconstructions of the AF resolution target using 137 raw images exhibits resolution improvement, even though the original raw images are significantly defocused. For recovery at z=+/− 150 μm, we used digital propagation to refocus the complex recovered image by these a-priori known sample defocus distances, as detailed in [8].

Third, the FPM algorithm outlined above is only a starting point for more involved recovery procedures. As previously explored within the realm of conventional ptychography, more advanced computational methods can simultaneously estimate any errors shared between captured images of a sequentially illuminated sample. For FPM, these errors correspond to the microscope objective lens’s optical aberration map. As demonstrated in [10], it is possible to simultaneously estimate any of the FPM system’s inherent optical aberrations during image reconstruction. This aberration map may in turn be iteratively removed from a final reconstructed image to sharpen its spatial resolution, which is especially beneficial in off-axis regions of the image plane. Because both the sample and aberration estimates are complex (i.e., include optical phase), this removal is not ill-conditioned (e.g., like point spread function deconvolution from intensity-only images). What’s more, it requires no initial calibration data. Further details regarding this correction procedure may be found in [10].

3. Phase imaging to measure scattering parameters

FPM’s captured phase map can also extract additional information about the scattering properties of a histology sample. The scattering properties of a tissue sample are directly tied to its spatially varying refractive index profile. Such refractive index maps can successfully distinguish between healthy and cancerous cells in digitized histology slide images [11]. Cancer can manifest itself within the cell nucleus as a slight index of refraction shift, which is both difficult for the clinical pathologist to detect upon direct observation and also difficult for a computer algorithm to recognize from an intensity-only image. Furthermore, a phase-derived refractive index map maintains its utility even in the presence of uneven histology sample staining [12], thus suggesting it as a robust computational aid that can fit within pathology’s well-established clinical diagnostic workflow. Following, we demonstrate how to transform a complex FPM image into a spatial scattering coefficient map, which may in turn be used as a diagnosis aid.

Recent work has revealed two close links between the statistics of a sample’s phase map and its scattering parameters [13, 14]. First, the spatial variance across the phase map is linearly related to the sample’s scattering coefficient, μ_s. Second, the variance of its phase gradient is related to the sample’s reduced scattering coefficient, ${\mu }_s^{\prime }$. We will denote FPM’s output phase map, whose quantitative accuracy is verified in [9], as ϕ(x, y). As outlined in [13, 14], a simple sequence of calculations transforms ϕ(x, y) into an estimate of the scattering and reduced scattering coefficients assuming the sample we are imaging is thin:

$${\mu }_s(x_0,y_0)=\frac{1}{L}{\langle {(\varphi (x,y)-{\mu }_{\varphi })}^2\rangle }_{\mathrm{\Delta }x,\mathrm{\Delta }y}$$ (1)

$${\mu }_s^{\prime }(x_0,y_0)=\frac{1}{2k_0^{2}L}{\langle {\mid \nabla \varphi (x,y)\mid }^2\rangle }_{\mathrm{\Delta }x,\mathrm{\Delta }y}$$ (2)

In Eq. (1), L is the thickness of the sample, μ_ϕ = 〈ϕ(x, y)〉 is the mean of the phase map, and 〈·〉_Δx,Δy denotes an expectation value over a finite spatial window(Δx, Δy). In short, Eq. (1) indicates that the scattering coefficient is given by examining the variance of the phase map ϕ(x, y) within a spatial window (Δx, Δy) centered on pixel (x₀, y₀). In Eq. (2), ∇ represents the 2D gradient operator. It effectively states that a spatial average of the square of the phase gradient over a similar sized window yields a sample’s reduced scattering coefficient.

In the following two sections, we first verify the accuracy of Eq. (1) and Eq. (2) for FPM’s data by imaging a slide of microspheres where μ_s and ${\mu }_s^{\prime }$ are known a-priori. Then, we apply the same computations to a histology slide with unknown μ_s and ${\mu }_s^{\prime }$. Doing so, we aim to demonstrate how FPM’s acquired quantitative phase map can offer extra information that is otherwise missing from intensity-only imagery.

Before turning to this experimental verification, it is worth summarizing several of the important benefits that quantitative phase may bring the digital pathologist. For many years, unique phase-sensitive microscope setups like differential-interference-contrast and phase-contrast offered revealing images of otherwise transparent sample features [15]. However, these early optical modifications were designed to ensure that a human operator could quickly and qualitatively detect areas of phase variation via changes in image brightness. Their resulting images thus contain little quantitative information about a sample’s varying optical thickness. Several recent optical microscope designs, all requiring digital image detection and post-processing, have since changed this paradigm. FPM is included in a continually growing list of optical setups that can quantitatively extract a thin sample’s varying optical thickness at sub-micron resolution, allowing one to now extract new sample measurements via computation.

With quantitative phase, a pathologist can immediately determine a spatial map of a sample’s refractive index via several direct calculations [16]. Or with an a-priori refractive index estimate, one can compute the sample’s three-dimensional surface profile. These simple computations can be combined with additional color information to find the index’s dispersion curve [17], or as suggested here, its scattering properties. As already demonstrated for refractive index [11], such sample-specific parameters may aid in the computational diagnosis of unhealthy cells or tissue. Beyond sample characterization, though, quantitative phase may also lead to enhanced microscope image sharpness. As noted above, digital refocusing of a captured blurry image is only possible when the phase of the defocused optical field is quantitatively measured. This also holds for aberration detection and removal [10]. Thus, quantitative phase not only transmits new sample information to potential algorithms that may assist with pathology diagnosis, it may also help improve possible imperfections within the regular intensity images that pathologists are accustomed to viewing.

4. Materials and methods

The specifics of the FPM microscope are as follows (with details in [8]). The microscope body is an Olympus BX 41, the objective lens is a 2X apochromatic (Plan APO 0.08 NA, Olympus), and the digital sensor is a 5.5 μm pixel-size charge-coupled device (CCD) camera (Kodak KAI-29050). The LED array is inserted 8 cm beneath the sample, which contains a 32×32 array of red, green and blue surface-mounted LEDs (SMD 3528) of center wavelength 632 nm, 532 nm and 472 nm and a brightness of 0.7, 1.0 and 0.4 W/m², respectively. All colors offered an approximate 20 nm bandwidth. Each LED is approximately 150 μm in diameter, leading to a quasi-spatially coherent field with a coherence length of ~1 mm at the sample plane. To minimize artifacts due to a non-ideally coherent field, image processing is performed over smaller image segments in parallel, as discussed below and examined in detail in [18].

For all of the included data, 137 LEDs were used for illumination to capture 137 images raw images (5,280 × 4,380 pixels each). Only the LEDs from the array’s central 12×12 area were necessary to achieve our final goal of a gigapixel image. Note that 7 of the 144 central LEDs were not utilized due to their tendency to scatter light off the objective lens aperture. The total acquisition time is approximately 3 minutes, which can be shortened with brighter LEDs. Image processing is currently carried out on an Intel i7 CPU, which requires approximately 10 minutes per full-resolution grayscale image. Processing speeds may be accelerated using multiple GPUs. As noted above, each raw image was split into smaller segments before applying the FPM pipeline. Image splitting enables parallel computation, reduces memory requirements, and minimizes issues related to the finite spatial coherence length of each LED.

5. Results

Fig. 4 displays an example 500 × 500 pixel FPM-recovered phase map of a monolayer of 2 sizes of microspheres (r = 3 μm and 6.5 μm, n = 1.6) immersed in oil (n = 1.48). This limited resolution image is one cropped area of a larger image. Selecting one 6.5 μm microsphere of interest as shown at top, we first apply Eq. (1) to find that μ_s = 0.696 μm⁻¹ and Eq. (2) to find that ${\mu }_s^{\prime }=0.0239\mathrm{\mu }{\mathrm{m}}^{-1}$. Here, we have used the fact that λ = 632 nm, estimated the sample thickness as L = 6.5 μm, and fit one imaged microsphere within a (Δx, Δy) = 22-pixel window to perform our spatial average. These two values may be combined to find the anisotropy factor $g=1-{\mu }_s^{\prime }/{\mu }_s=0.966$. We can compare these scattering coefficients to predictions from Mie theory code [19] for a sparse set of spheres of similar size and refractive index, which yields μ_s = 0.658 μm⁻¹, ${\mu }_s^{\prime }=0.0241$ and g = 0.963. These predicted values match our experimental measurements closely, thus confirming the accuracy of our quantitative phase map as well as verifying the validity of Eq. (1) and Eq. (2). The same set of calculations applied to one selected 3 μm microsphere (shown in Fig. 4, bottom row) offers similarly accurate results.

Fig. 4.

(a) FPM phase map of microspheres, allowing us to verify the accuracy of our tissue scattering parameter estimation procedure. (b) Cropped regions of the phase image around two different-sized microspheres. (c) Gradient of the phase in (b). Variances from the regions in (b) and (c) are used to determine the microsphere’s scattering and reduced scattering coefficients following Eq. (1) and Eq. (2), respectively.

Given a validated method of estimating a thin sample’s scattering and reduced scattering coefficients, we next apply the technique on biological tissue. Fig. 5(a) includes an example FPM histology slide image, shown here in color after combining three separate reconstructions under illumination from a set of red, green and blue LEDs. Fig. 5(b) displays the phase map of a 1200 × 1200 pixel area of interest, from which we calculate the phase and phase gradient variance for each image pixel following Eq. (1) and Eq. (2). As with the microspheres, we again assume (Δx, Δy) is a 22-pixel window and the sample thickness is 5 μm (an estimation). The result of these calculations are the spatial maps of μ_s and ${\mu }_s^{\prime }$ shown in Fig. 5(c)–(d), where regions of homogenous scattering and specific points of inhomogeneity are clearly observable. Specifically, the reduced scattering coefficient map highlights a boundary where the sample’s scattering properties clearly change (to yellow and red in Fig. 5(d)’s false color). This sharp change is not clear within the quantitative phase map in Fig. 5(a), thus suggesting these forms of computed scattering maps may be useful as a method of highlighting tissue boundaries, either for enhanced visualization or automated segmentation.

Fig. 5.

(a) full color FPM gigapixel image of a histology slide (120 mm² FOV). (b) Recovered phase from a 0.2 mm² region at the sample’s center. (c) Scattering coefficient μ_s for this area of tissue, computed using Eq. (1). (d) Reduced scattering coefficient ${\mu }_s^{\prime }$ for the same area of tissue from Eq. (2), here shown on log scale. False color scales are noted below each image.

6. Conclusion

In conclusion, we have presented a new imaging modality that can acquire high-resolution, wide-FOV gigapixel digital pathology images without requiring any mechanically moving parts (i.e., without physically scanning the slide or objective lens). FPM’s acquired phase information can also provide valuable information about the sample’s local scattering and reduced scattering coefficients. Future work first aims to implement FPM with a high-NA microscope objective (e.g., a 20X objective) to reach near diffraction-limited resolution, albeit over a narrower FOV. In addition, extending the principle to an epi-illumination configuration may allow gigapixel imaging of biological samples in-vivo. In general, we hope that our included findings encourage the future integration of FPM into digital pathology’s diagnosis pipeline, as its numerous unique benefits over conventional microscopy are likely to continue to expand.

Biographies

Roarke Horstmeyer is a graduate student in the Electrical Engineering department at the California Institute of Technology. Prior to joining Caltech, he received an M.S. degree from the Media Lab at the Massachusetts Institute of Technology in 2011, and a B.S. degree in Physics from Duke University in 2007. His research interests include computational imaging, algorithm design and cryptography.

Xiaoze Ou was born in China in 1988. He received a B.E. degree in optical engineering from Zhejiang University, China, in 2011, and a Masters degree in electrical engineering from California Institute of Technology, United States, in 2013. In 2011, he started working in the Biophotonics Laboratory at the California Institute of Technology as a research assistant. His main research areas of interest surround large field-of-view, high-resolution microscope systems.

Dr. Guoan Zheng is an Assistant Professor at the University of Connecticut, with a joint appointment in the Biomedical Engineering and the Electrical Engineering departments. His expertise lies in optical engineering, biophotonics, computational imaging, and lab-on-a-chip devices. His current research interests include Fourier ptychography, high-throughput imaging technologies, super-resolution imaging, and the development of optofluidics and chip-scale imaging solutions. He earned his M.S. and Ph.D. in Electrical Engineering from Caltech in 2008 and 2013, respectively. He is the recipient of the Lemelson-MIT Caltech Student Prize in 2011, and the Caltech Demetriades Thesis Prize in 2013. He has published more than 30 journal papers.

Dr. Phil Willems is currently the CEO of ePetri Inc. He received his Ph.D. in Physics at the California institute of Technology in 1997. Over the past 20 years he has worked on a wide range of optical technologies, both in industry and academia. These include the cooling and trapping of neutral atoms, optical fiber telecommunications, precision metrology, thermal compensation of high power optics, fiber lasers, and biological microscopy.

Prof. Changhuei Yang joined Caltech in 2003. He is an Electrical Engineering and Bioengineering professor. He received the NSF Career Award, the Coulter Foundation Early Career Phase I and II Awards, and the NIH Director’s New Innovator Award. Discover Magazine included Prof. Yang as one of the top 20 scientists under 40 in their list of Best Brains in Science 2008. His research area is biophotonics—the imaging and extraction of information from biological targets through the use of light.