High numerical aperture Fourier ptychography: principle, implementation and characterization

Abstract

Fourier ptychography (FP) utilizes illumination control and computational post-processing to increase the resolution of bright-field microscopes. In effect, FP extends the fixed numerical aperture (NA) of an objective lens to form a larger synthetic system NA. Here, we build an FP microscope (FPM) using a 40X 0.75NA objective lens to synthesize a system NA of 1.45. This system achieved a two-slit resolution of 335 nm at a wavelength of 632 nm. This resolution closely adheres to theoretical prediction and is comparable to the measured resolution (315 nm) associated with a standard, commercially available 1.25 NA oil immersion microscope. Our work indicates that Fourier ptychography is an attractive method to improve the resolution-versus-NA performance, increase the working distance, and enlarge the field-of-view of high-resolution bright-field microscopes by employing lower NA objectives.

1. Introduction

The microscope is an essential tool in most life sciences labs. Its resolving power is mainly determined by the numerical aperture (NA) of its objective lens, defined as $NA=n\cdot \sin \theta$. Here, n is the refractive index of the medium between the sample and lens, and $\sin \theta$ is the lens acceptance angle. While switching to a higher-NA objective lens in a conventional microscope improves image resolution, it also introduces several undesirable effects. First, the image field-of-view (FOV) is correspondingly reduced. Second, high-NA (i.e., large $\theta$) lenses also require a short working distance, which can make sample manipulation challenging. Third, a way of further increasing the lens NA is through a higher refractive index, n. However, while the introduction of a liquid immersion medium can push the NA beyond unity, it also increases the risk of sample contamination and microscope damage. Fourth, as $\theta$ increases, so do lens aberrations, which become increasingly difficult to correct for [1]. This last problem is the primary reason why good quality high NA objectives are generally expensive – they contain complex stacks of lens elements to correct aberrations. In previous works [2,3], we found that aberrations can be significant even within the specified field-of-view of an otherwise well-corrected objective lens system, which prevents diffraction limited performance. Other groups have reported their works on characterizing the residual aberrations for high NA optical systems and achieving improved performance [4–7].

Fourier ptychography (FP) is a recently developed super-resolution technique that offers an alternative way to increase the NA of a bright-field microscope [8,9]. Instead of changing the objective lens and possibly applying an immersion medium, we use an LED array to provide angularly varying illumination and acquire a sequence of images. Each off-axis LED shifts different amounts of high spatial frequency information, diffracted from the sample, into the acceptance angle of a dry objective lens. FP then uses a phase retrieval algorithm to fuse each uniquely illuminated image into a final output image with increased resolution. This paradigm shift comes with two notable consequences. First, for a fixed desired resolution, FP operates with a lower-NA objective lens as compared with a conventional microscope. This increases both the imaging FOV and sample-lens working distance. Second, by working with a lower-NA objective lens, we will need to contend less with residual (i.e., uncorrected) aberrations to potentially achieve a resolution that is more closely matched to the NA-predicted value.

This second point is practically important, because it could significantly alleviate the need to design and construct complicated, multi-element lens systems with many components included just to minimize aberrations. Furthermore, our prior work with microscope systems [2,3] suggests that the presence of residual aberrations still notably impacts the experimental formation of accurate, wide FOV images.

In this current work, our prime focus is thus to systematically study the true resolution of high-NA objective lenses and determine whether FP offers a significant advantage by working with lower-NA lenses. Of particular interest to us is the question of whether a free space high-NA FPM can be implemented to give comparable resolution performance to a commercially available standard oil-immersion high NA microscope. Such a system can potentially transform the messy process of working with oil-immersion objectives to a cleaner and faster process.

Here, we accomplish such a study by working towards two related goals. First, we construct two unique FP microscopes (FPMs) with effective system numerical apertures, NA_sys, greater than unity (using a dry 20X 0.5NA and 40X 0.75NA objective lenses, respectively). Second, we benchmark the performance of our new FPMs against other commonly available high-NA microscopes, including oil-immersion setup. Due to its inherent use of controllable illumination and computation, a direct comparison of FPM to conventional oil immersion images is somewhat nuanced. In this study, we chose to characterize the resolving ability to discern two closely spaced holes/slits as our benchmark, as this provides us with a robust and quantifiable measure of performance.

Here is the outline for this paper. In section 2, we discuss how Fourier ptychographic resolution improvement is analogous to coherent aperture synthesis [10–18]. We work towards a simple relationship defining the expected resolution performance of FPM as a function of its objective lens acceptance angle and maximum illumination angle. We then introduce the two experimentally implemented high-NA FPMs. One synthesizes images from a 20X 0.5 NA objective lens into an image with 1.2 system NA resolution performance. The other synthesizes images from a 40X 0.75 NA objective lens into an image with 1.45 system NA resolution performance. In Section 3, we use these two FPMs to test our theory of coherent aperture synthesis. Each microscope images a series of two-slit targets to reveal a Sparrow resolution limit that closely matches the theoretically predicted Sparrow resolution limit for our synthetic numerical aperture model. Simultaneously, we evaluated the performance of a conventional microscope with a 20x 0.5 NA lens, a 40x 0.75 NA lens and a 100x 1.25 NA oil lens, and found a larger deviation between theoretical and experimental resolution limits. In Section 4, we apply our oil-free high-NA FPMs to image a biological target, and demonstrate the ability to resolve features that only an oil immersion objective lens could otherwise normally capture. Finally, Section 5 concludes by summarizing the various tradeoffs and benefits of our high-NA FPM technique.

2. FP principle and high-NA FPM setup

The resolution improvement of FP may best be understood by examining the operation of a microscope, equipped with an array of LEDs beneath it, in the spatial frequency domain. In this section, we will assume each illumination LED creates a quasi-monochromatic, spatially coherent field that illuminates the sample plane from a fixed angle. Details regarding this approximation and its implications are provided in Appendix A. Assuming spatially coherent illumination allows us to model the resolution limit of a single microscope image using a coherent transfer function (CTF). It is shown in Ref [19] that the CTF of an aberration-free infinity-corrected objective lens is given by the geometric shape of its aperture. Typically, a microscope’s aperture is a circle. Thus, the CTF in our microscope model will remain a circular low pass filter (Fig. 1, left). This filter’s cutoff spatial frequency, $k_c$, is defined by the lens numerical aperture, $N{A}_{obj}$, and the illumination wavelength: $k_c=2\pi \cdot N{A}_{obj}/\lambda$. Since this microscope model is rotationally symmetric about the optical axis, it will be helpful to limit our attention to a 1D optical geometry through the remainder of this discussion.

Fig. 1.

Principle of Fourier ptychrography. The CTF of the microscope objective is a low pass filter with cutoff frequency k_c. When the sample is illuminated by normal incident plane wave (yellow line), the spatial frequency of the sample in the range [-k_c, k_c] passes through the CTF to form an image. Illuminating the sample with a tilted plane wave with wavevector k_i (blue line) shifts the sample spectrum. The CTF now defines the image’s spatial frequency support as [-k_i-k_c, -k_i + k_c]. After image capture, a phase retrieval algorithm stitches together the spatial frequency information from the unique support of each image. The resulting FP reconstruction is expected to exhibit a cutoff frequency of $k_{\max }+k_c$, corresponding to an expanded system NA, $N{A}_{sys}=N{A}_{obj}+N{A}_{illu}$.

When a thin sample s(x) is illuminated by a quasi-monochromatic, normally incident plane wave, the resulting optical field at the objective lens aperture plane is proportional to $\hat{s}(k_x)$, the Fourier transform of s(x) [19]. Here, x is the spatial coordinate of the sample plane and $k_x$ is its Fourier conjugate variable. The spectrum $\hat{s}(k_x)$ is then low-pass filtered by the lens CTF, which limits the highest sample spatial frequency reaching the image plane to $k_c$, the lens cutoff frequency (Fig. 1, right). In other words, $k_c$ defines the finest resolvable microscope image feature when illuminating the sample with an on-axis LED.

An off-axis LED will instead illuminate the sample with an oblique plane wave with wavevector $k_i=2\pi \cdot n_{illu}\sin \varphi /\lambda$ . Here, $n_{illu}$ is the refractive index of the medium separating the illuminator and the sample, and $\varphi$ is the angle between the LED illumination and the optical axis. If the sample is thin (see Appendix B for further discussion on sample thickness), the optical field exiting its top surface, e(x), can be written as the product, $e(x)=s(x)\cdot \exp (i{k}_i\cdot x)$. We may again determine the resulting optical field at the aperture plane by taking the Fourier transform $\mathbb{F}$ of e(x), which results in, $\mathbb{F}\{e(x)\}=\mathbb{F}\{s(x)\cdot \exp (i{k}_i\cdot x)\}=\hat{s}(k_x-k_i)$, a spatially shifted version of the sample spectrum. Because of this shift, a different range (i.e., support) of the sample spectrum will now pass through the fixed lens CTF to the image sensor. Specifically, whereas in the above normally incident illumination case the CTF defines the image spatial frequency support as $[-k_c,k_c]$, now in the off-axis case the CTF defines a shifted spatial frequency support as $[-k_i-k_c,-k_i+k_c]$.

FP repeats this shift-and-capture imaging process for N off-axis LED’s, extending to a maximum off-axis LED angle of ${\varphi }_{\max }$. After data capture, FP has thus acquired N images, each originating from a distinct spatial frequency support. A phase retrieval algorithm then digitally fuses all N captured spatial frequency supports together [8]. If the algorithm converges correctly, the final resulting spectrum will thus lie within a contiguous spatial frequency window $[-(k_c+k_{\max }),k_c+k_{\max }]$, where $k_{\max }=2\pi \cdot n_{illu}\sin {\varphi }_{\max }/\lambda$ is the illumination wave vector from the maximum off-axis LED angle. If we define $N{A}_{illu}=k_{\max }/(2\pi /\lambda )=n_{illu}\sin {\varphi }_{\max }$ as the illumination NA, the synthesized NA (i.e., NA_sys) of the FPM system is given by the sum,

$$N{A}_{sys}=N{A}_{obj}+N{A}_{illu}$$ (1)

This system NA is analogous to the resulting NA of synthetic aperture setups [10–18]. Unlike a true synthetic aperture, FP does not measure the phase of each shifted optical field at either the aperture plane or the image plane. Its unique phase retrieval procedure instead allows us to stitch together each shifted spatial frequency window when only the resulting intensities at the image plane are known. Details regarding this digital recovery process are in [8].

The system NA in Eq. (1) allows us to directly predict the theoretical resolution limit of an FP microscope. It forms the basis by which we compute such limits in the following section. We caution that this system NA cannot be directly compared to the native NA of a conventional microscope objective lens. To calculate resolution in both scenarios, the illumination conditions and whether the measurement process is coherent or incoherent are additional factors that need to be considered.

Previous demonstrations of FPM only applied this aperture synthesis process to low-NA microscope setups [8,9,20]. To extend FPM to the high-NA case, we start from a conventional microscope with a 20X 0.5NA objective lens (Olympus UPLFLN 20X) and a CCD camera (Kodak KAI-29050). An array of LEDs arranged in concentric rings is used to provide variable off-axis illumination, as shown in Fig. 2(a). Each LED consists of three active areas with center wavelengths at 632nm (red), 522nm (green) and 471nm (blue), which can separately acquire three color channels for an RGB image. The outmost ring has a radius of 40 mm and contains 12 LEDs. Two more inner rings, each containing 8 LEDs, are arranged to ensure enough overlap in the Fourier domain with radii of 16 mm and 32 mm, respectively. An Adafruit 32X32 RGB LED matrix panel is used in our experiment, and 3 rings of LEDs are selected within the panel such that their distances to the panel center match with the aforementioned parameters. The LED array is placed 41 mm away from the sample, providing an illumination NA of 0.7 (${\varphi }_{\max }={45}^{\circ }$). The total spatial frequency support that this arrangement covers is shown in Fig. 2(b). The center red circle represents the pass band edge (i.e., the CTF) set by the objective lens numerical aperture. During FP capture, we sequentially turn on each of the 28 LEDs in the illumination array and acquire an image. The unique spatial frequency support of each image is denoted by a white circle. In the reconstruction process, all of the information within each white circle is fused together to reconstruct an image with support defined by the large green circle, which is our synthetic system NA. For the present case, NA_sys = 1.2.

Fig. 2.

High-NA FPM setup and synthesized Fourier domain spectrum. (a) Our primary high NA FPM system consists of a conventional microscope with a 20X 0.5NA objective lens and a ring illuminator, offering an illumination NA of 0.7. (b) Each captured image is merged in the Fourier domain, forming an enlarged passband. Center red circle: Fourier support of the original microscope; white circle: Fourier support of one LED; green circle: synthesized Fourier support of the FPM system. (c1) Known sample intensity; (c2) image captured by a conventional 20X microscope corresponding to red circle in (b); (c3-c4) two images captured with different off-axis LEDs on, corresponding to two of the white circles; (c5) FPM reconstruction, corresponding to the green circle.

Just like with a conventional microscope, FP can switch to a higher NA objective lens and achieve a higher system NA. Alternatively, we can move the LED array closer to the sample to form a higher illumination NA, so long as the overlap between each image’s spatial frequency support remains sufficiently large (greater than ~60% area overlap) [21]. In this paper, we select the former option to construct a second FPM system, now using a 40X objective lens with NA_obj = 0.75 (Olympus UPLFLN 40X) and the same illumination setup as discussed above (NA_illu = 0.7). Following Eq. (1), we expect this second FPM to synthesize a system NA of 1.45. With these two unique setups, we hope to first validate Eq. (1), and then test whether the experimental performance of FP, due to its utilization of low-NA, low-aberration lenses, can have a better performance than its high-NA conventional microscope counterpart.

3. Resolution calibration

In this section, we propose an experimental procedure to verify the resolution of each of our new high-NA FPM systems. We select this procedure both to help verify our synthetic aperture model in Eq. (1), and to fairly compare the resolution of FPM to the resolution of a conventional incoherently (i.e., Kohler) illuminated microscope.

As noted in Section 2, Fourier ptychography ideally functions as a coherent imaging system. Given each LED emits light of suitable temporal and spatial coherence (see Appendix A), the formation of each FP image simply involves a multiplication of the complex sample spectrum $\hat{s}(k_x)$ with a suitably shifted objective lens CTF in the Fourier domain, defining the image’s spatial frequency support. The computational goal of FP is to determine $\hat{s}(k_x)$ by correctly fusing together the image measurements from each of these uniquely shifted support regions. Section 2 argues that this goal is equivalent to the formation of a large, coherent synthetic CTF, CTF_sys, with a cutoff frequency defined through Eq. (1). CTF_sys is a complex function that completely defines the ideal performance of FP.

Unlike the incoherent optical transfer function that solely depends upon image intensities, CTF_sys is sensitive to the input light’s phase at each spatial frequency [19]. Thus, while it would be ideal for us to characterize each FPM by measuring its CTF_sys, required stability at sub-wavelength scales presents an experimental challenge (e.g., small sample imperfections or setup instabilities can lead to large measurement errors). Instead of measuring an entire CTF_sys, it is common to define microscope performance using a single cutoff metric [19,22].

The approach by which the resolution of a coherent imaging system ought to be quantified with a single metric is not a settled matter in literature. In broad terms, published quantifications fall into two camps. One major approach is to use the width of a point-spread function (PSF) (or some related variant), which is widely used to characterize incoherent imaging systems. Significant papers using this method include [23–26]. Proponents of the second camp rightly pointed out that a point-spread function-defined resolution can systematically overestimate the ability of a coherent imaging system to actually resolve two points in close proximity [27]. As such, they argue that a two-point resolution measurement ought to be the defining way to characterize resolution. On the other hand, a counter-argument may be made that for a target with two points that are out of phase with each other, coherent systems can be expected to do a better job of resolving two points than an incoherent one, and as such, the two-point resolution measurement method unfairly penalizes coherent systems [19].

Here, we choose to risk underestimating instead of overestimating resolution performance. We characterize system resolution by simply identifying the minimum separation between two points or lines that the system can resolve. While alternative resolution measures exist [22], this well-known two-point/slit criterion lends itself nicely to comparing the resolution performance of coherent and incoherent imaging systems. Specifically, we may use the same two-point/slit target to mark FP’s performance against typical incoherent standard microscopes. Since the quantification metrics for coherent and incoherent performance are connected to imaging system NA by different constant factors [22,28], we caution readers seeking to compare our achieved resolutions to those of other reported systems to exercise due diligence. Furthermore, for this target, coherent imaging systems, such as the FPM, can be expected to systemically fare worse than incoherent imaging systems, such as a standard microscope, as the light transmitted through the two-point/slit would have the same phase. In comparison, a target with more phase variations can be expected to perform better for coherent systems. This means that the resolution we expect to measure here for the FPM is a base resolution quantity. For actual practical samples, the FPM may actually do better in resolving features.

We construct our resolution targets by forming aperture pairs of different separation with a focused ion beam on a gold coated (100 nm thickness) microscope slide. When illuminated from below, each aperture pair forms our two points at a unique separation. We tested two different aperture pair geometries. In the first set, we fabricated each aperture as a round hole with a 200 nm diameter. In the second set, each aperture is a slit of width 180 nm and length 4500 nm. For both target types, we fabricated multiple targets with varying aperture center-to-center distances ranging from 300 nm to 740 nm. For both tested FPM systems, we found that the more light-efficient two-slit target set led to less noisy images, and thus more reliable resolution measurements. The two-slit targets form the focus of this section, while we present and discuss our similar two-hole resolution measurements in Appendix C. The scanning electron microscope (SEM) images of 15 different two-slit targets are shown in the first column of Fig. 3(a). We mount each target with a #1 coverslip (to simulate our mounting of a biological sample) before imaging.

Fig. 3.

Resolution calibration using customized two-slit targets, illumination wavelength λ = 632nm. (a) SEM, conventional microscope, and FPM images of the two-slit targets (180 nm width, 4500 nm length). (b) Line plots of vertical intensity distribution across both slits, showing a Sparrow resolution limit of 615 nm for 20X 0.5 NA objective (b1), 455 nm for 40X 0.75 NA objective (b2), 315 nm for 100X oil immersion 1.25 NA objective (b3), 385 nm for 1.2 NA_sys FPM system (b4), and 335 nm for 1.45 NA_sys FPM (b5). Line plots of about 81% dip-to-peak ratio are also shown for a rough estimation of Rayleigh resolution limit [43].

To measure the two-point resolution for our FPM system, we first illuminate each of the targets with a sequence of red LEDs (center wavelength = 632nm) and capture an image set. We then apply our FP phase retrieval algorithm [8] to reconstruct a high-resolution image from each image set. We execute this entire procedure for our 1.2 NA_sys FPM setup first, with the resulting reconstructions shown in the second-to-last column of Fig. 3(a). We then repeat this procedure with our 1.45 NA_sys FPM setup. These reconstructions are in the last column of Fig. 3(a). Each reconstruction displays the image intensity in pseudo-color.

Next, we use the Sparrow resolution criterion [29] to determine the cutoff resolution of our two FPM setups from their target image sets. The Sparrow resolution limit is defined as the distance between two points/slits where the dip in brightness between each peak vanishes in an image. Vertical line traces through each slit pair help identify this resolution cutoff, which we plot in Fig. 3(b4-b5). For the 1.2 NA_sys FPM setup, we see this intensity dip between the slit peaks decrease as the slit center-to-center distance decreases (Fig. 3(b4)). It vanishes at a center-to-center distance of 380 nm, which suggests the measured Sparrow resolution limit of this FPM is approximately 385 nm. The theoretically predicted Sparrow resolution of a coherent illuminated, diffraction limited imaging system with an NA of 1.2 for two-slit target, defined as d = 0.68λ/NA (Appendix D shows our derivation of a suitable Sparrow resolution equation), is 358 nm. Thus, with only an 8% deviation between measurement and theory, we find that this 1.2 NA_sys FPM coherent synthetic aperture (computed via Eq. (1)) closely adheres to the theoretical limit. Furthermore, as our Sparrow limit measurement relies upon images of multiple targets, we can immediately ascertain if one target is tilted or misaligned (i.e., setting one aperture out of phase with the other), ensuring this measurement is robust against experimental error.

We also search for the same intensity dip vanishing point within image traces taken with our 1.45 NA_sys FPM setup (Fig. 3(b5)). These traces exhibit a higher contrast than the corresponding traces from the 1.2 NA_sys FPM setup, as expected. The intensity dip now disappears at a center-to-center spacing of 330 nm. This suggests our measured Sparrow resolution limit is approximately 335 nm, which deviates by 13% from theory (296 nm) for a 1.45 NA coherent microscope. In both cases, the small difference between theory and experiment is attributable to a mismatch in nominal NA and aberrations within the microscope objective that are not accounted for, thus concluding that Eq. (1) is an accurate model.

For comparison, we also image the same set of two-slit targets with a conventional incoherent microscope setup. We test the resolution performance of three different objective lenses: a 20X 0.5 NA objective, a 40X 0.75 NA objective and a 100X 1.25 NA oil immersion objective (Olympus PLN 100X). For each, we illuminate the sample with a halogen lamp beneath a condenser (i.e., Kohler illumination with matched illumination NA [30], here we use Olympus U-AC2 condenser for 20X 0.5NA and 40X 0.75NA objective and Olympus U-AAC oil immersible condenser for 100X 1.25NA objective), and place a red filter (Thorlabs FB630-10) in the light path to match its spectrum to the FPM LED illumination spectrum.

Sample images from the conventional microscope are shown in Fig. 3(a), columns two to four. We plot a vertical trace through the two-slit intensity distribution for each of these sample images, shown in Fig. 3(b1-b3). Under Kohler illumination imaging, the theoretical Sparrow resolution limit is given as d = 0.44λ/NA (Appendix D). A comparison between the theoretical value and each of our measured Sparrow resolution limits for the three tested incoherent microscope objectives is in Table 1.

Table 1. Sparrow resolution for microscope systems (λ = 632nm, two-slit targets).

	System Parameter	Theoretical Sparrow resolution (nm)	Measured Sparrow resolution (nm)	Deviation from theory
Conventional Microscope	20X 0.5NA	556	615	11%
	40X 0.75NA	371	455	23%
	100X 1.25NA	222	315	42%
FPM 1.2 NAsys	0.5NAobj + 0.7NAillu	358	385	8%
FPM 1.45 NAsys	0.75NAobj + 0.7NAillu	296	335	13%

The measurements for the incoherent microscope objectives showed significant and increasing deviations from theory as the NA increases. This mismatch is likely attributable to the deviation of their practical NA from their nominal NA, which includes the negative impact of uncorrected aberrations. It is generally known that due in part to a larger deviation from the paraxial approximation, aberrations are harder to eliminate within higher NA lenses [1]. Perhaps, of more pertinent importance is the observation that the 1.45 system NA FPM achieved a measured Sparrow resolution that is comparable to that of an incoherent 1.25 NA oil-immersion objective.

Finally, we would like to point out that the slightly larger Sparrow resolution limit for the 1.45 NA_sys FPM compare to the 100X 1.25 NA oil immersion objective does not necessarily means a vaguer image for practical samples, since the phase relationship of the sample will have an influence on the coherent system’s resolution performance. This point will be further elaborated upon in the next section.

Our observations here remind us that when using a high-NA objective lens, the nominal NA value (as marked on the lens casing) is not necessarily the best indicator for imaging system cutoff resolution, due to its high measurement-to-theory resolution deviation. Instead, the precise value should be calibrate via a test target. We suggest that our two-slit target sequence is a simple and robust procedure offering accurate results. At the same time, these tests reveal that FP offers a well-controllable way to improve resolution performance while preserving the longer working distance, larger FOV and less-aberration-challenge benefits of lower-NA microscope objectives.

At this point, we would like to point out that resolution provides a convenient and objective way for comparing microscope performance. The overall image quality is much more difficult to quantify, if at all possible. In fact, image quality can differ not just between systems, but is also dependent on the samples that are examined. The strong diffraction fringes observable for the 1.45 NA_sys FPM in Fig. 3 is attributable to the sharp cutoff in transfer function associated with a coherent imaging nature of the FPM. The dropoff of the optical transfer function for an incoherent system (conventional microscope) is much more gradual [19]. This does not imply that a coherent system is inferior in general, because system performance is highly sample dependent. This point is well explained in [19] and illustrative examples can be found in Fig. 6.17 and 6.21 of the book. The subjectivity of image quality versus the objectivity of resolution quantification is the reason we chose resolution as the way to benchmark and quantify the performance of our system. In the next section, we will look at the various system image performance with an actual biospecimen.

4. Imaging performance

In this section, we demonstrate how our high-NA FPM systems may benefit a particular medical imaging scenario: the diagnosis of malaria-infected human blood. We prepare a sample slide containing malaria-infected blood cells by first maintaining erythrocyte asexual stage cultures of the P. falciparum strain 3D7 in culture medium, following the protocol described in [31]. Then, we smear these cultures on glass slide, fix them with methanol and stain them with a Hema 3 stain set (a modified Wright-Giemsa stain).

To image the stained cells with a conventional microscope, we use the same incoherent Kohler illumination as the previous section, but now without a spectral filter. To obtain color images of the cells via FPM, we repeat the FP capture and process steps three separate times using red, green and blue LED illumination from the same LED array, and then place each reconstruction in the appropriate color channel for the final color image in Fig. 4(a). We apply gamma adjustment to this final color image to diminish its difference in color with the conventional microscope images, caused by differences in the spectrum of the illumination light. We detail imaging performance in two image sub-regions, marked by red squares in Fig. 4(a). The same sub-regions from our 1.2 NA_sys and 1.45 NA_sys FPM reconstructions are in Fig. 4(e-f). The pebbly pattern in the cells on 1.2 NA_sys FPM image and the colorized pattern in the background on 1.45 NA_sys FPM images are mainly caused by the variation of brightness between LED elements and between RGB chips within an LED, which are not fully corrected in the reconstruction process. Images from the conventional color microscope setup, using the same three different objective lenses as noted above, are in Fig. 4(b-d). Image clarity increases as the objective lens NA increases, but at a sacrifice of a smaller field-of-view (marked for each objective lens with dashed circles in Fig. 4(a)) and a smaller working distance (noted in each lens diagram).

Fig. 4.

Microscope images of a malaria infected blood smear. (a) Full-sized 1.2 NA_sys FPM reconstruction, which maintains the FOV and working distance of the 20X objective. The FOV of the 40X and 100X objective are marked with black and blue circles, respectively. (b1-b2) Two sub-regions from (a) (marked with red squares) captured by the 20X objective, (c1-c2) 40X 0.75 NA objective lens, and (d1-d2) 100X 1.25 NA objective lens. (e1-e2) 1.2 NA_sys FPM, (f1-f2) 1.45 NA_sys FPM images of cells from the same sub-regions. A malaria infected red blood cell from sub-region 2 are further zoomed in, showing particles (pointed by arrows) that are clearly resolved by 1.45 NA_sys FPM and vaguely resolved by 100X oil immersion microscope.

The 1.2 NA_sys FPM’s image is sharper compare to the 40X 0.75NA conventional microscope setup, while the 1.45 NA_sys FPM images contain details that are not resolved in any of the other images. For one example, a malaria infected red blood cell from Fig. 4(a) sub-region 2 are further zoomed in, showing particles (pointed by arrows) that are clearly resolved by 1.45 NA_sys FPM (Fig. 4(f2)). In comparison, for 100X oil immersion microscope, part of these particles are vaguely resolved and part of them are not resolved (Fig. 4(d2)), while all the particles are not resolved in the rest of the microscope setups.

As noted earlier, the Sparrow resolution measurements for each of our FPM setups was performed on a slit pair. Light transmitted through both slits undergoes the same phase retardation. A coherent imaging system (such as the FPM) can be expected to underperform for such a target than in an incoherent system (such as a standard microscope). Conversely, if the transmissions are not in phase, the two-point resolution cutoff can outperform for a coherent system [19,32]. As such, our Sparrow resolution measurements for our FPM systems establish base resolution (underestimation) scores for FPM. In a sample with significant phase variations (such as blood cells), the FPM can be expected to provide better resolution performance. Finally, we again note that differences in the nature of the transfer functions between the two systems can lead to variations in the FPM and standard microscopy images.

The FP technique simultaneously acquires quantitative sample phase during high-resolution intensity image reconstruction [9]. We can use the reconstructed sample phase to simulate other modalities typically offered by microscope systems, such as differential interference contrast (DIC) or dark-field imaging. This simulation requires no physical modification to the imaging system. Figure 5(a1-a2) shows the intensity and phase from a small region of the blood smear sample image in Fig. 4, taken with the 1.2 NA_sys FPM under red LED illumination. Phase gradient images in both directions are shown in Fig. 5(b1-b2), which have similar appearance as what we will see under DIC microscope. Also, a simulated dark field microscope image assuming a 0.5 NA objective lens and condenser with 0.65-0.7 NA illumination ring is in Fig. 5(c).

Fig. 5.

The amplitude and phase from FPM images may be post-processed into different modality of microscope (a1-a2) 1.2 NA_sys FPM intensity and phase image of the blood smear sample in Fig. 4. (b1-b2) Phase gradient images (similar appearance as DIC image), (c) Simulated dark field image using the data in (a).

5. Conclusion

In this paper, we first described a new interpretation of Fourier ptychography as a coherent aperture synthesis technique, arriving at the conclusion that its synthesized system NA equals the sum of its objective NA and illumination NA. Then, we demonstrated for the first time an FPM system with an NA_sys over unity. This demonstration was performed on two unique setups: a 1.2 NA_sys setup formed by a 0.5 NA objective lens with a 0.7 illumination NA LED ring, and a 1.45 NA_sys setup formed by a 0.75 NA objective lens and the same LED ring. We verify the predicted synthesized aperture sizes for each FPM setup using a simple Sparrow resolution limit measurement, finding good agreement with theory. Performing the same Sparrow limit measurement with several conventional microscope configurations led to a larger mismatch between measurement and theory, attributable to larger uncorrected aberrations within higher-NA objective lenses. We further found that the 1.45 NA_sys FPM gave comparable resolution performance to an incoherent 100X 1.25 NA oil immersion objective standard microscope. Finally, we used our FPM system to obtain comparable or better color imagery of a biological sample than a conventional 100X oil immersion objective lens.

This study substantiates our initial conjecture that the use of lower-NA objective lenses in FPM can yield resolutions that are competitive with those of standard microscopes using higher NA objectives. Particularly intriguing is our experimental result showing that an FPM employing a 40X 0.75 NA objective can give comparable resolution to that of commercially available standard microscope with an oil-immersion 100X 1.25 NA objective. We would like to stress that the observed competitive performance of the FPM with the 100X oil immersion objective is attributable to the inability of the oil immersion objective to deliver NA-limited resolution, rather than some extraordinary FPM ability. In other words, a perfect aberration-free oil immersion objective can be expected to perform better.

As a whole, these findings indicate that high-NA FPM offers five primary experimental benefits over conventional high-NA microscope counterparts: a wider FOV, longer working distance, larger depth-of-field, an ability to measure sample phase, and a mitigation of the need for an oil immersion medium in certain situations. These five primary advantages come with certain costs. First, FPM must acquire multiple images over time. Second, it now operates only with thin samples, and in its current configuration cannot improve the resolution of fluorescent samples. Finally, its image recovery process is an inverse problem that can be computationally demanding for large data sets. That said, a number of alternative applications may immediately gain from the above five advantages, given they are unaffected by or can tolerate these costs. Examples include the study of bacteria [33], differential leukocyte counting [34], muscle tissue examination [35], and, as we briefly demonstrated, malaria diagnosis [36].

Several future experimental steps may help improve high-NA FPM. First, the LED ring array we used for sample illumination in both FPMs was optimized for the 1.2 NA_sys FPM design. The 1.45 NA_sys FPM, using a 0.75 NA objective lens, can benefit from an even higher illumination NA. Second, an embedded pupil function recovery algorithm [3] can be implemented to simultaneously estimate and remove lens aberrations from our final FPM reconstruction. This additional step may lead to improved image quality. Finally, we conclude that Fourier ptychography offers a consistent technique to improve the resolution of conventional microscope objective lenses across all magnifications, and has the potential to scale up to even higher-NA configurations than this work includes.

Appendix A: Fourier Ptychography as a coherent imaging system

Here, we briefly discuss the spatial and temporal coherence properties of the FP microscope. While the light from each FPM LED is neither fully temporally or spatially coherent, we now argue why it is suitable to approximate it as such, to first-order. This approximation is mathematically equivalent to reducing the full description of each LED illumination via its cross-spectral density (CSD) function, $c(x_1,x_x,v_u)$, to a single coherent field, $U(x)$, which is the form of our phase retrieval solution. First, to drop the dependence of the CSD upon v, the quasi-monochromatic criterion ($v/\Delta v>M$, with $\Delta v$ the spectral bandwidth and M the number of sensor pixels along one axis) should be ideally fulfilled [37]. This requires a $\Delta v$ of several nanometers, which a highly temporally coherent LED may satisfy. (This assumption is independent of the FPM setup’s objective NA).

Second, a spatially coherent field (i.e., a single coherent mode) removes the CSD’s statistical dependence upon two spatial coordinates (i.e., we may assume $x_1=x_2=x$ if a single coherent mode is present). To fulfill this criterion, the field’s spatial coherence length, given by the Van-Cittert Zernike theorem as $l=\lambda z/w$, should extend across the entire image FOV at the sample plane. Here w is the LED active area width and z is distance between the LED and sample. Given image FOV decreases with an increased NA lens, we can expect required spatial coherence conditions to relax in high-NA setups, such as those current demonstrated. The inability to satisfy either of the above temporal or spatial coherence requirements does not fundamentally limit the FPM technique. Several proposed algorithms identify and account for the unknown (spatial and temporal) source incoherence by working with functions resembling the CSD, albeit at the cost of additional computation and possibly more required measurements [38,39].

Under the above conditions, each raw image in FPM contains the magnitude of a spatially coherent field, and the phase retrieval-based combination of these images is identical to forming a synthetic coherent aperture. The output of an ideal FPM operation is the complex field at the sample plane $s(x)$ filtered by a synthetic coherent transfer function, $CTF(k_x)$ (the larger rectangle in Fig. 1), which defines our system performance. $CTF(k_x)$ is most fairly compared to the physical CTF of coherently illuminated, high-NA physical system that also detects phase. Thus, we model FP as a coherent imaging system with a synthesized system NA of $N{A}_{obj}+N{A}_{illu}$.

Appendix B: The influence of sample’s thickness on Fourier ptychography

In previous discussions, we assume the sample to be infinitely thin in the axial direction such that a tilt in illumination results in a shift of the sample spectrum in Fourier domain. For practical samples with finite thickness, we now introduce a three dimensional spatial frequency representation to further analyze the problem [18].

As shown in Fig. 6(a1-a3)

Fig. 6.

Three dimensional spatial frequency space analysis of Fourier ptychography. (a1-a3) Normal angle illumination case, (b1-b3) oblique angle illumination case. Solid line red circles and arcs represents spatial frequency information of the sample captured with the microscope system under certain illumination, and dot rings depict the cross-sections of Ewald spheres.

, when a sample is illuminated with a normally incident plane wave, a spherical shell shape fraction of the sample’s 3D spatial frequency spectrum can pass through the system and be captured by the camera. The radius of the sphere is determined by the wavevector of light $k_0=2\pi /\lambda$, and the radius of the cross section in k_x-k_y space is limited by the numerical aperture of the objective lens$k_c=2\pi \cdot N{A}_{obj}/\lambda$, as shown in Fig. 6(a2). The center of the grey sphere (known as Ewald sphere) is on the surface of the red sphere, which also has a radius of k₀. In case of normal incident plane wave illumination, the center is also on k_z axis, as shown in Fig. 6(a3). When the sample is illuminated by an oblique angle plane wave, the sample’s spectrum is going to shift with an amount of ($k_{ix},k_{iy},k_{iz}$). Equivalently, the spherical shell is shift for the same amount to the opposite direction, as shown in Fig. 6(b1-b3). Since the illumination wavelength is fixed, i.e. ${k_{ix}}^2+{k_{iy}}^2+{k_{iz}}^2=k_0^2$, the center of the grey sphere remains on the surface of the red sphere. Here, we define $(k_{px},k_{py})$ as the coordinates of the Fourier spectrum of the light field that passes through our imaging system, and $(k_x,k_y)$ as the coordinates of 3D sample spatial frequency spectrum. In this particular case, $k_x=k_{px}-k_{ix},k_y=k_{py}-k_{iy}$.

For an illumination wave with wavevector ($k_{ix},k_{iy},k_{iz}$), the Fourier spectrum of the light field that pass through the imaging system corresponds to a particular 3D sample spatial frequency spectrum. As we can see in Fig. 6(b3), the k_z component of the sample spectrum is:

$$k_z=\sqrt{{k_0}^2-k_{px}^2-k_{py}^2}-\sqrt{{k_0}^2-k_{ix}^2-k_{iy}^2}$$ (2)

From Eq. (2), we can see that the same $(k_x,k_y)$ sample information acquired by different illumination wavevector $(k_{ix},k_{iy})$ will have different k_z, which means simply stitching in the spatial frequency domain for aperture synthesis might not be valid for thick sample. Thus, prior research studies in synthetic aperture required the sample to be thin [16,17].

Lee et al. [18] reported their further study of the influence of sample’s thickness on synthetic aperture for transmission geometry. They states that for a thick sample with h as the crude maximum thickness, the sample Fourier spectrum $\hat{s}(k_x,k_y,k_z)$ can be approximated as $\hat{s}(k_x,k_y,0)$ for $\left|k_z\right|<\sigma \cdot k_0$, where $\sigma =\pi /(k_0\cdot h)$ defined as thin-sample limit.

In case of FPM system with $N{A}_{obj}$ and $N{A}_{illu}$, according to Eq. (2),

$${\left|k_z\right|}_{\max }=\max ({k_0}^2-\sqrt{{k_0}^2-{(k_0\cdot N{A}_{obj})}^2},{k_0}^2-\sqrt{{k_0}^2-{(k_0\cdot N{A}_{illu})}^2})$$ (3)

The maximum thickness h that satisfy $\left|k_z\right|<\sigma \cdot k_0$ for all k_z can be calculated as $h_{\max }=\pi /{\left|k_z\right|}_{\max }$. For 1.2 NA_sys FPM, $N{A}_{illu}=0.7>N{A}_{obj}=0.5$, which give us $h_{\max }=1.75\lambda$. As for 1.45 NA_sys FPM, we have $N{A}_{obj}=0.75>N{A}_{illu}=0.7$, thus $h_{\max }=1.48\lambda$.

According to the derivation above, the ideal thickness of sample for 1.2 NA_sys FPM should be smaller than 822nm, while for 1.45 NA_sys FPM, this number should be 696nm. The resolution target which is 100nm thick gold falls into this range. For red blood cells, which have average thickness at the thickest point of 2–2.5 µm and a minimum thickness in the center of 0.8–1 µm [40], this limit is likely exceeded.

In our opinion, the requirement of $\left|k_z\right|<\sigma \cdot k_0$ is restrictively strict. It is valid for samples with complicated 3D Fourier spectrum distribution in the k_z direction. However, in the case of single layer blood smear sample, structure in the z direction is mainly extension of thickness of the same component. In this case, spectrum $\hat{s}(k_x,k_y,k_z)$ is $\hat{s}(k_x,k_y,0)$ with a scale down factor in a larger range of |k_z|. By rescaling captured captured images, these information can still be stitched together in Fourier domain– this is likely why we still get reasonable reconstructed images from our samples. However, the noise and artifacts introduced by this process should not be ignored and the rendered images can be significantly impacted if these effects are significant.

Appendix C: Resolution calibration using 2D circular hole pairs

In Fig. 7

Fig. 7.

Resolution calibration using a customized two-hole targets, illumination wavelength λ = 632nm. (a) SEM, FPM and conventional microscope image of holes (200nm diameter) on the target. (b) Line plots of vertical intensity distribution through the center of each hole pair, showing a Sparrow resolution of (b1) 635 nm for 20X 0.5 NA objective, (b2) 475 nm for 40X 0.75 NA objective, (b3) 405 nm for the 100X oil immersion 1.25 NA objective, and (b4) 405 nm for 1.2 NA_sys FPM setup.

we show our measurements of the Sparrow resolution limit, for the 1.2 NA_sys FPM and several conventional incoherent microscope setups, using targets consisting of two circular holes. We fabricated each two-hole calibration target on a gold-coated (100 nm coat thickness) microscope slide using a focused ion beam. Each hole has a diameter of 200 nm and the hole pairs have center to center distance varying from 400 nm to 740 nm. The SEM image of 10 sets of hole pairs are shown in the first row of Fig. 7(a). The resulting images are shown in the remaining rows of (a), while the traces through each intensity image are shown in Fig. 7(b). The result of the 1.45 NA_sys FPM is not shown here, because the captured images suffered from a high level of noise, due to the low brightness of each LED as compared to an incoherent halogen lamp. Finally, we list the corresponding experimentally measured and theoretically predicted Sparrow resolution limits in Table 2

Table 2. Sparrow resolution for microscope systems (λ = 632nm, two-hole targets).

	System Parameter	Theoretical Sparrow resolution (nm)	Measured Sparrow resolution (nm)	Deviation from theory
Conventional Microscope	20X 0.5NA	595	635	6%
	40X 0.75NA	396	475	20%
	100X 1.25NA	238	405	70%
FPM 1.2 NAsys	0.5NAobj + 0.7NAillu	384	405	5%

. The equations used to predict theoretical Sparrow resolution is listed in Appendix D.

Appendix D: Discussion on Sparrow resolution limitation for microscope system

The Sparrow resolution limit [29] of imaging system was first derived by Barakat [41,42]. The model he used is shown in Fig. 8(a)

Fig. 8.

Two-point image formation. (a) General imaging system (adapted from [28]), (b) Finite corrected microscope system (adapted from [43]), (c) Infinite corrected microscope system.

, $\epsilon \text{'}$ is the minimum resolvable separation in image space, q is the image distance, a is the pupil half-width and $k=2\pi /\lambda$ is the wavelength. In the derivation, he assumes the receiving plane is far away from the aperture ($q>>a^2/\lambda$) thus the image is a Fraunhofer diffraction of the E-field distribution at the aperture plane. The image plane illuminance distribution is analyzed and the separation $\epsilon \text{'}$ when the second derivative of the total distribution between two peak vanishes (definition of Sparrow limit) is given by the following relationships [28]:

$$\epsilon \text{'}=\alpha \times \frac{q}{ka}$$ (4)

$\alpha$ here is the coefficient determined by image system dimension and illumination condition, which is summarized in Table 3

Table 3. Sparrow resolution limit coefficient $\alpha$ in different circumstances.

	Coherent illumination	Incoherent illumination
Slit aperture (1D) system	4.164	2.606
Circular aperture (2D) system	4.600	2.976

[28]:

For finite corrected microscope system, the two-point image formation schematic is shown in Fig. 8(b). As have been point out by [43], according to the sine condition, $n\epsilon \sin \theta =n\text{'}\epsilon \text{'}\sin \theta \text{'}$, where n, n’ are refractive index on sample side and image side of the system, $\theta ,\theta \text{'}$ are the angles which the marginal ray make with the axis, and $\epsilon$ is the separation in the sample side. Given that n’=1 for microscope system, and $\sin \theta \text{'}\approx \theta \text{'}\approx a/q$ (long imaging distance). We have:

$$\epsilon \approx \frac{\alpha }{2\pi }\times \frac{\lambda }{n\sin \theta }=\beta \times \frac{\lambda }{NA}$$ (5)

Here $\beta =\alpha /2\pi$, and the value of $\beta$ in different circumstances is shown in Table 4

Table 4. Sparrow resolution limit coefficient $\beta$ in different circumstances.

	Coherent Illumination	Incoherent illumination
Slit aperture (1D) system	0.66	0.41
Circular aperture (2D) system	0.73	0.47

. This derivation might not be valid for high NA objective lenses. For an example, in order to satisfy the Fraunhofer diffraction condition, image distance q needs to be larger than 1.5 m for a 1 mm exit pupil objective lens illuminated by 632 nm light source, while the typical value of image distance is only 160mm.

As for infinity-corrected microscope system, which is commonly used in research-grade biomedical and industrial microscopes in the last decade, the image distance of the objective lens is set to infinity, and a tube lens is placed within the microscope to produce the image (Fig. 8(c)). In this case, the objective lens and the tube lens forms a low pass filter type 4f system, with is cutoff spatial frequency, $k_c$, defined by the lens numerical aperture, $N{A}_{obj}$, and the illumination wavelength: $k_c=2\pi \cdot N{A}_{obj}/\lambda$. Because of the Fourier transform property of the 4f system, no Fraunhofer diffraction assumption and small-angle approximation is needed. Using similar mathematical derivation as mentioned above [41,42], we can deduce Sparrow resolution limit as:

$$\epsilon =\gamma \times \frac{\lambda }{N{A}_{obj}}$$ (6)

which is suitable for both low NA and high NA occasions.

For the case when the object consists of two parallel lines in a 2-D circular aperture system (our scenario in section 3), the coefficient $\gamma$ is found from a simulations [44]. All the numbers of $\gamma$ are summarized in Table 5

Table 5. Sparrow resolution limit coefficient $\gamma$ in different circumstances.

		Coherent illumination	Incoherent illumination
Slit aperture (1D) system		0.66	0.41
Circular aperture (2D) system	Two-point target	0.73	0.47
	Two-slit target	0.68	0.44

.

Funding Information

This project was funded by the Caltech Innovation Initiative (CI2), Agency Award: 13520135; the National Institute of Health (NIH) Agency Award R01AI6226-01 and Clearbridge Biophotonics (Singapore).