Advances and challenges in cryo ptychography at the Advanced Photon Source

Abstract

Ptychography has emerged as a nondestructive tool to quantitatively study extended samples at a high spatial resolution. In this manuscript, we report on recent developments from our team. We have combined cryo ptychography and fluorescence microscopy to provide simultaneous views of ultrastructure and elemental composition, we have developed multi-GPU parallel computation to speed up ptychographic reconstructions, and we have implemented fly-scan ptychography to allow for faster data acquisition. We conclude with a discussion of future challenges in high-resolution 3D ptychography.

INTRODUCTION

Ptychography has been developed as a powerful coherent diffraction imaging (CDI) technique to reveal the details of extended objects at a high spatial resolution [1, 2]. In ptychography, a confined illumination probe is scanned across an extended object with some overlap between neighboring illuminated points, while diffraction patterns are recorded from each scan point. From these diffraction data, one can use a phase retrieval algorithm to reconstruct the object’s complex transmission function with a spatial resolution not limited by the optics but by the highest spatial frequencies measured in the diffraction patterns. In addition, the probe function can be reconstructed, which allows ptychography to be used for the characterization of focused beams [3]. Due to the short wavelength and high penetration of x-rays, x-ray ptychography has proven to be especially promising over a broad range of applications: it has been used to image subcellular structures of biological samples [4, 5], strains and dislocations of crystals [6, 7], magnetic domains in thin films [8], and chemical components of samples using x-ray absorption edges [9, 10]. When combined with angular projections, ptychographic tomography can provide 3D quantitative measurements of electron density [11, 12].

Although tremendous progress in ptychography has been made in recent years, there have still been limitations due to coherent flux, radiation damage, and the speed of data acquisition and reconstruction. We present here some advances in ptychography we have achieved at the Advanced Photon Source (APS) to address these challenges. We conclude with a discusison of the issues involved in high-resolution 3D ptychography.

SIMULTANEOUS CRYO PTYCHOGRAPHY AND FLUORESCENCE IMAGING

X-ray fluorescence microscopy (XFM) offers high sensitivity for quantitative mapping of elements in biological samples. However, a majority of biological structures are not easily visualized in XFM since XFM is relatively blind to the light elements (such as H, C, N, O) which are the main constituents of biological materials. Ptychography delivers quantitative phase contrast and high spatial resolution, providing a good solution to visualize biological ultrastructure. While the spatial resolution of ptychography can in theory reach the wavelength limit, radiation damage to biological samples affects the obtainable resolution. A good solution towards this problem is to work with frozen-hydrated biological specimens under cryogenic conditions. Cryogenic samples are better able to withstand beam-induced degradation and can provide high-fidelity structural and ionic elemental preservation, as well mitigate the effects of radiation damage. Therefore, we combined the above techniques (cryogenic conditions, fluorescence microscopy, and ptychography) together on the Bionanoprobe [13], a hard x-ray fluorescence nanorobe with cryogenic capabilities, to image frozen-hydrated biological samples. In this combination experiment, the fluorescence spectra and far-field diffraction patterns were recorded simultaneously from a frozen-hydrated Chlamydomonas alga scanned by a 5.2 keV x-ray focused beam with a spot size of ~ 85 nm [14]. Figure 1(a) shows P, S, K, and Ca fluorescence images, where the presence of K within the cell suggests good preservation of membrane integrity in the cryogenic sample preparation. The ptychographic image (Fig. 1(b)) reveals details of the organelles and membranes not seen in the fluorescence maps. With the combination of ptychographs and elemental maps, identification of the cellular components and quantification of elemental concentrations can become easier.

FIGURE 1.

Simultaneous x-ray fluorescence and ptychographic imaging of a frozen-hydrated Chlamydomonas reinhardtii alga obtained from a 167×151 point scan data [14]. (a) X-ray fluorescence maps reveal the distribution of several elements in the alga. (b) The ptychographic reconstruction shows the biological ultrastructure at sub-30 nm resolution. The ptychographic image reveals features not seen in the elemental maps and adds rich contextual information. (c) Multiple-GPU parallel reconstruction. The 167×151 diffraction patterns were partitioned across several GPUs to reduce processing time from hours to minutes, after which the reconstruction results from each GPU are stitched together to form one image as shown in (b).

RAPID PTYCHOGRAPHY RECONSTRUCTION WITH MULTI-GPUS PARALLEL COMPUTATION

Smaller nanofocused beam sizes and larger scan areas are leading to a rapid increase in the size of ptychography datasets for reconstruction. This is especially obvious in the combined ptychography-fluorescence approach shown above: the focused beam needs to be as small as possible for high-resolution fluorescence imaging, so the small beam size combined with the ptychographic overlap requirements leads to a large volume of datasets. The computational and memory requirements will become even more challenging when ptychography is extended to 3D imaging, since ptychographic scans will need to be acquired at many projection angles. Therefore, a fast and real-time ptychography reconstruction implementation is highly desirable. We have developed a rapid reconstruction code with multi-GPU implementation for real-time ptychographic phase retrieval [15]. Figure 1(c) shows an example of partitioning the 167×151 diffraction patterns (~6.2 GB) across 6 GPUs for independent reconstructions. The reconstruction results from each GPU were then stitched together as one (Fig. 1(b)) by a post-procedure of registration and phase normalization. Moreover, a function of information sharing on the neighboring boundaries instead of across all reconstruction area [16] is also provided in this developed code, which allows GPUs to communicate their results to each other during the reconstruction, yielding a result as if the reconstruction is performed using one GPU. The image reconstruction can be sped up by more than a factor of 100 by employing GPUs on one compute node, and can be faster when distributing the task over more than 100 nodes in the cluster. This parallel reconstruction method allows one to do real-time data analysis, which is extremely helpful to guide the running ptychography experiment.

FLY-SCAN PTYCHOGRAPHY FOR FAST DATA COLLECTION

X-ray ptychography was originally developed using a step-scan mode, in which the scanning microscope worked in a move-settle-acquire sequence for data acquisition at each point. The motor motion and settle time contributes an overhead time on scans, slowing down data acquisition. With diffraction limited storage rings expected to provide hundredfold gains in coherent flux [17], exposure times will decrease accordingly so the percentage of scan overhead could be larger than 90%. To address the scan overhead problem, a continuous motion approach has been proposed [18] and demonstrated [19, 20, 21] in ptychography. In fly scan ptychography, the probe moves relative to the object continuously within a scan line, while the data is acquired by detectors with a very small time interval of detector dead time. This continuous scan mode has already been well used in incoherent imaging systems, such as X-ray fluorescence microscopy [22]. However, diffraction patterns acquired from a continuously moving sample become blurry with degraded speckle visibility, bringing difficulties for ptychographic reconstruction. A recent important study [23] has shown that mixed-state issues can be well addressed using multiple object and/or probe modes in reconstruction. Since fly-scan diffraction patterns can be described as an incoherent sum of diffractions from discrete positions, the moving object can be recovered by using multiple probe modes in the reconstruction. We have explored the limits of probe overlap in fly-scan ptychography with multiple probe mode reconstruction [20].

The speedup that fly-scan ptychography can provide will become especially important when ptychography is extended to do 3D imaging of the sample [11]. Figure 2 shows a demonstration of fly-scan ptychographic tomography on a gold test sample with a thickness of 200 nm; the resolution of a single projection image (Fig. 2(a)) is about 18 nm as determined by line-cut method. The whole 3D dataset of 31 projections with 1200 scan points each was acquired within 1.5 hours, compared to the 10 hours that would have been required for step-scan ptychography. The phase contrast images from the 2D projections were then combined using TomoPy software package [24] to yield a tomographic three-dimensional view of the sample (see Fig. 2(b)).

FIGURE 2.

Fly-scan 3D ptychography on a gold test sample with a thickness of 200 nm. 31 projections of 40 × 30 scan points each were recorded in fly-scan mode with a 5.2 keV focused beam of ~100 nm, spanning a range of tilt angles from −60° to 60° with 4° angular spacing; the acquisition time per projection (covering 2 × 3 μm) is about 3 min with an exposure time of 110 ms per point, yielding the total collection time of ~1.5 hours for the whole 3D dataset. (a) Projection image reconstructed from ptychographic data at 0° using 5 probe modes. (b) 3D volume rendering of the tomographic reconstruction.

The ability to collect both ptychographic and fluorescence data in fly-scan mode enables fast multi-modal imaging. Fast data acquisition provided by the fly-scan ptychography combined with rapid parallel reconstruction allows for high-throughput ptychography, which will become increasingly important as brighter x-ray sources are developed.

3D PTYCHOGRAPHY AND DEPTH OF FIELD LIMITS

3D ptychographic imaging has been demonstrated using reconstructed phase contrast projection images in standard tomographic reconstruction [11]. What ptychography offers in this approach is a quantitative phase contrast image of extended objects, at a resolution beyond that of the probe-forming lens or pinhole. However, as spatial resolution of 3D ptychography continues to improve by, for example, the use of samples at cryogenic conditions (to mitigate radiation damage) and faster fly-scan approaches (to obtain sufficient angular sampling), the validity of treating ptychographic measurements as pure projection images becomes challenged.

In the case of a lens-based imaging system, the transverse resolution cutoff δ_t of a coherent image is limited by the numerical aperture θ of the lens by δ_t = 0.5λ/θ, where λ is the wavelength. The depth of field (DOF) goes like [25] DOF= λ/θ² (a factor of 2 beyond what is given by Chapman et al. [26]), giving a relationship of

$$\text{DOF}=4\frac{{\delta }_t^2}{\lambda }.$$ (1)

A necessary condition for the projection approximation in ptychography to be valid is when the sample thickness is less than the depth of field; in other words, the depth of field limits the maximum thickness of the sample for pure projection images at a desired resolution. Figure 3 plots the depth of field with respect to the transverse resolution at several x-ray energies. Each DOF line limits the maximum sample thickness for pure projection imaging at the desired resolution at that specific photon energy. As the spatial resolution is improved to smaller values, the corresponding DOF (or maximum sample thickness) decreases as the square of these improvements. This brings challenges to nanoscale 3D ptychography of samples with a size of microns to tens of microns, which is precisely the thickness range where x-ray microscopy offers significant advantages over electron microscopy. 3D pychographic tomography has been pioneered at the cSAXS beamline operating at 6.2 keV at the Swiss Light Source; we show as data points in Fig. 3 their achieved transverse resolution for objects with various thicknesses, along with our first demonstration of 3D fly-scan ptychography (Fig. 2). All of these experiments have been within the DOF thickness limit of Eq. 1, with the experiment by Holler et al. being closest to the limit (for details see Supplementary Information of [12]).

FIGURE 3.

Depth of field (DOF; Eq. 1) as a function of the achieved transverse resolution at a series of photon energies. As the transverse resolution improves, the depth of field decreases with the square of these improvements, putting a limit on the sample thickness for straightforward interpretation. The red curve is the DOF limit at 6.2 keV, the energy of ptychographic tomography demonstrations (red dots; see [11, 27, 4, 28, 12, 29, 30, 31, 32]) at the cSAXS beamline at the Swiss Light Source routinely conduct the 3D ptychography. The blue dot represents our fly-scan 3D ptychography study (Fig. 2) at 5.2 keV.

One solution to overcome the above challenges in high-resolution 3D ptychography is using hard x-rays with higher energy. Figure 3 shows that the depth of field increases with increases in photon energy for a fixed resolution. Hard x-rays have larger depth of field and better beam penetration as well, so that they are more suitable to study large samples in 3D compared to soft x-rays, though one also has less contrast for features of a given thickness. Another approach is to extend ptychography to deal with continuous 3D objects. A multislice reconstruction approach has been used in ptychography to image separated depth planes [33, 34]. Is there an extension of this approach that can be used for ptychographic tomography of continuous 3D objects that extend beyond the depth of focus limit of Eq. 1? This is an important question for the future.