Optimized Deconvolution for Maximum Axial Resolution in Three-Dimensional Aberration-Corrected Scanning Transmission Electron Microscopy

Abstract

Three-dimensional (3D) data sets were recorded of gold nanoparticles placed on both sides of silicon nitride membranes using focal series aberration-corrected scanning transmission electron microscopy (STEM). The deconvolution of the 3D datasets was optimized to obtain the highest possible axial resolution. The deconvolution involved two different point spread function (PSF)s, each calculated iteratively via blind deconvolution.. Supporting membranes of different thicknesses were tested to study the effect of beam broadening on the deconvolution. It was found that several iterations of deconvolution was efficient in reducing the imaging noise. With an increasing number of iterations, the axial resolution was increased, and most of the structural information was preserved. Additional iterations improved the axial resolution by maximal a factor of 4 to 6, depending on the particular dataset, and up to 8 nm maximal, but at the cost of a reduction of the lateral size of the nanoparticles in the image. Thus, the deconvolution procedure optimized for highest axial resolution is best suited for applications where one is interested in the 3D locations of nanoparticles only.

INTRODUCTION

The recording of 3D datasets using STEM focal series has become possible with a depth of field of several nanometers (van Benthem, et al., 2005) since the introduction of aberration-corrected STEM (Haider, et al., 2000; Krivanek, et al., 1999; Nellist, et al., 2004). Confocal STEM was already demonstrated earlier for micrometers-thick samples (Frigo, et al., 2002). Spherical aberration correction results in an increase of the electron probe convergence semi-angle α from 10 mrad up to 30-40 mrad, and a decrease of the depth of field from >50 nm in a conventional STEM, to a few nanometers in an aberration-corrected electron microscope. For objects equal in size, or smaller than the point spread function (PSF) of the microscope, e.g., single atoms, the depth of field is directly related to the axial resolution δz (Borisevich, et al., 2006). These focal series, however, show severe distortions in the shape of the objects: spherical objects appear as ellipsoids on account of the elongated PSF. For the imaging of objects larger than the PSF and in particular for highly scattering objects, the elongation becomes even more pronounced, with the axial elongation depending on the size of the nanoparticles (Behan, et al., 2009; de Jonge, et al., 2010b; Xin & Muller, 2009). It was demonstrated that this extension in the axial direction can be reduced significantly by rejecting the out-of-focus information using a pin hole aperture in front of the detector, as is utilized in a scanning confocal light microscopy (Frigo, et al., 2002; Hashimoto, et al., 2010; Nellist, et al., 2006). However, such a configuration inevitably leads to a loss of signal, and involves complex instrumentation.

Alternatively, the out-of-focus information can be effectively restored to its correct axial location by deconvolution of the image with the point spread function (PSF) of the probe (Pawley, 1995). An incoherent image I is the convolution of the true object O with the PSF, and can be described by (Puetter, et al., 2005):

$$I\left(x\right)=\int PSF(x,y)O\left(y\right)dy+N\left(x\right)$$ (1)

In this equation, x and y are three-dimensional vectors and N(x) is the image noise. The inverse operation can be performed if an accurate calculation of the PSF and an estimate of the noise can be provided, which is known as deconvolution. A previous study demonstrated that it was possible to increase the axial resolution of STEM focal series via deconvolution, (de Jonge, et al., 2010b; de Jonge, et al., 2007).

The deconvolution procedures are broadly classified as (1) deblurring methods in which the out-of-focus information in the 3D focal series is removed, and (2) image restoration methods in which the out-of-focus information is not removed but rather reassigned to its source, resulting in an increase in both the contrast and the axial resolution, and is, therefore, the preferred method (Parton & Davis, 2007; Wallace, et al., 2001). Image restoration acts on the entire dataset simultaneously, and requires the PSF to be known accurately. The PSF can be estimated either empirically or from theoretical calculations. The empirical PSF is usually obtained using sub-resolution spherical beads, which can be approximated to single point sources (Parton & Davis, 2007). Alternatively, both the PSF and the object can be estimated using a so-called “blind” deconvolution algorithm by maximum-likelihood approach that has been developed for widefield and confocal fluorescence microscopy (Holmes & O'Connor, 2000; Holmes, 1992; Pawley, 1995).

An important question is to what extent the deconvolution procedure can improve the axial resolution, while avoiding image distortion as much as possible. Here, we present the optimization of a blind deconvolution procedure in which both the PSF and the object are calculated (de Jonge, et al., 2010b). Silicon nitride membranes with gold nanoparticles deposited on both sides were used as test samples. This type of sample allows for measurements of the axial resolution, and of the image distortion. The purpose of this paper is to provide guidance on possible deconvolution strategies that one can select to deconvolve STEM focal series, depending on the specific needs of an experiment, and to provide a method to analyze the effect of the deconvolution quantitatively.

MATERIALS AND METHODS

Sample Preparation

Test samples consisted of silicon nitride (SiN) membranes with gold nanoparticles placed on both sides (Ramachandra, et al., 2011), as shown in Figure 1. SiN membranes with thicknesses of 50, 100, and 200 nm were used. They were supported by silicon microchips used as transmission electron microscopy specimen supports (SPI Supplies). Gold nanoparticles of diameters 2, 5, 10, and 30 nm in solution (Ted Pella) were combined, dispersed in an ultrasonic bath for 2 minutes, and diluted with ethanol. A separate solution of 1.4 nm diameter gold nanoparticles (Nanoprobes) was prepared. Both nanoparticle solutions were then applied in ~0.2 μl droplets on either side of the SiN membranes, and dried in air for ~3 minutes. The silicon microchips were then washed with ethanol, and plasma cleaned for ~30 sec on each side. Plasma cleaning led to some of the nanoparticles breaking into smaller fragments.

Figure 1.

Schematic of the test sample consisting of gold nanoparticles placed on top of and at the bottom of a silicon nitride (SiN) membrane with respect to the direction of the electron beam. The coordinate system is used for all figures.

Electron Microscopy

The samples were imaged with an aberration-corrected STEM (JEOL, 2200 FS) operated at 200 keV beam energy, and using the high annular dark field (HAADF) detector. The corrector was aligned on the same day as the imaging, with an electron probe convergence semi-angle α = 17.3 mrad. For recording datasets with α = 41 mrad (probe current of 83 pA), a second alignment was performed with α = 26.5 mrad (probe current of 30 pA), in order to reduce the higher order aberrations. Convergence of the corrector alignment was determined from the limits (|C1| < 5 nm, |A1| < 7nm, |A2| < 50 nm, |B2| < 50 nm, |A3| < 1.50 μm, |C3| < 2 μm, and |S3| < 0.50 μm) (Uhlemann & Haider, 1998). Each sample was pre-irradiated with an electron dose of ~10⁵ e^-/nm² to avoid contamination, by scanning it with a defocused beam at the eucentric height, with pixel dwell time of 8 μsec, and at a magnification of 50k. This process of exposing the sample to a low dose of electron irradiation before actual imaging is a so-called “beam shower”, which is known to reduce contamination (Egerton, et al., 2004). We found in a previous study that gold nanoparticles on a SiN support membrane did not move during STEM imaging for samples that were plasma cleaned and exposed to a “beam shower” (Ramachandra, et al., 2011). The high annular dark field (HAADF) detector was used for STEM imaging. The brightness and contrast of the detector (gain and offset) were adjusted for each dataset, such to acquire the entire dataset well within the dynamic range of detection (64 bit). A total of 4 focal series with 512×512 pixels were recorded at different microscope settings, see Table 1. The diffraction-limited focal depth (Lupini & de Jonge, 2011) for imaging with α = 27.5 mrad was calculated to be 5.9 nm; for α = 41 mrad it was be 2.6 nm.

Table 1.

Microscope settings used for the recording of the focal series in this study, with specimen thickness T, electron probe convergence semi-angle α, pixel dwell time t, magnification M, pixel size s, number of images in the dataset N, focus step Δz and total measured imaging time t_total (including t and the beam fly-back time). The best axial resolution $\overline{\delta z}$ of each data set is also included.

Dataset #	T (μm)	α (mrad)	t (μsec)	M	s (nm)	N	Δz (nm)	ttotal (min)	$\overline{\delta z}$
1	0.16	41	16	1.5M	0.19	180	1.5	20	8
2	0.09	41	32	500k	0.56	100	3	32	19
3	0.09	26.5	16	1.5M	0.19	100	3.5	12	9
4	0.05	41	16	500k	0.56	100	2	12	13

Deconvolution

The 3D datasets were deconvolved to increase the axial resolution. Data processing was done with 64 bit numbers. Several processing steps were conducted to render the data suitable for deconvolution (de Jonge, et al., 2010b). The noise was reduced using a convolution filter (Digital Micrograph, Gatan), with a kernel of (1,1,1; 1,3,1; 1,1,1), chosen such that the noise reduction did not result in a loss of the image sharpness. The optical density correction procedure of Autoquant deconvolution software (Media Cybernetics) was used to compensate for changes in the average detector signal per image of the focal series. The optical density correction reduced intensity differences between slices. It first took the average intensity at each slice. Then fitted a polynomial to these values. The intensity of each slice was adjusted such that the average of the slice matches the polynomial. The focal series was then aligned slice-by-slice,with respect to its nearest neighbors, to compensate for drift and rotation. For datasets #2 and #3 it was necessary to compensate for a small degree of image distortion occurring between the slices. This correction was done during alignment by setting the software to “tiny warp”. For this correction each slice was divided into a grid and the blocks of the grid were allowed to shear to compensate for small sample distortion with respect to adjacent slices. The pre-processed datasets were then deconvolved by a series of subsequent steps of blind, and fixed deconvolution. The STEM modality was implemented in the software. All optimization options used for light microscopy were switched off.

Particle Size Measurements

To measure the size of a nanoparticle, a line-scan (intensity versus lateral position) was obtained through the center of the nanoparticle in the in-focus image in the focal series. The line-scan was taken at ~ 45° to the image frame. The intensity values of the line-scan were normalized to unity with respect to the maximum value. The full width at half maximum (FWHM) of the peak in intensity was used as the measure of the size of the nanoparticle. To estimate the error in the measurement of nanoparticle diameters, the FWHMs of line-scans passing through the centers of 3 nanoparticles at five different orientations each were analyzed. The standard deviation of the values obtained for the different orientations was 7% of the FWHM, which was considered as the measurement error of the FWHM in this study.

Axial Resolution Measurements

To determine the axial resolution, the axial elongation of the 3D image of a nanoparticle was considered. A vertical line-scan (intensity versus vertical position) was obtained at the center coordinate of a particular gold nanoparticle, and normalized with respect to the maximum intensity. The vertical line-scans of 2 adjacent pixels were averaged in order to reduce noise. The axial resolution was measured as the FWHM of the intensity peak. The error in the measurement of axial resolution was estimated by calculating the axial resolution for ten different particles in 3 adjacent pixels each in the vertical direction. The standard deviation of the axial resolution for the adjacent pixels was 6%, which was considered as the measurement error in the calculation of axial resolution. The axial resolution, and the nanoparticle size were calculated at exactly the same pixel for all deconvolved datasets of a focal series, using an automated procedure programmed in ImageJ (NIH).

Measuring the Number of Nanoparticles in a Cropped Image

The sizes of nanoparticles in a selected small region of each deconvolved dataset were measured. To detect if a particular deconvolution created artifacts in the dataset, the data was also tested for consistency in the number of nanoparticles found in the analyzed region. The peak intensity of each nanoparticle in the cropped image was measured by obtaining a line-scan through the center of the nanoparticle. The threshold of the image was then adjusted such that the upper threshold was set to its maximum and the lower threshold was set to 50% of the peak intensity of the nanoparticle with the smallest peak intensity. This procedure was programmed in ImageJ (NIH). Next, the ‘analyze particles’ function of ImageJ was invoked to obtain the number of nanoparticles in the cropped image. Nanoparticles that were smaller than 0.5 nm were excluded from the count as they were assumed to be noise.

RESULTS AND DISCUSSION

3D Focal series of Gold Nanoparticles on a SiN Membrane

A 3D focal series was recorded of a 200 nm-thick SiN membrane containing gold nanoparticles on both sides. An aberration-corrected STEM was used with a 200 kV electron beam and a convergence semi angle of 41 mrad (see Table 1). Figure 2a shows one of the images of the focal series with the focus at the top of the SiN membrane. Gold nanoparticles of diameters ~1- 5 nm are visible in focus. Even the smallest nanoparticles are well resolved despite the thickness of the SiN membrane. This is because the contrast is formed with an electron probe scanning over the nanoparticles, after which scattered electrons travel through the sample towards the detector (Crewe & Wall, 1970). This small degree of interaction of the electron beam with the SiN membrane does not reduce the spatial resolution, as long as sufficient electrons reach the detector. There is also a blurred shape visible at the lower right in Figure 2a, which is the out-of-focus image of several nanoparticles at the bottom of the SiN membrane. A selected area of the image with the focus at the bottom of the SiN membrane is shown in Figure 2b. These nanoparticles appear more blurred than those at the top, because they were scanned with an electron probe that was broadened due to beam-sample interactions. The nanoparticles measured 2-9 nm in size from their full width at half maximum (FWHM). Smaller nanoparticles were also applied to the bottom of the SiN window, but these were not visible. The thickness of the specimen was measured from the focus difference between the top and bottom images, and amounted to 160 ± 16 nm. The error is the estimated calibration error in the conversion from lens current to focus position. The thickness was also measured by recording an image of the sample tilted by 20° and comparing the shifts of nanoparticles at the top and at the bottom, with respect to an image recorded without tilt. Via the parallax equation, the thickness was measured to be 179 ± 2 nm. The measured thickness is ~10% smaller than expected. Possibly, the SiN was thinner than indicated by the manufacturer.

Figure 2.

Images of the focal series #1 prior to deconvolution, showing gold nanoparticles on a 180 nm-thick SiN membrane recorded at an electron probe convergence semi-angle α = 41 mrad, and a focus step size of 1.5 nm. (a) Image with focus position at gold nanoparticles on top of the SiN membrane. The blurred shape in the lower right corner of the image is due to the out-of-focus information of gold nanoparticles at the bottom of the SiN membrane. (b) Selected region of the image with focus position at gold nanoparticles at the bottom of SiN membrane. (c) Side view projection (x-z) of a 388-pixel long and 18-pixel wide region of the focal series (the average intensity was calculated of the 18 pixels), before alignment. (d) Side view projection after automated alignment. The arrows 1 and 2 point to nanoparticles at the top and the bottom of the SiN membrane respectively. The region in the focal series used to generate (c), and (d) is shown as a dashed rectangle in (a), and (b).

The electron probe broadening due to elastic scattering though a material of thickness T can be expressed as the width of an intensity distribution across a sharp edge, where the intensity goes from 25% to 75% of the total intensity, x_25-75 (Reimer & Kohl, 2008):

$$x_{25-75}=1.05\cdot {10}^3{\left(\frac{\rho }{W}\right)}^{1∕2}\frac{Z(1+E∕E_0)}{E(1+E∕2E_0)}{T}^{3∕2}.$$ (2)

$$E_0=m_0{c}^2;E=Ue,.$$ (3)

with electron accelerating voltage U (in V), density ρ, atomic weight W, atomic number Z, m₀ the rest mass of the electron, c the speed of light, and e the electron charge. Experimentally, it was found that this equation provides a lower limit of the achievable resolution (Ramachandra, et al., 2011). The chemical composition of the SiN window approximately equals that of Si₃N₄, with ρ = 3.2×106 g/m³, W = 3/7×28 + 4/7×14 = 20.0 g/mol, and Z = √(3/7×142+4/7×72) = 10.6 (de Jonge, et al., 2010a). The calculation gives x_25-75 = 1.9 nm for the 180 nm SiN thickness, which is consistent with the experimental finding that nanoparticles of a diameter of a size of 2 nm are just visible at the bottom of the SiN membrane.

Figure 2c, shows the average intensity projection of the side view projection (x-z) of a 388-pixel long and 18-pixel wide region of the focal series (the average intensity was calculated of the 18 pixels); this region is shown as a dashed rectangle in Figures 2a and 2b. The side view shows a misalignment between the images. The maximal total drift that occurred during this focal series was 2.4 nm. We can also observe periodic changes in the intensity. These effects are likely caused by a combination of stage drift, periodic changes of the stage position, electromagnetic fields interfering with the scan position, and periodic changes of the electron probe current. The data was first corrected for variations of the probe current. The slices of the dataset were then aligned in lateral direction via an automated procedure. An advantage of the recording of 3D focal series over conventional tilt-series electron microscopy (Stahlberg & Walz, 2008) is the low drift and the ease at which the slices in the dataset can be aligned, which is the result of the absence of mechanical movements of the stage.

Figure 2d shows the same region of the data set after alignment, with most of the periodic effects and drift removed. The vertical elongation in the images of the nanoparticles is clearly visible. For extended nanoparticles with diameter d the observed intensity in axial direction is a measure of the specimen-limited axial resolution (Behan, et al., 2009; de Jonge, et al., 2010b; Xin & Muller, 2009):

$$\delta z\approx \frac{d}{\alpha }.$$ (4)

Figure 3 shows measurements of δz versus d for several gold nanoparticles at the top, and at the bottom of the SiN membrane. It can be observed that the experimental data exhibits the trend as predicted with equation (4).

Figure 3.

Axial resolution δz versus size d of nanoparticles before (red) and after deconvolution (blue), for both nanoparticles at the top (squares) and bottom (triangles) of dataset#1. The theoretical axial resolution is shown as a straight line.

Deconvolution

The 3D focal series was deconvolved using blind deconvolution available in commercial software. The merits and drawbacks of different deconvolution algorithms in restoring aberration-corrected 3D images are discussed elsewhere (Hovden, et al., 2011). The PSF of an aberration-corrected STEM typically has a lateral size of ~1 Å and an axial size of ~2-5 nm. Ideally, the focal series would be correctly sampled (Pawley, 1995), i.e. with a pixel size of ~ 0.5 Å and a focus step size of ~ 1-2 nm. The highest axial resolution was indeed obtained for the imaging of single atoms with similar settings (Borisevich, et al., 2006; van Benthem, et al., 2005). Unfortunately, these settings are difficult to apply to any samples. For example, many amorphous samples are not sufficiently stable when subjected to the high electron doses associated with the recording of a focal series. It is challenging to record focal series of single atoms that are not embedded in a fixed structure, such as a crystal lattice, requiring the atoms to remain in the same location throughout the portion of the focal series where they are visible. 3D imaging of atoms in a crystal lattice has its own set of challenges due to occurrence of channeling. When atomic resolution is not needed, these challenges are relaxed. Various groups have demonstrated the successful recording of focal series of nanoparticles, e.g., (Behan, et al., 2009; de Jonge, et al., 2010b; Xin & Muller, 2009), to study their 3D locations and shapes. The nanoparticles are typically sufficiently stable in structure and sufficiently strong adhered to their support for the recording of a focal series. The nanoparticles are typically imaged with a lateral pixel size of 0.5 nm or larger, i.e., at under-sampling conditions in the lateral direction, which caused some errors in the PSF estimations. In this report, the deconvolution strategy under the conditions of lateral under-sampling, are considered.

In previous work it was found that the axial resolution of a STEM focal series was improved by deconvolution using four steps of deconvolution (de Jonge, et al., 2010b): (1) A blind deconvolution of n1 iterations to provide a first estimate of the object and the PSF, (2) n2 iterations with fixed PSF (calculated in step 1) , (3) n3 iterations of a second round of blind axial deconvolution, resulting in a second PSF, and (4) n4 iterations with fixed PSF (from step 3). The first two steps represent a regular blind deconvolution, where the convergence angle was set equal to the angle used in the focal series. In the two steps of axial deconvolution the angle was set to zero, which was expected to lead to a stronger effect of the deconvolution procedure in the axial direction than in the lateral direction. As the convergence angle is set to zero, the PSF becomes axial because the PSF is below a pixel wide throughout the image field. In our initial work a deconvolution with n1 = 50, n2 = 50, n3 = 50, and n4 = 25 was used. The number of iterations in each step was chosen by trial and error. An important question is to what extent the deconvolution strategy can be optimized to yield the maximum axial resolution for the minimum possible image distortion.

To determine the optimal combination of iterations of the four different deconvolution steps a series of deconvolutions was performed on focal series #1. The different deconvolution strategies were divided into six categories. Four categories were based on the number of iterations of the regular deconvolution, i.e, the first two steps with n1, and n2 iterations, see Table 2. In each of the four categories several variations were made of the axial deconvolution, i.e., steps n3, and n4. The number of iterations of deconvolution increased from category 1 to category 4. The fifth category consisted of regular deconvolution only (i.e. without axial deconvolution), and the sixth category consisted of axial deconvolution only. A total of 51 deconvolved datasets were analyzed to determine the obtained axial resolution, and changes in the sizes of the nanoparticles. In the following the notation 50_50_50_25 will be used for a deconvolution series with n1 = 50, n2 = 50, n3 = 50 and n4 = 25, and so forth.

Table 2.

Categorization of different deconvolution strategies used for the focalseries #1, with n1 steps of blind deconvolution, n2 steps of fixed deconvolution, n3 steps of blind axial deconvolution, and finally, n4 steps of fixed axial deconvolution. In the fourth category an additional n5 steps of blind axial deconvolution and n6 steps of fixed axial deconvolution were performed. The fifth category consisted entirely of regular deconvolution, while the sixth category consisted entirely of axial deconvolution.

Category	n1	n2	n3	n4	n5	n6
1	25	0, 20	0, 25, 100	0, 10, 25, 50	0	0
2	50	0,25	0,50, 125	0,10, 25, 50	0	0
3	50	0,50	0,50, 100	0,10, 25, 50	0	0
4	100	0,50	0,100, 150, 200	0,10, 25, 50	0,100	0,50
5	100	0,50	0,100	0,50	0	0
6	100	0,50	0,100	0,50	0	0

Figure 4a shows a selected area of Figure 2a with eleven small nanoparticles on top of the SiN membrane imaged in focus. Figure 4b – 4f show several examples of the same area after different deconvolution strategies, demonstrating the effect of the deconvolution on the shape of the gold nanoparticles, and on the image noise. Figure 4b shows the first two steps of deconvolution within category 1, in which the PSF was estimated with n1 = 25 iterations of blind deconvolution, and the fixed deconvolution was then performed with n2 = 20 iterations. This deconvolution strategy is denoted as 25_20. The deconvolution resulted in a reduction of the image noise, and a better definition of the outlines of the nanoparticles. Figure 4c depicts the result of the subsequent axial deconvolution with n3 = 25, and n4 = 20, notated as 25_20_25_20. The effect of the deconvolution was much stronger than after the first two steps. Most of the noise disappeared, and the nanoparticles have become smaller. It was also observed that the deconvolution sometimes misinterpreted noise as a particle, resulting in the appearance of artifacts in the successive steps of deconvolution. Examples are visible at the arrows in Figures 4c-d. A small feature appeared in the deconvolved images at a position where there was clearly no nanoparticle in the original image (Figure 4a). To test if the deconvolution would improve by increasing the number of iterations of axial deconvolution a deconvolution with 25_20_100_50 was also included, as shown in Figure 4d. The effect of the deconvolution is more pronounced, with contours remaining of the nanoparticles. However, this deconvolution strategy did not remove the artifacts. It appeared to be necessary to increase the iterations of the first steps of deconvolution. Figure 4e shows deconvolution of category 3 with 50_50_50_50. The nanoparticles appear even smaller, and the artifacts have intensities much smaller than those of the nanoparticles, such that the artifacts appear as faint pixels that can be distinguished from the nanoparticles. The peak intensities of the artifacts in Figure 4e are smaller than 50% of the peak intensity of the smallest nanoparticle in the image. An even stronger deconvolution with 100_50_100_50 is shown in Figure 4f. The result is almost the same as that of Figure 4e, but now the artifacts are not visible anymore. The images of 100_50_100_50 (category 5), and 100_50_100_50 (category 6), using only regular or only axial deconvolution, respectively, did not show any difference from the 100_50_100_50 of category 4.

Figure 4.

Selected region of dataset #1 (dotted area in Figure 2a) showing eleven 0.8 – 1.8 nm diameter gold nanoparticles on top of the SiN membrane for different deconvolution strategies. (a) Original image. (b) Deconvolution 25_20. (c) 25_20_25_25. (d) 25_20_100_50. (e) 50_50_50_50. (f) 100_50_100_50. The arrows in the images (c) and (d) indicate the presence of artifacts appearing as additional nanoparticles. (g) – (l) Two 3.8 and 2.3 nm gold nanoparticles at the bottom of the SiN membrane for the original dataset, and for the same deconvolution strategies as in (b) – (f). (m) – (r) Side view projections (x-z) of a 388-pixel long and 18-pixel wide region (the average intensity was calculated of the 18 pixels) of the corresponding deconvolution strategies. The arrows 1 and 2 in image m point to nanoparticles at the top and the bottom of the SiN membrane respectively.

Figures 4g–l show images of two nanoparticles at the bottom of the membrane for the same datasets discussed above (following the same order as in Figures 4a–f). Figure 4g shows nanoparticles at the bottom of the membrane for the original dataset. The shape of the nanoparticles is blurred, and the number of nanoparticles in this image cannot be ascertained unambiguously. In Figure 4h, showing nanoparticles at the bottom for the 25_20 dataset, two distinct nanoparticles can clearly be observed. These two nanoparticles were further resolved with an increase in the number of iterations, as is evident from Figures 4i –l. It was also observed from Figure 4 that the images of larger nanoparticles both at the top and bottom of the membrane break up into collections of smaller peaks with an increasing number of iterations, which is similar to the ringing and explosion artifacts of optical deconvolution methods (Wallace, et al., 2001).

Figures 4m–r show the side view projections of a 388-pixel long and 18-pixel wide region (the average intensity was calculated of the 18 pixels) for the datasets discussed above (following the same order as in Figures 4a–f). Figure 4m shows that the intensity is extended in the axial direction for the original dataset, and is particularly prominent for nanoparticles at the bottom of the membrane, for which the intensity spread spans the entire depth of the focal series. For the deconvolution dataset 25_20 shown in Figure 4n, the axial intensity spread is significantly reduced, and the contrast of the image has also increased due to the restoration of the out-of-focus information to its original source. With an increasing number of iterations, the intensity was increasingly constrained to the location of the gold nanoparticles, as is evident from Figures 4n-p. The out-of-focus information disappeared almost entirely for the deconvolutions 50_50_50_50 and 100_50_100_500, shown in Figures 4q, and 4r respectively.

Quantitative Analysis of the Deconvolved Datasets

The various deconvolved datasets of focal series were quantitatively analyzed in order to select the best possible deconvolution procedure. For each of the 51 datasets, the same eleven nanoparticles located at the top of the SiN membrane were analyzed as shown in Figure 4. The average change in the sizes of the nanoparticles was measured, and the axial resolution was determined. The data was also checked for the occurrence of artifacts, i.e., disappearing of nanoparticles, or appearing of additional ones, by an automated particle counting procedure. The images of the eleven gold nanoparticles placed at the top of the SiN membrane shown in Figure 4a varied in size between 0.8 – 1.8 nm (the sizes of the nanoparticles were measured from line-scans in a direction 45° with respect to the image frame). As a measure of the change in the size the average change of the diameter of the eleven nanoparticles $\left(\overline{\Delta d}\right)$ was used. Since the lateral size of the PSF is much smaller than these nanoparticles, and the nanoparticles were in focus, it is reasonable to assume that the image in Figure 4a (the original image before deconvolution) contains their actual shape as projected on the xy plane. The value of $\overline{\Delta d}$ was calculated with respect to this original image as:

$$\overline{\Delta d}=\frac{1}{n}\sum _{particle=1}^n\frac{d_{deconvolved}-d_{original}}{d_{original}}.$$ (5)

The axial resolution ($\overline{\delta z}$) was determined from the average of the FWHM values measured from vertical line-scans in the 3D dataset at the locations of the eleven nanoparticles. The individual values of δz are shown in Figure 3. Remarkably, the values do not differ much between the different nanoparticle sizes after deconvolution, and it is thus correct to compute the average. The FWHM of the peak over a nanoparticle in a vertical line-scan is a sufficiently accurate measure of the resolution since the PSF extends in axial direction by a larger amount than the actual size of the nanoparticles. It was found in previous work that the measure of the FWHM was ~20% smaller than the value obtained from a more complex measurement of the axial resolution using the Raleigh criterion (de Jonge, et al., 2010b), a difference which was deemed acceptable for the optimization of the deconvolution procedure. Our goal was to find the highest possible axial resolution, i.e., the smallest possible value of $\overline{\delta z}$ (which was always much larger than d) for the smallest value of $\overline{\Delta d}$ (the minimal change of the size of the nanoparticles), while counting the same number of nanoparticles (eleven) as in the selected region of the original image.

Figure 5 depicts the values of $\overline{\delta z}$, and $\overline{\Delta d}$ for all 51 different deconvolution series (see Table 2). The deconvolution strategies that result in eleven nanoparticles are drawn in black, while the red color indicates that the number of nanoparticles did not equal eleven. It can be seen from Figure 5 that, for all choices of the number of iterations of the first step of deconvolution, $\overline{\Delta d}$ changed only by a small amount of typically less than -7%, while $\overline{\delta z}$ remained similar to that of the original dataset $(\overline{\delta z}=35\mathrm{nm})$. The second step of the deconvolution resulted in appreciable improvement of axial resolution. For example, the deconvolution of 100_50 resulted in $\overline{\delta z}=27$ nm, and $\overline{\Delta d}=\text{-}23\%$. However, this dataset was still noisy, and an artifact appeared in the image that triggered the count of an addition of a nanoparticle. The third step of deconvolution 50_100_50 resulted in a substantial decrease of $\overline{\delta z}$ to 17 nm, while $\overline{\Delta d}$ increased only slightly to -30%. This deconvolution resulted in a strong reduction of the image noise and the blurred signal from out-of-focus nanoparticles, and no artifacts were found.

Figure 5.

The average axial resolution $\overline{\delta z}$ (scale on right) and the average change in the particle size $\overline{\Delta d}$ (scale on left) for different deconvolution strategies applied to focal series#1. The red color indicates the occurrence of artifacts, either a loss or an addition of nanoparticles.

The strongest effect of the deconvolution occurred in the fourth step, resulting in a decrease of $\overline{\delta z}$ to 8 nm, a factor of four improvement with respect to the original data, while $\overline{\Delta d}$ was -44%. All datasets with n1 = 100, n2 = 50, and n3 = 100 (category 4 in Table 2) exhibited similar values of $\overline{\Delta d}$ and $\overline{\delta z}$ for different numbers of iterations of n4, and the deconvolution had thus converged. The deconvolution strategy of 100_50_100_50 can be selected as the best deconvolution - additional iterations are not needed.

To test if additional steps of deconvolution with a new PSF could improve the deconvolution, an additional fifth and sixth step of deconvolution were performed for category 4. The fifth step of deconvolution (100_50_100_50_100) resulted in a decrease of $\overline{\delta z}$ to 6 nm, while $\overline{\Delta d}$ increased to -58 %. The sixth step of deconvolution (100_50_100_50_100_50) resulted in a $\overline{\delta z}$ of 4nm, and a $\overline{\Delta d}$ of -65 %. However, both the fifth and the sixth step of deconvolutions induced an artifact in the form of loss of a particle, and deconvolution with a third PSF is thus not useful.

The effect of the axial deconvolution was also evaluated by deconvolving with either entirely regular deconvolution (category 5 in table 1), or entirely axial deconvolution (category 6), but found no significant difference with the results of category 4. It can thus be inferred that the axial deconvolution did not bring any appreciable increase in either the axial resolution or in the preservation of the nanoparticle size compared to the normal deconvolution. The increase of the axial resolution by a factor of four is rather achieved by using two subsequent deconvolutions with two different PSFs (each consisting of a blind and a fixed step). The process of iteratively refining both the PSF and the object is also used by others in optical blind deconvolution methods (Wallace, et al., 2001).

Finally, two nanoparticles at the bottom of the SiN membrane were analyzed, measuring 3.8, and 2.3 nm. The deconvolution strategy of 100_50_100_50, resulted in $\overline{\delta z}=9\mathrm{nm}$, which is almost the same as for nanoparticles at the top. The value of $\overline{\Delta d}$ was -73%, a larger difference than for nanoparticles on top of the SiN membrane. This is because the image at the bottom of the membrane was blurred due to beam broadening (equation 2), and the measured sizes of the nanoparticles are thus overestimates of the actual sizes, which led to a larger change in $\overline{\Delta d}$ upon deconvolution.

Deconvolved Images

Figure 6 includes images obtained with the best deconvolution strategy of 100_50_100_50 applied to focal series # 1. Figures 6a, and 6b show images of gold nanoparticles on top and at the bottom of the SiN membrane, respectively. The gold nanoparticles in these images appear much sharper than in the original dataset (Figs. 2a and b). In particular, the smallest nanoparticles at the bottom of membrane are well resolved, while they were difficult to distinguish in the original image (Figure 2b).

Figure 6.

Images of the dataset #1 processed with the optimized deconvolution strategy 100_50_100_50. (a) Image with focus position at the gold nanoparticles on top of the SiN membrane. (b) Image with focus position at gold nanoparticles at the bottom of SiN membrane. (c) Plot of the normalized intensity versus the axial position of the original dataset and of the deconvolved dataset over one particle. (d) Side view projection (x-z) of a 388-pixel long and 18-pixel wide region (the average intensity was calculated of the 18 pixels). The region in the focal series used to generate (d) is shown as a dashed rectangle in (a) and (b). The arrows 1 and 2 point to nanoparticles at the top and the bottom of the SiN membrane respectively.

The deconvolution did not work well for the larger nanoparticles, as their image became distorted into round shapes of bright and dark spots. Presumably, too much electron scattering occured in the larger nanoparticles such that the assumption of linear contrast-formation was not applicable at their corresponding locations in the dataset, which led to an incorrect deconvolution. The mean free path length for elastic scattering of 200 keV electrons in gold (corresponding to the total scattering cross section (Reimer & Kohl, 2008) amounts to 13 nm, which indicates that the nanoparticles should be smaller than 13 nm in order to prevent excessive scattering. The deconvolution is thus optimal for small nanoparticles. Figure 6c, compares the original and the deconvolved image via a vertical line-scan obtained over a 1.3 nm diameter nanoparticle on top of the SiN membrane. Most of the image noise disappeared, and δz of this nanoparticle improved from 43 nm to 8 nm. Figure 6d shows the average intensity projection of the side view for the 100_50_100_50 deconvolution dataset, for the same region shown in Figures 2c and 2d. The region in the focal series used to generate Figure 6d is shown as a dashed rectangle in Figures 6a and 6b. The axial profiles of the nanoparticles have a strong contrast for both nanoparticles at the top, and at the bottom, and the axial profiles are much more confined to actual locations of the nanoparticles than before deconvolution. In comparison, the side view projection of the original focal series in Figure 2d exhibited a strongly elongated axial profile, and a low contrast.

From Figures 4 and 6 and the discussion above, it is evident that deconvolution resulted in an increase of the axial resolution, and a decrease of the image noise. Deconvolution also led to a decrease of the size of the nanoparticles. Yet, a reduction in size is acceptable in many applications, for example, if one is mainly interested in the 3D location of nanoparticles. One should, however, be cautious of possible artifacts in the form of appearing, or disappearing of nanoparticles. In some cases, it may be beneficial to do only three steps of deconvolution, and not aim for the highest axial resolution. For example, in our recent work, three steps were used to deconvolve a 3D data set of a whole mount eukaryotic cell without any loss of gold nanoparticles, while simultaneously preserving the contrast on the stained cellular ultrastructure (Dukes, et al., 2011). In that work both the lateral and axial location were determined with a precision of 3 nm, despite a limited axial resolution (Dukes, et al., 2011). In another study, the 3D shape of a metal shadow cast biological sample was determined (clathrin coated pit and cytoskeleton) with four steps of deconvolution at less iterations (25_50_50_50) than what was used here to obtain the maximal axial resolution (de Jonge, et al., 2010b). In the latter study, the 3D shapes were well preserved. The choice of the deconvolution strategy will thus be set by the particular application. The maximal number of iterations of 100_50_100_50 can be used if one aims to determine the 3D locations of nanoparticles with the highest possible axial resolution, while lower number of iterations can be used to reconstruct datasets containing objects of mixed electron scattering strength, as in a stained biological ultrastructure. Figure 5 may serve as a guide to select the optimal balance between $\overline{\Delta d}$ and $\overline{\delta z}$ for a particular experiment.

Deconvolution of Other Datasets

To test the applicability of the deconvolution strategy three additional datasets #2 - #4 were analyzed (see Table 1). The difference between these datasets and dataset #1 is the thickness of the sample. Focal series #2 and #3 were recorded on a 100 nm thick SiN membrane, and focal series #4 was recorded on a 50 nm thick SiN membrane. The actual measured thicknesses for the 50 and 100 nm membranes were 46 ± 7, and 87 ± 8 nm, respectively, as measured from the focus positions. Datasets #2 - #4 also used different microscope settings than dataset #1. In Figure 7a-c it can be seen that also for these datasets the deconvolution of 100_50_100_50 provided the best results. The values of $\overline{\delta z}$ for nanoparticles at the top of the membrane before deconvolution were 72, 56, and 68 nm for focal series #2, #3, and #4, respectively. After deconvolution, the values were $\overline{\delta z}=19$, 9, and 13 nm for focal series #2, #3, and #4, respectively. For nanoparticles at the bottom it followed that $\overline{\delta z}=19$, 10, and 13 nm for focal series #2, #3, and #4, which is approximately the same as obtained for nanoparticles at the top. An improvement in the axial resolution by a factor of 4 – 6, was observed for the 4 different focal series.

Figure 7.

Values of $\overline{\delta z}$ and $\overline{\Delta d}$ for different deconvolution strategies applied to datasets #2 - #4. (a) Focal series #2 recorded of an 87 nm SiN membrane with α = 41 mrad. (b) Focal series #3 recorded of an 87 nm SiN membrane with α = 26.5 mrad. (c) Focal series #4 recorded of a 46 nm SiN membrane with α = 41 mrad.

Datasets #2, and #4 exhibited a lower axial resolution than the other datasets, although the improvement of the axial resolution by deconvolution was still a factor of 4-5. This difference is likely caused by the larger degree of under-sampling (Pawley, 1995) of the data in lateral direction, i.e., datasets #2 and #4 were recorded at a lower magnification than datasets #1 and #3 (see Table 1).

It is also remarkable that dataset #3, which was recorded at the smallest Δ and the largest Δz, and should thus exhibit the worst axial resolution of all datasets, still measured $\overline{\delta z}=9\mathrm{nm}$. Compared to datasets #2 and #4 this can be explained by the magnification. Datasets #2 and #4 were recorded at a lower magnification than #1 and #3, which resulted in a higher degree of under-sampling. But, most important, the lower magnification prevented the imaging of the smallest nanoparticles. The average size of nanoparticles that were analyzed in datasets #2, and #4 were 2.3, and 2.7 nm, while the average sizes were 1.2, and 0.9 nm for datasets #1, and #3, respectively. Therefore, datasets #2, and #4 measured a lower axial resolution than other datasets (see equation 4). The fact that dataset #3 had a similar axial resolution as dataset #1 is likely caused by the thicker sample support for dataset #1, leading to 1) less visibility of the smallest nanoparticles, and 2) larger beam broadening for the imaging of nanoparticles at the bottom. Under-sampling could possibly also be a cause for the general observed reduction in $\overline{\Delta d}$ after deconvolution in all datasets.

The change in size of the nanoparticles was not more than -45% for nanoparticles at the top of the membrane, consistent with dataset #1. The value of $\overline{\Delta d}$ was -58% at most for nanoparticles at the bottom, which is considerably smaller than the value of -73% obtained for the 200 nm membrane. From the point of view of beam broadening, the PSF for datasets #2 - #4 should be more homogeneous throughout the specimen than for dataset #1, since the latter was recorded on the thickest SiN membrane. This effect presumably explains the smaller $\overline{\Delta d}$ for datasets #2 - #4 compared to dataset #1. It is important to notice that the best deconvolution strategy did not change between the datasets obtained using different SiN thicknesses, which implies that the blind deconvolution finds the optimal PSF for the entire 3D volume.

CONCLUSIONS

Deconvolution of 3D STEM datasets of gold nanoparticles placed on SiN supporting membranes resulted in an increase of the axial resolution by a factor of 4-6 for the optimized deconvolution strategy. The best axial resolution as determined from the average FWHM values of axial profiles over a subset of nanoparticles was 8 nm. The deconvolution involved two different PSFs, each calculated via blind deconvolution. The best deconvolution included 100 iterations of blind deconvolution, 50 iterations of fixed deconvolution with the first PSF, 100 further iterations of blind deconvolution, and finally 50 iterations of fixed deconvolution with the second PSF. Additional iterations did not improve the axial resolution. The size of the nanoparticles measured in the lateral direction decreased with increasing number of deconvolution iterations. The images were searched for the occurrence of artifacts, such as the appearance, and disappearance of nanoparticles. The deconvolution found the optimal PSF in samples of thicknesses varying between 46 and 180 nm SiN, despite influences of beam blurring in the thicker samples. The deconvolution did not work well for nanoparticles of a size comparable to or larger than the mean free path length for elastic scattering in gold. The deconvolution strategy can be selected depending on the particular application. Low number of iterations can be used to reduce the image noise, and additional deconvolution can be used to obtain 3D information with moderate axial resolution, while preserving most of the structural information. The highest number of iterations is useful if one is mainly interested in the 3D locations of nanoparticles, while a reduction in the apparent size of the nanoparticles by ~50% is acceptable.