Correcting the linear and nonlinear distortions for atomically resolved STEM spectrum and diffraction imaging

We report a software tool for post-correcting the linear and nonlinear image distortions of atomically resolved 3D spectrum imaging as well as 4D diffraction imaging. This tool improves the interpretability of distorted scanning transmission electron microscopy spectrum/diffraction imaging data.

Abstract

Specimen and stage drift as well as scan distortions can lead to a mismatch between true and desired electron probe positions in scanning transmission electron microscopy (STEM) which can result in both linear and nonlinear distortions in the subsequent experimental images. This problem is intensified in STEM spectrum and diffraction imaging techniques owing to the extended dwell times (pixel exposure time) as compared to conventional STEM imaging. As a consequence, these image distortions become more severe in STEM spectrum/diffraction imaging. This becomes visible as expansion, compression and/or shearing of the crystal lattice, and can even prohibit atomic resolution and thus limits the interpretability of the results. Here, we report a software tool for post-correcting the linear and nonlinear image distortions of atomically resolved 3D spectrum imaging as well as 4D diffraction imaging. This tool improves the interpretability of distorted STEM spectrum/diffraction imaging data.

Introduction

Besides conventional imaging, an advantage of modern scanning transmission electron microscopy (STEM) is its compatibility with other techniques, such as microanalysis and diffraction techniques. The basic concept is rather simple; a focused STEM electron probe is systematically scanned across the sample and at each probe position, the imaging signal(s), i.e. annular dark-field (ADF), analytical signal(s) and/or diffraction patterns can be acquired simultaneously. The analytical signals are stored in a data cube where two of the cube axes correspond to spatial information, while the third dimension represents the spectrum, which can be an electron energy-loss spectrum (EELS) and/or an energy-dispersive X-ray spectrum (EDXS). This technique is referred to as spectrum imaging (SI) [1]. Similarly, the diffraction data are stored in a four-dimensional dataset where the third and fourth dimension are the reciprocal-space coordinates of the convergent beam electron diffraction patterns (CBED). Hence, it is referred to 4D-STEM [2]. The combination of spatial and spectral/diffraction information in a single measurement contains extremely localized information, opens up a wide range of possibilities for data analysis, and provides a powerful tool for materials characterization. With the advent of aberration correctors, monochromators, better detectors and cameras, as well as more stable electron optics, sample holders and instrument environments, it is becoming easier to probe various materials at atomic resolution using STEM-SI [3] or 4D-STEM [2, 4] techniques. Despite these improvements, raster scanning introduces artifacts in the measurements when the true STEM electron probe position does not match the desired position, which may be due to sample or stage drift, instabilities or fly-back distortion resulting from the sudden change of beam location from the end of a line to the beginning of the next line [5, 6]. These distortions become more crucial in STEM-SI and 4D-STEM experiments compared to conventional STEM imaging, as they require much longer dwell time in order to achieve an acceptable signal-to-noise ratio of each individual spectrum/diffraction pattern. Thus, the acquisition and interpretation of atomically resolved 3D and 4D datasets remain problematic due to heavy image distortions. Various methods have been proposed to minimize and correct these distortions in STEM imaging [5–12]. Popular atomic resolution STEM-SI acquisition techniques use on-the-fly drift-correction [13, 14] or direct sums of multiple fast-dwell-time SI that have been drift-corrected between separate SIs [15]. More recently, rigid and non-rigid registration techniques aligning the multi-frame SI have also been reported [16–19]. However, these experiments are not yet routine and require advanced registration algorithms to perform the alignment. Moreover, these techniques may not be applicable to the 4D-STEM due to the big data handling issue. Here, we report the development of a software tool, which we call STEM SI Warp, for post-correcting the image distortion of atomically resolved STEM-SI and 4D-STEM.

Algorithm

A perfect scan-grid is an equally spaced grid both along the fast- and slow-scan directions as schematically shown in Fig. 1a. In the presence of environmental distortions the fast-scan spacing remains approximately constant; however the linear (Fig. 1b and c) and nonlinear (Fig. 1d) offsets between slow-scan rows may corrupt the perfect scan-grid. As a consequence, these linear and nonlinear offsets create linear and nonlinear distortions in the resulting images, respectively. These distortions become visible as expansion, compression, and/or shearing of the crystal lattice. Here, we classify the linear and nonlinear distortions by checking whether or not the shear and expansion/compression ratio is constant/linear for the whole image. Figure 1 shows typical image distortions, i.e. linear shear (e) and expansion (f) distortion, and nonlinear distortion (g), in the atomically resolved spectrum image. With prior knowledge of the crystal structure of the sample (in this case the perovskite oxide structure), one can recognize that the straight lattice (indicated by the dotted lines) has been distorted into inclined or curved lines. Correcting these linear and nonlinear image distortion in the atomically resolved STEM-SI and 4D-STEM is the main task of this algorithm.

Fig. 1.

(a) Sketch of an ideal scan grid along the fast and slow scan directions. Rows and columns are equally spaced. Distorted scan grids, the fast scan mostly remains perfect but the slow scan becomes linearly (b, c) or nonlinearly (d) distorted, respectively. (e–g) Examples of these distortions on the spectrum image, respectively. Distortions are marked by solid straight and curved lines, respectively. The dotted lines mark the nominal trace of the atom planes.

The displacement due to linear and nonlinear distortion

Considering the displacement between a distorted image and an ideal image, it is reasonable to assign the distortion as the displacement of all the nodes between the distorted and the ideal grids. The distortion will have x- and y-components corresponding to the displacement along the x-axis (fast-scan direction) and y-axis (slow-scan direction), respectively.

We define $U\left(i,j\right)=∆X_{\mathit{ij}}+∆Y_{\mathit{ij}}$, where U(i, j) describes the displacement at grid-row i and grid-column j with identical grid spacing d_o.

For linear shear distortion as schematically shown in Fig. 2a, the fast-scan direction mostly remains perfect but the slow-scan direction becomes distorted. Thus, the displacement can be simplified as

$$U\left(i,j\right)\approx ∆X_{\mathit{ij}}=j·d_o·\mathit{tan}\theta ,$$ (1)

and the distortion can be corrected by performing an affine transformation in the x–y plane

$$\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right]=\left[\begin{array}{cc}1& a\\ 0& 1\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right],$$ (2)

where x’ and y’ are the coordinates of the warped image, x and y are the coordinates of the original image; and a is the shear coefficient ($\mathit{tan}\theta =\frac{∆X_{\mathit{ij}}}{j\ast d_o}$).

Fig. 2.

Displacement caused by (a, b) linear and (c) nonlinear distortions. (d) Flow chart of the script, starting with the STEM ADF image and ending in the results after removal of distortions.

For expansion/compression distortion as schematically shown in Fig. 2b, the displacement can be expressed as

$$U\left(i,j\right)\approx ∆X_{\mathit{ij}}=i·d_o·{\gamma }_x,$$ (3)

In case of presence of expansion/compression distortion along the y-axis

$$U\left(i,j\right)\approx ∆X_{\mathit{ij}}+∆Y_{\mathit{ij}}=i·d_o·{\gamma }_x+j\ast d_o·{\gamma }_y$$

and the distortion can be corrected by

$$\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right]=\left[\begin{array}{cc}1+{\gamma }_x& 0\\ 0& 1+{\gamma }_y\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right],$$ (4)

where x′ and y′ are the coordinates of the warped image, x and y are the coordinates of the original image; and γ_x and γ_y are the expansion/compression coefficient along the x- and y-axes (positive for expansion distortion and negative for compression distortion). In the case of Fig. 1f, expansion distortion is mainly along the x-axis direction (thus, ${\gamma }_y\approx 0$).

For nonlinear distortion, a row of the grid will be divided into sub-grids. In each section the x-displacement can be treated as a linear distortion, the y-displacement is simply corrected by letting each section have the same size in y (optional). Then, the nonlinear displacement can be expressed as

$$U\left(i,j\right)=∆X_{\mathit{ij}}+∆Y_{\mathit{ij}}=\sum _j\left[j_k·\mathit{tan}{\theta }_j+\left(d_j-d_o\right)\right],$$ (5)

and in each section the distortion will be corrected by

$$\left[\begin{array}{c}{x}_j^{\prime }\\ y_j^{\prime }\end{array}\right]=\left[\begin{array}{cc}1& a_j\\ 0& {1+\gamma }_j\end{array}\right]\left[\begin{array}{c}{x}_j\\ y_j\end{array}\right],$$ (6)

where $x_j^{\prime }$ and $y_j^{\prime }$ are the coordinates of the warped section, x_j and y_j are the coordinates of the original section; a_j is the shear coefficient ($\mathit{tan}{\theta }_j=\frac{∆X_{\mathit{ij}}}{j\ast d_o}$) of section j, and γ_j is expansion/compression coefficient of section j. In the software, the correction of the distortion along the y-axis is optional. If the correction is ‘on’, γ_j is set to $\frac{∆Y_{\mathit{ij}}}{d_o}$ and the program will correct all the sections to have the same size in y. If the correction is ‘off’, γ_j is set to 0.

Flow diagram and analysis steps

The software tool, written in the Digital Micrograph (DM, Gatan Inc.) scripting language [20], is divided into several sequential steps, which are summarized in the flow chart in Fig. 2d. The algorithm is based on first calculating the image distortions from the simultaneously acquired ADF image. The linear distortion is corrected by warping the distorted parallelogram or trapezoid back to a rectangle. For the nonlinear image distortion, the image and SI data are divided into subsections (corresponding to individual unit cells). Within each subsection the distortion can be corrected as a linear distortion. The division of the image into sections is based on the location of the center of selected atomic columns. To improve the precision of atomic column centering, filtering the input ADF image is usually necessary. After locating the center of the atomic columns, each selected atomic center will be the boundary of the sections and this is used to divide the image into sections. If the user would like to perform a correction of the distortions along the y-axis (to allow correction for compressive/tensile strain), one has to select a column of atoms along the y-direction and each unit cell along the y-direction will be one section. Finally, the calculated image distortion correction can be applied on both the ADF image and the multidimensional dataset. The STEM SI/DI data can be 2D images, i.e. computed elemental maps, reconstructed images, or 3D spectrum imaging data as well as 4D diffraction imaging data. In the script, the transformation of the coordinate system was performed by using the ‘warp’ function, which uses bilinear interpolation. Bilinear interpolation considers the closest 2 × 2 neighborhood of known pixel values surrounding the unknown pixel. It then takes a weighted average of these 4 pixels to arrive at its final interpolated value. The weight on each of the 4 pixel values is based on the computed pixel’s distance from each of the known points [21]. The software (plug-in for DM) is available by request to the authors. For advanced users, the multidimensional warp and offset functions are installed as libraries and can be called externally.

As can be seen from the following examples, this program does not fully correct the distortion. It maximally corrects the deformed lattice back to a marginally distorted one, thus improves the interpretability of distorted STEM spectrum/diffraction imaging data. It requires prior knowledge of the crystal structure, but not detailed lattice parameters. Comparing this off-the-fly method with the on-the-fly methods, i.e. multiple direction scanning [6, 9], multiple frame SI [15–19], it is rather simple. It does not require advanced registration algorithms to perform the alignment.

Experimental section

To illustrate the result of the STEM SI Warp program, three materials were chosen in this study, SrTiO₃ bulk material, a two-dimensional Ba-doped La₂CuO₄ (LCO) thin film grown on LaSrAlO₄ (LSAO) substrate, and a NdVO₃ film grown on NdGaO₃ (NGO) substrate. The epitaxial deposition of La₂CuO₄ by atomic layer by layer oxide molecular beam epitaxy has been described by Baiutti et al. [22]. The NdVO₃ film was grown on NGO substrate by pulsed laser deposition. TEM specimens were prepared by a standard procedure which includes mechanical grinding, tripod polishing, and argon ion beam milling in a stage cooled with liquid nitrogen. Before the STEM experiments, samples were plasma-cleaned in a Fischione plasma cleaner using a 75% argon–25% oxygen mixture for 4 min to eliminate possible hydrocarbon surface contamination. STEM investigations were performed using a JEOL JEM-ARM 200 CF scanning transmission electron microscope equipped with a cold field emission electron source, a DCOR probe corrector (CEOS GmbH), a 100 mm² JEOL Centurio EDX detector, and a Gatan GIF Quantum ERS electron energy-loss spectrometer. The microscope was operated at 200 kV with a semi-convergence angle of 20.4 mrad, giving rise to a probe size of 1 Å (0.8 Å for STEM diffraction imaging). A collection semi-angle of 111.5 mrad was used for EELS elemental mapping. The STEM diffraction image was acquired using a Gatan OneView camera and the STEM diffraction imaging module [23]. The de-noising of the STEM-SI data was performed by the multivariate weighted principal-component analysis (PCA) routine (MSA Plugin in Digital Micrograph) developed by M. Watanabe [24].

Applications

To demonstrate the capability of the software tool, we show examples for correcting the scanning imaging distortions of multidimensional data in the following.

(1) Correcting linear and nonlinear image distortions for STEM-SI

We present two experimental cases, two-dimensional Ba-doped LCO [25] and NdVO₃ thin film, for demonstrating the correction of linear and nonlinear image distortion, respectively. As it can be seen from the simultaneously acquired ADF images (Fig. 1e–g), typical images distortions are present in the images. Along the in-plane direction (vertical), the straight lattice as indicated by the dotted lines is distorted into inclined or curved lines. We classify them as linear and nonlinear image distortions, respectively. Such distortions can be corrected accordingly using the displacement functions and correction procedures described in the previous section depending on the type of the distortion. Figure 3a shows an ADF image of two-dimensional Ba-doped LCO with linear distortions corrected by the STEM SI Warp. The deformed orthogonal lattice has been fixed back, as it can be seen from the final warped and cropped RGB color figure, which shows overlaid elemental maps, Cu-L_2,3(red), La-M_4,5 (green) and Ba-M_4,5 (blue). These individual elemental maps as well as the elemental maps extracted from the raw SI can be found in the supporting information (see the supplementary Fig. S1 online). In the second example, as can be seen from Fig. 1f, the SI shows mainly expansion distortion along the fast-scan (x-axis) direction and the expansion ratio increases almost linearly along the slow-scan (y-axis) direction. After correcting these expansion distortions and a minor linear shear distortions, the final ADF image and the final RGB color figure are shown in Fig. 3c and d, respectively. In the nonlinear case, the atomically resolved EELS SI was acquired at the La_xNd_1–xVO₃/NdGaO₃ interface. As it can be seen from Fig. 1g, along the in-plane direction (vertical) the lattice is distorted nonlinearly. Figure 3e shows the final image where the nonlinear image distortion is corrected (without correcting the expansion/compression distortion along the y-direction). Figure 3f is the final warped and cropped RGB color-coded figure of overlaid elemental maps, Nd-M_4,5 (red), V-L_2,3 (green), Ga-L_2,3 (blue) and La-M_4,5 (magenta). Note that the compression distortion is present in Fig. 3f. With the option for correcting the expansion/compression distortion set to ‘on’, the compression distortion is largely corrected as can be seen from Fig. 3g. Individual elemental maps obtained by these two correction procedures can be found in the supporting information (Fig. S3). After correcting these image distortions, the averaged distribution profile along the in-plane lattice direction can be easily obtained from the atomically resolved elemental maps and the discussion of the dopant redistribution or the sharpness of the interface can be further carried out, which is impossible on the raw distorted SI.

Fig. 3.

Examples for demonstrating the correction of linear and nonlinear distortions. (a) and (c) Linear image distortion removed from the raw ADF image shown in Fig. 1e and f, respectively. (b) and (d) Final warped and cropped RGB color figure of the overlaid elemental maps, Cu-L_2,3 (red), La-M_4,5 (green) and Ba-M_4,5 (blue). (e) Nonlinear image distortion removed from the raw ADF image shown in Fig. 1g. Final warped and cropped RGB color figure of the overlaid elemental maps, Nd-M_4,5 (red), V-L_2,3 (green), Ga-L_2,3 (blue) and La-M_4,5 (magenta), without (f) and with (g) correction of the compression distortion along the y-direction.

(2) Warping the distortion for STEM diffraction imaging data

In traditional STEM imaging the signal within a certain range of scattering angles is integrated using an annular scintillator. This procedure poses a severe restriction as a huge amount of information within the electron diffraction pattern (Ronchigram) is discarded. The development of fast-readout cameras has enabled 4D diffraction (4D-STEM) imaging where 2D diffraction patterns are recorded at each probe position in a 2D scan with atomic scale electron probes. Both theoretical and experimental studies suggest that full acquisition of the Ronchigram at each spatial location allows, by post-processing of the data, for multiple imaging modes, will enable super resolution, phase contrast imaging, imaging of internal fields and 3D sample reconstruction [26–30]. Still limited by the frame rate of the camera and by the speed of writing and saving of large datasets on hard disks, 4D-STEM imaging uses longer dwell time than conventional STEM imaging. As a consequence, instrument and sample instabilities create substantial image distortions. These distortions prohibit atomic resolution and limit the interpretability of the result. Moreover, the methods that we summarized for suppressing the scan distortion for STEM imaging and SI, i.e. multi-frame techniques, may not work for the 4D-STEM experiments due to the challenge of big data acquisition, storage, and processing.

Here we demonstrate the feasibility of our STEM SI warp tool for correcting the image distortion of a 4D-STEM dataset. We acquired STEM diffraction data of SrTiO₃ with a Gatan OneView camera using the STEM diffraction imaging module. The 4D (108 × 101 × 128 × 128) dataset, which consists of 10908 CBED patterns with a size of 128 × 128 pixels, was collected with a dwell time of 4 ms. The total acquisition time was 114 s. As it can be seen from Fig. 4, image distortions are clearly present in the 4D-STEM data (Fig. 4a) and in the simultaneously acquired HAADF image (Fig. 4d): the square-shaped SrTiO₃ lattice has been distorted into a parallelogram. By integrating the diffraction image intensity over a specific radial range at each probe position, as schematically shown in Fig. 4b, one can synthesize different STEM images (Fig. 4c), i.e. 40–100 mrad for the ADF image, 11–22 mrad for the annular bright-field (ABF) image, and 0–11 mrad for the central bright-field (CBF) image [31]. The image distortions are identical in these reconstructed STEM images. To remove these image distortions, two approaches for data processing were compared: (i) first reconstruct the images, followed by warping the reconstructed images and (ii) directly warp the 4D diffraction data and then reconstruct the images. As can be seen from the comparison (Fig. 4g and h), the two approaches give identical results. The differences between the images extracted by the two approaches are presented in supplementary Fig. S4 online. These differences are smaller by a factor of 10⁶ compared to the image intensity. This shows that the 4D warping process did not create artifacts within the diffraction patterns. After correction of the image distortion, the dataset also provides the possibility to extract the differential phase contrast (DPC) signal [32] applying numerical segmented detectors [33] at atomic resolution.

Fig. 4.

Correcting the distortion for 4D STEM diffraction imaging (DI) data. Top: (a) the original DI data, (b) position-averaged CBED (PACBED) pattern overlaid by masks indicating radial integration ranges for later STEM image reconstruction, (c) ADF, ABF and CBF images reconstructed from the data in (a). (d) Original simultaneously acquired ADF image, (e) distortion-corrected 4D DI data, (f) distortion-corrected ADF image. Bottom right: Two approaches for data processing were compared; (g) approach 1: first reconstruct the images, followed by warping the reconstructed 2D images; (h) approach 2: directly warp the 4D diffraction data and then reconstruct the images.

Conclusion

We demonstrated the use of the STEM SI Warp software tool for post-correcting image distortions of atomically resolved STEM-SI and 4D-STEM data. Two typical distortions can be corrected, i.e. linear and nonlinear distortions. With prior knowledge of the crystal structure, the linear image distortion is corrected by warping the parallelogram or trapezoid back to a rectangle; the nonlinear image distortion is divided into different subsections. In each subsection, the x-component is corrected as linear distortion and the y-component is corrected to let each unit cell have equal size in y-direction. With practical examples, we demonstrated the correction of these linear and nonlinear distortions, feasibility of the script for 4D STEM diffraction imaging, and the application for maximizing the elemental mapping area. The warp and image-shift functions work for multidimensional data (2D to 4D). This software tool significantly improves the interpretability of distorted STEM spectrum/diffraction imaging data.