Extraction of Hidden Science from Nanoscale Images

Abstract

Scanning probe microscopies and spectroscopies enable investigation of surfaces and even buried interfaces down to the scale of chemical-bonding interactions, and this capability has been enhanced with the support of computational algorithms for data acquisition and image processing to explore physical, chemical, and biological phenomena. Here, we describe how scanning probe techniques have been enhanced by some of these recent algorithmic improvements. One improvement to the data acquisition algorithm is to advance beyond a simple rastering framework by using spirals at constant angular velocity then switching to constant linear velocity, which limits the piezo creep and hysteresis issues seen in traditional acquisition methods. One can also use image-processing techniques to model the distortions that appear from tip motion effects and to make corrections to these images. Another image-processing algorithm we discuss enables researchers to segment images by domains and subdomains, thereby highlighting reactive and interesting disordered sites at domain boundaries. Lastly, we discuss algorithms used to examine the dipole direction of individual molecules and surface domains, hydrogen bonding interactions, and molecular tilt. The computational algorithms used for scanning probe techniques are still improving rapidly and are incorporating machine learning at the next level of iteration. That said, the algorithms are not yet able to perform live adjustments during data recording that could enhance the microscopy and spectroscopic imaging methods significantly.

Graphical Abstract

INTRODUCTION

During the past four decades, computers have evolved as everyday instruments in research laboratories. Experimentalists benefit from data digitization that has fundamentally changed the process of data acquisition, analysis, and presentation. Theorists have leveraged the computational power and memory space of modern computers to obtain results from physical models previously unsolvable by pen-and-paper approaches. Nanoscale imaging is one of the fields revolutionized by the use of computers. One example that illustrates the power of the computer-aided processes is super-resolution fluorescence microscopy. Invented and pioneered by Hell,^1,2 Betzig,³ Moerner,^4,5 (S.) Weiss,⁶ Xie,⁷ Zhang,^2,8 and others, super-resolution microscopy took advantage of real-time computer fitting of point spread functions of the excitation laser profile to overcome the optical diffraction limit. This technique is now driving innovations throughout biological research,^2,9 including exploring protein localization patterns in bacteria,¹⁰ unraveling amyloid aggregates structures,¹¹ and direct observation of DNA and RNA dynamics in cells and bacteria.^12,13

Similar to super-resolution microscopy, scanning probe microscopy (SPM) also uses point-wise nanoscale data to reconstruct entire images. However, instead of shaping light beyond the diffraction limit, a nano-sized sharp tip is rastered near surfaces—SPM is a surface imaging technique. Depending on the feedback mechanism, different observables can be mapped along with topographical information, such as surface electronic structure, mechanical interactions, and electrochemical reactivity, via scanning tunneling microscopy (STM), atomic force microscopy (AFM), and scanning electrochemical microscopy (SECM), respectively, as well as a host of spectroscopic imaging methods that have been developed and applied.^14–22 The introduction of laser excitation into SPM adds another data dimension and enables hyperspectral investigation of charge-transfer phenomena and molecular vibrations at the single-molecule level.^23–26

The rich information content and the intrinsic oversampling nature of SPM images serve as ideal platforms for developing computer-aided imaging processing routines. This mini review covers the development, acquisition, and implementation of computer-aided imaging acquisition and processing in nanoscale surface images. Specifically, we focus on recent advances in the improvement of scanning patterns and data segmentation of SPM images. This mini review aims to give a general understanding of the progress of computer-aided image processing of SPM images and to look ahead to fully automated information extraction from multimodal date led by machine learning and artificial intelligence (vide infra).

METHODS

Image Acquisition

All SPM techniques share the same imaging principle of using a sharp tip as the probe for nanoscale surfaces. The first member of the SPM family is STM. A typical STM setup consists of a conductive/semiconducting surface and an atomically sharp metal tip brought within close proximity. When a voltage is applied across this nanoscale probe tip-sample junction, there is a net current of tunneling electrons from one side to the other (once any energy gaps are exceeded by the energy of the tunneling electrons). Moving the tip closer to the surface, the probability of electrons tunneling from one side to the other, and thus the current, increase exponentially. Using a feedback loop that maintains a constant tunneling current, the tip-sample distance is automatically adjusted to compensate for nanoscale changes in the surface electronic density of states. By recording the tip height at every scanned pixel, an STM topographic image can be reconstructed. The most common scan pattern is to raster the tip across the surface in a line-by-line fashion (Figure 1a).²⁷ For the first STM image, analog traces of piezoelectric locations were recorded by a chart recorder. Prior to computers, analog traces of piezoelectric locations were recorded by a chart recorder. A three-dimensional surface model was assembled from the original recorder traces with scissors, plexiglass, nails, and a camera to record the pictures of the glued model at different views (Figure 1b).^28,29 Later images have been recorded directly to computer and assembled in software.

Figure 1.

(a) In a linear tip raster pattern, the probe is moved in one direction (orange arrows), is turned around (red arrows) to retrace the line (green arrows), and then advanced to the following line (blue arrows). (b) The first scanning tunneling microscopic observation on the atomic reconfiguration of surface atoms of Si{111}. Image (a) reproduced with permission from ref 27. Copyright 2017 AIP Publishing. Image (b) reproduced with permission from ref 28. Copyright 1987 American Physical Society.

One of the major problems with the scanning pattern from Figure 1a, is the piezoelectric creep and hysteresis that occurs when the velocity of the tip changes by 180° at the end of the trace path (pink arrow in Figure 1a). This sudden change in direction distorts the image and is a time-inefficient method for scanning because the tip needs to return to its original starting position before the next line can be acquired. Some of these issues can be minimized by using a different scanning paradigm or by image processing.

An important contribution to image processing started with mitigating the effects of piezoelectric transducer creep and hysteresis, as STM and AFM both use piezoelectric materials to move the scanning probe tip.^30–32 Similarly, Yothers et al. and others have used image post-processing to decompose the components of thermal drift, hysteresis, and creep from their STM images.^27,33,34 The processing used a linear transformation correlating the actual tip location with the tip location shown on their images. Matrix elements are used to fit and to minimize the apparent distortions, which enabled recording images to larger scales that are more true to the actual surface. Using an algorithmic approach to image distortions can improve the quality of images acquired through SPM techniques; however, they do not improve the efficiency of image acquisition. In addition, changing the image path algorithm can minimize the effects of hysteresis and creep.

As discussed above, many SPM experiments are performed by rastering a scanning-probe tip across the surface in the fast direction, labeled as trace in Figure 1a, then the tip returns to its original position before it moves one pixel in the slow direction to begin the process again. Though the instrumental design is simple, it also introduces the distortions of creep and hysteresis. Researchers, including Moheimani, have developed a variety of non-rastering patterns to speed up acquisition times.^35,36 Ashby and coworkers engineered a version of the spiral raster pattern shown in Figure 2a that is more efficient in time and limits distortion effects.³⁷ By using a mixture of constant linear velocity and constant angular velocity, they have shown an improvement to the simple raster mechanism. To capitalize on the entire image area, they use a constant angular velocity when the tip is close to the center of the image then transition to a constant linear velocity around the edges. Figure 2b is an example of an optimized spiral scanning method with the large image corresponding to the complete scan area. The images to the right correspond to the hashed regions of the total scan area. While varying the scan pattern is not novel,³⁸ Fleming and coworkers found that using non-raster based patterns decreased the time required to image a surface at the expense of an increased vertical bandwith.³⁹ These varied scan patterns also enable researchers to undersample their images decreasing the time required for each subsequent scan.^39,40 For other AFM scanning methods, we invite the reader to read the review by Das et al.⁴¹

Figure 2.

(a) Spiral tip raster pattern implemented for addressing the raster scan paradigm, the probe is moving spirally to minimize the change in acceleration. (b) Optimized spiral-scanning image the three box overlays correspond to the center, middle, and the peripheral imaged regions. Adapted with permission from ref 35. Copyright 2017 IEEE.

Data Segmentation

In addition to improvements to simple image processing and data acquisition, computational methods have also been used for advanced data analysis in STM, particularly in fusing information from different spectroscopic imaging modes. The STM images reveal rich information about the structure of self-assembled monolayers (SAM) since they convey the chemical and physical properties of the studied material. However, the STM images are a convolution of the electronic and geometric information. This convolution makes the images challenging to analyze by a simple computational routine (Figure 3a), such as a planar background subtraction or simple Fourier filters.

Figure 3.

(a) Illustration of the mix of information in a scanning tunneling microscope (STM) image. We observe intensity differences over large regions as well as regions with different textural patterns. These two types of information correspond to specific chemical and physical properties. (b,c) images display the separated cartoon, u, and texture, v, components extracted from the STM image, while images (d,e) show the corresponding segmentation results. Image (a) reproduced with permission from ref 42. Copyright 2016 American Chemical Society. Images (b,c,d,e) reproduced with permission from ref 43. Copyright 2019 Springer Nature.

In Figure 3a, intensity differences occur over large regions in addition to textural (oscillating) patterns. Therefore, the ability to separate regions corresponding to variations of these features (such as significant changes of intensity or textural patterns at different scales and/or orientations) is of the utmost importance in extracting the physical and chemical properties of the observed materials. These properties can include buried functionality and interactions, molecular orientation, tilt angles of individual molecules, molecular conductance, contact conductance, charge redistribution and long-range order of molecular domains.^17,44–51

To extract chemical properties,⁴³ STM images can be analyzed using an automated image-processing framework. Since large constant intensity regions and textures carry different information, the first step consists of performing a cartoon (the term cartoon is used since large constant intensity regions look similar to actual cartoon art) and texture decomposition.

The image processing community has widely studied this type of decomposition following the seminal theoretical work of Y. Meyer.⁵² Most decomposition models consider that the cartoon part (denoted u) and the texture part (denoted v) belong to specific function spaces reflecting their intrinsic characteristics.^53–56 Denoting f, the input STM image, these models consist of minimizing some function based on the norms of these spaces, subject to the constraint that f = u+v. A linearized model⁵⁵ was executed for the simplicity of its numerical implementation as well as its low computational cost (described in detail in ref 57).

Figure 3b and 3c illustrate the cartoon (u) and texture (v) parts extracted from the input image f depicted in Figure 3a, respectively. The next steps are to design segmentation algorithms specialized for each type of component. First, the cartoon part u is segmented using the Local Multiphase Chan-Vese model.⁵⁸ The Ginzburg-Landeau approximation⁵⁴ was used to further reduce the expensive computation cost, which can be efficiently computed via the Merriman–Bence–Osher (MBO) scheme.^52,60 Figure 3d illustrates the segmentation of the cartoon part given in Figure 3b.

In the image processing community, textures are recognized as the most challenging objects to segment. This challenge is predominantly due to their variability, which avoids the existence of a theoretical model of textures. However, it is widely accepted that textures contain characteristics at different scales and eventually with different orientations. For these reasons, wavelets (and in particular curvelets)^59,61 have successfully extracted textures features used to feed some classifiers. Despite these successes, since standard wavelets (or curvelets) correspond to filter banks corresponding to fixed partitioning of the Fourier domain, they are not optimal in capturing the full variability of textures. To remedy this issue, the recently introduced empirical wavelets have been applied for segmentation.^62,63

The advantage of empirical wavelets lies in the fact that they are adaptive (i.e., data-driven) wavelets. The wavelets automatically detect the supports, in the Fourier domain, of each wavelet filter leading to the construction of a wavelet decomposition that is “optimal” to extract harmonic (i.e., oscillating) modes. Several types of wavelet, including curvelet, transforms have been revisited and generalized to empirical transforms. Given that both scales and orientations are important textural characteristics in STM images, one can use the energy of the obtained empirical curvelet coefficients to build the expected texture features (note that the superiority of such approaches has been demonstrated for more general texture segmentation/classification tasks in refs 64 and 65).

Finally, feeding the texture features into a multiclass MBO classifier results in the segmentation of the texture component. For example, Figure 3e illustrates the result corresponding to the texture component given in Figure 3c. Note that the algorithm detects the different textures and identifies the boundary regions between textures. In SPM images such boundaries are important clues to structural and other properties.^18,66–71

Segmentations obtained from the cartoon and texture parts provide complementary information that is beneficial in understanding the studied material’s physical and chemical properties.⁴⁷ The complete mathematical description and many more examples showing the efficacy of the proposed framework are available in ref 43. The corresponding Matlab code is freely available at https://github.com/kbui1993/Microscopy-Codes.

For other advanced computational SPM image processing techniques, we direct readers to more thorough reviews by Kalinin and coworkers for detailed discussions of machine learning in analyzing AFM images.⁷⁰

Using image segmentation, we have been able to differentiate nanoscale surface features in STM images that may be difficult to perceive visually.⁴⁶ In the field of chemistry, the reactive sites tend to be at the defect sites, including domain boundaries. Specifically, for extracting the chemical information in Figure 3a, we used the aforementioned wavelet algorithm to distinguish regions with similar angular coordinates and frequencies (Figure 4B).⁴² This image processing method enables grouping regions of the images with similar chemical information, such as the bonding directions of the domains. The regions with different chemical interactions have been given false colors (Figure 4C) to signify the different angular directions of the domains (Figure 4D). Note that multiple domains have similar spacings and orientations. However, viewing these images without these highlights makes grouping of similar domain regions difficult.

Figure 4:

a) Scanning tunneling microscopy image of a thermally annealed sample of AuCN on Au/mica substrate b) Wavelets were used to separate the components of image (a) by angle and frequency. c) False-color reconstruction from wavelet algorithm in (b). Images below show texture relating to the Fourier space below each texture. Reproduced with permission from ref 42 Copyright 2016 American Chemical Society.

We also used empirical wavelets segmentation to separate the textures of the STM image shown in Figure 5A. This process enabled correlating subdomain characteristics of the surfaces to molecular orientations (Figure 5B). Although some subdomain features are easily visible without processing, the interfacial regions can be more challenging to ascertain. Three regions have been assigned where regions 1 and 2 are well-ordered domains with an offset of 120°. Here, classes II, IV, and VI correspond to region 1, and classes I and V correspond to region 2. Finally, class III is composed of the disordered regions that tend to be more reactive than the ordered regions and are likely locations for surface manipulation.

Figure 5:

(A) Scanning tunneling microscope image of a self-assembled monolayer of AuCN on a Au/mica substrate (B) Cartoon image of three domains of image (A) composed of six classes (shown above). Adapted with permission from ref 42. Copyright 2016 American Chemical Society.

If complementary experimental measurables can be obtained simultaneously, for example, local barrier height measurments (LBH) and conventional topographic images, chemical information can be extracted by computing the correlations between the two spatially registered data sets. The connections between these two measurements enable resolution of long-range order within SAMs by correlating local maxima of the topography to local maxima of the LBH (Figure 6), which we have shown correspond physically to the exposed and buried ends of the molecules, respectively. Given two hypothetical SPM images containing local maximum data (i.e., p, the blue dots in the topography image and q, the red dots in the LBH image in Figure 6), the corresponding coordinates of p are grouped with q using a block-matching approach. This process is possible because both the topological image and the LBH are acquired simultaneously. The black square and the dashed black square are search windows used for the block-matching process. This procedure is repeated for each p, so p’s and q’s can be linked into a set of vectors that represent the orientations of the individual molecules in the SAM.

Figure 6:

Local maxima in the topography image and local barier height image are defined as p (blue dots) and q (red dots) respectively. The block-matching approach is used to match p and q coordinates. The red square represents the rastering direction. Adapted with permission from ref 46. Copyright 2015 American Chemical Society.

This approach was also used on self-assembled carboranethiol monolayers on Au{111} (Figure 7). Carboranes are composed of carbon–boron cage molecules with each atom bound to one hydrogen; the carboranethiols produce well-ordered monolayers on Au{111} with few defect sites, which by symmetry are limited to translational and rotational offsets in the registry of attachment in complete monolayers.⁴⁶ However, depending on the locations of the carbons within the caged molecules, there can be a dipole at a variety of angles with respect to the surface normal. Surprisingly, we found that the dipoles within neighboring carboranethiol molecules align within multiple structural domains at low temperature.

Figure 7:

(A) Scanning tunneling micrograph of o-9-carboranethiol (O9) on Au{111} where the blue dots show the local topographic maxima. (B) Simultaneously acquired local barrier height (LBH) image, which is inverted to highlight dipole orientations, with computed local maxima (red). Insets in (A) and (B) depict the fast Fourier transforms (FFTs) of the images. (C) Overlay of the topographic image with the computed molecular dipole orientations. (D) A ball-and-stick model of O9 where hydrogen atoms are omitted for clarity. Adapted with permission from ref 46. Copyright 2015 American Chemical Society.

RESULTS AND DISCUSSION

Hydrogen-Bonding Network and Images

Another use for LBH measurements is to elucidate hydrogen-bonding networks. We studied the long-range hydrogen-bonding networks buried within SAMs using image segmentation.⁴⁴ Similar to how the dipole directions were measured in Figure 7, the tilts of molecules were measured by correlating the centers of the molecules to the LBH measurements. Using 3-mercapto-N-nonylpropionamide (ATC9), which contains a buried amide group within the monolayer, we found that there were two phases that formed on Au{111} relating to the tilt of the molecules—one with a tilt of 18° while the other was essentially normal to the surface (Figures 8 and 9, respectively). The STM topographic images (Figures 8A and 9A) of both phases were difficult to differentiate; however, the LBH measurements enabled us to assess the relative positions of two parts of the molecule—the top of the molecule and the hydrogen-bonding amide functional group. Schematics of the molecules on the surface are shown in Figures 8C and 9C. It is apparent when the top and bottom of the molecules are not directly overlapping (as can be seen in Figure 8C), the molecular domain has a tilt with respect to the surface normal. Similarly, the local topographic maximums were generally less than 3 Å apart from the LBH maximums within the 0° SAM.

Figure 8:

(A) Scanning tunneling microscope topographic and (B) simultaneously acquired local barrier height (LBH) maps over an area of the more tilted (18°) structure of 3-mercapto-N-nonylpropionamide (1ATC9), with respect to the underlying Au{111} surface. The local maxima of both topography (red) and inverted LBH (blue) in B are computed. Insets depict fast Fourier transforms showing the expected topographic hexagonal nearest-neighbor spacing, which is also found within LBH images. (C) All maxima were connected within a defined radial range and orientation; best fit molecular orientations show the expected polar tilt angles. (D) A ball-and-stick model of 1ATC9 showing a polar chain tilt of 18° (for the molecular segment above the amide) and amide bonds nearly parallel to the substrate. Adapted with permission from ref 44. Copyright 2016 American Chemical Society.

Figure 9:

(A) Scanning tunneling microscope topographic and (B) simultaneously acquired local barrier height (LBH) map over the normally oriented (0°) structure, with respect to the underlying Au{111} substrate, within monolayers of 3-mercapto-N-nonylpropionamide (1ATC9). Local maxima in both topography (red) in A and inverted LBH (blue) in B are computed. Insets depict fast Fourier transforms of both topography and LBH images. (C) Computed molecular orientations overlaid onto the LBH map. (D) A ball-and-stick model of 1ATC9 normally oriented on Au substrates. Adapted with permission from ref 44. Copyright 2016 American Chemical Society.

In addition, it is possible to observe both domains within single image frames. Figure 10A and 10B correspond to a region where both domains, 0° and 18°, are in view, where Figure 10A is the topographic measurement and 10B is the LBH measurement. The topographic image appears to have multiple structural domains, however no orientation information can be obtained simply from the image itself. Using the simultaneous acquisition of the LBH measurements, however, the two domains were segmented and stitched together. The blue-shaded region corresponds to the normal phase of the molecules while the red-shaded region corresponds to the 18° tilt phase (Figure 10C). Therefore, to move beyond topography, which contains little chemical information, we can identify the molecular domains and the chemical orientations by correlating the topography with another measurement (in this case LBH). Experimental details and procedures can be found in ref 44.

Figure 10:

(A) Scanning tunneling micrograph of 3-mercapto-N-nonylpropionamide on Au{111} along a tilt. (B) Simultaneously acquired local barrier height (LBH) image of A. (C) We separate the upper (highlighted as red) and lower (highlighted as blue) domains in LBH based on relative work function differences. (D) Image histogram of C showing the energy cutoff used that was also fit with two Gaussian curves to solve for peak-to-peak image contrast differences. Adapted with permission from ref 44. Copyright 2016 American Chemical Society.

To our surprise, we found that the hydrogen-bonding networks continued across structural domain boundaries and even regions of disorder.⁴⁴ These observations of robust hydrogen-bonding networks explain the passivating effects of amide-containing alkanethiols, which were observed but not understood at the time of our prior work on the ultrasonic preparation of eutectic gallium-indium (EGaIn) liquid metal nanoparticles.⁷¹ In that case, when amide-containing monolayers were uses to protect the liquid EGaIn nanopartices as they were split (calved) to make smaller particles, little oxidation was observed. When n-alkanethiol capping ligands were used, there was significant oxidation of the EGaIn particles.

We have shown two examples in which chemical properties (work function, dipole moment directions, hydrogen-bonding directions, molecular orientations, etc.) were interpreted by using STM and LBH together. These measurements paint a chemical picture more detailed than simply the molecular location that traditional STM measurements yield. In order to understand how molecules are interacting with each other and their substrates, an understanding of the chemical environment is needed.

Functional Materials Properties

Although beyond the scope of the discussion here, we point interested readers to tremendous advances in using analogous machine-learning approaches to explore and to map electrostatic, ferroelectric, liquid crystal orientation, and other properties of functional materials.^72–77 Among the pioneers in this area are the Oak Ridge National Lab team, led by Kalinin and co-workers.

CONCLUSIONS AND PROSPECTS

Scanning probe techniques have opened up the atomic-scale world to exploration, including being able to measure bonding and other interactions between molecules and parts of molecules. Increasing power in computational algorithms in the forms of scanning methodologies, fusing complementary information from different imaging modalities, image processing, and image segmentation help elucidate chemical properties at these and larger scales. By correlating the orientations and continuity of interactions, new opportunities beyond chemical bonding distances have been uncovered, which have significant consequences in surface functionalization and passivation, structural biology, and other areas. With this understanding of extended interactions, better materials and assemblies can be designed and fabricated.

Further improvements in these methods will dramatically improve our ability to “see” and to explore the nanoscale world in multiple physical, chemical, and functional dimensions. One key step will be efficiently fusing together multiple acquisition modalities to extract information and understanding. The next generation of SPM techniques will involve more computational algorithms. These opportunities include compressive sensing methods to minimize the number of points acquired without losing information^78,79 and by using automated algorithms to manage the processes of sample preparation and image acquisition,^80–82 all of which will increase throughput and efficiency in acquiring a “complete” data set in terms of understanding the system under study.

As machine learning improves, gaining efficiency and speed, our ability to use algorithmic platforms will change the way both image collection and image processing are conducted. We target adaptive, real-time, image processing, so as to guide data acquisition to be more information rich and so as to be able to test and to constrain interpretations optimally, as data are being recorded. Though it is improving quickly, the technologies, as of this writing, are not yet able to make strong judgments about images without large libraries of training data. It is more realistic that, in the short term, we will use computational algorithms to increase the efficiency of image acquisition and separate images by quality and features. We will then be able to review images to understand how molecules and assemblies arrange, and with the support from complementary techniques, elucidate the chemistry of the interactions.