Semantic segmentation in crystal growth process using fake micrograph machine learning

Abstract

Microscopic evaluation is one of the most effective methods in materials research. High-quality images are essential to analyze microscopic images using artificial intelligence. To overcome this challenge, we propose the machine learning of “fake micrographs” in this study. To verify the effectiveness of this method, we chose to analyze the optical microscopic images of the crystal growth process of a Ge thin film, which is a material in which it is difficult to obtain a contrast between the crystal and amorphous states. By learning the automatically generated fake micrographs that mimic the crystal growth process, the machine learning model can now identify the low-resolution real micrographs as crystalline or amorphous. Comparing the three types of machine learning models, it was found that ResUNet ++ exhibited high accuracy, exceeding 90%. The technology developed in this study for the automatic and rapid analysis of low-resolution images is widely helpful in material research.

Introduction

The advancement of modern electronics has been supported by the progress in thin-film synthesis techniques for functional semiconductors. With the emergence of material informatics, which integrates material science and machine learning (ML), research on semiconductor thin-film crystal growth is entering a new stage^1–7. Bayesian optimization, a robust method for the mathematical and global optimization of black-box parameters, has already been used in the efficient exploration of composite materials and multi-parameter systems^8–11. In analyzing seemingly uncorrelated experimental conditions and measurement results, which is a challenge in materials research, dimension reduction methods such as principal component analysis and/or sparse modeling have provided valuable insights and understanding^12–16. By incorporating extensive databases and expert knowledge into the learning process, ML can automatically and swiftly identify materials from complex spectra and images^17–21. Introducing ML potentials has reduced computation time remarkably, addressing a significant bottleneck in the field of first-principles calculations^22–25.

Microscopic observation is one of the most common methods used in material evaluation, and applying ML to analyze various types of microscopic images have been proposed^26–29. In many semiconductor thin films, the state of the crystal determines its properties and microscopic observations have been used to distinguish between crystalline and amorphous areas. Differential interference microscopy efficiently contrasts crystals and amorphous materials because it highlights the differences in the refractive index through polarization. A technique for automatically clustering domains from high-quality microscopic images was also proposed^30,31. However, in many optical microscopes, it is difficult to distinguish between crystalline and amorphous areas. In-situ annealing observation through long-focus digital microscopes is an example, as shown in Fig. 1. Although this method helps to understand the solid-phase crystallization of amorphous thin films, obtaining high-resolution images with the contrast between the crystalline and amorphous areas is difficult. As shown in Fig. 2a–e, the in-situ observations of the solid-phase crystallization process are also tricky for the human eye to distinguish. However, Fig. 2f indicates that there are numerical differences in color distribution between amorphous (Fig. 2a) and crystalline (Fig. 2e) images. Therefore, traditionally, these microscopic images require color correction, followed by a time-consuming analysis based on the intuition of experienced researchers to distinguish between crystalline and amorphous areas.

Figure 1.

Concept of the study. The crystallization process was observed using an in-situ annealing system. The traditional method involves the manual detection of crystalline regions, which are difficult to discern in low-resolution micrographs. In contrast, this study introduces an ML automated process, enhancing clarity, efficiency, and reproducibility in distinguishing between amorphous and crystalline states.

Figure 2.

Representative in-situ micrographs showing the crystallization process of a Ge thin film. Micrographs obtained by in-situ observations for annealing time t = (a) 0, (b) 330, (c) 380, (d) 430, and (e) 600 min, where the annealing temperature was 350 °C. (f) Color histogram of the micrographs at t = 0 (solid line) and 600 min (dashed line).

Previously, we proposed a “fake micrograph” technique that automatically generates numerous images of the crystallization process and constructs an ML framework that calculates properties related to crystallization more accurately and quickly than humans²⁹. Building on the fake micrograph technique, this study proposes a novel semantic segmentation method for determining the crystalline and amorphous areas in low-quality micrographs. A suitable ML model trained on fake micrographs can automatically and rapidly generate images with clearly differentiated crystalline and amorphous areas from low-quality real micrographs.

Results and discussion

Experimental outline

Figure 3 shows a flowchart of the learning and prediction processes. The proposed analysis framework consists of two stages: (i) learning fake micrographs using an ML model and (ii) recognition rael micrographs using an ML model trained on fake micrographs. We aimed to develop an ML model that can accurately perform semantic segmentation of real in-situ micrographs. The model should reliably classify each pixel into one of the two categories: amorphous, represented as black with a value of zero, or crystalline, represented as white with a value of one. In this study, a Ge thin film synthesized on a glass substrate was used. It is particularly difficult to observe the contrast between the crystalline and amorphous phases in the optical microscopy images of Ge films. For example, in a typical microscope using a white light lamp (color temperature 3100 K), the difference in refractive index between crystalline and amorphous Ge is only 0.03, compared to 0.13 for Si³². This method of forming Ge thin films has already been reported^33,34. Fake micrographs, which mimic the growth process, were automatically generated by combining the surface micrographs of the samples before and after annealing, that is, wholly amorphous and crystalline on each surface. Previously, we used fake micrographs of overlaid circles cut from the crystalline images on the amorphous images²⁹. However, the boundary of the circle pasted as crystalline had sharp edges, resulting in unrealistic discrepancies in color and brightness. Additionally, in real in-situ annealing observation systems, the captured images of the sample surface are compressed and saved in JPEG format. These compressed images of amorphous and crystalline structures were used to create synthetic images. Although using JPEG format, which involves irreversible compression, may lead to degradation in image quality, there was hardly any difference in the resolution of in-situ micrographs before and after JPEG compression in our experimental system. This is because the dominant factor in image degradation is the long focal length of the microscope used in the in-situ annealing observation system. Due to the inclusion of an annealing apparatus, the microscope’s focal length must be extended, making it challenging to enhance the quality of observation images. The microscope used in this study combines a long focal length and high resolution at the highest level of current technology. This limitation in image quality, originating from the microscope, results in the domain contours of the synthesized fake images having finer edges than the block noise typically associated with the JPEG format. Therefore, in this study, we tested post-processed fake micrographs by adding blur effect to the fake micrographs as an augmentation during learning to make the process more similar to the actual system^35,36. This augmentation not only aligns the saved images and formats but also naturally blurs the edges of the crystal domains in generated fake micrographs. It is anticipated that this will enhance the model performance when analyzing more realistic image data. We used JPEG compression as post-processing for blurring. Typical results shown in Fig. S1 indicate that the edges of the contours are blurred due to the post-processing, making the fake micrographs more similar to the real micrographs. Other filters, such as low-pass filtering, color subsampling, and Gaussian blurring, may also enhance the efficiency of learning. Three representative ML models were considered for segmentation: base model³⁷, U-Net³⁸, and ResUNet ++³⁹, which are, respectively, an initially developed segmentation model, a model with a symmetric structure developed for application in biomedical engineering, and a model that adds residual connections to U-Net to alleviate the vanishing gradients problem and facilitate learning⁴⁰. The data size and format determine the optimal ML model^41,42. By training these models with post-processed fake micrographs, we attempted to segment real micrographs obtained during in-situ annealing observations, aiming for the automatic and rapid generation of images with enhanced crystalline/amorphous areas.

Figure 3.

Schematic of the experiment. The learning process begins with generating fake micrographs using wholly amorphous and crystalline phase images from real micrographs. These are subjected to post-processing to blur the edges the crystal domains in generated fake micrographs. The architecture and connection of layers within the ML models are illustrated. During the recognition process, the models are fed with real micrographs for semantic segmentation of crystal domains.

Evaluation of learning and inference processing

In our validation metrics, we used combo loss, defined as the sum of the Dice loss and cross-entropy loss⁴³. Figure 4 shows the learning curves while changing the model type and with or without post-processing. In all the cases, as the number of epochs increased, both the training and validation losses decreased. This suggests that the training using the fake micrographs was successful. A detailed comparison of each condition revealed the following trends. (i) The base model showed a slight gap between the training and validation losses, suggesting poor validation accuracy. (ii) U-Net exhibited large fluctuations during validation. (iii) ResUNet ++ demonstrated the lowest validation loss. These differences between the models likely stem from differences in structure, such as the form of connections between each layer and the output of the final layer. Conversely, the similarity in the final loss values observed between the same models in Fig. 4a,b can be surmised as due to the fixed presence (w/) or absence (w/o) of post-processing for the image sets during both training and validation under each condition. Models trained on post-processed images are evaluated on post-processed images, and vice versa; therefore, the values of this loss may not directly correspond to the predictive performance on real in-situ micrographs.

Figure 4.

Evaluating model performance. The graphs compare the training and validation losses across epochs for different ML models, depicting results (a) without and (b) with post-processing applied. The models compared include the base model, U-Net, and ResUNet++. Training loss is denoted with solid lines, while validation loss is represented with dashed lines, indicating the convergence behavior of each model.

Figures 5a–c compare the segmentation results obtained by the ML model trained under each condition with the manually annotated images and display the accuracy of the results as a function of the annealing time t. Additionally, we have presented the Dice coefficient and Intersection over Union, which are representative performance evaluation metrics for segmentation tasks, in Fig. S2. The manually annotated image results from a researcher examining real micrographs to classify each pixel as either amorphous or crystalline. It should be noted that the annotated images used in this study were created manually from low-resolution in-situ micrographs and do not always represent precise imagery. To obtain high-resolution images at the exact same positions as the in-situ micrograph, combining ex-situ measurements with microfabrication is conceivable. However, some crystal growth may occur during the cooling process required to remove the sample. For this reason, creating images that can be regarded as “ground truth” representations corresponding to in-situ micrographs is challenging. The ML model does not use any manually annotated images to learn segmentation because the purpose of this study is to implement a quick and objective analysis using fake micrographs. Recall refers to the ability to identify the crystalline pixels accurately. Precision quantifies the proportion of the pixels that were predicted to be crystalline and truly crystalline. The pixel-accuracy (PA) quantifies the agreement between the output labels and manually annotated images. These are standard metrics in the field of biomedical imaging diagnostics^39,44. In the region corresponding to the amorphous images, t < 200 min, the behavior of all pixels being zero in the manually annotated images was observed, for example, precision = 0. The PA showed high accuracy for models with post-processing. This suggests that the ML models could more accurately classify into amorphous and crystalline areas. In the region corresponding to the crystalline images, t > 500 min, almost one value was shown, except for the predictions by the base model trained without post-processing. This shows that almost all regions were accurately predicted to be crystalline while recognizing the image when the entire sample was crystallized.

Figure 5.

Comparative performance metrics of model predictions. Performance metrics, including (a) recall, (b) precision, and (c) pixel-accuracy, are depicted for the base model, U-Net, and ResUNet++ with and without post-processing. These metrics were computed by comparing the output of the ML models to the manually annotated images during the annealing time t = 200–500 min. (d) Bar plots summarize these metrics under each experimental condition and illustrate the distribution of maximum, third quartile (Q3), median, first quartile (Q1), minimum, and outliers for each model and post-processed case.

Figure 5d summarizes the statistical values of the metrics under each condition for t = 200–500, which is the most critical period for analyzing the crystal growth. Recall showed higher values without post-processing, whereas precision and PA were higher with post-processing. This difference numerically suggests that without post-processing, there is a possibility to mistakenly infer a wide area, including the amorphous area, as “crystallized.” In comparing PA, both the minimum value and the third quartile improved when post-processing was applied. Notably, the model that exhibited the highest median was ResUNet++ (PA = 90.0%).

Figure 6 shows the input and segmentation images obtained as output from the model trained under each condition. The original images correspond to real micrographs captured using an in-situ observation microscope. The normalized image equalizes the color histogram of the original image. The manually annotated image was an image annotated manually from the normalized image, which required approximately 20 min per image. The gradation from the top left to the bottom right in the normalized image reflects the direction of the light source. The output of the base model mostly blurred the outlines of the crystalline areas in both with and without post-processed cases and failed to produce the expected output. Unlike U-Net and ResUNet++, the base model lacks bottom-layer feature preservation using upsampling layers and concatenation, which leads to poor object position detection⁴⁵. This is also supported by the discrepancy in the training and validation losses, as shown in Fig. 4. In both U-Net and ResUNet++ without post-processing, the output reflected the light effect. However, with post-processing, this effect was appropriately removed in the output, contributing to a notable increase in the PA, as shown in Fig. 5d. In the curve shown in Fig. S2, the values only approach one at the final point of the annealing time when everything in the image becomes crystalline. While such behavior could be explained if the network were simply predicting “crystalline” for all pixels, the pixel-accuracy in Fig. 5 and the observations from Fig. 6 suggest that these metrics do not proportionally reflect the actual number of crystalline domains in the images, indicating that recognition is being performed with higher precision. The fact that prediction accuracy improved despite post-processing not having the capability to remove gradients suggests that, in learning tasks for segmentation, it is crucial that the trained fake micrographs faithfully mimic the real in-situ micrographs. From these output images, it was possible to discern the size and growth patterns of individual crystal domains with reasonable accuracy. Considering that the ML models used here were not pre-trained, there is significant potential to enhance accuracy by utilizing transfer learning⁴⁶. These results strongly support the importance of training post-processed fake micrographs to improve the accuracy of crystalline-area recognition.

Figure 6.

Visual assessment of model predictions. The first and fifth columns display the original micrographs at the start (t = 0 min, entirely amorphous) and end (t = 600 min, fully crystalline) of the annealing process, respectively. Columns two to four show the progression of the crystal domain growth at intermediate annealing times (t = 330, 380, and 430 min). The rows correspond to original images, normalized images, manually annotated images, and the segmentation results of the base model, U-Net, and ResUNet++ models, without (w/o) and with (w/) post-processing.

Conclusions

We aimed to develop a framework for the high-precision analysis of low-quality microscopic images, focusing on the crystal growth of Ge thin films, where distinguishing between the crystalline and amorphous states is challenging. The ML model trained on “post-processed fake micrographs” successfully identified crystalline and amorphous areas in low-quality real micrographs. Post-processing made the fake micrographs more similar to the real micrographs and significantly increased the accuracy. Among the three ML models, ResUNet++ demonstrated the highest accuracy (median: PA = 90.0%). Therefore, training on post-processed fake micrographs using the appropriate model enabled the automatic and rapid generation of images from low-quality real micrographs with clarified crystalline and amorphous areas. This semantic segmentation technique is likely to contribute to numerous material research fields that utilize microscopic observations.

Methods

Sample preparation and observation

A 200 nm thick As-doped amorphous Ge layer with an As concentration of 1.2 × 10²⁰ cm⁻³ was deposited on a SiO₂ glass base at a rate of 1.7 nm min⁻¹ in a vacuum evaporation system that operated at a base pressure of 1 × 10⁻⁷ Pa. Doping impurities into amorphous Ge aims to expand the domain size and enhance domain visibility in the resulting images^34,47. In-situ observation involved placing the samples in a furnace (Linkam 10042D) under a N₂ atmosphere and subjecting them to annealing at 350 °C for 200 h. The furnace was equipped with an integrated digital microscope (Keyence VH-5500) with an extended focal range, facilitating the sample viewing through the viewing portal of the furnace.

ML model deployment

We implemented three ML models (base model, U-Net, and ResUNet++) for segmentation by leveraging the PyTorch framework⁴⁸, where the architectures employed are identical to those described in Refs.^37–39. The input images were processed into 224 × 224 × 3 (height × width × channel) shapes, and the final layer output was 224 × 224 × 1. We chose to use RGB color images as input to provide an algorithm applicable to various experimental systems. During the training phase, we utilized 12,800 fake micrographs, where images of amorphous domains measuring 50 μm by 50 μm were superimposed with crystalline domains. These domains were synthesized according to a uniform probability distribution in terms of size, number, and placement (domain size < 120 μm, nuclei density < 1.3 × 10¹¹ cm⁻³). The learning rate was set at 1 × 10⁻². These models were trained using a batch size of 32 to prevent memory overflow. Computations were performed on an NVIDIA Tesla P100 GPU. The number of training epochs was limited to a maximum of 50. These performances were assessed using fivefold cross-validation.

Author contributions

T.I. and K.T. conceived of and designed the study. T.I. fabricated the samples, performed the analyses, and implemented the ML models. K.T. and T.S. managed and supervised the study. All the authors discussed the results and commented on the manuscript.

Data availability

The datasets used and/or analysed during the current study available from the corresponding author on reasonable request.

Competing interests

The authors declare no competing interests.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-024-70530-3.