Uncertainty aware machine learning for bridging simulation and experiment in high throughput materials characterization

Abstract

High-throughput materials characterization is essential for accelerating materials discovery. To enable high-throughput characterization, machine learning (ML) has been a powerful tool. However, the broader application of ML in experimental settings is limited by key challenges, i.e., the scarcity of labeled experimental data and the lack of uncertainty estimation in model predictions. In this work, we present Uncertainty-aware Simulation-to-Experiment Modeling (USEM), a novel approach that enables ML models trained on labeled simulation data to be adapted for analyzing unlabeled or sparsely labeled experimental data. The method uses adversarial domain adaptation in the latent space to reduce the distance between simulation and experiment data, while incorporating predictive uncertainty through spectral-normalized neural Gaussian processes (SNGP). We demonstrate the effectiveness of USEM on both synthetic and experimental X-ray diffraction (XRD) data, showing improved predictive accuracy and the ability to identify out-of-distribution samples. USEM offers a scalable and trustworthy solution for high-throughput characterization in ML-driven materials discovery.

Introduction

The need for high-throughput materials characterization is growing as researchers seek to accelerate materials data analysis and discovery. Machine learning (ML) has enabled high-throughput materials characterization through high-throughput data analysis^1–4. However, the bottleneck of the scarcity of labeled experimental data⁵ still hinders the application of ML in real experimental settings. Although high-throughput experiments generate large volumes of data, they are usually unlabeled (e.g., microscopy images collected prior to manual inspection), and manual labeling is time-consuming and labor-intensive. As a result, traditional supervised ML approaches cannot be trained using unlabeled experimental data. To mitigate this issue, physics-based simulation (e.g., crystal-based finite element formulations⁶) data is used to train the ML models for materials data analysis^7–9. However, due to the complexity in physical processes, there are discrepancies between simulated and experimental data^10,11, which limits the performance of ML on real experiment^8,12–14. This work focuses on adapting ML models trained on simulation data with high-dimensional inputs to apply in experimental materials characterization with uncertainty quantification. We demonstrate the developed methods in high-throughput crystal structure classification.

Integrating experiment data with simulation data for prediction is a widely examined topic in various scientific domains. Often we assume experiment data and simulation data share a similar domain of physics. Prediction based on model trained on simulation data often performs poorly due to domain shift, where the distribution of test data inputs differs from the distribution of training data inputs^15,16. To mitigate harmful effects of domain shift, several methods have been developed across various areas in both supervised learning^17–19 and deep reinforcement learning. In supervised learning, one method is hybrid modeling which mixes simulation and real data in training. Tercan et al.¹⁷ applied transfer learning for manufacturing process planning in injection molding by training the networks first on simulation data and subsequently adjusting them to real experimental data. Saadallah et al.¹⁸ integrated experiment and simulation by combining real and simulation data to form an enriched data set, and assembling models learned from real and simulated data separately. The methods were applied in real-time milling process monitoring and background suppression in astroparticle physics. Another method is domain adaptation, which aligns synthetic data distribution to real data. Alkhalifah et al.¹⁹ applied a series of linear operations so that simulation and real input data have similar distributions, and demonstrated in seismic area including seismic location identification and frequency prediction. The input data is from the simulation subset, and the auto-corrections are from the experiment subset. In deep reinforcement learning, it is often impractical or impossible to train in the real world. Osiński et al.²⁰ developed an autonomous driving system trained by simulation data and a semantic segmentator using mixed real-world and synthetic data, which are used to control a real-world vehicle. Kang et al.²¹ studied autonomous aerial vehicle collision avoidance by learning the physical behavior of the vehicle mostly from real-world data, and generalized the method using simulated experience.

Materials characterization can benefit from applying simulation-trained machine learning models to the analysis of experimental data. However, there are several challenges. First (Challenge 1), because labeled data in materials characterization is scarce and costly, hybrid approaches that mix labeled simulation and experimental data are less effective due to imbalanced data. Moreover, tasks like classification or regression are not sequential decision-making processes, making reinforcement learning unsuitable. Therefore, we adopt domain adaptation in this work. Domain adaptation aligns the unlabeled target domain (i.e., experimental data) with the labeled source domain (i.e., simulation data) in the latent space of inputs, making the two input distributions indistinguishable to the network. This enables the model to generalize effectively to both source and target data. For image data, domain adaptation can be attempted by modifying simulated data to look like experimental data before applying ML, such as adding noise to the simulations and scaling the pixel values such that the two datasets (simulated and experimental) have similar distributions. This manual distribution alignment method requires prior knowledge of simulation/experiment mismatch. Generative adversarial networks, using discriminator and generator to create experiment-like simulation images, have enabled automated distribution alignment and have been applied in scanning transmission electron microscopy images^22–28. In this work, we focus on automated distribution alignment which requires minimal manual image manipulation and omits the needs of generating new images. Instead, we directly adapt simulation to experiment in the latent space. This feature-level alignment is computationally less expensive than image-to-image translation.

Second (Challenge 2), materials characterization often involves analyzing high-dimensional data, such as 2D X-ray diffraction (XRD) images and 3D tomography, for which neural networks capable of handling high-dimensional complex data representations have been adopted²⁹. However, most existing neural network models used in materials characterization lack the ability to quantify uncertainty in their predictions^30,31. Without uncertainty estimation, it is difficult to assess the trustworthiness of model outputs. As a result, models are prone to failure when applied to novel materials that differ from the training data. This lack of awareness of out-of-distribution inputs significantly limits their usefulness in materials discovery. To address this limitation, we propose using the Spectral-normalized Neural Gaussian Process (SNGP) model. SNGP equips neural networks with uncertainty quantification by integrating a Gaussian Process into the neural network, allowing it to estimate distance-aware predictive uncertainty and detect unfamiliar inputs.

Both Challenges 1 and 2 become critical in high-throughput experimental studies for materials discovery and design³², where XRD is used to characterize the crystal structure of a sample (or a region of a combinatorial sample)²⁹. This is a complex problem because the structure may include multiple distinct crystalline phases (each with their own structure) whose identities and relative amounts are, in general, unknown. Automated, quantitative phase analysis of combinatorial specimens has recently been demonstrated but requires incorporation of extensive field-specific knowledge (such as first-principles calculations of thermodynamic data)³³. Although obtaining such detailed information is clearly desirable, it may not be feasible in general (especially if the range of possible phases is large) and may not be necessary if the goals of the experimental campaign can be met by a simpler classification scheme in which specimens (or regions of combinatorial specimens) are grouped according to the similarity of their structure. Unsupervised learning to classify structures from XRD has been demonstrated in several contexts but requires a robust data set and can be subject to overconfident predictions²⁹. The problem of data scarcity can be addressed through data augmentation³⁴, and there has been progress in assessing explainability of ML models^35,36. However, significant expert knowledge is required. Here, we develop an integrated data-driven approach that simultaneously addresses both the problem of data scarcity (Challenge 1) through domain adaptation and the problem of overconfidence (Challenge 2) through uncertainty estimation.

As a summary, we propose in this work the Uncertainty-aware Simulation-to-Experiment Modeling (USEM) for ML in materials characterization. The method offers two main innovations:

(1) Simulation-to-experiment transfer: ML models trained on simulation data can be applied to prediction using inputs from experimental data with an improved accuracy. This is achieved through domain adaptation in the latent space using unlabeled experimental data. Also, the method is flexible to incorporate labeled experimental data when available. (2) Uncertainty quantification (UQ): USEM provides predictive uncertainty estimates by using SNGP, which offers a measure of model trustworthiness and helps to identify model predictions with low confidence. Additionally, UQ enables detection of out-of-distribution samples, allowing the model to recognize novel materials different from training data for accelerating materials discovery.

The rest of the paper is organized as follows: Section “Technological background” introduces related work and technological background. Section “Proposed method” describes the proposed USEM method, including domain adaptation strategy, model architecture, and uncertainty quantification approach. Section “Data” presents the XRD datasets collected from simulation and experiment. Section “Results” and “Discussion” report the results of predictive performance, uncertainty calibration, and out-of-distribution detection in crystal structure classification. Section “Conclusions” concludes the paper with a summary of key contributions.

Technological background

We describe the key concepts behind the proposed method: domain adaptation and SNGP. Domain adaptation enables models trained on simulation data to be transferred and applied to experimental data. SNGP provides predictions with quantified uncertainties.

Domain adaptation

Deep neural networks, when trained on large-scale datasets, can learn representations for specific tasks. However, due to domain shift, the trained models do not generalize well to novel datasets and tasks. Domain adaptation methods attempt to mitigate the harmful effects of domain shift. In domain adaptation, this principle is to ensure that the network cannot distinguish between the distributions of its training and test domain examples^37–39.

Adversarial Discriminative Domain Adaptation is a realization of this type of approach which seeks to minimize an approximate domain discrepancy distance through an adversarial objective with respect to a domain discriminator (Fig. 1)^40–42. This is an unsupervised adversarial adaptation method. For image-based classification, the available data are images and labels drawn from a source domain distribution and unlabeled target images drawn from a target distribution . The aim is to learn a target representation and classifier so that target images can be correctly classified as one of categories during testing. Domain adaptation learns a model (source representation mapping and a source classifier ) using labeled source data, and then adapts that model to predict unlabeled target data. To achieve that, the main goal is to regularize the learning of the source and target mappings, and by minimizing the distance between the empirical source and target mapping distributions: and . After that, the source classifier can be directly applied to the target representations with .

Fig. 1.

Adversarial discriminative domain adaptation.

As shown in Fig. 1, first a source encoder CNN is trained using labeled source image data through the following optimization

1

where is an indicator function equal to 1 when the predicted class index matches the true label , and 0 otherwise. This is a standard cross-entropy loss function. Next, adversarial adaptation is performed by learning a target encoder CNN such that a discriminator that sees encoded source and target examples cannot reliably predict their domain label. This is achieved by

2

and

3

During testing, target images are mapped by the target encoder to the shared feature space and predicted by the source classifier/regressor. Networks with dashed-line boundaries in Modules 2 and 3 in Fig. 1 indicate fixed network parameters during training or testing. In this work, we model simulation data as source data, and experiment data as target data. We apply domain adaptation to align the distributions of inputs of simulation and experiment data.

Spectral-normalized Neural Gaussian Process (SNGP)

SNGP was developed⁴³ to quantify epistemic uncertainty in deep learning through distance-awareness. As shown in Fig. 2, SNGP modifies the traditional deep neural network in two key aspects, detailed as follows. First, as illustrated in the right-hand dashed-line box in Fig. 2, SNGP replaces the commonly used dense output layer with an approximated Gaussian Process (GP) layer which models the distance from the last hidden layer to the output layer. To handle the large dimensionality and large data size, Laplace-approximation inference is applied to the random Fourier features expansion of the GP layer⁴⁴. This operation enables the model to be distance-aware among data samples. Second, as illustrated in the left-hand dashed-line box in Fig. 2, SNGP preserves the distance approximately through adopting residual neural network architecture and up-bounding the spectral norm of the weights in the nonlinear residual blocks. The reasons are the following. The testing data which is dissimilar to the training data can be possibly mapped close to the training data in the hidden representation space⁴⁵. This is the feature collapse issue in neural network. Thus, it is needed to preserve the distance from the input layer throughout the hidden layers so that the distance in the hidden layers can reflect the distance in the input layer .

Fig. 2.

Spectral-normalized Neural Gaussian Process.

The SNGP outputs predictive logits that follow Gaussian distribution . For regression, the predictive mean and variance are and , respectively. For classification, the predictive probability⁴⁶ is

4

where for the output with classes. The Gaussian process component is used to impose prior over functions and to provide distance-aware uncertainty estimation, rather than to model the data distribution. As such, the neural network learns flexible, potentially non-Gaussian mapping, while the GP layer primarily affects uncertainty calibration rather than distributional expressiveness. In this work, SNGP is used to learn a mapping from materials characterization images to structures, and to quantify the uncertainty of predictions.

Proposed method

Uncertainty-aware Simulation-to-Experiment Modeling (USEM)

Figure 3 shows the workflow of the proposed Uncertainty-aware Simulation-to-Experiment Modeling (USEM) method. First, a simulation ML model is trained using the labeled simulation data. Next, the simulation ML model is adapted using experimental data. Since labeling the experimental data is time consuming, usually there is not sufficient labeled experimental data for model training. We only use unlabeled experimental data to adapt the simulation ML model to obtain experiment ML model. Following that, when available, a limited amount of labeled experimental data is used to further fine tune the experiment ML model. The details of simulation-to-experiment adaptation (Arrow 1 in Fig. 3) and experiment ML model tuning (Arrow 2 in Fig. 3) are described below.

Fig. 3.

Proposed framework for simulation-to-experiment modeling.

Figure 4a shows the simulation-to-experiment model adaptation details and the architecture of the ML models (ML indicated by Arrow 1 in Fig. 3).

Fig. 4.

Uncertainty-aware simulation-to-experiment modeling.

In the first module “Simulation Data Training”, the data generated from the simulation process is used for simulation ML model training. The input data is images (e.g., XRD images) and the output is materials structures (e.g., crystal structures). The ML model is a CNN-SNGP model which contains two components. The first component CNN-SN incorporates spectral normalization and residual connection in the CNN model⁴³. Following that, GP classifier layer is added as the last layer. This CNN-SNGP model can not only predict the materials properties from the images but also provide the predictive confidence by quantifying model uncertainty.

In the second module, “Sim-to-Exp Adversarial Adaptation”, we add another branch containing experiment inputs together with the “experiment CNN-SN” (Exp CNN-SN) machine learning model. This newly added branch is parallel with the branch which combines simulation inputs and “simulation CNN-SN" (Sim CNN-SN). The outputs from both branches are input to a discriminator to classify the domain labels (i.e., simulation or experiment images). Here the domain labels for simulation data and experiment data are 0 and 1 respectively. The simulation CNN-SN is kept unchanged during the training process, i.e., fixing parameters during training indicated by the dash-lined outline. This module is used to train an Exp CNN-SN so that the discriminator cannot distinguish the data from experiment or simulation.

In the third module “Experiment Data Testing”, the experiment CNN-SN from the second module and the GP classifier from the first module are concatenated, which forms the experiment ML model. This model can then be used for materials property prediction from experimental images.

For some cases the labels of a small portion of experimental data can be obtained, for example, through manual labeling. The labeled experimental data can then be used to further improve the model predictive performance (ML indicated by Arrow 2 in Fig. 3). This is achieved by fine tuning the experiment CNN-SN from the previous step using the labeled experimental data (Fig. 4b). The differences between the architectures used for simulation-to-experiment adaptation (Fig. 4a) and for fine tuning (Fig. 4b) are that, in the first module, the simulation data is replaced by labeled experiment data, and the simulation CNN-SN is replaced by experiment CNN-SN. We still apply adaptation (Modules 2 and 3) after training using the labeled experimental data (Module 1), which ensures that distributions in the latent space still align between labeled and unlabeled experimental data. It is noted that the fine-tuning step is not mandatory for USEM. USEM is developed under the assumption that no labeled experimental data are available, corresponding to an unsupervised domain adaptation setting. In practice, when a limited amount of labeled experimental data is available, USEM can be readily extended to incorporate these data to further improve domain adaptation performance. In this case, the framework operates in a semi-supervised domain adaptation regime.

Machine learning model architecture

The architecture of the CNN-SNGP model (CNN-SN and GP) is illustrated in Fig. 5, and the detailed specifications are provided in Table 1. The input is the diffraction images. The output corresponds to crystallographic phase classes (FCC, BCC, and HCP), treated as discrete categorical labels (i.e., 0, 1 and 2). We use the CNN model to process image data. There are 3 convolutional layers with 32 filters and a kernel size of 3 by 3. We added residual connections between adjacent convolutional layers, i.e.,

5

Fig. 5.

Machine learning model architecture.

Table 1.

Machine learning model layers and specifications.

Models	Layers	Specifications
CNN-SN	Input Layer	Shape: (106, 103)
	Convolutional layer 1	Kernel size: (3, 3), Filter size: 32, Activation: ReLU, Strides: 1, Padding: Same, Spectral normalization, Spectral norm bound: 0.95
	Pooling layer 1	Max pooling, Filter size: (2, 2), Strides:1, Padding: Same
	Residual layer 1	Input Layer + Pooling layer 1
	Convolutional layer 2	Same specifications as Convolutional layer 1
	Pooling layer 2	Same specifications as Pooling layer 1
	Residual layer 2	Residual layer 1 + Pooling 2
	Convolutional layer 3	Same specifications as Convolutional layer 1
	Pooling layer 3	Same specifications as Pooling layer 1
	Residual layer 3	Residual layer 2 + Pooling layer 3
	Flatten layer	Shape: (1, 106 × 103)
GP	Output layer	Random Feature Gaussian Process, Size: 1, Activation: ReLU
Discriminator	Dense layer	Size: 10, Activation: ReLU
	Dense layer	Size: 10, Activation: ReLU
	Output layer	Size: 1, Activation: Sigmoid

where and are the input and the output of the l-th layer, respectively. Strides of size 1 and “same” padding are added to keep the dimension of the filters unchanged after each convolutional operation to enable the summation in Eq. (5). The spectral normalization with spectral norm bound equal to 0.95 is applied in each convolutional layer. Spectral normalization is applied to each convolutional layer with a bound of 0.95 to ensure Lipschitz continuity and preserve distance awareness in the latent space. The output layer is a Random Feature Gaussian Process⁴³. The SNGP layer employs an RBF kernel approximated using 1024 random Fourier features. The length-scale parameter for the RBF kernel is 2. The hyperparameters follow prior SNGP literature⁴³.The Discriminator is a fully connected neural network. The inputs are the output from CNN-SN model. The outputs are probability of the images being from the same source. There are 2 hidden layers with 10 neurons and ReLU activation function. The models are trained using Adam optimizer⁴⁷ with categorical cross entropy loss function, learning rate of 1e-5, batch size of 32, and 500 epochs. The CNN architecture is lightweight and consistent across experiments to isolate the effect of domain adaptation and uncertainty modeling. The architecture design follows common XRD classification pipelines^29,48. Extending the framework to predict additional crystal systems (three of the seven crystal systems are considered in the current work) can be achieved by augmenting the output dimensions, but would require corresponding labeled training data³⁰. In contrast, extending the approach to predict all space groups and a wide range of structural variants is a highly challenging task due to the substantially larger output space (e.g., 230 space groups), which would require significant advances in machine learning models and data availability^48,49. This is beyond the scope of the present focus, i.e., uncertainty-aware simulation-to-experiment adaptation.

Data

X-ray diffraction (XRD) is a powerful technique used to study the crystal structure of materials⁴⁹. We aim to build an ML model to predict the crystal structure of metals and alloys from XRD images. Besides providing an example of our ML framework, this particular problem is of practical importance in the context of combinatorial materials discovery, in which a wide variety of material compositions are prepared and characterized, looking for either novel structures or useful combinations of properties³².

Simulation data

We generated 306 XRD data sets for the three most common crystal structures of metals: face-centered cubic (FCC), body-centered cubic (BCC), and hexagonal close-packed (HCP). Each data set includes a simulated XRD image and the corresponding crystal structure (102 for each of the three structures, in total 306 data points). Representative data are shown in Fig. 6a.

Fig. 6.

Materials characterization data from simulation and experiment. (a) Example of the simulated XRD images. (b) Experimental data from copper. The points in the left figure show the locations for collecting XRD images and the figure on the right shows an example of XRD image. (c) The experimental XRD images and their corresponding crystal structures.

The primary training data is generated by simulating diffraction patterns of polycrystalline samples, which allows us to control parameters that influence the diffraction pattern, including the lattice parameter, grain size, orientations of the grains in the scattering volume (“texture”)⁴⁹. The goal is to simulate a realistic powder pattern, including random fluctuations in intensity due to sampling statistics. The approach used for this is to simulate each individual grain in an aggregate of crystals. Here we use the term “grain” to refer to a crystal with a single lattice orientation, center of mass position, and volume.

Simulations are done using a MATLAB code package developed in Refs.^50,51 based on the methods described in Refs.^6,52. In short, given a crystal structure and the various parameters mentioned above, this method determines where on the x-ray detector diffraction spots will appear. For a given grain, the method determines the reciprocal lattice vectors for which the diffraction conditions are satisfied) for a given instrument setup (accounting for the x-ray beam divergence) and initial lattice orientation. This is repeated for all grains in the scattering volume, using vectorization and parallel processing to make the calculation reasonably fast for large numbers of lattice orientations. The intensity of each peak was then corrected by the Lorentz Polarization factor⁵³, the fraction of the diffraction volume accounted for by the grain, and a scaling factor to produce an intensity scale comparable to that observed in the experiments. The diffracted beams were then projected onto the detector^6,52, and after all intensities had been added to the pixels they directly landed on, the detector point spread function is applied.

The three most common structures for structural metals (FCC, BCC, and HCP) are simulated. To train the model, a range of lattice parameters are simulated for each structure spanning a range expected for metals. Specifically, we calculate lattice parameters for each structure based on an assumption close-packing of spherical atoms using the atomic diameter of niobium as the center of the range and then examine values within ± 25% of that central value, with a step size of 1% between simulations. Crystal orientations are generated approximately uniform sampling over orientation space, and positions are randomly generated uniformly over the entire diffraction volume, which is 0.22 × 0.22 × 0.2 mm (corresponding to a 200 μm thick foil with a 220 μm beam size, as discussed below). Two different populations of grains are generated for each structure and lattice parameter, one with 10,000 grains and one with 50,000. Changing the number of grains in the same sampling volume is equivalent to changing the averaging grain size, which corresponds to 12.27 μm and 7.18 μm (equivalent sphere diameter) for the two cases.

Experimental data

Transmission X-ray diffraction was performed using a custom X-ray diffractometer at an X-ray power of 700 W, beam energy of 24.2 keV and a spot size of 200 µm⁵⁴. Each diffraction pattern represents 30 s exposures and was collected using a CdTe Eiger 2R 1 M detector from Dectris at a detector distance of 50 mm. NIST standard alpha-alumina was used for calibration, and custom Python scripts written in Visual Studios, PyFAI⁵⁵, and Fit2D⁵⁶ were used as tools to analyze the diffraction data.

We collected 1280 experimental data from samples of nominally pure (ETP) copper purchased from McMaster-Carr (Fig. 6b). A 6" × 6" × 0.1870" sheet was cut into smaller (1 cm²) pieces using electro-discharge machining (EDM). The as-received specimens were annealed at 500 °C for 1 h in air induce some recovery (which sharpens the diffraction peaks). After annealing, the pieces were thinned and polished to a mirror finish using an Allied Tech Multiprep System (12″) Precision Polisher. In addition, to test the classifier we also collected fifteen XRD patterns from other materials (with the same instrument and same conditions) with a variety of different crystal structures (Table 2, and Fig. 6c).

Table 2.

Summary of additional experimental diffraction data collected.

Material	Number of images	Crystal structure
Al2O3	2	Corundum (Trigonal)
Al-7085	2	FCC
Cu	4	FCC
NbTi	4	BCC
SiC	1	Zincblende (ZnS)
Steel	2	BCC

In this work, energy-dependent detector response is expected to be negligible due to the use of a monochromatic X-ray beam. Charge-sharing effects may influence angular resolution, but their impact is presumed to be small relative to the intrinsic angular resolution of the MAXIMA instrument, which is intentionally limited as a design trade-off for high-throughput measurements. Beam divergence (parallax) effects, which can affect peak shape and intensity, are not explicitly modeled but are implicitly handled through the domain adaptation framework by aligning simulated and experimental data distributions.

Results

Prediction accuracy

We first use simulation data to evaluate the model prediction accuracy. ML models are built to predict the crystal structures from XRD images, using 80% of the data for model training and the remaining 20% for testing predictive accuracy. The inputs are normalized by dividing all values by the maximum intensity across the dataset, and the labels are one-hot encoded before training/testing split. Two ML models are built, a traditional CNN model and a CNN-SNGP model. Both models can produce 100% accuracy on testing data (Fig. 7 (a) and (b)). We conducted fivefold cross validation to test the model robustness. Both CNN and CNN-SNGP models can achieve 98% to100% accuracy across validations. This shows ML is capable of capturing features useful for crystal structure identification from the XRD images.

Fig. 7.

In-distribution classification using simulation data. (a, b) CNN and CNN-SNGP both achieve 100% accuracy on testing data. (c, d) Feature visualization using T-SNE. Circled points are testing data; non-circled points are training data. CNN-SNGP shows clear cluster separation, preserving input distances critical for uncertainty quantification.

Uncertainty quantification

We further investigated the extracted features by the two ML models (Fig. 7c,d). For visualization, the high dimensional features are shown in the two-dimensional space by reducing the dimensionality using T-distributed stochastic neighbor embedding⁵⁷ (t-SNE, a way of visualizing a high-dimensional data by giving each datapoint a location in a two or three-dimensional map). The CNN-SNGP extracted features can be fully separated in Fig. 7d. Each cluster represents a distinct crystal structure (FCC, BCC, or HCP). However, there is not a complete separation for the CNN extracted features in Fig. 7c. The improved cluster separation from Fig. 7c,d means that the CNN-SNGP can preserve the distances of input images when data is passed layer by layer. Complete cluster separation is not achieved by the traditional CNN, with the latent features of BCC and FCC close to each other. Without the color key, one cannot distinguish these two classes based solely on spatial locations. The distance preserving property in CNN-SNGP is critical for uncertainty quantification. Using this property, we can identify data points outside the distribution of training data such as crystal structure that is neither BCC, FCC nor HCP.

To investigate the question of out-of-distribution data, in Fig. 8 we train the model using only BCC and FCC data (80%). Again, the prediction accuracy is 100% for both CNN and CNN-SNGP models. Next, we test the model using the HCP data. Since the models were trained using only two classes, the model can never predict out-of-distribution class (HCP) and must (incorrectly) predict either BCC or FCC. Our results show that the CNN model has high confidence in its incorrect predictions (Fig. 8c) as the training, in-distribution testing and out-of-distribution testing data all have low uncertainty. On the other hand, the CNN-SNGP can identify out-of-distribution data by assigning high uncertainty for the predictions of the (unknown) HCP data (Fig. 8d). We can clearly separate the uncertainty distribution of in- and out-of-distribution data. This can inform the users that the model is unsure about its prediction since it has never seen such data. This may indicate that further human assessment is needed or the identification of a novel material. The uncertainty quantification capability of CNN-SNGP stems from its distance-preserving feature representations and the distance-aware uncertainty estimation provided by the GP layer. Similar to the discussion of Fig. 7c,d, we employ T-SNE to present the data distribution in the latent space using the extracted features from the last hidden layer. CNN-SNGP shows clear cluster separation between different classes of in-distribution data (i.e., BCC and FCC) in Fig. 8e, and between in- and out-of-distribution data (i.e., BCC / FCC vs HCP) in Fig. 8f. In contrast, a standard CNN without distance-preserving capability tends to produce mixed or overlapping latent feature distributions.

Fig. 8.

Out-of-distribution classification. (a, b) Models trained on BCC and FCC, tested on unseen HCP data. (c) CNN misclassifies the unknown HCP data with low uncertainty. (d) CNN-SNGP assigns high uncertainty, effectively detecting out-of-distribution data. (Note the difference in uncertainty scales between (e) and (f).) (e, f) Feature visualization using T-SNE. Points with black circles are testing data; non-circled points are training data. Points with red circles are out-of-distribution data. CNN-SNGP shows clear separation of different in-distribution classes, and between in- and out-of-distribution classes, preserving input distances critical for uncertainty quantification.

We also investigate uncertainty quantification and distance preservation using another in- and out-of-distribution data split. 80% FCC and HCP data is used as in-distribution data to train the standard CNN and CNN-SNGP models. Then the models are tested using the remaining 20% in-distribution data and all BCC data (out-of-distribution data). The results are presented in Fig. 9. Similar results to the previous discussion in Fig. 8 can be obtained. Although CNN and CNN-SNGP achieve the same accuracy for in-distribution data shown in the confusion matrices in Fig. 9a,b, CNN fails to differentiate out-of-distribution data from in-distribution data indicated by the low uncertainty for all testing data in Fig. 9c due to the lack of distance-preserving capability shown in Fig. 9e. In contrast, CNN-SNGP model clearly separate data of different classes in Fig. 9f and assign high uncertainty to out-of-distribution data (Fig. 9d).

Fig. 9.

Out-of-distribution classification. (a, b) Models trained on FCC and HCP, tested on unseen BCC data. (c) CNN misclassifies the unknown HCP data with low uncertainty. (d) CNN-SNGP assigns high uncertainty, effectively detecting out-of-distribution data. (e, f) Feature visualization using T-SNE. Compared to CNN, CNN-SNGP shows clear separation of different in-distribution classes, and between in- and out-of-distribution classes.

We further study distribution of simulation and experimental data in the latent space using the trained models CNN and CNN-SNGP in Section “Prediction accuracy”. The simulation and experimental XRD images (Fig. 6a, c, respectively) are used as inputs. In this case, experimental data is considered as out-of-distribution data. Figure 10 shows that, without color coding, CNN model cannot distinguish three in-distribution and out-of-distribution classes. The distance awareness is improved in CNN-SNGP model.

Fig. 10.

Simulation vs. experimental data in the latent space. Points represent both simulation and experimental samples. Non-circled points denote training simulation data. Testing simulation data are outlined with black circles, and experimental data are outlined with red circles. Without uncertainty-aware distance preservation, the standard CNN model does not clearly separate individual classes in the latent space. In contrast, the CNN-SNGP model exhibits improved distance awareness, leading to better separation between simulation and experimental classes.

Simulation-to-experiment adaptation

Validation on synthetic data

The proposed USEM method is first validated using synthetic data by modifying the simulated XRD images. We create modified simulation datasets to include noises which may exist in experimental data. Two sets of synthetic data are created by (1) blurring the simulation XRD images and (2) adding background noise, respectively. Blurred images are produced by applying a spatial averaging (box) filter with a kernel using OpenCV, which smooths local intensity variations to mimic reduced resolution. Images with background noise are generated by adding zero-mean Gaussian noise with unit variance to the original simulated patterns, i.e., , applied element-wise. Visualization of original and modified simulation XRD images is shown in Fig. 11. Although visually insignificant to human eyes, the modifications can provide adversarial attached and can provoke drastic changes in prediction⁵⁸. These synthetic data has a different domain compared to the original simulation data due to the added noise. Classification results for these data sets are shown in Figs. 12 and 13 and Table 3.

Fig. 11.

Data visualization for original simulation XRD images and modified images by image blurring and adding background noise. (The axes are in pixel coordinates).

Fig. 12.

Validation on synthetic data created by blurring the simulated images.

Fig. 13.

Validation on synthetic data created by adding background noise to the simulated images.

Table 3.

Predictive accuracy and uncertainty quantification.

	Synthetic data 1: Image blurring		Synthetic data 2: Background noise
	Accuracy (↑)	ECE (↓)	Accuracy (↑)	ECE (↓)
Source-only SNGP	0.45 ± 0.05	0.159 ± 0.011	0.61 ± 0.02	0.138 ± 0.012
Unsupervised DA-SNGP	0.64 ± 0.01	0.100 ± 0.009	0.86 ± 0.01	0.037 ± 0.005
Semi-supervised DA-SNGP	0.93 ± 0.01	0.034 ± 0.004	0.96 ± 0.01	0.025 ± 0.005
Semi-supervised Source-only SNGP	0.64 ± 0.01	0.116 ± 0.008	0.33 ± 0.02	0.166 ± 0.006

For the first synthetic dataset (i.e., the blurred simulated XRD images), first we use the CNN-SNGP model trained on simulation data (Source-only SNGP in Section “Machine learning model architecture”) to classify the data. The result is shown in Fig. 12a. The accuracy is 0.45. Next, the Source-only SNGP is adapted using unlabeled synthetic data. The predictive accuracy on the synthetic data is improved to 0.64. Next, we use 5% of labeled synthetic data to fine tune the DA-SNGP model. The predictive accuracy on the remaining 95% unlabeled synthetic data is significantly improved to 0.93. In order to show that the high accuracy is not just from fine turning, we use the same 5% labeled synthetic data to fine tune the CNN-SNGP model trained on simulation data without unsupervised domain adaptation (Semi-supervised Source-only SNGP). Results show that accuracy can only reach 0.64 compared to 0.93 (using the proposed domain adaptation approach).

Similar results can be obtained in the second set of synthetic data (i.e., the simulated XRD images with background noise, Fig. 13). The predictive accuracy on the synthetic data is improved from 0.61 (Simulation-only SNGP) to 0.86 (Unsupervised DA-SNGP). With 5% labeled experimental data, an accuracy of 0.96 is achieved (Semi-supervised DA-SNGP). We also test fine tune Simulation-only SNGP using the same labeled synthetic data, with the accuracy dropping from 0.61 (Simulation-only SNGP) to 0.33 (Semi-supervised Source-only SNGP). This emphasizes the fact that the domain alignment is critical for simulation-to-experiment adaptation. Simply fine tuning the simulation ML model by continuing training using a small amount of experimental data may harm the predictive performance on experimental data.

The significant performance degradation observed for the source-only SNGP model when applied to modified synthetic data highlights the sensitivity of simulation-trained models to domain shifts introduced by image blurring and background noise. While training directly on more realistic simulations could reduce this gap, the purpose of this study is to explicitly examine the impact of domain mismatch and to evaluate the effectiveness of domain adaptation in bridging this gap. Our results do not imply that highly realistic simulations are unnecessary; rather, they demonstrate that even modest discrepancies between simulation and measurement can substantially affect model performance^58,59, and that domain adaptation provides a flexible mechanism to address such discrepancies. We note that the balance between simulation fidelity and domain adaptation is application-dependent, and is worthy of investigation across broader areas.

The results presented in this section demonstrate that the proposed uncertainty-aware model achieves in-distribution predictive accuracy comparable to a standard convolutional neural network (interpolation), while significantly outperforming the regular CNN in out-of-distribution identification (extrapolation). Moreover, the model enables uncertainty quantification through a single training procedure, making it computationally more efficient than sampling or ensemble-based approaches.

When presented with multi-phase diffraction patterns, the model produces high predictive uncertainty with approximately equal probabilities across FCC, BCC, and HCP classes, indicating that the input lies outside the training distribution and beyond the model’s intended capability. Extending the framework to explicitly handle multi-phase mixtures would require multi-label or mixture-aware training data and is an important direction for future work.

An alternative representation of XRD data is radially integrated one-dimensional spectra, which significantly reduce input dimensionality and may lead to faster training and simpler models for phase-level classification tasks. While such representations may be sufficient when only crystal symmetry is of interest, they discard spatial information present in full 2D diffraction images, including ring continuity, texture, and anisotropy. In this work, we focus on 2D images to preserve this information and to support uncertainty-aware learning and out-of-distribution detection. A systematic comparison between models trained on 1D and 2D representations, including differences in training efficiency, extrapolation behavior, and uncertainty calibration, is beyond the current scope and left for future investigation.

Demonstration on experimental data

After validating the USEM using synthetic data, we applied the method to experimental data. First, the CNN-SNGP model trained on simulation data in Fig. 7 in Section “Machine learning model architecture” is applied to classify the 1280 copper data sets. In Fig. 14, the first three rows show 5 representative results from 1280 predictions. Using source-only SNGP, the accuracy is 0.32 (the first row of Fig. 14). This low accuracy is caused by the discrepancy between the source (simulation) and target (experiment) domains. With three classes, the accuracy is essentially equivalent to random guessing. Although the source-only SNGP model exhibits poor predictive performance, it differs from standard neural networks by producing a high level of uncertainty, as its predicted class probabilities are nearly identical, indicating appropriate uncertainty awareness. After unsupervised domain adaptation, the DA-SNGP can achieve accuracy of 1.0 (the second row of Fig. 14). This demonstrates the effectiveness of domain adaptation in improving the predictive performance of a model initially trained on simulation data when applied to experimental data.

Fig. 14.

Demonstration on real experimental data. The class with the orange color is the true label. The first three rows show 5 representative results from 1280 predictions. The last row shows the prediction of 3 out-of-distribution experimental data.

Next, we use 12 labeled in-distribution data (i.e., BCC and FCC) from the experimental data in Fig. 6c to fine tune the above DA-SNGP model. After that, the updated model is used to predict the copper data. The results in the third row in Fig. 14. With the semi-supervised DA-SNGP approach, the predictive performance can be further improved, indicated by the increases of the probability of true labels. Also, the probability of the three classes becomes more unevenly distributed, which shows the decreased model uncertainty for in-distribution data prediction when more labeled data are included for training. We also test the semi-supervised DA-SNGP on out-of-distribution data (i.e., Hexagonal in Fig. 6c). Results show that the model classifies the out-of-distribution data as BCC, FCC and HCP with almost identical probability. This means the model is uncertain about the predictions, as the model has never been trained by hexagonal crystal structure. An application of the developed uncertainty-aware ML model is data-driven materials discovery. Given input XRD images, the high model predictive uncertainty indicates new crystal structures different from the observed data.

It is important to note that the experimental out-of-distribution demonstration involves only three hexagonal samples, which is insufficient for statistical generalization. The experimental dataset is therefore intended as a qualitative proof-of-concept illustrating the behavior of USEM under real measurement conditions, rather than a comprehensive validation of out-of-distribution detection performance. Systematic evaluation of uncertainty calibration and OOD identification is conducted on synthetic datasets, where in-distribution and out-of-distribution conditions can be rigorously controlled. The experimental Cu dataset (1280 patterns) represents a single FCC phase and therefore primarily evaluates simulation-to-experiment transfer under a known in-distribution condition, rather than full multi-class experimental classification. While the model achieves 100% accuracy on this dataset, this result should be interpreted as evidence of successful feature transfer for FCC identification rather than comprehensive validation across multiple crystal classes. The additional experimental samples from other materials (Table 2), including BCC and FCC structures, are used for semi-supervised fine-tuning to demonstrate the flexibility of the USEM framework under limited labeled experimental data. A broader experimental benchmark spanning multiple independently verified crystal classes would further strengthen conclusions regarding general multi-class transfer performance.

Figure 14 illustrates how predictive uncertainty is conveyed through the class probability distribution. Confident predictions are characterized by a clearly dominant class probability, whereas uncertain predictions exhibit more evenly distributed probabilities across classes. In practice, this allows users to interpret uncertainty in an intuitive manner: samples with low dispersion can be accepted as reliable phase classifications, while samples with high dispersion indicate potential domain mismatch, multi-phase content, or structural features not represented in the training data. This probabilistic interpretation supports uncertainty-aware decision-making rather than replacing expert judgment.

In real metallic systems, defects such as stacking faults, twins, and microstrain are commonly present and may introduce HCP-like diffraction signatures within an FCC lattice. These effects are not explicitly incorporated in the idealized simulated training data used in this study, representing a source of mismatch between simulation and experiment. In the context of powder X-ray diffraction, however, defect- and strain-induced effects are generally subtle, primarily manifesting as small peak shifts which are often below the angular resolution of the instrument, and as changes in peak intensity and peak shape, rather than the appearance or disappearance of entire diffraction rings⁶⁰. The classification in this work is largely driven by the presence or absence of characteristic diffraction rings distinguishing FCC, BCC, and HCP phases, and is thus expected to be relatively robust to moderate levels of defects. Nevertheless, for materials with higher defect densities or significant strain, phase separability may be reduced. Incorporating defect- and strain-related diffraction effects into the simulation pipeline is an important direction for future work to further improve model robustness and generalizability.

Discussion

Distance preserving in USEM for uncertainty quantification

We investigate the distribution in the latent space of the simulation data and synthetic data. Figure 15 presents T-SNE visualizations of the latent feature representations learned by different models corresponding to Fig. 12. Colors indicate crystal classes (BCC, FCC, HCP). Points outlined in black correspond to synthetic data generated by blurring simulated images to mimic experimental conditions, while non-circled points denote original simulation data. In Fig. 15a Source-only SNGP, the blurred synthetic data exhibit noticeable shifts relative to the original simulation data, indicating domain misalignment. The low predictive accuracy in Fig. 12a results from the close proximity between synthetic BCC and HCP data distribution. In contrast, Fig. 15b DA-SNGP shows improved alignment between simulation and blurred synthetic data, demonstrating that domain adaptation effectively reduces domain discrepancy in the latent space. When limited labeled target data are incorporated, Fig. 15c Semi-supervised DA-SNGP still preserves cross-domain alignment, with tighter clustering and improved class separation across domains. This suggests that semi-supervised domain adaptation enhances both discriminability and domain invariance. Conversely, Fig. 15d Semi-supervised Source-only SNGP exhibits domain-dependent separation owing to the SNGP distance preserving property but cannot align simulation and synthetic data due to the lack of domain adaptation, indicating that fine-tuning alone is insufficient to fully address domain shift.

Fig. 15.

Feature visualization using T-SNE. Points with black circles are synthetic data by blurring the simulation images; non-circled points are simulation data. Points outlined in black correspond to synthetic data generated by blurring simulated images to mimic experimental conditions, and non-circled points denote original simulation data.

USEM for XRD data

The effectiveness of the proposed framework arises from the complementary roles of domain adaptation and uncertainty-aware modeling. Adversarial domain adaptation reduces systematic discrepancies between simulated and experimental XRD data, enabling feature alignment across domains. However, domain adaptation alone can obscure the model’s ability to recognize inputs that lie outside the supported data manifold. By integrating SNGP, the distance among data is retained in the adapted latent space. The distance-aware feature is critical for in- and out-of-distribution testing data distinction.

XRD data are particularly amenable to domain adaptation because class-defining features, such as diffraction ring presence, spacing, and symmetry, are governed by crystallographic structure and remain largely invariant between simulation and experiment⁴⁸. In contrast, domain discrepancies primarily arise from background intensity, detector noise, and resolution effects⁶¹. This separation between invariant structural features and domain-specific nuisance factors makes representation-level domain alignment effective for XRD-based phase classification.

Limitations

The proposed framework may be less effective in scenarios involving severe texture effects, strong peak overlap, multi-phase samples, or detector characteristics not represented in the training data. In such cases, domain adaptation may not fully align the feature distributions, and uncertainty estimates are expected to increase accordingly. These limitations delineate the intended scope of the method and motivate future extensions to explicitly model multi-phase behavior and more complex experimental conditions.

The objective of this work is not to claim universal superiority over existing methods, but to demonstrate a unified and practically deployable framework for simulation-to-experiment transfer in XRD-based materials characterization. This framework is appliable to other neural-network-based domain adaptation methods, including Domain-Adversarial Neural Network (DANN)³⁷, deep Correlation Alignment (CORAL)⁶², and Maximum Mean Discrepancy methods⁶³ to explicitly improve uncertainty estimation since SNGP can be applied to various neural network architectures without requiring fundamental changes to the underlying model structure⁴³. On the other hand, commonly used uncertainty quantification techniques such as Monte Carlo Dropout and Deep Ensembles require multiple stochastic forward passes or training of multiple models, resulting in increased computational cost⁴³. In contrast, the proposed framework integrates adversarial domain adaptation with distance-aware uncertainty estimation via SNGP, enabling both feature alignment and reliable identification of samples that remain outside the supported domain.

Conclusions

We have introduced USEM (Uncertainty-aware Simulation-to-Experiment Modeling), a machine learning framework designed to bridge the simulation-to-experiment gap in materials characterization. By combining domain adaptation and uncertainty quantification, USEM allows models trained on labeled simulation data to generalize to unlabeled or sparsely labeled experimental data. Our results show that USEM improves predictive performance on both synthetic and real experimental datasets, and enables the detection of out-of-distribution samples, which is essential for novel material discovery. This method offers a scalable and reliable approach to leverage simulation data for real-world experimental analysis, paving the way for broader application of ML in high-throughput materials research.

Several major conclusions can be obtained from the current study:

1. We propose a domain adaptation approach that allows ML models trained solely on simulation data to be applied to unlabeled experimental data, removing the need for extensive manual labeling.

2. USEM incorporates SNGP to provide calibrated predictive uncertainty, enhancing model interpretability and trustworthiness.

3. The model can identify experimental inputs that lie outside the training distribution, making it valuable for novel materials discovery.

4. We demonstrate USEM’s effectiveness on both synthetic data and real-world XRD datasets, showing improved performance in predicting the phase type over baseline models.

Looking forward, several avenues for future research are worth exploring. In terms of the method, more sophisticated domain adaptation strategies could be incorporated to handle larger distributional gaps between simulation and experiment. In terms of applications in materials characterization, extending USEM to multi-modal data, such as combining microscopy, spectroscopy, and tomography, could provide a more comprehensive understanding of complex material systems.

Overall, USEM provides a principled and generalizable approach to bridging simulation and experiment in science and engineering. Beyond materials characterization, the principles underlying USEM are broadly applicable to other domains where physics-based simulations are abundant but experimental data are scarce.

Author contributions

J.C., T.L., M.W., T.H., and W.C. designed research; J.C., T.L., and M.W. performed research; J.C. contributed analytic tools; M.W., and T.H. provided the simulation and experiment data; J.C., T.L., M.W., T.H., and W.C. analyzed data; and J.C., T.L., M.W., T.H., and W.C. wrote the paper.

Data availability

The dataset and the code for USEM are available at https://github.com/jcj7292/USEM.

Declarations

Competing interests

The authors declare no competing interests.