Examining graph neural networks for crystal structures: Limitations and opportunities for capturing periodicity

Abstract

Graph neural networks (GNNs) have recently been used to learn the representations of crystal structures through an end-to-end data-driven approach. However, a systematic top-down approach to evaluate and understand the limitations of GNNs in accurately capturing crystal structures has yet to be established. In this study, we introduce an approach using human-designed descriptors as a compendium of human knowledge to investigate the extent to which GNNs can comprehend crystal structures. Our findings reveal that current state-of-the-art GNNs fall short in accurately capturing the periodicity of crystal structures. We analyze this failure by exploring three aspects: local expressive power, long-range information processing, and readout function. To address these identified limitations, we propose a straightforward and general solution: the hybridization of descriptors with GNNs, which directly supplements the missing information to GNNs. The hybridization enhances the predictive accuracy of GNNs for specific material properties, most notably phonon internal energy and heat capacity, which heavily rely on the periodicity of materials.

Graph networks fail in capturing periodicity of crystals, and supplementing periodicity to GNNs improves prediction accuracy.

INTRODUCTION

Recently, machine learning (ML) has been widely used to predict properties of materials (1–7). Conversion of crystal structures into machine-readable numerical representations is one of the most critical steps for applications of ML in materials science (8). In general, there are two approaches to convert crystal structures into numbers: human-designed description and deep representation learning.

Human-designed descriptors are based on people’s understanding of compositions and structures of materials. Therefore, one can easily understand the meaning of the descriptors (9). Mean electronegativity and difference of atomic radius of elements in materials are examples of compositional descriptors, and mean bond length and difference of coordination number of atoms in crystal structures are examples of structural descriptors of materials. Beyond simple descriptors, researchers have recently proposed a series of more complex descriptors for materials, such as Magpie compositional descriptors (10), classical force-field inspired descriptors (11), Coulomb matrix (12), fragment descriptors (13), and the more recent Pointwise Distance Distributions, a set of unambiguous and invariant descriptors for periodic crystals (14). Although ML models based on human-designed descriptors have achieved some success in revealing the trend between human-understandable characteristics of materials and properties, by definition, these descriptors contain only known information. Consequently, using only these descriptors to learn and predict material properties could miss key structure-property relationships that are currently unknown to human.

Deep representation learning refers to ML models that learn the numerical representation of materials automatically during the training of the ML models. Although the learned representations are generally less understandable compared with human-designed descriptors, deep representation learning can uncover unknown patterns of structure-property relationships. Because materials can be intuitively represented as graphs, with atoms forming the nodes and bonds forming the edges, graph neural networks (GNNs) have become the state-of-the-art deep representation learning method for materials science. SchNet (15) and Crystal Graph Convolutional Neural Networks (CGCNN) (16) are two classic GNN architectures designed for materials. They update the representations of each atom by the types of neighboring atoms and the bond length between atoms and pool all the updated atom representations into an overall representation of the structure. Despite their broad usage and success in predicting some material properties (15, 16), there are several limitations associated with these distance-based classic message-passing GNNs: (i) Only encoding distance ignores many-body information and directional information important to properties; (ii) ambiguity of chemical bonding and the criterion of nearest neighbor to establish edges in the graphs might downplay the important interactions; (iii) limited range of receptive field and the problems of oversmoothing and oversquashing (17, 18) disable the model to capture long-range or global dependence of properties on structures.

To overcome the aforementioned problems, since the proposal of SchNet and CGCNN in 2018, we have witnessed a boost of development of GNNs for atomic structures. There are three main trends in the development: (i) incorporating more geometric details beyond bond length for higher local expressive power, such as explicitly incorporating angle information in GNNs such as DimeNet (19), GemNet (20), Atomistic Line Graph Neural Network (ALIGNN) (21), M3GNet (22), and arbitrary explicit body-order information in ALIGNN-d (23) and BOTNet (24), encoding relative vector instead of geometric features to encode directional information in representations equivariant to E(3) group in E3NN (25), NequIP (26), and Equiformer (27, 28), and simultaneously encoding canonical positional and structural information in Learnable Structural and Positional Representations (LSPE) (29) and Structure-aware Transformer (SAT) (30); (ii) introducing attention mechanism into GNNs in the graph convolution operation, which physically serves the role of defining the chemical bonding strength between atoms. Example models adapting attention mechanism include SAT (30), GATGNN (31), Matformer (32), and Equiformer (27); (iii) using more advanced readout function beyond average pooling such as global attention (31, 33, 34) to exchange information globally when collecting information from atoms. Example models adapting advanced readout functions include MEGNet (33), M3GNet (22), GATGNN (31), and GraphTrans (34). In addition to the development of GNNs for general atomistic systems, there are also efforts specifically targeted on crystal structures, such as space group equivariance and supercell identifiers in the study by Kaba and Ravanbakhsh. (28), explicitly encoding periodicity in GeoCGNN (35) and Matformer (32), incorporating information of structure motifs in AMDNet (36), and propagating information in the reciprocal lattice in Reciprocal Space Neural Network (RSNN) (37). The recent evolution of GNNs primarily exemplifies a bottom-up approach as illustrated in Fig. 1A: Researchers initiate modifications of the model anchored on a physical or mathematical concept, such as many-body information, equivariance, or attention. The revised GNNs are then subjected to testing to ascertain whether such enhancements contribute to more accurate property prediction. Conversely, a top-down approach as in Fig. 1B proves generally more challenging, because determining which physical or mathematical concept accounts for shortcomings in property prediction is not generally intuitive.

Fig. 1. Overview of approaches of development of GNNs.

(A and B) Illustration of the bottom-up and top-down approach of development of GNNs for prediction of materials properties, respectively. (C) Illustration of the top-down approach of examining GNNs and the followed bottom-up approach to improve property prediction in this work. (D) Schematic of analyzing whether a GNN can capture knowledge of crystal structures behind human-designed descriptors and whether hybridization of GNN and human-designed descriptors can improve prediction performance for material properties.

Although GNNs have achieved successes in learning materials properties, in general, prediction of materials properties is still challenging (21). For example, ALIGNN, one of the most powerful ML models for materials reported in the year of 2021 (38), still cannot achieve satisfactory regression results for about 80% materials properties (10, 21), and only limited (~10%) improvement in regression errors over ALIGNN was reported in the year of 2022 (32). Beyond the aforementioned theoretical limits, another challenge associated with using GNNs to predict materials properties is that, for many materials properties, the number of data points is very limited, often less than 10,000, which results in the phenomenon that for some properties, GNNs might have worse prediction performance compared with classic ML models based on human-designed descriptors (39–42). The superiority of human-designed descriptors for predicting some materials properties indicates that, on the one hand, more complex ML models might not always be better than simpler ML models, and on the other hand, there might still be some information in human-designed descriptors that are hard to be captured by GNNs.

In this work, as in Fig. 1C, we propose a top-down approach to analyze and quantify the limitations of GNNs for crystal structures and propose a straightforward yet general bottom-up approach to improve the GNN models for predicting materials properties. As illustrated in Fig. 1D, we use human-designed descriptors as a compendium of knowledge to test whether the current GNN models can capture certain knowledge of crystal structures. We test the GNNs by using them to learn and predict the human-designed descriptors, and we use the prediction accuracy as a quantitative metric for evaluation. The underlying assumption is that, if the model can accurately predict the descriptor, then the model can capture the knowledge behind the descriptor; otherwise, the model may not be able to capture certain pieces of information about crystal structures. We find that the GNNs do not capture the periodicity of crystal structures well, and we analyze the reasons for this failure in some detail. We further hybridize the deep learning models with the human-designed descriptors and test the descriptor-hybridized models on a range of important materials properties. Compared with original GNNs, we find that hybridization of GNNs and descriptors can result in notable decrease of errors for predictions of phonon-related properties, lattice thermal conductivity, and magnetization, which are highly dependent on periodicity. For material properties with limited amount of data, we find that hybridizing descriptors with simpler GNNs might have better prediction accuracy compared with more complex GNNs with or without hybridization. The main contributions of this work are summarized as below:

1) This work proposes a systematic top-down approach of development of GNNs for materials. Previously, it is hard to determine which aspects of GNNs should be responsible for unsatisfactory predictions of properties, while in this work, it is straightforward to observe the limitations of GNNs from poor predictions of descriptors.

2) This work shows that classic GNNs fail in capturing periodicity of crystals and shows the effects of many-body information, equivariance, embedding update, pooling function, and transformer-style graph convolution on the learnability of periodicity. Although the encoding of periodicity has been implemented in some previous GNNs (32, 35), the role of periodicity on property predictions of these GNNs has yet been systematically studied before this work.

3) This work systematically shows the power of hybridizing descriptors with GNNs for prediction of properties. For the properties (lattice thermal conductivity, phonon internal energy and heat capacity, and total magnetization) that are highly dependent on descriptors in this work, it is shown that hybridization with descriptors can improve predictions of five GNNs (CGCNN, ALIGNN, E3NN, MEGNet, and Matformer), which shows the generalizability of the hybridization method to improve prediction of different GNNs for specific properties.

4) This work reveals that, for predictions of some properties with limited number of data points, hybridization can make simpler GNNs more predictive than more sophisticated GNNs. This discovery paves the way for accurate property prediction within the confines of small datasets.

5) This work studies where and how to input additional information into GNNs for property predictions. In previous works, additional information is mostly input at the message-passing stage as descriptors or as modifications of graph construction such as periodic self-connection, while in this work, it is shown that input of additional information at the pooling layer might be more beneficial for some properties.

RESULTS

Brief overview of GNNs

In this work, we first choose CGCNN and ALIGNN as two classic examples of GNNs to investigate their ability to capture human-designed descriptors and as examples for improving prediction ability by hybridization with descriptors. Then, for descriptors that CGCNN and ALIGNN fail to capture and for properties where hybridization with descriptors leads to substantial improvement in prediction accuracy, we also experiment with E3NN, MEGNet, and Matformer to further understand the failure and verify the effectiveness of hybridization.

The architecture of CGCNN is summarized in Eqs. 1 to 3

$$a_i^{(n+1)}=a_i^{(n)}+\sum _{j,k}\mathrm{\sigma }(m_{(i,j{)}_k}^{(n)}{\mathit{W}}_{\mathrm{gate}}^{(n)})\odot g(m_{(i,j{)}_k}^{(n)}{\mathit{W}}_{\mathrm{message}}^{(n)})$$ (1)

$$m_{(i,j{)}_k}^{(n)}=a_i^{(n)}\oplus a_j^{(n)}\oplus b_{(i,j{)}_k}$$ (2)

$$\mathrm{Output}=\mathrm{AGG}(a_1^{(n^{\ast })},a_2^{(n^{\ast })},\dots ,a_N^{(n^{\ast })})$$ (3)

Here, $a_i^{(n)}$ denotes the representation of atom i at layer n, $b_{(i,j{)}_k}$ representation of the kth bond between atom i and j, n* the final convolution layer, ${\mathit{W}}_{\mathrm{gate}}^{(n)}$ the gate matrix at layer n, ${\mathit{W}}_{\mathrm{message}}^{(n)}$ the message matrix at layer n, $m_{(i,j{)}_k}^{(n)}$ the message from atom j to atom i via the kth bond, ⊙ elementwise multiplication, ⊕ concatenation, σ and g nonlinear activation functions, and AGG the aggregation (readout) function. In CGCNN, the implemented aggregation function can be written as

$$\mathrm{Output}=\mathrm{FCN}\left(\frac{1}{N_a}{\displaystyle \sum _{i=1}^{N_a}}{a}_i^{n^{\ast }}\right)$$ (4)

where the output is calculated by first taking the average of all atom representations and then feeding the averaged representation to a fully connected network. The reason for using average pooling (Eq. 4) is that intensive materials properties, such as bandgap and refractive index, are invariant to the (supercell) size of crystal structures. In summary, in each convolution layer, CGCNN uses neighboring atoms and bond length as messages to each atom and updates each atom representation by feeding the messages into a gate layer and a message-processing layer. After convolutions, CGCNN pools all atom representations by taking the average and input the pooled materials representation into a fully connected network to compute the property.

Beyond CGCNN, we incorporate four other GNNs that are equipped with more advanced features as listed in Table 1: ALIGNN explicitly encodes angle information; E3NN directly encodes relative vectors to construct representations equivariant to rotations; MEGNet uses a global attention-based aggregation function that allows for global information exchange at the pooling stage; and Matformer uses the transformer architecture to pass information between nodes at the convolution stage and offers the option to connect the same atoms at the adjacent cells to encode periodicity. Moreover, ALIGNN, MEGNet, and Matformer update embedding beyond node, including edge, angle, and state embeddings. The choice of the five GNNs strengthens the generalizability of the study, as it covers a classic GNN as well as a GNN with more geometric features, an equivariant GNN, a GNN with more advanced pooling function, and a graph transformer. In the following, we use the GNNs with their default settings unless otherwise specified.

Table 1. Comparison of architectures of CGCNN, ALIGNN, E3NN, MEGNet, and Matformer.

The features that are different from the others in the column and/or necessary to highlight are bolded.

Model	Geometry information	Embedding update	Pooling function	Convolution style
CGCNN (16)	Distance	Node	Average	Message passing
ALIGNN (21)	Distance + angle	Node + edge + angle	Average	Message passing
E3NN (25)	Relative vector	Node	Average	Message passing
MEGNet (33)	Distance	Node + edge + state	Global attention	Message passing
Matformer (32)	Distance	Node + edge	Average	Transformer

Learning and predicting human-designed descriptors

In this section, we use CGCNN and ALIGNN to learn and predict structural descriptors of a subset of crystal structures in the Materials Project database (43) (“MP dataset” as below; details in Methods) to examine the ability of the GNNs to capture certain knowledge behind the descriptors. As a baseline, we also use ROOST (44), one of the most powerful composition-only deep learning models, to learn and predict the structural descriptors.

In Table 2, we show the accuracies of predictions of some of the most basic local structural descriptors calculated by matminer (45) and pymatgen (46) from CGCNN, ALIGNN, and ROOST in terms of R² scores ($R^2=1-\frac{\sum {(y_i-y_{i,\mathrm{true}})}^2}{\sum {(y_{i,\mathrm{true}}-\overline{y})}^2}$, y_i predicted value, y_i,true true value, $\overline{y}$ mean of true values). We can see that, for most local structural descriptors, both CGCNN and ALIGNN can properly predict them with R² scores close to or higher than 0.8, and both of the two structure-based models outperform the composition-only model (ROOST). Because local descriptors in this work are essentially statistics of local environments around each atom, the explicit encoding of bond angles (three-body interaction) in ALIGNN might explain why ALIGNN outperforms CGCNN for learning local structural descriptors. The cases with lower R² scores, such as maximum relative bond length and standard deviation of average bond angles, can be attributed to the fact that average pooling (Eq. 4standard deviation) is used by both CGCNN and ALIGNN to obtain the mean statistics of atom representations, while the two descriptors here describe the maximum and standard deviation of a collection of atomic environments. For the coordination number defined by the Voronoi method (47), we can see that CGCNN cannot capture it very well, because the Voronoi coordination number is defined in a more complicated way than that defined by the method of nearest neighbors as in CGCNN and ALIGNN. The encoded angular information boosts the ability of ALIGNN to capture coordination number, as the algorithm to determine the Voronoi coordination number heavily uses the angular information (47). We anticipate that GNNs that determine neighbors by the Voronoi method, such as the iCGCNN (48), might have strong ability to capture the Voronoi coordination number.

Table 2. Prediction results of local structural descriptors calculated by matminer from CGCNN, ALIGNN, and ROOST, respectively.

For all descriptors, the R² score of regression results are reported. All values smaller than 0.8 are underlined. All models are used with intensive pooling.

Model	CGCNN	ALIGNN	ROOST
Mean absolute deviation in relative bond length	0.97 ± 0.01	0.98 ± 0.00	0.85 ± 0.02
Maximum relative bond length	0.75 ± 0.02	0.82 ± 0.02	0.65 ± 0.03
Minimum relative bond length	0.91 ± 0.01	0.93 ± 0.01	0.84 ± 0.02
Maximum neighbor distance variation	0.92 ± 0.01	0.95 ± 0.00	0.48 ± 0.04
Minimum neighbor distance variation	0.94 ± 0.01	0.96 ± 0.00	0.70 ± 0.02
Range neighbor distance variation	0.90 ± 0.01	0.96 ± 0.00	0.78 ± 0.03
Mean neighbor distance variation	0.98 ± 0.00	0.97 ± 0.00	0.60 ± 0.03
Standard deviation neighbor distance variation	0.92 ± 0.01	0.95 ± 0.01	0.77 ± 0.02
Mean average bond length	0.88 ± 0.01	0.90 ± 0.01	0.73 ± 0.02
Standard deviation average bond length	0.91 ± 0.01	0.94 ± 0.01	0.80 ± 0.01
Mean absolute deviation in atomic volume	0.94 ± 0.01	0.94 ± 0.01	0.73 ± 0.02
Mean average bond angle	0.83 ± 0.02	0.87 ± 0.01	0.56 ± 0.03
Standard deviation average bond angle	0.74 ± 0.02	0.81 ± 0.02	0.39 ± 0.05
Mean Voronoi coordination number	0.68 ± 0.03	0.83 ± 0.01	0.48 ± 0.05
Standard deviation Voronoi coordination number	0.81 ± 0.02	0.89 ± 0.01	0.58 ± 0.03
Structural complexity per atom	0.91 ± 0.01	0.86 ± 0.01	0.48 ± 0.06

In addition to basic local descriptors, we also test the ability of CGCNN, ALIGNN, E3NN, MEGNet, and Matformer to capture knowledge behind more global structural descriptors. In Table 3, we show the accuracies of predictions of the basic global structural descriptors calculated by matminer (45) and pymatgen (46) from CGCNN, ALIGNN, E3NN, MEGNet, Matformer, and ROOST. All the structure-based GNNs can accurately predict density, volume per atom, packing fraction, and number of atoms in the primitive cell [in this work, the “primitive cell” is defined as the Niggli reduced cell (49, 50)]. However, they cannot well capture structural complexity per cell (51) and lattice constants (a, b, c, α, β, and γ; in this work, a denotes the length of the longest lattice vector, c the shortest, and α denotes the largest lattice angle, γ the smallest). All structure-based models outperform the composition-only model, and ALIGNN, E3NN, MEGNet, and Matformer without encoded periodicity outperform CGCNN for number of atoms and lattice length. More discussion about the failure and comparison between models are provided in the next section. Note that the poor predictions of lattice constants and accurate predictions of density and vpa (volume per atom) do not contradict with each other, and more discussions are provided in the Supplementary Materials.

Table 3. Prediction results of global structural descriptors from CGCNN, ALIGNN, E3NN, MEGNet, Matformer without encoded periodicity, and ROOST, respectively.

Except crystal system where multiclass classification accuracy is reported, for all other descriptors, the R² scores of regression results are reported. All values smaller than 0.8 are underlined. All models are equipped with intensive pooling.

Model	CGCNN	ALIGNN	E3NN	MEGNet	Matformer without periodicity	ROOST
Structural complexity per primitive cell	0.32 ± 0.04	0.67 ± 0.02	0.44 ± 0.03	0.46 ± 0.03	0.61 ± 0.02	0.38 ± 0.03
Density	0.99 ± 0.00	0.99 ± 0.00	0.99 ± 0.00	0.99 ± 0.00	0.99 ± 0.00	0.87 ± 0.01
Volume per atom	0.80 ± 0.01	0.93 ± 0.01	0.89 ± 0.01	0.85 ± 0.01	0.90 ± 0.01	0.16 ± 0.06
Packing fraction	0.99 ± 0.00	0.99 ± 0.00	0.99 ± 0.00	0.99 ± 0.00	0.99 ± 0.00	0.90 ± 0.01
Number of atoms per primitive cell	0.77 ± 0.02	0.89 ± 0.01	0.95 ± 0.01	0.82 ± 0.01	0.83 ± 0.01	0.70 ± 0.02
a	0.14 ± 0.05	0.43 ± 0.04	0.24 ± 0.03	0.24 ± 0.04	0.42 ± 0.04	0.13 ± 0.06
b	0.41 ± 0.04	0.67 ± 0.03	0.54 ± 0.02	0.54 ± 0.04	0.64 ± 0.03	0.23 ± 0.04
c	0.54 ± 0.04	0.79 ± 0.02	0.61 ± 0.02	0.67 ± 0.03	0.75 ± 0.02	0.35 ± 0.04
α	0.15 ± 0.06	0.13 ± 0.06	0.10 ± 0.06	0.11 ± 0.07	0.11 ± 0.06	−0.10 ± 0.09
β	0.13 ± 0.07	0.15 ± 0.06	0.11 ± 0.06	0.11 ± 0.07	0.12 ± 0.06	−0.09 ± 0.09
γ	0.10 ± 0.08	0.08 ± 0.08	0.09 ± 0.07	−0.05 ± 0.09	0.01 ± 0.09	−0.09 ± 0.10
Crystal system	0.63 ± 0.03		0.72 ± 0.02			0.59 ± 0.03

In Table 3, we also show the accuracy of multiclass classification of crystal system of crystal structures from CGCNN, E3NN, and ROOST (we do not use the other three models as their training logics are structured with the style of binary classification), from which we can see that, although CGCNN can classify crystal systems of structures with around 60% accuracy, ROOST can also achieve similar accuracy, which agrees with CRYSPNet that the type of crystal structure of inorganic materials can be partially determined by composition (52). The similar classification ability of CGCNN and ROOST toward crystal system suggests that the geometric information encoded by CGCNN does not substantially help to classify crystal systems, which supports our conclusion that CGCNN cannot capture lattice constants well, as crystal system is determined by the relationship between lattice constants. E3NN shows higher classification accuracy than CGCNN and ROOST, which agrees with the higher ability of E3NN to capture periodicity than CGCNN.

Limitations of GNNs for capturing periodicity

Although previous works have suggested that lattice constants of crystal structures are learnable on the basis of only compositions (52, 53), the results in this work show that even with structures as input, GNNs without explicit encoding of periodicity cannot capture lattice constants well. In this section, we analyze the possible reasons for the failure and obtain insights for improving GNNs for crystal structures.

Lattice constants describe the periodicity of atomic structures. If A(r) describes the type of atom at position r (“none” if there is no atom at that position), and if R is a linear combination of lattice vectors, then periodicity requires that

$$A(r)=A(r+R)$$ (5)

In three-dimensional (3D) space, we need three linearly independent lattice vectors to describe the periodicity of atomic structures. Lattice constants describe the periodicity by the length of lattice vectors (a, b, and c) and angles between lattice vectors (α, β, and γ). To simplify the analysis, in addition to 3D crystal structures in the MP dataset, we also consider the toy cases of quasi-1D atomic chains as in Fig. 2A, where periodicity is imposed only along the x direction. In this quasi-1D space, we only need the length of the lattice vector (a) to describe the periodicity: A(r) = A(r + a).

Fig. 2. Limitations of GNNs for capturing periodicity.

(A) Illustration of the receptive field of an atom in a GNN and the periodicity of a 1D structure. Here, atom i receives information from atoms 1 to N, and two cases of periodicity are plotted: the short periodicity from atom 1 to 3 and the long periodicity from atom 1 to N + 1. (B) Illustration of 1D single carbon chains as toy structures. The chains are along the x direction with periodicity, with random displacement of each atom in the y and z directions. (C) Illustration of 1D chains with zigzag and armchair configuration, respectively. (D) Illustration of 1D double chain. (E and F) a_true versus a_pred of the datasets of 1D short chains and 1D long chains from default CGCNN. (G) R² scores of predictions of a of 1D short chains and 1D long chains, and a, b, and c of the MP dataset, from default CGCNN, CGCNN with eight convolution layers, and CGCNN connecting 18 nearest neighbors within 12 Å, respectively. (H) R² scores of prediction of a of 1D short chains and 1D long chains,and natoms, a, b, and c of the MP dataset from CGCNN with average pooling and CGCNN with sum pooling, respectively.

For GNNs with average pooling in Eq. 4, because they use the average local atomic environments to represent the atomic structures, they capture periodicity by learning how Eq. 5 affects the local atomic environments within the receptive fields of atoms in the GNNs. The receptive field of each atom describes the range of the space where information can be propagated to the atom through the GNNs, and it depends on the number of neighbors each atom can connect to and the number of convolution layers in the GNNs

$$\text{Range of receptive field}\propto \text{number of neighbors}\ast \text{number of convolutions}$$ (6)

If the length of the periodicity (length of lattice vector) is smaller than the length of the receptive fields of atoms in the GNNs, then the GNNs might be able to capture the short periodicity; however, if the length of the periodicity is larger than the length of the receptive fields of atoms, then in principle, the GNNs cannot capture the long periodicity. For example, as in Fig. 2A, if the periodicity is short, such as the top red arrow that requires that atom 1 and atom 3 (atom n and atom n + 2) have the same type and coordinates in the y and z directions, then the local atomic environment input to atom i is constrained by such periodicity, and the GNNs might be able to capture the constraint and periodicity. However, if the periodicity is long, such as the bottom red arrow describing that the periodicity is imposed between atom 1 and atom N + 1 (one atom beyond the receptive field), then there is no constraint inside the receptive field of atom i, and the GNNs cannot capture the long constraint and the periodicity.

To analyze the behaviors of GNNs on capturing periodicity, in this section, we introduce toy datasets of quasi-1D carbon chains as illustrated in Fig. 2B (“1D dataset” as below; details in Methods), and we create two versions of the 1D datasets: a short dataset where the periodicity of each chain is shorter than the receptive fields of atoms (1D, short), and a long dataset where the periodicity is longer than the receptive fields (1D, long). We use the default CGCNN to learn and predict the length of lattice vector (a) of the two datasets, and in Fig. 2 (E and F), we show the predicted a versus true a of the two datasets. We can see that, for the short chains, CGCNN can predict a well with the R² score larger than 0.8, while for the long chains, CGCNN cannot predict a well. The prediction results of a of the quasi-1D carbon chains support our analysis above that GNNs might be able to capture short periodicity while hard to capture long periodicity. Note that, the analysis for 1D long chains can represent thousands of materials in the Materials Project dataset where the receptive field of typical GNNs might be shorter than the periodicity of the crystal structures. More discussions are provided in fig. S5 in the Supplementary Materials.

Although the periodicity of most short chains in this work can be learned properly as in Fig. 2E, theoretically, GNNs with limited local expressive power are not able to fully determine the periodicity. Because Chen et al. (54) have proved the equivalence between the ability of GNNs to distinguish graphs and approximate graph functions, if a GNN cannot distinguish two atomic graphs with different periodicity, then the GNN cannot fully determine the graph function describing the periodicity. In Fig. 2C, we show two cases of 1D chains: a 1D zigzag chain and a 1D armchair chain, which represent structure prototypes of some real crystal structures such as organic crystals (55) and metal chalcogenides (56). If a GNN uses only diatomic distances to encode local atomic environments (such as CGCNN), and if the GNN only connect to the nearest neighbors (1 to 2), then the GNN cannot distinguish different zigzag and armchair 1D chains with the same bond length but different bond angles and cannot capture the angle dependence of a. If the GNN can connect to the second nearest neighbors (1 to 3), then the GNN is able to distinguish zigzag and armchair 1D chains with different bond angles; however, it is still not able to distinguish between zigzag and armchair chains with the same bond length and bond angle. The analysis suggests that, to improve the ability of GNNs to capture short periodicity, it might be helpful to increase the local expressive power of GNNs to distinguish structures with different periodicity.

In Fig. 2G, we show the effects of number of convolution layers and number of neighbors of CGCNN on capturing periodicity of 1D chains and 3D crystal structures. As in Eq. 6, both increasing number of convolution layers and increasing number of neighbors extend the receptive fields of atoms in CGCNN, and as the discussion of zigzag and armchair chains above, increasing number of neighbors can lead to higher local expressive power to distinguish graphs. From Fig. 2G, we can see that for short chains, increasing the number of neighbors leads to better prediction of a, which supports our suggestion above that improving the local expressive power can help to capture short periodicity. For long chains, both increasing number of neighbors and number of convolution layers result in better prediction of a, indicating that extending the receptive fields of atoms in CGCNN can help to capture long periodicity. As for the length of lattice vectors of real 3D structures in the MP dataset (mixed with short and long structures as in fig. S5 in the Supplementary Materials), we can see that both increasing number of neighbors and number of convolution layers lead to better prediction of a, b, and c. However, we find that increasing number of convolution layers by 133% and number of neighbors by 50% just leads to moderate improvement of prediction of a, b, and c. There are two possible reasons for the small degree of improvement: (i) as shown in eq. S4 in the Supplementary Materials, in real 3D closely packed materials, the radius of receptive field (r) is proportional to $\sqrt[3]{n}$ (n is the number of neighbors). Therefore, although in Fig. 2 we test CGCNN with 50% higher cutoff radius and maximum number of neighbors, for closely packed 3D crystal structures, the ratio of elongation of receptive field might just be $\sqrt[3]{1.5}\approx 1.14$, which limits the impact of increasing cutoff radius and maximum number of neighbors on capturing periodicity. (ii) Even if the receptive field can be effectively elongated, the problem of oversmoothing and oversquashing (17, 18) associated with GNNs with too many convolutions and neighbors might deteriorate the expressive power of GNNs, which also limits the ability of GNNs to capture periodicity. As shown in Fig. 2G, for 1D short chains, increasing the number of convolution layers deteriorates the power of CGCNN to capture periodicity, which might be attributed to oversmoothing and oversquashing (17, 18) as discussed above. The moderate improvement in this work also agrees with the recent report by Li et al. (57) that simply increasing number of layers and cutoff radius cannot lead to better prediction of long-range properties of molecules. Because the cost of graph convolution operations is proportional to number of convolution layers and neighbors, we suggest that simply increasing number of convolution layers and neighbors might not be an ideal way to improve the ability of GNNs to capture periodicity.

The analysis above is based on average pooling in Eq. 4. If we use sum pooling in Eq. 7 with size extensibility

$$\mathrm{Output}={\displaystyle \sum _{i=1}^{N_a}}\mathrm{FCN}(a_i^{n^{\ast }})$$ (7)

then the GNNs capture periodicity by summing contributions of each atom to the lattice vectors. In Fig. 2H, we show the R² scores of predictions of a of the 1D chains and natoms, a, b, and c of the MP dataset from CGCNN with average pooling and sum pooling, respectively. For 1D short chains, sum pooling can lead to better prediction of a than average pooling, which might be explained by the fact that sum pooling is more expressive than average pooling (58). For 1D long chains, sum pooling can result in substantially better prediction of a than average pooling, because average pooling requires that each atom encodes information from one end of the long primitive cell to the other end to capture the structural constraint imposed by the periodicity, while sum pooling needs only local contributions of each atom to the lattice vectors. Consistent with the results of a of the 1D chains, for natoms, a, b, and c of the MP dataset, sum pooling can also result in better prediction than average pooling. The stronger ability of sum pooling to capture periodicity might lead to better prediction of extensive material properties, and in table S1 and fig. S2 in the Supplementary Materials, we show that sum pooling can provide better prediction than average pooling for phonon internal energy (U), phonon heat capacity (C_v), and total magnetization (M).

Despite the improvement, we suggest that sum pooling is not an ideal solution to the challenge of capturing periodicity. Periodicity and lattice constants of the primitive cells do not scale with supercell size and are intensive characteristics of crystal structures. In principle, sum pooling cannot be used in ML of materials’ intensive properties because of the requirement of (supercell) size invariance (16). The improvement of sum pooling over average pooling in Fig. 2H and fig. S2 is based on the fact that primitive cells of crystals are used as input to the GNNs in this work. Even if only primitive cells are input to the GNNs, sum pooling might also fail to capture periodicity in some cases, as periodicity does not always scale with the number of atoms in the primitive cells. For example, in Fig. 2D we show the case of 1D double chains. Compared with 1D single chains in Fig. 2 (B and C), 1D double chains can have similar periodicity but twice number of atoms. In fig. S3, we show that, compared with the datasets with only 1D single chains, sum pooling is less powerful to capture the periodicity of the datasets mixed with 1D single and double chains.

From Table 3, we can see that ALIGNN, E3NN, MEGNet, and Matformer outperforms CGCNN in the prediction of natoms, a, b, and c of the MP dataset. The improved predictive ability of ALIGNN and E3NN could result from their stronger local expressive power than CGCNN, as ALIGNN explicitly encodes bond angles and E3NN directly encodes relative vectors that, in principle, contain full information of local environments; the improved predictive ability of MEGNet could result from its global attention-based pooling function and the associated stronger expressive power to global long-range information; and the improved ability of Matformer to capture periodicity could come from the more effective information propagation of the transformer architecture: in transformer, in each convolution the center atom of the receptive field can communicate with the atom at the end of the receptive field directly, while in normal message passing, the center atom could only communicate with the end atom together with information from all other intermediate atoms as in Eq. 1. In addition, the improved predictive ability of ALIGNN, MEGNet, and Matformer over CGCNN might also come from the embedding update of edge, angle, and state. The effect of embedding update can be observed from the comparison between ALIGNN and E3NN. Although theoretically E3NN has higher local expressive power than ALIGNN, ALIGNN updates node, edge, and angle representations during convolution, while E3NN only updates node representations. Because Topping et al. (59) has proved the effect of edge information update during convolution on alleviating the oversquashing problem (17) and consequently improving learnability of long-range information, ALIGNN with node, edge, and angle update has better ability to capture long-range information than E3NN with only node update.

In Table 3 and fig. S4, we show that GNNs cannot learn the lattice angles of the primitive cell well, and sum pooling, more convolutions, and more neighbors do not improve the prediction. Here, we partially attribute these results to the artificial choice of lattice angles. More discussions regarding the determination of the primitive cell are provided in the Supplementary Materials. In this work, we choose the set of six parameters (a, b, c, α, β, and γ) as a widely used rotationally invariant representation of lattice vectors, which might add artificial difficulty to the learning of periodicity. For example, in addition to the problems associated with learning and prediction of a in the 1D cases as above, for learning and prediction of the length of the longest lattice vector of 3D structures, the GNNs need to first identify which dimension is associated with the largest length and then determine the largest lattice length. For fairer evaluation, it is necessary to develop representations of periodicity that are equivariant to rotations to avoid this additional difficulty.

According to MLatticeABC (53) and CRYSPNet (52), lattice constants of high-symmetry materials are reported to be learnable on the basis of only compositions of materials, while here, we show that lattice constants are not learnable by the GNNs even with structures as input. In the previous works, materials with different symmetry are learned separately, and lattice constants of high-symmetry materials are reported to be more learnable than that of low-symmetry materials, while in this work, the MP dataset is mixed with different symmetries and is biased to materials with low symmetry. More details about the MP dataset are provided in Methods.

In this section, we discuss the limitations of the GNNs on capturing periodicity mainly in three aspects: limited local expressive power, difficulty of capturing long-range information, and average pooling as the readout function. For local expressive power, advancements of GNNs to capture more structural characteristics, such as ALIGNN-d for dihedral angles (23), equivariant representations for orientation of bond vectors (26, 60), and simultaneously encoding canonical positional and structural information in LSPE (29) and SAT (30), might be helpful to better capture periodicity of structures with lattice vectors shorter than the receptive fields. For long-range information, on the one hand, efforts to train very deep GNNs effectively and efficiently, such as DeeperGATGNN (61), are helpful to extend the receptive fields of atoms. On the other hand, the idea of topological message passing (62, 63), fragment-based message passing (57), periodic self-connecting (64), and propagating information in the reciprocal lattice (37) might be useful to capture long-range information by connecting nodes in the same cell complex that are far from each other, and the idea of implicit GNNs (65) might also be useful to bypass the problems associated with training very deep GNNs by obtaining implicitly defined state vectors from a fixed-point equilibrium equation. It is also necessary to further develop readout functions to collect the long-range information with size invariance, and the whole-graph self-attention based readout function used in GraphTrans (34) might be a good starting point to collect global information of crystals.

Descriptor-hybridized deep representation learning

From the results of learning human-designed descriptors, we know that GNNs might not capture all knowledge behind human-designed descriptors. One way to overcome the issue is to design better GNN architectures for specific information, such as long-range information. Another way to overcome the issue is to input the missing knowledge into the deep representation learning models. Although this idea is straightforward and used in previous works (35, 66, 67), the previous works did not explore the role of this added information systematically. In this section, we show the mechanisms of how inputting certain knowledge to GNNs improves the prediction of materials properties, and we find that the hybridization with descriptors can lead to a notable improvement for prediction of some material properties, especially vibrational properties that largely depend on periodicity. We also observe that, the relative prediction performance between different GNN models might change before and after the hybridization, especially for properties with limited number of data points.

We construct the descriptor-hybridized GNNs as below

$$\mathrm{Output}=\mathrm{FCN}(\frac{1}{N_a}{\displaystyle \sum _{i=1}^{N_a}}{a}_i^{n^{\ast }}\oplus \mathrm{descriptors})$$ (8)

In other words, we concatenate the vector of descriptors to the vector of learned representation and input the hybridized representation vector to the fully connected network. In the following, we hybridize all descriptors in Tables 2 and 3 and space group number with GNNs.

In Table 4, we show the prediction results of descriptor-hybridized CGCNN and ALIGNN (de-CGCNN and de-ALIGNN) on 13 material properties. The set of properties includes final total energy (E_fin.), bandgap (E_g), bulk and shear modulus (K and G), lattice thermal conductivity (κ), phonon internal energy and heat capacity at 300 K (U and C_v), Poisson ratio (v), modulus of the piezoelectric tensor (||e||_∞), electronic and total dielectric constant (ɛ_e. and ɛ_t.), refractive index (n), and total magnetization (M). The errors of the ML models are presented using the metric $\mathrm{MAE}/\mathrm{MAD}=\frac{\sum \mid y_i-y_{i,\mathrm{true}}\mid }{\sum \mid y_{i,\mathrm{true}}-\overline{y}\mid }$, which is invariant to scaling and used in the ALIGNN paper (21). Typically, a model with a MAE/MAD <0.2 is considered a good predictive model (10, 21). We can see that de-CGCNN has improved prediction performance for most properties compared with the original CGCNN, and de-ALIGNN has close to or larger than 10% improvement for four properties (κ, U, C_v, and M) and comparable performance for other properties compared with the original ALIGNN. Both de-CGCNN and de-ALIGNN outperform the descriptor-only model for all properties, regardless of whether CGCNN and ALIGNN outperform the descriptor-only model.

Table 4. Prediction results of 13 material properties from random forest, CGCNN, de-CGCNN, ALIGNN, and de-ALIGNN, respectively.

The properties without explicit unit have the unit of 1. The error in each cell under the column of model name is the MAE/MAD ratio. The number below each symbol of property is the dataset size. The lowest error in each row is bolded. Here, all the models are used with intensive pooling. More comparison between different pooling functions and normalizations is provided in table S1.

	Random forest with descriptors	CGCNN	de-CGCNN	ALIGNN	de-ALIGNN
Log(κ), 2.6k	0.36 ± 0.02	0.35 ± 0.03	0.26 ± 0.01	0.27 ± 0.02	0.25 ± 0.01
Efin. (eV/atom), 140k	0.40 ± 0.02	0.057 ± 0.006	0.042 ± 0.004	0.045 ± 0.004	0.052 ± 0.004
U (KJ/mol-cell), 1.5k	0.14 ± 0.01	0.71 ± 0.03	0.060 ± 0.005	0.53 ± 0.04	0.10 ± 0.01
Cv [J/(mol-cell*K)], 1.5k	0.11 ± 0.01	0.76 ± 0.04	0.058 ± 0.004	0.63 ± 0.04	0.089 ± 0.006
K (GPa), 13.3k	0.35 ± 0.01	0.22 ± 0.02	0.22 ± 0.01	0.20 ± 0.01	0.21 ± 0.01
G (GPa), 13.3k	0.49 ± 0.01	0.40 ± 0.02	0.36 ± 0.02	0.34 ± 0.02	0.33 ± 0.01
v, 13.3k	0.80 ± 0.02	0.80 ± 0.03	0.79 ± 0.03	0.72 ± 0.03	0.70 ± 0.02
Eg (eV), 140k	0.51 ± 0.01	0.23 ± 0.01	0.22 ± 0.01	0.19 ± 0.01	0.20 ± 0.01
||e||∞ (C/m2), 3.7k	0.79 ± 0.02	0.81 ± 0.03	0.76 ± 0.02	0.80 ± 0.03	0.74 ± 0.02
ɛe., 7.3k	0.50 ± 0.01	0.26 ± 0.02	0.24 ± 0.02	0.21 ± 0.02	0.21 ± 0.01
ɛt., 7.3k	0.67 ± 0.01	0.56 ± 0.02	0.55 ± 0.02	0.50 ± 0.02	0.49 ± 0.01
n, 7.3k	0.45 ± 0.01	0.24 ± 0.02	0.21 ± 0.01	0.20 ± 0.01	0.19 ± 0.01
M (μB/formula), 140k	0.62 ± 0.01	0.41 ± 0.02	0.34 ± 0.01	0.34 ± 0.02	0.31 ± 0.01

For κ, U, C_v, and M, in Table 5, we further compare the predictive power of E3NN, de-E3NN, MEGNet, de-MEGNet, state-MEGNet (inputting descriptors into the built-in state attribute in the message-passing layer of MEGNet), and Matformer with and without encoded periodicity by connecting atoms in adjacent cells and de-Matformer with and without encoded periodicity. We can see that input of periodicity information by all approaches (hybridization in this work, state attribute in MEGNet and periodic self-connection in Matformer) can improve prediction of the four properties, further confirming the effect of incorporating periodicity information that cannot be captured by backbone GNNs on property prediction. Note that all models in Tables 4 and 5 are used with intensive pooling, and more discussions about different pooling function and normalization for extensive properties are provided in the Supplementary Materials.

Table 5. Prediction results of ML models for κ, U, C_v, and M from random forest, CGCNN, de-CGCNN, ALIGNN, de-ALIGNN, E3NN, de-E3NN, MEGNet, state-MEGNet, de-MEGNet, Matformer with encoded periodicity, Matformer without encoded periodicity, de-Matformer with encoded periodicity, de-Matformer without encoded periodicity, and M3GNet-UMLFF.

The lowest error in each column is bolded. Except UMLFF, all other models are used with intensive pooling.

Model	Log(κ), 2.6k	U (KJ/mol-cell), 1.5k	Cv [J/(mol-cell*K)], 1.5k	M (μB/formula), 140k
Random forest with descriptors	0.36 ± 0.02	0.14 ± 0.01	0.11 ± 0.01	0.62 ± 0.01
CGCNN	0.35 ± 0.03	0.71 ± 0.03	0.76 ± 0.03	0.41 ± 0.02
de-CGCNN	0.26 ± 0.01	0.060 ± 0.005	0.058 ± 0.004	0.34 ± 0.01
ALIGNN	0.27 ± 0.02	0.53 ± 0.03	0.63 ± 0.03	0.34 ± 0.02
de-ALIGNN	0.25 ± 0.01	0.10 ± 0.01	0.089 ± 0.006	0.31 ± 0.01
E3NN	0.32 ± 0.01	0.48 ± 0.01	0.53 ± 0.02	0.45 ± 0.01
de-E3NN	0.29 ± 0.01	0.11 ± 0.01	0.14 ± 0.01	0.39 ± 0.01
MEGNet	0.31 ± 0.02	0.64 ± 0.02	0.71 ± 0.03	0.40 ± 0.01
state-MEGNet	0.29 ± 0.02	0.097 ± 0.006	0.098 ± 0.007	0.37 ± 0.01
de-MEGNet	0.27 ± 0.01	0.057 ± 0.005	0.058 ± 0.005	0.34 ± 0.01
Matformer without periodicity	0.31 ± 0.02	0.46 ± 0.02	0.44 ± 0.02	0.40 ± 0.02
de-Matformer without periodicity	0.27 ± 0.01	0.16 ± 0.01	0.15 ± 0.01	0.33 ± 0.01
Matformer with periodicity	0.29 ± 0.02	0.28 ± 0.02	0.30 ± 0.02	0.38 ± 0.02
de-Matformer with periodicity	0.26 ± 0.01	0.14 ± 0.01	0.13 ± 0.01	0.34 ± 0.01
M3GNet-UMLFF		0.111	0.130

In Table 5, we observe that ALIGNN, E3NN, MEGNet, and Matformer have better prediction for κ, U, C_v, and M than CGCNN, which aligns with Table 3 that the four GNNs with advanced features are more capable to capture global structural information than CGCNN. An exception is that, for M, E3NN is not observed to have better prediction performance than CGCNN, which agrees with Kaba et al. (28) that equivariant networks do not show notable improvement for magnetization compared with CGCNN and deserves further investigation. We also find that all hybridized models have substantial improvements compared with the original models for the prediction of for the four properties. For random forest, we find that for most properties except U and C_v, random forest has worse prediction than GNNs and hybridized GNNs. For U and C_v, because of the strong correlation between the descriptors and the two properties, random forest with descriptors has better prediction than all original GNNs and has similar or slightly better performance compared with de-Matformer, which might come from the less overfitting of random forest than concatenating two feature vectors with large information overlap (descriptors and representations from Matformer) then feeding to fully connected networks (more discussions about overfitting are provided below).

To understand these results, we show the feature importance spectrum of de-CGCNN for prediction of C_v in Fig. 3A (here, feature importance is estimated by inputting the hybridized representations in Eq. 8 to a random forest model which outputs feature importance; more details in Methods). We can see that the human-designed descriptors play important roles in learning C_v, with a being the most important feature, while the learned features are much less important. Therefore, the poor prediction ability of CGCNN, ALIGNN, E3NN, MEGNet, and Matformer without encoded periodicity for U and C_v can be explained by the fact that a is important to the two properties but these models without explicit periodicity information cannot learn a well as above. The distribution of feature importance agrees well with the phenomenon in Table 5 that using the ML model based on only human-designed descriptors can have lower errors for prediction of U and C_v compared with the GNNs.

Fig. 3. Feature importance of descriptor-hybridized GNNs.

(A to C) Relative feature importance of representations from de-CGCNN for C_v, κ, and M, respectively. (D) Ratio of feature importance of input human-designed descriptors to the total feature importance from de-CGCNN for the 13 material properties. Feature importance is estimated by inputting the hybridized representations in Eq. 8 to a random forest model which outputs feature importance.

The importance of the input human-designed descriptors to U and C_v can be justified physically as below. Approximately, if we only consider the acoustic phonons (collective vibrations for all atoms in the primitive cell), according to the Debye model of density of states, the phonon internal energy (U) and heat capacity (C_v) per primitive cell can be written as (68)

$$U\approx 9k_{B}T{\left(\frac{T}{\text{Θ}}\right)}^3{\displaystyle {\int }_0^{x_D}}dx\frac{x^3}{e^x-1}$$ (9)

$$C_{\mathrm{v}}={\left(\frac{\mathrm{\partial }U}{\mathrm{\partial }T}\right)}_{\mathrm{v}}\approx 9k_B{\left(\frac{T}{\text{Θ}}\right)}^3{\displaystyle {\int }_0^{x_D}}dx\frac{x^4{e}^x}{{(e^x-1)}^2}$$ (10)

$$x_D\equiv \frac{\text{Θ}}{T}$$ (11)

$$\text{Θ}=\frac{\hslash v}{k_B}{\left(\frac{6{\mathrm{\pi }}^2}{V}\right)}^{\frac{1}{3}}$$ (12)

where ϴ is the debye temperature, V is the volume of the primitive cell, and v is the velocity of sound, which can be approximated by the first-order Hooke’s law

$$v\approx \sqrt{\frac{C}{m}}d$$ (13)

where C is the effective spring constant, m is the mass of atoms in the primitive cell, and d is the effective distance between atomic planes along the direction of vibration. Therefore, with the information of V, C, m, and d, we can estimate acoustic U and C_v per primitive cell at given T within the Debye model. Because the set of descriptors in this work includes density and lattice constants, the information of V, m, and d can be directly obtained by ML models from the input descriptors. For C, because it is related to the bonding strength, it can be estimated by the bond length–related descriptors. Consequently, ML models based on the set of descriptors in this work can approximate U and C_v well within the Debye model, which explains why ML model based on only descriptors outperforms the original GNNs in Table 5. More discussions about the relation between this work and other direct predictions of phonon-related properties (60, 69, 70, 71) are provided in the Supplementary Materials.

In addition to direct prediction, phonon internal energy and heat capacity can also be predicted by ML force fields, such as in the study by Ladygin et al. (72) where a moment tensor potential is used to study phonon properties of Al, Mo, Ti, and U, Babaei et al. (73) where a Gaussian approximation potential is used to study phonon properties of Si, and Dhaliwal et al. (74) where a potential based on random Fourier features is used to study phonon properties of graphene. However, there still lacks a universal ML force field (UMLFF) that has shown the ability to calculate phonon properties for different materials with higher accuracy than direct prediction of phonon properties by data-driven ML models. One of the most recent UMLFFs is in the study by Chen et al. (22), where a M3GNet-based UMLFF is trained on the trajectories of geometry relaxation of structures in the Materials Project database. The UMLFF is shown to have accurate predictions of energy (R² = 0.959) and force (R² = 0.984) for different inorganic materials with almost all elements in the periodic table. To directly compare the performance of predicting phonon-related properties by our descriptor-hybridized GNNs and the UMLFF, we use the UMLFF to calculate U and C_v at 300K of materials in the same test set as de-CGCNN and de-ALIGNN. The MAE/MAD ratio of U and C_v by the UMLFF are 0.111 and 0.130, respectively. According to Table 5, the MAE/MAD ratio of U and C_v can be as low as 0.057 and 0.058, respectively, which demonstrates the importance of this work as it provides more accurate predictions of phonon-related properties than the current UMLFF. The lower error of direction prediction models compared with the UMLFF for U and C_v might result from the fact that the direct prediction models are specifically trained on the target properties (U and C_v), while the UMLFF is trained on energies and forces of geometric configurations near equilibrium and some of the distorted configurations important for phonon calculations might not be covered in the training set.

κ and M are another two properties with improvement from all descriptor-hybridized GNNs (around 10% in these cases). It is known that κ depends heavily on periodicity (75), and as shown in Fig. 3B, some input descriptors, including b, are important to the prediction of κ. As for M, as shown in Fig. 3C, some descriptors like structural complexity and lattice constants contribute to the prediction of M. In Fig. 3D, we show the ratio of feature importance from the human-designed descriptors to the total feature importance from de-CGCNN. We can see that most properties without notable improvement in Table 4 have low contributions from the input human-designed descriptors, with the exception of v and ||e||_∞ where all the models perform poorly. Here, we choose bandgap and total energy for more detailed discussion. For bandgap, from Rajan et al. (76), we know that bandgap is strongly correlated with volume per atom, which is reasonable as volume per atom reflects the degree of orbital overlap between atoms, and from Wu et al. (77), we know that bandgap is strongly correlated with bond angle, which is also reasonable as bond angle represents how atomic orbitals hybridize with each other (such as the difference between sp³ and sp² hybridization). Because the important features of bandgap, such as volume per atom and bond angle, can be well captured by the original GNNs as shown in Tables 2 and 3, it is reasonable that the descriptor-hybridized GNNs cannot improve predictions of bandgap as the descriptors do not provide useful information that the original GNNs cannot capture. As for total energy, we know that for 3D inorganic materials, short-range interactions dominate the total energy [this is straightforward in structures with covalent bonds; even for ionic systems or metallic systems, the electrostatic screening effect (78) diminishes the role of long-range interaction on total energy]. Because the descriptors about short-range interactions can be well captured by the original GNNs as in Table 2, it is also reasonable that the descriptor-hybridized GNNs cannot improve predictions of total energy.

The observation that hybridization with descriptors has larger improvement for CGCNN than the other four GNNs with advanced features might be explained by the fact that CGCNN captures these descriptors worse than the four GNNs with advanced features as in Table 3, therefore hybridization with descriptors provides more missing information to CGCNN than others. In addition to more improvement, in some cases, hybridization can make simpler GNNs more predictive than more sophisticated GNNs. For example, for U and C_v, although original ALIGNN, E3NN, and Matformer have better prediction than CGCNN and MEGNet, de-CGCNN and de-MEGNet have better prediction than de-ALIGNN, de-E3NN, and de-Matformer, and for κ, although CGCNN has worse prediction than MEGNet, E3NN, and Matformer, de-CGCNN has similar or slightly better prediction compared with de-MEGNet, de-E3NN, and de-Matformer. Because of the facts that (i) κ, U, and C_v all have very limited amount of data, (ii) they are all strongly correlated with descriptors as discussed above, and (iii) we do not observe similar reverse of relative prediction performance before and after hybridization for other properties with more data points, we propose that the better performance of hybridization with simpler GNNs than more complicated GNNs might result from the less information overlap between descriptors and representations from simpler GNNs and consequently less overfitting for the small datasets (79, 80). More discussion about information overlap between descriptors and representations from GNNs is provided in fig. S6 in the Supplementary Materials. The better prediction performance of hybridized simpler GNNs underscores a potential preference for their utilization over the construction of more complex GNNs when dealing with small datasets.

Besides hybridization of descriptors at the pooling layer as Eq. 8, we can also input information of periodicity by hybridizing descriptors at the message-passing stage in MEGNet and by connecting the same atoms at adjacent cells in Matformer. However, in Table 5, we can observe that de-MEGNet (hybridization at the pooling stage) has better prediction than state-MEGNet (hybridization at the message-passing stage), and de-Matformer without periodic self-connection has better prediction than Matformer with periodic self-connection, which suggests that providing more information at the pooling stage might be better than at the message-passing stage. We propose three possible reasons for the phenomenon: (i) inputting more information at the message-passing stage might lead to more severe information oversquashing problem (17, 57) as the message-passing stage has to process more information beyond local geometry, (ii) descriptors have to go through more layers to reach the final output layer when input at the message-passing stage, leading to more overfitting for properties with limited amount of data, and (iii) hybridization of descriptors at the pooling stage can bias the learned representations less correlated with the input descriptors, about which we provide more discussion in fig. S6 in the Supplementary Materials.

Other questions about hybridizing descriptors with GNN worth further investigation include (i) how the improvement scales with dataset size, (ii) how to choose the set of input descriptors for optimal performance. It will also be important to understand whether the two mentioned behaviors (scaling and selection of descriptors) are similar with or different from that of the descriptor-only models and pure deep representation learning model and (iii) how small geometric distortions, which might lead to the instability of some descriptors, such as coordination number, affect the predictions of descriptor-hybridized deep representation learning, as well as the proposed scheme of learning and predicting descriptors to exam the ability of deep representation learning.

In addition to human-designed structural and compositional descriptors, material properties can also be hybridized with GNNs to improve prediction of other target properties. Hybridization with properties can be potentially very effective if the input and target properties are highly correlated. However, compared with descriptors, a main limitation of hybridization with properties is that properties are generally harder to obtain than descriptors, and in some cases, input properties are not available for all target materials. As an example, in the study by Kong et al. (67), the electronic density of states (eDOS) is hybridized with a deep learning model to improve the prediction performance of photon absorption coefficient, and up to ~10% lower error is observed compared with the unhybridized model. To solve the problem of unavailability of eDOS, Kong et al. (67) used another deep learning module to learn and generate eDOS for target materials, which provides more opportunity for hybridizing properties with deep learning models.

DISCUSSION

In summary, we propose a systematic approach to analyze the representation power of GNNs for crystal structures. We use human-designed descriptors as a bank of knowledge to test whether GNNs can capture knowledge of crystal structures behind descriptors. We find that all GNNs can capture basic local structural descriptors well but cannot capture the periodicity of crystal structures. We analyze the limitations of the GNNs on capturing periodicity from three perspectives: local expressive power, long-range information, and pooling function. We also test the idea of hybridization with descriptors to improve the performance of GNN and show that descriptor-hybridized GNNs have better prediction performance for some material properties than the original GNNs, especially phonon internal energy and heat capacity, thermal conductivity, and total magnetization. We also find that for some properties with limited amount of data, hybridization can make simpler GNNs more predictive than more sophisticated GNNs, and hybridization at the pooling stage might result in better prediction than inputting more information at the message-passing stage.

All the descriptors in this work are calculated by pymatgen (46) and matminer (45), which are one of the most widely used materials infrastructure packages and one of the most widely used materials descriptors packages, respectively. We propose that, in the future, more materials descriptors beyond bond and distance should be examined and hybridized with GNNs, and the classic force-field inspired descriptors with plenty of angle-based and dihedral angle–based descriptors (11) might a good starting point. Despite the simplicity of the descriptors in this work, we still reveal an important drawback of current GNNs for materials, which is GNNs are hard to capture periodicity of crystals, one of the most important concepts in crystallography. Because in this work periodicity is the main knowledge that is found to be not learnable by GNNs, it is straightforward that inclusion of periodicity information into GNNs only leads to improvement of predictions of properties that are highly dependent on periodicity, which explains why predictions of more than half of properties are not substantially improved in this work. On the other hand, the properties highly correlated with periodicity are all more learnable to GNNs after hybridization with descriptors, which confirms the rule to know whether the inclusion of descriptors will be useful: (i) the degree of correlation between descriptors and properties and (ii) the degree of learnability of the descriptors to GNNs. In future studies, with more descriptors that are found to be not learnable by GNNs, we expect that more properties could be better predicted by the hybridization of descriptors with GNNs. Therefore, we believe that, with proper choice of descriptors, hybridizing descriptors with GNNs is potentially a general method to improve predictions of property.

This work develops the systematic top-down development approach of GNNs for crystal structures. The top-down analysis of examining GNNs and the followed hybridization study in this work can be easily extended to other atomistic systems such as molecules and amorphous materials. For example, aromaticity and chirality of molecules are two descriptors whose learnability to GNNs have been thoroughly examined (81, 82), and they are often embedded in GNNs to improve the predictive power (83). However, beyond the two classic chemical concepts, there are thousands of molecular descriptors not examined yet (84), which have huge potential to be hybridized with GNNs and to inspire further development of GNNs. An example of hybridizing dozens of unexamined molecular descriptors and GNNs is by Rahaman et al. (66), where a classic GNN is hybridized with human-designed molecular descriptors. This hybridized GNN outperforms the original GNN and classic ML models based on descriptors for predictions of total energy and HUMO/LUMO energies of molecules (~10% lower errors compared with the original GNN), yet the mechanism behind the improvement is not thoroughly explained. This example suggests that with future comprehensive study of GNNs’ ability on capturing knowledge behind different molecular descriptors, potentially one can hybridize GNNs with certain molecular descriptors that cannot be captured by GNNs to further enhance the predictive power of certain molecular properties.

This work shows that the fields of human-designed descriptors and deep representation learning can be developed synergically. For new deep representation learning models, their ability in representation of crystal structures can be tested by learning existing human-designed descriptors, and for new descriptors, they can be used to reveal how well the existing deep representation learning models capture the knowledge behind these descriptors, and they can also be hybridized with deep representation learning models for improved prediction performance. Beyond property prediction, this work can potentially promote the development of inverse design of materials from three aspects: property optimization, structure generation, and structure optimization conditioned on descriptors. For property optimization, because the gradient of property to structure is typically required to search structures in the structural latent space (85), an accurate property prediction model is a premise of effective search, as inaccurate prediction models cannot provide reliable gradient. Therefore, the more accurate prediction of properties reported in this work can potentially contribute to more effective property optimization by providing more reliable gradient of property to structure. For structure generation, the disability of GNNs on capturing periodicity impedes the learning and prediction of lattice constants during the structure encoding and decoding process, respectively. As a result, current generative models for crystal structures often bypass this issue by generating crystal structures with constrained lattice [such as cubic lattice only (86)] or normalized lattice constants [such as lattice length divided by cubic root of number of atoms (87)], which might lower the diversity of generated structures as the generated lattice constants are not sampled from the original distribution of lattice constants of all known crystals. Therefore, directly encoding periodicity in GNNs might enable the generative model to learn the original distribution of lattice constants of all known crystals and generate more diverse and valid structures. For optimizing structures with desired properties, if the properties are heavily dependent on some descriptors, such as U and C_v that are heavily dependent on periodicity, then the structure optimization and generation can be conditioned on the structural descriptors, which might lead to higher efficiency of structure optimization. We hope this work may inspire further development of deep representation learning, human-designed descriptors, and hybridized property prediction models and inverse design models for crystal structures and materials science.

METHODS

Datasets

In this work, we choose 28 (in Tables 2 and 3) human-designed descriptors to test their learnability to GNNs and hybridize them (and space group number) with the five GNNs to test the prediction performance. The criterion for choosing the descriptors is that they are easy to understand and obtain from crystal structures. Number of atoms and lattice constants of the primitive cell are determined by the Niggli reduction (49) implemented in the Structure class in pymatgen (46), and other descriptors are calculated by matminer (45). For the descriptor “standard deviation average bond length” (and similar descriptors), the calculation procedure is first calculating the average bond length for each atom, then calculating the SD for the average bond length of all atoms.

In this work, most real 3D crystal structures (primitive cells) and material properties are downloaded from the Materials Project database (V2021.03.22) (43), and those for κ are from the TEDesignLab database (88). U and C_v are calculated by the PhononDos class in pymatgen (46) based on the phonon density of states from the Materials Project database (43). The dataset size for each property depends on the number of structures that have the recorded property in the two materials databases. For ML of material properties in Tables 4 and 5, we randomly split the datasets into 60, 20, and 20% as the training, validation, and test set.

For the dataset used for testing whether GNNs can capture human-designed descriptors of crystal structures, because we know that lattice constants of high-symmetry materials are reported to be more learnable than that of low-symmetry materials (52, 53) based on compositions, we create a subset of the Materials Project database (“MP dataset”) by removing some structures randomly based on their space group number

$$\mathrm{Probability}(\mathrm{removed})=\frac{\text{Space group number}}{\text{Space group number}+15}$$

where 15 is the space group number of the C2/c group, the last space group in the class of monoclinic Bravais lattice. Consequently, we have a dataset with 47,862 crystal structures biased to materials with low symmetry to test whether GNNs can learn human-designed descriptors from crystal structures. To facilitate the analysis about failure of CGCNN to capture lattice constants, we create a dataset of random 1D carbon chains (“1D dataset”). The random 1D chains are created by the pseudo-code provided in section S9 in the Supplementary Materials. For classification of crystal system in Table 3, all the structures in the Materials Project database are used. For the dataset of (1D, short), the number of atoms is set to be between [2, 9), and for the dataset of (1D, long), the number of atoms is set to be between [37, 51). In total, both datasets have 1400 data points. For ML of human-designed descriptors in Tables 2 and 3, we split the dataset into 80, 10, and 10% as the training, validation, and test set.

Models

In this work, we use the default architecture of CGCNN (31), ALIGNN (21), E3NN (25), MEGNet (33) and Matformer (32) for learning human-designed descriptors in Tables 2 and 3 unless specifically mentioned. The reason for using the default architectures is that, as in Fig. 2 (G and H), although intentionally revising their architectures can improve learning performance for some descriptors, in this work, we try to show the representation power and limit of GNNs in a setting close to those in real applications. For learning material properties in Tables 4 and 5, hyperparameter search based on the search spaces in tables S2 to S6 is conducted. All the neural networks are trained for 300 epochs (21) on a Quadro RTX 6000 graphics processing unit. For feature importance in Fig. 3, because the permutation feature importance of deep neural networks is very expensive to calculate, we estimate the feature importance by extracting the representations in Eq. 8 and then feed the representations into a random forest model to calculate the feature importance. In all tables and figures that show prediction accuracy, the error bars are the SD of errors/accuracies of five models initialized by different random seeds.

Supplementary Materials

This PDF file includes:

Supplementary Text

Figs. S1 to S6

Tables S1 to S6