Feature Blending: An Approach toward Generalized Machine Learning Models for Property Prediction

Abstract

From studying the atomic structure and chemical behavior to the discovery of new materials and investigating properties of existing materials, machine learning (ML) has been employed in realms that are arduous to probe experimentally. While numerous highly accurate models, specifically for property prediction, have been reported in the literature, there has been a lack of a generalized framework. Herein we propose a novel feature selection approach that enables the development of a unified ML model for property prediction for several classes of materials. It involves an ingenious blending of selected features from various classes of data such that the resultant feature set equips the model with global data descriptors capturing both class-specific as well as global traits. We took accurate band gaps of three distinct classes of 2D materials as our target property to develop the proposed feature blending approach. Using Gaussian process regression (GPR) with the blended features, the ML model developed here resulted in an average root-mean-squared error of 0.12 eV for unseen data belonging to any of the participating classes. The feature blending approach proposed here can be extended to additional classes of materials and also to predict other properties.

Introduction

The three paradigms of science namely empirical, theoretical, and computational have not only been contributing to and benefiting from each other through decades, but have also resulted in the generation of a huge amount of data over these years leading to a fourth paradigm. This fourth paradigm shift in materials science with computational methods leading to material discovery and property predictions is now driving the era of materials-informatics.^1,2 The workflow of material-informatics involves extraction of knowledge via data-driven machine learning (ML) methods from large amounts of unexplored computational and experimental data. ML has been utilized in the field of material science for various applications such as the discovery of new materials,³⁻⁵ force-field generation,⁶⁻¹⁰ microstructure analysis,¹¹⁻¹⁵ and property prediction.¹⁶⁻³⁰ Despite immense progress in the application of ML in material science, the applicability of all of these models thus far has been for a specific family/class of materials. Several attempts to develop multiclass generalized property prediction models surpassing the barrier of descriptor dependence have also been made. These include development of crystal graph convolutional neural networks,³¹ universal graph networks,³² SchNet,⁹ and message passing neural network,^33,34 etc. Similarly, a general-purpose ML framework using a Random Forest method was proposed by Ward et al.³⁵ in which the data set was partitioned into sets of similar materials based on their composition, followed by the designing of several models for the various classes of materials.

Since ML-based predictions depend primarily on the availability of a pristine data set employed for training, the resultant models focused on a particular class of materials show poor transferability across different classes. Increasing the variability in the data results in high prediction errors compared to specific class data since the descriptors are incapable of describing global data trends. This insufficiency can only be compensated by increasing the amount of training data of any new incoming class, which is not always feasible. Therefore, to make predictions on a class of materials with insufficient training data, it would be highly desirable to have a pre-existing generalized ML model that has previously been trained on the desired class using a bigger data set of the same class, belonging, however, to a different database. The lack of such a generalized ML framework that works across data sets poses a serious challenge in realizing the full potential of data-driven approaches and calls for a scheme that enables the development of generalized multiclass unified ML models.

In this work, we propose a feature blending approach to develop one such unified ML model generalized to unseen data from multiple participating classes of materials. A data set of 272 materials belonging to two different databases was utilized by splitting it into three different classes based on their spacegroups. The GW gap of these materials served as the target property. Blended features capture the patterns and trends elicited by the individual as well as mixed class data. The best GPR model using this feature set resulted in an rmse of 0.14/0.14 eV and R² of 0.99/0.99 for train/test sets and an rmse of 0.14/0.09/0.13 eV (average of 0.12 eV) and R² of 0.99/0.96/0.99 for validation sets belonging to the three considered classes P6̅m2, P3m1 and P3̅m1, respectively. The unprecedented accuracy can be attributed to the judicious selection of features using this approach, that represent each focused class and at the same time can also capture the complex details of the combined data set. Furthermore, some of the features belonging to the final blended set have been shown to display a universal empirical relationship with the target property that can be utilized to accelerate the estimation of the bandgap for materials with no training data.

Methodology

For developing the proposed generalized feature selection approach, structure, first-principle properties, and the property of interest (GW band gap) are extracted from open material databases.^29,36,37 Our initial feature set comprised DFT calculated properties³⁸ and standard deviation and mean of elemental properties,^39,40 resulting in 44 primary features. These properties, along with their corresponding symbols, have been listed in Table S1 in the Supporting Information. Other than the computed properties, elemental features (Table S1) are considered in the feature set due to their ease of availability and their role in structure formation (bonding). Selection of relevant features is performed using least absolute selection and shrinkage operator (LASSO) and neighborhood component analysis (NCA)(section 3 in Supporting Information).

Once the reduced set of features is obtained, to develop the prediction models, Gaussian process regression (GPR) is implemented along with automatic relevance determination (ARD) kernel. ARD kernel provides an opportunity to find the relevance of different features (proportion of contribution in the model) by selecting a different length scale for each. The relevance of features for the given output is obtained by calculating the inverse of the length scale for each feature. A larger length scale suggests smaller variation over the distribution of the function; hence, a smaller effect on prediction or lesser contribution and vice versa. GPR has been discussed further in the Supporting Information.

Results and Discussion

The 2D-materials data set was collected from two online repositories, that is, the computational 2D-materials database (C2DB)^36,37 and aNANt.⁴¹ C2DB hosts 2D-materials of numerous structural, thermodynamic and electronic properties from 30 different crystal classes, whose properties are calculated using DFT within Perdew–Burke–Ernzerhof (PBE) approximation.³⁶ This C2DB contains well converged many-body perturbation theory (GW) quasi-particle band gaps for ∼232 2D-materials. The majority of data belonged to classes P6̅m2 and P3̅m1 comprising 109 and 79 compounds, respectively. Around 44 other compounds belonging to space groups such as P1, Pm, P4/nmm, Pma2, Pmmn, P3m1, and Pmn21 were collected and only utilized for testing the generalizability of the derived empirical model. The GW band gap of C2DB spans from 1 to 10 eV for different classes. MXenes selected from the aNANt(41) database are a relatively new class of 2D-materials with chemical formula M_n+1X_nT₂ (n = 4–1), where M is early transition metals, X is either C or N, and T is functional groups attached to the top and bottom surface of MXene. This family of layered materials belongs to either space group P3m1 or P3̅m1⁴²⁻⁴⁴ and the GW band gap value ranges from 0 to 3 eV (shown in Figure 1a). In this data set, 84 compounds belonging to P3m1 space group are randomly selected. Thus, the final data set used here for ML comprised 272 compounds belonging to classes P6̅m2, P3̅m1, and P3m1.

Figure 1.

(a) Voilin plot showing the data distribution for model development with the GW band gap range, along with the template structure for different space groups. (b) Feature blending schematic: Stepwise selection and blending of various features to obtain global feature set for all participating classes. I_i denotes the individual feature for i^th class, C_ij denotes combined class features for i^th and j^th class and B_ij is the blended feature set for these classes.

The feature blending algorithm for the three classes was performed in three stages and can be understood pictorially from Figure 1b. Here, for any given class i with dataset D_i its individual class feature set derived after feature selection is represented by I_i. Similarly, C_ij denotes combined class features set obtained after applying feature selection on mixed data of ith and jth classes. B_ij on the other hand is the feature set which is obtained when three feature sets, individual sets I_i, I_j, and combined feature set (C_ij), are simply appended together and feature reduction is performed. Thus, C_ij = F(D_i + D_j) while B_ij = F(C_ij + I_i + I_j) where F(.) represents feature selection. When a third class is to be included, B_ij is taken as individual class feature set I_ij for the first two classes and process of finding combined class and blended feature sets is replicated with a new class as the second class. This procedure can be generalized for any n number of classes.

Individual Class Models

At the first stage, each class was considered individually. The first data set to be utilized was P6̅m2. Feature selection using LASSO followed by NCA resulted in eight individual-class features namely T_b^mean, C_g, C_g^std, C_mol, H_e^std, T_m, r_cov^std and E_g. During the process of feature selection for any class, it was ensured that the features had a low correlation with each other and, at the same time, have moderate to high Pearson correlation with the response variable, E_g^GW (PCC (E_g)). For class P6̅m2 this has been shown in Figure 2a and Figure 2c, respectively. From Figure 2d, it can be noticed that E_g^PBE is the most important feature in the model, also having the highest correlation with the target property. T_b is another feature that has a high correlation with E_g^GW; however, it has moderate relevance in the model. C_mol has the least relevance for this GPR model.

Figure 2.

(a) Correlation heatmap for feature selected using P6̅m2 data set with all three data sets; (b) scatter plot for the model developed using P6̅m2 data set; (c) Pearson correlation coefficient of features with GW band gap (PCC (E_g^GW)); and (d) feature importance (inverse of length scale).

After removing the 10% validation data, the remaining 90% data set was further split into a 90–10% ratio and used for the training-testing purpose. This process of splitting the train-test data was performed 2000 times, and a GPR model was built iteratively for each train-test combination by optimizing the kernel hyperparameters. The best-optimized hyperparameters were selected based on model performance on test data. This best model gave an rmse and R² of 0.10/0.11 eV and 0.99/0.99, respectively, for train/validation sets.

At this stage, the transferability of the model is ascertained, which is done by utilizing the optimized hyperparameters obtained for class P6̅m2 and training with additional 90% data from the remaining two classes, one at a time, along with the original P6̅m2 training data. On combining class P3m1 data while training, the model gave an rmse and R² of 0.21/0.26 eV and 0.98/0.70, respectively, for train/validation sets. Likewise, when trained with additional P3̅m1 data, model resulted in an rmse and R² of 0.21/0.37 eV and 0.99/0.98, respectively, for train/validation sets. The performance of this model for all three classes has been shown in Figure 2b. Evidently, this model only performed well for its parent class.

Similarly, the features selection process was repeated for class P3m1, and the above-mentioned validation process was performed on the 10% validation data. The features obtained were P^mean, C_g^mean, H_e, χ_p^std, m^std, r_atom, κ_l^std, and E_g. Using these individual-class features, the rmse and R² were 0.08/0.08 eV and 0.96/0.94 for train and validation data, respectively. The correlation plot (Figure S2c) for these features with E_g^GW showed a low correlation between E_g and E_g^GW; however, the relevance of E_g in the model was quite significant (Figure S2d). Feature selection on the third class P3̅m1 resulted in features namely, T_b^mean, EA^mean, ρ^mean, χ_p and E_g^PBE. The model for this class gave an rmse and R² of 0.10/0.10 eV and 0.99/0.99, respectively, on train/validation sets. Next, transferability of both P3m1 and P3̅m1 individual class models was tested on the remaining two classes. The resultant performance has been displayed in Figure S3 in the Supporting Information and the class features have also been listed in Table S2 there. In Table S3, we can see that all of the above-developed models using individual-class features were highly efficient in making predictions for the parent class, however, they were not sufficient for predictions on another class despite the inclusion of data of new class while retraining the models. Thus, the localized behavior of the class features impedes generalization to other classes.

Two-Class Combined and Blended Models

At the second stage, we mixed the data of two classes and performed feature selection to obtain combined class features. The first combination comprised classes P6̅m2 and P3m1 (Figure S4). The selected features of combined data set included H_e^mean, G^mean, r_atom, κ_l^mean, T_b, EA^std, G^std, r_atom^std, r_cov, E_CBM, and E_g^PBE. The hyperparameters for the best-optimized model were obtained for this set of features, and the resultant model gave an rmse and R² of 0.17/0.17 eV and 0.98/0.97, respectively, for the train/test sets. When this model was tested on the validation data of the two classes, it gave an rmse and R² of 0.20/0.18 eV and 0.99/0.87 for P6̅m2/P3m1 data as shown in Figure 3a. The prediction accuracy using these two-class combined features was better than the accuracies obtained for validation using any of the individual-class features. A comparison of P6̅m2 and P3m1 individual-class features, with these features revealed that other than a few common ones, the combined feature set comprised several additional features. The improvement in the performance of the model for both classes can be attributed to these additional descriptors that provided a better representation of the latent behavior of this mixed data.

Figure 3.

Scatter plots for 2 class (a) combined and (b) blended feature, and 3 class (c) combined and (d) blended features, respectively.

To harness the capabilities of both individual and combined class features, feature blending for the two classes was performed. For the two classes P6̅m2 and P3m1, when their individual features were simply appended to their combined feature set, 23 features were obtained, which were reduced using LASSO and NCA. The new set termed as two-class blended feature set consisted of 10 features including E_g^PBE, χ_p, κ_l^std, r_cov, C_g^std, H_e, T_b^mean, G^mean, r_atom, and T_b^std. The best model using these blended features gave an rmse and R² of 0.14/0.14 eV and 0.99/0.98 for combined train/test sets, respectively. Further, on the 10% validation data, the rmse and R² were 0.13/0.16 eV and 0.99/0.90 for P6̅m2/P3m1 data as shown in Figure 3b. This model with blended features gave a better accuracy than the models using individual or combined class features, thus improving generalizability. Two-class combined and blended features were next derived for pairs P6̅m2 and P3̅m1, as well as P3m1 and P3̅m1, and similar observations were made. The features and the results for these combinations have been tabulated in Tables S2 and S3 and displayed in Figure S4. The observation that generalizability has been achieved with improvement in prediction accuracy at the same time, can be credited to the fact that this feature set reflects a balanced blend of both localized and group behavior.

Three-Class Combined and Blended Models

The feature blending scheme was next extended to all three classes, that is, P6̅m2, P3m1, and P3̅m1. For this, we begin by generating the combined class features for all three class mixed data as done in the case of two classes. A set of 13 three-class combined features were obtained. This combined feature set consisted of H_e^mean, IE₁, χ_A^mean, T_m, κ_l^mean, C_g, EA^std, IE₁^std, m^std, r_atom, r_vdw^std, E_VBM and E_g. The best model using these combined features gave an rmse of 0.16/0.16 eV and R² of 0.99/0.99 for combined train/test data. For validation data, rmse of 0.20/0.14/0.22 eV and R² of 0.99/0.90/0.99 was obtained for classes P6̅m2/P3m1/P3̅m1, respectively, as shown in Figure 3c. This again was a significant improvement compared to the performance of individual class models on validation sets.

For obtaining blended features for any number of classes, all we need is two sets of individiual class features and a set of their combined features. Thus, a remarkable facet of the feature blending approach is that it can always be reduced to a two-class problem whenever a new class needs to be included. For the three-class scenerio, this was achieved by utilizing the 10 two-class blended features obtained in stage two as the first individual feature set (for P6̅m2 and P3m1 combined data) along with the seven individual-class features of the new incoming class P3̅m1 (obtained at stage one) as second individual feature set and finally the 13 three-class combined features. Feature reduction on this set gave eight features, namely, χ_P^std, T_b, r_cov^std, E_g, G^mean, r_atom^mean, H_e, and κ_l^mean. The unified model designed using these three-class blended features gave an rmse of 0.14/0.14 eV and R² of 0.99/0.99 for combined train/test data, respectively, whereas an unprecedented rmse of 0.14/0.09/0.13 eV and R² 0.99/0.96/0.99 were obtained for 10% validation data of P6̅m2/P3m1/P3̅m1, respectively, as shown in Figure 3d.

Similar calculations were performed by changing the sequence in which the classes are selected for inclusion. When we begin with P6̅m2 and P3̅m1 and consider P3m1 as the third incoming class, the three-class blended feature set obtained comprised T_b^mean, H_e, κ_l^mean, EA^std, IE_l, χ_P^std, m^std, E_VBM, and E_g. The model using these features gave an rmse of 0.12/0.14 eV and R² of 0.99/0.99 for combined train/test data, whereas an rmse of 0.13/0.14/0.12 eV and R² 0.99/0.91/0.99 were obtained for 10% validation data of P6̅m2/P3m1/P3̅m1, respectively. With P3̅m1 and P3m1 as the first two classes and P6̅m2 as the third class, the feature set T_b^mean, H_e, IE₁^mean, κ_l, C_g^std, χ_P, r_cov^std, and E_g resulted in an rmse of 0.14/0.15 eV and R² of 0.99/0.99 for combined train/test data and an rmse of 0.14/0.15/0.14 eV and R² of 0.99/0.90/0.99 for 10% validation data of P6̅m2/P3m1/P3̅m1, respectively. It was interesting to note that the order in which the classes were included for blending, did not have a major impact on the performance of the final unified three-class blended model. This was due to the fact that all three blended feature sets had majority features namely T_b^mean, H_e, κ_l^mean, χ_P, and E_g^PBE in common. The performance of the above-discussed models has been shown in Figure S5.

Two intriguing observations emerged as a result of the feature blending process. First, an analysis of the individual, combined, and blended models showed that the relevance of each feature of the blended feature set in the unified model, whether belonging to an individual or the combined feature set, was almost the same as its relevance in its parent model, that is, individual or combined class model. This was true for two as well as three-class models. This relevance or importance of the selected features in the regression model for the first set of three-class blended features, shown in Figure S6(a,b), can be seen to be comparable to those in their parent model. This conservation of relevance of each feature in the unified model plays a significant role in the consistent performance of blended features for all its participating classes.

Second, apart from promoting generalizability in prediction models, the blending process emanated a few features that recurred in almost all feature sets. Some of these features not only had high relevance in the statistical model, but they also displayed a notable relationship with each other and with the E_g^GW. Features, namely χ_P, T_b^mean, and E_g, present in all the three final three-class blended feature sets, were found to not only have high Pearson correlations with E_g^GW but also captured the right physics in the trends in the band gap. From Figure 4a two conclusions were drawn: first, increase/decrease in bandgap at PBE levels shows a similar shift at the GW level, and second, E_g varies with χ_P^std and T_b, that is, increase in the standard deviation of elemental electronegativity, and subsequent decrease in the mean elemental boiling temperature leads to increase in the bandgap. The increase in χ_P^std indicates a broader spread (larger difference) in the electronegativity value from the mean, hence, strong interaction among the atoms, such firm overlap causing a larger splitting and consequently a wider bandgap. Besides, the boiling temperature of elements with larger electronegativity (>2.6) is comparably low (<1000 K); however, for elements with smaller χ_P (<2.6), the boiling temperature varies (up to 5000 K). Therefore, materials with a larger χ_P usually have lower T_b^mean. Nevertheless, depending upon the constituent elements, the mean boiling temperature for a system could be higher with the larger value of χ_P. For example, a material with the combination of C and F would have both larger χ_P^std and T_b. Furthermore, other pairwise distribution plots of the individual class feature provide further physical insights between different features of the particular class. Additional discussion on this is included in the Supporting Information and pictorially explained in Figure S7.

Figure 4.

(a) The three-dimensional color map showing the variation of E_g^PBE with χ_P^std and T_b, and variation in their E_g^GW, shown by colorbar. (b) Pair distribution plot between χ_P and T_b^mean, with colored zone for different class of materials.

Moreover, the relationship between χ_P^std and T_b was also found to partition different classes of materials and can thus be exploited for expediting selection of 2D materials in general, having bandgap in the desired range as shown in Figure 4b. P3m1 lies at the top right with large electronegativity and boiling temperatures, P6̅m2 lies in the middle, followed by P3̅m1 at the bottom of the plot. The interpretation of these three relationships provided an opportunity to model an empirical relation between them for estimating the GW band gap. This relation was derived by generating a set of compound features by applying various mathematical operations on these three selected features and correlating them with the calculated GW bandgap.²⁶ Some of the mathematical operations used here were x², x³, 1/x, log (x), exp (x), etc., where x is any feature. As the dimensionality increases, x will include more than one feature. Combination of these mathematical operations on these three features gave an empirical formula represented as

1

The estimated value E_g^estimated using this empirical formula showed a strong correlation of 0.93 with E_g and the fitted linear equation results in a high coefficient of correlation R² of 0.83, as shown in Figure S8a. This equation was also utilized for estimating bandgaps of the 44 compounds belonging to space groups P1, Pm, P4/nmm, Pma2, Pmmn, Pmn21, and P3m1 from the CMR database that were neither sufficient to build an individual class model nor for feature blending. As shown in Figure S8b, the small residuals obtained exhibit the potential of this relationship in making predictions on unseen classes and corroborate the significance of the selection of the right features.

Conclusion

In summary, we attempted to extend the transferability of any pretrained machine learning model across different databases by introducing the concept of feature blending. Feature blending is shown to enable the selection of features with global relevance in contrast to local scope offered by individual class features, by iteratively mixing and selecting features obtained for various classes at different stages. Thorough validation of models at different stages of development has been performed and significance of judicious selection of features for a general-purpose machine learning framework has been instantiated. The reported unified model developed for bandgap prediction, has been shown to perform on previously unseen data from all the participating classes with comparable accuracy. The most prominent outcome of the scheme was identification of certain universal features which exhibited vital influence on the band gap of all the participating classes in a similar manner. A detailed study of physical relationships between these cross cutting features made it possible to develop a generic empirical relation for the approximation of E_g^GW for 2D materials. This developed equation using the three important features was found to be applicable on all 9 classes of 2D materials available in the initial data set with reasonable accuracy. Although classes in this work were formed based on space groups, classification can be done based on any other parameter. The applicability of the proposed feature blending approach can be seemingly extended to large classes of materials.

Supporting Information Available

The Supporting Information is available free of charge at https://pubs.acs.org/doi/10.1021/acsphyschemau.1c00017.

• Additional information including features selection techniques, machine learning methods, feature selection using P3m1, P3̅m1, and two/three-class combined and blended features. Performance test for two-class, mixed class, feature relevance, empirical relation and physical significance of features. (PDF)

Author Contributions

^‡ S.S. and A.M. contributed equally.

The authors declare no competing financial interest.