Skip to main content
Journal of Cheminformatics logoLink to Journal of Cheminformatics
. 2025 May 22;17:80. doi: 10.1186/s13321-025-01004-5

Benchmarking molecular conformer augmentation with context-enriched training: graph-based transformer versus GNN models

Cecile Valsecchi 1,✉, Jose A Arjona-Medina 2, Natalia Dyubankova 3,✉, Ramil Nugmanov 3
PMCID: PMC12100853  PMID: 40405272

Abstract

The field of molecular representation has witnessed a shift towards models trained on molecular structures represented by strings or graphs, with chemical information encoded in nodes and bonds. Graph-based representations offer a more realistic depiction and support 3D geometry and conformer-based augmentation. Graph Neural Networks (GNNs) and Graph-based Transformer models (GTs) represent two paradigms in this field, with GT models emerging as a flexible alternative. In this study, we compare the performance of GT models against GNN models on three datasets. We explore the impact of training procedures, including context-enriched training through pretraining on quantum mechanical atomic-level properties and auxiliary task training. Our analysis focuses on sterimol parameters estimation, binding energy estimation, and generalization performance for transition metal complexes. We find that GT models with context-enriched training provide on par results compared to GNN models, with the added advantages of speed and flexibility. Our findings highlight the potential of GT models as a valid alternative for molecular representation learning tasks.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13321-025-01004-5.

Keywords: Drug discovery, QSAR, Cheminformatics, Graph, Transformers, Deep learning, Pretraining

Scientific Contribution

This study advances the understanding of molecular representation by providing a comparative analysis of Graph-based Neural Networks with a focus on Graph-based 2D, 3D and 4D Transformer models. In addition, we identify the benefits of context-enriched training in Graph-based Transformers.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13321-025-01004-5.

Introduction

In recent years, there has been a progressive shift in the focus of molecular representation learning. Traditional molecular descriptors are increasingly being replaced by patterns identified by models trained exclusively on molecular structures [1, 2]. These models exploit representations in form of strings (such as SMILES) or graphs [3, 4].

Molecular string-based representations are less close to the actual molecular structures and are often referred as 1D representations, they allow for effective data augmentation techniques, which are useful in low-data regimes [5, 6]. Widely used 2D molecular descriptors such as Extended Connectivity Fingerprints (ECFPs), might miss deeper interactions and the spatial arrangement of atoms and bonds [7]. On the other hand, graph-based representations provide a seemingly more realistic depiction of molecules [3].

While chemical language models have been successful in various applications, this study will focus exclusively on graph-based methods [4], while ECFPs will be used as baseline representations.

Graphs encode basic chemical information in nodes (e.g., atomic number, chirality, charge, hybridization) and in bonds (e.g., type of bond). Some models even integrate these graphs with properties based on Density Functional Theory (DFT)[8, 9].

Graph-based representations can be categorized into two types: 2D-based, which support limited augmentation options such as contrastive learning methods, and 3D-based, which allow for conformer-based augmentation [10–12]. Leveraging the collective power of many different conformer structures has proven to be beneficial in some cases[10], while limited improvements have been made in addressing chirality, despite advancements in application of chirality-aware graph-based deep learning models like ChIRo [13] and ChIENN [14].

Graph Neural Networks (GNNs) and Graph-based Transformer models (GTs) are two emerging paradigms in the field of graph-based modeling [15]. Recent applications of GNNs highlight their versatility, including the integration of conformer ensembles [10, 16] and the utilization of resource-intensive density functional theory (DFT)-based features [8] as well as properties derived from docking studies [17]. These advancements illustrate the novel capabilities of GNN models [18].

While GNNs have been the conventional approach, GT models, such as Graphormer [19] and Transformer-M [20], have emerged as valid and flexible alternatives. This is evidenced by the dominance of GT models in the leaderboard of the 2022 Open Graph Benchmark (OGB) challenge [21–24], with notable examples including UniMol [22], GPS [23] and Equiformer [24]. GTs provide flexibility through their easier implementation and customizable input, enabling them to handle various data formats in a multimodal manner [25]. Furthermore, GT architectures have proven effective across different data modalities, particularly in managing both 2D and 3D molecular structures [19, 20, 26].

However, the increasing complexity of these models and the proliferation of variants for each architecture complicate the identification of genuine improvements across diverse datasets and chemical spaces [21, 22, 24].

In addition, the training procedure-comprising types of losses, loss modification (e.g., to handle qualifiers), prompting, and pretraining on relevant chemical properties-can impact performance. These methods allow to introduce domain-specific knowledge and biases that often have a more influence than the complexity of the architecture itself [25, 27, 28]. Therefore, we attempt to balance between model complexity and effective training strategies.

This work aims to contribute to this ongoing discourse by providing insights into the performance of simpler, flexible GT models compared to traditional GNNs, including 2D, 3D and conformer ensemble (4D) representations. In the GT landscape, we limited the analysis to a Graphormer-based architecture [29, 30].

Methods

In this section, we present the architectures, methods, and datasets used in our comparative study of GT and GNN models. We detail the key features of the models, covering both 2D and 3D-based approaches, and provide an overview of the datasets, which include a variety of chemical structures.

Selected modeling approaches

Table 2 and Fig. 1 summarize characteristics of the Graph Neural Networks (GNN) and Generative Transformer (GT) models employed in this study. To enhance comparability, we set the hidden dimension of the embeddings to a uniform size of 128. We also explored variations by increasing this value to 256 and decreasing it to 64; the results can be found in the Appendix B (Fig. 6). GNN As GNN representative methods, we selected two 2D-based models (ChemProp and GIN-VN) and two 3D-based models (SchNet, PaiNN). The selection was made in line with previous studies and as a trade-off between performance and implementation easiness [10].

  • Chemprop consists of four modules: (1) a local feature encoding function, (2) a directed message passing neural network to learn atomic embeddings from the local features, (3) an aggregation function to join atomic embeddings into molecular embeddings, and (4) a standard feed-forward neural network for the transformation of molecular embeddings to target properties [31].

  • Graph Isomorphism Networks (GIN) is a message passing neural network created to generalize the Weisfeiler-Lehman graph isomorphism test[32]. GIN with Virtual Node (GIN-VN) augments the vanilla GIN by incorporating a virtual node to aggregate the features of all nodes in the graph.

  • Chiral InterRoto-Invariant Neural Network (ChIRo) is GNN model that processes torsion angles of a 3D molecular conformers[13]. It includes 2D message passing phase to embed node states based on graph connectivity, two feed-forward networks to encode 3D bond distances and bond angles, a torsion encoder to explicitly encode chirality with invariance to internal bond rotations and a readout phase for property estimation.

  • Polarizable Atom Interaction Neural Network (PaiNN) is a rotationally equivariant message passing neural network exploiting 3D geometries [33].

  • SchNet is a message passing neural network for learning properties of 3D conformers, updates node states using messages informed by radial basis function expansions of interatomic distances between neighboring nodes through continuous filter convolutional layers [34].

GT As GT representative method, the implementation used in this study is based on the Graphormer[35] architecture (https://github.com/chython/chytorch).

  • 2D-GT For each structure, the input consists of a vector of heavy atom types and a vector of neighbor counts, both of the same length. Additionally, topological distances, represented as shortest distance paths, are included. The architecture is highly flexible and allows for the incorporation of additional atom information, such as chirality tags, or prompt encoders to enable multimodality. This flexibility also enables training with auxiliary tasks by incorporating additional input vectors or switching from molecular properties to atomic properties. In the architecture, the topological distances are used in a multi-head bias, similar to the original Graphormer implementation. These distances are added to the atom features in a distance-biased dot-product attention layer, as illustrated in Fig. 1. After normalization, a final feedforward layer is included in the architecture.

  • 3D-GT is a variant of the 2D version that considers binned distances instead of shortest-path distances. The binning of distances is customizable, but in this study, a shortest possible distance of 0.9 Angstrom, a visible neighbor sphere radius of 5 Angstrom, and a precision of 0.5 Angstrom were used (see example in SI Fig. 5). This 3D approach provides more granularity in distances compared to the 2D version, allowing for the differentiation of different types of interactions. However, this increased granularity can also introduce noise depending on the task and data.

Table 2.

Summary statistics of the considered models

Type Name Params Features Avg Train/inference time (s)
2D ChemProp 106,369 9 atom, 3 bond 21.5/2.3
2D Gin-VN 240,769 9 atom, 3 bond 16.2/2.4
2D GT 1,608,544 2 atom, distances 3.7/0.4
3D ChIRo 834,436 9 atom, 3 bond, angles 49.1/6.9
3D PaiNN 1,244,161 1 atom, distances 20.7/3.9
3D SchNet 149,167 1 atom, distances 15.9/3.1
3D GT 1,608,544 2 atom, distances 3.9/0.4
4D PaiNN 1,244,288 1 atom, distances 147.1 / 31.3
4D SchNet 149,294 1 atom, distances 99.7 / 24.4
4D GT 1,608,544 2 atom, distances 22.0/2.7

The smallest parameter sizes and the lowest average training times for each type (2D, 3D and 4D) are highlighted in bold

The time reported is the average time for one epoch over the three datasets

Fig. 1.

Fig. 1

An illustration of the graph-based transformer (GT) and general graph neural network (GNN) architectures and context-enriched training and conformer-ensemble (4D) methods for the considered three datasets: BDE, Kraken and tmQMg

Fig. 6.

Fig. 6

Barplot of R2 score on the test set varying the hidden dimension of embeddings

Fig. 5.

Fig. 5

Comparison of 2D distance (shortest path) and 3D binned distance as input to GT models

Baseline In addition to the methods described above, we also include eXtreme Gradient Boosting (XGBoost) operating on molecular fingerprints as a baseline for our experiments. For the baseline model, each structure was represented by a binary vector composed of concatenated fingerprints. We used ECFPs with 3072 bits and a radius of 3, RDKit topological fingerprints with a maximum of 5 bonds, and MACCS keys. For the baseline models we optimized in cross-validation parameters such as the number of estimators, the depth and learning rate.

Datasets and tasks

We exploited three small-to-large scale datasets involving eight regression tasks. Those datasets include molecules with high structural flexibility, evidenced by an average number of rotatable bonds exceeding 6 and a molecular weight of 538 Da (more details provided in SI Fig. 4).

  • BDE [10, 36]is a dataset focused on reaction-centric properties. It contains 5,915 organometallic catalysts ML1L2 consisting of a metal center (M = Pd, Pt, Au, Ag, Cu, Ni) coordinated to two flexible organic ligands (L1 and L2), each selected from a 91-membered ligand library. The data includes conformations of each unbound catalyst, as well as conformations of the catalyst when bound to ethylene and bromide after oxidative addition with vinyl bromide. The task to be modeled is the binding energy from the graphs of the unbound and bound catalyst [36].

  • Kraken[37]is a dataset of 1,552 monodentate organophosphorus (III) ligands along with their DFT-computed conformer ensembles. As in [10], we consider four 3D ligand descriptors exhibiting significant variance among conformers: Sterimol B5, Sterimol L, buried Sterimol B5, and buried Sterimol L. The goal is to estimate the Boltzmann-averaged value of each property across the conformer ensemble from the molecular graph, a single conformer, or the set of conformers.

  • tmQMg [8] is a dataset of more than 60,000 transition-metal complexes (TMCs). TMCs are more difficult to express as graphs due to the metal d orbitals, which yield larger valences and multi-center bonds. In this context, the representation of a TMC can become ambiguous, with multiple possible graphs of different topology. This may include disconnected graphs limiting the applicability of GNNs. Natural bond orbital theory[38] and second-order perturbation analysis[38] were used to define the topology. We consider four quantum properties: HOMO-LUMO gap, Gibbs energy, dipole moment and polarizability. The goal is to estimate each property from a single conformer. Graphs that were informed with classical descriptors.

Fig. 4.

Fig. 4

Boxplot of molecular weight (MW) and rotatable bond count (RB) for the three considered datasets and histograms of the rescaled eight considered tasks

Experimental setup

This section outlines the experimental setup used to evaluate the performance of simpler representations and flexible GT and traditional GNN models. We detail the processes of dataset compilation and curation, the representation of molecular structures, and the training approaches employed. Additionally, we describe the use of context-enriched training strategies, including pretraining and multitask learning, to enhance the models’ generalization capabilities. These setups ensure a robust and fair comparison of the models across various datasets.

Dataset compilation and curation

For 2D GNN modeling, molecules were represented as graphs, where atoms were encoded as nodes with nine features, and bonds were encoded as edges with three features. Both node and edge features were classical descriptors easily calculated through RDKit [39], including atomic numbers, chirality tags, formal charges, number of directly-bonded neighbors, number of implicit hydrogens, number of radical electrons, hybridization, aromaticity, ring connection, bond type, stereo configuration, and conjugated bonds. For the ChIRo model, additional descriptors such as dihedral angles were also computed. The 3D GNN and GT models relied solely on atomic numbers, hydrogen count, and distances as input representations.

To guarantee homogeneity and fair comparison we used the same training/validation/test split for all tested models and we ran three independent trainings with three different seeds. We pruned structures with more than one atom at the same 3D coordinates, and structures with less than 3 atoms. The final data sets are summarized in Table 1

Table 1.

Summary statistics of the considered datasets (training/validation/test sets)

Dataset Task Molecules Conformers Additional atomic species
Kraken 4 1096/156/286 11244/1490/2997 B, Fe, I, S, Se, Si, Sn
tmQMg 3 27032/3859/7881 27032/3859/7881 Sc, Rh, Re, Au, Hf, B, Fe, Se,
Pt, Ir, Zr, Si, Ni, Os, Cd, La,
Ru, Ta, Ag, Cr, V, I, Cu, Pd,
Hg, Zn, Mn, Mo, As, Tc, Co,
Y, S, Nb, Ti, W
BDE 1 4144/640/1131 71384/10784/20369 Ag, Au, Cu, Ni, Pd, Pt

Column named Additional atomic species refers to elements in addition to the common set: Br, C, Cl, F, H, N, O, P

Conformer ensembles

For both 3D GNN and 3D GT models, two training approaches were employed: training on a single conformer per structure (3D) and training on a pool of conformers per structure (4D).

In the 4D training approach, multiple conformers from the original data sources were used for each structure. Conformer embeddings were generated for each conformer, and a maximum pooling operation was applied to aggregate the conformer-level information into a single representation. This pooling operation retains the most prominent features across the conformers, enabling the model to capture conformer-specific patterns that may influence overall molecular properties.

By training on either individual conformer representations (3D) or aggregated conformer representations (4D), the models can capture the inherent features of individual conformers as well as the collective properties of all conformers within a structure.

Training details

We explored the impact of context-enriched training strategies on the performance of GT models, focusing specifically on the benefits of pretraining and training with auxiliary tasks.

For pretraining, we utilized a freely available quantum mechanical (QM) dataset [40], which consists of 136,209 small molecules and provides atom-level properties such as NMR shielding constants, Fukui electrophilic and nucleophilic indices, and partial charges. This dataset allowed the GT models to learn representations that capture key features of molecular structures before being fine-tuned on the target tasks [25].

For datasets containing multiple tasks - Kraken and tmQMg - we implemented auxiliary task training. In this multitask approach, the GT models were trained with different tokens representing each task, a technique known as prompting. The final feedforward layer of the models included additional parameters to learn token-specific scaling, enabling the models to effectively learn and differentiate between the tasks simultaneously. Multitask learning involves the simultaneous learning of multiple related responses or tasks [41]. It is anticipated that these tasks will share information during this parallel tuning process, allowing the modeling of one specific task to enhance the performance of others. However, the extent of improvement depends on the degree of relatedness among the tasks. Some tasks can potentially introduce noise into the training process.

Regarding the tmQMg dataset, which includes only single-conformer structures, we did not train 4D models. The 4D models are specifically designed to incorporate multiple conformers per structure, which is not applicable to single-conformer datasets. By incorporating these context-enriched training strategies, we aimed to evaluate how additional information and prior knowledge could enhance the performance and generalization capabilities of the GT models.

Results

Our results encompass an evaluation of the efficiency of GT models against GNNs, examining their computational speed, reproducibility, and the impact of 2D, 3D, and 4D molecular representations. Additionally, we explored the benefits of context-enriched training and assess the uncertainty associated with each model’s outcome.

Results on inference

Although the GT implementation has a higher number of trainable parameters compared to the GNN models, it is five times faster on average for both training and inference on a single GPU NVIDIA A10G AWS instance (see Table 2). Despite the GT models having a greater number of parameters than the GNN models, no significant difference in overfitting was observed. More results in the AppendixD, Fig. 7.

Fig. 7.

Fig. 7

The extent of overfitting was assessed by measuring the difference in performance between the training and test sets. The boxplot illustrates the distribution of this difference, represented in terms of the R2 score, categorized by model dimension and type

Reproducibility

We evaluated the reproducibility of our results by comparing the performance of our models with those reported in previous studies on the Kraken and BDE datasets [10]. Despite potential differences in the training/validation/test splits, we found that our overall performance was reasonably comparable to the literature. For instance, the ChemProp model exhibited lower mean absolute percentage error (MAPE) across all tasks in our study, while PaiNN 3D showed improved performance on the BDE dataset. However, some models, such as PaiNN 2D and SchNet 2D, performed better in the previous study. Additionally, we compared our results on the tmQMg dataset with those obtained in the original study [8]. The use of classical descriptors led to worse performance compared to the DFT-based descriptors used in the original work.

Baseline vs GNN vs GT performance

The baseline XGBoost model was effective for the buried L task, achieving the second best performance after PaiNN 4D considering the R2 score and the best performance considering MAPE results on the test set. In several tasks, the XGBoost model achieved similar performance to the best performing model in terms of MAPE, e.g., tmQMg dipole moment and homo lumo gap and Kraken sterimol burB5.

2D vs 3D vs 4D representation

For the Kraken and BDE tasks, transitioning from 2D to 3D and from 3D to 4D geometries in GT models did not result in significant degradation in performance, with average R2 score improvements of +0.11 and +0.04, respectively, on the test set (see Fig. 2 and Table 3). The MAPE remained comparable or lower in the 4D GT models compared to the 3D GT models. Conversely, in the tmQMg tasks, the inclusion of 3D information led to a degradation in performance, indicating that in these tasks, the 2D representation was less noisy.

Fig. 2.

Fig. 2

Performance (R2 score on test set) for each modeling strategy for Kraken sterimol B5 task and aggregated results for all tasks

Table 3.

Performance (R2 score/MAPE on test set, MAPE the lower the better, R2 score the higher, the better) of 2D, 3D and 4D (conformer ensemble) GNN and GT modeling strategies on the three considered datasets and eight tasks

Type Model Kraken tmQMg BDE
B5 L BurB5 BurL HLG Gibbs DM
2D XGBoost 0.80/1.3 0.68/2.0 0.71/1.0 0.66/1.4 0.68/2.6 0.89/1.3 0.50/3.3 0.96/.5
ChemProp 0.85/.9 .71/2.1 0.75/1.5 0.51/3.2 0.69/2.5 0.88/1.5 0.43/3.3 0.93/.5
GIN-VN 0.86/.9 0.65/1.6 0.73/1.2 0.54/3.8 0.71/2.6 0.98/.3 0.50/4.0 0.88/.5
GT 0.82/1.2 0.43/2.1 0.68/2.3 0.18/2.7 0.68/2.7 0.99/0.1 0.40/3.9 0.96/.4
GT-m 0.78/1.9 0.53/2.1 0.62/1.6 0.25/2.5 0.64/3.2 0.99/.2 0.39/3.5 −
GT-pre 0.87/1.0 0.60/2.3 0.71/1.7 0.49/5.7 .75/2.2 0.99/0.1 0.53/3.3 0.97/.4
GT-pre-m 0.85/1.2 0.53/2.4 0.70/1.1 0.52/3.7 0.72/2.5 0.99/0.1 0.51/3.2 −
3D ChIRo 0.69/1.4 0.60/1.8 0.63/1.7 0.45/2.6 0.63/2.8 0.98/0.5 0.37/3.8 0.94/.5
PaiNN 0.85/1.0 0.72/2.1 0.76/1.0 0.1/7.5 0.53/3.7 1.0/.0 0.29/5.5 0.98/.3
SchNet 0.73/1.5 0.56/2.0 0.66/1.8 0.39/2.0 0.53/3.2 0.99/0.2 0.25/4.6 0.96/.5
GT 0.84/1.0 0.61/1.9 0.71/1.6 0.41/2.5 0.69/3.0 0.99/0.1 0.39/3.9 0.97/.4
GT-m 0.86/1.1 0.67/2.0 0.74/1.3 0.47/5.6 0.66/3.3 0.99/0.2 0.36/4.8 −
GT-pre 0.89/.9 0.72/1.7 0.77/1.1 0.58/2.1 0.73/2.7 0.99/0.1 0.47/4.1 0.98/.3
GT-pre-m 0.88/.7 0.70/2.3 0.77/1.0 0.58/2.5 0.69/2.9 0.99/0.2 0.42/4.4 −
4D PaiNN 0.91/.7 0.80/1.2 0.84/.9 0.69/1.9 – – – 0.98/.3
SchNet 0.83/1.3 0.59/2.4 0.69/1.8 0.45/2.5 – – – 0.97/.4
GT 0.89/.7 0.68/1.5 0.74/1.2 0.46/3.4 – – – 0.97/.4
GT-m 0.87/.8 0.66/1.8 0.76/1.1 0.43/2.8 – - – −
GT-pre 0.87/.8 0.75/1.8 0.78/.9 0.59/4.4 – – – 0.97/.3
GT-pre-m 0.90/.7 0.73/1.8 0.77/1.0 0.59/4.0 – – − −

The model with the highest score for each type and dataset is highlighted in bold, while the best model for each dataset is highlighted in italics

In the case of GNN models, 3D PaiNN and SchNet, on average, performed worse than their 2D counterparts GIN-VN and ChemProp by -0.7 in R2 score on the test set. However, PaiNN models performed well when considering multiple conformers (4D), making them the best modeling strategy for four out of five tasks where multiple conformers were available. For the BDE dataset, most models showed a degradation in results when using 4D geometries, except for SchNet, which showed improvements in both R2 score (+0.01) and MAPE (-0.32) on the test set.

GNN vs GT

For the tmQMg tasks, 3D GT models generally outperformed 3D GNN models in terms of R2 score, with the exception of the Gibbs energy task, where PaiNN 3D was the top performer. Notably, the MAPE values for the GNN and GT models are comparable. For the BDE and Kraken tasks, except for the sterimol L and burL tasks, GT models performed slightly worse but were still comparable to GNN models.

Context-Enriched GT

Pretraining on QM properties was beneficial for all tasks and input types, particularly for 2D GT models, resulting in an average R2 score improvement of +0.11 on the test set. Before pretraining, GNN models significantly outperformed GT models trained from scratch in certain tasks. However, with QM pretraining, GT models performed similarly to or better than GNN models (see Fig. 2 and Table 3). In AppendixE Fig. 8 we evaluated the effect of using only a random half of the pretraining QM data. Multitasking was not always be beneficial.

Fig. 8.

Fig. 8

We compared the R2 scores on the test set after pretraining the models using only a random half of the QM dataset. Although the effect is not substantial, training on the entire dataset generally results in improved performance on average

Results on uncertainty

We used the standard deviation of estimations over three independent runs as a measure of uncertainty (see Fig. 3). The baseline XGBoost models had an average uncertainty comparable to GT and GNN models except for Kraken sterimol burL and tmQMg dipole moment tasks, where the baseline uncertainty is the lowest and for tmQMg Gibbs energy where XGBoost is less consistent than graph-based models. The SchNet model with conformer ensembles (4D SchNet) showed the highest or second-highest interquartile range of uncertainty in all tasks that included multiple conformers. The context-enriched GT models generally had lower uncertainty compared to the corresponding GT models trained from scratch, but not in all cases.

Fig. 3.

Fig. 3

Distribution of uncertainty across models and across tasks. The standard deviation over three independent runs is considered as a proxy of uncertainty. For each task, models are sorted by descending MAPE

Discussion

In this section, we discuss the key findings from our study, analyzing the model efficiency, reproducibility, and the impact of different geometric representations and context-enriched training on performance. We aim to highlight the strengths and limitations of the different modeling strategies.

Inference

The efficiency of the GT implementation, as shown in Table 2, is attributed to specific architectural and design choices that optimize computation and memory usage. These optimizations enable faster training and inference times, even with a higher number of parameters compared to other models. This highlights the importance of thoughtful model design in achieving both high performance and computational efficiency.

Reproducibility and benchmark comparisons

Our analysis showed that the overall performance of our models was comparable to those reported in the literature [5], particularly for the Kraken and BDE datasets [10]. However, slight variations in performance were observed, which are attributed to differences in the specific training/validation/test splits used across studies. The observed discrepancies are related to the sensitivity of model performance to dataset splits and indicate that only standardized datasets enhance consistent comparisons across studies. To further support reproducibility, we share the train/validation/test splits used in our experiments.

In the case of the tmQMg dataset, our results were worse than those reported in the original study [8] due to the use of classical descriptors instead of DFT-based descriptors. We did not incorporate QM properties as node and edge features in our GNN models. The decision to use the same RDKit-calculated input features across all GNN models in our study aimed to maintain consistency but resulted in a performance trade-off. This performance decrease emphasizes the value of incorporating quantum mechanical properties as input features when available, as they can significantly enhance model accuracy.

Impact of 2D, 3D, and 4D representations

For Kraken and BDE tasks, GT models are robust across different geometric representations. However, the lack of substantial improvement with 4D geometries indicates that while conformer ensembles can be beneficial, their impact is task-dependent. For instance, in the BDE task, SchNet benefitted from the use of 4D geometries, but this was not a universal trend across all models and tasks.

In contrast, the tmQMg tasks revealed a different pattern, where the inclusion of 3D information actually degraded performance compared to 2D representations. This observation indicates that the 3D information in these tasks introduces noise rather than clarity, due to the limited quality or relevance of single-conformer data.

The consistent improvement seen with 4D GNN models, particularly for the PaiNN model, is explained by the ability of a set of conformers to capture more comprehensive chemical patterns that single-conformer representations may miss. However, the improvement is also attributed to the different GNN architectures used in 2D and 3D/4D settings.

Baseline vs GNN vs GT performance

While XGBoost models were effective for specific tasks like the buried L task, their overall performance was less consistent across other tasks. GT models generally performed comparably to or slightly worse than GNN models in tasks like BDE and Kraken, except for a few specific cases. This indicates that while GT models are highly flexible, their performance does not always surpass that of GNN models without further optimization.

Notably, the 3D GT models outperformed 3D GNN models in most tmQMg tasks, except for the Gibbs energy task where PaiNN 3D excelled. This highlights the potential of GT models to effectively handle complex 3D information.

Benefits of context-enriched training

The benefits of pretraining on QM properties were evident across all tasks and input types, particularly for 2D GT models. This pretraining allowed GT models, which were previously outperformed by GNN models in certain tasks, to close the performance gap or even exceed GNN performance (see Fig. 2B). The success of pretraining indicates that incorporating domain-specific knowledge into model training is beneficial, as it helps the model develop a deeper understanding of the underlying molecular structures.

However, the benefits of multitasking were less consistent. While training all tasks simultaneously for the same dataset showed advantages in a few cases (see Fig. 2A), the assumption that related properties should be modeled together did not universally hold true. This indicates that while multitasking can be beneficial, it requires careful consideration of the relationships between the tasks to avoid introducing unnecessary complexity or noise into the model training process.

Uncertainty and model robustness

The analysis of uncertainty revealed interesting insights into model robustness. While lower uncertainty generally correlated with better performance, this was not always the case (see Fig. 3). For instance, the SchNet model with conformer ensembles showed high uncertainty, indicating difficulties in convergence, yet it performed well in certain tasks. This demonstrates that uncertainty measures, while useful, are not always be directly indicative of model performance. Additionally, the observation that context-enriched GT models did not consistently show lower uncertainty compared to models trained from scratch demonstrating that initializing models with the same weights does not guarantee convergence to similar solutions, underscoring the complexity of model training dynamics.

Overall, these findings reinforce the importance of careful model selection and training strategy design, particularly when dealing with complex datasets and tasks. The results demonstrate that while advanced models and training techniques can offer performance improvements, they must be applied judiciously to ensure that their potential benefits are fully realized.

Conclusion

The main takeaways of this study are consistent with previous research [5], which suggests that simpler 2D models, which do not require 3D coordinates, can yield comparable results. This is evident in our study for the dipole moment and HOMO-LUMO gap for the tmQMg dataset. Additionally, the baseline XGBoost model delivered overall competitive results, outperforming all the neural networks in the sterimol BurL task.

Our results show that 2D GNN architectures such as ChemProp and GIN-VN often outperform GNNs like PaiNN and SchNet that are trained on a single conformer per structure. Thus, the inclusion of additional node and bond properties can compensate for or improve upon the introduction of 3D geometry information. However, when multiple conformers are provided for the same structure (conformer ensemble), the estimation capabilities of PaiNN and SchNet improve, demonstrating the importance of conformer ensembles in capturing more comprehensive chemical information. This points to the value of leveraging 3D geometry when multiple conformers are available, particularly for tasks that require a nuanced understanding of molecular structures.

GT models trained with context-enriched training consistently provided comparable or even better results compared to the best GNN models, with the added advantage of requiring less training and inference time. The overall trend for GT models indicates a benefit from considering 3D geometry and conformer ensembles, as well as from context-enriched training, particularly through pretraining on quantum mechanical properties.

GT models can be considered as a valid alternative to GNN models, offering the advantages of being faster, more flexible, and customizable to the specific task.

Acknowledgements

The authors thank Aline Hartgers for Fig. 1.

Appendix A: Datasets and tasks

See Figs. 4 and 5.

Appendix B: Effect of hidden embedding size on performance

See Fig. 6.

Appendix C: Hyperparameter optimization for baseline models

Hyperparameter optimization was conducted for the XGBoost regression models, employing the RandomizedSearchCV method from the scikit-learn library. A parameter grid was defined, which included variations for key hyperparameters: the number of boosting rounds set to either 250 or 500, maximum tree depth options of 3, 6, or 10, learning rates of 0.01 or 0.1, subsample ratios of 0.8 or 1.0, and feature sampling rates of 0.8 or 1.0. The optimization process utilized 5 iterations of random sampling for hyperparameter combinations, evaluated through 3-fold cross-validation, and aimed to minimize the negative mean squared error as the scoring metric. The best hyperparameters were subsequently identified and employed to fit the final model, enabling effective predictions on both the validation and test datasets. The performance of the optimized model was quantified using mean absolute percentage error (MAPE) and the coefficient of determination (R2) on the respective datasets.

Appendix D: Overfitting assessment

See Fig. 7.

Appendix E: Pretraining data size assessment

See Fig. 8.

Supplemementary Information

Supplementary material 1. (59.8KB, xlsx)

Author contributions

C.V.: Conceptualization; writing-original draft; formal analysis; writing-review and editing. J.A.M.: Methodology; Writing-original draft. N.D.: Conceptualization; writing-review and editing. R.N.: Conceptualization; Methodology; Writing-review and editing.

Data availability

No datasets were generated or analysed during the current study.

Footnotes

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Contributor Information

Cecile Valsecchi, Email: cvalsecc@its.jnj.com.

Natalia Dyubankova, Email: ndyubank@its.jnj.com.

References

  • 1.Chuang KV, Gunsalus LM, Keiser MJ (2020) Learning molecular representations for medicinal chemistry: miniperspective. J Med Chem 63(16):8705–8722 [DOI] [PubMed] [Google Scholar]
  • 2.Tu Z, Stuyver T, Coley CW (2023) Predictive chemistry: machine learning for reaction deployment, reaction development, and reaction discovery. Chem Sci 14(2):226–244 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Atz K, Grisoni F, Schneider G (2021) Geometric deep learning on molecular representations. Nature Machine Intell 3(12):1023–1032 [Google Scholar]
  • 4.Grisoni F (2023) Chemical language models for de novo drug design: challenges and opportunities. Current Opinion Struct Biol 79:102527 [DOI] [PubMed] [Google Scholar]
  • 5.van Tilborg D, Brinkmann H, Criscuolo E, Rossen L, özçelik R, Grisoni F, (2024) Deep learning for low-data drug discovery: hurdles and opportunities. Current Opinion Struct Biol 86:102818. 10.1016/j.sbi.2024.102818 [DOI] [PubMed] [Google Scholar]
  • 6.Bjerrum EJ, Sattarov B (2018) Improving chemical autoencoder latent space and molecular de novo generation diversity with heteroencoders. Biomolecules 8(4):131 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Rogers D, Hahn M (2010) Extended-connectivity fingerprints. J Chem Inform Model 50(5):742–754. 10.1021/ci100050t. (PMID: 20426451) [DOI] [PubMed] [Google Scholar]
  • 8.Kneiding H, Lukin R, Lang L, Reine S, Pedersen TB, De Bin R et al (2023) Deep learning metal complex properties with natural quantum graphs. Digital Disc 2(3):618–633 [Google Scholar]
  • 9.Haas B, Hardy M, SV SS, Adams K, Coley C, Paton R, et al (2024) Rapid Prediction of Conformationally-Dependent DFT-Level Descriptors using Graph Neural Networks for Carboxylic Acids and Alkyl Amines. ChemRxiv. 10.26434/chemrxiv-2024-m5bpn [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Zhu Y, Hwang J, Adams K, Liu Z, Nan B, Stenfors B, et al (2023) Learning Over Molecular Conformer Ensembles: Datasets and Benchmarks. In: The Twelfth International Conference on Learning Representations
  • 11.Zankov D, Madzhidov T, Varnek A, Polishchuk P (2024) Chemical complexity challenge: Is multi-instance machine learning a solution? Wiley interdisciplinary reviews. Computat Mol Sci 14(1):e1698 [Google Scholar]
  • 12.Wang Y, Wang J, Cao Z, Barati Farimani A (2022) Molecular contrastive learning of representations via graph neural networks. Nature Machine Intell 4(3):279–287 [Google Scholar]
  • 13.Adams K, Pattanaik L, Coley CW (2021) Learning 3d representations of molecular chirality with invariance to bond rotations. arXiv preprint arXiv:2110.04383
  • 14.Gaiński P, Koziarski M, Tabor J, Śmieja M (2023) ChiENN: Embracing Molecular Chirality with Graph Neural Networks. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer; p. 36–52
  • 15.Isert C, Atz K, Schneider G (2023) Structure-based drug design with geometric deep learning. Current Opin Struct Biol 79:102548 [DOI] [PubMed] [Google Scholar]
  • 16.Zankov DV, Matveieva M, Nikonenko AV, Nugmanov RI, Baskin II, Varnek A et al (2021) QSAR modeling based on conformation ensembles using a multi-instance learning approach. J Chem Inform Model 61(10):4913–4923 [DOI] [PubMed] [Google Scholar]
  • 17.Kyaw Zin PP, Borrel A, Fourches D (2020) Benchmarking 2D/3D/MD-QSAR models for Imatinib derivatives: how far can we predict? J Chem Inform Model 60(7):3342–3360 [DOI] [PubMed] [Google Scholar]
  • 18.Liu L, He D, Ye X, Zhang S, Zhang X, Zhou J, et al (2023) Pre-Training on Large-Scale Generated Docking Conformations with HelixDock to Unlock the Potential of Protein-ligand Structure Prediction Models. arXiv preprint arXiv:2310.13913
  • 19.Ying C, Cai T, Luo S, Zheng S, Ke G, He D et al (2021) Do transformers really perform badly for graph representation? Adv Neural Inform Process Syst 34:28877–28888 [Google Scholar]
  • 20.Luo S, Chen T, Xu Y, Zheng S, Liu TY, Wang L, et al (2022) One transformer can understand both 2d & 3d molecular data. In: The Eleventh International Conference on Learning Representations
  • 21.Hussain MS, Zaki MJ, Subramanian D (2024) Triplet Interaction Improves Graph Transformers: Accurate Molecular Graph Learning with Triplet Graph Transformers. arXiv preprint arXiv:2402.04538
  • 22.Lu S, Gao Z, He D, Zhang L, Ke G (2023) Highly accurate quantum chemical property prediction with uni-mol+. arXiv preprint arXiv:2303.16982
  • 23.Masters D, Dean J, Klaser K, Li Z, Maddrell-Mander S, Sanders A, et al (2022) Gps++: An optimised hybrid mpnn/transformer for molecular property prediction. arXiv preprint arXiv:2212.02229
  • 24.Liao YL, Smidt T (2022) Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. arXiv preprint arXiv:2206.11990
  • 25.Arjona-Medina J, Nugmanov R. Analysis of Atom-level pretraining with Quantum Mechanics (QM) data for Graph Neural Networks Molecular property models. https://arxiv.org/abs/2405.14837
  • 26.Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN et al (2017) Attention is all you need. Adv Neural Inform Process Syst 30:10 [Google Scholar]
  • 27.Mitchell TM (1980) The Need for Biases in Learning Generalizations. Laboratory for Computer Science Research, Rutgers University, Department of Computer Science
  • 28.He K, Zhang X, Ren S, Sun J (2016) Deep Residual Learning for Image Recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); p. 770–778
  • 29.Luong KD, Singh A (2024) Application of transformers in cheminformatics. J Chem Inform Model 10:12 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Müller L, Galkin M, Morris C, Rampášek L. Attending to Graph Transformers. https://arxiv.org/abs/2302.04181
  • 31.Heid E, Greenman KP, Chung Y, Li SC, Graff DE, Vermeire FH et al (2023) Chemprop: A machine learning package for chemical property prediction. J Chem Inform Model 64(1):9–17 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Hu W, Fey M, Zitnik M, Dong Y, Ren H, Liu B et al (2020) Open graph benchmark: datasets for machine learning on graphs. Adv Neural Inform Process Syst 33:22118–22133 [Google Scholar]
  • 33.Schütt K, Unke O, Gastegger M (2021) Equivariant message passing for the prediction of tensorial properties and molecular spectra. In: International Conference on Machine Learning. PMLR; p. 9377–9388
  • 34.Schütt K, Kindermans PJ, Sauceda Felix HE, Chmiela S, Tkatchenko A, Müller KR (2017) Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. Adv Neural Inform Process Syst 30:12 [Google Scholar]
  • 35.Nugmanov R, Dyubankova N, Gedich A, Wegner JK (2022) Bidirectional graphormer for reactivity understanding: neural network trained to reaction atom-to-atom mapping task. J Chem Inform Model 62(14):3307–3315 [DOI] [PubMed] [Google Scholar]
  • 36.Meyer B, Sawatlon B, Heinen S, Von Lilienfeld OA, Corminboeuf C (2018) Machine learning meets volcano plots: computational discovery of cross-coupling catalysts. Chem Sci 9(35):7069–7077 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Gensch T, dos Passos Gomes G, Friederich P, Peters E, Gaudin T, Pollice R et al (2022) A comprehensive discovery platform for organophosphorus ligands for catalysis. J Am Chem Soc 144(3):1205–1217 [DOI] [PubMed] [Google Scholar]
  • 38.Glendening ED, Landis CR, Weinhold F (2012) Natural bond orbital methods. Wiley Interdiscipl Rev Computat Mol Sci 2(1):1–42 [Google Scholar]
  • 39.Landrum G. RDKit: Open-source cheminformatics. http://www.rdkit.org/
  • 40.Guan Y, Coley CW, Wu H, Ranasinghe D, Heid E, Struble TJ et al (2021) Regio-selectivity prediction with a machine-learned reaction representation and on-the-fly quantum mechanical descriptors. Chem Sci 12(6):2198–2208 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Caruana R (1997) Multitask learning. Machine Learn 28:41–75 [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

No datasets were generated or analysed during the current study.


Articles from Journal of Cheminformatics are provided here courtesy of BMC

RESOURCES